Diffusion Twigs with Loop Guidance for Conditional Graph Generation
Authors: Giangiacomo Mercatali
Yogesh Verma
Andre Freitas
Vikas Garg
Venue: NeurIPS 2024 poster (2024)
Keywords: graph network
conditional generation
diffusion
generative models
molecule design
molecule optimization
Primary area: diffusion_based_models
TL;DR: We introduce a novel score-based  diffusion framework for conditional generation that  co-evolves multiple heterogeneous flows, and leverages a new strategy called loop guidance.

Abstract: We introduce a novel score-based diffusion framework named  Twigs that incorporates multiple co-evolving flows for enriching conditional generation tasks. Specifically, a central or trunk diffusion process is associated with a primary variable (e.g., graph structure), and additional offshoot or  stem processes are dedicated to dependent variables (e.g., graph properties or labels). A new strategy, which we call loop guidance, effectively orchestrates the flow of information between the trunk and the stem processes during sampling. This approach allows us to uncover intricate interactions and dependencies, and unlock new generative capabilities. We provide extensive experiments to demonstrate strong performance gains of the proposed method over contemporary baselines in the context of conditional graph generation, underscoring the potential of Twigs in challenging generative tasks such as inverse molecular design and molecular optimization. 
Code is available at https://github.com/Aalto-QuML/Diffusion_twigs.

Diffusion Twigs with Loop Guidance for Conditional Graph Generation
Giangiacomo Mercatali 1
Yogesh Verma Aalto University yogesh. wwr.a@aalto...ro..m
HES-SO Genève
University of Manchester
giangiacomo.
htwp:aacoooom.aaaaaaaaaaaaaaaaaa www.mmrrclaaaaaammamm. ihhsge..h
Andre Freitas
Vikas Garg Yai Ltd & Aalto University vgargdesail mit. edu
Idiap Research Institute University of Manchester NBC, CRUK Manchester Institute andre frrerttitaeeee treitas@idaapp.. ch
Abstract
We introduce novel score-based diffusion framework named Twigs that incor- porates multiple co-evolving flows for enriching conditional generation tasks. Specifically, central or trunk diffusion process is associated with primary vari- able (e.g.. graph structure), and additional offshoot or stem processes are dedicated dependent variables (e.g.. graph properties labels). A new strategy, which we call loop guidance, effectively orchestrates the flow of information between the trunk and the processes during sampling. This approach allows uncover intricate interactions and dependencies, and unlock new generative capabilities. We provide extensive experiments demonstrate strong performance gains of the proposed method over contemporary baselines in the context of conditional graph generation, underscoring the potential of Twigs in challenging generative tasks such as inverse molecular design and molecular optimization. Code is available at https://githur com Attp/Aalto--aaa QuML/Diffuue
Introduction
Conditional graph generation a fundamental problem scientific domains such as de novo drug design [21 43 74 and material design (39]. However, searching for new molecules with desired physicochemical properties poses significant challenges to traditional brute-force methods due to the vast combinatorial spaces (64). With the advent of neural networks [44], deep generative models
have emerged as powerful tool for learning informative conditional representations of molecules,
facilitating the development of in silico methods for chemical design [6
31
61 73).
Score-based diffusion generative models (SGMs) and denoising probabilistic diffusion models (DDPMs) 24 67 have recently emerged powerful techniques for training deep networks graph- structured data, with applications spanning molecular design [37 53 36 81), molecular docking (6) molecular dynamics simulations (78). protein folding (79]. and backbone modeling [70]. Notably, diffusion models exhibit superior capabilities for conditional graph generation, excelling in both
discrete 26 75 49 and continuous 3 28 45 settings. The training of the mentioned conditional
diffusion models is achieved by two types of diffusion guidance algorithms: classifier-bssed guidance
'quual Contribution. Order decided via coin flip. Work done while the University of Manchester
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Classifier guidance
Classifier free guidance
Twigs with loop guidance
Forward procass
M4,0
y/,,7
Turk Rooeess
Na.0
Forward SDE
Forward SDE
Forward SDE
Es
Forward SDE
{sh.
Stan Process
Revarse procees
Ve,T
M
V4.7
Trunk Precess
V4.0
Reverse SDE (Ae)
Reverse SUE
Reverse
Sten Prosees
3/C
Reverse SDE (sa.) h
ferote
9C
f+
G 0.5
(orvo 0.8
y/²
Overview of the proposed method (Twigs). We define two types of diffusion processes: (1) multiple Stem processes (, which unravel the interactions between graph structure and single properties, and (2) the Trunk process, which orchestrates the combination of the graph structure score from se with the stem process contributions from sø.- During the forward process, the structure y. and the properties {yr}* co-evolve toward noise. In each step of the reverse process, the structure is first denoised and subsequently used to denoise the properties (indicated by the green-dashed line). Such de-noised properties are then utilized, in tumn, to further denoise the structure (red line), in process that resembles guidance loop.
Figure 1:
81. which involves training separate property predictor model alongside the diffusion model; and classifier-free guidance [23], which integrates scores from both unconditional and conditional diffusion models. While these guidance techniques have been found to be effective, the algorithm design is not tailored to encompass the intricate hierarchical or multi-resolution elements inherent in conditional generation. Consequently, it is plausible that this inadequacy may contribute suboptimal representations, particularly notable in tasks such as conditional graph generation. The recent success of hierarchical diffusion flows in various domains, such as modeling interactions between node and edge features (37. multi-resolution modeling 225, decision-makinn [47], and conditional image generation
underscores the need to integrate hierarchical information beyond the capabilities of classifier based and classifier-free guidance.
We assert that conditional diffusion models for structured spaces, such as graphs, could be enhanced with hierarchical conditional processes. Specifically, rather than treating heterogeneous structural and label information uniformly within the hierarchy, we advocate for the co-evolution of multiple processes with distinct roles (asymmetric). These roles encompass a primary process governing the structural evolution alongside multiple secondary processes responsible for driving conditional content. We aim to propose an alternative to existing conditional graph diffusion techniques (outlined in Table D by bestowing the models with finer control over two key aspects: 1) the evolution of
structural graph components, including nodes and edges, and 2) the co-adaptation of the graph structure in conjunction with one or more associated properties.
Towards this objective, we present a novel diffusion framework for conditional generation named Twigs, drawing analogies from the trunk and offshoots of a tree. Concretely, we establish a central trunk process governing primary variable, which interacts with several stem processes, each associ- ated with a secondary variable. contrast with classifier-free and classifier-based methodologies, a novel conditional mechanism, termed loop guidance, orchestrates information exchange between the trunk and the stem processes (refer to Figure J. Our methodology facilitates the acquisition of flexible representations, capitalizing on the disentanglement of intricate interactions and dependen- cies. We formalize our framework by drawing upon the theory of denoising score matching [67] and leveraging tools derived from stochastic differential equations (SDEs) [I]. The effectiveness of Twigs substantiated through compelling empirical validation across various conventional constrained generation tasks, utilizing both molecular and generic graph datasets.
1.1 Contributions
In summary, this paper makes the following key contributions:

---

Table 1: Comparison of related methodologies. Twigs is the first method that enables seamless
orchestration of multiple asymmetric property-oriented hierarchical diffusion processes via SDEs.
Method
Conditional
Asymmetric
Multiple flows
Continuous (SDEs)
GDSS
37
EEGSDE
MOOD 45
JODO
28
X
EDGE
X
GruphMaker 46
Nisonoff al.
X
X
Gruver
Klarner et al.
40
X
Twigs (ours)
(Conceptual and methodological) The introduction of a new score-based, end-to-end trainable, con-autorerresive generative model Twigs designed for acquiring conditional representations. Our approach enables precise guidance of multiple property copeert-coondttooee diffusion processes.
(Technical) We present robust mathematical framework, including novel strategy called loop guidance, that employs tools from Stochastic Differential Equations (SDEs) to derive both the forward diffusion process and its corresponding reverse SDE for conditional generation. This framework is designed seamlessly integrate additional contexts conditioning information.
(Empirical) We showcase the versatility of the proposed diffusion mechanism (Twigs) through extensive empirical evidence across various challenging conditional graph generation tasks, consis- tently surpassing contemporary baselines.
Related works
In Table I we provide an overview of the similarities and differences between Twigs and related methods. refer the reader to Appendix for additional related work.
Diffusion guidance is typically applied to regulate the diffusion process for conditional generation. Previous approaches that perform class-conditional generation divided into classifier-based (8) and classificr-fres guidance (23). While some works model diffusion with multiple flows (5) 37 46)
they treat nodes and edges symmetric way; i.c., they associate multiple flows for nodes and edges that have equivalent contributions (in other words, these flows have the same roles). We instead abstract graph properties as secondary processes that branch from, and interact with, the main process that pertains to the graph structure. In addition, while other guidance methods are related [8 40 52]
they do not leverage multiple diffusion flows. To our knowledge, the proposed method the first to incorporate multiple diffusion flows in a hierarchical fashion for conditional generation. We formalize Table - how Twigs differs, mathematically. from classifier-fre and classifier-based methods.
Conditional Diffusion for Graphs Recent advancements in generative modeling have prominently featured score-based techniques (SGM), utilizing diffusion stochastic differential equations (SDEs) [3 [5] [8] 26 40 45 46 52 72 75 82
32 35 37 48), including for graph generation [
Guidance methods have been adopted in conditional molecule generation settings. The works
28 Xu et al. [22] are classifier free approaches, while Bao al. B. Vignac et 755, Lee et al. focus on classifier-basse methods. Diverging from these approaches, we explicitly model the dynamic interaction between primary variables (e.g.. graph structure) and dependent variables (c.g., graph properties) using dedicated diffusion processes to achieve more expressive representations and improve performance for conditional generation.
from Hoogeboom et al. [26.. Huang et al. 22
3 Diffusion Twigs
Method overview We extend score-based techniques 667 for training conditional diffusion models over graphs. Differently from current guidance methods, as summarised in Table 2 we leverage finer control over the structure and graph properties diffuse multiple hierarchical processes, toward achieving more robust representation. Our method, Twigs, defines trunk process over the primary

---

Table 2: Twigs comparison Classifier-based [ and Classifier free [2 guidance, applied for conditional generation in Diffusion models. Here y. represents the graph structure, {yr represent the k- p-ropeerties of graph. The fo function is the classifier, co and $0.6 are learnable score models.
Method
Diffusion Scheme
Appracc
Class 50e hised
(dd glOdw
Vy., kopp (1iha) y... lopp Vy.. lopil Vad) (Pace) Vy, log (yyl (rad)
dy. U - of Vy. logzeldees (rrhallett goow
flv.a. fide g(0dw
P.., ((r [Waee) V.,,, logg [xx]a) Vy,, logplrad (ro (Kra (x9)aa
ay. Tiy. - Ve,,, logpetyr.. (D-)o)fitt gool
Deige
dy. flyaa 4 glodw, [dy.l fi fjdt glw
F. lo (oa (Marle) Ve logowlora)) 2n Vy. bi yud
Fiy. of Ve.. loggelseon (orahalldd godw
(dyi)e Mar yy log yulldt gudw
y.., koppvilaee true (Maeee) (g.00 + 2 Aa.00
variable (graph structure) ys. and stem process each dependent variable y, R (e.g., graph property). We achieve the desired flexibility with a variable y. that encompasses both node features and the adjacency matrix well the coordinates. The details of the dimensions of y, given in Section B. for the 3D case, and in Section B.2 for the 2D case.
Forward process We define multiple forward processes within hierarchy that co-evolves data and properties into noise. The trunk forward process for the graph structure ys is defined as
dy = fy tjdt + (t)dw
where f, and are corresponding diffusion and drift functions, and dw is the Wiener noise. The stem forward process over the dependent variables y ........y' is defined
(t)
Fp.y (p)
dy(t)= =
(2)
(t)
E.Yr, gp (t)
Here, fp and Op denote the diffusion and drift functions, respectively, for the i stem processes. Collectively, along with the trunk forward process, they constitute Twigs. These operations introduce random Gaussian noise, iteratively, to the data toward a prior (typically Gaussian) distribution.
Reverse Process The Twigs reverse process starts from the prior distribution (Gaussian noise) towards the data distribution. A key difference with Song et [67] is that here our variable yt comprises both structure and properties, leading to the following modification of the overall diffusion process:
dy: (f(,,,) o. Vy, log Pog)) g.dw
where Y: (ys,,, (yr.)
(3)
We derive Equation 3 in Section A.1 The joint distribution over the trunk and stem processes is assumed to factorize as
(r s,to y1,4 Sa.--yaa Vk,.) pr I pe(yi, Ysaa)
In turn, the score function simplifies in Equation
5 leading to the decomposition in Equation (6)
Vy,
yk.)) Vy. logpe(y..))
E Vy, logpe yya))
(5)
dy. tf(y o y log (y..)) E Vy, Iy ly,,))]dt + g.
(6)
Conditional modeling We expand our proposed approach to enable conditional generation with an extemnl context = {y. E C}. where C' S (1..... .......,,, k}. The context can be represented as a scalar or vector, describing a particular value associated with a data-dependent variable. For example, in case of molecules, it could represent one or more of the k properties such as the Synthetic Accessibility (SA) score or the Quantitative Estimate Drug likeness (QED). This extension modifies the joint distribution for the score function in Equation

---

Reverse SDE under conditioning context
The reverse SDE for y; (ys,t, (yr,ek) give an external conditioning context yc is shown below (details in Appendix A.2).
- () of Vy. logpe(y + g.dw
(7)
We resort the following factorization of the distribution, conditioeed on the context yc:
(y {yi,e b:: = II, ( (x..s.)))((......C)))
As result, the factorization of the score function Vy, log (Ys,t, (yi, k.. amounts to
Vy, logp(yy... e Vy, log y.,) C Σ 8 lo (ys. Ve,t.
(8)
The above-factorice score function parameterizes our reverse diffusion process, thus offering a novel approach integrate external contextual information into conditional generation.
Training We propose to train Twigs by incorporating the factorization from Equation ( within a score-matching objective function (30 67.. Algorithm shows the training procedure to learn two types of time-dependent score-based models: 80.2, which approximates the trunk variable, and 8 which approximates the coupling between the stem variable and the trunk variable. The objective function for optimizing the score networks 8a, 86., is given as follows:
ming,0, E. (Ay, Ey) Ey.y. Se), Et (Vi,
Vy, yc)) (9)
where Ey. = Ey,,o.r.. and Ey, = Vi,,- is worth noting that the influence introduced by the variable $e. provides the directions for the diffusion model to converge into distributions with the desired properties. Such property-orient information provided by 8a, resulting in novel form of guidance that orchestrated by a branching diffusion process, named Loop guidance.
knowledge operates in conjunction with the structural
Algorithm Training Twigs
Algorithm 2 Generating with Twigs
Input: Dataset D, iterations Witers batch size B, number batches B, K properties to consider Initialize parameters s0.1, (s ] for Score Networks
Input: models 80,1, fs K Time step schedule {1) Langevin MCMC step size a, External context yo
K
for k = 1, .,... Riter do
for I,.... n1 do
Yor (3iT) 1)
t 4(0
for t = T. T...., ......0 do
Db = ((ya.) (K
K B yc}
K $8,1 +- s0,1 (ys: {Br, 1i yc)
Ls + Eq. 9
{s 1K
[i=1
{s Ys. yi.es Uc) vc)].K [=
end for
Ye. f- Ys, + 2 S0. Vazii 24 ~ (0,1
0, (dx) K optim 1 Σn Lb)
Yi, < Yi. + 2 s + Vaz; 2, ~ ((0,
ng
end for
end for
Sampling Given trained conditional Twigs model, our generative process begins by sampling an external context or conditioning value yo. which can also supplied externally. We then simulate the reverse diffusion process, similar to the one described in Equation 8 but with modified score function to generate the data. The proposed algorithm for generating new data samples with Twigs given in Algorithm [n involves loop of updates between processes: the stem score network sa, evolves the property y,. integrating information from the structure ye. and subsequently, the updated property information from $s, is integrated into the main process by the score network sg.
4 Experiments
We conduct a set of comprehensive experiments to demonstrate that Twigs improves over con- temporary conditional generation methods. Benchmarks include: molecule generation conditioned over single (8 4.T), D), and multiple ($ (.2)) properties on QM9, as well as molecule optimization on ZINC250K and network-graph generation
gennraatonnnnnn conditioned on desired properties ($ 4.4
5

---

Figure 2: First row: Samples by Twigs for 3D molecules conditioned single properties QM9.
Second row: KDE and KL divergence results between target and predicted properties.
Twigs JODO Data
FL3-00-04
KL-S de
BL9.7e-04
KL:3-46 02
KL-7 20
Cv
Table 3: MAEL results on single target quantum property for the QM9 dataset.
Method
C.
a
Ac
CHOMO
ELUMO
EDM
1.065 (= 0.010)
1.123
( 0.013)
2.78
(1: 0.04)
():5)
(d: 2)
(d: 7)
GM
1,025 na)
1.108
na)
2.37
(+
587 na)
340 (: ma)
522 (:: na)
EEGSDE
0.941 (= 0.005)
0.777
[ 0.007)
2.50
(1:: 0.02)
487 (1.3)
(1 2)
530 (1
EquiFM
1,033 na)
1.106
na)
2.41
591 (a)
337
ma)
TEDMol
0.847 (= na)
0.840
= na)
2.24
( na)
443 (a)
279 ( ma)
412 ( na)
JODO
0.58 0.001)
0.628
0.008)
T.42
335
226
256
Twigs
0.559
0.627
0.001)
1.36
(: 0.01)
323
225
244 (1.
Table 4: Novelty, atom & molecule stability for QM9 single property.
Noveltyt
Atom Stability?
Mol Stability?
Noveltyt
Atom Stability?
Mol Stability
C
EDM
83.64.1 0.30)
98.25
0.02)
80.82 0.321
83.93 0.11)
98. 18.17:a 0.84)
80.25 0.40)
EEGSDE
83.53 0.18)
98.25
0.06)
80.83
83.85
98. 18.11::. +
80.25
TEDMol
83.82
98,27
80.83
84,88
98.22
$0.31 mz)
JODO
91.21(4..22 0.22)
97.74 0.299
91.75 0.111
91.22 0.02)
99.02 :2 0.02)
92.86 0.15)
Twigs
93.16 +: 0.166)
99.14
0.04)
92.72 = 0.07
92.90 0.08)
99.25 0.05)
93.91 0.03)
ÉHOMO
83.93 0.45)
98.
0.04)
81.95 0:271
84.35 c. 0.51)
98. 17.1 0.07)
79.6 0.32)
EEGSDE
84.09
98.18
80.99
84.44
98.
79.8
TEDMol
84,92 m))
98. 19th
79.82: na)
84,58 (5
98.22: ma)
80.97
JODO
91.02 0.119)
98 42,42 0.0))
93.32 0.00
91.38 d 0.02)
98. 19 0.55)
92.02; 0.03)
Twigs
92.70 0.00
99.31 0.01
94.12 =: 0.11)
93.02 0.21)
99.26 0.04)
94,11
ELUMO
98. k3 0.04)
79.33 0:300
84.62/11 0.28)
26 0.04)
81.34
EEGSDE
84.19
98.26 0.03)
80.95
84.83
98. 14 0.0))
80.00
TEDMol
85.82 d:
98.42
82.03
84.90
98.31
81.40
JODO
90.15 0.02)
78 0.50
94.03 = 0:31
90.78 (8 0.42)
98. 84:4 0.0)
94.02 0.3)
Twigs
92.88 0.13)
0.13)
94.12 0.00)
92.48
99.29 0.17)
94.11
4.1 Single Quantum properties QM9
Setup. We evaluate the effectiveness of Twigs for generating molecules with a single desired quantum property, sourced from the QM9 dataset (58). specifically, we consider C., 14, Ae, ÉLUMO and еномо- To ensure consistency and comparability with the baselines, which include JODO 228), EDM (26). EEGSDE B3, GeoLDM (82), TEDMol [49), EquiFM (68), we adhere to the identical dataset preprocessing, training/test data partitions, and evaluation metrics outlined by Huang et al. 28 Regarding parameterization of Twigs, we follow the attention architecture defined in Section I.I with single stem process.
Results. In T we report the Mean Absolute Error (MAE) results, and in Table the Novelty, Atom stability and Molecule stability. Our method outperforms the evaluated baselines across the specified properties. In Figure 2 the bottom row provides Kernel Density Estimation (KDE) visualization which shows that Twigs achieves a more accurate distribution for the property values when compared with JODO, while the top row shows some 3D molecule samples by model.

---

Figure 3: Samples of coltiple-properyy conditional molecules by Twigs (C, and (4) for QM9.
Table 5: MAE (4) for conditional generation on QM9 with multiple properties.
C.
Ac
EDM
1.097ct 0.007)
1.156 0.011)
683
1.130 0.007)
2.76 0.01)
1.158 0.002)
EEGSDE
0.981 0.008)
0.912 0.006)
563
0.866 0.003)
2.61 0.01)
0.855 0.007)
TEDMol
0.645 n/a)
0.836 n/a)
c8 n/a)
0.843 nla)
2.27 (z n/a
0.809 m/)
JODO
0.634 0.002)
0.716 0.006)
350 4)
0.752 (:: 0.006)
1.52 d: 0.01)
0.717 c:: 0.006)
Twigs
0.602 0.001)
0.708 0.002)
343 + 2)
0.740 c:: 0.003)
1.46
0.712 = 0.002)
4.2 Multiple Quantum properties on QM9
Setup. This experiment evaluates the capability to combine multiple desired properties in the generated molecule. Specifically we follow Huang et al. [29] and consider all possible combinations of properties involving μu: (C', Jt). (Ac, p)). (a, p)) Since we model two properties, we test our Twigs with two stem networks within the attention architecture described in Section B.] We benchmark
against several contemporary baselines, including EDM 226], EEGSDE and JODO [28].
Results. In Table we present the Mean Absolute Error (MAE) results obtained from the property predictors introduced by Huang et al. [28 for the various property pairs under consideration. The superior performance of Twigs across all baselines reinforces the findings from the single property experiment (Section 4.1). emphasizing the benefits of learning multiple hierarchical stem processes.
OED 0.84
SA.0.62
QED 0.69 SA 0.90
QED 0.60
0.00
Q.58
SA 0.50
DED 0.70
SA 0.50
QED 0.53
SA 0.74
D5 -5.6
05 -5.9
DS -1.
Ds -99
DS-
DS -15
OED 0.66
SA 0.39
QED 0:30
SA 0.53
OED 0.68 5A.0.67
OED 0.75
SA066
0
SA 0.54
OED 0.64
SA 0.45
05-7.3
DS -7.5
D6 -7.3
DS -33
D5-
DS 8.88
Figure 4: Molecules generated by Twigs from ZINC250K conditioned on fa7 (top), parpl (bottom).
4.3 Molecule optimization ZINC250K
Setup. The goal is to generate molecules from the ZINC250K dataset that exhibit optimal binding affinity, drug-likeness, and synthesizability for the following five target proteins: parpl, fa7, Shulb, braf. jak2. We adhere to the evaluation protocol established by Lee et al. [45]. which involves generating 3000 molecules and assessing them using two metrics that constrain the desired properties, including docking score (DS), likgelikeness (QED), and synthetic accessibility (SA).
The first metric, Novel hit ratio (%), represents the fraction of unique hit molecules that have maximum Tanimoto similarity of less than 0.4 with the training molecules. Hit molecules are defined as those meeting the criteria: DS < (the median DS of the known active molecules), QED 0.5, and SA < The second metric, Novel top 5% docking score, is the average DS of the top 5% unique molecules that satisfy QED 0.5 and SA 5, with a maximum similarity of less than 0.4 to the training molecules.

---

reinforcement learning (RL) model that utilizes prior sequence model, MORLD 33 RL model that uses QED and SA scores as intermediate rewards and docking scores as final rewards, HierVAE [3 VAF-based model that utilizes hierarchical
Baselines. We consider REINVENT 155
molecular representation and active learning. GDSS 37 a score-based diffusion model that evolves nodes and edge information with a system of SDEs, MOOD 45] score-based diffusion model based on GDSS that trains an additional property predictor to improve conditional generation. For MOOD we consider the version without the out-of-distributtin (OOD) control, to have a fair comparison with our method. For Twigs we follow the GCN-based architecture described in Section B.2 with multiple stem processes (one for each target protein).
Results. In Table O we report the results for top 5% docking scores. We observe that Twigs achieves the highest score across all properties, excluding braf, where it achieves the second-bess score after MOOD, In Table 7 we report the results for Novel hit ratio. The outcomes confirm that our model improving the performance substantially over all the considered properties, except for braf, on which Twigs is the second-bert performing model after MOOD. In Figure 4 we provide some samples of the molecules obtained by Twigs with the respective QED, SA, and docking score. Additionally, in Table 13 we report the MAE values for generating molecules with a desired target protein property, we compare the inference cost of Twigs against MOOD,
and in Table 14
Table 6: Novel top 5% docking score ZINC250K. Best is boldfaced, second-best is in gray
Model
parpl
fa7
5hrlb
braf
jak2
REINVENT
8,702 0.523)
7.205 0.264)
8.770 0.316)
8.392 0.400)
8,165 0.277)
MORLD
7.532 ce 0.260)
6.263 0.165)
7,869 (:: 0.650)
8.040 0.337)
7.816 c: 0.133)
ViE
9.487 0.278)
6.812 0.274)
8.08 ( 0.252)
8.978
0.525)
8.285 (L:: 0.370)
GDSS
9.967 0.028)
7.775 cE 0.039)
9.459 =: 0.101)
9.224
d: 0.068)
8.926 0.089)
MOOD
10.409 de 0.030)
7.947 ct: 0.034)
10.487 c:: 0.069)
10.421
0.050)
9.575 0.075)
Twigs
10.449 =: 0.009)
8.182 + 0.012)
10.542 :: 0.025)
10.343 ce 0.024)
9.678 0.032)
Table Novel hit ratio (1) results on ZINC250K.
Model
parpl
fa7
Shilb
braf
jak2
REINVENT
0.480 0.344)
0.213 0.081)
2.453 0.561)
0.127ct 0.088)
0..13 0.167)
MORLD
0.047 0.050)
0.007 0.013)
0,880 c: 0.735)
0.047 0.040)
0,227 0.118)
VierVAE
0.553 0.214)
0.007
0.507 0,278)
0.207 0.220)
0.227
GDSS
1.933 0.208)
0.368
0.103)
4.667 ct: 0.306)
0.167 c:: 0.134)
1.167 0.281)
MOOD
3.400) 4 0.1
0.433 0.063)
11.873 0.522)
2.207 <:: O. 0.165)
3.953
Twigs
3.733 :: 0.081)
0.900 0.012)
16.366 06366.:000000 0.029)
1.933 0.023)
5.100:10.13)
4.4
Generation of Network graphs with desired properties
Setup. We follow the data processing delineated by Jo et al. 337 and provide results for the Community-small [60] and Enzymes datasets [62]. To test the capabilities to generate conditional graphs, we extract four properties via the NetworkX library (20), including density, clustering, assortativity, and transitivity. Considering graph G with n nodes and m edges, we have: (1) Density:
n n(=-1) II. (2) Clustering coefficient: the average C 1 Evec C.. (3) Assortativity; measures the
2m
similarity of connections in the graph with respect to the node degree. (4) Transitivity: the fraction of all possible triangles present in G. Possible triangles are identified by the number of "triads" (two
edges with shared vertex). The transitivity is T 3 #triangles
#triads
Baselines. In terms of baselines, we first consider two versions of MOOD 45 (two OOD coefficients). and we train the property predictors using the codes from the authors. Our second baseline is GDSS (77,, which we modify to be equipped with a classifier-free guidance scheme. We also consider the version of GDSS based on transformers, which leverages the graph--mlti-head attention [2]. Finally, we consider Digress 75). which is classifier- guidance diffusion model based on attention mechanisms. We parameterize our Twigs model with our GCN architecture described in Section B.2 with single stem process.
8

---

Enzymes Community
Density
Clustering
Assortativity
Transitivity
Community
KL-0 003
KL-0.00C KL:0.030
KL-a
KL-B 808
KL.0 005
KL:5.500 2,500
KL.0 023
0.25
0.30
0.55
0.40
02
0.4
0.8
-0.
0.0
0,8
Enzymes
KL-0 045
KL.0 061
KL.1-43
KL.0.060
Twigs
MOOD
Data
Figure 5: Visualization of Community-small and Enzymes datasets. First and second samples generated by Twigs. Third and fourth rows: KDE plots and corresponding KL divergence values.
Table 8: MAE (1) values on Community-smal and Enzymes, conditioned on single properties.
Community Small
Enzymes
Model
Density
Clustering
Assortativity
Transitivity
Density
Clustering
Assortativity
Transitivity
GDSS
2.95
12.1
8.04
1.98
2.55
GDSS-T
2.30
11.5
19.2
10.1
9.25
3.27
2.03
2.68
Digress
2.34
10.6
17.8
9.42
8.04
2.39
1.95
2.55
MOOD-1
2.35
11.1
18.8
10.5
7.94
2.34
1.83
2.12
MOOD-4
2.12
11.3
16.7
8.76
7.98
2.44
1.99
2.43
Twigs
2.07
9.67
15.2
8.35
7.35
2.23
1.72
2.03
Results. Table 8 reports the MAE average of three runs, demonstrating that Twigs consistently outperforms the considered baselines on all cases across the two datasets. MOOD is the second-best performing model in the majority of the cases. We further strengthen the MAE results by providing in Figure (bottom) the KDE plots of the property distributions of the graph generated by Twigs and MOOD. The Figure demonstrates that Twigs can achieve a higher fidelity the data, which is also confirmed by the lower KL divergence values. Figure (top) depicts some random graph samples generated by Twigs.
4.5 Ablation study on multiple properties
Setup. Assuming conditional independence among the properties ax бномо- ÉLUMO. Ae, μe, and Cp given the molecular graph can simplify the modeling process. This assumption leverages the fact that the molecular graph captures the essential structural dependencies, allowing us to treat the properties independent for computational efficiency and case of interpretation, even if slight interdependencies exist.
Results. Here we show that such modeling assumption can work practically. Table reppors the MAE on molecular graphs for QM9 on three properties, showing that our method consistently achieves lower eтror on all the properties. Table I0 shows that on generic graphs Twigs can achieve lower MAE on all the considered cases, the cases of two and three properties.
Table MAE values over three properties for QM9,
Model
A
JODO
2.749 (# 0.03)
1.162 (+: 0.04)
717 (+:
Twigs 2.544 0.05)
1.094 (#: 0.02)
640 (::

---

Table 10: MAE results for two and three properties on community small.
Model
Pairl
Pair2
Triplet Clustering
Density
Clustering
Density
Assortativity
Density
Assortativity
GDSS
2.95
13.3
2.61
19.8
2.97
12.5
19.4
Digress
2.82
12.1
2.52
18.1
2.65
11.2
18.2
MOOD
2.43
12.0
2.40
17.2
2.53
11.4
17.3
Twigs
2.34
11.0
2.39
16.7
2.27
10.6
16.1
4.6 Training time
In Table 11 we study the impact of multiple diffusion flows on the community-small and Enzymes datasets. Specifically, we report the average time for the overall training for Twigs with one and three secondary diffusion flows. We observe that our models encounter small overhead compared to GDSS and Digress, however, we believe it is good tradeoff because it achieves a lower MAE.
Table 11:
Overall training time for 5,000 epochs (hours and minutes) for Twigs with different secondary diffusion flows, GDSS, and Digress on the Community-smal and Enzymes datasets.
Dataset
Twigs p
Twiga p 33
GDSS
Digress
Community Enzymes
Oh 22m
Oh 24m
Oh 19m
Oh 20m
6h 45m
6h 59m
6h 42m
6h 43m
Conclusion, Broader Implications, and Limitations
We introduced novel approach model conditional information within generative models tailored for graph data. Twigs incorporates the novel mechanism of loop guidance to control the overall generative process by first bifurcating the diffusion flow into multiple stem processes and then re-integrating them into the trunk process, resembling a loop. Our experimental results showcase the performance gains of Twigs when compared current state-of-the-as baselines across various conditional graph generation tasks.
of the most exciting avenues within machine learning Conditional generation is fast emerging classifier- and classifier- free schemes, making our and would benefit from techniques beyond method applicable settings beyond this work. Indeed, while the current work has focused on graph settings, Twigs might find use in other domains (e.g., image, text, and audio). However, whether Twigs is effective such settings needs be investigated in future works.
Training multiple properties (stem processes) might require training additional parameters, incurring additional computation and training time. Our ablation study training time due to multiple processes (Section (4.6) suggests that Twigs could provide a good tradeoff (lower MAE compared to some prominent existing methods at the expense of small additional computational overhead).
Finally, assuming factorization of the distribution over stem processes conditioned on the trunk process might not always be realistic. Our experiments in Section 4.5 suggest that Twigs might still be able to achieve strong performance when considering multiple properties. case some prior knowledge is available about some properties that violate this assumption, we could, in principle, adapt Twigs by grouping them into single stem process while factorizing with the remaining ones.
Acknowledgments
YV and VG acknowledge support from the Research Council of Finland for the "Human-steered next-geneee -ext-eenerttion machine learning for reviving drug design" project (grant decision 342077). VG also acknowledges Jane and Aatos Erkko Foundation (grant 7001703) for "Biodesign: Use of artificial intelligence in enzyme design for synthetic biology". GM acknowledges support from the Engineering and Physical Sciences Research Council (EPSRC) and the BBC under iCASE. AF is partially funded by the CRUK National Biomarker Centre, by the Manchester Experimental Cancer Medicine Centre and the NIHR Manchester Biomedical Research Centre.
10

---

References
[1] Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and
their Applications. 12(3):113332, 1982. ISSN 0304-4149. doi: https://doi.orr/100..00
08044-4149082)900551-5. URL https: / //wp:///www..... ccckennedirect..com//ceece.....e
p11/0304414982900515
[2] Jinheon Baek, Minki Kang, and Sung Ju Hwang. Accurate learning of graph representations
with graph multiset pooling. In International Conference on Learning Representations, 2021.
URL https://opeerreveew http:p///ppppaeeeeeeeeeee/o/ooua wttp://oooooooidddioioiooooooo
[3] Fan Bao, Min Zhao, Zhongkai Hao, Peiyao Li, Chongxuan Li, and Jun Zhu. Equivariant energy-guided SDE for inverse molecular design. In The Eleventh International Conference on Learning Representations, 2023. URL URL
[4] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. arXiv preprint 1 13, 2023.
[5] Xiaohui Chen, Jiaxing He, Xu Han, and Liping Liu. Efficient and degree- generation via discrete diffusion modeling. In International Conference on Machine Learning, pages 4585-4610. PMLR, 2023.
degree-guided guided graph
[6] Gabriele Corso, Hannes Stärk, Bowen Jing, Regina Barzilay, and Tommi Jaakkola. Diffdock: Diffusion steps, twists, and turns for molecular docking. arXiv preprint arXiv:2210.01776, 2022.
[7] Alex O Davies, Nirav S Ajmeri, al. Hierarchical gnns for large graph generation. arXiv preprint arXiv:2306. 1412, 2023.
[8] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021.
[9] Anh-Dung Dinh, Daochang Liu, and Chang Xu. Rethinking conditional diffusion sampling with progressive guidance. Thinty-seventh Conference on Neural Information Processing Systems, 2023. URL /tLpattt:://opeeeeviir http:///ogpaaaeeeeoooooooo//oaa/ net./forumiiddgghhaiinegg
[10] Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Amaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and meme. In International Conference on Machine Learning, pages 1889-8510. PMLR, 2023.
[11] Alexandru Dumitrescu, Dani Korpela, Markus Heinonen, Yogesh Verma, Valerii Iakovlev, Vikas Garg, and Harri Lähdesmäki. Field-based molecule generation. arXiv preprint 20Xiv:2402.15864, 2024.
[12] Peter Eckmann, Kunyang Sun, Bo Zhao, Mudong Feng, Michael K Gilson, and Rose Yu. Limo: Latent inceptionism for targeted molecule generation. arXiv preprint arXiv:2206.0001 2022.
[13] Niklas Gebauer, Michael Gastegger, and Kristof Schütt.. cemmettr-aadapter generation of 3d point sets for the targeted discovery of molecules. Advances in neural information processing systems, 32, 2019.
[14] Niklas WA Gebauer, Michael Gastegger, Stefaan SP Hessmann, Klaus-Robert Müller, and Kristof T Schütt. Inverse design of 3d molecular structures with conditional generative neural networks. Nature communications, 13(1 13(1),973, 2022.
[15] Zijie Geng, Shufang Xie, Yingce Xia, Lijun Wu, Tao Qin, Jie Wang, Yongdong Zhang, Feng Wu, and Yie-Yan Liu. De novo molecular generation via connection-awarc motif mining. In International Conference on Learning Representations, 2023. URL htttp:://ppereviii net/forwarid-Q_JeelBB-
[16] Rafael
Jennifer Wei, David Duvenaud, José Miguel Hernández-Looatto, Benjamín Sánchez-LLenellnng Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D Hirzel, Ryan Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules. ACS central science, 4(2):268-22) 2018.
11

---

[17]
Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as Alug-and-play priors, Advances in Neural Information Processing Systems, 35:14715-14728, 2022.
[18] Nate Gruver, Samuel Stanton, Nathan Frey, Tim GJ Rudner, Isidro Hotzel, Julien Lafrance- Vanasse, Arvind Rajpal, Kyunghyun Cho, and Andrew G Wilson. Protein design with guided discrete diffusion. Advances in neural information processing systems, 36, 2023.
[19] Florentin Guth, Simon Coste, Valentin De Bortoli, and Stephane Mallat. Wavelet score-based generative modeling, 2022.
[20] Arie Hagberg, Pieter Swart, and Daniel S Chult. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Lab.(L.NN.)), Los Alamos, NM (United States), 2008
[21] Philip J Hajduk and Jonathan Greer. A decade of fragment-based drug design: strategic advances and lessons learned. Nature reviews Drug discovery, 6(3):211-21) 1-219, 2007.
[22] Yutong He, Naoki Murata, Chich-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, Zico Kolter, Ruslan Salakhutdinov, et al. Manifold preserv- ing guided diffusion. arXiv preprint arXiv:231 166424, 2023.
[23] Jonathan Ho and Tim Salimans. Classifier free diffusion guidance, 2022.
[24] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arxiv:2006. 11399, 2020.
[25] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, 23(1):2249-22) 2281, 2022.
[26] Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International conference on machine learning, pages 8867-8887. PMLR, 2022.
[27] Han Huang, Leilei Sun, Bowen Du, Yanjie Fu, and Weifeng Lv. Graphgdp: Generative diffusion processes for permutation invariant graph generation. In 2022 IEEE International Conference on Data Mining (ICDM), pages 201-210. IEEE, 2022.
[28] Han Huang, Leilei Sun, Bowen Du, and Weifeng Lv. Learning joint 2d & 3d diffusion models for complete molecule generation. arXiv preprint arXiv:2305. 12347, 2023.
[29] Yinan Huang, Xingang Peng, Jianzhu Ma, and Muhan Zhang. 3dlinker: (3) equivariant variational autoencoder for molecular linker design. arXiv preprint 2xXv:2205.00300 2022.
[30] Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
[31] John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph-based protein design. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL http:atttps://pooeeeeiiii http::////ppoceediinnnnneeeepepp cc/ppapr_ffilss pdf
[32] John Ingraham, Max Baranov, Zak Costello, Vincent Frappier, Ahmed Ismail, Shan Tie, Wujie Wang, Vincent Xue, Fritz Obermeyer, Andrew Beam, et al. Illmiiatiin protein space with a programmable generative model. BioRxiv, pages 2022-12 12, 2022.
[33] Woosung Jeon and Dongsup Kim. Autonomous molecule generation using reinforcement learning and docking to develop potential novel inhibitors. Scientific reports, 10(1):22104, 2020.
[34] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hierarchical generation of molecular graphs using structural motifs. In International conference on machine learning, pages 4839- 4848, PMLR, 2020.
12

---

[35] Bowen Jing, Gabriele Corso, Renato Berlinghieri, and Tommi Jaakkola. Subspace diffusion generative models. In Lecture Notes in Computer Science. pages 274-289. Springer Nature Switzerland, 2022. doi:
2-7. URL https::/doi..om http///bb...og11110070///./1 1007%
21978-3--03--0055---11
[36] Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi Jaakkola. Torsional diffusion for molecular conformer generation. Advances in Neural Information Processing Systems, 35:24240-24253, 2022.
[37] Jachyeong Jo, Seul Lee, and Sung Ju Hwang. Score-based generative modeling of graphs via the system of stochastic differential equations. In International Conference on Machine Learning, pages 10362-10383. PMLR, 2022.
[38] Jachyeong Jo, Dongki Kim, and Sung Ju Hwang. Graph generation with destination-driven diffusion mixture. arXiv preprint arXiv:2302.03556 2023.
[39] Kisuk Kang, Ying Shirley Meng, Julien Breger, Clare P Grey, and Gerbrand Ceder. Electrodes with high power and high capacity for rechargeable lithium batteries. Science, 31 31166))3): 977-980, 2006.
[40] Leo Klarner, Tim GJ Rudner, Garrett Morris, Charlotte M Deane, and Yee Whye Teh. Context-guided diffusion for out-of-distribution molecular and protein design. arXiv preprint 0rXiv22407.19442 2024.
[41] Lingkai Kong, Jiaming Cui. Haotian Sun, Yuchen Zhuang, B Aditya Prakash, and Chao Zhang Autoregressive diffusion model for graph generation. In International Conference on Machine Learning, pages 17391-17440 17408. PMLR, 2023.
[42] Xiangzhe Kong, Wenbing Huang, Zhixing Tan, and Yang Liu. Molecule generation by principal subgraph mining and assembling. Advances in Neural Information Processing Systems, 35: 2560-2563, 2022.
[43] Najwa Laabid, Severi Rissanen, Markus Heinonen, Arno Solin, and Vikas Garg. Alignment key for applying diffusion models to retrosynthesis. arXiv preprint arXiv:2405. 2024.
[44] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436-444 2015.
[45] Seul Lee, Jachyeong Jo, and Sung Ju Hwang. Exploring chemical space with score-based out-of-distribution generation. Proceedings of the 40th International Conference on Machine Learning, 2023.
[46] Mufei Li, Eleonora Kreačié, Vamsi Potluru, and Pan Li. Graphmaker: Can diffusion models generate large attributed graphs? arXiv preprint arXiv:2310.138333 2023.
[47] Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning, pages 20035-20064. PMLR, 2023.
[48] Meng Liu, Keqiang Yan, Bora Oztekin, and Shuiwang Ji. Graphebm: Molecular graph genera- tion with energy-based models. arXiv preprint arXiv:2102.00546, 2021.
[49] Yanchen Luo, Sihang Li, Zhiyuan Liu, Jiancan Wu. Zhengyi Yang, Xiangnan He, Xiang Wang, and Qi Tian. Text-guided diffusion model for 3d molecule generation, 2024. URL nttps://openrevie:: wett//ooumiiddaaatattte
[50] Lukasz Maziarka, Agnieszka Pocha, Jan Kaczmarczyk, Krzysztof Rataj, Tomasz Danel, and Michat Warchot. Mol-cyclegan: a generative model for molecular optimization. Journal of Cheminformatics, 12(1):1-18, 2020.
[51] Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint 210.022000..173..... 2021.
13

---

[52] Hunter Nisonoff, Junhao Xiong, Stephan Allenspach, and Jennifer Listgarten. Unlocking guidance for discrete state-space diffusion and flow models. arXiv preprint arXiv:2406.01572. 2024.
[53] Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permutation invariant graph generation via score-based generative modeling. In International Conference on Artificial Intelligence and Statistics, pages 4474 4484. PMLR, 2020.
[54] Juhwan Noh, Dae- Woong Jeong, Kiyoung Kim, Sehui Han, Moontae Lee, Honglak Lee, and Yousung Jung. Path-aware and structure-preservingg generation of synthetically accessible molecules. In International Conference on Machine Learning, pages 16952-16968. PMLR, 2022.
[55] Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. Journal of cheminformatics, 9()::1114, 2017.
[56] Yidong Ouyang, Liyan Xie, and Guang Cheng. Improving adversarial robustness through the sondrestive-guided diffusion process. In International Conference Machine Learning, pages 26699-26723. PMLR, 2023.
[57] Bo Qiang, Yuxuan Song, Minkai Xu, Jingjing Gong, Bowen Gao, Hao Zhou, Wei-Ying Ma. and Yanyan Lan. Coarse-to-fine: a hierarchical diffusion model for molecule generation in 3d. In International Conference on Machine Learning, pages 28277-28688 28299. PMLR, 2023.
[58]
Raghunathan Ramakrishnan, Pavio O Dral, Matthias Rupp, and O Anatole Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 2014.
[59] Arne Schneuing, Yuanqi Du, Charles Harris, Arian Jamasb, Ilia Igashov, Weitao Du, Tom Blundell, Pietro Lió, Carla Gomes, Max Welling, et al. Structure-based drug design with equivariant diffusion models. arXiv preprint arXiv:2210. 2022.
[60] Ida Schomburg, Antje Chang, Christian Ebeling, Marion Gremse, Christian Heldt, Gregor Huhn, and Dietmar Schomburg. Brenda, the enzyme database: updates and major new developments. Nucleic acids research, 32(supp__1)D4311-4433 2004.
[61] Daniel Schwalbe-Kond Koda and Rafael cömez-Bombarelli, Generative models for automatic chemical design. Machine Learning Meets Quantum Physics, pages 445-467, 2020.
[62] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- Rad. Collective classification in network data. Al magazine, 29(3):93-93, 2008.
[63] Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular conformation generation. In International conference on machine learning, pages 9568-9568. PMLR, 2021.
[64] Gregory Sliwoski, Sandeepkumar Kothiwale, Jens Meiler, and Edward W Lowe. Computational methods in drug discovery. Pharmacological reviews, 66(1):334-39) 2014.
[65] Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse- diffusion models for inverse problems. In International Conference on Learning Representations, 2023.
URL htpp:://///peeereeeee nwt. htte//oormmiddddep -ooouumidd-9.gMMMMEEE.
[66] Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-plan controllable generation. In International Conference Machine Learning, pages 12483-32499 PMLR, 2023.
[67] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[68] Yuxuan Song, Jingjing Gong, Minkai Xu, Ziyao Cao, Yanyan Lan, Stefano Ermon, Hao Zhou, and Wei-Ying Ma. Equivariant flow matching with hybrid probability transport for 3d molecule generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:t::://// wtt.pppnrreiieeeeeeeoeuuuaaaaaaa
14

---

[69] Brian L Trippe, Jason Yim. Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi Jaakkola. Diffusion probabilisticc modeling of protein backbones in 3d for the motif-scafii scaff--cafoolding problem. arXiv preprint arXiv:2206.044 19, 2022.
[70] Brian L. Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi S. Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif- scaffolding problem. In The Eleventh International Conference on Learning Representations, 2023. URL htttps:///ppnreeeeeeee
[71] Alex M Tseng, Tommaso Biancalani, Max Shen, and Gabriele Scalia. Hierarchically branched diffusion models for efficient and interpretable multi-class conditional generation. arXiv preprins (7Xiv:22/2.1077: 2022.
[72] Alex M Tseng, Nathaniel Diamant, Tommaso Biancalani, and Gabriele Scalia, Graphguide: interpretable and controllable conditional graph generation with discrete bernoulli diffusion. arXiv preprint urXiv:2302.03777 2023.
[73] Yogesh Verma, Samuel Kaski, Markus Heinonen, and Vikas Garg. Modular flows: Differential molecular generation. In Advances in Neural Information Processing Systems, volume 35, pages 12409-12421. Curran Associates, Inc., 2022.
[74] Yogesh Verma, Markus Heinonen, and Vikas Garg. Abode: Ab initio antibody design using conjoined odes. International Conference on Machine Learning, pages 35037-35083 PMLR, 2023.
[75] Clement Vignac, Igor Krawezuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. In The Eleventh Inter- national Conference on Learning Representations, 2023. URL htttps://openreviex net/ forumPid=UaAD-Muu8h0
[76] Wujie Wang. Minkai Xu, Chen Cai, Benjamin Kurt Miller, Tess Smidt, Yusu Wang, Jian Tang. and Rafael Homeez-Bombarelii Generative coarse-graining of molecular conformations. arXiv preprint 1xXiv:2201.12170 2022.
[77] Zichao Wang, Weili Nie, Zhuoran Qiao, Chaowei Xiao, Richard Baraniuk, and Anima Anand- kumar. Retrieval-based controllable molecule generation. arXiv preprint arXiv:2208. /26, 2022.
[78] Fang Wu and Stan Z Li. Diffmm: geometric diffusion model for molecular dynamics simu- lations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 5321-5329, 2023.
[79] Kevin Eric Wu, Kevin K Yang, Rianne van den Berg, James Zou, Alex Xijie Lu, and Ava Amini. Protein structure generation via folding diffusion, 2023. URL /ttpsspp:///ppeerviii net/forrumidd-NRaTAA...
[80] Lemeng Wu, Chengyue Gong, Xingchao Liu, Mao Ye, and Qiang Liu. Diffusion- based molecule generation with informative prior bridges. Advances in Neural Information Processing Systems, 35:36533-36545, 2022.
[81] Minkai Xu, Lantao Yu, Yang Song, Chence Shi. Stefano Ermon, and Jian Tang. Geodiff: geometric diffusion model for molecular conformation generation. In International Con- ference Learning Representations, 2022. URL http::://oppereeeeeeeeeee// net/ wtt.://wmneeeooo/mmmaaa formmiid
PZcvxEMzvQC
[82] Minkai Xu, Alexander S Powers, Ron O Dror, Stefano Ermon, and Jure Leskovec. Geometric latent diffusion models for 3d molecule generation. In International Conference on Machine Learning, pages 38592-38610. PMLR, 2023.
15

---

A Proofs
A.1 Derivation of the reverse SDE
For Stochastic Differential Equation (SDE) of the form,
dx far, l)dl + g(x,, ()dw
(10)
where f()) and (()) are diffusion, drift function and dw is the weiner noise. The evolution of the distribution of 24 governed by the Kolmogorov Forward Equation (KFE) as,
1 (x,) -r. If ( + 0e, [9e pxx)
(11)
Kolmogrov Forward/Backward Equation (KFE/KBE). Essentially KFE describes the evolution of a probability distribution p(a+) forward time. The reverse-time SDE can be derived by solving the Kolmogorov Backward Equation (K.B.E) as derived in Anderson [[]. It can be defined for t 2 to as,
-pp (It, xr.)) = (mt,) Deeg P (Ie, xro) + 50 (#t.) 02 (2t, (to)
(12)
where Lt, and It, are distributions at the respective time steps. Specifically, it models how the distribution dynamics a later point in time changes as we change to an earlier time.
In our case, we consider the diffusion over structure y, and properties {y ykk.. The KFE of the system y V1s..... is given by,
1 of [g² (y.) 0.p (y:) -y (f+)))
(13)
Independence Factorization. We can factorize p (y+) based (yI.t.... ) are independent conditioned on the structure ys,t as
assumption that the properties
ply:) = P(y.,, ...1)
p(ys.))(yy,.. b...... ... Ys,s)
p(ys.) I Plya ys,))
(14)
Leveraging this factorization, we define system of SDEs with KFEs for each variable, leading us to the SDE system defined in Eq. and Eq. 2
Reverse SDE: In the reverse case, aim to denoise the full vector y {y., VI. ....... .kk where y. denotes the diffusion over structure and {yI...., Y over the properties via reverse SDE. Expressing in the form of Eq. 12 we note that for t 2 to.
-0p Vto) = f (Y Py.P (ye, Yta) 2 (y Sg Py, y1o)
(15)
Anderson 1 defines joint distribution over the time-ordered variables yt, and yt, to derive the reverse SDE We utilize their analysis and define a joint distribution
p Ne.
(y.. e. V1,4 ... Vk.t. a Ss.to V1.to Yk.ta)
= = (y V1,6 Vk.h, Ya,to .k...o p(ys,oo V1.6o..... ..........,....))
(16)
We denote p(y..o=
Vk..)) by p(y,.). ), and note that it can be decomposed similarly as in Eq. [4] Taking the time derivative of Eq. [6] we get
Vi.to..... -
-P,p (to) O.P 2..1 V1A Vko Ya Vi,fo ... Vk.ta) p(yta)
Opp(y,o)) (y V1,t Sket, Y1.to Vk.to)
(17)
Comparison with KFE/KBE. observe that O.p (ystis V1, ... , Ystoo V1
Yk.to)
corresponds to the KBE in Eq. 15 and dpp(y1.) to the KFE in Eq. 13
Denoting
16

---

V1.1 :
yksid H by Yt. we immediately get
(yn, p(y+)) Orp((to)p(ys yto)
(ye, vto) + 20 (yto) 0g (yt, (too) plyco) + + (yt, yto)) (Bri [S (yro) p (yro)l
= ((1(yt)) Oxr) (y
(18)
2 8 [o_ (y p
The derivatives can be handled, by following standard differentiation rules as,
(yt,, yto)
Dy. P (yr,
Vto))
dy..
p (Yr.)
(19)
Oy., Nto)
p (y Vto) Oy (y
(y:o)
p² (yto)
Evaluating the derivative of the products in the forward Kolmogorov equation and substituting the derivatives accordingly we obtain,
1
-d,p (yt, (to) = dy:o
Vt.) +
2 (y..) yip (ye, yt.) p(y..)
(20)
1
2 (yr, yt,)) oy [gº (yt,)) p(yt.)]
Matching the terms of the second-order derivatives with the expansion of the derivative and doing some algebraic manipulations, we obtain
-ú,p -y (yr, Vo.) = -Oy.p If (Fr.) p Ne,)) +
2 Nr, Ip (Vy,, N1o) 9² (yt.))
(21)
- -y, (ve, Via) dy., 1 (y1))((vt))))
which can be written as
-O.p (Yt., Nto) -yeo p( (- (f((r))) + p(y..)
1 ply 2y.o (y ((w)))))))))
+ (22)
1 oe (y,, o² (yto)]
(23)
2
Comparison with KFE. The above result is in the form of a Kolmogorov forward equation with the joint probability distribution ( The time-ordering is t > to and the term - ~èp (t.,) describes the change probability distribution as we move backward in time. We can marginalize over f1, using the Leibniz rule, to obtain
-P,p (yt)) dy., p(sw)) (-f + p(y.)) Py.o (g² (yr (yw) oo 2
(y (yr))
(24)
This finally gives a stochastic differential equation analogous to the Bokker-Planck/foowaala Kol- mogorov equation that can be solved backward in time:
dyt. = --- 1) + (vv.)) Oy (g² (yt,) (v)))(()))) dt + g (y+,) dw
(25)
We keep 9² (yt)) independent of yto- Applying the log-derivative trick, the SDE simplifies
dy (f oh Vyr, log pogp(tt))) + 9t.
dw dw
(26)
A.2 Conditional score factorization
We extend our method incorporate an external context or conditional information for conditional generation, similar to classifier-based [8] and classifier-fres [23] guidance. Following similar notation, the reverse SDE (67, given an external context yc can written as
dy, [f(y,,t) t) - gf log (v) g.dw
(27)
17

---

Here Yt Y1,...... Vk.t, and yc = (ye c C) is an external context or conditioning variable. This external context can be a scalar or vector describing property value of the primary variable like QED or plogp in the case of molecules
image labels in the case of images. The Vy; log (y.: Yc) term pertains to the score function which guides the process (see table for com- parison with both classifier-based and classifier-free guidance). Under our condition independence assumption, the score function factorizes as
K Vidt .1.s........... = II (y V.s.N.))
(28)
Vy, log Vi.t..... Vc = Vy lygpi
Σ
Vy,
lyg Yos)
iec
(29)
k
6 sime Vy, log Ya,sYo)
B
Parameterizations
Here we describe two instances of Twigs based on architecture choices: Attention networks, and graph convolution networks (GCNs). Twigs with attention is used and 4.2 while Twigs with GCNs is used in 4.3 and 4.4
B.1 Twigs with graph attention
We denote the variable y. as 3D graph G (A, x, h), with node coordinates x (x) ....,. VV RN33 node features h (h' ,...., h R.Nxd and edge information A NxNx2 The variable C E I denotes the conditional information, which is obtained by adding the noise level log(a, (a(a//o]), the perturbed property y, (0 Z) R, and the fixed property yс R. The context C combined with y. by multilayer perceptions (MLP), after projecting (h, A, x) respectively into
H. E, P:
AdaLN (1 LLN
(30)
M MHA(AdaLN(H, C), (dE C),
where MHA is the multi-head attention, and AdaLN is Adaptive LayerNorm (LN) function. Subse- quently, we leverage the Scale function Scale(h, C) = MLP(CC) hh, and the Feed Forward Network (FFN) to obtain the Diffusion Graph Transformer (DGT) block, as defined in (28), which is described by Eq (2)
DGT first computes the intermediate representations for the 1-th layer as:
M' MHA(AdaLN(HH C), AdaLN(E, C')
(31)
H Scale(M's C) + H'
E = (M + M C) + +
then computes the 1 + layer as:
E-1 Scale(TPN(AdaaL C)),
(32)
H -Scalee Scalc(FFNNAdaLLN)) C)), C) H
p(+1 Eij PP:P1
P.-P/ tanh (MLP (E)))))))))
The Twigs trunk process sy is parameterized as:
8p = DGT + E. PDGT,
(33)
where PDGT, resembles the stem process networks 8a,, which obtained by pooling to one- dimensional variable by an MI. operation, over the output of the DGT block. To optimize Eqq ( DGT minimizes the denoising score matching objective from (28] for node, edge and position information (h, A, x), while PDGT; for the perturbed property yt.
18

---

B.2 Twigs with graph convolutions
In the case of 2D graphs with N nodes we consider the variable y, (X, A) RN R where F is the dimension of the node features, X E RNx are node features, A E RNNN is weighted adjacency matrix. We define the perturbed property y: R and the (fixed) property yc R. The stem process network 8q. is given as:
S6. (GNN(, A)); P. [xv]llvv.lvvv
(34)
where v; and ve are vectors obtained by repeating N times the perturbed property y; and the fixed property Ye respectively, and concatenating them into the node features matrix X. The Twigs trunk process is obtained by combining the contributions from the properties y, derived by the stem processes sd; and the structure ys. as follows:
= S8* [X, A, + Σ, (X,
(35)
where sax is conditional node feature
network: 8gx MLP(GNN(XX yc, A)). Finally, following 37 A is co-evolved together with the node features, by the adjacency score model $6
884 = MLP ([GGH (H; Af) 0.p
K.P.
(36)
where GMH is graph multi-head attention
(2). which employs higher-order adjacency matrices A" and K denotes the number of GMH layers. The optimization for the Twigs objective function ( is
obtained by minimizing the denoising score matching for A, X,P,
The GMH block employs higher-order adjacency matrices A to represent the long-range dependen- cies and is provided as: 88a (G.) = MLP ([(GMH (H,, A!) K.P
==)
C
Additional experimental results
C.1
QM9 dataset
Further details for generation conditioned quantum properties from Section 4.1
Molecular quality, Additional results for molecular stability in 2D and Fréchet ChemNet Distance (FCD) for 2D and 3D are given in Table I2
Table 12: Molecule quality results.
Property
Mol-S-2D
FCD-2D |
FCD-3D
C
98.88
0.107
0,871
98.93
0.125
0.842
a
98.71
0.106
0.867
Ac 98.82
0.105
0.787
<номо ÉLUMO
98.95
0.111
0,827
98.52
0.117
0.846
C.2 ZINC250K dataset
Conditional generation. predictors released from 45 which given molecule G, are trained to predict
The evaluation is performed by measuring the MAE of the pre-trained
Obj = DS.).) QED(G.) x SA(G.)
(37)
where DS is the normalized docking score (DS) of the considered target protein, QED is the drug- likeness, and SA is the normalized synthetic accessibility (SA).
In terms of baselines, we consider the MOOD model [45] which leverages classifier-based guidance scheme, and we also implement diffusion guidance version of GDSS [37 based on the classifier- free
scheme. Our Twigs method is parameterized by the architecture described B.2 with single stem process. The models conditioned on the function in Equation (37).
19

---

Table 13: MAE for ZINC250K conditioned on single properties.
parp1
fa7
5htlb
braf
jak2
GDSS
5.56
4.76
5.78
5.73
5.98
MOOD ood=0.04
5.42
4.33
5.52
5.37
5.10
MOOD ood=0.01
5.41
4.33
5.52
5.36
5.09
Twigs
5.38
4.30
5.43
5.28
5.01
Results. In T3] report the mean MAE values over multiple runs computed from the generated molecules using the pre-trained classifiers from [45]. We can observe that the Twigs consistently achieves lower error, demonstrating improved control over generating molecules with the desired
target proteins.
Runtime. We have incorporated the runtime for molecule generation at inference time for large- scale dataset (ZINC250K) as for Section [4.3] in Table [4] A comparison with MOOD [ indicates that our model incurs certain overhead, as anticipated. However, it demonstrates improved alignment when generating conditional molecules.
Table 14: Runtime for inference on molecule generation.
model
Seconds per molecule
Twigs MOOD
0.378 0.267
D Experimental details
D.1
Computational resources
All experiments are performed with GPUs, Nvidia A100 or v100.
D.2
Models details
We follow the data splits from Huang et al. 28 for 4.1 the data splits from et
4.2 the ones from Lee et al.
45 for 4.3 and
for We use Adam optimizers on all experiments.
For Sections 4.1 and 4.2 we follow the same hyperparameters from Huang et al. 22 For Section 4.3
we follow the hyperparameters from Lee al. [45), for the MOOD baseline, explore OOD
coefficients between 0.01 and 0.09. For Section 4.4 we follow the hyperparameters from Jo et al.
37
E
Additional Related Works
This section extends the discussion presented in Section by exploring additional related works in the field. In Table we summarise related methods including score-sdes, hierarchical models (not necessarily conditional), and hierarchical conditional models.
Conditional molecular diffusion. Guidance techniques have also been adopted in conditional molecule generation settings: in the context of classifier-free approaches, Hoogeboom et al. [26
proposes equivariant approach based on DDPM for 3D molecules; Huang et al. (28) explores attention mechanisms within SGM models; and Xu et al. investigates DDPMs in latent space settings.
In terms of classifier-based guidance, Bao al. incorporate energy guidance into a diffusion model by leveraging stochastic differential equation; Vignac et al. [75 provide DDPM coupled with a classifier over quantum molecular properties; and Lee al. 45
operate over pre-trained SGM and train an additional predictor for fine-tuning the desired protein target properties.
20

---

Table 15: Comparison with related works.
Method
Score-based SDE
Hierarchical modeling
Hierarchical conditional diffusion
EDM [26
x
X
EEGSDE
X
Digress 75
X
HierVAE 34
X
XX
X
GraphGuide
X
GeoL.DM 82
X
HierGraph [57
X
JODO 288
Twigs (this work)
Guidance methods. Recent works utilize multiple diffusion processes: cascaded diffusion 25 provides a flow for each resolution, and GDSS (37] has a joint system of diffusion processes one for nodes and the other for edge features, but it does not cover mechanisms for conditional generation. Tseng et 771) define hierarchy of branching points within a single diffusion flow.
Other Diffusion methods for Graphs. Other works related to ours focus on hierarchical diffusion processes 777, diffusion applied protein backbones [69], geometry-based models [59 82) and autoregressive models [1] In the realm of stochastic differential equation (SDE)-based approaches, the literature includes bridge methods [38, permutation invariance [27], torsional modeling [36.. and docking [6]. Additionally, [63] introduces the ConfGF approach, estimating gradient fields of atomic coordinates, while [80] proposes method steering the training of diffusion-basee based generative models using physical and statistical prior information.
Autoencoder-Bann graph models. This category includes works employing autoencoders, such retrieval-based models (77 [2), scaffold modeling (50), link design [29), and coarse-grain model- ing (76). Notably, [54] proposes raction-embeddee and structure-conditioned variational autoen- coder, while [42] defines the concept of principal subgraphs, relevant to informative patterns within molecules.
Conditional Diffusion. In the realm of diffusion generative models, several noteworthy approaches have been developed to enhance their performance and versatility. Du et al. [10] introduce energy-based parameterization of diffusion models, allowing the integration of novel compositional operators and Metropolis-corres correcctd samplers. Building this, He al. [22] contribute training- free conditional generation framework, leveraging pretrained diffusion models focusing on the manifold hypothesis refine guided diffusion steps and introduce shorteut algorithm. Meanwhile, Meng et al. [51] employ a stochastic differential equation (SDE) in synthesizing realistic images, iterating through denoising steps guided by a pretrained diffusion model.
In a different vein, Song et al. [66] propose guiding denoising diffusion models with general differen- tiable loss functions in plug-and-play manner, facilitating controllable generation without additional training. Addressing the challenge inferring high- high-dimensional data within the context of diffusion models, Graikos al. [17] present model consisting of prior and auxiliary differentiable constraint. Dinh et al. 9 tackle diversity and adversarial effects classifier guidance for diffusion generative models by allowing relevant classes' gradients to contribute shared information con- struction during noisy early sampling steps. Furthermore, Song et al. (65) put forth a method for estimating conditional scores without additional training. Lastly, Ouyang al. (56) propose the Contrastive-Guided Diffusion Process (Contrastive-DP)) integrating contrastive loss to guide the diffusion model in data generation. These diverse contributions collectively advance the field by addressing various challenges and expanding the capabilities of diffusion generative models.
21

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We demonstrate with theoretical results and comprehensive set of experi-
ments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations provided in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
Proof' provided in Section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Section 4 we provide details reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
The implementation details are in appendix to run the experiments. The used datasets are public and can be accessed with the reference paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/ppp wwwpppblccgguiecuceccccceooooope /eegguddsse/Ceebbbiniinn htt.SSbbbiiiiooooiiiiiooooii for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Described in D.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report mean and standard deviation in our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Described
D.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://uurr....ccpplccc/caaaaaaa
Answer: [Yes]
Justification:
We conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Provided in Section 5
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Does not apply for our paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The resources that we used are cited, the source code we used is released on open licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve Crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve IRB.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "cQkOlav5HU", "reviewer": "Reviewer_GMeF", "summary": "This paper proposes a novel score-based diffusion framework called Twigs that incorporates multiple co-evolving flows to capture complex interactions and dependencies for enriching conditional generation tasks. It consists of a central trunk process and additional stem processes, coordinated by a loop guidance strategy during sampling. Extensive experiments on conditional graph generation demonstrate Twigs' strong performance gains over baselines, highlighting its potential for challenging generative tasks like inverse molecular design.", "questions": "1. Is the external context $y_C$ one of the $k$ dependent graph properties $y_k$ in Eq. 2?", "limitations": "Not applied.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Based on my knowledge, the idea of incorporating multiple co-evolving diffusion processes proposed in this paper is very novel, especially in the field of graph generation.\n2. The paper is well written and provides sufficient background information, such as Table 2, to help understand the differences between the method proposed in this paper and other classifier-based and classifier-free methods.\n3. The experiments are very comprehensive and thorough, and the experimental results fully demonstrate the outstanding performance of the proposed method.", "weaknesses": "1. The content in Section 3 is written too generally and does not address the specific characteristics of graph data very well. Additionally, the mathematical symbols used appear somewhat disorganized.\n2. I am a little bit confused about the the dimension graph structure $y_s \\in \\mathbb{R}^{N\\times D}$ on top of Eq. (1). Does $D$ covers a lot more information, e.g. node coordinates (dim 3), node features (dim $d_1$), and  edge information (dim $N \\times d_2$), as indicated in Appendix B.1?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your thoughtful comments. We address all your concerns, as described below.  \n\n**RE: Section 3 is written too generally and does not address the specific characteristics of the graph (W1)**\n\nWe maintain the section in a general format to accommodate multiple cases, specifically one for 2D graphs and another for 3D graphs. We achieve the desired flexibility by introducing a variable $y_s$ that encompasses both node features and the adjacency matrix (2D case), as well as the coordinates (3D case). Below are the detailed descriptions of the $y_s$ variable.\n\n**RE: Dimension of the graph structure in Eqn (1) (W2)**\n\nWe address two cases:\n\n**3D Case (Appendix B.1)**: We denote the variable $y_s$ as a 3D graph $G = (A, x, h)$, where node coordinates are represented as $x = (x^1, \\ldots, x^N) \\in \\mathbb{R}^{N \\times 3}$, node features as $h = (h^1, \\ldots, h^N) \\in \\mathbb{R}^{N \\times d_1}$, and edge information as $A \\in \\mathbb{R}^{N \\times N \\times d_2}$.\n\n**2D Case (Appendix B.2)**: Here, we denote $y_s$ as a 2D graph with $N$ nodes. The variable is defined as $y_s = (X, A) \\in \\mathbb{R}^{N \\times F} \\times \\mathbb{R}^{N \\times N}$, where $F$ represents the dimension of the node features. In this case, $X \\in \\mathbb{R}^{N \\times F}$ are the node features, and $A \\in \\mathbb{R}^{N \\times N}$ is the weighted adjacency matrix. Additionally, we define the perturbed property $y_i \\in \\mathbb{R}$ and the fixed property $y_C \\in \\mathbb{R}$.\n\n\n**RE: External Context $y_C$ and Dependent Graph Properties (Q1)**\n\nIndeed, the external context $y_C$ can be **one or more** of the $k$ dependent graph properties. Specifically, $y_C$ represents the graph property (or properties) upon which we are conditioning. For instance, in the context of conditional generative modeling for drug design, we might seek a molecule with a specific $\\epsilon_{\\text{LUMO}}$ value. In this scenario, $\\epsilon_{\\text{LUMO}}$ serves as the $y_C$ variable. The properties listed in Equation (2) encompass all relevant characteristics of the molecule, such as $\\alpha$, $C_v$, $\\epsilon_{\\text{HOMO}}$, and others. Our model aims to perform multiple $k$ diffusion processes for each property, with each process conditioned on the  context $y_C$ and the property $y_k$. \n\n\n**Final Comment**\n\nThank you so much for your constructive feedback. If you believe we sufficiently addressed your concerns, we would appreciate an increase in your score for this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lHGLjtvf1Z", "reviewer": "Reviewer_WGvA", "summary": "This paper introduces Twigs, a new score-based diffusion model for graph generation conditioned on graph-level properties. It employs two diffusion processes, one for graph data (trunk process) and one for graph-level properties (stem process). The underlying generation process corresponds to a factorization of the joint distribution into the unconditional distribution of the graph data and conditionally independent distributions of graph properties given the graph data. Empirical studies across diverse benchmarks demonstrate the effectiveness of the proposed approach.", "questions": "In addition to the issues mentioned in the \"Weaknesses\" section, \n\n**Q1**: Why did you not include the results of EDM and TEDMol in table 4, as reported in [1]?\n\n[1] Luo et al. Text-guided Diffusion Model for 3D Molecule Generation.", "limitations": "The paper assumes conditional independence between multiple graph properties, which does not necessarily hold in practice.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**S1:** The overall presentation is clear and easy to follow.\n\n**S2:** Extensive empirical studies demonstrate the effectiveness of the method for conditional molecule and graph structure generation.", "weaknesses": "**W1**: The idea of hierarchical (conditional) diffusion model for graph generation has been explored in previous works. For example, GDSS [1] and EDGE [2] explore the factorization of the joint distribution of graph structure and node attributes for unconditional molecule generation. GraphMaker [3] explores this idea for node-label-conditioned generation of large attributed graphs.\n\n**W2**: The assumption of the conditional independence of the multiple graph properties may be too strong.\n\n**W3**: The empirical studies performed consider at most modeling two graph properties at a time and there is a lack of understanding in how the model performance changes as more graph properties are modeled simultaneously. In addition, there is a lack of understanding in how the model performance varies as the properties get more correlated.\n\n**W4**: Some presentation details can be further clarified. For example:\n- The terms \"hierarchical modeling\" and \"hierarchical conditional diffusion\" are used without a formal definition in table 1.\n- I guess $y_{s,t}$ between L110 and 111 refers to $y_{s}$ at time $t$, but it is not formally defined.\n- L106 says that $y_s$ is the primary variable (graph structure) and takes shape $\\mathbb{R}^{N\\times D}$. Does this consist of both adjacency matrix and node features like atom types?\n\n[1] Jo et al. Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations.\n\n[2] Chen et al. Efficient and Degree-Guided Graph Generation via Discrete Diffusion Modeling.\n\n[3] Li et al. GraphMaker: Can Diffusion Models Generate Large Attributed Graphs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks. \n\n**Re: related works (W1)**  \n\nThe suggested works, including GDSS (Jo et al. 2022), GraphMaker (Li et al. 2024), and EDGE (Chen et al. 2023), are relevant to our research and will be included in our paper. Below we report the differences with our method. Refer to the table of the global comment for a summary.\n\nFirst, these methods use different hierarchical notions in their frameworks. In EDGE and GDSS, edges and nodes are modeled symmetrically. In contrast, we introduce a unique hierarchical approach by learning multiple asymmetric diffusion flows. This distinction is crucial: while the mentioned methods use diffusion flows in the same roles, our method uses a secondary process to learn interactions with properties, feeding back into the main process to learn the graph structure. \n\nSecondly, our methodology is designed specifically for generating graphs with desired conditional properties, which the mentioned papers do not focus on. GDSS and EDGE do not consider conditional modeling, and while GraphMaker explores node-label conditioning, this differs from our goal of generating graphs constrained by specific properties. Our method learns a joint distribution of graph structures (nodes, edges) and properties, unlike the mentioned methods that only learn a joint distribution of nodes and edges. \n\n\n**RE: conditional independence (W2)**  \n\nAssuming conditional independence among the properties $\\alpha$, $\\epsilon_{\\text{HOMO}}$, $\\epsilon_{\\text{LUMO}}$, $\\Delta \\epsilon$, $\\mu$, and $C_v$ given the molecular graph can simplify the modeling process. This assumption leverages the fact that the molecular graph captures the essential structural dependencies, allowing us to treat the properties as independent for computational efficiency and ease of interpretation, even if slight interdependencies exist.\n\n\n**RE: Modeling multiple properties (W3)**  \n\nWe show here two additional results on 2 and 3 properties. We study the models over the `community-small` dataset. Specifically, our method (Twigs) is trained using multiple secondary processes (one for each property).\n\n**Two properties**\n\nWe show the MAE results for property pairs for density. Our model achieves the lowest MAE in both cases. We notice that while the properties may present some form of correlation, our model can achieve a good performance in generating the graphs with desired properties.\n\n\n| Model   | MAE Density | MAE Clustering |\n| ------- | ----------- | -------------- |\n| GDSS    | 2.95        | 13.3           |\n| Digress | 2.82        | 12.1           |\n| MOOD    | 2.43        | 12.0           |\n| Twigs   | **2.34**    | **11.0**       |\n\n| Model   | MAE Density | MAE Assortativity |\n| ------- | ----------- | ----------------- |\n| GDSS    | 2.61        | 19.8              |\n| Digress | 2.52        | 18.1              |\n| MOOD    | 2.40        | 17.2              |\n| Twigs   | **2.39**    | **16.7**          |\n\n\n**Three properties**\n\nThe Table below shows that our method achieves the lowest MAE values (the lower the better) across all three required properties. \n\n| Model   | MAE Density | MAE Clustering | MAE Assortativity |\n| ------- | ----------- | -------------- | ----------------- |\n| GDSS    | 2.97        | 12.5           | 19.4              |\n| Digress | 2.65        | 11.2           | 18.2              |\n| MOOD    | 2.53        | 11.4           | 17.3              |\n| Twigs   | **2.27**    | **10.6**       | **16.1**          |\n\n\n\n\n**RE: Hierarchical conditional diffusion (W4.1)**  \n\nHere's a clarification of \"Hierarchical Conditional Diffusion\" and the distinction from \"unconditional hierarchical models\", which will be integrated into the paper.\n\nIn lines 39-43, we define \"Hierarchical Conditional Diffusion\" as follows: \".. rather than treating heterogeneous structural and label information uniformly within the hierarchy, we advocate for the co-evolution of multiple processes with distinct roles. These roles encompass a primary process governing structural evolution alongside multiple secondary processes responsible for driving conditional content.\"\n\nTo distinguish our method from \"unconditional\" hierarchical models by Jin et al. (2020) and Qiang et al. (2023), we label those models as \"Hierarchical Modeling\" in Table 1. While they model hierarchical structures, our approach is unique in leveraging a hierarchy of branching diffusion processes for conditional generation based on desired properties.\n\n**References**\n\nJin et al. 2020. Hierarchical generation of molecular graphs using structural motifs.  \n\nQuiang et al. 2023. Coarse-to-fine: a hierarchical diffusion model for molecule generation in 3d.\n\n\n**RE: Variable $y_s$ (W4.2)**  \nIn the variable $y_{s,t}$ t indicates time. We will add to the main text.\n\n\n**RE: Dimension of graph (W4.3)**   \n\nWe have two cases: \n\nThe 3D case in (Appendix B.1), we denote the variable $y_s$ as a 3D graph $G = (A, x, h)$, with node coordinates $x= (x^1, \\ldots, x^N) \\in \\mathbb{R}^{N \\times 3}$, node features $h = (h^1, \\ldots,h^N) \\in \\mathbb{R}^{N \\times d1}$, and edge information $A \\in \\mathbb{R}^{N \\times N \\times d2}$.\n\nThe 2D case in Appendix B.2:  we denote $y_s$ as a 2D graph with $N$ nodes we consider the variable $y_s=(X,A)\\in \\mathbb{R}^{N\\times F}\\times\\mathbb{R}^{N\\times N}$, where $F$ is the dimension of the node features, $X\\in\\mathbb{R}^{N\\times F}$ are node features, $A\\in\\mathbb{R}^{N\\times N}$ is weighted adjacency matrix. We define the perturbed property $y_i \\in \\mathbb{R}$ and the (fixed) property $y_C \\in \\mathbb{R}$.\n\n\n**RE: EDM and Tedmol results (Q1)**  \n\nApologies for the misleading results presentation. Due to space constraints, we placed the results for EDM and Tedmol in Table 10 of Appendix C.1. We will move it to the main text. These methods were included in the appendix because they are less competitive compared to the other baselines.\n\n\n**Final comment**\n\nWe hope our response addresses your concerns and reinforces your support for this work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T2gwqe6dce", "reviewer": "Reviewer_8xx4", "summary": "This study proposed a conditional generative model with guided diffusion. The authors introduce a new mechanism called loop guidance to include conditions. Empirical analysis includes small molecule generation on a diverse of datasets and properties, with both quantitative evaluation and visualizations.", "questions": "1. What are the meanings of the properties $C_v$, $\\mu$, $\\alpha$, $\\Delta\\epsilon$, $\\epsilon_{\\text{LUMO}}$, and $\\epsilon_{\\text{HOMO}}$? Why are they important impact factors for molecule generation? Are they practically easy to obtain? Since these properties are used as guides when generating small molecules, do the generated molecules actually have properties close to the guide? \n2. What is the relationship and difference between the proposed method and other recent works on guided diffusion, such as https://arxiv.org/pdf/2406.01572, https://arxiv.org/abs/2305.20009, and https://openreview.net/pdf?id=8NfHmzo0Op? (Note: I'm not requiring a quantitative comparison, especially for the first one, which was uploaded after the conference submission deadline. However, for the latter two papers, as they have been published for more than 3 months, I believe they should be included in the paper.)\n3. How should the top row of Figure 2 be interpreted? Are they different samples from the same guidance, or are they the same sample shown from different views?\n4. Similarly, it is not clear how to interpret the results in Figure 3 (and this figure was not referenced in the main text). Are all the samples generated based on the same input properties? How can the legitimacy of these generations be validated? What does the ground-truth molecule look like? To what level they are better than the generation performance of the baseline methods?", "limitations": "The authors listed one limitation of the current work as not generalizing the framework to other types of data (text, image). No additional analysis on the broader impact has been provided.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a well-structured framework for conditional generation with guided diffusion, including a neat summary of relevant methods in this field.\n2. The presentation of the methods and results is clear and easy to follow.\n3. The authors provide extensive experimental comparisons, offering a thorough evaluation of the proposed method against various baselines.", "weaknesses": "The weaknesses mainly arise from the insufficient empirical evidence:\n\n1. No comparison has been provided regarding the computational cost, making it unclear how efficient the proposed method is relative to others.\n2. Although the authors claimed that conditional generation is a fast-developing research field, the baseline methods used for comparison are generally from before 2023, which may not reflect the current SOTA.\n3. No ablation study or hyperparameter selection/analysis has been reported. \n4. Some terminologies could be introduced and analyzed more carefully (see question 1 below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks! \n\n**RE: computational cost  (W1)** \n\nIn Table 12 of paper, we present the inference times of our model compared to MOOD, showing that while we encounter a small overhead, we achieve superior performance. \n\nIn addition, we report below the average time (in seconds) required to train a single epoch for the `community-small` and `Enzymes`. For Twigs, we consider one property (density).  \n\n\n| Dataset         | Twigs p=1 | GDSS   | Digress |\n| --------------- | --------- | ------ | ------- |\n| Community-small | 0.2747    | 0.2294 | 0.2382  |\n| Enzymes         | 4.8669    | 4.8260 | 4.8451  |\n\n\n\n**baseline methods (W2)**  \n\nWe provide a list of the recent methods (after 2023) in our experiments.\nIn Sections 4.1 and 4.2, we compare Twigs with the following methods on QM9: TEDMol (Luo et al 2024), Jodo (Huang et al 2023), EEGSDE (Bao et al 2023), GeoLDM (Xu et al 2023), EquiFM (Song et al 2023). In Section 4.3, we compare with Lee et al (2023). In Section 4.4, we compare with Lee et al (2023), and Vignac et al (2023) on the `community-small` dataset.\n\n\n\n\n\n**Hyperparameters (W3)**  \n\nHere we report the hyperparameters used in the experiments.\nFor Sections 4.1 and 4.2 we follow the same hyperparameters from Huang et al (2023).\nFor Section 4.3 we follow the hyperparameters from Lee et al (2023), for the MOOD baseline, we explore the OOD coefficient between $0.01$ and $0.09$.\nFor Section 4.4 we follow the hyperparameters from Jo et al (2022).\n\n\n**Ablation study (W3)**  \n\nWe have added a new study to understand the impact of multiple properties over the `community-small` dataset. We report the results in the \"Additional experiments with multiple properties\" part of the [global comment](https://openreview.net/forum?id=fvOCJAAYLx&noteId=SQNeH19zc9).\n\n\n**Refs**\n\nHuang et al (2023). Learning joint 2d & 3d diffusion models for complete molecule generation.\n\nLee et al (2023). Exploring chemical space with score-based out-of-distribution generation.\n\nJo et al (2022). Score-based generative modeling of graphs via the system of stochastic differential equations. \n\nBao et al 2023. Equivariant Energy-Guided SDE for Inverse Molecular Design.\n\nLuo et al 2024. Text-guided diffusion model for 3d molecule generation.\n\nXu et al 2023. Geometric latent diffusion models for 3d molecule generation.\n\nSong et al 2023. Equivariant flow matching with hybrid probability transport for 3d molecule generation.\n\nVignac et al 2023. Digress: Discrete denoising diffusion for graph generation\n\n\n\n**properties close to the guide (Q1)**  \n\nThe generated molecules' properties closely align with the target values, as evidenced by the MAE values presented in Tables 3, 5, and 8. The MAE quantifies the deviation between the desired ground truth properties and those of the generated molecules. \n\n\n**meanings of the properties (Q1)**  \n\nThe mentioned properties are standard quantum properties used in the QM9 dataset for modeling molecules (Hoogeboom et al 2022). The molecule properties are obtained from the data. \n\n- $\\alpha$ Polarizability: Tendency of a molecule to acquire an electric dipole moment when subjected to an external electric field.\n\n- $\\epsilon_{\\text{HOMO}}$: Highest occupied molecular orbital energy.\n\n- $\\epsilon_{\\text{LUMO}}$: Lowest unoccupied molecular orbital energy.\n\n- $\\Delta \\epsilon$ Gap: The energy difference between HOMO and LUMO.\n\n- $\\mu$: Dipole moment.\n\n- $C_v$: Heat capacity at 298.15K.\n\n\n**Refs:**  \n\nHoogeboom et al 2022. Equivariant Diffusion for Molecule Generation in 3D. ICML.\n\n\n**related works (Q2)**\n\nOur approach differs from the mentioned methods in two key ways, which we will elaborate on in the related works section of our paper.\n\n**Framework**: The first two studies (Nisonoff et al. 2024, Gruver et al. 2023) are based on discrete frameworks. The third study (Klarner et al. 2024) adopts a plug-and-play approach using a diffusion model, which is agnostic to the underlying framework. Differently from the above, our method operates within a continuous framework utilizing stochastic differential equations (SDEs).\n\n**Contributions**:  Nisonoff et al. 2024 primary contribution is to provide a framework for diffusion guidance within discrete spaces. On the other hand, the main contribution of Gruver et al. (2023), addresses the challenges of optimizing discrete sequences by a novel sampling technique.\nKlarner et al. 2024 propose to learn two models one diffusion without labels and an additional discriminator model with property labels.\nOur contribution is separate from all the above methods, as we uniquely leverage multiple diffusion flows, defined over two distinct types of flows within a hierarchical structure. Specifically, we define a primary flow for capturing structure and a secondary flow for modeling properties.\n\n\n**Refs**  \n\nNisonoff et al (2024). Unlocking Guidance for Discrete State-Space Diffusion and Flow Models.  \n\nGruver et al (2023). Protein Design with Guided Discrete Diffusion.  \n\nKlarner et al (2024). Context-Guided Diffusion for Out-of-Distribution Molecular and Protein Design.  \n\n\n\n**Clarification on Figs (Q3-Q4)**\n\nFigure 2: Depits uncurated samples obtained via our model conditioned on $C_v$ property. The performance of conditioning is given in Table 3 First column ($C_v$). In addition, to validate the legitimacy, those molecules are representative of the \"Molecule quality\" results from Table 4.\n\nFigure 3. It shows uncurated samples of molecules produced under two desired properties ($C_v$ and $\\mu$). The performance corresponding to Fig 3 is represented in Table 5 first two columns  ($C_v$ and $\\mu$). \n\nNote that in this setup, our goal is not to reconstruct a specific molecule there could be multiple different molecules that have the desired properties and do not reconstruct the training set. In this sense, the target properties should be considered separately from the molecule.\n\nWe would be grateful if this could be reflected in an increased score for our work. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3598wf24xl", "reviewer": "Reviewer_Gpe5", "summary": "The paper presents Twigs, a score-based diffusion framework for enhancing conditional generation tasks. It includes a central trunk diffusion process for graph structure and stem processes for graph properties. The innovative loop guidance strategy manages information flow between trunk and stem processes. Experiments show Twigs significantly outperforms contemporary baselines, especially in tasks like inverse molecular design.", "questions": "1. How will the sampling step size affect the generation? Can we use different step sizes when sampling $y_s$ and $y_i$?\n2. Classifier guidance helps the diffusion model to generalize to new conditions. Can Twigs provide guidance of unseen conditions by, for example, separating the training of $s_{\\theta}$ and $s_{\\phi}$?\n3. How will the incorporation of multiple $s_{\\phi}$ influence the total training time of Twigs? Can you provide the training time comparison as well?", "limitations": "The limitations mentioned in the manuscript are more like future direction of applying the similar strategy to other domains. Please provide more discussion if possible.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. A new guidance strategy that explicitly models the coupling effects between condition and data.\n2. Impressive experimental results.", "weaknesses": "1. The main drawback of the proposed method is the limited flexibility. The loop guidance requires the joint training of $s_{\\theta}$ and $s_{\\phi}$ and it cannot provide guidance for unseen conditions. \n2. Some of the experimental settings and evaluation pipeline are unclear. Please at lease specify them in the appendix.\n3. Minor: Please put back the guidelines of paper checklist.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your many excellent suggestions! We have acted on all of these, and additionally, address all your questions and comments below.\n\n**RE: Twigs cannot guide unseen conditions (W1) (Q2)** \n\nThanks for raising this point. Based on your suggestion we have added an experiment to tackle the issue that you raised. The Twig framework is very flexible, and it can be trained with unconditional setting, and combined with a classifier, to achieve generalization, as you mentioned. We provide here an example.\n\nSpecifically, we first train an unconditional version of Twigs, which consists of a diffusion model over $k$ graph properties. Secondly, we train a separate discriminator for the new property to achieve prediction guidance. This idea is the same as in classifier guidance, and in MOOD (Lee et al 2023), which allows the model to generalize to properties not seen during the training of the diffusion model. \n\nThe table below shows the unconditional version of Twigs trained with three properties (excluding the unseen property) and a separate property predictor (for the unseen property), on the `community-small` dataset. The results show that Twigs receives benefits from having the diffusion over multiple properties.\n\n| Property      | MAE values |          |         |      |\n| ------------- | ---------- | -------- | ------- | ---- |\n|               | Twigs p=3  | MOOD     | Digress | GDSS |\n| Density       | **2.12**   | **2.12** | 2.34    | 2.95 |\n| Clustering    | **9.94**   | 11.3     | 10.6    | 12.1 |\n| Assortativity | **15.8**   | 16.7     | 17.8    | 19.6 |\n| Transitivity  | **8.68**   | 8.76     | 9.42    | 11.4 |\n\n\n**RE: experimental settings and evaluation pipeline are unclear (W2)**  \n\nThank you, based on your comment, we have added the hyperparameters used in the experiments, which we will integrate into the main text. In particular, we will clarify the following settings.\nFor Sections 4.1 and 4.2 we follow the same hyperparameters from Huang et al (2023).\nFor Section 4.3 we follow the hyperparameters from Lee et al (2023), for the MOOD baseline, we explore the OOD coefficient between $0.01$ and $0.09$.\nFor Section 4.4 we follow the hyperparameters from Jo et al (2022).\n\n\n**Guidelines checklist (W3):**\nYes, we will bring it back.\n\n\n**RE: how step sizes affect generation. Q1**  \n\nFor step-size Sections 4.1 and 4.2, we follow the same hyperparameters from Huang et al (2023), including step-sizes of the sampling procedure. For Section 4.3 we follow the hyperparameters from Lee et al (2023), which leverages GDSS.\n\n\n**RE: can we have different main and secondary process stepsizes. Q1**\nTo generalize to unseen properties, we do not need to separate the diffusion, we show above that we can train a separate predictor on unseen properties. Therefore we can maintain the same stepsizes in the diffusion flows.\n\n\n\n**RE: provide the training time comparison in multiple diffusion (Q3)** \n\nWe show the impact of multiple diffusion flows on the community-small and Enzymes datasets. Specifically,  we report the average time (in seconds) for the training of a single epoch for Twigs with one and three secondary diffusion flows.\nWe observe that our models encounter a small overhead compared to GDSS and Digress, however, we believe it is a good tradeoff for performance.\n\n\n| Dataset         | Twigs p=1 | Twigs p=3 | GDSS   | Digress |\n| --------------- | --------- | --------- | ------ | ------- |\n| Community-small | 0.2747    | 0.2997    | 0.2294 | 0.2382  |\n| Enzymes         | 4.8669    | 5.0304    | 4.8260 | 4.8451  |\n\n\n**Final comment**\n\nWe are grateful for your thoughtful review. We hope our response has addressed your questions and concerns, and will appreciate the same being reflected in your stronger support for this work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents Twigs, which aims to enhance the representation power of graph generators by focusing on the relationships between graph structure and target properties. Twigs introduces a hierarchy of diffusion flows, comprising multiple Stem processes and a Trunk process. The Stem processes focus on disentangling the interactions between the graph structure and individual properties, while the Trunk process orchestrates these contributions to generate conditional representations. The experimental results demonstrate that Twigs outperforms existing guidance methods, such as MOOD and JODO, in diffusion-based graph generation.

**Strengths:**
1. Reviewers liked the hierarchical diffusion processes, which is innovative and addresses the challenge of capturing complex interactions between graph structure and target properties.
2. The loop guidance mechanism, which iteratively refines the contributions from Stem to Trunk processes, is well-motivated.
3. The method's performance improvements over existing approaches, particularly MOOD and JODO, are OK.

**Weaknesses:**
1. The writing in the main paper needs some improvement. Reviewers initially provided negative feedback due to the lack of detail in Section 3 and incomplete information about the method in Section 4. The rebuttal successfully addressed these concerns, and these clarifications should be incorporated into the main text. My recommendation assumes the authors will make the necessary changes to the paper to improve its readability.
2. The scalability on the number of properties is a limitation of the method that needs to be better discussed in the main paper.
3. The comparison with baseline methods was less-than-ideal in the original version. Please add the new experiments to the paper.

**Recommendation:**
Overall, Twigs represents a good contribution to score-based diffusion graph generation, with multiple co-evolving flows to capture complex interactions and dependencies for enriching conditional generation tasks. With some additional clarity on the theoretical aspects and computational considerations, this work could have greater impact. I recommend acceptance under the assumption the authors will do the promised improvements to the paper (i.e., camera-ready will take time and effort). If the paper is accepted, I will check to see if the authors made the promised changes.
Decision: Accept (poster)