Title: Graph Diffusion Transformers for Multi-Conditional Molecular Generation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Graph Diffusion Transformers

Inverse Molecular Design

Molecular Generation

Polymer Generation
Abstract: Inverse molecular design with diffusion models holds great potential for advancements in material and drug discovery. Despite success in unconditional molecule generation, integrating multiple properties such as synthetic score and gas permeability as condition constraints into diffusion models remains unexplored. We present the Graph Diffusion Transformer (Graph DiT) for multi-conditional molecular generation. Graph DiT has a condition encoder to learn the representation of numerical and categorical properties and utilizes a Transformer-based graph denoiser to achieve molecular graph denoising under conditions. Unlike previous graph diffusion models that add noise separately on the atoms and bonds in the forward diffusion process, we propose a graph-dependent noise model for training Graph DiT, designed to accurately estimate graph-related noise in molecules. We extensively validate the Graph DiT for multi-conditional polymer and small molecule generation. Results demonstrate our superiority across metrics from distribution learning to condition control for molecular properties. A polymer inverse design task for gas separation with feedback from domain experts further demonstrates its practical utility. The code is available at https://github.com/liugangcode/Graph-DiT.

Graph Diffusion Transformers for Multi-Conditional Molecular Generation

Abstract

Inverse molecular design with diffusion models holds great potential for advancements in material and drug discovery. Despite success in unconditional molecular generation, integrating multiple properties such as synthetic score and gas permeability as condition constraints into diffusion models remains unexplored. We present the Graph Diffusion Transformer (Graph DiT) for multi-conditional molecular generation. Graph DiT integrates an encoder to learn numerical and categorical property representations with the Transformer-based denoiser. Unlike previous graph diffusion models that add noise separately on the atoms and bonds in the forward diffusion process, Graph DiT is trained with a novel graph-dependent noise model for accurate estimation of graph-related noise in molecules. We extensively validate Graph DiT for multi-conditional polymer and small molecule generation. Results demonstrate the superiority of Graph DiT across nine metrics from distribution learning to condition control for molecular properties. A polymer inverse design task for gas separation with feedback from domain experts further demonstrates its practical utility.

1 Introduction

Diffusion models for molecular graphs are essential for inverse design of materials and drugs by generating molecules and polymers (macro-molecules) [40, 46], because the models can be effectively trained to predict discrete graph structures and atom/bond types in denoising processes [43]. Practical inverse designs consider multiple factors such as molecular synthetic score and various properties [15], known as the task of multi-conditional graph generation.

Existing work converted multiple conditions into a single one and solved the task as single-condition generation [5, 25]. However, multi-property relations may not be properly or explicitly defined [5]. First, the properties have diverse scales and units. For example, the synthetic complexity ranges from 1 to 5 [8], while the gas permeability varies widely, exceeding 10,000 in Barrier units [4]. This gap makes it hard for models to balance the conditions. Second, multi-conditions consist of a mix of categorical and numerical properties. The common practice of addition [47] or multiplication [25] is inadequate for combination.

Figure 1(a) empirically illustrates the challenges in multi-conditional generation, i.e., discovering molecules meeting multiple properties. We used a test set of 100 data points with three properties: synthesizability (Synth.) [12], O 2 and N 2 permeability (O 2 Perm and N 2 Perm) [4]. A single-conditional diffusion model generated up to 30 graphs for each condition, resulting in a total of 90 graphs for three conditions. We sort the 30 graphs in each set using a polymer property Oracle (see appendix B.3). Then, we check whether a shared polymer structure that meets multi-property constraints can be identified across different condition sets. If we find the polymer, its rank K (where K is between 1 and 30) indicates how high it appears on the lists, considering all condition sets. If not, we set K as 30. Figure 1(a) shows the frequency distribution of K on the 100 test cases. The median K was

Synth.

Ranking

Median:

30

Generated

Generated

Generated

…

…

…

Shared

from Top

K

If  found:

K

else:

30

721

H

TestExamples

64

56

48 40 324

#

16

8

0

0

5

10

15

20

25

30

Synth.

Ranking …

…

…

K

K

…

Median=

4

Median=

11

Median=

9

K

Test Examples

22

9

Examples

10

10

15

02

25

30

10

15

25

30

Q

O

10

15

20

25

30

30, indicating that the multiple properties were not met on over half of the test polymers despite generating a large number of graphs.

To address these challenges, we project multi-properties into representations by learning , thereby guiding the diffusion process for molecule generation. We propose the Graph Diffusion Transformer (Graph DiT) for graph denoising under conditions. Graph DiT has a condition encoder for property representation learning and a graph denoiser. The condition encoder utilizes a novel clustering-based method for numerical properties and one-hot encoding for categorical ones to learn multi-property representations. The graph denoiser first integrates node and edge features into graph tokens, then denoise these tokens with adaptive layer normalization (AdaLN) in Transformer layers [19, 34]. AdaLN replaces the molecular statistics (mean and variance) in each hidden layer with those from the condition representation, effectively outperforming other predictor-based and predictor-free conditioning methods [22, 43, 34], as shown in Section 4.4. We observe that existing forward diffusion processes [43, 22] apply noise separately to atoms and bonds, which may compromise the accuracy of Graph DiT in noise estimation. Hence, we propose a novel graph-dependent noise model that effectively applies noise tailored to the dependencies between atoms and bonds within the graph.

Results in Figure 1(b) show that the polymers generated by Graph DiT closely align with multiproperty constraints. For each test case, we have one graph generated from Graph DiT conditional on three properties. The Oracle determines the rank of this graph among 30 single-conditionally generated graphs for each condition. We find the median ranks are 4, 9, and 11, for Synth., O 2 Perm, and N 2 Perm, respectively, all much higher than 30. Note that the ranked set of 30 graphs was very competitive because the model was trained on the specific condition dedicatedly.

In experiments, we evaluate model performance on one polymer and three small molecule datasets. The polymer dataset includes four numerical conditions for multi-conditional evaluation. Our model has the lowest average mean absolute error (MAE), significantly reducing the error by 17.86% compared to the best baseline. It also excels in small molecule tasks, achieving over 0.9 accuracy on task-related categorical conditions, notably surpassing the baseline accuracy of less than 0.6. We also examine the model's utility in inverse polymer designs for O 2 /N 2 gas separation, with domain expert feedback highlighting our model's practical utility in multi-conditional molecular design.

2 Problem Definition

2.1 Multi-Conditional Inverse Molecular Design

A molecular graph G = ( V, E ) consists of a set of nodes (atoms) V and edges (bonds) E . We follow [43] and define 'non-bond' as a type of edge. There are N atoms and each atom has a one-hot encoding, denoting the atom type. We represent it as X V ∈ R N × F V , where F V is the total number of atom types. Similarly, the bond features are a tensor X E ∈ R N × N × F E , representing both the graph structure and F E bond types.

Let C = { c 1 , c 2 , . . . , c M } be a set of M numerical and categorical conditions. The task is: q ( G | c 1 , c 2 , . . . , c M ) ∝ q ( G ) q ( c 1 , c 2 , . . . , c M | G ) , where q represents observed probability. We use a model parameterized by θ for multi-conditional molecular generation p θ ( G | C ) . The evaluation involves both distribution learning q ( G ) [35] and condition control q ( c 1 , c 2 , . . . , c M | G ) . We follow previous work in assuming that there exist different oracle functions O that can independently evaluate each conditioned property [14]: q ( c 1 , c 2 , . . . , c M | G ) = ∏ M i =1 O i ( c i | G ) . Note that the oracles are not used in the training of p θ .

2.2 Diffusion Model on Graph Data

Diffusion models consist of forward and reverse diffusion processes [17]. We refer to the forward diffusion process as the diffusion process following [17]. The diffusion process q ( G 1: T | G 0 ) = ∏ T t =1 q ( G t | G t -1 ) corrupts molecular graph data ( G 0 = G ) into noisy states G t . As timesteps T →∞ , q ( G T ) converges a stationary distribution π ( G ) . The reverse Markov process p θ ( G 0: T ) = q ( G T ) ∏ T t =1 p θ ( G t -1 | G t ) , parameterized by neural networks, gradually denoises the latent states toward the desired data distribution.

Diffusion Process One may perturb G t in a discrete state-space to capture the structural properties of molecules [43]. Two transition matrices Q V ∈ R F V × F V and Q E ∈ R F E × F E are defined for nodes X V and edges X E , respectively [43]. Then, each step q ( G t | G t -1 , G 0 ) = q ( G t | G t -1 ) in the diffusion process is sampled as follows.

where Cat( X ; p ) denotes sampling from a categorical distribution with probability p . We remove the subscript ( V/E ) when the description applies to both nodes and edges. It is assumed that the noise Q i ( i ≤ t ) is independently applied to X in each step i , allowing us to rewrite q ( X t | X t -1 ) as the probability of the initial state q ( X t | X 0 ) = Cat ( X t ; p = X 0 ¯ Q t ) , where ¯ Q t = ∏ i ≤ t Q i .

Noise Scheduling Transition matrices Q V and Q E control the noise applied to atom features and bond features, respectively. Vignac et al. [43] defined π ( G ) = ( m X ∈ R F V , m E ∈ R F E ) as the marginal distributions of atom types and bond types. The transition matrix at timestep t is Q t = α t I +(1 -α t ) 1m ′ for atoms or bonds, where m ′ denotes the transposed row vector. Therefore, we have ¯ Q t = ¯ α t I +(1 -¯ α t ) 1m ′ , where ¯ α t = ∏ t τ =1 α τ . The cosine schedule [32] is often chosen for ¯ α t = cos(0 . 5 π ( t/T + s ) / (1 + s )) 2 .

Reverse Process With the initial sampling G T ∼ π ( G ) , the reverse process generates G 0 iteratively in reversed steps t = T, T -1 , . . . , 0 . We use a neural network to predict the probability p θ ( ˜ G 0 | G t ) as the product over nodes and edges [1, 43]:

p θ ( ˜ G 0 | G t ) could be combined with q ( G t -1 | G t , G 0 ) to estimate the reverse distribution on the graph p θ ( G t -1 | G t ) . For example, p θ ( v t -1 | G t ) is marginalized over predictions of node types ˜ v ∈ ˜ x v , which applies similarly to edges:

The neural network could be trained to minimize the negative log-likelihood [43].

Condition

Encoder

…

𝑞

𝐺

!"#

#

𝐺

$

, 𝐺

!

𝑝

%

( #

𝐺

$

∣ 𝐺

!

, 𝒞)

𝒞

T

t

t-1

0

Representations

of Conditions

𝐺

*

#

𝐺

+

Adaptive Layer Normalization

with Conditions' Statistics

…

𝒞

𝐺

,

𝑀

properties

…

…

Layer

Layer

…

…

Graph

Denoiser

Clustering

Encoding

One-Hot

Encoding

Numerical

Categorical

(b) Condition Encoder

(a) Denoising Process

Multi-Head

Attention

AdaLN

MLP

AdaLN

(with Transformer Layers)

where x ∈ G 0 denotes the node or edge features. Typically, the reverse process in diffusion models does not consider molecular properties as conditions. While there have been efforts to introduce property-related guidance using additional predictors, the more promising approach of predictor-free guidance [16], particularly in multi-conditional generation, remains underexplored.

3 Multi-Conditional Graph Diffusion Transformers

We present the denoising framework of Graph DiT in Figure 2. The condition encoder learns the representation of M conditions. The statistics of this representation like mean and variance are used to replace the ones from the molecular representations [19] (see Section 3.2). Besides, we introduce a new noise model in the diffusion process to better fit graph-structured molecules (see Section 3.1).

3.1 Graph-Dependent Noise Models

The transition probability of a node or an edge should rely on the joint distribution of nodes and edges in the prior state. However, as an example shown in Eq. (1), current diffusion models [22, 43, 25] treat node and edge state transitions as independent, misaligning with the denoising process in Eq. (3). This difference between the sampling distributions of noise in the diffusion and reverse processes introduces unnecessary challenges to multi-conditional molecular generations.

To address this, we use a single matrix X G ∈ R N × F G to represent graph tokens for G , with F G = F V + N · F E . Token representations are created by concatenating the node feature matrix X V and the flattened edge connection matrix from X E . Each row vector in X G contains features for both nodes and edges, representing all connections and non-connections. Hence, we could design a transition matrix Q G considering the joint distribution of nodes and edges. Q G ∈ R F G × F G is constructed from four matrices Q V , Q EV ∈ R F E × F V , Q E , Q V E ∈ R F V × F E , denoting the transition probability ('dependent old state' → 'target new state' ) node → node ; edge → node ; edge → edge ; node → edge, respectively.

where ⊗ denotes the Kronecker product, 1 N , 1 ′ N , and 1 N × N represent the column vector, row vector, and matrix with all 1 elements, respectively. According to Eq. (5), the first F V columns in Q G determine the node feature transitions based on both node features (first F V rows) and edge features (remaining N · F E rows). Conversely, the remaining N · F E columns determine the edge feature transitions, depending on the entire graph. We introduce a new diffusion noise model:

where ˜ p is the unnormalized probability and ˜ Cat denotes categorical sampling: The first F V columns of ˜ p are normalized to sample X t V , while the remaining N · E dimensions are reshaped and normalized to sample edges X t E . These components are combined to form X t G , completing the ˜ Cat sampling.

Choice of Q V E and Q EV Similar to the definitions of m V and m E [43], we leverage the prior knowledge within the training data for the formulation of task-specific matrices, Q EV and Q V E . We calculate co-occurrence frequencies of atom and bond types in training molecular graphs to obtain the marginal atom-bond co-occurrence probability distribution. For each bond type, each row in m EV represents the probability of co-occurring atom types. m V E is the transpose of m EV and has a similar meaning. Subsequently, we define Q EV = ¯ α t I +(1 -¯ α t ) 1m ′ EV and Q V E = ¯ α t I +(1 -¯ α t ) 1m ′ V E .

3.2 Denoising Models with Multi-Property Conditions

We present Graph DiT as the denoising model to generate molecules under multi-conditions C = { c 1 , c 2 , . . . , c M } without extra predictors.

Predictor-Free Guidance The predictor-free reverse process ˆ p θ ( G t -1 | G t , C ) aims to generate molecules with a high probability q ( C | G 0 ) . This could be achieved by a linear combination of the log probability for unconditional and conditional denoising [16]:

where s denotes the scale of conditional guidance. Unlike classifier-free guidance [16], which typically predicts noise, we directly estimate p θ ( ˜ G 0 | G t , C ) . We one one denoising model f θ ( G t , C ) for both p θ ( ˜ G 0 | G t ) and p θ ( ˜ G 0 | G t , C ) . Here, f θ ( G t , C = ∅ ) computes the unconditional probability by substituting the original conditional embeddings with the null value. During training, we randomly drop the condition with a ratio, i.e., C = ∅ , to learn the embedding of the null value. f θ ( G t = X t G , C ) comprises two components: the condition encoder and the graph denoiser. An overview of the architecture is presented in Figure 2.

Condition Encoder We treat the timestep t as a special condition and follow [31] to obtain a D -dimensional representation t with sinusoidal encoding. For property-related numerical or categorical condition c i ∈ C , we apply distinct encoding operations to get D -dimensional representation. For a categorical condition, we use the one-hot encoding. For a numerical variable, we introduce a clustering encoding method. This defines learnable centroids, assigning c i to clusters, and transforming the soft assignment vector of condition values into the representation. It could be implemented using two Linear layers and a Softmax layer in the middle as: Linear (Softmax (Linear( c i ))) . Finally, we could obtain the representation of the condition as c = ∑ M i =1 encode( c i ) , where encode is the specific encoding method based on the condition type. For numerical conditions, we evaluate our proposed clustering-based approach against alternatives like direct or interval-based encodings [28]. As noted in Section 4.4, the clustering encoding outperforms the other methods.

Graph Denoiser: Transformer Layers Given the noisy graph at timestep t , the graph tokens are first encoded into the hidden space as H = Linear( X t G ) , where H ∈ R N × D . We then adapt the standard Transformer layers [42] with self-attention and multi-layer perceptrons (MLP), but replace the normalization with the adaptive layer normalization ( AdaLN ) controlled by the representations of the conditions [19, 34]: H = AdaLN( H , c ). For each row h in H :

where µ ( · ) and σ ( · ) are mean and variance values. ⊙ indicates element-wise product. γ θ ( · ) and β θ ( · ) are neural network modules in f θ ( · ) , each of which consists of two linear layers with SiLU activation [11] in the middle. We have a gated variant AdaLN gate for residuals:

We apply the zero initialization for the first layer of γ θ ( · ) , β θ ( · ) , and α θ ( · ) [34]. There are other options to learn the structure representation from the condition [34]: In-Context conditioning adds condition representation to the structure representation at the beginning of the structure encoder, and Cross-Attention calculates cross-attention between the condition and structure representation. We observe in Section 4.4 that AdaLN performs best among them.

Graph Denoiser: Final MLP We have the hidden states H after the final Transformer layers, the MLP is used to predict node probabilities ˜ X 0 V and edge probabilities ˜ X 0 E at t = 0 :

We split the output X G into atom and bond features ˜ X 0 V , ˜ X 0 E . The first F V dimensions of ˜ X 0 G represent node type probabilities, and the remaining N · F E dimensions cover probabilities for N edge types associated with the node, as detailed in Section 3.1.

Generation to Molecule Conversion A common way of converting generated graphs to molecules selects only the largest connected component [43], denoted as Graph DiT-LCC in our model. For Graph DiT, we connect all components by randomly selecting atoms. It minimally alters the generated structure to more accurately reflect model performance than Graph DiT-LCC.

4 Experiment

RQ1 : We validate the generative power of Graph DiT compared to baselines from molecular optimization and diffusion models in Section 4.2. RQ2 : We study a polymer inverse design for gas separation in Section 4.3. RQ3 : We conduct further analysis to examine Graph DiT in Section 4.4.

4.1 Experimental Setup

We use datasets with over ten types of atoms and up to fifty nodes in a molecular graph. We include both numerical and categorical properties for drugs and materials, offering a benchmark for evaluation across diverse chemical spaces. Model performance is validated across up to nine metrics, including distribution coverage, diversity, and condition control capacity for various properties.

Datasets and Input Conditions We have one polymer dataset [40] for materials, featuring three numerical gas permeability conditions: O 2 Perm, CO 2 Perm, and N 2 Perm. For drug design, we create three class-balanced datasets from MoleculeNet [46]: HIV, BBBP, and BACE, each with a categorical property related to HIV virus replication inhibition, blood-brain barrier permeability, or human β -secretase 1 inhibition, respectively. We have two more numerical conditions for synthesizability from synthetic accessibility (SAS) and complexity scores (SCS) [12, 8].

Evaluation We randomly split the dataset into training, validation, and testing (reference) sets in a 6:2:2 ratio. Evaluations are conducted on 10,000 generated examples with metrics [35] (1) molecular validity (Validity); (2) heavy atom type coverage (Coverage); (3) internal diversity among the generated examples (Diversity); (4) fragment-based similarity with the reference set (Similarity); (5) Fr´ echet ChemNet Distance with the reference set (Distance) [36]; MAE between the generated and conditioned (6) synthetic accessibility score [12] (Synth.); (7) ∼ (9) MAE/Accuracy for the numerical/categorical task conditions (Property). The evaluation Oracle uses random forest trained on all task-related molecules [14]. Lower MAE or higher accuracy indicates stronger model controllability.

Baselines We select strong and popular molecular optimization baselines from recent studies [14]: Graph-GA [20], MARS [47], JTVAE [21] with Bayesian optimization (JTVAE-BO), LSTM [6] on SMILES with Hill Climbing (LSTM-HC). We include the most recent diffusion models: GDSS[22], DiGress [43], and their conditional version with extra predictors: MOOD [25], and DiGress v2 [43]. We train multi-task predictors using the same architecture for MOOD and DiGress v2 models to provide additional guidance for generation. For molecular optimization, we formulate the condition set of each test data point as a combined goal, minimizing the sum of the normalized errors between generated and input properties. We train a random forest model for each property using the training data to optimize the molecular structure.

4.2 RQ1: Multi-Conditional Molecular Generation

We have the observations from Table 1 and Table 2:

Chemical Validity High validity may not accurately represent the model's generative performance if hard-coded rules are introduced in the algorithm. For example, GraphGA could eliminate non-valid molecules during mutation and crossover iterations to achieve perfect validity in the final evaluation. Without rule checking in the generation-to-molecule step, DiGress, GDSS, and MOOD show a marked performance decline, with validity often dropping from 0.99 to below 0.6. In contrast, Graph DiT often maintains over 0.8 validity without any rule-based processing.

Distribution Learning GraphGA is a simple yet effective baseline for generating in-distribution molecules, e.g., on BBBP and HIV generation datasets. Diffusion model baselines such as DiGress and MOOD could produce diverse molecules but often fail to capture the original data distribution in multi-conditional tasks. Graph DiT shows the competitive performance of diffusion models in fitting complex molecular data distributions. Using fragment-based similarity and neural network-based distance metrics [36], we achieve the best in the polymer task and rank second in the HIV small molecule task, involving up to 11 and 29 types of heavy atoms, respectively.

Condition Controllability LSTM-HC surpasses many baselines, achieving lower average MAE on polymer properties and higher rankings on small molecular properties. However, its control over synthetic scores in polymer tasks is relatively poor. Conversely, MARS effectively manages synthetic scores for polymers but exhibits a larger MAE in gas permeability conditions compared to other baselines. GDSS performs well in gas permeability control but underperforms Graph GA and MARS in terms of the synthetic score condition. DiGress v2 and MOOD, although equipped with the predictor guidance, still exhibit limited condition control compared to their unconditional counterparts over polymer and small molecule tasks. These baselines struggle to balance and control multiple conditions in generation. In contrast, Graph DiT significantly improves diffusion models and achieves the best multi-conditional performance in all tasks. In polymer tasks, Graph DiT reduces MAE on all gas permeability conditions, averaging +17.8% improvement over the best baseline LSTM-HC. For small molecule tasks, Graph DiT consistently ranks top-1 in condition controllability with over 0.9 accuracy in categorical conditions. Compared to Graph DiT-LCC, we observe that Graph DiT, which connects all generated graph components, shows better controllability performance due to minimal rule-based post-generation processing.

4.3 RQ2: Polymer Inverse Design for Gas Separation

We aim to design polymers with high O 2 and low N 2 permeability, demonstrating the models' precise control over related properties. Following Robeson [38]'s definition of high-performance polymers based on the O 2 /N 2 permeability ratio, we selected 16 polymers meeting this criterion from 609 examples as our test/reference set. The remaining data is used for training and validation. Subsequently, we generated 1,000 polymers conditioned on test set labels.

UtS: 0.75 / AS: 1.00

Rank:

(1)

UtS: 0.49 / AS: 0.97

Rank:

(2)

UtS: 0.41 / AS: 0.87

Rank:

(3)

UtS: 0.13 / AS: 1.00

Rank: (9)

UtS: 0.07 / AS: 0.98

Rank: (13)

UtS: 0.09 / AS: 0.99

Rank: (11)

UtS: 0.08 / AS: 0.94

Rank: (12)

UtS: 0.04 / AS: 1.00

Rank: (15)

UtS: 0.06 / AS: 0.21

Rank: (14)

UtS: 0.2 / AS: 0.97

Rank: (4)

UtS: 0.19 / AS: 0.93

Rank: (5)

UtS: 0.11 / AS: 0.96

Rank: (10)

UtS: 0.14 / AS: 0.70

Rank: (8)

UtS: 0.16 / AS: 1.00

Rank: (6)

UtS: 0.16 / AS: 0.77

Rank: (6)

(a) MARS

(b) LSTM-HC

(c) DiGress

(d) MOOD

(e) Ours

1.00

0.76

0.53

1.00

0.69

0.48

1.00

0.76

0.54

1.00

1.02

1.03

1.00

0.98

0.59

1.00

0.21

0.06

O

2

Perm

N

2

Perm

CO

2

Perm

Diversity

Similarity

Distance

Cluster

Direct

Interval

1.00

0.59

0.57

1.00

0.56

0.53

1.00

0.61

0.59

1.00

1.04

1.03

1.00

0.60

0.57

1.00

0.05

0.06

O

2

Perm

N

2

Perm

CO

2

Perm

Diversity

Similarity

Distance

AdaLN

In-Context

Cross-Attn

1.00

0.53

0.51

1.00

0.49

0.46

1.00

0.55

0.52

1.00

1.01

1.04

1.00

0.95

0.59

1.00

0.06 0.24

O

2

Perm

N

2

Perm

CO

2

Perm

Diversity

Similarity

Distance

Depedent

Separate

DiGress v2

(a) Numerical Condition Encodings

(b) Condition Architectures

(c) Graph Dependent Noise Model

In Figure 3, we present the top three polymers generated by each model for a case study with expertise. Initially, a random forest algorithm identifies the top five polymers per method based on average MAE in two gas permeability. These 25 polymers are then shuffled and evaluated by four polymer scientists, who rank them from 1 to 25 using their domain knowledge. Rankings are normalized to a Utility Score (UtS) ranging from 0 to 1, with higher scores indicating greater utility. The variance in UtS is converted into an Agreement Score (AS) for further evaluation. As shown in Figure 3, there is a high consensus among experts that the three polymers generated by Graph DiT are the most promising for successful polymer inverse design tasks. More details are in appendix D. By comparing generated examples from different models, we have further observations:

DiGress and MOOD struggle to capture polymerization points, marked with asterisks ('*'), which is one of the most important features that distinguish polymers from small molecules. Additionally, the two methods frequently feature excessive carbon atoms and overly large cycles. These molecular configurations with significant distortion from the canonical geometry of stable compounds may lead to poor synthesizability [35, 7].

LSTM-HC may result in too-long carbon chains with limited diversity. MARS produces examples with asymmetrical graph structures, challenging polymer synthesis [10, 2].

Graph DiT generates structurally diverse and symmetric polymers with two polymerization points, indicative of more valid and synthesizable polymer structures. The first two, which are polyimides, imply effective gas separation performance [24].

4.4 RQ3: Ablation Studies and Model Analysis

Model Components In light of Table 1, we analyze three components that impact our model's learning in various conditions. Our assessment of relative performance is based on the ratio between our method and comparative approaches. The first component is numerical conditional encoding . Results in Figure 4(a) highlight the superiority of clustering encoding over direct and interval-based encoding, particularly in controlling gas permeability, despite its slightly lower diversity. The second component concerns the neural architecture for conditions . As shown in Figure 4(b), similar to Figure 4(a), AdaLN surpasses both In-Context Conditioning and Cross-Attention in learning distribution with better condition controllability. The third component validates the importance of the graph-dependent noise model compared to separately applying noise to atoms and bonds. It also shows the improvement of the predictor-free Graph DiT over the predictor-guided DiGress v2, even without the graph-dependent noise model. More results on model controllability are in appendix E.

Oracle Selections We analyze the robustness of Oracles in evaluating six task-related properties (three gas permeability and three small molecule properties) across six conditional generation tasks. Oracles are switched from Random Forest to Gaussian Process or Support Vector Machines for ranking generative model performance. Results in Table 3 show consistent rankings (Graph DiT, LSTM-HC, MARS, JTVAE-BO, MOOD, GDSS, GraphGA). It indicates that while perfectly approximating the truth properties of generated molecules is difficult, we could effectively compare the relative performance of various models.

Graph DiT consistently ranked first among baselines.

5 Related Work

Diffusion Models for Molecules: Score-based diffusion models applied noise and denoising in continuous space [33, 22]. DiGress [43] used discrete noise as transition matrices based on marginal distributions of atom and bond types. Extra predictor models are studied to guide the generation process in DiGress and GDSS [25]. Diffusion models could also be used for molecular property prediction [27], for conformation [48] and molecule generation with 3D atomic coordinates [18, 49, 3]. We focus on molecular graph generation, considering the high computational cost of accurate 3D coordinates for larger molecules like polymers [23]. We explore predictor-free diffusion guidance, instead of the classifier guidance [9, 44], for generating molecules under categorical and numerical conditions. It can be integrated with diffusion models for atomic coordinates in future research.

Molecular Optimization: Optimization algorithms could optimize molecules towards property constraints, including genetic algorithms [20], Bayesian optimization [39, 50], REINFORCE [45], and reinforcement learning [30]. Both sequential and graph-based generative models [6, 21, 30], along with diverse sampling methods [47, 13], are used in conjunction with these algorithms to produce desirable molecules. These methods have been applied to both single-objective and multi-objective optimization, the latter by manually integrating multiple property conditions into a single one [5, 25]. Several challenges in molecular optimization methods remain underexplored, including the inadequate or unclear definition of multi-property relations when integration into a single objective [5], and the inaccessibility of the oracle function for property-oriented optimization during the training phase [14].

6 Conclusion

In this work, we solved inverse molecular design using properties as predictor-free diffusion guidance. The proposed Graph DiT performed diffusion based on the joint distribution of atoms and bonds in both forward and reverse processes. It introduced representation learning for multiple categorical and numerical properties and utilized a Transformer-based graph denoiser for conditional graph denoising. Results on multi-conditional generations and polymer inverse designs showed the remarkable generative capabilities of Graph DiT, making it suitable for designing promising molecules.

References

Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems , 34:17981-17993, 2021.

Alexandru T Balaban. Symmetry in chemical structures and reactions. In Symmetry , pages 999-1020. Elsevier, 1986.

Fan Bao, Min Zhao, Zhongkai Hao, Peiyao Li, Chongxuan Li, and Jun Zhu. Equivariant energy-guided SDE for inverse molecular design. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=r0otLtOwYW .

J Wesley Barnett, Connor R Bilchak, Yiwen Wang, Brian C Benicewicz, Laura A Murdock, Tristan Bereau, and Sanat K Kumar. Designing exceptional gas-separation polymer membranes using machine learning. Science advances , 6(20):eaaz4301, 2020.

Camille Bilodeau, Wengong Jin, Tommi Jaakkola, Regina Barzilay, and Klavs F Jensen. Generative models for molecular discovery: Recent advances and challenges. Wiley Interdisciplinary Reviews: Computational Molecular Science , 12(5):e1608, 2022.

Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling , 59(3):1096-1108, 2019.

Robert F Bruns and Ian A Watson. Rules for identifying potentially reactive or promiscuous compounds. Journal of medicinal chemistry , 55(22):9763-9772, 2012.

Connor W Coley, Luke Rogers, William H Green, and Klavs F Jensen. Scscore: synthetic complexity learned from a reaction corpus. Journal of chemical information and modeling , 58 (2):252-261, 2018.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Jack D Dunitz. Symmetry arguments in chemistry. Proceedings of the National Academy of Sciences , 93(25):14260-14266, 1996.

Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks , 107:3-11, 2018.

Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. Journal of cheminformatics , 1:1-11, 2009.

Tianfan Fu, Wenhao Gao, Cao Xiao, Jacob Yasonik, Connor W Coley, and Jimeng Sun. Differentiable scaffolding tree for molecular optimization. arXiv preprint arXiv:2109.10469 , 2021.

Wenhao Gao, Tianfan Fu, Jimeng Sun, and Connor Coley. Sample efficiency matters: a benchmark for practical molecular optimization. Advances in Neural Information Processing Systems , 35:21342-21357, 2022.

Niklas WA Gebauer, Michael Gastegger, Stefaan SP Hessmann, Klaus-Robert M¨ uller, and Kristof T Sch¨ utt. Inverse design of 3d molecular structures with conditional generative neural networks. Nature communications , 13(1):973, 2022.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Emiel Hoogeboom, Vıctor Garcia Satorras, Cl´ ement Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International conference on machine learning , pages 8867-8887. PMLR, 2022.

Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision , pages 1501-1510, 2017.

Jan H Jensen. A graph-based genetic algorithm and generative model/monte carlo tree search for the exploration of chemical space. Chemical science , 10(12):3567-3572, 2019.

Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference on machine learning , pages 2323-2332. PMLR, 2018.

Jaehyeong Jo, Seul Lee, and Sung Ju Hwang. Score-based generative modeling of graphs via the system of stochastic differential equations. In International Conference on Machine Learning , volume 162, pages 10362-10383. PMLR, 2022.

Soumil Y Joshi and Sanket A Deshmukh. A review of advancements in coarse-grained molecular dynamics simulations. Molecular Simulation , 47(10-11):786-803, 2021.

Michael Langsam. Polyimides for gas separation. In Polyimides , pages 697-742. CRC Press, 2018.

Seul Lee, Jaehyeong Jo, and Sung Ju Hwang. Exploring chemical space with score-based out-ofdistribution generation. In International Conference on Machine Learning , pages 18872-18892. PMLR, 2023.

Gang Liu, Tong Zhao, Jiaxin Xu, Tengfei Luo, and Meng Jiang. Graph rationalization with environment-based augmentations. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1069-1078, 2022.

Gang Liu, Eric Inae, Tong Zhao, Jiaxin Xu, Tengfei Luo, and Meng Jiang. Data-centric learning from unlabeled graphs with diffusion model. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id=DmakwvCJ7l .

Gang Liu, Tong Zhao, Eric Inae, Tengfei Luo, and Meng Jiang. Semi-supervised graph imbalanced regression. In 29th SIGKDD Conference on Knowledge Discovery and Data Mining , 2023.

Ruimin Ma and Tengfei Luo. Pi1m: a benchmark database for polymer informatics. Journal of Chemical Information and Modeling , 60(10):4684-4690, 2020.

Amina Mollaysa, Brooks Paige, and Alexandros Kalousis. Goal-directed generation of discrete structures with conditional generative models. Advances in Neural Information Processing Systems , 33:21923-21933, 2020.

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162-8171. PMLR, 2021.

Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permutation invariant graph generation via score-based generative modeling. In International Conference on Artificial Intelligence and Statistics , pages 4474-4484. PMLR, 2020.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4195-4205, 2023.

Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, et al. Molecular sets (moses): a benchmarking platform for molecular generation models. Frontiers in pharmacology , 11:565644, 2020.

Kristina Preuer, Philipp Renz, Thomas Unterthiner, Sepp Hochreiter, and Gunter Klambauer. Fr´ echet chemnet distance: a metric for generative models for molecules in drug discovery. Journal of chemical information and modeling , 58(9):1736-1741, 2018.

Philipp Renz, Dries Van Rompaey, J¨ org Kurt Wegner, Sepp Hochreiter, and G¨ unter Klambauer. Onfailure modes in molecule generation and optimization. Drug Discovery Today: Technologies , 32:55-63, 2019.

Lloyd M Robeson. The upper bound revisited. Journal of membrane science , 320(1-2):390-400, 2008.

Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE , 104 (1):148-175, 2015.

A Thornton, L Robeson, B Freeman, and D Uhlmann. Polymer gas separation membrane database, 2012.

Austin Tripp and Jos´ e Miguel Hern´ andez-Lobato. Genetic algorithms are strong baselines for molecule generation. arXiv preprint arXiv:2310.09267 , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209.14734 , 2022.

Tomer Weiss, Eduardo Mayo Yanes, Sabyasachi Chakraborty, Luca Cosmo, Alex M Bronstein, and Renana Gershoni-Poranne. Guided diffusion for inverse molecular design. Nature Computational Science , pages 1-10, 2023.

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8:229-256, 1992.

Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science , 9(2):513-530, 2018.

Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. Mars: Markov molecular sampling for multi-objective drug discovery. arXiv preprint arXiv:2103.10432 , 2021.

Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id= PzcvxEMzvQC .

Minkai Xu, Alexander S Powers, Ron O Dror, Stefano Ermon, and Jure Leskovec. Geometric latent diffusion models for 3d molecule generation. In International Conference on Machine Learning , pages 38592-38610. PMLR, 2023.

Yiheng Zhu, Jialu Wu, Chaowen Hu, Jiahuan Yan, Tingjun Hou, Jian Wu, et al. Sampleefficient multi-objective molecular optimization with gflownets. Advances in Neural Information Processing Systems , 36, 2024.

A Details on the Denoising Model Component

A.1 Numerical Condition Encoding

We explore several approaches for encoding numerical conditions. In addition to the clustering-based method, we consider:

The direct encoding approach, which employs a linear layer to map a continuous number into a high-dimensional space.

The interval-based approach, as described in [28], divides the label space into N Interval intervals. It then converts the number into an interval index, allowing us to apply one-hot encoding for the number.

A.2 Neural Architecture for Conditions

Besides the AdaLN , there are two more options to integrate condition representation into molecular graph representations [34]:

The In-Context conditioning approach adds the condition representation c to each row of the molecular graph representation H after mapping the X t G into H using the linear layer in the structure encoder.

The Cross-Attention approach concatenates the timestep encoding vector with the condition representation from synthesis scores or task-related properties into a two-length sequence. In each Transformer encoder layer, this is followed by a cross-attention layer at the end of the standard multi-head self-attention layer.

B Details on Datasets and Evaluation Methods

All experiments can be run on a single A6000 GPU card.

B.1 Datasets and Task Conditions

As presented in Table 4, we collect popular datasets in prediction tasks for more challenging molecular generation tasks. We include a polymer dataset [40, 26] for material design. It consists of conditions of O 2 , CO 2 , and N 2 , which study the numerical gas permeability for oxygen, carbon dioxide, and nitrogen, respectively. Additionally, we also study the generative performance of different models separately on the polymer data with O 2 , CO 2 , or N 2 , as illustrated in Table 4. We also create three class-balanced molecule datasets from [46] for drug design: HIV, BBBP, and BACE, which study categorical properties related to the inhibition of HIV virus replication, blood-brain barrier permeability, and inhibition of human β -secretase 1, respectively. We aim to generate synthesizable

*

C

N

O

F

Si

P

S

Cl

Ge

Br

1e-05

0.0001

0.001

0.01

0.1

1.0

(a) Histogram for Heavy Atom Type Distribution

SINGLE

DOUBLE

TRIPLE

AROMATIC

0.0001

0.001

0.01

0.1

1.0

Training

MCD (Ours)

DiGress

DiGress v2

GDSS

MOOD

Graph GA

MARS

LSTM-HC

JTVAE-BO

molecules. Therefore, we add two numerical conditions for synthetic complexity scores [12, 8] for each of the tasks. For the gas separation polymer design task, we consider joint conditions for O 2 and N 2 and measure the selectivity O 2 /N 2 as the ratio between two gas permeability scores. All polymer gas permeabilities are scaled in the log space following previous work [29]. We focus on experiments for polymers and molecules within 50 nodes.

B.2 Evaluation and Metrics

We randomly split the dataset into training, validation, and testing (reference) sets in a 6:2:2 ratio. We investigate more than eight metrics to systematically evaluate the generation performance. First, we assess generation validity (Validity). Second, we evaluate the distribution learning capacity of different models by measuring heavy atom type coverage (Coverage), internal diversity among the generated examples using Tanimoto similarity (Diversity), fragment-based similarity with the reference set (Similarity), and the Fr´ echet ChemNet Distance with the reference set (Distance). Third, we evaluate the model's controllability by measuring the mean absolute error (MAE) between the generated condition score and the actual condition scores if the condition is numerical; otherwise, we measure the accuracy score. We follow previous work [14] to use the random forest trained on all the available data as the Oracle evaluation function. For molecular optimization algorithms, we train random forest predictors on the training set for conditional generation. For predictor-guided diffusion models, DiGress v2 and MOOD, we use the same architecture as their denoising models to train predictors for diffusion guidance. Given the conditions in the test set, we report the generation performance by generating 10,000 examples for the six multi-conditional generation tasks and 1,000 examples for the polymer inverse design problem focused on selectivity.

B.3 Datasets and Tasks in Figure 1

Using the same dataset from the O 2 /N 2 polymer inverse design task, we keep 100 polymers to provide condition sets for testing and split the rest into training and validation sets in a 0.65:0.35 ratio. We apply our proposed Graph DiT for both single-conditional and multi-conditional approaches, focusing on three properties: (1) Synth. score for synthesizability [12], (2) O 2 permeability, and (3) N 2 permeability. The single-conditional approach generates 30 polymers per condition for each test data point, totaling 9,000 polymers. In contrast, the multi-conditional approach generates 30 polymers for each set of conditions per test data point, resulting in 3,000 polymers. We rank these polymers based on the mean absolute error between the generated properties (evaluated by a random forest model trained on all the data to simulate the Oracle function) and the conditional property. For each test data point, we also rank the best multi-conditional polymer in different single-conditional

Median: 30

(a) Single-Conditional Generation

Median: 16

(b) Multi-Conditional Generation

72

12

# Test Examples

4 56 8 40 2 416

：TestExamples

10

8

6

4

#

2

8

0

0

0

5

10

15

20

25

30

0

5

10

15

20

25

30

sets. For the single-conditional approach, we identify a common polymer meeting various properties and visualize the minimum top K value distribution across all 100 test points.

In addition to Figure 1, we compute the median rankings of the multi-conditionally generated polymers within the single-conditional sets. The results are shown in Figure 6. Both Figure 1 and Figure 6 demonstrate the advantages of multi-conditional generation over single-conditional generation.

C Details on Multi-Conditional Generation Results

Weshow results for three polymer generation tasks in Table 1 and molecule generation tasks in Table 2. As complementary results for Table 1, we present new results on generation using only one gas permeability in Table 5. We also compare the distributions of atom and bond types between generated and training data in Figure 5. Furthermore, Figure 7 visualizes the two-dimensional molecular data distribution of both training and generated molecules across various generative models.

C.1 Discussion on Diffusion Model Baselines

While diffusion models like GDSS [22] and DiGress [43] show promise in unconditional tasks, their performance in multi-conditional generations needs improvement for fitting training distributions and achieving more controllable results. As indicated by Figure 5(a), the generation of GDSS [22] often collapses to carbon elements with Gaussian noise in the continuous diffusion state-space. MOOD[25] improves atom type coverage by adding predictor guidance and an out-of-distribution hyper-parameter, but it is hard to fit the training distribution, as visualized in Figure 7(h). DiGress and its predictor-guided variant [43] (i.e., DiGress v2), using discrete state-space and transition matrices for diffusion noise, outperform GDSS and MOOD in distribution fitting and internal diversity in Tables 1, 2 and 5. However, as indicated in Figure 7(e) and Figure 7(f), these two models still generate too many out-of-distribution examples without justification of the generalization capacity. While GDSS and MOOD show lower average MAE in polymer conditional generation tasks, their subpar distribution learning performance and the results from Table 2 suggest that this may be due to the carbon element, which may be a confounder and affect the evaluation of the correlation between the model and controllability in polymer tasks.

C.2 Discussion on Molecular Optimization Baselines

Popular molecular optimization baselines are competitive in molecular generation tasks. Earlier studies have noted their strong performance: Gao et al. [14] showed their effectiveness in the standard molecular optimizations with a combined optimization target, and Tripp and Hern´ andez-Lobato [41] found that genetic algorithm often outperforms recent methods in unconditional generation. We

observe their competitive performance in multi-conditional settings, characterized by high validity, good atom type coverage, and distribution similarity in Tables 1, 2 and 5. MARS [47] and JTV AE [21] perform well for controlling the synthetic accessibility score [12] but are less effective at controlling specific task properties like gas permeability, BACE, BBBP, and HIV. For example, in generation tasks with the categorical task condition, the generated examples only achieve around 50% accuracy in hitting the input condition.

(a) Graph GA

(b) MARS

(c) LSTM-HC

(d) JTVAE-BO

(g) GDSS

(h) MOOD

O Task

N Task

CO Task

O Task

N Task

CO Task

BBBP Task

BACE Task

HIV Task

BBBP Task

BACE Task

HIV Task

O Task

N Task

CO Task

O Task

N Task

CO Task

BBBP Task

BACE Task

HIV Task

BBBP Task

BACE Task

HIV Task

O Task

N Task

CO Task

O Task

N Task

CO Task

BBBP Task

BACE Task

XSL AIH

BBBP Task

BACE Task

HIV Task

O Task

N Task

CO Task

O Task

N Task

CO Task

BBBP Task

BACE Task

HIV Task

BBBP Task

BACE Task

HIV Task

O2Task

N2 Task

CO2Task

BBBPTask

BACE Task

HIV Task

C.3 Discussion on Training Dynamics

In Figure 8, we illustrate changes in various indicators on the validation set during model training. We note an increase in generated validity and similarity to the validation reference set, along with a decrease in distance to the reference set and errors between generated properties and conditions, indicating gradual improvement in conditional generation over epochs. However, a trade-off between distribution fitting and internal diversity is observed in our current model, suggesting that further work on enhancing generation diversity could be promising.

2000

4000

6000

8000

0.50

0.60

0.70

0.80

0.90

Percentage

Validity w/o Check

2000

4000

6000

8000

0.90

0.90

0.91

0.91

0.92

To Reference Set

Diversity

2000

4000

6000

8000

0.60

0.70

0.80

0.90

To Reference Set

Similarity

2000

4000

6000

8000

Epoch

6.00

8.00

10.00

12.00

14.00

16.00

To Reference Set

Distance

2000

4000

6000

8000

Epoch

2.30

2.40

2.50

2.60

2.70

2.80

2.90

3.00

3.10

MAE(SA)+MAE(SC)

Synthesizability

2000

4000

6000

8000

Epoch

0.80

0.90

1.00

1.10

1.20

1.30

1.40

1.50

MAE(Property)

Property

(a) O

2

Perm Only

2000

4000

6000

8000

10000

0.40

0.50

0.60

0.70

0.80

0.90

Percentage

Validity w/o Check

2000

4000

6000

8000

10000

0.89

0.89

0.90

0.90

0.91

0.91

To Reference Set

Diversity

2000

4000

6000

8000

10000

0.40

0.50

0.60

0.70

0.80

0.90

To Reference Set

Similarity

2000

4000

6000

8000

10000

Epoch

6.00

8.00

10.00

12.00

14.00

16.00

18.00

20.00

To Reference Set

Distance

2000

4000

6000

8000

10000

Epoch

2.40

2.60

2.80

3.00

3.20

MAE(SA)+MAE(SC)

Synthesizability

2000

4000

6000

8000

10000

Epoch

0.80

0.90

1.00

1.10

1.20

1.30

1.40

MAE(Property)

Property

(b) CO

2

Perm Only

2000

4000

6000

8000

10000

0.40

0.50

0.60

0.70

0.80

0.90

Percentage

Validity w/o Check

2000

4000

6000

8000

10000

0.88

0.89

0.89

0.90

0.90

0.91

0.91

0.92

To Reference Set

Diversity

2000

4000

6000

8000

10000

0.30

0.40

0.50

0.60

0.70

0.80

0.90

To Reference Set

Similarity

2000

4000

6000

8000

10000

Epoch

6.00

8.00

10.00

12.00

14.00

16.00

18.00

20.00

22.00

To Reference Set

Distance

2000

4000

6000

8000

10000

Epoch

2.40

2.60

2.80

3.00

3.20

MAE(SA)+MAE(SC)

Synthesizability

2000

4000

6000

8000

10000

Epoch

1.00

1.20

1.40

1.60

1.80

MAE(Property)

Property

(c) N

2

Perm Only

2000

4000

6000

8000

10000

0.20

0.40

0.60

0.80

Percentage

Validity w/o Check

2000

4000

6000

8000

10000

0.81

0.82

0.82

0.82

0.82

0.82

0.83

0.83

0.83

To Reference Set

Diversity

2000

4000

6000

8000

10000

0.65

0.70

0.75

0.80

0.85

To Reference Set

Similarity

2000

4000

6000

8000

10000

Epoch

8.00

10.00

12.00

14.00

16.00

18.00

20.00

22.00

To Reference Set

Distance

2000

4000

6000

8000

10000

Epoch

0.80

1.00

1.20

1.40

1.60

1.80

2.00

MAE(SA)+MAE(SC)

Synthesizability

2000

4000

6000

8000

10000

Epoch

0.60

0.70

0.80

0.90

Accuracy(Property)

Property

(d) BACE

2000

4000

6000

8000

10000

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

Percentage

Validity w/o Check

2000

4000

6000

8000

10000

0.89

0.89

0.90

0.90

0.91

To Reference Set

Diversity

2000

4000

6000

8000

10000

0.75

0.78

0.80

0.83

0.85

0.88

0.90

0.93

To Reference Set

Similarity

2000

4000

6000

8000

10000

Epoch

13.00

14.00

15.00

16.00

17.00

18.00

19.00

20.00

To Reference Set

Distance

2000

4000

6000

8000

10000

Epoch

1.50

1.75

2.00

2.25

2.50

2.75

3.00

MAE(SA)+MAE(SC)

Synthesizability

2000

4000

6000

8000

10000

Epoch

0.75

0.80

0.85

0.90

0.95

Accuracy(Property)

Property

(e) BBBP

2000

4000

6000

8000

10000

0.10

0.20

0.30

0.40

0.50

0.60

0.70

Percentage

Validity w/o Check

2000

4000

6000

8000

10000

0.89

0.90

0.90

0.91

0.91

0.92

0.92

0.93

To Reference Set

Diversity

2000

4000

6000

8000

10000

0.50

0.60

0.70

0.80

0.90

To Reference Set

Similarity

2000

4000

6000

8000

10000

Epoch

7.50

10.00

12.50

15.00

17.50

20.00

22.50

25.00

27.50

To Reference Set

Distance

2000

4000

6000

8000

10000

Epoch

1.00

1.25

1.50

1.75

2.00

2.25

2.50

2.75

3.00

MAE(SA)+MAE(SC)

Synthesizability

2000

4000

6000

8000

10000

Epoch

0.60

0.70

0.80

0.90

1.00

Accuracy(Property)

Property

(f) HIV

C.4 Discussion on Uniqueness, and Novelty

We evaluate the model's performance on Novelty and Uniqueness. Unlike unconditional generation, multi-conditional generation involves generating multiple possible molecules under the same condition. Therefore, we compute these metrics across different sets of conditions rather than for generated molecules under the same conditions. Results are presented in Table 7.

Graph DiT demonstrates reasonable performance on these metrics. However, higher Novelty and Uniqueness values do not necessarily indicate better performance, as they may not reflect the model's ability to design satisfactory molecules with desirable properties. Moreover, these values risk leading to misleading conclusions. For instance, AddCarbon achieves nearly perfect scores (99.94% Novelty

and 99.86% Uniqueness) according to [37, 41], yet it randomly adds carbon atoms to existing molecules, resulting in new molecules that are not practically useful [37].

D Details on Polymer Inverse Design

We aim to design polymers with high O 2 and low N 2 permeability, showing refined control of models over these properties. This is reflected in the selectivity, defined as the O 2 /N 2 permeability ratio. Robeson [38] has identified an inherent trade-off between gas permeability and selectivity, known as the upper-bound. Ideally, high-performance polymers should fall in the above-the-bound region, demonstrating an effective combination of permeability and selectivity. We have 609 polymers with annotated permeability values for both gases. 16 above-the-bound polymers are included in the test/reference set and excluded from the training set. We generate 1,000 polymers conditional on test set labels.

D.1 Survey Setup on Generated Polymers

Figure 9: The reference polymer structure in the case study has conditions { SAS=3.8, SCS=4.3, O 2 Perm=34.0, N 2 Perm=5.2 } .

We aim to gather expert evaluations on the generation performance of various methods. We conduct a study using a test data point from the O 2 /N 2 gas separation inverse design task, taking its properties as conditions. The structure of the selected data point is presented in Figure 9. We display 25 generated polymers, each with its properties, alongside three real polymers from the training dataset as references. The first real polymer serves as the test reference, while the other two, similar in properties to the first, also aid experts in assessing the generated polymers. The properties of these generated polymers are predicted using a well-trained random forest model. Experts are asked to rank the generated polymers from 1 to 25, considering: (1) Structures of real polymers with desirable properties; (2) Predicted properties of generated polymers, displayed beneath each visualization. Here, a rank of 1 represents the best example as per domain knowledge, while 25 is the least favorable. Rankings are then converted to utility scores (UtS) ranging from 0 to 1 using 1 ranking , allowing us to quantify the relative performance of different generation methods. The agreement score (AS) could be obtained by exp( -25 × Variance( UtS 1 , UtS 2 , UtS 3 , UtS 4 )) , where UtS i denotes the utility score from the i -th domain expert. (3) Finally, we select top-3 polymers for each generative models and present them in Figure 3.

D.2 Results on Inverse Design

We present the inverse design results of all 1,000 generated polymers in validity, distribution learning, and condition control in Table 6. # UB is the count of generated polymers successfully identified as upper bound instances. Higher # UB indicates that Graph DiT has a higher likelihood of generating candidates for excellent O 2 and N 2 gas separation. The smallest MAE across most properties and a 9.9% average MAE improvement over baselines highlight Graph DiT's superior control in generating examples closely aligned with multiple conditions.

0

200

400

600

800

1000

N

2 property values

0.6

0.7

0.8

0.9

Value from test set

Validity

(a) Changes of Validity

0

200

400

600

800

1000

N

2 property values

1.0

1.2

1.4

1.6

Value from test set

Synth

.

(b) Changes of MAE to the target Synthesizability

0

200

400

600

800

1000

N

2 property values

2

4

6

Value from test set

N

2

(c) Changes of MAE to the target

N

2

0

200

400

600

800

1000

N

2 property values

0.6

0.8

1.0

1.2

1.4

1.6

Value from test set

O

2

(d) Changes of MAE to the target

O

2

E Details and More Results on Model Analysis

E.1 Case Studies on Generation Controllability

We conduct a new case study on the O 2 /N 2 polymer dataset, studying the controllability on three properties synthesizability score, O 2 and N 2 properties with varied N 2 property values. We select a polymer example from the test set and vary its N 2 while keeping other properties fixed. The N 2 property from the test polymer is 213.75, and we vary it from 0 to 1000. We sample 50 values within this range and generate 100 polymer graphs conditioned on multiple properties with each sampled value. We evaluate various metrics, including the chemical validity of the generated polymers.

We visualize results in Figure 10. We consistently observe that validity and controllability performance improve as the values approach 213.75, derived from a real test polymer. Conversely, performance deteriorates when the sampled N 2 values are closer to the extremes of the sampling range (0 or 1000). This observation underscores the interdependency between conditions, where less frequent combinations of different properties may be more challenging to learn. Moreover, the model performs well across a elatively large range from 0 to 1000 in terms of validity, O 2 , and synthesizability score control. This demonstrates good generalization of the proposed method in capturing complex condition interdependencies.

1.00

0.61

1.00

0.58

1.00

0.62

1.00

0.99

1.00

1.00

1.00

0.86

O

2

Perm

N

2

Perm

CO

2

Perm

Diversity

Similarity

Distance

MLP

Linear

E.2 Ablation Studies on Final MLP

In addition to the three components related to conditioning effectiveness of Graph DiT studied in Section 4.4, we also examine the importance of the final layer MLP for conditional graph denoising. Results in Figure 11 show that MLP significantly outperforms a linear layer [34].

E.3 Details on Oracle Simulation

We train three types of Oracles based on Random Forest, Gaussian Process, and Support Vector Machine on all polymers or molecules in a task to evaluate the properties of generated polymers conditional on O 2 Perm only, N 2 Perm only, CO 2 Perm only, BACE, BBBP, or HIV. The training performance (MAE or AUC) is presented in Table 8. Our findings show that the random forest achieves the lowest MAE and highest AUC scores, leading us to select it for simulating oracles in our generation evaluation process.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Discussions on the proposed method and empirical support can be found in sections 1, 3, and 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Discussions on generation diversity and Oracle evaluations are found in Sections 4.2 and 4.4.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper focuses on empirical studies.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Details are in Section 4, appendix, and supplementary code.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code is provided in the supplementary materials. Data and code will be on Github after publication.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details are in Section 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Improvement percentages ( > 10%) are discussed in Section 4.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Discussions are in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conforms the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The work has positive societal impacts on designing molecules for drugs and materials, as discussed in Sections 1 and 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The work does not present issues of high-risk misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All data sources and baseline models are open-sourced. They have been properly credited and mentioned, as outlined in Section 4.1.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The experiment results are verified through a survey conducted with a small group of domain experts. However, no crowdsourcing experiments were conducted.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn molecular generation tasks, there is often a desire for the generator to produce molecules that satisfy multiple properties simultaneously. Previous architectural designs did not pay particular attention to the scenario of optimizing multiple constraints in tandem. Therefore, the author has proposed a Multi-Conditional Graph Diffusion Transformer. In the noise addition phase, the author has employed a novel noise addition method, successfully establishing connections between the graph's nodes and edges. At the same time, the author has designed a new condition encoder and a graph diffusion transformer architecture. This architecture is capable of addressing both numerical and categorical conditions simultaneously. Experiments have demonstrated that the proposed method exhibits excellent performance.\n\nSTRENGTHS:\n- The writing is mostly clear and easy to follow\n- The evaluation is comprehensive.\n\nWEAKNESSES:\n- some comparisons are not fair, see the questions for detail.\n- some metrics of evaluation is not so suitable, see questions for detail\n\nQUESTIONS:\n**Major Issues**\n\n- The comparison presented in Figure 1 is not fair. Molecules that rank low on a certain single property may not necessarily rank low on other properties. Therefore, for a fair comparison, the statistical graph shown in Figure 1(b) should follow the same format as in Figure 1(a). That is, the authors should calculate and visualize the minimal $ K $ at which a molecule can satisfy all the constraints within rank $ K $ for all the generated molecules.\n- All the datasets used for drug design suffer from a severe class imbalance issue, i.e., the ratio of positive to negative samples is less than 1 to 20. Could the authors specify the approximate ratio of positive to negative samples in the input condition (label) when calculating the accuracy in Table 2? If the ratio of positive to negative samples follows the original dataset's proportion, using ROC-AUC seems to be a more reasonable metric.\n- How is the Avg. Rank calculated in Table 2?\n- In section 3.1, if the setting of $\\mathbf{Q}\\_V$ follows the setting of Digress, and $\\mathbf{Q}\\_{EV}$ is derived from the marginal distribution of $V$, then we have, for any $k$, $\\sum\\_{i=1}^{F_V}\\mathbf{P}\\_{k,i}=N+1 \\neq 1$, where $\\mathbf{P}=\\mathbf{X}\\_G^{t-1}\\mathbf{Q}\\_G^t$, $\\mathbf{P}\\_{k,i}$ represents the probability of node $k$ being of category $i$. The edges have the same issue. Therefore, $q(\\mathbf{X}\\_G^t|\\mathbf{X}_{G}^{t-1})$, as defined by the formula in the article, does not seem to be a distribution. However, in the specific implementation process, the graph is sampled from the \"distribution\" instead of directly using the probability as the edge and node features, which means the flaws mentioned above  theoretically should not affect the model's performance. But the authors need to make corrections in the writing.\n\n**Minor Issues**\n\n- Many diffusion-based generation methods for conditional generation only require the introduction of conditions during the generation process, without the need to incorporate conditions during the training process. Can the proposed method achieve the same?\n\n\n**If you can address all the major issues, I would be very willing to raise my score**\n\nLIMITATIONS:\nI believe the societal impacts and limitations of this work are discussed to a sufficient extent.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Graph DiT, which initially learns conditions (including categorical and numerical properties) through clustering and one-hot encodings. Subsequently, Graph DiT utilizes a Transformer architecture during the diffusion denoising phase to refine noisy molecular graphs incorporating these conditions. Experimental results underscore Graph DiT's efficacy in multi-conditional generation and polymer inverse design, emphasizing its capability to innovate in molecule creation.\n\nSTRENGTHS:\n- **This paper makes somewhat novel contributions**: For example, unlike previous studies that treat node and edge state transitions independently, potentially misaligning with the denoising process, this research proposes a graph-dependent noise model. Graph DiT constructs a transition matrix based on the joint distribution of nodes and edges, enhancing the coherence of the denoising process.\n\n- **The experiments address three primary questions**: (1) The authors validated the generative capabilities of Graph DiT against baseline models from molecular optimization and diffusion. (2) The authors explored polymer inverse design for gas separation. (3) They conducted additional analysis to further examine the capabilities of Graph DiT.\n\n- A portion of the experimental results showed superior performance compared to the baselines.\n\nWEAKNESSES:\n- The paper should emphasize the main contributions and acknowledge its limitations.\n\n- The presentation of the paper should be improved as it is difficult to follow.\n\n- **Lack of experiments**: Similar to DiGress, the paper should compute statistics such as uniqueness, novelty, and VUN (Valid, Unique, and Novel graphs) to provide a comprehensive assessment of the method's efficacy and innovation in graph generation and optimization. These metrics are crucial for evaluating the quality and originality of the generated graphs, ensuring a thorough comparison with existing approaches in the field.\n\n- The experimental results in Table 2 did not demonstrate superior performance compared to other baseline methods. The effectiveness of the proposed Graph DiT appears somewhat trivial to some extent. Only the results in conditional control show good performance.\n\nQUESTIONS:\n- Please refer to the \"weaknesses\" section above.\n\n- The authors argue in lines 25 to 27 that the diverse scales and units of properties present a significant challenge in multi-property optimization. This diversity can complicate the comparison and combination of different properties, potentially leading to skewed optimization results. In my view, a straightforward solution to this issue could involve scaling the properties to a common range, such as 0 to 1. This approach would normalize the scales and facilitate fair comparisons and effective optimization across different properties, thereby addressing the challenge effectively.\n\n- What does \"LCC\" signify in DiT-LCC?\n\n- What visualization method is used in Figure 6? Is it a linear technique like PCA, or a non-linear method such as UMAP, utilized for dimensionality reduction?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis research introduces the Graph Diffusion Transformer (Graph DiT) for generating molecules with multiple properties, such as synthetic score and gas permeability. Unlike previous models, Graph DiT uses a new noise model and a Transformer-based denoiser to better handle molecular structures. Experiments show that it performs well in generating both polymers and small molecules.\n\nSTRENGTHS:\n1. The paper successfully applies the DiT framework from computer vision.\n\n2. The method jointly applies noise between atoms and bonds,\n\nWEAKNESSES:\n1. **Limited Demonstration of Multi-Condition Capability:** The paper demonstrates conditional generation with only two conditions, such as Synth. and HIV, yet claims capability for multiple conditions. To substantiate this claim, it would be beneficial to test on more complex condition sets such as GSK3β+JNK3+QED+SA, as evaluated in the MARS framework and also many other methods. Or at least three properties like Synth, HIV, and BBBP to validate multi-conditional generative capacity.\n\n2. **Unexplored Texture Conditions:** Given that the model incorporates strategies from DiT, it is imperative to assess how it performs under texture conditions. Testing under these conditions would provide a more comprehensive evaluation of the model's versatility and adherence to the principles derived from DiT.\n\n3. **Incomplete Evaluation Metrics for Unconditional Generation:** The evaluation metrics employed for unconditional generation do not adequately measure key aspects such as uniqueness, novelty, and FCD (Fréchet ChemNet Distance). Including these metrics would offer a more rounded understanding of the model’s performance in generating novel and diverse molecular structures.\n\n4. **Ambiguity in Graph-Dependent Noise Schedule Impact:** While the advantages of a graph-dependent noise schedule are discussed, Figure 4(c) does not provide clear empirical evidence of its impact, particularly in comparison with separate discrete diffusion schedules.\n\nQUESTIONS:\n1. How do you determine the $\\text{rank}\\( K \\)$ of molecules under single-condition constraints, and how does this compare to the $\\text{rank}\\( K \\)$ in multi-condition scenarios?\n\n2. Could you clarify the methodology used by the Oracle to rank these graphs, particularly how closely attributes of ranked molecules match the conditional attributes?\n\n3. Does the graph-dependent noise schedule enhance the validity of your method compared to a separate discrete diffusion schedule? Could you provide comparative results?\n\n4. Given the higher similarity and lower distance of your generated molecules to a reference set, how do you ensure that the novelty is not compromised?\n\n5. What advantages does using a learnable dropping embedding offer over simply excluding the condition encoder from the training process in unconditional generation?\n\n6. Have you explored generation with three or more conditions, such as Synth., HIV, and BBBP? If not, what constraints prevent such multi-condition generation?\n\n7. Could you provide examples of true polymers corresponding to the conditions shown in Figure 3, to facilitate a direct comparison and enhance the evaluation of your model’s performance?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This work proposes Graph Diffusion Transformer (Graph DiT), which is a a molecular generation model based on multi-condition, and diffusion process. Graph DiT enables multi-conditional molecular generation, integrating multiple properties, eg. synthetic score and gas permeability.Graph DiT employs a graph-dependent noise model (instead of node level or edge level), and is claimed to improve noise estimation accuracy in molecules. Empirical validation across polymer and small molecule generation tasks shows Graph DiT’s better performance in condition control and distribution learning.

STRENGTHS:
- The topic of how diffusion models can be further made effective for multi-conditional molecular generation is focused and respective limitation of existing work is illustrated with examples. Although there can be multiple directions to improve the integration of conditions, which are discussed.
- The use of graph-dependent noise model forms the basis of enhancing noise estimation beyond what is seen in DIGRESS and similar models. While doing this, several challenges can arise. These challenges are discuss upon by the paper to a god extent.
- Empirical results show improvements in multiple metrics compared to Digress, MOOD and other baselines.

WEAKNESSES:
- Due to the graph dependent noise, the model may be limited or unscalable to medium or large scale graphs.
- Minor format comment: the color shades in the charts like in Figure 4 are inconsistent with their labels.

QUESTIONS:
- Since the model is Graph DiT, the methodology is unclear on how 'graph structure' is modeled within the architecture. If not, it would resemble as not a specific graph transformer based model.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15