Graph Diffusion Transformers for Multi-Conditional Molecular Generation

Authors: gang  liu
jiaxin  xu
tengfei  luo
meng  jiang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 18:40:41

Abstract: Inverse molecular design with diffusion models holds great potential for advancements in material and drug discovery. Despite success in unconditional molecule generation, integrating multiple properties such as synthetic score and gas permeability as condition constraints into diffusion models remains unexplored. We present the Graph Diffusion Transformer (Graph DiT) for multi-conditional molecular generation. Graph DiT has a condition encoder to learn the representation of numerical and categorical properties and utilizes a Transformer-based graph denoiser to achieve molecular graph denoising under conditions. Unlike previous graph diffusion models that add noise separately on the atoms and bonds in the forward diffusion process, we propose a graph-dependent noise model for training Graph DiT, designed to accurately estimate graph-related noise in molecules. We extensively validate the Graph DiT for multi-conditional polymer and small molecule generation. Results demonstrate our superiority across metrics from distribution learning to condition control for molecular properties. A polymer inverse design task for gas separation with feedback from domain experts further demonstrates its practical utility. The code is available at https://github.com/liugangcode/Graph-DiT.

Main paper: 
 introduction: 
 Diffusion models for molecular graphs are essential for inverse design of materials and drugs by generating molecules and polymers (macro-molecules) [40, 46] , because the models can be effectively trained to predict discrete graph structures and atom/bond types in denoising processes [43] . Practical inverse designs consider multiple factors such as molecular synthetic score and various properties [15] , known as the task of multi-conditional graph generation.Existing work converted multiple conditions into a single one and solved the task as single-condition generation [5, 25] . However, multi-property relations may not be properly or explicitly defined [5] . First, the properties have diverse scales and units. For example, the synthetic complexity ranges from 1 to 5 [8] , while the gas permeability varies widely, exceeding 10,000 in Barrier units [4] . This gap makes it hard for models to balance the conditions. Second, multi-conditions consist of a mix of categorical and numerical properties. The common practice of addition [47] or multiplication [25] is inadequate for combination.Figure 1 (a) empirically illustrates the challenges in multi-conditional generation, i.e., discovering molecules meeting multiple properties. We used a test set of 100 data points with three properties: synthesizability (Synth.) [12] , O 2 and N 2 permeability (O 2 Perm and N 2 Perm) [4] . A single-conditional diffusion model generated up to 30 graphs for each condition, resulting in a total of 90 graphs for three conditions. We sort the 30 graphs in each set using a polymer property Oracle (see appendix B.3). Then, we check whether a shared polymer structure that meets multi-property constraints can be identified across different condition sets. If we find the polymer, its rank K (where K is between 1 and 30) indicates how high it appears on the lists, considering all condition sets. If not, we set K as 30. Figure 1 30, indicating that the multiple properties were not met on over half of the test polymers despite generating a large number of graphs.To address these challenges, we project multi-properties into representations by learning, thereby guiding the diffusion process for molecule generation. We propose the Graph Diffusion Transformer (Graph DiT) for graph denoising under conditions. Graph DiT has a condition encoder for property representation learning and a graph denoiser. The condition encoder utilizes a novel clustering-based method for numerical properties and one-hot encoding for categorical ones to learn multi-property representations. The graph denoiser first integrates node and edge features into graph tokens, then denoise these tokens with adaptive layer normalization (AdaLN) in Transformer layers [19, 34] . AdaLN replaces the molecular statistics (mean and variance) in each hidden layer with those from the condition representation, effectively outperforming other predictor-based and predictor-free conditioning methods [22, 43, 34] , as shown in Section 4. 4 . We observe that existing forward diffusion processes [43, 22] apply noise separately to atoms and bonds, which may compromise the accuracy of Graph DiT in noise estimation. Hence, we propose a novel graph-dependent noise model that effectively applies noise tailored to the dependencies between atoms and bonds within the graph.Results in Figure 1(b) show that the polymers generated by Graph DiT closely align with multiproperty constraints. For each test case, we have one graph generated from Graph DiT conditional on three properties. The Oracle determines the rank of this graph among 30 single-conditionally generated graphs for each condition. We find the median ranks are 4, 9, and 11, for Synth., O 2 Perm, and N 2 Perm, respectively, all much higher than 30. Note that the ranked set of 30 graphs was very competitive because the model was trained on the specific condition dedicatedly.In experiments, we evaluate model performance on one polymer and three small molecule datasets. The polymer dataset includes four numerical conditions for multi-conditional evaluation. Our model has the lowest average mean absolute error (MAE), significantly reducing the error by 17.86% compared to the best baseline. It also excels in small molecule tasks, achieving over 0.9 accuracy on task-related categorical conditions, notably surpassing the baseline accuracy of less than 0.6. We also examine the model's utility in inverse polymer designs for O 2 /N 2 gas separation, with domain expert feedback highlighting our model's practical utility in multi-conditional molecular design.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Discussions on the proposed method and empirical support can be found in sections 1, 3, and 4. 
problem definition 2.1 multi-conditional inverse molecular design: 
 A molecular graph G = (V, E) consists of a set of nodes (atoms) V and edges (bonds) E. We follow [43] and define "non-bond" as a type of edge. There are N atoms and each atom has a one-hot encoding, denoting the atom type. We represent it as X V ∈ R N ×F V , where F V is the total number of atom types. Similarly, the bond features are a tensor X E ∈ R N ×N ×F E , representing both the graph structure and F E bond types.Let C = {c 1 , c 2 , . . . , c M } be a set of M numerical and categorical conditions. The task is: q(G | c 1 , c 2 , . . . , c M ) ∝ q(G)q(c 1 , c 2 , . . . , c M | G), where q represents observed probability. We use a model parameterized by θ for multi-conditional molecular generation p θ (G | C). The evaluation involves both distribution learning q(G) [35] and condition control q(c 1 , c 2 , . . . , c M | G). We follow previous work in assuming that there exist different oracle functions O that can independently evaluate each conditioned property [14] :q(c 1 , c 2 , . . . , c M | G) = M i=1 O i (c i | G).Note that the oracles are not used in the training of p θ .Diffusion models consist of forward and reverse diffusion processes [17] . We refer to the forward diffusion process as the diffusion process following [17] . The diffusion process q(G 1:T | G 0 ) = T t=1 q(G t | G t-1 ) corrupts molecular graph data (G 0 = G) into noisy states G t . As timesteps T → ∞, q(G T ) converges a stationary distribution π(G). The reverse Markov process p θ (G 0:T ) = q(G T ) T t=1 p θ (G t-1 | G t ), parameterized by neural networks, gradually denoises the latent states toward the desired data distribution.Diffusion Process One may perturb G t in a discrete state-space to capture the structural properties of molecules [43] . Two transition matricesEQUATIONwhere Cat(X; p) denotes sampling from a categorical distribution with probability p. We remove the subscript ( V /E ) when the description applies to both nodes and edges. It is assumed that the noise Q i (i ≤ t) is independently applied to X in each step i, allowing us to rewrite q(X t | X t-1 ) as the probability of the initial state q(X t | X 0 ) = Cat X t ; p = X 0 Qt , where Qt = i≤t Q i .Noise Scheduling Transition matrices Q V and Q E control the noise applied to atom features and bond features, respectively. Vignac et al. [43] definedπ(G) = (m X ∈ R F V , m E ∈ R F E )as the marginal distributions of atom types and bond types. The transition matrix at timestep t is Q t = α t I + (1 -α t )1m ′ for atoms or bonds, where m ′ denotes the transposed row vector. Therefore, we have Qt = ᾱt I + (1 -ᾱt )1m ′ , where ᾱt = t τ =1 α τ . The cosine schedule [32] is often chosen for ᾱt = cos(0.5π(t/T + s)/(1 + s)) 2 .Reverse Process With the initial sampling G T ∼ π(G), the reverse process generates G 0 iteratively in reversed steps t = T, T -1, . . . , 0. We use a neural network to predict the probability p θ ( G0 | G t ) as the product over nodes and edges [1, 43] :EQUATIONp θ ( G0 | G t ) could be combined with q(G t-1 | G t , G 0 ) to estimate the reverse distribution on the graph p θ (G t-1 | G t ). For example, p θ (v t-1 | G t) is marginalized over predictions of node types ṽ ∈ xv , which applies similarly to edges:EQUATIONThe neural network could be trained to minimize the negative log-likelihood [43] . where x ∈ G 0 denotes the node or edge features. Typically, the reverse process in diffusion models does not consider molecular properties as conditions. While there have been efforts to introduce property-related guidance using additional predictors, the more promising approach of predictor-free guidance [16] , particularly in multi-conditional generation, remains underexplored.L = E q(G 0 ) E q(G t |G 0 ) -E x∈G 0 log p θ x | G t (4) Condition Encoder … 𝑞 𝐺 !"# # 𝐺 $ , 𝐺 ! 𝑝 % ( # 𝐺 $ | 𝐺 ! , 𝒞) 𝒞 T t t-Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Discussions on generation diversity and Oracle evaluations are found in Sections 4.2 and 4.4.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
multi-conditional graph diffusion transformers: 
 We present the denoising framework of Graph DiT in Figure 2 . The condition encoder learns the representation of M conditions. The statistics of this representation like mean and variance are used to replace the ones from the molecular representations [19] (see Section 3.2). Besides, we introduce a new noise model in the diffusion process to better fit graph-structured molecules (see Section 3.1).The transition probability of a node or an edge should rely on the joint distribution of nodes and edges in the prior state. However, as an example shown in Eq. ( 1), current diffusion models [22, 43, 25] treat node and edge state transitions as independent, misaligning with the denoising process in Eq. ( 3). This difference between the sampling distributions of noise in the diffusion and reverse processes introduces unnecessary challenges to multi-conditional molecular generations.To address this, we use a single matrix X G ∈ R N ×F G to represent graph tokens for G, withF G = F V + N • F E .Token representations are created by concatenating the node feature matrix X V and the flattened edge connection matrix from X E . Each row vector in X G contains features for both nodes and edges, representing all connections and non-connections. Hence, we could design a transition matrix Q G considering the joint distribution of nodes and edges.Q G ∈ R F G ×F G is constructed from four matrices Q V , Q EV ∈ R F E ×F V , Q E , Q V E ∈ R F V ×F E ,denoting the transition probability ("dependent old state" → "target new state") node → node; edge → node; edge → edge; node → edge, respectively.EQUATIONwhere ⊗ denotes the Kronecker product, 1 N , 1 ′ N , and 1 N ×N represent the column vector, row vector, and matrix with all 1 elements, respectively. According to Eq. ( 5), the first F V columns in Q G determine the node feature transitions based on both node features (first F V rows) and edge features (remaining N • F E rows). Conversely, the remaining N • F E columns determine the edge feature transitions, depending on the entire graph. We introduce a new diffusion noise model:EQUATIONwhere p is the unnormalized probability and Cat denotes categorical sampling: The first F V columns of p are normalized to sample X t V , while the remaining N •E dimensions are reshaped and normalized to sample edges X t E . These components are combined to form X t G , completing the Cat sampling. Choice of Q V E and Q EV Similar to the definitions of m V and m E [43] , we leverage the prior knowledge within the training data for the formulation of task-specific matrices, Q EV and Q V E . We calculate co-occurrence frequencies of atom and bond types in training molecular graphs to obtain the marginal atom-bond co-occurrence probability distribution. For each bond type, each row in m EV represents the probability of co-occurring atom types. m V E is the transpose of m EV and has a similar meaning. Subsequently, we defineQ EV = ᾱt I + (1 -ᾱt )1m ′ EV and Q V E = ᾱt I + (1 -ᾱt )1m ′ V E .We present Graph DiT as the denoising model to generate molecules under multi-conditions C = {c 1 , c 2 , . . . , c M } without extra predictors.Predictor-Free Guidance The predictor-free reverse process pθ (G t-1 | G t , C) aims to generate molecules with a high probability q(C | G 0 ). This could be achieved by a linear combination of the log probability for unconditional and conditional denoising [16] :EQUATIONwhere s denotes the scale of conditional guidance. Unlike classifier-free guidance [16] , which typically predicts noise, we directly estimate p θ ( G0 | G t , C). We one one denoising model f θ (G t , C) for both p θ ( G0 | G t ) and p θ ( G0 | G t , C). Here, f θ (G t , C = ∅) computes the unconditional probability by substituting the original conditional embeddings with the null value. During training, we randomly drop the condition with a ratio, i.e., C = ∅, to learn the embedding of the null value. f θ (G t = X t G , C) comprises two components: the condition encoder and the graph denoiser. An overview of the architecture is presented in Figure 2 .Condition Encoder We treat the timestep t as a special condition and follow [31] to obtain a Ddimensional representation t with sinusoidal encoding. For property-related numerical or categorical condition c i ∈ C, we apply distinct encoding operations to get D-dimensional representation. For a categorical condition, we use the one-hot encoding. For a numerical variable, we introduce a clustering encoding method. This defines learnable centroids, assigning c i to clusters, and transforming the soft assignment vector of condition values into the representation. It could be implemented using two Linear layers and a Softmax layer in the middle as: Linear (Softmax (Linear(c i ))). Finally, we could obtain the representation of the condition as c = M i=1 encode(c i ), where encode is the specific encoding method based on the condition type. For numerical conditions, we evaluate our proposed clustering-based approach against alternatives like direct or interval-based encodings [28] . As noted in Section 4.4, the clustering encoding outperforms the other methods.Graph Denoiser: Transformer Layers Given the noisy graph at timestep t, the graph tokens are first encoded into the hidden space as H = Linear(X t G ), where H ∈ R N ×D . We then adapt the standard Transformer layers [42] with self-attention and multi-layer perceptrons (MLP), but replace the normalization with the adaptive layer normalization (AdaLN) controlled by the representations of the conditions [19, 34] : H = AdaLN(H, c). For each row h in H:EQUATIONwhere µ(•) and σ(•) are mean and variance values. ⊙ indicates element-wise product. γ θ (•) and β θ (•) are neural network modules in f θ (•), each of which consists of two linear layers with SiLU activation [11] in the middle. We have a gated variant AdaLN gate for residuals:EQUATIONWe apply the zero initialization for the first layer of γ θ (•), β θ (•), and α θ (•) [34] . There are other options to learn the structure representation from the condition [34] : In-Context conditioning adds condition representation to the structure representation at the beginning of the structure encoder, and Cross-Attention calculates cross-attention between the condition and structure representation. We observe in Section 4.4 that AdaLN performs best among them.Graph Denoiser: Final MLP We have the hidden states H after the final Transformer layers, the MLP is used to predict node probabilities X0V and edge probabilities X0EQUATIONWe split the output X G into atom and bond features X0 V , X0 E . The first F V dimensions of X0 G represent node type probabilities, and the remaining N • F E dimensions cover probabilities for N edge types associated with the node, as detailed in Section 3.1.A common way of converting generated graphs to molecules selects only the largest connected component [43] , denoted as Graph DiT-LCC in our model. For Graph DiT, we connect all components by randomly selecting atoms. It minimally alters the generated structure to more accurately reflect model performance than Graph DiT-LCC. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Code is provided in the supplementary materials. Data and code will be on Github after publication.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setup: 
 We use datasets with over ten types of atoms and up to fifty nodes in a molecular graph. We include both numerical and categorical properties for drugs and materials, offering a benchmark for evaluation across diverse chemical spaces. Model performance is validated across up to nine metrics, including distribution coverage, diversity, and condition control capacity for various properties.We have one polymer dataset [40] for materials, featuring three numerical gas permeability conditions: O 2 Perm, CO 2 Perm, and N 2 Perm. For drug design, we create three class-balanced datasets from MoleculeNet [46] : HIV, BBBP, and BACE, each with a categorical property related to HIV virus replication inhibition, blood-brain barrier permeability, or human βsecretase 1 inhibition, respectively. We have two more numerical conditions for synthesizability from synthetic accessibility (SAS) and complexity scores (SCS) [12, 8] . (5) Fréchet ChemNet Distance with the reference set (Distance) [36] ; MAE between the generated and conditioned (6) synthetic accessibility score [12] (Synth.); ( 7)∼( 9) MAE/Accuracy for the numerical/categorical task conditions (Property). The evaluation Oracle uses random forest trained on all task-related molecules [14] . Lower MAE or higher accuracy indicates stronger model controllability.EvaluationBaselines We select strong and popular molecular optimization baselines from recent studies [14] : Graph-GA [20] , MARS [47] , JTVAE [21] with Bayesian optimization (JTVAE-BO), LSTM [6] on SMILES with Hill Climbing (LSTM-HC). We include the most recent diffusion models: GDSS [22] , DiGress [43] , and their conditional version with extra predictors: MOOD [25] , and DiGress v2 [43] .We train multi-task predictors using the same architecture for MOOD and DiGress v2 models to provide additional guidance for generation. For molecular optimization, we formulate the condition set of each test data point as a combined goal, minimizing the sum of the normalized errors between generated and input properties. We train a random forest model for each property using the training data to optimize the molecular structure.We have the observations from Table 1 and Table 2 :Chemical Validity High validity may not accurately represent the model's generative performance if hard-coded rules are introduced in the algorithm. For example, GraphGA could eliminate non-valid molecules during mutation and crossover iterations to achieve perfect validity in the final evaluation. Without rule checking in the generation-to-molecule step, DiGress, GDSS, and MOOD show a marked performance decline, with validity often dropping from 0.99 to below 0.6. In contrast, Graph DiT often maintains over 0.8 validity without any rule-based processing. Distribution Learning GraphGA is a simple yet effective baseline for generating in-distribution molecules, e.g., on BBBP and HIV generation datasets. Diffusion model baselines such as DiGress and MOOD could produce diverse molecules but often fail to capture the original data distribution in multi-conditional tasks. Graph DiT shows the competitive performance of diffusion models in fitting complex molecular data distributions. Using fragment-based similarity and neural network-based distance metrics [36] , we achieve the best in the polymer task and rank second in the HIV small molecule task, involving up to 11 and 29 types of heavy atoms, respectively.Condition Controllability LSTM-HC surpasses many baselines, achieving lower average MAE on polymer properties and higher rankings on small molecular properties. However, its control over synthetic scores in polymer tasks is relatively poor. Conversely, MARS effectively manages synthetic scores for polymers but exhibits a larger MAE in gas permeability conditions compared to other baselines. GDSS performs well in gas permeability control but underperforms Graph GA and MARS in terms of the synthetic score condition. DiGress v2 and MOOD, although equipped with the predictor guidance, still exhibit limited condition control compared to their unconditional counterparts over polymer and small molecule tasks. These baselines struggle to balance and control multiple conditions in generation. In contrast, Graph DiT significantly improves diffusion models and achieves the best multi-conditional performance in all tasks. In polymer tasks, Graph DiT reduces MAE on all gas permeability conditions, averaging +17.8% improvement over the best baseline LSTM-HC. For small molecule tasks, Graph DiT consistently ranks top-1 in condition controllability with over 0.9 accuracy in categorical conditions. Compared to Graph DiT-LCC, we observe that Graph DiT, which connects all generated graph components, shows better controllability performance due to minimal rule-based post-generation processing.We aim to design polymers with high O 2 and low N 2 permeability, demonstrating the models' precise control over related properties. Following Robeson [38] 's definition of high-performance polymers based on the O 2 /N 2 permeability ratio, we selected 16 polymers meeting this criterion from 609 examples as our test/reference set. The remaining data is used for training and validation. Subsequently, we generated 1,000 polymers conditioned on test set labels. Current Value for other metrics.In Figure 3 , we present the top three polymers generated by each model for a case study with expertise. Initially, a random forest algorithm identifies the top five polymers per method based on average MAE in two gas permeability. These 25 polymers are then shuffled and evaluated by four polymer scientists, who rank them from 1 to 25 using their domain knowledge. Rankings are normalized to a Utility Score (UtS) ranging from 0 to 1, with higher scores indicating greater utility. The variance in UtS is converted into an Agreement Score (AS) for further evaluation. As shown in Figure 3 , there is a high consensus among experts that the three polymers generated by Graph DiT are the most promising for successful polymer inverse design tasks. More details are in appendix D. By comparing generated examples from different models, we have further observations:• DiGress and MOOD struggle to capture polymerization points, marked with asterisks ("*"), which is one of the most important features that distinguish polymers from small molecules. Additionally, the two methods frequently feature excessive carbon atoms and overly large cycles. These molecular configurations with significant distortion from the canonical geometry of stable compounds may lead to poor synthesizability [35, 7] . • LSTM-HC may result in too-long carbon chains with limited diversity. MARS produces examples with asymmetrical graph structures, challenging polymer synthesis [10, 2] .• Graph DiT generates structurally diverse and symmetric polymers with two polymerization points, indicative of more valid and synthesizable polymer structures. The first two, which are polyimides, imply effective gas separation performance [24] .Model Components In light of Table 1 , we analyze three components that impact our model's learning in various conditions. Our assessment of relative performance is based on the ratio between our method and comparative approaches. The first component is numerical conditional encoding.Results in Figure 4 (a) highlight the superiority of clustering encoding over direct and interval-based encoding, particularly in controlling gas permeability, despite its slightly lower diversity. The second component concerns the neural architecture for conditions. As shown in Figure 4  
related work: 
 Diffusion Models for Molecules: Score-based diffusion models applied noise and denoising in continuous space [33, 22] . DiGress [43] used discrete noise as transition matrices based on marginal distributions of atom and bond types. Extra predictor models are studied to guide the generation process in DiGress and GDSS [25] . Diffusion models could also be used for molecular property prediction [27] , for conformation [48] and molecule generation with 3D atomic coordinates [18, 49, 3] . We focus on molecular graph generation, considering the high computational cost of accurate 3D coordinates for larger molecules like polymers [23] . We explore predictor-free diffusion guidance, instead of the classifier guidance [9, 44] , for generating molecules under categorical and numerical conditions. It can be integrated with diffusion models for atomic coordinates in future research.Molecular Optimization: Optimization algorithms could optimize molecules towards property constraints, including genetic algorithms [20] , Bayesian optimization [39, 50] , REINFORCE [45] , and reinforcement learning [30] . Both sequential and graph-based generative models [6, 21, 30] , along with diverse sampling methods [47, 13] , are used in conjunction with these algorithms to produce desirable molecules. These methods have been applied to both single-objective and multi-objective optimization, the latter by manually integrating multiple property conditions into a single one [5, 25] . Several challenges in molecular optimization methods remain underexplored, including the inadequate or unclear definition of multi-property relations when integration into a single objective [5] , and the inaccessibility of the oracle function for property-oriented optimization during the training phase [14] . 
conclusion: 
 In this work, we solved inverse molecular design using properties as predictor-free diffusion guidance.The proposed Graph DiT performed diffusion based on the joint distribution of atoms and bonds in both forward and reverse processes. It introduced representation learning for multiple categorical and numerical properties and utilized a Transformer-based graph denoiser for conditional graph denoising. Results on multi-conditional generations and polymer inverse designs showed the remarkable generative capabilities of Graph DiT, making it suitable for designing promising molecules. We explore several approaches for encoding numerical conditions. In addition to the clustering-based method, we consider:1. The direct encoding approach, which employs a linear layer to map a continuous number into a high-dimensional space. 2. The interval-based approach, as described in [28] , divides the label space into N Interval intervals. It then converts the number into an interval index, allowing us to apply one-hot encoding for the number.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Details are in Section 4.1.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: Improvement percentages (>10%) are discussed in Section 4.2.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The work has positive societal impacts on designing molecules for drugs and materials, as discussed in Sections 1 and 6.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The work does not present issues of high-risk misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Besides the AdaLN, there are two more options to integrate condition representation into molecular graph representations [34] :1. The In-Context conditioning approach adds the condition representation c to each row of the molecular graph representation H after mapping the X t G into H using the linear layer in the structure encoder. 2. The Cross-Attention approach concatenates the timestep encoding vector with the condition representation from synthesis scores or task-related properties into a two-length sequence.In each Transformer encoder layer, this is followed by a cross-attention layer at the end of the standard multi-head self-attention layer.All experiments can be run on a single A6000 GPU card.As presented in Table 4 , we collect popular datasets in prediction tasks for more challenging molecular generation tasks. We include a polymer dataset [40, 26] for material design. It consists of conditions of O 2 , CO 2 , and N 2 , which study the numerical gas permeability for oxygen, carbon dioxide, and nitrogen, respectively. Additionally, we also study the generative performance of different models separately on the polymer data with O 2 , CO 2 , or N 2 , as illustrated in Table 4 . We also create three class-balanced molecule datasets from [46] for drug design: HIV, BBBP, and BACE, which study categorical properties related to the inhibition of HIV virus replication, blood-brain barrier permeability, and inhibition of human β-secretase 1, respectively. We aim to generate synthesizable molecules. Therefore, we add two numerical conditions for synthetic complexity scores [12, 8] for each of the tasks. For the gas separation polymer design task, we consider joint conditions for O 2 and N 2 and measure the selectivity O 2 /N 2 as the ratio between two gas permeability scores. All polymer gas permeabilities are scaled in the log space following previous work [29] . We focus on experiments for polymers and molecules within 50 nodes.We randomly split the dataset into training, validation, and testing (reference) sets in a 6:2:2 ratio. We investigate more than eight metrics to systematically evaluate the generation performance. First, we assess generation validity (Validity). Second, we evaluate the distribution learning capacity of different models by measuring heavy atom type coverage (Coverage), internal diversity among the generated examples using Tanimoto similarity (Diversity), fragment-based similarity with the reference set (Similarity), and the Fréchet ChemNet Distance with the reference set (Distance). Third, we evaluate the model's controllability by measuring the mean absolute error (MAE) between the generated condition score and the actual condition scores if the condition is numerical; otherwise, we measure the accuracy score. We follow previous work [14] to use the random forest trained on all the available data as the Oracle evaluation function. For molecular optimization algorithms, we train random forest predictors on the training set for conditional generation. For predictor-guided diffusion models, DiGress v2 and MOOD, we use the same architecture as their denoising models to train predictors for diffusion guidance. Given the conditions in the test set, we report the generation performance by generating 10,000 examples for the six multi-conditional generation tasks and 1,000 examples for the polymer inverse design problem focused on selectivity.Using the same dataset from the O 2 /N 2 polymer inverse design task, we keep 100 polymers to provide condition sets for testing and split the rest into training and validation sets in a 0.65:0.35 ratio. We apply our proposed Graph DiT for both single-conditional and multi-conditional approaches, focusing on three properties: (1) Synth. score for synthesizability [12] , (2) O 2 permeability, and (3) N 2 permeability. The single-conditional approach generates 30 polymers per condition for each test data point, totaling 9,000 polymers. In contrast, the multi-conditional approach generates 30 polymers for each set of conditions per test data point, resulting in 3,000 polymers. We rank these polymers based on the mean absolute error between the generated properties (evaluated by a random forest model trained on all the data to simulate the Oracle function) and the conditional property. For each test data point, we also rank the best multi-conditional polymer in different single-conditional Median: 30 We compare the average ranking of generated polymers with desirable properties. First, we generate three sets of polymers using a single-conditional approach for each condition. In (a), as shown in Figure 1 , we find the ranking of the shared structure for each multi-condition requirement.In (b), polymers are generated using a multi-conditional approach, and for each, we identify the highest ranking among the three single-conditional sets. Then, we calculate the median of these maximum ranking positions, which is 16, approximately 2× better than single-conditional generation, which has a median value greater than 30.sets. For the single-conditional approach, we identify a common polymer meeting various properties and visualize the minimum top K value distribution across all 100 test points.In addition to Figure 1 , we compute the median rankings of the multi-conditionally generated polymers within the single-conditional sets. The results are shown in Figure 6 . Both Figure 1 and Figure 6 demonstrate the advantages of multi-conditional generation over single-conditional generation.We show results for three polymer generation tasks in Table 1 and molecule generation tasks in Table 2 .As complementary results for Table 1 , we present new results on generation using only one gas permeability in Table 5 . We also compare the distributions of atom and bond types between generated and training data in Figure 5 . Furthermore, Figure 7 visualizes the two-dimensional molecular data distribution of both training and generated molecules across various generative models.While diffusion models like GDSS [22] and DiGress [43] show promise in unconditional tasks, their performance in multi-conditional generations needs improvement for fitting training distributions and achieving more controllable results. As indicated by Figure 5 (a), the generation of GDSS [22] often collapses to carbon elements with Gaussian noise in the continuous diffusion state-space. MOOD [25] improves atom type coverage by adding predictor guidance and an out-of-distribution hyper-parameter, but it is hard to fit the training distribution, as visualized in Figure 7 (h). DiGress and its predictor-guided variant [43] (i.e., DiGress v2), using discrete state-space and transition matrices for diffusion noise, outperform GDSS and MOOD in distribution fitting and internal diversity in Tables 1, 2 and 5. However, as indicated in Figure 7 (e) and Figure 7 (f), these two models still generate too many out-of-distribution examples without justification of the generalization capacity. While GDSS and MOOD show lower average MAE in polymer conditional generation tasks, their subpar distribution learning performance and the results from Table 2 suggest that this may be due to the carbon element, which may be a confounder and affect the evaluation of the correlation between the model and controllability in polymer tasks.Popular molecular optimization baselines are competitive in molecular generation tasks. Earlier studies have noted their strong performance: Gao et al. [14] showed their effectiveness in the standard molecular optimizations with a combined optimization target, and Tripp and Hernández-Lobato [41] found that genetic algorithm often outperforms recent methods in unconditional generation. We observe their competitive performance in multi-conditional settings, characterized by high validity, good atom type coverage, and distribution similarity in Tables 1, 2 and 5. MARS [47] and JTVAE [21] perform well for controlling the synthetic accessibility score [12] but are less effective at controlling specific task properties like gas permeability, BACE, BBBP, and HIV. For example, in generation tasks with the categorical task condition, the generated examples only achieve around 50% accuracy in hitting the input condition. In Figure 8 , we illustrate changes in various indicators on the validation set during model training.We note an increase in generated validity and similarity to the validation reference set, along with a decrease in distance to the reference set and errors between generated properties and conditions, indicating gradual improvement in conditional generation over epochs. However, a trade-off between distribution fitting and internal diversity is observed in our current model, suggesting that further work on enhancing generation diversity could be promising. The generated validity and similarity to the validation reference set have increased, accompanied by a decrease in distance to the reference set and errors between generated properties and conditions. We also observe a trade-off between distribution fitting and internal diversity over epochs.We evaluate the model's performance on Novelty and Uniqueness. Unlike unconditional generation, multi-conditional generation involves generating multiple possible molecules under the same condition. Therefore, we compute these metrics across different sets of conditions rather than for generated molecules under the same conditions. Results are presented in Table 7 .Graph DiT demonstrates reasonable performance on these metrics. However, higher Novelty and Uniqueness values do not necessarily indicate better performance, as they may not reflect the model's ability to design satisfactory molecules with desirable properties. Moreover, these values risk leading to misleading conclusions. For instance, AddCarbon achieves nearly perfect scores (99.94% Novelty and 99.86% Uniqueness) according to [37, 41] , yet it randomly adds carbon atoms to existing molecules, resulting in new molecules that are not practically useful [37] .We aim to design polymers with high O 2 and low N 2 permeability, showing refined control of models over these properties. This is reflected in the selectivity, defined as the O 2 /N 2 permeability ratio.Robeson [38] has identified an inherent trade-off between gas permeability and selectivity, known as the upper-bound. Ideally, high-performance polymers should fall in the above-the-bound region, demonstrating an effective combination of permeability and selectivity. We have 609 polymers with annotated permeability values for both gases. 16 above-the-bound polymers are included in the test/reference set and excluded from the training set. We generate 1,000 polymers conditional on test set labels. We aim to gather expert evaluations on the generation performance of various methods. We conduct a study using a test data point from the O 2 /N 2 gas separation inverse design task, taking its properties as conditions. The structure of the selected data point is presented in Figure 9 . We display 25 generated polymers, each with its properties, alongside three real polymers from the training dataset as references. The first real polymer serves as the test reference, while the other two, similar in properties to the first, also aid experts in assessing the generated polymers. The properties of these generated polymers are predicted using a well-trained random forest model. Experts are asked to rank the generated polymers from 1 to 25, considering: (1) Structures of real polymers with desirable properties; (2) Predicted properties of generated polymers, displayed beneath each visualization.Here, a rank of 1 represents the best example as per domain knowledge, while 25 is the least favorable. Rankings are then converted to utility scores (UtS) ranging from 0 to 1 using 1 ranking , allowing us to quantify the relative performance of different generation methods. The agreement score (AS) could be obtained by exp (-25 × Variance(UtS 1 , UtS 2 , UtS 3 , UtS 4 )), where UtS i denotes the utility score from the i-th domain expert. (3) Finally, we select top-3 polymers for each generative models and present them in Figure 3 .We present the inverse design results of all 1,000 generated polymers in validity, distribution learning, and condition control in We visualize results in Figure 10 . We consistently observe that validity and controllability performance improve as the values approach 213.75, derived from a real test polymer. Conversely, performance deteriorates when the sampled N 2 values are closer to the extremes of the sampling range (0 or 1000). This observation underscores the interdependency between conditions, where less frequent combinations of different properties may be more challenging to learn. Moreover, the model performs well across a elatively large range from 0 to 1000 in terms of validity, O 2 , and synthesizability score control. This demonstrates good generalization of the proposed method in capturing complex condition interdependencies. In addition to the three components related to conditioning effectiveness of Graph DiT studied in Section 4.4, we also examine the importance of the final layer MLP for conditional graph denoising. Results in Figure 11 show that MLP significantly outperforms a linear layer [34] .We train three types of Oracles based on Random Forest, Gaussian Process, and Support Vector Machine on all polymers or molecules in a task to evaluate the properties of generated polymers conditional on O 2 Perm only, N 2 Perm only, CO 2 Perm only, BACE, BBBP, or HIV. The training performance (MAE or AUC) is presented in Table 8 . Our findings show that the random forest achieves the lowest MAE and highest AUC scores, leading us to select it for simulating oracles in our generation evaluation process.• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper contributes Graph Diffusion Transformer (Graph DiT), a diffusion-based model which learns representations of multiple conditions using an additional encoder. The paper contributes also a novel graph-dependent noise model for molecules. During the rebuttal period, the Authors have performed additional analyses and provided novel results, specifically, values of Validity, Novelty and Uniqueness, as requested by the Reviewers. These values were not the best among the compared models. However, other metrics showing favourable performance have been provided.

The contribution is of interest for the NeurIPS community.

Reviews: review: paper_summary: paper_summary: This work proposes Graph Diffusion Transformer (Graph DiT), which is a a molecular generation model based on multi-condition, and diffusion process. Graph DiT enables multi-conditional molecular generation, integrating multiple properties, eg. synthetic score and gas permeability.Graph DiT employs a graph-dependent noise model (instead of node level or edge level), and is claimed to improve noise estimation accuracy in molecules. Empirical validation across polymer and small molecule generation tasks shows Graph DiT’s better performance in condition control and distribution learning.
strength_weakness: strength_weakness: - Due to the graph dependent noise, the model may be limited or unscalable to medium or large scale graphs.
- Minor format comment: the color shades in the charts like in Figure 4 are inconsistent with their labels.
questions: questions: - Since the model is Graph DiT, the methodology is unclear on how 'graph structure' is modeled within the architecture. If not, it would resemble as not a specific graph transformer based model.
limitations: limitations: yes
paper_summary: paper_summary: This research introduces the Graph Diffusion Transformer (Graph DiT) for generating molecules with multiple properties, such as synthetic score and gas permeability. Unlike previous models, Graph DiT uses a new noise model and a Transformer-based denoiser to better handle molecular structures. Experiments show that it performs well in generating both polymers and small molecules.
strength_weakness: strength_weakness: 1. **Limited Demonstration of Multi-Condition Capability:** The paper demonstrates conditional generation with only two conditions, such as Synth. and HIV, yet claims capability for multiple conditions. To substantiate this claim, it would be beneficial to test on more complex condition sets such as GSK3β+JNK3+QED+SA, as evaluated in the MARS framework and also many other methods. Or at least three properties like Synth, HIV, and BBBP to validate multi-conditional generative capacity.

2. **Unexplored Texture Conditions:** Given that the model incorporates strategies from DiT, it is imperative to assess how it performs under texture conditions. Testing under these conditions would provide a more comprehensive evaluation of the model's versatility and adherence to the principles derived from DiT.

3. **Incomplete Evaluation Metrics for Unconditional Generation:** The evaluation metrics employed for unconditional generation do not adequately measure key aspects such as uniqueness, novelty, and FCD (Fréchet ChemNet Distance). Including these metrics would offer a more rounded understanding of the model’s performance in generating novel and diverse molecular structures.

4. **Ambiguity in Graph-Dependent Noise Schedule Impact:** While the advantages of a graph-dependent noise schedule are discussed, Figure 4(c) does not provide clear empirical evidence of its impact, particularly in comparison with separate discrete diffusion schedules.
questions: questions: 1. How do you determine the $\text{rank}\( K \)$ of molecules under single-condition constraints, and how does this compare to the $\text{rank}\( K \)$ in multi-condition scenarios?

2. Could you clarify the methodology used by the Oracle to rank these graphs, particularly how closely attributes of ranked molecules match the conditional attributes?

3. Does the graph-dependent noise schedule enhance the validity of your method compared to a separate discrete diffusion schedule? Could you provide comparative results?

4. Given the higher similarity and lower distance of your generated molecules to a reference set, how do you ensure that the novelty is not compromised?

5. What advantages does using a learnable dropping embedding offer over simply excluding the condition encoder from the training process in unconditional generation?

6. Have you explored generation with three or more conditions, such as Synth., HIV, and BBBP? If not, what constraints prevent such multi-condition generation?

7. Could you provide examples of true polymers corresponding to the conditions shown in Figure 3, to facilitate a direct comparison and enhance the evaluation of your model’s performance?
limitations: limitations: NA
paper_summary: paper_summary: This paper presents Graph DiT, which initially learns conditions (including categorical and numerical properties) through clustering and one-hot encodings. Subsequently, Graph DiT utilizes a Transformer architecture during the diffusion denoising phase to refine noisy molecular graphs incorporating these conditions. Experimental results underscore Graph DiT's efficacy in multi-conditional generation and polymer inverse design, emphasizing its capability to innovate in molecule creation.
strength_weakness: strength_weakness: - The paper should emphasize the main contributions and acknowledge its limitations.

- The presentation of the paper should be improved as it is difficult to follow.

- **Lack of experiments**: Similar to DiGress, the paper should compute statistics such as uniqueness, novelty, and VUN (Valid, Unique, and Novel graphs) to provide a comprehensive assessment of the method's efficacy and innovation in graph generation and optimization. These metrics are crucial for evaluating the quality and originality of the generated graphs, ensuring a thorough comparison with existing approaches in the field.

- The experimental results in Table 2 did not demonstrate superior performance compared to other baseline methods. The effectiveness of the proposed Graph DiT appears somewhat trivial to some extent. Only the results in conditional control show good performance.
questions: questions: - Please refer to the "weaknesses" section above.

- The authors argue in lines 25 to 27 that the diverse scales and units of properties present a significant challenge in multi-property optimization. This diversity can complicate the comparison and combination of different properties, potentially leading to skewed optimization results. In my view, a straightforward solution to this issue could involve scaling the properties to a common range, such as 0 to 1. This approach would normalize the scales and facilitate fair comparisons and effective optimization across different properties, thereby addressing the challenge effectively.

- What does "LCC" signify in DiT-LCC?

- What visualization method is used in Figure 6? Is it a linear technique like PCA, or a non-linear method such as UMAP, utilized for dimensionality reduction?
limitations: limitations: None
paper_summary: paper_summary: In molecular generation tasks, there is often a desire for the generator to produce molecules that satisfy multiple properties simultaneously. Previous architectural designs did not pay particular attention to the scenario of optimizing multiple constraints in tandem. Therefore, the author has proposed a Multi-Conditional Graph Diffusion Transformer. In the noise addition phase, the author has employed a novel noise addition method, successfully establishing connections between the graph's nodes and edges. At the same time, the author has designed a new condition encoder and a graph diffusion transformer architecture. This architecture is capable of addressing both numerical and categorical conditions simultaneously. Experiments have demonstrated that the proposed method exhibits excellent performance.
strength_weakness: strength_weakness: - some comparisons are not fair, see the questions for detail.
- some metrics of evaluation is not so suitable, see questions for detail
questions: questions: **Major Issues**

- The comparison presented in Figure 1 is not fair. Molecules that rank low on a certain single property may not necessarily rank low on other properties. Therefore, for a fair comparison, the statistical graph shown in Figure 1(b) should follow the same format as in Figure 1(a). That is, the authors should calculate and visualize the minimal $ K $ at which a molecule can satisfy all the constraints within rank $ K $ for all the generated molecules.
- All the datasets used for drug design suffer from a severe class imbalance issue, i.e., the ratio of positive to negative samples is less than 1 to 20. Could the authors specify the approximate ratio of positive to negative samples in the input condition (label) when calculating the accuracy in Table 2? If the ratio of positive to negative samples follows the original dataset's proportion, using ROC-AUC seems to be a more reasonable metric.
- How is the Avg. Rank calculated in Table 2?
- In section 3.1, if the setting of $\mathbf{Q}\_V$ follows the setting of Digress, and $\mathbf{Q}\_{EV}$ is derived from the marginal distribution of $V$, then we have, for any $k$, $\sum\_{i=1}^{F_V}\mathbf{P}\_{k,i}=N+1 \neq 1$, where $\mathbf{P}=\mathbf{X}\_G^{t-1}\mathbf{Q}\_G^t$, $\mathbf{P}\_{k,i}$ represents the probability of node $k$ being of category $i$. The edges have the same issue. Therefore, $q(\mathbf{X}\_G^t|\mathbf{X}_{G}^{t-1})$, as defined by the formula in the article, does not seem to be a distribution. However, in the specific implementation process, the graph is sampled from the "distribution" instead of directly using the probability as the edge and node features, which means the flaws mentioned above  theoretically should not affect the model's performance. But the authors need to make corrections in the writing.

**Minor Issues**

- Many diffusion-based generation methods for conditional generation only require the introduction of conditions during the generation process, without the need to incorporate conditions during the training process. Can the proposed method achieve the same?


**If you can address all the major issues, I would be very willing to raise my score**
limitations: limitations: I believe the societal impacts and limitations of this work are discussed to a sufficient extent.

Comments: title: Summary of Discussion with Reviewer N56G
Thank you for your feedback
Response to follow up comment
Response to follow up comment
Response to Authors
Response to follow up comment
Response to follow up comment
Response to Authors' Rebuttal
Thank you for raising the score
Final Response
Further response about Major 2
Further Questions about Major 2
comment: We thank reviewer N56G for the engaging discussion. Below is a concise summary of our discussion in a question-and-answer format to ensure clarity:

---
# Are Uniqueness and Novelty metrics important?
**Yes.** Uniqueness measures the diversity of generated molecules at the instance level, while Novelty ensures the generative model creates new molecules rather than merely memorizing training examples. 

We agree with the reviewer and have reported these metrics in our initial rebuttal. Graph DiT has a Novelty value of 0.9702 and a Uniqueness value of 0.8919, demonstrating its ability to generate diverse and novel examples.

# Do high Uniqueness and Novelty always indicate better model performance?

**Not necessarily.** High uniqueness and novelty do not guarantee that generated molecules will meet desirable multi-property criteria. Another example is the AddCarbon algorithm. While it has high uniqueness and novelty [1], it has been considered less useful in previous research [2].

# Do low Uniqueness and Novelty always indicate worse model performance?

**Not entirely.** In molecular discovery, even a single new and useful molecule can have significant real-life impacts. A model with relatively low Uniqueness, but with strong condition controllability, may still identify a new valuable molecule, even if it is repeated multiple times in generation.

# Can Uniqueness alone represent generation diversity?

**Probably not.** While uniqueness is an important measure of instance-level diversity, it only yields a binary result (0 or 1) when measuring a pair of molecules. The metric of internal diversity used in the paper provides a more nuanced evaluation by calculating structural differences with values between 0 and 1.

# Can Uniqueness alone represent generation distribution learning?
**No.** It's important to consider distribution fitting metrics like Fréchet ChemNet Distance (FCD) and structure-based similarity (Similarity), as used in our paper. These metrics are defined on the reference set (i.e., test set) in the paper to avoid potential overfitting concerns.

# Is there a one-size-fits-all metric for evaluating molecular discovery?

**To the best of our knowledge, no.** This is why we extensively evaluate models using up to 9 metrics and conduct surveys to gather feedback from material scientists. We aim to comprehensively present model performance for higher chances of discovering new and useful molecules that meet multiple properties. We appreciate the reviewer's suggestion to include Novelty and Uniqueness for a more comprehensive evaluation.

---

We sincerely appreciate the reviewer's time and effort in our discussion. We hope this summary clarifies any misunderstandings and aligns us on the metrics of molecular discovery. If there are any remaining concerns, we would be grateful for the opportunity to continue the discussion in the remaining valuable time.


# Reference

[1] Genetic Algorithms are Strong Baselines for Molecule Generation. 2023.

[2] On failure modes in molecule generation and optimization. Drug Discovery Today: Technologies. 2019.
Thank you for your feedback on the model's multi-conditional performance. We will update the paper accordingly to better present the model performance. We welcome any further discussion on the points the reviewer feels have not been fully addressed.
Thank you for your response. The success rate of 93.3% appears reasonable. It would be beneficial to include this result, along with the corresponding table in W1, in the revision.
Thank you for your prompt response.

# Regarding "I think the authors have ...", 

We apologize for any misunderstanding. To clarify, we previously stated (Initial Rebuttal):

> Randomly adding carbons to existing molecules can yield almost 100% novelty and uniqueness.

In our second round of responses, we mentioned:

> Table 1 in Reference 1 indicates that the AddCarbon method achieves 99.94% Novelty and 99.86% Uniqueness, both approaching 100%.

We believe these two statements are consistent. The confusion may stem from our earlier comment (Initial Rebuttal):

> We did not choose uniqueness and novelty as major metrics because recent research shows they can be flawed.

As stated in our last response, we agree that Uniqueness and Novelty are important. In our discussion, the additional thoughts we want to share is: relying **solely** on these metrics may be insufficient, as their ceiling may be the AddCarbon method, and as quoted from the new reference [1]: 

> The fact, that the simple AddCarbon model is useless in practice and still obtains good scores, casts some doubt if currently used metrics are sufficient to estimate performance.

This raises concerns and explains why we originally thought Uniqueness and Novelty might be flawed metrics. We appreciate the reviewer’s feedback and believe it would be more appropriate to revise the original sentence to: "Evaluating solely based on Uniqueness and Novelty may not be sufficient."

# Regarding "Diversity or similarity ... do not conflict with uniqueness and novelty ...":

Yes, we agree with the reviewer. 

# Regarding "Therefore, I believe that ..."

We appreciate the reviewer's suggestions and concur that evaluating Uniqueness and Novelty is important. We kindly note that this does not conflict with our additional thoughts on the issues of Novelty and Uniqueness discussed earlier.

Thank you again for your prompt response. We greatly appreciate the reviewer's time and suggestions and hope this discussion helps address the concern.

# New Reference 

[1] On failure modes in molecule generation and optimization. Drug Discovery Today: Technologies. 2019
Thank you for your prompt response.

Our multi-conditional generation setting differs from [1,2] in that we use true property combinations from the test set, rather than focusing on optimizing molecules toward a single target combination, i.e., the reviewer suggested QED $\geq$ 0.6, SA $\leq$ 4 (before scaling, according to [2]), GSK3 $\geq$ 0.5, JNK3 $\geq$ 0.5. This approach allows us to flexibly condition on various property combinations, especially for continuous conditions.

We can adjust the input conditions for Graph DiT to generate molecules with specific properties as requested by the reviewer. Below, we present the results for 1,000 generated molecules based on the input conditions: (1) QED randomly sampled from [0.6, 0.9], (2) SA randomly sampled from [1, 4] (before scaling), (3) GSK3β=1, and (4) JNK3=1. The success rate is 93.37%.

We greatly appreciate the reviewer's time and suggestions and hope this discussion helps address the concern.

# Reference

[1] Xie, Yutong, et al. "MARS: Markov Molecular Sampling for Multi-objective Drug Discovery." International Conference on Learning Representations, 2021.

[2] Jin, Wengong, Regina Barzilay, and Tommi Jaakkola. "Multi-objective molecule generation using interpretable substructures." International conference on machine learning. PMLR, 2020.
The results of W1 appear promising. However, I am uncertain whether these results demonstrate the model's ability to generate molecules that concurrently satisfy QED $\ge$ 0.6, SA $\le$ 0.4, the inhibition scores of GSK3β and JNK3 $\ge$ 0.5 [2]. Could you report the success rate (SR) of four properties as described in MARS [1] and RationalRL [2]; or alternatively, provide the top-$k$ average property score (APS) of four properties as described in DST[3] (Appendix C3).

MARS: Success rate (SR) is the percentage of generated molecules that are evaluated as positive on all given objectives (QED $\ge$ 0.6, SA ≥ 0.67 \{This should be SA $\le$ 0.4 \}, the inhibition scores of GSK3β and JNK3 $\ge$ 0.5); 

[1] Xie, Yutong, et al. "MARS: Markov Molecular Sampling for Multi-objective Drug Discovery." International Conference on Learning Representations, 2021.

[2] Jin, Wengong, Regina Barzilay, and Tommi Jaakkola. "Multi-objective molecule generation using interpretable substructures." International conference on machine learning. PMLR, 2020. 

[3] Fu, Tianfan, et al. "Differentiable Scaffolding Tree for Molecule Optimization." International Conference on Learning Representations, 2022.
I think the authors have a misunderstanding of evaluation metrics. The results in Table 1 of Ref [1] do not support the authors' claims. Adding carbon atoms directly leads to a validity score of 1, rather than indicating deficiencies in uniqueness and novelty. Diversity or similarity checks can examine similar substructures in generated molecules, but this does not conflict with uniqueness and novelty. For example, if a large number of the generated molecules already exist in the training set, diversity may be high, but novelty will be low. Therefore, I believe that evaluating uniqueness and novelty is necessary.
Thank you for your follow-up comment.

Regarding the evidence: Table 1 in Reference 1 indicates that the AddCarbon method achieves 99.94% Novelty and 99.86% Uniqueness, both approaching 100%. This supports the claim that "randomly adding carbons to existing molecules can yield..."

Based on this, Uniqueness may not always work as expected, particularly in cases as mentioned by the reviewer when evaluating a model "whether it only generates simple carbon chains"

For Novelty, consider three examples: `C` (a single carbon), `CC` (two carbons), and `c1ccccc1` (an aromatic ring). The Novelty metric produces a score of 1 when comparing `C` with either `CC` or `c1ccccc1`, yet the internal diversity metric, as used in the paper, better reveals structural differences when comparing `C` with `CC`/`c1ccccc1`.

We agree with the reviewer that Uniqueness and Novelty are important indicators and appreciate the feedback. Graph DiT has achieved a Validity of 0.9760, Novelty of 0.9702, and Uniqueness of 0.8919, which we believe demonstrates its ability to successfully model the data distribution of complex molecules. Additionally, we want to emphasize the outstanding performance of the condition control in Graph DiT. It is the primary focus of our paper, as previous methods have struggled to generate desirable small molecules or polymers for drug and material discovery. 

For further discussion on distribution learning, we refer to Lines 229-235, which remain relevant with the updated Novelty and Uniqueness results:
>GraphGA is a simple yet effective baseline for generating in-distribution molecules, e.g., on BBBP and HIV generation datasets. Diffusion model baselines such as DiGress and MOOD could produce diverse molecules but often fail to capture the original data distribution in multi-conditional tasks. Graph DiT shows the competitive performance of diffusion models in fitting complex molecular data distributions. Using fragment-based similarity and neural network-based distance metrics, we achieve the best in the polymer task and rank second in the HIV small molecule task, involving up to 11 and 29 types of heavy atoms, respectively. 
>

We appreciate the reviewer's follow-up questions and welcome further discussion on any concerns the reviewer feels have not yet been fully addressed.
# W1: Conditions on Gas Permeability

Thank you for your feedback. We believe that conditional generation based on multiple gas permeability properties is a challenging task.

First, **the condition space for gas permeability is vast**, as noted in Line 26, where gas permeability varies widely, exceeding 10,000 Barrier units. This variability presents significant technical challenges in controlling polymer generation across such a broad condition space.

Second, **the task relatedness in multiconditions does not reduce the difficulty but rather increases it**. As detailed in Section 4.3, polymers often require high permeability for one gas and low permeability for another, making it challenging for the generation model to capture the crucial differences between various gas permeability properties. The GSK3β and JNK3 properties mentioned by the reviewer also have relatedness, as both are serine/threonine protein kinases.

Finally, we have provided additional results to further clarify our model's performance (best results are highlighted for distribution learning and condition control).

| Method     | Validity ↑ | Coverage ↑ | Diversity ↑ | Similarity ↑ | Distance ↓  | Synth. (MAE↓) | QED (MAE↓) | GSK3$\beta$ (Acc ↑) | JNK3 (Acc ↑) |
|------------|------------|------------|-------------|--------------|-------------|---------------|------------|---------------|--------------|
| Graph GA   |          1 |        7/7 |      0.8727 |       0.9438 | **17.5076** |        0.8405 |     0.1864 |        0.6050 |       0.7240 |
| MARS       |          1 |        7/7 |      0.7010 |       0.6568 |     39.2837 |        0.7961 |     0.2053 |        0.6440 |       0.7390 |
| LSTM-HC    |      0.999 |        7/7 |      0.8739 |       0.9313 |     18.7856 |        0.8366 |     0.1864 |        0.6096 |       0.7608 |
| JTVAE-BO   |          1 |        5/7 |      0.6695 |       0.8567 |     48.8672 |        0.8614 |     0.2346 |        0.6280 |       0.7170 |
| DiGress    |      0.251 |        7/7 |  **0.8977** |       0.6508 |     32.5904 |        3.1438 | **0.1670** |        0.6693 |       0.7331 |
| DiGress v2 |      0.265 |        7/7 |      0.8976 |       0.7475 |     31.4630 |        2.9744 |     0.1772 |        0.6566 |       0.7358 |
| Graph DiT  |      0.852 |        7/7 |      0.8647 |   **0.9458** |     19.7717 |    **0.7430** |     0.1805 |    **0.9416** |   **0.9777** |

Due to time constraints, we sampled a subset of 600 data points from the kinase dataset provided by the MARS paper and split them into training, validation, and test sets in the same manner as described in the paper. We then generated 1,000 examples for evaluation. The results show that Graph DiT significantly outperforms other methods on the GSK3β and JNK3 properties, with strong distribution matching to the reference set.

# W3: VUN Metrics

Our model demonstrates reasonable values on the VUN metrics. We believe that a Validity of 0.9760, Novelty of 0.9702, and Uniqueness of 0.8919 illustrate Graph DiT's ability to successfully model the data distribution of complex molecules.

We agree with that reviewer that Novelty and Uniqueness are important perspectives in evaluating molecular generation models. We would also like to share additional thoughts on drug and material discovery. This field may focus on individual instances that have significant real-world impacts. A model capable of suggesting a single valid, novel, and unique drug or material that satisfies diverse property requirements, even if it produces many invalid ones (resulting in lower VUN metrics), may be promising as well compared to other models that generates numerous valid, novel, and unique suggestions but fails to meet specific condition requirements. 

We appreciate the reviewer's follow-up questions and welcome further discussion on any concerns the reviewer feels have not yet been fully addressed.
Thank you for the authors' responses. The authors' rebuttal does not convince me.

Firstly, I disagree with the authors' claim that "randomly adding carbons to existing molecules can yield almost 100% novelty and uniqueness. Instead, we use internal diversity to better reflect generation diversity." Additionally, I could not find evidence of the alleged deficiencies in uniqueness and novelty as mentioned in reference [1]. Could the authors provide evidence supporting the above claims? On the contrary, simply adding carbon atoms, such as carbon chains, tends to result in nearly 100% validity. Uniqueness is an effective measure to assess the model's distribution learning capability (i.e., whether it only generates simple carbon chains), while novelty serves as an indicator of whether the model is experiencing overfitting.

Based on the additional experimental results, the proposed DiT model's uniqueness is not high, at only 0.89. Furthermore, the validity (second to last) and novelty (third to last) do not outperform the baseline models. This suggests that the model's ability to learn molecular graphs (validity), distribution learning (uniqueness), and resistance to overfitting (novelty) are relatively trivial.
Thank you for your response. The feedback addressed several of my concerns; however, I still have a few remaining issues. Regarding W1, the conditions $O_2$, $N_2$, and $CO_2$ represent similar conditions without any competitive dynamics. Consequently, I would appreciate further clarification on the performance of the proposed method in relation to GSK3β+JNK3+QED+SA. For W3, Uniqueness and Novely may be flawed. However, a high VUN value does not necessarily indicate high performance of generation, whereas a low VUN value indicates underperformance in the generative process.
We are encouraged that your concerns have been addressed. Thank you again for your valuable and helpful feedback! We welcome any new discussion.
Your response has addressed all my concerns. I would like to increase my rating to 6.
Thank you for your prompt feedback. Yes, it is the balanced subset with 2,372 examples. All the information about the dataset is provided in Table 4 (Appendix).
Am I correct in assuming that the dataset employed in your study is a balanced subset of the HIV dataset, which includes an equal number of positive and negative samples, rather than the complete dataset?