Extracting Training Data from Molecular Pre-trained Models

Authors: renhong  huang
jiarong  xu
zhiming  yang
xiang  si
xin  jiang
hanyang  yuan
chunping  wang
yang  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:04:42

Abstract: Graph Neural Networks (GNNs) have significantly advanced the field of drug discovery, enhancing the speed and efficiency of molecular identification. However, training these GNNs demands vast amounts of molecular data, which has spurred the emergence of collaborative model-sharing initiatives. These initiatives facilitate the sharing of molecular pre-trained models among organizations without exposing proprietary training data. Despite the benefits, these molecular pre-trained models may still pose privacy risks. For example, malicious adversaries could perform data extraction attack to recover private training data, thereby threatening commercial secrets and collaborative trust. This work, for the first time, explores the risks of extracting private training molecular data from molecular pre-trained models. This task is nontrivial as the molecular pre-trained models are non-generative and exhibit a diversity of model architectures, which differs significantly from language and image models. To address these issues, we introduce a molecule generation approach and propose a novel, model-independent scoring function for selecting promising molecules. To efficiently reduce the search space of potential molecules, we further introduce a Molecule Extraction Policy Network for molecule extraction. Our experiments demonstrate that even with only query access to molecular pre-trained models, there is a considerable risk of extracting training data, challenging the assumption that model sharing alone provides adequate protection against data extraction attacks. Our codes are publicly available at: \url{https://github.com/renH2/Molextract}.

Main paper: 
 introduction: 
 Deep learning has revolutionized various scientific disciplines, inspiring researchers to adopt these advanced techniques in drug discovery to accelerate molecule identification while reducing costs. Molecules are commonly represented by molecular graphs, capturing essential structural information. Consequently, Graph Neural Networks (GNNs) have demonstrated effectiveness in tasks like property prediction [13, 59] , drug discovery [53, 32] , and drug design [34] . However, training these GNNs faces a significant challenge known as "data hunger" [17] ; that is, a substantial amount of molecular data is required for training. For instance, developing a new drug often involves understanding intricate molecular behaviors and responses, which can only be achieved through the analysis of extensive molecular data. This massive data requirement often exceeds what any single organization can collect and maintain on its own. This limitation leads to the necessity of collaborative efforts. Nevertheless, the direct sharing of data often raises concerns about commercial confidentiality and information privacy [8, 42] . In view of this, various graph pre-training techniques emerge as viable solutions. These techniques have demonstrated remarkable generalizability across various molecular datasets [54, 19, 22, 52] , facilitating model-sharing collaboration. Organizations can leverage these advancements by sharing molecular pre-trained models that have been trained on proprietary molecular datasets without compromising data privacy. Then, model users are able to query these pre-trained models without access to any training data.However, such model-sharing collaboration, while highly beneficial, is not devoid of vulnerabilities. One significant risk is the susceptibility to data extraction attacks [6, 7] , where those with malicious intentions may attempt to access the private molecular training data. Such breaches could potentially compromise commercial secrets, violate privacy regulations, and undermine trust among collaborative partners [28] . As the first in the literature, this work studies the problem of molecular data extraction attacks, aiming to explore the risks of extracting training data from molecular pre-trained models.Extensive work has been done on data extraction attacks in the realms of image and text, suggesting that training data can be extracted from pre-trained models due to memorization effects [6, 7] . Yet, these methods are not applicable to molecular pre-trained models for three key reasons. Firstly, most of the existing data extraction attacks target at generative models (e.g., transformers and diffusion models) in text and image domains. From generative models, training data can be easily inferred via simple prompts (as depicted in Figure 1 (a)). In comparison, most molecular pre-trained models are not generative. Instead, users of these models can only query the model with molecular graph and then obtain corresponding representations of this graph [62, 53, 18, 19, 55] , which obstructs the direct extraction of molecular data (see Figure 1 (b)). Secondly, while models in image and text domains typically employ widely recognized architectures like diffusion models [44] and transformers [43] , molecular pre-trained models feature a much greater diversity in their architectures and training tasks, such as contrastive learning [47, 62, 53] , context prediction [18, 55, 51] , and graph reconstruction [19, 45] . These architectures often remain undisclosed to potential adversaries, adding an additional layer of complexity to any attempt at data extraction. Lastly, the vast combinatorial possibilities of molecules, estimated to number around 10 60 [38] , introduce a level of complexity that necessitates highly efficient and specialized methods for extracting molecular graphs.In this paper, we first generate molecule candidates by combining a defined template structure, motif banks, and bond connectivity, all within the bounds of established chemical constraints. This serves as an alternative approach for molecule generation when direct data extraction from non-generative pre-trained models is not feasible. With these molecule candidates, we introduce a novel scoring function to determine whether potential molecules belong to the training data of a pre-trained model. This scoring function is model-independent, making it applicable across various architectures of molecular pre-trained models. To further reduce the search space for molecules and efficiently extract them, we introduce a Molecule Extraction Policy Network for generating those with high-scoring functions and meet the valency rule through reinforcement learning (RL). Extensive experiments break the illusion that sharing molecular pre-trained models, rather than raw data, adequately protects against data extraction attacks: despite only having query access to these black-box models, our findings reveal a significant risk of training data being extracted with an average precision of 49.0%. 
problem formulation: 
 In this section, we outline the scenario, describe the adversary's knowledge, and define the problem associated with molecular graph extraction attacks.Scenario. Adversary's knowledge. The adversary (in this case Company B) has black-box access to the molecular pre-trained model. This access allows the adversary to query the model with a molecular graph and receive the corresponding graph representation in return, without any insight into the model architecture or the specific pre-training tasks it underwent. This setting mirrors common situations in the industry, particularly for models that are accessed via an API while keeping their internal workings undisclosed [2, 23, 27] .Additionally, adversaries may possess an auxiliary dataset (G aux ), which can be used to assist with data extraction efforts. This assumption is reasonable given that such an auxiliary dataset can be sourced from publicly available molecular databases like ChEMBL [14] , PubChem [51] , ZINC15 [46] , or it could be some data held by the adversaries themselves.Molecular graph extraction attack. In this paper, we explore the potential risk of molecular pretrained models leaking their training molecule data. To thoroughly investigate this risk, we formally define the problem as follows:Problem 1 (Molecular Graph Extraction Attack) Given the molecular pre-trained model f that has been pre-trained on a private dataset G, adversaries who only have query access to the model and access to an auxiliary dataset aim to obtain a subset of graphs G adv that exist within G.Here, the private information is defined as the molecular graphs within G. Since we are investigating the risks posed by molecular graph extraction attacks, we consider it sufficiently risky to deduce only a portion of the graphs or those similar to G. 
methodology: 
 We first introduce the molecule generation process to generate molecule candidates in §3. [39] , then the diversity of the extracted molecules would be restricted. Rings uniquely meet both criteria: they are prevalent across numerous molecular families, satisfying the common criterion, and as a functional structure, they significantly influence molecular stability, reactivity, and interactions with other molecules [9, 48] . Recognizing these advantages, we select rings as the template structure for molecule generation.Motif bank. After a template structure is selected as the starting point for molecule generation, adversaries can then attach various molecule building blocks. Common choices include atoms [61, 31] and motifs [25, 60] . Yet, atoms might not be informative attachments, due to the limited structural information an individual atom can provide. Moreover, atoms may form atypical chemical fragments, such as alternating bond patterns that form incomplete aromatic rings [33] . Therefore, we opt for motifs as the building blocks for molecule generation. In our implementation, we use the 91 common motifs extracted by [33] . These common motifs constitute the motif bank.Bond connectivity. Once we establish the template structure and motif banks, the next step is to consider the connectivity between the template structure and the motifs. Molecules are generated by forming bonds through specific attachment positions on the template structure and motifs. However, it is often not legal to attach an arbitrary bond to an arbitrary position. So expert chemical knowledge is needed in this process, and common chemical constraints should be satisfied. In this work, we obtain feasible attachment positions by utilizing CReM [37] for the decomposition of molecules in the auxiliary dataset G aux .Given a set of template structures and a motif bank, we first select a template structure R as the starting point and choose a motif M from the motif bank. Then, a bond B = {a R , a M } is formed between the template structure R and the motif M , subject to the satisfaction of chemical constraints, where a R and a M represent the attachment points in R and M respectively. This bonding results in Ĝ as the union of R and M with the bond B, which can be represented as Ĝ := R ∪ B M . In subsequent generation steps, we can take Ĝ as our new starting point and select additional motifs and bonds. By repeating this process, we gradually construct a potential molecule.This subsection introduces a scoring function to determine the probability of the existence of Ĝ in the private training dataset G. The scoring function should be independent of any specific model architectures or pre-training tasks. In the following, we first define the scoring function, and then explain its rationality.Since adversaries can only query the molecular pre-trained model to obtain representations, we derive insights from the representations of template structure R, the motif M , and their combined structure with bond B, denoted as Ĝ, as provided by the pre-trained model f . We define the scoring function as follows:EQUATIONwhere α ∈ [0, 1] is a hyper-parameter, f ( Ĝ), f (R), and f (M ) are representations of Ĝ, R, and M respectively, and Sim(•, •) can be defined as cosine similarity or other forms of similarity measure.Rationality of scoring function. The crux of the scoring function's rationality lies in the observation that, for molecular pre-trained model, the relationship between representations f ( Ĝ), f (R), and f (M ) exhibits distinct patterns depending on whether Ĝ is present in the private training dataset.Consider Figure 2 sequently, there exists a specific relationship: f ( Ĝ) can be effectively approximated as a linear combination of the representations of two other molecules, f (R) and f (M ), in the representation space, that is,f ( Ĝ) ≈ αf (R) + (1 -α)f (M ). On the other hand, from the bottom row of this figure, if Ĝ does not exist in G, it is highly likely that f ( Ĝ) is dissimilar with αf (R) + (1 -α)f (M ).Additionally, we show that the scoring function is related to specific molecular pre-trained models with various α values in Eq. ( 1). For instance, when the molecular pre-trained model employs bond-deletion augmentation in graph contrastive learning [62, 53] , the value of α in Eq. ( 1) is approximately 1 2 . For the molecular pre-trained model that employs subgraph augmentations in graph contrastive learning [45] , the value of α in Eq. ( 1) is approximately 1. Detailed proofs supporting these examples, as well as rationale behind the scoring function can be found in Appendix A.5. Furthermore, our analysis of the distribution of α values across different pre-trained models in §4.2 provide additional evidence.Learning scoring function using auxiliary dataset. However, directly relying on the scoring function Eq (1) to extract training data still presents two challenges.Firstly, the graph representations obtained from the pre-trained model may not be optimally suited for data extraction due to the discrepancy between the pre-training tasks and the task of data extraction. To mitigate this issue, we introduce an adapter g θ , instantiated as an MLP with learnable parameters θ. This adapter is designed to project the representations obtained from the pre-trained model f into another representation space that is more conducive to facilitating a graph extraction attack. The transformation of representations is achieved through the mapping g θ • f (•) = g θ (f (•)), with the hope that the output is specifically tailored for the extraction task.Secondly, treating α as a fixed hyper-parameter in the scoring function introduces challenges in adaptability. A fixed α may not adjust dynamically to different contexts or datasets, potentially limiting the attack's adaptability and leading to suboptimal performance. To overcome this limitation, we introduce a more adaptive mechanism that can optimize α in response to changing contexts, by modeling α as a function ofα = h ϕ ([f (R); f (M ); f ( Ĝ)]).Based on the above solutions, we transform the scoring function Eq. ( 1) into a learnable form:EQUATIONWe utilize the information contained in the auxiliary dataset G aux to learn the parameters {θ, ϕ}.The key idea is that if the scoring function, parameterized by {θ, ϕ}, can effectively determine the presence of a graph Ĝ within G aux , it is likely to generalize to the private training dataset G. The training process can be formalized as follows:EQUATIONwhere ℓ CE is the cross-entropy function, 1 {•} is the indicator function, and E Ĝ is the mathematical expectation taken over all the possible generated molecules Ĝ.To conduct the molecular extraction attack, the most straightforward way is to enumerate all possible generated molecules and rank them using a specified scoring function, and then select those with the highest scores. This approach, inevitably, leads to an exponential increase in complexity due to the vast number of possible combinations.Given that the molecular generation process involves the iterative selection of template structures and motifs to form bonds, it naturally aligns with the Markov Decision Process (MDP) framework [41] ,where each decision is based on the current state and leads deterministically to a new state. This sequential decision-making property allows for a structured exploration of the molecular space. We therefore introduce a Molecule Extraction Policy Network for molecular graph extraction attacks through RL, which significantly narrows the search space for molecules. This network strategically guides the selection of motifs and attachment points, focusing on the most promising options. We further detail our design.State space. The state at time step t, denoted as S t , is defined as the graph Ĝt generated up to that point. The initial state Ĝ0 represents the template structure R, serving as the starting point for the molecular generation process.Action space. At time step t, the RL agent selects a motif M t from the motif bank and determines the best attachment positionsB t = {a Ĝt-1 , a Mt }, resulting in the updated graph Ĝt := Ĝt-1 ∪ Bt M t .More specifically, the action at step t involves three stages: (1) Selecting attachment position a Ĝt-1 on Ĝt-1 ; (2) Choosing a motif M t from the motif bank; (3) Selecting attachment position a Mt on M t to form the bond B t . In summary, the action at step t can be expressed asA t = {a Ĝt-1 , M t , a Mt }.Reward design. We employ both delayed reward and intermediate reward to guide the molecular generation. For the delayed reward, we instantiate the reward r as the scoring function and extend it over multiple steps as follows:EQUATIONwhere β i represents the weight for combining rewards from different Ĝi on the trajectory. Intuitively, if Ĝt-1 exists in G, then the generated Ĝt based on Ĝt-1 is more likely to exist in G. Therefore, we consider the molecule generation process as a whole and accumulate rewards by summation. Additionally, when t is small, the corresponding weight of the reward should be relatively high, whereas when t is large, the weight should be relatively low. Here, we set β i to 0.99 i .Regarding intermediate rewards, a positive reward δ is allocated when the generated molecules do not violate valency rules [36] , ensuring that each atom has not exceeded its maximum possible valency. For molecules that fail to pass valency rules, the intermediate rewards are set to zero.Policy network. To enable the RL agent to predict actions effectively, obtaining accurate molecular representations is crucial. We utilize a GNN to learn representations from molecules, a method proven effective for learning molecular representations [10] . We can then obtain representations of attachment position z(a Ĝt-1 ) and z(a Mt ). For graph representations, such as motif representations, we apply sum pooling to derive the graph representation, represented as z(M t ).Based on the representations, three networks (π first , π second , and π third ) are designed to predict the action A t = {a Ĝt-1 , M t , a Mt } across three stages. For the first stage, the RL agent selects an attachment position from Ĝt-1 according to the network π first , i.e.,EQUATIONwhere π first outputs the probability distribution p first t of a Ĝt-1 . We then obtain a Ĝt-1 by sampling according to the probability π first . For the second stage, the RL agent tries to select the motif M t from the motif bank based on selected a Ĝt-1 , i.e.EQUATIONwhere π second takes in the representations of attachment position a Ĝt-1 selected in stage 1 and motif M t , outputs the probability p second t (M t ) of selecting motif M t . Finally, given selected a Ĝt-1 and M t , the agent selects attachment position a Mt in motif M t as:EQUATIONwhere π third outputs the probability distribution of a Mt . In the implementation, the three policy networks, π first , π second , and π third , consist of MLP layers with ReLU activations, followed by a softmax layer to predict the probabilities p first , p second , and p third , respectively.Policy gradient training. To enhance the exploration capability of the RL agent in capturing more molecules from G, we leverage the Soft Actor-Critic framework [15] . Soft Actor-Critic integrates the entropy measure of the policy into the reward to promote the exploration of molecular generation. By maximizing entropy, we can obtain molecules with both high scores and diversity. Specifically, the policy network is trained with the objective as follows:EQUATIONwhereH(π(• | S i ))is the entropy measure of the action distribution given the state S i and τ , known as the temperature parameter, controls the trade-off of exploration for molecules. The detailed modifications to the Soft Actor-Critic optimization can be found in Appendix A.3.Reward function initialization and update. Since the reward function depends on the quality of the scoring function's training, and the scoring function's training, in turn, depends on the quality of the generated graphs, we consider initializing the scoring function for a warm-up phase. We first enumerate all possible molecules constructed by appending a motif to the template structure, and use them as the distribution of the generated graph to pre-train adapters g α and h ϕ using Eq. ( 3). During the training process, as the quality of the generated graphs improves, the generated molecules can, in turn, enhance the RL learning framework. Specifically, we adjust the scoring function using the generated molecules as the distribution of the generated graph and training with Eq. (3).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments: 
 In this section, we evaluate the performance of molecular extraction attacks against different molecular pre-trained models. Besides, we conduct case studies, and runtime analyses to underscore the effectiveness of our approach. Additional results can be found in Appendix A.4.Dataset. In our experiment, we used datasets containing 2 million molecules sampled from ZINC15 [46] as the pre-training dataset G, and an additional 20,000 molecules as the auxiliary dataset G aux . The detailed statistics information is provided in the Appendix A.3.Molecular pre-trained models. We selected the most common and widely used molecular pretrained models from each category to demonstrate the versatility of the proposed method. These methods include: (1) Contrastive Learning: GraphCL [62] , SimGRACE [56] InfoGraph [47] ; (2) Graph Reconstruction: GraphMAE [18] , AttrMasking [19] , EdgePred [16] , Mole-BERT [55] ; (3) Context Prediction: ContextPred [19] , Grover [45] . Notably, the encoder architectures of Mole-BERT and Grover are based on Transformers or BERT structures and all the molecular pre-trained models are trained using the default hyper-parameters specified in original papers.Baselines. Since existing methods are not designed for molecular extraction, we first tailor other methods to fit our setting. We enumerate all the potential molecules (constructed within one or two steps generation) and use metrics to select molecules as the prediction of G aux . Our baselines can be roughly categorized into two groups: chemical property-based methods and learning-based methods.For chemical property-based methods, we compare the QED score [3] , a common estimation of drug-likeness, which predicts the drug-like potential of a molecule. Additionally, the SA(Synthetic Accessibility) score is considered to measure the synthetic accessibility and rationality of molecules [11] .We also introduce the Docking score [49] as our scoring function baseline to estimate the binding affinity between a ligand (small molecule) and a receptor (protein target). Specifically, we obtain A molecular graph extraction attack is considered successful if a graph in G adv exists in G or if G adv is similar to G. Therefore, assuming the model has generated G adv with K molecules, we adopt the following metrics to measure the performance of the extraction attack:• Precision measures the ratio of generated molecules that exist within the G. The larger the precision is, the better the performance of the molecular extraction attack. • FCD, also known as Fréchet ChemNet Distance, offers a distance measure between G and G adv .This metric leverages ChemNet [40] to capture the differences in both the chemical and biological properties of the molecules. A lower FCD indicates that G and G adv are similar in terms of chemical and biological properties, suggesting better extraction performance.Molecular extraction results. Table 1 demonstrates the superior performance of our model over baselines across various molecular pre-trained models. Chemical property-based methods generally underperform, likely due to the infrequency of target properties in pre-trained datasets (e.g., QED). The better performance of SA indicates that molecular stability could be a significant indicator of molecule presence in real datasets. It is evident that our reinforcement learning approach significantly outperforms MLP method on precision and FCD across several pre-trained models, with average improvements of 30.9% and 3.97%, respectively, highlighting the effectiveness of our method.We can also observe that AttrMasking is the most vulnerable to privacy leakage among the molecular pre-trained models. Furthermore, we have also compared the performance of proposed model under different model frameworks, and it consistently succeeds in molecular graph extraction across various Case study. We further investigate the behavior of the scoring function under various molecular pre-trained models. We explore the α distribution as shown in Figure 3 . It is evident that selfsupervised tasks within the same category exhibit a similar pattern in their α distributions, whereas models from different categories display distinct distribution patterns. For graph reconstruction-based models, the α distributions are predominantly centered around 0.35, with a peak indicating a high concentration. In contrast, for contrastive learning-based models, the α distributions are flat, which may be attributed to the inherent randomness in the augmentations used in contrastive learning. As for context prediction tasks, the α distributions are centered around 0.54. This phenomenon provides an explanation for the rationality of the scoring function in black-box scenarios.Runtime analysis. Table 3 compares the runtime of the proposed method with the baselines across the two categories: chemical property-based methods, and learning-based methods. The proposed method exhibits superior extraction attack performance while maintaining a runtime that is comparable to others. This efficiency is achieved through the explorative RL framework, which replaces the need for exhaustive enumeration of thousands of molecules, significantly reducing the time required for molecular graph extraction attacks. 
related works: 
 Molecular pre-trained models. Molecular pre-trained models utilize GNNs to capture the intricate non-Euclidean structure of molecular graphs, employing various self-supervised pre-training tasks to enhance generalization [57, 22, 12, 5, 21, 58] . Training on extensive molecular graphs, molecular pre-trained models can acquire generalized molecular graph representations and patterns, thereby benefiting various downstream tasks in the molecular domain [13, 53, 1, 30, 32] . Molecular pretrained models typically employ self-supervised tasks as follows.(1) Contrastive Learning [47, 62, 53] . The objective of the contrastive pre-training task is to capture the similarities and dissimilarities between instances of subgraphs at the molecular level or motif level. (2) Graph Reconstruction [18, 55, 51] . Certain components (such as atoms, bonds, properties of atoms, and fragments) of molecules are masked out, and models are trained to recover components based on the remaining information.(3) Context Prediction [19, 45] . The objective of graph context prediction is to utilize subgraphs to make predictions of surrounding graph structures. This is achieved by classifying whether a specific neighborhood component and surrounding context belong to the same node within the ego-graph.Effectiveness and reliability of model can be compromised by adversarial attacks in various forms [63, 58] . Pre-trained models contain a large amount of knowledge, and data extraction attacks are among the methods aimed at extracting training data from these models [4, 24] .Research in this area can be broadly classified into two categories: one uses membership inference to deduce information from generative models, while the other exploits the memorization mechanism of networks to carry out attacks. In the first category, [6] generates text from pre-trained language models and performs membership inference attacks to filter the generated text for extraction. In the second category, [26] demonstrated that the effectiveness of data extraction is due to duplication in commonly used web-scraped training sets [26] . [20] analyzed the extracted text from pre-trained language models and found these models do leak personal information as a result of memorization. However, all the aforementioned studies focus solely on the extraction from generative pre-trained models and do not adequately address the challenge of extracting data from graph pre-trained models. 
broader impacts: 
 We recognize that our investigation into Molecular Graph Extraction Attacks on graph-pretrained models could be misused, particularly in collaborative model-sharing, where it may lead to privacy risks. However, we emphasize that our primary objective is to identify vulnerabilities in graph pretrained models, and support the creation of more effective defense strategies. To this end, the paper assesses the susceptibility of mainstream graph-pretrained models to the attack, underscoring the need for enhanced defense measures for existing work.Furthermore, we propose the following potential defense strategies: (1) Behavior Detection: Implement systems for continuous monitoring and identification of malicious queries in shared models to protect data integrity. (2) Prediction Perturbation: Since the efficacy of model extraction attacks is influenced by embeddings, we suggest introducing minor noise into the final outputs of graphpretrained models without significantly affecting performance. We believe this ongoing interplay between attack and defense will foster a more robust research community, contributing to future studies on defense strategies. 
conclusion: 
 The for each environment setup do 6:Obtain A t = {a Ĝt-1 , M t , a Mt } by sequentially compute Eq. ( 5), Eq. ( 6) and Eq. ( 7). 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: sufficient information on the computer resources (type of compute workers, memory, time of execution are provided in §4 and A.3Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The paper discusses the negative societal impacts of attacks in § 2 and § 6.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: the paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification: this paper does not use existing assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Ĝt:= Ĝt-1 ∪ Bt M t 8:Compute the overall reward r(S t , A t ) and intermediate reward.Optimization for Soft Actor-Critic.10:end for 11: end for 12: Utilizing the trained reinforcement learning agent, generate K molecules for G adv .The detailed statistics of the pre-training dataset and auxiliary dataset. The pre-training dataset consists of 1,883,524 molecules, while the auxiliary dataset contains 20,000 molecules. There is an overlap of 103 molecules between these datasets, indicating a relatively small overlap ratio.To enhance computational efficiency, we subdivide both the pre-training and auxiliary datasets based on template structures. For each template structure, we select molecules containing the corresponding template from the original dataset to create a tailored dataset. Subsequently, we perform attacks and evaluations using these customized pre-training and auxiliary datasets.Additionally, to evaluate the robustness of our attack method, we utilize a set of hard auxiliary datasets. For a given template structure, we select 80% of the molecules from the auxiliary dataset to form a hard auxiliary dataset. This hard auxiliary dataset has a higher Fréchet ChemNet Distance (FCD) from the pre-training dataset compared to the original, resulting in reduced similarity and increased difficulty of the attack. Below are the statistics for some of these datasets: Baseline model description. The models we have chosen include:Index• GraphCL [62] is a contrastive self-supervised learning method for GNNs, which learns representations by maximizing the agreement between differently augmented views of the same graph.• SimGRACE [56] utilizes the original graph as input, and employs a GNN model along with its perturbed variant as dual encoders. Then model conducts two correspondingly linked perspectives for contrastive learning without the need for data augmentation. • InfoGraph [50] is a pre-training approach that maximizes the mutual information between the local patch representations and global graph representations, encouraging the model to capture local and global graph structures. • GraphMAE [18] is a graph-based model that uses a masked autoencoder framework for pretraining, which learns to reconstruct masked parts of input graphs, enabling the model to capture intrinsic graph structures. • Attribute Masking [19] aims to capture domain knowledge by learning the regularities of the node/edge attributes distributed over graph structure. • EdgePred [16] is a pre-training task where the model learns to predict whether an edge (relationship) exists between two nodes in a graph, which helps the model to understand the connectivity and relationship between nodes. • Mole-BERT [55] Description of baselines. We compare with the following baselines.• MLP trains a multi-layer classifier to predict the graph Ĝ's existence in G. The classifier takes in Ĝ's representation f ( Ĝ), and is trained to predict Ĝ's existence in G aux .• QED score [3] stands for quantitative estimation of drug-likeness, predicting the drug-like potential of a molecule. The higher the QED score, the more likely the molecule is a drug. Based on the assumption that a drug-like molecule is likely to appear in G, a potential molecule with a high QED, is predicted to exist in the pre-training dataset. • SA score [11] stands for synthetic accessibility score, estimating the ease of synthesizing a particular molecule. SA scores typically range from 1 (easily synthesizable) to 10 (difficult to synthesize), with a lower SA score indicating that the molecule is easier to synthesize. Therefore, we use the negative SA score to score molecules. • Docking score [49] estimates the binding affinity between a ligand (small molecule) and a receptor (protein target). A more negative docking score indicates a stronger binding interaction between the ligand and the receptor, implying a more favorable binding event. Therefore, we use the negative docking score to score the potential molecule Ĝ. Specifically, we get 3 variants of docking scores with three different protein targets FA7, PARP-1, 5-HT1B.Detailed implementation details. Since existing methods are not tailored for molecular extraction attacks, we initially adapt other methods to our specific scenario. We enumerate all potential molecules (constructed through either one-step or two-step generation) and employ metrics to select molecules as predictions for G aux .For chemical property-based methods, we utilize the rdkit toolkit [29] for computations. Meanwhile, for learning-based methods, we employ a two-layer MLP classifier with a hidden size of 300.In constructing our reinforcement learning framework, we employ the Soft Actor-Critic (SAC) algorithm as implemented in OpenAI's SpinningUp, as well as the molecular reinforcement learning efforts [60] . Regarding the parameters for the RL agent, we set the total number of training epochs to 100, with β i as the weight for delayed reward and δ = 0.05 for intermediate rewards. The graph adapter within the scoring function of the delayed reward is also a two-layer MLP with a hidden size of 300. We employ the Adam optimizer with a learning rate of 0.005 for 100 epochs during the pre-training phase. After 5 epochs, the scoring function is trained using the generated molecules. For policy training, we implement three policy networks with two-layer MLPs and a hidden size of 128. The graph representation network utilizes a two-layer GCN with a hidden size of 128. We update the policy network after generating 256 molecules, and set the temperature τ to 1. The policy networks are trained with the Adam optimizer, using a learning rate of 0.01 and a weight decay of 1e-6. All experiments are conducted on a single machine of Linux system with an Intel Xeon Gold 5118 (128G memory) and a GeForce GTX Tesla P4 (8GB memory).Modification to the optimization Soft Actor-Critic. Similar to the optimization process in [15] , calculating entropy requires computing log π(A i | S i ). In our setup, the specific method for this calculation is as follows:EQUATIONRationality behind ring structures. We here clarify rationality behind choosing ring structures as a starting template is that ring structures are very common in chemical datasets. In the ZINC15 dataset with 2 million unlabeled molecules, we identified 1,990,890 molecules containing ring structures, constituting the majority of the dataset. Furthermore, the diversity of ring structures offers a wide range of options for template structures (such as tetrahydrofuran and cyclobutane).We further incorporate five ring-free scaffold templates (i.e., alkanes) into the template bank. The performance is shown as follows. It can be observed that incorporating these ring-free templates has led to a slight improvement in performance. Result for multiple step generation. Our approach based on reinforcement learning can be extended to multiple steps. Here, we extend the time-step, and the results are as follows (we only take "Random" as a baseline considering the runtime of other baselines). It can be observed that as the time-step increases, performance may decline due to the increased difficulty of extraction caused by the complexity of the molecules. Nevertheless, our model still outperforms the baseline model in precision, indicating the effectiveness of our extraction method. Performance with PebChem as the auxiliary dataset. We also sampled 20,000 molecules from PubChem [51] as the auxiliary dataset to ensure a difference from the ZINC pre-training dataset.The results based on the GraphCL pre-trained model are shown as follows. We observed that under these scenarios, the performance of our method had a slight decline. However, it still demonstrates comparable efficacy and showcases robustness. Performance under regression task. We further explored data extraction attacks on regression models. When integrating regression tasks, we adapted our model by replacing the final output of the regression with the representation in Eq. ( 2). In our implementation, we chose the real-world chemical dataset FreeSolv [35] and regressed the hydration-free energy, utilizing 5% of the molecules from FreeSolv as an auxiliary dataset. The detailed results are as follows. We discovered that our model still performs well with the regression model. Here, we provide a detailed explanation of the example presented in § 3.2 along with its proof, which further indicates that our scoring function can effectively characterize different molecular pre-training tasks.Example 1 When the pre-trained model performs specific subgraph masking as described in [45] , the value of α in Eq. ( 1) is approximately 1. When the pre-trained model is involved in using bond-deletion augmentation in graph contrastive learning, as in [62, 53] , the value of α in Eq. ( 1) is approximately 0.5. EQUATIONWhen a pre-trained model employs subgraph masking, it can ensure that the representation of the graph with masked motifs remains consistent with the original graph's representation. That is,f ( Ĝ) = f (R). In this case, the form of our scoring function is as follows:EQUATION) By combining Eq. ( 10) and Eq. ( 11), we have:EQUATIONBy adjusting α in Eq. ( 12), we can observe that the scoring function reaches its maximum when α = 1.When a pre-trained model is involved in using bond-deletion augmentation in graph contrastive learning, where the representation of the graph with the bond between M and R removed is made similar, it is easy to obtain the following relationship:f (R) ≈ f (R), f (M ) ≈ f (M ). By substituting it into Eq. ( 11), we can obtain the following results:EQUATIONApparently, the scoring function reaches its maximum value α = 0.5.Proof for the rationale behind the scoring function. Assume that graph G is composed by G := G 1 ∪ G 2 and the graph pre-trained model is represented by f . Let the loss function for the pre-training task be L, which takes graph representation as input. Further, we assume that L is a bijection and linear mapping.Without loss of generality, we assume that the loss function belongs to the category of weighted sum, that is L(f (G)) = α 1 L (f (G 1 )) + α 2 L (f (G 2 )), with α 1 and α 2 serve as hyper-parameters (This assumption is common among various tasks. For instance, in the most common case of cross-entropy for the classification task,α 1 = |G 1 | /|G| and α 2 = |G 2 | /|G|.). We can infer that f (G) = L -1 (α 1 L (f (G 1 )) + α 2 L (f (G 2 ))) = α 1 f (G 1 ) + α 2 f (G 2 ).Therefore, given the theoretical analysis, we can observe that the relationship between f (G), f (G 1 ), and f (G 2 ) is akin to a weighted combination, which justifies our design of score function. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: All reviewers except one (qKJK) argued for accepting the paper. For qKJK their main conerns were on (a) how realistic the problem setup is, (b) the score function lacking theoretical or chemical justification, (c) missing regression tasks. The authors respond convincingly to each point. For (a) they point out, as the reviewer does, that sharing a classification/regression model poses a higher privacy risk because of the potential leakage of confidential label data. For this reason there are at least two papers which have proposed sharing graph representation learning models instead. Two other papers have also highlighted the necessity for a generic pre-trained representation learning model. As far as I am aware this paper is the first to point out the issue of extracting training data from these models and to give an attack to extract such data. The authors give a convincing real-world example of a collaboration between 10 pharmaceutical companies who jointly trained a model for structure-activity. They cite a paper that describes how the companies had privacy concerns regarding their molecule structures, treating them as confidential. They cite two other papers on how molecular structures are considered sensitive IP in various settings.. For (b) they provide a better intuition of the score function, and for (c) they provide new results on a regression tasks. In response, Reviewer qKJK still argued against (a), saying that the motivation is not realistic. They argue that because there are large public databases of molecular structure then this shouldn’t be an issue. This completely ignores the author’s response that there are many scenarios where molecular structure privacy is an issue. Therefore, I find this point resolved. As all concerns are resolved, I vote to accept. Authors: you’ve already made improvements to respond to reviewer changes, if you could double check their comments for any recommendation you may have missed on accident that would be great! The paper will make a nice contribution to the conference!

Reviews: review: paper_summary: paper_summary: This paper investigates the issue of data leakage risks when using pre-trained molecular models in a shared environment and proposes a method to extract training data from such pre-trained models. Specifically, the authors employ a molecular generation method based on templates and a candidate motif bank to attempt to generate potential training data. By designing a scoring function, the model can distinguish substructures that belong to the training data, and using a policy-based reinforcement learning method, the exploration space for generation is narrowed. Finally, the authors evaluated the method on various models trained using ZINC15 dataset.
strength_weakness: strength_weakness: While I recognize the research significance and model solution of this work, some design aspects could be improved, specifically:

- Regarding the selection of template structures, the paper ultimately chooses rings as the starting structures for subsequent generation/growth. However, a proportion of molecular data does not contain rings, so this heuristic choice of starting templates might not be the best strategy. The authors could consider constructing a template bank to avoid this issue.

- Are the 20k molecules used in the experiments on line 269 also obtained from ZINC15? Is this a strong iid assumption? I would like to see the effect of data extraction when the auxiliary dataset is replaced with molecules having significantly different distributions, for example, selecting some molecules from the PubChem dataset.

- The meaning of "step" in Table 1 seems unclear. Is it referring to the time-step mentioned earlier? If so, why only explore the case of one and two steps? What characteristics would molecules generated with more steps have? This seems to lack discussion and analysis.

- The author explores few model backbones, so I suggest the authors also conduct experimental evaluation on Graph Transformer or other architectures. And the pertaining data set is also limited, which would be more convincing if another one dataset was added.
questions: questions: A key metric in the experimental evaluation is Precision. How do you determine whether the generated molecules exist in the dataset? Since molecules lack unique identifiers and it is easy to consider two identical molecules as different mistakenly, I hope the authors can provide clearer details on how the presence of data is judged.
limitations: limitations: Please refer to Weaknesses and Questions.
paper_summary: paper_summary: This paper explores the vulnerabilities of molecular pre-trained models to data extraction attacks. The authors introduce a novel molecule generation approach and a model-independent scoring function to identify molecules potentially originating from private datasets. They also present a Molecule Extraction Policy Network to optimize the search process for high-scoring molecules.
strength_weakness: strength_weakness: 1. The experiments might not fully capture the diversity and complexity of real-world datasets, which could affect the generalizability of the findings. I recommend conducting additional experiments using a wider variety of datasets to better evaluate the model's performance in real-world scenarios.
2. The scoring function is based on G, which is a linear combination of the representations of R and M. However, considering that the target model is treated as a black-box model, it is unclear whether there is a linear relationship between the representations of R and M obtained by the black-box model and the representation of G. To clarify this point, I suggest providing a thorough analysis of the relationship between the representations, as well as discussing any assumptions made in the scoring function.
questions: questions: See above
limitations: limitations: Yes
paper_summary: paper_summary: This paper tackles the problem of extracting private training molecular data from pre-trained models. To address this problem, the authors propose a machine learning method based on a model-independent scoring function and a molecule extraction policy network. The privacy of the training data is an emerging issue in scientific applications, and this paper showed that the proposed method has practical potential in the problem of extracting private training molecular data.
strength_weakness: strength_weakness: 1. Although the authors made a scenario to clarify the problem, it is not realistic because we can easily hide the molecular representations of classification and regression models.
---
2. If the authors claim the importance of the training data privacy in the molecular representation learning tasks, I cannot agree on it because extensive molecular structures for representation learning are already available in public databases, such as PubChem and ChEMBL. Note that the molecular data for supervised learning, rather than representation learning, is expensive because measuring the target physical and chemical properties of the molecules is time-consuming.
---
3. The score function in Eq. (1) is trivial, and the manuscript does not present some theoretical or chemical justification for the score function.
---
4. The proposed method should be evaluated in the regression tasks to demonstrate its practical potential in real-world chemical applications.
questions: questions: Please refer to the Weaknesses section.
limitations: limitations: Please refer to the Weaknesses section.

Comments: title: Response to Reviewer qKJK.
Boarder Impact
Boarder Impact
Any unanswered questions yet?
Any unanswered questions yet?
Any unanswered questions yet?
Additional Rebuttal
comment: We appreciate your perspective in chemical science and understand the concerns you have raised. Firstly, we recognize the importance of developing classification or regression models in reducing the need for time-consuming experiments. Correspondingly, we have conducted experiments on the FreeSolv regression task and achieved promising results, indicating that our approach is also suitable for regression-type models.

Secondly, we believe that due to the confidential nature of label information, and in order to ensure the transferability of the model when dealing with Out-of-Distribution data, selecting to release the intermediate layer representations of the classification/regression model is a reasonable approach in a practical scenario. Specifically, our attack on graph representation is suitable for the scenario.

Moreover, when publicly available datasets’ distribution diverge from private data, and the privacy concerns of data owners who are unwilling to disclose any information about the labels, the choice would still be to employ SSL pre-training to obtain graph representations.

Lastly, we will revise the problem formulation of our manuscript and incorporate the three cases discussed above. In conclusion, we are committed to refining our manuscript to better align with the concerns and expectations of the chemical science research community. We believe that our work can make a significant contribution to the field, and we are grateful for the opportunity to improve our submission based on your valuable feedback.
We recognize that  our investigation into the Molecular Graph Extraction Attack on graph-pretrained models could be misused. Within the context of collaborative model-sharing initiatives in drug discovery, it may lead to privacy breaches. However, we wish to underscore that our primary objective in this paper is to identify vulnerabilities in graph-pretrained models, rather than to create new data extraction attacks. We aim to assist in the exploration of more effective defense strategies. To this end, the paper assesses the susceptibility of mainstream graph-pretrained models to the attack, underscoring the imperative for enhanced defense measures for existing work.

Moreover, for molecular graph pre-trained researchers, we can offer additional perspectives on working methods, such as potential defense strategies:
1.Behavior Detection: Identify malicious queries and implement continuous monitoring systems on shared pre-trained models to safeguard data.
2.Prediction Perturbation: The impact of model extraction attacks is influenced by embeddings. We suggest introducing a small amount of noise into the final output of graph-pretrained models as a defense measure, without compromising their performance.
We believe that the game of attack and defense will make the research community safer and more robust. We anticipate that this will be of interest for a study on defense strategies in future work.
We recognize that  our investigation into the Molecular Graph Extraction Attack on graph-pretrained models could be misused. Within the context of collaborative model-sharing initiatives in drug discovery, it may lead to privacy breaches. However, we wish to underscore that our primary objective in this paper is to identify vulnerabilities in graph-pretrained models, rather than to create new data extraction attacks. We aim to assist in the exploration of more effective defense strategies. To this end, the paper assesses the susceptibility of mainstream graph-pretrained models to the attack, underscoring the imperative for enhanced defense measures for existing work.

Moreover, for molecular graph pre-trained researchers, we can offer additional perspectives on working methods, such as potential defense strategies:
1.Behavior Detection: Identify malicious queries and implement continuous monitoring systems on shared pre-trained models to safeguard data.
2.Prediction Perturbation: The impact of model extraction attacks is influenced by embeddings. We suggest introducing a small amount of noise into the final output of graph-pretrained models as a defense measure, without compromising their performance.
We believe that the game of attack and defense will make the research community safer and more robust. We anticipate that this will be of interest for a study on defense strategies in future work.
Thank you for the careful response.

However, as a researcher in chemical science, I still do not agree with the scenario of this work.
Researchers and engineers in academic and industrial chemistry want to build a classification or regression model for their molecular datasets because they need to skip time-consuming chemical experiments to observe the physical and chemical properties of target molecules.

The graph representation of the molecule is just intermediate information for classification and regression in most real-world chemical applications. Furthermore, as I mentioned, many large molecular databases containing extensive 2D and 3D molecular structures are publicly accessible, such as PubChem, ChEMBL, and QM. For this reason, training data for molecular representation learning is not expensive and private.

The authors need to handle the validity of their problem definition before developing the method.
Thank you for the responses. And I have also read other reviewer's comments. The rebuttal has addressed some of my concerns. I have raised my rating accordingly.
Thanks for the response. I've read through other reviewers' feedback and responses as well. I believe this work holds practical value, so I have provided continued support. Although, as other reviewers have mentioned, the method itself may not be particularly fancy, I think it offers a new perspective and a feasible solution for safely applying molecular pretraining models in collaborative scenarios.

Therefore, I will increase my rating to `7`.

Best regards, 

Reviewer JnnP
Dear Reviewer qKJK,

Thanks again for your detailed review and questions. We have provided detailed answers to them in the rebuttal. As we are approaching the end of the discussion phase, we would like to kindly ask if you still have any unanswered questions about our paper?

Best, Authors
Dear Reviewer o6iJ,

Thanks again for your detailed review and questions. We have provided detailed answers to them in the rebuttal. As we are approaching the end of the discussion phase, we would like to kindly ask if you still have any unanswered questions about our paper?

Best, Authors
Dear Reviewer JnnP,

Thanks again for your detailed review and questions. We have provided detailed answers to them in the rebuttal. As we are approaching the end of the discussion phase, we would like to kindly ask if you still have any unanswered questions about our paper?

Best, Authors
>Q4: The author explores few model backbones, so I suggest the authors also conduct experimental evaluation on Graph Transformer or other architectures. And the pertaining data set is also limited, which would be more convincing if another one dataset was added.

Sorry for not explaining clearly. In our experiments, we have employed various encoder architectures for molecular pre-trained models. The Grover graph pre-trained model included in Table 1 is based on the GTransformer [2], a Graph Transformer. The performance demonstrates the effectiveness of our model across various pre-trained model architectures.

As for the limitation of pre-training dataset, we further introduced a new dataset. We employed the PubChem dataset used in [1]. We sampled 20k molecules from the PubChem dataset to serve as the auxiliary dataset and employed the other molecules as the pre-training dataset. The specific results are as follows and the performance on the PubChem dataset indicates the broad applicability of our method.

|~|One Step||
|-|-|-|
|~|K = 50||
|~|Prec.|FCD|Prec.|FCD|Prec.|FCD|Prec.|FCD|
|MLP|0.43|18.26|0.30|**16.75**|0.35|18.32|0.25|**16.52**|
|Ours|**0.46**|**17.29**|**0.37**|17.42|**0.42**|**16.54**|**0.34**|17.25|

>Q5: A key metric in the experimental evaluation is Precision. How do you determine whether the generated molecules exist in the dataset? Since molecules lack unique identifiers and it is easy to consider two identical molecules as different mistakenly, I hope the authors can provide clearer details on how the presence of data is judged.

Sorry for not explaining it clearly. We utilized the RDKit library to determine molecular identity [3]. Specifically, it assesses whether there exists an atomic mapping such that each atom in the query molecule can be paired with a corresponding atom in the target molecule, with the same connectivity as in the query. Detailed explanation will be included in the revised version.

Reference:

[1] Y. Wang, J. Wang, et al. "MolCLR: Molecular Contrastive Learning of Representations via Graph Neural Networks." Nat. Mach. Intell., 2022.

[2] Rong, Yu, et al. "Self-supervised graph transformer on large-scale molecular data." NeurIPS, 2020.

[3] Landrum, Greg. "RDKit: A software suite for cheminformatics, computational chemistry, and predictive modeling." 2013.