Cross-view Contrastive Unification Guides Generative Pretraining for Molecular Property Prediction

Authors: junyu  lin
yan  zheng
xinyue  chen
yazhou  ren
xiaorong  pu
jing  he
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 07:11:06

Abstract: Multi-view based molecular properties prediction learning has received widely attention in recent years in terms of its potential for the downstream tasks  in the  field of drug discovery. However, the consistency of different molecular view representations and the full utilization of complementary information among them in existing multi-view molecular property prediction methods remain to be further explored. Furthermore, most current methods focus on generating global level representations at the graph level with information from different molecular views (e.g., 2D and 3D views) assuming that the information can be corresponded to each other. In fact it is not unusual that for example the conformation change or computational errors may lead to discrepancies between views. To addressing these issues, we propose a new Cross-View contrastive unification guides Generative Molcular pre-trained model, call MolCVG. We first focus on common and private information extraction from 2D graph views and 3D geometric views of molecules, Minimizing the impact of noise in private information on subsequent strategies. To exploit both types of information in a more refined way, we propose a cross-view contrastive unification strategy to learn cross-view global information and guide the reconstruction of masked nodes, thus effectively optimizing global features and local descriptions. Extensive experiments on real-world molecular data sets demonstrate the effectiveness of our approach  for molecular property prediction task.

Main paper: 
 introduction: 
 In the field of computer-aided drug discovery and development, the molecular property prediction (MPP) plays a vital role [3] . When applying deep learning models for MPP, effective learning and integration of different expression forms of molecules is essential for constructing accurate and comprehensive molecular characterizations [4, 11] .Recently, the field of multi-modal and multi-view learning has had a vast impact [25, 29] , and we explore MPP from the perspective of multi-view learning. MPP can be categorized into two types of approaches, single-view and multi-view, according to the diversity of data perspectives. Single-view MPP focuses on a single type of data, such as fingerprints [10, 24] , SMILES [6, 23] , or 2D structure graphs [7, 8, 19, 34] to depict molecular properties. However, single-view based MPP methods are limited by a single data source and cannot capture molecular properties in an all-encompassing way, which can easily lead to incomplete information and insufficient feature coverage. In comparison, multi-view MPP is aimed at effectively integrating information from multiple views through clever design and strategy to capture a broader range of contextual information [13, 15, 36] . Nevertheless, there may be noise, redundancy or irrelevant information among the views, which may interfere with the learning process of common semantics when they are not properly addressed. Therefore, exploring the further enhancement of multi-view MPP has received extensive attention. One of the main problems hindering MPP is the fact that obtaining labeled data are often challenging within the field of molecular biochemistry. Drawing on successful practices in the utilization of unlabeled data in other domains, especially extracting useful information by self-supervised learning techniques, provides insights into solving this dilemma [14, 21, 35, 38] . In this context, the framework of pre-training and fine-tuning has been applied in many studies. Combined with the fact that there are multiple view representations of molecules, optimization of multi-view MPP is performed. This approach aims to extract high-quality representations from a large amount of unlabeled molecular data, and then make them better serve the task of molecular property prediction through a fine-tuning process. In the pre-training phase, a crucial step is to explore an effective proxy task. Since multi-view data naturally contains both positive and negative pairs, contrastive self-supervised tasks have been widely adopted in previous GNN-based multi-view approaches for molecules [34, 37] . For example, Liu et al. [13] creates supervised signals for contrastive SSL on molecular 2D and 3D views.However, such contrastive approaches are not always the optimal choice. As shown in Figure 1 , we summarize the prevailing self-supervised frameworks based on graph neural networks on molecular property prediction from single-view and multi-view perspectives. Figure 1 (a) shows a typical graph mask autoencoder framework utilizing a single view for a rational design, and Graph-MAE is one of the representatives among this type of methods. Figure 1 (b) is a typical graph contrastive learning framework, for example, GraphCL. Figure 1(c ) is an approach based on combining contrastive and generative tasks under multi-view learning, and a specific representative method is GraphMVP. It is worth noting that the contrastive and generative tasks in Figure 1(b-c ) are based on the alignment at the coarse-grained level under the molecular global domain or graph-level representation, which may be insufficient for the learning of some discriminative information such as discriminative atoms or functional groups. Figure 1 (e-f) demonstrates that the reconstruction of node features under single view achieves superior results to the contrastive methods. This confirms to some extent that there is potential for the learning of sufficient local details for performance improvement in molecular property prediction tasks.Further, when exploring the limitations of multi-view molecular property prediction methods, there are problems shown below: (1) Ignoring view specificity and information filtering. Fewer studies have focused on the extraction of common information and viewspecific private information for multiple views of a molecule at the same time, which may result in failing to fully utilize the complementarity between views as well as effectively filtering out irrelevant noise. Different views may contain their own unique biological or chemical signals that are critical for property-specific prediction, and simply merging views may overwhelm these signals, causing models to suffer from irrelevant or misleading information. (2) Overreliance on global contrasts and reconstructions. Current methods generally focus on implementing contrast learning and mutual reconstruction strategies on graph-level representations, emphasizing on global-scale feature alignment and consistent matching of crossmodal information [13, 34, 37] . such methods essentially perform similarity inference at a high level abstraction, assuming that the global information under different views can directly correspond to each other. This is not the case in practice. Local details may be neglected in such methods.In this paper, we propose a cross-view contrastive unification guided generative pre-training method for molecular property prediction, called MolCVG, which utilizes cross-view learning to guide the graph autoencoder such generative pre-training. Specifically, the overview of MolCVG is shown in Figure 2 , and we consider a 2D topological view and a 3D geometric view of the molecule. These two views provide information on the planar chemical structure of the molecule and the 3D spatial distribution of the molecule, respectively. MolCVG encodes these two views through an encoder after randomly masking the nodes. We then propose Common and Private Information Separation (CP-Info Separation) strategy to specifically learn and separate public and private information on molecular views. Weakening the interdependence between common and private information among different views to cope with the problem (1). Moreover, molecular representations are learned by MolCVG through the strategy like graph autoencoder. Unlike previous work, we propose cross-view contrastive unification to guide the reconstruction of the masked nodes to solve the problem (2) . By deeply exploring and fusing the core common features of views, it not only contributes to a more accurate reconstruction of node features, but also effectively guards against the potential misdirection of view-specific information to the cross-view fusion process.Our contributions in this work are as follows:• We propose a new multi-view learning method for molecular property prediction called MolCVG. It deeply unites molecular 2d topological views and 3d geometric views to mine effective features.• To the best of our knowledge, we are the first to propose molecular cross-view contrastive unification to guide node-level generative pretraining tasks. While ensuring that overall consistency is fully emphasized, we also focus on learning local detailed features.• In MolCVG, we introduce the Common and Private Information Separation strategy to better capture molecular universality pattern and differences between molecular views.• Extensive experiments demonstrate the effectiveness of Mol-CVG, achieving superior performance on multiple molecular property prediction benchmark datasets. 
related work 2.1 single-view molecular property prediction: 
 Single-view molecular property prediction is mainly based on molecular representation in one dimension, SMILES strings, molecular fingerprints, 2D molecular structure graphs or 3D conformations, and other single data sources. In chemoinformatics, traditional chemical descriptors such as molecular fingerprints (e.g., ECFP, MACCS) [18, 28] , etc., are utilized to establish quantitative relationships between molecular structures and properties by statistical learning methods (e.g., support vector machines, random forests, logistic regression, etc.). In recent years, deep learning techniques have played an important role in single-view molecular property prediction. High-level abstract features in molecules are learned through deep neural networks. Some sequence-based approaches with Transformer, etc. focus on the contextual information contained in the molecular sequences [23] . Some graph neural network-based methods directly deal with graphical structural representations of molecules. Hu et al. [8] designed a graph pretraining model that utilizes advanced graph neural network (GNN) techniques aimed to extract rich node and graph level representation information. You et al. [34] performed graph augmentation of molecular graphs to compare and learn from each other in a contrastive learning framework. Hou et al. [7] designed a generative self-supervised method incorporating masked feature reconstruction and re-masking strategies. Liu et al. [12] introduced denoising to realize a pre-training framework for atom-to-distance denoising on molecular 3D views. Xia et al. [27] optimized the atomic vocabulary to mitigate the negative migration problem by alleviating the difference in the number of different atoms.Multi-view molecular property prediction integrates multiple representations of molecules, including but not limited to 2D structures, 3D conformations, SMILES sequences, chemical descriptors, and possibly experimental data. It aims to reveal the complex relationship between molecular properties and structures more comprehensively by integrating information from multiple views. Chen et al. [2] combine Algebraic Graph Theory and Transformers to encode 3D molecular information into graph invariants and fuse the complementary molecular descriptors generated by the two to enhance the representation. Guo et al. [5] proposed a pre-training process by combining the SMILES representation of the molecule and IUPAC. St 𝑎rk et al. [22] optimize the matching degree between the learned 3D geometric structures and the 2D graphs. Liu et al. [13] design generative and contrastive interface tasks aimed at injecting information about 3D geometric views into 2D encoders. Zhu et al. [37] adopt four forms of molecular characterization to do comparative learning after fusion. 
methodology 3.1 problem setting: 
 3.1.1 Notation. Given a molecular graph, we can denote it as 𝐺 = (𝐴, 𝑉 , 𝐶, 𝑋, 𝐸). Here 𝐴 ∈ R 𝑁 ×𝑁 is the adjacency matrix of nodes with values taking only 0 and 1, 𝑉 = [𝑣 1 , 𝑣 2 , . . . , 𝑣 𝑁 ] is the node set, 𝐶 = [𝑐 1 , 𝑐 2 , . . . , 𝑐 𝑁 ] is the nodes' 3D coordinate matrix with each 𝑐 𝑖 ∈ R 1×3 , 𝑋 ∈ R 𝑁 ×𝑑 is denoted as a d-dimensional matrix of atom attributes, and 𝐸 ∈ R 𝑁 ×𝑁 ×𝑑 𝐸 records information about the properties of chemical bonds in the molecule where 𝑑 𝐸 represents the dimension of the bond attribute. 𝑁 is the number of nodes.We aim to obtain effective molecular representations by training models through self-supervised learning. Considering the limitations of single-view representation of molecules, we employ a multi-view approach to enrich the molecular representation to contribute to more accurate molecular property prediction, which has been shown potential. Specifically, we consider 2D views and 3D geometric views of molecules, pre-trained on a large unlabeled dataset. Cross-view disparity contrasts in MolCVG drive multi-view fusion to guide the reconstruction of the masked nodes. For the fine-tuning phase, we finetune the pretrained 2D model on the downstream task dataset (2D molecular maps available). This method compensates for the limitations imposed by a single-view, while considering both local and global information.In MolCVG, we intermingle two basic descriptions, 2D topological view and 3D geometric view, in a dual perspective within a self-supervised learning framework. The framework of MolCVG is illustrated in Figure 2 . Inspired by GraphMAE [7] , node feature reconstruction shows key influence in node and graph level classification tasks. Masking node features is achieved by randomly selecting a percentage (e.g., 0.25) of nodes in the graph data and masking their feature information. Operationally, we select a partial subset 𝑉 within the node set 𝑉 , and subsequently implement a hiding process for all the features of each node in the subset, with a predefined [MASK] marking symbol to accomplish this feature masking action. Its masked node attribute vector 𝑥 𝑖 can be obtained in the following formulation:EQUATIONMasked node set 𝑉 is determined by random sampling mechanism.For the node feature update mechanism, it can be clarified as follows: the process of feature reconstruction of any node 𝑣 𝑖 , is determined by the information of other nodes in the neighborhood to which the node belongs, in essence. In 2D space, a molecular structure can be represented as a graph where atoms are considered as nodes and chemical bonds are depicted as edges connecting these nodes. We can consider the topological representation of the 2D molecular graph 𝐺 2𝐷 as a mapping function 𝑓 2𝐷 : 𝐺 2𝐷 → R 𝑁 ×𝑑 𝑧 , where 𝑑 𝑧 is the dimension of the node latent representation. The function 𝑓 2𝐷 receives three parameters, i.e., the adjacency matrix 𝐴, atom attributes vectors set 𝑋 , and bond attributes vectors that may be integrated in the set 𝐸. For a known 2D molecular graph, the representations 𝑍 2𝐷 are obtained by processing it through a 2D graph neural network (e.g. graph isomorphism network [30] ):EQUATIONIn 3D space, molecular characterization involves describing the spatial arrangement of molecules, etc. Similarly, the mapping function 𝑓 3𝐷 : 𝐺 3𝐷 → R 𝑁 ×𝑑 𝑧 can be defined as a process of mapping from a 3D molecular graph 𝐺 3𝐷 to a set of embedding vectors in a continuous vector space. Different from 2D, 𝑓 3𝐷 has the ability to capture information such as the 3D spatial distances between atoms inside a molecule, which is crucial for understanding and predicting the physicochemical properties of molecules. 𝑓 3𝐷 receives the atom attributes matrix 𝑋 and the atom 3D coordinates 𝐶 as parameters. For a given 𝐺 3𝐷 , its latent representation can be obtained by 3D graphical neural networks (e.g., Schnet [20] ):EQUATIONThe pre-training objective of MolCVG is to decode the obtained latent representations of different molecular views to reconstruct the masked node features. However, due to the limitations of a single-view, we advocate utilizing data from multiple views to improve reconstruction performance. To address the situation where the information from different molecular views cannot actually correspond to each other, we attempt to learn the common and private information from multiple views. Specifically, effective discrimination and extraction of information is achieved by weakening the inter-correlation between common and private information.Based on this foundation, we deeply contrast and fuse the common information of views from different sources, and these common information resources are utilized to guide the node feature reconstruction work. Such a strategy is effective because by deeply exploring and fusing the common core features shared by each view, node attributes can be reproduced more accurately while preventing private information from misleading cross-view fusion.Most molecular multi-view methods normally implement alignment operations between views only at the graph level (global) or at a coarse-grained scale. This alignment approach ignores the multi-level and complexity of the view's intrinsic structure and fails to fully reveal and utilize the fine-grained associations between views. Analyzing from the level of information extraction, existing methods tend to focus on mining the common information among views. Instead, they are not sufficient in mining the private information that is unique to a view and different from other views. To address the limitations faced by previous molecular multi-view approaches in extracting common information, we propose Common and Private Information Separation (CP-Info Separation) strategy to separate common and private information of different molecular views, as shown in CP-Info Separation in Figure 2 . Also this facilitates the subsequent cross-view fusion strategy. Specifically, we first obtain the latent representations (i.e., 𝑍 2𝐷 and 𝑍 3𝐷 ) by encoding on 2D and 3D views, respectively. Then, we map the latent representations to the corresponding common and private features by the Common Information Learning (CIL) layer and Private Information Learning (PIL) layer with independent parameter sets:EQUATIONEQUATIONwhere 𝑍 𝑖 𝑐𝑜𝑚 represents common features, 𝑍 𝑖 𝑝𝑟𝑖 represents private features. 𝑊 𝑐𝑜𝑚 , 𝑊 𝑝𝑟𝑖 , 𝑏 𝑐𝑜𝑚 , and 𝑏 𝑝𝑟𝑖 are the learnable parameters. 𝜎 is an activation function.For molecular 2D and 3D views of common and private information, the two parts of the information are intertwined to some extent. For example, the basic chemical structure in the common information and the energetics in the private information. Our goal is to make these two parts of information as unrelated as possible. Inspired by some previous work [1, 9, 16] , the independence assumption is introduced to force the common information to be statistically independent or approximately independent from the private information in the feature space. The inter-correlation loss is set to weaken the statistical correlation between the common and private information by minimizing the Pearson's correlation coefficient of them:EQUATIONwhere 𝐶𝑜𝑣 (•) denotes covariance and 𝜎 denotes standard deviation. Non-linear amplification of the inter-correlation coefficient is performed by the exponential function, which exerts stronger penalties on higher inter-correlation strengths.Previous approaches have built on the premise that there are clear correspondences between views, but intuitive contrast learning or mindless fusion of views without rigorous filtering tends to confuse perceptions of inherent consistency within the views. Therefore, we propose a cross-view contrastive unification fusion strategy that aims to accurately extract and fuse common information from both 2D and 3D views. This establishes the essential foundation for guiding node feature reconstruction.We first concatenate the common information vectors 𝑍 2𝐷 𝑐𝑜𝑚 and 𝑍 3𝐷𝑐𝑜𝑚 of the two views to construct the joint information vector𝐽 𝐶 = 𝑍 2𝐷 𝑐𝑜𝑚 ||𝑍 3𝐷𝑐𝑜𝑚 , where || is the concatenation operation. 𝐽 𝐶 is processed by a multilayer perceptron (MLP) to obtain the crossview fusion representation 𝑍 𝑓 𝑢𝑠𝑒 . It fuses the multi-dimensional information of 𝑍 2𝐷𝑐𝑜𝑚 and 𝑍 3𝐷 𝑐𝑜𝑚 , and realizes the knowledge crossfertilization between the views through nonlinear mapping:EQUATIONwhere 𝑀 𝜂 is a learnable matrix. We introduce contrastive learning [17] to discover and emphasize those elements that are consistent. First, we apply the readout function to 𝑍 𝑓 𝑢𝑠𝑒 , 𝑍 2𝐷 𝑐𝑜𝑚 , 𝑍 3𝐷 𝑐𝑜𝑚 to obtain 𝐻 𝑓 𝑢𝑠𝑒 , 𝐻 2𝐷 , 𝐻 3𝐷 respectively. Taking the 2D view as an example, we mark the positive pairs as {(ℎ𝑐𝑜𝑚,𝑗 )|𝑖 = 𝑗 }, and negative pairs as{(ℎ 𝑓 𝑢𝑠𝑒 𝑖 , ℎ 2𝐷 𝑐𝑜𝑚,𝑗 )|𝑖 ≠ 𝑗 }.For the contrastive loss under the 2D view, the calculation formula can be expressed as follows:EQUATION)EQUATIONwhere 𝜙 (•) denotes the dot product similarity between the two vectors and 𝜏 is the temperature hyperparameter. Similarly, we calculate the contrastive loss for 3D views:EQUATION)The total cross-view contrastive unification loss function is as follows:EQUATIONThe pre-training goal of MolCVG is to reconstruct mask node features. From GraphMAE, we recognize the potential of this generative pre-training and the existence of some improvement possibilities. It is limited by a single data source to fully capture molecular features, which easily leads to incomplete information and insufficient feature coverage. For this reason, we propose Cross-view contrastive unification to guide node feature reconstruction while focusing on global consistency and local details learning. We first concatenate the common and private representations, and then obtain the pre-reconstructed node representations 𝑍 𝛿 𝑝𝑟 through an Linear funtion:EQUATIONwhere 𝑊 𝛿 and 𝑏 𝛿 are the learnable weights and biases, respectively. This is currently still in the single-view case. With multi-view enhancement, the features of cross-view contrastive unification are leveraged to enrich the recognition capability of the model. Specifically, the knowledge or information provided by the crossview fusion feature 𝑍 𝑓 𝑢𝑠𝑒 guides the pre-reconstructed node feature set 𝑍 𝛿 𝑝𝑟 for reconstruction adaptation:EQUATIONwhere 𝑀 𝛿 𝜃 is a single-layer neural network. Finally, we decode the cross-view guided reconstruction features 𝑃 to obtain the reconstructed outputs. Formally, we denote the reconstructed node feature set as 𝑍 𝛿 𝑟𝑒 = 𝑓 𝑑 (𝑍 𝛿 𝑔 ). 𝑓 𝑑 is implemented as a single-layer MLP. The node reconstruction loss is calculated by computing the cosine similarity between the reconstructed node feature set 𝑍 𝛿 𝑟𝑒 and the corresponding original node features. It is worth noting that only mask nodes are computed for node reconstruction loss. The loss can be expressed as follows:EQUATIONwhere Ω = {2𝐷, 3𝐷 }, |Ω| represents the total number of elements in the set Ω, | 𝑉 | denotes the total number of elements contained in the mask node set 𝑉 .In the pre-training phase the overall loss of MolCVG consists of three components: the node reconstruction loss 𝐿 𝑟𝑒𝑐𝑜𝑛 , the intercorrelation loss 𝐿 𝑖𝑐 and the cross-view contrastive unification loss 𝐿 𝑐𝑣𝑐 . The formula is as follows:EQUATIONwhere 𝛼 and 𝛽 are hyperparameters for regulating weights.The fine-tuning phase, the pretrained 2D GNN model is applied for fine-tuning on specific downstream tasks. At this point, it relies only on the 2D molecular map data. MolCVG is fine-tuned on each of the eight benchmark molecular datasets used for the classification task. The probability of the output molecule 𝑖 to be determined as a positive class is denoted as 𝑝 𝑖 after the encoded features are processed by the prediction header. The loss of the prediction and the target label is calculated as follows: 16) where 𝑦 𝑖 is the label of the molecule 𝑖. MolCVG implemented optimizations for 2D characterization of molecules in a generative pre-training task, and subsequent ablation studies demonstrated these enhancements. EQUATION 
experiment 4.1 dataset: 
 The pre-training dataset we used was obtained from the PubChemQC database, which contains about 4 million molecules with 3D geometric configurations. These data were accurately computed by using Density Functional Theory (DFT). These data are optimized by Molecule3D [32] to be a suitable form for deep learning model processing. For pre-training, we selected a subset containing 1 million molecular 3D geometric conformation.For the fine-tuning dataset, we evaluated the performance of MolCVG on 8 public datasets on MoleculeNet [26] , including BACE, BBBP, ClinTox, Tox21, SIDER, ToxCast, MUV, and HIV, which are widely adopted in molecular and drug representation learning and property prediction tasks. For each benchmark dataset, the scaffold splitting scheme is applied to divide them. These molecules are assigned to the training set, validation set, and test set in the ratio of 80%, 10%, and 10%. The statistics for these eight datasets are summarized in Table 1 .Baselines. To demonstrate the effectiveness of MolCVG, we selected eight baselines as benchmark references. Among them are ContextPred [8] , AttrMasking [8] , GraphCL [34] , JOAO [33] , GraphLoG [31] , GraphMAE [7] and Mol-BERT [27] for single-view molecular property prediction, and GraphMVP [13] , 3D Infomax [22] , MEMO [37] for multi-view molecular property prediction.Evaluation metric. During the evaluation process, we follow previous work to adopt the area under the receiver operating characteristic curve (ROC-AUC) as the primary metric. For each experiment, we ensure that three independent executions were performed using different seeds and finally recorded the mean of each experiment and its corresponding standard deviation as the presentation of the experimental results.Implementation details. In the process of constructing molecular 2D view representations, we draw on the successful practices of previous studies and choose the Graph Isomorphism Network [30] with a five-layer structure as the core to construct the backbone model. For encoding the molecular 3D conformations, the Schnet [20] model is adopted as the basic backbone to effectively encode and analyze the molecular 3D views. In the pre-training phase, we set the masking ratio to 0.25. The Adam optimizer is adopted for the pre-training process of 100 training epoch, initializing the learning rate to 8 × 10 -4 and setting the batch size to 256. We set the mask ratio and temperature coefficient by default to 0.25 and 0.1 respectively. In addition, we apply a cosine annealing learning rate scheduling strategy to dynamically adjust the learning rate throughout the pre-training process. In the fine-tuning phase, the Adam optimizer is still followed for model optimization. We determine the learning rate, dropout, batch size, and epoch with grid search. All experiments are done in RTX 3080 GPU environment.Table 2 summarizes the performance of the comparison methods for molecular property prediction on the eight datasets. It is demonstrated that our MolCVG model exhibits excellent performance on all 8 downstream task datasets, with 6 of them achieving the best results and realizing an absolute performance improvement of up to 1.9% in average performance compared to the existing methods.The outstanding results validate the superiority of our proposed method. It shows that MolCVG can better capture the commonalities and differences between molecular 2D and 3D views, and it can fully utilize the multi-view information of molecules. Compared to the baseline of single-view molecular property prediction, MolCVG outperforms almost all baselines. Specifically, for example, when evaluated on the BACE and ClinTox datasets, Mol-CVG performed significantly better by 1.5% and 8.3%, respectively. We found that MolCVG slightly underperforms Mol-Bert on the Tox21 dataset by 0.1%, which may be due to dataset-specificity and randomization factors, but both also have comparable performance. Overall an absolute improvement of 2.3% was obtained on average. Single-view molecular property prediction method emphasizes the deep mining and learning of a single-view and it fails to enjoy the complementary information from multi-views. GraphMAE and GraphCL are the typical generative and contrastive approaches, respectively, which both produce suboptimal results compared to MolCVG. This goes some way to show that it is quite feasible for our cross-view contrastive unification fusion to guide generative tasks. These results demonstrate that MolCVG has superior hidden representation learning capability and can effectively handle the coordination of multi-view information under the molecular property prediction task.Compared with the multi-view molecular property prediction baseline, MolCVG also outperforms almost all multi-view MPP baselines. Specifically, an overall average improvement of 1.9% is obtained. We notice that MolCVG underperforms MEMO by 0.5% on the Toxcast dataset. We speculate that this is a result of MEMO utilizing more complementary information from the four molecular views for processing, improving its performance on the Toxcast dataset. In addition, GraphMVP and 3D Infomax both utilize 2D and 3D views for self-supervised methods. The experimental results show that MolCVG outperforms them both. It resonates with our motivation that there is a performance bottleneck in global contrast and reconstruction based on the assumption that multiview information can be matched one-to-one, which may overlook important local details. inter-correlation loss 𝐿 𝑖𝑐 and the cross-view contrastive unification loss 𝐿 𝑐𝑣𝑐 . We report the performance of each loss component on the three datasets BACE, BBBP, and ClinTox in Table 3 . The experimental results show that our method with a complete loss function achieves optimal performance. Specifically, it improves 1.9%, 0.9%, and 8.3% on the BACE, BBBP, and ClinTox datasets, respectively, compared to only with node reconstruction loss. This indicates that the loss of our method is significantly effective. Further, we observe a performance decrease on the BBBP dataset when our method lacks the cross-view contrastive unification fusion loss, although the overall performance is improved. This corresponds to our previous observational analysis, where direct fusion without differentiating the data across views may misleading information cause interference. In addition, the complete MolCVG obtains a larger increase in performance compared to the case where intercorrelation loss is lacking. This also suggests that neglecting view specificity and information filtering limits model to learn effective representations.The effectiveness of the strategies of MolCVG. Table 4 demonstrates the effectiveness study of two strategies for MolCVG. The experimental results show that our proposed method achieves superior performance. Note that there are differences in the setup of this strategy ablation and loss ablation, and we aim to explore the effects of these two strategies. Specifically, the C-V Contrastive Guidance setting item in Table 4 refers to whether the fused features obtained through cross-view contrastive unification fusion are used to guide node feature reconstruction, but the cross-view contrastive unification loss function remains. CP-Info Separation refers to whether to extract and separate the common and private information of each view or not, unlike loss ablation this setting eliminates CIL and PIL, and at the same time inter-correlation loss also its cannot be constructed. This is because it is not possible to preserve the inter-correlation loss function without adopting an information separation strategy. We can observe that both strategies contribute to the performance of the model. It is worth noting that only lacking the CP-Info Separation strategy or the C-V Contrastive Guidance, although the performance is improved compared to the version that lacks both, there is still a large gap compared to the full version of MolCVG. In particular, on the ClinTox dataset, there is still a 4% significant gap. This phenomenon demonstrates that MolCVG combines the two strategies well and achieves significant results. It also shows that the successful integration of these two strategies in the MolCVG framework effectively addresses and solves the two challenges mentioned in above.Ratio. The plot on the right of Figure 3 , shows the effect of the mask ratio. In most cases, when a low mask ratio is adopted for the feature reconstruction task, it is not challenging enough to motivate the model to learn beneficial features. If the mask ratio is set too high, it may result in certain key features not being effectively recovered, which may cause performance degradation. As the mask ratio increases from 0.15 to 0.25, the model performance shows significant improvement, demonstrating that moderately increasing the difficulty of feature reconstruction contributes to the model's ability to learn richer and more valuable intrinsic patterns. However, as the mask ratio continues to increase to 0.75, there is a slight decrease in the overall performance at first both, after that the model shows a relatively stable trend. Based on experience we set the mask ratio to 0.25.We performed an ablation study of the hyperparameters in MolCVG, and the detailed results are shown in Figures 3 and 4 . In Eq. ( 15), the trade-off coefficient hyperparameters 𝛼 and 𝛽 are included. We set 𝛼, 𝛽 ∈ [0.2, 1.0] with an interval of 0.2, respectively, to explore the effects of these two parameters in MolCVG. As shown in Figure 4 , our experimental results demonstrate that our method is robust to the choice of 𝛼 and 𝛽. Moreover, we perform ablation experiments on the temperature coefficient 𝜏 in Eq. ( 9) and Eq. (10) . We set 𝜏 ∈ [0.1, 0.9] with an interval of 0.2 to explore the effect of temperature hyperparameter in MolCVG. As shown in the left plot of Figure . 3, there is good robustness of MolCVG to 𝜏 over the experimental range we set. 
conclusion: 
 In this paper, we present MolCVG, a cross-view contrastive unification guides generative pre-training method , which considers molecular 2d topological views and 3d geometrical views jointly for molecular property prediction. In MolCVG, after encoding the 2D and 3D views of molecule, the common and private information are extracted and separated from the different views of molecule.Focusing on global consistency while taking into account the preservation and learning of local details, MolCVG designs a cross-view contrastive unification to guide node feature reconstruction. Extensive experiments on eight public molecular property prediction benchmarks demonstrate the superiority of our method. 
 
 Appendix:  


Decision: This paper received ba, ba, br from all of reviewers after rebuttal. I have read the paper and believe the paper is of high quality. I am happy to recommend to accept this paper. Please carefully revise the final manuscript according to the comments and discussions.

Reviews: review: paper_summary: paper_summary: This paper presents MolCVG, a novel method for molecular property prediction that leverages cross-view contrastive learning and generative pretraining. The approach utilizes two different views of molecular data: the 2D topological view and the 3D geometric view. MolCVG aims to address limitations in current multi-view molecular property prediction methods, specifically the issues of view specificity and the loss of local detail information. Authors introduced a cross-view contrastive unification to guide node-level generative pretraining, ensuring both global consistency and local detail preservation. Extensive experiments demonstrate that MolCVG achieves state-of-the-art performance on multiple molecular property prediction benchmark datasets, surpassing existing methods in most cases.
strength_weakness: strength_weakness: 1. The paper proposes a novel method that integrates cross-view contrastive learning with generative pretraining.
2. The CP-Info Separation strategy is a strong point as it effectively distinguishes between common and unique information from different views, enhancing the model’s ability to utilize complementary information from both 2D and 3D representations.
limitations: limitations: 1. The proposed model’s complexity and computational requirements are not discussed in depth. Given that it integrates multiple views and involves sophisticated strategies like CP-Info Separation, it is likely to be computationally intensive.
2. While the paper demonstrates effectiveness on benchmark datasets, there is limited discussion on the scalability of the approach to larger datasets or in real-world applications. The dataset scales shows in Table 1 are small.
3. The performance slightly varies across different datasets.  (e.g., underperforming on the Tox21 dataset compared to Mol-Bert), suggesting that the method may have dataset-specific dependencies​.
4. The contrastive learning strategy is common in biological area, such GearNet[1]. 
5. As for the hyperparamerts, 𝛼 and 𝛽, and the the temperature coefficient $\tau$, why the model is robust to the choices of these parameters? 

[1] Zhang, Z., Xu, M., Lozano, A. C., Chenthamarakshan, V., Das, P., & Tang, J. (2024). Pre-training protein encoder via siamese sequence-structure diffusion trajectory prediction. Advances in Neural Information Processing Systems, 36.
paper_summary: paper_summary: ## Summary
The paper presents a novel method, MolCVG, which combines multi-view learning approaches to improve molecular property prediction. The core innovation lies in integrating 2D topological and 3D geometric views of molecules through a process involving encoding, information separation, and cross-view contrastive guidance. The methodology encompasses several components:

1. **Encoding of molecular views**: Utilizing Graph Neural Networks (GNNs) to encode both 2D and 3D views into feature representations.
2. **Common and Private Information Separation**: Distinct layers are used to extract common (universal across views) and private (unique to each view) information, enhancing the adaptability of the model to various molecular structures.
3. **Cross-View Contrastive Unification**: Employing cross-view learning to enhance feature reconstruction and ensure that information learned from one view can enhance the learning from another.

### Questions:


1. **Statistical Independence in Feature Separation**:
How do you quantitatively measure and ensure the statistical independence between common and private features? What specific metrics or tests are used to evaluate the effectiveness of this separation in practical scenarios?

2. **Model Complexity and Efficiency**:
Given the apparent complexity of the MolCVG model with its multi-component structure involving encoders, decoders, and separate pathways for common and private information, what are the computational requirements? How does the model's performance scale with increasing dataset sizes or molecular complexity?

3. **Generalisation and Overfitting**:
What measures are in place within MolCVG to prevent overfitting, especially given the deep and complex architecture? Are there specific regularisation techniques or model validation strategies that have proven effective?

4. **Fine-Tuning and Adaptability**:
During the fine-tuning phase, how adaptable is the model to different types of downstream tasks? Are there specific examples where MolCVG has shown significant improvements over existing models?
strength_weakness: strength_weakness: 1. **Innovative Integration of Multi-View Data**: The methodology effectively utilizes the complementary nature of 2D and 3D molecular data, which is a significant advancement in molecular modeling. This integration helps in capturing both the structural and relational aspects of molecules more comprehensively than using either view in isolation.

2. **Robust Information Separation Strategy**: The approach of separating common and private information is methodologically sound and innovative. This separation allows the model to better generalize across different molecular datasets by leveraging universal features while preserving view-specific details that might be crucial for certain properties.

3. **Enhanced Learning Through Cross-View Contrastive Guidance**: This component is particularly notable for its potential to refine feature learning by using insights gained from one view to inform the feature reconstruction in another. This could potentially lead to more accurate predictions by ensuring that the model fully exploits all available information.
limitations: limitations: 1. **Conceptual Overlap with Existing Techniques**: 
While MolCVG’s approach is framed as innovative, parts of its methodology closely resemble existing strategies in machine learning that involve feature integration and contrastive learning, potentially diluting its perceived novelty.

2. **Complexity and Computational Demands**: The complex structure, while powerful, could pose challenges in terms of computational efficiency and scalability, especially when deployed in resource-limited settings.

3. **Empirical Validation Needs**: While the methodological innovations are clear, the actual performance benefits of these complex strategies would benefit from more extensive empirical validation across diverse and large-scale datasets.

4. **Handling of Conflicting Information**: The methodology does not explicitly address how contradictions between 2D and 3D information are resolved during the feature reconstruction phase.
paper_summary: paper_summary: This work proposed MolCVG, a method for molecular property prediction that leverages cross-view contrastive unification and generative pretraining. The method integrates 2D topological and 3D geometric views of molecules to enhance the prediction accuracy of molecular properties. The authors demonstrate that MolCVG outperforms existing methods on multiple benchmark datasets, showing the effectiveness of their approach.
strength_weakness: strength_weakness: 1.  The paper is of high quality and is clearly written.

2.  The key observations and basic ideas behind this method are particularly noteworthy. By separating common and private information from different molecular views, MolCVG effectively addresses the issue of noise and inconsistency between views. The cross-view contrastive unification strategy ensures that the decoder benefits from the rich, discriminative features learned during the contrastive phase.

3.  The experimental results are comprehensive, and confirm the merits of the proposed method.
limitations: limitations: 1.  The improvements are mainly observed on the ClinTox and BACE datasets. On other datasets, the performance is only marginally better or sometimes slightly worse compared to existing methods.

2.  The pre-training dataset used in MolCVG contains 1 million molecular 3D geometric conformations, which is significantly larger than the datasets used in previous works such as GraphMVP and MEMO. While the large dataset is a strength, it also raises questions about the generalizability and scalability of the approach. The manuscript does not sufficiently discuss how MolCVG performs with smaller datasets or the computational resources required for handling such a large dataset. A comparison of performance on smaller pre-training datasets or a discussion on the impact of dataset size on model performance would provide a more comprehensive evaluation.