A Sample-driven Selection Framework: Towards Graph Contrastive Networks with Reinforcement Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-30 17:10:59

Abstract: Graph Contrastive Learning (GCL) applied in real-world scenarios aims to alleviate label scarcity by harnessing graph structures to disseminate labels from a limited set of labeled data to a broad spectrum of unlabeled data. Recent advancements in amalgamating neural network capabilities with graph structures have demonstrated promising progress. However, prevalent GCL methodologies often overlook the fundamental issue of semi-supervised learning (SSL), relying on uniform negative sample selection schemes such as random sampling, thus yielding suboptimal performance within contexts. To address this challenge, we present GraphSaSe, a tailored approach designed specifically for graph representation tasks. Our model consists of two pivotal components: a Graph Contrastive Learning Framework (GCLF) and a Selection Distribution Generator (SDG) propelled by reinforcement learning to derive selection probabilities. We introduce an innovative strategy whereby the divergence between positive graph representations is translated into a reward mechanism, dynamically guiding the selection of negative samples during training. This adaptive methodology aims to minimize the divergence between augmented positive pairs, thereby enriching graph representation learning crucial for applications. Comprehensive experimentation across diverse real-world datasets validates the effectiveness of our algorithm, positioning it favorably against contemporary state-of-the-art methodologies.

Main paper: 
 graph contrastive learning.: 
 Existing graph contrastive learning methods [4, 14, 46] train an encoder to measure the divergence in latent space by contrasting samples from a distribution that contains depict statistical dependencies of interest and those that do not [9] . Their basic idea is to make multiple views from the same instance under various graph transformations agree with each other to optimize model parameters. Despite the similarity in principle, these methods are elaborately designed and differ from each other in various aspects, such as network architecture, augmented view design, and contrastive objectives. For instance, InfoGraph [28] maximizes the mutual information between the graph-level representation and the representations of substructures of different scales. GraphCL [39] first designs four types of graph augmentations to incorporate various priors and then systematically study the impact of various combinations of graph augmentations. The theoretical analysis sheds light on the reasons behind their success [46] . Objectives used in these methods can be seen as designing different graph augmentation strategies to enhance the graph representation, while little effort from previous works for negative sample selection strategy. In contrast, our method adopts the RL to optimize the data selection process, which can be trained with GNNs in an end-to-end manner.Reinforcement learning (RL) is one of the effective machine learning paradigms, aiming to learn how to make decisions for maximizing the cumulative future rewards [43] . Previous empirical studies [5, 21] have shown that RL generally does not require numerous labeled datasets but obtains samples for training through ongoing interactive trial and error among the environment, which is closer to the human learning process. Recent trends in RL field is to combine CNNs with RL algorithms for solving high-dimensional complex problems, such as object localization [45] and object detection [11] . Unlike their works, we leverage RL to guide GCL and narrow the divergence between the augmented positive pairs, so as to further improve graph representations. 
preliminary: 
 Graph Neural Networks. Graph neural networks (GNNs) have emerged as a promising approach for analyzing graph-structured data recently [39] . We use G = (V, E) to denote an undirected graph and define X ∈ R 𝑁 ×𝐷 as the feature matrix, where V = {𝒗 𝑖 } 𝑁 𝑖=1 represent a set of 𝑁 nodes, E = 𝒆 𝑖,𝑗 ∈ R 𝑁 ×𝑁 denotes the adjacency matrix, 𝒙 𝑖 = X [𝑖, :] 𝑇 is the 𝐷-dimensional attribute vector of the node 𝒗 𝑖 ∈ V. GNNs use a neighborhood aggregation approach, whose propagation process can be writen as:EQUATIONwhere h 𝑘 𝑖 represents the output of the 𝑘-th network layer with h 0 𝑖 = 𝒙 𝑖 i.e, the initial feature of node. N 𝑖 is a set of vertices adjacent to 𝒗 𝑖 . 𝒆 𝑖,𝑗 represents the edge feature between node 𝑖 and 𝑗, which is an option. 𝛿 denotes differentiable, permutation invariant function, such as sum or mean. 𝚯 and 𝜙 represent differentiable functions or network layers, such as multi-layer perceptron (MLP). After the 𝐾-layer propagation, the output embedding for G is summarized on layer embeddings through the READOUT function:EQUATIONProblem Definition. Prior to going further, we first provide the problem definition used in this paper. For self-supervised graph representation learning, a set of unlabeled graphsG = {G 1 , G 2 , • • • , G 𝑛 }are given, and we aim to learn a 𝑑-dimensional vector 𝒛 G 𝑖 ∈ R 𝑑 for each graph G 𝑖 ∈ G under the guidance of the data itself, whose representation can be used in the downstream tasks. 
methodology: 
 The main goal of this paper is to train a GNN encoder with an RL-based adaptive sampling strategy. As shown in Figure 2 , our method consists of two critical components: A graph contrastive learning framework (GCLF), and selection distribution generator (SDG) in the following.This section discusses how to perform contrastive learning on graphs. We aim to capture the intrinsic patterns and properties of input graph data without using human-provided labels. The process of GCLF is generating augmented examples from original graph data, learning their representations with a graph encoder, and selecting negative samples for the final graph contrastive learning.Graph Augmentations. Previous work [39] suggests that structural feature expansion can effectively improve the performance of graph classification. Data augmentation aims to create novel and realistically rational data by applying certain transformations without affecting the semantics label. To perform GraphSaSe on graphs, given a original graph G ∈ G 𝑞 : 𝑞 ∈ 𝑄 in the dataset of 𝑄 graphs, we perform graph augmentation to formulate augmented graphs Ĝ from the original graphs satisfying: Ĝ ∼ M ( Ĝ|G), where M (•|G) is the augmentation distribution conditioned on the original graph, which is pre-defined, representing the human prior for data distribution. We focus on graph-level augmentations and adopt four basic data augmentation strategies to construct positive pairs of graphs [39] , which are node dropping, edge perturbation, attribute masking and subgraph, respectively. More details can be seen in Appendix A. After that, augmented graphs Ĝ could preserve taskrelevant information, while simultaneously minimizing irrelevant information across views. Graph Encoder. The purpose of training a graph encoder is to extract the most important information that, is data self-correlation, preserve task-relevant properties, and prevent the model from learning results that may lead to brittle representations. Notice that our method allows various choices of network architecture without any constraints. Formally, considering the general GNN framework in Table 1 , we take GIN [37] as an example to instantiate the GNN encoder in the anomaly problem. GIN calculates the representation for each node via a sum-like neighborhood aggregation function. We initialize ℎ (0) 𝑖 = 𝑥 𝑣 𝑖 . After 𝑘 rounds of aggregation, each node 𝑣 𝑖 ∈ V obtains its representation ℎ (𝑘) 𝑖 ∈ R 𝑑 , aggregated from their neighbors N 𝑣 𝑖 . The other three GNN frameworks have similar neighborhood aggregation and can be generalized to more datasets for graph embeddings. Finally, we obtain the output embedding 𝑓 (H) through the READOUT function (See Eq.( 2)). Then a multilayer perceptron (MLP) is adopted, e.g., 𝑧 𝑖 = MLP(𝑓 (H)), for the graph-level downstream task (e.g., classification or regression).Table 1 : Neighborhood aggregation schemes.Aggregation and combination functions for round 𝑘 (1 ≤ 𝑘 ≤ 𝐾)General GNN ℎ (𝑘 ) 𝑖 = COMBINE (𝑘 ) ℎ (𝑘-1) 𝑖 , AGGREGATE (𝑘 ) ( ℎ (𝑘-1) 𝑗 : 𝑣𝑗 ∈ N𝑣 𝑖 )GCN [16] ℎ(𝑘 ) 𝑖 = Θ 𝑣 𝑗 ∈N𝑣 𝑖 ∪ {𝑣𝑖 } 1 ( N𝑣 𝑖 +1) •( N𝑣 𝑗 +1) • W (𝑘-1) • ℎ (𝑘-1) 𝑣 𝑗GIN [37] ℎ(𝑘 ) 𝑖 = Θ (1 + 𝜖) • ℎ (𝑘-1) 𝑖 + 𝑣 𝑗 ∈N𝑣 𝑖 ℎ (𝑘-1) 𝑗 GraphSAGE [8] ℎ (𝑘 ) 𝑖 = Θ W (𝑘-1) • ℎ (𝑘-1) 𝑖 ∥ 𝑣 𝑗 ∈N𝑣 𝑖 ℎ (𝑘-1) 𝑗 GAT [32] ℎ (𝑘 ) 𝑖 = Θ 𝑣 ′ ∈N𝑣 ∪{𝑣} 𝑎 (𝑘-1) • 𝑖,𝑗 W (𝑘-1) • ℎ (𝑘-1) 𝑗Recent GCL works fail to consider the class information, leading to more miniature discriminative graph representations issue [44] .More concretely, without access to labels, the randomly selected negative samples may have the same label as the positive sample in a situation, causing a performance drop. Thus, we attempt to exploit these probabilities to opt for the negative samples to widen the differences between their classes. Specifically, a sequence of 0-1 states is obtained through Bernoulli distribution to sample negative instances. However, how to produce the selection probabilities matters, which will discuss in the following. Obviously, a multi-layer perceptron (MLP) model is used as the SDG, which learns the selection policy optimized by the reward from the divergence between positive graphs representation based on RL. The reward can be trained by SDG and then generate selection probabilities that stay away from positive samples. In what follows, we will introduce a novel negative sampling method by adopting RL. The main idea is to opt for negative samples far away from positive samples during training based on the selection probabilities. For a more rigorous description, given a graph dataset D ∈ {𝑥 1 , 𝑥 2 , • • • , 𝑥 𝑛 }, we divide it into 𝑁 mutually exclusive batch sets marked as D = {𝐵 1 , 𝐵 2 , . . . , 𝐵 𝑁 }, where 𝐵 𝑗 = 𝑥 ( 𝑗-1)𝑛/𝑁 +1 , 𝑥 ( 𝑗-1)𝑛/𝑁 +2 , . . . , 𝑥 𝑗𝑛/𝑁 , 𝑗 ∈ {1, 2, . . . , 𝑁 }. 𝑛 is the total number of a dataset and we marked 𝐵 𝑗 as the number of samples in one batch set. Subsequently, we denote the collection of graph representations with the aforementioned graph encoder for the batch sets asΦ 𝐵 𝑗 = 𝑏 𝑗 1 , 𝑏 𝑗 2 , • • • , 𝑏 𝑗 |𝐵𝑗 |, where 𝑏 𝑗 𝑙 (𝑙 = 1, 2, • • • , 𝐵 𝑗 ) is the vector of the 𝑙-th sample in 𝐵 𝑗 . Then SDG maps the graph representation Φ 𝐵 𝑗 or Φ 𝐴 𝑗 (𝐴 𝑗 is the augmented graphs of 𝐵 𝑗 ) to generate a selection vectorΓ 𝐵 𝑗 = 𝑣 𝑗 1 , 𝑣 𝑗 2 , • • • , 𝑣 𝑗 |𝐵𝑗 | ∈ R 1×|𝐵 𝑗 |, which represents the probability for each instance on the confidence of select. Generally speaking, RL is defined by specifying three ingredients: state, action, and reward, described as follows.State. We then define a state collection marked as 𝑠𝑡𝑎𝑡𝑒 = 𝑠 1 , 𝑠 2 , . . . , 𝑠 𝑗 , . . . , 𝑠 𝑁 , which includes a collection of states for all 𝑗 with respect to 𝑁 batch sets, where each 𝑠 𝑗 indicates a state including selected positive graph instances B𝑗 sampled from 𝐵 𝑗 according to the distribution vector Γ 𝐵 𝑗 , For simplicity we use Φ B𝑗 and Φ Â𝑗 ( Â𝑗 is the augmented graphs of B𝑗 ) to represent state 𝑠 𝑗 . Action. For each state, the action space A is a 0-1 judgment of how to select positive and negative samples. We take instances (1) to choose positive pairs, while instances (0) for negative ones.We get a list of actions𝑎 = {𝑎 𝑙 } |𝐵𝑗 | 𝑙=1 ∈ {0, 1} |𝐵𝑗 | from Γ B𝑗, whose process of assigning the value, i.e., 1 or 0, to 𝑙-th element of action 𝑎 can be formulated by sampling from a Bernoulli distribution. After each action, the framework gives new Φ B𝑗 , then the satate 𝑠 𝑗 is changed to 𝑠 ′ 𝑗 . The policy is defined as P 𝜔 (𝑎 | 𝑠). Reward. Reward mechanism is the core of RL, here we set a reward 𝑟 (𝑠, 𝑎, 𝑠 ′ ) to assess the distance between Φ B𝑗 and Φ Â𝑗 in the current state (𝑠 ′ ) and its previous state (𝑠):EQUATIONwhere 𝑑 (•, •) is a distribution discrepancy measurement (DDM) implemented by different information-bearing functions, discussed in the following part. 𝛾 ∈ (0, 1) is a discounting constant that decreases the impact from future distribution divergences. Note that our reward is related not only to the current status of the 𝑗-th selected batch set B𝑗 , but also to the status of the (𝑗-1)-th selected samples B𝑗-1 . Eq.( 6) is conducted in a sequential manner based on two adjacent batch sets 𝐵 𝑗-1 and 𝐵 𝑗 , of whichΦ 𝑠 ′ B𝑗 is impacted by Φ 𝑠 B𝑗-1via parameters of the GCN Encoder updated by B𝑗-1 .When better instances are selected, the reward is then expected to produce a higher value because the measurement for the previous state𝑑 (Φ 𝑠 B𝑗-1 , Φ 𝑠 Â𝑗-1) is supposed to give a larger distance betweenΦ 𝑠 B𝑗-1 and Φ 𝑠 Â𝑗-1than that for the current state. Our design aims to ensure that the actions obtained from SDG can also have a particular gain on the selection of the next batch set, rather than only affecting the current batch set, which makes the selection stable. MMD: The maximum mean discrepancy [1] ,EQUATIONThe symmetric Rényi divergence [25] , we set 𝛼 = 0.99 following [30] .EQUATIONEQUATIONJS: The Jensen-Shannon divergence [20] ,EQUATION)EQUATIONNote that our method can adopt any related graph difference measurements, not limited to the MMD, RENYI, and JS. Just examples in this part.How to select negative samples adaptively? As shown in Figure 3 , for instance, suppose a graph batch is 6, and a 0-1 selection vector [0, 1, 1, 0, 1, 0] is obtained through SDG based on RL. In this way, a batch of 6 graphs will be divided into 3 graphs closed to positive pairs and 3 graphs far away from positive pairs as negative samples memory bank for the following contrastive learning process. Note that due to the award generated by the divergence between the positive pairs, such that samples with the same label will roughly gather in feature space finally and we can use this information to help samples in the same class closer. Finally, choosing the graph representation far away from the positive pairs as corresponding negative samples memory bank.The following object is optimized to obtain the optimal distribution generation policy:𝐽 (𝜔) = E 𝑝 𝜔 (𝑎 |𝑠)       𝑁 𝑗=1 𝛾 𝑗-1 𝑟 𝑠 𝑗 , 𝑎 𝑗       (9)Then the parameters of the SDG, i.e., 𝜔, is updated via policy gradient [17] by: 𝜔 ← 𝜔 + 𝜏∇ 𝜔 J (𝜔) (10) where 𝜏 is the discounting learning rate and can be self-adapted by optimizer, such as Adam. The gradient ∇ 𝜔 𝐽 (𝜔) is approximated by:EQUATION)with 𝑗 referring to the 𝑗-th step (corresponding to the 𝑗-th batch set) in Reinforcement Learning, and 𝑘 is the 𝑘-th selection process to estimate ∇ 𝜔 𝐽 (𝜔), which is updated after every 𝑇 times of selection over all 𝑁 data batch sets.To maximize the consistency between positive pairs 𝑧 𝑖 , 𝑧 𝑗 compared with negative pairs {𝑧 𝑘 } 𝐾 𝑘=1 , we adopt the noise-contrastive estimation loss [31] :EQUATIONwhere 𝜏 denotes the temperature parameter. 𝐾 denotes the number of negative sample in one batch. To simplify the calculation, we use dot product as the similarity metric function 𝑠𝑖𝑚(•, •). Finally, we combine L 𝑁𝐶𝐸 and 𝐽 (𝜔) to get the total loss function:EQUATION)where 𝜆 is a hyper-parameter to control the magnitude of reinforcement learning task. The entire learning process is described in Appendix B. 
experiments: 
 In this section, we aim to showcase the rapid and robust learning capabilities of our self-supervised model, GraphSaSe, in the realm of graph representation, which holds immense significance for various industrial applications. To achieve this, we conduct extensive comparisons with state-of-the-art methods (SOTAs) across unsupervised and transfer learning scenarios for graph classification tasks, providing insights into GraphSaSe's effectiveness in industrial contexts. Through these comparisons, we seek to highlight the superior performance and efficiency of GraphSaSe in learning highquality representations from graph-structured data, thus bolstering its potential impact on industrial processes and innovation.Datasets. To evaluate our model, we conduct experiments on eight real-world datasets2 in three fields, including three molecules datasets: MUTAG, PTC, NCI-1, three social network datasets: REDDIT-BIN, IMDB-BINARY (IMDB-B), IMDB-MULTI (IMDB-M), and two bioinformatics datasets: PROTEINS, D&D. More details can be found in Appendix C. Baseline Methods. The following models, which are the advanced and closely related works, including HGCL [15] , AD-GCL [29] , SimGRACE [34] , GraphMAE [12] , LaGraph [35] , CuCo [3] , GraphCL [39] , InfoGraph [28] , JOAO [38] , LG2AR [10] , are used as representative baselines to evaluate the performance of the proposed model. More details can be found in Appendix D.Experimental Setting. For graph classification tasks, we adopt the same procedure of previous works [3, 23, 28, 39] to make a fair comparison and used 10-fold cross-validation accuracy to report the classification performance. Experiments are repeated 5 times. For some classical supervised learning algorithms, we report results from previous papers since we have the same experimental setup. If results are not previously reported, we implement them and conduct a hyper-parameter search according to the original paper. For all methods, the parameters of downstream classifiers are independently tuned using cross-validation on training folds of data. The best average classification accuracy is reported for each method. The classification accuracies are computed using LIBSVM [2] , and the 𝐶 parameter was selected from 10 -3 , 10 -2 , • • • , 10 2 , 10 3 . For our proposed model, we adopt a four-layer GNNs with 32dimensional hidden units and a sum pooling readout function for performance comparisons. To measure each original and augmented graph, we select RÉNYI to calculate the DDM for a reward. 2 shows the results of evaluating unsupervised graph-level representations using downstream graph classification tasks on multiple fields datasets. We have the following observations: (1) Overall, our proposed GraphSaSe consistently achieves top-tier performance across all datasets and architectures, affirming the effectiveness of our approach. Notably, on the MUTAG dataset, where existing baselines have already achieved high performance, GraphSaSe continues to push the boundaries further. This underscores the robustness and efficacy of our method in selecting negative samples for graph contrastive learning, even in scenarios where performance levels are already elevated. (2) Additionally, we observe that our method consistently outperforms recent unsupervised baselines. This superiority can be attributed to our deliberate consideration of the advantages stemming from high-quality and well-selected negative samples, which play a pivotal role in the training process. While approaches like CuCo also address this limitation by selecting negative samples, their reliance on comparisons among all graphs in the dataset demands substantial computational resources. As a result, our approach offers a more efficient and resource-friendly alternative while achieving competitive performance.Our reward is based on RL, and its formula is :EQUATIONwhere we assess the distance between the current state (e.g.,𝑑 (Φ 𝑠 ′ B𝑗 , Φ 𝑠 ′ Â𝑗 ))and its previous state (e.g., 𝑑 (Φ 𝑠 B𝑗-1, Φ 𝑠 Â𝑗-1)). We use the 𝛾 to control the magnitude of the current status for getting better reward We set (𝛾 = 1) to conduct the situation that the current status is equally important as the previous status. As for our data selection, we set 𝛾 = 0.9 that the current status is slightly smaller than the previous status. We choose datasets: PTC, IMDB-M, D&D as examples and conduct experiments. As exhibited in Figure 4 , when we set 𝛾 = 0.9, we get the best performance. The main reason is that 𝛾 = 0.9 puts the *reward* in a state of dynamic equilibrium, in which the *reward* goes down or up to select the balanced samples (e.g., the balance positive and negative samples). Although 𝛾 = 1 also has the same effect, 𝛾 = 0.9 has the trend that the reward is expected to produce a higher value which the previous state can affect the current state.As shown in Table 3 , we can see that our model yields excellent performance over the up-to-date baselines and has a 2-4% relative improvement on three datasets (e.g., we choose MUTAG, NCI-1, PROTEINS as examples). Furthermore, we perform an additional experiment to evaluate the training efficiency of all models. The values of the last column in Table 3 represent the training time of models in one epoch on three graph datasets. Unfortunately, the efficiency of our model is not outstanding, but it gets ahead of some baselines. The main reason is that our model employs reinforcement learning to select samples for better training. Lastly, we simply apply RL to generate selection probabilities, which require less computational cost and can achieve greater performance improvements, which indeed is also a bright spot of our method.Our GraphSaSe pairs reinforcement learning with contrastive learning to automatically select the appropriate negative samples in the training data through the probability produced based on RL. To verify where the performance improvement of the selection probability in our model comes from, we conduct experiments on three datasets to investigate the contribution of each component. Specially, we design two variant versions of GraphSaSe: w/o Probability: We do not use any probability, and randomly select negative samples. w/o Negative: We do not use any negative samples, and use the BYOL framework [7] to conduct the experiment. As shown in Table 5 , for dataset NCI-1, compared with the variant w/o Probability, GraphSaSe yields a result of 81.5% in Accuracy, which brings a 2.3% improvement. We infer that our negative sample selection strategy can select higher-quality negative samples to further enhance graph representations. Likewise, GraphSaSe outperforms the w/o Negative by 3.4%, demonstrating that pushing away the embedding vectors from different samples (i.e., negative instances) in representation space can also prevent the training from complete collapsed representations, so as to widen the differences between positive and negative samples for contrastive learning. These trends continued to be the same on the dataset IMDB-B and PROTEINS. We conclude that each component is necessary and contributes to performance improvement. Meanwhile, it also reflects that RL is a prospective strategy for data selection in leveraging pertinent data, which benefits graph contrastive learning. In our study, we delve into the realm of transfer learning applied to two critical domains: molecular property prediction in chemistry and protein function prediction in biology. Following [14] , our approach involves a comprehensive pre-training and fine-tuning strategy to assess the transferability of the pre-training scheme across different real-world datasets. To elaborate, our methodology begins with pre-training a GNN model on a source dataset. Subsequently, we fine-tune this pre-trained model using a limited subset of labeled data from the target dataset. This fine-tuning process allows the model to adapt its learned representations to the specifics of the target task, effectively leveraging the knowledge gained during pre-training. Moreover, we aim to ascertain whether the pre-trained representations capture generalizable features that can be beneficially transferred across diverse datasets and domains.Setting. In our experimental setup, we meticulously adhere to a rigorous train-test and model selection protocol, akin to the methodology outlined in Xu et al. [36] . This entails conducting 10-fold cross-validation and meticulously selecting the epoch with the optimal cross-validation performance, averaged across the 10 folds. Our chosen evaluation metric is the ROC-AUC score, renowned for its effectiveness in assessing model performance across diverse datasets. When fine-tuning our models for downstream tasks, we augment the pre-trained GNN with a linear classifier. This supplementary classifier enables us to adapt the learned representations to the specific nuances of the target task. Importantly, all reported results are meticulously averaged over five independent runs, ensuring robustness and reliability under consistent configurations. In our transfer learning endeavors, we use OGB [13] datasets, a widely recognized real-wolrd dataset in the field. Notably, traditional unsupervised methods lack the prowess to transfer knowledge to datasets from disparate domains. To benchmark our approach effectively, we compare against six baselines, encompassing both non-pretrained (direct supervised learning) methods and six state-of-the-art GNN self-supervised learning techniques. These include EdgePred [8] , InfoGraph [28] , AttrMasking [14] , ContextPred [14] , CuCo [3] and GraphPartition [41] . These baselines serve as essential reference points for assessing the efficacy and generalizability of our proposed approach across diverse molecular property prediction tasks.Comparison. Table 4 presents a comprehensive comparison of our proposed GraphSaSe method with other stateof-the-art approaches in the transfer learning setting. We focus our evaluation on two critical domains: molecular property prediction in chemistry and protein function prediction in biology. These domains represent fundamental areas of research with significant implications for drug discovery, molecular design, and understanding biological mechanisms. (1) Our meticulous analysis reveals that GraphSaSe consistently outperforms all baseline methods. This achievement is particularly notable given the complexity and diversity of the datasets involved in molecular property and protein function prediction tasks. (2) Furthermore, our method exhibits a notable 7% performance enhancement over the non-pretrained baseline, underscoring its efficacy in leveraging self-supervised learning techniques for knowledge transfer across diverse domains. This improvement not only validates the effectiveness of Graph-SaSe but also highlights its potential to drive advancements in both computational chemistry and biology. (3) In summary, our findings underscore the pivotal role of GraphSaSe in advancing transfer learning methodologies across critical domains. By harnessing the power of self-supervised learning, GraphSaSe holds promise for accelerating research and innovation in drug discovery, molecular design, and biological understanding. 
conclusion: 
 In this paper, we systematically investigate the common issues encountered in current graph contrastive learning methods, and propose a novel framework named GraphSaSe. Indeed, the ability to automatically select and learn from the most relevant training instances holds promise for enhancing graph representation tasks in various industrial applications. Moreover, while the research on combining reinforcement learning and graph contrastive learning is still in its nascent stages, our work highlights its potential for further exploration and development. The fusion of these methodologies presents exciting opportunities for advancing industrial processes and innovation.  
: 
 domains, including social networks [27] , protein molecules [6] , and transportation networks [19] . However, the practical application of GNNs in contexts often encounters challenges due to the scarcity of labeled data [3] . For instance, in industrial settings, such as manufacturing processes or supply chain management, obtaining labeled data for graph learning can be arduous and resource-intensive. Additionally, in fields like chemical engineering, where understanding molecular structures is crucial for product development and optimization, acquiring labeled data for graph learning may involve costly and time-consuming experiments. To address these challenges, researchers are increasingly exploring self-supervised or unsupervised learning approaches for graph representation learning in applications. By leveraging techniques that require limited or no labeled data, such as contrastive learning or generative models, these methods aim to enhance graph learning efficiency and scalability in contexts. This shift towards unsupervised learning paradigms offers promising avenues for advancing graph-based solutions, where labeled data may be scarce or costly to obtain.Due to their demonstrated effectiveness, graph contrastive learning (GCL) has emerged as a leading self-supervised approach, aiming to extract discriminative representations without relying on human annotations. These methodologies typically achieve this by learning representations that remain invariant to data augmentations, achieved by maximizing the agreement between embedding vectors derived from various distortions of the same graph [39] . Central to contrastive learning are two essential components: the concept of similar (positive) pairs 𝑥, 𝑥 + and dissimilar (negative) pairs (𝑥, 𝑥 -) of data points. Subsequently, the training objective guides the learned representation to map positive pairs closer together while pushing negative pairs farther apart [26] . However, the effectiveness of these methods heavily relies on the design of these positive and negative pairs, which cannot fully exploit true similarity information due to the lack of supervision.While existing literature has predominantly focused on refining positive pairs through diverse graph augmentation strategies like node dropping and edge perturbation, the significance of adept negative sample selection in graph contrastive learning (GCL) has often been overlooked. Surprisingly, in practical scenarios, superior or comparable test performance can be achieved by meticulously selecting relevant negative samples for GCL. Recent findings underscore the unequal contributions of individual samples to training, with lower-quality samples hindering progress and undermining overall performance. Consequently, the careful curation of highquality negative samples emerges as pivotal for optimizing performance and safeguarding against training stagnation. In industrial contexts, the meticulous selection of negative samples holds significant promise for enhancing various processes. For instance, in manufacturing, precise negative sample selection can aid in optimizing production workflows and identifying potential inefficiencies. Similarly, in supply chain management, leveraging graph contrastive learning with adept negative sample selection can offer insights into network relationships among suppliers, manufacturers, and distributors, thereby enhancing decision-making processes and supply chain efficiency. Moreover, in fields like chemical engineering and materials science, where molecular structure analysis is paramount, effective negative sample selection can facilitate the prediction of material properties and the design of novel materials with tailored characteristics. Thus, refining negative sample selection methodologies holds potential for driving efficiency and innovation across diverse industrial domains. However, devising effective negative samples presents its own set of challenges. Current methodologies frequently resort to random sampling from batches, a practice depicted in Figure 1 . Yet, this approach risks the inadvertent inclusion of invalid-negative samples sharing labels with the anchor, potentially exacerbating the training process. Although some studies, like Cuco [3] , recognize the importance of negative samples, their methods entail comparing entire datasets during training, demanding substantial computational resources. In light of these considerations, a pertinent inquiry arises: is there a more refined approach to selecting appropriate negative samples for graph contrastive learning?We should eliminate the influence of "invalid" negative samples in graph contrastive learning. Careful treatments and additional strategies are usually needed to circumvent invalid-negative samples and pick hard negative ones [26] . Thus, making graph contrastive learning self-adapted to select the negative samples is a challenge and has become increasingly important to learn the representations of entire graphs. Considering that data selection is, in general, a combinatorial optimization problem with complexity, it is impossible to try all possible combinations of training instances [21] . In this way, we hope that the previous data selection of GCL should influence the later data decisions. In this case, reinforcement learning (RL) can be an appropriate vehicle, which has proved to be a prospective approach for data training selection in leveraging pertinent data [5] . Such reinforcement mechanisms make GCL especially appealing in selecting a significant portion of training data for graphs. However, how to effectively take advantage of RL to benefit GCL remains largely under-explored, which is still a significant challenge.In this paper, we present Graph Sample Selection Contrastive Learning (GraphSaSe), a novel approach that integrates reinforcement learning (RL) with graph contrastive learning (GCL) to address the aforementioned challenge effectively. By dynamically selecting appropriate negative samples from training data, Graph-SaSe enhances graph representations, making it particularly suitable for real-world applications. Comprising a graph contrastive learning framework (GCLF) and a selection distribution generator (SDG), GraphSaSe optimizes the contrastive objective function by maximizing similarity between graph augmentations (positive views) while minimizing similarity to other samples within a batch (negative views). Through comprehensive experimentation, Graph-SaSe demonstrates its efficacy in enhancing graph representation learning across a spectrum of applications. Its ability to dynamically adapt to diverse datasets and optimize representation learning makes it a promising tool for addressing challenges in real-world scenarios.The main contributions of this work are summarized as follows: Graph Convolutional Networks (GCNs) represent a groundbreaking extension of the powerful Convolutional Neural Networks (CNNs) into the realm of graph-structured data [18] . The applications of GCNs extend far beyond traditional domains into various industrial contexts. For example, in smart grid management, GCNs can model complex power distribution networks, optimizing energy flow and predicting potential failures. Hence, the integration of GCNs with contrastive learning not only advances the field of graph representation learning but also holds immense promise for enhancing industrial processes and innovation. This process seamlessly integrates connectivity patterns and feature attributes inherent in graph-structured data, leading to superior performance over many state-of-the-art methods on various benchmarks [37] , including L2-GCN [40] and AM-GCN [33] . To address the scalability challenge posed by spectral-based GCNs when dealing with large-scale images, spatial-based GCNs have emerged rapidly [8] . For instance, PATCHY-SAN addresses this issue by sorting nodes and selecting a fixed number of adjacent points for graph convolution, inspired by GCN principles [24] . Similarly, the Graph Attention Network (GAT) introduces an attention mechanism to define graph convolutional operations [32] . In this study, we harness the potential of contrastive learning techniques to tap into the rich information contained within unlabeled graphs. By applying contrastive learning, we aim to extract meaningful representations from unlabeled data, thus enhancing the performance and scalability of graph-based models, which is crucial for various real-world applications. 
 
 Appendix: Eight widely used datasets are publicly available at https://ls11www.cs.tudortmund.de/staff/morris/graphkerneldatasets. 


Decision: This paper received positive scores (ba,ba,ba,wa,wa) from all of reviewers after rebuttal. This paper introduces reinforcement learning for graph contrastive learning. Reviewers believe that the method is innovative and the experiments are extensive. There are also some concerns from reviewers such as weak baselines and lack of discussion in experiments. Authors should solve these issues in their final version.

Reviews: review: paper_summary: paper_summary: The authors developed GraphSaSe, a novel Graph Contrastive Learning (GCL) method that addresses the limitations of traditional semi-supervised learning by using reinforcement learning to adaptively select negative samples. It features two key components: a Graph Contrastive Learning Framework (GCLF) and a Selection Distribution Generator (SDG). The method translates the divergence between positive graph representations into a reward mechanism to guide negative sample selection dynamically, enhancing graph representation learning. Extensive experiments validate its effectiveness, showing significant improvements over existing methods.
strength_weakness: strength_weakness: 1. Adaptive Negative Sample Selection: GraphSaSe uses reinforcement learning to adaptively select negative samples, overcoming the limitations of traditional methods that rely on random sampling, thus improving model performance.
2. Innovative Approach: It introduces reinforcement learning for negative sample selection in graph contrastive learning, providing a novel approach to this challenge.
limitations: limitations: 1. The reinforcement learning component may require extensive parameter tuning to ensure the effectiveness of the selection distribution generator, necessitating additional experiments and time.
2. The authors may consider providing an analysis of the class distribution balance in the existing datasets (optional) and offer insights and discussions on the application of this method to imbalanced data distributions.
paper_summary: paper_summary: This paper mainly focuses on improving negative sampling in the graph contrastive learning framework. Specifically, it adopts reinforcement learning to select the most appropriate negative samples, i.e., those furthest away from the positive pairs. It regards the differences observed between positive graph representations as the reward function. Then a negative sample selection distribution generator is devised, enabling adaptive selection of negative samples to effectively guide the representation learning process.
strength_weakness: strength_weakness: 1. The paper is the first to utilize RL to select the appropriate negative samples in graph contrastive learning.
2. The experiments are very extensive, proving the effectiveness of the method.
limitations: limitations: 1. Assumption about selecting distant negative samples:
The assumption that we should select negative samples that are far away from the positive samples, which requires further theoretical or empirical support. Previous work [1] has suggested that "hard" negative samples (similar to positives but with different labels) can be more beneficial for contrastive learning.
The authors need to clarify the validity of their assumption and position it in the context of related findings. They should provide a stronger justification for why distant negative samples are preferable in their framework, compared to other negative sampling strategies.
2. Clarity of Figure 1:
The current Figure 1 is not informative enough. It is unclear what the authors want to emphasize with this figure.
If the goal is to show that the proposed method can avoid selecting nodes with the same class as negatives, this does not seem well-aligned with the claimed motivation of minimizing the similarity between positive and negative batches.
The authors should either improve the figure to better illustrate the key ideas or consider removing it if it does not significantly contribute to the paper's narrative.
3.Time complexity analysis:
There is a good observation that the proposed method does not seem to have high time complexity, as it can even outperform GraphCL in some cases (as shown in Table 3).
The authors should provide more analysis and description in the experiments section to explain the factors that contribute to the reduced time complexity. A deeper exploration of the time complexity analysis and comparison to related methods would strengthen the technical contributions of the paper.
4. Model optimization:
It's not clear whether the distribution generation policy and graph representation learning are jointly optimized or optimized in an alternating manner.
The authors should clarify the details of the optimization procedure in the methodology section.
5. Paper writing improvements:
the introduction section seems unnecessarily verbose, especially the long list of potential application cases.
The authors should streamline this section and focus on concisely explaining how improved negative sampling techniques can benefit graph representation learning and downstream tasks, without going into excessive detail about specific applications.
Regarding the figures, the authors should ensure that the font size and overall presentation are clear and readable. Adjusting the font size in Figure 1, would be a good improvement.
[1] Generating Counterfactual Hard Negative Samples for Graph Contrastive Learning
paper_summary: paper_summary: The paper presents a novel approach named GraphSaSe, which aims to enhance GCL by integrating reinforcement learning. GraphSaSe consists of two main components: a Graph Contrastive Learning Framework (GCLF) and a Selection Distribution Generator (SDG). The SDG uses reinforcement learning to dynamically select negative samples, optimizing the contrastive learning process. The effectiveness of this approach is validated through comprehensive experiments across diverse real-world datasets, demonstrating its superiority over state-of-the-art methods.
strength_weakness: strength_weakness: 1. The integration of reinforcement learning with graph contrastive learning to address the negative sample selection problem is innovative. 
2. The paper provides extensive experimental results on multiple datasets, showcasing the robustness and generalizability of the proposed method.
3. The contributions of the paper are well-defined, including the novel GraphSaSe framework and its components, as well as the specific improvements in negative sample selection.
limitations: limitations: 1. Could the authors provide a more detailed theoretical justification for the use of reinforcement learning in this context? How does it specifically improve the selection of negative samples?
2. How does the proposed method scale with very large graphs or datasets? Are there any inherent limitations that need to be addressed?
3. Missing important baselines in both the related work part and the experimental part: Bringing Your Own View: Graph Contrastive Learning without Prefabricated Data Augmentations; GPS: Graph Contrastive Learning via Multi-scale Augmented Views from Adversarial Pooling; Graph Self-supervised Learning with Augmentation-aware Contrastive Learning; Self-supervised Graph-level Representation Learning with Adversarial Contrastive Learning.
4. How sensitive is the performance of GraphSaSe to the choice of hyperparameters, especially those related to reinforcement learning?
5. Can the authors provide more insight into how the selection distribution generator operates in various graph contexts? Are there specific types of graphs where it performs better or worse?
6. It is best to provide a code link to enhance the repeatability of the paper.
paper_summary: paper_summary: This paper introduces a new graph contrastive learning approach named GraphSaSe, designed to address the problem of negative sampling. The approach comprises two components: Graph Contrastive Learning Framework (GCLF) and Selection Distribution Generator (SDG). By integrating Reinforcement Learning (RL) with graph contrastive learning, the framework dynamically selects appropriate negative samples during training. Extensive experiments across various real-world datasets demonstrate the effectiveness of GraphSaSe.
strength_weakness: strength_weakness: 1) The paper studies a key problem in the prevalent graph contrastive learning: positive and negative sampling. 

2) The idea of integrating reinforcement learning into graph contrastive learning for dynamic negative sample selection is innovative. 

3) The paper is well-written and easy to follow.
limitations: limitations: 1) The paper primarily focuses on graph representation learning, with limited discussion on the model's adaptability to other data types or tasks, e.g., images in CV. In my opinion, the proposed approach could extend to tasks and models beyond the current scope. Thus, I suggest that the authors discuss adaptability, any understanding is valuable. 

2) The baselines are outdated. Although the proposed approach shows promise, it is crucial to include a comparative analysis with the latest models to validate its effectiveness comprehensively.

3)  The paper lacks a detailed exposition on the selection of hyper-parameters.
paper_summary: paper_summary: This paper proposes a graph contrastive learning framework named **GraphSaSe** that combines reinforcement learning with graph contrastive learning. The main idea is to use an RL model to adaptively select negative samples during training. The approach consists of two main components: a GCL framework and a selection distribution generator driven by reinforcement learning. Experiments on graph classification and transfer learning tasks show that GraphSaSe outperforms several state-of-the-art graph representation learning methods.

**Reinforcement learning model**:
state: a state represents the current set of selected positive and negative graph instances.
action: determine which samples in the batch are positive, and which are negative.
reward: minimize the divergence for positive pairs while ensuring the selected negative samples are sufficiently dissimilar

**Positive Samples**: Positive samples are generated by applying graph augmentations, just like the traditional GCL paradigm.
**Negative Samples**: Negative samples are selected by the above RL model,
strength_weakness: strength_weakness: 1. This novel approach combines reinforcement learning and contrastive learning for adaptive negative sample selection in graph representation learning. This is the first time I see a paper bring RL and GCL together.
2. GraphSaSe uses reinforcement rewards, instead of solely contrastive losses, to minimize the divergence between positive graph samples.
3. The detailed ablation study and hyperparameter analysis validate GraphSaSe's effectiveness.
4. GraphSaSe is an efficient training pipeline, which is only slightly slower than GCN.
limitations: limitations: Below are my main considerations, listed in descending order of importance.
1. The compared baseline methods are weak. I rarely see methods published in 2023 or later. The authors should include more recent state-of-the-art graph pre-training baselines.
2. There's no exploration of alternative reinforcement learning algorithms or reward-shaping strategies. Will the performance gains vary if we change the RL algorithm?
3. Since GraphSaSe still follows the contrastive learning paradigm, the model's performance could influenced by the pre-training and downstream batch size. The RL module may also be sensitive to the number of samples in a batch. The discussion over this hyperparameter is lacking.