FUG: Feature-Universal Graph Contrastive Pre-training for Graphs with Diverse Node Features

Authors: jitao  zhao
di  jin
  meng ge
lianze  shan
xin  wang
dongxiao  he
zhiyong  feng
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-05 13:53:30

Abstract: Graph Neural Networks (GNNs), known for their effective graph encoding, are extensively used across various fields. Graph self-supervised pre-training, which trains GNN encoders without manual labels to generate high-quality graph representations, has garnered widespread attention. However, due to the inherent complex characteristics in graphs, GNNs encoders pre-trained on one dataset struggle to directly adapt to others that have different node feature shapes. This typically necessitates either model rebuilding or data alignment. The former results in non-transferability as each dataset need to rebuild a new model, while the latter brings serious knowledge loss since it forces features into a uniform shape by preprocessing such as Principal Component Analysis (PCA). To address this challenge, we propose a new Feature-Universal Graph contrastive pre-training strategy (FUG) that naturally avoids the need for model rebuilding and data reshaping. Specifically, inspired by discussions in existing work on the relationship between contrastive Learning and PCA, we conducted a theoretical analysis and discovered that PCA's optimization objective is a special case of that in contrastive Learning. We designed an encoder with contrastive constraints to emulate PCA's generation of basis transformation matrix, which is utilized to losslessly adapt features in different datasets. Furthermore, we introduced a global uniformity constraint to replace negative sampling, reducing the time complexity from $O(n^2)$ to $O(n)$, and by explicitly defining positive samples, FUG avoids the substantial memory requirements of data augmentation. In cross domain experiments, FUG has a performance close to the re-trained new models. The source code is available at: https://github.com/hedongxiao-tju/FUG.

Main paper: 
 introduction: 
 Graph Neural Networks (GNNs) [1] , renowned for their effectiveness in encoding both graph topology and feature information, have emerged as the mainstream methods in graph representation learning. They are extensively applied in various scenarios such as protein molecule prediction [2] , e-commerce recommendation systems [3] , and community detection [4] . Training high-quality GNNs typically relies on extensive manually labeled data [5] , which is costly and limits their applications. To address this challenge, graph self-supervised pre-trainings have gained significant attentions [6, 7, 5] . These approaches leverage self-supervised signals mined from the graph data themselves and design proxy tasks to train models without manually labeled data.Unfortunately, due to the inherent complexity of graph data, models trained on one graph dataset often fail to perform well on another. This complexity arises from two main aspects: 1). topological diversity, where a node in a graph can connect to any number of other nodes, and 2). feature diversity, where nodes represent a variety of information such as textual or float data. In domains like vision [8] , text [9] , and speech [10] , data shares common latent semantics (such as RGB channels in pixels, word vectors, or audio samples), and have relative positional information like spatial relations in vision or sequence relations in text and audio. Unlike these domains, features in graphs are unordered and chaotic. Even within the same citation dataset category using identical feature extraction methods, such as Cora, CiteSeer, and PubMed [11, 12] , the node features have different shapes. Existing GNNs can address the issue of topological diversity through welldesigned message-passing and aggregation methods [13, 14] . However, due to the neural networks' requirement for fixed input-output dimensions, GNNs struggle with feature diversity. This necessitates retraining a new graph neural network for datasets with different node feature shapes, significantly restricting the applicability of existing graph self-supervised pre-training models. This leads to some graph pre-training works [6, 15] that have to ignore node attributes and encode only the structural information. Recent "All-In-One" work in graph attempts to reshape node features to the same dimensions through preprocessing methods like Principal Component Analysis (PCA) [16, 17] , feature similarity embedding [18] and language models [17] . However, these methods inevitably lose crucial semantic information related to downstream tasks during data preprocessing. Classical machine learning methods such as PCA cannot participate in training, and they primarily encode the information with the highest variance as principal components, overly focusing on differences between node features and losing substantial homophily information in node features. Encoding informative features into single-dimensional similarity also brings obvious information loss. Language models suffer assumption biases, presuming that all input data is textual, thus ineffectively handling extensive float features. Furthermore, using large language model interfaces or involving language models in training incurs additional significant costs. Additionally, sliding window-based neural networks, such as 1D-CNNs [19] and LSTMs [20] , also fail to address the issue of feature diversity. While they can handle inputs of varying dimensions, they operate under the prior assumption that the data is sequential, which is not satisfied by graph node features.PreprocessingAmong the feature-unifying approaches discussed earlier, only PCA-like methods exhibit no assumption bias and are adaptable to various feature forms. Their primary limitation lies in the misalignment between their model objectives and the proxy tasks of graph pre-training, coupled with their inability to train based on specific loss functions. Noting discussions in existing works that link contrastive learning with PCA-like methods, which regard contrastive pre-training as a generalized form of nonlinear PCA [21] , we draw inspiration to propose a question: Could a graph contrastive pre-training principle based on PCA theory be developed to address inherent issues of PCA and broadly applied to node features of different shapes?In this paper, we theoretically demonstrate that classical PCA and contrastive learning fundamentally follow the same theoretical framework. PCA's optimization objective is a special case of that in contrastive learning. The adaptability of PCA-like methods to various feature shapes stems from their intrinsic component that acts as a columnar encoder for values under specific feature dimension, creating a basis transformation matrix, which we term dimensional encoding. The loss and parameter update mechanisms of contrastive learning could offer traditional PCA methods new goals and training mechanisms that align better with the graph data and downstream tasks. Based on these analyses, we propose a new strategy for Feature-Universal Graph pre-training (FUG strategy), which includes three steps: dimensional encoding, graph encoding, and relative relationship optimization task.Guided by this theory, we instantiate a simple FUG model. Simulating PCA, this model utilizes sampling to capture the distribution among different nodes within specific feature dimension for dimensional encoding, and generates basis transformation vectors for feature dimensions which are used to losslessly unify the node features. It employs GNN encoder for graph encoding. We also design an improved and more efficient graph contrastive loss as the relative relationship optimization task. According to this task, FUG effectively learns the relative relations within graph data, and transfers this encoding capability effectively to other datasets with different node feature shapes.We conducted extensive experiments to validate FUG's efficacy. In in-domain self-supervised learning experiments, FUG demonstrated competitive performance with existing advanced graph self-supervised models. In the cross-domain learning experiments, the FUG trained and tested on different datasets shows similar performance to the model trained and tested on the same dataset. These results prove the strong adaptability and transferability of FUG. 
notations and preliminary: 
 In this section, we introduce some preliminary concepts and notations to facilitate analysis in later. We consider a graph data G = (V, E), where V = {v 1 , v 2 , ..., v n } represents the set of nodes and E ⊆ V × V represents the set of edges. X ∈ R n×d denotes the feature matrix of the nodes. A is the adjacency matrix representing the topological relationships between nodes, with a ij = 1 if e ij ∈ E and a ij = 0 otherwise.Graph Neural Networks (GNNs) . GNNs can be uniformly described as H (t+1) = AW H (t) , where H represents the embeddings in hidden layer, with t = 0 denoting the node features X. A is the message passing matrix, which can be derived through neighbor sampling [13] , random walks, or deep learning [22] techniques to mine graph data. W is a learnable parameter matrix. Under the successful design of existing works, the message passing matrix A can generically encode graph data with varying edges. However, the presence of the parameter matrix W means that a trained GNN model can only be directly applied to graphs with the same node feature dimensionality as the training dataset, significantly limiting the application scenarios of GNNs. L InfoNCE = -log(vi,vj )∈P S e Sim(g(vi),g(vj ))/τ (vi,vj )∈P S e Sim(g(vi),g(vj ))/τ + (vi,vj )∈N S e Sim(g(vi),g(vj ))/τ , (1) where P S denotes the data sampled as positive pairs, and N S denotes the data sampled as negative pairs. Sim(•) represents the cosine similarity calculation, g(•) is a non-linear projector, τ is a temperature parameter. The training objective is to increase the similarity between positive pairs and decrease the similarity between negative pairs. Principal Component Analysis (PCA). Given data X with any number of dimensions d, PCA aims to reduce the dimensionality to k dimensions, where k ≤ d, while maximizing the retention of variance between the data. PCA based on eigendecomposition is shown as:PCA(X) = XP, P = Norm[Concat{c i |c i ∈ C, λ c ≥ λ (k) }], (Λ, C) = δ[Cov( X :,1 , X :,2 , ..., X :,d )],(2) where P represents the basis transformation matrix, X = X -Mean(X). c i denotes the i-th eigenvector, λ i denotes the corresponding eigenvalue, λ (k) denotes the k-th largest eigenvalue. Concat(•) represents concatenating the eigenvectors within the set, and Norm(•) represents normalization. C and Λ represent the matrix composed of eigenvectors and the corresponding eigenvalues, respectively. Cov(•) represents covariance computation, and δ denotes matrix decomposition. Note that since the covariance matrix is symmetric, the extracted eigenvectors are orthogonal to each other. This formulation mathematically ensures that the dimensionality-reduced embeddings maximize the retention of variance between the original data and minimize homogeneity among the data. 
graph contrastive pretrained models from the pca perspective: 
 In this section, we first theoretically summarize the framework that PCA essentially follows. We then analyze contrastive learning, demonstrating that it also adheres to this framework, and the PCA optimization task is a special case of that of CL. Furthermore, we answer the question why the PCA embeddings generally underperform graph contrastive pre-training in downstream tasks within the same framework, and why graph contrastive pre-training models lose the ability to adapt datasets with varying node feature shapes. Based on these findings, we propose a new strategy for Feature-Universal Graph pre-training (FUG strategy). First we define a class of objectives: Definition 3.1 (Data relative relationship optimization task L RT ) . Given a dataset X and a model ϕ(•), L RT involves maximizing or minimizing the l2-distance between certain pairs of data points, formulated as:EQUATIONwhere X + represents the set of data pairs that need to be encoded as similar, whereas X -denotes the set of pairs that should be encoded as dissimilar. And z i denotes the embedding of x i .Notably, unlike the contrastive loss, X + represents positive pairs within the raw data, rather than positive pairs between the augmented views. It means L RT focuses solely on the relative semantic relationships among the data points, disregarding their absolute semantic.Definition 3.2 (Encoder Dim(•)) Dim(•) refers to a series of dimension-specific encoders that directly take the data of all samples under a specific dimension i, X T :,i , and output the embedding of that dimension, DimEmb i , formalized as DimEmb i = Encode(X T :,i ), DimEmb ∈ R D×N ′ .Definition 3.3 (Encoder Fea(•)) Fea(•) refers to a series of sample-specific encoders that take the data of all dimensions of a specific sample j, X j , and output the embedding of that sample, F eaEmb j , formalized as F eaEmb j = Encode(X j , ϵ), F eaEmb ∈ R N ×D ′ , where ϵ represents additional input.Theorem 3.1 (PCA framework) . Consider Dim(•) as an encoder modeling the distribution of data across a specific dimension (as in Definition 3.2), Fea(•) as an encoder modeling the relationships between different dimensions of data (as in Definition 3.3). PCA is an instantiation model for these two encoders. Assuming that data representations have equal length m, when X + = ∅ in Definition 3.1 of L RT , PCA adheres to the following form:EQUATIONRigorous proof can be found in Appendix A.1. Here,L P CA = -E[||PCA(x i ) -mean(PCA(X))|| 2 2] is interpreted as an instantiation of L RT , based on the principle of maximizing the retention of data variance. We consider the covariance function in Eq. 2, Cov(•) , as an implementation of Dim(•), since it essentially encodes the distribution information of a single dimension through the computation of covariance. The matrix decomposition and normalization concatenation in Eq. 2 are viewed as an implementation of Fea(•), as they fundamentally encode the relationships among different data dimensions. This interpretation also applies to PCA methods based on Singular Value Decomposition. Thus, we summarize the framework that PCA adheres to as shown in Eq. 4.PCA framework has powerful transfer ability. From Eq. 4, it can be inferred that PCA essentially functions as a neural network model trained to reach a global optimum. The parameters of PCA's two encoders learn to encode the covariance of dimensional data and to extract and concatenate eigenvectors to form the basis transformation matrix, respectively. Extensive researchers have validated PCA's success across various fields with the same parameters [23, 24, 25] , demonstrating that under the guidance of the L RT loss, the model's capacity to encode difference in data demonstrates formidable transferability. This transform capability originates from the model's focus on encoding the differential relationships among data rather than their absolute semantics.PCA is capable of adapting to arbitrary Euclidean data and demonstrates strong transfer performance. However, its effectiveness generally falls short of advanced NN-based models, and it lacks the ability to encode structure information. To address the issue of node feature generality in graph pre-training models, inspired by existing work [26] , we attempt to further theoretically analyze the relationship between existing graph contrast pre-training models and the PCA framework. We start by analyzing the InfoNCE loss which is commonly used in contrastive learning. Theorem 3.2 (Equivalence of Contrastive Loss and L RT ) . Assuming that data augmentation in contrastive learning does not alter the essential semantics of node pairs for downstream tasks, it follows:MinimizeEQUATIONDetailed proof can be found in Appendix A.2. Here we offer an intuitive explanation: the negative sampling component in InfoNCE loss is widely acknowledged for encoding dissimilarities within the data. We contend that the augmentation-based positive sampling part, inherently encodes similar relationship between data points rather than their absolute semantics. Consider a pair of data points, x i and x j , with the same semantics. Ideally, when data augmentation does not alter the essential semantics of the nodes, the augmented data, Aug(x i ) and Aug(x j ), should be identical. Consequently, Maximize [Sim(x i , Aug(x i )) + Sim(x j , Aug(x j ))], essentially equates to Maximize [Sim(x i , x j ). Thus, data augmentation is essentially looking for positive samples in the dataset, aligning with the views expressed in work [27] . Furthermore, this view explains why augmentation is less critical in graph contrastive learning compared to other domains [28] , leading to the proposal of many augmentation-free graph contrastive studies [29, 30, 31] , as the edges in graph data (especially in homophilic graphs) inherently convey similarities between data points.Corollary 3.2.1 Consider a specific instantiation of a dimension encoder defined as Dim(X) = X, and the contrastive model viewed as an instantiation of an Fea(•) = ContrastiveModel(•). Under this the contrastive model also adheres to the framework in Theorem 3.1, which states:EQUATIONFrom this, we can conclude that the PCA optimization objective is, in fact, a special case of the CL loss, detailed proofs are in Appendix A.2. As demonstrated by Theorem 3.1 and Corollary 3.2.1, PCA and contrastive pre-training fundamentally follow the same framework. Our analysis extends the existing discussions [26] on the relationship between the two. Within this framework, we can more clearly analyze why PCA is versatile across various data, while advanced contrastive pre-training models generally outperform PCA when directly used as embeddings in downstream tasks. learns the data distribution within a specific dimension to generate a basis transformation matrix T , which converts any form of node features to a unified shape. The GE(•) encodes both the structure and the node features to obtain graph representations. L RT-FUG is an improved contrastive loss based on the L RT concept. Compared to the classical InfoNCE loss, L RT-FUG more effectively guides the model in encoding the relative semantic information in the data. Furthermore, in line with the earlier discussion on augmentation essentially serving as positive sampling, we explicitly define positive sampling within L RT-FUG to replace data augmentation.We designed a non-linear dimension encoder, DE(•), to encode the distribution information of a specific dimension. If it encodes the features of all nodes under one dimension, with the input dimension fixed at the number of nodes in the training set n t , this encoder would be constrained by the number of input nodes. Therefore, we randomly sample n s nodes from the training set as the input for this encoder, approximating the feature distribution of the entire data through the sampled nodes. The dimension distribution information, after being processed by DE(•), generates a basis transformation vector for each dimension as follows:EQUATIONwhere X = Sample(X) and MLP represents a Multi-Layer Perceptron. X :,i represents the i-th column of the X matrix, that is, all the data under the i-th feature dimension. Eq. 7 can be seen as DE(•) assigning a point to each feature dimension in a hyperspherical embedding space which the center at the origin and radius one. The basis transformation vectors should maximally reflect the differences among the feature dimensions while retaining inter-dimension correlations. In terms of the embedding space, this means the base vectors should be uniformly distributed over the hypersphere globally. This implies that the mean of the base vectors should approximate the origin. Therefore, we designed a separate loss for DE(•) as follows:EQUATIONwhere d represents the number of dimensions, T denotes the basis transformation matrix. Eq. 8 ensures the global uniformity of the basis transformation vectors, while the local similarity or dissimilarity is controlled by the dimension's inherent information and the overall model's loss.Following numerous existing works [5, 32, 33] , we implement our graph encoder using GNNs, which effectively encode the feature and structural information of graphs. We use H = XT as the node feature embedding input to GE(•). This approach standardizes any shape of node features to the same dimension, making GE(•) generally applicable across various datasets.Relative Relationship Optimization Task L RT-FUG . Based on the analysis of L RT in Section 3 , the L RT-FUG loss should be divided into positive and negative parts, L RT-FUG + and L RT-FUG -. For negative part, although the InfoNCE loss has demonstrated strong performance in prior studies [8, 5] , it faces challenges with false negative sampling [34, 35] and high computational complexity (which is O(n 2 ), see Appendix B.1). Moreover, existing studies have shown that InfoNCE-based loss functions face conflicts in graph data, making the model gradient unable to descend [36] . Thus, we propose a new type of loss based on the analysis in L RT . Ideally, representations should be able to be divided into multiple clusters based on their intrinsic semantics, with nodes within the same cluster close together (positive pairs) and nodes between clusters far apart (negative pairs). Without prior knowledge, accurately identifying negative pairs is difficult and prone to sampling bias. Therefore, we opt not to explicitly determine negative pairs but instead constrain the overall representations, as:EQUATIONwhere z i represents the embedding of node v i . Similar to Eq. 8, we first constrain the representations to a unit sphere and then force the mean of the representations to be close to the origin. From Eq. 9, we reduce the complexity of the negative sampling in InfoNCE loss which isO(n 2 ) to O(n).For positive sampling, inspired by existing work [37, 31, 30] and based on the natural homophily assumption in graphs, which means connected nodes have similar semantics, we consider node neighbors as positive samples. As Eq. 8 and 9 both use the l2 distance, for loss balance, we also adopt l2 loss here:EQUATIONwhere E represents the set of edges. Note that Eq 9 focuses on encoding global differences, while Eq. 10 focuses on encoding local homogeneity, complementing each other without conflict. The relationship between of L RT and L RT-FUG is in Appendix A.3. The final loss is defined as:EQUATIONwhere, λ 1 , λ 2 , λ 3 are hyper-parameters which adjust the weight of the three losses. Compared to the classical InfoNCE contrastive loss, L FUG better encodes the relative relationships between nodes and offers a lower computational complexity of O(n + e + d) (Please see Appendix B.1 for details). Notably, although the final loss involves three hyper-parameters, in fact, FUG has only one additional hyper-parameter compared to classical GCLs (or even fewer, as FUG does not require tuning multiple augmentation-related parameters). Details can be found in Appendix E.3. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: We provide instructions for each of our conclusions in the main paper and formal proofs in the appendix.• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experimental result reproducibility: 
 Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We provide full explanations and implementation details for the proposed FUG strategy and the instantiated FUG model in the main paper and in the appendix. And we provide the core code as supplementary material in the submission. All datasets required for the experiments are public datasets. In addition, we will release the source code after the paper is accepted.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
experiment settings: 
 To comprehensively evaluate the effectiveness of FUG, we selected commonly used graph datasets with diverse node feature shapes, including Cora, CiteSeer, PubMed [11, 12] , Photo, Computers [38] , CS, and Physics [39] . Detailed information about these datasets can be found in Appendix E.1. We primarily compared four types of methods: supervised GNN model (GCN [1] ), classic machine learning algorithms (PCA [16] , DeepWalk [40] ), advanced self-supervised contrastive models (GRACE [5] , BGRL [32] , GBT [41] DGI [33] , GCA [42] , ProGCL [34] ), and recent graph universal pre-training models (OFA [17] , GraphControl [18] ). In both self-supervised model-rebuilding tests and cross-domain transfer tests, FUG is initially trained in a self-supervised manner. The trained model is then frozen, and node representations are generated on the test datasets. These representations are subsequently used as input for downstream predictors. For the node classification experiments, to simulate a few-shot scenario, we split the dataset into train, valid, and test sets by 10%/10%/80%. We train a simple downstream l2 classifier [35] using only 10% labels. Detailed experimental settings can be found in Appendix E.3. We conduct experiments to test FUG's performance in cross-domain learning, as shown in Table 2 . FUG-C represents a fixed FUG model where all hyperparameters remain constant during pre-training. This presents a more challenging scenario, as it means we cannot adjust parameters based on changes in the training datasets. In Table 2 , we report the optimal accuracy of the state-of-the-art graph universal pre-training models, OFA and GraphControl, which are capable of training across datasets with different node features. These models have access to richer data during pre-training than FUG, which is not fair to FUG. However, as Table 2 shows, FUG's overall performance in cross-domain learning significantly surpasses both models. This is because OFA and GraphControl employ data preprocessing methods that significantly degrade the information in the node features (OFA uses LLM to reshape node features to the same dimension, and GraphControl replaces original node features with feature similarity). This validates our views made in Section 1 and Appendix D. It also demonstrates that FUG's dimension encoder can almost losslessly adapt to graph data with various node feature shapes.Additionally, we derived some interesting findings from Table 2 : 1). Counterintuitively, FUG-C's performances are not necessarily the best when pre-trained and tested on the same dataset. This is mainly because our loss functions focus on the relative semantic relationships between nodes, allowing the encoder to learn the knowledge of relative relationships. This leads to: a) This knowledge is generalizable across different datasets, so even though the datasets are from different domains, this knowledge remains the same. In other words, relative semantic relationship knowledge has weak cross-domain limitations, as stated in Section 3. b) The learning of relative semantic knowledge primarily relies on the distributional differences between nodes and between different feature dimensions, rather than on the absolute semantics contained in the data. Therefore, it is common in Table 2 to see that FUG-C pre-trained on other datasets performs better than when pre-trained and tested on the same dataset. 2). The performances of cross-domain trained FUG-C are close to that of FUG which is trained and tested on the same datasets with tuned hyper-parameters. In fact, on the Photo dataset, FUG-C (Computers) outperforms FUG. This demonstrates FUG's strong cross-domain learning capability and the success of our proposed dimension encoding and relative relationship loss. It proves FUG's robust applicability to various graph data. In Table 3 , we designed node classification experiments to evaluate FUG in intra-domain selfsupervised scenarios. We also report the computational overhead of FUG when the hidden layer is fixed to 256 as shown in Table 5 , and the complete results are in Appendix B.2. From Table 3 and 5, we can observe that FUG demonstrates competitive performance. In Table 3 , we observed that: 1). FUG outperforms GRACE across all datasets, reaffirming that the learnable DE(•) encoder in FUG can almost losslessly standardize node features to the same shape. This also supports our views in Section 3 that both positive and negative sampling in the InfoNCE loss inherently constrain the relative relationships between nodes. Furthermore, it demonstrates that our loss function maintains excellent performance while reducing computational overhead. 2). FUG significantly surpasses PCA on all datasets. Although PCA, FUG, and GRACE follow the same framework, PCA is limited by its inability to encode graph structure and the relative semantics between data points, leading to poor performance. This validates our previous views. FUG simulates PCA's encoding process through graph contrastive learning, thereby addressing PCA's shortcomings and significantly improving performance. 3). FUG generally outperforms the negative-sample-free GBT and BGRL, while maintaining comparable time costs as shown in Table 5 . This indicates that considering negative relative relationships between data points is effective, and FUG successfully reduces the time and space overhead associated with these negative considerations to match that of methods that do not consider negative samples. Additionally, we compare FUG with latest advanced in-domain self-supervised methods [43, 44, 45] in Appendix E.4.Figure 3 : Hyper-parameter analysis of FUG.To validate the effectiveness of each component in FUG, we conducted ablation studies and parameter analysis, as shown in Table 4 and Fig. 3 .From Table 4 , it is evident that FUG outperforms other models without part of loss functions across all datasets, demonstrating the effectiveness of the three proposed loss functions. Furthermore, we observed that FUG w/o L RT-FUG + generally achieves the second-best performance. This is primarily due to L RT-FUG -constraining global uniformity without focusing on the distance between local node pairs, preventing the model from over-encoding differences between nodes. Additionally, FUG w/o L DE generally exhibits the worst results. The main reason is that, L RT -F U G + and L RT -F U G -are effective when based on L DE , especially for datasets with high quality features. As described in Section 5.4, when L DE is absent, DE is not sufficiently trained, which leads to DE embedding X almost randomly to generate H, which is an important input for GE. Furthermore, without the guidance of L DE , GE only needs to extract relationships in H to reduce the loss, ignoring whether H contains sufficient information from X. Therefore, in FUG w/o L DE , the H input to GE is of poor quality, resulting in worst performances on some datasets. From Fig. 3 , we can observe that as the number of sampled nodes and the dimensionality of the basis transformation vectors increase, the overall model performance improves until it stabilizes at around 1024. This indicates that a limited number of sampled nodes can effectively represent the overall data distribution. Moreover, once the dimensionality of the basis transformation vectors reaches a certain threshold, it can embed dimensional information nearly losslessly, thereby unifying data features to the same shape. This further validates the design of the dimensional encoder.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We provide the source code as supplementary material in the submission.The source code will be released after the paper is accepted, and we will also release a FUG model trained on the 7 datasets mentioned in the paper for easily use. In addition, we believe that the main paper and appendix provide sufficient experimental details to ensure the reproducibility of our model.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
discussions and future works: 
 In this paper, we explore the connection between contrastive learning and PCA, and propose the FUG strategy, which includes specific dimension encoding, inter-dimensional encoding, and relative semantic relationship loss. The models instantiated under this framework achieve near-lossless unification of node attributes with different shapes without data preprocessing, making them applicable to graph data and demonstrating strong cross-domain learning capabilities. We instantiated a FUG model and validated its effectiveness through extensive experiments. Additionally, our theoretical analysis shows that the complexity of the GCL loss involving negative samples is reduced from O(n 2 ) to O(n), further lowering FUG's training costs.In future works, the FUG strategy can support research on graph prompt tuning, universal graph models, and general large models based on graph networks, enabling graph models to train directly on various data. Moreover, upon acceptance of this paper, we will release a pre-trained FUG model on existing seven datasets, allowing for direct use of FUG-generated graph representations as priors for further extensions. While the FUG strategy is general, the instantiated model has some limitations, such as only considering homophilic graphs in positive sampling, which may perform poorly on heterophilic graphs, and not constraining absolute semantic information in the loss function. Future work can further refine the universal graph contrastive pre-training model based on the FUG strategy.A.1 Proof for Theorem 3.1The proof of Theorem 3.1 is divided into two parts: framework equivalence and objective equivalence. We first present the framework equivalence part, where PCA can be decomposed into the Fea(•) and Dim(•) components.Given an input data X with a standardized matrix X = X -M ean(X), PCA is as follows.EQUATIONwhere (Λ, C) represents the eigenvalues and eigenvectors matrix, N orm(•) represents normalization, Concat(•) represents concatenation, λ (k) represents the k-th largest eigenvalue, Cov(•) represents covariance computation, and δ(•) represents eigenvalue and eigenvector computation. This can be decomposed into the forms of Dim(•) and Fea(•) as follows:EQUATIONwhere W 1 is a special parameter matrix, W 1 = X, and φ(•) represents the operation to solve W 2 as:EQUATIONFrom the above, Encode 1 (•) and Encode 2 (•) satisfy linear or nonlinear mappings. So, we derive:EQUATIONThus, PCA can be rigorously decomposed into Dim(•) and Fea(•) as described in Theorem 3.1.Then, we prove that MinimizeL P CA ∝ MinimizeL RT .We start with the loss function of PCA, which maximizes the variance:EQUATIONwhere Z denotes the PCA embeddings. To obtain a form corresponding to Eq. 3, we consider an arbitrary pair of data points z i and z j , as shown below::EQUATIONWhere c represents the sum of the representations of all other nodes. From Eq. 2, Z = XP and M ean( X) = 0. So M ean( XP ) = Z = 0. Therefore, we approximate c to 0 here. The following equation can be derived:EQUATIONhere we use the assumption that all representations have the same length m. When X + =, Eq. 3 can have the following form:MinimizeEQUATIONAssumption Analysis. In Theorem 3.1, we assume that all representations have the same length m.This assumption is reasonable because in many deep learning scenarios, normalizing is a common practice, specifically normalizing the length of the embeddings to 1.Similar to Appendix A.1, the instantiation part of the theorem has already been clearly explained in the main text. We consider Dim(X) = X as a special form of dimensional encoding instantiation, given that CL essentially lacks a dimensional encoding component. We view the neural network in CL as the instantiation of Fea(•). Here, we prove that Minimize[L CL ] ∝ Minimiaze [L RT ].Proof: First, we define a set X + A representing the positive pairs in contrastive learning. (x i , x j ) ∈ X + A means that x i and x j are two augmented data points derived from the same original data point. Then we have:EQUATION)where z i means the embedding of data point x i . Considering a positive pair (x a , x + a ) ∈ X + A , we can obtain:EQUATIONwhere c p , c n are constants Considering a negative pair (x a , x - a ) ∈ X -, we can obtain:EQUATIONSo, we have:EQUATIONHere we get the conclusion that minimizing the contrastive InfoNCE loss is maximizing the similarity between positive pairs and minimizing the similarity between negative pairs. Based on our assumption, data augmentation does not alter the essential semantics of the data. We define H as the data stripped of all irrelevant and redundant information by data augmentation, containing only the essential semantics. For a data point x a , maximizing (xa,xj )∈X + A Sim(x a , x j ) is equivalent to maximizing (xa,xj )∈X + A Sim(x a , h a ). For downstream tasks, data of the same category embodies the same essential semantics. Therefore, for a set D of data points have similar essential semantics, their h values are identical. For any two data points x a , x b ∈ D ∥ , we have:EQUATIONComparing equations 19 and 24, we can conclude that the PCA optimization objective is, in fact, a special case of the contrastive loss when X + = ∅. □ Assumption Analysis. We assume that data augmentation does not alter the essential semantics of the data, which is an idealized condition. Many works in contrastive learning aim to achieve this through augmentation. Therefore, this assumption is reasonable. Additionally, this assumption is made for the sake of clarity. Under this assumption, the set D is correctly defined, meaning all data within the set share the same essential semantics. If the assumption does not hold, the essential semantics within the set D may not be identical across all nodes. However, this does not affect the correctness of the proof, as the set D still exists.In this section, we discuss the relationship between L RT-FUG and L RT .Theorem A.1 Assuming the representations have the same length m, then:EQUATIONProof:The L RT-FUG consists of both positive and negative parts, which we discuss separately. For L RT-FUG + , we have:EQUATIONFor L RT-FUG -take any two nodes v i , v j then we have:EQUATIONhere, we also use a similar trick as in Appendix A.1, treat the constant vector c as 0:EQUATIONHence, from Eq. 28 and 26, we can obtain:EQUATION□ As shown in Eq. 9, L RT-FUG in the negative sampling part only requires calculating the mean of all node representations after normalization and computing the distance to the origin. In fact, since Mean(•) -0 = Mean(•), we only need to perform operations directly on the mean. Thus, its computational complexity is O(n). The negative sampling part of the InfoNCE loss is well-known for its high complexity, as it must compute the similarity between every pair of nodes, resulting in O(n 2 ) complexity. Therefore, we state in the main paper that L RT-FUG , as an optimized contrastive loss, reduces the original time complexity from O(n 2 ) to O(n).As shownin Eq. 10, for the positive sampling part, we discard the costly data augmentation and, guided by Theorem 3.2, explicitly define neighbors as positive samples. Hence, we only need to compute the distance for the two points linked by an edge. Its time complexity is O(e), where e is the number of edges. Therefore, the overall complexity of the RT-FUG loss is O(n + e).DE Loss. Similar to L RT-FUG -, we compute the mean of the basis transformation matrix for each dimension and its distance to the origin. Thus, its time complexity is O(d).Overall Time Complexity. From the above, the overall time complexity of the loss is O(n + e + d).Furthermore, considering d is the number of dimensions of node features, it is negligible compared to the number of nodes n when the data is large enough. Additionally, even if the number of nodes is very large, the average degree of nodes is generally small. Assuming the average node degree is a fixed constant a, the overall time complexity becomes O((2a + 1)n), which can be regarded as O(n). For accuracy, we still state the complexity of L FUG as O(n + e + d) in the main paper.In this section, we present the complete computational cost experiments reported in To further illustrate how FUG works, we provide pseudocode as shown in Algorithm 1 Algorithm 1 FUG Training Input: Data {X (g) , A (g) |(X (g) , A (g) ) ∈ G (g) , g = 1,X (g) = Sample(X (g) ) 5: T = Norm(DE θ ( X (g)))6:H = X (g) T 7: Z = GE ϕ (H, A(g)) 8:/* Calculate loss */ 9:EQUATION10:EQUATION11:EQUATION12:L FUG = λ 1 L RT-FUG -+ λ 2 L RT-FUG + + λ 3 L DE 13:/* Update parameters */ Graph Contrastive Learning (GCL). Graph Contrastive Learning is dedicated to learning graph representations in a self-supervised manner [35] . Initially inspired by the success of contrastive learning in the visual domain [8, 46, 47] , approaches such as GRACE [5] were proposed. These methods generate augmented views by randomly perturbing graph data, treating nodes augmented from the same original node as positive samples, and other nodes as negative samples. They employ the InfoNCE loss [48] to maximize mutual information between positive pairs while minimizing it between negative pairs. Numerous improvements have been made within this framework. To address the computational burden of negative sampling, some studies [32, 41] have explored training methods that do not require negative sampling. BGRL [32] utilizes two distinct encoders, Online and Target, with the Online encoder learning to replicate the Target's output for positive samples. GBT [41] proposes a loss to reduce redundancy in node representations. To mitigate the semantic distortion caused by random graph data augmentations, GCA [42] assesses the importance of nodes based on degree centrality, increasing perturbations for less important nodes. HomoGCL [28] uses community detection algorithms to assess the importance of edges and node features for designing augmentation. AD-GCL [49] introduces a learnable data augmentation strategy through a minimax approach. SpCo [50] proposes a graph spectral-based augmentation strategy. Works like AFGRL [31] , AFGCL [30] , and NeCo [29] have introduced contrastive learning algorithms that do not use data augmentations. To resolve sampling biases in positive and negative sampling, LocalGCL [37] and NeCo [29] suggest using neighbors as positive samples. ProGCL [34] utilizes a Beta distribution fitting approach to select negative samples. While these methods have shown great success, they all use GNNs [1, 22] as encoders, meaning the trained models are not applicable to other datasets with different forms of node features.Unified Graph Pre-training Models Inspired by the recent success of unified large language models [51] , numerous studies have attempted to propose unified, generic graph pre-training models for graph data. All-In-One [52] employs transferable graph prompts, allowing graph pre-training models to be applicable across various tasks, though it still cannot handle graphs with diverse features and relies on methods like PCA to standardize node feature dimensions first. OFA introduces a pre-training and fine-tuning pipeline that transforms all downstream tasks into link prediction tasks, with the model itself lacking adaptability to arbitrary node features. It uses large language models to convert node features into a uniform format. Classic approaches like GCC [6] and recent methods like UniLink [15] discard original node features in favor of encoding new topology-based positional attributes, allowing application across diverse data but resulting significant information loss. Very recently, GraphControl [18] converts distances between node features into a matrix similar to adjacency matrices, utilizing the universality of GNNs for arbitrary topologies to achieve attribute universality. However, this approach, which only encodes distances between node features, still significantly harms the semantic integrity of the original node features.Although these methods have extended the applicability of existing GNNs, they still lead to considerable loss of node feature information during preprocessing. In contrast, our FUG retains the semantic integrity of node features while ensuring the universality of the graph pre-training model.We follow many prior works [5, 42, 32, 33] and evaluated our performance on seven widely used public datasets:• Cora, CiteSeer, and PubMed are the citation datasets from PlantoID [11, 12] . Nodes represent scientific papers, edges represent citation relationships, node features are bag-ofwords vectors of papers, and labels represent domains of papers.• Photo and Computers are co-purchase graphs from Amazon [38] . Nodes represent products, edges represent frequent co-purchase relationships, node features are bag-of-words vectors of product reviews, and labels represent product categories.• CS and Physics are from the Microsoft Academic Graph [39] . The nodes represent researchers, the edges represent paper co-authorship relationships, the node features are the bag-of-words vectors of the authors' papers, and the labels represent the research fields of the authors.The statistical information is presented in Tab. 7. We follow the exist work [5, 35] , utilized the Python library "torch_geometric.dataset" to load the datasets. Bidirectional links were established for all edges. For all the data, we divided the training, validation, and test data sets with 10%, 10%, and 80%, to simulate the scenario with fewer tags downstream. To further showcase FUG's performance in in-domain scenarios, we compare FUG with the latest state-of-the-art algorithms, as shown in Table 10 . The results for GraphACL [43] and ROSEN [44] are obtained using the same experimental settings as described above. For POT [45] , a plug-in method with similar experimental settings to ours, we directly report its best performances as stated in its original paper. As shown in Table 10 , even in intra-domain scenarios (which are not our primary focus), FUG demonstrates competitive performance, achieving the best results on three out of five datasets. This highlights the effectiveness of FUG and supports the claims made in our paper. To fully illustrate the interactions among the three losses included in FUG, we report the analysis experiments on the weight parameters λ 1 , λ 2 , and λ 3 across all datasets, as shown in Figure 3 .Here, we describe the limitations of the proposed FUG model. The FUG model serves as a straightforward implementation to validate our proposed FUG strategy and has some imperfections. For example, the loss L RT-FUG + in FUG model is based on the assumption of graph homophily assumption, which leads to good performance on homophily datasets. However, it might face challenges on heterophilic datasets because neighboring nodes in such datasets often have different semantics.Additionally, the encoder used in FUG is a GCN, which also operates under the graph homogeneity assumption. Furthermore, the GCN encoder we use can only handle weighted ordinary edges and homogeneous nodes. Therefore, when dealing with heterogeneous graphs, where node types within the same dataset differ, the model still faces applicability challenges. These issues can be addressed by following the FUG strategy to construct new instantiated models, which is a direction we aim to explore in future works.Here, we discuss the social border impacts related to the FUG strategy and model. Justification: We have stated the limitations of the proposed FUG model in the last section of the main paper. In addition, we set up a separate section in the appendix to state the limitations of the proposed FUG theory and the instantiated model. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We present the experimental setup in the main text and provide details in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: For all experiments with the FUG model, we provide the standard deviation of 20 runs of the random split to describe the randomness of the model. We provide the details in the appendix. Guidelines: Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: FUG is a basic research on graph pre-training models. The training datasets are all public datasets. We believe that FUG does not have this risk. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: The authors introduce Feature-Universal Graph Pre-training (FUG), a novel framework inspired by Principal Component Analysis (PCA) designed to enhance the generalizability of graph pre-training models. FUG features a parameterized dimension encoding component that unifies various node attributes while preserving essential semantics. By integrating the principles of PCA and Graph Contrastive Learning (GCL), FUG aims to learn invariant knowledge across different datasets. The model's innovative approach addresses the challenge of generalizing to graph data with diverse node feature shapes, and extensive experiments demonstrate its effectiveness, particularly in cross-domain training scenarios.

The reviewers praise the paper's clarity, solid theoretical foundation, and innovative approach that unifies Principal Component Analysis (PCA) with Graph Contrastive Learning (GCL) both methodologically and theoretically. The Feature-Universal Graph Pre-training (FUG) strategy is recognized as a significant contribution to the field of graph representation learning. While some issues were raised concerning the theoretical analyses and experimental evaluations, the reviewers were generally satisfied with the authors' further explanations provided during the discussion period.

Reviews: review: paper_summary: paper_summary: In this paper, the authors try to answer the question: Could a graph contrastive pre-training principle based on PCA theory be developed to address inherent issues of PCA and broadly apply to node features of different shapes? Then the authors start with some theoretical analysis and then propose a method namely FUG, which has good transferability. The method is validated on several datasets to test the performance on cross-domain and inter-domain tasks.
strength_weakness: strength_weakness: -   The theoretical analysis is somewhat loose. For instance, in Theorem 3.1, $Dim(\cdot)$ and $Fea(\cdot)$ are loose formulations since their mathematical properties are not well defined. The authors should clarify that a function or mapping is a Dim as long as it satisfies some pre-conditions. The theoretical analysis may not be qualified for NeurIPS. Moreover, the equivalence between $\mathcal{L}_{RT}$ and InfoNCE is quite similar to some existing works, such as [1]. So the contribution from the theoretical aspect may be over-claimed.  
-   From the ablation experiments, it seems that CE is the most important part. However, it is unclear why CE can significantly improve the performance. A theoretical proof will be convincing. 
-   The pre-training of GNNs has been studied recently. However, the proposed FUG is not compared with these methods.  Here is a survey as an example [2].
-   There are totally 3 hyperparameters in the final loss, which may result in great difficulty in tuning them in practice.  

[1] A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses

[2] Graph Prompt Learning: A Comprehensive Survey and Beyond
questions: questions: -   What is the connection between the proposed FUG and graph prompt tuning? Which one is more practical in practice?
-   Why does CE improve performance significantly? 
-   Can RT-FUG- and RT-FUG+ be replaced by InfoNCE or other contrastive loss?

For more questions, please see Weaknesses.  

Overall, I'd like to discuss this paper with the authors and other reviewers during rebuttal. And then update my rating.
limitations: limitations: Yes
paper_summary: paper_summary: The authors propose a universal graph pretraining framework called FUG. Inspired by PCA, FUG includes a parameterized dimension encoding component to unify different forms of node attributes, thereby preventing the loss of important semantics. The authors design an optimization objective based on relative semantics to learn invariant knowledge across datasets. FUG demonstrates excellent performance in cross-domain training.
strength_weakness: strength_weakness: 1. The authors introduce a new loss, which, in my view, has a primary advantage over the InfoNCE in terms of faster training speed. However, its effectiveness on positive sampling compared to existing GCLs is questionable as noise based augmentation not only aligns positive samples but also enhances the model's robustness. Using neighbors as positives might make the model sensitive to noise.  
2. Although this paper is well-written, I still recommend that the authors emphasize the two questions raised in Section 3. This would enable readers to quickly find the questions corresponding to the two answers.  
3. In Theorem 3.1, dividing PCA into Dim. and Fea. seems inappropriate. In fact, I tend to believe that PCA cannot be clearly separated into these two parts. Matrix decomposition belongs to Fea., but the ordering of eigenvalues might be more reasonably viewed as Dim., as it can be considered a nonlinear mapping based on the relative relationships between dimensions.
questions: questions: Please see the weakness.
limitations: limitations: Yes
paper_summary: paper_summary: The paper is generally well-presented. It aims to address the generalizability of graph pre-training models to graph data with arbitrary node feature shapes. By comparing the Principal Component Analysis (PCA) with existing graph contrastive learning methods, the authors discovered that PCA follows the same framework as these graph contrastive learning methods. Based on this observation, they proposed a Feature-Universal Graph Pre-training (FUG) strategy and instantiated a FUG model. Extensive experiments have been conducted to validate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. Equation 4 is somewhat unclear. Although the authors provide an intuitive explanation, a more effective approach would be to formally decompose PCA according to the form on the right side of the equation, making it easier to understand.
2. The authors state that FUG is inspired by existing works discussing the relationship between PCA and contrastive learning. The authors should more clearly introduce these works and compare them with the proposed theory to highlight the theoretical contributions.
 
Minor issues and typos: 
1. The subscript font for symbols should be consistently either italic or upright. Additionally, Table 5 should specify the units of the values.
2. Line 180: Maximize $[Sim(x_i, x_j )$
questions: questions: Please see the weaknesses.
limitations: limitations: Please see the weaknesses.
paper_summary: paper_summary: In this paper, it is discovered that contrastive learning and PCA follow the same framework. By combining the two, the authors propose a graph pre-training model that offers both the generalization capability of PCA for arbitrary node shapes and the powerful embedding ability of GCLs. This approach enhances the generality of existing graph pre-training frameworks.
strength_weakness: strength_weakness: - The primary advantage of the proposed model lies in its robust cross-domain encoding capability. However, the paper lacks a discussion of existing works on graph cross-domain learning [1].  
- The authors claim that PCA is limited by its inability to encode homogeneous information within the data and the inflexibility of the encoder. However, they do not explain why they did not directly use a graph encoder on the PCA-generated embeddings.  
- Equation 10 reduces the Euclidean distance between neighboring node representations. This implies forcing the alignment of neighbor features across each dimension, which seems to contradict the authors' idea of relative semantic encoding. It appears that a constraint on the similarity would better align with this idea.  
- The proposed model seems to lack a projector between representations and the loss, a component widely present in GCLs. The authors should provide more discussion to explain this, or it may be that they omitted the introduction of it.  

[1] Unsupervised Domain Adaptive Graph Convolutional Networks, WWW’ 20.
questions: questions: - Table 2 demonstrates the remarkable transfer learning capabilities of FUG. However, does this indicate that collaborative training on multiple datasets does not significantly improve the model's performance in cross-domain scenarios? A similar observation is made in GCC, where models pre-trained on multiple datasets do not show a substantial performance improvement compared to those pre-trained on a single dataset.
limitations: limitations: Yes, they have.
paper_summary: paper_summary: In this paper, the authors design a feature-universal graph contrastive pre-training strategy for model rebuiding and data reshaping. They focus on the inherent complex characteristics in graphs, and introduce a theoretical analysis for their method. The gloabl uniformity constraint reduces the time complexity from O(n^2) to O(n).
strength_weakness: strength_weakness: 1. Line 59 points out the shortcomings of the language model, but with the development of LLM, numerous LLM4graph papers have been proposed, effectively solving the problem of feature diversity.

2. The results in Table 3 demonstrate certain effectiveness of the designed method, but the methods compared are not the latest [1, 2, 3, 4]. The improvement in performance is border. The effectiveness of the design method is not convincing enough.

3. The results in Table 4 show that DE has a significant impact on model performance, while the other two parts have little impact on the model. How can the effectiveness of these two module designs be proven?

[1] Xiao, Teng, et al. "Simple and asymmetric graph contrastive learning without augmentations." Advances in Neural Information Processing Systems 36 (2024).

[2] Yu, Yue, et al. "Provable training for graph contrastive learning." Advances in Neural Information Processing Systems 36 (2024).

[3] Zhuo, Jiaming, et al. "Graph contrastive learning reimagined: Exploring universality." Proceedings of the ACM on Web Conference 2024. 2024.

[4] Bo, Deyu, et al. "Graph contrastive learning with stable and scalable spectral encoding." Advances in Neural Information Processing Systems 36 (2024).
questions: questions: 1. There is a lack of sensitivity analysis experiments for the hyperparameters in Equation 11.

2. In Equation 9, z is made to approach 0. Will this operation lose the diversity of features?
limitations: limitations: The use of Gaussian function can also achieve uniformity. Please analyze the advantages and disadvantages compared to the method designed in the paper.

Comments: title: About the responses
Response to authors
comment: Thank you for the response, which well addressed my main concerns. I have carefully read the response and the other reviews. In my opinion, the authors proposed a highly universal graph pre-training strategy, supported by theoretical and experimental evidence, which is both sound and interesting to me. So, I support the acceptance of this paper.
Thank you again for your review and the score. We will further improve our paper follow your suggestions.
We sincerely appreciate the time and effort you invested in reviewing our paper, as well as your new score! We will carefully revise our paper according to your suggestions. Thank you again for your valuable comments, which have greatly enhanced the theoretical part of our work.
I thank the authors for the response. I have read the response and other reviews. I admit that the paper has some merits. I would raise my rating to 5.
Thanks for your rebuttal. I would keep the original score.
We sincerely thank you for your recognition of our work and these valuable suggestions! We will carefully fix these issues in future versions and double-check our paper to improve its quality.
Thank you for your response. I would like to maintain my previous scores. By the way, please revise them carefully in your future version.