FUG: Feature-Universal Graph Contrastive Pre-training for Graphs with Diverse Node Features
Authors: Jitao Zhao
Di Jin
Meng Ge
Lianze Shan
Xin Wang
Dongxiao He
Zhiyong Feng
Venue: NeurIPS 2024 poster (2024)
Keywords: Graph pre-training
Graph contrastive learning
Graph foundation model
Principal component analysis
Primary area: graph_neural_networks

Abstract: Graph Neural Networks (GNNs), known for their effective graph encoding, are extensively used across various fields. Graph self-supervised pre-training, which trains GNN encoders without manual labels to generate high-quality graph representations, has garnered widespread attention. However, due to the inherent complex characteristics in graphs, GNNs encoders pre-trained on one dataset struggle to directly adapt to others that have different node feature shapes. This typically necessitates either model rebuilding or data alignment. The former results in non-transferability as each dataset need to rebuild a new model, while the latter brings serious knowledge loss since it forces features into a uniform shape by preprocessing such as Principal Component Analysis (PCA). To address this challenge, we propose a new Feature-Universal Graph contrastive pre-training strategy (FUG) that naturally avoids the need for model rebuilding and data reshaping. Specifically, inspired by discussions in existing work on the relationship between contrastive Learning and PCA, we conducted a theoretical analysis and discovered that PCA's optimization objective is a special case of that in contrastive Learning. We designed an encoder with contrastive constraints to emulate PCA's generation of basis transformation matrix, which is utilized to losslessly adapt features in different datasets. Furthermore, we introduced a global uniformity constraint to replace negative sampling, reducing the time complexity from $O(n^2)$ to $O(n)$, and by explicitly defining positive samples, FUG avoids the substantial memory requirements of data augmentation. In cross domain experiments, FUG has a performance close to the re-trained new models. The source code is available at: https://github.com/hedongxiao-tju/FUG.

FUG: Feature-Universal Graph Contrastive Pre-training for Graphs with Diverse Node Features
Jitao Zhao', Di Jin Meng Ge', Lianze Shan', Xin Wang', Dongxiao He', Zhiyong Feng' CColleee of Intelligence and Computing, Tianjin University, China DDpartmentt of Electrical and Computer Engineering, National University of Singapore, Singapore (zjtao, jindi, shanlz21 19, wangx, hedongxiao, zyfeng) @tju.edu.cn, ?gemeng nus.edu.sg
Abstract
Graph Neural Networks (GNNs), known for their effective graph encoding, are extensively used across various fields. Graph self-supervised pre-training, which trains GNN encoders without manual labels to generate high-quality graph rep- resentations, has garnered widespread attention. However, due the inherent complex characteristics in graphs, GNNs encoders pre-trained on one dataset struggle to directly adapt to others that have different node feature shapes. This typically necessitates either model rebuilding or data alignment. The former results in con-traasferability as each dataset need to rebuild new model, while the latter brings serious knowledge loss since it forces features into uniform shape by preprocessing such as Principal Component Analysis (PCA). address this chal- lenge, propose new Teature-Universoe Graph contrastive pre-training strategy (FUG) that naturally avoids the need for model rebuilding and data reshaping. Specifically, inspired by discussions in existing work the relationship between contrastive Learning and PCA, we conducted a theoretical analysis and discovered that PCA's optimization objective is a special case of that in contrastive Learning. We designed an encoder with contrastive constraints to emulate PCA's generation of basis transformation matrix, which is utilized to losslessly adapt features in different datasets. Furthermore, we introduced a global uniformity constraint to replace negative sampling. reducing the time complexity from O(n²) to O(n), and by explicitly defining positive samples, FUG avoids the substantial memory requirements of data augmentation. In cross domain experiments, FUG has performance close to the re-trained new models. The source code available at:
Introduction
Graph Neural Networks (GNNs) []. renowned for their effectiveness in encoding both graph topology and feature information, have emerged as the mainstream methods in graph representation learning. They extensively applied in various scenarios such as protein molecule prediction (2) e-commerce
recommendation systems (3] and community detection [4. Training high-quality GNNs typically relies extensive manually labeled data (5), which is costly and limits their applications. To address this challenge, graph self-supervised pre-trainings have gained significant attentions [6] 5] These
approaches leverage self-supervised signals mined from the graph data themselves and design proxy tasks to train models without manually labeled data.
Unfortunately, due to the inherent complexity of graph data, models trained graph dataset often fail to perform well on another. This complexity arises from two main aspects: 1). topological
Corresponniig author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

diversity, where a node in graph can connect to any number of other nodes, and 2). feature diversity, where nodes represent a variety of information such textual or float data. In domains like vision [8] text [9, and speech [0], data shares common latent semantics (such as RGB channels in pixels, word vectors, or audio samples), and have relative positional information like spatial relations in vision or sequence relations in text and audio. Unlike these domains, features in graphs are unordered and chaotic. Even within the same citation dataset category using identical feature extraction methods, such Cora, CiteSeer, and PubMed [II, the node features have different shapes.
Existing GNNs can address the issue of topologiaal diversity through well- designed message-passingg passing and aggregation methods [T3] 14]. However, due the neural networks' requirement for fixed input-output dimen- sions, GNNs struggle with feature diversity. This necessitates retraining new graph neural network for datasets with different node feature shapes, significantly restricting the applicability of existing graph self-supervised pre-training models. This leads to some graph pre-training works [6] [5] that have to ignore node attributes and encode only the structural infor- mation. Recent "All-In-One" work in graph attempts to reshape node features to the same dimensions through preprocessing methods like Prin- cipal Component Analysis (PCA) [16 171, feature similarity embedding [18 and language models [17]. However, these methods inevitably lose crucial semantic information related to downstream tasks during data preprocessing. Classical machine learning methods such as PCA cannot participate in training, and they primarily encode the information with the highest variance as principal components, overly focusing differences between node features and losing substantial homophily information in node features. Encoding informative features into single-dimensional similarity also brings obvious information loss. Language models suffer assumption biases, presuming that all input data is textual, thus inef-
Train Dataset
Test Dataset
UNN Encoda
Goph Pre-uni Midkls
Prososen
FUG
Figure I: Motivation.
fectively handling extensive float features. Furthermore, using large language model interfaces or involving language models in training incurs additional significant costs. Additionally, sliding window- based neural networks, such as ID-CNNs [19] and LSTMs (20), also fail to address the issue of feature diversity. While they can handle inputs of varying dimensions, they operate under the prior assumption that the data is sequential, which is not satisfied by graph node features.
Among the feature-unifying approaches discussed earlier, only PCA-like methods exhibit no assump- tion bias and are adaptable to various feature forms. Their primary limitation lies in the misalignment between their model objectives and the proxy tasks of graph pre-training, coupled with their inability to train based on specific loss functions. Noting discussions in existing works that link contrastive learning with PCA-like methods, which regard contrastive pre-training as generalized form of non- linear PCA (21) we draw inspiration to propose a question: Could graph contrastive pre-training principle based on PCA theory be developed to address inherent issues of PCA and broadly applied to node features of different shapes?
In this paper, theoretically demonstrate that classical PCA and contrastive learning fundamentally follow the same theoretical framework. PCA's optimization objective is a special case of that in contrastive learning. The adaptability of PCA-like methods various feature shapes stems from their intrinsic component that acts as columnar encoder for values under specific feature dimension, creating a basis transformation matrix, which term dimensional encoding. The loss and parameter update mechanisms of contrastive learning could offer traditional PCA methods new goals and training mechanisms that align better with the graph data and downstream tasks. Based on these analyses, we propose a new strategy for Univeeral Graph pre- pee-traniig (FUG strategy), which includes three steps: dimensional encoding, graph encoding, and relative relationship optimization task.
Guided by this theory, we instantiate simple FUG model. Simulating PCA, this model utilizes sampling to capture the distribution among different nodes within specific feature dimension for dimensional encoding, and generates basis transformation vectors for feature dimensions which used to losslessly unify the node features. It employs GNN encoder for graph encoding. We also design an improved and more efficient graph contrastive loss the relative relationship optimization task. According to this task. FUG effectively learns the relative relations within graph data, and transfers this encoding capability effectively other datasets with different node feature shapes.

---

We conducted extensive experiments to validate FUG's efficacy. In in-domain self-supervvsed learning experiments, FUG demonstrated competitive performance with existing advanced graph self-supervised models. In the cross-domain learning experiments, the FUG trained and tested on different datasets shows similar performance to the model trained and tested on the same dataset. These results prove the strong adaptability and transferability of FUG.
2 Notations and Preliminary
In this section, introduce some preliminary concepts and notations facilitate analysis in later. We consider a graph data 9 = (V,&), where V = {vi, V2,,-, vn) represents the set of nodes and E S V x V represents the set of edges. X E gx denotes the feature matrix the nodes. A is the adjacency matrix representing the topological relationships between nodes, with dij 1 if eij E and 0ij = 0 otherwise.
Graph Neural Networks (GNNs). GNNs can be uniformly described as HT(+1) = AW H where
H represents the embeddings in hidden layer, with t = 0 denoting the node features X. A is the message passing matrix, which can be derived through neighbor sampling [3] random walks, or deep learning [22] techniques to mine graph data. W is a learnable parameter matrix. Under the successful design of existing works, the message passing matrix À can generically encode graph data with varying edges. However, the presence of the parameter matrix W means that a trained GNN model can only be directly applied graphs with the same node feature dimensionality the training dataset, significantly limiting the application scenarios of GNNs.
Graph Contrastive Pre-training. It is based on Contrastive Learning (CL). Given graph G, graph contrastive pre-training aims to train a GNN encoder f(-) in a self-supervised manner, which generates low-dimensional dense graph representations 7 = =G) for downstream tasks. Typically, InfoNCE loss functions are used for training, formulated as:
LIsTONCE loo
(I)
[(rr,))EPS (S)n + Σ (VI.)))NS (Sim(g(()).ovv))))
where PS denotes the data sampled as positive pairs, and NS denotes the data sampled as negative pairs. Sim(-) represents the cosine similarity caleulation, 9(-) is a non-linear projector, is temperature parameter. The training objective is to increase the similarity between positive pairs and decrease the similarity between negative pairs.
Principal Component Analysis (PCA). Given data with any number of dimensions d, PCA aims to reduce the dimensionality to k dimensions, where k S d, while maximizing the retention of variance between the data. PCA based on eigendecomposition shown as:
PCA(X) XP, P Norm [Concat E C, do 2 (x) }}, (A, C) 8[Cov(X,,
where P represents the basis transformation matrix, X = - Mean(X). c2 denotes the i-th eigen- vector, A; denotes the corresponding eigenvalue, (kk denotes the k-th largest cigenvalue. Concat(-) represents concatenating the cigenvectors within the set, and Norm(-) represents normalization. C and A represent the matrix composed of eigenvectors and the corresponding eigenvalues, respectively. Cov(-) represents covariance computation, and 5 denotes matrix decomposition. Note that since the covariance matrix is symmetric, the extracted eigenvectors are orthogonal to each other. This formu- lation mathematically ensures that the dimensionality-redueee reduced embeddings maximize the retention of variance between the original data and minimize homogeneity among the data.
3 Graph Contrastive Pretrained Models from the PCA perspective
In this section, we first theoretically summarize the framework that PCA essentially follows. We then analyze contrastive learning. demonstrating that also adheres to this framework, and the PCA optimization task is special case of that of CL. Furthermore, we answer the question why the PCA embeddings generally underperform graph contrastive pre-training in downstream tasks within the same framework, and why graph contrastive pre-training models lose the ability to adapt datasets with varying node feature shapes. Based on these findings, we propose a new strategy for Universal Graph pre- pre-training (FUG strategy). First we define class of objectives:
3

---

Definition 3.1 (Data relative relationship optimization task LRT). Given dataset X and model Φ()), Lur involves maximizing or minimizing the 12-distance between certain pairs of data points, formulated as:
Minimize (Crer] & Minimize (x),)))E [Sim(zi, 2y)1 ((x,)))ex [Sim(zi,2] 25)]],
(3)
where X represents the set of data pairs that need to be encoded as similar, whereas X denotes the set of pairs that should be encoded as dissimilar, And 2; denotes the embedding of T.-
Notably, unlike the contrastive loss, X represents positive pairs within the raw data, rather than positive pairs between the augmented views. means LeT focuses solely on the relative semantic relationships among the data points, disregarding their absolute semantic.
Definition 3.2 (Encoder Dim(.)) Dim(.) refers to a series of dimension-speciis encoders that di- rectly take the data of all samples under specific dimension i, XT and oulput the embedding of that dimension, DimEmb, formalized as DimEmb, = Encode(X (XT,). DimEmb E D
Definition 3.3 (Encoder Fea(.)) Fea(.) refers to series of sample-specific encoders that take the data of all dimensions of a specific sample j. x,, and output the embedding of that sample, Fea Embg. formalized as FeaEmb; = Emcode(X,;, FeaEmb E R.N x where e represents additional imput.
Theorem 3.1 (PCA framework) Consider Dim(.) (-) as an encoder modeling the distribution of data across specific dimension (as in Definition 3.2). Fea(.) as an encoder modeling the relationships between different dimensions of data (as in Definition 3.3). PCA is an instantiation model for these two encoders. Assuming that data representations have equal length m, when X+ Definition 3. 3. of RT,, PCA adheres to the following form:
Minimize ([rCA(PCA(X)) Minimize (Fea (X))]-
Rigorous proof can be found in Appendix A.1 Here, LPCA - ([ mean mean(PCA(X) (PCA(X)) is interpreted as an instantiation of Crr, based on the principle of maximizing the retention of data variance. We consider the covariance function in Eq. 2 Cov(-)
implementation of Dim(.). since it essentially encodes the distribution information of a single dimension through the computation of covariance. The matrix decomposition and normalization concatenation in Eq. aare viewed as an implementation of Fea(.), as they fundamentally encode the relationships among different data dimensions. This interpretation also applies PCA methods based on Singular Value Decomposition. Thus, we summarize the framework that PCA adheres to as shown in Eq. 4
PCA framework has powerful transfer ability. From Eq. it can be inferred that PCA essentially functions as a neural network model trained to reach a global optimum. The parameters of PCA's two encoders learn to encode the covariance of dimensional data and to extract and concatenate eigenvectors to form the basis transformation matrix, respectively. Extensive researchers have vali- dated PCA's success across various fields with the same parameters [23] 24] 25], demonstrating that under the guidance of the LRT loss, the model's capacity to encode difference in data demonstrates formidable transferability. This transform capability originates from the model's 's focus on encoding the differential relationships among data rather than their absolute semantics.
PCA is capable of adapting to arbitrary Euclidean data and demonstrates strong transfer performance. However, its effectiveness generally falls short of advanced NN-based models, and it lacks the ability to encode structure information. To address the issue of node feature generality in graph pre-training models, inspired by existing work (26), we attempt to further theoretically analyze the relationship between existing graph contrast pre- nee-rrainnng models and the PCA framework. We start by analyzing the InfoNCE loss which is commonly used in contrastive learning.
Theorem 3.2 (Equivalence of Contrastive Loss and LRT) Assuming that data augmentation in contrastive learning does not alter the essential semantics of node pairs for downstream tasks, it follows:
Minimize [ingaNce] & Minimize [Lnr].
(5)
Detailed proof can be found in Appendix A.2 Here we offer an intuitive explanation: the negative sampling component in InfoNCE loss is widely acknowledged for encoding dissimilarities within

---

the data. We contend that the augmentation-basee positive sampling part, inherently encodes similar relationship between data points rather than their absolute semantics. Consider pair of data points, and zj, with the same semantics. Ideally, when data augmentation does not alter the essential semantics of the nodes, the augmented data, Aug(a;) and Aug(z;), should be identical. Consequently, Maximize [Sim(r;, Aug(z;)) + Sim(ry, Aug(z;))) essentially equates to Maximize [Sim(z;,,g) (z)). Thus, data augmentation is essentially looking for positive samples in the dataset, aligning with the views expressed in work [27]. Furthermore, this view explains why augmentation less critical in graph contrastive learning compared to other domains 228] leading to the proposal of many augmentation-fiet graph contrastive studies (29) 30 311, as the edges in graph data (especially in homophilic graphs) inherently convey similarities between data points.
Corollary 3.2.1 Consider a specific instantiation of dimension encoder defined as Dim( X, and the contrastive model viewed as an instantiation of an Fea(-) = ContrastiveModel() Under this the contrastive model also adheres the frumework in Theorem 3. which states:
Minimize (Lngonce(Conrrasii (ConrastiveModee(() (X)) x Minimize (Fea(FeeDim
(6)
From this, we can conclude that the PCA optimization objective is, in fact, special case of the CL loss, detailed proofs are in Appendix A.2 As demonstrated by Theorem [3. and Corollary [.2.1] PCA and contrastive pre-training fundamentally follow the same framework. Our analysis extends the existing discussions [26] on the relationship between the two. Within this framework, we can more clearly analyze why PCA is versatile across various data, while advanced contrastive pre-training models generally outperform PCA when directly used as embeddings in downstream tasks.
The robust versatility of PCA across various types of Euclidean data is at- tributed to its inclusion of the Dim (-) compo- nent. As illustrated in Table PCA, in contrast to contrastive learning models, incorporates the Dim(-) component. Most existing neural net- work layers encode features individual data points rather than encoding across the same di- mension for different data. Due to these layers' fixed input-output dimensionality requirements, they cannot universally adapt to data with var-
Answer 1.
Table 1: Comparative Analysis of Model Com- ponents in PCA, Contrastive Pre-training Models (CL), and the proposed FUG
Dim(-)
Fea(-)
X
X
PCA CL.
V
x
v
X
/
V
FUG (Ours)
V
/
/
ied feature forms except using sliding window strategy. In contrast, PCA encodes different data within the same dimension by computing covariance between dimensions, inherently encoding the data distribution of that dimension. This enables PCA to generate transform matrix and integrate data into embeddings within a unified shape.
The superior performance of contrastive pre-training models over PCA primarily stems from their additional encoding of data homogeneity and a more flexible Fea(.) component. As noted in [ key advantage of contrastive learning (CL) compared to PCA is its consideration of homogeneity in the optimization objective, rather than merely encoding differences between data points. Furthermore, thanks to variety of neural network layers, these models are capable of extracting richer data information and are adaptable to -uccliddd graph data.
Answer 2.
Ceature-Universar Graph Contrastive Pre-training. Based on the above analysis, we propose that a Feature- Universal Graph Contrastive Pre-training model should include comeenion--spccfic data distribution encoder Dim(-), a cross dooss-dimesionall data information encoder Fea(-), and an optimization objective LTT that encodes both similarities and differences between data. Models designed under this theory can universally adapt to graph data with any node feature shapes. Moreover, guided by LRT, the model captures the relative semantic relationships among data, rather than absolute semantics. This type of knowledge, which is more easily transferable, ensures that the model can generalize across different datasets.
Feature-Universal Graph Pre-training Model
Guided by the theories presented in Section 3 we instantiated a Univers- Graph pre-training model (FUG model), as depicted in Fig. 2 This model incorporates a Dimension Encoding component
DE(-), Graph Encoder GE(-), and relative relationship optimization task LRT-FUG The DE(-)
5

---

Graph Encoding
LRT-FUG
X Rexd
H E B
GE(-)
Zougeer Alganeess
DE [[Mean()) 0
DE(-)
T Raxk
8' E R
Meen Origin
Monne () RE Chitual Unaieesdly
Heprocebiian
Dimensional Encoding
Figure 2: The overview of the proposed FUG model. the node features X and structure A of a graph, X is first processed through a learnable dimensional encoding component, DB()), to generate basis transformation matrix T, which used to embed features into a unified shape H = XT E R"* H and A are then input into the graph encoder GE(-) to produce representations Z. We set three losses, LUE, LRIFUG+ and LRT-FUG, which collaboratively train GE(-) and DE(-).
learns the data distribution within specific dimension generate basis transformation matrix T, which converts any form of node features to a unified shape. The GE(-) encodes both the structure and the node features to obtain graph representations. FRTFUG is improved contrastive loss based on the Le concept. Compared to the classical InfoNCE loss, LRT-FUG more effectively guides the model in encoding the relative semantic information in the data. Furthermore, in line with the earlier discussion on augmentation essentially serving as positive sampling, we explicitly define positive sampling within CRI-FUG replace data augmentation.
Dimension Encoder DE We designed a non-linear dimension encoder, DE(-), ). to encode the distribution information of specific dimension. If it encodes the features of all nodes under one dimension, with the input dimension fixed at the number of nodes in the training set n,, this encoder would be constrained by the number of input nodes. Therefore, we randomly sample n. nodes from the training set as the input for this encoder, approximating the feature distribution of the entire data through the sampled nodes. The dimension distribution information, after being processed by DE(-), generates a basis transformation vector for each dimension as follows:
DE(X ,) Norm (ML((X, }}
where X = Sample(X) and MLP represents a Multi-Layer Perceptron. X represents the i-th column of the X matrix, that is, all the data under the i-th feature dimension. Eq. can be seen as (E(-) assigning a point to each feature dimension in hyperspherical embedding space which the center the origin and radius one. The basis transformation vectors should maximally reflect the differences among the feature dimensions while retaining inter-dimension correlations. In terms of the embedding space, this means the base vectors should be uniformly distributed over the hypersphere globally. This implies that the mean of the base vectors should approximate the origin. Therefore, we designed separate loss for DE(.) as follows:
d Σ 44
CDE(T) 1
(8)
1=1
where represents the number of dimensions, T denotes the basis transformation matrix. Eq. [ eensrres the global uniformity of the basis transformation vectors, while the local similarity or dissimilarity is controlled by the dimension's 's inherent information and the overall model's loss.
we implement our graph encoder using GNNs, which effectively encode the feature and structural information of graphs. We use H XT as the node feature embedding input to GE(-). This approach standardizes any shape of node features to the same dimension, making GE(-) generally applicable across various datasets.
Graph Encoder GE(.). Following numerous existing works 5 32 33
Relative Relationship Optimization Task LRTFUG- Based on the analysis of Car in Section 3 the LRT.FG loss should divided into positive and negative parts, PRIFUG+ and ERTFUG- For negative part, although the InfoNCE loss has demonstrated strong performance in prior studies g 5 it faces challenges with false negative sampling [34 355 and high computational complexity (which is (n?)) Appendix B.I. Moreover, existing studies have shown that InfoNCE-basce loss functions face

---

conflicts in graph data, making the model gradient unable to descend [36]. Thus, we propose new type of loss based on the analysis Ląr. Ideally, representations should be able to be divided into multiple clusters based on their intrinsic semantics, with nodes within the same cluster close together (positive pairs) and nodes between clusters far apart (negative pairs). Without prior knowledg ledge accurately identifying negative pairs is diffcclt and prone to sampling bias. Therefore, we opt not to explicitly determine negative pairs but instead constrain the overall representations, as:
CRIFUG = 1 Σ Norm 0
where 2; represents the embedding of node v4- Similar to Eq. we first constrain the representations to unit sphere and then force the mean of the representations to be close to the origin. From Eq. g we reduce the complexity of the negative sampling in InfoNCE loss which is On to O(z).
For positive sampling, inspired by existing work [37 31 30 and based on the natural homophily assumption in graphs, which means connected nodes have similar semantics, we consider node neighbors as positive samples. As Eq. g and 9 both use the 12 distance, for loss balance, we also
adopt 12 loss here:
PRI-FUG
Σ (PgM))EE
1 - -
(10)
E
where & represents the set of edges. Note that Eq focuses encoding global differences, while Eq. [0 focuses on encoding local homogeneity, complementing each other without conflict. The relationship between of RT and LRI-FUG is in Appendix A.3 The final loss is defined as:
LFGG = X LRTFUG + A2LRTFIGH AạCDE,
(11)
where, ^1, A2, Ag are purer-parameters which adjust the weight of the three losses. Compared to the classical InfoNCE contrastive loss, LFUG better encodes the relative relationships between nodes and offers a lower computational complexity of ( + E + d) (Please see Appendix [. for details). Notably, although the final loss involves three syper-parameters, in fact, FUG has only one additional hyper-paramete compared to classical GCLs (or even fewer, as FUG does not require tuning multiple ciaieenttationrllatee parameters). Details can be found in Appendix E.3
Table 2: Cross-domain Node Classification. The performances of OFA and GraphControl reported
from 17 18 FUG-C (Cora) means FUG-C (Cora) means FUG-C model pre-trained on Cora dataset.
Method
Cora
CiteSeer
PubMed
Photo
Computers
CS
Physics
OFA 17
75.90 # 1.26
78.25 + 0.71
GraphControl
94.31
1.12
FUG-C (Cora)
82.35 2.60
67.13 2.80
84.58 0.57
92.78
1.23
86.27 # 1.60
92.78
0.66
95.41
0.42
FUG-C (CiteSeer)
82.28 4 2.67
68.53
2.43
84.57 a 0.35
92.89 + 1.06
85.76 : 1.59
92.62 + 0.80
95.41
1: 0.41
FUG (PuhMed)
83.31 2.72
1.49
87.70
0.72 92.65
95.20 0.39
FUG-C (Photo)
83.35 1.94
2.09
92.93 0.43
88.01 1.38
92.44 0.69
95.25 2 0.41
FUG-C C (Computers)
83.31 2.25
69.01
1.79
84.37 4 0.47
93.11 2 1.33
88.23
1.31
92.50 1 0.65
95.58 à 0.40
FUG-C (CS)
8.37
2.64
91.96 1.11
83.28
1.55
92.54 + 0.78
95.34
0.37
FUG-C C (Physics)
60.85
2.222
84.26
0.61
84,59 1.80
73.43
1: 2.08
92.13 1: 0.66
95.35 à: 0.42
FUG (Rehuild)
84.45 2.45
7
85.47 + 1.13
93.07 + 0.82
88.42 + 0.98
92.89 + 0.45
95.45 + 0.27
5 Experiments
5.1 Experiment Settings
To comprehensively evaluate the effectiveness of FUG, we selected commonly used graph datasets with diverse node feature shapes, including Cora, CiteSeer, PubMed [1]11] Photo, Computers CS, and Physics [39]. Detailed information about these datasets can be found in Appendix
38
We primarily compared four types of methods: supervised GNN model (GCN [.), classic
machine learning algorithms (PCA [16), DeepWalk [40]), advanced self-supervised contrastive models
(GRACE 5. BGRL (32) GBT [1 DGI 33 GCA 42 ProGCL (34) ). and recent graph universal
pre-training models (OFA [7.. GraphControl [8]. both self-supervised model-rebuilding tests
and cross-domain transfer tests, FUG is initially trained in a self-supervised manner. The trained
7

---

model is then frozen, and node representations are generated on the test datasets. These representations are subsequently used as input for downstream predictors. For the node classification experiments, to simulate few-shot scenario, we split the dataset into train, valid, and test sets by 10%/10%/80% We using only 10% labels. Detailed experimental settings
train a simple downstream 12 classifier can be found in Appendix E.3
35
Table 3: Intra-domain Model Re-building Node Classification. Data without standard deviation are reported from previous works 42 32). Aside from GCN, the best results are highlighted in bold, and the second-best results are underlined. OOM indicates Out-Of-Memory on RTX 3090 GPU.
Method
Cora
CiteSeer
PubMed
Photo
Computers
Physics
Raw features
57,90 3.90
57.60
79.55
80.99 1.65
75.59 1.69
89.92 +: 0.95
93.18 + 0.45
PCA
56.36 + 4.14
48.29
3.18
82.80 0.91
74.92 + 1.82
71.26 + 1.34
87.12 = 0.73
92.48 + 0.47
DeepWalk
40
75.70 a 0.00
50.50 a 0.00
80.50 a: 0.00
89.44 1: 0.00
85.68 a 0.00
84.61 +: 0.00
91.77 a 0.00
Deep Feattrreee
73.10 0.00
47.60
0.00
83.70 0.00
90.05
0.00
86,28 0.00
87.70 + 0.00
94.90 + 0.00
GRACE 5
=
70.99
85,46 = 0.54
91.93 +: 0.83
91.44 0.82
OOM
BGRL 32
81.57 a: 2.07
70.10
83.67 =: 0.84
92.34 1: 0.73
86.51 +: 1.53
92.12 + 0.63
95.42 a 0.41
GBT 41
82.45 1.97
85.69 1.22 2..16
92.44 0.48
87.46
92.39 0.78
95.07 0.23
DGI [38]
83,24
71.23
84.62 = 0.83
9
1
92.47 = 0.60
94.47 + 0.50
GCA [+2
82,83 2:29
72.06 1.91
85.69 + 0.68
92.63 + 1.12
87.78 + 0.78
92. + 0.49
OOM
ProGCL 84
82.65 1.74
72.85 : 2.17
OOM
92.87 :: 0.44
OOM
OOM
OOM
FUG (Rebuild)
84,45 2.45
72.43 2.99
85.47 + 1.13
93.07 + 0.82
88.42 + 0.98
92.89 + 0.45
95.45 a 0.27
GCN (Supervised)
82.80 + 0.00
72.00 # 0.00
84.80 + 0.00
92,42 + 0.00
86.51 + 0.00
93.03 + 0.00
95.65 + 0.00
Table 4: Ablation study of the loss functions of FUG.
Cora
CiteSeer
PubMed
Photo
Computers
CS
Physies
FUG wlo Coe
81.32 + 2.56
71.74 2.88
84.68 # L.19
57.37 = 3,38
50.55 = 1.69
91.25 # 0.75
93.77 + 0.40
FUG w/o Cresue-
83.31 + 2.85
68.44 : 2.63
85.18 101
92.80 0.71
88.29 = 1.09
92.71 # 0,53
95.18 0.29
FUG w/o Ceroist
84.30 2.88
70.48 2.47
85.24 1.05
92.89 0.82
88.27 92.74 0.86
95.29 0.11
FUG
84.45 2.45
72.43 2.92
85.47 = 1.13
93.07 0.82
88.42 0.98
92.89 a: 0.45
95.45 + 0.27
5.2 Evaluating on Cross-domain Learning
We conduct experiments to test FUG's performance in cross-domain learning, as shown in Table 2 FUG-C C represent a fixed FUG model where all hyperparameters remain constant during pre-training. This presents a more challenging scenario, as it means we cannot adjust parameters based on changes in the training datasets. In Table 2] we report the optimal accuracy of the state-of-the-art graph universal pre-training models, OFA and GraphControl, which are capable of training across datasets with different node features. These models have access richer data during pre-training than FUG, which is not fair to FUG. However, as Table[]shows FUG's ' overall performance in cross-domain learning significantly surpasses both models. This is because OFA and GraphControl employ data preprocessing methods that significantly degrade the information in the node features (OFA uses LLM to reshape node features to the same dimension, and GraphControl replaces original node features with feature similarity). This validates our views made in Section [ and Appendix [ It also demonstrates that FUG's s dimension encoder can almost losslessly adapt to graph data with various node feature shapes.
Additionally, we derived some interesting findings from Table 1). Counterintuitively, FUG- C's performances are not necessarily the best when pre-trained and tested on the same dataset. This mainly because our loss functions focus on the relative semantic relationships between nodes, allowing the encoder to learn the knowledge of relative relationships. This leads to: a) This knowledge generalizable across different datasets, so even though the datasets are from different domains, this knowledge remains the same. In other words, relative semantic relationship knowledge has weak cross-domain limitations, as stated Section B b) The learning of relative semantic knowledge primarily relies on the distributional differences between nodes and between different feature dimensions, rather than on the absolute semantics contained in the data. Therefore, it is common in Table 2 to see that FUG-C pre-trained on other datasets performs better than when pre-trained and tested on the same dataset. 2). The performances of cross-domain trained FUG-C are close to that of FUG which is trained and tested on the same datasets with tuned papper-praamterss.. In
8

---

fact, on the Photo dataset, FUG-C (Computers) outperforms FUG. This demonstrates FUG's strong cross-domain learning capability and the success of our proposed dimension encoding and relative relationship loss. It proves FUG's robust applicability to various graph data.
5.3 Evaluating In-domain Learning
In Table we designed node classification ex- periments evaluate FUG in intra-domain self- supervised scenarios. We also report the computa- tional overhead of FUG when the hidden layer fixed to 256 shown in Table 5 and the complete results are in Appendix B.2 From Table 3 and we can observe that FUG demonstrates compet- itive performance. In Table 3
Table 5: Comparison of computational cost
GRACE
BGRI.
GBT
FUG
X
CS CS VRAM
we observed that: 1). FUG outperforms GRACE across all datasets, reaffirming that the learnable DE(-) encoder FUG can almost losslessly standardize node fea-
11,960
4,482
3,324
2,482
Physics Time Physies VRAM
OOM
0.1084
0.1320
0.1047
0OM
10,278
8,002
5,190
tures to the same shape. This also supports our views in Section that both positive and negative sampling in the InfoNCE loss inherently constrain the relative relationships between nodes. Fur- thermore, it demonstrates that our loss function maintains excellent performance while reducing computational overhead. 2). FUG significantly surpasses PCA all datasets. Although PCA, FUG, and GRACE follow the same framework, PCA is limited by its inability to encode graph structure and the relative semanties between data points, leading poor performance. This validates our previous views. FUG simulates PCA's encoding process through graph contrastive learning, thereby addressing PCA's shortcomings and sigmificantly improving performance. 3). FUG generally outperforms the Serstive-sample--e GBT and BGRL, while maintaining comparable time costs as shown in Table 5 This indicates that considering negative relative relationships between data points is effective, and FUG successfully reduces the time and space overhead associated with these negative considerations match that of methods that do not consider negative samples. Additionally, we compare FUG with latest advanced in in-domain self-supervised methods 43 in Appendix E.4
5.4 Model Analysis
To validate the effective- ness of each component in FUG, we conducted ab- lation studies and param-
0.8
eter analysis, as shown
Accurace
0.7
in Table 4 and Fig. 3
0.8
From Table 4 it is ev-
ident that FUG outper-
Computon
forms other models with- out part of loss functions across all datasets, demon- strating the effectiveness of the three proposed loss functions. Furthermore, observed that FUG w/o
04
Chisker
0.3
PolMel
Plysss
PulMed
Plysks
0.2
128 258 10242048
32 64
128
Sample size
Base transformation vector size
Figure 3: papeeeearrmmter analysis of FUG.
fRTFUG+ generally achieves the second-best performance. This is primarily due to PRTFUG- con- straining global uniformity without focusing on the distance between local node pairs, preventing the model from over-encoding differences between nodes. Additionally, FUG w/o L generally exhibits the worst results. The main reason is that, LRT-FUG and LRT-FUG are effective when based on LDE, especially for datasets with high quality features. As described in Section 5.4, when LDg is absent, DE is not sufficiently trained, which leads to DE embedding almost randomly generate H, which is an important input for GE. Furthermore, without the guidance of LDg. GE only needs to extract relationships in to reduce the loss, ignoring whether contains sufficient information from X. Therefore, in FUG w/o LDE, the H input to GE is of poor quality, resulting in worst performances on some datasets. From Fig. 3 we can observe that as the number of sampled nodes and the dimensionality of the basis transformation vectors increase, the overall model performance

---

February 1, 2019, pages improves until it stabilizes at around 1024. This indicates that a limited number of sampled nodes can effectively represent the overall data distribution. Moreover, once the dimensionality of the basis transformation vectors reaches a certain threshold, it can embed dimensional information nearly losslessly, thereby unifying data features to the same shape. This further validates the design of the dimensional encoder.
Discussions and Future Works
In this paper, we explore the connection between contrastive learning and PCA, and propose the FUG strategy, which includes specific dimension encoding, dite-ddimenioonal encoding, and relative semantic relationship loss. The models instantiated under this framework achieve near-lossless unifi- cation of node attributes with different shapes without data preprocessing, making them applicable to graph data and demonstrating strong cross-domain learning capabilittis. We instantiated a FUG model and validated its effectiveness through extensive experiments. Additionally, theoretical analysis shows that the complexity of the GCL loss involving negative samples is reduced from Onn") to (n), further lowering FUG's training costs.
In future works, the FUG strategy can support research on graph prompt tuning, universal graph models, and general large models based on graph networks, enabling graph models to train directly on various data. Morcover, upon acceptance this paper, we will release a pre-trained FUG model on existing seven datasets, allowing for direct use of FUG-generated graph representations priors for further extensions. While the FUG strategy is general, the instantiated model has some limitations, such as only considering homophilic graphs in positive sampling, which may perform poorly on heterophilic graphs, and not constraining absolute semantic information in the loss function. Future work can further refine the universal graph contrastive pre- pre-training model based on the FUG strategy.
Acknowledgement
This work was supported by the National Natural Science Foundation of China [grants num- bers 62422210, 62276187, 92370111, 62372323, 62472311] and the Ant Group Research Fund (2023061517131).
References
[1] Thomas N. Kipf and Max Welling. Semi- supervise classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[2] Sebastian Vlaie, Theresia Conrad, Christian Tokarski Schnnkll,,,enele,,..ee Mika Gustafsson, Uta Dahmen, Reinhard Guthke, and Stefan Schuster. Modulediscoverer: Identification of regulatory modules in protein-protein interaction networks. Scientific reports, 8(1):433, 2018
[3] Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. Session- based recommendation with graph neural networks. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 346-353. AAAI Press, 2019.
[4] Yao Zhang, Yun Xiong, Yun Ye, Tengfei Liu, Weiqiang Wang, Yangyong Zhu, and Philip S Yu. Seal: Learning heuristics for community detection with generative adversarial networks. In KDD '20: The 26th ACM SIGKDD Conference Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 100--111,, ACM, 2020.
[5] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. CoRR, 0is/2006604131.. 2020.
[6] Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. GCC: graph contrastive coding for graph neural network pre-training. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash, editors, KDD '20: The 26th
10

---

ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA. USA, August 23-27, 2020. pages 1150-1160. ACM, 2020.
[7] Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. GPT-GNN: gen- erative pre-training of graph neural networks. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash, editors, KDD 20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 1857-1867.
ACM, 2020.
[8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 1 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 2020.
[9] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics, 2019.
[10] Shigeki Karita, Xiaofei Wang, Shinji Watanabe, Takenori Yoshimura, Wangyou Zhang. Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, and Ryuichi Yamamoto. A comparative study on transformer vs RNN in speech applications. In IEEE Automatic Speech Recognition and Understanding Workshop. ASRU 2019, Singapore, December 14-18, 2019, pages 446- IEEE, 2019.
[11] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Gallagher, and Tina Eliassi- Rad. Collective classification in network data. Al Mag.. -933))93-106, 2008.
[12] Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. Revisiting semi- superviid learning with graph embeddings. Floriaa Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 40 JMLR.org. 2016.
[13] William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning large graphs. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, Long Beach, CA, USA, pages 1024-1034, 2017.
[14] Xiao Wang, Deyu Bo, Chuan Shi, Shaohua Fan, Yanfang Ye, and Philip S. Yu. survey on heterogeneous graph embedding: Methods, techniques, applications and sources. IEEE Trans. Big Data, 955):415-436 2023.
[15] Kaiwen Dong, Haitao Mao, Zhichun Guo, and Nitesh V. Chawla. Universal link predictor by in-context learning on graphs. CoRR, abs/2402.07738, 2024.
[16] Andrzej Maćkiewicz and Waldemar Ratajezak. Principal components analysis (pca). Computers Geosciences, 1933):00-332, 1993.
[17] Hao Liu, Jiarui Feng, Lecheng Kong. Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang. One for all: Towards training one graph model for all classification tasks. CoRR, abs/2310.001 2023.
[18) Yun Zhu, Yaoke Wang, Haizhou Shi, Zhenshuo Zhang, Dian Jiao, and Siliang Tang. Graphcon- trol: Adding conditional control to universal graph pre-trained models for graph domain transfer learning. In Tat-Seng Chua, Chong-War Wah Ngo, Ravi Kumar, Hady W. Lauw, and Roy Ka-Wei Lee, editors, Proceedings of the ACM on Web Conference 2024. WWW 2024. Singapore, May 13-17, 2024, pages 539-550, ACM, 2024.
[19] Serkan Kiranyaz, Onur Avei, Osama Abdeljaber, Turker Ince, Moncef Gabbouj, and Daniel J Inman. 1d convolutional neural networks and applications: A survey. Mechanical systems and signal processing, 151:107398, 2021.

---

[20] Alex Graves and Alex Graves. Long short-term memory. Supervised sequence labelling with recurrent neural networks. pages 37-45, 2012.
[21] Zhenmei Shi, Jiefeng Chen, Kunyang Li, Jayaram Raghuram, Xi Wu, Yingyu Liang, and Somesh Jha. The trade-off between universality and label efficiency of representations from contrastive learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[22] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In 6th International Conference Learning Representations. ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[23] Andreas Daffertshofer, Claudine JC Lamoth, Onno G Meijer, and Peter J Beek. Pea studying coordination and variability: tutorial. Clinical biomechanics, 19(4):415 2004.
[24] Jian Yang, David Zhang, Alejandro F. Frangi, and Jing- Yu Yang. Two-dimensional PCA: new approach to uppearance-basee face representation and recognition. IEEE Trans. Pattern Anal. Mach. Intell., 26(1):131-11) 137, 2004.
[25] Daniel Granato, Jânio S Santos, Graziela B Escher, Bruno L. Ferreira, and Rubén M Maggio. Use of principal component analysis (pea) and hierarchical cluster analysis (hea) for multivari- ate association between bioactive compounds and functional properties in foods: A critical perspective. Trends in Food Science & Technology, 72:83-90, 2018.
[26] Zhenmei Shi, Jiefeng Chen, Kunyang Li, Jayaram Raghuram, Xi Wu, Yingyu Liang, and Somesh Jha. The trade-off between universality and label efficiency of representations from contrastive learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[27] Benoit Dufumier, Carlo Alberto Barbano, Robin Louiset, Edouard Duchesnay, and Pietro Gori. Integrating prior knowledge in contrastive learning with kernel. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 8851-8878. PMLR, 2023.
[28] Wen-Zhi Li, Chang-Dong Wang, Hui Xiong, and Jian- Huang Lai. Homogel: Rethinking homophily in graph contrastive learning. In Ambuj K. Singh, Yizhou Sun, Leman Akoglu, Dim- itrios Gunopulos, Xifeng Yan, Ravi Kumar, Fatma Ozean, and Jieping Ye, editors, Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6-10, 2023, pages 1341-1352 ACM, 2023.
[29] Dongxiao He, Jitao Zhao, Rui Guo, Zhiyong Feng, Di Jin, Yuxiao Huang, Zhen Wang, and Weixiong Zhang. Contrastive learning meets homophily: Two birds with one stone. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pages (2775-12789 PMLR, 23-29 Jul 2023.
[30] Haonan Wang, Jieyu Zhang, Qi Zhu, and Wei Huang. Augmentation-free graph contrastive learning with performance guarantee. arXiv preprint 0rXiv:2204.04874, 2022.
[31] Namkyeong Lee, Junseok Lee, and Chanyoung Park. Augmentation-free self-supervised learning on graphs. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Chirty-Fourthy Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 March 1, 2022, pages 7372-7380. AAAI Press, 2022.
[32] Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Rémi Munos, Petar Veličković, and Michal Valko. Bootstrapped representation learning on graphs. In ICLR 2021 Workshop Geometrical and Topological Representation Learning, 2021.
12

---

[33] Petar Velickovic, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[34] Jun Xia, Lirong Wu, Ge Wang, Jintao Chen, and Stan Z. Li. Progel: Rethinking hard negative mining in graph contrastive learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 24332-24346 PMLR, 2022.
[35] Yanqiao Zhu, Yichen Xu, Qiang Liu, and Shu Wu. An empirical study of graph contrastive learning. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks I, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021.
[36] Dongxiao He, Jitao Zhao, Cuiying Huo, Yongqi Huang. Yuxiao Huang, and Zhiyong Feng. A new mechanism for eliminating implicit confiic in graph contrastive learning. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirly-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, FAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 12340-12348. AAAI Press, 2024.
[37] Hengrui Zhang, Qitian Wu, Yu Wang, Shaofeng Zhang, Junchi Yan, and Philip S Yu. Localized contrastive learning on graphs. arXiv preprint 2.0xix::222.046000 2022.
[38] Julian J. McAAley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, Santiago, Chile, August 9-13, 2015, pages 43-52. ACM, 2015.
[39] Arnab Sinha, Zhihong Shen, Yang Song. Hao Ma, Darrin Eide, Bo-June Paul Hsu, and Kuansan Wang, An overview of microsoft academic service (MAS) and applications. In Proceedings of the 24th International Conference on World Wide Web Companion, WWW 2015, Florence, Italy, May 18-22, 2015 Companion Volume, pages 243-246. ACM, 2015.
[40] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '14, New York, NY, USA August 24- 27, 2014, pages 701-710. ACM, 2014.
[41] Piotr Bielak, Tomasz Kajdanowicz, and Nitesh V. Chawla. Graph barlow twins: A self- supervised representation learning framework for graphs. Knowl. Based Syst., 256:109631 2022.
[42] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In WWW '21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021, pages 2069-2080. ACM IW3C2, 2021.
[43] Teng Xiao, Huaisheng Zhu, Zhengyu Chen, and Suhang Wang. Simple and asymmetric graph contrastive learning without augmentations. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[44] Jiaming Zhuo Can Cui. Kun Fu, Bingxin Niu, Dongxiao He, Chuan Wang. Yuanfang Guo, Zhen Wang. Xiaochun Cao, and Liang Yang. Graph contrastive learning reimagined: Exploring universality. In Tat-Seng Chua, Chong-We Wah Ngo, Ravi Kumar, Hady W. Lauw, and Roy Ka-Wei Lee, editors, Proceedings of the ACM on Web Conference 2024, WWW 2024, Singapore, May 13-17, 2024, pages 641-651. ACM, 2024.
[45] Yue Yu, Xiao Wang, Mengmei Zhang, Nian Liu, and Chuan Shi. Provable training for graph contrastive learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36:
13

---

Anmual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[46] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural 2020.
information processing systems, 33:21271
[47] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pullern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 9726-9735. Computer Vision Foundation / IEEE, 2020.
[48] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1807.000 2018.
[49] Susheel Suresh, Pan Li, Cong Hao, and Jennifer Neville. Adversarial graph augmentation to improve graph contrastive learning. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 1990-11933,, 2021.
[50] Nian Liu, Xiao Wang, Deyu Bo, Chuan Shi, and Jian Pei. Revisiting graph contrastive learning from the perspective of graph spectrum. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[51] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901 2020.
[52] Xiangguo Sun, Hong Cheng, Jia Li, Bo Liu, and Jihong Guan. All in one: Multi-task prompting for graph neural networks. In Ambuj K. Singh, Yizhou Sun, Leman Akoglu, Dimitrios Gunopu- los, Xifeng Yan, Ravi Kumar, Fatma Ozcan, and Jieping Ye, editors, Proceedings of the 29th
ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, ACM, 2023.
CA, USA, August 6- 10, 2023, pages
[53] Yizhen Zheng, Shirui Pan, Vincent Lee, Yu Zheng, and Philip S Yu. Rethinking and scaling up graph contrastive learning: An extremely efficient approach with group discrimination. Advances in Neural Information Processing Systems, 35: 15:10809-10820, 2022.
14

---

A Proofs
A.1 Proof for Theorem 3.1
The proof of Theorem 3.1 is divided into two parts: framework equivalence and objective equivalence.
We first present the framework equivalence part, where PCA can be decomposed into the Fea(-) and Dim(.) components.
Proof:
Given an input data X with standardized matrix X X Mean(X), PCA is as follows.
PCA(X) = XP, Norm(Concal((llcs E C, 2 (()) (A, C) = 8(Coo(X))
(12)
where (A, C.) represents the eigenvalues and eigenvectors matrix, Norm(.) represents normalization, Concat(.) represents concatenation, (4) represents the k-th largest eigenvalue, Cov(.) represents covariance computation, and 6(.) represents eigenvalue and eigenvector computation. This can be decomposed into the forms of Dim(.) and Fea(-) as follows:
DimEmb = Encode (X) X Wi = X,
(13)
FeaEmb Encodez (X) PCA(X)= XW2, W2 P (DDimEmb)
where W1 is a special parameter matrix, W1 =x, and (p-)) represents the operation to solve W2 as:
W2 P sp Emim) (orm[Comaat(alle E C,A. J), 0) FmDm)
(14)
From the above, Encoder() (-) and Encode2(-) satisfy linear or nonlinear mappings. So, we derive:
PCA(X) Fea(X)- F XW2 = XX(Dim(XX) Fea(X, Dim(()))
(15)
Thus, PCA can be rigorously decomposed into Dim(.) and Fea(-) as described Theorem 3.1.
Then, prove that MinimizeCPcA MinimizeCgr
Proof:
We start with the loss function of PCA, which maximizes the variance:
Minimize [PCC)] =
Minimize
E PCA((z)) (eCan(PCA(()])))))
(16)
Minimize E8 24
(2)
where denotes the PCA embeddings. To obtain form corresponding Eq. 3 we consider an arbitrary pair of data points 24 and zy, as shown below::
21
+
22
24 +
2j
4
2n
Minimize [C 24)] = Minimize [-|]
2 2j +
Minimize
|-1|
n
11 [-1]
Minimize
2;
2j + d
n
(17)
Where represents the sum of the representations of all other nodes. From Eq. 2
z = XP and
Mean(X) 0. So Mean(XP) Z = 0. Therefore, approximate to 0 here. The following
15

---

equation can be derived:
n 1
1
Minimize LPCA(Z, 2j) Minimize
[-|]
2j +
cl2
n
11 1
Minimize
24
11
n2
T2
(n - 1)²
2n - 2
Minimize
||=|| 12
||
n
n2a
n²
(1 1)²
1
2n 2
Minimize
m²
m11
2;2z]]
n
n
n
(18)
Minimize
[-(
1)²
m11
2
+
1
m1
2n 2
111
| 2;2g
n
n2a
n
Minimize |-(
-
1)²
m
+
1
m
2n 2
m
Siiiimmi (2, 2,))]
n²
n1
n
(n 1)2
1
2n
2
2
m
n1
m²
+
n²
(Minimize (Sim (2) 2;)])
Minimize [Sim(=1, =j)]
here we use the assumption that all representations have the same length m. When X+ =, Eq. can have the following form:
Minimize [Crrl Minimize (x.,a)ex [Sim 2,)] (2,))) (Sim(z,
Minimize (x.,))e (Sim 2,)]]
(19)
Minimize [Crcal
we assume that all representations have the same length m. This assumption is reasonable because in many deep learning scenarios, normalizing is common practice, specifically normalizing the length of the embeddings to 1.
Assumption Analysis. Theorem 3.1
A.2 Proofs for Theorem3.2 and Corollary 3.2.1
Similar to Appendix A. the instantiation part of the theorem has already been clearly explained in the main text. We consider Dim(XX) X as a special form of dimensional encoding instantiation, given that CL essentially lacks dimensional encoding component. We view the neural network in CL the instantiation of Fea(-). Here, we prove that Minimize [Ccu. x Minimiaze [Cnr].
Proof: First, we define set X representing the positive pairs in contrastive learning. (x,, az)) augmented data points derived from the same original data point.
X means that and z, Then we have:
e Sim(2..25)/r
LinfONCE = = -o
(20)
Exns))ex (im(aa,,))/T + E(Ere)EE Sim{
where z; means the embedding of data point vy. Considering positive pair (va; 2 E X we can obtain:
Sim(2o xt// + Cp
Minimize (LuttoNC xx]]
Minimize [-
log (im(=a 1/5 + Cn
Cn @Sim(znstd) 1/7 Cp + Cn
Minimize [-
log
(21)
On
log
Maximize
(e Sim(za.ad 1/7 | + Cp +Cn
On
log
Maximize [Sim] (a 11/T Cp + Cn
x Maximize [Sim(za: =]
16

---

where Cp: On are constants Considering a negative pair (T) E _ we can obtain:
Minimize [CIntoNCE J.
Minimize log
eSimlza 1/7 + Cp + Gn
Cp
log Minimize (eSim(ZasAu 1/7 + +Cp t +Cn
(22)
Cp
log
Minimize [Sim 11/T + Cp + cn
x Minimize Sim(za: 25
So, have:
Minimize [CinfoNCE] & Minimize
Σ (x)1)))E
Sim(z 2j)] & Maximize
Sim 24)]
(RAR))EXX
& Minimize
Σ
Sim(zn)a)
Σ
Sim(z4,
(EX
(mLa))eXT
(23)
Here get the conclusion that minimizing the contrastive InfoNCE loss is maximizing the similarity between positive pairs and minimizing the similarity between negative pairs. Based on our assumption, data augmentation does not alter the essential semantics of the data. We define H the data stripped of all irrelevant and redundant information by data augmentation, containing only the essential semantics. For data point Ta maximizing E (x,,,,)) Sim 2y) ) equivalent to maximizing (Exx)))xx; Sim(za,ha), For downstream tasks, data of the same category embodies the same essential semantics. Therefore, for set 'D of data points have similar essential semantics, their h values are identical. For any two data points Faz Th E Di1, we have:
Minimize [CIatNNC]] & Minimize
Σ
Sim 22)
Sim (2, 2,
(xx,,))E--
IIED)
Minimize
Σ (EX-
Sim(zi, zj)
Sim ha)
k= 2,ED
(24)
Minimize
Σ (xI,T))EX
Sim(z;; 23) )
Σ
Σ
Sim 24 25)1
kml #iiaED
Minimize
Σ
Sim(zi, zz)
Σ
Sim(z; (2, 2,)
(Ex
Ex,
Minimize [CRT]
Comparing equations 19 and can conclude that the PCA optimization objective is, in fact, special case of the contrastive loss when X = V.
Assumption Analysis. We assume that data augmentation does not alter the essential semantics of the data, which an idealized condition. Many works in contrastive learning aim to achieve this through augmentation. Therefore, this assumption is reasonable. Additionally, this assumption is made for the sake clarity. Under this assumption, the set is correctly defined, meaning all data within the set share the same essential semantics. If the assumption does not hold, the essential semantics within the set D may not be identical across all nodes. However, this does not affect the correctness of the proof, as the
still exists.
A.3 The Relationship Between the RT-FUG Loss and RT Loss
In this section, we discuss the relationship between LRT.FU and LRT-
Theorem A.1 Assuming the representations have the same length m, then: Minimize [LRT=FUG] Minimize [LRT]
(25)
17

---

Proof:
The FUTFUG consists of both positive and negative parts, which discuss separately. For LRTFUG+.
we have:
Minimize [ARTFUG+]
Minimize
Σ
|| - =,ا1
(N.,)))x*
Minimize
Σ (v.,e))EX+
(2) 23 - -2242
Minimize
Σ
(2m² 2m²
2/ )
1|2/|||122512
(oLo))EXt
(26)
Minimize
Σ
(2m² 2m Som'Sim(za) (2,,
(n),,))EX+
2n - Maximize
Σ
(Sim 2y)))
(N..O))EX
x Minimize
|-
Sim(z 2y)1
(EL+))EX+
For CRT-FUG- take any two nodes v, by then we have:
Minimize [CRT-FUG (vi, v;)]
Minimize
0
71
Minimize
|||(( m
1
3j
+ c)
0
m
[1-((-) 2i
3j + e
Minimize
(27)
here, we also use similar trick as in Appendix A. treat the constant vector O:
Minimize [CRT-FUG (o%, uy)
Minimize
[||
+
2j
+
c)
n
m
m
Minimize
[||
|||-(a)
1
2i
+
3j
>
m
Minimize
|| + 2511
Minimize
(2, + 2) + 22424)
7177
Minimize
(2m² 2212,)]
(28)
nm
2:25 }]
Minimize
(2m² + 2m || [zi|||||=|11
Minimize
(2m + 2m Sim(z;, 2;)
1 (2m² + 2m Minimize Sim(2; 2,)))
nm
x Minimize (2i,
Hence, from Eq. 28 and 26
we can obtain:
Minimize [RRT-FG
Minimize
[REFUG+ CRTFUC-]
Minimize
[RTFFG+ + Minimize [CRTFUG]]
Minimize
CRT.FUG+ Minimize [CRT.FUG-]
(29)
8 Minimize
[x [Sim(z, 27)] -(x+,,) [Sim(2,, 2;)]]
8 Minimize [[rr]
18

---

Table 6: Comparison of Computational Cost (Completed)
GRACE
BGRL
GBT
FUG
X-
X
Cora Time
0.0106
0.0055
0.0095
0.0091
Cora VRAM
680
532
526
674
CiteSeer Time
0.0153
0.0075
0.0126
0.0148
CiteSeer VRAM
910
788
658
848
PubMed Time
0.2154
0.0201
0.0257
0.0165
PubMed VRAM
11,610
1,254
1,432
1,340
Photo Time 0.0552
0.0225
0,0270
0.0220
Photo VRAM
3,692
1,336
2,158
1,962
Computers Time
0.1397
0.0440
0.0492
0.0407
Computers VRAM
8,952
2,316
2,570
6,418
CS Time CS VRAM
0.2169
0.0462
0.0593
0.0526
11,960
4,482
3,324
2,482
Physics Time
OOM
0.1084
0.1320
0. 001477
Physics VRAM
OOM
10,278
8,002
5,190
B Analysis of Computational Costs
B.1 Computational Complexity
RT-FUG Loss. As shown in Eq. 9 CRI-FUG in the negative sampling part only requires calculating the mean of all node representations after normalization and computing the distance to the origin. In fact, since Mean(-) - = Mean(-), we only need to perform operations directly on the mean. Thus, its computational complexity is (n). The negative sampling part of the InfoNCE loss is well-known for its high complexity, as it must compute the similarity between every pair of nodes, resulting in O(n²) complexity. Therefore, state in the main paper that LRT.FUG, an optimized contrastive loss, reduces the original time complexity from (n²) to O(n).
As shownin Eq. TO for the positive sampling part, we discard the costly data augmentation and, guided by Theorem 3.2, explicitly define neighbors as positive samples. Hence, we only need to compute the distance for the two points linked by edge. Its time complexity is O(e), where is the number of edges. Therefore, the overall complexity of the RT-FUG loss O(n + e).
DE Loss. Similar to LRTFUG we compute the mean of the basis transformation matrix for each dimension and its distance the origin. Thus, its time complexity O(d).
Overall Time Complexity. From the above, the overall time complexity of the loss is O + e + d). Furthermore, considering d is the number of dimensions of node features, it is negligible compared to the number of nodes n when the data is large enough. Additionally, even if the number of nodes is very large, the average degree of nodes is generally small. Assuming the average node degree is fixed constant a, the overall time complexity becomes O((2a + 1)n), which can be regarded as Onn). For accuracy, we still state the complexity of LFUG O(n e + d) in the main paper.
B.2 Computational overhead experiments
In this section, we present the complete computational cost experiments reported in Table 3 of the main text, as shown in Table The implementation details of the comparative methods are provided in Appendix E.2 To eliminate the influence of hidden layer size on model comparison, we fixed the hidden layer to 256 dimensions and reported the memory usage and computational cost per epoch. As observed in the table, FUG, which considers negative samples, has time and space complexity comparable to methods that only consider positive samples. Furthermore, FUG's advantages become more pronounced as the number of nodes in the dataset increases. As stated in Appendix B. when
19

---

the dataset is large enough, the time complexity of FUG is O(n) rather than O(n + c d). Therefore, FUG performs comprehensively best on the Physics dataset. However, on smaller datasets like Cora and CiteSeer, FUG's advantage is not evident because the feature dimensions are large relative to the number of nodes. Table further demonstrates FUG's superiority computational efficiency.
C Algorithm
To further illustrate how FUG works, we provide pseudocode shown in Algorithm |
Algorithm 1 FUG Training
Input: Data (X((s) A | (x())) ) G(o), g 9= 1,2, ..., 9n
Dimensional encoder DEg(-); Graph encoder GE
Hyper-parameters: Learning rate Loss weights \1, X2, Output: Trained DE and (-)(
1: for Number of training epochs do
2: for 1, ..... do
3: 1 Generate basis transformation matrix */
4:
(()) Sample(X((s)
5:
T = Norm(DE, (x ))
6:
H X(9)T
7:
GE A(g))
8:
Calculate loss $/
9:
CRIFUG Meea (Norm
10:
CRTFUG -= 1|
LD = Mean
12:
LFUG A1 LRTFUG- + *2 + AgLDE
1* Update parameters $/
14:
δ + adof; 0 0 aBee
15: end for
16:
end for
17:
return DEa(-), (-)
D Related works
Graph Contrastive Learning (GCL). Graph Contrastive Learning is dedicated to learning graph representations in a self-supervised manner [35). Initially inspired by the success of contrastive learning in the visual domain 88 46 47), approaches such as GRACE [5] were proposed. These methods generate augmented views by randomly perturbing graph data, treating nodes augmented from the same original node positive samples, and other nodes negative samples. They employ the InfoNCE loss [48] to maximize mutual information between positive pairs while minimizing between negative pairs. Numerous improvements have been made within this framework. To address the computational burden of negative sampling, some studies [32 41 have explored training methods that do not require negative sampling. BGRL 32 utilizes two distinct encoders, Online and Target, with the Online encoder learning to replicate the Target's output for positive samples. GBT [41] proposes loss reduce redundancy node representations. To mitigate the semantic distortion caused by random graph data augmentations, GCA [42] assesses the importance of nodes based on degree centrality, increasing perturbations for less important nodes. HomoGCL (88 uses community
detection algorithms to assess the importance of edges and node features for designing augmentation.
AD-GCL (49] introduces a learnable data augmentation strategy through minimax approach. SpCo
augmentation strategy. Works like AFGRL [31], AFGCL [30]. and NeCo (29] have introduced contrastive learning algorithms that not use data augmentations.
(50 proposes graph spectral-
To resolve sampling biases in positive and negative sampling, LocalGCI. 337 and NeCo [2] suggest
using neighbors as positive samples. ProGCI. (34 utilizes Beta distribution fitting approach to
select negative samples. While these methods have shown great success, they all use GNNs
22
20

---

as encoders, meaning the trained models are not applicable to other datasets with different forms of node features.
Unified Graph Pre-training Models Inspired by the recent success unified large language models [5], numerous studies have attempted to propose unified, generic graph pre-training models for graph data. All-In-One [52] employs transferable graph prompts, allowing graph pre-training models be applicable across various tasks, though it still cannot handle graphs with diverse features and relies on methods like PCA to standardize node feature dimensions first. OFA [7 introduces pre-training and fine-tuning pipeline that transforms all downstream tasks into link prediction tasks, with the model itself lacking adaptability to arbitrary node features. It uses large language models to convert node features into a uniform format. Classic approaches like GCC [6] and recent methods like UniLink [15] discard original node features in favor of encoding new topology- based positional attributes, allowing application across diverse data but resulting in significant information loss. Very recently, GraphControl [18] converts distances between node features into matrix similar adjacency matrices, utilizing the universality of GNNs for arbitrary topologies achieve attribute universality. However, this approach, which only encodes distances between node features, still significantly harms the semantic integrity of the original node features.
Although these methods have extended the applicability of existing GNNs, they still lead to consider- able loss of node feature information during preprocessing. contrast, our FUG retains the semantic integrity of node features while ensuring the universality of the graph pre- rre-training model.
E Experimental Supplement
E.1
Datasets
We follow many prior works 5
42
32
33
and evaluated our performance on seven widely used
public datasets:
Nodes represent scientific papers, edges represent citation relationships, node features are bag-of- words vectors of papers, and labels represent domains papers.
Cora, CiteSeer, and PubMed are the citation datasets from PlantolD ||| 12
Photo and Computers are co-purchase graphs from Amazon 388. Nodes represent products, edges represent frequent co-purchase relationships, node features bag-of-words vectors of product reviews, and labels represent product categories.
CS and Physics are from the Microsoft Academic Graph searchers, the edges represent paper oo-authorship relationships, the node features are the bag-of-wo log-oof-ordds vectors of the authors' papers, and the labels represent the research fields of the authors.
39 The nodes represent re-
The statistical information presented Tab. We follow the exist work 5 35}, utilized the Python library "torch crometric.dataset to load the datasets. Bidirectional links were established for all edges. For all the data, we divided the training, validation, and test data sets with 10%, 10%, and 80%, to simulate the scenario with fewer tags downstream.
Table 7: Dataset statistics
Name
Nodes
Edges
Features
Classes
Cora
2,708
4,732
1,433
7
CiteSeer
3,327
5,429
3,703
6
PubMed
19,717
44,338
500
3
Photo
7,650
119,081
745
8
Computers
13,752
245,861
767
10
CS
18,333
81,894
6,805
15
Physics
34,493
247,962
8,415
5
21

---

E.2 Compared Methods
In this section, we detail the implementations of the methods we compare against. For supervised GCN [1 as well the classic Deepwalk and Deepwallk+Fettures [40] methods, directly report the accuracy from [ 42 [2] for comparison. For the "Raw features" method in T we directly
use the node features from the dataset embeddings for the downstream classifier. For the PCA [I6] in T we use node features as PCA input, reduce the dimensions uniformly to 100, and then feed them into the downstream classifier. For GRACE [5) GCA 42),
DGI (33, and ProGCL (34, we use the official code for implementation. To ensure a fair comparison, we replace the downstream classifiers with unified classifier (details Appendix E.3 For BGRL, since the official code is entirely based on "PyTorch" while other methods are based on "torch "erree_gemmetrc" or "DGL" libraries, we use the implementation from BT for fair comparison. For GBT [4I, since the official code is based on Docker, we use the implementation from (35 for fair comparison. For the graph pre-training models OFA [17] and GraphControl [18), due their use of large language models or massive datasets for training, we were unable to reproduce the models due to computational constraints. Therefore, we directly report the best accuracies from their respective papers.
E.3
Detail Settings
Training, Embedding and Testing. For all experiments, we first train our models in unsupervised manner on the training dataset, and then generate representations on the test dataset which are fed
into a downstream classifier. Following the prior work (35), we set up a simple /2 classifier, training it with only 10% of the data and validating as well as testing with the remaining 90%. Moreover, for
edge-sparse datasets like Cora and CiteSeer, adhere to the method proposed in [53] by further
enhancing the similarity relations between connected nodes through topological propagation during embedding.
Evaluation Methods. The number of propagation layers is denoted by "num hop". All reported results are averaged over 20 runs with different splits, providing mean and variance. The mean and standard deviation are directly calculated using "torch.mean" and "torch.std", respectively.
Installation of FUG. For FUG in all scenarios, we use a two-layer GCN as the graph encoder. We chose Adam as the Optimizer, the learning rate is set to 0.00001, the weight decay is set to 0.00001, and PReLU is selected as the activation function. The dimension encoder is MLP, (PReLLLLLLLL (-))) The number of nodes sampled and the dimension of the basis transfor- mation vector are both 1024. In addition, in order to reduce the fluctuation caused by random, directly select the first 1024 nodes as the sampling nodes for all datasets. The random seed is fixed to "66666" in all scenarios. All experiments were conducted on a device equipped with Intel 12400 CPU and an NVIDIA RTX 3090 GPU. For FUG-C in Table 2 we set the same shown in Table 8
as The hyper-parameters for FUG (Rebuild) Table Jar depicted in Table g
The Selection of Hyper-parameters. Although Equation II includes three tlper-parametern in fact, tuning hyper-parameters of FUG is not complex. Because the FUG model has only one additional hyper-parameter compared to classical GCLs (or even fewer, as FUG does not require tuning multiple augmentatiion-reltte parameters). Specifically, (1) in Equation [1] x is fixed at 1, requiring only two
to be tuned, while classic GCLs require tuning the temperature parameter т. Therefore, FUG has just one more hyper-pann papee-prreettrr to be tuned. (2) As shown in Table 9 (in the appendix), the search spaces for X2 and Xg are small, specifically [0.1, 0.3, 0.5] and [20, 50, 100, 200, 300], respectively. They exhibit clear unimodality in sensitive analysis in Figure making the tuning process straightforward. (3) As shown in Table 2 and Figure 4, the model can achieve good results without meticulous tuning, demonstrating the robustness of FUG.
Table 8: Hyper of FUG-Cin in cross-domain learning
#Epoch
#Hidden Units
A
X2
FUG-C
500
1,024
1
0.5
200
22

---

Table 9: Hyper-parameters of FUG in in-domain learning
#Epoch
#Hidden Units
A2
Num Hop
Cora
300
512
1
0.3
100
CiteSeer
500
512
1
0.5
50
PubMed
700
2,048
1
0.1
20
Photo
500
1,024
1
0.5
200
Computers
500
1,024
1
0.1
300
CS
300
1,024
1
0.1
200
Physics
300
1,024
1
0.1
200
E,4 Further Comparing FUG with Recent Advanced in-domain Methods
To further showcase FUG's performance in-domain scenarios, we compare FUG with the latest state-of-the-ar algorithms, shown in Table TO The results for GraphACL 43 and ROSEN 44 are obtained using the same experimental settings as described above. For POT with similar experimental settings to ours, directly report its best performances as stated in its original paper. As shown in Table [O] even intra-domain scenarios (which are not our primary focus), FUG demonstrates competitive performance, achieving the best results three out of five datasets. This highlights the effectiveness of FUG and supports the claims made in our paper.
45 a plug-in method
Table 10: Additional Intra-domain Model Re-building Node Classification.
Method
Cora
CiteSeer
PubMed
Photo
Computers
GraphACL 43
83.40 + 2.12
73.37 # 1.77
85.07 # 0.62
92.93 # 0.79
89.50 # 0.31
ROSEN 44
83.35 # 1.77
73.01 + 2.01
92.82 = 0.72
88.87 = 0.42
POT 45
80.50 = 0.90
69.00 + 0.80
92.40 # 0.30
89.10 + 0.30
FUG (Rebuild)
84.45 2.45
72.43 2.92
85.47 1.13
93.07 = 0.82
88.42 + 0.98
E,5
Hyper-Parameters Analysis
To fully illustrate the interactions among the three losses included in FUG, report the analysis experiments on the weight parameters Y₁ X2, and Xg across all datasets, as shown in Figure
Limitations
Here, we describe the limitations of the proposed FUG model. The FUG model serves as straight- forward implementation validate our proposed FUG strategy and has some imperfections. For example, the loss LREFUG+ FUG model is based on assumption graph homophily assumption, which leads to good performance homophily datasets. However, it might face challenges on heterophilic datasets because neighboring nodes in such datasets often have different semantics. Additionally, the encoder used in FUG is GCN, which also operates under the graph homogeneity assumption. Furthermore, the GCN encoder we use can only handle weighted ordinary edges and homogeneous nodes. Therefore, when dealing with heterogeneous, graphs, where node types within the same dataset differ, the model still faces applicability challenges. These issues can addressed by following the FUG strategy to construct new instantiated models, which is direction aim to explore in future works.
G Border Impacts
Here, we discuss the social border impacts related to the FUG strategy and model. a foundational study in graph pre-training models, FUG itself does not have direct Border Impacts. Furthermore, all datasets used for training the FUG model are public, eliminating privacy concerns. Given the broad applications of Graph Neural Networks (GNNs), further research on FUG could potentially advance
23

---

0.00
0.00
6.2
810
35
Lambbde
Cors
676
20
55
103
200
490
Coca
800
070
03
1.mbria7
Cor
Lambda
0.70
0.74
0.20
0,72
6.70
2000
0:0
Clicserr
CiteSoer
Chicser
Limbdde
12
20
53
133
200
403
800
02 Limbrda
Lumbda
1.505
0.90
2 1.90
0.00
2.000
03c0
PakMec
0a2
PubMel
015
PutMel
Lumbila
0.00
15
20
50
1E3
200
403
800
0.2
Limbia
1.50
1.40
1.00
0,000
1.005
Photo
244
Photo
1.90
1903
Photo
12
Lambae 1
17
20
53
193
200
493
800
Lambba
Lambda
1.00
1.0
0.00
6.75
0.7
0.70
Computere
Computers
Comguters
Lanbaaa
10
100
230
400
890
Lanbala 5
Lambia
0.234
0.34
0.90
6.00
1.80
0.2
6322
6528
Landdla
10
Limbia
100
233
400
890
Lantla
0.50
1900
2334
0.0
0.54
1352
682
1345
054
Prycce
Physics
2300
Physio
Lendda
20
13
193
200
Lanbda
Figure 4: Parameter Analysis of \1, X2 and 0
24

---

multiple fields and serve as foundational research for large graph-based based models. While FUG, like any neural network, might face societal risks, these risks are too remote for preliminary theoretical study like this and should be addressed in subsequent research.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We simulate the PCA process by utilizing graph contrastive learning, thus solving the proposed problem that graph pre-training models cannot be directly generalized to graphs with different node feature shapes. We also improve the contrastive loss to make pre-training faster.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have stated the limitations of the proposed FUG model in the last section of the main paper. In addition, we set up separate section the appendix to state the limitations of the proposed FUG theory and the instantiated model.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
26

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide instructions for each of our conclusions in the main paper and formal proofs in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide full explanations and implementation details for the proposed FUG strategy and the instantiated FUG model in the main paper and in the appendix. And we provide the core code as supplementary material in the submission. All datasets required for the experiments are public datasets. In addition, we will release the source code after the paper is accepted.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
27

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the source code as supplementary material in the submission. The source code will be released after the paper is accepted, and will also release FUG model trained on the 7 datasets mentioned the paper for easily use. addition, we believe that the main paper and appendix provide sufficient experimental details to ensure the reproducibility of our model.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
poollccgguies/// Coweeuumissiiooooooo misscnyyyyy for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips.cc//// wwwpc//ccwppcp.c..e.eeee
http://eeeuummiiiioooooooooooo.e for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We present the experimental setup in the main text and provide details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
28

---

Justification: For all experiments with the FUG model, we provide the standard deviation of 20 runs of the random split describe the randomness of the model. We provide the details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
We provide the experimental platform in the main paper and appendix, includ- ing the settings hardware, dyyer-parametent and so on, as well as providing the time complexity of the model.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Yes we followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
29

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We are committed to analysing and improving the existing graph pre-training strategies and methods, and propose FUG. It does not have any negative social impact, from this view we should choose NA. But Considering the potential problems widely possessed by neural networks, we provide discussion about the social implications of FUG in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: FUG is basic research on graph pre-training models. The training datasets are all public datasets. We believe that FUG does not have this risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual- should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
30

---

Answer: [Yes]
The datasets and comparison methods involved in this paper are all publicly available. We quote them correctly in the paper.
Justification:
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: In fact, we have no new assets except our FUG model. All the libraries and datasets used are publicly available, and we referenced them as required. We have provided the source code with the submission as supplementary material. In addition, the code will made public upon acceptance of the paper. Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Crowd-related experiments are not included in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
31

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: These kinds of experiments are not included this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "lsEHGplVTT", "reviewer": "Reviewer_vBXZ", "summary": "In this paper, the authors try to answer the question: Could a graph contrastive pre-training principle based on PCA theory be developed to address inherent issues of PCA and broadly apply to node features of different shapes? Then the authors start with some theoretical analysis and then propose a method namely FUG, which has good transferability. The method is validated on several datasets to test the performance on cross-domain and inter-domain tasks.", "questions": "-   What is the connection between the proposed FUG and graph prompt tuning? Which one is more practical in practice?\n-   Why does CE improve performance significantly? \n-   Can RT-FUG- and RT-FUG+ be replaced by InfoNCE or other contrastive loss?\n\nFor more questions, please see Weaknesses.  \n\nOverall, I'd like to discuss this paper with the authors and other reviewers during rebuttal. And then update my rating.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-   This paper is pretty well-written and easy to follow. \n-   The paper is well-motivated. It is motivated by the connection between PCA and Contrastive Learning since PCA has good transferability. \n-   A different contrastive loss is proposed to replace InfoNCE.\n-   The ablation experiments show some interesting conclusions.", "weaknesses": "-   The theoretical analysis is somewhat loose. For instance, in Theorem 3.1, $Dim(\\cdot)$ and $Fea(\\cdot)$ are loose formulations since their mathematical properties are not well defined. The authors should clarify that a function or mapping is a Dim as long as it satisfies some pre-conditions. The theoretical analysis may not be qualified for NeurIPS. Moreover, the equivalence between $\\mathcal{L}_{RT}$ and InfoNCE is quite similar to some existing works, such as [1]. So the contribution from the theoretical aspect may be over-claimed.  \n-   From the ablation experiments, it seems that CE is the most important part. However, it is unclear why CE can significantly improve the performance. A theoretical proof will be convincing. \n-   The pre-training of GNNs has been studied recently. However, the proposed FUG is not compared with these methods.  Here is a survey as an example [2].\n-   There are totally 3 hyperparameters in the final loss, which may result in great difficulty in tuning them in practice.  \n\n[1] A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses\n\n[2] Graph Prompt Learning: A Comprehensive Survey and Beyond", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough review of our paper and your comments. Here are our responses.\n\n# R1 for W1.\n(a) We first give the definition of $Dim(\\cdot)$ and $Fea(\\cdot)$. \n\nGiven a data matrix $X\\in\\mathbb{R}^{N\\times D}$, $N$ represents the number of samples and $D$ represents the number of dimensions. \n\n$Dim(\\cdot)$ refers to a serious of dimension-specific encoders that directly accept multi-samples input with a specific dimension $X_{:,i}^T$ and output the embedding of that dimension, denoted as $DimEmb_i = Encode(X_{:,i}^T)$, $DimEmb\\in \\mathbb{R}^{D\\times N’}$. In contrast, $Fea(\\cdot)$ refers to a serious of sample-specific encoders that accept all dimensions of a specific sample as input and output the embedding of that sample, denoted as $FeaEmb_j = Encode(X_j, \\epsilon)$, $FeaEmb\\in\\mathbb{R}^{N\\times D’}$. Here, $Encode$ represent linear or non-linear functions, $\\epsilon$ denotes additional inputs. \n\nWith the clear definitions, PCA can be decomposed into $Dim(\\cdot)$ and $Fea(\\cdot)$, with a rigorous proof, which can be found in Global Rebuttal.\n\n(b) Secondly, although Ref [1] you provided decomposes InfoNCE into the tightness part and the contrastive part, our work is theoretically entirely different from it.\n- **Different Contributions.** In [1], the authors aim to analyzing the relationships between several CE-based losses, whereas we aim to unify the PCAs and CL frameworks. Work [1] might be somewhat similar to a small part (task analysis) of our theory. Apart from that, we extra-analyze the augmentation, encoder and sampling strategy.\n- **Relative Semantic Perspective.** In task analysis, we provided a new relative semantic perspective, whereas work [1] tried to connect InfoNCE with labels $y$ (as shown in Equations 3 and Table 2 of work [1]). Given that labels are unknown in SSL, our provided perspective, which only focus on the relationship between data, is more applicable.\n- **Positive Part Analyzing.** Work [1] only demonstrated that positive pairs in InfoNCE are aligned. In contrast, our work reveals that aligning augmented views of the same data inherently aligns representations between different data.\n\nWe will increase these in future versions to refine the theory. We will also greatly hope to discuss the theoretical part with the reviewer to further improve this work.\n# R2 for W2 and Q2.\nWe guess here you may refer to $L_{DE}$ (in Table 4) rather than CE since FUG does not include a CE component.\n\nThe main reason of its significance is that, $L_{RT-FUG^+}$ and $L_{RT-FUG^-}$ are effective when based on $L_{DE}$, especially for datasets with high quality features. As described in Section 5.4, when $L_{DE}$ is absent, DE is not sufficiently trained, which leads to DE embedding $X$ almost randomly to generate $H$, which is an important input for GE. Furthermore, without the guidance of $L_{DE}$, GE only needs to extract relationships in $H$ to reduce the loss, ignoring whether $H$ contains sufficient information from $X$. Therefore, in FUG w/o $L_{DE}$, the $H$ input to GE is of poor quality, resulting in worst performances on some datasets. We will include more explanations in future versions to facilitate understanding.\n\n# R3 for W3 and Q1.\nConsider that Ref [2] you provided in W3 is also about Graph Prompt Learning, so we combine our responses to your W3 and Q1.\n- First, it seems that Prompt Learning (PL) and Pre-Training (PT) are two distinct directions for improving graph learning. To be specific, PT (that you provided) focuses on making models learn knowledge applicable to various data and generate representations, whereas PL (which our FUG belongs to) aims to generalize pre-trained models to different downstream tasks, including few-shot and zero-shot scenarios. Thus, as a universal graph pre-training model, our FUG is primarily compared to some advanced PT models in Tables 2 and 3. Notably, in Table 2, our FUG significantly outperforms some latest SOTA universal PT models, which demonstrates the effectiveness of FUG as a universal graph pre-training approach.\n- Second, PL and PT are not mutually exclusive while may work together to achieve better performance. As a flexible PT strategy, FUG can accept various graph inputs and can be integrated with different PLs, thereby extending their generalizability across diverse graph datasets. Additionally, integrating PLs also enhances FUG's performance in few-shot and zero-shot scenarios. Exploring the combination of FUG with various PLs will be our future research direction.\n\n# R4 for W4.\nIn fact, the FUG model has only one more hyper-parameter compared to classical GCLs (or even fewer, as FUG does not require tuning multiple parameters related to augmentation).\n- First, in Equation 11, $\\lambda_1$ is fixed at 1, requiring only two hyper-parameters to be tuned, while classic GCLs require tuning the temperature parameter $\\tau$. Therefore, FUG has just one more hyper-parameter to be tuned.\n- Second, as shown in Table 9 (in the appendix), the search spaces for $\\lambda_2$ and $\\lambda_3$ are small, specifically [0.1, 0.3, 0.5] and [20, 50, 100, 200, 300], respectively. They exhibit clear unimodality in sensitive analysis in Figure 4, making the tuning process straightforward.\n- Furthermore, as shown in Table 2 and Figure 4, the model can achieve good results even without meticulous tuning, demonstrating the robustness of FUG.\n\nWe will add these discussions in the future versions.\n\n# R5 for Q3.\nYes, they can be replaced by other contrastive losses such as InfoNCE since FUG is a highly inclusive theory. However, compared to our losses, InfoNCE have some drawbacks. For example, as described in Section 3 and Appendix B, InfoNCE introduce negative sampling bias and has high computational complexity. These issues are critical for universal graph pre-training models.\n\n**Finally,** thank you again for your comments. We look forward to further discussing with you to address your concerns and to continually improve our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5WyK42MATv", "reviewer": "Reviewer_nhsa", "summary": "The authors propose a universal graph pretraining framework called FUG. Inspired by PCA, FUG includes a parameterized dimension encoding component to unify different forms of node attributes, thereby preventing the loss of important semantics. The authors design an optimization objective based on relative semantics to learn invariant knowledge across datasets. FUG demonstrates excellent performance in cross-domain training.", "questions": "Please see the weakness.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This work is highly contributive. It has the potential to positively impact the broad field of graph representation learning, as it proposes a pretraining framework that is universally applicable to graph data.  \n2. The paper is well-written and easy to understand.  \n3. The authors' theoretical foundation is solid, offering a new perspective on traditional dimensionality reduction methods and advanced graph contrastive learning.", "weaknesses": "1. The authors introduce a new loss, which, in my view, has a primary advantage over the InfoNCE in terms of faster training speed. However, its effectiveness on positive sampling compared to existing GCLs is questionable as noise based augmentation not only aligns positive samples but also enhances the model's robustness. Using neighbors as positives might make the model sensitive to noise.  \n2. Although this paper is well-written, I still recommend that the authors emphasize the two questions raised in Section 3. This would enable readers to quickly find the questions corresponding to the two answers.  \n3. In Theorem 3.1, dividing PCA into Dim. and Fea. seems inappropriate. In fact, I tend to believe that PCA cannot be clearly separated into these two parts. Matrix decomposition belongs to Fea., but the ordering of eigenvalues might be more reasonably viewed as Dim., as it can be considered a nonlinear mapping based on the relative relationships between dimensions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for recognizing our work. Our responses are blow.\n\n# R1 for W1. \nThe robustness improvement brought by data augmentation mainly stems from its ability to encode perturbed data into the same representation as clean data. Although FUG does not use data augmentation, it retains this characteristic through positive neighbor sampling. Specifically, in homogeneous graphs, linked nodes are considered to have the same semantics, even if their features and topologies are not entirely identical. Therefore, the differences between linked nodes can be seen as a form of perturbation. Compared to data augmentation, this is a method that does not require hyper-parameter tuning and adaptively adds noise, allowing it to ignore noise without encoding inter-class nodes into the same representation.\n\n# R2 for W2. \nThanks for your suggestions. We will add them in future versions.\n\n# R3 for W3.\nIndeed, as a classic and effective machine learning algorithm, it is challenging to completely separate PCA into two unrelated parts. We have added a more precise definition of $Dim(\\cdot)$ and $Fea(\\cdot)$, where $Dim(\\cdot)$ is formalized as $DimEmb_i = Encode(X_{:,i}^T)$, and $Fea(\\cdot)$ is formalized as $FeaEmb_j = Encode(X_j, \\epsilon)$, where $Encode$ represents a linear or nonlinear mapping and  $\\epsilon$ represents additional inputs (details can be found in global rebuttal). Under this definition, PCA can be decomposed into $Dim(\\cdot)$ and $Fea(\\cdot)$ as described in Section 3, because the ordering of eigenvalues can be seen as the process of solving the weights of the $Encode$ in $Fea(\\cdot)$. We will include this explanation in future versions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UBKiA2xF9T", "reviewer": "Reviewer_FBBq", "summary": "The paper is generally well-presented. It aims to address the generalizability of graph pre-training models to graph data with arbitrary node feature shapes. By comparing the Principal Component Analysis (PCA) with existing graph contrastive learning methods, the authors discovered that PCA follows the same framework as these graph contrastive learning methods. Based on this observation, they proposed a Feature-Universal Graph Pre-training (FUG) strategy and instantiated a FUG model. Extensive experiments have been conducted to validate the effectiveness of the proposed method.", "questions": "Please see the weaknesses.", "limitations": "Please see the weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed FUG strategy is novel and significant. Pre-trained graph models struggle to generalize across various graph datasets. The FUG strategy introduces the concept of dimensional encoding, which nearly losslessly unifies features to the same dimension. This innovative and interesting approach can provide new insights for researchers.\n2. The idea of relative semantic encoding is interesting, as it unifies the optimization objectives of contrastive learning and PCA. Additionally, this objective shows potential in learning domain-invariant knowledge. The instantiated loss function is effective and efficient.\n3. The authors conduct a theoretical analysis showing that classical PCA and contrastive learning fundamentally follow the same theoretical framework。\n4. The proposed FUG model demonstrates significant performance in cross-domain learning experiments.", "weaknesses": "1. Equation 4 is somewhat unclear. Although the authors provide an intuitive explanation, a more effective approach would be to formally decompose PCA according to the form on the right side of the equation, making it easier to understand.\n2. The authors state that FUG is inspired by existing works discussing the relationship between PCA and contrastive learning. The authors should more clearly introduce these works and compare them with the proposed theory to highlight the theoretical contributions.\n \nMinor issues and typos: \n1. The subscript font for symbols should be consistently either italic or upright. Additionally, Table 5 should specify the units of the values.\n2. Line 180: Maximize $[Sim(x_i, x_j )$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments, which are incredibly helpful for improving our paper.\n\n# R1 for W1. \nWe apologize for not explaining this clearly. Here, we provide a formal definition of $Dim(\\cdot)$ and $Fea(\\cdot)$. $Dim(\\cdot)$ is defined as a dimension-specific encoder, formalized as $DimEmb_i = Encode(X_{:,i}^T)$, and $Fea(\\cdot)$ is defined as a sample-specific encoder, formalized as $FeaEmb_j = Encode(X_j, \\epsilon)$, where $Encode$ represents a linear or nonlinear mapping, $\\epsilon$ represents additional inputs. With these definitions, PCA can be decomposed as follows:\n$$DimEmb = CovEncode(\\widetilde{X}^T) =\\widetilde{X}^T *\\widetilde{X}= \\widetilde{X}^TW_1$$ \n$$FeaEmb = Encode(\\widetilde{X})=W_2 X, W_2 = \\varphi(DimEmb)$$\nwhere $\\widetilde{X}$ is the standardized feature matrix, $W_2$ is the parameter matrix for $Fea(\\cdot)$, and $\\varphi(DimEmb)$ represents the process of solving $W_2$, with $\\varphi$ denoting matrix decomposition, sorting and concatenating based on eigenvalues. This clearly demonstrates how the PCA method can be decomposed into the $Dim(\\cdot)$ and $Fea(\\cdot)$ components. \n\nMore detailed definitions and prove can be found in global rebuttal. We will incorporate this into the theoretical section to improve our paper.\n\n# R2 for W2.\nAs mentioned in Sections 1 and 3, our work is primarily inspired by [1]. In [1], the InfoNCE loss is briefly explored as a generalized nonlinear PCA objective function. We further propose a relative semantic relationship encoding objective to better unify the contrastive loss and the PCA objective, and we provide additional analysis of the positive and negative parts. Additionally, as research for the CL and PCA frameworks, we extra-analyze aspects such as data augmentation, encoder, and sampling strategy. We will add these discussions in future versions.\n\n# R3 for W3. \nThank you very much for your suggestions. We will correct them in future versions.\n\n[1] The trade-off between universality and label efficiency of representations from contrastive learning, ICLR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mtBcEAxK0J", "reviewer": "Reviewer_5FZS", "summary": "In this paper, it is discovered that contrastive learning and PCA follow the same framework. By combining the two, the authors propose a graph pre-training model that offers both the generalization capability of PCA for arbitrary node shapes and the powerful embedding ability of GCLs. This approach enhances the generality of existing graph pre-training frameworks.", "questions": "- Table 2 demonstrates the remarkable transfer learning capabilities of FUG. However, does this indicate that collaborative training on multiple datasets does not significantly improve the model's performance in cross-domain scenarios? A similar observation is made in GCC, where models pre-trained on multiple datasets do not show a substantial performance improvement compared to those pre-trained on a single dataset.", "limitations": "Yes, they have.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors present a novel dimensional encoding neural network, which appears to be a simple yet effective approach for encoding global information.  \n- The paper innovatively unifies GCL with traditional PCA both theoretically and methodologically.\n- Extensive experimental analysis is provided to validate the effectiveness of the proposed model.", "weaknesses": "- The primary advantage of the proposed model lies in its robust cross-domain encoding capability. However, the paper lacks a discussion of existing works on graph cross-domain learning [1].  \n- The authors claim that PCA is limited by its inability to encode homogeneous information within the data and the inflexibility of the encoder. However, they do not explain why they did not directly use a graph encoder on the PCA-generated embeddings.  \n- Equation 10 reduces the Euclidean distance between neighboring node representations. This implies forcing the alignment of neighbor features across each dimension, which seems to contradict the authors' idea of relative semantic encoding. It appears that a constraint on the similarity would better align with this idea.  \n- The proposed model seems to lack a projector between representations and the loss, a component widely present in GCLs. The authors should provide more discussion to explain this, or it may be that they omitted the introduction of it.  \n\n[1] Unsupervised Domain Adaptive Graph Convolutional Networks, WWW’ 20.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks you very much for your comments. The following are our response. \n\n# R1 for W1. \nThe works related to Graph Domain Adaptation (GDA) primarily focus on knowledge transfer between domains. They differ significantly from FUG in several ways. \n- First, GDAs are typically supervised in the source domain and often use data from the target domain during training. In contrast, graph pre-training models are mostly fully self-supervised or unsupervised. \n- Second, Existing GDAs use datasets with fixed feature shapes. For example, in the work you mentioned, the feature dimensions of the datasets are fixed at 7537. In reality, a trained GDA cannot be universally applied to diverse graph data. \n\nThank you for your suggestion, we will supplement our discussion with GDA-related works in future versions.\n\n# R2 for W2. \nThis is mainly due to two factors: \n- First,  as described in Section 1, PCA, as a classic machine learning algorithm, can only serve as a data preprocessing method and cannot participate in graph pre-training. Consequently, during data preprocessing, PCA loses a substantial amount of important information that cannot be recovered through pre-training. \n- Second, another disadvantage of PCA methods is their inability to transform features to arbitrary shapes. PCA can only perform dimension reduction, not dimension increase. The dimension of node features varies greatly, as shown in Table 7 (Appendix), where the feature dimension of Physics is 8415, while PubMed is only 500. Using PCA preprocessing can only reduce the dimension of Physics to 500 to match PubMed, resulting in severe information loss.\n\nWe will add these discussions in the future versions.\n\n# R3 for W3.\nAdmittedly, using similarity can also align the relative semantics between positives. However, using the l2 distance has advantages, mainly in that. \n- First, using the l2 distance can better balance the gradients. The other two losses are defined based on the l2 distance, so defining $L_{RT-FUG+}$ using the l2 distance can make the values of the three losses more comparable, thereby balancing the gradients. \n\n- Second, the l2 distance and similarity are equivalent. As mentioned in Section 3 and Appendix A, reducing the l2 distance and increasing similarity are equivalent under the assumption that the length of the representation vectors is the same. Additionally, both $L_{DE}$ and $L_{RT-FUG-}$ include normalization operations, which essentially act as flexible similarity constraints between representations. The l2 distance constraint in $L_{RT-FUG+}$ can indirectly ensure that the representations satisfy the assumption of equal lengths. \n\nWe will add relevant discussions in future versions.\n\n# R4 for W4. \nFUG does not require a projector, demonstrating the effectiveness of the proxy tasks designed for FUG. We believe that the existing of projectors prevents overfitting to the proxy tasks, thereby increasing the quality of representations. Essentially, the overfitting problem stems from the proxy tasks, as existing CL losses bring bias from negative sampling. However, $L_{RT-FUG-}$ avoids negative sampling by utilizing the concept of global uniformity, thus avoiding artificially defined negative samples. Consequently, our proxy task is more effective and can directly constrain representations, leading to better model performance.\n\n# R5 for Q1.\nThis is a very valuable question. As you mentioned, like GCC, multi-datasets collaborative training only yields slight improvements for FUG. As stated in Section 5.2, we believe this is mainly because the proposed RT task is easy to learn and can be generalized across datasets. Therefore, a small amount of data training can achieve performance close to that in intra-domain scenarios. Additionally, this might be because existing tasks like node classification and link prediction are relatively easy, and the information quality provided by the datasets is high, facilitating knowledge transfer. We will attempt to challenge more complex graph scenarios in future works to explore more effective graph pre-training models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SS4VYcFPBD", "reviewer": "Reviewer_C58N", "summary": "In this paper, the authors design a feature-universal graph contrastive pre-training strategy for model rebuiding and data reshaping. They focus on the inherent complex characteristics in graphs, and introduce a theoretical analysis for their method. The gloabl uniformity constraint reduces the time complexity from O(n^2) to O(n).", "questions": "1. There is a lack of sensitivity analysis experiments for the hyperparameters in Equation 11.\n\n2. In Equation 9, z is made to approach 0. Will this operation lose the diversity of features?", "limitations": "The use of Gaussian function can also achieve uniformity. Please analyze the advantages and disadvantages compared to the method designed in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces a theoretical analysis about their designed method.\n\n2. The authors propose a feature-universal graph contrastive pre-training strategy.\n\n3. The designed method is clearly described.", "weaknesses": "1. Line 59 points out the shortcomings of the language model, but with the development of LLM, numerous LLM4graph papers have been proposed, effectively solving the problem of feature diversity.\n\n2. The results in Table 3 demonstrate certain effectiveness of the designed method, but the methods compared are not the latest [1, 2, 3, 4]. The improvement in performance is border. The effectiveness of the design method is not convincing enough.\n\n3. The results in Table 4 show that DE has a significant impact on model performance, while the other two parts have little impact on the model. How can the effectiveness of these two module designs be proven?\n\n[1] Xiao, Teng, et al. \"Simple and asymmetric graph contrastive learning without augmentations.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[2] Yu, Yue, et al. \"Provable training for graph contrastive learning.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[3] Zhuo, Jiaming, et al. \"Graph contrastive learning reimagined: Exploring universality.\" Proceedings of the ACM on Web Conference 2024. 2024.\n\n[4] Bo, Deyu, et al. \"Graph contrastive learning with stable and scalable spectral encoding.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Our responses are as follows.\n# R1 for W1.\nWhile there have been some LLM4graphs works which seems achieve feature diversity problem, as highlighted in our paper, they encounter many challenges which is correctly our focus.\n- **Restricted Feature Input.** LLMs require node features to be in text form, whereas FUG can handle node features in various formats, including an amount of floating-point data. Moreover, arbitrarily converting graphs or nodes into text can lead to serious domain biases.\n- **Structure Neglect.** LLMs are unable to process structured data, often resulting in the separate encoding of features and structures in LLM4Graphs. In contrast, FUG can simultaneously encode features and structures.\n- **Model Complexity.** Due to their large number of parameters, LLMs struggle to be optimized by graph tasks in pre-training. FUG, being lightweight, is easy to train and applicable to various scenarios.\n\nTherefore, using LLMs to solve the feature diversity problem is often limited and sub-optimal. As shown in Table 2, OFA is a recent SOTA LLM4Graph approaches, yet its performances on the text-rich graph datasets Cora and PubMed are significantly inferior to that of FUG. On the PubMed dataset, OFA's performance is even worse than simply using the raw bag-of-words node features as input for downstream tasks.\n\nFurthermore, FUG strategy may have the potential to facilitate the development of large graph models, which is more suitable for graph data.\n# R2 for W2.\nThank you very much for the provided the references.\n\nFirst, please kindly note that, the main focus of our work is to develop a graph pre-training model which is generalizable across different datasets, while the four cited methods cannot be applied to cross-domain scenarios with varying feature shapes. Besides, the results in Table 2 rather than Table 3 validates our main contribution. As shown in Table 2, FUG demonstrates significant performance advantages compared to the SOTA cross-domain graph pre-training methods, which demonstrate the effectiveness of our method.\n\nIn the intra-domain scenario, follow your suggestions, we compare FUG with these works, as shown in the table below. GraphACL and ROSEN results are obtained using the same experimental settings as in our paper. POT, being a plug-in method with experimental settings similar to ours, is directly reported with its best performances from its paper. Due to the unavailability of Sp2GCL code and the considerable differences in the datasets used in their paper, we do not compare with this method.\n| |Cora|Citeseer|PubMed|Photo|Computers|\n|-|-|-|-|-|-|\n|GraphACL [1]|83.40±2.12|**73.37±1.77**|85.07±0.62|92.93±0.79|**89.50±0.31**|\n|ROSEN [3]|83.35±1.77|73.01±2.01|85.33±0.84|92.82±0.72|88.87±0.42|\n|POT [2]|80.50±0.90|69.00±0.80|82.80±2.00|92.40±0.30|89.10±0.30|\n|FUG (Ours)|**84.45±2.45**|72.43±2.92|**85.47±1.13**|**93.07±0.82**|88.42±0.98|\n\nAs shown in the table above, even in intra-domain scenarios (which are not our main focus), FUG demonstrates competitive performance, achieving the best results on three out of five datasets. This proves the effectiveness of FUG and is consistent with the arguments in our paper. We will add these results and discussions in the future versions.\n\n# R3 for W3.\nIn fact, all the three losses are crucial and effective, which is why FUG outperforms the other three methods across all datasets in Table 4. We explain W3 from two aspects.\n\n- In fact, these three losses are not independent; the two RT-FUG losses rely on $L_{DE}$, as it ensures that the input $H$ of GE retains sufficient information from $X$. This is evident in datasets with high-quality features like Photo and Computers, where without $L_{DE}$, the other two losses do not function properly.\n- Notably, the two RT-FUG losses have a greater impact on performance in datasets with high-quality structures, such as CiteSeer. This proves the effectiveness of the RT-FUG losses, as they guide GE to encode structures effectively with a low-quality $H$.\n\nThus, as a pre-training model suitable for a wide range of datasets, all three losses are indispensable for FUG.\n# R4 for Q1.\nIn fact, we have already conducted the sensitivity analysis experiments. But due to limited space in the main text, we included the results in Appendix F, Figure 4. We will add a guide in the main text in future versions to help readers find these results.\n# R5 for Q2.\nWe apologize for any misunderstanding. In fact, in Equation 9, $z$ is not made to approach $0$. Instead, we enforce $||z||_2 = 1$. Equation 9 ensures that the **mean** of the normalized representations approaches $0$, which allows the representations to be uniformly distributed on a hyper-sphere centered at the origin with a radius of 1. This guarantees the diversity of the representations.\n# R6 for L1.\nThank you very much for pointing out the limitation we had overlooked. In our understanding, the Gaussian function you mentioned refers to the \"Gaussian potential kernel\" provided in \"Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere\". (If this is not the case, we will be grateful if you could provide new references.)\n\nUsing the Gaussian function to achieve uniformity has many advantages. The uniformity metric is asymptotically correct, and the uniformity metric is empirically reasonable with finite number of points. However, it has two problems. a) It relies on correct negative sampling, and b) it has a high time complexity of $O(n^2)$. In contrast, $L_{RT-FUG^-}$ avoids these drawbacks. $L_{RT-FUG^-}$ eliminates the need for manual negative sampling, thereby avoiding sampling bias. Additionally, the time complexity of $L_{RT-FUG^-}$ is $O(n)$. We will include relevant explanations in future versions.\n\n**Lastly,** thanks for your comments. We would be very grateful if you could engage in further discussions with us. This would be highly beneficial for our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the five reviewers for their constructive suggestions and comments, which have greatly helped us improve our paper. We also extend our gratitude to the NeurIPS Chairs for reviewing our paper.

We have noticed that some reviewers raised concerns regarding Theorem 3.1. In our paper, we provided an intuitive explanation of the two proposed encoders, $Dim(\cdot)$ and $Fea(\cdot)$, and offered a rigorous proof of the equivalence between the CL loss, the PCA objective, and the proposed RT task. Here, we present a clearer definition of $Dim(\cdot)$ and $Fea(\cdot)$ and a theoretical proof of how PCA can be decomposed into these two parts, to enhance our theoretical foundation.

Given a set of data $X \in \mathbb{R}^{N \times D}$, where $N$ represents the number of samples, and $D$ represents the number of dimensions. Let $Encode$ represent a series of linear or nonlinear mapping functions.

**Definition of $Dim(\cdot)$.** $Dim(\cdot)$ refers to a series of dimension-specific encoders that directly take the data of all samples under a specific dimension $i$, $X_{:,i}^T$, and output the embedding of that dimension, $DimEmb_i$, formalized as $DimEmb_i = Encode(X_{:,i}^T)$, $DimEmb \in \mathbb{R}^{D \times N'}$.

**Definition of $Fea(\cdot)$.** $Fea(\cdot)$ refers to a series of sample-specific encoders that take the data of all dimensions of a specific sample $j$, $X_{j}$, and output the embedding of that sample, $FeaEmb_j$, formalized as $FeaEmb_j = Encode(X_j, \epsilon)$, $FeaEmb \in \mathbb{R}^{N \times D'}$, where $\epsilon $ represents additional input.

Based on the above definitions, PCA can be decomposed into $Fea(\cdot)$ and $Dim(\cdot)$ parts, with a detailed proof as follows:

> Given an input data $X$ with a standardized matrix $\widetilde{X}=X-Mean(X)$, PCA is as follows.
> $$PCA(X) = \widetilde{X}P, P=Norm[Concat(c_i|c_i\in C, \lambda_c \ge \lambda^{(k)})], (\Lambda, C) = \delta\[Cov(\widetilde{X})],$$
where $(\Lambda, C)$ represent the eigenvalues and eigenvectors matrix, $Norm(\cdot)$ represents normalization, $Concat(\cdot)$ represents concatenation, $\lambda^{(k)}$ represents the k-th largest eigenvalue, $Cov(\cdot)$ represents covariance computation, and $\delta(\cdot)$ represents eigenvalue and eigenvector computation. This can be decomposed into the forms of $Dim(\cdot)$ and $Fea(\cdot)$ as follows:
> $$ DimEmb =  Encode_1(\widetilde{X}) = \widetilde{X}^TW_1, W_1=\widetilde{X},$$
> $$FeaEmb = Encode_2( \widetilde{X}) = PCA(X) = \widetilde{X}W_2, W_2 = P= \varphi(DimEmb),$$
> where $W_1$ is a special parameter matrix, $W_1=\widetilde{X}$, and $\varphi(\cdot)$ represents the operation to solve $W_2$ as follows.
> $$W_2 = P = \varphi(DimEmb) = Norm[Concat(c_i|c_i\in C, \lambda_c \ge \lambda^{(k)})], (\Lambda, C) = \delta(DimEmb).$$
> From the above, $Encode_1(\cdot)$ and $Encode_2(\cdot)$ satisfy linear or nonlinear mappings. Therefore, we derive:
> $$PCA(X) = \widetilde{X}P = \widetilde{X}W_2 = \widetilde{X} \delta(Dim(\widetilde{X})) = Fea(\widetilde{X}, Dim(\widetilde{X})).$$
> Thus, PCA can be rigorously decomposed into $Dim(\cdot)$ and $Fea(\cdot)$ parts as described in Theorem 3.1.

We will include the above content in future versions of this paper to further support our theory. Additionally, we have provided relevant content and detailed discussion in each reviewer's response.

Once again, we thank all the reviewers. We have carefully read all the comments and made every effort to address all concerns. We hope that the rebuttal phase can be informative and pleasant for all reviewers.

Meta-review: The authors introduce Feature-Universal Graph Pre-training (FUG), a novel framework inspired by Principal Component Analysis (PCA) designed to enhance the generalizability of graph pre-training models. FUG features a parameterized dimension encoding component that unifies various node attributes while preserving essential semantics. By integrating the principles of PCA and Graph Contrastive Learning (GCL), FUG aims to learn invariant knowledge across different datasets. The model's innovative approach addresses the challenge of generalizing to graph data with diverse node feature shapes, and extensive experiments demonstrate its effectiveness, particularly in cross-domain training scenarios.

The reviewers praise the paper's clarity, solid theoretical foundation, and innovative approach that unifies Principal Component Analysis (PCA) with Graph Contrastive Learning (GCL) both methodologically and theoretically. The Feature-Universal Graph Pre-training (FUG) strategy is recognized as a significant contribution to the field of graph representation learning. While some issues were raised concerning the theoretical analyses and experimental evaluations, the reviewers were generally satisfied with the authors' further explanations provided during the discussion period.
Decision: Accept (poster)