Full-Rank Unsupervised Node Embeddings for Directed Graphs via Message Aggregation
Authors: Ciwan Ceylan
Kambiz Ghoorchian
Danica Kragic
Venue: Accepted by TMLR ()

Abstract: Linear message-passing models have emerged as compelling alternatives to non-linear graph neural networks for unsupervised node embedding learning, due to their scalability and competitive performance on downstream tasks. However, we identify a fundamental flaw in recently proposed linear models that combine embedding aggregation with concatenation during each message-passing iteration: rank deficiency. A rank-deficient embedding matrix contains column vectors which take arbitrary values, leading to ill-conditioning that degrades downstream task accuracy, particularly in unsupervised tasks such as graph alignment. We deduce that repeated embedding aggregation and concatenation introduces linearly dependent features, causing rank deficiency. To address this, we propose ACC (Aggregate, Compress, Concatenate), a novel model that avoids redundant feature computation by applying aggregation to the messages from the previous iteration, rather than the embeddings. Consequently, ACC generates full-rank embeddings, significantly improving graph alignment accuracy from 10% to 60% compared to rank-deficient embeddings, while also being faster to compute. Additionally, ACC employs directed message-passing and achieves node classification accuracies comparable to state-of-the-art self-supervised graph neural networks on directed graph benchmarks, while also being over 70 times faster on graphs with over 1 million edges.

Published in Transactions on Machine Learning Research (05/2025)
Full-Rank Unsupervised Node Embeddings for Directed Graphs via Message Aggregation
Ciwan Ceylan
ciwan@kth.ss
Division of Robotics, Perception and Learning KTH Royal Institute of Technology
Kambiz Ghoorchian
kambiz hta.bbii.ghhorrhaannnnoo Bscb. com
SEB Croup
Danica Kragic
dani@kth.
Division of Robotics, Perception and Learning KTH Rogal Institute of Technology
Reviewed on OpenReview: https: // openreview. net/ forum? id= ad=3EcEE2g213
Abstract
Linear message-passing models have emerged compelling alternatives to non-linear graph neural networks for unsupervised node embedding learning, due to their scalability and competitive performance on downstream tasks. However, we identify a fundamental flaw in recently proposed linear models that combine embedding aggregation with concatenation during each pesssge-passingg iteration: rank deficiency. A rank-deficient embedding matrix contains column vectors which take arbitrary values, leading ill-conditioning that degrades downstream task accuracy, particularly in unsupervised tasks such as graph alignment. We deduce that. repeated embedding aggregation and concatenation introduces linearly depen- dent features, causing rank deficiency. To address this, propose ACC (Aggregate, Com- press, Concatenate), novel model that avoids redundant feature computation by applying aggregation to the messages from the previous iteration, rather than the embeddings. Con- sequently, ACC generates full-rank embeddings, significantly improving graph alignment accuracy from 10% to 60% compared to rank-deficient embeddings, while also being faster to compute. Additionally, ACC employs directed message- passiin and achieves node clas- sification accuracies comparable to state-of-the-art self-supervised graph neural networks directed graph benchmarks, while also being over 70 times faster graphs with over 1 million edges.
1 Introduction
Node embeddingg,, which represent nodes in a graph as vectors, have proven highly effective for various graph-related tasks, including node classification (Veličković et al., 2018; Rossi et al., 2023), node clustering (Henderson et al., 2012; Donnat et al., 2018), and graph alignment (Heimann et al., 2018; Skitsas et al., 2023). Consequently, there has been substantial research focused on developing algorithms
compute these em-
beddings, resulting in wide array of models (Kipf & Welling, 2017; Wu et al., 2019; 2021; Rossi et al., 2023).
Given the scarcity of labelled data in real-world graphs (Veličković et al., 2019), our work focuses on unsupervised embedding models. Unlike supervised models, which target specific downstream tasks, unsu- pervised models aim to extract and compress the information inherent in the graph structure into individual embedding vectors. The dominant approach for this extraction is message- message-passing (Gilmer et al., 2017).
In a nessage-passing algorithm, each node initially assigned a feature vector, serving as its initial em- bedding. These embeddings are iteratively refined by incorporating information from increasingly larger

---

Published in Transactions on Machine Learning Research (05/2025)
PCAPass
ACC
Initial embeddings
Embedding aggregation
Messages
Initial messages
Message aggregation
Messages
Concatenate embeddings and messages
PCA compression
Graph
Compressed messages
Embeddings
Embeddings
PCA
compression
Concatenate with previous messages
Figure Overview of directed message-passing in PCAPass (Sadowski al., 2022) and our model, ACC. Each vertically stacked rectangle represents matrix corresponding to node in the graph (bottom- left). The representations are colour-coded: blue for node embeddings, and orange and purple for messages. Orange denotes aggregation following edge directions, while purple indicates reverso-direction aggregation. Node 1, highlighted in green in the graph, is used to illustrate these aggregations. In PCAPass, messages are concatenated with the embeddings from the previous iteration, creating a feedback loop that leads to feature duplication and rank deficiency over multiple iterations. ACC avoids this issue through message aggregation, where only the message matrices are propagated between iterations. The embedding matrix is constructed by concatenating the messages outside the feedback loop.
neighbourhoods. Conceptually, each iteration consists of two steps: the aggregation step, where information from node's neighbourhood is summarized into message, and the update step, where these messages are integrated into the existing embeddings.
When the aggregation and update steps consist of parameterized and non-linear functions, the message- passing model is referred to as a Graph Neural Network (GNN) (Wu et al., 2021). While the parameterization and non-linearity of GNNs make them highly expressive, these features also limit their scalability. Training GNNss requires non-convex optimization, typically through gradient descent, to minimize a self-supervised loss function. This training process involves hundreds even thousands of training epochs, each consisting of multiple message-pasee paassng iterations (Zhang et al., 2021; Thakoor et al., 2022; Hou et al., 2022; 2023).
In their seminal work, Wu et al. (2019) addressed the scalability challenge of GNNs by introducing a linear musssgo-passingg model called SGCN. They demonstrated that SGCN could achieve accuracics comparable to GNNs on popular node classification benchmarrk while being significantly more scalable, requiring only a single execution of the message- messagge-assing procedure. However, despite these advancements, SGCN encounters pesssag--assingg models: over-smoothing (Li et al., 2018; Chen et al., 2020a), where the
common issue repeated summation of embeddings across iterations leads to gradual loss of information.
Computing multi-scale embeddings, i.e., concatenating vectors that capture information from various neigh- bourhood sizes, is a common method for mitigating information loss in unsupervised node embedding models (Tang et al., 2015; Cao et al., 2015; Perozzi et al., 2017; Donnat et al., 2018; Rozemberczki et al., 2021). ).. A re- cent example of this approach is PCAPass (Sadowski et al., 2022). As shown on the left Figure 1, PCAPass aggregates node embeddings in each pessagc-passing itcration create new fcatures, called messages, which are then combined with the existing embeddings via horizontal concatenation. To prevent the embedding dimensionality from doubling with each iteration, the concatenated matrix is compressed using PCA.
While the concatenation approach addresses over-smoothing, it introduces a new issuc: rank deficiency (Hansen, 1998, Ch. 1). A rank-deficient matrix characterized by a cluster of singular values close lo zero, each corresponding to an arbitrary and non-informative feature column. This phenomenon affects the PCAPass embedding matrix and degrades the quality of the embeddings. Unsupervised downstream tasks, which rely on distances between embeddings, are particularly sensitive to this issue, raising concerns since such tasks well-suited for unsupervised node embedding models.
2

---

Published in Transactions on Machine Learning Research (05/2025)
We identify that the repeated embedding aggregation and concatenation in PCAPass leads to the creation of linearly dependent features, which results in rank deficiency. To address this, we propose ACC² (Aggregate, Compress, Concatenate), linear message-passing model designed to prevent rank deficiency while main- taining scalability by generating embeddings in a single forward pass. As shown in Figure 1, ACC applics aggregation and PCA compression to the message malrices from the previous iteration, rather than the em- beddings, and constructs the embeddings via concatenation separately. This message aggregation approach breaks the feedback loop present in PCAPass, and thereby avoids computing the redundant features that cause rank deficiency.
In support of ACC, we demonstrate the rank deficiency issue present in PCAPass, focusing particularly on its negative impact on unsupervised embedding- based graph aligument (Heimann et al., 2018). Although it is technically possible to address the rank deficiency in PCAPass through singular value thresholding, this approach is not only difficult due to numerical inaccuracies but also inefficient, as computation is used to generate the redundant features in the first place. Consequently, ACC consistently achieves higher graph alignment accuracies and computes embeddings faster than PCAPass.
We also demonstrate ACC's effectiveness in learning node embeddings on directed graphs, underexplored challenge for unsupervised message- paaaaggggpssingggggo. models that has only recently been addressed in the supervised learning context (Rossi et. al., 2023). Using standard directed graph node classífication benchmarks without hyperparameter tuning, we show that ACC achieves accuracies comparable to state-of- the---rrtth--rtttte self-supervised GNNs. Notably, ACC is significantly faster, at least 70 times faster on the Arxiv Year dataset (Lim et al., 2021) with GPU computations and 270 times faster on Snap Patents (Leskovec et al., 2005) with CPU.
2 Background: Node embeddings via message-passing and embedding aggregation
The message-passingg framework (Gilmer et al., 2017) forms the basis for wide range of graph models (Kipf & Welling, 2017; Hamilton et al., 2017; Wu al., 2019), including our model, ACC. In this section, we introduce the principles and mathematical notation for node embedding learning through message- message-passing, followed by a description of the embedding aggregation and concatenation approach used by Sadowski et al. (2022) for PCAPass. Additionally, we outline directed message-passing for PCAPass, following Rossi et al. (2023).
Node embedding message-passing models take as input a graph g = MM) and a matrix X E Raxd and output an embedding matrix Z E Raxp, The graph 9 consists of n nodes N, with n = [N], connected by m edges M, where [M] The matrix X contains initial feature vectors of length d for each node, and the embedding matrix Z holds the resulting p-dimensional node embeddings. In this work, we assume transductive setting where G is fully observed during the compulation of Z, which is common in unsupervised node embedding models (Zhang et al., 2021; Thakoor et al., 2022; Hou et al., 2022; Sadowski et al., 2022).
The goal of pressag--passing is to gather graph structure information into the embeddings of cach nodc. This is achieved by ileratively updating each node embedding by incorporating information from the nodes' respec- tive neighbourhood, resulting in sequence of progressively refined embedding matrices. We denote the em- bedding matrix after k passage-passing iterations as H (4) E RR= where the embedding dimensionality pk may vary across iterations. The final embeddings are obtained after K iterations, represented as Z = H (K),
Each messago-passinng passing iteration consists of two key steps: aggregation and update. In the aggregation step, each node collects information from its immediate neighbours, aggregating these inputs into a new vector called a message. We denote the matrix of all messages at iteration k as (*),, In the update step, each node integrates the new information by combining its received message with its current embeddings. This iterative process allows information to propagate through the graph, enabling each node to gather data from increasingly distant nodes, effectively expanding its receptive field with each iteration.
We now describe the aggregation and update operations used by PCAPass in detail. Let A E Raxx represent the graph's 's adjacency matrix, where the element Ar, indicates the presence of an edge from node j to node i.
IACC model code: https://gitthnn http:///gihhbb.cocc/c/aaaeeeee/s Experiments code: http:://ttttttccccccceccccccccce
3

---

Published in Transactions on Machine Learning Research (05/2025)
The out-degree and in-degree of node i are denoted as dego (i) and degr(1), respectively:
An. { 1 if ( otherwise,
dego (i)
Σ
Ak.
degi (i) = Σ A k=
(1)
k=1
Additionally, let Do be diagonal matrix containing the out-degrees, with Do dega and similarly, let Dr be a diagonal matrix of in-degreos. The matrices D5 and DJ" represent their respective inverses, with elements corresponding to nodes with zero out-degree or in-degree set to 0.
For undirected graphs the adjacency matrix is symmetric, A AT, and each node has single degree, so D = Do = Dr. The normalized adjacency matrix is then defined as Au = D-IA. Using the above ionis the embedding aggregation step used by PCAPass can be formulated as ( = A, (k-1), where the message for node i, M, is the average of its neighbours' embeddings.
PCAPass updates its embeddings in each iteration using a concatenation and compression approach. This update can be expressed as ( (k--)) M ((k), where the brackets indicate horizontal concate- nation. The matrix (()) E 2 1xPb is derived through PCA (Murphy, 2012, Ch. 12.2) and serves to compress the embeddings. Compression is essential because, without it, the dimensionality of the embed- ding space would double with each message-passing iteration, resulting in a final dimension of 2 d. This exponential growth would impose considerable memory and computational overhead, and could negatively impact downstream tasks due to the curse of dimensionality.
Although Sadowski al. (2022) originally formulated PCAPass for undirected graphs, it can be straight- forwardly exlended directed graphs by following the approach of Rossi et al. (2023). In directed graphs, each node has two distinct sets of neighbours: those connected by incoming edges and those connected by outgoing edges. To capture the information from these distinct neighbourhoods, two separate aggregation operators are employed: A; = = A and A = DT AT. Here, Ar uses the adjacency matrix A, while As uses its transpose AT, corresponding to a graph where all edge directions are reversed. The forward operator Ap aggregates messages based on a node's incoming edges and normalizes by in-degree, while the backward operator Ag aggregates based on outgoing edges and normalizes by out-degree.
(k) = (1 and With these operators, the directed aggregation step for PCAPass is defined as M
M (+-1, In the update step, both forward and backward messages are concatenated with the previous embeddings and compressed: H (k) = (k-1) M
M (k) (kk,, with (ik) e 1
3 Embedding aggregation and concatenation results in rank deficiency
As outlined above, PCAPass (Sadowski al., 2022) leverages embedding aggregation, concatenation, and compression to generate node embeddings. This approach aims to address the over-smoothing issue that often plagues massang--assing models (Li et al., 2018; Chen et al., 2020a). However, the repeated process of embedding aggregation and concatenation introduces and retains redundant features in the embeddings. This not only leads to inefficient computation but also results in rank deficiency, a condition where the embedding matrix becomes ill-conditioned, adversely affecting the quality and usefulness of the embeddings.
3.1 Origin of rank deficiency
A matrix Z Raxp is considered rank-deficient if it exhibits cluster of small singular values, with clear gap between the large and small singular values (Hansen, 1998, Ch. 1.). This situation arises when the columns of Z are not linearly independent, indicating the presence of redundant features.
To illustrate the relationship between redundant features and small singular values, consider a simple ex- ample. Let X E Raxd be full-rank matrix with the singular value docomposition (SVD) X UΣVT, Here, U E Raxd and V E Raxd have orthogonal columns with unit norms, and Σ is non-negative diagonal matrix containing the singular values in descending order (Golub & Van Loan, 2013, Ch. 2.4).
4

---

Published in Transactions on Machine Learning Research (05/2025)
Now, define matrix Z = columns in Z redundant. The SVD of Z can then be expressed as Z = U2 V2, where
X X]
R"x2d, where each column of X is duplicated, making the last d
V2 0 0
VT
VT
U2 U Tu].
Σg =
V2
(2)
V2
r
Y
Here, the block matrices Yr E Baxd Tv, Rdxd, and Tv, E Rdxd are orthonormal and satisly T =0 and TV v= r V O. We can verify this decomposition through matrix multiplication:
V2E
1 V
VT TV
V
VT T
U3Σ2VZ U Tul
IUE
0]
[UEV UEVT]. (3)
M
m
(See Appendix A for verification of the orthonormality of Uz and Vz.)
The expression for Σz in Equation reveals two clusters of singular values: the d values contained in E and d singular values equal to zero. Each of the zero singular values corresponds to a column in Yu, which we refer to as singular dimensions. These singular dimensions span the left mull space of Z and do not contain any information about X. All information about X is already encapsulated in U, Σ, and V, demonstrated by Equation 3. The columns of Yo are only constrained by their orthogonality to the columns in U, and their elements can otherwise be chosen arbitrarily. Similarly, Tv. and Tv, are only constrained by their orthogonality to V.
Although the values of Yu, Tv, and Tv, are not uniquely defined, in practical applications, they are determined by a combination of numerical inaccuracies and the arbitrary implementation choices of the specific algorithm used compute the SVD. Since they do not carry any meaningful information about X, they effectively be considered noise. Furthermore, numerical implementations of SVD will not produce singular values that exactly zero due to rounding errors, reflected in the definition of rank deficiency.
3.2 Rank deficiency in PCAPass
Having established the connection between redundant features and rank deficiency, we now analyse the wessage-passing mechanism in PCAPass, as described in Section 2, to show how redundant features con- tribute to rank deficiency in its embeddings. For simplicity, we focus on undirected message- message-passing; however, the results readily extend to directed graphs.
Let Pk denote the dimension of the embeddings after compression in the kth iteration, with Po d, and let (k) 2 xPP denote the compression matrix for iteration k. Then, the PCAPass message passagg with the first two iterations given by update step can be expressed (k) [ (k-1) ( Wkk,),ith
1)) X
A W
(4)
(2) (I))
(, W(2)
By inserting (l) into the expression for (2),, and factoring out W(), we obtain the following result:
(2) [H(I)
A. (
= [[X A&X] W() Ag X
A
W w
(())
0
(5)
X
A&X A,X
0
w
A X
(2)
( = [
A
A,X |
E Baxdd
(6)
x
(11) 0
(2(2)
2 E @ddxpa
(7)
0
W(1)
5

---

Published in Transactions on Machine Learning Research (05/2025)
We see that H A W(2,, where A (2) consists of four submatrices of aggregated node features. We can express ( the same block matrix form:
H (3) (2)
(2
W
(
=|X
AgX XxX X AxX XIX X X
W
(8)
vi
A (3)
W3(3) X
Ax (3)
[X A,
X X A,X X A A] E @Raxdd
(9)
33
22
0
W 3) E 8dxps
(10)
O
(
Continuing this pattern, we see that ( A (k) Wk),, where A A E gax2*d and W (k) E R2
Notice that the submatrix Ay is repeated twice in A and three times in A (3) x where A also appears three times. These redundant submatrices contribute to the rank deficiency of the PCAPass embeddings. In fact, only one linearly independent submatrix is introduced in each message- message-passing iteration, specifically X in iteration k, implying that vank(Ax (k)) $ (k + 1)d. Consequently, we obtain upper bound the rank of H (k) by applying Sylvester's inequality (Horn & Johnson, 2012, Ch. 0.4.5):
x
( < min(rank(A)) ))) rank W (1)) S rank( x S (k + 1)d.
(11)
Thus, we have Pk 2 Pk-(k (k 1)d, with Ps being the number of singular values close to zero for ( RRxxP This bound indicates that the PCAPass embeddings will be rank-deficient unless pk S (k + 1)d. However, note that setting ps (k: + 1)d does not guarantee full-rank embeddings, additional sources of rank deficiency may be present, such as the potential rank deficiency of X itself.
Having established the rank-deficiency of the PCAPass embeddings regardless of the specific compression method used to compute each W(k), we now examine the PCA compression utilized by Sadowski et al. (2022) in greater detail. Recall that PCA is typically and efficiently implemented through centring and SVD (Murphy, 2012, Ch. 12.2.3). Therefore, if all singular dimensions are retained, the final PCAPass embeddings can be expressed as Z [UE TE Here, UE [=*((+1)d represents the informative part of the embeddings, while EE E Rax captures the singular dimensions, with elements in Ezo being close to zero.
Applying compression in each iteration has the potential to address the rank deficiency by removing the singular dimensions Y. However, the effectiveness of this approach depends on the specifies of the PCA compression. Sadowski et al. (2022) implement a maximum embedding dimension, Pmax, such that Ps = min(pmax; 2dd), where 2 is the embedding dimension without compression for undirected message- passing. This implies that the PCAPass embedding will remain rank-deficient in each iteration until (k + 1)d 2 Pmax. Thus, using a fixed value for Pmax does not ensure the removal of singular dimen- sions. Instead, the number of retained dimensions would need to vary with each iteration k to guarantee the elimination of all singular dimensions. This introduces additional algorithmic complexity, especially since the bound on ps must be adapted for directed message-passing and for message-passing with multiple aggregation operations (Jin et al., 2019; Corso et al., 2020).
An alternative approach is to apply a threshold the singular values during compression, removing any dimensions where Σx,t 0E,,, where Σt, is the largest singular value and 0 E [o, 1] a relative tolerance. However, setting 0 either too low or too high can lead to loss of accuracy in downstream tasks, see Section 5.1.
Moreover, even if the singular dimensions are successfully eliminated, computational resources have still been unnecessarily spent on computing the redundant features in the first place. Given that the number of singular dimensions can grow rapidly, as suggested by the bound Pk (2* k 1)d, and that PCA compression has quadratic time complexity with respect to the number of features (Golub Van Loan, 2013, Ch. 2.4), the amount of unnecessary computation is substantial.
6

---

Published in Transactions on Machine Learning Research (05/2025)
X
B
C
+
D
E
+
F
10
UMAP dim 2
UMAP dim
UMAP dim
UMAP dim
6
6
10 UMAP dim
2.5
5.0
7.5
2.5
5.0
7.5
10
12
UMAP dim
UMAP dim
UMAP dim
(a) Original, X
(b) X duplicated 1 time
(c) X duplicated 2 times
(d) X duplicated 4 times
Figure 2: The effect of increasing rank deficiency on cluster structures. Figure 2a shows UMAP projection of X, comprising 200 data points from 6 classes originally sampled on the surface of a 6D sphere. Figures 2a to 2d illustrate the cluster structure deterioration as X is horizontally concatenated with itself and whitened.
3.3 The negative effects of rank-deficient embeddings on clustering
Rank-deficient or otherwise ill-conditioned embedding matrices can violate fundamental assumptions made by downstream machine learning models, like the linear independence of features, and contribute to the instability of numerical computations (Trefethen & Bau, 1997, Pt. 3). Tn supervised learning, the adverse effects of rank deficiency are often mitigated through regularization techniques. For example, ridge regression (Hoerl & Kennard, 1970) effectively addresses this issuc. However, regularization and model selection become much more challenging in unsupervised settings, where the absence of guiding supervision signal complicates the process (Ma et al., 2023). As result, unsupervised tasks that rely on embedding distances, such as clustering and graph alignment, are particularly vulnerable to the negative impacts of rank deficiency.
The rank-deficien embedding matrices produced by PCAPass are especially problematic when subjected to common preprocessing operations like standardization (Aggarwal, 2015, Ch. 2.3.3) and whitening (Hyvärinen et al., 2009). We illustrate this with a simple clustering example. We generate a matrix X E IR 200xx consisting of 200 six-dimensional samples. The data is divided into six classes, each corresponding to normal distribution centred at one of six equidistant points. This setup ensures that samples from each Gaussian blob are well-separated. We visualize this in Figure 2a using 2D UMAP projections (McInnes et al., 2018).
To simulate the introduction of redundant features, we concatenate X with itself, forming X = [X X] Applying PCA-based whitening to X yields the simulated embedding matrix Z I UU r]. This expression highlights the connection to PCAPass embeddings that have been standardized to unit variance.
Figure 2 visually demonstrates how cluster separability diminishes as the number of duplicate features increases. This degradation in separability can also be quantified by running k-means clustering (Arthur & Vassilvitskii, 2007). The average normalized mutual information (NMI) (Danon al., 2005) over 10 seeds is 1.0 for the original data, indicating perfect separation of the classos. However, X is duplicated, the NMI decreases to 0.78, and then to 0.71 with two duplicates, and down to 0.47 with four duplicates.
Note that the information required to distinguish the clusters remains present in Z, and a regularized supervised classifier could still achieve high classification accuracy. However, the relevant information has been diluted by the singular dimensions, Y, which significantly impairs the class's clustering separability.
In this toy example, the negative effects of the singular dimensions can be mitigated by using Z 3-U0 rE as input for clustering, which simulates not preprocessing the PCAPass embeddings. However, in practice, preprocessing is often necessary for achieving high task accuracy. Embedding- based graph alignment using structural input features is one such example.
3.4 The negative effects of rank-deficient embeddings on graph alignment
Graph alignment is the task of finding node correspondences between two graphs, 9, (N, M,) and G2 (N, Ml2). We assume, for simplicity, that while the node set N is shared, while the edge sets differ. When Ml, is subset or superset of M,, we refer to the missing or added edges as noise edges.
7

---

Published in Transactions on Machine Learning Research (05/2025)
4
4
displacement
10
displacement
10
2
2
10
10"
10-8
10-
-
2
10 11
10
10
-255
0.0
2.5
-25
0.0
2.5
Li singular value
E singular value
(a) No whitening
(b) With whitening
(c) Arenas
(d) Enron
Figure 3: Visualization explaining the need for whitening in embedding- basedd graph alignment. In Figure 3a, the goal is to match the blue plus with the blue circle. However, the Euclidean distance fails to account for the data's shape, causing the erroneous orange squares to appear closer. Whitening corrects this issue, shown in Figure 3b. The assumption in Figure 3a is that the direction of displacemenni aligns with the data's 's variation. Figures 3c and 3d provide empirical evidence of a strong correlation between the displacement along axis and the corresponding singular value, Σj based on PCAPass embeddings.
Embedding- based graph alignment (Heimann et al., 2018) is greedy approach where node emboddings are used to match the nodes. Specifically, the nodes in G2 are matched to the nodes in G1 with the most similar embeddings in terms of Euclidean distance, which can be computed efficiently using KD-trees (Bentley, 1975). When the graphs lack node attributes, structural node embeddings are used (Jin et al., 2021). These embeddings can generated by messagg-ppssing models where X consists of structural features such as node degrees and local clustering coefficients (Fagiolo, 2007).
For this approach to be effective, the embeddings need to be whitened before matching. This is detrimental to the alignment accuracy of PCAPass embeddings, since applying whitening removes the scaling of Σ=o from the singular dimensions, allowing Y to introduce noise into the distance computation. Consequently, the PCAPass embeddings struggle to produce high graph alignment accuracies, as we demonstrate in Section 5.
Figure 3 provides a visual explanation for why whitening is necessary. The gray scatter points in Figure 3a represent the embeddings for each node G1, and the ellipse highlights variations along the principal axes of the embedding matrix. The coloured points show magnified version of the alignment process. The blue circle represents embedding vector in G1, and the blue plus represents the corresponding node in G2. The orange squares represent embeddings for other nodes in G1.
As shown, both orange squares are closer to the blue plus than the blue circle in terms of Euclidean distance. Thus, this node would be incorrectly matched. However, if the covariance of the data considered in the
Algorithm 1: The ACC algorithm for a directed graph G with node features X using K message- passsing iterations, desired dimensionality Paax, minimal message size Guin and relative volerance 0 1).
Algorithm 2: PCA compression computed via SVD as used for ACC. Here X Rxxp is real-valued matrix, an integer s.t. c CS P. and 0 E [0, 1] relative tolerance.
def ACC(G, X, K, Pmax: Cuin 2, 0 = 10-8): max ([Pmax/ (K + 1)J, Cmin)
def PCA(X, c, 0):
X centreeeeeeec cow.urreeeocuo_(e(_ (X)
(0) PCA(X c, H = (0)
U, E, VT SVD(X)
# Find the index of the smallest singular value which exceeds the given tolerance relative to the largest singular value Σ1,1 hy ===ma max(j Eg, 2 /E,11 Keep at most Ào dimensions. k = min(c, ko)
for l in range(1, (1, K 1):
Compute aggregations in both direction.
(k) = (k-)) (k-)
(A) (*) 0,
# Concatenate with existing embeddings.
(h)) (k-1) (
X, =V=VV.. return X.
XV.
return (K)
8

---

Published in Transactions on Machine Learning Research (05/2025)
distance calculation, the blue plus is closer to the blue circle, indicated by the green ellipse. Whitening spheres the data, equalizing the variance along each principal axis. Therefore, Euclidean distance provides the correct matching in the whitened space, as shown in Figure 3b.
This explanation assumes that the direction of the embedding displacement, indicated by the black arrow in Figure 3a, correlates with the data variation. Figures 3c and 3d provide empirical support for this assumption. These figures show the singular values of the PCAPass embeddings for G1, denoted z on the x-axes. The y-axes show the average embedding displacement along the corresponding principal axis, defined as 6, 1 (l) - z V.all, where V is basis for the principal axes of Z As seen, the correlation between the directions of variation and displacement is strong, supporting our assumption.
In summary, the rank-deficient embeddings produced by PCAPass contain arbitrary column vectors that act as noise for unsupervised downstream tasks such as clustering and graph alignment. This issue is particularly problematic when normalizing preprocessing steps are required for optimal performance, as is the case with whitening for graph aligument. Therefore, is crucial to avoid rank-deficient node embeddings.
4 The ACC model and message aggregation
To address the issues of rank-delicient embeddings, we introduce the ACC, which stands for Aggregale, Compress, and Concatenate. Performed in this order, these operations result in a message-passing approach where the message matrices, rather than the embeddings, are passed between message-passing iterations. We refer to this message aggregation, and as we will demonstrate, it avoids computing the redundant features that cause the rank deficiency observed in PCAPass. Furthermore, ACC is designed to work with directed graphs, expanding its applicability and versatility.
The complete ACC algorithm is detailed in pseudocode in Algorithm 1. Initially, the feature matrix X is compressed into a c-dimensional message matrix, ( E BR*e, using Principal Component Analysis (PCA) (Murphy, 2012, Ch. 12.2). Once this initial compression is complete, the message-passingg procedure begins. The main distinguishing feature of ACC, as shown on Line 6 of Algorithm 1, is message aggregation. This means that ACC applies aggregation operators directly to the message matrices from the previous iteration, (A) [A, AgM(A1)] This approach is crucial for avoiding the crcation of redundant features, as seen in PCAPass when aggregating and concatenating embedding matrices in Equation 9.
The new message matrix (6) is then compressed to c dimensions using PCA and concatenated with the em- bedding matrix, (R) (k-1 M(]]. Consequently, the final embedding matrix is a concatenation of all message matrices, H (k))= = (0) M()... (()]], with a total embedding dimension of (K + 1)c.
Typically, the value of e is chosen as the largest integer such that the total embodding dimension does not exceed a desired limit, (K + 1)c S Pmax- However, this condition would result e = 0 if (K + 11 > Pmax- To prevent this, ACC enforces a minimal value for c, denoted as Cmin: as specified on Line of Algorithm 1. By default, set Cmin = 2 to account for the two messago-passing directions in dirccted graphs.
For undirected graphs, ACC is slightly modified on Line 6, as message-passing is conducted using only the undirected matrix Ag, (k) = (k-(,-1),, In this scenario, the compression steps on Line 7 is technically unnecessary since M (k) is already o-dimensional. Nonetheless, for consistency across our experiments, we always perform PCA.
To dcmonstrate that ACC avoids computing the rodundant features seen for PCAPass in Equa- tion 9, we write out the expression for the undirected ACC embeddings after K iterations: (i))= = [XW(%) AnXW(1), Af XW). In this formulation, each ((4) d is a compres- sion matrix derived from successive applications of PCA. Unlike PCAPass, the ACC embeddings exclude the repeated submatrices found in Equation 9. Specifically, (K) in ACC contains exactly one instance of A X for each & E (O...., K}, representing the non-redundant information produced al each iteration.
We can further verify that the ACC embeddings are not rank-deficient by examining the number of singular values close to zero, denoted as PK. where K represents the number of message-passing iterations. To compute PK, we count the number of singular values for which Σ, S 10-6E Figure 4 illustrates pk for
9

---

Published in Transactions on Machine Learning Research (05/2025)
10
MCC
10a
10
PCAPPs
PCAASs
2hd
110
1..0
107
10
10
PCAPE
2.5
5.0
10.0 2.5
0.0
2.5
5.0
10.0
100
(a) Arenas, # En R
(b) Arenas, dimensions
(c) Enron, # Σi, RM 0
(d) Enron, dimensions
Figure 4: Figures 4a and 4c show the number of small singular values, PK, as function of the number of message-passing iterations, K, for two graphs: Arenas and Enron. The dashed line in 4a represents the the- oretical prediction for PK discussed in Section 3.2. Figures 4b and 4d display the embedding dimensionality Рк- Both ACC and PCAPass use the maximum dimensionality Punx = 512, resulting in the curve cut-off.
both the ACC and PCAPass embeddings, along with the corresponding embedding dimensions pk, for two graphs: Arenas, an undirected graph, and Enron, directed graph. The initial number of features is d for Arenas and d = 6 for Enron. In both cases, the maximum embedding dimensionality Pmax 512 is applied.
We observe that the number of small singular values remains at zero for all values of K for ACC, indicating that its embeddings are full rank. In contrast, for PCAPass, PK grows rapidly for both graphs until the embedding dimensionality pk reaches the maximum value Pmax For Arenas, the growth of px closely follows our predicted lower bound (2* - - 1)d, as depicted by the dashed black curve in Figure 4a. For Enron, PK initially increases more rapidly than for Arenas, consistent with the fact that the dimensionality Pk expands more quickly for directed graphs due to the concatenation of forward and backward aggregations. This also results in Pmax being reached sooner, and as consequence, the singular dimensions are replaced by non- redundant features after K = 4, causing Pk to drop to zero. This analysis highlights the interaction between the PCAPass rank deficiency, the number of message- message-passing iterations, and the maximum dimensionality used for PCA compression.
While message aggregation effectively avoids computing the redundant features highlighted for PCAPass in Equation 8, other sources of singular dimensions can still arise. For instance, singular dimensions can be introduced if X itself is not full rank, if the adjacency matrix contains eigenvalues close to zero. To mitigate these issues, ACC employs a small threshold 0 during the PCA compression to discard embedding dimensions with small singular values, as outlined in Lines 4 to 6 of Algorithm 2.
However, singular dimensions can also emerge due to correlations across message-passing iterations. For example, if X contains an eigenvector of Ag, this would introduce a singular dimension in [X AxX], ACC cannot remove these correlations during message-passing because the messages from each iteration are compressed independently. This further implies that the dimensions of the final ACC embeddings, Z, may exhibit correlations, which could impact certain downstream tasks.
To address these concerns, decorrelation and removal of singular dimensions from H after message-pasingg could be beneficial. However, in our experiments, we have not incorporated these steps by default, treating them instead as potential preprocessing steps for specific downstream tasks.
4.1 Time complexity, scalability, and the inductive learning setting
The time complexity of ACC is primarily determined by the message aggregation and PCA compression steps (lines 3, 6, and 7 in Algorithm 1). For an input matrix of size p, PCA has complexity O(mp²), arising from both the complexity of SVD (Golub & Van Loan, 2013, Ch. 5.5.6) and the matrix multiplication in Algorithm 2, line 6. Thus, line 3 has complexity O(nd)). The message aggregation step (line 6) involves sparse-dense matrix multiplications with complexity O(mc)), where ej denotes the dimension of the message matrix (k-1) from the previous iteration. The PCA compression step (line 7) has complexity O(nc)), which also varies with iteration k. Using the bound Cg < Pmax from line 2, we derive the overall worst-case K+1
K complexity for K iterations as O ndd2 + K+I mipmax + (K+1)= mp P
K
10

---

Published in Transactions on Machine Learning Research (05/2025)
ACC
PCAPass
ACC + whitening
PCAPass whitening
0.5
0:75
0.6
Accuacy
Accuracy
0.4
Acmuracy
0.60
Accuacy
Accurcy
0.4
0.0
0:20
0.3
0
0.0
6.00
0.0
0.0
K
K
(a) Arenas
(b) PPI
(c) Enron
(d) Polblogs
(e) Magua
Figure 5: ACC and PCAPass for graph aligument with 15% noise edges, The x-axis shows the mumber of message-passing iterations K. Markers and shaded areas represent the average and standard deviation over seeds. Without whitening, both algorithms perform poorly. Whitening benefits ACC across all datasets, with its accuracy reaching 60% on Arenas compared to 10% for PCAPass. Whitening only benefits PCAPass when embeddings are not rank-deficient, as shown in Figure 4.
The linear dependence on n and m highlights ACC's scalability. Furthermore, both message aggregation and PCA compression benefit from parallelization and GPU acceleration, supported by prevalent graph learning librarics (Paszke al., 2019; Fey & Lenssen, 2019). Importantly, ACC learns emboddings in a single forward pass without gradient-based optimization. This is in contrast to self-supervised graph neural networks, which typically require hundreds or thousands of training epochs (Hou et al., 2022), making ACC a computationally efficient alternative.
PCAPass similarly features linear complexity in n and m. However, while ACC restricts message matrix dimensionality with c S Panxs the corresponding bound for PCAPass is cæ S Pmax. Cousequently, the worst- case complexity for PCAPass over K iterations is O(Kmpmax + KKpp'ax) which represents an increase by a factor of K+1 in the term involving m, and by a factor of (K +1) in the term involving n, compared to ACC.
K+1
ACC is primarily designed for transductive learning, where the entire graph available during training. Given its efficiency, recomputation of embeddings upon graph updates remains straightforward. However, ACC can also be adapted for inductive learning using SraphSAGG-inspieee approach (Hamilton et al., 2017). Given a new node with known features and connections to existing nodes, its embedding can be computed by aggregating messages from neighbors within a sampled subgraph.
The primary challenge for inductive learning with ACC is the PCA-based compression step, since the pro- jection matrices (k) are derivod from the original training graph. One practical solution is to project new message matrices (k) onto the previously computed PCA bases, ensuring consistency with existing embeddings. Additionally, incremental PCA methods could be employed to dynamically update the bases without full recomputation (Balsubramani et al., 2013; Yu et al., 2017).
Both subgraph sampling and incremental PCA introduce potential sources of noise, necessitating careful empirical evaluation. Given ACC's computational efficiency, inductive extensions are primarily relevant for very large graphs and we thus leave a detailed empirical investigation of such extensions to future work.
5 Experiments
In this scction, we empirically demonstrate the superior graph alignment accuracy of the ACC model com- pared to PCAPass, highlighting the negative impact of rank deficiency. Additionally, we compare ACC to saate-of-the-rrs self-supervised graph neural networks (SSGNNs) across five standard node classification benchmarks for directed graphs (Rossi et al., 2023). Our results show that ACC over 70 times faster on the largest datasets while achieving better accuracy with default hyperparameters compared to the SSGNNs.
5.1 Graph alignment: ACC vs. PCAPass
To evaluate alignment accuracy, we adopt the established experimental protocol from Heimann al. (2018). Specifically, we generate a second graph G2 from reference graph G1 by permuting node indices and removing 15% of the edges. We conduct experiments using four datasets: Arenas, PPT, Enron, and Polblogs. Arenas
11

---

Published in Transactions on Machine Learning Research (05/2025)
and PPI are widely used undirected benchmark graphs (Heimann et al., 2018; Jin al., 2021; Skitsas et al., 2023), while Enron and Polblogs provide examples of directed graphs.
Additionally, we evaluate performance on the real-world Magna dataset (Saraph & Milenković, 2014), where the perturbed graph G2 contains 15% more edges than G1. These noisy edges were selected by Saraph & Milenković (2014) based on observed node interaction probabilities. Real-world datasets like Magna are rare due to the complexity inherent to the graph alignment problem. Detailed dataset statistics are provided in Appendix C.1.
For both ACC and PCAPass, we set the maximum embedding dimension to Punx 512 and use node structural features as input, X E Raxd, For undirected graphs, the input features are the node degree and local clustering coefficient (d = 2). For directed graphs, we expand the feature set to include the in-degree, out-degree, and the four local clustering coefficients outt in, cycle, and middleman as defined by Fagiolo (2007), resulting in d= 6.
We first compare ACC and PCAPass across varying numbers of message- passing iterations K, with and without whitening, as shown in Figure 5. Without whitening, both ACC and PCAPass yield low accuracy, as expected given the necessity of whitening for embedding-based graph alignment (sce discussion in Section 3.4). With whitening, ACC accuracy markedly improves on all datasets, reaching 60% on Arenas and 75% on Enron. In contrast, PCAPass accuracy remains low for undirected graphs, peaking at only 10% on Arenas. For directed graphs, accuracy initially increases, then deteriorates, and eventually recovers. This fluctuation closely linked to the presence of singular dimensions in PCAPass embeddings. As shown in Figure 4c, accuracy drops when the number of small singular values is high, particularly for K E (2,3,4) 4). These results highlight the detrimental impact of rank deficiency on alignment accuracy.
We next investigate how singular value thresholding can mitigate rank deficiency in PCAPass. Specifically, we apply a threshold 0, removing embedding dimensions with singular values M1, S f2,,,, where E1, is the largest singular value. We use K = 10 for undirected graphs and K = for directed graphs, settings where PCAPass exhibits maximum rank deficiency.
The results are shown in Figure 6, with the threshold 0 varying along the x-axis. We observe that selecting appropriate 9 effectively addresses rank deficiency, allowing PCAPass to achieve accuracics comparable to ACC. However, choosing 0 requires careful consideration: it must be high enough to remove singular dimen- sions but not so high as to eliminate important information. The optimal range for 0 depends on numerical precision and dataset characteristics; for Magna, only 0 E [10-6, 10-51 yields accuracies matching ACC.
Moreover, ACC consistently outperforms PCAPass in terms of computational speed by avoiding the com- putation of redundant features. As illustrated Figures 6c and 6d, this speed advantage is substantial. On the Magna dataset, for instance, ACC is approximately 80% faster than PCAPass, with runtimes around 55 ms compared to 100 ms for PCAPass.
Finally, we compare ACC's graph alignment accuracy against three state-of-the-art graph alignment methods: S-GWL (Xu et al., 2019), CONE (Chen et al., 2020b), and FUGAL (Bommakanti et al., 2024). Here, ACC is significantly more efficient approximately 100 times faster than CONE and FUGAL, and over 1,000 times faster than S-GWL. Additionally, ACC achieves the highest accuracy on the Enron dataset due to its offectiveness on dense, directed graphs, a scenario challenging for other methods. Detailed results of this comparison are provided in Appendix E.
5.2 Node classification: ACC vs self-supervised graph neural networks
To compare ACC with self-supervised graph neural networks (SSGNNs), we evaluate node classification accu- racy on five standard directed graph datasets from the literature (Pei et al., 2020; Lim et al., 2021; Platonov et al., 2023; Rossi et al., 2023). Dataset details are provided in Appendix C.1. For initial node features X, we use both the dataset- provided node features and the structural features used for graph alignment.
We train a gradient boosting classifier on top of the embeddings, following the setup of Sadowski et al. (2022). Specifically, we employ Scikit-learn's implementation of LightGBM (Ke et al., 2017) with default
12

---

Published in Transactions on Machine Learning Research (05/2025)
0.4
0.5
AFI:
0.5
ACC
Accurary
.0.3
Accuracy
(1.6
0.1
PCAPuu
Run time (4)
PCAFa
0.2
0.4
time
0.3
0.3
ADC PCARas
0.2
ACC: PCAP
0.2
0.00
0.1
0.0
00.
10-1 Relative tolerance,
10
10 10 18
10-1
10
10
10-
-0
10-1
10-1
10
10:5 10 10 10 Relative tolerance,
Relative tolerance,
Relative tolerance,
(a) Magna, accuracy
(b) Enron, accuracy
(c) Magna, run time
(d) Enron, run time
Figure 6: ACC and PCAPass for graph aligument with 15% noise edges, K = 10 for Magna, and K = 4 for Enron. The x-axes show the relative tolerance O used to remove dimensions with small singular values. Figures 6a and 6b show accuracy on the y-axis, while 6c and 6d show run time. Markers and shaded areas in- dicate the average and standard deviation over 5 sceds. ACC achieves equal or superior accuracy for all 8, and retains its accuracy for 0 = 0, as its embeddings are full rank. ACC also consistently faster than Pass
hyperparameters. To ensure robustness, we perform three repeats of 5-fold cross- valsdatin with five different random sceds, reporting mean and standard deviation statistics for the classification accuracy.
All models are executed in a Google Cloud g2-standard-32 environment with one Nvidia L4 24GB GPU, 32 vCPUs @ 2.20GHz, and 128 GB of memory. Reported run times are averages over the five random seeds.
We adapt all bascline models to use directed messago-passing following the approach outlined by Rossi et al. (2023), except for MVGRL (Hassani & Khasahmadii, 2020) and GREET (Liu et al., 2023), which feature non-standard architectures. Additionally, we introduce BGRL-GS and GraphMAEv2-GG by replacing the default GNNs in BGRL (Thakoor et al., 2022) and GraphMABv2 (Hou et al., 2023) with the GraphSAGE model (Hamilton et al., 2017). SGCN and PCAPass are also included in our comparisons. For all models, including ACC, we use K = 2 message-passing iterations and p = 512 embedding dimensions, both of which are commonly used values in the literature (Hamilton et al., 2017; Veličković et al., 2018; Zhang et al., 2021; Thakoor et al., 2022; Hou et al., 2022). We use default values for optimizer and loss function hyperparameters. In the Appendix H, we provide a discussion on hyperparameter tuning of unsupervised learning models, well as analysis of huperparameter tuning results for the number of training epochs and embedding dinmensions for the SSGNNs.
Table displays the accuracies and run times for all models and datasets. ACC stands out with the highest accuracy across all datasets. Morcover, these results demonstrate ACC's superior scalability compared to SSGNNs. On the Arxiv Year dataset, with milion edges, ACC is over 70 times faster than CCA-SSG, the fastest SSGNN, Moreover, whereas most SSGNNs take a full day or more to run on Snap Patents, the largest dataset with 14 million edges, ACC requires only 27 seconds.
The linear model SGCN is the only baseline faster than ACC. However, SGCN suffers from over-smoothing, leading to ACC's substantial accuracy advantage on the Roman Empire dataset. This underscores how ACC's concatenation approach mitigates over-smoothing. Further results are available in the Appendix D.
Additionally, we observe a significant accuracy discrepancy between ACC and PCAPass on the Squirrel dataset, which cannot be attributed the PCAPass rank deficiency. Our detailed analysis, available in Appendix F, highlights another benefit of ACC's message aggregation. Unlike PCAPass, which compresses its full embeddings in each iteration and may discard informative but low-variance features, ACC compresses features separately at each iteration, preserving information more evenly. Consequently, the ACC embeddings retains more information than PCAPass from the dass-informattve but low-variance fcatures provided by AgX in the Squirrel dataset, resulting in higher accuracy.
6 Rank deficiency in SSGNNs: Beyond the linear setting
As demonstrated by our theoretical analysis in Section and experimental results in Section 5, the embedding concatenation approach of PCAPass leads to rank deficiency, negatively impacting embedding quality. A natural question is whether similar issues affect self-supervised graph neural networks (SSGNNs), and if so, whether ACC's message aggregation approach could mitigate them.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Table 1: Gradient boosting node classification results. The top 3 accuracies are highlighted in bold. Snap Patents results were gathered using CPU only due to GPU memory limits. 00M abbreviates out of memory.
MODEL
CHAMELEON
SQUIRREL
ROMAN EMPIRE
ARXIV YEAR
SNAP-PATENTS
ACCURACY
TIME
ACCURACY
TIME
ACCURACY
TIME
ACCURACY
TIME
ACCURACY
TIME
No MODEL, X
64.6 2.3
0M8
52.1 1.4
OMS
70.4 0.7
OMS
0.2
OMS
56.1 0.1
OMS
GAE'
62.2 2.2
11s
44.8 + 1.5
1M 48
53.4 + 3.1
168
42.3 + 0.3
7M 34s
51.9 = 0.1
3u 15M
DGI
61.5 2.3
14s
44.2
+ 1.6
1M 40s
73.4 + 0.9
4M 28
47.1 + 0.2
6и
TIMEOUT
22411
MVGRL3
06.
26M 44s
46.4 + 1.5
26M 325
64.3 + 1.0
39M
138
0OM
2128 GB
OOM
2128 GB
BGRL
66.1
7M
49s
46.7 + 1.3
31M 45s
76.0 - 0.8
25M
24s
0.3
4H 52M
TIMEOUT
224H
BGRL-GS
65.3 2.2
8M
31s
44.8 1.5
33M 21s
74.4 0.7
21M
528
4.2
311 8M
TIMEOUT
22411
CCA-SSG"
2.1
48
59.9 1.2
6..7
10s
4.2
1M 11s
56.1 0.0
11 40M
GRAPHMAL'S
68.9 2.0
278
56.9 1.8
59s
53.9 + 1.0
4.3
8M 55S
45.6 0.1
168
ARA2
69.6 1.9
368
51.2
2.5
1M 148
-
1M
0.3
14M
0.0
23H
GRAPHMAEV2-GS'
74.1 1.11:2. 2.3
298
57.0
1.4
1M 68
80.0 0.7
1M 358
46.310..3
10M 46s
53.8
18H
GREETS
61.4
1M 258
44.2
1.6
6M 458
80.1
0.5
56M
OOM
2128
OOM
2128 GB
SPGCL*
66.3 2.2
168
45.8
1.5
1M 238
73.6 1 0.8
558
0.2
52M
OOM
2128 CB
SGCN20
74.7 1.8
374MS
68.3
1.4
723MS
45.4
393MS
0.2
1s
50.2
26s
PCAPASS"
71.7 2.2
28
61.5
1.3
22s
852MS
46.3 0.2
61.2
20s
ACC
76.6 1.9
18
71.5
1: 1.3
18
81.5 1 0.6
432M8
49.4 1 0.3
18
62.6
278
1 KIPF& & WELLING (2017)
2 VELIČKOVI ET AL. (2019)
3 HASBANI & KHASARMADI (2020)
TTHAKOOR ET AL. (2022)
5 ZHANG ET AL. (2021)
HoU ET AL. (2022)
Hou ET AL. (2023)
LLU ET AL. (2023)
9 WANG ET AL. (2023)
Wu ET AL. (2019)
11 SADowsKI ET AL. (2022)
The two main differences between SSGNNs and the PCAPass ACC models are:
GNNs employ non-linear activation functions during message- message-passing.
Self-supervised GNNs require optimization of loss function using non-convex methods due these non-linearities.
These differences introduce challenges for both theoretical and empirical analyses. First, the non-lineesssies in GNNs invalidate the rank-deficiency analysis presented in Section 3. Consequently, more sophisticated math- ematical frameworks necessary, similar to ongoing theoretical investigations in deep learning (Roberts et al., 2022). Such advanced analyses are beyond the scope of this paper and constitute important directions for future research.
Empirical analysis is complicated by the diversity of existing SSGNN architectures, which differ significantly in terms of message--psssing schemes (c-g., GCNs (Kipf & Welling, 2017) add self-loops, whereas GATs (Veličković et al., 2018) utilize attention mechanisms) and loss functions (Veličković et al., 2019; Zhang et al., 2021; Thakoor et al., 2022; Hou et al., 2022). Additionally, SSGNNs contain many hyperparameters inftuencing embedding quality, related to both architecture and optimization.
A comprehensive empirical study of rank deficiency and embedding quality across various SSGNN setups would therefore be extensive and is beyond this paper's scope. Nonetheless, we perform a small-scale analysis of SSGNN versions of PCAPass and ACC in Appendix G, summarized here for completeness.
Specifically, we convert PCAPass into an SSGNN model similar to GraphSAGE (Hamilton et al., 2017) by using the cessage-passingg update:
(
(K-1))
M(
( w(()),
(12)
where a is the PRcLU activation function (He et al., 2015), ( E 1 are learnable weight matrices,
and dimensions are set Po = d and pk Pmax for k > 1. The message matrices M (k) and M (k)
are
computed as in Section 2. We refer to this model as PCAPass-GNT
For the corresponding ACC-GNN model, we modily lines 3 and 6-7 in Algorithm as follows:
O =a (XW("),
M(M) =a (AA-M()))) (g w())..
(13)
where ( dxe and W(h) R2xxe for k > 1, with Pmmax K 1), consistent with standard ACC.
14

---

Published in Transactions on Machine Learning Research (05/2025)
We train LrAPass-GNN and ACC-GNN using the GraphMAEv2 loss function (Hou et al., 2023), evaluating performance at 0, 20, and 200 training epochs to observe changes induced by weight learning. We assess both the embedding rank and the graph aligmment accuracy across varying numbers of message-passing iterations, using a fixed embedding dimensionality of pmax = 512. Full results are provided in Appendix G; key insights are summarized below.
Most notably, we find that LAPPas--GN embeddings do not exhibit rank deficiency, as evidenced by the absence of significant gaps in their singular value spectra across all training epochs. This suggests that the inclusion of activation functions mitigates the explicit rank collapse observed in the linear setting. In terms of downstream performance, PCAPass-GNN shows substantial improvement over the original lincar PCAPass model in graph aligument tasks and even surpasses ACC on one dataset.
Despite this improvement, two critical issues remain. First, PCAPass- GNN'ss alignment accuracy peaks around K 5, after which performance declines as K incrcases, suggesting a progressive loss of discriminative information. This patlern mirrors the behaviour observed for linear PCAPass in Figure 5, and stands in contrast to ACC-GNN, which, like ACC, maintains stable or plateauing accuracy acioss larger values of K.
Second, we observe that PCAPass-Gal achieves its highest aligmment accuracy without training (0 epochs), with performance deteriorating after 20 and 200 training epochs. This performance deterioration appear to correlate with variability in the singular value spectra. Although the embeddings are not rank-deficient, we find that the condition number, i.e, the ratio between the largest and smallest singular values, increases significantly in many cases where alignment accuracy drops. This effect is particularly pronounced for undirected graphs, after 200 epochs of training, and at high message-passing depths (e.g., K = =100)
In contrast, ACC-GNN does not exhibit such degradation, instead maintaining more stable condition num- bers and accuracies across training epochs, albeit with higher variance compared to linear ACC. However, these trends are not entirely consistent across all datasets and models, warranting further investigation.
From this exploratory analysis, we conclude that while PCAPass-GNV does not suffer from explicit rank defi- ciency, its embedding quality and behaviour closely resemble those of the linear PCAPass model. Meanwhile, ACC-GNN continues to mirror the robustness of linear ACC, This consistency suggests a deeper underlying reason for the benefits of ACC's message aggregation strategy one that manifests as rank deficiency in the linear case and as instability in the non-linear setting. We hypothesize that large fluctuations in singular val- ues may be linked to the observed performance degradation in PCAPass-GNN GNN. As previously noted, formal theoretical analysis is needed to rigorously understand these effects, and this remains an important direction for future research in unsupervised node embedding models.
7 Conclusion and future work
In this paper, we addressed the issue of rank-deficient node embeddings generated by linear message-passing models, which not only lead to computational inefficiencies but also risk degrading downstream task perfor- mance. To overcome this, we introduced ACC, a novel unsupervised node embedding model that leverages message aggregation to avoid redundant feature computations, thereby ensuring full-rank embeddings.
Beyond our analysis of linear models, we also provided initial insights into rank instability in self-supervised graph neural networks, pointing to the existence of deeper underlying mechanisms behind the observed deficiencies. These findings motivate further theoretical and empirical investigation.
A promising direction for future work to explore whether the layerwise PCA-based compression schemes used in ACC and PCAPass can be reformulated as the optimization of a global loss function. Such a formulation could deepen our theoretical understanding of these models and extend to broader class of unsupervised nessngo-passing architectures, including self-supervised graph neural networks.
Moreover, ACC's message aggregation stralegy inherently ties each embedding feature to a specific hop distance within the graph. Replacing PCA with a feature selection mechanism could further enhance inter- protability by explicitly linking each embedding dimension to a single input feature. This extension could be particularly valuable for applications requiring transparency, such as anomaly detection.
15

---

htt.dd..rrr.000.000/007755555//3 Published in Transactions on Machine Learning Research (05/2025)
References
Lada A. Adamic and Natalic Glance. The Political Blogosphere and the 2004 U.S. Election: Divided They Blog. In LinkKDD '05, pp. 36 43, 2005. URL https://doi.orgee 1145/1134771.11442
Charu C. Aggarwal. Data Mining: The Tertbook. Springer, first edition, 2015. URL https://dol org/
0007/978-3-33 3-1-----11-----------
David Arthur and Sergei Vassilvitskii. K-means+++ The Advantages Careful Seeding. In SODA '07, pp- 1027 1035, 2007. URL httpp://d1l wtt.:aammm..o/////. http:///i/1...555122////3 1283494.
acm.
Akshay Balsubramanii, Sanjoy Dasgupta, and Yoav Freund. The fast convergence of incremental pea. NewrIPs '1, volume 26, 2013. URL
htt.///i...rrg1...////////////// 1501. 03796.
1501
Jon Louis Bentley. Multidimensionall Binary Search Trees Used for Associative Searching. Communications of the ACM, 18(9):509 517, 1975. URL https://doit httt::://oo..rrg/1011145533 /ttp.///r....111//101000.0.00000
Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Veriiee 2006. URL
Aditya Bommakanti, Harshith Reddy Vonteri, Konstantinos Skitsas, Sayan Ramu, Davide Mottin, and Pana- giotis Karras. FUGAL: Feature-fortified Unrestricted Graph Alignment. In NewrIPS '24, 2024. URL https://openreviev..t wttp:://w.....ooooo/oaaaaaaaaaaa
Bobby-Joe Breitkreutz, Chris Stark, Teresa Reguly, Lorrie Boucher, Ashton Breitkreutz, Michael Livstone, Rose Oughtred, Daniel H. Lackner, Jürg Bähler, Valerie Wood, Kara. Dolinski, and Mike Tyers. The BioGRID Interaction Database: 2008 update. Nucleic Acids Research, 36(suppl_ 1):D637- 1116677--660,, 2007.
URL https://dol org/10 10. w099//arr///1000/
Shaosheng Cao, Wei Lu, and Qiongkai Xu. Grarep: Learning graph representations with global structural information. In CTKM '15 pp. 891-900, 2015. URL https: http::///oooooooo////
org/ 10. wtt.:/...//.././//..111 145/2806411.28866122 2806512.
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun, Measuring and Relieving the Over-Smootiine Smoothing Problem for Graph Neural Networks from the Topological View. AAAI '20, 34((4))3438-38)) 3445, 2020a. URL
https://doi. ort.//rg///// 10. 1609/aaa8 v34104.5747.
Xiyuan Chen, Mark Heimann, Fatemeh Vahedian, and Danai Koutra. CONE-AAlign: Consistent Network Aligument with Proxinity-PPreservnng Node Embedding. In CIKM 220 pp. 1985 1988, 2020b. URL https:///oi. org/10. 10111111113351 34121..
Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Velickovic. Principal Neighbourhood Aggregation for Graph Nets. NewrIPS 2020. URL https://proceedii newr.ps. wwwtrips.ccpaaaaeee paper/ hptp:h/haaaaoaaaaaaaarrreeaeooep wwt.:ooaaee-ttttttt.hhah.
Loon Danon, Albert Díaz-Guilera, Jordi Duch, and Alex Arenas. Comparing community structure iden-
tification. Journal of Statistical Mechanics: Theory and Experiment, 2005(09), 2005. URL https:
Claire Donnat, Marinka Zitnik, David Hallac, and Jure Leskovec. Learning Structural Node Embeddings Via Diffusion Wavelets. In KDD'1, pp. 1320-1329, 2018. URL https://doi org/ 10. 1145/3219819. 3220025.
Giorgio Fagiolo. Clustering in complex directed networks. Physical Reviewo E, 76(2), 2007. URL https:
1ttp://o....//ty/eeee...... 76. 026107.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019. URL https://///o.. http:ass://ooooorooo.ete555 org/10..855// 1xXiv.103..00 02428. 192....28..2...
16

---

Published in Transactions on Machine Learning Research (05/2025)
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural Message Passing for Quantum Chemistry. In JCML '1, pp. 1263-1272, 2017. URL https://proceeddigg: mlr.
press//v0//gimmrr7aa.. hatm. html.
Gene H. Golub and Charles F. Van Loan. Matrix Computations. JHU Press, fourth edition, 2013. URL https://epubs. siam. wwtg/oo/aa////.ooooooooo 1ttpa//...11771..781222244221331
Roger Guimera, Leon Danon, Albert Diaz-Gri Guilera, and Francesc Giraltand Alex Arenas. Self-similar community structure in network of human interactions. Physical Review E, 68:065103, 2003. URL
https://doi. org.oo.g///../.ooo.ooooo... 11/1100..111/// 1w..100//hyyeee..... 68.06510336.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive Representation Learning on Large Graphs. In NewrIPS'17, 2017. URL batps://doi.. h1tg///0..855///// arxiv. 1706. .2216..
Per Christian Hansen. Rank-Deficient and Discrete Ill-Posed Problems. Society for Industrial and Applied Mathematics, 1998. URL https://epubs.. siam.
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive Multi- View Representation Learning on Graphs. In ICML '20, pp. 4116 4126, 2020. URL https://proceedinggs.mo mlr presss//1119hhaasnn220 httpp/ppwaaaaaaarrrrrrrrrressaaa html.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving Deep into Rectifiers: Surpassing Human- Level Performance on ImageNet Classification. In ICCV '15, pp. 1026-1034, 2015. URL https://doi. 1/5/10.1109/ICCV 2015.121.
Mark Heimann, Haoming Shen, Tara Safavi, and Danai Koutra. Rogal: Representation Learning-Base Graph Alignment. In C1KM'18, pp. 117 126, 2018. URL https://ddi.orgg.e 1145/3269206. 3271788.
Keith Henderson, Brian Gallagher, Tina Eliassi-Rai Hanghang Tong, Sugato Basu, Leman Akoglu, Danai Koutra, Christos Faloutsos, and Lei Li. RolX: structural role extraction & mining in large graphs. Tn KDD'12, pp. 1231 1239, 2012. URL https://doi.
22t/://2333353...3773.. 2339723.
Nicholas J. Higham and Awad H. Al-Mohy, Computing matrix functions. Acta Numerica, 19:159-208, 2010.
URL attpp:///oi..oo org/ 1017 00077//896229999000000666
Arthur E. Hoerl and Robert W. Kennard. Ridge Regression: Biased Estimation for Nonorthogonal Problems.
Technometrics, 12(1):55 67, 1970. URL http://wwrt jstor org/stable//267335
Roger A. Horn and Charles Johnson. Matria Analysis. Cambridge University Press, edition, 2012. URL https://doi.oon org/ /0t1:/000007779991111001111
Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. GraphMAB: Self-Supervised Masked Graph Autoencoders. In KDD'22 pp. 594-604, 2022. URL https://doi.org/ 10. N0.1145/3534678.3888 3539321.
Zhenyu Hou, Yufci He, Yukuo Cen, Xiao Liu, Yuxiao Dong, Evgeny Kharlamov, and Jie Tang. GraphMAE2: A Decoudigg-Ehannced Masked Self-Supervised Graph Learner. In WWW'23, pp. 737 787-746, 2023. URL https: /etps://doi..m org/10 1145/3543507 3583379.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open Graph Benchmark: Datasets for Machine Learning Graphs. In NewrIPS '20, pp. 22118 22133, 2020. URL htttps://dol: org/ 10. 48550/arXiv 2000. 00687.
Aapo Hyvärinen, Jarmo Hurri, and Patrik O. Hoyer. Principal Components and Whilening, pp. 93-130. Springer London, 2009. URL
Di Jin, Ryan A. Rossi, Eunyee Koh, Sungchul Kim, Anup Rao, and Danai Koutra. Latent Network Sum- marization: Bridging Network Embedding and Summarization. In KDD'1, pp. 987-997, 2019. URL https://doi.org: http://....15//2200.3333000
17

---

Published in Transactions on Machine Learning Research (05/2025)
Junchen Jin, Mark Heimann, Di Jin, and Danai Koutra. Toward Understanding and Evaluating Structural Node Embeddings. ACM TKDD, 16(3), 2021. URL httpps//doi..or /tt..o////...///111155531330 1114//4481699.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. LBM: A Highly Efficient Gradient Boosting Decision Tree. In NewrlPS117, pp. 3149 3157, 2017.
URL https://dl. acm. 95t0///99..555/229999966 32950744.
Thomas N. Kipf and Max Welling. Semi-Supervised Classification with Graph Convolutional Networks. In
ICLR '17 2017. URL
Bryan Klimt and Yiming Yang. The Enron Corpus: A New Dataset for Email Classification Research. In Machine Learning: ECML 2004, pp. 217 226. Springer Berlin Heidelberg, 2004. URL https://doi..ooo org/ 10. 1-.077978-3-540--305 3--44-00011---_22
Jure Leskovec, Jon Kleinberg, and Christos Faloutsos. Graphs over time: densification laws, shrinking diameters and possible explanations. In KDD '05, pp. 177-187, 187, 2005, URL attps://doi.orgg 1145/
1081893.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper Insights Into Graph Convolutional Networks for Semi- Supervised Learning. In AAAI'18, 2018. URL attps://don htt////ii.og////////// hts//aaii.vvii..
Derek Lim, Felix Matthew Hohne, Xiuyu Li, Sijia Linda Huang, Vaishnavi Gupta, Omkar Prasad Bhalerao, and Ser-Nam Lim. Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods. In NewrlPS'21 2021. URL https://doi. org/10. 48550/arXiv 2110. 14446.
Yixin Liu, Yizhen Zheng, Daokun Zhang, Vincent Lee, and Shirui Pan. Beyond Smoothing: Unsupervised Graph Representation Learning with Edge Heterophily Discriminating. In AAAI'23 pp. 4516-4552 2023. URL https://doi. 10t/ (0000//aaa... v3714. 01774.25573.
Martin Q. Ma, Yue Zhao, Xiaorong Zhang, and Leman Akoglu. The Need for Unsupervised Outlier Model Se- lection: A Review and Evaluation Internal Evaluation Strategies, ACM SIGKDD Explovations Newslet- ter, 25(1):19-35 35, 2023. URL https://doi. /ttp:////ooo.o./..././//// http://.......00111111160/00.0.. 116.1///6667744366667777
Leland Mclnnes, John Healy, and James Melville. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. ArXiv e-prints, 2018. URL https://doi... org/ 10. 485500 arXiv.
Kevin P. Murphy. Machine Learning: A Probabilistic Perspective. //mitpress.nt www.tieee..mm... edu/9980022018802//
MIT press, 2012. URL https:
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In NeurlPs '19, pp. 8035, 2019. URL attps://ax acm. org. /tti//oo///a .tt////...55//42787.3344443 3455008.
Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-GCN: Geometric Graph Convolutional Networks. In ICLR 20, 2020. URL /ttps://openrevieet http://wwww..eeeeeeoooooo///aaaa
Bryan Perozzi, Vivek Kulkarni, Haochen Chen, and Steven Skiena. Don't walk, skip! online learning of multi-scale network embeddings. In ASONAM '17 pp. 258-265, 2017. URL https://doli oroeiiorgg1.... 1145/
Oleg Platonov, Denis Kuznedelev, Michael Diskin, Artem Babenko, and Liudmila Prokhorenkova. A critical look at evaluation of gnns under heterophily: Are we really making progress? In ICLR 22, 2023. URL
/ttpp://doi.orgg 4ttt//...885///......... 2302. 11640.
Daniel A. Roberts, Sho Yaida, and Boris Hanin. The Principles of Deep Learning Theory. Cambridge University Press, 2022. URL https///bepleemmnnnherr com.
18

---

Published in Transactions on Machine Learning Research (05/2025)
Emanuele Rossi, Bertrand Charpentier, Francesco Di Giovanni, Fabrizio Frasca, Stephan Günnemann, and Michael M Bronstein. Fdge Directionality Improves Learning Heterophilic Graphs. In LoG '23, 2023.
URL https://doi.orrg/00 http://o....rr1.1....886666/ 2349.10498.
Benedek Rozemberczki, Carl Allen, and Rik Sarkar. Multi-Scale attributed node embedding. Journal of Complex Networks, 9(2), 2021. URL https://doft www.ai.......ooo/eeeooooeoeeoooe 10930.....10991 hat/9//comet///aaaa cnab014.
Krzysztof Sadowski, Michal Szarmach, and Eddie Mattia. Dimensionality Reduction Meets Message Passing for Graph Node Emboddings. ArXiv e-prints, 2022. URL https://doi.or http:///....0r11...4555///////// 2202. 00408.
Vikram Saraph and Tijana Milenković. MAGNA: Maximizing Accuracy in Global Network Aligument. Bioinformatics, 20(20):29312294)) 2014. URL https://doi http:://............0.ooo/oooooa htt.://///oooooorriiiaaa//aaoaa
Konstantinos Skitsas, Karol Orlowski, Judith Hermanns, Davide Mottin, and Panagiotis Karras. Com- prehensive Evaluation of Algorithms for Unrestricted Graph Alignment. In EDBT'23, 2023. URL
https://doi. wwo//0..//0/........ 487.8/004488///// het.s//dd.../// edbt.2023.21 20.
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mci. Line: Lerge-Scale Information Network Embedding. In WWW '15, 2015. URL https //to///oii//i org/10. 1145/2736277 27772777777777998..
Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Mehdi Azabou, Eva L. Dyer, Remi Munos, Petar Veličkovié, and Michal Valko. Lerge-Seale Representation Learning on Graphs via Bootstrapping. In TCLR 22, 2022. URL https: / /ttp///o.ooo//// org/ 10. 48550/ aarivv//rriv../ 2102. 06514.
Lloyd N. Trefethen and David Bau. Numerical Linear Algebra. Society for Industrial and Applied Mathe- matics, 1997. URL https://epubs. siam. www....g/mmoo.iiiioooooiio.o.ooe
10. 1Tt.00111//11.911119977777
Petar Veličković, William Fedus, William L. Hamilton, Pictro Liò, Yoshua Bengio, and R Devon Hjelm. Deep Graph Infomax. In ICLR (19 2019. URL
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph Attention Networks. In ICLR '8, 2018. URL https://opennerree: http://wwwwwetttoooooooooo/or/oo
Haonan Wang, Jieyu Zhang, Qi Zhu, Wei Huang, Kenji Kawaguchi, and Xiaokui Xiao. Single-Pass Con- trastive Learning Can Work for Both Homophilic and Heterophilic Graph. Transactions on Machine Learning Research, 2023. URL
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying Graph Convolutional Networks. 1CML 19, pp. 6861 6871, 2019. URL arXiv. Stt.v..992.07155.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A Comprehensive Survey on Graph Neural Networks. IEEE Transactions on Neurul Nelworks and Learning Systems, 32(1): 4 24, 2021. URL attps://doi. 10. 1109/1111ISSSS 2020. 2978386.
Hongteng Xu, Dixin Luo, and Lawrence Carin. Scalable ing and Matching, In NewrlPS '19, 2019. URL http:://oggenerveeeeree///////od
Learning for Graph Partition-
Wenjian Yu, Yu Gu, Jian Li, Shenghua Liu, and Yaohang Li. Single-Pass PCA of Large Dign-nDimesiionl Data. In IJCAI '17, pp. 3350-3356, 2017. URL https://do1 wrw.o.og///oo 10tg/10.o0 24963/1jccai 2017/446.
Hengrui Zhang, Qitian Wu, Junchi Yan, David Wipf, and $ Yu Philip. From Canonical Correlation Analysis to Self-supervised Graph Neural Networks. In NewrIPS'21, 2021. URL https:///bii.org/1... arXiv. 2106 12484.
19

---

Published in Transactions on Machine Learning Research (05/2025)
ACC (K 4)
* PCAPass (K- 4)
ACC (K 10)
PCAPass (K 10)
108
10
10
10
10
10
A 10
10
10-1
10
0.0
0.5
1.0
0.0
0.5 i/pk
1/pk
1/pK
(a)
Arenas
(b) PPI
(c) Magna
10
10
10
10
10
0.0
0.5
1.0
0.0
0.5
10
6/pk
6/PK
(d) Enron
(e) Polblogs
Figure 7: The singular values spectrum for ACC and PCAPass on all graph aligmment datasots. Spectrums using both K and K 10 message-passing iterations are shown. The y-axes show the singular values, and the x-axes their index in descending order, normalized using the number of embedding dimensions pk-
A SVD of rank-deficient matrix
In Section 3.1, we stated the SVD of the matrix Z
X X E Rax2d as Z =UZZE where
(2E 0
VT
VT
Uz [U Tu].
Σg
V =
(14)
0
v2
r
r
and the block matrices Yu E Raxd Yv, E Raxd and Yv. E Rxd each have orthogonal columns and satisfy the conditions TTU = 0, r = I, r V r V = 0, and rT Tv. = r Tv. =I
Here, we verify the orthonormality of Uz and Vz via matrix multiplication:
ULUz- = Ut r
UV UT
0 = 0 I 1 VVV + VTV
U
Tul =
r N
1 VT
VT
V Yv
VT + VTTV T + r Tv.
I 0 0 I
VzVZ-
2
rt,
r
V Iv 2 [rr V + V
=
B Singular values of ACC and PCAPass
Figure 7 presents the singular value spectra for the ACC and PCAPass embedding matrices across all graph alignment datasets. Both models use a maximum embedding dimension of Pmax = 512. However, this does not result in the same final embedding dimension, Pk, for each model. To facilitate comparison, we normalize the singular value index i (x-axis) by the embedding dimension.
For the undirected graphs (Arenas, PPI, and Magna), the singular value gap indicating rank deficiency is clearly visible for PCAPass, with both K 4 and K 10 message-passing itcrations. On the dirccted graphs, this gap only appears for K 4, consistent with the results in Figure 4c. In contrast, ACC shows such singular value gaps, further confirming that it produces full-rank embeddings.
20

---

Published in Transactions on Machine Learning Research (05/2025)
Table 2a shows graph statistics for the graph alignment datasets. Specifically, it shows the number of nodes and edges, the number of weakly and strongly connected components, the global clustering coefficient, Cg, and the average path length, (4path). Table 2b shows basic information regarding the node classification datasets: the number of nodes, edges and node features, and the number of node classes.
Table 2:
(a) Graph alignment datasets and statistics.
(b) Node classification datasets.
DATASET
DIR.
#
CC
SCC
Cg
(ptth)
DATASET
m
FRAT.
CLs.
ARENAS
1.1K
11K
X
0.17
3.6
CHAMELEON
2.3K
36K
2325
PPI
3.9K
76K
X
35
0.09
3.1
SQUIRREL
5.2K
217K
2089
POLBLOGS
1.5K
19K
V
268
688
0.25
3.4
ROMAN EMPIRE
23K
33K
300
18
ENRON
7.9K
142K
V
58
861
0.16
3.5
ARXIV YEAR
169K
1.2M
128
MAGNA
1K
17K
X
0.62
5.5
SNAP PATENTS
2.9M
14M
269
C
Additional experiments information and results
C.1
Datasets
Table 2a provides statistics for the graph alignment datasets. Arenas (Guimera al., 2003) is an undirected email network, where each edge represents email communication between two students. Similar to Magna, PPI (Breitkreutz et al., 2007) is protein-protcin interaction graph, with nodes representing proteins and edges denoting interactions between them. Polblogs (Adamic & Glance, 2005) is a hyperlink graph of political blogs, while Enron (Klimt & Yang, 2004) is an email communication network, where each node corresponds to an email address. Specifically, we use a subgraph of the full Enron datasct for our experiments.
Table 2b summarizes the node classification datasets. These datasets were used in recent work by Rossi et al. (2023) on directed pessang-passing for supervised graph neural networks.
The Chameleon and Squirrel datasets are both hyperlink networks, where each node represents a Wikipedia article, and edges indicate hyperlinks between articles. Node features are binary variables indicating the presence of specific nouns, while labels reflect the average monthly traffic for cach webpage. Originally proposed by Rozemberczki et al. (2021) for regression tasks, they were later adapted for node classification by Pei et al. (2020).
The Roman Empire dataset, introduced by Platonov et al. (2023), is a word co-occurrence network basod the Wikipedia page for the Roman Empire. Nodes correspond to words, and edges represent synlactic dependencies between them, resulting in a graph that closely resembles chain structure. The node labels represent syntactic roles, while the features are word embeddings.
Arxiv Year and Snap Patents were introduced by Lim el al. (2021) to benchmark GNNs on large-scale graphs. The Arxiv Year dataset is derived from the OGB Arxiv citation network (Hu et al., 2020), where nodes represent papers and features are derived from their abstracts. Unlike OGB Arxiv, which uses subject areas for labels, Arxiv Year assigns labels based on the publication year.
Snap Patents is patent citation network, where nodes represent patents and edges indicate citations. Originally studied by Leskovec et al. (2005) to investigate the evolution of citation networks over time, the dataset used by Lim et al. (2021) assigns labels based on the year each patent was granted. Node features are generated from patent metadata.
C.2 Additional graph alignment results
Figure 8 extends the results shown in Figure 6 from the main paper, now including the Arenas, PPI, and Polblogs datasets. The figure presents graph alignment accuracies and run times across varying relative tolerances 0 for thresholding singular values.
The trends in accuracy and run time closely resemble those observed for Magna and Enron in Figure Specifically, PCAPass can match ACC's accuracy within a certain range of values, but its accuracy sharply drops to outside this range. In contrast, ACC maintains stable accuracy, only declining for very
21

---

Published in Transactions on Machine Learning Research (05/2025)
0.6
0.6
0.4
Accuracy
0.44
Accuracy
Accuracy
0.4
0.2
0.2
0.2
ACC PCAPass
ACC PCAPass
ACC
PCAPss
0.0
0.0
0.0
0
10-5
10-6
10-
10-3
10"
0
10-8
10-6
10
10-2
10
0
10-6
10-6 10
10-2
10
Relative tolerance, 0
Relative tolcrance, 0
Relative tolerance, 0
(a) Arenas, accuracy
(b) PPI, accuracy
(c) Polblogs, accuracy
0.450
0.7
ACC PCAPas
ACC PCAPass
ACC PCAPass
0.425
Run time ?
0.8
Run time
Run time
0.400
0.5
0.6
0.375
0.4
0.4
0.350
0
10-H
10
10
10
10
0
10
10
10
10
10
0
10-3
10
10 10"h 10
Relative tolerance, 0
Relative tolerance, 0
Relative tolerance,
(d) Arenas, run time
(e) PPI, run time
(f) Polblogs, run time
Figure 8: ACC and PCAPass for graph alignment with 15% noise edges, K = 10 for Arenas and PPI, and K = 4 for Polblogs. The x-axes show the relative tolerance 0 applied to remove dimensions with small singular values. Figures 8a to 8c show accuracy on the y-axis, while 8d to 8f show run time. Markers and shaded areas indicate the average and standard deviation over seeds.
high values of 0, which leads to excessive removal of embedding features. Additionally, ACC consistently outperforms PCAPass in terms of run time.
C.3 Node classification baselines
Below, we list the baselines used in our node classification experiment, including references to the respective model implementations and their licences. For models licensed under the MIT or Apache 2.0 licences, we also release our directed extensions as part of this paper's code repository. Additionally, we specify the default number of epochs used for training, as this directly influences the reported model run times. For other hyperparameter defaults, please refer to our code.
GAE (Kipf& & Welling, torch_geometric.c model..ll. GAE html, MIT Licence, 200 epochs.
DGI (Veličković et al., 2019):
MIT licence, 100 epochs.
MVGRL (Hassani & Khasahmadi, 2020): http:://gghhh..ccamaavvveaeeaaa No licence, 3000
epochs.
BGRL (Thakoor et al., 2022): httpss//ggthhh..oo///aassaa22/g/ Apache Licence 2.0, 10000 epochs.
CCA-SSG (Zhang et al., 2021): http:://gggtthh..oo////ggggggggg Apache Licence 2.0, 100 epochs.
GraphMAE (Hou et al., 2022): httpss//ggtabb.cco/////o///gg/pp MIT licence, 1000 epochs.
GraphMAEv2 (Hou et al., 2023): http:://gtaubb.oo///U////pappppp MIT licence, 1000 epochs.
GREET (Liu et al., 2023): https///ggaaub.oo///xxxxn//////2 MIT licence, 400 epochs.
SPGCL (Wang et al., 2023): http:://ggthaal.cc///aoaaaa//// No licence, 500 epochs.
22

---

Published in Transactions on Machine Learning Research (05/2025)
Table 3: Node classification results using a logistic regression classifier. OOM abbreviates out of memory. All Snap Patents results were gathered using CPU only as the SSGNNs exceeded our GPU memory limit. The top accuracies for each dataset are highlighted in bold.
MODEL
CHAMELEON
SQUIRREL
ROMAN EMPIRE
ARXIV YEAR
SNAP-PATENTS
ACCURACY
TIME
ACCURACY
TIME
ACCURACY
TIME
ACCURACY
TIME
ACCURACY
TIME
No MODEL, X
52.3
2.2
0M8
35.3 1.4
0MS
69.8 1 0.7
OMS
43.4 0.2
OMS
50.7 0.1
OMS
GAE!
54.8 + 2.6
118
38.2 + 1.9
1M 48
66.1 + 1.8
168
43.2 0.3
7M 348
49.3 + 0.1
3H 15M
DGT2
54.0 2.8
148
40.4 + 1.7
1M 408
76.5 + 0.7
4M 28
49.4 0.3
5H
TIMROUT
224H
MVGRL3
3.0
26M
44s
40.0 + 1.4
26M 32s
65.1 + 0.9
39M
138
0OM
2128 GB
OOM
2128 GB
BGRL
55.9
7M
49s
42.8 + 1.5
31M 45s
79.0 = 0.7
25M
24s
49.4
4n 52M
TIMEOUT
224H
BGRL-GS
58.6 2.1
31s 8M
41.6 1.7
33M 21s
78.9 + 0.7
21M
528
4.3
3H SM
TIMEOUT
224H
CCA-SSG"
59.5 +.7
48
41.6 1.4
7s
70.0 0.7
10s
50.9 0.3
1M 118
55.3 + 0.1
1H 40M
GRAPHMAE
278
43.5 1.7
59s
5..7
43.8
8M 55S
42.7 4.1
168
GRAPHMAEV2
65.1 +
368
40.4 2.1
1M 14s
0.9
1M 468
4.3
14M 535
23B
GRAPHMAEV2-GS
70.6 2.3
298
48.6
1M 68
80.2
1M 358
46.9 0.3
10M 46s
5.07
188
GREETS
53.9 1:
258
37.
1.5
6M 458
77.5
56M
OOM
1128
OOM
2128 GB
SPGCL"
58.2 2.3
168
42.5
1.5
1M 238
76.2 0.7
55s
48.2.00.3 0.3
52M
OOM
2128 CB
SGCN
49.8 + 2.7
374M8
35.3 + 1.2
723M8
39.2 + 0.7
393M8
4.2
18
42.3
268
PCAPASS"I
48.7 2.1
28
40.5
1.3
22s
77.6 + 0.7
852M8
49.3
28
54.5 0.1
2M 20s
ACC
60.7 + 2.5
18
44.1 + 1.4
18
79.3 + 0.6
432M8
49.4 + 0.3
18
56.6 + 0.0
278
1 KIPF & WELLING (2017)
2 VELIČKOVI ET AL. (2019)
HASSANI & KHASAHMADI (2020)
TTHAKOOR ET AL. (2022)
5 ZHANGG ET AL. (2021)
6 Hou ET AL, (2022) 7 Hou ET AL. (2023)
"Lu ET AL. (2023)
9 WANG ET AL. (2023)
10 WU ET AL. (2019) 11 SADOWSKI ET AL. (2022)
SGCN (Wu et al., 2019):
geometric conv. SGConv. html, MIT licence.
PCAPass (Sadowski et al., 2022): The original PCAPass implementation available at https://githi: htt/////yyztoo--maailllel//eeeee
under the MIT License. However, we use our reimplementation for this paper, available alongside ACC in our online code repository.
C.4 Node classification results using logistic regression
Table 3 presents the node classification test accuracies for each embedding model and dataset using a logistic regression classifier. The experimental setup heheewise follows the description in Section 5.2. We observe that the accuracies obtained with logistic regression are generally lower than those achieved using the gradient boosting classifier in the main paper (Table 1). This is expected, gradient boosting is more expressive and less biased model capable of capturing highly non-linear class boundaries.
Overall, ACC embeddings perform competitively against the SSGNNs also using the logistic regression classifier, achieving the highest accuracy on Snap Patents, the second-highest on Arxiv Year and Roman Empire, and ranking third on Squirrel and fourth on Chameleon.
On the Chameloon and Squirrel datascts, we observe a notable drop ACC's performance with logistic regression compared to gradient boosting: from 76.6% to 60.7% on Chameleon, and from 71.5% to 44.1% on Squirrel. A similar trend is seen for other linear embedding models like SGCN and PCAPass. This suggests that non-linear classification model is necessary to fully exploit the information in these embeddings.
In contrast, the performance gap is smaller for several SSGNNs. For instance, GraphMAEv2-CG MAMM--AEv--SS sees only a modest decline in accuracy from 74.1% to 70.6% on Chameleon. This indicates that the inherent non- lincarity of GNNs can compensate for the simplicity of logistic regression. However, the extent which this potential realized depends on the specific GNN architecture and training, as many SSGNN models still achieve lower accuracies than ACC with logistic regression.
23

---

Published in Transactions on Machine Learning Research (05/2025)
8
16
32
64
128
256
512
0.6
0.5
0.4
Accuracy
0.4
Accuracy
00.4
Accurccy
0.3
Accuracy
0.1
0.2
0.2
0.2
0.2
0.0
0
10
0.0
10
0.0
10
0.0
K
K
K
K
(a) Arenas
(b) PPI
(c) Magna
(d) Enron
10
K =
displacement
10-1
0.6
K
0.6
K 12
Accuracy
10
Accuracy
0.5
0.4
10-3
0.4
0.2
0.3
10
0.0
10
10a
10-6
10-6
10-1
10-2
10"
K
Eg - -iinuular value
Relative tolerance, 0
(e) Polblogs
(I) Polblogs, ACC, Pmax 512
(g) Polblogs, ACC, Pmax 512
Figure Figures 9a to 9e present the ACC graph aligmment accuracy (y-axis) for varying numbers of message-passing iterations K (x-axis) and embedding dimensions Pmax (indicated by hue and line style). Figures 9f and 9g provide deeper insights into the results for the Polblogs dataset. Similar to Figure 3c, Figure 9f plots the singular values of the embedding matrix (x-axis) against the average displacement along the corresponding principal axis, & (y-axis). For K = 6 and K = 12, the correlation between singular value and displacement observed at = 2 is disrupted, with some features exhibiting larger displacements than expected based on their singular values. These features cause the accuracy drop shown in Figure 9e. Finally, Figure 9g confirms that by removing the dimensions corresponding these high-displacement, low-singular- value features, the alignment accuracy can be restored.
D Effect of the number of message message-passing iterations and embedding dimensions
In this section, we investigate how the quality of ACC embeddings is influenced by the two primary hyperpa- rameters: the number of message-passing iterations, K, and the maximum embedding dimensionality, Pmax- To assess their impact, replicate the graph alignment and node classification experiments, measuring the accuracy of ACC embeddings across grid of K and Pmax values.
D.1 Graph Alignment
Figures 9a present the grid evaluation results for graph alignment, where the x-axes denote the number of passage-passing iterations, K. The hue and line style differentiate the Pmax values.
The first key obsorvation is that accuracy increases steadily with K for each undirected graph, eventually plateauing. This illustrates ACC's ability to preserve information across multiple message-passing iterations.
Secondly, for the undirected graphs, the value of Pmax appears to have no significant effect. This is because the initial number of features, d = 2, matches the minimum compression dimensionality, Cmin = 2, for these datascts. Therefore, regardless of the value of Pmax, the resulting ACC embeddings will have dimensionality
Pk = 2 (K+1) 1).
24

---

Published in Transactions on Machine Learning Research (05/2025)
On the directed graphs, we observe different behaviour. Starting with Enron in Figure 9d, when Pmax is sulficiently high, the accuracy follows similar pattern to the undirected graphs increasing steadily before levelling off. However, for Pmax E 8, 16, 32, a different trend emerges: the accuracy initially increases, but then decreases, eventually aligning with the curve for Pmax =
This seemingly unusual behaviour stems from the formula used to determine the number of compression dimensions, c c= max ((Pmax/ + 1)J, Cmin), which in turn defines the final embedding dimensionality, pk = c. (K + 1). As K increases, this formula can lead to decrease in the final embedding dimensionality, p. Consequently, more information must be compressed into fewer dimensions, resulting in loss of information and a corresponding drop in accuracy. However, once K + 1 becomes a factor of Pmaxx, the value of pk increases again, restoring some of the lost accuracy.
For example, with Pmax = 8, we obtain po = and p1 = 8 for K = and K = 1, but then P2 = 6 for K = 2, before increasing again to Ps 2==8 8. Beyond K 2 4, the formula sets c= Cmin = 2, which why the curve for Pmax Pmax E 16, 32, shown in Figure 9d.
converges with the curve for pmax
A similar explanation applies to the curves for
In Figure 9c, we observe the same effect for Puax E {8, 16, 32] on Polblogs as we did for Enron. However, we also notice a distinctly different behaviour: as K increases, the accuracy begins to drop for Pmax 2 64.
This decline in accuracy is neither due to information loss from compression nor rank deficiency. As seen in Figure 7e, the singular value spectrum for ACC on Polblogs with K = 10 does not exhibit any singular value gaps. Instead, the drop in accuracy occurs because increasing K generates embedding features that are disproportionately noisy.
We demonstrate this effect in Figure 9f. Similar to Figures 30 and 3d in the main paper, Figure 96 plots the singular values of the embedding matrix against the average displacement along the corresponding principal axis due to noise from graph aligmment. As a reminded from the main paper, let ( represent the embedding matrix for graph 91, and (2) the embedding matrix for the noisy graph G2. Using the singular value decomposition U, Σ, VT = (1), where V contains the principal axes for 2(1), the x-axis shows the singular values Mi,s, and the y-axis shows the displacement & (1 V.,ll2 along the ith principal axis.
For K = 2, we observe that the displacement 6; is proportional to the singular values E,,,, consistent with our observations across the other four graph alignment datasets. However, for K = 6 and K = 10, this linear correlation breaks down, and features with disproportionately large displacements emerge. This is seen as the curvature of the point clouds in Figure 9f.
We further verify that these high-displacement features cause the accuracy drop by applying singular value thresholding. The results for K = 12 and pmax = 512 are shown in Figure 9g. As can be seen, the alignment accuracy improves dramatically, from 30% to 60%, once the dimensions with small singular values and high displacements are removed.
We do not yet fully understand why the high-displacement dimensions appear in the Polblogs embeddings. Looking at the graph statistics in Table 2a, two potential causes stand out: the large number of weakly connected components and the high global clustering coefficient for Polblogs. However, we can rule out the former, as we observe the same behaviour when running the graph alignment experiment on the largest connected component of Polblogs. This leaves the high global clustering coefficient as the most likely cause. Further research is needed to verify this hypothesis and to explore the underlying mechanisms that might lead to the emergence of these high-displacement dimensions.
D.2 Node classification
Figure 10 illustrates the effect of K and Pmax on ACC node classification accuracies, measured using logistic regression classifier. Regarding Pmax, we observe a consistent increase in accuracy across all values of K. This is expected, as higher embedding dimensionality allows the embeddings to capture more information, facilitating better classification performance.
The effect of increasing the number of messago- passggggppsnnggggg iterations K depends on the embedding dimensionality Pmax- When Pmax is sufficiently large, classification accuracy rises and eventually plateaus for all four datasets.
25

---

Published in Transactions on Machine Learning Research (05/2025)
64
128
256
512
1024
2048
0.8
0.6
0.6
Accuracy
Accuracy
Accuracy
Accuracy
0..5
0.4
0.4
0.4
0.2
0.4
02
0.0
0.3
K
K
K
(a) Chameleon
(b) Squirrel
(c) Roman Empire
(d) Arxiv Year
Figure 10: These figures the ACC node classification accuracy (y-axis) for various number of musssage-assiing
iterations K (x-axis), and embedding dimensions Pmax (hue and style).
0.8
0.8
134
Acoureyy
0.6
Accuray
Accuracy
0.3
0.6
0.4
0.4
0.2
0.2
0.2
10
0.3
10
10
0.0
K
K
K
(a) Roman Empire
(b) Arxiv Year
(c) Magna
(d) Enron
Figure 11: These figures show the SGCN (Wu et al., 2019) node classification accuracy (Figures 11a and 11b) and graph alignment accuracy (Figures 1lc and 11d) the y-axis, plotted against the number of message-passing iterations K (x-axis). The legends indicate the number of SGCN embedding dimensions, which are always equal to the number of input features d.
However, when Praxx too small, accuracy can decline as increases. This effect particularly notable for the Roman Empire dataset, as seen in Figure 10c.
The drop in accuracy is due to the increased need for compression K grows, meaning that less information from each scale of the graph is retained. Specifically, when K = 0, the initial d features are compressed into Pmax embedding dimensions, but when K = 12, these features are compressed into only [Pmax/ /13] dimensions. The loss of information due to this increased compression results in an accuracy decline for datasets where local features (i.e., small K) are especially important for classification. The Roman Empire datasct exemplifies this behaviour, where the information contained X, ApX, and Ag is critical for achieving high accuracy.
D.3 Over-smoothing in SGCN
To highlight the advantage of the concatenation update used by ACC, we provide results using SGCN (Wu et al., 2019) in Figure 11. SGCN employs summation rather than concatenation to update its embeddings. This results loss of information with each message-passing iteration, leading over-smoothing (Li al., 2018; Chen et al., 2020a).
This issue is particularly noticeable the Roman Empire dataset, where the accuracy of SGCN drops from 70% to 10% as K increases. In contrast, ACC's accuracy remains stable as long a sufficiently high embedding dimension is used, as shown in Figure 10c.
Additionally, SGCN embeddings maintain a fixed dimension for all message- message-passing iterations, PK = d. This limitation hampers graph alignment accuracy, as it prevents the integration of information from different scales to form more distinct emboddings. Consequently, the alignment accuracy for SGCN lower, evidenced in Figures 11 and 11d.
26

---

Published in Transactions on Machine Learning Research (05/2025)
ACC
PCAPass
CONE
S-GWL
FUGAL
ACC-FUGAL
1.0
1.0
Accuracy
Accuracy
Accurnay
Accurncy
Accurary
0.5
65
6.6
0.0
0
10
0.0
0.0
0.0
0.0
K
RC
K
K
K
(a) Arenas
(b) PPI
(c) Enron
(d) Polblogs
(e) Magna
Figure 12: Graph alignment accuracy comparison between ACC, PCAPass, and graph aligument baselines
under 15% edge noise. The x-axis denotes the mumber of message-passing iterations used for ACC and PCAPass. For each method (CONE, S-GWL, FUGAL, ACC-FUGAL), markers and shaded regions indicate the mean and standard deviation over three random seeds. The results for ACC and PCAPass are repeated from Figure 5, where five were used.
E Comparison to state-of-the-art graph alignment methods
In this section, we compare the graph aligmment accuracy of ACC against three state-of-the-art graph alignment algorithms: S-GWL (Xu et al., 2019), CONE (Chen et al., 2020b), and FUGAL (Bommakanti et al., 2024). None of these methods natively support directed graphs, which is a limitation shared by most recent graph alignment algorithms (Skitsas et al., 2023). Fortunately, S-GWL naturally generalizes to directod graphs, and FUGAL can be extended in a straightforward manner since it is node baaoure Specifically, we compute node features on both the original and transposed graphs, and concalenate the resulting feature vectors input to FUGAL's optimization protocol. This simple modification enables FUGAL to process directed graphs. In contrast, adapting CONE to directed graphs is non-trivial due to its architecture. For all methods, we adopt the recommended hyperparameter settings from Skitsas et al. (2023).
Figure 12 shows the previously reported graph alignment accuracy results for ACC and PCAPass (with whitening), now presented alongside the newly added results for each baseline method. The x-axis represents the number of meessagepassing iterations K used for ACC and PCAPass. The corresponding run times for all methods provided in Figure 13.
We begin our comparison by examining ACC alongside CONE, which is the weakest of the three baseline methods, As expected, CONE fails entirely on the directed graphs Enron and Polblogs, reflecting its lack of support for directed edges. On the undirected datasets, CONE and ACC achieve similar levels of accuracy: CONE slightly outperforms ACC on Arenas, ACC performs better on Magna, and the two models yield identical results on PPI. However, when it comes to computational efficiency, ACC holds clear advantage. While CONE exhibits quadratic time complexity in the number of nodes, O(n²), ACC operates with linear complexity, as discussed in Section 4.1. In practice, this difference is substantial and ACC is approximately 100 times faster than CONE, shown in Figure 13.
S-GWL is the strongest overall baseline in terms of alignment accuracy, achieving the highest performance on Arenas, PPI, and Polblogs. This result is consistent with the benchmark evaluation by Skitsas et al. (2023), where S-GWL was identified as the top-performing method.
However, S-GWL also by far the most computationally expensive of the three basclines, as shown Figure 13. On the Enron dataset, S-GWL failed to converge even after two hours of run time, at which point we terminated its execution. In contrast, ACC is highly efficient; approximately 1000 times faster across the five datasets. This efficiency stems from two key factors. First, S-GWI has a time complexity of O(n" l) (Skitsas et al., 2023), while ACC scales linearly with the number of nodes. Second, S-GWL is an iterative optimization-basel method, whereas ACC generates embeddings in a single forward pass.
FUGAL the second-best bascline in terms of alignment accuracy, achicving the highest performance on Magna and tying with S-GWL for best results on Arenas. Notably, our extension of FUGAL to directed graphs allows it to perform well on Polblogs, where its accuracy closely matches that of both S-GWL and
27

---

Published in Transactions on Machine Learning Research (05/2025)
ACC
PCAPass
CONE
S-GWL
FUGAL
ACC-FUGAL
10
Ram time (6)
101
Run time (6)
Run (6)
10
(6)
100
Run (6)
10
107
10P
10
10-1
10
10
10
10
K
K
K
K
RC
(a) Arenas
(b) PPI
(c) Enron
(d) Polblogs
(e) Magna
Figure 13: Run time comparison of ACC and PCAPass to graph aligument baselines. The x-axis shows the number of pesssage-assiing iterations K and the y-axis the algorithm run time in seconds. Markers and shaded areas represent the average and standard deviation over three scods for the graph alignment baselines, and five seeds for ACC and PCAPass.
ACC. However, FUGAL fails completely on PPI and Enron, achieving only trivial alignment. accuracy. In contrast, ACC maintains high accuracy on both datasets while also being significantly more efficient, FUGAL has cubic time complexity, O(n²).
To understand why FUGAL fails on PPT and Enron, we examine the graph statistics in Table 2a. These two graphs are simultaneously the densest in the benchmark and exhibit the lowest global clustering coefficients. While Bommakanti et al. (2024) showed that FUGAL performs well across varying graph densitics, thoir experiments were conducted in noise-free free settings. We therefore hypothesize that the combination of edge- removal noise and high density poses challenges for FUGAL.
This limitation could stem from either FUGAL's initial feature extraction step or its subsequent optimization procedure. To investigate, we replace FUGAL's node features with ACC embeddings computed using K = 6 and Pmax = 512. The resulting method, denoted ACC-FUGAL, is shown in Figures 12 and 13. As illustrated, ACC-FUGAL also fails on PPI and Enron, indicating that the bottleneck lies in FUGAL's optimization stage rather than its input features.
On the other three datasets, however, ACC-FUGAL achieves the highest accuracy among all methods. This suggests that, if FUGAL's limitations dense graphs can be addressed, combining ACC embeddings with the FUGAL optimization framework could yield highly effective alignment algorithm. In the meantime, using ACC on its own offers a compelling alternative as it delivers strong accuracy with far greater efficiency, and scaling well to large graphs due to its lincar time complexity.
F Analysis of node classification accuracy on the Squirrel dataset
In our node classification benchmark results using the gradient boosting classifier, as shown in Table 1, ACC achieves an accuracy of 72% on the Squirrel dataset, whereas PCAPass achieves only 52%. In this section, we explore the source of this discrepancy by analysing the Squirrel dataset and comparing the features generated by message aggregation and embedding aggregation.
To identify the key features contributing high classification accuracy on Squirrel, we perform a single ceessage-passing iteration to obtain three feature matrices: X Bxd A,X Raxd and A,X Raxd We refer to these feature groups.
By training a gradient boosting classifier on each feature group with an 80-20 training-test split, we find the following test. accuracies: 51% for X, 33% for A₃X,, and 84% for AgX. These results indicate that the features in Ag are particularly crucial for achieving high classification accuracy on the Squirrel dataset.
Next, we investigate how well each feature group preserved in the PCAPass and ACC embeddings. Ignoring the column centring step of PCA, can express the PCAPass embeddings after one message-passing
28

---

Published in Transactions on Machine Learning Research (05/2025)
iteration
Wx
- = A,X, A,X XWx A;XW; AgXWB,
W
WF
(15)
W
where W E _ddxP the projection matrix learned via PCA. We divide this matrix vertically into three sub- matrices: Wx E Rdxp, WF E Rdxp, and WB Rdxp, These matrices compress X, ArX, and Ag respec- tively. By analysing these matrices,
assess how much information from each feature group is preserved.
Specifically, for cach of the 512 features, we compute the proportion of cach feature group that contributes to the embedding. Since each column in W has unit norm, these proportions can be calculated as follows:
d Wx = Σ ( 2=1
WF
WF 1
Wa
(WB)
2
(16)
1=1
1=1
where wx, WF, and W3 denote the proportions of each feature group represented in the embeddings. Note that Wx + WF + Wa = 1p. where 1 is a length-p vector of oncs.
We can perform similar analysis for ACC. In this case, the embeddings are given by Z [M MW). where M XVx and
V/ V
M (1) = (Ar A = V + AVVVV
(17)
VE
Here, the matrices Vx Iade, Vx E Raxe, and Vx Vo E Rdxe are used to compress the three feature groups.
An important difference from PCAPass that Vx is computed separately via PCA from V₂ and V.. mcaning that Vx forms an orthogonal basis by itself, i.e., VIVx = I.. Consequently, the column norms of Vx Vj are equal to the column norms in V3, and similarly for Vx V2 and Vg. This can be demonstrated by considering the norm of the ith column in Vx Ve:
Vx V VV.. V VoL. VI.. = VV....12.
(18)
Therefore, the ACC proportion vectors are
vx= (Vx..) lo:
VF Σ (Fi..)
VB
V Bi.:
(19)
where VF + VB=
1c.
In Figure 14, we visualize the proportion vectors as heat maps. The heat maps on the left show the PCAPass vectors, Wx, Wy, and Wa, while the heat maps on the right display the ACC vectors, Va and Vg. The colours in the heat maps represent the proportion of information derived from each feature group. The bright colours in the first. two rows for PCAPass indicate that most of the information captured from the features X and A In contrast, ACC captures more information from AgX, as evidenced by the more uniform colour distribution in its heat map.
We can further quantify this difference by computing the effective number of features extracted from each feature group. We denote this p(=ff) where the star is either X, F, or for each respective feature group. These quantities are computed the sum of each proportion vector:
Px
w
(efl) Pe
WFk:
PB
(eff)
WB
(20)
k=
for PCAPass, and
(eff) Px
ex
256,
Df
(eff)
UFki
PB
(eff)
Uski
(21)
1=1
k=
k=
29

---

Published in Transactions on Machine Learning Research (05/2025)
1.0
1.0
X
0.8
0.8
F
0.6
0.6
Model
Px
(eff)
PF
(eff)
Pb
(cff)
P
0.4
PCAPass
309
140
63
B
ACC
256
160
96
0.2
0.2
43
86
129
172
21.5
258
301
344
387
430
473
0.0
22
44
66
88
110
132
154
176
198
220
242
0.0
(a) PCAPass
(b) ACC
Figure 14: Visualization of the projection matrices used in the first passing iteration for PCAPass and ACC. Each column represents embedding dimension, while each row corresponds to one of three fea- ture groups: the input features X, the forward aggregation features ArX, and the backward aggregation features AgX. The colour indicates the proportion each feature group represented in each embedding dimen- sion. These proportions are calculated using Equations 16 and 19.
Table The effective number of embedding dimensions used per feature group after one message- passing iteration using ACC and PCAPass. The effective dimen- sions are calculated using Equa- tions 20 and 21.
for ACC. Note that Px
(eff) 256 since ACC always includes features per message-passing iteration.
The effective number of embedding dimensions is shown in Table 4. Compared to PCAPass, ACC effectively uses 53 fewer features from X and 33 more features from A which represents more than a 50% increase compared to 63 for PCAPass. The inclusion of these class- informanfree features explains the higher accuracy achieved by message aggregation compared to embedding aggregation on the Squirrel datasct.
G Experimental investigation of rank deficiency in self-supervised graph neural networks
G.1 Singular value spectra and graph alignment experiments
In this section, present experimental results supporting the discussion in Section 6 on rank deficiency in self-supervised graph neural networks (SSGNNs). As described there, we evaluate SSGNN versions of PCAPass and ACC, trained by minimizing the GraphMAEv2 loss (Hou et al., 2023), and assess their behaviour after 0, 20, and 200 training epochs.
We first revisit the experiments from Section B and compute the singular value spectra of the resulting embeddings for K = 4 and K 10. The results are shown in Figure 15. Comparing the spectra of "CAPass-GNY to those of the linear PCAPass model in Figure 7, observe that PCAPass-GND does not exhibit rank deficiency, as there is no clear spectral gap. This confirms that the inclusion of non-linearities in the model prevents the explicit rank collapse seen in the linear setting.
The next question is whether the absence of rank deficiency in PCAPass-GNt implies improved embedding quality. To investigate this, we repeat the graph alignment experiments from Section 5.1 using the learned embeddings of both PCAPass-GNM and ACC-GNN.
The results are shown in Figure 16. Notably, we observe significant increase in variance across random seeds for both PCAPass-GNV GNN and ACC-GNN, compared to their lincar counterparts. This increased variability stems from the random initialization of the learnable weight matrices introduced in Equations 12 and 13, which introduces greater sensitivity to seed selection.
In terms of accuracy, PCAPass-GNT substantially outperforms the original lincar PCAPass model (cf. Figure 5), suggesting that resolving rank deficiency does indeed improve embedding quality. However, performance degrades with increased training: PCAPass-Or trained for 200 epochs performs poorly across
30

---

Published in Transactions on Machine Learning Research (05/2025)
ACC-GNN.0
ACC-GNN:20
ACC- 200:
PCAPass GNNoo
PCAPass 2CN GNN220
PCAPass GNN
10
10
10
10F
10
10
65
27
10
10"
10
10-5 8.
7
0.0
0.5
10"7 7
0.5
10
1.0
10-7
1.0
s/pk
k/pK
s/px
1/px
(a) Arenas, K = 4
(b) Arenas, K = 10
(c) PPI, =4
(d) PPI, K = 10
10
109
10
10
28 109
2 109
10-
10-3
10
10
10 0.0
0.5
1.0
10" 0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
13
s/pg
i/pR
8/P8
4/pK
(e) Magna, K = 4
(I) Magna, K = 10
(4) Enron, K = 4
(h) Enron, K 10
107
10
10a
10a
A 10-1
oT 10-1
10-5
10-3
0.0
0.5
1.0
0.0
0.5
1.0
1/Pk
3/pw
(i) Polblogs, K 4
(j) Polblogs, K = 10
Figure 15: The singular values spectrum for the SSGNN versions of ACC and PCAPass on all graph alignment datascts using K = and K = 10. The colours correspond to 0, 20 and 200 training epochs for ACC- GNN and PCAPass- GNNN respectively. The y-axes show the singular values, and the x-axes their index in descending order, normalized using the number of embedding dimensions PK-
all datasets. Additionally, the number of message-passing iterations K increases, PCAPass-GNa initially improves, reaching a peak accuracy, but then consistently declines beyond a certain point.
In contrast, ACC-GNN exhibits greater robustness. Its accuracy does not degrade with increasing training epochs or message- meessgg-passsng iterations on the undirected datasets. The only dataset where increased training leads to a noticeable drop in accuracy is Enron, see ACC-GNN:200, On Polblogs, accuracy decreases with increasing K, reflecting trend also observed for the linear ACC model, as discussed in Section D.1.
Despite the absence of a clear spectral gap, we can still relate the behaviour of the graph alignment accuracies to the singular value spectra of PCAPass- and ACC-GNN. To this end, we consider the condition number x ()) of the embedding matrix Z, defined as the ratio between its largest and smallest singular values:
max Z
R( Z)
(22)
min (
The condition number is elosely related to the numerical sensitivity of matrix operations, particularly matrix inversion (Higham & Al-Mohy, 2010, Ch. 5.8). In general, higher condition numbers indicate greater sensitivity and potential numerical instability. While rank deficiency often leads to high condition number, the converse does not necessarily hold: a matrix can have a large condition number even in the absence of an explicit spectral gap.
Table reports the condition numbers of PCAPass-GNT and ACC-GNN for K = and K 10, across training epochs 0, 20, and 200, and for each dataset. For comparison, we also include the condition numbers of the linear PaAPass and ACC embeddings.
31

---

Published in Transactions on Machine Learning Research (05/2025)
ACC-GNN:20
ACC- 200:
PCAPass GNNo
PCAPass 2N GNN:20
PCAPass GNN:200
0.6
0:75
0.
Accumey
0.4
Accurary
00.4
Accunyy
6.50
Accuncy
Accuney
0.4
0.2
1222
0:20
0.2
0.0
K
(a) Arenas
(b) PPI
(c) Enron
(d) Polblogs
(e) Magna
Figure 16: SSGNN versions of ACC and PCAPass for graph alignment with 15% noise edges. The x-axis
shows the number of meessage-psssingg iterations K. The colour shades represent distinct number of training epochs. Markers and shaded areas represent the average and standard deviation over 3 seeds.
The general (rend is that, for a fixed value of K, embeddings with lower condition numbers x tend to yield higher graph alignment accuracies. This pattern is clearly observed for ACC and PCAPass on the undirected datascts (Arenas, PPI, and Magna), where ACC exhibits significantly lower к values alongside substantially higher alignment performance. A similar relationship holds for Enron and Polblogs at 4. At K 10, however, the condition numbers and aligmment accuracies of ACC and PCAPass become more comparable, reflecting the effects of deeper message-passenn passing and greater compression. These observations align with the rank deficiency analysis presented in the main paper.
A similar trend appears in the GNN variants. For 200 training epochs, PCAPass-GNN typically exhibits much higher condition numbers than ACC-GNN, and correspondingly lower aligument accuracies. At 0 epochs of training, their values are more similar, and so are their accuracies, suggesting that weight optimization in PCAPass PaN amplifies instability in the embeddings.
Nonetheless, there are notable exceptions to this trend. For instance, on Polblogs at K = 10, PCAPass-GNt
with 0 epochs has much lower condition number than ACC-GNN, but does not oulperform it in accuracy. Additionally, both ACC and ACC-GNN tend to have higher condition numbers at K = 10 than at K = 4, yet their accuracies also increase, highlighting that x is not a sufficient standalone prodictor of embedding quality.
In conclusion, while the SSGNN variant of PCA Pass is not explicitly rank-deficient, there appears to be a cor- relation between information redundancy induced by its embedding aggregation and concatenation strategy, elevated condition numbers, and reduced embedding quality. These issucs mitigated by ACC's message aggregation approach, both in linear and non-linear settings. However, analysing the non-linear GNN case remains significantly more complex and calls for deeper theoretical and empirical investigation in future work.
G.2 Node classification results for ACC-GNN and PCAPass GNN
We also evaluate node classification accuracy for both ACC-GNN and PCAPass-GNNN Results using a gradient boosting classifier are reported in Table 6, and those using logistic regression are shown in Table 7.
Table 5: The condition number of the embedding matrices for ACC and PCAPass, and the ACC-GNN and PCAPass-GNT models for 0, 20 and 200 epochs of training. Displayed values are log x(()).
(a) K 4
(b) K 10
MOUEL
AsteNas
PPI
MAGIN
EsBOS
POLILOGS
MODEL
AHEZAS
PPI
MAGNA
POLILOUS
ACC
1.83
2.23
1.84
2.
3.41
ACC
4.34
4.70
4.26
4.41
5.20
PCAPASS
8.61
8.82
8.53
9.16
0.54
PCAPASS
9.35
9.32
0.74
3.06
3.85
ACC-GNN-0
2.29
2.09
1.62
4.49
4.72
ACC-GNN:0
4.87
5.14
4.31
7.33
7.71
PCA NA
3.99
4.06
4.23
3.38
3.63
PCAPASS-GNN-
4.80
4.96
4.94
4.14
4.60
ACC-GNN-20
2.67
1.96
3.27
4.50
5.12
ACC-GNN:21
4.54
5.46
4.11
7.46
8.14
PC
6.53
5.82
5.64
6.32
7.36
PCAPASS G-N-20
4.88
7.81
8.70
7.65
7.96
ACC-GNN-200
1.98
2.38
2.26
4.35
4.63
ACC-GNN:200
5.78
4.71
4.24
5.58
9.16
PAN
5.41
5.48
5.81
8.03
9.34
8.69
9.46
5.95
7.52
32

---

Published in Transactions on Machine Learning Research (05/2025)
Table 6: Gradient boosting node classification results for SSGNN versions of ACC and PCAPass. The results for ACC and the best performing SSGNNs from Table are included for reference. The highest accuracy is highlighted in bold.
MODEL
CHAMBLEON
SQUIRREI
ROMAN EMPIRE
ARXIV YEAR
SNAP-PATENTS
ACCURACY
TIME
ACCURACY
TIME
ACCURACY
TIME
AACCRAAYY
TIME
ACCURACY
TIME
No MODEL,
64.6 + 2.3
0MS
52.1 1.4
0MS
70.4 0.7
0MS
44.77
0M8
56.1 0.1
0MS
CCA-SSG!
71.0 2.1
48
59.9 + 1.2
7s
63.7 + 0.7
10s
48.4 0.2
1M 11s
56.1 + 0.0
In 40M
GRAPHMAEV2-GES
74.1 +: 2.3
296
57.0 1.4
1M 68
80.0 + 0.7
1M 358
46.3 + 0.3
10M 468
53.8 + 0.1
18H
ACC
76.6 + 1.9
1s
771.5 + 1.3
1s
8815 + 0.6
432MS
|9.4 0.3
1s
62.6 02.6:000 0.1
27s
ACC-GNN
73.8 1
906MS
66.7 1: 1.4
18
78.0 1
47.9 0.2
88
62.3 1: 0.1
11M
PCAPASS-GNN
66.1 1 2.1
989MS
45.0 1.5
18
66.2 T 0.9
28
45.2 0.3
168
60.0
0.1
16M 17s
ZHANG ET AL. (2021)
2 Hou ET AL. (2023)
Table 7: Logistic regression node classification results for SSGNN versions of ACC and PCAPass. The rosults for ACC and the best performing SSGNNs from Table are included for reference. The highest accuracy is highlighted in bold.
MODEL
CHAMELEON
SQUIRREL
ROMAN EMPIRE
ARXIV YEAR
SNAP-PATENTS
ACCURACY
TIME
ACCURACY
TIME
ACCURACY
TIME
ACCURACY
TIME
ACCURACY
TIME
No MODEL,
52.3 2.2
OMS
35.3 1.4
UMS
69.8 0.7
OMS
43.4
0MS
50.7 0.1
OMS
CCA-SSG'
59.5 + 2.7
48
41.6 1.4
78
70.0 0.7
10s
50.9 0.3
1M 11s
55.3 0.1
1n 40M
GRAPHMAEV2-GS
70.6 + 2.3
296
48.6 1.6
1M 68
80.2 = 0.7
1M 358
46.9 + 0.3
10M 468
54.7
0.0
18H
ACC
60.7 1 2.5
1s
44.1 1 1.4
18
79.3 1 0.6
432Ms
49.4 0.3
18
56.6 1. 0.0
27s
ACC-GNN
57.8 + 3.0
906MS|
42.2 + 1.7
18
77.7 + 0.6
18
48.7 + 0.4
88
61.3 + 0.1
11M 238
PCAPASS-GNN
56.0 + 2.9
989MS
39.0 + 2.0
1s
74.4 : 0.8
28
47.7 + 0.4
16s
60.6 + 0.1
16M 17s
1 ZZHANG ET AL. (2021)
2 Hou ET AL. (2023)
For ease of comparison, also include the best-performing results from the main paper. All experiments were conducted with 20 training epochs and consistent hyperparameter settings: K 2 and Pmax 512.
Consistent with earlier findings, ACC-GNN generally outperforms PCAPass-GNV across datasets, suggest- ing that message aggregation produces higher-quality embeddings than embedding aggregation, even in the presence of non-linearity. Furthermore, although self-supervised graph neural networks (SSGNNs) are theo- retically more expressive, they do not outperform ACC in practice under default hyperparameters. Due their reliance on iterative gradient descent, these models also significantly loss efficient, particularly large-scale datasets such as Snap-Palents.
H Hyperparameter tuning of self-supervised graph neural network baselines
For the node classification benchmarks presented in Section 5.2, we fix the number of message- passing iterations to K = 2 and the embedding dimension to 512 across all embedding models, while keeping all other hyperparameters at their default settings. This ensures a fair comparison by standardizing two of the most influential factors across models.
Since K and p have well-documented impact on embedding quality, as demonstrated for ACC Scction D, using consistent values for these parameters eliminates variation due to their influence. We selected K = and p 512 based on their frequent use in prior work (Veličković et al., 2019; Zhang et al,, 2021; Thakoor et al., 2022; Hou et al., 2022; Wang et al., 2023). For all other hyperparameters, which vary between models, we use the authors' default configurations, as these have typically been optimized and validated during the original model development.
An alternative approach would be to perform hyperparameter tuning and select the best configuration for each model and dataset. However, there are several reasons to avoid this strategy. First, hyperparameter tuning is notoriously difficult. in real-world unsupervised learning scenarios (Ma et al., 2023). This difficulty
33

---

Published in Transactions on Machine Learning Research (05/2025)
arises from the fact that tuning typically relics on ground-truth labels to guide the selection process. In un- supervised learning, however, such label information is not available by design. Indeed, the absence of labels is precisely why unsupervised methods are used in the first place. Consequently, evaluating unsupervised embedding models under their default hyperparameters provides a more realistic and practical assessment of their utility in real-world applications.
Second, hyperparameter tuning is computationally expensive. Each self-supervised graph neural network (SSGNN) includes numerous hyperparamcters, and evaluating different combinations requires training and validating the model multiple times. Ideally, this process must also be repeated over several random seeds to account for initialization variance. As shown in Table 3, training SSGNNs is significantly more time- consuming than training ACC. Thus, comprehensive hyperparameter tuning would require substantial com- pute resources and cloud GPU hours, which are unfortunately beyond the financial means of the authors.
While comprehensive hyperparameter tuning is infcasible, we conduct a limited tuning experiment to esti- mate the patential accuracy gains that SSGNNs might achieve with optimized settings. Specifically, we tune the embedding dimension p, which we previously found to significantly influence node classification accuracy for ACC (Figure 10), and the number of training epochs, common tuning parameter in SSGNN stud- ies (Veličković et al., 2019; Hassani & Khasahmadi, 2020; Zhang et al., 2021; Hou et al., 2022; 2023; Wang et al., 2023). To reduce computational overhead, we use logistic regression rather than gradient boosting for evaluation, as the latter substantially more expensive to train and validate.
The classification accuracy results for each baseline model and dataset are presented in Figures 17 and 18. Each coloured line represents different embedding dimension p, and the x-axis shows the number of training epochs. The orange marker highlights the best-performing configuration observed during the sweep, while the black dashed line indicates the accuracy reported in Table 3 using p 512 and the model's default nunber of training epochs (see Section C.3). Figures 19 and 20 show the corresponding loss values across training.
The primary observation from the tuning experiment is that, for the majority of models and datascts, accu- racy lends to increase with the embedding dimension p. This expected: larger embedding space allows the classifier to model more complex decision boundaries, reducing bias. The accompanying increase in variance is mitigated by the large number of training samples and the of regularization in the classifier (Bishop, 2006, Ch. 3.2). However, increasing p also leads to greater computational cost. These findings support decision to evaluate all models using a common embedding dimensionalityy of p 512, as comparing models with different values would introduce unfair advantages to those using nigher-dimensions embeddings.
That said, we also observe that for GAE, GraphMAE, and GraphMAEv2, performance degrades when p = 2048 on the Chameleon and Squirrel datasets. In these cases, accuracy drops noticeably compared to p 512, and this drop is accompanied by sharp increase in loss values. We speculate that this behaviour stems from these models' reliance autoencoding losses, which may overfit or become unstable when the embedding space is too large relative to the dataset size. This hypothesis is supported by the fact that Chameleon and Squirrel are the two smallest datasets in terms of node count. However, further investigation is required to confirm the underlying cause.
Regarding the number of training epochs, we find no consistent benefit from deviating from the default values. For most models with p = 512, accuracy cither peaks near the default epoch sctting or platcaus around the same value. The exceptions are BGRL and MVGRL, where accuracy improves with longer training. Notably, their default epoch values- 10,000 for BGRL and 3,000 for MVGRL- the maximum number of epochs tested in our sweep, suggesting that the defaults are already well-chosen in these cases.
34

---

Published in Transactions on Machine Learning Research (05/2025)
64
256
512
1024
2048
Best p & epoch
Table (p 512)
Accuracy
0.6
Accuracy
0.4
0.75
Accuracy
0.75
Accuracy
0.5
0.50
0.4
0.3
0.4
1000
1000
1000
1000
Epoch
Epoch
Epoch
Epoch
GAF., Chameleon
GAE, Squirrel
GAE, Roman Empire
GAE,, Arxiv Year
0.6
0.50
Accuracy
Accuracy
0.4
Accuracy
0.75
Accuracy
0.45
0.4
0.3
0.50
1000
1000
1000
1000
Epoch
Epoch
Epoch
Epoch
DGL, Chameleon
DGL, Squirrel
DGI, Roman Empize
DGL, Arxiv Year
0.6
Accuracy
Accuracy
0.4
Accuracy
0.75
Accuracy
0.475
0.450
0.4
0.3
0.50
0.425
1000
1000
1000
1000
Epoch
Epoch
Epoch
Epoch
BGRL, Chameleon
BGRL, Squirrel
BGRL, Roman Empire
BGRL, Arxiv Year
Accuracy
0.6
Accuracy
0.4
Accuracy
Accuracy
0.45
0.6
0.3
0.4
0.40
1000
1000
1000
1000
Epoch
Epoch
Epoch
Epoch
BGRL-GS, Chameleon
BGRL-GS, Squirrel
BGRL-GS, Roman Empire
BGRL-GS, Arxiv Year
0.5
0.7
Accuracy
0.6
Accuracy
Accuracy
0.7
Accuracy
0.50
0.4
0.6
0.5
0.3
0.4
0.45
1000
1000
1000
0
1000
Epoch
Epoch
Epoch
Epoch
CCA-SSG, Chameleon
CCA-SSG, Squirrel
CCA-SSG, Roman Empire
CCA-SSG, Arxiv Year
Figure 17: Node classification accuracy across training epochs for GAE, DGI, BGRL, BGRL-GS, and CCA- SSG. Each subplot corresponds to a dataset, while line colours and marker shapes represent the embedding dimension p. The orange marker indicates the highest observed accuracy across all settings. The black dashed line shows the accuracy reported in Table for p = 512 and the model's default number of training epochs. Shaded areas represent standard deviations computed over five repeated 5-fold cross vols-valio runs for the classifier, using five different random seeds for the embedding model.
35

---

Published in Transactions on Machine Learning Research (05/2025)
p & epoch
Table 512)
0.45
Accuracy 0.5
Accuracy
Accuracy
0.6
Accuracy
0.6
0.4
0.40
0.3
0.4
1000
1000
1000
1000
Epoch
Epoch
Epoch
Epoch
GraphMAE, Chameleon
GraphMAE, Squirrel
GraphMAE, Roman Empire
GraphMAE, Arxiv Year
Accuracy
Accuracy
Accuracy
0.6
Accuracy
0.45
0.6
0.4
0.40
0.3
0.4
0.4
1000
1000
1000
1000
Epoch
Epoch
Epoch
Epoch
GraphMAEv2, Chameleon
GraphMAEv2, Squirrel
GraphMAEv2, Roman Empire
GraphMAEv2, Arxiv Year
Accuracy
0.75
Accuracy
Accuracy
0.8
Accuracy
0.45
0.50
0.50
0.6
0.40
1000
0.25 0)
1000
1000
0
1000
Epoch
Epoch
Epoch
Epoch
GraphMABv2-GS, Chameleon
GraphMAEv2-GS, Squirrel
GraphMABV2-GS, Roman Em-
GraphMABv2-GS, Arxiv Year
pire
Accuracy
Accuracy
Accuracy
0.7
Accuracy
0.6
Accuracy
0.4
0.5
0.6
0.4
0.3
0.5
0.4
0.3
1000
1000
1000
1000
1000
Epoch
Epoch
Epoch
Epoch
Epoch
MVGRL, Chameleon
MVGRL, Squirrel
MVGRL, Roman Empire
GREET, Chameleon
GREET, Squirrel
0.8
Accuracy
Accuracy
Accuracy
0.4
Accuracy
0.75
0.6
0.4
0.50
1000
1000
1000
1000
Epoch
Epoch
Epoch
Epach
GREET, Roman Empire
SPGCL, Chameleon
SPGCL, Squirel
SPGCL, Roman Empire
Figure 18: Node classification accuracy across training epochs for GraphMAE, GraphMAEv2, GraphMAEv2 GS, MVGRL, GREET, and SPGCL. Each subplot corresponds to a dataset, while line colours and marker shapes represent. the embedding dimension p. The orange marker indicates the highest observed accuracy across all settings. The black dashed line shows the accuracy reported in Table 3 for p = 512 and the model's default number of training epochs. Shaded areas represent standard deviations computed over five repeated 5-fold cross vols-valiiotion runs for the classifier, using five different random seeds for the embedding model.
36

---

Published in Transactions on Machine Learning Research (05/2025)
64
128
256
512
1024
2048
Loss
20
Loss
20
Loss
20
Loss
20
0
1000
1000
1000
1000
Epoch
Epoch
Epoch
Epoch
GAF, Chamelron
GAF, Squirrel
GAF, Roman Empire
GAE, Arxiv Year
10000
500
Loss
1000
Loss
Loss
10000
Loss
0
0
0
0
1000
0
1000
0
1000
0
1000
Epoch
Epoch
Epoch
Epoch
DGL, Chamelenn
DGI, Squirre
DGL, Roman Empire
DGL, Arxiv Year
2
2
2
Loss
Loss
Loss
Loss
1000
1000
1000
0
1000
Epoch
Epoch
Epoch
Epoch
BGILL Chameleon
BGRLL, Squirrel
BGRL, Roman Empire
BGRLL Arxiv Year
2
2
2
2
Loss
Loss
Loss
Loss
1
1000
1000
1000
1000
Epoch
Epoch
Epoch
Epoch
BGRL-GS, Chameleon
BGRL-GS, Squirrel
BGRL-GS, Roman Empire
BGRL-GS, Arxiv Year
-250
Loss
Loss
Loss
Loss
-1000
-500
-750
-2000
-2000
1000
0
1000
0
1000
1000
Epoch
Epoch
Epoch
Epoch
CCA-SSG, Chameleon
CCA-SSG, Squirrel
CCA-SSG, Roman Empire
CCA-SSG, Arxiv Year
Figure 19: Loss curves across training epochs for GAF DGI, BGRL, BGRL-GS, CCA-SSG. Each subplot corresponds to a dataset, while line colours represent the embedding dimension p. Shaded areas represent standard deviations over 5 different seeds.
37

---

Published in Transactions on Machine Learning Research (05/2025)
64
128
256
512
1024
2048
1.0
1.0
1.0
1.0
Loss
Loss
Loss
Loss
0.5
0.5
0.5
0.5
1000
1000
1000
1000
Epoch
Epoch
Epoch
Epoch
GraphMAE, Chameleon
GraphMAE, Squirrel
GraphMAE, Roman Empire
GraphMAE, Arxiv Yeur
2
2
2
Loss
Loss
Loss
Loss
1000
1000
1000
1000
Epoch
Epoch
Epoch
Epoch
GraphMAEv2, Chameleon
GraphMAEv2, Squirrel
GraphMAEv2, Roman Empire
GraphMAEv2, Arxiv Year
2
2
2
2
Loss
Loss
Loss
Loss
1000
1000
1000
1000
Epoch
Epoch
Epoch
Epoch
GraphMAFY2-GGS Chameleon
GraphMAFv2-GGS Squirrel
TTaphMAEv2-CC2 Roman Fm-
MAR Arxiv Year
pire
8.5
Loss
Loss
Loss
Loss
Loss
8.0
1000
1000
1000
1000
1000
Epoch
Epoch
Epoch
Epoch
Epoch
MVGRL,, Chameleon
MVGRL, Squirrel
MVGRL, Empire
GREET, Chameleon
GREET, Squirrel
10
Loss
Loss
Loss
50
Loss
50
1000
1000
1000
1000
Epoch
Epoch
Epoch
Epoch
GREET, Roman Empire
SPGCL, Chameleon
SPGCL, Squirrel
SPGCL, Roman Empire
Figure 20: Loss curves across training epochs for GraphMAE, GraphMAEv2, GraphMAEV2-GS, MVGRL. GREET, and SPGCL. Each subplot corresponds to dataset, while line colours represent the embedding dimension p. Shaded areas represent standard deviations over 5 different seeds.
38


Reviews:
[{"review_id": "X6nCvaLpAj", "reviewer": "Reviewer_DC2u", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengthnesses:\n1. This paper provides novel theoretical discussion about the rank deficiency due to repeated embedding aggregation and concatenation in existing linear message-passing models (e.g., PCAPass). And it derives the lower bound $\\rho_k \\geq  (2^k − k − 1)d$\n2. The proposed ACC novelly shifts the aggregation process from embedding aggregation to message aggregation, preventing redundancy and ensuring full-rank embeddings.\n3. The empirical evaluation on graph alignment and node classification shows both the effectiveness and the computational efficiency of the propose ACC.\n\n### Weaknesses:\n1. This paper adopts the transductive setting, where the entire graph is available during training. However, many real-world applications involve new nodes appearing dynamically such as social networks and recommendation systems.\n2. The time complexity analysis of the proposed method is missing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper identifies the rank deficiency problem in existing linear models. This paper presents formal proof of how embedding aggregation leads to rank deficiency.\n\nThen this paper introduces ACC (Aggregate, Compress, Concatenate), a novel linear message-passing model designed to overcome the issue of rank deficiency in node embeddings for directed graphs. The proposed ACC ensures embeddings remain full rank, improving alignment and classification tasks without additional computational cost.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. discuss the time complexity of the proposed method\n2. discuss the inductive setting", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8uNhu6pbDS", "reviewer": "Reviewer_iHLV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Weakness:\n\nThe paper proves redundancy through a simple example of duplicated columns, but doesn't rigorously prove why PCAPass necessarily leads to rank deficiency in the general case. It leaves open the possibility that PCAPass's rank deficiency might be implementation-dependent rather than fundamental.\n\nThe hyperparameters chosed in the paper could be further discussed. For example, do optimal hyperparameter values differ between tasks (e.g., graph alignment vs. node classification)?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper:\n\nIdentifies and analyzes a fundamental flaw (rank deficiency) in linear message-passing models that use embedding aggregation and concatenation, particularly in PCAPass.\n\nIntroduces ACC (Aggregate, Compress, Concatenate).\n\nProvides theoretical analysis of why rank deficiency occurs in previous approaches and how ACC resolves this issue.\n\nDemonstrates empirically that ACC is 70-270x faster than existing methods while maintaining competitive or superior accuracy on large-scale graphs.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Consider including guidelines for selecting optimal hyperparameters.\n\nInclude more detailed analysis of how different compression strategies might affect performance.\n\nVerify the proof of rank deficiency in general case.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CJveiR6BZA", "reviewer": "Reviewer_7mXH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- This paper is well-written and the presentation is clear.\n- The proposed method solved the rank deficiency issue in PCAPass and can learn more stable node representations.\n\n**Weakness**\n- Traditional self-supervised graph neural networks already perform very well. This work is an improvement on PCAPass [1]. So from my perspective, even the groundwork for this paper, PCAPass, has *little practical application*.\n- The experiment settings in 'Section 5.2 Node classification: ACC vs self-supervised graph neural networks' do not make sense to me. The authors claim that 'We do not perform hyperparameter tuning, as it is well-established that SSGNNs can surpass linear models\nlike ACC in accuracy with sufficient tuning.............' However it is not practical for them to use the default hyperparameter set on all datasets. The comparison is not fair.\n\n[1] Sadowski, Krzysztof, Michał Szarmach, and Eddie Mattia. \"Dimensionality reduction meets message passing for graph node embeddings.\" arXiv preprint arXiv:2202.00408 (2022).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors aim to solve the rank deficiency in one previous method PCAPass. It first proves the rank deficiency existed in PCAPass and then improves it to eliminate this effect. Experiments show the effectiveness of the proposed method compared to PCAPass.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please kindly refer to the Strengths And Weaknesses section. I suggest the authors tune self-supervised graph neural networks.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper identifies a problem with a previously presented unsupervised node embedding algorithm, PCAPass: while concatenation of multi scale embeddings helps avoid over-smoothing, it can lead to rank deficiency, which can negatively impact performance. The paper proposes an alternative approach that avoids this problem.

The reviewers were somewhat lukewarm on the paper due to its potential lack of impact. I agree with the reviewers that the impact may be small, since the field is dominated by SSGNNs, and since PCAPass is not a commonly used approach. However, per TMLR guidelines, this is not sufficient justification to reject the paper. I believe the paper will be of interest to some in the field, and disagree with the idea that there is no value to large-scale graph methods that are not inductive.

I would like to see some expansions of the discussion/analysis, however:
1. Does the concatenation issue identified in PCAPass also impact SSGNNs that use concatenation?
2. How does the (big-O) computational cost compare with PCAPass?
3. For the graph alignment section, it is hard to see whether ACC is performing "well", vs "better than PCAPass". I would like to see a strong baseline here (you don't necessarily need to beat a strong baseline, but it would help to know whether this method would be a good choice for graph alignment).

I am also sympathetic to Reviewer 7mXH's concern about the hyper parameters of the SSGNNs. I would consider performing a lightweight hyper parameter search if computational budget allows, or including published results on the same datasets.
Decision: Accept (poster)