Node2ket: Efficient High-Dimensional Network Embedding in Quantum Hilbert Space
Authors: Hao Xiong
Yehui Tang
Yunlin He
Wei Tan
Junchi Yan
Venue: ICLR 2024 poster (2024)
Keywords: high-dimensional network embedding
quantum mechanics
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Network embedding (NE) is a prominent technique for network analysis where the nodes are represented as vectorized embeddings in a continuous space. Existing works tend to resort to the low-dimensional embedding space for efficiency and less risk of over-fitting. In this paper, we explore a new NE paradigm whose embedding dimension goes exponentially high w.r.t. the number of parameters, yet being very efficient and effective. Specifically, the node embeddings are represented as product states that lie in a super high-dimensional (e.g. $2^{32}$-dim) quantum Hilbert space, with a carefully designed optimization approach to guarantee the robustness to work in different scenarios. In the experiments, we show diverse virtues of our methods, including but not limited to: the overwhelming performance on downstream tasks against conventional low-dimensional NE baselines with the similar amount of computing resources, the super high efficiency for a fixed low embedding dimension (e.g. 512) with less than 1/200 memory usage, the robustness when equipped with different objectives and sampling strategies as a fundamental tool for future NE research. As a relatively unexplored topic in literature, the high-dimensional NE paradigm is demonstrated effective both experimentally and theoretically.

Published as a conference paper at ICLR 2024
NODE2KET: EFFICIENT HIGH-DIMENSIONAL NET- WORK EMBEDDING IN QUANTUM HILBERT SPACE
Hao Xiong, Yehui Tang, Yunlin He, Wei Tan, Junchi Yan"
Department of Computer Science and Engineering, Shanghai Jiao Tong University (taxuexh, yehuitang, anonimit.y wilson. yaw.aaannnjsssseeeeeee @ajjt.......... edu. https://gtt /ttt::///tttbbb com/ ShwwWh/ncde22eee...e
ABSTRACT
Network embedding (NE) is a prominent technique for network analysis where the nodes represented as vectorized embeddings in continuous space. Ex- isting works tend to resort to the low-dimensional embedding space for efficiency and less risk of over-fitting. In this paper, explore new NE paradigm whose embedding dimension goes exponentially high w.r.t the number parameters, yet being very efficient and effective. Specifically, the node embeddings are rep- resented as product states that lie in super nigh-dimensional (e.g. 232 2d2-dim) quan- tum Hilbert space, with carefully designed optimization approach to guarantee the robustness to work in different scenarios. In the experiments, we show diverse virtues of our methods, including but not limited to: the overwhelming perfor- mance on downstream tasks against conventional low-dimensional NE baselines with the similar amount of computing resources, the super high efficiency for a fixed low embedding dimension (e.g. 512) with less than 1/200 memory usage, the robustness when equipped with different objectives and sampling strategies fundamental tool for future NE research. As relatively unexplored topic in literature, the high-dimensional NE paradigm is demonstrated effective both experimentally and theoretically.
Hilbert space is big place!
-Carlton M. Caves, cited in Quantum Computation and Quantum Information, 10th Anniversary Edition. Massachusetts Institute of Technology, 2010.
INTRODUCTION
Network embedding (NE), has been extensively studied for decades. By representing nodes in networks as semantically informative embeddings that are mathematically continuous vectors of fixed dimension (called embeddings), NE makes it possible to apply advanced machine learning algorithms to network data (Perozzi et al., 2014). Numerous studies have shown that by encoding semantic features (c.g. neighborhood information, topology similarity, etc.) into node embeddings, machine learning (ML) approaches can significantly outperform conventional non-ML methods on various downstream tasks in network analysis including node classification (Tang et al., 2015; Per- ozzi et al., 2014), link prediction (Wang et al., 2016), network alignment (Liu et al., 2016; Man et al., 2016), community detection (Keikha et al., 2018), etc. Different from the current popular graph neural networks that require node attributes as the input (Kipf & Welling, 2016; Hamilton et al., 2017), NE plays an unsubstitutable role in dealing with non-attributed networks.
Low-dimensional embedding is natural idea for NE which starts from the perspective of dimension reduction (Belkin & Niyogi, 2002; Roweis & Saul, 2000; Yan et al., 2006). It has been well studied academically in the decades (Tang al., 2015; Perozzi et al., 2014; Grover & Leskovec, 2016), and widely adopted in industry due to the good efficiency and scalability (Zhang et al., 201 2018a). Although NE has become basic principle in literature (Goyal & Ferrara, 2018), there is the problem that low-dimension embedding is merely a low-rank approximation of some information matrix describing how nodes interact that is usually high-rank. Some works
oiw-dimeniional (e.g.
*Correspondance author. This work was in part supported by NSPC (62222607, 72342023) and SITU Trans-med Awards Research (STAR) 20210106.

---

Published as a conference paper at ICLR 2024
(Goyal & Ferrara, 2018; Yin & Shen, 2018) tried increasing the embedding directly but ended with a common finding that high-dimensional embeddings suffer from the problem of over-fitting and the subsequent descent of model performance in the inference tasks. Another drawback of a high emhedding dimension is that the high computational complexity scales linearly to the dimension, which afterward would cause problems such increasing training time and reducing inference efficiency (Yin & Shen, 2018; Wu et al., 2016). In this paper, we are going to overturn these common belief' by presenting a brand new NE paradigm which is high-dimensional but endowed with a low computational complexity (Sec. 3), competent in the preservation of high-rank information (Sec. 5.2), showing good performance on both information preservation and inference (Sec. 6).
Embedding in the exponentially large quantum Hilbert space was firstly proposed in word2ket' (Panahi et al., 2020) for model quantization. By representing the word embeddings as 'pseudo' entangled states (see Appendix E.2 for interpretation of 'pseudo" entangled quantum state) whose dimension grows exponentially as the amount of used parameters, it achieves to save parameters in the embedding model. Since the adopted neural decoder suffers a high complexity for high- dimensional embeddings, word2ket is restricted in a low-dimensional embedding space where the dimension is up 8,000 reported in experiments (much smaller than the vocabulary size). Though high-dimensional embedding is not explored in word2ket (Panahi et al., 2020), it sheds light on a new representation form that uses fewer parameters than the dimension to construct embeddings.
Compared with entangled states, product states can generate the same high-dimensional embedding with fewer parameters, with the complexity linear to the number of parameters in computing the fidelity (equivalent to inner product). Therefore, in the proposed high-dimensional NE paradigm node2ket, we represent node embeddings as the product states (Sec. 3) and adopt inner product- based objectives for embedding training to achieve a low complexity. We also find that extra con- straints are necessary since the tensorized embedding brings instability to model learning. These constraints include that all the embeddings are normalized (for stability) and the inner product be- tween embeddings should be positive (for interpretability, see Sec. 4.1 for more details). Due to these constraints, the origin SGD which empirically works well for previous conventional NE meth- ods (Tang et al., 2015; Perozzi et al., 2014; Grover & Leskovee, 2016) would fail in embedding optimization of node2ket. To make the optimization procedure work and improve the robustness of the embedding learning process under different possible application scenarios while balancing the running efficiency, we develop optimizer for stable learning of nigh-dimensional NE, which learns embeddings on the unit hypersphere by Riemannian optimization, keeps monitoring inner products between embeddings and makes them positive, and updates embeddings by Adagrad in asynchronous batches (Sec. 4). Based on node2ket, we propose a variant node2ket+ which intro- duces parameter sharing in the embedding construction step of node2ket (Sec. 3.1). Compared with existing works of NE, node2ket+ can explicitly encode semantic information and is endowed with a significantly higher time/space efficiency. The main contributions are:
1) At the algorithm level, we open the path to efficient high -igh--imensional embedding which has been rarely studied in NE literature before, and specifically devise two NE methods node2ket and node2ket+ as a starting point. We theoretically analyze the essence of our methods from the per- spective of implicit matrix factorization and give a sufficient condition when produces full-rank approximation of the latent information matrix of the given network.
2) At the engineering level, we implement and integrate all the proposed techniques into a C++ li- brary named as LIBN2K, which is fundamental toolkit for efficient high-dimensional NE algorithm development with the following features as robust tool for future research in NE:
Flexible and robust. It is highly flexible with plug-in modules including flexible embedding con- struction, sampling strategies, objectives, two types of embedding architectures. The developed optimizer designed to deal with the proposed two necessary constraints ensures that the system runs stably and robustly in different scenarios.
Highly efficient and scalable. It runs in distributed manner with multiple CPU threads, which enables it to handle large-scale networks on a single machine. The only-CPU style makes it easy be deployed on most devices where only CPUs are available.
Compatible with both networks and sequences. Considering that most NE methods either learn from a highly struccurrd network or implicitly structured node sequences, it accepts both the two
'ket' is a representation of quantum states, see Appendix E. for more details.
2

---

Published as a conference paper at ICLR 2024
Low- Oimenninnlla
High-Drmansional
mave corpoct higher infermatirn explcity enanded
Conventional Embedding
b) WordSkel
NodeZket
NodeSkate
Emboddding Formulation
2. the rumber of cach rode
Use
X. R
Fesleddag of
Contralnt Juch = Mhe,,
X, R
Marc
Conntraiat
(B Re) > 0,76,
Qsaetum
perada craargted
queam eyarms
Enbeddiig Optimitaation
2.
Ercal Ercal& peptart
Time & space (lle/011))
Alp compang
Time & Aapace complesiy: (lp))
Stop Drmp
6 1 E 00
coepes oreGAR
Tisc-omminn
qure
Dettoestreer by wonterteg feaer
12
redfaces pliing 20021
poodent deny glvn
Time space conglesityy Oly)
Forward plopog wordfow
53 1011 aeikkii
Oip lake
Transing pistine rodee www.-eedduttteee otjechive
1ar prodect vollec
dinerrional
Lose rininiiiaiaaall
Quarrns
Hab-diraaieel with low coupleskyy LIINIK: fendanee koikker for afficisnn high divnansionnl
Piporestisl lecrasing complesit
NI sggether oprant mesrach
Figure I: Technical points of node2ket compared with conventional embedding and word2ket.
types of data as inputs. Learning from node sequences in streams also further decreases memory usage since the whole network is not loaded into the memory.
3) Experiments are conducted on downstream tasks including network reconstruction, link predic- tion, and node classification. We show the overwhelming performance of our methods that much fewer parameters to achieve the best scores among all the baselines. The high running speed, low memory usage, and low local space consumption for embedding storage especially for large-scale networks are impressive. Note our setting follows the widely adopted node attribute-free free protocol which in fact can avoid privacy breaking, and thus GNNs not compared done in peer literature.
RETROSPECT OF EXISTING METHODS
We define a non-attributed network as G (V, where V { is the node set and & S V x V is the edge set. NE aims to represent each node i E V as continuous vector. Suppose that for each node i we use p parameters to represent the embedding x,. Based on the above framework, different existing embedding methods can be described as follows.
Conventional Embedding, Fig. a). Embedding x; is directly constructed as a vector x; E R with the p parameters (p & N). Because of its simplicity and low time & space complexity that are linear to for model optimization, NE methods based upon the convention embedding are usually easy to be implemented, time-efficient and scalable. With different objectives and sampling strategies over diverse types of data and application scenarios, the conventional embedding is adopted as the default in all works in NE to the best of our knowledge (see Appendix B for related works).
Word2ket, Fig. 1 b). Divide the p parameters into R rows and C' columns, then each block on r-th row e-th column contains p/RC parameters formulating a 'sub-embedding" Uire E RP/RCC The node embedding is constructed by x; = E X c=1 Uire E R(e/RCy where @ denotes the tensor product (equal to outer product here). In physical view, the formulation of x; looks like an entangled quantum state but not true one (details in Appendix E.2). It is more precise to be interpreted as a rank-R order-C CP Decomposition in mathematics (Hitchcock, 1927). During the model optimization, a concrete x; should be obtained as the input to the following neural network- based decoder. That causes two problems: i) High time & space complexity: an O(p/ time complexity for the decoder to process x, and an (((pRC))) space complexity to store the temporal variable x; in the memory; ii) Low embedding dimension: Due to the high complexity, the seemingly high embedding dimension should be low enough afford. Furthermore, because x; is not strictly a quantum state and neural networks are not coantum--fiendde word2ket is still a quantum-inspired algorithm but not one that can be implemented by quantum computing.
3 EMBEDDING CONSTRUCTION BY NODE2KET
Embedding Construction (shown in Fig. 1 c). We divide the p parameters (referred in See. 2) into C' columns, with each column containing p/C' parameters which defines a sub-embedding uie E RP/C Then the embedding is constructed by x; 0== Uie R(//C) With the constraint
3

---

Published as a conference paper at ICLR 2024
||uell22 introduced in Sec. 4.1, embedding by node2ket will be strict 'product state' following
the normalization rule ||x||| = 1. The constraint distinguishes node2ket's embeddings from the embeddings of rank-1 1 version of word2ket.
The (pp) Time and Space Complexity in Training. To achieve the O(p) complexity in training, the objective function must be based the inner produet, which is very loose condition and indeed very common in NE objectives, e.g. the Skip-Gram with Negative Sampling (SGNS) and the
Marginal Triplets (MT) (see Appendix C.2 for details). The O(p) complexity of node2ket comes from the convenience in inner product computation which is given by:
xx) = II (uee uje?,
(I)
where each (Uie, Uje) takes O(p/C) time. Since the embedding x; and xj are not computed, the space complexity is also (p). Furthermore, in other cases such as computing Euclidean distance, the time and space complexity are also O(p) by ||x; x,l12 = V (x,,xx x + (x,, 2(x,,x))
3.1 node2ket+: MORE COMPACT WITH SEMANTIC INFORMATION EXPLICITLY ENCODED
Conventionally, semantic information (c.g. various orders of proximity (Cao et al., 2015), structural similarity (Ribeiro et al., 2017), etc.) are preserved by designing fancy sampling strategy that implicitly defines nodes' similarities in the embedding space. As node2ket divides the parameters into sub-embeddings, provides an opportunity to explicitly encode semantic information in the step of embedding construction, which is achieved by parameter sharing between nodes that share common information. As illustrated in Fig. d), if we have found two semantically proximate nodes and j. then we can share the sub-emba uub-embeddings between them. To technically achieve that, we need assign each node C indices to retrieve the eubeembeddings. We use a table I(i, c) to record the index of c-th sub-embedding of node i. Then if T(i, c) T(J, c), embeddings the same e-th sub-embedding. In the example of Fig. d), we share the parameters of u22 and u22 by fixing [(i, = [ 2). The parameter sharing scheme of node2ket+ would also reduce space overhead. According to this point, we apply node2ket+ on compressive network embedding.
and x; shares
3.2 APPLY NODE2KET+ ON COMPRESSIVE NETWORK EMBEDDING
One promising applications of node2ket+ is compressive NE (CNE) especially for devices where only limited computational resources are available. By parameter sharing, the memory usage and local space for embedding storage can be significantly reduced. One feasible approach for CNE to equip node2ket+ with graph partition algorithms,, e.g. Louvain partition (Blondel et al., 2008), to generate [(i, c) table. Louvain partition algorithm is a fast and scalable unsupervised partition algorithm that can divide a network into parts with a given resolution A higher resolution will result in a larger number of partitions. To explore community features of different granularity, we run multiple times of Louvain partition with different resolutions. If two nodes and are divided into a common community in the c-th time of Louvain partition, then we set I(i, c) = [jj.c).
The following Proposition 1 shows the lower bound of the number of parameters to train fixed g-dimensional NE. The proof is given in Appendix D. As a concrete example, when N = 10 and q 100, by setting C 4, we only need 400 parameters to store the sub-embeddings and 4 x 10 to store index table (i, c), far less than conventional embedding methods that require 10 parameters.
Proposition 1 (Capability of Node2ket+ on Embedding Compression). To train q-dimensional em- beddings for N nodes, if it requires obtaining N different embeddings (x,} by node2ket+, we need at least C(Ng)=/C parameters to store sub- sub-embeddinggs and NC parameters to store the index table ci, where C the number of sub-embedddings for each node.
4 LEARNING SUB-EMBEDDINGS UNDER CONSTRAINTS
In general, a typical NE algorithm can be formalized as iterations of the following steps (illustrated in Fig. 4, Appendix C): i) sample node pair (i. J) the positive sample by some strategy P(i,j(C) (e.g. directly sample from edges), ii) compute the gradients according to the objective Oii,j) de- fined over i and j. and iii) update node embeddings given gradients. Hence the complete objective of such an NE algorithm can be described as follows:
maximize ((s))P((ss ((,,))
(2)
4

---

Published as a conference paper at ICLR 2024
Great efforts of research in NE have been paid in designing expressive objectives, and exploring network structures by diverse task-oriented sampling strategies (see Appendix B for related works). We do not pay incremental efforts these two aspects, but aim to provide general approach for model optimization that works for any inner product-based objectives. By 'inner product-based objective' we refer to objective that is function of inner products between one multiple em- bedding pairs. In this paper, we investigate two sampling strategies, random walk (RW) and random walk with restart (RWR) (details in Appendix C.I), and two inner product-based objective functions, skip-gram with negative sampling (SGNS) and marginal triplets (MT) (details in Appendix C.2).
In the following part of this section, we will discuss two constraints in embedding learning, for which we will also show how to optimize the derived constrained objective.
4.1 CONSTRAINTS IN EMBEDDING LEARNING
The Normalization Constraint ||u:c.le l2 = 1, Vi, With the constraint, the embedding can be- come a strict quantum state (see Appendix E.3), which makes the algorithm quantum-friendif since the embedding can be loaded into a quantum machine without further normalization and the inner product can be obtained by computing the fidelity of quantum states, e.g. by simple swap test (Barenco et al., 1997) or quantum matrix multiplication (Shao, 2018). The constraint is also empir- ically necessary for stable embedding learning, preventing the sub-embeddings from collapsing to zero vectors that would make the program crash.
The Positive Inner Product Constraint (use: Uja) > O, J. In Eq. 1, the inner product of two embeddings (x), can be decomposed into the product of the inner products of the sub- embeddings (Uie; Uje)- So it is rather likely to happen that 'two negatives make a positive', e.g. (U,1, (U1) < 0, (U,2, Uz2) < 0 yielding (x,, x,} = (u,11 U51) (ug, U22) > O. Such cases bring chaos in model optimization and also make the embedding results less interpretable for visualization.
The Objective Under Constraints. The objective Eq. under the above two constraints now is:
maximize ((ss)-P(ass))(((....) (i.,),
s.t. ||liell2 1, Vi,e, and (urer UjO) > 0, V.,
4.2 GRADIENT COMPUTING
Note that the gradient computing here is very different from the explicit forward/backward propa- gation as done in the deep learning frameworks such as Pytorch and Tensorflow. Specifically, we compute the gradients of uub-embedding U, directly through mathematical derivation. In this way, the step of computing the loss can be skipped thus accelerating training. And most importantly, the computation of x; is skipped, which makes training embeddings in the super high-dimensional (can be even larger than 2 2228 -dimensional) Hilbert space possible on a single machine.
For embedding x; X 1= Uic. Xj c=1 Ujer the partial derivative of the inner product-based objective Oii, j) w.r.t. the sub-embedding Use is:
jO(i,)) aO(i.)) 0(x,, a0(i,)) Ou,c 0(x,,x)) Ou:e a(x, x,) Uje ( Uje).
(4)
d'
where a0(i.)) x(x)))) depends on the objective (see Appendix C.2 for examples of MT and SGNS).
4.3 TRAINING PIPELINE
We optimize Eq. 3 by asynchronously updating embeddings on CPUs in line with the major litera- ture on network embedding (Mikolov et al., 2013b; Tang et al., 2015; Perozzi et al., 2014; Grover & Leskovee, 2016). Although the embedding optimization by asynchronous SGD (ASGD) (Recht et al., 2011) empirically works well for conventional embedding methods, the simple ASGD fails in our method where the gradients are much more complex with further constraints. Thus we intro- duce new techniques optimize the proposed compressive embedding model asynchronously with multiple CPU threads under the constraints stated in Sec. 4.1. Details of an iteration of embedding training is shown in 1. The features and explanations of Alg. 1 are listed follows:
i) Embeddings are learned on the unit hypersphere (see the function RiemannAdag in Alg. 1, for motivation please recall the normalization constraint in Sec. 4.1). For an embedding u,
5

---

Published as a conference paper at ICLR 2024
denote its Euclidean gradient as Grad(u). We first apply the Adagrad algorithm (Duchi et al., 2011) to compute the updating step Au for embedding u. Adagrad can automatically modify the step size of Au and accelerate model convergence. Our target is to update according to Au with the constraint that the updated embedding still falls on the hypersphere. As shown in lines 3-8 of the function Ri emannAdagrad in Alg. 1, we introduce three optimization strategies achieve the target. In the order-0 strategy, we update the embedding u in Euclidean space by the step Au and then map the updated embedding back to the hypersphere; in the order-1 strategy, we first map Au to the tangent space of u by Au- = Au - {u, Au)u, u, then update u by the step Au, and finally normalize the updated embedding; in the order-2 strategy, (Manfredo, 1992) which moves u in the direction of Áu with velocity || | along the geodesic of the manifold. The number of computational steps increases with the order of the updating strategy.
adopts exponential mapping
iii Inner products between embeddings are mostly positive (see line 3-9 of func- tion node2ketIter in Alg. 1, for mo- tivation please recall the second constraint (Uio; Uja) > 0, Vi, c in Sec. 4.1), We deal with this constraint by maximizing the term Eijey Ees1 min( Uje), 0). In practice, we keep monitoring the inner products between embeddings during the embedding learning procedure. If we find pair of embedding (Uae, Uja) < O, we max- imize their inner product.
Algorithm 1: An iteration of node2ket.
Input: suh-embeddiing dimension number suh-embedddings each node C, sampling function P(JGG) and PU]s, G). initial learning rate = 0.2, batch size be 8; function kode2kettter I: Sample a node pair (+. as the positive sample by P(J]G),
j ~ (joz(jji G):
2: Sample nodes (3'15" Pa [G)] so that (t, J') forn negative samples (pusitive/oegative samples are concepts from constrastive learning Chen et (2020);
3: & Prsitive inner product constraint
4:
for n E U U (J)* _ =,.....
5:
if (Mie; Unee) then Grad(u:e) +- Grac( +
6:
of unel
Uie
7: Grad(une) + Grad(Une) 2K Mnee
Mac
iii) Asynchronous batch optimization is adopted (see the function capchte
8: end if
9: end for
10: > Compute gradients and update embeddings
in Alg. 1). Though adopted by previous
11: for e (b.j) u (J') "''"".." 1, C do
works and working well in conventional
12: Grad(une) e- Gra)(Ume) Bung
a063))
embedding methods, updating embeddings once after obtaining gradients (Tsitsulin et al., 2018; Perozzi et al., 2014; Grover & Leskovee, 2016) can be too frequent for embedding learning thus causing great overhead during training. On the other hand, gradients of an embedding can be very dissimilar for different positively or negatively paired nodes, which brings dif- ficulties to model convergence. We asyn- chronously implement the batch optimiza- tion algorithm with a variable Count (u)
13: BatchUpdate(UMee Grad(U...))
Update batch
14: end for
function eatchupdate(u, crad(u))
1: Count(u) e- co)nt(u) +
Batch counter
2: Count(u) 2 be then
3: Grad(u)
Grad(r)))))) (u)aad((/Coutt(u))
Ri emmennndegrad u, Gred(u))
Normalization constraint
S:
Count(u) + 0: Grad(u) + 0: end if
6:
7:
function Ri mimmmmddggood crooo))) crod(u))
1: Statesum(u) -- atessmm()) + [[Grad(u)]]/dd
Adagrad
2: Au Vitttedur(u)++ Grad(u)
10-8
3: > Learning embeddiing on the Riemann manifold
4: [Order=0]
Morn(n + Au);
recording how many times that 'back-
5: > Map Au onto tangent space of
propagation' is performed on the embed-
6: [Order=l or 2) Áu + Au - Auju:
7: [Onder=l] u e- Norn(u + An);
ding u. When Count (u) reaches the batch size, the program updates the embedding u.
8: [Order=2] u + cos(I]āūl)ju++ + stn (1 Nu:
[dul
THEORETICAL ANALYSIS
5.1 PRELIMINARIES: NETWORK EMBEDDING IS IMPLICIT MATRIX FACTORIZATION
Previous studies (Levy & Goldberg, 2014; Qiu et al., 2018) have shown that the essence of network embedding is implicit matrix factorization (IMF) over an information matrix A(G) E N of the given network G. A indicates the latent node proximity and is implicitly defined by the whole training procedure, including the sampling strategy, the objective, and also constraints over embeddings. There are two types of IMF for NE (Armandpour et al., 2019; Xiong et al., 2022):
Type-I IMF: minimize [|A[G) x X|I,, Type-Il IMF: minimize ||A(G) - X H |r, (5) X.H
X
where X = X2.... x is the node embedding matrix, and H = h, he....................... ., hv] is called the hidden embedding matrix (a.k.a. context embedding in some works (Mikolov et al., 2013b; Tang

---

Published as a conference paper at ICLR 2024
Tranpped Kiwss Reu peried
Klamikaa prilat
@ pore x Xiriis
X2
H gionfo
1%% Vio
X
A(O)
A(O)
Aqi(G)
(G))
a) IMF of Conventional Embedding
b) IMF of Node2ket
Figure 2: Implicit matrix factorization (IMF) of conventional embedding methods and node2ket.
et al, 2015)) which is constructed in the same way of X. The type-1l II IMF achieves to encode asym- metric information while the type-I IMF can only preserve symmetric information. Some works also that the type-I objective focuses on the first-order information of A while type-Il II empha- sizes more on the second-ordes information (Tang et al., 2015; Xiong et al., 2022). Technically, the type-I IMF can be regarded as special case of the type-II IMF by fixing H X.
5.2 NODE2KET PERFORMS HIGH-RANK RANK APPROXIMATION OF THE INFORMATION MATRIX
Since the type-Il IMF is more general, we take the type-Il IMF the example for analysis while the conclusion also fits the type-I IMF.
Conventional Embedding a Low-Rank Approximation of (G) (illustrated in Fig. 2 a). As shown in Sec. 2, in convention embedding the embedding matrix X and H have the shape p x N, p denoting the number parameters for each node. It is obvious that rank(X T H) S p. In real-world networks, (G) can usually be very complex and high-rank. When p & (ank(A(G)) and the singular values of A (G) are close, only a very small portion of information can be preserved.
Node2ket Performs High-Rank Approximation of A(G) (illustrated in Fig. 2 b). We denote the e-th column sub-embedding matrix as ((c) [uies U2e1 UNo] and the e-th column sub-hidden-
embedding matrix V(e) [Vies V2cy .... V.el. Then by
8 Uie and h, X c1 Vic.
we have X = U() o... o (c)) and = ((1) o V Vic) where denotes the Khatri-Rao product (a.k.a. column-wise Kronecker product) (Liu, 1999). Then we have the following formula:
H = O ... (G),
(6)
where O denotes Hadamard product, (G) (c) Vie) is the information matrix reconstructed
by e-th column of uub-embedding and sub-hidden he--hhddd--ebdddingg.. with rank (A()) (G)) < p/C. One of the properties of Hadamard product is that the rank can be exponentially increasing. More precisely, we have the following theorem over the rank of X HH whose proof is put in Appendix D.2:
Theorem 1. By node2ket, denote (G)
Kze (G) we have rank X H N
if there 3c s.t. (G) and (G) are positive semidefinite, and max (G) + rank(A (A) (G)) rank A (G)) + K-rank (arank(A()))) (o)()) > N, and K-rank denotes the Kruskal rank.
To intuitively see the supremacy of node2ket over conventional embedding in approximating high- rank information matrices given the same p. we give toy examples for both type-I (Example 1, Appendix D.3) and type-II (Example 2, Appendix D.4) IMF, Example gives a G of 36 nodes with an information matrix (an((A(G)) approximation given = 8. And Example 2 gives G of 9 nodes with an information matrix (ank(A(G)) = for which conventional embedding gives a rank-6 approximation give p = 6. In comparison, node2ket can fully recover A (G)) with the same p.
16, for which conventional embedding gives a rank-8
Difficulties of Solving Node2ket by Explicit Matrix Factorization (EMF), When A can be explicitly expressed (e.g. (Cao et al., 2015; Qiu et al., 2018; 2019)), obtaining embeddings of node2ket by solving matrix factorization is still very difficult. Different from EMF by conven- tional embedding whose solution can be obtained through SVD for AGG), EMF by node2ket in- volves an NP-hard problem, rank-1 CP Tensor Decomposition (Hitchcock, 1927; Kiers, 2000). A feasible solution is: i) conduct SVD A(G)
PEQ ii) Expand P from the shape N x N to shape N x (p/C)C by concatenating a zero matrix then do a transpose: = P.O ili Conduct rank-1 1 CP Decomposition for each column of X which can be written as the Khatri-Rao product:
7

---

Published as a conference paper at ICLR 2024
X U-) o.... (C)) Step iii) is NP-hard (Phan et al., 2019; Krijnen et al., 2008), indicating the difficulties of node2ket by EMF.
EXPERIMENTS
6.1 PROTOCOL AND OVERVIEW
Details of settings including datasets, baselines, tasks, and metrics are in Appendix F.I.
Hardware Contigurations. All of the experiments run on single machine with Intel(R) Core(TM) 19-7920X CPU @ 2.90GHz with 24 logical cores and 128GB memory.
Datasets. We study five public real-world datasets of different scales, different densities, and con- taining different types of information. All of the networks are processed as undirected ones. The detailed statisties of the networks given in Table 5. We give brief introduction of the datasets as follows: YouTube (Tang & Liu, 2009b) is social network with millions of users on youtube.com BlogCatalog (Tang & Liu, 2009a) (BC) is social network of bloggers who have social connec- tions with each other; PPI (Breitkreutz et al., 2007) subgraph of the PPI network for Homo Sapiens; Arxiv GR-QC (Leskovec al., 2007) is collaboration network from arXiv and covers scientific collaborations between authors with papers submitted to General Relativity and Quantum Cosmology category; DBLP (Ley, 2002) is a citation network DBLP, where each node denotes publication, and each edge represents a citation.
Baselines. The investigated NE methods include three based on training with SGNS objectives (LINE, node2vec, VERSE), two based matrix factorization (ProNE and NetSMF), one based on hyperbolic embedding (HHNE), one based on graph partition (LouvainNE), one based on graph neural networks (GraphSAGE), and word2ket that constructs embeddings in the Hilbert space.
Tasks. We study three downstream tasks, including a preservation task, network reconstruction (NR), and two inference task, link prediction (LP), and node classification (NC).
We summarize all the experimental points and conclusions as follows:
Strong model performance against conventional baselines. In Sec. F.2, we compare the perfor- mance of node2ket and node2ket+ with baselines with same or varying number parameters. Results on small networks demonstrate that our methods can achieve the best performance all three tasks with the least parameters. We also find that as the dimension goes exponentially high, the model performance all the three tasks NR, LP, and NC improves.
Sec. F.3, the scalability of the model in shown by experiments on a Million-scale network. Our methods achieve top-level performance, with impressive high running speed, low memory usage, and small space for embedding storage.
Performance with varying C' and In Appendix F.41,, we show that when either C' is too small (C 1) or large (C p/2) the performance degenerates, indicating that our embedding constitution as product state of entangled states simultaneously improves the expressiveness and compactness.
Necessity of the proposed two constraints shown in Sec. F.4.2.
Fast convergence brought by the Adagrad-based optimizer is demonstrated in Sec. F.4.3.
Robustness. We demonstrate that our methods are robust with different sampling strategies (Ap- pendix F.5.1) objectives (Appendix F.5.2), and types of embeddings (Appendix F.5.3).
Parallclizability. In Sec. F.6 show that the parallelization decrease the running time significantly and will not affect model performance.
Visualization. In Sec. F.7, visualize the reconstructed adjacency matrix (G), which shows that sub-embeddings uie with different index c can orthogonally preserve the latent information matrix, which makes the approximated (G)) high-rank one.
We tend to conclude that the excellent performance of node2ket comes from two aspects:
Using the same amount of parameters, the embedding dimension of node2ket in the Hilbert space is much higher than conventional embedding methods, resulting in stronger expressiveness. Our implementation also carefully avoids the computation of high-dimensional embeddings.
8

---

Published as a conference paper at ICLR 2024
() PPT
hJ YouTube
Table I: Network reconstruction Table 2: Link prediction pre- precisions medium-scale net- cisions medium-scale net- works. 'CR.' is short for com- works. 'CR.' is short for the pressive ratio (see main text for compressive ratio (see main text the detailed definition).
40-
Mazzo (%)
30
-2-
20-
for the detailed definition).
10
BC
DBLP
GB-QC
PeI
VTC
BC
DSLP
GR-QC
VIC
LINE
4,80
18.24
2.34
54.19
15.18
0.10
LINE
53.08
77.35
75.45
05.85
V1.23
55.15
$2.70
16.55
0.103 0.7 0.6 5.01
Labol Foction
Label Portion
0.50
0.09
VERSS
87.59
LINE
VEStee
Ne45500
ProMts
17.55
30.9
015
PrNNE
75:27
15:31
03.55
LousanNE
6.79
21.33
63.47
12.79
11.00
LemsaNE
$7.99
73.44
86.90
$7.90
12
ProNE
HINK
0.99
1188
11.40
0.55
40.90
033
1825
100
2001
035
HINN
16.77
3F.
89.07
9031
V130
10300
68.7
17.0
200
K3
Figure 3: Results of node clas- sification. The compressive ra- tio of node2ket+ is 0.90 on PPI, and 0.73 on YouTube.
74.77
51.75
15.77
36.3
Willowni
86-57
96.37
V1.95
60.70
50.90
30,33
89.07
07.00
$1.56
74.05
Di mide
89:25
95.37
96.55
84.54
91.3
34.81
8923
9000
87.8
04.7
V179
WERY
CR.
0.76
CR.
0.33
The sampling strategy and objective function design are important. For the sampling strategy, experiments show that the model prefers low-order information of the network (e.g. with smaller window size by random walk as shown in Sec. 1.5. For the objective, experiments in Sec. F.5.2 show that the MT loss is more suitable for structural tasks NR and LP, while the logistic loss is more suitable for the semantic task NC.
6.2 BRIEF EXPERIMENTAL RESULTS ON AMALL-SCALE NETWORKS
We evaluate all the baselines and node2ket on the three tasks, NR, LP, and NC. In NR and LP, we set p = 128 for all methods, C 4 and R = 2 for w2k+n2k, and C' = 8 for our methods. In the experiments of NC, we set p = 32 for all methods, C' = 2 and R = 2 for w2k+n2k, and C = conduct Louvain partition on the former 4 sub-embeddings
for methods. For node2ket+, with resolution [100, 500, 1000, 1500], and do not do partition on the later 4 sub-embeddings. The compressive ratio of node2ket+ in Table and Table is defined as the ratio of the number of parameters used by node2ket+ to that used by other methods.
Analysis for Results. We give the results of NR Table LP in Table 2, NC in Fig. 3. i) Our methods consistently outperform the baselines on all three tasks, demonstrating the superiority of our embedding construction in the high-dimensional Hilbert space compared with other low- dimensional NE methods with the same amount of parameters. ii) By comparing node2ket and node2ket+, the compressive node2ket+ outperforms in NR, comparable NC and L.P. This indicates that the utilization of community information brings benefits to the preservation task NR and that appropriate compression won't have an adverse influence. iili The embeddings learned by the GNN model GraphSAGE do not perform well the three tasks. It is because GNN models highly rely on node features, which makes it difficult applying GNN methods on non-attinn attributedd networks.
7 CONCLUSIONS AND FUTURE WORKS
This paper makes contributions in two folds: On the algorithm level, we formalize the problem of nigh-dimensional network embedding, propose two approaches node2ket and node2ket+ con- stitute embeddings, and develop an optimizer to learn the embeddings stably with introduced con- straints in embedding learning: On the engineering level, we develop a fundamental toolkit in c++ named as LIBN2K for further high-dimensional NE algorithm development.
There are also limitations for future work. On geometry space of embeddings. Though node2ket and node2ket+ are designed in quantum Hilbert space that is nigh-dimensional spherical space, the approach of constructing compressive embeddings via tensor product also applicable other geometric spaces (e.g. Euclidean space, hyperbolic space). However, it may require specific model optimization techniques for embedding learning. This paper only discusses optimization in the quantum Hilbert space. On exploitation of graph properties. Previous works (sce Sec. B) have developed numerous methods exploiting graph structures by designing novel sampling techniques. We only show that our methods work robustly in different cases, e.g. random walk and random walk with restart, but do not pay extra attention to structure exploitation. On dealing with attributed networks. Just as many NE methods, the methods are not designed for attributed networks.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENTS
The used data, the source code of node2ket, node2ket+, and the LIBN2K library, the compilation instructions, and all the scripts to run experiments in Sec. 6, are provided in the link shown right below author names in the first page.
REFERENCES
Mohammadreza Armandpour, Patrick Ding, Jianhua Huang, and Xia Hu. Robust negative sampling for network embedding. Proceedings of the AAAI Conference on Artificial Intelligence, 33:3191- 3198, 07 2019. doi: 10.1609/aaii. v33101.33011191.
Adriano Barenco, Andre Berthiaume, David Deutsch, Artur Ekert, Richard Jozsa, and Chiara Mac- chiavello. Stabilization of quantum computations by symmetrization. SIAM Journal on Comput- ing, 2665):1541-1557, 1997.
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps and spectral techniques for embedding and clustering. In NIPS, pp. 585-591, 2002.
Ayan Kumar Bhowmick, Koushik Meneni, Maximilien Danisch, Jean-Loup Guillaume, and Bivas Mitra. Louvainne: Hierarchical louvain method for high quality and scalable network embedding. In Proceedings of the 13th International Conference on Web Search and Data Mining, 2020.
Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities large networks. Journal of Statistical Mechanics: Theory and Experiment, 2008(10), 2008
Antoine Bordes, Nicolas Usumier, Alberto Garcia- Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In NIPS, pp. 2787-2795, 2013.
Bobby-Joe Breitkreutz, Chris Stark, Teresa Reguly, Lorrie Boucher, Ashton Breitkreutz, Michael Livstone, Rose Oughtred, Daniel H Lackner, Jürg Bähler, Valerie Wood, et al. The biogrid inter- action database: 2008 update. Nucleic acids research, 366supp
2007.
Shaosheng Cao, Wei Lu, and Qiongkai Xu. Grarep: Learning graph representations with global structural information. In CIKM, pp. 891-900. ACM, 2015.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Peng Cui, Xiao Wang, Jian Pei, and Wenwu Zhu. A survey on network embedding, IEEE transac- tions on knowledge and data engineering, 31(5):833-8522 2018
A. M. Dirac. new notation for quantum mechanics. Mathematical Proceedings of the Cam- bridge Philosophical Society, 35(3):116-418 1939. doi: 02.1017/0030500410002
Yuxiao Dong, Nitesh V. Chawla, and Ananthram Swami. metapath2vec: Scalable representation learning for heterogeneous networks. In Proceedings of the 23rd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, 2017.
Xingbo Du, Junchi Yan, Rui Zhang, and Hongyuan Zha. Cross-networt skip-gram embedding for joint network alignment and link prediction. TKDE, 2022.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Rong-En Fan, Kai-Wei Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. Liblinear: A library for large linear classification. JMLR, 1(Aug):171-11844 2008.
Guobing Gan, Peng Zhang, Sunzhu Li, Xiuqing Lu, and Benyou Wang. MorphTE: Injecting Injectin mor- phology in tensorized embeddings. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
10

---

Published as a conference paper at ICLR 2024
Palash Goyal and Emilio Ferrara. Graph embedding techniques, applications, and performance: A survey. Knowledge-Basd Based Systems, 151:78-998 94, 2018.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In KDD, pp- 855-864. ACM, 2016.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010.
Will Hamilton, Zhitao Ying. and Jure Leskovec. Inductive representation learning on large graphs. In Advances in neural information processing systems, pp. 1024-1034, 2017.
Frank L Hitchcock. The expression of tensor polyadic as of products. Journal of Mathemattics and Physics, (11-4):164-199) 189, 1927.
Roger A. Horn and Zai Yang. Rank of a hadamard product. Linear Algebra and its Applications, 591:87-98, 2020, ISSN 0024-3795, doi: https///oii.rr//1.. 111.
Oleksii Hrinchuk, Valentin Khrulkov, Leyla Mirvakhabova, Elena Orlova, and Ivan Oseledets. Ten- sorized embedding layers. Findings of the Association for Computational Linguisties: EMNLP 2020, pp. 4860, Online, November 2020. Association for Computational Linguistics. doi: 18.18653/v 1/2020. htt.:/..inn.nnnsseeee.. mondings-emnlp-455
Chenqing Hua, Guillaume Rabusseau, and Jian Tang. High-order pooling for graph neural networks with tensor decomposition. Advances in neural information processing systems, 2022.
Mohammad Mehdi Keikha, Maseud Rahgozar, and Masoud Asadpour. Community aware random walk for network embedding. Knowledge-Basse Systems, 148:47-54, 2018.
Henk AL Kiers. Towards a standardized notation and terminology in multiway analysis. Journal of Chemometrics: A Journal of the Chemometrics Society, 14(3):105-122, 2000.
Thomas N Kipf and Max Welling. Semi-super classification with graph convolutional net-
works. arXiv preprint arXiv: wtpp::i000009977 2016.
Wim Krijnen, Theo K Dijkstra, and Alwin Stegeman. On the non-existence of optimal solutions and the occurrence of "degeneracy" in the candecomp/parafac model. Psychometrika, 73(3): 431-439, 2008
Jure Leskovec, Jon Kleinberg, and Christos Faloutsos. Graph evolution: Densification and shrinking diameters. ACM transactions on Knowledge Discovery from Data (TKDD), 1(1):2-es, 2007.
Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. Advances in neural information processing systems, 27, 2014.
Michael Ley. The DBLP computer science bibliography: Evolution, research issues, perspectives.
In Proc. Int. Symposium on String Processing and Information Retrieval, pp. 1-10, 2002.
Li Liu, William K Cheung, Xin Li, and Lejian Liao. Aligning users across social networks using network embedding. In IJCAI, pp. 1774-1780, 2016.
Shuangzhe Liu. Matrix results on the khatri-rao and tracy-singh products. Linear Algebra and its Applications, 289(1):267-277, 1999. ISSN 0024-3795. doi: http:://do..og/11.0001 30024-3795(98)
Weiyi Liu, Pin- Yu Chen, Sailung Yeung. Toyotaro Suzumura, and Lingli Chen. Principled multi- layer network embedding. In 2017 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 134-141. IEEE, 2017.
Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Ming Zhou, and Dawei Song. A tensorized transformer for language modeling. Advances in neural information processing systems, 32, 2019.
11

---

Published as a conference paper at ICLR 2024
Tong Man, Huawei Shen, Shenghua Liu, Xiaolong Jin, and Xueqi Cheng. Predict anchor links across social networks via an embedding approach. In ICAA,, volume 16, pp. 1823-1829, 2016.
Manffrdo. do carmo. riemannian geometry, 1992.
Yu Meng, Jiaxin Huang. Guangyuan Wang, Chao Zhang, Honglei Zhuang, Lance Kaplan, and Jiawei Han. Spherical text embedding. In Advances in neural information processing systems, 2019.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. Computer Science, 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. In NIPS, pp. 3111-3119, 2013b.
Michael A Nielsen and Isaac Chuang. Quantum computation and quantum information, 2002.
Giannis Nikolentzos and Michalis Vazirgiannis. Random walk graph neural networks. In Proceed- ings of the 34th Conference on Neural Information Processing Systems, pp. 16211-16224 2020.
Mingdong Ou, Peng Cui, Jian Pei, Ziwei Zhang, and Wenwu Zhu. Asymmetric transitivity preserv- ing graph embedding. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1105-1114, 2016.
Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab, 1999.
Aliakbar Panahi, Seyran Saeedi, and Tom Arodz. word2ket: Space-efficient word embeddings inspired by quantum entanglement. In International Conference on Learning Representations, 2020.
Bryan Perozzi, Rami Al-Rfou. and Steven Skiena. Deepwalk: Online learning of social representa- tions. In KDD, pp. 701-710. ACM, 2014.
Anh-Huy Phan, Petr Tichavský, and Andrzej Cichocki. Error preserving correction: A method for ep decomposition at a target error bound. IEEE Transactions on Signal Processing, 67(5): 1175-1190, 2019, doi: 10. 10.1109/TS..218.222 2887888.........81........
Jiezhong Qiu, Yuxiao Dong, Hao Ma, Jian Li, Kuansan Wang, and Jie Tang. Network embedding as matrix factorization: Unifying deepwalk, line, pte, and node2vee. KDD, pp. 459-467, ACM, 2018.
Jiezhong Qiu, Yuxiao Dong, Hao Ma, Jian Li, Chi Wang. Kuansan Wang, and Jie Tang. Netsmf: Large-scale network embedding as sparse matrix factorization. In The World Wide Web Confer- ence, pp. 1509-1520. ACM, 2019.
Jiezhong Qiu, Laxman Dhulipala, Jie Tang. Richard Peng, and Chi Wang. Lightne: A lightweight graph processing system for network embedding. SIGMOD '21, pp. 2281-2289, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781 9781450383431. 3448016.3457329.
doi: 10.1145)
Meng Qu, Jian Tang, Jingbo Shang, Xiang Ren, Ming Zhang, and Jiawei Han. An attention-based collaboration framework for multi-view network representation learning. In CIKM, 2017.
Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In NIPS, pp. 693-701, 2011.
Radim Rehůřek and Petr Sojka. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pp. 45-50, Valletta, Malta, May 2010. ELRA.
Leonardo FR Ribeiro, Pedro HP Saverese, and Daniel R Figueiredo. struc2vec: Learning node representations from structural identity. In KDD, pp. 385-394. ACM, 2017.
Sam T Roweis and Lawrence Saul. Nonlinear dimensionality reduction by locally linear embed- ding. science, 290(5500) 2000.
12

---

Published as a conference paper at ICLR 2024
Changpeng Shao. Quantum algorithms to matrix multiplication. arXiv preprint arXiv: arXhv:1803.0160) 2018.
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang. Jun Yan, and Qiaozhu Mei. Line: Large-scale information network embedding. In Proceedings of the 24th International Conference on World Wide Web, pp. 1067-1077. WWW, 2015.
Lei Tang and Huan Liu. Relational learning via latent social dimensions. In KDD, pp. 817-826.
ACM, 2009a.
Lei Tang and Huan Liu. Scalable learning of collective behavior based on sparse social dimensions. In CIKM, pp. 1107-1116. ACM, 2009b.
Anton Tsitsulin, Davide Mottin, Panagiotis Karras, and Emmanuel Müller Verse: Versatile graph embeddings from similarity measures. In Proceedings of the 2018 World Wide Web Conference, pp. 539-548. International World Wide Web Conferences Steering Committee, 2018.
Daixin Wang. Peng Cui, and Wenwu Zhu, Structural deep network embedding. In KDD, pp. 1225- 1234. ACM, 2016.
Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang- Deep graph library: A graph-centric, highly-performant package for graph neural networks. arXiv preprint KKV:1199000115. 2019.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao. Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint 16Xiv: 044.. 2016.
Hao Xiong, Junchi Yan, and Li Pan. Contrastive multi-view multiplex network embedding with applications robust network alignment. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery; Data Mining, 2021.
Hao Xiong, Junchi Yan, and Zengfeng Huang. Learning regularized noise contrastive estimation for robust network embedding. IEEE Transactions on Knowledge and Data Engineering, pp. 1-1, 2022. doi: 10. 20099TTDEE2022..3144888
Shuicheng Yan, Dong Xu, Benyu Zhang, Hong-Jiang Zhang. Qiang Yang, and Stephen Lin. Graph embedding and extensions: A general framework for dimensionality reduction. IEEE transactions on pattern analysis and machine intelligence, 29(1):40-5,, 2006.
Zi Yin and Yuanyuan Shen. On the dimensionality of word embedding. Advances in neural infor- mation processing systems, 31, 2018
Le Yu, Leilei Sun, Bowen Du, Tongyu Zhu, and Weifeng Lv. Label-enhanced graph neural network for semi-supervised node classification. IEEE Transactions on Knowledge and Data Engineering, 2022.
Jie Zhang, Yuxiao Dong, Yan Wang, Jie Tang, and Ming Ding. Prone: Fast and scalable network representation learning. In IJCAI, 2019.
Yiding Zhang, Xiao Wang, Nian Liu, and Chuan Shi. Embedding heterogeneous information net- work in hyperbolic spaces. ACM Transactions on Knowledge Discovery from Data (TKDD), 16 (2):1-23, 2021.
Ziwei Zhang, Peng Cui, Haoyang Li, Xiao Wang, and Wenwu Zhu. Billion-scale network embed- ding with iterative random projection. In 2018 IEEE International Conference on Data Mining (ICDM), pp. 787-796. IEEE, 2018a.
Ziwei Zhang, Peng Cui, Xiao Wang, Jian Pei, Xuanrong Yao, and Wenwu Zhu. Arbitrary-orde proximity preserved network embedding. In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery Data Mining, pp. 2778-2786. 20.8b.
13

---

Published as a conference paper at ICLR 2024
Da Zheng, Xiang Song, Chao Ma, Zeyuan Tan, Zihao Ye, Jin Dong, Hao Xiong, Zheng Zhang, and George Karypis. Dgl-ke: Training knowledge graph embeddings at scale. In Proceedings of the 43rd International ACM SIGIR Conference Research and Development in Information Retrieval, SIGIR '20, pp. 739-748, New York, NY, USA, 2020. Association for Computing Ma- chinery.
Zhaocheng Zhu, Shizhen Xu, Meng Qu, and Jian Tang. Graphvite: high-performance epu-gpu hybrid system for node embedding. In The World Wide Web Conference, pp. 2494-2500 ACM, 2019.
14

---

Published as a conference paper at ICLR 2024
Appendix
CONTENTS
Notations
16
B Related Works
16
Investigated Sampling Strategies and Learning Objectives
17
C.1 Sampling Strategies
17
C,2 Objective Functions
18
Theoretical Analysis
19
D.I Complexity Analysis From the Perspective of Model Compression
19
D.2 Proofs to Theorem
19
D.3 Example for Type-I Embedding
20
D.4 Example 2 for Type-II Embedding
21
E Connections to Quantum States
21
E.1
Preliminaries of Quantum States
E.2 Why Word2ket Constructs 'Pseudo' Entangled States?
22
E,3
Quantum Interpretation of Node2ket
22
F
Experiments
22
F.1
Experimental Settings
22
F.2
Experimental Results Compared with Conventional Baselines
25
F2.1
Results with the Same Number of Parameters
25
F.2.2
Results with Varying Training Parameters.
26
F3
Experiments on the Million-Scall Network
26
F.3.1
Overhead Comparison with LightNE and Gensim
28
E.4
Ablation Studies
28
F4.1
Model Performance with Varying C' and p
28
F.4.2
Ablation Study the Proposed Constraints
29
R4.3
Ablation Study on the Optimizer
29
F55
Model Performance in Different Scenarios
................................
30
F.5.1
Sampling Strategies
30
E.5.2
Objectives
30
E55.3
Embedding Types
F6
Parallelizability
31
F7
Case Study by Interpretable Visualization
15

---

Published as a conference paper at ICLR 2024
A
NOTATIONS
The used notations are listed Table 3.
Table 3: Main notations and descriptions.
Notations
Descriptions
G = (V,8)
Non-attributed network G with the vertex set V and the edge set & Quantum states
(8)
The number of parameters to construct an embedding The embedding dimension (when fixed in Sec. 3.2) Embedding of node i e-th sub-embedding of node Number of nodes of network G
use
N
Number of sub-embeddings of each node embedding Dimension of u4e (d p/C C/ in node2ket)
d
c(),c)
(for node2ket+) the index of Uge in the look-up table
O(1,5)
Objective of positive node pair (i,j)
The 'margin' for the marginal triplets objective
K
Number of negative samples for the skip-gram with negative sampling objective
P(G,jl))
Probability that (i, is sampled as a positive sample Window size for random walk Probability that random walk restarts The updating step of in the Fuclidean space
Au
Au
The updating step of u in the tangent space
B RELATED WORKS
Network Embedding. Previous works of NE can be divided into works on algorithm design and works on efficient NE systems. Algorithms. NE methods based on the language model word2vec (Mikolov et al., 2013b) are one of the biggest branches of the NE research. In this branch, NE methods have been designed for different types of networks, including homogeneous networks (Perozzi et al., 2014; Grover Leskovee, 2016; Tang et al., 2015), heterogeneous net- works (Dong et al., 2017) and multiplex networks (Liu et al., 2017: Qu et al., 2017; Xiong al., 2021). By designing the input node sequences, NE methods achieve to preserve specific types of information, c.g. low-order proximity (LINE (Tang et al., 2015).) structure similarity (struc2vec (Ribeiro et al., 2017)), versatile similarity measures (VERSE (Tsitsulin et al., 2018)), and cross-network alignment relations (CENALP (Du et al., 2022)). Another line of NE research is developing NE methods based on matrix factorization. GraRep (Cao et al., 2015) solves the embed- ding by matrix factorization for random walk and Skip-Gram while also taking high-order proximity information into consideration. NetMF (Qiu et al., 2018) proposes to unify some word2vec-basse NE methods including LINE, Walk, and node2vec, etc. within matrix factorization frame- work, and NetSMF (Qiu et al., 2019) treats the network embedding task as sparse matrix factor- ization. AROPE and HOPE (Zhang et al., 2018b; Ou et al., 2016) propose to preserve multi-order proximity by matrix factorization with some mathematical tricks. Methods based on deep neural networks (Wang et al., 2016) especially graph neural networks (Hamilton al., 2017) are also in- fluential in literature of NE.
Systems and Toolkits Developing efficient network embedding systems is also important for in- dustrial applications of NE algorithms. Graph Vite (Zhu et al., 2019) uses GPUs to accelerate em- bedding training for algorithms including LINE, DeepWalk, and node2vec. LightNE (Qiu et al., 2021) achieves to train embeddings for billion-scale networks based on ProNE (Zhang et al., 2019) and NetSMF (Qiu et al., 2019) on a single machine of 1.5TB memeory. Though endowed with high efficiency, these works a basic tool for developing new NE methods. There are also some works that aim to provide tools for developing NE methods, e.g. Gensim (Rehüřek & Sojka, 2010), and DGL (Wang et al., 2019). Gensim is library for word embedding training. based which many famous NE methods includ- ing DeepWalk (Perozzi et al., 2014), node2vec (Grover & Leskovee, 2016), and struc2vec (Ribeiro al., 2017) developed. These works obtain node embeddings by feeding the carefully designed
highly packaged for specific NE algorithms and hardly can they serve as
16

---

Published as a conference paper at ICLR 2024
Table 4: Comparison between the word embedding library Gensim (Rehüřek & Sojka, 2010) and our high-dimensional NE library LIBN2K.
Gensim
Proposed LIBN2K
Field
Word embedding
Network embedding
Dimension
Low
High
Supported
Skip-gram with negative sampling,
Skip-Gram, CBOW
Objectives
marginal triplets
Optimizer
SGD
Constrained Riemannian Adagrad
Supported Data
Sequences
Networks and sequences Random walk, random walk with restart
Sampler
N/A
DeepWalk (Perozzi al., 2014),
Implemented
node2vec (Grover & Leskovec, 2016), struc2vec (Riheiro et al,, 2017), etc.
node2ket, node2kett, w2k+n2k
Methods
Language
Cython
C++
Common Points
Good scalability, high efficiency, distributed training, CPU-only
node sequences into gensim. word2vec. DGL is a library for general deep graph learning mostly on GNN models. DGL also provides implementation for NE methods including DeepWalk and LINE, with GPUs accelerating training.
In the aspects of algorithm design and system developing, we design the first nigh-dimensional NE algorithms node2ket(+) and develop toolkit LIBN2K for future -igh-dimensional NE algorithm development. The significance of LIBN2K for high-dimensional NE is similar to Gensim for NE, where the technical differences between them are summarized in Table 4.
Tensorized Models for Model Compressing, Tensors are a generalization of vectors and matrices. Thanks to the capability of representing and manipulating high-dimensional subjects, tensor has been successfully applied to exploit complex physical systems (Nielsen & Chuang, 2002). Notic- ing that tensors can be decomposed thus reducing the total number of parameters of models, searchers start to explore the application of tensor decomposition techniques model compression. These works include compressing word embeddings in NLP (Panahi al., 2020; Hrinchuk et al., 2020; Gan et al., 2022), compressing transformer models (Ma et al., 2019), learning high-order fea- tures by Graph Neural Networks (Hua et al., 2022). Compared with word2ket (Panahi et al., 2020) which adopts a similar embedding construction approach for deep neural networks, we substantially decrease computational overhead especially memory usage by skipping the computation of full- dimensional embeddings, thus fully utilizing the strong expressiveness of super-high dimensional Hilbert space (our p > 232 v.s. p 8000 in word2ket). The tensorized models inspire us to conduct high-dimensional NE with a small number of parameters.
C INVESTIGATED SAMPLING STRATEGIES AND LEARNING OBJECTIVES
C.I SAMPLING STRATEGIES
The sampling strategies P)) directly affect the performance of NE by implicitly defining prox- imity between the nodes. We decompose the probability as P(i.jlG P(jC)) G), where P(i = Σ., degtej is the probability node i being sampled as the first node with deg(i) referring to the degree of node i, and P(jli, G) is the probability that node j is sampled
deg(s)
positive node for i. We consider two strategies of P(jiú é, G). random walk (RW) and random walk with restart (RWR): RW is widely used in exploring graph structures in graph-related MI. models (Perozzi et al., 2014; Nikolentzos & Vazirgiannis, 2020). RWR is closely associated with Personalized PageRank (PPR) (Page al., 1999), which is popular for measuring node similarities. In Alg. 2, we give the implicit definition of P(jli, G) of RW and RWR by sampling. Note that for RW of window size w, we will perform the function RW(i, G, w) for w times, with ú 1, W.
17

---

Published as a conference paper at ICLR 2024
Embedding Look-up
Distributed Trainer
Input Data
Table
network
Conventional Embedding
Sampler
update embedding
samples
pustive & negative samples
node2ket
Objective
gradients
node sequences
node2ket+
Optimizer
write
Figure 4: Considering such common embedding learning pipeline that iteratively samples node pairs (yellow arrows), backward propagates gradients (the purple arrow), and updates node embed- dings (blue arrows), highlight our contributions in red including compressive embedding look-up tables by node2ket and node2ket+, and optimizer to efficiently learn embeddings in a stable man- ner.
C.2 OBJECTIVE FUNCTIONS
The basic idea for objective design is in accordance with the prevalent contrastive learning (Gutmann & Hyvärinen, 2010; Chen et al., 2020), making proximate nodes close in the embedding space and unrelated nodes far away.
Skip-Gram with Negative Sampling (SGNS). Initially, the objective is to maximize the following log softmax function:
(i. log exp((x;, Xz))
Ejev exp( (x,, xy'))
which, however, is computationally intractable for a large V, and thus surrogate objectives are pro- posed, including Hierarchical Softmax and Negative Sampling (Mikolov et al., 2013b). In this paper, only discuss typical objective based on the popular Negative Sampling in NE literature:
Oii,)) log (x X3) +
BPen log (-x x):),
positive samples
nogative samples
where (xx) = 1/(1 exp(-x)) is the sigmoid function, K is the number of negative samples for each positive sampled pair, Preg is the distribution of negative samples j' which is empirically set proportional degree((')U.78 in SGNS-based NE methods (Perozzi et al., 2014; Tang et al., 2015; Grover & Leskovec, 2016; Liu et al., 2016).
Gradient Derivation SGNS. Given the label (1 for positive samples and otherwise), the partial derivative of logsigmoid function is:
log (2(1) 0.5)(x,, (1, (xx, H(x,,
(9)
Ou,
Ouic
where 1, is an indicator returning 1 if and otherwise. The Eq. used computing the gradient of SGNS loss.
Marginal Triplets (MT). The objective more often used in knowledge graph embedding (Bordes et al., 2013; Zheng et al., 2020), word embedding (Meng et al., 2019), and also shows strong per- formance on structure-relate tasks e.g. network alignment (Xiong al., 2021). aims keep the margin between inner product positive pairs and negative pairs larger than hyperparameter ry:
(ii.)) max ((x,, Xyy (x,, x3) + +7,0).
(10)
where j' the sampled negative node as j' in SGNS. The gradient of the MT objective is obvious so we omit it here.
18

---

Published as a conference paper at ICLR 2024
Algorithm 2: Sampling strategies G).
Input: network G. starting node i, window size ur (for RW), restart probability o; function RW(i, w)
1: + i;
2: for cuc 1.2,..., do
3: j SN(J,, ==-SNjjG)) G);
Sample from j's neighbors
4: end for
5: return j;
function RWR(i, G, a)
1: j
2: while Rand() < do
3: j SN(j, G))
4: end while
5: return j:
D THEORETICAL ANALYSIS
D.1 COMPLEXITY ANALYSIS FROM THE PERSPECTIVE OF MODEL COMPRESSION
Proposition 1 (Capability of Node2ket+ on Embedding Compression). To train q-dimensional em- beddings for N nodes, if requires obtaining different embeddings {x:) by node2ket+, we need at least C(Nq)) /C parameters to store eub-embeddings and NC parameters to store the index table [(i, c), where C is the number of sub- uub-embeddings for each node.
Proof. We use [T(,c)) to denote the number of different indices of e-th sub-embeddings, then the total number of different embeddings that these sub-embeddings can constitute is II (=) So should make sure that I T(,cc) 2 N.
The total number of -ubeednedd is Ee (((,c)):
Eu((o)) 2 (II I/C 2 CCN/C
(11)
where the first '2' is by Cauchy inequality which takes '=' when [T(:.c) is constant. The di- q²/C, So for the number of parameters for all the sub-
mension of each lebbedmbeding embeddings is - [A(,,c) 2 C(Nq) The index table [(i, c) needs NC parameters.
D.2 PROOFS THEOREM
Before the proof, we introduce the following lemma:
Lemma 1 (Rank of Hadamard Product(Horn & Yang. 2020)), Let A, E N be positive semidefinite. If A and B have no zero main diagonal entries and
+ rank (Bnk rank(A) K-rank > N,
(12)
then A < B is positive definite.
Theorem 1. By node2ket, we denote (G) Ok4o (G)) we have rank(X ( N if there 3e s.t. (G) and (G) are positive semidefinite, and max (K-rank (Aco + rank A (G)) rank A (G)) + K-rank(A (G) > N, and K-rank denotes the Kruskal rank.
Proof. Notice that the conditions in the theorem meet the requirements of Lemma so we have A(G) (G) (G) is positive definite, which indicates that rank(A(G)) =
19

---

Published as a conference paper at ICLR 2024
D.3 EXAMPLE 1 FOR TYPE-I EMBEDDING
Example 1. For undirected graph G 36 nodes (N = 36), whose adjacency matrix is defined as Fig. 5:
1 2
O
2
O
2
0
2
1
Figure 5: The adjacency matrix of the 36-node example.
Suppose the embedding target is to preserve the adjacency matrix as the latent information matrix (G). Given the number of parameters for each embedding p can be factorized by node2ket without loss of information by setting:
t 1,,
t2 1,1,,
t3 1,00]
t4 1,,0,
(13)
ts [1,0,1,0],
to = 1,,
x; t///6] & 1,%6. 1.2.
we would obtain that X = A(G) where X xx X36l- Since A is a rank- 16 matrix, it never be factorized as the multiplication of two 36 x 8 matrix by conventional embedding without losing information.
20

---

Published as a conference paper at ICLR 2024
D.4 EXAMPLE 2 FOR TYPE-II EMBEDDING
Example 2. For directed ring G of nodes (N = 9), suppose we use the adjacency matrix as the latent information matrix AG), i.e.
A(G)
(14)
where the unfilled elements are as the defeult. Given the number of parameters for each embedding = 6, it can be factorized by node2ket without loss of information by setting:
X1 [0, 1,0 & 1,0,0),
X2 00,0, 1] @ [1,0,0]
X3 11,0.0 @ [0,1,0],
X4 0, @ [0,1,0],
X5 00.011
[0,1,0] [,,10,)
(15)
X6 1,0.0 @ 0,0,1].
X7 = 00, 1. 8 00,
X5 00 0, 10 00,
Xg [1 0, 0,00
@ [1,0,0],
and
h 11,0, 0 @ [1,0,0].
h 1,, @ 11,0,),
ha 00,0,1
[1,0,0],
h4 [1,0,0 @ 00,1,0,
h; = 1,, @ 0.1,0].
(16)
he [0,0,
0,1,0),
by [1,0,0 8 0,0,1],
hg 10 0 0,0,1]
hg [0, 0, 1] 00,
would obtain that H A() where X [x..... .x] and [h..... hn].
Since it is a rank-9 9 matrix, it can never be factorized as the multiplication two x 6 matrix without losing information.
E CONNECTIONS TO QUANTUM STATES
E.1 PRELIMINARIES OF QUANTUM STATES
The quantum state is physical concept to describe the state of quantum system. It can be rep- resented by a state vector, wave function, or density matrix. Paul Dirac introduced the bra-ket notation to denote quantum states complex vectors in Hilbert space (Dirac, 1939). In matrix representations, kets are column vectors denoted as (ı) (where the name node2ket comes) and bras vectors that the conjugate transpose of the kets denoted as (16).
In physical experiments, the spin of an electron can be either up or down, therefore the quantum state of electron is wwo-dimensional. A two-dimensional quantum system is known qubit (quantum bit), which basic unit in quantum computing. The state qubit is defined as a 2-dimensional complex vector in the Hilbert space C A state (w) [ is linear superposition
21

---

Published as a conference paper at ICLR 2024
of two orthonormal basis states (0) = [P] and [1) [b]. and have (&8) a{0) - B]1), where
o, 9 E C are probability amplitudes satisfying Joi - 1812 1. That means, experimentally
will
get the probability that (16) collapse to state [0) with probability |a|2 and state [1) with probability 1812. Indeed, any quantum state (eb) should be normalized, i.e. the 22-norm of state vector (e)) should be I, which indicates that the quantum fidelity (vb(p) is equal to I in the language of quantum mechanics.
Considering two quantum systems A and B with corresponding Hilbert spaces HA and HB. the composite space of A and B H. H. @ Hg where denotes tensor product (can be seen as Kronecker product for two vectors). If we use ( and (6) ø to denote states of system A and B respectively, then a state (6) E H. that can be written as (Ø) (Ø) A @ (0) is called a product state or separable state, otherwise it is an entangled state. A most well-known entangled states is the Bell state (Ø+) / (10)
X (0) + [1] & [1)) = (7.0 0,0, v2 One can easily check that (Ø+)
cannot be written (v) a | e for any possible | ( B C². The properties of 2-qubit system can be easily extended to n-qubit quantum system that is in the Hilbert space C
E.2 WHY WORD2KET CONSTRUCTS 'PSEUDO' ENTANGLED STATES?
In Sec. E.1 have reviewed the concepts of quantum states, which is written in the language of physics. To transfer quantum states to embedding, we just need to substitute the complex number field C with the real number field R and quantum states (vb) with embedding vectors and
Word2ket represent a node embedding x; (i is the node index) as:
R
Uires
(17)
r=l c=1
where Uire is the state of quantum subsystem. Though it is similar the form of an entangled state, we find that it hardly follows the normalization rule quantum state, i.c. ||x:||2 = 1:
xx 2
uirel 2
Σ r:
Uirrer
Uirze).
(18)
c=1
c=1
To make ||>;1|2 1. one feasible solution is to add constraint |/urrell 1/R and (uiries Usrze) Vc, F1 * r2. The latter constraint is too strict to be practical. Such constraints are not shown in word2ket in model training. Therefore, embeddings by word2ket are finally the so-called 'pseudo" entangled states.
E.3 QUANTUM INTERPRETATION OF NODE2KET
The embedding constructed by node2ket can be regarded as the product state of several separate quantum sub-systems of which each state is an entangled state. We use toy example explain this: Suppose we define the embedding x; as the vectorized 2-order tensor: un1 U22 R then the sub-embedding U; can be used represent any state of 3-qubit quantum system (since 2 9) in the real number field, no matter it is product state or an entangled state, and the embedding x, the product state composed of two separate quantum sub-systems represented by u, and U;2 respectively.
uii @ u,2 where
F EXPERIMENTS
F.1 EXPERIMENTAL SETTINGS
Hardware Configurations. All of the experiments are a single machine with Intel(R) Core(TM) i9-7920X CPU @ 2.90GHz with 24 logical cores and 128GB memory.
Datasets. We study five public real-world datasets of different scales, different densities, and con- taining different types of information. All of the networks are processed as undirected ones. The detailed statistics the networks are given in Table 5. We give a brief introduction of the datasets as follows: You Tube (Tang & Liu, 2009b) is social network with millions of users youtube.com;
22

---

Published as a conference paper at ICLR 2024
Table Statistics datasets.
Dataset
[V]
[|]
Avg. degre
Type of info.
#Labels
PPI
3,890
76,584
39.37
Biological
50
GR-QC
5,242
14,496
5.53
Academic
N/A
BlogCatalog (BC)
10,312
333,983
64.78
Social
N/A
DBLP
12,591
49,627
7.88
Academic
N/A
YouTube-Cur (YTC)
22,579
95,596
8,46
Social
YouTube
1,138,499
2,990,443
5,25
Social
47
BlogCatalog (Tang & Liu, 2009a) (BC) is a social network of bloggers who have social connec- tions with each other; PPI (Breitkreutz al., 2007) subgraph of the PPI network for Homo Sapiens; Arxiv GR-QC (Leskovec et al., 2007) is a collaboration network from arXiv and covers scientific collaborations between authors with papers submitted General Relativity and Quantum Cosmology category: DBLP (Ley, 2002) is citation network of DBLP, where each node denotes publication, and each edge represents a citation.
Baselines and Used Parameters. The investigated NE methods include three based on train- ing with SGNS objectives (LINE, node2vec, VERSE), based on matrix factorization (ProNE and NetSMF), one based on hyperbolic embedding (HHNE), one based on graph partition (Lou- vainNE), and one based on graph neural networks (GraphSAGE). Note that no baselines in ten- sorized/coopreesivee NE are provided since this is the first work in this line of research. Brief introductions of baselines and used parameters are follows:
LINE² (Tang et al., 2015) models first-order and second-order proximity on the adjacency matrix, trains them separately with edge sampling, and concatenates the representations after normalization on node embedding. The number of samples is set as x 10
node2vec2 (Grover & Leskovee, 2016) uses biased random walk to explore both breadth-first and derth-first structure, and train node embedding and hidden embedding by negative sampling. We set p = and = the experiments, and window size 10, walk length 80 and the number of walks for each node 80.
VERSE* (Tsitsulin et al., 2018) uses node similarity by Personalized PageRank (PPR), Adajcency Similarity, and SimRank, and learns node embedding by LINE-Ist. We use the default version with PPR similarity, damping factor a = 0.85. We run 10 epochs for each node.
N (Qiu et al., 2019) obtains node embeddings by sparse matrix factorization dense random walk matrix. We set the rank as 512, window size as 10, and rounds as 1000.
sparsified
ProNE (Zhang et al., 2019) initializes embedding by sparse matrix factorization, and then enhances it via spectral propagation. We run the Python version in the repository with the default settings. We the enhanced embedding the node embedding result.
LouvainNE' (Bhowmick et al., 2020) adopts Louvain graph partition algorithm (Blondel et al., 2008) and then learns node embeddings from the partitions. We set the damping parameter 0.01 of the method which is the default value in the repository.
HHNES (Zhang al., 2021) adopts hyperbolic embedding for heterogeneous networks. We feed random walk sequences into the hyperbolic embedding core. We set window size as 5, the number of negative samples as 10. The other parameters follow the default setting of the official repository.
hhtps://ihhubbccm//nggaaauuuuuu
hhapss//githubbccokkgg///eeee
http://ghhhbb.cmmmmmaaaa//////aa
23

---

Published as a conference paper at ICLR 2024
GraphSAGE" (Hamilton et al., 2017) To make the GNNs applicable to non-attributed networks, we adopt the same approach in (Hamilton et al., 2017) which treats the one hot encoding of node degree as imput node features. We train the model in the unsupervised manner. We set the hidden dimension as 256, output dimension as 128, learning rate as 0.01 and rounds as 500. It should be noted that comparison of NE methods with GNN methods is not typical (Cui et al., 2018) because of their different settings: NE methods deal with networks with only structural information while GNNs deal with networks with node features.
Work2ket (Panahi et al., 2020) represent embeddings in the R-rank C-order CP Decomposition form with neural network- based decoder. It cannot be directly run for NE, nor can the embeddings be optimized stably without the proposed optimization techniques in this paper. We name the variant that constructs embeddings like word2ket but optimized with techniques of node2ket as w2k+n2k. As the default, we set R = 2.
Tasks. Investigated tasks include the following three benchmark tasks in NE literature (Tang et al,, 2015; Cao et al., 2015; Wang et al., 2016; Grover & Leskovec, 2016; Perozzi et al., 2014; Qiu et al., 2021): i) network reconstruction, preserving known links of the origin network, ii) link prediction, predicting unknown links in networks, and iii) node classification, predicting node labels by training an extra classifier on the learned node embeddings. The former two tasks can be categorized as structural tasks since they are only related to structural information. The node labels usually contain semantic information (Yu et al., 2022), so we categorize node classification as a semantic task. Detailed descriptions of the tasks and the metries
below.
Network Reconstruction (NR). Network reconstruction is used to measure an embedding model's ability to preserve nodes' origin local structure. In line with (Wang et al., 2016), for baseline meth- ods, we measure the score of a pair of node embeddings by negative Euclidean distance. During the reconstruction process,
calculate the scores between all possible pairs of the embeddings, namely [ 1)/2 undirected node pairs. Then we sort the scores and select the top-(E) node pairs, denoted E. Then the precision is defined as Precision E n E'//EE. For large-scale networks, we use Prec@N to approximately measure the precision.
Link Prediction (LP). We randomly mask 1% edges of the given network, and create the same number of edges as noise edges by randomly connecting nodes in the network. Then, we conduct network embedding on the network where masked edges are removed, and predict the masked edges from the mixture of masked edges and noise edges. The used metric is precision.
Node Classification (NC), As an experimental paradigm in line with previous works (Perozzi et al., 2014; Wang et al., 2016; Tang et al., 2015), we first train node embeddings unsupervisedly by uti- lizing merely structural information. For evaluation, we divide all the labels into train and test sets, and then we train an independent classifier to classify the nodes according to nodes' embeddings and labels in the training set. Then the trained classifier is used to classify the nodes in the test set by the learned node embeddings. Concretely, we conduct experiments on PPI and YouTube. After embeddings are obtained, LIBLINEAR (Fan et al., 2008) is used to train one-vs-rest Logistic Re- gression classifiers. We range the portion of training data from 1% to 9% for YouTube and 10% to 90% for PPI. We use Macro-Fl as the metric. For each embedding method, we perform 10 times of trials and report the average scores.
Default Parameter Settings. For all methods that can set the number of threads (node2ket and baselines except ProNE and LouvainNE),
use 8 threads the default. For our method, we use
the MT objective on the tasks of NR and LP, and the SGNS objective on the NC task. For the MT objective, the margin is set as 7 0, and for the SGNS objective, the number of negative samples is set as K = 5. We use random walk with window size w 2 for LP. w = 1 for NR and NC. The default Riemannian optimization order is O. The default sampling number is 100M. For YouTube the sampling number is 1000M. Default input is data in the network form. Default embedding type type-I.
24

---

Published as a conference paper at ICLR 2024
Table 6: Network reconstruction precisions on medium-scale networks. 'CR.' is short for com- pressive ratio (see main text for the detailed def- inition).
Table 7: Link prediction precisions on medium- scale networks. 'CR.' is short for the compres- sive ratio (see main text for the detailed defini- tion).
BC
DBLP
GR-QC
PPI
YTC
BC
DBLP
GR-QC
PPI
YTC
LINE
4.80
2.34
54.19
15.18
0.10
LINE
53.08
75.45
95.86
55.15
62.76
node2vec
2.05
18.74
59.92
43.46
1.00
node2vec
77.58
82.09
91.72
77.58
76.78
VERSE
10.96
16.96
56.08
27.59
7.26
VERSE 33.35
68.81
93.79
51.55
59.83
NctSMF
7.64
18.08
30.26
17.14
0.28
NetSMF 59.34
82.09
87.59
72.16
74.58
ProNE
10.10
17.76
56.37
24.02
0.15
ProNE 75.27
85.31
93.79
73.97
81.07
LouvainNE
6.79
21.33
43.47
17.77
11.90
LouvaInNE
57.99
73.44
86.90
57.99
82.11
HHNNE
0.09
11.40
50.50
18.23
<0.01
HANEE
38.77
35.92
93.10
65.72
83.89
GraphSAGE
0.88
0.23
0.93
0.91
0.25
GraphSAGE
51.5
50.91
60.69
47.68
<0.01
w2k++2k
29.77
51.25
80.98
35.77
26.37
w2k+n2k
86.57
94.37
95.86
84.79
90.59
node2ket
97.90
74.62
node2ket
89.28
95.37
node2ket+
54.61
89.63
98.07
87.35
76.43
node2ket+
89.31
94,97
93.79
84.54
91.63
CR.
0.86
0.72
0.76
0.88
0.74
CR.
0.83
0.69
0.73
0.84
0.70
PPI
b) YouTube
40
20
Macro-FF F1 (%)
15-
10
5-
10
0.1
0.3
0.5
0.7
0.9
0.01
0.05
0.09
Label Portion
Label Portion
LINE VERSE
node2vec ProNE
NeLSMF LouvainNE
HINE GraphSAGE
w2k+n2k
mode2ket+
node2ket
Figure 6: Results of node classification. The compressive ratio of node2ket+ is 0.90 on PPI, and 0.73 on YouTube.
F.2 EXPERIMENTAL RESULTS COMPARED WITH CONVENTIONAL BASELINES
F.2.1 RESULTS WITH THE SAME NUMBER PARAMETERS
We evaluate all the baselines and node2ket on the three tasks, NR, LP, and NC. In NR and LP, we set p 128 for all methods, C 4 and R 2 for w2k+n2k, and C = 8 for methods. In the experiments NC, we set p 32 for all methods, C' = and R = 2 for w2k+n2k, and C = for our methods. For node2ket+, we conduct Louvain partition on the former 4 sub-embeddings with resolution [100, 500, 1000, 1500], and do not do partition on the later 4 sub-embeddings. The compressive ratio of node2ket+ in Table 6 and 7 defined as the ratio of the number of parameters used by node2ket+ to that used by other methods.
Analysis for Results. We give the results of NR in Table LP in Table 7, NC in Fig. i) Our methods consistently outperform the basclines including conventional methods and w2k+n2k on all three tasks, demonstrating the superiority of our embedding construction the high-dimensional Hilbert space compared with other low-dimensional NE methods with the same amount of parame- ters. ii) By comparing node2ket and node2ket+, we observe that the compressive node2ket+ shows better performance in NR, comparable in NC and LP. This indicates that the utilization of commu- nity information brings benefits to the preservation task NR and that appropriate compression won't 't have adverse influence GraphSAGE do not perform well on the three tasks. is because GNN models highly rely on node features, which makes it difficult applying GNN methods non-attributed networks.
model performance. iii) The embeddings learned by the GNN model
25

---

Published as a conference paper at ICLR 2024
a) Network Recon.
b) Link Prediction
c)
Node Classification
20.0
Precision (%)
80
Precision (%)
80
Macro-FI (%)
17.5
60
60
40
15.0
40-
20
12.5
0
1
256
512
1
256 512
256 PPraams/Node
512
# ParaamsNNode
# #Params/Node
LINE
VERSE
node2vec
w2k+n2k
node2ket
Figure 7: Results with varying parameters on PPI. The more close to the upper left corner, the better efficiency of parameter utilization.
F.2.2 RESULTS WITH VARYING TRAINING PARAMETERS.
We evaluate the trainable baselines of which the used training parameters can be quantified. Ex- periments for NR, LP, and NC are conducted on the PPI. We give the results in Fig. 7, where #Params/Node (can be different from p for conventional methods) for node2ket is equal to Cd. Specially, in Fig. 7 c), we only report the cases when #Params/Node< 32. That is because when #Params/Node= Cd ing capacity of the classifier. In practice, the consumed memory is always positively related to the number of parameters (experiments in Sec. F.3)
* 64, the embedding dimension q = dc 224 exceeds the process-
Analysis for Results. i) In all the three tasks, node2ket achieves to use least parameters to yield the best results. Concretely, we observe that node2ket with only 32 parameters/node outperforms all the baselines with 512 parameters/node, which is 16 times node2ket. The results indicate that our embedding models have better expressiveness. ii) The performance of conventional embedding methods is likely get worse when the embedding dimensions are larger than specific threshold. However, the phenomenon is not obvious on node2ket. The performance of node2ket improves almost consistently learning high-dimensional embeddings.
parameters used, which indicates the better ability of node2ket in
F.3 EXPERIMENTS ON THE MILLION-SCALE NETWORK
Metrics. Network reconstruction. For large-scale networks, it is infeasible compute the scores of all [V[[V] - 1) node pairs and rank them to reconstruct the raw networks. So we approximately estimate the reconstruction precision by sampling in line with previous works e.g. SDNE (Wang et al., 2016). First we sample (N T)|E| randomly matched node pairs as the 'noise" node pairs, denoted as _". Then we calculate the scores of all the NN edges in Ed and select the (op-(E] pairs as the reconstructed edges. So far, the computation of precision Prec@N becomes obvious. The metrics of link prediction is the precision, the same as described in Sec. F.I; For node clas- sification the metrics are Macro-/Miero-Fi In NC. evaluate the embeddings with 1% portion of label information as the train data. Settings. We train 512-dimension embeddings for all meth- set C 3 and p 24, which also yields 512-dimensional
ods. For node2ket and node2ket+, embeddings. For node2ket+, we do Louvain partition on the former sub-embeddings with reso- lutions [500, 1000]. The comparison of model performance is given in Table The comparison between the computational overhead is shown in Fig. 8. In Fig. 8 a), we plot the running time of the methods. For node2ket+, the time does not contain the procedure of building sub-embedding index table by Louvain partition which is regarded as data pre- preporcessiing and takes about 625 seconds on Tube. In Fig. b), we plot the maximum memory usage (VmPeak) which basically has three parts, the loaded data, temporary variables, and the node embeddings. In Fig. 8 c), plot used local space for embedding storage. Embeddings of node2ket and other baselines are stored as 'mpy' binary files. Sub-embeddings and the indices of sub-embeddings by node2ket+ stored pickle binary files. The methods "ode2ket(net)" and 'node2ket(seq)" in the results mean that node2ket uses network node sequences by random walk the input.
26

---

Published as a conference paper at ICLR 2024
Table 8: Results on YouTube. The compressive ratio of node2ket+ is 0.02.
NR
LP
NC
Method
Prec@2
Prec
55
Precision
Macro-FI
Micro-FI
LINE 81.17
66.40
55.95
55.27
15.75
29.77
node2vec
45.59
11.30
4.90
51.09
27.13
37.17
VERSE
37.32
26.45
22.91
23.77
12.82
23.04
ProNE
96.11
88.34
76.38
79.24
29.50
36.83
LouvainNE:
85.65
70.50
62.92
76.06
18.57
31.06
HHNE
85.17
66.16
48.02
79.94
17.24
33.31
NetMMF & GraphSAGE are not available on our machine
w2k+n2k(net)
99.20
96.78
90.72
88.37
25.30
38.20
node2ket(net)
97.97
92.01
81.06
86.06
28.67
39.29
node2ket(seq)
98.02
91.67
78.03
86.22
25.71
37.44
(nee2ket+(net)
96.71
87.83
72.48
86.06
26.56
36.66
mode2ket+(seq)
95.73
85.47
69.18
86.69
23.67
34.25
Running Time (Training 1/O)
2.0
#2k+n2k(net)
Time (h)
LouvainNE
1.0
n2k(net)
(ne+(net))
n2k(seq)
n2k+ (seq)
node2vec
VERSE
ProNE
HHNE
LINE
0.5
b) Peak of Memory Usage
40
VmPeak (GB)
w2k+n2k(net)
30
LouvainNE
n2k+(net)
n2k + (seq)
n2k(net)
n2k(seq)
node2vec
VERSE
ProNE
HANE NE
LINE
10
c) Embedding Storage
Storage (MB)
2000
1500
Baselines
1000
5.3x of w2k+n2k
10.5x of n2k 22.7% 7x of n2k+
500
w2k+n2k
node2ket
node2ket+
Figure 8: Computational overhead on YouTube, 'n2k' is short for 'node2ket',
Analysis. i) Table shows that our methods show overwhelming performance on structural tasks NR and LP, and comparable performance with the best baseline ProNE on the semantic task NC. ii) Fig. 8 shows that our methods use much fewer computational resources compared with baselines. the result of both much fewer parameters and the light-weight implementation of sampling. In our program, variables in memory are mostly the training uub-embeddings. In comparison, the baseline node2vec will store huge transition matrix memory. And ProNE, NetSMF will also consume huge memory for large-scale matrix factorization. LouvainNE. does not contain these parts so is also highly light-weight. iii) Compared with node2ket, though sacrificing part of the model performance and slightly increasing the running time, node2ket+ moderately reduce memory usage and significantly reduce local storage while still outperforming most baselines. iv) Compared with the input of network, the input of sequences but sacrifices the local storage for node sequences since the file of node sequences is usually much larger than the network file.
much running time and memory usage,
27

---

Published as a conference paper at ICLR 2024
F.3.1 OVERHEAD COMPARISON WITH LIGHTNE AND GENSIM
Settings. LightNE (Qiu et al., 2021) and Gensim" 1
(Rehüřek Sojka, 2010) are two CPU-only multi-threaded embedding tools with high efficiency for NE methods based matrix factorization and word2vec respectively (See Sec. for more details). We compare our node2ket (window size 2) with Walk implemented with Gensim-4.3.1 and NetSMF integrated in LightNE on their ability to learn network structures. The thread number set as 24 for all the methods. NR precisions given in Table 9. The comparison of running time and the maximum memory usage (VmPeak) are shown in Fig. a) and b) respectively.
Result Analysis. Results show that node2ket only needs 2.3GB memory for 312-dimensional em- bedding for the YouTube network of 1.IM nodes. In comparison, the state-of-the-ar high-efficiency NE system LightNE requires 94.0GB memory (40.0x of node2ket) and Gensim requires 17.5GB memory (7.4x of node2ket). Using much less memory. the precision of node2ket is only slightly lower than LightNE with about more 8 minutes of running time.
Time (h)
b) VmPeak (GB)
Table 9: NR precisions.
1.0
75-
Prec@N
0.5
50
N=2
N=10
N=50
DeepWalk
92.40
84.31
75.96
0.0
LightNE
98.78
97.60
94.35
node2ket (LIBN2K)
LightNE
DeepWalk (Gensim)
node2ket
98.12
95.12
89.38
Figure 9: Overhead.
In experiments, we also find that memory usage will increase with the number of threads. We record VmPeak of our methods with I thread Table 10. shows that node2ket+ with sequences as input only consumes 0.32GB memory. In comparison, LightNE (94.0GB) consumes 297.6x of the memory and Gensim (17.5GB) consumes 55.2x of the memory.
Table 10: VmPeak of our methods with thread.
n2k(net)
n2k+(net)
n2k(seq) n2k+(seg)
VmPeak (GB)
0.79
0.58
0.53
0.32
F.4 ABLATION STUDIES
F.4.1 MODEL PERFORMANCE WITH VARYING C' AND p
We conduct ablation studies with main concerns parameters C and p on node2ket, while the conclusions are also suitable for node2ket+. We conduct experiments on the three tasks with varying C' and p. Results are given in Fig. 10. The analysis is from the following aspects:
Model performance with the number of parameters = Cd fixed (d is the dimension of sub- embeddings). Blocks in each row in the sub-figures of Fig. 10 have the same Cd, indicating that the numbers of parameters are the same. We have the following findings: i) By observing the rows, can find that in most cases, C performance of our methods as shown in Sec. F.2 comes from the tensorized embedding constitution the Hilbert space with the same objective function, sampling strategy, and optimization technique. ii) Observing the rows, we can find that when C' is very big, though the compressive ratio is impres- sive, model performance decreases. It is probably because the embeddings become too sparse the Hilbert space.
is better than C 1. So, we can safely claim that the strong
Model performance with the embedding dimension q = d fixed. By the comparison between two cases of 64-dimensional embedding, C - 1, Cd 64 and C = 2, p Cd 16 in the figures, we can find that the former will yield better results. It indicates that when the embedding dimension dC is fixed, though the embedding model can be compressed by increasing C, the model performance will also be sacrificed.
10
"I
28

---

Published as a conference paper at ICLR 2024
The compressive power of node2ket. Impressively, as plotted in Fig. 10, the compressive ratio can be as small as 2 2-57 -57
Network Reconstruction
Link Prediction
c) Node Classification
8
4.31
3,87
8
00.94
64.00
16
19.47
26.90
35.02
10.31
36.08
12.74
12
10.30
12"
12
16
00.19
86.02
RR
87.35
87.66
05:77
32
64
27.00
30.89
40.13
37.
(16)
(8)
(128)
40.10
128
4.36
132)
12 13)
25)
12 57)
120
07:05
09.54
09.02
70
00
128
38.20
41.32
(32)
Figure 10: Model performance with varying C and p Cd. The number at the top of each block is the score, and the number at the bottom of each block within a bracket is the inverse number of the compressive ratio w.r.t. the conventional counterpart, which is calculated by č²- for example, 20 means that the output embedding dimension = d is 20 times as the used parameter = Cd. The compressive ratios b) are the same as a) so we hide them.
F.4.2 ABLATION STUDY ON THE PROPOSED CONSTRAINTS
In Sec. 4.1, we introduced the two adopted constraints for embedding learning: i) sub-embeddings should be normalized, and ii) inner product between arbitrarily two sub-embeddings should be pos- itive. We are going to investigate the necessity of the two constraints by removing them during embedding learning, and compare the model performance with different orders of Riemannian opti- mization whose algorithm is given in Alg. 1. We run experiments with random seeds varying from 1 to 5, then compute the average and std value of the results. The studies conducted on PPI with other parameters and experimental settings keeping the same as in Sec. F.2. Results are presented in Table 11.
Analysis for Constraint i). Removing this will cause program crashes, as shown by the first row in Table 11. To learn embeddings with the constraint, we propose three orders of Riemannian optimization in Alg. 1, Sec. 4.3. We investigate how they affect the model performance, and present the results as shown by the bottom three rows of Table 11. The results show that Riemannian order=0 is recommended for its good performance and fast rumning speed.
Analysis for Constraint ii). We find that removing this constraint won't 't obviously affect the model performance by comparing the second row of Table 11 with the bottom three rows. However, by an- alyzing the sub-embeddings, we find that the phenomenon 'two negative makes positive' happens 66,814 times without this constraint, which decreases to 24,414 times when we add the constraint. By visualizing the embeddings in Fig. 11, we also find that adding this constraint will endow the leamned embeddings with better clustered local structure.
Table 11: Ablation studies for the adopted constraints on PPI. 'N/A' means that the program crashes.
NR
LP
NC
Remove constraint i)
N/A
N/A
N/A
Remove constraint ii)
87.87+0.07
84.48+0.46
16.54:10.20
Riemannian order=)
88.02
80.39
16.25
Riemannian order=1
88.2440.08
84.74
16.3941040
Riemannian order=2
86.57#0.25
83.92:00.76
11.1740.26
F,4.3 ABLATION STUDY ON THE OPTIMIZER
In this paper we use Adagrad (Duchi et al., 2011) to adaptively modify the learning rate for the sake of stable learning with different sampling strategies and objectives. We compare the embedded Ada- grad optimizer with other possible choices, including the SGD with a linearly decreasing learning rate from 0.02 0.0001, which empirically works best in training conventional embedding methods (Mikolov et al., 2013a; Perozzi et al., 2014; Tsitsulin et al., 2018; Grover & Leskovec, 2016; Xiong al., 2022), and RMSProp by G. Hinton, which slows down the learning rate decaying of Adagrad.
29

---

Published as a conference paper at ICLR 2024
a) Without Constraint ii)
b) With Constraint ii)
50
50
0
-50
-50
50
50
-50
0
50
Figure 11:
Visualization of learned node embeddings of PPI by t-SNE. The constraint ii) refers that we try to keep inner products between sub-embedddings positive, One can see that embeddings b) better clustered.
Results are given in Fig. 12 and Table 12, showing the superiority of Adagrad in both accelerating convergence and improving model performance on downstream tasks.
Loss of MT
SGD RMSProp
0.04
Adagrad
0.02
Table 12: Model performance with dif- ferent optimizers on downstream tasks.
0.00
5
20
NR
L.P
NC
Loss of SGNS
SGD
48.58
88.18
38.42
-1.40
RMSProp
7.94
82.01
30.52
Adagrad
76.43
91.63
38.96
-1.45
5
10
15
20
25
30
Number of Samples (Million)
Figure 12: Loss of objectives.
F.5 MODEL PERFORMANCE IN DIFFERENT SCENARIOS
F.5.1 SAMPLING STRATEGIES
We investigating the model performance under different sampling strategies the model bustness. Experiments are conducted on YouTube-Cut. For NC, the embedding results are evaluated with 9% portion of all the labels as training data. We vary the window size for random walk, and vary the parameter for random walk with restart. Results are given in Fig. 13. We can see that i) random walk with a small window size yields the best results, and ii) though the model performance fluctuates as the parameters w and a change, the fluctuation is still in an acceptable range which can be seemed to brought by the change of sampling strategies. Our carefully designed optimization techniques endow the embedding learning procedure with strong robustness when equipped with different sampling strategies.
F.5.2 OBJECTIVES
We evaluate the model performance of node2ket the dataset YouTube-Cuut with two objectives, Marginal Triplets (MT) and Skip-Gram with Negative Sampling (SGNS), as described in Sec. C.2. The scores of node2ket on the three tasks NR, LP, and NC, are given in Table 13. NC uses 5% portion of train data. Results show that MT is more suitable for structural tasks NR and LP, while SGNS is more suitable for the semantic task NC. The conclusions in accordance with previous works that use SGNS for NC task (Perozzi et al., 2014; Tsitsulin et al., 2018) and use MT for structural tasks (Xiong al., 2021).
30

---

Published as a conference paper at ICLR 2024
a) Random Walk
b) Random Walk with Restart
92
NR Precision (%)
70
LP Precision (%)
NC Macro (%)
NR Precision(%
LP Precision (%)
30
NC Macro-F1 (%)
-37.0
60
20
88
90
20
50-
36.5
10-
NR
86
89
LP
40
88
+36.0
0
NC
84
-0
3
4 5 6
7
8 9
10
0.1
0.3
0.5
0.7
0.9
Window Size w
Figure 13: Performance of node2ket on YouTube- Cutbee-Ctttte under different sampling strategies.
Table 13: Scores of objectives MT and SGNS on the three tasks.
NR (precision)
LP (precision)
NC (Macro-F1)
MT
74.62
91.63
31.01
SGNS
0.32
87.13
38.96
F.5.3 EMBEDDING TYPES
In Sec. 5.1 we have introduced two types of embedding architectures. We evaluate the model performance of node2ket on the dataset YouTube- with the two embedding types. The scores of node2ket on the three tasks NR, LP, and NC, are given in Table 14. NC uses 5% portion of train data. Results show that the type-I embedding yields better results for node2ket and also demonstrate the robustness for different types of embedding.
Table 14: Scores of objectives MT and SGNS on the three tasks.
NR (precision)
LP (precision)
NC (Macro-FI)
Type-I Type-1I
74.62
91.63
38.96
18.29
88.91
37.06
F.6 PARALLELIZABILITTY
We evaluate the model performance and running time with regard the number threads on GR- QC. We set C= = 8, p- 128. We plot the running time of 100M iterations and results of NR and LP with different numbers of threads in Fig. 14. We can see that: i) Used time decreases significantly as the thread number increases, indicating that parallelization ding procedure. ii) The performance stays stable as the thread number increases, which indicates that asynchronously updating embeddings by multiple threads would not affect the model perfor- mance. iii) When the number of threads is greater than 8, the acceleration brought by parallelization becomes less significant, which means that the advantages of our method are more pronounced on machines with limited number of threads.
be applied to accelerate the embed-
F.7 CASE STUDY BY INTERPRETABLE VISUALIZATION
In this section, we conduct visualization to show that how our methods preserve high-rank informa- tion with small portion of parameters.
We conduct node embedding via node2ket on circular ladder graph of 20 nodes, which is plotted in Fig. a) and the adjacency matrix visualized in Fig. 15 f). For node2ket, we set = and p 8. After obtaining the sub-embeddings, we reconstruct the adjacency matrix from the and U,2, and visualize them Fig. 15 b) and c). Then, from the final node embeddings, we compute the reconstructed adjacency matrix and visualize it in real-valued vaalvveee/biiaiizeeddd form shown in Fig. 15 d) and e). From the results, we have the findings: i) by comparing Fig. 15 b). c), and d), we can see that column 1 and column preserves structures in approximately orthogonal manner, which means the extracted information is different among aubb-emeeddigg, and ii) by comparing Fig. 15 e) with the ground truth f), node2ket fully reconstructs the input adjacency matrix.
31

---

Published as a conference paper at ICLR 2024
29
lime
NR Precision (%)
precision
LP Precision (%)
100
precision
156.96
100
Time (s)
27
91.72
94
95
26
53.98
85
49.75
91
88
90
8
16
16
2
4 # TThrads
8
16
# Threads
#Threads
Figure 14: Parallelizability evaluation on the GR-QC dataset.
Recon. by Am @ Ag (real-valued)
Recon. (binarized)
a) Input
b) Am)GG) by Un
c) Aca(G) by Up, Uça
Ground
truth
Figure 15: Visualization of the reconstructed adjacency matrix at different levels from the input 18- node circular ladder graph. Darker means a higher value. In b) and c), a color block at the i-th row and j-th column shows the inner product of two sub-embeddings, i.c. (uies uze). And the heatmap becomes the visualization of (G) = U ((e). In d), color block the i-th row and j-th column visualizes the inner product of node embeddings, i.c. (x) In e), we binarize the results given by d). And f) is the visualization of the ground truth adjacency matrix of the input graph.
To compare the reconstruction performance of different methods on different data, we further in- vestigate Hypercube graph, 3D-Grid graph, and the real-world Karate Club graph in Fig. 16. The results directly show the overwhelming performance of node2ket on preserving information especially the high-rank information.
32

---

Published as a conference paper at ICLR 2024
Circular Gruph
Hlypercube
a) Input Graph
00
bo Ground -ruth
0000
00
CJ nodee
20
di LING
e) nod822ees
00 WERSE
99 Pake
hJ LouvainNE E
Figure 16: Visualization of reconstruction results of different embedding methods..
33


Reviews:
[{"review_id": "W7W6mBaWwa", "reviewer": "Reviewer_VRn5", "summary": "For the standing and important task of network embedding in data mining and machine learning, the paper proposes explores the exponentially high embedding space for network embedding, which largely differs from existing works dewelling on the low-dimensional embedding. This is achieved by product quantum states in a super high-dimensional quantum Hilbert space. The experiments show surprisingly strong performance of the approach, in terms of both high memory and running efficiency with strong robustness across different tasks of network reconstruction, link prediction, and node classification. The authors also provide the source code to ensure the soundness of the experiments.", "questions": "1) Comaperd with Fig. 1, can the authors provide a more succinct plot to convey the main idea of the paper? Fig. 1 is still a bit busy which is useful yet a more direct illustration in the begining of the paper is welcomed. Something like Fig. 2 is better.\n2) Can the approach be useful for solving combinatorial problems especially for large-scale ones? As there is little attributes need to be considered in these cases thus it seems suited to the proposed methods?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) This paper innovatively resorts to the high-dimensional embedding space for network embedding, which quite departures from existing literature.\n2) The paper is well presented and the overview plot in Fig. 1 is very informative and useful to readers. The paper is well organized with strong content in appendix that signifcantly enriches the paper.\n3) The experiments are impressive. Provided with the source code, I am convinced by the strong performance.\n4) The authors give strong theoretical understanding of the essence of their approaches, which I really appreciate.", "weaknesses": "As the authors emphasized, the presented techniques are mainly suited for the structure networks, without considering the attributes. I understand this setting and think it is reasonable in practice. It also fits with many previous works in literature that have also been compred in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3qVsrXwsbb", "reviewer": "Reviewer_RkWJ", "summary": "The paper introduces a groundbreaking paradigm for network embedding (NE), departing from traditional low-dimensional embeddings and exploring high-dimensional quantum state representations. The authors propose two NE methods, node2ket and node2ket+, and implement them in a flexible, efficient C++ library (LIBN2K). Experimental results showcase the good performance of their proposal, boasting advantages in parameter efficiency, running speed, and memory usage.", "questions": "No questions at the moment.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well organized. The insights offered in the paper have the potential to inspire the development of other quantum-inspired methods and contribute to the broader application of quantum computing in the field of network embedding.", "weaknesses": "The primary concern in this submission pertains to the technical contribution. First, the extension of the word2ket concept to product states appears relatively straightforward. Second, the utilization of product states might limit the embedding's expressivity since these states occupy a smaller portion of the Hilbert space and result in a low-dimensional representation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pzPWozAHGn", "reviewer": "Reviewer_8cxP", "summary": "This work proposes two high-dimensional network embedding methods called node2ket and node2ket+ that outperform standard methods such as word2ket. As evidence of these claims, they perform experiments studying several tasks, obtaining better compressive ratio than other approaches. Additionally, they provide an implementation in a library called LIBN2K.", "questions": "- What is the significance of the embeddings being designed for \"quantum Hilbert space\"? To me, the relationship to quantum computation is not clear and seems more like an afterthought. Quantum computers are known to be good at problems with certain structure, and it's not clear to me what structure is being leveraged here (and what benefits are obtained as a result).\n- The fact that pure quantum states are normalized leads to the constraint $\\\\|\\mathbf{x}_i\\\\|=1$. Is there any consideration for eliminating a global phase, which I assume would affect the embedding? Also, I wonder what embedding might be developed for more general quantum states, such as mixed states.\n- Forgive my ignorance, but what is the definition of positive node pairs and negative nodes? Does it just mean the inner product is positive or negative?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper improves upon existing network embedding techniques by proposing a high-dimensional embedding with better efficiency over previous quantum-inspired methods such as word2ket.\n- The theoretical analysis is clear, showing that node2ket gets a high-rank approximation of the information matrix\n- The experiments appear to be quite thorough and shows advantages over existing methods, as promised.\n- It is nice that the code is made available.", "weaknesses": "- This is a quantum-inspired algorithm for classical machine learning, which suffers from the lack of a clear connection to quantum computation (see Questions).\n- There are gaps for actually making this algorithm \"quantum-friendly,\" as it is generally not easy to load classical information into a quantum device.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)