Title: Reconciling Competing Sampling Strategies of Network Embedding
Venue: NeurIPS 2023
Accepted: yes
Keywords: Network embedding
Abstract: Network embedding plays a significant role in a variety of applications. To capture the topology of the network, most of the existing network embedding algorithms follow a sampling training procedure, which maximizes the similarity (e.g., embedding vectors' dot product) between positively sampled node pairs and minimizes the similarity between negatively sampled node pairs in the embedding space. Typically, close node pairs function as positive samples while distant node pairs are usually considered as negative samples. However, under different or even competing sampling strategies, some methods champion sampling distant node pairs as positive samples to encapsulate longer distance information in link prediction, whereas others advocate adding close nodes into the negative sample set to boost the performance of node recommendation. In this paper, we seek to understand the intrinsic relationships between these competing strategies. To this end, we identify two properties (discrimination and monotonicity) that given any node pair proximity distribution, node embeddings should embrace.
Moreover, we quantify the empirical error of the trained similarity score w.r.t. the sampling strategy, which leads to an important finding that the discrimination property and the monotonicity property for all node pairs can not be satisfied simultaneously in real-world applications. Guided by such analysis, a simple yet novel model (SENSEI) is proposed, which seamlessly fulfills the discrimination property and the partial monotonicity within the top-$K$ ranking list. Extensive experiments show that SENSEI outperforms the state-of-the-arts in plain network embedding.

Reconciling Competing Sampling Strategies of Network Embedding

Abstract

Network embedding plays a significant role in a variety of applications. To capture the topology of the network, most of the existing network embedding algorithms follow a sampling training procedure, which maximizes the similarity (e.g., embedding vectors' dot product) between positively sampled node pairs and minimizes the similarity between negatively sampled node pairs in the embedding space. Typically, close node pairs function as positive samples while distant node pairs are usually considered as negative samples. However, under different or even competing sampling strategies, some methods champion sampling distant node pairs as positive samples to encapsulate longer distance information in link prediction, whereas others advocate adding close nodes into the negative sample set to boost the performance of node recommendation. In this paper, we seek to understand the intrinsic relationships between these competing strategies. To this end, we identify two properties ( discrimination and monotonicity ) that given any node pair proximity distribution, node embeddings should embrace. Moreover, we quantify the empirical error of the trained similarity score w.r.t. the sampling strategy, which leads to an important finding that the discrimination property and the monotonicity property for all node pairs can not be satisfied simultaneously in real-world applications. Guided by such analysis, a simple yet novel model (SENSEI) is proposed, which seamlessly fulfills the discrimination property and the partial monotonicity within the topK ranking list. Extensive experiments show that SENSEI outperforms the state-of-the-arts in plain network embedding.

1 Introduction

In the era of big data, network embedding [36, 14, 21, 7, 42] maps nodes in the network to lowdimensional vectors in the embedding space, which plays an important role in many tasks such as node recommendation [54, 61, 38, 8, 20, 19, 1, 2], networked time series imputation [44, 18, 48, 9], knowledge graph completion [49, 46, 47, 45, 28], and network alignment [52, 53, 60, 62, 16, 27, 59]. To distill the topology information of the network, most existing network embedding methods follow a sampling training procedure. Given any central node to be considered, existing network embedding methods build a positively sampled node pair set and a negatively sampled node pair set. Then, they optimize the embeddings of nodes by maximizing/minimizing the similarity between the positively/negatively sampled node pairs in the embedding space. Explicitly or implicitly, these methods are based on an assumption that nodes close to the central node should be included in the positively sampled node pair set, whereas distant nodes are likely to be considered as negative samples. For example, DeepWalk [36] and metapath2vec [7] explicitly construct random walk containing positively sampled nodes and select negative nodes according to the degree distribution of the network. The message-passing mechanism in GraphSAGE [15] and graph auto-encoder (GAE) [22] is built on graph Laplacian regularization [21, 57], which makes connected node pairs to be

similar. In this case, the close/distant nodes become implicit positive/negative samples for the central node, which can be regarded as a general form of such sampling training procedure.

Numerous network embedding algorithms have been proposed, which focus on improving either the positive sampling strategy or the negative sampling strategy based on various intuitions. Consequentially, disparate or even competing sampling strategies emerge in various methods. To name a few, rather than solely favor nodes within two hops as positive samples in LINE [41], node2vec [14] allows a longer random walk length. For graph convolution network (GCN) [21], APPNP [23] utilizes personalized pagerank [33] to sample more distant nodes w.r.t the central node for feature aggregation. The aforementioned two algorithms accommodate farther nodes in the positively sampled node pair set. Meanwhile, SPNE [12], KBGAN [4] and RecNS [56] encourage choosing closer nodes as difficult negative samples to promote the performance of node recommendation.

Thus, some fundamental questions arise: what is the theoretic root cause behind such antithetical sampling strategies? what are possible and what are impossible for sampling of network embedding? how can we develop a practical embedding algorithm that simultaneously embraces these competing strategies?

In this paper, we hammer at bringing the intrinsic relationships behind these competing sampling strategies in light. Concretely, we start from two fundamental tasks of graph learning: the link prediction task and the node recommendation task. To tackle the above two tasks synchronously, we identify two desirable properties, including the discrimination property and the monotonicity property, that network embedding vectors should satisfy given a node pair proximity distribution. The discrimination property means that the node pair with high proximity should be distinguished from the node pair owning low proximity in the embedding space ( link prediction ). For the monotonicity property, the ranking list of nodes recommended to the query node needs to be consistent with the proximity list in the descending order. Theoretically, we analyze the general form of network embedding algorithms' loss functions. We show that the negative sampling distribution should be negatively correlated with the node pair proximity distribution. Furthermore, we show that in the ideal case where the algorithm can sample an infinite number 1 of positive nodes for the central node, both the discrimination property and the monotonicity property can be fulfilled (i.e., possibility result). However, due to the limited sample size of the real-world data, there exists an inevitable error between the ideal optimal similarity scores and the empirical optimal similarity scores in the embedding space. Regardless of the specific sampling strategy, the discrimination property and the monotonicity property for all node pairs in the network can not be fulfilled at the same time (i.e., impossibility result).

Fortunately, in real-world applications, ranking all nodes in the network for the query node is often unnecessary. In many cases, only the topK recommendation list and its internal order matters, which suggests that we can first achieve the discrimination property to detect candidate nodes to be recommended. After that, we can attain the monotonicity property within the topK ranking list ( partial monotonicity ). Guided by this intuition and the theoretical results, we propose a simple yet novel model (SENSEI). In detail, SENSEI adopts a commonly used proximity measurement (personalized pagerank [33]) and decomposes the embedding process into two steps. The first step is to satisfy the discrimination property. In addition to sampling nodes with large proximity as positive samples, SENSEI also includes nodes with intermediate proximity in the positively sampled node pair set, which reduces the empirical error of the similarity scores for these nodes. Then, in the second step, SENSEI pays attention to the monotonicity property within the positively sampled node pairs from the first step. Nodes are ranked w.r.t. their proximities and importantly, some positively sampled nodes are turned into negative samples, which resembles the strategy of selecting difficult negative samples in some existing methods [4, 55]. In this way, SENSEI creatively integrates these two competing sampling strategies in one integral framework. The experiments demonstrate that SENSEI greatly outperforms various baselines.

To summarize, our contributions are three folds:

Theoretical Analysis. We reveal the underlying relationships between the competing sampling strategies of existing network embedding methods. Specially, we prove that any

sampling strategy bears an inevitable error gap between the empirical and ideal optimal embedding similarity scores.

Simple yet Novel Model. Based on the theoretical results, we propose a two-step model SENSEI which creatively integrates competing sampling strategies in one network embedding framework. It satisfies the discrimination property in the first step and obtains the partial monotonicity property within the positively sampled node pair set in the second step.

Experimental Results. Extensive experiments show that SENSEI outperforms the state-ofthe-arts in plain network embedding.

Problem setting. We primarily focus on plain network embedding, in which given the adjacency matrix of a plain network A , we aim to output the embedding matrix F .

2 Analysis

In this section, we uncover the intrinsic relationships of existing competing sampling strategies. First, we describe the sampling-based network embedding process mathematically. Then, we propose two desired properties for the learned embeddings: the discrimination property and the monotonicity property, which correspond to two common learning tasks: link prediction and node recommendation . Theoretically, we analyze the general form of network embedding algorithms' loss functions. We show that the negative sampling distribution should be negatively correlated with the given node pair proximity distribution. Guided by this critical insight, we give both possibility and impossibility results for network embedding. First (possibility result), we show that, in the ideal case where an infinite number of positive samples can be obtained, both the discrimination property and the monotonicity property can be satisfied with the optimal solution of the loss function. Second (impossibility result), for the empirical loss function with a limited number of positive samples, we prove that there always exists an error gap between the empirical optimal solution and the ideal optimal solution. Regardless of the specific sampling strategy, the discrimination property and the monotonicity property can not be simultaneously satisfied for all node pairs in the network in the empirical situation.

Current sampling-based network embedding algorithms can be divided into two phases. The first phase is to construct a node pair proximity distribution p ( u | v ) , which serves as the positive sampling distribution in the training process. For example, personalized pagerank [33] in APPNP [23] is an explicit node pair proximity distribution, while random walk in DeepWalk [36] and node2vec [14] implicitly constructs such node pair proximity distribution. The second phase is to design a negative sampling distribution p n ( u | v ) and to utilize the network embedding loss function to train the embedding model. Similar to previous works [55, 56], we start from analyzing the second phase of the sampling-based network embedding process as follows:

Definition 1. Sampling-based Network Embedding Process in the Second Phase. Given a node proximity distribution p , where p ( u | v ) refers to the proximity of node u w.r.t. the central node v and ∑ u p ( u | v ) = 1 , the algorithm designs a negative sampling distribution p n and adopts a loss function J to obtain similarity scores s ( u, v ) ∈ [0 , 1] of all node pairs ( u, v ) , which are calculated by the node embedding matrix F .

The similarity score s ( u, v ) defined here is general, which can be implemented with various similarity measurements (e.g., the sigmoid function σ ( F ( u, :) ⊤ F ( v, :)) or the cosine function cos ( F ( u, :) , F ( v, : )) ) in various algorithms.

Then, to solve both link prediction and node recommendation tasks, we propose that the general similarity score s ( u, v ) calculated by node embeddings should possess two properties: the discrimination property and the monotonicity property, which are defined as the following:

Definition 2. Discrimination and Monotonicity.

Discrimination Property: Given the central node v , node pair ( u, v ) with large p ( u | v ) should be clearly distinguished with node pair ( w,v ) with small p ( w | v ) in the embedding space, i.e., lim s ( u,v ) s ( w,v ) = + ∞ when p ( w | v ) p ( u | v ) → 0 .

Monotonicity Property: Given the central node v and two arbitrary nodes u , w in the network, if p ( u | v ) > p ( w | v ) , then s ( u, v ) > s ( w,v ) .

First category

of methods

𝑠(𝑢

!

, 𝑣)

𝑠 → 1

𝑠(𝑢

"

, 𝑣)

𝑠(𝑢

#$

, 𝑣)

𝑠(𝑢

%

, 𝑣)

𝑠(𝑢

&

, 𝑣)

𝑠(𝑢

'

, 𝑣)

𝑠(𝑢

#

, 𝑣)

𝑠(𝑢

(

, 𝑣)

𝑠(𝑢

)

, 𝑣)

𝑠(𝑢

*

, 𝑣)

𝑠(𝑢

#

, 𝑣)

𝑠(𝑢

(

, 𝑣)

𝑠(𝑢

#$

, 𝑣)

𝑠(𝑢

%

, 𝑣)

𝑠(𝑢

"

, 𝑣)

𝑠(𝑢

!

, 𝑣)

𝑠(𝑢

)

, 𝑣)

𝑠(𝑢

'

, 𝑣)

𝑠(𝑢

&

, 𝑣)

𝑠(𝑢

*

, 𝑣)

Second category

of methods

𝑠 𝑢

#

, 𝑣

= 0.7

𝑠 𝑢

!

, 𝑣

= 0.8

𝑠 𝑤, 𝑣

= 0.01

Case 1

𝑠 𝑢

#

, 𝑣

= 0.51

𝑠 𝑢

!

, 𝑣

= 0.50

𝑠 𝑤, 𝑣

= 0.49

Case 2

𝑠 → 0

The discrimination property bears subtle difference from the monotonicity property. Let us look at an illustrative example in the first line in Figure 1. For the central node v , we assume that p ( u 1 | v ) = 0 . 45 , p ( u 2 | v ) = 0 . 4 and p ( w | v ) = 0 . 001 . Case 1 : If s ( u 1 , v ) = 0 . 7 , s ( u 2 , v ) = 0 . 8 and s ( w,v ) = 0 . 01 , the final node embeddings satisfy the discrimination property ( s ( w,v ) s ( u 1 ,v ) = 0 . 0143 and s ( w,v ) s ( u 2 ,v ) = 0 . 0125 , both approaching zero) but do not satisfy the monotonicity property ( p ( u 1 | v ) > p ( u 2 | v ) but s ( u 1 , v ) < s ( u 2 , v ) ). Case 2 : If s ( u 1 , v ) = 0 . 51 , s ( u 2 , v ) = 0 . 50 and s ( w,v ) = 0 . 49 , the final node embeddings fulfill the monotonicity property ( s ( u 1 , v ) > s ( u 2 , v ) > s ( w,v ) ) but do not satisfy the discrimination property ( s ( u 1 ,v ) s ( w,v ) = 1 . 04 and s ( u 2 ,v ) s ( w,v ) = 1 . 02 ). From the above example, we can see that the discrimination property focuses on making node pairs with large proximity closer and pushing away node pairs with small proximity in the embedding space, which is desirable for tasks like binary classification for link prediction (i.e., predicting the existence of a link between two nodes). On the other hand, the monotonicity property pays more attention to the rank of the similarity scores of node pairs, which is critical for tasks like ranking for node recommendation.

2.1 Possibility Results in the Ideal Case

We conduct theoretic analysis about the loss function of network embedding. Previous works [24, 37, 55] have demonstrated that most existing network embedding algorithms can be regarded as implicit matrix factorization and there exists a general form of loss function for the given central node v . If the sigmoid function is adopted as the similarity function. The loss function is:

where σ ( · ) is the sigmoid function, F ( u, :) , F ( v, :) and F ( w, :) are the embeddings of u, v and w respectively, and k is the number of negative samples for each positive sample. If the sigmoid function is replaced by other similarity measurements, we can obtain a more general form of this loss function:

where s ( · , · ) is a general form of similarity measurement.

In the ideal case where an infinite number of positive samples from p ( u | v ) are available, we have the following theorem about the optimal solution of s :

Theorem 1. Optimal Solution of s . The optimal solution of the similarity function s satisfies that for each node pair ( u, v ) ,

Proof.

Since s ( u, v ) is the parameter to be optimized, we can calculate ∇ s ( u,v ) J as following:

Let the derivative to be 0 , we can obtain s ( u, v ) = p ( u | v ) p ( u | v )+ kp n ( u | v ) , which is the optimal solution for J .

With the optimal solution of s ( u, v ) , we give the following proposition about the design of p n :

Proposition 1. The Design of Negative Sampling. In the sampling-based network embedding process, the negative sampling distribution p n should be negatively correlated with the given node pair proximity distribution p to fulfill the monotonicity property.

Proof. Here we prove the above proposition in the ideal case by contradiction: In the ideal case, assume that the best embeddings and corresponding similarity scores to satisfy the discrimination property and the monotonicity property are obtained with p n , and p n is not negatively correlated with p . It is equivalent to the situation that for one central node v , there must exist two nodes u , w with p ( u | v ) > p ( w | v ) and p n ( u | v ) > p n ( w | v ) . From Theorem 1 we know that s ( u, v ) = p ( u | v ) p ( u | v )+ kp n ( u | v ) and s ( w,v ) = p ( w | v ) p ( w | v )+ kp n ( w | v ) . Now, we interchange p n ( u | v ) with p n ( w | v ) and we can get new similarity scores s ′ ( u, v ) and s ′ ( w,v ) for ( u, v ) and ( w,v ) . s ′ ( u, v ) = p ( u | v ) p ( u | v )+ kp n ( w | v ) > p ( u | v ) p ( u | v )+ kp n ( u | v ) = s ( u, v ) . In addition, s ′ ( w,v ) = p ( w | v ) p ( w | v )+ kp n ( u | v ) < p ( w | v ) p ( w | v )+ kp n ( w | v ) = s ( w,v ) . So, we get two new similarity scores that can satisfy the monotonicity property better, which is contradictory with the assumption that we have already achieved the best embeddings and corresponding similarity scores.

Intuitively, this proposition makes sense because if one node is more likely to be sampled as a positive sample for the central node, it is less likely that this node also acts as a negative sample.

From Theorem 1, we can see that if p n is negatively correlated with p , when p ( u | v ) is large and p n ( u | v ) is small, s ( u, v ) → 1 ; and when p ( u | v ) is small and p n ( u | v ) is large, s ( u, v ) → 0 , which satisfies the discrimination property. For all nodes in the network with p ( u 1 | v ) > p ( u 2 | v ) > · · · > p ( u n | v ) , if we set p n ( u 1 | v ) < p n ( u 2 | v ) < · · · < p n ( u n | v ) , the monotonicity property can be fulfilled with s ( u 1 , v ) > s ( u 2 , v ) > · · · > s ( u n , v ) .

2.2 Impossibility Results in the Empirical Case

However, in real-world applications, we can not sample an infinite number of nodes from p ( u | v ) . Empirically, existing algorithms set a fixed number T of positive samples for the central node and the general loss function in Eq. (2) turns into:

where u i is a positive sample from p ( u | v ) and w i is a negative sample from p n ( w | v ) . We use S = [ s ( u 1 , v ) , s ( u 2 , v ) , . . . , s ( u n , v )] to denote the final similarity scores to be optimized w.r.t. the central node v . For the general loss function in the ideal case, the optimal solution of S is S ∗ , where S ∗ = [ p ( u 1 | v ) p ( u 1 | v )+ kp n ( u 1 | v ) , . . . , p ( u n | v ) p ( u n | v )+ kp n ( u n | v ) ] . For the loss function J e in the empirical case with T positive samples from p , we represent the optimal solution as S e . For the mean squared error between S ∗ and S e , we have the following theorem:

Theorem 2. Mean Squared Error . For the mean squared error between the empirical optimal solution S e and the ideal solution S ∗ , we have

where E is the expectation and ( S e -S ∗ ) u = s e ( u, v ) -s ∗ ( u, v ) is the error between the optimal similarity scores obtained in the empirical and ideal cases respectively.

The proof for Theorem 2 is attached in Appendix A.1. If the sigmoid function is adopted as the similarity function, the expectation of the similarity error in Theorem 2 degenerates to its special case:

where F e ( u, :) and F ∗ ( u, :) are the optimal embedding vectors of node u in J e and J . Eq. (8) was first discovered in MCNS [55]. Based on Eq. (8), MCNS advocates that, in order to bound the dot product error for nodes with large p ( u | v ) , p n should be positively correlated with p . Unfortunately, this is not always the best choice due to the following reason. In MCNS, the authors set p n ( u | v ) ∝ p ( u | v ) a , where 0 < a < 1 . With this negative sampling distribution, the right part of Eq. (8) changes into 1 T ( 1 p ( u | v ) (1 + p ( u | v ) 1 -a c ) -1 -1 k ) , where c is a constant. If p ( u | v ) is large, this term can indeed be bounded. However, for distant nodes with small p ( u | v ) (i.e., p ( u | v ) → 0 ), its positively correlated negative sampling probability p n ( u | v ) is also very small and p n ( u | v ) → 0 , which leads to an extremely large error considering that the term 1 p ( u | v ) + 1 kp n ( u | v ) →∞ , which may fail to fulfill the discrimination property.

Discussion . Let us analyze the implications of Theorem 2. If p n ( u | v ) is negatively correlated with p ( u | v ) , it means that kp n ( u | v ) p ( u | v ) → 0 for nodes with the largest p ( u | v ) s and kp n ( u | v ) p ( u | v ) →∞ for nodes with the smallest p ( u | v ) s. Let us consider the case that p ( u | v ) is large and kp n ( u | v ) p ( u | v ) → 0 . E [ || ( S e -S ∗ ) u || 2 ] < 1 T ( p ( u | v ) kpn ( u | v ) ) 2 ( 1 p ( u | v ) + 1 kp n ( u | v ) -1 -1 k ) < 1 T ( kp n ( u | v ) p ( u | v ) ) 2 ( 1 p ( u | v ) + 1 kp n ( u | v ) ) . It can be rewritten as 1 T (( kp n ( u | v ) p ( u | v ) ) 2 1 p ( u | v ) + kp n ( u | v ) p ( u | v ) 1 p ( u | v ) ) . Since p ( u | v ) is among the largest proximity scores, 1 p ( u | v ) is approximately bounded by a finite number 1 n , where n is the number of nodes in the network. As a result, kp n ( u | v ) p ( u | v ) 1 p ( u | v ) → 0 and ( kp n ( u | v ) p ( u | v ) ) 2 1 p ( u | v ) → 0 . Therefore, for nodes with large p ( u | v ) , the empirical embedding similarity score s e ( u, v ) ≈ s ∗ ( u, v ) . With a similar analysis on nodes with small p ( u | v ) , we reach the same conclusion that s e ( u, v ) ≈ s ∗ ( u, v ) . Based on the above analysis, we can see that the empirical optimal solution can achieve the discrimination property with a negatively correlated p n : for node pairs with large/small p ( u | v ) , their similarity scores approach the ideal optimal solution. However, there might exist some nodes with p ( u | v ) ≈ kp n ( u | v ) , which are referred to as nodes with intermediate p ( u | v ) . For these nodes, the error in Theorem 2 is larger than nodes with small/large p ( u | v ) because (2 + kp n ( u | v ) p ( u | v ) + p ( u | v ) kp n ( u | v ) ) 2 achieves its minimum when p ( u | v ) = kp n ( u | v ) . This can also be explained intuitively as follows. Nodes with large p ( u | v ) and large p n ( u | v ) (small p ( u | v ) ) are always sampled as positive samples and negative samples, while nodes with intermediate p ( u | v ) and p n ( u | v ) might be ignored and are less likely to be sampled as positive samples or negative samples. As such, the embedding similarity scores for these nodes bear more uncertainty and large error between the empirical optimal solution and the ideal optimal solution (Theorem 2).

With these findings, we can now unveil the underlying reason for the competing sampling strategies in existing methods. The first category of methods (e.g., [36, 14, 23]) champion sampling nodes with longer distance from the given central node as positive samples. The essence of this positive sampling strategy is to include more nodes with intermediate p ( u | v ) in the positively sampled node pair set. In this way, these methods realize the goal to minimize the error in Theorem 2 for these nodes with intermediate p ( u | v ) . However, this design has cost. When choosing more nodes with intermediate p ( u | v ) as positive samples, it implicitly decreases the p ( u | v ) for nodes with large p ( u | v ) (since ∑ u p ( u | v ) = 1 ). As shown on the second line in Figure 1, it hurts the monotonicity between these nodes (round nodes) and nodes with large p ( u | v ) (star nodes). On the contrary, for the second category of methods (e.g., [4, 56, 55]), they prefer to select nodes with intermediate p ( u | v ) as difficult negative samples in the recommendation task, they successfully fulfill the monotonicity property between nodes with large p ( u | v ) (star nodes) and nodes with intermediate p ( u | v ) (round nodes). Unfortunately, as our discussion about Eq. (8) has demonstrated, this positively correlated negative sampling distribution could make the empirical error of the embedding similarity scores of nodes with small p ( u | v ) quite large. It does harm the discrimination property, which is shown on the last line in Figure 1 (e.g., s ( u 8 , v ) ).

To conclude, due to the limited number T of positive samples, the empirical loss tends to sample nodes with large p ( u | v ) or p n ( u | v ) and there always exist some nodes that can not be sampled as positive/negative samples and thus are ignored. Therefore, the discrimination property and the monotonicity property for all node pairs in the network can not be satisfied simultaneously in the empirical situation.

3 Model

In many real-world applications, given the query node, we primarily care about the topK recommendation list. This suggests that we only need to satisfy the discrimination property and the partial monotonicity property within nodes that are likely to appear in the topK recommendation list. Guided by this and the theoretical results in Section 2, we propose a simple yet novel model named SENSEI.

Key Idea. The key idea of SENSEI is to seamlessly integrate two competing sampling strategies into one model together. Concretely, it means that we can decompose the proposed SENSEI into two steps. The first step is to satisfy the discrimination property and the second step is to achieve the monotonicity property within nodes that are likely to appear in the topK recommendation list. As shown in Figure 2, in the first step, SENSEI samples nodes with intermediate p ( u | v ) , which is the strategy by the first category of methods mentioned in Section 2. SENSEI constructs a combined positive sample set, including nodes with large p ( u | v ) and nodes with intermediate p ( u | v ) . During the training process, SENSEI treats these nodes equivalently as positive samples and maximizes their similarity scores with the given central node. At the same time, it minimizes the similarity scores for nodes with small p ( u | v ) (large p n ( u | v ) ) to fulfill the discrimination property. In the second step, SENSEI focuses on the difficult negative samples, which is the strategy by the second category of methods in Section 2. SENSEI pays attention to the monotonicity property within the combined positive sample set in the first step. In detail, the previously sampled positive nodes have two roles: they act as the positive samples compared to nodes with smaller p ( u | v ) in the first step and meanwhile function as the difficult negative samples compared to nodes with larger p ( u | v ) in the second step.

Details. For p ( u | v ) , we run personalized pagerank [33], which is used throughout SENSEI. In addition, we normalize node embeddings to have unit L2 norm and adopt the dot product ( F ( u, :) ⊤ F ( v, :) ) as the similarity function s ( u, v ) .

Step 1: Fulfill the Discrimination Property. In this step, to satisfy the discrimination property, we construct a combined positive sample set P ( v ) for v in A , which includes nodes with large and intermediate p ( u | v ) . Specifically, we rank the proximity p ( u | v ) for all nodes in A to obtain a descending list [ p ( u 1 | v ) , p ( u 2 | v ) , . . . , p ( u n | v )] . Then, we set a threshold τ . Nodes with p ( u | v ) > τ form the combined positive sample set P ( v ) for the central node v and the remaining nodes are added into the set N ( v ) , where negative samples will be randomly selected. So, the loss for node v is:

𝑠(𝑢

!

, 𝑣)

𝑠 → 1

𝑠 → 0

𝑠(𝑢

"

, 𝑣)

𝑠(𝑢

#

, 𝑣)

𝑠(𝑢

$

, 𝑣)

𝑠(𝑢

%

, 𝑣)

𝑠(𝑢

&

, 𝑣)

𝑠(𝑢

'

, 𝑣)

𝑠(𝑢

#(

, 𝑣)

𝑠(𝑢

)

, 𝑣)

𝑠(𝑢

*

, 𝑣)

Step 1

𝑠(𝑢

'

, 𝑣)

𝑠 → 0

𝑠(𝑢

$

, 𝑣)

𝑠(𝑢

#(

, 𝑣)

𝑠(𝑢

!

, 𝑣)

𝑠(𝑢

%

, 𝑣)

𝑠(𝑢

&

, 𝑣)

𝑠(𝑢

#

, 𝑣)

𝑠(𝑢

"

, 𝑣)

𝑠(𝑢

)

, 𝑣)

𝑠(𝑢

*

, 𝑣)

𝑠(𝑢

#

, 𝑣)

𝑠(𝑢

"

, 𝑣)

𝑠(𝑢

#(

, 𝑣)

𝑠(𝑢

!

, 𝑣)

𝑠(𝑢

%

, 𝑣)

𝑠(𝑢

'

, 𝑣)

𝑠(𝑢

$

, 𝑣)

𝑠(𝑢

&

, 𝑣)

𝑠(𝑢

)

, 𝑣)

𝑠(𝑢

*

, 𝑣)

Step 2

𝑠 → 1

where k is the negative sample number for each positive sample, u is the positively sampled node for v and w m ∈ N ( v ) is the negatively sampled node. By adding the loss for all v s in A , we obtain the

overall loss in Step 1 as:

Step 2: Fulfill the Partial Monotonicity Property. In this step, we focus on the partial monotonicity property within the combined positive sample set in Step 1. For any two nodes u l and u m in the positive sample set P ( v ) , if p ( u l | v ) > p ( u m | v ) , we want to retain the monotonicity between them with F ( u l , :) ⊤ F ( v, :) > F ( u m , :) ⊤ F ( v, :) . Node u m changes from a positive sample for v in Step 1 to a negative sample for v compared with u l , which is consistent with the idea of sampling difficult negative samples. The margin-based ranking loss is:

✶ where u l , u m ∈ P ( v ) , ✶ p ( u l | v ) >p ( u m | v ) is the indicator function and γ is a positive margin. When p ( u l | v ) > p ( u m | v ) , ✶ p ( u l | v ) >p ( u m | v ) = 1 , otherwise, ✶ p ( u l | v ) >p ( u m | v ) = 0 . Similar to Eq. (10) in Step 1, we have the loss L p in Step 2 as:

In fact, Step 2 can be regarded as a fine-tuning process to fulfill the partial monotonicity property. Therefore, this step usually has a small learning rate and a small positive margin γ . The pseudocode of this algorithm can be found in Appendix A.2.

Complexity Analysis. We give a complexity analysis of SENSEI here. For SENSEI, the time complexity can be analyzed as the following: (1) Calculating personalized pagerank has the complexity O ( iter max · ∥E∥ ) , where ∥E∥ is the number of edges in the graph and iter max is the maximum iterations or hops for personalized pagerank; (2) the time complexity for sorting the proximity distribution p ( u | v ) and sampling positive/negative nodes for node v is O ( n · log( n )) ; (3) Calculating J ( v ) takes O ( ∥P ( v ) ∥ · k · d ) , where ∥P ( v ) ∥ is the positive sample set, k is the number of negative samples and d is the dimension of node embedding. Obtaining J P takes O ( n ) time; (4) Computing L ( v ) in Eq. (11) takes O ( ∥P ( v ) ∥ 2 · d ) and calculating L p costs O ( n ) . Most of these computations can be further parallelized.

4 Experiment

In this section, we evaluate the effectiveness of the proposed algorithm (SENSEI) for solving link prediction and node recommendation simultaneously in plain networks.

4.1 Experimental Setup

Datasets. We use 4 public real-world datasets to evaluate the proposed SENSEI model: C.ele [51], Cora [39], Citeseer [39], NS [31].

Baselines. We compare the proposed SENSEI with the following 5 plain network embedding methods: node2vec [14], VGAE [22], GAT [43], ARGVA [35] and RBGE [17].

Metrics. For link prediction and node recommendation in plain networks, we randomly split edges in every dataset into 70/10/20% for training, validation, and test. 2 The same amount of additionally sampled non-existent edges are taken as negative edges for training, validation and test in the link prediction task.

Settings. For the link prediction task in plain networks, we use Area Under the ROC and PrecisionRecall Curves (i.e., AUC-ROC and AUC-PR) as the metrics to evaluate the performance of different methods. For the node recommendation task in plain networks, we take Hit@ K as the metric. Only when the model ranks the correct node as topK in the recommendation list, it is counted as a hit for this query node in the test set. Therefore, the average hit number for all query nodes is Hit@ K . We set K = 10 for SENSEI.

Additional Contents. In Appendix, we provide more contents related to experiments, including (1) dataset statistics (Appendix A.3); (2) implementation details (Appendix A.3); (3) the comparison with additional GCN-based methods: GraphSAGE [15], Cluster-gcn [5], APPNP [13], GPRGNN [6] and H2GCN [63] (Appendix A.4).

4.2 Effectiveness of SENSEI

The results of link prediction on all 4 plain networks are presented in Table 1. Our proposed SENSEI generally outperforms all baselines on all 4 datasets. In particular, SENSEI achieves about 2% improvement in AUC-PR compared to the best competitor (e.g., VGAE on Citeseer and node2vec on C.ele). For the node recommendation task, as shown in Table 2, node2vec obtains the best Hit@10 on Cora and NS among all baselines, which indicates that including more positive samples ( T ) by setting a larger walk length (80) can indeed reduce the error in Theorem 2. Notice that the proposed SENSEI still outperforms node2vec on all 4 datasets, which demonstrates the ability of SENSEI to fulfill the partial monotonicity property .

4.3 Ablation Study and Sensitivity Study

In this subsection, we conduct the ablation study to validate the importance of Step 2 of SENSEI. As illustrated in Table 2, for the node recommendation task (Hit@10), Step 2 consistently improves the performance of SENSEI. The results verify that Step 2 is indeed beneficial by fine-tuning node embeddings to satisfy the partial monotonicity property. In addition, we carry out a sensitivity study on different K s for Hit@ K , which is presented in Figure 3. We observe that when K increases from 1 to 30, Hit@ K gradually increases and Hit@30 is much higher than Hit@1, which is expected.

5 Related works

Network Embeddings. Network embedding maps nodes in the network to low dimensional vectors. It can be traced back to matrix low rank approximation [26] and spectral clustering [40]. DeepWalk [36] and node2vec [14] rely on random walk to encode the topological information of homogeneous networks. For multi-layered networks or heterogeneous networks, metapath2vec [7] and Hin2vec [10] design metapaths to capture the connectivity between different layers or different types of nodes. Since graph convolutional network (GCN) [21] emerges, many GCNbased network embedding methods have been proposed. For example, variational graph autoencoder (VGAE) [22] adopts the same message-passing mechanism as GCN to embed the homogeneous network. More recently, GCN has been integrated with random walk based heterogeneous network embedding methods, which leads to heterogeneous graph convolutional networks like MAGNN [11] and HAN [50]. MANE [25] successfully employs network embedding on multilayered networks and DMGC [29] accomplishes both network embedding and clustering by utilizing cross-layer links as regularization. In addition, many embedding algorithms for knowledge graph have been proposed such as translational distance model [3] and semantic matching model [32].

Sampling in Network Embedding. Sampling is an important technique, which appears in recommendation [34] and text embedding [30] to speed up the training process. For network embedding algorithms, both the positive sampling strategy and the negative sampling strategy have been applied and improved. For matrix factorization [26], spectral clustering [40] and LINE [41], direct or two-hop neighboring nodes act as positive samples and negative nodes are sampled uniformly or according to the degree distribution. In DeepWalk [36], node2vec [14] and metapath2vec [7], the positive sampling strategy is modified based on truncated random walk starting from the central node. The positive and negative sampling are implemented implicitly with a message-passing mechanism in GCN-based methods, where nodes to be aggregated can be viewed as positive samples and the

0

5

10

15

20

25

30

K

0

20

40

60

80

100

Hit@

K

(%)

Citeseer

NS

C.ele

remaining nodes are implicit negative samples. For the GCN related works, efforts are mainly made to improve the positive sampling strategy. For example, APPNP [13] manipulates positive samples with personalized pagerank [33] to catch long distance information and PGNN [58] randomly fixes an anchor node set for aggregation in each training epoch. Recently, more attention has been paid to improve the negative sampling strategy. Many works suggest to sample closer nodes as difficult negative samples. For instance, MCNS [55] proposes that the negative sampling distribution should be positively correlated with the positive sampling distribution. KBGAN [4] samples most similar entities to replace the groundtruth positive entity in knowledge graph embedding and RecNS [56] tends to sample negative nodes from the intermediate distance region.

6 Conclusion and Limitations

In this paper, we study the sampling strategies of network embedding. To uncover the underlying relationships of existing competing sampling strategies, we conduct theoretical analysis on the sampling-based network embedding process. In the analysis, we identify two desirable properties for the similarity scores of node embedding, including the discrimination property and the monotonicity property. Furthermore, we prove that there always exists an error gap between the empirical and ideal optimal embedding similarity scores. Guided by such analysis, we propose a simple yet novel model (SENSEI), which creatively integrates the two competing sampling strategies to fulfill the the discrimination property and the partial monotonicity property. The effectiveness of SENSEI is verified by extensive experiments.

This paper studies sampling strategies of network embedding, which has no negative ethical impacts on society. The limitations of our paper lie in that the theoretical analysis is conducted on proximity scores and the proposed SENSEI model is designed for plain (non-attributed) networks rather than attributed networks. The key to generalize the proposed properties/SENSEI model to attributed networks is a new definition of the node pair similarity, which we leave for future exploration.

References

Yikun Ban, Yuchen Yan, Arindam Banerjee, and Jingrui He. Ee-net: Exploitation-exploration neural networks in contextual bandits. arXiv preprint arXiv:2110.03177 , 2021.

Yikun Ban, Yuchen Yan, Arindam Banerjee, and Jingrui He. Neural exploitation and exploration of contextual bandits. arXiv preprint arXiv:2305.03784 , 2023.

Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems , 26, 2013.

Liwei Cai and William Yang Wang. Kbgan: Adversarial learning for knowledge graph embeddings. arXiv preprint arXiv:1711.04071 , 2017.

Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 257-266, 2019.

Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. arXiv preprint arXiv:2006.07988 , 2020.

Yuxiao Dong, Nitesh V Chawla, and Ananthram Swami. metapath2vec: Scalable representation learning for heterogeneous networks. In KDD '17 , pages 135-144. ACM, 2017.

Boxin Du, Si Zhang, Yuchen Yan, and Hanghang Tong. New frontiers of multi-network mining: Recent developments and future trend. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 4038-4039, 2021.

Dongqi Fu, Liri Fang, Ross Maciejewski, Vetle I Torvik, and Jingrui He. Meta-learned metrics over multi-evolution temporal graphs. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 367-377, 2022.

Tao-yang Fu, Wang-Chien Lee, and Zhen Lei. Hin2vec: Explore meta-paths in heterogeneous information networks for representation learning. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management , pages 1797-1806, 2017.

Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King. Magnn: Metapath aggregated graph neural network for heterogeneous graph embedding. In Proceedings of The Web Conference 2020 , pages 2331-2341, 2020.

Hongchang Gao and Heng Huang. Self-paced network embedding. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 1406-1415, 2018.

Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05997 , 2018.

Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining , pages 855-864, 2016.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Xiaotian Han, Zhimeng Jiang, Ninghao Liu, and Xia Hu. G-mixup: Graph data augmentation for graph classification. In International Conference on Machine Learning , pages 8230-8248. PMLR, 2022.

Zexi Huang, Arlei Silva, and Ambuj Singh. A broader picture of random-walk based graph embedding. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining , pages 685-695, 2021.

Baoyu Jing, Hanghang Tong, and Yada Zhu. Network of tensor time series. In Proceedings of the Web Conference 2021 , pages 2425-2437, 2021.

Baoyu Jing, Yuchen Yan, Kaize Ding, Chanyoung Park, Yada Zhu, Huan Liu, and Hanghang Tong. Sterling: Synergistic representation learning on bipartite graphs. arXiv preprint arXiv:2302.05428 , 2023.

Baoyu Jing, Yuchen Yan, Yada Zhu, and Hanghang Tong. Coin: Co-cluster infomax for bipartite graphs. arXiv preprint arXiv:2206.00006 , 2022.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 , 2016.

Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05997 , 2018.

Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. Advances in neural information processing systems , 27, 2014.

Jundong Li, Chen Chen, Hanghang Tong, and Huan Liu. Multi-layered network embedding. In Proceedings of the 2018 SIAM International Conference on Data Mining , pages 684-692. SIAM, 2018.

Chih-Jen Lin. Projected gradient methods for nonnegative matrix factorization. Neural computation , 19(10):2756-2779, 2007.

Hongyi Ling, Zhimeng Jiang, Meng Liu, Shuiwang Ji, and Na Zou. Graph mixup with soft alignments. arXiv preprint arXiv:2306.06788 , 2023.

Lihui Liu, Boxin Du, Jiejun Xu, Yinglong Xia, and Hanghang Tong. Joint knowledge graph completion and question answering. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1098-1108, 2022.

Dongsheng Luo, Jingchao Ni, Suhang Wang, Yuchen Bian, Xiong Yu, and Xiang Zhang. Deep multi-graph clustering via attentive cross-graph association. In Proceedings of the 13th international conference on web search and data mining , pages 393-401, 2020.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems , 26, 2013.

Mark EJ Newman. Finding community structure in networks using the eigenvectors of matrices. Physical review E , 74(3):036104, 2006.

Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. A three-way model for collective learning on multi-relational data. In Icml , 2011.

Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab, 1999.

Rong Pan, Yunhong Zhou, Bin Cao, Nathan N Liu, Rajan Lukose, Martin Scholz, and Qiang Yang. One-class collaborative filtering. In 2008 Eighth IEEE International Conference on Data Mining , pages 502-511. IEEE, 2008.

Shirui Pan, Ruiqi Hu, Guodong Long, Jing Jiang, Lina Yao, and Chengqi Zhang. Adversarially regularized graph autoencoder for graph embedding. arXiv preprint arXiv:1802.04407 , 2018.

Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 701-710, 2014.

Jiezhong Qiu, Yuxiao Dong, Hao Ma, Jian Li, Kuansan Wang, and Jie Tang. Network embedding as matrix factorization: Unifying deepwalk, line, pte, and node2vec. In Proceedings of the eleventh ACM international conference on web search and data mining , pages 459-467, 2018.

Shane Roach, Connie Ni, Alexei Kopylov, Tsai-Ching Lu, Jiejun Xu, Si Zhang, Boxin Du, Dawei Zhou, Jun Wu, Lihui Liu, et al. Canon: Complex analytics of network of networks for modeling adversarial activities. In 2020 IEEE International Conference on Big Data (Big Data) , pages 1634-1643. IEEE, 2020.

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina EliassiRad. Collective classification in network data. AI magazine , 29(3):93-93, 2008.

Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence , 22(8):888-905, 2000.

Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Largescale information network embedding. In Proceedings of the 24th international conference on world wide web , pages 1067-1077, 2015.

Yuxin Tang, Zhimin Ding, Dimitrije Jankov, Binhang Yuan, Daniel Bourgeois, and Chris Jermaine. Auto-differentiation of relational computations for very large scale machine learning. arXiv preprint arXiv:2306.00088 , 2023.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

Dingsu Wang, Yuchen Yan, Ruizhong Qiu, Yada Zhu, Kaiyu Guan, Andrew Margenot, and Hanghang Tong. Networked time series imputation via position-aware graph enhanced variational autoencoders. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2256-2268, 2023.

Ruijie Wang, Baoyu Li, Yichen Lu, Dachun Sun, Jinning Li, Yuchen Yan, Shengzhong Liu, Hanghang Tong, and Tarek F Abdelzaher. Noisy positive-unlabeled learning with self-training for speculative knowledge graph reasoning. arXiv preprint arXiv:2306.07512 , 2023.

Ruijie Wang, Zheng Li, Dachun Sun, Shengzhong Liu, Jinning Li, Bing Yin, and Tarek Abdelzaher. Learning to sample and aggregate: Few-shot reasoning over temporal knowledge graphs. Advances in Neural Information Processing Systems , 35:16863-16876, 2022.

Ruijie Wang, Zheng Li, Jingfeng Yang, Tianyu Cao, Chao Zhang, Bing Yin, and Tarek Abdelzaher. Mutually-paced knowledge distillation for cross-lingual temporal knowledge graph reasoning. In Proceedings of the ACM Web Conference 2023 , pages 2621-2632, 2023.

Ruijie Wang, Zheng Li, Danqing Zhang, Qingyu Yin, Tong Zhao, Bing Yin, and Tarek Abdelzaher. Rete: Retrieval-enhanced temporal event forecasting on unified query product evolutionary graph. arXiv preprint arXiv:2202.06129 , 2022.

Ruijie Wang, Yuchen Yan, Jialu Wang, Yuting Jia, Ye Zhang, Weinan Zhang, and Xinbing Wang. Acekg: A large-scale knowledge graph for academic data mining. In Proceedings of the 27th ACM international conference on information and knowledge management , pages 1487-1490, 2018.

Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Heterogeneous graph attention network. In The World Wide Web Conference , pages 2022-2032, 2019.

Duncan J Watts and Steven H Strogatz. Collective dynamics of 'small-world'networks. nature , 393(6684):440-442, 1998.

Yuchen Yan, Lihui Liu, Yikun Ban, Baoyu Jing, and Hanghang Tong. Dynamic knowledge graph alignment. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 4564-4572, 2021.

Yuchen Yan, Si Zhang, and Hanghang Tong. Bright: A bridging algorithm for network alignment. In Proceedings of the Web Conference 2021 , pages 3907-3917, 2021.

Yuchen Yan, Qinghai Zhou, Jinning Li, Tarek Abdelzaher, and Hanghang Tong. Dissecting cross-layer dependency inference on multi-layered inter-dependent networks. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management , pages 2341-2351, 2022.

Zhen Yang, Ming Ding, Chang Zhou, Hongxia Yang, Jingren Zhou, and Jie Tang. Understanding negative sampling in graph representation learning. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 1666-1676, 2020.

Zhen Yang, Ming Ding, Xu Zou, Jie Tang, Bin Xu, Chang Zhou, and Hongxia Yang. Region or global a principle for negative sampling in graph-based recommendation. IEEE Transactions on Knowledge and Data Engineering , pages 1-1, 2022.

Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning , pages 40-48. PMLR, 2016.

Jiaxuan You, Rex Ying, and Jure Leskovec. Position-aware graph neural networks. In International Conference on Machine Learning , pages 7134-7143. PMLR, 2019.

Zhichen Zeng, Boxin Du, Si Zhang, Yinglong Xia, Zhining Liu, and Hanghang Tong. Hierarchical multi-marginal optimal transport for network alignment. arXiv preprint arXiv:2310.04470 , 2023.

Zhichen Zeng, Si Zhang, Yinglong Xia, and Hanghang Tong. Parrot: Position-aware regularized optimal transport for network alignment. In Proceedings of the ACM Web Conference 2023 , pages 372-382, 2023.

Qinghai Zhou, Liangyue Li, Nan Cao, Lei Ying, and Hanghang Tong. Admiring: Adversarial multi-network mining. In 2019 IEEE International Conference on Data Mining (ICDM) , pages 1522-1527. IEEE, 2019.

Qinghai Zhou, Liangyue Li, Xintao Wu, Nan Cao, Lei Ying, and Hanghang Tong. Attent: Active attributed network alignment. In Proceedings of the Web Conference 2021 , pages 3896-3906, 2021.

Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems , 33:7793-7804, 2020.

A Appendix

The appendix is structured as follows:

Subsection A.1 gives the proof of Theorem 2;

Subsection A.2 gives the detailed algorithm of SENSEI;

Subsection A.3 introduces the statistics of datasets and the implementation details of SENSEI;

Subsection A.4 compares the performance of SENSEI with 5 additional GCN baselines (Cluster-gcn, GraphSAGE, APPNP, GPRGNN and H2GCN) on Cora and Citeseer.

A.1 Proofs

Theorem 2. Mean Squared Error of S between the Ideal Loss and the Empirical Loss . For the mean squared error between S ∗ and S e , we have

where E is the expectation and S ∗ = [ p ( u 1 | v ) p ( u 1 | v )+ kp n ( u 1 | v ) , . . . , p ( u n | v ) p ( u n | v )+ kp n ( u n | v ) ] .

Proof. In this proof, since S = [ s ( u 1 , v ) , s ( u 2 , v ) , . . . , s ( u n , v )] is the parameter to be optimized, we can write the empirical loss J e as a function of S as J e ( S ) . We prove this theorem with the help of Taylor expansion of ∇ S J e ( S e ) around S ∗ . Because S e is the solution to minimize J e , ∇ S J e ( S e ) = 0 . So, it can be expressed as:

Thus, up to terms of order O ( || S e -S ∗ || 2 ) , we obtain

Then, we discuss the term -( ∇ 2 J e ( S ∗ )) -1 and the term ∇ S J e ( S ∗ ) respectively.

For the term -( ∇ 2 J e ( S ∗ )) -1 , let e be the one-hot vector, where only the u i -th entry is 1.

S S ( u i )

Since s ∗ ( u i , v ) = p ( u i | v ) p ( u i | v )+ kp n ( u i | v ) and s ∗ ( w i , v ) = p ( w i | v ) p ( w i | v )+ kp n ( w i | v ) ,

Therefore,

where the last line is because that we combine same terms when u i = w i .

Then, we consider the second term ∇ S J e ( S ∗ ) . We analyze the expectation and variance of ∇ S J e ( S ∗ ) to get the final E [ || ( S e -S ∗ ) u || 2 ] .

For the expectation, as ∇ S J e ( S ) is as follows:

the expectation is:

For the variance of ∇ S J e ( S ∗ ) , since E [ ∇ S J e ( S ∗ )] = 0

Based on Eq. (15), Eq. (18), Eq. (22) and E [ || ( S e -S ∗ ) u || 2 ] = Cov ( S e -S ∗ )( u, u ) , we have

which can be simplified as:

A.2 The Algorithm of SENSEI

In this section, we give the detailed algorithm of SENSEI in Algorithm 1.

A.3 Dataset Statistics and Implementation Details.

In this section, we introduce the statstics of datasets and the implementation details of SENSEI. The statistics of datasets are present in Table 3.

Implementation Details. For all the methods, we run 5 times to calculate the standard deviation and for all baselines, we use their default parameters. For SENSEI on 4 datasets: {C.ele, Cora, Citeseer,

NS}, we set the threshold τ as { 0 . 008 , 0 . 01 , 0 . 005 , 0 . 05 } , the number of epochs in Step 1 as {40, 100, 20, 20}, the number of epochs in Step 2 as {40, 100, 50, 20}, the learning rate in Step 1 as {0.02, 0.1, 0.1, 0.2}, the learning rate in Step 2 as {0.01, 0.01, 0.005, 0.1}, the positive margin γ as {0.05, 0.0001, 0.0001, 0.1} and the negative sample number k as 40 on all 4 datasets. All experiments are run on a Tesla-V100 GPU. 3

A.4 Additional Baselines of Plain Network Embedding

We have added another 5 GNNs as additional baselines: GraphSAGE [15], Cluster-gcn [5], APPNP [13], GPRGNN [6] and H2GCN [63]. the results are demonstrated in Table 4. We can see that for both link prediction and node recommendation, SENSEI generally beats these GNN methods.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper first introduces two desirable properties for network embedding. The discrimination property expresses how well the model is able to distinguish node pairs that are similar from those which are dissimilar. The monotonicity property expresses how the similarity scores obtained from the embedding follow the same order as the probability of a pair in the original dataset. The authors first define these two properties, argue why they are important for network embedding methods, and illustrate them with an example.\nThen, they consider a general framework for network embedding algorithm.\n\nThey present a standard expression of the likelihood of such models, discuss the elaboration of positive and negative samples sets, and derive elementary results of such models regarding the monotonicity and discrimination properties. They show both can be satisfied in an ideal case where infinite data is available (possibility results), and how both cannot be simultaneously satisfied for finite dataset (impossibility results). Doing so, they derive an optimal expression for the similarity between pairs of nodes, and how the positive sampling distribution should (negatively) correlate with the negative sampling distribution in order to satisfy monotonicity. Finally, they analyse the expected gap between empirical optimal solutions (finite dataset) and theoretical optimal solutions (infinite dataset).\n\nThey discuss the implication of these demonstrations. In particular, they detail some caveats of state-of-the-art sampling strategies in light of their findings.\n\nFinally, they design a simple embedding procedure (SENSEI) that focuses on satisfying the discrimination and monotonicity properties to some extent. They show on 4 real-world datsets how their approach outperforms existing methods both in terms of link prediction (related to the discrimination property) and node recommendation (related to the monotonicity property).\n\nThey conclude the paper with a short overview of existing works on network embedding.\n\nSTRENGTHS:\n- The problem discussed is general; many previous works could be studied in the light of these findings\n- Writing is clear and sections are well organized.\n- Experiments are well carried out, with ablation tests, numerous baselines and four real-world datasets.\n- The discussion of previous works limits in the light of some findings (Thm.2) is interesting and insightful.\n\nWEAKNESSES:\n- Illustrations could be improved. The first line of Fig.1 is not very helpful, gray bars are not explicitly defined, and the symbolic representation of p(u|v) is unclear. I believe simpler visualisations are doable. Also, figures should take the entire width of the page, in my opinion, instead of floating in the middle of the text.\n- There are several mentions of \"large\" and \"small\" values of p and s. While it helps get a sense of what is discussed, I believe these should be better defined, as they are key words in the demonstrations. In line 233, the authors introduce the \"intermediate\" values with a rigorous definition; I believe discussing this above in the text might help to reach an explicit definition of \"large\" and \"small\".\n- All demonstrations are in the appendix. I understand that available space is an issue for large mathematical formulas, but proofs for Th.1 and Prop.1, that are key in the text, are short enough to be included in the main text. In their absence, the main paper cannot be considered as self-sufficient.\n\nQUESTIONS:\n- l.232-233 - observing the deviation of p(u|v) entries could help choosing an optimal number for k?\n\n- l.141 - Definition of discrimination property : because s \\in [0;1] and p \\in [0;1], the definition seems to boil down to s(w|v)->0 when p(w|v)->0. Also, the definition only considers extreme cases where two nodes are so dissimilar the ratios of p and s diverge; what about intermediate cases, such as the ones given in the paragraph under the definitions? What does \"clearly distinguished in the embedding space\" mean?\n\n- l.144-166 - While I understand this is an illustrative example, I believe there should be at least a small discussion the chosen values of s(.,.). For instance, why would s(u1,v)<s(u2,v) given p(u1|v)>p(u2|v)? Does it happen often? Same for the very close values of s in case 2.\n\n- Eq.7 - Why does k appear along with positive samples, when it is absent from Eq.4?\n\nAdditional comments:\n\nl.107 - the link between discrimination/monotonicity and link pred/node rec should be stated higher in the text\n\nFig.1 - Maybe a matter of taste, but text should not wrap figures. The signification of the two-way gray arrows should be explicited in the legend. Besides, I am not sure that the figure helps the understanding of lines 144-166.\n\nEq.1 - I think the transition between Eq.1 and Eq.2 would be clearer if sigma(-F(w,:)^T F(v,:)) would be directly written as 1-sigma(F(w,:)^T F(v,:)). Also, there should be a reference to Def.1, because some s(.,.) could take values that are not in [0,1]. Dissimilar vector can yield a negative cosine similarity, for instance.\n\nAppendix A.2, Th.1 - Because there is no space constraint, it would be clearer to explicit the definition of the expectation E before differentiating.\n\nAppendix A.2, Th.2, l.600 - Typo \"is is\".\n\nl.206 - Specify the theorem holds for a 1st order Taylor expansion.\n\nl.211-220 - Nice discussion.\n\nl.241-256 - Fig.1 should be placed and explained here. Plotting fictive values of p(u|v) on top of the s(.,.) lines (instead of symbols) would ease the understanding.\n\nTables - Results with overlapping std should be emphasized in a similar way (bold or underline)\n\nLIMITATIONS:\nIn the words of the authors, with which I agree: \n\"The limitations of our paper lie in that the theoretical analysis is conducted on proximity scores and the proposed SENSEI model is designed for plain (non-attributed) networks rather than attributed networks. The key to generalize the proposed properties/SENSEI model to attributed networks is a new definition of the “proximity” (i.e., node pair similarity). Currently, the proximity refers to the topological aspect of the node pair similarity.\"\nThese limitations lay the ground for broader future theoretical works on the topic.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work targets on network embedding under different strategies. The authors first theoretically demonstrate the dilemma in existing network embedding algorithms, especially the sampling-based framework, that the sampling-based embedding performance can have an error gap due to the competing sampling strategies.  Then, they propose two properties, i.e., discrimination and monotonicity, that should be obeyed, and eventually, under this perspective, they propose a novel network embedding approach called SENSEI to perform the (plain) network embedding. The SENSEI includes two steps, where the first step samples nodes with intermediate distribution following the first category of sampling strategies to satisfy the discrimination property, while the second step follows the second category of sampling strategies to partially satisfy the monotonicity property. Experiments on several datasets show the effectiveness of the proposed approach.\n\nSTRENGTHS:\n+ This work has a clear motivation to devise a new reconciling strategy to satisfy the discrimination and monotonicity properties. \n+ The theoretical analysis is sound and details the significance of the two steps. Corresponding discussions clearly tell the advantages of the proposed embedding approach.\n+ The performance of SENSEI is not incremental in both link prediction and node recommendation tasks. Though the work mainly performs on plain networks, the authors still give the results on multi-layer networks.\n+ The proposed approach seems not to be difficult to reproduce.\n\nWEAKNESSES:\n- This work can only be effective on sampling-based embedding algorithms, though I think it is valuable for this field.\n- According to the results of Table 2, the promotion of step 2 is incremental. However, will step 2 harm the performance of the link prediction task? (See my second question in `Questions’) If so, the necessity of reconciling the competing sampling strategies is not very strong.\n\nQUESTIONS:\n+ Why is this approach called `SENSEI’?\n+ According to intuition, the SENSEI-P (w/o Step 2) will perform better than SENSEI-P on the link prediction task. Is it right?\n\nLIMITATIONS:\nYes. The authors claim that the attribute could have helped promote the approach.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGraph embedding methods typically involve negative sampling, where a set of nodes are sampled as positive samples (usually nodes of close proximity, such as edges) and a set of nodes are sampled as negative samples. While early work tended to randomly sample negatives from non-edges, more recent work has adopted different approaches for negative sampling. The authors identify two main classes of methods for negative sampling in the literature. Their main contribution is identifying two properties that they claim any good set of node embeddings should satisfy: discrimination and monotonicity. They then demonstrate that these two properties cannot be simultaneously satisfied with a finite number of samples. They propose the SENSEI method that satisfies discrimination and a *partial* monotonicity property and demonstrate strong empirical performance.\n\n*After rebuttal:* The authors have clarified my concerns, and I continue to support the paper. I also did not see anything from the other reviews that would negatively affect my opinion.\n\nSTRENGTHS:\n- Provides an explanation unifying different negative sampling strategies for graph embeddings based on different desirable properties. This is a novel way of thinking about the graph embedding problem.\n- Interesting theoretical results with practical applicability: a possibility theorem to satisfy the two desirable properties in with infinite samples along with an *impossibility* theorem to satisfy the two desirable properties in practice with finite samples.\n- New SENSEI algorithm that satisfies a weaker version of the two desirable properties and shows strong empirical performance.\n\nWEAKNESSES:\n- Generality of results is questionable. The proposed properties of discrimination and monotonicity are very much tied to two assumed tasks: link prediction and node recommendation. If one is only interested in one of these tasks (or perhaps a different task altogether), then the trade off between these two properties may not apply.\n- Discussion of limitations should be moved to the main paper.\n\n\nTypos and minor issues:\n- Figure 1 appears very early on in the paper, before the detailed discussion on the first and second category of methods on page 6. I did not understand it until reading page 6. I suggest the authors place a shorter summary of the first and second categories in the figure caption or earlier in the paper around the first time Figure 1 is used.\n\nQUESTIONS:\n1. The analysis in Section 2 is quite general with respect to the similarity function $s(u,v)$. Sigmoid and cosine functions are mentioned as examples. What requirements must the similarity function satisfy in order for your analysis to be valid? Would, for example, a Euclidean distance-based similarity such as $\\sigma(||\\mathbf{F}(u,:) - \\mathbf{F}(v,:)||_2)$ meet the requirements?\n\n2. It seems that the two properties of discrimination and monotonicity are very much tied to the two tasks that you assume are important: link prediction and node recommendation. However, if one is only interested in link prediction, is there a reason they should care about monotonicity rather than just discrimination?\n\nLIMITATIONS:\nLimitations are discussed, but not in much detail and only in the supplement. Such a discussion should be moved to the main paper, with a longer discussion in the supplement if necessary.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}