SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuchen Yan, Baoyu Jing, Lihui Liu, Ruijie Wang, Jinning Li, Tarek Abdelzaher, Hanghang Tong
University of Illinois at Urbana Champaign, IL, USA {yucheny5, baoyuj2, lihui12, ruijiew2, jinning4, zaher, htong}@illinois.edu
Network embedding plays a significant role in a variety of applications. To capture the topology of the network, most of the existing network embedding algorithms follow a sampling training procedure, which maximizes the similarity (e.g., embedding vectors' dot product) between positively sampled node pairs and minimizes the similarity between negatively sampled node pairs in the embedding space. Typically, close node pairs function as positive samples while distant node pairs are usually considered as negative samples. However, under different or even competing sampling strategies, some methods champion sampling distant node pairs as positive samples to encapsulate longer distance information in link prediction, whereas others advocate adding close nodes into the negative sample set to boost the performance of node recommendation. In this paper, we seek to understand the intrinsic relationships between these competing strategies. To this end, we identify two properties (discrimination and monotonicity) that given any node pair proximity distribution, node embeddings should embrace. Moreover, we quantify the empirical error of the trained similarity score w.r.t. the sampling strategy, which leads to an important finding that the discrimination property and the monotonicity property for all node pairs can not be satisfied simultaneously in real-world applications. Guided by such analysis, a simple yet novel model (SENSEI) is proposed, which seamlessly fulfills the discrimination property and the partial monotonicity within the top- $K$  ranking list. Extensive experiments show that SENSEI outperforms the state-of-the-arts in plain network embedding.
In the era of big data, network embedding [36, 14, 21, 7, 42] maps nodes in the network to low-dimensional vectors in the embedding space, which plays an important role in many tasks such as node recommendation [54, 61, 38, 8, 20, 19, 1, 2], networked time series imputation [44, 18, 48, 9], knowledge graph completion [49, 46, 47, 45, 28], and network alignment [52, 53, 60, 62, 16, 27, 59]. To distill the topology information of the network, most existing network embedding methods follow a sampling training procedure. Given any central node to be considered, existing network embedding methods build a positively sampled node pair set and a negatively sampled node pair set. Then, they optimize the embeddings of nodes by maximizing/minimizing the similarity between the positively/negatively sampled node pairs in the embedding space. Explicitly or implicitly, these methods are based on an assumption that nodes close to the central node should be included in the positively sampled node pair set, whereas distant nodes are likely to be considered as negative samples. For example, DeepWalk [36] and metapath2vec [7] explicitly construct random walk containing positively sampled nodes and select negative nodes according to the degree distribution of the network. The message-passing mechanism in GraphSAGE [15] and graph auto-encoder (GAE) [22] is built on graph Laplacian regularization [21, 57], which makes connected node pairs to be
similar. In this case, the close/distant nodes become implicit positive/negative samples for the central node, which can be regarded as a general form of such sampling training procedure.
Numerous network embedding algorithms have been proposed, which focus on improving either the positive sampling strategy or the negative sampling strategy based on various intuitions. Consequently, disparate or even competing sampling strategies emerge in various methods. To name a few, rather than solely favor nodes within two hops as positive samples in LINE [41], node2vec [14] allows a longer random walk length. For graph convolution network (GCN) [21], APPNP [23] utilizes personalized pagerank [33] to sample more distant nodes w.r.t the central node for feature aggregation. The aforementioned two algorithms accommodate farther nodes in the positively sampled node pair set. Meanwhile, SPNE [12], KBGAN [4] and RecNS [56] encourage choosing closer nodes as difficult negative samples to promote the performance of node recommendation.
Thus, some fundamental questions arise: what is the theoretic root cause behind such antithetical sampling strategies? what are possible and what are impossible for sampling of network embedding? how can we develop a practical embedding algorithm that simultaneously embraces these competing strategies?
In this paper, we hammer at bringing the intrinsic relationships behind these competing sampling strategies in light. Concretely, we start from two fundamental tasks of graph learning: the link prediction task and the node recommendation task. To tackle the above two tasks synchronously, we identify two desirable properties, including the discrimination property and the monotonicity property, that network embedding vectors should satisfy given a node pair proximity distribution. The discrimination property means that the node pair with high proximity should be distinguished from the node pair owning low proximity in the embedding space (link prediction). For the monotonicity property, the ranking list of nodes recommended to the query node needs to be consistent with the proximity list in the descending order. Theoretically, we analyze the general form of network embedding algorithms' loss functions. We show that the negative sampling distribution should be negatively correlated with the node pair proximity distribution. Furthermore, we show that in the ideal case where the algorithm can sample an infinite number<sup>1</sup> of positive nodes for the central node, both the discrimination property and the monotonicity property can be fulfilled (i.e., possibility result). However, due to the limited sample size of the real-world data, there exists an inevitable error between the ideal optimal similarity scores and the empirical optimal similarity scores in the embedding space. Regardless of the specific sampling strategy, the discrimination property and the monotonicity property for all node pairs in the network can not be fulfilled at the same time (i.e., impossibility result).
Fortunately, in real-world applications, ranking all nodes in the network for the query node is often unnecessary. In many cases, only the top- $K$  recommendation list and its internal order matters, which suggests that we can first achieve the discrimination property to detect candidate nodes to be recommended. After that, we can attain the monotonicity property within the top- $K$  ranking list (partial monotonicity). Guided by this intuition and the theoretical results, we propose a simple yet novel model (SENSEI). In detail, SENSEI adopts a commonly used proximity measurement (personalized pagerank [33]) and decomposes the embedding process into two steps. The first step is to satisfy the discrimination property. In addition to sampling nodes with large proximity as positive samples, SENSEI also includes nodes with intermediate proximity in the positively sampled node pair set, which reduces the empirical error of the similarity scores for these nodes. Then, in the second step, SENSEI pays attention to the monotonicity property within the positively sampled node pairs from the first step. Nodes are ranked w.r.t. their proximities and importantly, some positively sampled nodes are turned into negative samples, which resembles the strategy of selecting difficult negative samples in some existing methods [4, 55]. In this way, SENSEI creatively integrates these two competing sampling strategies in one integral framework. The experiments demonstrate that SENSEI greatly outperforms various baselines.
To summarize, our contributions are three folds:
- Theoretical Analysis. We reveal the underlying relationships between the competing sampling strategies of existing network embedding methods. Specially, we prove that any
Instead of having an infinite number of nodes in the network, we could sample positive pairs infinite times to approximate the true distributions.
sampling strategy bears an inevitable error gap between the empirical and ideal optimal embedding similarity scores.
- Simple yet Novel Model. Based on the theoretical results, we propose a two-step model SENSEI which creatively integrates competing sampling strategies in one network embedding framework. It satisfies the discrimination property in the first step and obtains the partial monotonicity property within the positively sampled node pair set in the second step.
- Experimental Results. Extensive experiments show that SENSEI outperforms the state-of-the-arts in plain network embedding.
Problem setting. We primarily focus on plain network embedding, in which given the adjacency matrix of a plain network  $\mathbf{A}$ , we aim to output the embedding matrix  $\mathbf{F}$ .
In this section, we uncover the intrinsic relationships of existing competing sampling strategies. First, we describe the sampling-based network embedding process mathematically. Then, we propose two desired properties for the learned embeddings: the discrimination property and the monotonicity property, which correspond to two common learning tasks: link prediction and node recommendation. Theoretically, we analyze the general form of network embedding algorithms' loss functions. We show that the negative sampling distribution should be negatively correlated with the given node pair proximity distribution. Guided by this critical insight, we give both possibility and impossibility results for network embedding. First (possibility result), we show that, in the ideal case where an infinite number of positive samples can be obtained, both the discrimination property and the monotonicity property can be satisfied with the optimal solution of the loss function. Second (impossibility result), for the empirical loss function with a limited number of positive samples, we prove that there always exists an error gap between the empirical optimal solution and the ideal optimal solution. Regardless of the specific sampling strategy, the discrimination property and the monotonicity property can not be simultaneously satisfied for all node pairs in the network in the empirical situation.
Current sampling-based network embedding algorithms can be divided into two phases. The first phase is to construct a node pair proximity distribution  $p(u|v)$ , which serves as the positive sampling distribution in the training process. For example, personalized pagerank [33] in APPNP [23] is an explicit node pair proximity distribution, while random walk in DeepWalk [36] and node2vec [14] implicitly constructs such node pair proximity distribution. The second phase is to design a negative sampling distribution  $p_n(u|v)$  and to utilize the network embedding loss function to train the embedding model. Similar to previous works [55, 56], we start from analyzing the second phase of the sampling-based network embedding process as follows:
Definition 1. Sampling-based Network Embedding Process in the Second Phase. Given a node proximity distribution  $p$ , where  $p(u|v)$  refers to the proximity of node  $u$  w.r.t. the central node  $v$  and  $\sum_{u} p(u|v) = 1$ , the algorithm designs a negative sampling distribution  $p_n$  and adopts a loss function  $J$  to obtain similarity scores  $s(u,v) \in [0,1]$  of all node pairs  $(u,v)$ , which are calculated by the node embedding matrix  $\mathbf{F}$ .
The similarity score  $s(u, v)$  defined here is general, which can be implemented with various similarity measurements (e.g., the sigmoid function  $\sigma(\mathbf{F}(u, :)^{\top} \mathbf{F}(v, :))$  or the cosine function  $\cos(\mathbf{F}(u, :), \mathbf{F}(v, :))$ ) in various algorithms.
Then, to solve both link prediction and node recommendation tasks, we propose that the general similarity score  $s(u,v)$  calculated by node embeddings should possess two properties: the discrimination property and the monotonicity property, which are defined as the following:
Definition 2. Discrimination and Monotonicity.
Discrimination Property: Given the central node  $v$ , node pair  $(u, v)$  with large  $p(u|v)$  should be clearly distinguished with node pair  $(w, v)$  with small  $p(w|v)$  in the embedding space, i.e.,  $\lim_{s \to 0} \frac{s(u, v)}{s(w, v)} = +\infty$  when  $\frac{p(w|v)}{p(u|v)} \to 0$ .
Monotonicity Property: Given the central node  $v$  and two arbitrary nodes  $u, w$  in the network, if  $p(u|v) > p(w|v)$ , then  $s(u,v) > s(w,v)$ .
Figure 1: Discrimination and monotonicity properties. We use star nodes to denote nodes with large  $p(u|v)$ , round nodes to denote nodes with intermediate  $p(u|v)$  and rectangle nodes to denote nodes with small  $p(u|v)$ . We assume that  $p(u_1|v) > p(u_2|v) > \dots > p(u_{10}|v)$ .
The discrimination property bears subtle difference from the monotonicity property. Let us look at an illustrative example in the first line in Figure 1. For the central node  $v$ , we assume that  $p(u_1|v) = 0.45$ ,  $p(u_2|v) = 0.4$  and  $p(w|v) = 0.001$ . Case 1: If  $s(u_1, v) = 0.7$ ,  $s(u_2, v) = 0.8$  and  $s(w, v) = 0.01$ , the final node embeddings satisfy the discrimination property  $\left( \frac{s(w, v)}{s(u_1, v)} = 0.0143 \right.$  and  $\frac{s(w, v)}{s(u_2, v)} = 0.0125$ , both approaching zero) but do not satisfy the monotonicity property  $(p(u_1|v) > p(u_2|v)$  but  $s(u_1, v) < s(u_2, v))$ . Case 2: If  $s(u_1, v) = 0.51$ ,  $s(u_2, v) = 0.50$  and  $s(w, v) = 0.49$ , the final node embeddings fulfill the monotonicity property  $(s(u_1, v) > s(u_2, v) > s(w, v))$  but do not satisfy the discrimination property  $\left( \frac{s(u_1, v)}{s(w, v)} = 1.04 \right.$  and  $\frac{s(u_2, v)}{s(w, v)} = 1.02$ . From the above example, we can see that the discrimination property focuses on making node pairs with large proximity closer and pushing away node pairs with small proximity in the embedding space, which is desirable for tasks like binary classification for link prediction (i.e., predicting the existence of a link between two nodes). On the other hand, the monotonicity property pays more attention to the rank of the similarity scores of node pairs, which is critical for tasks like ranking for node recommendation.
2.1 Possibility Results in the Ideal Case
We conduct theoretic analysis about the loss function of network embedding. Previous works [24, 37, 55] have demonstrated that most existing network embedding algorithms can be regarded as implicit matrix factorization and there exists a general form of loss function for the given central node  $v$ . If the sigmoid function is adopted as the similarity function. The loss function is:
$$
J = - \mathbb {E} _ {u \sim p (u | v)} \log \sigma (\mathbf {F} (u,:) ^ {\top} \mathbf {F} (v,:)) - k \mathbb {E} _ {w \sim p _ {n} (w | v)} \log (1 - \sigma (\mathbf {F} (w,:) ^ {\top} \mathbf {F} (v,:))) \tag {1}
$$
where  $\sigma (\cdot)$  is the sigmoid function,  $\mathbf{F}(u,:),\mathbf{F}(v,:)$  and  $\mathbf{F}(w,:)$  are the embeddings of  $u,v$  and  $w$  respectively, and  $k$  is the number of negative samples for each positive sample. If the sigmoid function is replaced by other similarity measurements, we can obtain a more general form of this loss function:
$$
J = - \mathbb {E} _ {u \sim p (u | v)} \log (s (u, v)) - k \mathbb {E} _ {w \sim p _ {n} (w | v)} \log (1 - s (w, v)) \tag {2}
$$
where  $s(\cdot, \cdot)$  is a general form of similarity measurement.
In the ideal case where an infinite number of positive samples from  $p(u|v)$  are available, we have the following theorem about the optimal solution of  $s$ :
Theorem 1. Optimal Solution of  $s$ . The optimal solution of the similarity function  $s$  satisfies that for each node pair  $(u, v)$ ,
$$
s (u, v) = \frac {p (u | v)}{p (u | v) + k p _ {n} (u | v)} \tag {3}
$$
$$
J = - \mathbb {E} _ {u \sim p (u | v)} \log s (u, v) - k \mathbb {E} _ {w \sim p _ {n} (w | v)} \log (1 - s (w, v)) \tag {4}
$$
Since  $s(u,v)$  is the parameter to be optimized, we can calculate  $\nabla_{s(u,v)}J$  as following:
$$
\nabla_ {s (u, v)} J = - p (u | v) \frac {1}{s (u , v)} - k p _ {n} (u | v) \frac {1}{s (u , v) - 1} \tag {5}
$$
Let the derivative to be 0, we can obtain  $s(u,v) = \frac{p(u|v)}{p(u|v) + kp_n(u|v)}$ , which is the optimal solution for  $J$ .
With the optimal solution of  $s(u, v)$ , we give the following proposition about the design of  $p_n$ :
Proposition 1. The Design of Negative Sampling. In the sampling-based network embedding process, the negative sampling distribution  $p_n$  should be negatively correlated with the given node pair proximity distribution  $p$  to fulfill the monotonicity property.
Proof. Here we prove the above proposition in the ideal case by contradiction: In the ideal case, assume that the best embeddings and corresponding similarity scores to satisfy the discrimination property and the monotonicity property are obtained with  $p_n$ , and  $p_n$  is not negatively correlated with  $p$ . It is equivalent to the situation that for one central node  $v$ , there must exist two nodes  $u, w$  with  $p(u|v) > p(w|v)$  and  $p_n(u|v) > p_n(w|v)$ . From Theorem 1 we know that  $s(u,v) = \frac{p(u|v)}{p(u|v) + kp_n(u|v)}$  and  $s(w,v) = \frac{p(w|v)}{p(w|v) + kp_n(w|v)}$ . Now, we interchange  $p_n(u|v)$  with  $p_n(w|v)$  and we can get new similarity scores  $s'(u,v)$  and  $s'(w,v)$  for  $(u,v)$  and  $(w,v)$ .  $s'(u,v) = \frac{p(u|v)}{p(u|v) + kp_n(w|v)} > \frac{p(u|v)}{p(u|v) + kp_n(u|v)} = s(u,v)$ . In addition,  $s'(w,v) = \frac{p(w|v)}{p(w|v) + kp_n(u|v)} < \frac{p(w|v)}{p(w|v) + kp_n(w|v)} = s(w,v)$ . So, we get two new similarity scores that can satisfy the monotonicity property better, which is contradictory with the assumption that we have already achieved the best embeddings and corresponding similarity scores.
Intuitively, this proposition makes sense because if one node is more likely to be sampled as a positive sample for the central node, it is less likely that this node also acts as a negative sample.
From Theorem 1, we can see that if  $p_n$  is negatively correlated with  $p$ , when  $p(u|v)$  is large and  $p_n(u|v)$  is small,  $s(u,v) \to 1$ ; and when  $p(u|v)$  is small and  $p_n(u|v)$  is large,  $s(u,v) \to 0$ , which satisfies the discrimination property. For all nodes in the network with  $p(u_1|v) > p(u_2|v) > \dots > p(u_n|v)$ , if we set  $p_n(u_1|v) < p_n(u_2|v) < \dots < p_n(u_n|v)$ , the monotonicity property can be fulfilled with  $s(u_1,v) > s(u_2,v) > \dots > s(u_n,v)$ .
2.2 Impossibility Results in the Empirical Case
However, in real-world applications, we can not sample an infinite number of nodes from  $p(u|v)$ . Empirically, existing algorithms set a fixed number  $T$  of positive samples for the central node and the general loss function in Eq. (2) turns into:
$$
J _ {e} = - \frac {1}{T} \sum_ {i = 1} ^ {T} \log (s (u _ {i}, v)) - \frac {1}{T} \sum_ {i = 1} ^ {k T} \log (1 - s (w _ {i}, v)) \tag {6}
$$
where  $u_{i}$  is a positive sample from  $p(u|v)$  and  $w_{i}$  is a negative sample from  $p_{n}(w|v)$ . We use  $S = [s(u_{1}, v), s(u_{2}, v), \ldots, s(u_{n}, v)]$  to denote the final similarity scores to be optimized w.r.t. the central node  $v$ . For the general loss function in the ideal case, the optimal solution of  $S$  is  $S^{*}$ , where  $S^{*} = \left[\frac{p(u_{1}|v)}{p(u_{1}|v) + kp_{n}(u_{1}|v)}, \ldots, \frac{p(u_{n}|v)}{p(u_{n}|v) + kp_{n}(u_{n}|v)}\right]$ . For the loss function  $J_{e}$  in the empirical case with  $T$  positive samples from  $p$ , we represent the optimal solution as  $S_{e}$ . For the mean squared error between  $S^{*}$  and  $S_{e}$ , we have the following theorem:
Theorem 2. Mean Squared Error. For the mean squared error between the empirical optimal solution  $S_{e}$  and the ideal solution  $S^{*}$ , we have
$$
\mathbb {E} [ \| (S _ {e} - S ^ {*}) _ {u} \| ^ {2} ] = \frac {1}{T (2 + \frac {k p _ {n} (u | v)}{p (u | v)} + \frac {p (u | v)}{k p _ {n} (u | v)}) ^ {2}} (\frac {1}{p (u | v)} + \frac {1}{k p _ {n} (u | v)} - 1 - \frac {1}{k}), \tag {7}
$$
where  $\mathbb{E}$  is the expectation and  $(S_e - S^*)_u = s_e(u,v) - s^*(u,v)$  is the error between the optimal similarity scores obtained in the empirical and ideal cases respectively.
The proof for Theorem 2 is attached in Appendix A.1. If the sigmoid function is adopted as the similarity function, the expectation of the similarity error in Theorem 2 degenerates to its special case:
$$
\mathbb {E} [ | | \mathbf {F} _ {e} (u,:) ^ {\top} \mathbf {F} _ {e} (v,:) - \mathbf {F} ^ {*} (u,:) ^ {\top} \mathbf {F} ^ {*} (v,:) | | ^ {2} ] = \frac {1}{T} \left(\frac {1}{p (u | v)} + \frac {1}{k p _ {n} (u | v)} - 1 - \frac {1}{k}\right), \tag {8}
$$
where  $\mathbf{F}_e(u,:)$  and  $\mathbf{F}^* (u,:)$  are the optimal embedding vectors of node  $u$  in  $J_{e}$  and  $J$ . Eq. (8) was first discovered in MCNS [55]. Based on Eq. (8), MCNS advocates that, in order to bound the dot product error for nodes with large  $p(u|v)$ ,  $p_n$  should be positively correlated with  $p$ . Unfortunately, this is not always the best choice due to the following reason. In MCNS, the authors set  $p_n(u|v)\propto p(u|v)^a$ , where  $0 < a < 1$ . With this negative sampling distribution, the right part of Eq. (8) changes into  $\frac{1}{T}\left(\frac{1}{p(u|v)}\left(1 + \frac{p(u|v)^{1 - a}}{c}\right) - 1 - \frac{1}{k}\right)$ , where  $c$  is a constant. If  $p(u|v)$  is large, this term can indeed be bounded. However, for distant nodes with small  $p(u|v)$  (i.e.,  $p(u|v)\to 0$ ), its positively correlated negative sampling probability  $p_n(u|v)$  is also very small and  $p_n(u|v)\to 0$ , which leads to an extremely large error considering that the term  $\frac{1}{p(u|v)} +\frac{1}{kp_n(u|v)}\to \infty$ , which may fail to fulfill the discrimination property.
Discussion. Let us analyze the implications of Theorem 2. If  $p_n(u|v)$  is negatively correlated with  $p(u|v)$ , it means that  $\frac{kp_n(u|v)}{p(u|v)} \to 0$  for nodes with the largest  $p(u|v)$ s and  $\frac{kp_n(u|v)}{p(u|v)} \to \infty$  for nodes with the smallest  $p(u|v)$ s. Let us consider the case that  $p(u|v)$  is large and  $\frac{kp_n(u|v)}{p(u|v)} \to 0$ .  $\mathbb{E}[||(S_e - S^*)_u||^2] < \frac{1}{T(\frac{p(u|v)}{kp_n(u|v)})^2} (\frac{1}{p(u|v)} + \frac{1}{kp_n(u|v)} - 1 - \frac{1}{k}) < \frac{1}{T} (\frac{kp_n(u|v)}{p(u|v)})^2 (\frac{1}{p(u|v)} + \frac{1}{kp_n(u|v)})$ .