SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \lim _ {T \to + \infty} - \nabla_ {S} ^ {2} J _ {e} (S ^ {*}) = - \sum_ {u _ {i}} p (u _ {i} | v) \frac {(p (u _ {i} | v) + k p _ {n} (u _ {i} | v)) ^ {2}}{p (u _ {i} | v) ^ {2}} \mathbf {e} _ {(u _ {i})} \mathbf {e} _ {(u _ {i})} ^ {\top} \\ - \sum_ {w _ {i}} k p _ {n} \left(w _ {i} \mid v\right) \frac {\left(p \left(w _ {i} \mid v\right) + k p _ {n} \left(w _ {i} \mid v\right)\right) ^ {2}}{\left(k p _ {n} \left(w _ {i} \mid v\right)\right) ^ {2}} \mathbf {e} _ {\left(w _ {i}\right)} \mathbf {e} _ {\left(w _ {i}\right)} ^ {\top} \tag {18} \\ = - \sum_ {u _ {i}} \frac {\left(p (u _ {i} | v) + k p _ {n} (u _ {i} | v)\right) ^ {3}}{k p _ {n} (u _ {i} | v) p (u _ {i} | v)} \mathbf {e} _ {(u _ {i})} \mathbf {e} _ {(u _ {i})} ^ {\top} \\ \end{array}
$$
where the last line is because that we combine same terms when  $u_{i} = w_{i}$ .
Then, we consider the second term  $\nabla_{S}J_{e}(S^{*})$ . We analyze the expectation and variance of  $\nabla_{S}J_{e}(S^{*})$  to get the final  $\mathbb{E}[||(S_e - S^*)_u||^2 ]$ .
For the expectation, as  $\nabla_{S}J_{e}(S)$  is as follows:
$$
\nabla_ {S} J _ {e} (S) = - \frac {1}{T} \sum_ {i = 1} ^ {T} \frac {1}{s \left(u _ {i} , v\right)} \mathbf {e} _ {\left(u _ {i}\right)} - \frac {1}{T} \sum_ {i = 1} ^ {k T} \frac {1}{s \left(w _ {i} , v\right) - 1} \mathbf {e} _ {\left(w _ {i}\right)} \tag {19}
$$
the expectation is:
$$
\begin{array}{l} \mathbb {E} \left[ \nabla_ {S} J _ {e} \left(S ^ {*}\right) \right] = - \sum_ {u _ {i}} p \left(u _ {i} \mid v\right) \frac {p \left(u _ {i} \mid v\right) + k p _ {n} \left(u _ {i} \mid v\right)}{p \left(u _ {i} \mid v\right)} \mathbf {e} _ {\left(u _ {i}\right)} \\ + \sum_ {w _ {i}} k p _ {n} \left(w _ {i} \mid v\right) \frac {p \left(w _ {i} \mid v\right) + k p _ {n} \left(w _ {i} \mid v\right)}{k p _ {n} \left(w _ {i} \mid v\right)} \mathbf {e} _ {\left(w _ {i}\right)} \tag {20} \\ = 0 \\ \end{array}
$$
For the variance of  $\nabla_S J_e(S^*)$ , since  $\mathbb{E}[\nabla_S J_e(S^*)] = 0$
$$
\begin{array}{l} \operatorname {C o v} \left(\nabla_ {S} J _ {e} \left(S ^ {*}\right)\right) = \mathbb {E} \left[ \nabla_ {S} J _ {e} \left(S ^ {*}\right) \nabla_ {S} J _ {e} \left(S ^ {*}\right) ^ {\top} \right] \\ = \frac {T}{T ^ {2}} \mathbb {E} _ {u \sim p (u | v)} \left(\frac {1}{s ^ {*} (u , v) ^ {2}}\right) \mathbf {e} _ {(u)} \mathbf {e} _ {(u)} ^ {\top} \\ + \frac {k T}{T ^ {2}} \mathbb {E} _ {w \sim p _ {n} (w | v)} \left(\frac {1}{(1 - s ^ {*} (w , v)) ^ {2}}\right) \mathbf {e} _ {(w)} \mathbf {e} _ {(w)} ^ {\top} \\ + \sum_ {u _ {i}, u _ {j}} \frac {T (T - 1)}{T ^ {2}} p \left(u _ {i} \mid v\right) p \left(u _ {j} \mid v\right) \frac {1}{s ^ {*} \left(u _ {i} , v\right)} \frac {1}{s ^ {*} \left(u _ {j} , v\right)} \mathbf {e} _ {\left(u _ {i}\right)} \mathbf {e} _ {\left(u _ {j}\right)} ^ {\top} \tag {21} \\ - \sum_ {u _ {i}, w _ {j}} \frac {k T ^ {2}}{T ^ {2}} p (u _ {i} | v) p _ {n} (w _ {j} | v) \frac {1}{s ^ {*} (u _ {i} , v)} \frac {1}{1 - s ^ {*} (w _ {j} , v)} \mathbf {e} _ {(u _ {i})} \mathbf {e} _ {(w _ {j})} ^ {\top} \\ - \sum_ {u _ {j}, w _ {i}} \frac {k T ^ {2}}{T ^ {2}} p (u _ {j} | v) p _ {n} (w _ {i} | v) \frac {1}{s ^ {*} (u _ {j} , v)} \frac {1}{1 - s ^ {*} (w _ {i} , v)} \mathbf {e} _ {(w _ {i})} \mathbf {e} _ {(u _ {j})} ^ {\top} \\ + \sum_ {w _ {i}, w _ {j}} \frac {k T (k T - 1)}{T ^ {2}} p _ {n} (w _ {i} | v) p _ {n} (w _ {j} | v) \frac {1}{1 - s ^ {*} (w _ {i} , v)} \frac {1}{1 - s ^ {*} (w _ {j} , v)} \mathbf {e} _ {(w _ {i})} \mathbf {e} _ {(w _ {j})} ^ {\top} \\ \end{array}
$$
Because  $s^*(u_i, v) = \frac{p(u_i | v)}{p(u_i | v) + kp_n(u_i | v)}$ ,  $s^*(u_j, v) = \frac{p(u_j | v)}{p(u_j | v) + kp_n(u_j | v)}$ ,  $s^*(w_i, v) = \frac{p(w_i | v)}{p(w_i | v) + kp_n(w_i | v)}$  and  $s^*(w_j, v) = \frac{p(w_j | v)}{p(w_j | v) + kp_n(w_j | v)}$ , Eq. (21) becomes:
$$
\begin{array}{l} \operatorname {C o v} \left(\nabla_ {S} J _ {e} \left(S ^ {*}\right)\right) = \frac {1}{T} \sum_ {u} \frac {\left(p (u | v) + k p _ {n} (u | v)\right) ^ {3}}{k p _ {n} (u | v) p (u | v)} \mathbf {e} _ {(u)} \mathbf {e} _ {(u)} ^ {\top} \\ + \sum_ {u _ {i}, u _ {j}} (1 - \frac {1}{T}) (p (u _ {i} | v) + k p _ {n} (u _ {i} | v)) (p (u _ {j} | v) + k p _ {n} (u _ {j} | v)) \mathbf {e} _ {(u _ {i})} \mathbf {e} _ {(u _ {j})} ^ {\top} \\ - \sum_ {u _ {i}, w _ {j}} (p (u _ {i} | v) + k p _ {n} (u _ {i} | v)) (p (w _ {j} | v) + k p _ {n} (w _ {j} | v)) \mathbf {e} _ {(u _ {i})} \mathbf {e} _ {(w _ {j})} ^ {\top} \\ - \sum_ {u _ {j}, w _ {i}} \left(p \left(u _ {j} \mid v\right) + k p _ {n} \left(u _ {j} \mid v\right)\right) \left(p \left(w _ {i} \mid v\right) + k p _ {n} \left(w _ {i} \mid v\right)\right) \mathbf {e} _ {\left(w _ {i}\right)} \mathbf {e} _ {\left(u _ {j}\right)} ^ {\top} \tag {22} \\ + \sum_ {w _ {i}, w _ {j}} (1 - \frac {1}{k T}) (p (w _ {i} | v) + k p _ {n} (w _ {i} | v)) (p (w _ {j} | v) + k p _ {n} (w _ {j} | v)) \mathbf {e} _ {(w _ {i})} \mathbf {e} _ {(w _ {j})} ^ {\top} \\ = \frac {1}{T} \sum_ {u} \frac {\left(p (u | v) + k p _ {n} (u | v)\right) ^ {3}}{k p _ {n} (u | v) p (u | v)} \mathbf {e} _ {(u)} \mathbf {e} _ {(u)} ^ {\top} \\ - (1 + \frac {1}{k}) \frac {1}{T} \sum_ {u _ {i}, u _ {j}} (p (u _ {i} | v) + k p _ {n} (u _ {i} | v)) (p (u _ {j} | v) + k p _ {n} (u _ {j} | v)) \mathbf {e} _ {(u _ {i})} \mathbf {e} _ {(u _ {j})} ^ {\top} \\ \end{array}
$$
Based on Eq. (15), Eq. (18), Eq. (22) and  $\mathbb{E}[||(S_e - S^*)_u||^2] = \mathrm{Cov}(S_e - S^*)(u,u)$ , we have
$$
\begin{array}{l} \mathbb {E} [ | | \left(S _ {e} - S ^ {*}\right) _ {u} | | ^ {2} ] = \\ \frac {1}{T} \frac {k p _ {n} (u | v) p (u | v)}{\left(p (u | v) + k p _ {n} (u | v)\right) ^ {3}} \left(\frac {\left(p (u | v) + k p _ {n} (u | v)\right) ^ {3}}{k p _ {n} (u | v) p (u | v)} \right. \tag {23} \\ \left. - \left(1 + \frac {1}{k}\right) \left(p (u | v) + k p _ {n} (u | v)\right) ^ {2}\right) \frac {k p _ {n} (u | v) p (u | v)}{\left(p (u | v) + k p _ {n} (u | v)\right) ^ {3}} \\ \end{array}
$$
which can be simplified as:
$$
\begin{array}{l} \mathbb {E} \left[ | | \left(S _ {e} - S ^ {*}\right) _ {u} | | ^ {2} \right] \\ = \frac {1}{T} \frac {\left(k p _ {n} (u | v) p (u | v)\right) ^ {2}}{\left(p (u | v) + k p _ {n} (u | v)\right) ^ {4}} \left(\frac {k p _ {n} (u | v) + p (u | v)}{k p _ {n} (u | v) p (u | v)} - 1 - \frac {1}{k}\right) \tag {24} \\ = \frac {1}{T (2 + \frac {k p _ {n} (u | v)}{p (u | v)} + \frac {p (u | v)}{k p _ {n} (u | v)}) ^ {2}} (\frac {1}{p (u | v)} + \frac {1}{k p _ {n} (u | v)} - 1 - \frac {1}{k}) \\ \end{array}
$$
A.2 The Algorithm of SENSEI
In this section, we give the detailed algorithm of SENSEI in Algorithm 1.
A.3 Dataset Statistics and Implementation Details.
In this section, we introduce the statistics of datasets and the implementation details of SENSEI. The statistics of datasets are present in Table 3.
Table 3: Dataset Statistics.
Dataset	Layers	Nodes	Edges
C.ele	1	297	2,148
Cora	1	2,708	5,429
CiteSeer	1	3,327	4,732
NS	1	1,589	2,742
Implementation Details. For all the methods, we run 5 times to calculate the standard deviation and for all baselines, we use their default parameters. For SENSEI on 4 datasets: {C.ele, Cora, CiteSeer,
Algorithm 1 The algorithm of SENSEI
Input: (1) the input plain network A; (2) training epochs in Step 1  $(E_1)$  and in Step 2  $(E_2)$ , hyper-parameters  $\tau$ ,  $k$  and  $\gamma$ .  
Output: Node embeddings F. Initialize F; for each  $v$  in A do Run PPR to construct  $\mathcal{P}(v)$  and  $\mathcal{N}(v)$  with the threshold  $\tau$ ; end for # Step 1: Fulfill Discrimination Property  $e = 0$ ; while  $e < E_1$  do for each  $v$  in A do Randomly sample  $k$  negative nodes from  $\mathcal{N}(v)$ ; Compute  $J(v)$  using Eq. (9); end for Add all  $J(v)$ s to obtain  $J_P$  in Eq. (10); Back-propagate the loss and update F;  $e = e + 1$ ; end while # Step 2: Fulfill Partial Monotonicity Property  $e = 0$ ; while  $e < E_2$  do for each  $v$  in A do Compute  $L(v)$  using Eq. (11); end for Add all  $L(v)$ s to obtain  $L_P$  in Eq. (12); Back-propagate the loss and update F;  $e = e + 1$ ; end while return F.
NS}, we set the threshold  $\tau$  as  $\{0.008, 0.01, 0.005, 0.05\}$ , the number of epochs in Step 1 as  $\{40, 100, 20, 20\}$ , the number of epochs in Step 2 as  $\{40, 100, 50, 20\}$ , the learning rate in Step 1 as  $\{0.02, 0.1, 0.1, 0.2\}$ , the learning rate in Step 2 as  $\{0.01, 0.01, 0.005, 0.1\}$ , the positive margin  $\gamma$  as  $\{0.05, 0.0001, 0.0001, 0.1\}$  and the negative sample number  $k$  as 40 on all 4 datasets. All experiments are run on a Tesla-V100 GPU. $^3$
A.4 Additional Baselines of Plain Network Embedding
We have added another 5 GNNs as additional baselines: GraphSAGE [15], Cluster-gcn [5], APPNP [13], GPRGNN [6] and H2GCN [63]. the results are demonstrated in Table 4. We can see that for both link prediction and node recommendation, SENSEI generally beats these GNN methods.
Table 4: The AUC-ROC and AUC-PR of link prediction and Hit@10 of node recommendation on Cora and Citeseer(%)
Models	Cora	Citeseer
AUC-ROC	AUC-PR	Hit@10	AUC-ROC	AUC-PR	Hit@10
GraphSAGE [15]	64.13	65.36	3.92	66.10	68.94	4.74
Cluster-gcn [5]	65.93	72.39	4.65	67.33	70.45	5.42
APPNP [13]	68.77	72.39	11.72	66.45	71.57	11.96
GPRGNN [6]	64.43	70.15	13.31	62.93	68.69	10.64
H2GCN [63]	61.90	61.89	1.96	60.95	61.94	3.53
SENSEI	81.27	83.59	20.51	76.28	82.02	19.03
3The simplified code of SENSEI is on: https://github.com/yucheny5/SENSEI.