SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zihan Luo $^{1}$ , Hong Huang $^{1*}$ , Jianxun Lian $^{2}$ , Xiran Song $^{1}$ , Xing Xie $^{2}$ , Hai Jin $^{1}$
$^{1}$  National Engineering Research Center for Big Data Technology and System, Service Computing Technology and Systems Laboratory, Cluster and Grid Computing Lab, School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China  $^{2}$  Microsoft Research Asia, Beijing, China {zihanluo,honghuang,xiransong,hjin}@hust.edu.cn, {jianxun.lian,xingx}@microsoft.com
Recently, the bias-related issues in GNN-based link prediction have raised widely spread concerns. In this paper, we emphasize the bias on links across different node clusters, which we call cross-links, after considering its significance in both easing information cocoons and preserving graph connectivity. Instead of following the objective-oriented mechanism in prior works with compromised utility, we empirically find that existing GNN models face severe data bias between internal-links (links within the same cluster) and cross-links, and this inspires us to rethink the bias issue on cross-links from a data perspective. Specifically, we design a simple yet effective twin-structure framework, which can be easily applied to most GNNs to mitigate the bias as well as boost their utility in an end-to-end manner. The basic idea is to generate debiased node embeddings as demonstrations and fuse them into the embeddings of original GNNs. In particular, we learn debiased node embeddings with the help of augmented supervision signals, and a novel dynamic training strategy is designed to effectively fuse debiased node embeddings with the original node embeddings. Experiments on three datasets with six common GNNs show that our framework can not only alleviate the bias between internal-links and cross-links but also boost the overall accuracy. Comparisons with other state-of-the-art methods also verify the superiority of our method.
Recently, due to the strong capability in learning the latent representation of graph structure data, Graph Neural Networks (GNNs)-based link prediction methods have received increasing research interests and show excellent performance in recommendation systems [33, 34], bioinformatics [7, 16], and knowledge graph [24, 40]. However, these link prediction methods often prioritize performance without considering potential bias on the sensitive attributes of nodes, such as genders or regions, thus leading to social risks or the creation of informa
Figure 1: A toy graph for illustrating the concept of topological communities. The proportions of two kinds of links also are provided on the right.
*Hong Huang is the corresponding author.
tion cocoons. For instance, in social recommendations, models tend to recommend users with friends belonging to the same region, limiting users' opportunities to connect with the outside world [46]. Similarly, in political news recommendations, recommends prefer to deliver content that aligns with users' partisan beliefs, filtering out different perspectives and narrowing users' scopes [23]. To tackle these issues, researchers have put forward several solutions. For example, FLIP [28] employs adversarial learning to encourage the model to predict outcomes that are independent of sensitive attributes. Similarly, CFC [4] aims to debias the node embeddings with compositional adversarial constraints. FairAdj [20] learns edge weights to generate a fair adjacency matrix, which is used for subsequent link prediction tasks. UGE [38] explores unbiased node embeddings from an unobserved graph, and proposes two kinds of methods based on reweighting and regularization constraints.
Although these methods make great progress in mitigating the bias in link prediction, there are two issues remaining unsolved: 1) All these methods generally modify the objective functions, like adding regularization constraints [4, 28] or reweighting [38] for mitigating the bias. However, the mechanism of simply modifying objective function may influence the optimization trajectory of the model and result in convergence towards a sub-optimal status. The experimental trade-off between debias and utility on multiple prior works [4, 20, 28, 38] also supports this claim. 2) Existing methods simply focus on sensitive attributes of nodes while ignoring the bias based on graph topology. In fact, due to GNNs' heavy dependence on the aggregation of neighborhood messages,
for the link prediction tasks, GNNs tend to wire new links inside the local community $^2$  (denoted as internal-links), while ignoring the links connecting different communities (denoted as cross-links), and this kind of bias may leave the graph in danger of being trapped in information cocoons or disconnected [21, 28], as shown in Figure 1. Empirical evidence can also be provided in Figure 2, in which we illustrate the top 10 commonest labels' distributions in different communities. It can be seen that each community contains its own specific information pattern [26], making it challenging for one single community to encompass all diverse information in a network. This observation implies the propensity of graphs to form information cocoons with insufficient cross-links as bridges $^3$ . Motivated by these findings, we aim to design a GNN model that could boost the link prediction performance on cross-links and mitigate the bias between internal-links and cross-links without sacrificing utility.
By rethinking the problem of bias between internal-links and cross-links with a fresh insight from the data perspective, we statistics the proportion of internal-links and cross-links in three real-world datasets in Figure 1, and we can find that the number of internal-links far exceeds that of cross-links. This kind of data bias may be further enlarged and perpetuated by stacking GNN aggregation operations and finally leads to a biased link prediction. In light of this, we propose a simple yet effective twin-structure framework, which consists of two independent GNN models to mitigate the bias between internal-links and cross-links. Specifically, we first divide the whole graph into multiple communities that share similar topological locations and differentiate links into internal-links and cross-links. After that, in order to alleviate the data bias between internal-links and cross-links, supervision augmentation based on multiple rules is proposed to increase the supervised signals for cross-links, which could help the GNNs to better capture the patterns of cross-links and further generate debiased node embeddings. Subsequently, to avoid utility degradation, we design an embedding fusion module to merge original node embeddings and debiased node embeddings with a dynamic training strategy. In this way, the embedding fusion module could effectively preserve the performance of internal-links while alleviating the bias between internal-links and cross-links. Our main contributions can be summarized as follows:
- We reveal a significant link prediction bias based on the graph topology, i.e. the bias between internal-links and cross-links, and design a model-agnostic framework, which can help most of the GNNs address such bias without compromising utility.
${}^{2}$  Generally, it refers to a cluster of nodes that enjoy similar topological location, which can be detected by community detection algorithms. In this paper, we deploy the Louvain algorithm [2] as the default detection method.
Further analysis on elaborating the role of cross-links is provided in Section A of the Appendix.
- We propose three core components in our framework, including supervision augmentation, twins-GNN, and embedding fusion module. The combination of supervision augmentation and twins-GNN can generate debiased embeddings from a data aspect as demonstrations, while the embedding fusion module can effectively filter out the noise and preserve the utility of GNN.
- We evaluate our method on three real-world datasets with six common GNNs as base models and as well as compare them with other state-of-the-art baselines. The experimental results consistently demonstrate that our framework could effectively reduce the bias between internal-links and cross-links with even improved overall performance.
Link Prediction As one of the most important applications in graph learning, link prediction algorithms have received extensive attention in the last two decades [3, 12, 14, 29]. The traditional link prediction methods are mainly based on some heuristic metrics on the graph structure, such as CN [25], AA [1], and Jaccard coefficient. In recent years, shallow embedding-based link prediction methods such as DeepWalk [29], Node2Vec [12], and LINE [36] have also emerged. With the powerful capability in representation learning on graph structure data, GNNs have achieved great progress in link prediction tasks as well [3, 13, 14, 19, 37, 48]. For example, LightGCN discovers the redundancy of non-linear activation and feature transformation and achieves better performance on recommendations [14]. PPRGo [3] uses a Personalized Page Rank matrix to efficiently approximate the multi-hop aggregation, which breaks the classical message-passing paradigm of GNNs. Following LightGCN, UltraGCN [27] approximates the infinite-layer aggregation for better recommendations. Besides, instead of learning from the whole graph, another line of research focuses on learning from relatively small subgraphs, such as SEAL [44] and SUREL+ [42], which also achieve remarkable performance in link prediction.
Bias Related Issues in GNNs Recently, bias-related issues have been widely concerned in graph neural networks, termed as "debias" or "fairness". Concentrating on node classification tasks, there is a branch of works on achieving fair results that are uncorrelated with sensitive attributes, including FairGNN [8], FairVGNN [39], EDITS [9], and EqGNN [32]. As for bias in link prediction, based on node2vec [12], Fairwalk [30] proposes a more fair random walk strategy, but fails to avoid the decrease in models' utility after achieving satisfactory fairness. Both FLIP [28] and CFC [4] try to mitigate the dyadic bias by proposing adversarial constraints in the loss function. Li et al. [20] designed a model named FairAdj, which could generate a fair adjacency matrix with different edge weights to address the dyadic bias with competitive utility. Similar to FairAdj, UGE [38] derives unbiased embeddings from an unobserved graph that involves no sensitive attribute information. However, all these methods modify the original objective functions and face a trade-off between debias and utility.
3 Methodologies
In this work, a graph is denoted by  $\mathcal{G} = (\mathcal{V},\mathcal{E})$ , and it consists of  $N = |\mathcal{V}|$  vertices and  $|\mathcal{E}|$  links. In the real world, nodes on the graph spontaneously form local communities [26], such as social circles in social networks, and various bundles of also-buy items on product graphs. We use  $\mathcal{C}(v)\in \mathbb{N}$  to denote the community membership of a given node  $v$  and define cross-links and internal-links:
Definition 1. Cross-links and Internal-links. Given a link  $\langle u, v \rangle$ , it belongs to cross-links  $\mathcal{E}_{cr}$  if its endpoints' memberships satisfy  $\mathcal{C}(u) \neq \mathcal{C}(v)$ ; otherwise, it belongs to internal-links  $\mathcal{E}_{in}$  as it satisfies  $\mathcal{C}(u) = \mathcal{C}(v)$ . So according to the endpoints' memberships, the links  $\mathcal{E}$  in a graph could be divided into  $\mathcal{E}_{in}$  and  $\mathcal{E}_{cr}$  two sets exclusively, i.e.  $\mathcal{E}_{in} \cap \mathcal{E}_{cr} = \emptyset$  and  $\mathcal{E}_{in} \cup \mathcal{E}_{cr} = \mathcal{E}$ .
Centering on the link prediction based on GNNs, each node will be mapped to an  $D$  dimensional embedding vector  $\mathbf{z} \in \mathbb{R}^{D}$  by a GNN encoder. The dot product score of a pair of nodes' embeddings implies the GNN model's confidence on whether there will be a potential link between two given vertices. Our goal is to mitigate the bias between internal-links and cross-links, i.e.:
Definition 2. Bias between internal-links and cross-links (Bias). Given a trained GNN model, a confidence matrix  $\mathbf{P} \in \mathbb{R}^{N \times N}$  can be inferred by calculating the dot product of endpoints'
embeddings, and a higher  $\mathbf{P}_{uv}$  indicates that model predicts that it very likely exists a link between node  $u$  and node  $v$ . Let the notation  $\mathcal{M}$  represent an evaluation metric in link prediction tasks, and we use the difference between two kinds of links' average performance to define the bias:
$$
\operatorname {B i a s} = \left| \mathbb {E} \left[ \mathcal {M} (\mathbf {P}) \mid \mathcal {E} = \mathcal {E} _ {i n} \right] - \mathbb {E} \left[ \mathcal {M} (\mathbf {P}) \mid \mathcal {E} = \mathcal {E} _ {c r} \right] \right| \tag {1}
$$
Task definition. To address such bias, a GNN recommender should balance the performance on internal-links and cross-links, so that the final model could show equivalent capability on mining true internal-links and cross-links. The goal of this paper is to design a model-agnostic framework, which can enhance the embedding ability of a given GNN model so that the bias metrics are reduced while the accuracy metrics are at least not decreased:
$$
\operatorname {B i a s} \left(\mathcal {M} (\mathbf {P} | \mathbf {Z})\right) <   \operatorname {B i a s} \left(\mathcal {M} \left(\mathbf {P} | \mathbf {Z} ^ {\prime}\right)\right), \quad \mathcal {M} (\mathbf {P} | \mathbf {Z}) \geq \mathcal {M} (\mathbf {P} | \mathbf {Z} ^ {\prime}) \tag {2}
$$
where  $\mathbf{Z}'$  represents the original node embeddings given by a GNN, and  $\mathbf{Z}$  represents the node embeddings enhanced by our framework.
3.2 Framework Overview
We propose a twin-structure GNN-enhancement framework, which can reduce the bias between cross-links and internal-links without hurting the overall performance. An overview of our framework is illustrated in Figure 3. Its key components include supervision augmentation, Twins GNNs, and an embedding fusion module. Details will be provided in the following sections. In traditional graph embedding models, existing links on the graph are used as supervision signals. In our framework, we first generate a certain number of pseudo cross-links and form the augmented supervision signals. Then, we use two GNNs with the same architecture to model the two sets of supervision signals (original supervision and
Figure 3: The overview of our framework
augmented supervision) independently, leading to original node embeddings and debiased node embeddings. Lastly, to avoid the impact of noise during supervision augmentation, an embedding fusion module is proposed, which ensures that the final embeddings will retain the merits of both original node embeddings and debiased node embeddings.
3.3 Supervision Augmentation
As revealed in Figure 1, in most datasets, usually the vast majority of links on the graph are internal-links, and this kind of disparity may cause the learned model to be biased to internal-links while neglecting the performance of the cross-links in order to obtain better overall metrics. Therefore, we design an augmentation step to alleviate the supervision signal sparsity issue.
Jaccard based augmentation. Inspired from the concept of edge strength [11], a set of  $K$  pairs of nodes  $\{\langle \hat{u}_1,\hat{v}_1\rangle ,\dots,\langle \hat{u}_K,\hat{v}_K\rangle \}$ , which satisfy  $\mathcal{C}(\hat{u}_i)\neq \mathcal{C}(\hat{v}_i)$ $\forall i\in \{1,\ldots ,K\}$ , and have the top  $K$  highest Jaccard coefficient score  $\mathcal{J}(\hat{u},\hat{v})$ , are wired as pseudo cross-links. Specifically, the  $\mathcal{J}(\hat{u},\hat{v})$  can be formulated as:
$$
\mathcal {J} (\hat {u}, \hat {v}) = \frac {\left| \mathcal {N} (\hat {u}) \cap \mathcal {N} (\hat {v}) \right|}{\left| \mathcal {N} (\hat {u}) \cup \mathcal {N} (\hat {v}) \right|} \tag {3}
$$
and  $K = k * (|\mathcal{E}_{in}| - |\mathcal{E}_{cr}|)$ , where  $k$  is a hyper-parameter to control the supervision augmentation size  $K$ , and  $\mathcal{N}(\cdot)$  represents the neighbor nodes of a given node.  $\mathcal{J}(\cdot, \cdot)$  measures the proportion of common neighbors between two nodes, and intuitively, a pair of nodes with more common neighbors are more likely to form a potential link, which ensures the high confidence of augmented signals.
Random walk based augmentation. However, when a network is extremely dense, adopting Jaccard-based augmentation will consume too much computing resource due to the "neighbor explosion" issue. What's more, the Jaccard-based supervision augmentation may easily only cover nodes around the border of communities, and nodes in the center of a community may still fail to benefit from the
augmentation. To solve these problems, we propose a new supervision augmentation method based on random walks. Specifically, we take random walks in the original graph, and each pair of nodes  $\langle \hat{u},\hat{v}\rangle$  that appear in the same walk path will be recorded. Next, cross-community node pairs, i.e.,  $\mathcal{C}(\hat{u})\neq \mathcal{C}(\hat{v})$ , with the top  $K$  highest co-occurrence frequency will be screened out as augmented supervision signals. Similarly,  $K = k*\left(|\mathcal{E}_{in}| - |\mathcal{E}_{cr}|\right)$  represents the supervision augmentation size.
Note that, we only generate pseudo supervision signals, and the original graph structure, which is used for message passing in GNNs, remains unchanged. We believe that the lack of supervision plays a much more important role in causing the bias between internal-links and cross-links and adding supervision signals is a more straightforward way to alleviate this kind of bias. Let  $\mathcal{E}^P$  denote the newly constructed edge set. The original supervision signals, which are the original links on the graph, are denoted by  $\mathcal{E}^O$ , and the final augmented supervision signals are denoted by  $\mathcal{E}^A = \mathcal{E}^O \cup \mathcal{E}^P$ .
3.4 Twins GNNs
Recently, the retrieval-based learning paradigm, in which the core idea is to provide supporting examples that can be used as references for the model, has extracted increasing research interest in the field of natural language processing [35]. Inspired by this, it would be a promising way to generate debiased node embeddings as references for our framework to mitigate the bias between internal-links and cross-links. Specifically, we let  $\mathcal{E}^O$  and  $\mathcal{E}^A$  to guide the training of two twin GNNs, respectively. The two twin GNNs share the same model architecture but have independent sets of parameters. To this end, the twin-structure GNNs could generate two kinds of embeddings named original node embeddings  $\mathbf{Z}^O$  and debiased node embeddings  $\mathbf{Z}^A$ , respectively. In order to make sure that the twin-structure GNNs could precisely generate original embeddings and debiased embeddings for further embedding fusion, we follow the literature [14, 34] and employ one of the most classic loss functions in recommendation systems - BPRLoss to design our two auxiliary loss functions for the twin-structure GNNs, which can be generalized as:
$$
\mathcal {L} = - \frac {1}{| \mathcal {E} |} \sum_ {\langle u, v \rangle \in \mathcal {E}} \log \sigma \left(r _ {u, v} - r _ {u, \hat {v}}\right) \tag {4}
$$
where  $\mathcal{E}$  could be  $\mathcal{E}^O$  or  $\mathcal{E}^A$  depending on the type of generated embeddings, and the auxiliary loss functions, are denoted as  $\mathcal{L}^O$  and  $\mathcal{L}^A$ , respectively.  $\sigma$  is an activation function like sigmoid.  $\langle u,\hat{v}\rangle$  denotes a negative sample, and  $r_{u,v}$  denotes the prediction score of a given node pair  $\langle u,v\rangle$ . Note that, other alternative loss functions with multiple negative samples can be also easily deployed to our method, which is determined by the practitioners according to their practical applications. Intuitively, with the balanced proportion of two types of links in  $\mathcal{E}^A$  after supervision augmentation, the debiased node embeddings  $\mathbf{Z}^A$  could show excellent capability in eliminating the bias between internal-links and cross-links. The debiased node embeddings  $\mathbf{Z}^A$  will become a good demonstration for subsequent embedding fusion modules in the further learning of final embeddings.
3.5 Embedding Fusion
Although we set several heuristic rules in Section 3.3 to ensure that the generated signals have high confidence, the supervision augmentation stage will inevitably introduce noise. Thus,  $\mathbf{Z}^A$  is trained with impure supervision. If fusing  $\mathbf{Z}^A$  into  $\mathbf{Z}^O$  in a naive way, such as a simple averaging, although the performance on the cross-links can be improved, the internal-links' will be considerably degraded.
Thus, we adopt an embedding fusion module  $F(\cdot)$  that merges two kinds of embeddings, i.e. original node embeddings  $\mathbf{Z}^O$  and debiased node embeddings  $\mathbf{Z}^A$ , to refine meaningful information from the debiased embeddings and fuse them into the original embeddings:
$$
\mathbf {H} _ {i} = \operatorname {C O N C A T} \left(\mathbf {Z} _ {i} ^ {O}, \mathbf {Z} _ {i} ^ {A}\right) \tag {5}
$$
$$
\mathbf {Z} _ {i} = F \left(\mathbf {H} _ {i}, \theta^ {F}\right) = \mathbf {W} _ {2} \sigma \left(\mathbf {W} _ {1} \mathbf {H} _ {i} + \mathbf {b} _ {1}\right) + \mathbf {b} _ {2} \tag {6}
$$
where  $\mathbf{W}_1\in \mathbb{R}^{D\times 2D}$ ,  $\mathbf{b}_1\in \mathbb{R}^D$ ,  $\mathbf{W}_2\in \mathbb{R}^{D\times D}$ ,  $\mathbf{b}_2\in \mathbb{R}^D$  are the trainable parameters of  $F(\cdot)$  and can be abbreviated as  $\theta^F$  for conciseness.  $\sigma$  is the activation function and  $D$  is the embedding dimension. Note that, both  $\mathbf{Z}_i^O$  and  $\mathbf{Z}_i^A$  are output embeddings from GNN models supervised by auxiliary loss, and we believe that there is no need to further deploy a complex network during embedding fusion, hence we simply show a 2-layer MLP in Eq.(6).
As the core component of our framework, we argue that the final embeddings  $\mathbf{Z}$  after embedding fusion function  $F(\cdot)$  should filter signal noise in  $\mathcal{E}^A$  while preserving as much correct augmented information as possible. Therefore, we optimize embedding fusion module  $F(\cdot)$  and twin-GNNs by minimizing the following objective function:
$$
\mathcal {L} ^ {F} = - \frac {1}{| \mathcal {E} ^ {O} |} \sum_ {\langle u, v \rangle \in \mathcal {E} ^ {O}} \log \sigma \left(r _ {u, v} - r _ {u, \hat {v}}\right) \tag {7}
$$
Algorithm 1 Proposed training process
Input: Graph  $\mathcal{G}$  with link set  $\mathcal{E}^O$ . Hyper-parameters:  $\alpha$ ,  $\beta$ , and  $T$ , learning rate  $\gamma^O$ ,  $\gamma^A$ .
Output: Twin GNNs' parameters  $\theta^{\bar{O}},\theta^{A}$  , and embedding fusion module parameters  $\theta^F$
1: Randomly initialize twins GNN models with  $\theta^O$ ,  $\theta^A$ , embedding fusion module with  $\theta^F$ .
2: Split  $\mathcal{G}$  into  $|\mathcal{C}|$  communities and categorize links into internal-links and cross-links.
3: Select augmented supervision signals  $\mathcal{E}^A$  by the Jaccard coefficient or co-occurrence frequency.
4: while not converged do
5: Compute  $\mathcal{L}^O$  and  $\mathcal{L}^A$  by Eq.(4)
6: Update twins GNN models:  $\theta^O\gets \theta^O +\gamma^O\cdot \nabla_{\theta^O}\mathcal{L}^O$ $\theta^A\gets \theta^A +\gamma^A\cdot \nabla_{\theta^A}\mathcal{L}^A$
7: Compute learning rate  $\gamma_t^{F'}$  and step size  $S_{t}$  by Eq.(8)
8: for step  $= 1$  to  $S_{t}$  do
9: Compute  $\mathcal{L}^F$  by Eq.(7)
10: Update embedding fusion module:  $\theta^F\gets \theta^F +\gamma_t^F\cdot \nabla_{\theta^F}\mathcal{L}^F$
11: Update twins GNN models:  $\theta^O\gets \theta^O +\gamma_t^F\cdot \nabla_{\theta^O}\mathcal{L}^F$ $\theta^A\gets \theta^A +\gamma_t^F\cdot \nabla_{\theta^A}\mathcal{L}^F$
12: end for
13: end while
14: return  $\theta^O, \theta^A, \theta^F$
Algorithm 2 Proposed inference process