SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Input: Graph  $\mathcal{G}$  with link set  $\mathcal{E}^O$ . Trained Twin GNNs' parameters  $\theta^O, \theta^A$ , and embedding fusion module parameters  $\theta^F$
1: Get node embeddings  $Z^{O}$  through parameters  $\theta^{O}$
2: Get node embeddings  $Z^{A}$  through parameters  $\theta^{A}$
3: Get node embeddings  $Z$  through combining  $Z^{O}$  and  $Z^{A}$  with embedding fusion module  $\theta^{F}$
4: Evaluation through test node pairs with node embeddings  $Z$
3.6 Implementation Algorithms
For a better understanding of our method, here we summarize the detailed training process and inference process, and the pseudo codes are provided in Algorithm 1 and Algorithm 2, respectively.
Training process: Firstly, during the preprocessing stage,  $C = |\mathcal{C}|$  communities are detected through Louvain algorithm [2] and all links are categorized into internal-links or cross-links, respectively (line 2). In addition, newly constructed supervision signals  $\mathcal{E}^P$  are also generated during supervision augmentation (line 3). After that, original node embeddings  $\mathbf{Z}^O$  and debiased node embeddings  $\mathbf{Z}^A$  are generated by minimizing the auxiliary loss functions in Section 3.4 (lines 5-6) with learning rate  $\gamma^O$  and  $\gamma^A$ , respectively. The two sets of generated embeddings will be forwarded to the embedding fusion module  $F(\cdot)$  to get the final embeddings  $\mathbf{Z} = F(\mathbf{Z}^O, \mathbf{Z}^A)$ , and minimize the final loss function with multiple training steps (lines 8-12.) Essentially, the embedding fusion module  $F(\cdot)$  is highly related to the quality of original embeddings  $\mathbf{Z}^O$  and debiased embeddings  $\mathbf{Z}^A$ , thus we introduce a dynamic training strategy to avoid excessive training for embedding fusion module when  $\mathbf{Z}^O$  and  $\mathbf{Z}^A$  have not been stable yet. Specifically, the learning rate  $\gamma_t^F$  and training step  $S_t$  for  $F(\cdot)$  at epoch  $t$  is:
$$
\gamma_ {t} ^ {F} = \alpha * \frac {1}{1 + e ^ {- (t - T)}} \quad , \quad S _ {t} = \beta * \frac {1}{1 + e ^ {- (t - T)}} \tag {8}
$$
where  $\alpha$ ,  $\beta$ , and  $T$  are hyper-parameters to control the learning rate, training steps, and the epoch that  $F(\cdot)$  can believe the quality of input embeddings have been stable, respectively.
Inference process: During the inference stage, we use the node embeddings  $\mathbf{Z}$  for evaluation (line 4), which are the output of the embedding fusion module. Specifically, we first get node embeddings  $\mathbf{Z}^O$  and  $\mathbf{Z}^A$  through the trained twins GNN in advance (lines 1-2), and then perform embedding fusion operation (line 3) to get the final node embeddings  $\mathbf{Z}$ .
Datasets. We conduct our experiments on three real-world networks from  $\mathrm{SNAP^4}$  and RecBole [47] public datasets. Specifically, to verify our framework's extendibility, we choose two kinds of networks according to the types of interaction between nodes, including user-user (Epinions, DBLP), and user-item (LastFM). Epinions is a who-trust-whom social network extracted from
Table 1: Dataset Statistics
Datasets	Users	Items	Interactions	Density	Type
Epinions	75,879	-	508,837	0.000088	User-user
DBLP	317,080	-	1,049,866	0.000010	User-user
LastFM	1,892	17,632	92,834	0.002783	User-item
the online review site Epinions.com. Each node represents a consumer, and each directed link represents a consumer-to-consumer trust connection. DBLP is a co-author social network, which is collected from the DBLP computer science bibliography, and two author nodes are connected if they used to publish at least one paper together. LastFM dataset contains users' listening information from the Last.fm online music system and each listening event represents a user-artist interaction. The detailed statistics for these three datasets are reported in Table 1, and other experimental settings on datasets are described in Section B in the Appendix.
Base models. Because our proposed method is a model-agnostic framework, which is compatible with almost all GNN-based link prediction models, here we choose six common and effective GNNs as our base models, including GraphSAGE [13], GIN [41], GAT [37], LightGCN [14], PPRGo [3], and UltraGCN [27]. We verify if our framework could mitigate the bias between internal-links and cross-links without hurting the original overall performance.
Reproducibility. All experiments are run on machines with the same configuration: Intel(R) Xeon(R) CPU E5-2680 and Tesla V100 GPU with 32GB memory. How to detect the communities of a network is not the research focus of this paper, and here we apply the Louvain algorithm [2] to detect the communities in a graph after considering its high speed and effectiveness. Other alternative community detection algorithms and hyper-parameters will be discussed in the Appendix. The source code and data are available at https://github.com/CGCL-codes/Cross-links-Bias.
In this part, we show the performance of our method in terms of both link prediction utility and debias effectiveness after applying it to various GNN models.
The main results on three datasets are reported in Table 2. For Epinions and DBLP, we use Jaccard-based augmentation here, and considering the high density of LastFM, we use random walk-based augmentation. More results with random walk-based augmentation on Epinions and DBLP are listed in Section C in the Appendix for reference. The internal-links performance (Internal.), cross-links performance (Cross.), and overall performance (Overall) are calculated on internal-links in the test set, the cross-links in the test set, and the whole test set respectively. To verify our method's capability of addressing the bias, Bias in Eq.(1) is also computed to evaluate the performance difference between internal-links and cross-links. Table 2 shows the results of our approach and base models, and we can have the following observations:
- The performance of internal-links outperforms that of cross-links under all settings, which indicates the link prediction bias between internal-links and cross-links is widespread. Note that, after proposing our framework, we can only ease this kind of bias to some extent, which inspires us the reasons for the poor performance on cross-links may not only come from the data perspective, and we leave this problem as a future work.
4https://snap.stanford.edu/data/index.html
<sup>5</sup>https://recbole.io/index.html
Table 2: Link prediction performance (Hits@50) of internal-links, cross-links, and the whole link set of our methods and corresponding base models on three real-world datasets. The results are reported in percentage (\%). We bold the results when our framework improves the base GNN model.
Epinions	DBLP	LastFM
Internal.↑	Cross.↑	Overall↑	Bias↓	Internal.↑	Cross.↑	Overall↑	Bias↓	Internal.↑	Cross.↑	Overall↑	Bias↓
SAGE	Orig.	31.68	28.91	30.69	2.77	69.27	14.62	56.41	54.65	32.84	14.84	26.80	18.00
Debias	36.98	34.45	36.08	1.63	77.18	31.54	66.23	45.64	32.73	15.13	26.80	17.60
GIN	Orig.	33.49	30.97	32.59	2.52	56.66	16.86	47.29	39.80	33.63	16.91	28.00	16.72
Debias	40.56	39.39	40.20	1.26	60.61	34.03	54.35	26.58	32.13	19.88	28.00	12.25
GAT	Orig.	39.30	34.90	37.73	4.40	66.25	22.47	55.94	43.78	32.28	12.76	25.70	19.52
Debias	39.58	36.13	38.35	3.45	75.66	37.46	66.66	38.20	34.24	16.32	28.20	17.92
PPRGo	Orig.	42.86	28.75	37.83	14.11	85.71	41.14	75.28	44.58	34.84	17.51	29.00	17.33
Debias	47.44	42.34	45.62	5.10	90.54	55.64	82.32	34.90	35.14	19.88	30.00	15.26
LightGCN	Orig.	46.43	37.11	43.11	9.32	85.95	47.41	76.88	38.54	38.16	16.62	30.90	21.54
Debias	51.49	45.31	49.29	6.18	93.55	65.33	86.90	28.22	38.31	17.51	31.30	20.80
UltraGCN	Orig.	30.62	5.81	21.78	24.81	95.74	63.82	88.22	31.92	32.73	15.73	27.10	17.00
Debias	44.13	43.66	43.96	0.67	97.14	71.71	91.15	25.43	35.29	18.40	29.60	16.89
- After proposing our method, all base models get improved on both internal-links and cross-links in most cases, which results in an improvement in the overall performance. The reasons for this impressive improvement are two-fold. For one thing, supervision augmentation directly helps boost the performance of cross-links. For another, the augmented supervisions play a regularization role in the learning of node embeddings, thus, the overall quality of representations can be improved.
- The increase of cross-links is much higher than that of internal-links, and this contributes to the decrease of bias. For instance, when our framework is applied to LightGCN on DBLP, the performance of internal-links improves from  $85.95\%$  to  $93.55\%$ , and the performance of cross-links improves from  $47.41\%$  to  $65.33\%$ , thus the bias decreases from  $38.54\%$  to  $28.22\%$ .
In this section, we want to explore: 1) the impact of supervision augmentation on our framework; 2) the impact of the embedding fusion module on our framework; 3) the impact of two auxiliary loss functions in Eq.(4); 4) the impact of dynamic training strategy during training embedding fusion module. As an example, we show the results of LightGCN on two datasets in Table 3. All results are based on Jaccard-based augmentation.
We first replace the augmented crosslinks signals, which are selected through
Table 3: Ablation studies with LightGCN as the base model. The results (Hits@50) are reported in percentage (\%). The best results are **bold**, and the runner-up is **underlined**.
Epinions	DBLP
Internal.↑	Cross.↑	Overall↑	Internal.↑	Cross.↑	Overall↑
Original	46.43	37.11	43.11	85.95	47.41	76.88
Ours	51.49	45.31	49.29	93.55	65.33	86.90
- Augment	49.43	40.97	46.77	92.76	60.97	85.24
- Fusion	44.31	46.39	44.87	72.92	66.97	71.53
- Auxiliary Loss	49.63	41.55	46.75	85.31	60.61	84.85
- Dynamic	48.63	41.58	46.12	93.09	59.08	85.08
Jaccard coefficient, with random node pairs across different communities, and denote this variant as "-Augment". There are two main observations: 1) After adding random-augmented signals, the overall performance still shows a significant improvement compared to the base model, which indicates that the embedding fusion module can effectively filter out useful information from the augmented embeddings and fuse it into the original embeddings; 2) Compared with our framework, the random-augmented framework demonstrates unsatisfactory results on reducing the performance disparity between cross-links and internal-links, such as slightly reducing this kind of bias on Epinions dataset from  $9.32\%$  (46.43% - 37.11%) to  $8.46\%$  (49.43% - 40.97%), and this indicates the importance of our supervision augmentation method on debias.
Secondly, after removing the embedding fusion module (denoted as "-Fusion"), although the cross-links performance of LightGCN gets significant improvement, i.e., from  $37.11\%$  to  $46.39\%$  on the Epinions dataset and from  $47.41\%$  to  $66.97\%$  on the DBLP dataset, the internal-links performance and overall performance get severe influence comparing with our framework. Especially on DBLP, the internal-links performance of LightGCN is deceased by  $13.03\%$  (from  $85.95\%$  to  $72.92\%$ ), while our framework could even slightly improve the performance of internal-links. It indicates that, after proposing the embedding fusion module, our framework could effectively filter out noisy augmented signals to avoid performance deterioration.
Finally, after discarding the two auxiliary loss functions of twin GNNs in Eq.(4) ("-Auxiliary Loss") or setting a fixed learning rate  $\gamma^{F*}$  and training step  $S^*$  in the embedding fusion module ("-Dynamic"), all metrics deteriorate slightly compared to our proposed framework. This matches our expectation, because both the lack of guidance from auxiliary loss and a fixed training strategy would inevitably influence the quality of two sets of input embeddings, thus influencing the final embeddings.
4.4 Comparison with Other Competitors
In this section, we aim to compare our methods with several powerful competitors on reducing the bias between internal-links and cross-links. Specifically, we choose five highly related methods as our baselines, including FairWalk [30], CFC [4], FairAdj [20], FLIP [28], and UGE [38], and we replace the required sensitive attributes in these methods with community memberships detected by Louvain algorithm [2] in advance. To
Table 4: Comparison with several competitors on both utility and debias. The average results (in Hits@50) are reported in percentage (\%) after repeating each method three times. The best results are **bold**, and the runner-up is **underlined**.
Epinions	DBLP	LastFM
Overall↑	Bias↓	Overall↑	Bias↓	Overall↑	Bias↓
FairWalk	23.75 ± 1.1	7.16 ± 0.7	48.81 ± 1.3	50.00 ± 2.1	23.05 ± 0.6	21.78 ± 1.0
CFC	27.35 ± 0.4	2.01 ± 0.3	54.12 ± 2.1	49.37 ± 1.6	24.28 ± 0.5	17.79 ± 0.3
FairAdj	-	-	-	-	23.97 ± 1.2	19.05 ± 0.9
FLIP	25.65 ± 0.6	4.65 ± 0.5	55.23 ± 1.3	45.93 ± 1.1	22.73 ± 0.4	18.23 ± 0.5
UGE	27.22 ± 0.9	1.20 ± 0.2	54.66 ± 1.8	54.36 ± 2.3	25.50 ± 0.9	14.00 ± 0.7
GraphSAGE	30.69 ± 0.9	2.77 ± 0.3	56.41 ± 1.5	54.65 ± 1.6	26.80 ± 0.5	18.00 ± 0.6
Debias-SAGE	36.08 ± 0.7	1.63 ± 0.2	66.23 ± 1.9	45.64 ± 1.7	26.80 ± 0.4	17.60 ± 0.3
be fair, all models' embedding dimensions are set to 64, and other details on hyper-parameters are listed in Section B in the Appendix for saving space. Note that, due to the algorithm's requirement on the multiplication of the adjacency matrix, FairAdj reports "OOM" errors when deployed on Epinions and DBLP. For comparison, we report the results of GraphSAGE [13] trained with our framework (denoted as Debias-SAGE). The same as Section 4.2, we take Jaccard-based augmentation on Epinions and DBLP and take random walk-based augmentation on LastFM.
As is shown in Table 4, our method achieves state-of-the-art results on overall performance with top-ranking debias results. Especially on the DBLP dataset, our method's overall performance far exceeds that of all other baselines, while achieving the best results in reducing the bias between internal-links and cross-links. We believe the reasons are two-fold: 1) With the model-agnostic design of our framework, we could easily deploy our method on many powerful GNN models. 2) Instead of adding extra constraints in the loss function or modifying the objective functions, our method aims at addressing the bias between internal-links and cross-links from a data perspective, and the supervision augmentation plays a regularization role in boosting the overall performance.
4.5 Empirical Findings on Easing Information Cocoons
With emphasizing the performance on cross-links, one merit of our model is the potential capability to ease information cocoons, which can be empirically verified from the following two aspects.
Debiased Recommendation. In this part, we further explicitly reveal whether our framework can provide unbiased recommendations for nodes to ease the information cocoons. In detail, we train
several powerful GNN models, including LightGCN [14], PPRGo [3], and UltraGCN [27] on Epinions and DBLP. Next, we randomly select 2000 source nodes and observe the average proportion of internal-links in their historical data and their recommendation lists given by GNNs. As is shown in Figure 4, the recommendation lists given by GNN models significantly magnify the nodes' original preference on internal-links. From the perspective of the users (or nodes receiving recommendations), such biased recommendations will distort users' true preference by ignoring their niche interests (crosslinks) while emphasizing the mainstream interests (internal-links), and make users feel con
(a) Epinions
(b) DBLP
Figure 4: The internal-links proportion of historical distribution and multiple GNNs' recommendations before and after applying our method. "His.", "Rec.", and "Ours." denote the historical interactions, original recommendations, and our debiased recommendations, respectively.
fined to limited domains. For comparison, we provide the results of our enhanced GNN models as well, and it can be seen that after proposing our method (denoted as "Ours"), the distributions of
(b) DBLP
Figure 5: The visualization of subgraphs reconstructed by the embeddings learned through the base model (above) and ours (below). Note that, each column shows the visualization of the same subgraph, and subgraphs generated by our approach show minor topological isolation phenomena.
(c) LastFM
recommendations become closer to that of historical data, which indicates that our approach could effectively address GNN models' biased recommendation and thus ease the information cocoons.
Visualization of Breaking Filter Bubbles. To visualize our proposed framework's effect on alleviating information cocoons, we use Gephi<sup>6</sup> to show the graphs reconstructed by node embeddings learned through the base model and our model, respectively. To be specific, we take UltraGCN [27] as the base model here and only a subset of nodes from different communities are shown for conciseness. Note that, all the hyper-parameters in the Gephi platform are set to be the same, and the visualization results reflect the natural layout of the reconstructed networks. The visualization results are shown in Figure 5, and node colors represent the community tags. To better understand the visualization results, we also provide the modularity value in each subgraph, which measures the strength of the division of a network. It can be seen that after proposing our framework, except for some extreme outliers, the connections between most communities are becoming closer, and the isolation between communities is broken down. The reduced modularity values can also support our claims. These results indicate that the embedding generated by our model can effectively facilitate the generation of cross-links in a network, thus alleviating information cocoons.
In this paper, we aim to address the bias in GNN-based link prediction, especially the bias based on cross-links (links cross communities) after considering its specific roles in easing information cocoons and connecting different communities. We further break the paradigm of modifying objective functions in prior works while pursuing debiased performance, and turn to address the bias issue on cross-links from a data perspective, which could effectively alleviate the bias by even boosting models' prediction utility. Specifically, by borrowing the concept in retrieval-based learning paradigms, debiased node embeddings are generated as demonstrations. An embedding fusion module with dynamic training strategies is also proposed to ensure that the final embeddings could retain the merits of both original embeddings and debiased embeddings. Experiments on three real-world datasets with six base GNNs indicate that our framework could not only reduce the performance disparity between internal-links and cross-links but also significantly improve the overall performance, and further alleviate the potential information cocoons.
Limitations & Discussions. Firstly, although we have greatly reduced the bias between internal-links and cross-links, the experimental results indicate that the bias is not clearly eliminated, which implies that data bias may not be the only reason. What's more, despite the supervision augmentation having a potential regularization effect, and leading to improved performance on internal-links, we currently lack theoretical underpinnings and rigorous analysis for this phenomenon. However, we find some potential connections between our work and counterfactual learning. In our settings, supervision augmentation will introduce plenty of unobserved/counterfactual samples for GNN learning. This kind of counterfactual learning may help GNN to better capture the intrinsic relationships between a pair of nodes, thus giving a more accurate prediction.
$^{6}$ https://gephi.org/
The work is supported by the National Natural Science Foundation of China (No.62172174, No.61932004).
[44] M. Zhang and Y. Chen. Link prediction based on graph neural networks. In Proceedings of the Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 5171-5181, 2018.
[45] M. Zhang, P. Li, Y. Xia, K. Wang, and L. Jin. Labeling trick: A theory of using graph neural networks for multi-node representation learning. In Proceedings of the Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 9061-9073, 2021.
[46] T. Zhao, G. Liu, D. Wang, W. Yu, and M. Jiang. Learning from counterfactual links for link prediction. In Proceedings of the International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 26911-26926. PMLR, 2022.
[47] W. X. Zhao, S. Mu, Y. Hou, Z. Lin, Y. Chen, X. Pan, K. Li, Y. Lu, H. Wang, C. Tian, Y. Min, Z. Feng, X. Fan, X. Chen, P. Wang, W. Ji, Y. Li, X. Wang, and J. Wen. Recbole: Towards a unified, comprehensive and efficient framework for recommendation algorithms. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021, pages 4653-4664. ACM, 2021.
[48] W. Zhou, H. Huang, R. Shi, X. Song, X. Lin, X. Wang, and H. Jin. Temporal heterogeneous information network embedding via semantic evolution. IEEE Transactions on Knowledge and Data Engineering, pages 1-12, 2023.
A Further Analysis on the Role of Cross-links
A.1 The relationship between cross-links and information cocoons
To fully understand the relationship between cross-links and information cocoons, we conduct the following experiments for analysis.
- Experimental settings. Based on the communities detected by Louvain algorithm [2] in advance, we get the internal-links and cross-links of a network, and here we take Epinions and DBLP, two real-world social networks as examples. The detailed dataset information is described in Section 4.1. Next, we borrow the concept of message propagation from the Friedkin-Johnsen dynamics model [6] and revise its formula to simulate the information propagation with randomly initialized node embeddings:
$$
\mathbf {Z} _ {i} ^ {t} = \frac {\mathbf {Z} _ {i} ^ {t - 1} + \sum_ {j \in \mathcal {N} _ {i}} w _ {i j} \mathbf {Z} _ {j} ^ {t - 1}}{| \mathcal {N} _ {i} | + 1} \tag {9}
$$
where  $\mathbf{Z}_i^t$  denotes the embedding of node  $i$  at iteration  $t$ , and  $\mathcal{N}_i$  represent the neighbors of node  $i$ .  $w_{ij}$  is a manually controllable reweight scalar determined by the type of link  $\langle i, j \rangle$ . At each iteration, each node will update its embedding with the weighted average embeddings from its neighbors and itself. For simulating the lack of cross-links, we weaken the role of cross-links in information propagation by tuning the  $w$  for cross-links from 1 to 0, and setting  $w$  for internal-links to 1.
We further use Calinski-Harabasz (CH) index [5, 10] to measure the extent of the information cocoons phenomenon in a network, which can be calculated as follows:
$$
C H _ {C} = \frac {S S B _ {C} \left(\mathbf {Z} ^ {t}\right)}{S S W _ {C} \left(\mathbf {Z} ^ {t}\right)} \cdot \frac {N - C}{C - 1} \tag {10}
$$
where  $SSW_{C}$  and  $SSB_{C}$  are functions to measure the within-cluster dispersion and between-cluster dispersion, respectively [10].  $N$  denotes the number of nodes, and  $C$  denotes the number of communities. A higher CH index score indicates that node embeddings are more polarized among communities, which further illustrates the extent of the information cocoon problem in the current network.
(b) DBLP
Figure C1: The distribution of CH index score wrt. the weight of cross-links during propagation. A higher value indicates more severe information cocoons. The dashed lines indicate the CH index score under a normal setting.