Title: Combating Bilateral Edge Noise for Robust Link Prediction
Venue: NeurIPS 2023
Accepted: yes
Keywords: Robust link prediction

Edge noise
Abstract: Although link prediction on graphs has achieved great success with the development of graph neural networks (GNNs), the potential robustness under the edge noise is still less investigated. To close this gap, we first conduct an empirical study to disclose that the edge noise bilaterally perturbs both input topology and target label, yielding severe performance degradation and representation collapse. To address this dilemma, we propose an information-theory-guided principle, Robust Graph Information Bottleneck (RGIB), to extract reliable supervision signals and avoid representation collapse. Different from the basic information bottleneck, RGIB further decouples and balances the mutual dependence among graph topology, target labels, and representation, building new learning objectives for robust representation against the bilateral noise. Two instantiations, RGIB-SSL and RGIB-REP, are explored to leverage the merits of different methodologies, i.e., self-supervised learning and data reparameterization, for implicit and explicit data denoising, respectively. Extensive experiments on six datasets and three GNNs with diverse noisy scenarios verify the effectiveness of our RGIB instantiations. The code is publicly available at: https://github.com/tmlr-group/RGIB.

Combating Bilateral Edge Noise for Robust Link Prediction

Abstract

Although link prediction on graphs has achieved great success with the development of graph neural networks (GNNs), the potential robustness under the edge noise is still less investigated. To close this gap, we first conduct an empirical study to disclose that the edge noise bilaterally perturbs both input topology and target label , yielding severe performance degradation and representation collapse. To address this dilemma, we propose an information-theory-guided principle, Robust Graph Information Bottleneck (RGIB), to extract reliable supervision signals and avoid representation collapse. Different from the basic information bottleneck, RGIB further decouples and balances the mutual dependence among graph topology, target labels, and representation, building new learning objectives for robust representation against the bilateral noise. Two instantiations, RGIB-SSL and RGIB-REP, are explored to leverage the merits of different methodologies, i.e. , self-supervised learning and data reparameterization, for implicit and explicit data denoising, respectively. Extensive experiments on six datasets and three GNNs with diverse noisy scenarios verify the effectiveness of our RGIB instantiations. The code is publicly available at: https://github.com/tmlr-group/RGIB .

1 Introduction

Link prediction [25] is the process of determining whether two nodes in a graph are likely to be connected. As a fundamental task in graph learning, it has attracted growing interest in real-world applications, e.g. , drug discovery [18], knowledge graph completion [3], and question answering [17].

While recent advances in graph neural networks (GNNs) [21, 12, 22, 48, 55] have achieved superior performances, the poor robustness under the edge noise is still a practical bottleneck to the current deep graph models [11, 10, 36, 8]. Early works explore improving the robustness of GNNs under the node label noise [7, 24] by means of the smoothing effect of neighboring nodes. Other methods achieve a similar goal via randomly removing edges [30] or actively selecting the informative nodes or edges and pruning the task-irrelevant ones [50, 27]. However, when applying these noise-robust methods to the link prediction with noise, only marginal improvements are achieved (Sec. 5). The attribution is that different from the label noise, the edge noise here is bilateral : it can naturally deteriorate both the input graph topology and the target edge labels, as illustrated in Fig. 1. Such a bilateral noise is practical in real-world graph data [11, 36]. Nonetheless, previous works that only consider unilateral noise in input space or label space cannot effectively deal with such a coupled scenario. This raises a new challenge of tackling the bilateral edge noise for robust link prediction.

noisy inputs A

node repre. U

edge repre.H

GNN

GNN

Predict

encoding

decoding

unseen edges

clean edges

noisy edges

In this paper, we first disclose that the bilateral edge noise leads to the severe edge representation collapse behind the performance degradation (Sec. 3.1). Note the collapsed representation is reflected by much lower alignment and poorer uniformity (Sec. 3.2), which are used for quantitive and qualitative analysis of edge noise from representation perspective [35]. To solve this, we propose an information-theory-guided principle named Robust Graph Information Bottleneck (RGIB) (Sec. 4.1).

Conceptually, the RGIB principle is with new learning objectives that decouple the mutual information (MI) among noisy inputs, noisy labels, and representation. As illustrated in Fig. 2, RGIB generalizes the basic GIB [39] to learn a robust representation that is resistant to the bilateral edge noise. Technically, we provide two instantiations of RGIB based on different methodologies: (1) the RGIB-SSL utilizes contrastive pairs with automatically augmented views to form the informative regularization in a selfsupervised learning manner, which benefits from

GIB

RGIB

A

Y

Y

2

3

Noisy

Clean

一

2

4

3

Noisy

Clean

① = I(A;H|Y) ② = I(A;Y) ③ = I(Y;H|A)

optimalH

④= 1(A;|H) (②+③) = I(Y;H)

its intrinsic robustness to label noise (Sec. 4.2); (2) the RGIB-REP explicitly purifies the graph topology and supervision targets with the reparameterization mechanism, which enables further modeling of the mutual patterns of noise from both input and label spaces (Sec. 4.3). Both instantiations are equipped with adaptive designs, aiming to effectively estimate and balance the informative terms in a tractable manner, i.e. , the hybrid augmentation algorithm and self-adversarial alignment loss for RGIB-SSL; the relaxed information constraints on topology space and label space for RGIB-REP. Empirically, the two instantiations of RGIB work effectively under extensive noisy scenarios and can be seamlessly integrated with various GNNs (Sec. 5). Our contributions are summarized as follows.

To our best knowledge, we are the first to study the robustness problem of link prediction under bilateral edge noise. We reveal that the bilateral noise can bring severe representation collapse and performance degradation, and such negative impacts are general to common datasets and GNNs.

We propose a general learning framework, RGIB, with new representation learning objectives to promote GNNs' robustness. Two instantiations, RGIB-SSL and RGIB-REP, are proposed upon different methodologies that are equipped with adaptive designs and extensive theoretical analysis.

Without modifying the GNN architectures, the RGIB achieves state-of-the-art results on 3 GNNs and 6 datasets under various noisy scenarios, obtaining up to 12 . 9% promotion in AUC. The distribution of learned representations is notably recovered and more robust to the bilateral noise.

2 Preliminaries

Notations. We denote V = { v i } N i =1 as the set of nodes and E = { e ij } M ij =1 as the set of edges. With adjacent matrix A and node features X , a graph is denoted as G =( A,X ) , where A ij =1 means there is an edge e ij between v i and v j . X [ i, :] ∈ R D is the D -dimension node feature of v i . The link prediction task is to indicate the existence of query edges that are not observed in A . The binary labels of these query edges are denoted as Y , where 0 for negative edges and 1 for positive edges. The I ( · ; · ) indicates the mutual information between two variables. Detailed notations are in Tab. 9.

GNNs for Link Prediction. Here, we follow the encode-decode framework (Fig. 1) for link prediction [22], where the GNN architecture can be GCN [21], GAT [33], or SAGE [15]. Given a L -layer GNN f w ( · ) with learnable weights w , the node representation U ∈ R |V|× D for each node v i ∈V are obtained by a L -layer message propagation as the encoding process. For decoding, the logits φ e ij of each query edge e ij are computed by projecting the edge representation h ij = u i glyph[circledot] u j to probability. Note that the representation of all query edges is denoted as H and namely f w ( A,X )= H . Finally, the optimization objective is to minimize the binary classification loss with logits φ e ij and labels Y ij .

Topological denoising approaches. A natural way to tackle the input edge noise is to directly clean the noisy graph. Sampling-based methods, such as DropEdge [30], NeuralSparse [50], and PTDNet [27], are proposed to remove the task-irrelevant edges. Besides, as GNNs can be easily fooled by adversarial networks with only a few perturbed edges [4, 52, 9, 6], defending methods GCN-jaccard [38] and GIB [39] are designed for pruning adversarial edges from the poisoned graphs.

Label-noise-resistant techniques. To tackle the general problem of noisy labels, sample-selectionbased methods such as Co-teaching [16] let two neural networks teach each other with small-loss samples based on the memorization effect [2]. Besides, peer loss function [26] pairs independent peer examples for supervision and works within the standard empirical risk minimization framework without noise priors. For tackling label noise on graphs, label propagation techniques [7, 24] are designed to propagate the reliable signals from clean nodes to noisy ones, which are nonetheless entangled with the node annotations and node classification task that cannot be directly applied here.

3 An Empirical Study of Bilateral Edge Noise

In this section, we attempt to figure out how GNNs behave when learning with the edge noise and what are the latent mechanisms behind it. We first present an empirical study in Sec. 3.1, and then investigate the negative impact of noise through the lens of representation distribution in Sec. 3.2.

3.1 How GNNs perform under bilateral edge noise?

To quantitatively study the impact of edge noise, we simulate different levels of perturbations properly on a range of GNNs benchmark datasets, as elaborated in Def. 3.1. Note the data split manner adopt by most relevant works [22, 48, 55] randomly divides partial edges as observations and the others as prediction targets. Hence, the noisy edges will be distributed to input ˜ A and labels ˜ Y . With the bilateral noise defined in Def. 3.1, we then conduct a comprehensive empirical study. A further discussion on the edge noise is in Appendix B.1, and full evaluations can be found in Appendix D.1.

Definition 3.1 (Bilateral edge noise) . Given a clean training data, i.e., observed graph G = ( A,X ) and labels Y ∈ { 0 , 1 } of query edges, the noisy adjacence ˜ A is generated by directly adding edge noise to the original adjacent matrix A while keeping the node features X unchanged. The noisy labels ˜ Y are similarly generated by adding edge noise to the labels Y . Specifically, given a noise ratio ε a , the noisy edges A ′ ( ˜ A = A + A ′ ) are generated by flipping the zero element in A as one with the probability ε a . It satisfies that A ′ glyph[circledot] A = O and ε a = | nonzero ( ˜ A ) |-| nonzero ( A ) | / | nonzero ( A ) | . Similarly, noisy labels are generated and added to the original labels, where ε y = | nonzero ( ˜ Y ) |-| nonzero ( Y ) | / | nonzero ( Y ) | .

Observation 3.1. As illustrated in Fig. 3, the bilateral edge noise causes a significant drop in performance, and a larger noise ratio generally leads to greater degradation. It means that the standardly trained GNNs are vulnerable to the bilateral edge noise, yielding a severe robustness problem. As will be shown in Sec. 5, the performance drop brought by the bilateral noise is much greater than that of the unilateral input noise or label noise. However, none of the existing methods can effectively tackle such bilateral noise, since they only consider unilateral noise either in input space or label space. Thus, it is necessary to devise a robust method accordingly. To this end, we further investigate the underneath noise effect as follows.

clean

=20%

=40%

=60%

0.70

0.75

0.80

0.85

0.90

Test AUC

GCN-Cora

GAT-Cora

SAGE-Cora

GCN-Citeseer

GAT-Citeseer

SAGE-Citeseer





















√}√]√]√˜(√∐⌉√⌈˜√





















{˜˜∐√]√˜(√∐⌉√⌈˜√

(a) Clean data





















√}√]√]√˜(√∐⌉√⌈˜√





















{˜˜∐√]√˜(√∐⌉√⌈˜√

(b)

ε

= 20%





















√}√]√]√˜(√∐⌉√⌈˜√





















{˜˜∐√]√˜(√∐⌉√⌈˜√

(c)

ε

= 40%









































(d)

ε

= 60%

√}√]√]√˜(√∐⌉√⌈˜√

{˜˜∐√]√˜(√∐⌉√⌈˜√

3.2 Understanding the noise impact via inspecting the representation distribution

Recall that the edge representation is extracted by a forward pass as H = f w ( ˜ A,X ) and the model is optimized by stochastic gradient descent as w := w -η ∇ w L ( H , ˜ Y ) . When encountering noise within ˜ A and ˜ Y , the edge representation H can be directly influenced, since the training neglects the adverse effects of data corruption. Besides, the GNN readouts the edge logit φ e ij based on node representations h i and h j , which are possibly collapsed under the bilateral edge noise [14, 28].

To analyze the edge noise from the perspective of representation H , we use two recent concepts [35] in representation learning: (1) alignment quantifies the stability of GNN when encountering edge noise in the testing phase. It is computed as the distance of representations between two randomly augmented graphs. A higher alignment means being more resistant to input perturbations; (2) uniformity qualitatively measures the denoising effects of GNN when learning with edge noise in the training phase. Overall, a greater uniformity implies that the representation of query edges are more uniformly distributed on the unit hypersphere, preserving more information about the original data.

Observation 3.2. As can be seen from Tab. 1 and Fig. 4, a poorer alignment and a worse uniformity are brought by a severer edge noise. As the noise ratio glyph[epsilon1] increases, the learned GNN f w ( · ) is more sensitive to input perturbations as the alignment values are sharply increased, and the learned edge representations tend to be less uniformly distributed and gradually collapse into individual subregions. That is, the learned edge representation H is severely collapsed under the bilateral edge noise, which is strongly correlated with the observation in Sec. 3.1 and reflects an undesirable property of GNNs.

4 Robust Graph Information Bottleneck

Without prior knowledge like the noise ratio, without the assistance of auxiliary datasets, and even without modifying GNN architectures, how can representations be resistant to the bilateral noise? Here, we formally build a method to address the above problems from the perspective of robust graph information bottleneck (Sec. 4.1) and design its two practical instantiations (Sec. 4.2 and Sec. 4.3).

4.1 The principle of RGIB

Recall in Sec. 3.2, the edge representation H is degraded. To robustify H , one can naturally utilize the information constraint based on the graph information bottleneck (GIB) [39, 46], i.e. , solving

where the hyper-parameter γ constrains the MI I ( H ; ˜ A ) to avoid H from capturing excess taskirrelevant information from ˜ A . The basic GIB can effectively defend the input perturbation [39]. However, it is intrinsically vulnerable to label noise since it entirely preserves the label supervision and maximizes the noisy supervision I ( H ; ˜ Y ) . Our empirical results in Sec. 5 show that optimization only with Eq. 1 is ineffective when learning with the bilateral edge noise defined in Def. 3.1.

(a) RGIB (Eq. 2)

(b) RGIB-SSL (Eq. 3)

(c) RGIB-REP (Eq. 4)

uniformityH(H1),H(H2)

H(H)

A1

H

topologyconstraint

labelconstraint

A

I(H;AIY)

H

I(H;Y|A)

Y

A

1

人~

A

ZA

H

Zy

Y

A2

H

2

I(A; ZA)

I(Y;Zy)

alignment I(H;H2)

Deriving the RGIB principle. The basic GIB of Eq. 1 decreases I ( H ; ˜ A | ˜ Y ) by directly constraining I ( H ; ˜ A ) to handle the input noise. Symmetrically, the label noise can be hidden in I ( H ; ˜ Y | ˜ A ) , but trivially constraining I ( H ; ˜ Y ) to regularize I ( H ; ˜ Y | ˜ A ) is not ideal, since it conflicts with Eq. 1 and also cannot tackle the noise within I ( ˜ A ; ˜ Y ) . Thus, it is crucial to further decouple the dependence among ˜ A , ˜ Y , and H , while the noise can exist in areas of I ( H ; ˜ Y | ˜ A ) , I ( H ; ˜ A | ˜ Y ) , and I ( ˜ A ; ˜ Y | H ) . Analytically, I ( ˜ A ; ˜ Y | H )= I ( ˜ A ; ˜ Y )+ I ( H ; ˜ Y | ˜ A )+ I ( H ; ˜ A | ˜ Y ) -H ( H )+ H ( H | ˜ A, ˜ Y ) , where I ( ˜ A ; ˜ Y ) is a constant and redundancy H ( H | ˜ A, ˜ Y ) can be minimized. Thus, the I ( ˜ A ; ˜ Y | H ) can be approximated by H ( H ) , I ( H ; ˜ Y | ˜ A ) and I ( H ; ˜ A | ˜ Y ) . Since the two later terms are also noisy, a balance of these three informative terms can be a solution to the problem of bilateral edge noise.

Definition 4.1 (Robust Graph Information Bottleneck) . Based on the above analysis, we propose a new learning objective to balance informative signals regarding H , as illustrated in Fig. 5(a), i.e.,

Specifically, constraints on H ( H ) encourage a diverse H to prevent representation collapse ( >γ -H ) and also limit its capacity ( <γ + H ) to avoid over-fitting. Another two MI terms, I ( H ; ˜ Y | ˜ A ) and I ( H ; ˜ A | ˜ Y ) , mutually regularize posteriors to mitigate the negative impact of bilateral noise on H . The complete derivation of RGIB and a further comparison of RGIB and GIB are in Appendix B.2.

Note that MI terms, e.g. , I ( H ; ˜ A | ˜ Y ) are usually intractable. Therefore, we introduce two practical implementations of RGIB, i.e. , RGIB-SSL and RGIB-REP, based on different methodologies. RGIBSSL explicitly optimizes the representation H with the self-supervised regularization, while RGIBREP implicitly optimizes H by purifying the noisy ˜ A and ˜ Y with the reparameterization mechanism.

4.2 Instantiating RGIB with self-supervised learning

Recall that the edge representation has deteriorated with the supervised learning paradigm. Naturally, we modify it into a self-supervised counterpart by explicitly regularizing the representation H (see Fig. 5(b)) to avoid collapse and to implicitly capture reliable relations among noisy edges as

where margins λ s , λ u , λ a balance one supervised and two self-supervised regularization terms. When λ s ≡ 1 , λ u ≡ 0 , the RGIB-SSL can be degenerated to the basic GIB. Note H 1 and H 2 are the representations of two augmented views ˜ A 1 and ˜ A 2 , namely, H 1 = f w ( ˜ A 1 , X ) and H 2 = f w ( ˜ A 2 , X ) .

Instantiation. To achieve a tractable approximation of the MI terms in Eq. 2, we adopt the contrastive learning technique [5], and contrast pair of samples, i.e. , perturbed ˜ A 1 , ˜ A 2 that are sampled from the augmentation distribution P ( ˜ A ) . Based on which we approximate the expected supervision term by E [ I ( H ; ˜ Y | ˜ A )] ≤ E [ I ( H ; ˜ Y )] = E ˜ A s ∼ P ( ˜ A ) [ I ( H s ; ˜ Y )] ≈ 1 / 2 ( L cls ( H 1 ; ˜ Y )+ L cls ( H 2 ; ˜ Y )) . Note the approximation also supports multiple samples. Similarly, we approximate the entropy term E [ H ( H )] by 1 / 2 ( H ( H 1 )+ H ( H 2 )) , where a higher uniformity leads to a high entropy H ( H ) as proved in Prop. 4.2. Lastly, the I ( H ; ˜ A | ˜ Y ) is estimated by the alignment term I ( H 1 ; H 2 ) (refer to Prop. 4.3).

Proposition 4.2. A higher information entropy H ( H ) of edge representation H indicates a higher uniformity [35] of the representation's distribution on the unit hypersphere. Proof. See Appendix A.3.

Proposition 4.3. A lower alignment I ( H 1 ; H 2 ) indicates a lower I ( H ; ˜ A | ˜ Y ) . Since I ( H ; ˜ A | ˜ Y ) ≤ I ( H ; ˜ A ) ≤ 1 / 2 ( I ( H 1 ; H 2 )+ I ( ˜ A 1 ; ˜ A 2 ) ) = 1 / 2 ( I ( H 1 ; H 2 )+ c ) , a constrained alignment estimated by I ( H 1 ; H 2 ) can bound a lower I ( H ; ˜ A | ˜ Y ) and I ( H ; ˜ A ) . Proof. See Appendix A.4.

However, directly applying existing contrastive methods like [5, 20] can be easily suboptimal, since they are not originally designed for graph data and neglect the internal correlation between topology ˜ A and target ˜ Y . Here, we propose the following two designs to further improve robust learning.

Hybrid graph augmentation. To encourage more diverse views with lower I ( A 1 ; A 2 ) and to avoid manual selection of augmentation operations, we propose a hybrid augmentation method with four augmentation operations as predefined candidates and ranges of their corresponding hyper-parameters. In each training iteration, two augmentation operators and their hyper-parameters are automatically sampled from the search space. Then, two augmented graphs are obtained by applying the two operators on the original graph adjacency A . The detailed algorithm is elaborated in Appendix C.

glyph[negationslash]

Self-adversarial loss terms. With edge representations H 1 and H 2 from two augmented views, we build the alignment objective by minimizing the representation similarity of the positive pairs ( h 1 ij , h 2 ij ) and maximizing that of the randomly sampled negative pairs ( h 1 ij , h 2 mn ) , e ij = e mn . The proposed self-adversarial alignment loss is R align = ∑ N i =1 R pos i + R neg i . 2 Importantly, softmax functions p pos ( · ) and p neg ( · ) aim to mitigate the inefficiency problem [32] that aligned pairs are not informative. Besides, the uniformity loss is R unif = ∑ K ij,mn e -‖ h 1 ij -h 1 mn ‖ 2 2 + e -‖ h 2 ij -h 2 mn ‖ 2 2 with the Gaussian potential kernel, where edges e ij and e mn are respectively sampled from E pos and E neg .

Optimization. Regarding Eq. 3, the objective of RGIB-SSL is L = λ s L cls + λ a R align + λ u R unif . Remark 4.1. The collapsed representation comes from trivially minimizing the noisy supervision I ( H ; ˜ Y ) . The alignment and uniformity terms in Eq. 3 can alleviate such noise effects (see Sec. 5).

4.3 Instantiating RGIB with data reparameterization

Another realization is by reparameterizing the graph data on both topology space and label space jointly to preserve clean information and discard noise (as Fig. 5(c)). We propose RGIB-REP that explicitly models the reliability of ˜ A and ˜ Y via latent variables Z to learn a noise-resistant H , i.e. ,

where latent variables Z Y and Z A are clean signals extracted from noisy ˜ Y and ˜ A . Their complementary parts Z Y ′ and Z A ′ are considered as noise, satisfying ˜ Y = Z Y + Z Y ′ and ˜ A = Z A + Z A ′ . When Z Y ≡ ˜ Y and Z A ≡ ˜ A , the RGIB-REP can be degenerated to the basic GIB. Here, the I ( H ; Z Y ) measures the supervised signals with selected samples Z Y , where the classifier takes Z A ( i.e. , a subgraph of ˜ A ) as input instead of the original ˜ A , i.e. , H = f w ( Z A , X ) . Constraints I ( Z A ; ˜ A ) and I ( Z Y ; ˜ Y ) aim to select the cleanest and most task-relevant information from ˜ A and ˜ Y .

Instantiation. For deriving a tractable objective regarding Z A and Z Y , a parameterized sampler f φ ( · ) sharing the same architecture and weights as f w ( · ) is adopted here. f φ ( · ) generates the probabilistic distribution of edges that include both ˜ A and ˜ Y by P = σ ( H φ H glyph[latticetop] φ ) ∈ (0 , 1) |V| × |V| , where representation H φ = f φ ( ˜ A,X ) . Bernoulli sampling is then used to obtain high-confidence edges, i.e. , Z A = SAMP ( P | ˜ A ) and Z Y = SAMP ( P | ˜ Y ) , where | Z A | ≤| ˜ A | and | Z Y | ≤| ˜ Y | .

Proposition 4.4. Given the edge number n of ˜ A , the marginal distribution of Z A is Q ( Z A ) = P ( n ) ∏ n ˜ A ij =1 P ij . Z A satisfies I ( Z A ; ˜ A ) ≤ E [ KL ( P φ ( Z A | A ) || Q ( Z A ))] = ∑ e ij ∈ ˜ A P ij log P ij τ + (1 -P ij ) log 1 -P ij 1 -τ = R A , where τ is a constant. The topology constraint I ( Z A ; ˜ A ) in Eq. 4 is bounded by R A , and the label constraint is similarly bounded by R Y . Proof. See Appendix A.5.

Proposition 4.5. The supervision term I ( H ; Z Y ) in Eq. 4 can be empirically reduced to the classification loss, i.e., I ( H ; Z Y ) ≥ E Z Y , Z A [log P w ( Z Y | Z A )] ≈ -L cls ( f w ( Z A ) , Z Y ) , where L cls is the standard cross-entropy loss. Proof. See Appendix A.6.

Optimization. Arelaxation is then conducted on the three MI terms in Eq. 4. With derived bounds in Prop. 4.4 ( i.e. , regularization R A and R Y ) and Prop. 4.5 ( i.e. , L cls ), the final optimization objective is formed as L = λ s L cls + λ A R A + λ Y R Y , and the corresponding analysis Thm. 4.6 is as follows.

Theorem 4.6. Assume the noisy training data D train = ( ˜ A,X, ˜ Y ) contains a potentially clean subset D sub =( Z ∗ A , X, Z ∗ Y ) . The Z ∗ Y and Z ∗ A are the optimal solutions of Eq. 4 that Z ∗ Y ≈ Y , based on which a trained GNN predictor f w ( · ) satisfies f w ( Z ∗ A , X ) = Z ∗ Y + glyph[epsilon1] . The random error glyph[epsilon1] is independent of D sub and glyph[epsilon1] → 0 . Then, for arbitrary λ s , λ A , λ Y ∈ [0 , 1] , Z A = Z ∗ A and Z Y = Z ∗ Y minimizes the RGIB-REP of Eq. 4. Proof. See Appendix A.7.

Remark 4.2. Note that the methodologies take by RGIB-SSL and RGIB-REP, i.e., self-supervised learning and data reparameterization are not considered in the original GIB [39]. In Sec. 5, we justify that they are effective in instantiating the RGIB to handle the bilateral edge noise. A detailed comparison of the two instantiations is conducted in Appendix B.3. More importantly, it is possible that new instantiations based on other methodologies are inspired by the general RGIB principle.

5 Experiments

Setup. 6 popular datasets and 3 types of GNNs are taken in the experiments. The edge noise is generated based on Def. 3.1 after the commonly used data split where 85% edges are randomly selected for training, 5% as the validation set, and 10% for testing. The AUC is used as the evaluation metric as in [48, 55]. The software framework is the Pytorch [29], while the hardware is one NVIDIA RTX 3090 GPU. We repeat all experiments five times, and full results can be found in Appendix E.

Baselines. As existing robust methods separately deal with input noise or label noise, both kinds of methods can be considered as baselines. For tackling input noise , three sampling-based approaches are used for comparison, i.e. , DropEdge [30], NeuralSparse [50], and PTDNet [27]. Besides, we also include Jaccard [38], GIB [39], VIB [31], and PRI [47], which are designed for pruning adversarial edges. Two generic methods are selected for label noise , i.e. , Co-teaching [16] and Peer loss [26]. Besides, two contrastive learning methods are taken into comparison, i.e. , SupCon [20] and GRACE [54]. The implementation details of the above baselines are summarized in Appendix C.

5.1 Main results

Performance comparison. As shown in Tab. 2, the RGIB achieves the best results in all 6 datasets under the bilateral edge noise with various noise ratios, especially on challenging datasets, e.g. , Cora and Citeseer, where a 12 . 9% AUC promotion can be gained compared with the second-best methods. As for the unilateral noise settings shown in Tab. 3, RGIB still consistently surpasses all the baselines ad hoc for unilateral input noise or label noise by a large margin. We show that the bilateral edge noise can be formulated and solved by a unified learning framework, while the previous denoising methods can only work for one specific noise pattern, e.g. , the unilateral input noise or label noise.

Remark 5.1. The two instantiations of RGIB can be generalized to different scenarios with their own priority according to the intrinsic graph properties. Basically, the RGIB-SSL is more adaptive to sparser graphs, e.g., Cora and Citeseer, where the edge noise results in greater performance degradation. The RGIB-REP can be more suitable for denser graphs, e.g., Facebook and Chameleon.

Combating with adversarial attacks. Adversarial attacks on graphs can be generally divided into poisoning attacks that perturb the graph in training time and evasion attacks that perturb the graph in testing time. Here, we conduct the poisoning attacks based on Nettack [57] that only perturbs the graph structure. Notes that Nettack generates perturbations by modifying graph structure or node attributes such that perturbations maximally destroy downstream GNN's predictions. Here, we apply Nettack on Cora and Citeseer datasets as representatives. As shown in Tab. 4, the adversarial attack that adds noisy edges to the input graph also significantly degenerates the GNN's performance. And comparably, the brought damage is more severe than randomly added edges [57, 53, 42]. Crucially, we empirically justify that RGIB-SSL and RGIB-REP can also promote the robustness of GNN against adversarial attacks on the graph structure, remaining the adversarial robustness of the GIB.

The distribution of learned representation. We justify that the proposed methods can effectively alleviate the representation collapse. Compared with the standard training, both RGIB-REP and RGIB-SSL bring significant improvements to the alignment with much lower values, as in Tab. 5. At the same time, the uniformity of learned representation is also enhanced: it can be seen from Fig. 6 that the various query edges tend to be more uniformly distributed on the unit circle, especially for the negative edges. It shows that the distribution of edge representation is effectively recovered. Besides, as RGIB-SSL explicitly constrains the representation, its recovery power on representation distribution is naturally stronger than RGIB-REP, resulting in much better alignment and uniformity.

Learning with clean data. Here, we study how the robust methods behave when learning with clean data, i.e. , no edge noise exists. As shown in Tab. 7, the proposed two instantiations of RGIB can also boost the performance when learning on clean graphs, and outperforms other baselines in most cases.

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(a) Standard

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(b) RGIB-REP

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(c) RGIB-SSL

0

0.01

0.1

1.0

10

100

u

0

0.01

0.1

1.0

10

100

a

0.70

0.72

0.74

0.76

0.78

0.80

0.82

0.84

0.86

0

0.01

0.1

1.0

10

100

Y

0

0.01

0.1

1.0

10

100

A

0.64

0.66

0.68

0.70

0.72

0.74

0.76

0.78

0.80

Optimization schedulers. To reduce the search cost of coefficients in the objectives of RGIB-SSL and RGIB-REP, we set a unified optimization framework that can be formed as L = α L cls +(1 -α ) R 1 + (1 -α ) R 2 . Here, we attempt 5 different schedulers to tune the only hyper-parameter α ∈ [0 , 1] , including (1) constant, α ≡ c ; (2) linear, α t = k · t , where t is the normalized time step; (3) sine, α t = sin( t · π / 2 ) ; (4) cosine, α t = cos( t · π / 2 ) ; and (5) exponential, α t = e k · t . As empirical results summarized in Tab. 6, the selection of optimization schedulers greatly affects the final results. Although there is no golden scheduler to always perform the best, the constant and sine schedulers are generally better than others among the 5 above candidates. A further hyper-parameter study with grid search of the multiple λ in Eq. 3 of RGIB-SSL and Eq. 4 of RGIB-REP are illustrated in Fig. 7.

5.2 Ablation study

Hybrid augmentation. As shown in Tab. 8, RGIB-SSL benefits from the hybrid augmentation algorithm that automatically generates graphs of high diversity for contrastive learning. e.g. , compared with the fixed augmentation, the hybrid manner brings a 3 . 0% average AUC promotion on Cora.

Self-adversarial alignment loss. Randomly-sampled pairs with hierarchical information to be contrasted and learned from. The proposed re-weighting technique further enhances high-quality pairs and decreases low-quality counterparts. It refines the signal and brings up to 2 . 1% promotion.

1.0

1.0

1.0

topology constraint

supervised loss

1.0

0.8

0.8

0.8

label constraint

0.8

SS07

0.6

supervised loss

alignment regularization

SS07

0.6

supervisedloss

alignment regularization

SS07

0.6

SS07

0.6

0.4

uniformityregularization

0.4

uniformityregularization

0.4

0.4

0.2

0.2

0.2

0.2

supervised loss

topology constraint

label constraint

0.0

0.0

0.0

0.0

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

Epoch

Epoch

Epoch

Epoch

Information constraints. Label supervision contributes the most among the three informative terms, even though it contains label noise. As can be seen from Tab. 8, degenerating RGIB-SSL to a pure self-supervised manner without supervision ( i.e. , λ s =0 ) leads to an average 11 . 9% drop in AUC.

Edge / label selection. The two selection methods adopted in RGIB-REP are nearly equally important. Wherein, the edge selection is more important for tackling the input noise, as a greater drop will come when removed; while the label selection plays a dominant role in handling the label noise.

Topological / label constraint. Tab. 8 also shows that the selection mechanisms should be regularized by the related constraints, otherwise sub-optimal cases with degenerated effects of denoising will be achieved. Besides, the topological constraint is comparably more sensitive than the label constraint.

Learning curves. We draw the learning curves with constant schedulers in Fig. 8, where the learning processes are generally stable across datasets. A detailed analysis with more cases is in Appendix E.1.

6 Conclusion

In this work, we study the problem of link prediction with the bilateral edge noise and reveal that the edge representation is severely collapsed under such a bilateral noise. Based on the observation, we introduce the Robust Graph Information Bottleneck (RGIB) principle, aiming to extract reliable signals via decoupling and balancing the mutual information among inputs, labels, and representation to enhance the robustness and avoid collapse. Regarding the instantiations of RGIB, the selfsupervised learning technique and data reparameterization mechanism are utilized to establish the RGIB-SSL and RGIB-REP, respectively. Extensive studies on 6 common datasets and 3 popular GNNs verify the denoising effect of the proposed RGIB methods under different noisy scenarios.

References

I. Ahmad and P. Lin. A nonparametric estimation of the entropy for absolutely continuous distributions (corresp.). IEEE Transactions on Information Theory , 1976.

D. Arpit, S. Jastrz˛ ebski, N. Ballas, D. Krueger, E. Bengio, M. S Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio, et al. A closer look at memorization in deep networks. In ICML , 2017.

A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko. Translating embeddings for modeling multi-relational data. In NeurIPS , 2013.

J. Chen, Z. Shi, Y. Wu, X. Xu, and H. Zheng. Link prediction adversarial attack. arXiv , 2018.

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In ICML , 2020.

Y. Chen, H. Yang, Y. Zhang, K. Ma, T. Liu, B. Han, and J. Cheng. Understanding and improving graph injection attack by promoting unnoticeability. In ICLR , 2022.

E. Dai, C. Aggarwal, and S. Wang. Nrgnn: Learning a label noise resistant graph neural network on sparsely and noisily labeled graphs. In SIGKDD , 2021.

E. Dai, T. Zhao, H. Zhu, J. Xu, Z. Guo, H. Liu, J. Tang, and S. Wang. A comprehensive survey on trustworthy graph neural networks: Privacy, robustness, fairness, and explainability. arXiv , 2022.

N. Entezari, S. Al-Sayouri, A. Darvishzadeh, and E. Papalexakis. All you need is low (rank) defending against adversarial attacks on graphs. In WSDM , 2020.

E. Ferrara, O. Varol, C. Davis, F. Menczer, and A. Flammini. The rise of social bots. Communications of the ACM , 2016.

B. Gallagher, H. Tong, T. Eliassi-Rad, and C. Faloutsos. Using ghost edges for classification in sparsely labeled networks. In SIGKDD , 2008.

J. Gilmer, S. S Schoenholz, P. F Riley, O. Vinyals, and G. E Dahl. Neural message passing for quantum chemistry. In ICML , 2017.

C. Gong, Q. Wang, T. Liu, B. Han, J. You, J. Yang, and D. Tao. Instance-dependent positive and unlabeled learning with labeling bias estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

Florian Graf, Christoph Hofer, Marc Niethammer, and Roland Kwitt. Dissecting supervised constrastive learning. In ICML , 2021.

W. Hamilton, Z. Ying, and J. Leskovec. Inductive representation learning on large graphs. In NeurIPS , 2017.

B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS , 2018.

X. Huang, J. Zhang, D. Li, and P. Li. Knowledge graph embedding based question answering. In WSDM , 2019.

V. N Ioannidis, D. Zheng, and G. Karypis. Few-shot link prediction via graph neural networks for covid-19 drug-repurposing. arXiv , 2020.

H. Kamigaito and K. Hayashi. Comprehensive analysis of negative sampling in knowledge graph representation learning. In ICML , 2022.

P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. In NeurIPS , 2020.

T. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In ICLR , 2016.

T. Kipf and M. Welling. Variational graph auto-encoders. In NIPS Workshop on Bayesian Deep Learning , 2016.

H. Li, X. Wang, Z. Zhang, and W. Zhu. Out-of-distribution generalization on graphs: A survey. arXiv , 2022.

Y. Li, J. Yin, and L. Chen. Unified robust training for graph neural networks against label noise. In PAKDD , 2021.

D. Liben-Nowell and J. Kleinberg. The link-prediction problem for social networks. Journal of the American society for information science and technology , 2007.

Y. Liu and H. Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In ICML , 2020.

D. Luo, W. Cheng, W. Yu, C. Zong, J. Ni, H. Chen, and X. Zhang. Learning to drop: Robust graph neural network via topological denoising. In WSDM , 2021.

D. Nguyen, R. Levie, J. Lienen, G. Kutyniok, and E. Hüllermeier. Memorization-dilation: Modeling neural collapse under noise. arXiv , 2022.

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in PyTorch. In ICLR , 2017.

Y. Rong, W. Huang, T. Xu, and J. Huang. Dropedge: Towards deep graph convolutional networks on node classification. In ICLR , 2020.

Q. Sun, J. Li, H. Peng, J. Wu, X. Fu, C. Ji, and S. Philip. Graph structure learning with variational information bottleneck. In AAAI , 2022.

Z. Sun, Z. Deng, J. Nie, and J. Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. In ICLR , 2019.

P. Veliˇ ckovi´ c, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio. Graph Attention Networks. In ICLR , 2018.

Q. Wang, B. Han, T. Liu, G. Niu, J. Yang, and C. Gong. Tackling instance-dependent label noise via a universal probabilistic model. In AAAI , 2021.

T. Wang and P. Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML , 2020.

B. Wu, J. Li, C. Hou, G. Fu, Y. Bian, L. Chen, and J. Huang. Recent advances in reliable deep graph learning: Adversarial attack, inherent noise, and distribution shift. arXiv , 2022.

B. Wu, J. Li, J. Yu, Y . Bian, H. Zhang, C. Chen, C. Hou, G. Fu, L. Chen, T. Xu, et al. A survey of trustworthy graph learning: Reliability, explainability, and privacy protection. arXiv , 2022.

H. Wu, C. Wang, Y. Tyshetskiy, A. Docherty, K. Lu, and L. Zhu. Adversarial examples on graph data: Deep insights into attack and defense. arXiv , 2019.

T. Wu, H. Ren, P. Li, and J. Leskovec. Graph information bottleneck. In NeurIPS , 2020.

H. Xu and S. Mannor. Robustness and generalization. Machine learning , 2012.

Z. Yang, M. Ding, C. Zhou, H. Yang, J. Zhou, and J. Tang. Understanding negative sampling in graph representation learning. In SIGKDD , 2020.

J. Yao, B. Han, Z. Zhou, Y. Zhang, and I. Tsang. Latent class-conditional noise model. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen. Graph contrastive learning with augmentations. In NeurIPS , 2020.

Y. You, T. Chen, Z. Wang, and Y. Shen. Bringing your own view: Graph contrastive learning without prefabricated data augmentations. In WSDM , 2022.

F. Yu, Z. Qin, C. Liu, L. Zhao, Y. Wang, and X. Chen. Interpreting and evaluating neural network robustness. In IJCAI , 2019.

J. Yu, T. Xu, Y . Rong, Y . Bian, J. Huang, and R. He. Graph information bottleneck for subgraph recognition. arXiv , 2020.

S. Yu, F. Alesiani, W. Yin, R. Jenssen, and J. Principe. Principle of relevant information for graph sparsification. In UAI , 2022.

M. Zhang and Y. Chen. Link prediction based on graph neural networks. In NeurIPS , 2018.

Y. Zhang, Z. Zhou, Q. Yao, X. Chu, and B. Han. Adaprop: Learning adaptive propagation for knowledge graph reasoning. In SIGKDD , 2023.

C. Zheng, B. Zong, W. Cheng, D. Song, J. Ni, W. Yu, H. Chen, and W. Wang. Robust graph representation learning via neural sparsification. In ICML , 2020.

Z. Zhou, C. Zhou, X. Li, J. Yao, Q. Yao, and B. Han. On strengthening and defending graph reconstruction attack with markov chain approximation. In ICML , 2023.

D. Zhu, Z. Zhang, P. Cui, and W. Zhu. Robust graph convolutional networks against adversarial attacks. In SIGKDD , 2019.

J. Zhu, J. Zhang, B. Han, T. Liu, G. Niu, H. Yang, M. Kankanhalli, and M. Sugiyama. Understanding the interaction of adversarial training with noisy labels. arXiv , 2021.

Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang. Deep Graph Contrastive Representation Learning. In ICML Workshop on Graph Representation Learning and Beyond , 2020.

Z. Zhu, Z. Zhang, L. Xhonneux, and J. Tang. Neural bellman-ford networks: A general graph neural network framework for link prediction. In NeurIPS , 2021.

Z. Zhu, X. Yuan, M. Galkin, S. Xhonneux, M. Zhang, M. Gazeau, and J. Tang. A*net: A scalable path-based reasoning approach for knowledge graphs. In NeurIPS , 2023.

D. Zügner, A. Akbarnejad, and S. Günnemann. Adversarial attacks on neural networks for graph data. In SIGKDD , 2018.

Appendix

A Theoretical justification

A.1 Notations

As an expansion of the notations in Sec. 2, we summarize the frequently used notations in Tab. 9.

A.2 Preliminaries for mutual information

Definition A.1 (Informational Divergence) . The informational divergence (also called relative entropy or Kullback-Leibler distance) between two probability distributions p and q on a finite space X (i.e., a common alphabet) is defined as

Remark A.1. D ( p || q ) measures the distance between p and q . However, D ( ·||· )) is not a true metric, and it does not satisfy the triangular inequality. D ( p || q ) is non-negative and zero if and only if p = q . Definition A.2 (Mutual Information) . Given two discrete random variables X and Y , the mutual information (MI) I ( X ; Y ) is the relative entropy between the joint distribution p ( x, y ) and the product of the marginal distributions p ( x ) p ( y ) , namely,

Remark A.2. I ( X ; Y ) is symmetrical in X and Y , i.e., I ( X ; Y ) = H ( X ) -H ( X | Y ) = H ( Y ) -H ( Y | X ) = I ( Y ; X ) .

Proposition A.3 (Chain Rule for Entropy) . H ( X 1 , X 2 , · · · , X n ) = ∑ n i =1 H ( X i | X 1 , X 2 , · · · , X i -1 ) .

Proposition A.4 (Chain Rule for Conditional Entropy) . H ( X 1 , X 2 , · · · , X n | Y ) = ∑ n i =1 H ( X i | X 1 , X 2 , · · · , X i -1 , Y ) .

Proposition A.5 (Chain Rule for Mutual Information) . I ( X 1 , X 2 , · · · , X n ; Y ) = ∑ n i =1 I ( X i ; Y | X 1 , X 2 , · · · , X i -1 ) .

Corollary A.6. ∀ A,Z i , Z j , I ( A ; Z i , Z j ) ≥ max ( I ( A ; Z i ) , I ( A ; Z j ) ) .

Proof. As I ( A ; Z i | Z j ) ≥ 0 , I ( A ; Z i , Z j ) = I ( A ; Z i ) + I ( A ; Z i | Z j ) ≥ I ( A ; Z i ) . Similarly, I ( A ; Z i , Z j ) ≥ I ( A ; Z j ) can be obtained. Thus, we have I ( A ; Z i , Z j ) ≥ max ( I ( A ; Z i ) , I ( A ; Z j ) ) .

Proposition A.7 (Chain Rule for Conditional Mutual Information) . I ( X 1 , · · · , X n ; Y | Z )= ∑ n i =1 I ( X i ; Y | X 1 , · · · , X i -1 , Z ) .

A.3 Proof for Proposition 4.2

Proof. As introduced in Sec. 4.2, the uniformity loss on two augmented graphs is formed as R unif = ∑ K ij,mn e -‖ h 1 ij -h 1 mn ‖ 2 2 + e -‖ h 2 ij -h 2 mn ‖ 2 2 . For simplicity, we can reduce it to one graph, i.e. , R unif = E ij,mn ∼ p E [ e -‖ h ij -h mn ‖ 2 2 ] . Next, we obtain its upper bound by the following derivation.

Here, ˆ p vMF-KDE ( · ) is a von Mises-Fisher (vMF) kernel density estimation based on N training samples, and Z vMF is the vMF normalization constant. ˆ H ( · ) is the resubstitution entropy estimator of H = f w ( · ) [1]. As the uniformity loss L unif can be approximated by entropy ˆ H ( H ) , a higher entropy H ( H ) indicates a lower uniformity loss, i.e. , a higher uniformity.

A.4 Proof for Proposition 4.3

Proof. Recall the hybrid graph augmentation technique is adopted by RGIB-SSL for contrasting, we can approximate the expectation E [ I ( H ; ˜ A | ˜ Y )] by 1 / N ∑ N i =1 I ( H i ; ˜ A i | ˜ Y ) with N augmented graphs. Based on this, we drive the upper bound of I ( H ; ˜ A | ˜ Y ) as follows.

According to the definition of interaction information w.r.t three variables, we know that I ( X ; Y ; Z ) = I ( X ; Y ) -I ( X ; Y | Z ) ≥ 0 under the condition of Markov chain X → Z → Y . Thus, we can obtain I ( X ; Y ) ≥ I ( X ; Y | Z ) .

Specifically, the first inequality, i.e. , 1 N ∑ N i =1 I ( H i ; ˜ A i ) ≈ 1 2 ( I ( H 1 ; ˜ A 1 ) + I ( H 2 ; ˜ A 2 ) , is to approximate the expectation of N views by 2 representative views in practice. As these two views are generated by random and unbias perturbation in each training iteration, the approximation error induced here is expected to be reduced.

As for the second inequality, i.e. ,

is due to H ( H 1 , ˜ A 1 ) + H ( H 2 , ˜ A 2 ) ≥ H ( H 1 , H 2 ) + H ( ˜ A 1 , ˜ A 2 ) . Besides, I ( H 1 ; H 2 ) ≥ I ( H 1 ; ˜ A 1 ) and I ( ˜ A 1 ; ˜ A 2 ) ≥ I ( H 1 ; ˜ A 1 ) .

That is, the informative variables located in the same representation space are naturally more correlated.

Then, we have

Thus, we can obtain H ( H 1 , ˜ A 1 ) + H ( H 2 , ˜ A 2 ) ≥ H ( H 1 , H 2 ) + H ( ˜ A 1 , ˜ A 2 ) . And finally, a lower I ( H 1 ; H 2 ) can upper bounded as lower I ( H ; ˜ A ) and I ( H ; ˜ A | ˜ Y ) .

A.5 Proof for Proposition 4.4

Proof. From the perspective of data processing, the collected noisy edges in the training set can be randomly split into the observed graph ( i.e. , the input of GNN) or the predictive graph ( i.e. , the query edges for the GNN to predict). Considering the noisy edges might come from similar sources, e.g. , biases from human annotation, the corresponding noise patterns can also be similar between these two kinds of noise that are naturally coupled together to some extent. We treat the label constraint and topology constraint in the same way here, without injecting any additional heuristics or assumptions. With the marginal distribution Q ( Z A ) = ∑ n P φ ( P | n ) P ( ˜ A = n ) = P ( n ) ∏ n ˜ A ij =1 P ij , we drive the upper bound of MI I ( Z A ; ˜ A ) as:

where the KL divergence on two given distribution P ( x ) and Q ( x ) is defined as KL ( P ( x ) || Q ( x )) = ∑ x P ( x ) log P ( x ) / Q ( x ) . Thus, we obtain the upper bound of I ( Z A ; ˜ A ) as Eq. 11. Similarly, the label constraint is bounded as I ( Z Y ; ˜ Y ) ≤ ∑ e ij ∈ ˜ Y P ij log P ij τ +(1 -P ij ) log 1 -P ij 1 -τ = R Y .

A.6 Proof for Proposition 4.5

Proof. We derive the lower bound of I ( H ; Z Y ) as follows.

Thus, the MI I ( H ; Z Y ) can be lower bounded by the classification loss, and min -λ s I ( H ; Z Y ) in RGIB-REP (Eq. 4) is upper bounded by min λ s / | Z Y | ∑ e ij ∈ Z Y L cls ( f w ( Z A ) , Z Y ) as Eq. 12.

A.7 Proof for Theorem 4.6

Proof. From the theoretical perspective, RGIB-SSL explicitly optimizes the representation H with self-supervised regularizations, i.e. , alignment I ( H 1 ; H 2 ) and uniformity H ( H 1 ) , H ( H 2 ) . By contrast, RGIB-REP implicitly optimizes H by purifying the noisy ˜ A and ˜ Y with the reparameterization mechanism to extract clean signals in the forms of latent variables Z Y and Z A . The information constraints I ( Z A ; ˜ A ) , I ( Z Y ; ˜ Y ) are directly acting on Z Y and Z A and indirectly regularizing the representation H . The Z ∗ Y and Z ∗ A are the optimal solutions of RGIB-REP in Eq. 4, and that is the reason of clean labels are expected to be recovered, i.e. , Z ∗ Y = Y .

Then, with the relaxation via parametrization in A.6, we first relax the standard RGIB-REP to its upper bound as follows.

As Z ∗ Y ≈ Y , Eq. 13 can be reduced to min -λ s I ( Z A ; Y ) + λ A I ( Z A ; ˜ A ) + λ Y H ( Y ) , where I ( Y ; ˜ Y ) = H ( Y ) as Y ⊆ ˜ Y . Removing the final term with constant H ( Y ) , it can be further reduced to min -I ( Z A ; Y )+ λI ( Z A ; ˜ A ) , where λ = λ A / λ s . Since to minimize the -I ( Z A ; Y )+ λI ( Z A ; ˜ A ) is equal to maximize the I ( Z A ; Y ) -λI ( Z A ; ˜ A ) , next, we conduct the following derivation:

Since MI I ( ˜ A ; Y | Z A ) ≥ 0 and I ( Z A ; ˜ A | Y ) ≥ 0 are always true, the optimal Z ∗ A should makes -(1 -λ ) I ( ˜ A ; Y | Z A ) -λI ( Z A ; ˜ A | Y ) = 0 to reach the optimal case. Thus, it should satisfy I ( ˜ A ; Y | Z A ) = 0 and I ( Z A ; ˜ A | Y ) = 0 simultaneously. Therefore, Z A = Z ∗ A maximizes I ( Z A ; Y ) -λI ( Z A ; ˜ A ) , which is equal to minimize -I ( Z A ; Y ) + λI ( Z A ; ˜ A ) , i.e. , the RGIB-REP. Symmetrically, when Z A ≡ A , Z Y = Z ∗ Y maximizes I ( A ; Z Y ) -λI ( Z Y ; ˜ Y ) as I ( A ; Z Y ) -λI ( Z Y ; ˜ Y ) = (1 -λ ) I ( ˜ Y ; A ) -(1 -λ ) I ( ˜ Y ; A | Z Y ) -λI ( Z Y ; ˜ Y | A ) .

B Further discussions

B.1 Definition of the bilateral edge noise

Basically, the edge noise can exist in the form of false positive edges or false negative noise. Specifically, the false positive edges are treated as existing edges with label 1, but in fact, such edges should not exist. This is the standard noisy label learning problem. Conversely, the false negative edges are about the labels of the non-observed edges, which is usually considered as hard negative mining in GNNs. The predictive probabilities of such edges will be minimized. In terms of dropping edges, it actually will not inject noise into the graph data but only decrease the positive observations, which is beyond the research focus.

Here, we would highlight that our work focuses on the false positive edges as it is more practical and common in real-world scenarios since the data annotating procedure can produce such a kind of noise [37]. Thus, if the bilateral edge noise exists, it is more likely to be false positive samples, while the false negative samples are often intractable to be collected and annotated in practice. Actually, investigating the influences of the false negative samples is another line of research, such as [41, 19], which is orthogonal and complementary to our work. Learning from a clean graph can also encounter the problem of false negative samples, which is usually due to the random sampling of negative nodes/edges. Note that tackling the false negative samples is a well-known problem in the area of link prediction while handling the false positive samples is also valuable but still remains under-explored.

The input noise and label noise do look quite similar and unilateral. From the perspective of data processing, the collected noisy edges in the training set can be randomly split into the observed graph ( i.e. , the input of GNN) or the predictive graph ( i.e. , the query edges for the GNN to predict). Considering the noisy edges might come from similar sources, e.g. , biases from human annotation, the corresponding noise patterns can also be similar between these two kinds of noise that are naturally coupled together to some extent. However, we would claim and justify that the two kinds of noise are fundamentally different.

Although both noises can bring severe degradation to empirical performances, they actually act in different places from the model perspective. As the learnable weights are updated as w := w -η ∇ w L ( H , ˜ Y ) , the label noise ˜ Y acting on the backward propagation can directly influence the model. By contrast, the input noise indirectly acts on the learned weights as it appears in the front end of the forward inference, i.e. , H = f w ( A,X ) . Empirically, as results shown in Tab. 2 and Tab. 3, the standard GNN (without any defenses) performs quite differently under the same proportion of input noise and label noise. Such a phenomenon can inspire one to understand the intrinsic denoising mechanism or memorization effects of the GNN, and we would leave that as further work.

More importantly, from the perspective of defending, it could be easy and trivial to defend if these two kinds of noise are the same. However, none of the existing robust methods can effectively defend such a bilateral noise. As can be seen from Table 2, only marginal improvements are achieved when applying the existing robust methods to the bilateral noise. While in Tab. 2 and Tab. 3, these robust methods work effectively in handling the unilateral noise, i.e. , only the structure noise or label noise exists. The reason is that the properties of bilateral noise are much more complex than unilateral noise. Both sides of the information sources, i.e. , ˜ A , and ˜ Y , should be considered noisy, based on which the defending mechanism could be devised.

As pointed out by recent works [11, 37], graph noise can be produced in the data generation process that requires manually annotating the data. As in [37], in general, graph noise here refers to irreducible noises in graph structures, attributes, and labels. Thus, the studied bilateral edge noise in our work is in line with the generic concept of graph noise in the literature, which is recognized as a common problem in practice but also an underexplored challenge in both academic and industrial scenes.

In this work, we focus on the edge noise on adjacency A but not the node feature X . Specifically, we focus on the investigation of the false positive edges as the bilateral noise and design the simulation approach in Definition 3.1 as it is closer to real-world scenarios. Therefore, in the optimization objective of the proposed RGIB principle, we assume that X is generally clean and attaches more importance to the adjacency A . Besides, the presentation H is encoded from the node feature X . Thus, regularizing H in RGIB objectives can indirectly balance its dependence on X .

Input noise

Observed graph

Predictive graph

Label noise

GNN

A=A+A'

=Y + y'

Dtrain

= (A, X, Y)

B.2 The difference with GIB and RGIB

In short, RGIB generalizes the GIB with improvements in both theories and methodologies to learn a robust representation more resistant to the bilateral edge noise.

Conceptually, we would clarify that GIB is intrinsically susceptive to label noise since it entirely preserves the label supervision with maximizing. Thus, GIB cannot provide an ideal solution to the inherent edge noise investigated in this work. Different from the GIB, the RGIB takes both noisy inputs and noisy labels into the design of its information bottleneck. RGIB decouples and balances the mutual dependence among graph topology, target labels, and representation, building new learning objectives toward robust representation.

Analytically, the GIB is intrinsically not robust to label noise since it entirely preserves the label supervision with maximizing I ( H ; ˜ Y ) , as in Eq. 1. As illustrated in Fig. 2, the GIB decreases I ( H ; ˜ A | ˜ Y ) by directly constraining I ( H ; ˜ A ) to handle the input noise. Symmetrically, the label noise can be hidden in the area of I ( H ; ˜ Y | ˜ A ) , but trivially constraining I ( H ; ˜ Y ) to regularize I ( H ; ˜ Y | ˜ A ) is not ideal, since it will conflict with Eq. 1.

Besides, it cannot tackle the noise within I ( ˜ A ; ˜ Y ) , where the two kinds of noise can share similar patterns as the random split manner does not change their distributions in expectation. GIB only indirectly regularizes the MI term I ( H ; ˜ A | ˜ Y ) , as we introduce Sec. 4.1, which can only solve partial noisy information. By contrast, RGIB takes all the related MI terms, i.e. , H ( H ) , I ( H ; ˜ Y | ˜ A ) and I ( H ; ˜ A | ˜ Y ) , based on which to provide a solution that balances these MI terms as a more strict information bottleneck.

Thus, it is crucial to further decouple the mutual dependence among ˜ A , ˜ Y , and H . Regarding the representation H , noise can exist in areas of I ( H ; ˜ Y | ˜ A ) , I ( H ; ˜ A | ˜ Y ) , and I ( ˜ A ; ˜ Y | H ) , as shown in Fig. 2. As I ( ˜ A ; ˜ Y | H ) = I ( ˜ A ; ˜ Y ) + I ( H ; ˜ Y | ˜ A ) + I ( H ; ˜ A | ˜ Y ) -H ( H ) + H ( H | ˜ A, ˜ Y ) , where I ( ˜ A ; ˜ Y ) is a constant and redundancy H ( H | ˜ A, ˜ Y ) can be regularized simultaneously. Therefore, the I ( ˜ A ; ˜ Y | H ) containing the shared noise patterns of ˜ A and ˜ Y can be approximated by the other three terms, i.e. , H ( H ) , I ( H ; ˜ Y | ˜ A ) and I ( H ; ˜ A | ˜ Y ) . Since the two later terms are also with noise, a balance of these three informative terms can be a solution to the problem.

Based on the above analysis, we derive the RGIB principle in Def. 4.1 that balances the three important information terms H ( H ) , I ( H ; ˜ Y | ˜ A ) and I ( H ; ˜ A | ˜ Y ) . It works as an information bottleneck to filter out the noisy signals in both ˜ A and ˜ Y , utilizing the supervision I ( H ; ˜ Y ) at the same time.

From the view of methodology, we also provide two instantiations, RGIB-SSL and RGIB-REP, to approximate the aforementioned MI terms. These two instantiations benefit from different methodologies, i.e. , self-supervised learning and data reparameterization, for implicit and explicit data denoising, respectively. Note that these two methodologies are not considered in the original GIB. However, we show that they can be utilized to implement the specific RGIB instantiation and build effective information bottlenecks for handling the inherent noise.

Empirically, we justify the effectiveness of the two instantiations of RGIB that outperform GIB in several noisy scenarios. Besides, the GIB is highly coupled with the GAT. By contrast, RGIB does not require any modifications to GNN architecture. It can be seamlessly integrated with various existing GNNs and promote their robustness against inherent noise.

B.3 The relationship between RGIB-SSL and RGIB-REP

Based on the empirical study of several datasets and GNNs, it is found that the two instantiations can be generalized to different scenarios. Generally, RGIB-SSL is more adaptive to sparser graphs, while RGIB-REP can be more suitable for denser graphs. More importantly, they can be complementary to each other with flexible options in practical applications, as summarized in Remark 5.1.

From the theoretical perspective, RGIB-SSL explicitly optimizes the representation H with selfsupervised regularizations, i.e. , alignment I ( H 1 ; H 2 ) and uniformity H ( H 1 ) , H ( H 2 ) . By contrast, RGIB-REP implicitly optimizes H by purifying the noisy ˜ A and ˜ Y with the reparameterization mechanism to extract clean signals in the forms of latent variables Z Y and Z A . The information constraints I ( Z A ; ˜ A ) , I ( Z Y ; ˜ Y ) are directly acting on Z Y and Z A and indirectly regularizing H .

Besides, from the perspective of methodology, both instantiations are equipped with adaptive designs for obtaining an effective information bottleneck. We also conduct a further comparison and analysis of the two instantiations that are summarized as follows.

More importantly, it is also possible that new instantiations based on other kinds of methodology are inspired by the RGIB principle. We believe that such a bidirectional IB that strictly treats the information signals on both the input and label side is helpful to noisy scenarios in practice. For example, one can generalize RGIB to the reasoning task on knowledge graphs [49, 56], where the multi-relational edges characterize more diverse and complex patterns.

B.4 Assumption

The primary assumption of our work is that the edges of collected graph data can be potentially noisy. As in the submission, we present two ways of realizing the RGIB principle, which actually corresponds to some assumptions in building the objectives with the proper approximation.

RGIB-SSL assumes that the learned representation can be improved with higher uniformity and alignment. As RGIB-SSL directly acts on the graph representation, it is more suitable for recovering the distribution of representation, especially when encountering representation collapse due to the severe edge noise.

RGIB-REP explicitly purifies the input graph's topology and target labels with the jointly reparameterized Z A and Z Y . Here, the latent variables Z A , Z Y are expected to be more clean and informative than the noisy ˜ A , ˜ Y . RGIB-REP assumes that the GNN model can identify these latent variables and further benefit its learning procedure against noise.

In addition, any clean labels and adjacency are not used in training. On the contrary, the model directly learns from the noisy adjacency ˜ A and label ˜ Y , which is practical as the collected data is potentially noisy in real-world applications. Besides, RGIB and all the baselines are run without any noise priors, e.g. , noise type or noise ratio.

Empirically, RGIB-SSL is more adaptive to sparser graphs, e.g. , Cora and Citeseer, where the edge noise results in severer representation collapse. RGIB-REP can be more suitable for denser graphs, e.g. , Facebook and Chameleon, where the latent variables of edge data are extracted by RGIB-REP. More importantly, the two RGIB instantiations be complementary to each other with flexible options in practical applications.

B.5 Broader impact

In this work, we propose the RGIB framework for denoising the bilateral edge noise in graphs and for improving the robustness of link prediction. We conceptually derive the RGIB and empirically justify that it enables the various GNNs to learn the input-invariant and label-invariant edge representations, preventing representation collapse and obtaining superior performances against edge noise. Our objective is to protect and enhance the current graph models, and we do not think our work would have negative societal impacts.

Besides, A similar problem has been noticed in an early work [11], where robots tend to build connections with normal users to spread misinformation on social networks, yielding the degeneration of GNNs for robot detection. In addition, as pointed out by a recent survey [37], the bilateral noise can be produced in the data generation process that requires manually annotating the data. As introduced in [37], the bilateral noise here refers to irreducible noises in graph structures, attributes, and labels. In short, the studied edge noise in our work is in line with the generic concept of graph bilateral noise in the literature, which is recognized as a common problem in practice but also an under-explored challenge in both academic and industrial scenes.

However, several existing benchmarks, e.g. , Cora and Citeseer, are generally clean and without annotated noisy edges. Inevitably, there is a gap when one would like to study the influence of bilateral noise on these common benchmarks. Thus, to fill this gap, it is necessary to simulate the bilateral edge noise properly to investigate the impact of noise. If the edge noise exists, it should be false positive samples, while the false negative samples are often intractable to be collected. Thus, we focus on the investigation of the false positive edges as the noise and design the simulation approach in Def. 3.1 as it is closer to the real-world scenarios.

Besides, to our knowledge, we are the first to study the robustness problem of link prediction under the bilateral edge noise. One of our major contributions to this research problem is that we reveal that the bilateral noise can bring a severe representation collapse and performance degradation, and such negative impacts are general to common datasets and GNNs. What's more, it is also possible that new instantiations based on other kinds of methodology are inspired by the robust GIB principle. We believe that such a bidirectional information bottleneck strictly treating the information source on both the input side and label side is helpful in practice, especially for extremely noisy scenarios.

B.6 General robustness of GNNs

Here, we provide a discussion of the robustness based on our understanding, which is in two folds.

(1) Training-phase robustness. Briefly, the training data is "hard" due to it can be perturbed with input or label noise or with imbalance problems, e.g. , long-tail sample distributions. By contrast, the testing data is comparably "easy" because it can be clean and well-balanced. Here, the robustness is usually evaluated by how the model trained on the "hard tasks" performs on "easy tasks", i.e. , the performance on testing data. Considering the "hard tasks" can be with different difficulties, e.g. , low or high noise ratios, the change in model performances is also a natural metric to quantify the robustness. A more robust model shall have lower performance changes while maintaining good performance simultaneously [13, 34].

In addition to test accuracy, the loss visualization can be utilized to establish quantitative metrics to evaluate the intrinsic robustness of the model, e.g. , the KL divergence between two predictions on an original input and a perturbed input [45]. It can provide more insights into the model compared with the aforementioned accuracy-based robustness estimation.

(2) Testing-phase robustness. Here, the training data is "easy" and the testing data can be "hard", e.g. , with perturbation generated by adversarial attacks, or encountering the out-of-distribution samples in testing time. The robustness is evaluated here for a trained model in the testing phase, where the property is that if a testing sample is 'similar' to a training sample, i.e. , the in-distribution samples, then the testing error is close to the training error [51].

On the other hand, how this model performs on the out-of-distribution samples is usually hard to estimate, whether these samples are naturally collected or manually perturbed. Thus, measuring the performance of these "hard" testing samples, and be used to estimate the robustness of the model. An early work [40] derives generalization bounds for learning algorithms based on their robustness. And recently, out-of-distribution generalization on graphs has made great progress and attracted more attention from the research community [23]. Besides, adversarial training has been demonstrated to improve model robustness against adversarial attacks and out-of-distribution generalization ability.

C Implementation details

C.1 GNNs for link prediction

We provide a detailed introduction forward propagation and backward update of GNNs in this part.

Formally, let glyph[lscript] = 1 . . . L denote the layer index, h glyph[lscript] i is the representation of the node i , MESS ( · ) is a learnable mapping function to transform the input feature, AGGREGATE ( · ) captures the 1 -hop information from neighborhood N ( v ) in the graph, and COMBINE ( · ) is the final combination between neighbor features and the node itself. Then, the l -layer operation of GNNs can be formulated as m glyph[lscript] v = AGGREGATE glyph[lscript] ( { MESS ( h glyph[lscript] -1 u , h glyph[lscript] -1 v , e uv ): u ∈ N ( v ) } ) , where the representation of node v is h glyph[lscript] v = COMBINE glyph[lscript] ( h glyph[lscript] -1 v , m glyph[lscript] v ) . After L -layer propagation, the final node representations h L e of each e ∈ V are obtained. Then, for each query edge e ij ∈E train unseen from the input graph, the logit φ e ij is computed with the node representations h L i and h L j 3 with the readout function, i.e. , φ e ij = READOUT ( h L i , h L j ) → R . Finally, the optimization objective can be defined as minimizing the binary cross-entropy loss, i.e. , min L cls = ∑ e ij ∈E train -y ij log ( σ ( φ e ij )) -(1 -y ij ) log (1 -σ ( φ e ij )) where σ ( · ) is the sigmoid function, and y ij =1 for positive edges while y ij =0 for negative ones.

In addition, we summarize the detailed architectures of different GNNs in the following Tab. 11.

C.2 Details of RGIB-SSL and RGIB-REP

As introduced in Sec. 4.2, the graph augmentation technique [43, 44] is adopted here to generate the perpetuated graphs of various views.

To avoid manually selecting and tuning the augmentation operations, we propose the hybrid graph augmentation method with the four augmentation operations as predefined candidates and the ranges of their corresponding hyper-parameters. The search space is summarized in Tab. 12, where the candidate's operations cover most augmentation approaches except for those operations modifying the number of nodes that are unsuitable for the link prediction task.

In each training iteration, two augmentation operators T 1 ( · ) and T 2 ( · ) and their hyper-parameters θ 1 and θ 2 are randomly sampled from the search space as elaborated in Algorithm. 1. The two operators will be performed on the observed graph G , obtaining the two augmented graphs, namely,

G 1 = T 1 ( G| θ 1 ) and G 2 = T 2 ( G| θ 2 ) . The edge representations are gained by h 1 ij = f ( G 1 , e ij | w ) = h 1 i glyph[circledot] h 1 j , where the node representations h 1 i and h 2 i are generated by the GNN model f ( ·| w ) with learnable weights w , and so it is for h 2 ij with G 2 .

glyph[negationslash]

Then, we learn the self-supervised edge representations by maximizing the edge-level agreement between the same query edge of different augmented graphs (positive pairs) and minimizing the agreement among different edges (negative pairs) with their representations. Note the h ij here is the edge representation. Specifically, we minimize the representation similarity of the positive pairs ( h 1 ij , h 2 ij ) and maximize the representation similarity of the randomly-sampled negative pairs ( h 1 ij , h 2 mn ) , where e ij = e mn .

Algorithm 1 Hybrid graph augmentation.

Require: number of augmentation operators n ;

1: initialize ∅ → T ( · )

2: for i = 1 . . . n do

3: randomly selects one augmentation operator from the operator space T i ( · ) ∈ H T ;

4: uniformly sample the corresponding hyper-parameter θ i ∼ U ( θ T i ) ;

5: stores the newly sampled operator T ( · ) ∪ {T i ( ·| θ i ) } → T ( · ) ;

6: end for

7: return the hybrid augmentation operators T ( · )

C.3 A further comparison of RGIB-SSL and other contrastive learning methods

Regarding the three information terms of RGIB-SSL in Eq. 3, we elaborate a detailed clarification by comparing it with other contrastive learning methods in the following three folds.

(1) Supervision term I ( H 1 ; ˜ Y )+ I ( H 2 ; ˜ Y ) . Compared with the common manner of self-supervised contrastive learning, RGIB-SSL also considers utilizing noisy labels ˜ Y . We show that, although being potentially noisy, the labels ˜ Y can also benefit learning. Besides, our experiments show that both the supervised contrastive learning method ( e.g. , SupCon) and the self-supervised contrastive learning method ( e.g. , GRACE) perform poorly when learning with the bilateral edge noise. An intuitive explanation is that SupCon entirely trusts and utilizes the noisy ˜ Y while GRACE does not adopt ˜ Y . From the data perspective, RGIB-SSL performs a hybrid mode of supervised and self-supervised learning. It effectively extracts and utilizes the informative signals in ˜ Y without entirely absorbing its noisy signals. This is achieved with the synergy of two self-supervision terms as follows.

(2) Self-supervision alignment term I ( H 1 ; H 2 ) . Here, the main differences with common contrastive learning methods are data augmentation and loss function. Specifically, RGIB-SSL utilizes the hybrid augmentation algorithm and self-adversarial loss. Compared with the fixed augmentation manner of other contrastive learning methods [20, 54], i.e. , performs random perturbation on node features and edges, we propose a hybrid augmentation method with four augmentation operations. The motivation here is to encourage more diverse views with lower MI I ( A 1 ; A 2 ) and to avoid manual selection of augmentation operations. Besides, Prop. 4.2 provides a theoretical analysis of hybrid augmentation from information-theoretical perspectives. Empirically, compared with fixed augmentation, the hybrid brings a 3.0% average AUC promotion on Cora (please refer to Tab. 8). This means that a stronger augmentation scheme with more diverse views can help better deal with severer edge noise. Besides, the self-adversarial loss further enhances high-quality pairs and decreases low-quality counterparts. It refines the signal and brings up to 2.1% promotion.

g1 = T1(G|01)

g² = T2(G102)

Latent space H

G

Observed graph

Augmented graph I

Augmented graph 2

hj

dpos:

J...3oup

ik

h=hOh}

(3) Self-supervision uniformity term H ( H 1 ) + H ( H 2 ) . As for the uniformity term, the understanding part in Sec. 3.2 shows that a severer edge noise brings a worse uniformity. To learn a more robust representation, we add this uniformity term to form the final loss of RGIB-SSL and adopt the Gaussian potential kernel for implementation, which is usually not considered in other contrastive learning methods. The ablation studies in Sec. 5.2 also illustrate that the uniformity term is essential, especially in dealing with label noise. Besides, the uniformity of learned representation is also enhanced (see Fig. 6), and the various query edges tend to be more uniformly distributed on the unit circle, especially for the negative edges.

C.4 Baseline implementations

All baselines compared in this paper are based on their own original implementations. We list their source links here.

DropEdge, https://github.com/DropEdge/DropEdge .

NeuralSparse, https://github.com/flyingdoog/PTDNet .

PTDNet, https://github.com/flyingdoog/PTDNet .

GCN Jaccard, https://github.com/DSE-MSU/DeepRobust .

GIB, https://github.com/snap-stanford/GIB .

VIB, https://github.com/RingBDStack/VIB-GSL .

PRI, https://github.com/SJYuCNEL/PRI-Graphs .

Co-teaching, https://github.com/bhanML/Co-teaching .

Peer loss functions, https://github.com/weijiaheng/Multi-class-Peer-Loss-functions

SupCon, https://github.com/HobbitLong/SupContrast .

GRACE, https://github.com/CRIPAC-DIG/GRACE .

.

D A further empirical study of noise effects

This section is the extension of Sec. 3 in the main content.

D.1 Full evaluation results

Evaluation settings. In this part, we provide a thorough empirical study traversing all combinations of the following settings from 5 different aspects.

3 GNN architectures: GCN, GAT, SAGE.

3 numbers of layers: 2 , 4 , 6 .

4 noise types: clean, bilateral noise, input noise, label noise.

3 noise ratios: 20% , 40% , 60% .

6 datasets: Cora, CiteSeer, PubMed, Facebook, Chameleon, Squirrel.

Then, we summarize the entire evaluation results of three kinds of GNNs as follows. As can be seen, all three common GNNs, including GCN, GAT, and SAGE, are vulnerable to bilateral edge noise.

Tab. 27: full evaluation results with GCN.

Tab. 28: full evaluation results with GAT.

Tab. 29: full evaluation results with SAGE.

Data statistics. The statistics of the 6 datasets in our experiments are shown in Tab. 13. As can be seen, 4 homogeneous graphs (Cora, CiteSeer, PubMed, and Facebook) are with the much higher homophily values than the other 2 heterogeneous graphs (Chameleon and Squirrel).

D.2 Further investigation on the unilateral noise

Loss distribution We visualize the loss distribution under different scenarios to further investigate the memorization effect of GNNs. As shown in Fig. 11, two clusters are gradually separated apart with clean data, but such a learning process can be slowed down when training with noisy-input data, which confuses the model and leads to overlapped distributions. As for the label noise, the model cannot distinguish the noisy samples with a clear decision boundary to separate the clean and noisy samples apart. Besides, it is found that the model can gradually memorize the noisy edges according to the decreasing trend of the corresponding losses, where the loss distribution of noisy edges is minimized and progressively moving towards that of the clean ones.

0

1

2

0

10

Density

Clean, epoch=100

positive

negative

0

1

2

0

10

Density

Ratio=0.4, epoch=100

positive

negative

0

1

2

0

10

Density

Clean, epoch=500

positive

negative

0

1

2

0

10

Density

Ratio=0.4, epoch=500

positive

negative

0

1

2

Training Loss

0

10

Density

Clean, epoch=1000

positive

negative

0

1

2

Training Loss

0

10

Density

Ratio=0.4, epoch=1000

positive

negative

0

1

2

0

10

Density

Clean, epoch=100

clean

0

1

2

0

10

Density

Ratio=0.4, epoch=100

clean

noisy

0

1

2

0

10

Density

Clean, epoch=500

clean

0

1

2

0

10

Density

Ratio=0.4, epoch=500

clean

noisy

0

1

2

Training Loss

0

10

Density

Clean, epoch=1000

clean

0

1

2

Training Loss

0

10

Density

Ratio=0.4, epoch=1000

clean

noisy

(a) Training loss of positive/negative samples with clean data (left) and 40% input-noisy data (right). (b) Training loss of clean/noisy samples with clean data (left) and 40% label-noisy data (right).

Visualization of edge representations As shown in Fig. 12, the dimensionality reduction technique, T-SNE, is utilized here to measure the similarity of the learned edge representations h ij = [ h i , h j ] with edges e ij in the test set E test . Compared with the representation distribution of clean data (Fig. 12(a)), which present comparatively distinct cluster boundaries, both the input noise (Fig. 12(b)) and the label noise (Fig. 12(c)) are with overlapped clusters of the positive and negative samples. Namely, the GCN model is confused by the two patterns of noise, and fails to discriminate the true or false edges with their representations.

30

20

10

0

10

20

30

Component 1

30

20

10

0

10

20

30

Component 2

positive

negative

(a) Clean data

30

20

10

0

10

20

30

Component 1

30

20

10

0

10

20

30

Component 2

positive

negative

(b) Input noise

30

20

10

0

10

20

30

Component 1

30

20

10

0

10

20

30

Component 2

positive

negative

(c) Label noise

D.3 Decoupling input noise and label noise

For a further and deeper study of the bilateral noise, we use the edge homophily metric to quantify the distribution of edges. Specifically, the homophily value h homo ij of the edge e ij is computed as the cosine similarity of the node feature x i and x j , i.e. , h homo ij = cos ( x i , x j ) . As shown in Fig. 13, the distributions of edge homophily are nearly the same for label noise and structure noise, where the envelope of the two distributions is almost overlapping. Here, we justify that the randomly split label noise and structure noise are indeed coupled together.

40

40

noisyinputs

noisylabels

noisylabels

Density

30

cleaninputs

Density

20

cleanlabels

Density

30

noisyinputs

20

20

10

10

10

Q

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.1

0.2

0.3

0.4

Edgehomophily

Edge homophily

Edgehomophily

(a) Homophily distributions of input edges (b) Homophily distributions of output edges (c) Homophily distributions of noisy edges

Based on the measurement of edge homophily, we then decouple these two kinds of noise in simulation. Here, we consider two cases that separate these two kinds of noise apart, i.e. , (case 1) input noise with high homophily and label noise with low homophily shown in Fig. 14, and (case 2) input noise with low homophily and label noise with high homophily in Fig. 15.

40

40

noisyinputs

15

noisylabels

noisylabels

Density

30

cleaninputs

Density

10

cleanlabels

Density

30

noisyinputs

20

20

10

5

10

D

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Edgehomophily

Edgehomophily

Edgehomophily

(a) Homophily distributions of input edges (b) Homophily distributions of output edges (c) Homophily distributions of noisy edges

30

30

10.0

noisylabels

noisylabels

Density

7.5

cleaninputs

Density

20

cleanlabels

Density

20

noisyinputs

5.0

10

10

2.5

0.0

D

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Edgehomophily

Edgehomophily

Edgehomophily

(a) Homophily distributions of input edges (b) Homophily distributions of output edges (c) Homophily distributions of noisy edges

D.4 Large-scale datasets

We conduct an empirical study on two large-scale datasets, i.e. , PPI (56,944 nodes and 1,612,348 edges) and Yelp (716,847 nodes and 13,954,819 edges). Here, we first evaluate the performance without any defenses ( i.e. , standard training) with a 4-layer GCN that learns on different noisy cases. As shown in the Tab. 14 below, the added edge noise also degenerates the performance of GNN on these two large-scale datasets.

Next, we evaluate the effectiveness and efficiency of the proposed methods on these two datasets with glyph[epsilon1] = 40% inherent noise. As shown below, RGIB can also bring promotions compared with standard training. Besides, RGIB indeed brings extra computing costs, and specifically, RGIB-SSL can be comparably more expensive than RGIB-REP. However, the extra computing costs are not so high and within an acceptable range.

D.5 Node classification task

Weconduct experiments with a 2-layer GCN on Cora and Citeseer datasets with random label noise on nodes. As the empirical results are shown below, we justify that the RGIB framework can generalize to the node classification tasks with inherent label noise in nodes, where the two instantiations of RGIB also significantly outperform the standard training manner.

D.6 Combating feature noise

Although our RGIB methods focus on edge noise, we conduct the following experiments to verify the point. Specifically, we compare standard training, RGIB-REP, and RGIB-SSL using a 4-layer GCN on six datasets with various ratios of feature noise. Here, the noisy feature is generated by adding glyph[epsilon1] f % random noise within the range of [0 , 1] to the normalized node feature.

The evaluation results of mean AUC are shown in the below Tab. 18 - Tab. 23. As can be seen, the noisy node feature also significantly degenerates the GNN's performance, and the degradation becomes severer as the noise ratio increases. Interestingly, compared with the standard training manner, RGIB-SSL and RGIB-REP can also promote the robustness of GNN against feature noise. For example, when learning with glyph[epsilon1] f = 10% feature noise, RGIB-SSL can bring 16 . 6% and 6 . 5% improvements in AUC on Cora and Citeseer datasets, respectively.

We speculate that as the graph representation H is encoded from the node feature X , regularizing H in RGIB objectives can also balance its dependence on X and thus shows some potential robustness, even though we originally designed these objectives to handle edge noise. The above experiments show that our RGIB principle also has some merits in combating feature noise. We should note that this might be an initial verification, and a more comprehensive study will be conducted to have a rigorous conclusion in future explorations.

E Full empirical results

E.1 Further ablation studies on the trade-off parameters λ

We conduct an ablation study with the grid search of several hyper-parameters λ in RGIB. For simplicity, we fix the weight of the supervision signal as one, i.e. , λ s = 1 . Then, the objective of RGIB can be formed as L = L cls + λ 1 R 1 + λ 2 R 2 , where the information regularization terms R 1 / R 2 are alignment and uniformity for RGIB-SSL, while topology constraint and label constraint for RGIB-REP, respectively. As the heatmaps illustrated in Fig. 16, the λ 1 , λ 2 are better in certain ranges. Neither too large nor too small value is not guaranteed to find a good solution.

0

0.01

0.1

1.0

10

100

u

0

0.01

0.1

1.0

10

100

a

0.70

0.72

0.74

0.76

0.78

0.80

0.82

0.84

0.86

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

u

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

a

0.83

0.84

0.85

0.86

0

0.01

0.1

1.0

10

100

Y

0

0.01

0.1

1.0

10

100

A

0.64

0.66

0.68

0.70

0.72

0.74

0.76

0.78

0.80

(c) RGIB-REP: grid search in range of

{

0

,

10

-

2

,

10

-

1

,

10

0

,

10

1

,

10

2

}

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

Y

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

A

0.66

0.68

0.70

0.72

0.74

0.76

0.78

0.80

(d) RGIB-REP: grid search in range of

{

0

,

0

.

1

,

0

.

2

, ...,

0

.

9

,

1

.

0

}

In addition to fixing the hyper-parameters λ as constants, we also attempt dynamic tuning of these hyper-parameters within the training procedures, i.e. , using the optimization schedulers. We attempt 5 different schedulers in total, including constant, linear, sine, cosine, and exponential. As shown in Table 24, the selection of optimization schedulers greatly influences the final results, but there is no gold scheduler that consistently performs the best. Nonetheless, the constant and sine are generally better than the others.

E.2 Combination of RGIB-REP and RGIB-SSL

We attempt to jointly combine and use RGIB-REP and RGIB-SSL simultaneously. Specifically, the "RGIB-REP+RGIB-SSL" integrates both data reparameterization technique and self-supervised learning method and the final objective for optimization equals minimizing Eq. 4 and Eq. 3 simultaneously. Then, we compare "RGIB-REP+RGIB-SSL" with standard training, RGIB-REP, and RGIB-SSL based on a 4-layer GCN on all six datasets with glyph[epsilon1] = 40% bilateral noise.

As shown in the Tab. 25, the combination "RGIB-REP+RGIB-SSL" achieves considerable performances as RGIB-REP and RGIB-REP in most cases and outperforms these two instantiations on the Facebook dataset. Although we believe that a more careful finetuning of the hyper-parameters will bring further improvements to the "RGIB-REP+RGIB-SSL" combination, we suggest using one of the instantiations in practice to keep the learning objective simple and sufficient to combat the bilateral edge noise.

E.3 Further case studies

Alignment and uniformity of baseline methods. The alignment of other methods is summarized in Tab. 26, while the uniformity is visualized in Fig. 17 and Fig. 18. Specifically, the mean alignments values on different datasets ( i.e. , the x-axis) and various noise settings ( i.e. , the y axis) from clean to extremely noisy. Besides, the x and y axes in the uniformity figures are the projection dimensions as the common TSNE plots. Here, we keep the same ranges of x and y axis from 0 to 128 in each plot to provide a fair and consistent overview of representation distributions under various noisy scenarios.

The intuition is that the alignment measures how the representations can be influenced when encountering topological perturbations, while uniformity describes how the representations distribute on the hyperplane. In other words, when taking the edge noise into consideration, the alignment actually measures the stability of GNN when the input graph is with extra edge noise in the testing phase, and the uniformity measures the denoising robustness of GNN when learning with inherent edge noise in the training phase.

As introduced in Sec. 3.2, desired representations should be invariant to the topological perturbations, i.e. , with a higher alignment. On the other hand, they should also be uniformly distributed on the hyperplane to preserve as much information as possible. Based on the concepts of alignment and uniformity, we then provide quantitive and qualitative analyses of the learned representations under edge noise in Sec. 3.2. As shown, with a higher ratio of edge noise, the alignment and uniformity can be damaged to a greater extent. Thus, these two metrics can help to comprehend the learned representations by GNN under edge noise. Thus, both metrics are indeed closely connected with the edge noise here, and we reveal that a severer edge noise brings a poorer alignment and a worse uniformity.

We have the following three observations. First, the sampling-based methods, e.g. , DropEdge, PTDNet, and NeuralSparse, can also promote alignment and uniformity due to their sampling mechanisms to defend the structural perturbations. Second, the contrastive methods, e.g. , SupCon and GRACE, are with much better alignment but much worse uniformity. The reason is that the learned representations are severely collapsed, which can be degenerated to single points seen from the uniformity plots but stay nearly unchanged when encountering structural perturbations. Third, the remaining methods are not observed with significant improvements in alignment or uniformity. When connecting the above observations with their empirical performances, we can draw a conclusion. That is, both alignment and uniformity are important to evaluate the robust methods from the perspective of representation learning. Besides, such a conclusion is in line with the previous study [35].

Learning curves of RGIB. We draw the learning curves of RGIB with constant schedulers in Fig. 19, Fig. 20, Fig. 21, and Fig. 22. We normalize the values of each plotted line to (0 , 1) for better visualization and have the following observations.

For RGIB-SSL, the uniformity term, i.e. , H ( H ) , converges quickly and remains low after 200 epochs. Similarly, the alignment term, i.e. , I ( H 1 ; H 2 ) also converges in the early stages and keeps stable in the rest. At the same time, the supervised signal, i.e. , I ( H ; Y ) gradually and steadily decreases as the training time moves forward. The learning processes are generally stable across different datasets.

As for RGIB-REP, we observe that the topology I ( Z A ; ˜ A ) and label constraints I ( Z Y ; ˜ Y ) can indeed adapt to noisy scenarios with different noise ratios. As can be seen, these two regularizations converge more significantly when learning on a more noisy case. That is, when noise ratio glyph[epsilon1] increases from 0 to 60% , these two regularizations react adaptively to the noisy data ˜ A, ˜ Y . Such a phenomenon shows that RGIB-REP with these two information constraints works as an effective information bottleneck to filter out the noisy signals.

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(a) standard

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(b) Dropedge

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(c) Neural Sparse

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(d) PTDNet

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(e) co-teaching

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(f) peer loss

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(g) Jaccard

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(h) GIB

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(i) SupCon

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(j) GRACE

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(k) RGIB-REP

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(l) RGIB-SSL

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(a) standard

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(b) Dropedge

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(c) Neural Sparse

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(d) PTDNet

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(e) co-teaching

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(f) peer loss

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(g) Jaccard

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(h) GIB

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(i) SupCon

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(j) GRACE

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(k) RGIB-REP

0

32

64

96

128

0

32

64

96

128

positive samples

0

32

64

96

128

0

32

64

96

128

negative samples

(l) RGIB-SSL

(a) clean

(b)

ε

=20%

(c)

ε

=40%

(d)

ε

=60%

1.0

1.0

1.0

1.0

0.8

0.8

0.8

0.8

SS07

0.6

supervised-loss

alignment regularization

SS07

0.6

supervised loss

alignment regularization

LoSS

0.6

supervised loss

alignment regularization

SS07

0.6

supervisedloss

alignmentregularization

0.4

uniformityregularization

0.4

uniformityregularization

0.4

uniformityregularization

0.4

uniformityregularization

l

0.2

0.2

0.2

0.2

0.0

0.0

0.0

0.0

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

Epoch

Epoch

Epoch

Epoch

(a) clean

(b)

ε

=20%

(c)

ε

=40%

(d)

ε

=60%

1.0

1.0

supervised loss

1.0

supervised loss

1.0

supervised loss

topology constraint

topology constraint

topologyconstraint

0.8

0.8

label constraint

0.8

label constraint

0.8

labelconstraint

SS07

0.6

supervised loss

SS07

0.6

LoSS

0.6

SS07

0.6

topology constraint

label constraint

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.0

0.0

0.0

0.0

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

Epoch

Epoch

Epoch

Epoch

(a) clean

(b)

ε

=20%

(c)

ε

=40%

(d)

ε

=60%

1.0

1.0

1.0

1.0

0.8

0.8

0.8

0.8

0.6

alignment regularization

supervised loss

SS07

0.6

supervised loss

alignmentregularization

SSO7

0.6

alignment regularization

supervised loss

SS07

0.6

supervised loss

alignment regularization

0.4

uniformityregularization

0.4

uniformityregularization

0.4

uniformityregularization

0.4

uniformityregularization

0.2

0.2

0.2

0.2

0.0

0.0

0.0

0.0七

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

Epoch

Epoch

Epoch

Epoch

(a) clean

(b)

ε

=20%

(c)

ε

=40%

(d)

ε

=60%

1.0

1.0

supervisedloss

1.0

1.0

supervised loss

topology constraint

topology constraint

0.8

0.8

labelconstraint

0.8

0.8

labelconstraint

LoSS

0.6

supervised-loss

SS07

0.6

SS07

0.6

LoSS

0.6

topology constraint

0.4

label constraint

0.4

0.4

0.4

0.2

0.2

0.2

supervised loss

topology constraint

0.2

label constraint

0.0

0.0

0.0

0.0

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

Epoch

Epoch

Epoch

Epoch

E.4 Full baseline comparison with bilateral and unilateral noise

The entire evaluation with 10 baselines and two proposed methods are conducted keeping the same settings as in D.1.

Results on each dataset are summarized as follows.

Tab. 30, Tab. 31, Tab. 32 : full results of GCN/GAT/SAGE on Cora dataset.

Tab. 33, Tab. 34, Tab. 35 : full results of GCN/GAT/SAGE on CiteSeer dataset.

Tab. 36, Tab. 37, Tab. 38 : full results of GCN/GAT/SAGE on PubMed dataset.

Tab. 39, Tab. 40, Tab. 41 : full results of GCN/GAT/SAGE on Facebook dataset.

Tab. 42, Tab. 43, Tab. 44 : full results of GCN/GAT/SAGE on Chameleon dataset.

Tab. 45, Tab. 46, Tab. 47 : full results of GCN/GAT/SAGE on Squirrel dataset.

GCN

layers

method

bilateral noise

input noise

label noise

20%

40%

60%

20%

40%

60%

20%

40%

60%

L=2

Standard

.9473

±

.0019

.9271

±

.0027

.9141

±

.0041

.9590

±

.0022

.9468

±

.0022

.9337

±

.0016

.9646

±

.0024

.9637

±

.0021

.9597

±

.0022

DropEdge

.9394

±

.0025

.9155

±

.0027

.8994

±

.0036

.9467

±

.0022

.9302

±

.0021

.9146

±

.0022

.9594

±

.0026

.9558

±

.0027

.9519

±

.0019

NeuralSparse

.9479

±

.0021

.9251

±

.0039

.9120

±

.0029

.9558

±

.0019

.9315

±

.0033

.9269

±

.0014

.9654

±

.0015

.9525

±

.0023

.9588

±

.0017

PTDNet

.9467

±

.0018

.9264

±

.0025

.9111

±

.0032

.9554

±

.0026

.9320

±

.0035

.9272

±

.0027

.9651

±

.0018

.9616

±

.0021

.9584

±

.0021

Co-teaching

.9502

±

.0085

.9335

±

.0104

.9160

±

.0096

.9510

±

.0098

.9331

±

.0054

.9255

±

.0113

.9676

±

.0018

.9560

±

.0127

.9608

±

.0098

Peer loss

.9500

±

.0034

.9339

±

.0044

.9140

±

.0138

.9558

±

.0068

.9397

±

.0019

.9283

±

.0090

.9615

±

.0175

.9521

±

.0178

.9545

±

.0144

.9496

±

.0039

.9325

±

.0087

.9235

±

.0081

.9554

±

.0019

.9327

±

.0067

.9230

±

.0159

.9580

±

.0032

.9532

±

.0086

.9591

±

.0054

GIB

.9509

±

.0205

.9268

±

.0074

.9131

±

.0066

.9599

±

.0303

.9303

±

.0047

.9228

±

.0234

.9559

±

.0076

.9573

±

.0040

.9597

±

.0099

SupCon

.9345

±

.0020

.9257

±

.0017

.9118

±

.0031

.9583

±

.0015

.9345

±

.0030

.9214

±

.0023

.9625

±

.0021

.9522

±

.0018

.9506

±

.0010

GRACE

.9341

±

.0032

.9319

±

.0027

.9154

±

.0049

.9409

±

.0046

.9321

±

.0032

.9225

±

.0078

.9489

±

.0034

.9516

±

.0027

.9511

±

.0022

RGIB-REP

.9537

±

.0013

.9368

±

.0018

.9270

±

.0036

.9579

±

.0022

.9467

±

.0023

.9365

±

.0016

.9696

±

.0033

.9690

±

.0023

.9671

±

.0015

RGIB-SSL

.9585

±

.0022

.9471

±

.0032

.9301

±

.0021

.9425

±

.0088

.9305

±

.0098

.9121

±

.0125

.9719

±

.0020

.9724

±

.0015

.9711

±

.0012

L=4

Standard

.8870

±

.0041

.8748

±

.0031

.8641

±

.0041

.8854

±

.0051

.8759

±

.0031

.8651

±

.0040

.9030

±

.0082

.9039

±

.0029

.9070

±

.0062

DropEdge

.8711

±

.0149

.8482

±

.0045

.8354

±

.0062

.8682

±

.0158

.8456

±

.0036

.8376

±

.0046

.9313

±

.0071

.9201

±

.0091

.9240

±

.0077

NeuralSparse

.8908

±

.0080

.8733

±

.0022

.8630

±

.0049

.8931

±

.0090

.8720

±

.0043

.8649

±

.0041

.9272

±

.0108

.9136

±

.0117

.9089

±

.0084

PTDNet

.8872

±

.0071

.8733

±

.0036

.8623

±

.0050

.8903

±

.0087

.8776

±

.0078

.8609

±

.0055

.9219

±

.0122

.9099

±

.0104

.9093

±

.0101

Co-teaching

.8943

±

.0090

.8760

±

.0117

.8638

±

.0093

.8931

±

.0045

.8792

±

.0036

.8606

±

.0083

.9315

±

.0075

.9291

±

.0327

.9319

±

.0324

Peer loss

.8961

±

.0130

.8815

±

.0099

.8566

±

.0057

.8917

±

.0076

.8811

±

.0127

.8643

±

.0129

.9126

±

.0116

.9101

±

.0046

.9210

±

.0095

.8872

±

.0036

.8803

±

.0060

.8512

±

.0136

.8987

±

.0221

.8764

±

.0099

.8639

±

.0073

.9098

±

.0110

.9135

±

.0116

.9096

±

.0132

GIB

.8899

±

.0239

.8729

±

.0205

.8544

±

.0051

.8932

±

.0256

.8808

±

.0053

.8618

±

.0317

.9037

±

.0089

.9114

±

.0065

.9064

±

.0059

SupCon

.8853

±

.0061

.8718

±

.0110

.8525

±

.0108

.8867

±

.0080

.8739

±

.0033

.8558

±

.0042

.9131

±

.0068

.9108

±

.0095

.9162

±

.0125

GRACE

.8922

±

.0034

.8749

±

.0098

.8588

±

.0042

.8810

±

.0034

.8795

±

.0099

.8593

±

.0040

.9234

±

.0088

.9252

±

.0052

.9255

±

.0043

RGIB-REP

.9017

±

.0044

.8834

±

.0082

.8652

±

.0038

.9008

±

.0033

.8822

±

.0054

.8687

±

.0056

.9357

±

.0028

.9343

±

.0062

.9332

±

.0062

RGIB-SSL

.9225

±

.0125

.8918

±

.0065

.8697

±

.0053

.9126

±

.0046

.8889

±

.0052

.8693

±

.0049

.9594

±

.0026

.9604

±

.0028

.9613

±

.0023

L=6

Standard

.8870

±

.0056

.8731

±

.0032

.8640

±

.0036

.8855

±

.0025

.8742

±

.0029

.8652

±

.0041

.9050

±

.0083

.9112

±

.0059

.9063

±

.0036

DropEdge

.8623

±

.0039

.8421

±

.0044

.8342

±

.0058

.8623

±

.0054

.8407

±

.0051

.8328

±

.0058

.9140

±

.0054

.9102

±

.0082

.9092

±

.0072

NeuralSparse

.8814

±

.0049

.8586

±

.0029

.8603

±

.0060

.8792

±

.0072

.8695

±

.0031

.8612

±

.0046

.9140

±

.0065

.9130

±

.0047

.9080

±

.0077

PTDNet

.8807

±

.0053

.8610

±

.0026

.8518

±

.0040

.8791

±

.0040

.8708

±

.0032

.8619

±

.0040

.9123

±

.0074

.9086

±

.0079

.9105

±

.0072

Co-teaching

.8850

±

.0126

.8698

±

.0101

.8568

±

.0050

.8773

±

.0023

.8767

±

.0113

.8691

±

.0095

.9103

±

.0372

.9311

±

.0290

.9204

±

.0162

Peer loss

.8860

±

.0125

.8633

±

.0079

.8534

±

.0048

.8734

±

.0023

.8775

±

.0107

.8668

±

.0042

.9155

±

.0070

.9172

±

.0089

.9251

±

.0107

.8852

±

.0092

.8658

±

.0085

.8555

±

.0055

.8711

±

.0031

.8794

±

.0126

.8670

±

.0200

.9045

±

.0137

.9118

±

.0108

.9065

±

.0059

GIB

.8807

±

.0046

.8738

±

.0140

.8605

±

.0193

.8718

±

.0070

.8767

±

.0110

.8623

±

.0114

.9081

±

.0131

.9136

±

.0147

.9155

±

.0106

SupCon

.8716

±

.0090

.8627

±

.0043

.8533

±

.0045

.8705

±

.0108

.8733

±

.0026

.8643

±

.0047

.9232

±

.0081

.9294

±

.0075

.9218

±

.0092

GRACE

.8798

±

.0079

.8664

±

.0031

.8584

±

.0051

.8806

±

.0043

.8675

±

.0035

.8591

±

.0045

.9296

±

.0064

.9190

±

.0052

.9110

±

.0049

RGIB-REP

.8846

±

.0071

.8715

±

.0035

.8640

±

.0043

.8818

±

.0054

.8716

±

.0021

.8646

±

.0047

.9161

±

.0081

.9156

±

.0075

.9129

±

.0095

RGIB-SSL

.8915

±

.0062

.8737

±

.0029

.8633

±

.0036

.8891

±

.0057

.8732

±

.0034

.8639

±

.0051

.9450

±

.0042

.9488

±

.0065

.9484

±

.0027

GAT

layers

method

bilateral noise

input noise

label noise

20%

40%

60%

20%

40%

60%

20%

40%

60%

L=2

Standard

.9173

±

.0028

.8984

±

.0030

.8884

±

.0033

.9255

±

.0024

.9176

±

.0035

.9102

±

.0030

.9306

±

.0038

.9271

±

.0030

.9232

±

.0034

DropEdge

.9102

±

.0023

.8970

±

.0032

.8837

±

.0034

.9208

±

.0030

.9177

±

.0028

.9087

±

.0037

.9374

±

.0031

.9309

±

.0020

.9267

±

.0019

NeuralSparse

.9106

±

.0021

.8952

±

.0031

.8840

±

.0029

.9179

±

.0029

.9210

±

.0032

.9141

±

.0038

.9310

±

.0039

.9297

±

.0026

.9252

±

.0021

PTDNet

.9119

±

.0015

.8943

±

.0030

.8844

±

.0037

.9068

±

.0027

.9210

±

.0026

.9143

±

.0025

.9311

±

.0031

.9280

±

.0034

.9267

±

.0024

Co-teaching

.9163

±

.0044

.8924

±

.0050

.8881

±

.0100

.9058

±

.0089

.9180

±

.0089

.9136

±

.0068

.9556

±

.0098

.9364

±

.0207

.9376

±

.0249

Peer loss

.9156

±

.0115

.8968

±

.0083

.8823

±

.0041

.9147

±

.0055

.9171

±

.0089

.9173

±

.0070

.9407

±

.0225

.9296

±

.0084

.9219

±

.0229

.9128

±

.0057

.8940

±

.0094

.8831

±

.0086

.9042

±

.0092

.9181

±

.0175

.9228

±

.0214

.9304

±

.0039

.9355

±

.0127

.9282

±

.0109

GIB

.9136

±

.0061

.8965

±

.0110

.8881

±

.0115

.9052

±

.0313

.9208

±

.0097

.9383

±

.0268

.9366

±

.0069

.9291

±

.0065

.9277

±

.0089

SupCon

.9072

±

.0036

.8881

±

.0037

.8763

±

.0027

.9076

±

.0025

.9080

±

.0033

.9002

±

.0037

.9202

±

.0032

.9123

±

.0040

.9059

±

.0027

GRACE

.8230

±

.0512

.7882

±

.0693

.7914

±

.0767

.8273

±

.0671

.8053

±

.0498

.7993

±

.0695

.8792

±

.0342

.8926

±

.0257

.8946

±

.0211

RGIB-REP

.9190

±

.0025

.9034

±

.0017

.8939

±

.0056

.9250

±

.0033

.9164

±

.0024

.9099

±

.0029

.9311

±

.0036

.9306

±

.0025

.9276

±

.0022

RGIB-SSL

.9223

±

.0027

.9054

±

.0032

.8960

±

.0027

.9183

±

.0021

.9071

±

.0023

.8977

±

.0029

.9405

±

.0027

.9407

±

.0022

.9378

±

.0018

L=4

Standard

.8610

±

.0045

.8434

±

.0042

.8339

±

.0048

.8668

±

.0040

.8547

±

.0050

.8476

±

.0037

.8817

±

.0042

.8772

±

.0040

.8696

±

.0062

DropEdge

.8600

±

.0034

.8462

±

.0028

.8325

±

.0044

.8605

±

.0041

.8557

±

.0057

.8454

±

.0029

.8852

±

.0052

.8851

±

.0055

.8871

±

.0034

NeuralSparse

.8691

±

.0034

.8505

±

.0025

.8402

±

.0055

.8612

±

.0042

.8572

±

.0048

.8459

±

.0038

.8906

±

.0059

.8806

±

.0076

.8796

±

.0075

PTDNet

.8614

±

.0038

.8568

±

.0026

.8372

±

.0338

.8640

±

.0035

.8541

±

.0050

.8562

±

.0027

.8909

±

.0052

.8855

±

.0054

.8785

±

.0064

Co-teaching

.8688

±

.0125

.8450

±

.0065

.8413

±

.0081

.8666

±

.0066

.8556

±

.0103

.8468

±

.0118

.8792

±

.0107

.8946

±

.0335

.8968

±

.0132

Peer loss

.8613

±

.0040

.8493

±

.0105

.8342

±

.0108

.8652

±

.0061

.8504

±

.0143

.8471

±

.0059

.8851

±

.0110

.8839

±

.0085

.8735

±

.0069

.8630

±

.0089

.8502

±

.0082

.8365

±

.0136

.8677

±

.0198

.8495

±

.0065

.8485

±

.0116

.8863

±

.0078

.8846

±

.0060

.8750

±

.0119

GIB

.8616

±

.0036

.8534

±

.0143

.8370

±

.0116

.8814

±

.0189

.8486

±

.0059

.8421

±

.0220

.8828

±

.0111

.8827

±

.0066

.8700

±

.0076

SupCon

.8534

±

.0061

.8292

±

.0128

.8048

±

.0164

.8625

±

.0085

.8447

±

.0069

.8360

±

.0141

.8768

±

.0057

.8686

±

.0062

.8629

±

.0065

GRACE

.8355

±

.0477

.8202

±

.0387

.8046

±

.0409

.8439

±

.0344

.8078

±

.0649

.7822

±

.0686

.7878

±

.0839

.8185

±

.0702

.8008

±

.0794

RGIB-REP

.8656

±

.0043

.8443

±

.0057

.8339

±

.0033

.8674

±

.0058

.8528

±

.0043

.8392

±

.0036

.8909

±

.0084

.8925

±

.0051

.8833

±

.0062

RGIB-SSL

.8891

±

.0037

.8651

±

.0036

.8480

±

.0068

.8878

±

.0039

.8650

±

.0044

.8489

±

.0055

.9276

±

.0042

.9277

±

.0022

.9264

±

.0024

L=6

Standard

.8315

±

.0059

.8116

±

.0073

.8040

±

.0092

.8374

±

.0036

.8201

±

.0321

.8067

±

.0306

.8480

±

.0086

.8414

±

.0116

.8313

±

.0071

DropEdge

.8468

±

.0407

.8236

±

.0050

.8004

±

.0071

.8655

±

.0037

.8303

±

.0044

.7915

±

.0334

.8501

±

.0058

.8568

±

.0068

.8523

±

.0381

NeuralSparse

.8426

±

.0035

.8234

±

.0032

.8098

±

.0067

.8323

±

.0342

.8258

±

.0090

.7977

±

.0295

.8622

±

.0062

.8563

±

.0091

.8577

±

.0075

PTDNet

.8384

±

.0319

.8117

±

.0384

.8031

±

.0380

.8508

±

.0044

.8251

±

.0284

.8204

±

.0034

.8635

±

.0064

.8498

±

.0251

.8452

±

.0065

Co-teaching

.8360

±

.0122

.8111

±

.0122

.8081

±

.0113

.8391

±

.0101

.8246

±

.0398

.8071

±

.0312

.8729

±

.0090

.8647

±

.0107

.8424

±

.0079

Peer loss

.8354

±

.0128

.8186

±

.0111

.8044

±

.0104

.8416

±

.0122

.8277

±

.0359

.8133

±

.0336

.8504

±

.0132

.8563

±

.0215

.8310

±

.0089

.8333

±

.0140

.8191

±

.0073

.8044

±

.0082

.8408

±

.0221

.8340

±

.0424

.8216

±

.0497

.8528

±

.0100

.8507

±

.0210

.8304

±

.0108

GIB

.8316

±

.0125

.8266

±

.0055

.8064

±

.0241

.8359

±

.0136

.8205

±

.0345

.8274

±

.0430

.8551

±

.0149

.8493

±

.0134

.8361

±

.0072

SupCon

.8218

±

.0236

.7763

±

.0306

.7113

±

.0241

.8137

±

.0821

.8025

±

.0286

.7555

±

.0297

.8543

±

.0139

.8390

±

.0370

.8540

±

.0071

GRACE

.7773

±

.0737

.7729

±

.0344

.6821

±

.1217

.7015

±

.1340

.6953

±

.0805

.6643

±

.1009

.5212

±

.0262

.5455

±

.0459

.5743

±

.0549

RGIB-REP

.8324

±

.0039

.8162

±

.0088

.7934

±

.0068

.8248

±

.0070

.8145

±

.0088

.7892

±

.0107

.8518

±

.0056

.8496

±

.0083

.8492

±

.0113

RGIB-SSL

.8738

±

.0051

.8443

±

.0033

.8124

±

.0084

.8709

±

.0023

.8432

±

.0046

.8187

±

.0050

.9208

±

.0043

.9250

±

.0029

.9235

±

.0031

GCN

layers

method

bilateral noise

input noise

label noise

20%

40%

60%

20%

40%

60%

20%

40%

60%

L=2

Standard

.9880

±

.0007

.9866

±

.0007

.9843

±

.0010

.9878

±

.0008

.9852

±

.0006

.9834

±

.0011

.9892

±

.0006

.9888

±

.0008

.9888

±

.0007

DropEdge

.9865

±

.0007

.9845

±

.0008

.9821

±

.0009

.9764

±

.0010

.9835

±

.0007

.9708

±

.0012

.9783

±

.0006

.9879

±

.0008

.9881

±

.0007

NeuralSparse

.9877

±

.0007

.9861

±

.0008

.9837

±

.0009

.9772

±

.0007

.9852

±

.0007

.9732

±

.0010

.9788

±

.0006

.9885

±

.0007

.9888

±

.0005

PTDNet

.9876

±

.0008

.9862

±

.0008

.9838

±

.0008

.9776

±

.0007

.9848

±

.0008

.9728

±

.0011

.9859

±

.0006

.9786

±

.0009

.9886

±

.0006

Co-teaching

.9871

±

.0052

.9722

±

.0104

.9876

±

.0009

.9738

±

.0018

.9848

±

.0077

.9750

±

.0002

.9716

±

.0266

.9873

±

.0121

.9825

±

.0009

Peer loss

.9755

±

.0095

.9737

±

.0033

.9895

±

.0096

.9797

±

.0024

.9855

±

.0051

.9720

±

.0019

.9763

±

.0001

.9732

±

.0110

.9774

±

.0034

.9759

±

.0043

.9796

±

.0056

.9707

±

.0006

.9739

±

.0098

.9807

±

.0040

.9793

±

.0150

.9729

±

.0030

.9773

±

.0044

.9729

±

.0042

GIB

.9752

±

.0009

.9735

±

0.021

.9720

±

.0154

.9796

±

.0245

.9820

±

.0173

.9707

±

.0024

.9724

±

.0017

.9774

±

.1404

.9717

±

.0086

SupCon

.9779

±

.0008

.9748

±

.0014

.9665

±

.0022

.9724

±

.0009

.9803

±

.0011

.9783

±

.0010

.9713

±

.0060

.9569

±

.0160

.9701

±

.0024

GRACE

.8883

±

.0120

.8811

±

.0077

.8412

±

.0224

.8950

±

.0081

.8792

±

.0119

.8530

±

.0221

.8864

±

.0077

.8826

±

.0051

.8784

±

.0104

RGIB-REP

.9881

±

.0007

.9866

±

.0008

.9846

±

.0009

.9880

±

.0007

.9854

±

.0005

.9841

±

.0011

.9895

±

.0007

.9891

±

.0008

±

.0008

RGIB-SSL

.9845

±

.0005

.9810

±

.0005

.9765

±

.0014

.9840

±

.0011

.9808

±

.0012

.9787

±

.0009

.9862

±

.0005

.9859

±

.0010

.9866

±

.0007

L=4

Standard

.9829

±

.0020

.9520

±

.0424

.9438

±

.0402

.9819

±

.0015

.9668

±

.0147

.9622

±

.0154

.9882

±

.0007

.9880

±

.0007

.9886

±

.0006

DropEdge

.9811

±

.0028

.9682

±

.0096

.9473

±

.0120

.9803

±

.0016

.9685

±

.0033

.9531

±

.0112

.9673

±

.0008

.9771

±

.0008

.9776

±

.0009

NeuralSparse

.9825

±

.0020

.9638

±

.0039

.9456

±

.0067

.9712

±

.0022

.9691

±

.0045

.9583

±

.0071

.9781

±

.0007

.9781

±

.0008

.9784

±

.0008

PTDNet

.9725

±

.0017

.9674

±

.0131

.9485

±

.0094

.9725

±

.0014

.9668

±

.0089

.9493

±

.0267

.9879

±

.0003

.9880

±

.0008

.9783

±

.0008

Co-teaching

.9820

±

.0113

.9526

±

.0439

.9480

±

.0450

.9712

±

.0084

.9707

±

.0141

.9714

±

.0146

.9762

±

.0070

.9797

±

.0167

.9638

±

.0154

Peer loss

.9807

±

.0016

.9536

±

.0503

.9430

±

.0437

.9758

±

.0011

.9703

±

.0204

.9622

±

.0176

.9769

±

.0085

.9750

±

.0128

.9734

±

.0137

.9794

±

.0031

.9579

±

.0471

.9428

±

.0452

.9784

±

.0127

.9702

±

.0295

.9638

±

.0340

.9702

±

.0016

.9725

±

.0064

.9758

±

.0035

GIB

.9773

±

.0182

.9608

±

.0548

.9417

±

.0441

.9796

±

.0260

.9647

±

.0200

.9650

±

.0373

.9742

±

.0073

.9703

±

.0023

.9771

±

.0030

SupCon

.9588

±

.0067

.9508

±

.0100

.9297

±

.0121

.9647

±

.0088

.9517

±

.0113

.9401

±

.0135

.9647

±

.0164

.9567

±

.0149

.9553

±

.0133

GRACE

.8899

±

.0100

.8865

±

.0226

.8315

±

.0108

.9015

±

.0049

.8833

±

.0285

.8395

±

.0157

.8913

±

.0036

.8972

±

.0046

.8887

±

.0123

RGIB-REP

.9832

±

.0026

.9770

±

.0032

.9519

±

.0063

.9833

±

.0014

.9723

±

.0062

.9682

±

.0087

.9884

±

.0006

.9883

±

.0007

.9889

±

.0007

RGIB-SSL

.9829

±

.0023

.9711

±

.0025

.9643

±

.0029

.9821

±

.0019

.9707

±

.0021

.9668

±

.0040

.9857

±

.0010

.9881

±

.0008

.9857

±

.0007

L=6

Standard

.9798

±

.0013

.9609

±

.0138

.9368

±

.0179

.9764

±

.0034

.9502

±

.0096

.9469

±

.0160

.9863

±

.0013

.9865

±

.0012

.9876

±

.0010

DropEdge

.9774

±

.0030

.9635

±

.0063

.9500

±

.0103

.9540

±

.0034

.9579

±

.0083

.9411

±

.0084

.9556

±

.0014

.9642

±

.0006

.9559

±

.0012

NeuralSparse

.9774

±

.0020

.9596

±

.0067

.9435

±

.0114

.9449

±

.0029

.9496

±

.0073

.9490

±

.0087

.9564

±

.0012

.9562

±

.0011

.9465

±

.0016

PTDNet

.9767

±

.0066

.9606

±

.0131

.9449

±

.0175

.9555

±

.0034

.9568

±

.0064

.9416

±

.0145

.9560

±

.0011

.9460

±

.0014

.9569

±

.0009

Co-teaching

.9638

±

.0051

.9565

±

.0235

.9426

±

.0261

.9520

±

.0051

.9568

±

.0148

.9489

±

.0228

.9474

±

.0068

.9458

±

.0143

.9561

±

.0169

Peer loss

.9699

±

.0045

.9557

±

.0141

.9362

±

.0221

.9554

±

.0053

.9494

±

.0135

.9445

±

.0234

.9498

±

.0063

.9581

±

.0200

.9650

±

.0021

.9641

±

.0032

.9553

±

.0137

.9425

±

.0178

.9507

±

.0064

.9616

±

.0125

.9480

±

.0170

.9528

±

.0060

.9525

±

.0075

.9423

±

.0057

GIB

.9649

±

.0145

.9602

±

.0198

.9416

±

.0236

.9511

±

.0333

.9552

±

.0261

.9555

±

.0346

.9556

±

.0099

.9430

±

.0106

.9578

±

.0052

SupCon

.9462

±

.0050

.9377

±

.0054

.9282

±

.0113

.9460

±

.0073

.9355

±

.0118

.9253

±

.0110

.9411

±

.0088

.9550

±

.0087

.9510

±

.0122

GRACE

.8939

±

.0118

.8563

±

.0206

.8259

±

.0019

.9071

±

.0077

.8582

±

.0228

.8274

±

.0029

.8962

±

.0093

.9026

±

.0035

.8977

±

.0100

RGIB-REP

.9774

±

.0031

.9680

±

.0097

.9501

±

.0134

.9780

±

.0024

.9619

±

.0069

.9423

±

.0085

.9868

±

.0008

.9863

±

.0010

.9872

±

.0011

RGIB-SSL

.9751

±

.0023

.9641

±

.0037

.9395

±

.0122

.9763

±

.0034

.9633

±

.0040

.9451

±

.0084

.9845

±

.0008

.9843

±

.0012

.9847

±

.0011

SAGE

layers

method

bilateral noise

input noise

label noise

20%

40%

60%

20%

40%

60%

20%

40%

60%

L=2

Standard

.9858

±

.0008

.9827

±

.0008

.9788

±

.0012

.9881

±

.0008

.9862

±

.0007

.9862

±

.0008

.9867

±

.0006

.9840

±

.0006

.9825

±

.0014

DropEdge

.9745

±

.0008

.9807

±

.0011

.9765

±

.0011

.9766

±

.0007

.9749

±

.0011

.9752

±

.0008

.9761

±

.0008

.9739

±

.0010

.9719

±

.0011

NeuralSparse

.9753

±

.0008

.9619

±

.0011

.9683

±

.0007

.9675

±

.0008

.9660

±

.0006

.9659

±

.0011

.9667

±

.0007

.9640

±

.0007

.9627

±

.0011

PTDNet

.9755

±

.0007

.9723

±

.0009

.9779

±

.0012

.9775

±

.0007

.9761

±

.0008

.9754

±

.0007

.9766

±

.0007

.9743

±

.0007

.9726

±

.0014

Co-teaching

.9759

±

.0101

.9722

±

.0004

.9728

±

.0100

.9702

±

.0050

.9742

±

.0023

.9739

±

.0081

.9764

±

.0125

.9788

±

.0205

.9700

±

.0282

Peer loss

.9765

±

.0052

.9720

±

.0072

.9790

±

.0103

.9787

±

.0005

.9781

±

.0105

.9787

±

.0075

.9721

±

.0019

.9842

±

.0136

.9793

±

.0031

.9747

±

.0018

.9720

±

.0009

.9721

±

.0096

.9729

±

.0166

.9760

±

.0139

.9752

±

.0102

.9733

±

.0053

.9706

±

.0059

.9716

±

.0098

GIB

.9798

±

.0089

.9780

±

.0040

.9765

±

.0020

.9774

±

.0139

.9767

±

.0287

.9778

±

.0006

.9780

±

.0012

.9713

±

.0067

.9772

±

.0050

SupCon

.9482

±

.0166

.9288

±

.0046

.9141

±

.0139

.9793

±

.0043

.9782

±

.0072

.9786

±

.0078

.9278

±

.0048

.9252

±

.0074

.9156

±

.0041

GRACE

.9113

±

.0116

.8907

±

.0295

.9077

±

.0089

.9012

±

.0113

.8885

±

.0177

.9031

±

.0104

.9053

±

.0126

.9050

±

.0173

.9067

±

.0137

RGIB-REP

.9860

±

.0009

.9834

±

.0008

.9802

±

.0014

.9878

±

.0009

.9863

±

.0007

.9863

±

.0011

.9875

±

.0006

.9860

±

.0007

.9855

±

.0011

RGIB-SSL

.9840

±

.0007

.9829

±

.0010

.9811

±

.0006

.9838

±

.0007

.9821

±

.0008

.9823

±

.0004

.9841

±

.0008

.9839

±

.0007

.9844

±

.0010

L=4

Standard

.9824

±

.0015

.9783

±

.0025

.9698

±

.0040

.9849

±

.0015

.9815

±

.0024

.9815

±

.0025

.9844

±

.0015

.9817

±

.0007

.9809

±

.0009

DropEdge

.9818

±

.0026

.9764

±

.0034

.9722

±

.0022

.9625

±

.0015

.9601

±

.0043

.9600

±

.0023

.9649

±

.0008

.9627

±

.0017

.9615

±

.0012

NeuralSparse

.9811

±

.0031

.9776

±

.0021

.9740

±

.0031

.9636

±

.0023

.9600

±

.0026

.9615

±

.0016

.9648

±

.0008

.9628

±

.0014

.9613

±

.0025

PTDNet

.9823

±

.0012

.9757

±

.0041

.9735

±

.0032

.9623

±

.0041

.9609

±

.0031

.9607

±

.0022

.9652

±

.0007

.9632

±

.0009

.9623

±

.0024

Co-teaching

.9660

±

.0009

.9653

±

.0075

.9659

±

.0114

.9600

±

.0098

.9634

±

.0091

.9626

±

.0079

.9654

±

.0253

.9706

±

.0136

.9628

±

.0263

Peer loss

.9668

±

.0105

.9676

±

.0064

.9716

±

.0030

.9681

±

.0079

.9641

±

.0044

.9603

±

.0112

.9671

±

.0204

.9609

±

.0088

.9660

±

.0032

.9621

±

.0019

.9715

±

.0056

.9772

±

.0068

.9662

±

.0143

.9675

±

.0094

.9782

±

.0056

.9725

±

.0107

.9726

±

.0063

.9705

±

9.276

GIB

.9774

±

0.000

.9758

±

.0200

.9732

±

.0025

.9708

±

.0195

.9626

±

.0286

.9679

±

.0171

.9765

±

.0101

.9714

±

.0018

.9710

±

.0107

SupCon

.9169

±

.0144

.9096

±

.0191

.9001

±

.0143

.9324

±

.0208

.9188

±

.0100

.9185

±

.0192

.9295

±

.0114

.9328

±

.0158

.9325

±

.0148

GRACE

.8683

±

.0292

.8355

±

.0696

.8545

±

.0415

.8585

±

.0424

.8721

±

.0261

.8413

±

.0364

.8108

±

.0213

.8527

±

.0375

.8376

±

.0330

RGIB-REP

.9828

±

.0016

.9804

±

.0020

.9750

±

.0024

.9843

±

.0014

.9811

±

.0030

.9808

±

.0030

.9857

±

.0004

.9833

±

.0025

.9830

±

.0019

RGIB-SSL

.9817

±

.0016

.9797

±

.0014

.9768

±

.0009

.9828

±

.0018

.9798

±

.0014

.9797

±

.0013

.9821

±

.0009

.9820

±

.0011

.9826

±

.0010

L=6

Standard

.9751

±

.0055

.9603

±

.0365

.9616

±

.0091

.9715

±

.0077

.9672

±

.0057

.9517

±

.0228

.9761

±

.0136

.9788

±

.0045

.9749

±

.0151

DropEdge

.9652

±

.0039

.9676

±

.0088

.9578

±

.0149

.9734

±

.0050

.9616

±

.0126

.9665

±

.0073

.9631

±

.0022

.9606

±

.0046

.9651

±

.0145

NeuralSparse

.9640

±

.0075

.9577

±

.0222

.9672

±

.0061

.9647

±

.0048

.9694

±

.0156

.9673

±

.0036

.9632

±

.0017

.9618

±

.0037

.9615

±

.0051

PTDNet

.9668

±

.0029

.9633

±

.0142

.9617

±

.0111

.9774

±

.0028

.9693

±

.0053

.9642

±

.0055

.9629

±

.0035

.9736

±

.0010

.9708

±

.0037

Co-teaching

.9644

±

.0116

.9699

±

.0463

.9658

±

.0182

.9730

±

.0107

.9738

±

.0076

.9583

±

.0239

.9742

±

.0138

.9744

±

.0106

.9720

±

.0202

Peer loss

.9644

±

.0141

.9602

±

.0369

.9681

±

.0085

.9613

±

.0167

.9728

±

.0053

.9524

±

.0288

.9655

±

.0220

.9769

±

.0028

.9650

±

.0256

.9750

±

.0119

.9666

±

.0453

.9686

±

.0094

.9666

±

.0261

.9612

±

.0056

.9683

±

.0383

.9634

±

.0170

.9784

±

.0110

.9781

±

.0178

GIB

.9739

±

.0197

.9656

±

.0564

.9749

±

.0125

.9743

±

.0351

.9717

±

.0034

.9615

±

.0291

.9607

±

.0153

.9779

±

.0059

.9743

±

.0222

SupCon

.9007

±

.0112

.8859

±

.0189

.8730

±

.0140

.8997

±

.0142

.8940

±

.0104

.9006

±

.0232

.9146

±

.0114

.9166

±

.0143

.9185

±

.0122

GRACE

.8202

±

.0365

.7225

±

.0840

.7901

±

.1065

.8018

±

.0575

.8224

±

.0547

.7960

±

.0915

.8061

±

.0367

.8262

±

.0306

.8103

±

.0351

RGIB-REP

.9770

±

.0040

.9557

±

.0119

.9574

±

.0149

.9749

±

.0046

.9683

±

.0072

.9584

±

.0093

.9838

±

.0020

.9829

±

.0015

.9814

±

.0029

RGIB-SSL

.9773

±

.0032

.9720

±

.0031

.9678

±

.0030

.9781

±

.0027

.9739

±

.0023

.9692

±

.0050

.9774

±

.0016

.9786

±

.0012

.9786

±

.0017

GAT

layers

method

bilateral noise

input noise

label noise

20%

40%

60%

20%

40%

60%

20%

40%

60%

L=2

Standard

.9725

±

.0027

.9650

±

.0018

.9625

±

.0018

.9767

±

.0026

.9747

±

.0020

.9759

±

.0018

.9746

±

.0023

.9743

±

.0017

.9711

±

.0041

DropEdge

.9716

±

.0023

.9630

±

.0036

.9563

±

.0023

.9747

±

.0026

.9717

±

.0030

.9708

±

.0025

.9752

±

.0018

.9726

±

.0026

.9698

±

.0035

NeuralSparse

.9733

±

.0029

.9667

±

.0028

.9615

±

.0028

.9762

±

.0028

.9748

±

.0028

.9741

±

.0015

.9762

±

.0022

.9742

±

.0033

.9704

±

.0042

PTDNet

.9736

±

.0028

.9657

±

.0024

.9620

±

.0027

.9768

±

.0023

.9735

±

.0021

.9739

±

.0022

.9670

±

.0021

.9631

±

.0028

.9611

±

.0058

Co-teaching

.9617

±

.0071

.9629

±

.0047

.9626

±

.0051

.9624

±

.0084

.9645

±

.0027

.9647

±

.0104

.9684

±

.0279

.9610

±

.0139

.9688

±

.0092

Peer loss

.9675

±

.0064

.9646

±

.0101

.9683

±

.0037

.9602

±

.0046

.9621

±

.0112

.9611

±

.0078

.9638

±

.0045

.9626

±

.0074

.9620

±

.0089

.9662

±

.0063

.9665

±

.0062

.9697

±

.0073

.9652

±

.0130

.9649

±

.0051

.9616

±

.0123

.9608

±

.0076

.9612

±

.0047

.9603

±

.0134

GIB

.9628

±

.0018

.9639

±

.0115

.9625

±

.0208

.9681

±

.0091

.9643

±

.0257

.9601

±

.0285

.9644

±

.0119

.9688

±

.0096

.9631

±

.0113

SupCon

.9709

±

.0059

.9611

±

.0195

.9607

±

.0032

.9752

±

.0011

.9721

±

.0026

.9706

±

.0022

.9692

±

.0035

.9701

±

.0030

.9689

±

.0057

GRACE

.7908

±

.0348

.7829

±

.0494

.7598

±

.0693

.7992

±

.0371

.7960

±

.0371

.7913

±

.0418

.7905

±

.0261

.7985

±

.0322

.7894

±

.0340

RGIB-REP

.9736

±

.0031

.9668

±

.0021

.9629

±

.0026

.9769

±

.0018

.9752

±

.0016

.9752

±

.0018

.9766

±

.0021

.9753

±

.0029

.9732

±

.0041

RGIB-SSL

.9635

±

.0037

.9593

±

.0037

.9536

±

.0041

.9640

±

.0062

.9587

±

.0048

.9545

±

.0035

.9709

±

.0015

.9728

±

.0028

.9719

±

.0046

L=4

Standard

.9721

±

.0035

.9652

±

.0023

.9605

±

.0031

.9741

±

.0028

.9686

±

.0030

.9674

±

.0027

.9740

±

.0037

.9738

±

.0027

.9712

±

.0047

DropEdge

.9666

±

.0031

.9572

±

.0034

.9469

±

.0043

.9659

±

.0028

.9598

±

.0024

.9496

±

.0040

.9749

±

.0024

.9747

±

.0016

.9718

±

.0031

NeuralSparse

.9687

±

.0027

.9603

±

.0026

.9536

±

.0033

.9699

±

.0025

.9600

±

.0036

.9502

±

.0037

.9748

±

.0022

.9756

±

.0022

.9727

±

.0035

PTDNet

.9707

±

.0041

.9640

±

.0031

.9549

±

.0034

.9708

±

.0027

.9602

±

.0025

.9548

±

.0058

.9754

±

.0025

.9761

±

.0021

.9709

±

.0041

Co-teaching

.9726

±

.0096

.9648

±

.0030

.9639

±

.0088

.9551

±

.0037

.9503

±

.0027

.9566

±

.0028

.9691

±

.0039

.9664

±

.7565

.9622

±

.0242

Peer loss

.9745

±

.0092

.9714

±

.0088

.9668

±

.0103

.9637

±

.0041

.9602

±

.0088

.9658

±

.0064

.9549

±

.0073

.9531

±

.0096

.9587

±

.0068

.9776

±

.0089

.9692

±

.0098

.9663

±

.0112

.9729

±

.0024

.9677

±

.0214

.9744

±

.0012

.9587

±

.0078

.9519

±

.0092

.9573

±

.0137

GIB

.9600

±

.0148

.9646

±

.0041

.9641

±

.0089

.9639

±

.0197

.9642

±

.0189

.9636

±

.0047

.9646

±

.0044

.9624

±

.0103

.9609

±

.0064

SupCon

.9644

±

.0033

.9537

±

.0106

.9381

±

.0125

.9675

±

.0028

.9631

±

.0042

.9555

±

.0068

.9594

±

.0131

.9442

±

.0136

.9386

±

.0102

GRACE

.5429

±

.0465

.6085

±

.1132

.6273

±

.1475

.6178

±

.0940

.5839

±

.1022

.6513

±

.1544

.5534

±

.0439

.6195

±

.1055

.6333

±

.0818

RGIB-REP

.9704

±

.0035

.9645

±

.0027

.9591

±

.0036

.9713

±

.0023

.9654

±

.0046

.9572

±

.0047

.9741

±

.0040

.9745

±

.0028

.9718

±

.0039

RGIB-SSL

.9507

±

.0034

.9432

±

.0034

.9389

±

.0059

.9526

±

.0037

.9451

±

.0045

.9360

±

.0083

.9619

±

.0037

.9645

±

.0042

.9643

±

.0038

L=6

Standard

.9659

±

.0029

.9573

±

.0036

.9482

±

.0054

.9644

±

.0033

.9543

±

.0075

.9474

±

.0074

.9722

±

.0043

.9688

±

.0055

.9698

±

.0065

DropEdge

.9461

±

.0082

.9295

±

.0063

.9062

±

.0218

.9469

±

.0059

.9293

±

.0087

.9182

±

.0063

.9698

±

.0036

.9689

±

.0046

.9694

±

.0037

NeuralSparse

.9563

±

.0079

.9391

±

.0057

.9185

±

.0157

.9473

±

.0149

.9338

±

.0043

.9250

±

.0037

.9688

±

.0037

.9705

±

.0026

.9696

±

.0063

PTDNet

.9508

±

.0075

.9359

±

.0222

.9288

±

.0153

.9535

±

.0082

.9424

±

.0053

.9332

±

.0043

.9723

±

.0033

.9717

±

.0036

.9697

±

.0046

Co-teaching

.9699

±

.0033

.9627

±

.0129

.9501

±

.0141

.9709

±

.0129

.9569

±

.0115

.9482

±

.0155

.9673

±

.0285

.9698

±

.0260

.9631

±

.0176

Peer loss

.9736

±

.0116

.9570

±

.0133

.9485

±

.0076

.9659

±

.0060

.9555

±

.0157

.9469

±

.0086

.9704

±

.0136

.9701

±

.0196

.9682

±

.0078

.9680

±

.0057

.9607

±

.0055

.9511

±

.0106

.9815

±

.0185

.9563

±

.0145

.9468

±

.0100

.9750

±

.0124

.9732

±

.0093

.9772

±

.0120

GIB

.9652

±

.0138

.9666

±

.0113

.9612

±

.0165

.9776

±

.0272

.9610

±

.0322

.9624

±

.0365

.9587

±

.0079

.9514

±

.0078

.9585

±

.0138

SupCon

.9481

±

.0185

.9310

±

.0171

.9015

±

.0169

.9455

±

.0267

.9490

±

.0157

.9194

±

.0239

.9413

±

.0206

.9370

±

.0174

.9241

±

.0144

GRACE

.5681

±

.0785

.5107

±

.0287

.5147

±

.0361

.5599

±

.0776

.5046

±

.0124

.5167

±

.0502

.5850

±

.0827

.5864

±

.0795

.5518

±

.0404

RGIB-REP

.9612

±

.0042

.9410

±

.0069

.9258

±

.0169

.9531

±

.0054

.9417

±

.0058

.9313

±

.0092

.9716

±

.0049

.9747

±

.0036

.9707

±

.0056

RGIB-SSL

.9367

±

.0058

.9233

±

.0046

.9199

±

.0053

.9418

±

.0085

.9236

±

.0056

.9205

±

.0050

.9576

±

.0066

.9562

±

.0079

.9557

±

.0083

GCN

layers

method

bilateral noise

input noise

label noise

20%

40%

60%

20%

40%

60%

20%

40%

60%

L=2

Standard

.9621

±

.0018

.9519

±

.0020

.9444

±

.0024

.9610

±

.0028

.9490

±

.0031

.9401

±

.0036

.9744

±

.0013

.9731

±

.0010

.9722

±

.0011

DropEdge

.9571

±

.0015

.9480

±

.0032

.9370

±

.0028

.9587

±

.0011

.9486

±

.0026

.9349

±

.0026

.9623

±

.0015

.9612

±

.0008

.9604

±

.0014

NeuralSparse

.9611

±

.0019

.9528

±

.0027

.9409

±

.0023

.9621

±

.0017

.9495

±

.0016

.9364

±

.0022

.9741

±

.0013

.9724

±

.0018

.9722

±

.0010

PTDNet

.9612

±

.0018

.9513

±

.0026

.9424

±

.0024

.9619

±

.0016

.9488

±

.0035

.9363

±

.0022

.9744

±

.0011

.9723

±

.0015

.9718

±

.0009

Co-teaching

.9701

±

.0076

.9535

±

.0044

.9465

±

.0059

.9599

±

.0097

.9525

±

.0120

.9454

±

.0109

.9629

±

.0047

.9666

±

.0304

.9631

±

.0004

Peer loss

.9610

±

.0015

.9608

±

.0057

.9451

±

.0097

.9636

±

.0107

.9491

±

.0125

.9436

±

.0091

.9694

±

.0012

.9628

±

.0134

.9676

±

.0149

.9613

±

.0027

.9568

±

.0046

.9509

±

.0014

.9622

±

.0159

.9594

±

.0111

.9397

±

.0119

.9528

±

.0017

.9405

±

.0089

.9401

±

.0010

GIB

.9522

±

.0195

.9478

±

.0094

.9489

±

.0159

.9494

±

.0260

.9461

±

.0091

.9432

±

.0068

.9457

±

.0099

.9462

±

.0017

.9492

±

.0060

SupCon

.9494

±

.0013

.9439

±

.0009

.9458

±

.0024

.9421

±

.0008

.9549

±

.0013

.9489

±

.0030

.9444

±

.0011

.9423

±

.0015

.9318

±

.0009

GRACE

.9292

±

.0011

.9316

±

.0010

.9330

±

.0013

.9290

±

.0012

.9317

±

.0010

.9327

±

.0016

.9054

±

.0018

.9058

±

.0015

.9052

±

.0018

RGIB-REP

.9614

±

.0019

.9554

±

.0026

.9471

±

.0035

.9611

±

.0021

.9512

±

.0032

.9392

±

.0043

.9761

±

.0015

.9753

±

.0011

.9745

±

.0015

RGIB-SSL

.9571

±

.0019

.9476

±

.0029

.9402

±

.0020

.9534

±

.0016

.9486

±

.0027

.9451

±

.0031

.9752

±

.0023

.9749

±

.0019

.9738

±

.0023

L=4

Standard

.9432

±

.0036

.9406

±

.0031

.9386

±

.0025

.9416

±

.0042

.9395

±

.0011

.9411

±

.0040

.9720

±

.0016

.9720

±

.0016

.9710

±

.0021

DropEdge

.9439

±

.0063

.9377

±

.0024

.9365

±

.0045

.9426

±

.0042

.9376

±

.0033

.9358

±

.0023

.9608

±

.0008

.9603

±

.0013

.9698

±

.0016

NeuralSparse

.9494

±

.0065

.9309

±

.0028

.9297

±

.0033

.9469

±

.0047

.9403

±

.0028

.9417

±

.0062

.9633

±

.0009

.9626

±

.0016

.9625

±

.0016

PTDNet

.9485

±

.0056

.9326

±

.0046

.9304

±

.0048

.9469

±

.0048

.9400

±

.0030

.9379

±

.0026

.9633

±

.0018

.9623

±

.0012

.9626

±

.0012

Co-teaching

.9461

±

.0099

.9352

±

.0064

.9374

±

.0051

.9462

±

.0101

.9425

±

.0032

.9306

±

.0053

.9675

±

.0292

.9641

±

.0115

.9655

±

.0264

Peer loss

.9457

±

.0117

.9345

±

.0117

.9286

±

.0044

.9362

±

.0038

.9386

±

.0058

.9336

±

.0043

.9636

±

.0012

.9694

±

.0033

.9696

±

.0006

.9443

±

.0047

.9327

±

.0079

.9244

±

.0054

.9388

±

.0179

.9345

±

.0146

.9240

±

.0055

.9529

±

.0029

.9512

±

.0089

.9501

±

.0120

GIB

.9472

±

.0127

.9329

±

.0155

.9302

±

.0202

.9390

±

.0045

.9406

±

.0196

.9397

±

.0249

.9641

±

.0025

.9628

±

.0032

.9601

±

.0042

SupCon

.9473

±

.0010

.9348

±

.0017

.9301

±

.0029

.9372

±

.0010

.9343

±

.0037

.9305

±

.0033

.9516

±

.0042

.9595

±

.0011

.9511

±

.0040

GRACE

.9394

±

.0008

.9380

±

.0007

.9363

±

.0016

.9392

±

.0011

.9378

±

.0008

.9363

±

.0017

.9171

±

.0019

.9174

±

.0014

.9166

±

.0013

RGIB-REP

.9509

±

.0046

.9455

±

.0071

.9434

±

.0059

.9495

±

.0053

.9432

±

.0059

.9405

±

.0050

.9735

±

.0013

.9733

±

.0015

.9737

±

.0009

RGIB-SSL

.9499

±

.0040

.9426

±

.0019

.9425

±

.0021

.9479

±

.0051

.9429

±

.0017

.9429

±

.0024

.9727

±

.0015

.9729

±

.0019

.9726

±

.0011

L=6

Standard

.9484

±

.0049

.9429

±

.0038

.9408

±

.0039

.9489

±

.0057

.9408

±

.0021

.9386

±

.0022

.9688

±

.0028

.9675

±

.0027

.9656

±

.0034

DropEdge

.9460

±

.0063

.9375

±

.0028

.9349

±

.0017

.9474

±

.0057

.9409

±

.0030

.9365

±

.0022

.9669

±

.0022

.9652

±

.0031

.9638

±

.0038

NeuralSparse

.9508

±

.0049

.9429

±

.0047

.9391

±

.0029

.9532

±

.0060

.9412

±

.0044

.9388

±

.0036

.9681

±

.0020

.9647

±

.0018

.9671

±

.0019

PTDNet

.9535

±

.0047

.9451

±

.0050

.9395

±

.0033

.9514

±

.0041

.9432

±

.0068

.9380

±

.0020

.9690

±

.0023

.9667

±

.0035

.9649

±

.0025

Co-teaching

.9480

±

.0144

.9454

±

.0074

.9463

±

.0076

.9538

±

.0051

.9424

±

.0048

.9379

±

.0012

.9672

±

.0179

.9694

±

.0283

.9654

±

.0195

Peer loss

.9539

±

.0051

.9420

±

.0127

.9405

±

.0060

.9445

±

.0139

.9446

±

.0012

.9462

±

.0081

.9684

±

.0182

.9639

±

.0188

.9624

±

.0189

.9519

±

.0138

.9459

±

.0100

.9398

±

.0115

.9483

±

.0070

.9429

±

.0203

.9428

±

.0105

.9697

±

.0074

.9687

±

.0062

.9673

±

.0077

GIB

.9447

±

.0103

.9459

±

.0026

.9410

±

.0108

.9431

±

.0289

.9413

±

.0301

.9386

±

.0238

.9633

±

.0041

.9619

±

.0114

.9612

±

.0064

SupCon

.9590

±

.0023

.9507

±

.0019

.9463

±

.0028

.9580

±

.0024

.9519

±

.0018

.9464

±

.0026

.9638

±

.0051

.9524

±

.0048

.9419

±

.0012

GRACE

.9399

±

.0007

.9372

±

.0006

.9348

±

.0017

.9397

±

.0012

.9370

±

.0009

.9349

±

.0017

.9213

±

.0017

.9218

±

.0013

.9211

±

.0014

RGIB-REP

.9556

±

.0028

.9492

±

.0060

.9463

±

.0063

.9570

±

.0018

.9496

±

.0060

.9420

±

.0056

.9677

±

.0029

.9676

±

.0031

.9662

±

.0018

RGIB-SSL

.9462

±

.0029

.9471

±

.0020

.9448

±

.0032

.9509

±

.0032

.9474

±

.0017

.9449

±

.0030

.9642

±

.0012

.9659

±

.0027

.9647

±

.0010

SAGE

layers

method

bilateral noise

input noise

label noise

20%

40%

60%

20%

40%

60%

20%

40%

60%

L=2

Standard

.9680

±

.0015

.9626

±

.0015

.9570

±

.0016

.9737

±

.0010

.9736

±

.0010

.9721

±

.0011

.9692

±

.0015

.9643

±

.0012

.9606

±

.0017

DropEdge

.9657

±

.0011

.9602

±

.0015

.9535

±

.0020

.9622

±

.0010

.9625

±

.0009

.9609

±

.0020

.9593

±

.0009

.9530

±

.0009

.9593

±

.0007

NeuralSparse

.9572

±

.0008

.9517

±

.0012

.9562

±

.0019

.9534

±

.0007

.9534

±

.0010

.9521

±

.0016

.9589

±

.0012

.9534

±

.0009

.9591

±

.0010

PTDNet

.9578

±

.0009

.9513

±

.0013

.9559

±

.0021

.9630

±

.0010

.9504

±

.0007

.9524

±

.0010

.9596

±

.0014

.9529

±

.0010

.9596

±

.0019

Co-teaching

.9526

±

.0031

.9560

±

.0071

.9526

±

.0029

.9403

±

.0050

.9585

±

.0061

.9585

±

.0101

.9533

±

.0022

.9533

±

.0306

.9539

±

.0026

Peer loss

.9552

±

.0079

.9507

±

.0050

.9588

±

.0066

.9512

±

.0052

.9534

±

.0019

.9599

±

.0013

.9525

±

.0105

.9555

±

.0178

.9510

±

.0187

.9658

±

.0111

.9619

±

.0068

.9630

±

.0027

.9600

±

.0189

.9649

±

.0173

.9617

±

.0171

.9620

±

.0105

.9608

±

.0068

.9603

±

.0044

GIB

.9656

±

.0079

.9660

±

.0077

.9661

±

.0060

.9632

±

.0260

.9517

±

.0213

.9565

±

.0010

.9565

±

.0009

.9593

±

.0002

.9548

±

.0111

SupCon

.9628

±

.0015

.9541

±

.0019

.9409

±

.0040

.9585

±

.0061

.9585

±

.0101

.9533

±

.0022

.9533

±

.0306

.9539

±

.0026

.9400

±

.0021

GRACE

.8543

±

.0076

.8572

±

.0094

.8528

±

.0166

.8563

±

.0068

.8601

±

.0087

.8528

±

.0075

.8477

±

.0119

.8459

±

.0105

.8452

±

.0141

RGIB-REP

.9701

±

.0010

.9647

±

.0014

.9597

±

.0018

.9757

±

.0009

.9747

±

.0007

.9737

±

.0011

.9735

±

.0010

.9707

±

.0007

.9682

±

.0016

RGIB-SSL

.9536

±

.0016

.9480

±

.0027

.9448

±

.0030

.9734

±

.0014

.9739

±

.0015

.9803

±

.0024

.9765

±

.0021

.9702

±

.0015

.9694

±

.0005

L=4

Standard

.9584

±

.0107

.9577

±

.0076

.9541

±

.0021

.9637

±

.0092

.9630

±

.0079

.9607

±

.0090

.9663

±

.0020

.9612

±

.0049

.9612

±

.0020

DropEdge

.9518

±

.0030

.9577

±

.0021

.9515

±

.0033

.9656

±

.0051

.9612

±

.0093

.9573

±

.0111

.9683

±

.0020

.9628

±

.0014

.9608

±

.0012

NeuralSparse

.9604

±

.0083

.9570

±

.0040

.9513

±

.0040

.9644

±

.0066

.9599

±

.0084

.9605

±

.0061

.9652

±

.0035

.9632

±

.0023

.9605

±

.0021

PTDNet

.9624

±

.0050

.9581

±

.0039

.9536

±

.0033

.9667

±

.0045

.9668

±

.0057

.9649

±

.0052

.9675

±

.0043

.9635

±

.0040

.9607

±

.0026

Co-teaching

.9518

±

.0181

.9468

±

.0093

.9479

±

.0024

.9525

±

.0086

.9577

±

.0159

.9528

±

.0137

.9625

±

.0244

.9610

±

.0220

.9675

±

.0097

Peer loss

.9632

±

.0196

.9662

±

.0085

.9602

±

.0088

.9674

±

.0145

.9626

±

.0120

.9643

±

.0145

.9637

±

.0004

.9617

±

.0170

.9691

±

.0016

.9622

±

.0135

.9606

±

.0157

.9587

±

.0105

.9647

±

.0084

.9630

±

.0253

.9640

±

.0278

.9505

±

.0026

.9508

±

.0063

.9599

±

.0039

GIB

.9527

±

.0122

.9538

±

.0275

.9531

±

.0174

.9603

±

.0081

.9620

±

.0115

.9580

±

.0330

.9598

±

.0031

.9523

±

.0064

.9559

±

.0091

SupCon

.9478

±

.0042

.9412

±

.0059

.9342

±

.0039

.9267

±

.0045

.9368

±

.0028

.9249

±

.0026

.9375

±

.0034

.9335

±

.0012

.9317

±

.0021

GRACE

.8859

±

.0115

.8751

±

.0085

.8609

±

.0212

.8824

±

.0113

.8805

±

.0261

.8633

±

.0178

.8386

±

.0130

.8466

±

.0103

.8347

±

.0180

RGIB-REP

.9611

±

.0058

.9578

±

.0035

.9536

±

.0037

.9682

±

.0049

.9667

±

.0051

.9584

±

.0074

.9689

±

.0014

.9664

±

.0027

.9635

±

.0022

RGIB-SSL

.9498

±

.0032

.9443

±

.0047

.9422

±

.0044

.9531

±

.0071

.9599

±

.0023

.9551

±

.0019

.9623

±

.0021

.9603

±

.0087

.9601

±

.0091

L=6

Standard

.9555

±

.0065

.9528

±

.0038

.9461

±

.0054

.9592

±

.0053

.9600

±

.0036

.9551

±

.0042

.9574

±

.0192

.9540

±

.0207

.9583

±

.0023

DropEdge

.9544

±

.0069

.9411

±

.0200

.9463

±

.0041

.9566

±

.0070

.9473

±

.0172

.9519

±

.0052

.9464

±

.0023

.9402

±

.0084

.9396

±

.0025

NeuralSparse

.9563

±

.0060

.9515

±

.0036

.9457

±

.0054

.9591

±

.0046

.9460

±

.0052

.9415

±

.0061

.9371

±

.0025

.9344

±

.0017

.9319

±

.0033

PTDNet

.9565

±

.0060

.9524

±

.0050

.9463

±

.0056

.9593

±

.0045

.9584

±

.0044

.9492

±

.0038

.9384

±

.0023

.9342

±

.0018

.9236

±

.0018

Co-teaching

.9423

±

.0071

.9396

±

.0126

.9251

±

.0137

.9382

±

.0063

.9433

±

.0054

.9388

±

.0088

.9344

±

.0232

.9496

±

.0204

.9374

±

.0083

Peer loss

.9473

±

.0108

.9371

±

.0040

.9333

±

.0084

.9339

±

.0053

.9312

±

.0058

.9331

±

.0066

.9396

±

.0211

.9375

±

.0195

.9335

±

.0035

.9335

±

.0058

.9305

±

.0057

.9225

±

.0146

.9309

±

.0059

.9386

±

.0084

.9433

±

.0147

.9348

±

.0200

.9210

±

.0285

.9273

±

.0029

GIB

.9437

±

.0258

.9323

±

.0155

.9204

±

.0182

.9492

±

.0239

.9406

±

.0137

.9322

±

.0335

.9407

±

.0269

.9364

±

.0274

.9394

±

.0079

SupCon

.9392

±

.0049

.9330

±

.0098

.9221

±

.0138

.9266

±

.0024

.9313

±

.0102

.9309

±

.0052

.9264

±

.0021

.9202

±

.0084

.9196

±

.0021

GRACE

.8418

±

.0983

.8470

±

.0823

.8732

±

.0151

.8747

±

.0151

.8126

±

.1348

.8716

±

.0148

.8385

±

.0097

.8370

±

.0295

.8487

±

.0121

RGIB-REP

.9558

±

.0047

.9474

±

.0066

.9455

±

.0052

.9583

±

.0044

.9566

±

.0035

.9447

±

.0030

.9662

±

.0057

.9646

±

.0021

.9634

±

.0030

RGIB-SSL

.9437

±

.0026

.9391

±

.0036

.9366

±

.0042

.9441

±

.0064

.9549

±

.0012

.9469

±

.0012

.9678

±

.0031

.9613

±

.0076

.9601

±

.0041

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the robustness of graph neural networks (GNNs) in the presence of edge noise during link prediction on graphs. The authors empirically investigate the impact of edge noise on both the input topology and target labels, revealing significant performance degradation and representation collapse. To address this challenge, they propose a novel principle called Robust Graph Information Bottleneck (RGIB), which leverages information theory to extract reliable supervision signals and prevent representation collapse. The paper explores two instantiations of RGIB, RGIB-SSL and RGIB-REP, which utilize self-supervised learning and data reparametrization techniques, respectively, for implicit and explicit data denoising.\n\nSTRENGTHS:\nOverall, this paper makes a valuable contribution to understanding and enhancing the robustness of GNNs in link prediction tasks affected by edge noise.\n\nThe main novelty is based on systematic study of Bilateral Edge Noise including its empirical influence, visualization, and two instantiations of Robust Graph Information Bottleneck with data reparameterization and data augmentation. \n\nThis paper also provides theoretical analysis for noisy dataset convergence relaxations.\n\nWEAKNESSES:\nThis paper provides a novel method and analysis for the edge noise issue for the GNN, through the lens of information theory. The method is clearly explained, and the performance is supported with grounded evaluations with visualizations. I did not carefully check the theory proofs, but from the main paper, there is no obvious limitation.\n\nQUESTIONS:\nIf the SSL and REP versions are parallel methods, could the authors discuss the guidelines on method selection? Are they be simultaneously used while still improve? It would be better to add a section of such discussions with supportive case evidence.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

comments: SUMMARY:
This paper proposes to tackle the bilateral edge noise via mutual information. The authors start from empirical observations that existing GNNs are vulnerable to bilateral edge noises. To tackle this issue, the authors propose a robust graph information bottleneck which is information-theory guided. In practice, a self-supervised regularization and a purification mechanism are proposed. The extensive evaluation on various datasets as well as models demonstrates the effectiveness of proposed algorithm.

STRENGTHS:
1. The paper is well-written and well-organized. 
2. The introduced bilateral edge noise is an interesting and challenging problem in GNNs.
3. The proposed robust graph information bottleneck is well-motivated by both sufficient empirical observations as well as theoretical analysis.
4. The experiments are extensive, including various datasets and GNNs. Various ablation studies demonstrate the effectiveness of RGIB-SSL and RGIB-REP.

WEAKNESSES:
I have several concerns:
1. In RGIB-SSL, the authors introduce hybrid graph augmentation which shows superiority over other contrastive learning methods. However, how the proposed augmentation outperforms other graph contrastive methods needs more clarification to provide some insights of this design.
2. It is recommended to demonstrate the robustness of the proposed algorithm under attacks, such as [4, 8, 46].

QUESTIONS:
1. Please clarify the motivation of hybrid graph augmentation and the highlight the contribution compared to those in [18, 47].
2. Please include the evaluation under attacks [4, 8, 46] if possible, or discuss the generalization of the proposed algorithm against adversarial attacks.

LIMITATIONS:
The authors have not discussed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-30

comments: SUMMARY:
This paper focuses on the robustness of GNNs under the edge noise. The authors disclose the influence of bilateral edge noise and the corresponding robustness issue via a series of empirical studies on edge noise. Based on the observations of bilateral noise, the authors propose an information-theory-guided principle, Robust Graph Information Bottleneck (RGIB) and its two instantiations, RGIB-SSL and RGIB-REP. Experimental results verify the effectiveness of RGIB instantiations.

STRENGTHS:
- The paper is well written and structured.
- The emprical studies and relative experiments are clear and convincing.
- Experiments are comprehensive and complete.
- The proposed RGIB principle is simple yet effective.

WEAKNESSES:
- There is a lack of more comprehensive studies on other graph representation learning tasks, such as node classification.
- RGIB is proposed under the assumption of edge noise, however the motivation statement is not very convincing. The relationship between GIB and bilateral noise should be further explained.

QUESTIONS:
- I wonder how RGIB improves other graph representation learning tasks, like node classification?
- Why the model performance on original graphs are not reported? Since the graphs without additional noises are not considered as completely clean, RGIB should be able to provide performance gain as well.

LIMITATIONS:
Please refer to the weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
The authors extend the Graph Information Bottleneck (GIB) to "bilateral" structural noise and label noise. That is, both the adjacency matrix and the labels are being randomly perturbed. The authors observe that the bilateral noise leads to "poorer alignment and a worse uniformity". To handle this noise, the authors decompose the term used in GIB and, due to its intractability, propose two efficient, practical instantiations.

STRENGTHS:
1. The authors propose two methods that significantly and consistently outperform the baselines
1. The approach is principled due to its roots in the powerful concept of information bottleneck
1. The experiments are extensive and are convincing that the method has some merit for certain applications

The paper is generally well-written and logically structured.

WEAKNESSES:
1. The method seems to rely heavily on the assumption that the node features are clean. It would be good to study how sensitive the model is to feature noise.
1. The authors could elaborate more on the assumptions etc., that make their instantiations tractable and when these assumptions are met.

Minor: margins between lines 215 & 216 seem violated.

QUESTIONS:
1. Would it make sense / be possible also to evaluate the method using an attack that optimizes for $\tilde{A}$ or $\tilde{Y}$, e.g., using first-order optimization?
1. As you do not provide code, I wonder if the clean labels/adjacency are used in any way during training?

LIMITATIONS:
The authors should elaborate more and more prominently on what the computational requirements of their method are and how it compares to the other baselines.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This paper tackles the challenge of link prediction on graphs in the presence of edge noise, a topic that has seen little exploration despite the advancements in graph neural networks (GNNs). Through an empirical study, the authors reveal that edge noise can adversely affect both input topology and target labels, leading to performance degradation and representation collapse. In response, the paper introduces an information-theory-guided principle named Robust Graph Information Bottleneck (RGIB), which aims to extract reliable supervision signals and prevent representation collapse. RGIB achieves this by decoupling and balancing the mutual dependencies among graph topology, target labels, and representation, which creates new learning objectives for building robust representations in the face of bilateral noise. The authors present two specific implementations of RGIB, namely RGIB-SSL (which employs self-supervised learning) and RGIB-REP (which uses data reparametrization), for implicit and explicit data denoising respectively. The effectiveness of the proposed RGIB methods is validated through extensive experiments on six datasets and three GNNs under various noisy conditions.

STRENGTHS:
The paper is well-written with clear motivation and structure. Overall interesting problem; good mathematical exposition; solid theoretical results and analysis. This paper has a very comprehensive experimental analysis including the experiments in the appendix.

WEAKNESSES:
This paper is overall good.

QUESTIONS:
Most of my concerns are addressed in the appendix. I have no further questions or suggestions for this work.

LIMITATIONS:
Board impact is discussed in the appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07