SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we provide additional experimental results to further showcase the effectiveness of the RES in enhancing the robustness of GCL against various adversarial attacks. More specifically,
(a) GRACE, OGB-arxiv
(b) DGI, OGB-arxiv
(c) GraphCL, OGB-molhiv
(d) BGRL, OGB-molhiv,
Figure 7: Certified accuracy of smoothed GCL on OGB-arxiv and OGB-molhiv
(a) GraphCL, MUTAG
Figure 8: Certified accuracy of smoothed GCL on MUTAG and PROTEINS
(b) BGRL, MUTAG
(c) GraphCL, PROTEINS
(d) BGRL, PROTEINS
in Sec. I.1, we show the comparison results of RES with the baselines on graph classification. In Sec. I.2, we conduct experiments to demonstrate GCL with RES is resistant to different levels of structural noises. In Sec. I.3, we present the comparison results of three target GCL methods (i.e., GRACE, BGRL, DGI) against three types of structural attacks (i.e., Random, CLGA and PRBCD) on node classification. In Sec. I.4, we present additional experimental results of two advanced GCL methods (i.e., ADGCL [58] and RGCL [66]) on graph classification.
I.1 Robust Performance on Graph Classification
In this subsection, we conduct experiments on graph classification to demonstrate the effectiveness of our method in this downstream task. Due to the limited availability of open-source attack methods specifically designed for graph classification, we utilize random attacks as the attack method. However, we believe that our method is robust against other attack methods as well. Specifically, we select GraphCL as the target GCL methods. The perturbation rate of random attack is 0.1. The smoothed version of GraphCL is denoted as RES-GraphCL. The results on MUTAG and PROTEINS are given in Fig. 9. From the figure, we observe: (i) When no attack is applied to the raw graphs, RES-GraphCL achieves comparable performance to the baseline GraphCL. (ii) When attacks are conducted on the noisy graphs, RES-GraphCL consistently outperforms the baseline on both the MUTAG and PROTEINS datasets. This result demonstrates the effectiveness of our method in enhancing the robustness of GraphCL against adversarial attacks.
(a) Clean
Figure 9: Robust accuracy of GraphCL for graph classification against random attack
(b) Noisy
I.2 Robustness Under Different Noisy Levels.
To demonstrate the ability of our method to improve the robustness of Graph Contrastive Learning (GCL) against different levels of structural noise, we compare the robust accuracy of the GCL methods w/o applying our RES under evasion attacks for node classification. Specifically, we select GRACE, BGRL and DGI as target GCL methods. We set  $(1 - \beta) = 0.05$  and  $\mu = 50$ . We consider two targeted attack methods, random attack and Nettack [11] to conduct targeted attacks. The attack budget is set from 0 to 5. We randomly select  $15\%$  of the test nodes as the target nodes to compute the robust accuracy. The results on the Cora and Pubmed datasets are reported in Fig. 10 and Fig. 11. We observe: (i) The robust accuracies of the baseline methods exhibit a significant drop as the
(a) Cora, GRACE
(b) Cora, BGRL
(c) Cora, DGI
(d) Pubmed, GRACE
(e) Pubmed, BGRL
(f) Pubmed, DGI
Figure 10: Robust accuracy under different perturbation sizes of random attack
(a) Cora, GRACE
(d) Pubmed, GRACE
Figure 11: Robust accuracy under different perturbation sizes of Nettack
perturbation sizes increase, which is expected. In contrast, the performances of RES are much more stable and consistently outperform the baselines. This demonstrates the robustness of GCL with RES against various levels of structural noise. (ii) The high robust accuracies of the three GCL models demonstrate that our RES is effectively applicable to various GCL method.
I.3 Additional Results of Robust Performance on Node Classification
In this subsection, we provide additional experimental results from Section 6.2, focusing on node classification. We select GRACE, BGRL, and DGI as the target GCL methods and evaluate their performance on four types of graphs: raw graphs, random attack perturbed graphs, CLGA perturbed graphs, and PRBCD perturbed graphs. The perturbation ratio is set to 0.1. Table 3 presents the results of these experiments. The highlighted results denote the best performance for each pair of GCL and RES-GCL. Note that BGRL, implemented based on PyGCL [65], encounters out-of-memory (OOM) errors in our platform, and hence the results for BGRL on OGB-arxiv are left blank. From
the table, we observe that all three GCL methods, when combined with RES, achieve state-of-the-art performance. This demonstrates the effectiveness of our method in enhancing the robustness of various GCL methods. By incorporating RES, the GCL models are more resilient to adversarial attacks and exhibit improved performance across different types of perturbed graphs.
Moreover, we further consider two graph injection attack methods, i.e., TDGIA [60] and AGIA [61], as baselines to demonstrate the robustness of RES. Specifically, we select GRACE as the target GCL methods and evaluate them on three types of graphs: raw graphs, TDGIA perturbed graphs and AGIA perturbed graphs. We insert the same number of fake nodes as the target nodes. We set  $(1 - \beta) = 0.1$  and  $\mu = 50$ . The comparison results on four datasets are shown in Table 4. From the results, we observe that (i) RES-GRACE consistently outperforms the baselines across 4 datasets in defending graph injection attacks. (ii) Both TDGIA and AGIA are much more powerful than the structural attack methods in Table 3 against GCL.
Table 3: Robust accuracy results of GCL methods for node classification.
Dataset	Graph	GRACE	RES-GRACE	BGRL	RES-BGRL	DGI	RES-DGI
Cora	Raw	77.1±1.6	79.7±1.0	78.5±1.6	79.9±1.2	81.3±0.7	81.4±0.8
Random	74.5±2.1	79.7±1.0	76.2±1.2	79.6±1.1	77.5±1.0	79.2±0.6
CLGA	74.9±2.0	78.2±1.0	75.8±1.6	79.4±0.9	79.5±0.5	80.9±1.0
PRBCD	75.8±2.5	78.5±1.7	76.4±0.6	79.5±0.8	79.6±1.4	80.9±1.1
Pubmed	Raw	79.5±2.9	79.5±1.2	79.9±1.4	81.5±0.6	80.1±0.9	80.0±0.8
Random	75.0±1.0	78.2±0.9	74.0±1.0	81.0±0.7	76.7±0.7	78.8±0.6
CLGA	76.6±2.5	78.3±1.1	77.9±0.3	81.6±0.2	79.6±0.6	80.0±1.2
PRBCD	73.2±2.3	78.8±1.7	71.6±2.4	80.9±0.5	75.4±0.9	78.0±0.6
Physics	Raw	94.0±0.4	94.7±0.2	95.3±0.1	95.6±0.1	93.5±0.6	94.1±0.3
Random	92.6±0.5	94.2±0.3	94.0±0.2	95.5±0.2	91.5±0.9	92.2±0.4
PRBCD	89.2±0.6	94.1±0.2	92.3±0.2	95.4±0.2	88.8±0.5	90.0±0.4
OGB-arxiv	Raw	65.1±0.5	65.2±0.1	-	-	65.0±0.2	64.8±0.1
Random	59.0±0.2	60.0±0.1	-	-	58.0±0.1	58.9±0.1
PRBCD	55.7±0.4	58.3±0.4	-	-	56.9±0.9	57.2±0.3
Table 4: Robust accuracy results of GCL methods against injection attacks.
Dataset	Graph	GRACE	RES-GRACE
Cora	Raw	77.1±1.6	79.7±1.0
TDGIA	22.4±1.5	78.7±1.1
AGIA	21.1±1.4	78.4±0.9
Pubmed	Raw	79.5±3.1	79.5±1.2
TDGIA	50.6±1.2	77.0±0.4
AGIA	50.2±1.6	77.5±1.0
Physics	Raw	94.0±0.4	94.7±0.2
TDGIA	52.2±1.0	93.8±0.1
AGIA	52.3±0.5	94.0±0.2
OGB-arxiv	Raw	65.1±0.5	65.1±0.1
TDGIA	40.3±0.4	46.9±0.1
AGIA	40.4±0.3	47.0±0.1
I.4 Additional Results of RES for Advanced GCLs
(a) ADGCL, Clean
(b) ADGCL, Noisy
Figure 12: Robust accuracy of ADGCL and RGCL for graph classification against random attack
(c) RGCL, Clean
(d) RGCL, Noisy
To further demonstrate the effectiveness of RES, we further add ADGCL [58] and RGCL [66] as baselines and implement RES-ADGCL and RES-RGCL. We set graph classification as the downstream task. The hyperparameter is tuned based on the performance of the validation set. We use random attack to get the noisy graphs and the perturbation rate is 0.1. Each experiment is conducted 5 times and the average results are reported. Comparison results on MUTAG and PROTEINS are
(a) MUTAG
(b) PROTEINS
Figure 13: Certified accuracy of RGCL on MUTAG and PROTEINS
shown in Fig. 12. From this figure, we observe that all RES-GCL methods achieve comparable performances to the baselines on raw graphs and consistently outperform the baselines in the noisy graphs of two datasets, which validates the effectiveness of RES in any GCL model.
We also report the certified accuracy of RES-RGCL on the two datasets. The results are shown in Fig. 13. From the figure, we can observe there is a tradeoff between certified robustness and model utility, which is similar to that of Sec. 6.3.
(a) Cora
Figure 14: Ablation Studies of Training with RES on Cora and OGB-arxiv
(b) OGB-arxiv
(a) Clean
Figure 15: Ablation Studies of Comparisons RES with FLIP
J Additional Results of Ablation Studies
J.1 Ablation Studies on Training with RES
In this subsection, we conduct ablation studies to investigate the effect of our training method for robust GCL. To demonstrate that our training method improves the robustness of GCL, we do not inject random edgedrop noise during the training phase and obtain a variant called NoRES. We select PRBCD as the attack method to generate noisy graphs, and set  $\mu = 50$ . We vary the value of  $\beta$  as  $\{0.5, 0.6, 0.7, 0.8, 0.9\}$  and compare the robust accuracy of RES and NoRES on both raw graphs and PRBCD-perturbed graphs. The results of the variant NoRES on raw and PRBCD-perturbed graphs are denoted as NoRES-Raw and NoRES-Ptb, respectively. We report the results on the Cora and OGB-arxiv datasets in Fig. 14. From the figure, we observe the following: (i) RES consistently outperforms NoRES on both raw and perturbed graphs in all settings, implying the effectiveness of our training method for robust GCL. (ii) The variance of RES is lower than that of NoRES. This is because we inject randomized edgedrop noise into the graphs during the training phase, helping the models better understand and generalize from the data with randomized edgedrop noise, thereby ensuring the robustness and utility of the models.
(a)  $\mu$
(b)  $(1 - \alpha)$
Figure 16: Hyperparameter Sensitivity Analysis on Cora
(a)  $\mu$
(b)  $(1 - \alpha)$
Figure 17: Hyperparameter Sensitivity Analysis on Coauthor-Physics
J.2 Additional Results of Comparisons RES with FLIP
In this subsection, we present additional experimental results in Sec. 6.4 to further demonstrate that our RES method is significantly more effective in GCL compared to vanilla randomized smoothing [24]. We introduce two variants of our model, namely FLIP and FLIP/T, which replace the randomized edgedrop noise with binary random noise [24]. This noise flips the connection status within the graph with a probability of  $\beta$ . The key difference between FLIP and FLIP/T is that FLIP injects binary random noise into the graphs during both the training and inference phases, while FLIP/T only injects binary random noise during the inference phase. For our RES method, we set  $\beta = 0.9$  and  $\mu = 50$ . For FLIP and FLIP/T, to ensure a fair comparison, we set  $\mu = 50$ , and vary  $\beta$  over  $\{0.1, 0.2, \dots, 0.9\}$  and select the value that yields the best performance on the validation set of clean graphs. We select GRACE as the targeted GCL method and compare the robust accuracy of our RES with FLIP and FLIP/T on the clean and noisy graph under Nettack with an attack budget of 3. The average robust accuracy and standard deviation on Cora and Pubmed datasets are reported in Fig. 15. We observe: (i) RES consistently outperforms FLIP and FLIP/T on clean and noisy graphs of Cora and Pubmed datasets, further validating the effectiveness of RES in providing certified robustness for GCL. (ii) FLIP and FLIP/T exhibit comparable performance on all graphs, but significantly lower than RES. This finding confirms our analysis that vanilla randomized smoothing introduces numerous spurious/noisy edges to the graph, resulting in poor representation learning by the GNN encoder and compromising downstream task performance.
K Hyperparameter Sensitivity Analysis
We further investigate how hyperparameter  $(1 - \alpha)$  and  $\mu$  affect the performance of robustness certificates of our RES, where  $(1 - \alpha)$  and  $\mu$  control the confidence level and the number of Monte Carlo samples used to compute the certified accuracy. We vary the value of  $(1 - \alpha)$  as  $\{90\%, 95\%, 99\%, 99.9\%, 99.99\% \}$  and fix  $\mu$  as 200, vary  $\mu$  as  $\{20, 50, 200, 500, 1000\}$  and fix  $(1 - \alpha)$  as  $99\%$ , respectively.  $\beta$  is set as 0.9. We report the certified accuracy of RES-GRACE on Cora and Coauthor-Physics dataset in Fig. 16 and Fig. 17. From the figures, we observe: (i) As  $\mu$  increases, the certified accuracy curve becomes higher. The reason for this is that a larger value of  $\mu$  makes the estimated probability bound  $\underline{p_{\mathbf{v}^{+},h}}(\mathbf{v} \oplus \epsilon)$  and  $\overline{p_{\mathbf{v}_i^{-},h}}(\mathbf{v} \oplus \epsilon)$  in Eq. (8) tighter, resulting in a higher certified perturbation size of a test sample. (ii) As  $(1 - \alpha)$  increases, the certified accuracy curve becomes slightly lower. This is because a higher confidence level leads to a looser estimation of  $\underline{p_{\mathbf{v}^{+},h}}(\mathbf{v} \oplus \epsilon)$  and  $\overline{p_{\mathbf{v}_i^{-},h}}(\mathbf{v} \oplus \epsilon)$  in Eq. (8), meaning that fewer nodes satisfy Theorem 2 under the same perturbation size, resulting in a smaller certified perturbation size of a test sample.
L Limitations
In this paper, we propose a unified criteria to evaluate and certify the robustness of GCL. Our proposed approach, Randomized Edgedrop Smoothing (RES), injects randomized edgedrop noise into graphs to provide certified robustness for GCL on unlabeled data. Moreover, we design an effective training method for robust GCL by incorporating randomized edgedrop noise during the training phase. The theoretical analysis and extensive experiments show the effectiveness of our proposed RES.
**Limitation & future work:** Our current results are limited mainly to GCL while we believe it is also interesting to develop new techniques to other graph self-supervised methods e.g. generative and neighborhood prediction methods based on our framework, which we leave for immediate future work. We hope that this work could inspire future certifiably defense algorithms of adversarial attacks. Additionally, in this paper, we only focus on the graph-structured data. Thus, it is also interesting to investigate how to extend it to other domains, e.g., images and texts. Furthermore, in this paper, we utilize Monte Carlo algorithms to calculate robustness certificates for GCL, potentially increasing the computational demands. Therefore, it is also worthwhile to investigate methods to improve the efficiency of the robustness certification for GCL. Due to the nature of this work, there may not be any potential negative social impact that is easily predictable.