SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Minhua Lin, Teng Xiao, Enyan Dai, Xiang Zhang, Suhang Wang  
The Pennsylvania State University  
{mf15681, tengxiao, emd5759, xzhang, szw494}@psu.edu
Graph Contrastive Learning (GCL) has emerged as a popular unsupervised graph representation learning method. However, it has been shown that GCL is vulnerable to adversarial attacks on both the graph structure and node attributes. Although empirical approaches have been proposed to enhance the robustness of GCL, the certifiable robustness of GCL is still remain unexplored. In this paper, we develop the first certifiably robust framework in GCL. Specifically, we first propose a unified criteria to evaluate and certify the robustness of GCL. We then introduce a novel technique, RES (Randomized Edgedrop Smoothing), to ensure certifiable robustness for any GCL model, and this certified robustness can be provably preserved in downstream tasks. Furthermore, an effective training method is proposed for robust GCL. Extensive experiments on real-world datasets demonstrate the effectiveness of our proposed method in providing effective certifiable robustness and enhancing the robustness of any GCL model. The source code of RES is available at https://github.com/ventr1c/RES-GCL.
Graph structured data are ubiquitous in real-world applications such as social networks [1], finance systems [2], and molecular graphs [3]. Graph Neural Networks (GNNs) have emerged as a popular approach to learn graph representations by adopting a message passing scheme [4, 5, 6], which updates a node's representation by aggregating information from its neighbors. Recently, graph contrastive learning has gained popularity as an unsupervised approach to learn node or graph representations [7, 8, 9]. The graph contrastive learning creates augmented views and minimizes the distances among positive pairs while maximizing the distances among negative pairs in the embedding space.
Despite the great success of GNNs, some existing works [10, 11, 12, 13] have shown that GNNs are vulnerable to adversarial attacks where the attackers can deliberately manipulate the graph structures and/or node features to degrade the model's performance. The representations learned by Graph Contrastive Learning (GCL) are also susceptible to such attacks, which can lead to poor performance in downstream tasks with a small amount of perturbations [14, 15, 16]. To defend against graph adversarial attacks, several empirical-based works have been conducted in robust GNNs for classification tasks [17, 18] and representation learning with contrastive learning [14, 15]. Nevertheless, with the advancement of new defense strategies, new attacks may be developed to invalidate the defense methods [19, 20, 21, 22], leading to an endless arms race. Therefore, it is important to develop a certifiably robust GNNs under contrastive learning, which can provide certificates to nodes/graphs that are robust to potential perturbations in considered space. There are several efforts in certifiable robustness of GNNs under supervised learning, which requires labels for the certifiable robustness analysis [23, 24, 25, 26]. For instance, [23] firstly analyze the certifiable robustness against the perturbation on node features. Some following works [24, 25, 26] further study the certified robustness of GNN under graph topology attacks. However, the certifiable robustness of GCL, which is in unsupervised setting, is still rarely explored.
Certifying the robustness of GCL is rather challenging. First, quantifying the robustness of GCL consistently and reliably is a difficult task. Existing GCL methods [27, 14, 15] often use empirical robustness metrics (e.g., robust accuracy) that rely on the knowledge of specific downstream tasks (e.g., task types, node/graph labels) for evaluating GCL robustness. However, these criteria may not be universally applicable across different downstream tasks. The lack of clarity in defining and assessing robustness within GCL further compounds the difficulty of developing certifiably robust GCL methods. Second, the absence of labels in GCL makes it challenging to study certifiable robustness. Existing works [23, 28, 29, 25, 24] on certifiable robustness of GNNs are designed for supervised or semi-supervised settings, which are not applicable to GCL. They typically try to provide certificates by analyzing the worst-case attack on labeled nodes. However, it is difficult to conduct such analyses due to the absence of labels in GCL. Additionally, some works [29, 24] that rely on randomized smoothing to obtain robustness certificates can not be directly applied to GCL as they may require injecting noise into the whole graph during data augmentation, introducing too many spurious edges and hurting downstream task performance, particularly for large and sparse graphs. Third, it is unclear whether certifiable robustness of GCL can be converted into robust performance in downstream tasks. The challenge arises due to the misalignment of objectives between GCL and downstream tasks. For instance, GCL usually focuses on discriminating between different instances, while some downstream tasks may also require understanding the relations among instances. This discrepancy engenders a divide between unsupervised GCL and its downstream tasks, making it difficult to prove the robust performance of the certified robust GNN encoder in downstream tasks.
In this work, we propose the first certifiably robust GCL framework, RES (Randomized Edgedrop Smoothing). (i) To address the ambiguity in evaluating and certifying robustness of GCL, we propose a unified criteria based on the semantic similarity between node or graph representations in the latent space. (ii) To address the label absence challenges and avoid adding too many noisy edges for certifying the robustness in GCL, we employ the robustness defined at (i) for unlabeled data and introduce a novel technique called randomized edgedrop smoothing, which can transform any base GNN encoder trained by GCL into a certifiably robust encoder. Therein, some randomized edgedrop noises are injected into the graphs by stochastically dropping observed edges with a certain probability, preventing the introduction of excessive noisy edges and yielding effective and efficient performance. (iii) We theoretically demonstrate that the representations learned by our robust encoder can achieve provably robust performance in downstream tasks. (iv) Moreover, we present an effective training method that enhances the robustness of GCL by incorporating randomized edgedrop noise. Our experimental results show that GRACE [30] with our RES can outperform the state-of-the-art baselines against several structural attacks and also achieve  $43.96\%$  certified accuracy in OGB-arxiv dataset when the attacker arbitrarily adds at most 10 edges.
Our main contributions are: (i) We study a novel problem of certifying the robustness of GCL to various perturbations. We introduce a unified definition of robustness in GCL and design a novel framework RES to certify the robustness; (ii) Theoretical analysis demonstrates that the representations learned by our robust encoder can achieve provably robust performance in downstream tasks; (iii) We design an effective training method for robust GCL by incorporating randomize edgedrop noise; and (iv) Extensive empirical evaluations show that our method can provide certifiable robustness for downstream tasks (i.e., node and graph classifications) on real-world datasets, and also enhance the robust performance of any GNN encoder through GCL.
Graph Contrastive Learning. GCL has recently gained significant attention and shows promise for improving graph representations in scenarios where labeled data is scarce [31, 32, 9, 30, 27, 15, 33, 32, 34, 35]. Generally, GCL creates two views through data augmentation and contrasts representations between two views. Several recent works [16, 36] show that GCL is vulnerable to adversarial attacks. Despite very few empirical works [9, 15, 14] on robustness of GCL, there are no existing works studying the certified robustness of GCL. In contrast, we propose to provide robustness certificates for GCL by using randomized edgedrop smoothing. More details are shown in Appendix A.1. To the best of our knowledge, our method is the first work to study the certified robustness of GCL.
Certifiable Robustness of GNNs. Several recent studies investigate the certified robustness of GNNs in the (semi)-supervised setting[23, 28, 29, 26, 24, 25, 37]. Zunger et al. [23] are the first to explore
certifiable robustness with respect to node feature perturbations. Subsequent works[28, 26, 29, 24, 38] extend the analysis to certifiable robustness under topological attacks. More details are shown in Appendix A.2. Our work is inherently different from them: (i) we provide an unified definition to evaluate and certify the robustness of GCL; (ii) we theoretically provide the certified robustness for GCL in the absence of labeled data, and this certified robustness can provably sustained in downstream tasks; (iii) we design an effective training method to enhance the robustness of GCL.
3 Backgrounds and Preliminaries
3.1 Graph Contrastive Learning
Notations. Let  $\mathcal{G} = (\mathcal{V},\mathcal{E},\mathbf{X})$  denote a graph, where  $\mathcal{V} = \{v_{1},\dots,v_{N}\}$  is the set of  $N$  nodes,  $\mathcal{E}\subseteq \mathcal{V}\times \mathcal{V}$  is the set of edges, and  $\mathbf{X} = \{\mathbf{x}_1,\dots,\mathbf{x}_N\}$  is the set of node attributes with  $\mathbf{x}_i$  being the node attribute of  $v_{i}$ .  $\mathbf{A}\in \mathbb{R}^{N\times N}$  is the adjacency matrix of  $\mathcal{G}$ , where  $\mathbf{A}_{ij} = 1$  if nodes  $v_{i}$  and  $v_{j}$  are connected; otherwise  $\mathbf{A}_{ij} = 0$ . In this paper, we focus on unsupervised graph contrastive learning, where label information of nodes and graphs are unavailable during training for both node- and graph-level tasks. For the node-level task, given a graph  $\mathcal{G}$ , the goal is to learn a GNN encoder  $h$  to produce representation  $\mathbf{z}_v$  for  $v$ , which can be used to conduct prediction for node  $v$  in downstream tasks. For the graph-level task, given a set of graph  $\mathbb{G} = \{\mathcal{G}_1,\mathcal{G}_2,\dots\}$ , the goal is to learn the latent representation  $\mathbf{z}_{\mathcal{G}}$  for each graph  $\mathcal{G}$ , which can be used to predict downstream label  $y_{\mathcal{G}}$  of  $\mathcal{G}$ . For simplicity and clarity, in this paper, we uniformly denote a node or graph as a concatenation vector  $\mathbf{v}$ . The representation of  $\mathbf{v}$  from encoder  $h$  is denoted as  $h(\mathbf{v})$ . More details are shown in Appendix E.2.
Graph Contrastive Learning. Generally, GCL consists of three steps: (i) multiple views are generated for each instance through stochastic data augmentation. Positive pairs are defined as two views generated from the same instance; while negative pairs are sampled from different instances; (ii) these views are fed into a set of GNN encoders  $\{h_1, h_2, \dots\}$ , which may share weights; (iii) a contrastive loss is applied to minimize the distance between positive pairs and maximize the distance between negative pairs in latent space. To achieve the certified robustness of GCL to downstream tasks, we introduce latent class to formalize the contrastive loss for GCL, which is inspired by [39, 40].
Definition 1 (Latent Class). We utilize the latent class to formalize semantic similarity of positive and negative pairs in GCL [39]. Consider a GNN encoder  $h$  learnt by GCL. Let  $\mathcal{V}$  denote the set of all possible nodes/graphs in the input space. There exists a set of latent class, denoted as  $\mathcal{C}$ , where each sample  $\mathbf{v} \in \mathcal{V}$  is associated with a latent class  $c \in \mathcal{C}$ . Each class  $c \in \mathcal{C}$  is associated with a distribution  $\mathcal{D}_c$  over the latent space of samples belonging to class  $c$  under  $h$ . The distribution on  $\mathcal{C}$  is denoted as  $\eta$ . Intuitively,  $\mathcal{D}_c(\mathbf{v})$  captures how relevant  $\mathbf{v}$  is to class  $c$  in the latent space under  $h$ , similar to the meaning of class in supervised settings. The latent class is related to the specific downstream task. For instance, when the downstream task is a classification problem, the latent class can be interpreted as the specific class to which the instances belong.
Typically, GCL is based on the label-invariant augmentation intuition: the augmented operations preserve the nature of graphs and make the augmented positive views have consistent latent classes with the original ones [41]. Let  $c^{+}, c^{-} \in \mathcal{C}$  denote the positive and negative latent classes drawn from  $\eta$ ,  $\mathcal{D}_{c^{+}}$  and  $\mathcal{D}_{c^{-}}$  are the distributions to sample positive and negative samples, respectively. For each positive pair  $(\mathbf{v}, \mathbf{v}^{+}) \sim \mathcal{D}_{c^{+}}^{2}$  associated with  $n$  negative samples  $\{\mathbf{v}_i^{-} \sim \mathcal{D}_{c^{-}} | i \in [n]\}$ , the widely-used loss for GCL, which is also known as InfoNCE loss [42], can be written as follows:
$$
\mathcal {L} _ {G C L} = \underset {c ^ {+}, c ^ {-} \sim \eta^ {2}} {\mathbb {E}} \left[ \underset {\mathbf {v}, \mathbf {v} ^ {+} \sim \mathcal {D} _ {c ^ {+}} ^ {2}, \mathbf {v} _ {i} ^ {-} \sim \mathcal {D} _ {c ^ {-}}} {\mathbb {E}} \left[ - \log \left(\frac {e ^ {h (\mathbf {v}) ^ {\top} h (\mathbf {v} ^ {+})}}{e ^ {h (\mathbf {v}) ^ {\top} h (\mathbf {v} ^ {+})} + \sum_ {i = 1} ^ {n} e ^ {h (\mathbf {v}) ^ {\top} h (\mathbf {v} _ {i} ^ {-})}}\right) \right] \right], \tag {1}
$$
3.2 Threat Model
Attacker's Goal. We consider an attacker conducts an evasion attack in GCL. Given a well-trained GNN encoder  $h$  on a clean graph  $\mathcal{G}$  via GCL, an attacker aims to degrade the performance of  $h$  in downstream tasks by injecting noise into the graph. For instance, an attacker may attempt to manipulate a social network by injecting fake edges, which could affect the performance of a well-trained  $h$  in tasks such as community detection, node classification, or link prediction. The noise can take different forms, including adding/deleting nodes/edges or augmenting node features.
Attacker's Knowledge and Capability. We assume that the attacker follows the grey-box setting to conduct an evasion attack. Specifically, the attacker has access to a benign GNN encoder  $h$  trained on a clean dataset through GCL, and the training data to train downstream classifier is available to the attacker. The model architecture and other related information of the GNN encoder are unknown to the attacker. During inference phase, the attacker is capable of injecting noises to the graph within a given budget to degrade the performance of target nodes/graphs in downstream tasks. In this paper, we focus on perturbations on the graph structure  $\mathbf{A}$ , i.e., only structural noises such as adding new edges/nodes are injected into the graph as structure attack is more effective than feature attack [11, 24, 43]. We leave the extension to feature attack and defense of GCL as future work. We denote  $\delta \in \{0,1\}^N$  as the structural noise to a node/graph  $\mathbf{v}$ , where  $N$  denote the number of nodes on the graph  $\mathbf{v}$  or  $K$ -hop subgraph of node  $\mathbf{v}$  and  $\delta_i = 1$  indicates the addition of a noisy edge to  $\mathbf{v}$  in the  $i$ -th entry, and its  $L_0$ -norm  $||\delta||_0$  indicates the number of noisy edges. We then have the perturbed version of  $\mathbf{v}$  by the attacker, denoted as  $\mathbf{v}' = \mathbf{v} \oplus \delta$ .
3.3 Problem Statement
Our objective is to develop a certifiably robust GCL. We aim to train a GNN encoder that exhibits provably benign behaviors in downstream tasks. The problem can be formulated as follows:
Problem 1. Given a GNN encoder  $h$ , a node or graph  $\mathbf{v}$ . Let  $\mathbf{v}' = \mathbf{v} \oplus \delta$  denote the perturbed version of  $\mathbf{v}$  by the attacker, where the structural noise  $\delta$  with  $\| \delta \|_0 \leq k$  is injected into  $\mathbf{v}$ . Suppose that  $c^*$  is the latent class of  $\mathbf{v}$  and  $f$  is the downstream classifier. Our goal is to develop a certifiably robust GCL for  $h$  such that for any  $c \neq c^*$ , the representation of  $\mathbf{v}'$  under  $h$  can satisfy the following requirement in downstream tasks:
$$
m \left(\mathbf {v} ^ {\prime}, c ^ {*}; h\right) = \min  _ {c \neq c ^ {*}} \left[ \mathbb {P} \left(f \left(h \left(\mathbf {v} ^ {\prime}\right)\right) = c ^ {*}\right) - \mathbb {P} \left(f \left(h \left(\mathbf {v} ^ {\prime}\right)\right) = c\right) \right] > 0, \tag {2}
$$
where  $m(\mathbf{v}', c^*; h)$  is the worst-case margin of  $h(\mathbf{v}')$  between  $c^*$  and  $y$  in the downstream task for any  $c \neq c^*$ , and  $\mathbb{P}(f(h(\mathbf{v}')) = c)$  denotes the probability of the representation  $h(\mathbf{v}')$  being classified into class  $c$  by  $f$  in the downstream task. Eq. (2) implies that  $f(h(\mathbf{v}')) = f(h(\mathbf{v})) = c^*$  for any  $\delta$  within the budget  $k$ , i.e.,  $h$  is a certifiably robust at  $(\mathbf{v}, c^*)$  when perturbing at most  $k$  edges.
4 Certifying Robustness for GCL
In this section, we present the details of our RES, which aims to provide certifiable robustness for GCL models. There are mainly three challenges: (i) how to define the certified robustness of GCL; (ii) how to derive the certified robustness; and (iii) how to transfer the certified robustness of GCL to downstream tasks. To address these challenges, we first define certified robustness for GCL in Sec. 4.1. We then propose the RES method to derive the certificates and theoretically guarantee certifiable robustness in Sec. 4.2. Finally, we show that the certifiably robust representations learned from our approach are still provably robust in downstream tasks in Sec. 4.3.
4.1 Certified Robustness of GCL
To give the definition of certified robustness of GCL, we first give the conditions for successfully attacking the GNN encoder. The core idea behind it comes from supervised learning, where the objective is to judge whether the predictions of target nodes/graphs are altered under specific perturbations. Inspired by [40], we consider the following scenario of a successful attack against the GNN encoder.
Given a clean input  $\mathbf{v}$ , with positive and negative samples  $\mathbf{v}^{+}$  and  $\mathbf{v}^{-}$  used in the learning process of GCL, respectively, we consider  $\mathbf{v}' = \mathbf{v} \oplus \delta$ , where  $\delta$  represents structural noise on  $\mathbf{v}$  and  $\mathbf{v}^{-}$  is the attack target of  $\mathbf{v}$ . The attacker's goal is to produce an adversarial example  $\mathbf{v}'$  that can deceive the model into classifying  $\mathbf{v}$  as similar to  $\mathbf{v}^{-}$ . Formally, given a well-trained GNN encoder  $h$  via GCL, we say that  $h$  has been successfully attacked at  $\mathbf{v}$  if the cosine similarity  $s(h(\mathbf{v}'), h(\mathbf{v}^{+}))$  is less than  $s(h(\mathbf{v}'), h(\mathbf{v}^{-}))$ . This indicates that  $\mathbf{v}'$  is more similar to  $\mathbf{v}^{-}$  than to  $\mathbf{v}^{+}$  in the latent space. Otherwise, we conclude that  $h$  has not been successfully attacked. This definition of successful attack provides the basis for the definition of certified robustness of GCL. The formal definition of certified robustness problem for GCL is then given as
Definition 2 (Certified Robustness of GCL). Given a well trained GNN encoder  $h$  via GCL. Let  $\mathbf{v}$  be a clean input.  $\mathbf{v}^{+}$  is the positive sample of  $\mathbf{v}$  and  $\mathbf{V}^{-} = \{\mathbf{v}_{1}^{-},\dots ,\mathbf{v}_{n}^{-}\}$  denotes all possible negative
samples of  $\mathbf{v}^{+}$ , which are sampled as discussed in Sec. 3.1. Suppose that  $\mathbf{v}'$  is a perturbed sample obtained by adding structural noise  $\delta$  to  $\mathbf{v}$ , where  $||\delta||_0 \leq k$ , and  $s(\cdot, \cdot)$  is a cosine similarity function. Then,  $h$  is certifiably  $l_0^k$ -robust at  $(\mathbf{v}, \mathbf{v}^{+})$  if the following inequality is satisfied:
$$
s \left(h \left(\mathbf {v} ^ {\prime}\right), h \left(\mathbf {v} ^ {+}\right)\right) > \max  _ {\mathbf {v} ^ {-} \in \mathbf {V} ^ {-}} s \left(h \left(\mathbf {v} ^ {-}\right), h \left(\mathbf {v} ^ {\prime}\right)\right), \forall \delta : \| \delta \| _ {0} \leq k. \tag {3}
$$
Similar to the supervised certified robustness, this problem is to prove that the point  $\mathbf{v}' = \mathbf{v} \oplus \delta$  under the perturbation within budget  $k$  is still the positive sample of  $\mathbf{v}^+$ . However, unlike the supervised learning, where we can estimate the prediction distribution of  $\mathbf{v}$  by leveraging the labels of the training data, it is difficult to estimate such a distribution in GCL because the label information is unavailable. To address this issue, we first consider a space  $\mathbb{B}$  for a sample  $\mathbf{v}^+$ , where each sample within it is the positive sample of  $\mathbf{v}^+$ . Formally, given a sample  $\mathbf{v}^+$  with the latent class  $c^+$  from the probability distribution  $\mathcal{D}_c^+$ ,  $\mathbb{B}(\mathbf{v}^+)$  is defined as
$$
\mathbb {B} \left(\mathbf {v} ^ {+}\right) := \left\{\mathbf {v} \mid \mathbf {v} \in c ^ {+} \right\}. \tag {4}
$$
GCL maximizes the agreement of the positive pair in the latent space and assume access to similar data in the form of pairs  $(\mathbf{v},\mathbf{v}^{+})$  that comes from a distribution  $\mathcal{D}_{c^+}^2$  given the latent class  $c^+$ . Thus, it is natural to connect the latent class and the similarity between representations. The probability that  $\mathbf{v}$  is the positive sample of  $\mathbf{v}^{+}$  can be given by the following theorem.
Theorem 1. Let  $\mathbb{B}(\mathbf{v}^{+})$  be a space around  $\mathbf{v}^{+}$  as defined in Eq. (4). Given an input sample  $\mathbf{v}$  and a GNN encoder  $h$  learnt by GCL in Eq. (1), the probability of  $\mathbf{v}$  being the positive sample of  $\mathbf{v}^{+}$  is:
$$
P r (\mathbf {v} \in \mathbb {B} (\mathbf {v} ^ {+}); h) = \exp \left[ - \left(\frac {1 - s (h (\mathbf {v}) , h (\mathbf {v} ^ {+}))}{a}\right) ^ {\sigma} \right], \tag {5}
$$
where  $s(\cdot, \cdot)$  is the cosine similarity function.  $a, \sigma > 0$  are Weibull shape and scale parameters [44]. The proof is in Appendix C.1. Theorem 1 manifests that we can derive the probability of  $\mathbf{v}$  being the positive sample of  $\mathbf{v}^+$  based on the cosine similarity under  $h$ , providing us a feasible way to study the certified robustness problem for GCL without the label information.
4.2 Certifying Robustness by the Proposed Randomized EdgeDrop Smoothing (RES)
With the definition of certifiable robustness of GCL, we introduce our proposed Randomized Edgedrop Smoothing (RES), which provides certifiable robustness for GCL. RES injects randomized edgedrop noise by randomly dropping each edge of the input sample with a certain probability. We will also demonstrate the theoretical basis for the certifiable robustness of GCL achieved by RES.
Firstly, we define an randomized edgedrop noise  $\epsilon$  for  $\mathbf{v}$  that remove each edge of  $\mathbf{v}$  with the probability  $\beta$ . Formally, given a sample  $\mathbf{v}$ , the probability distribution of  $\epsilon$  for  $\mathbf{v}$  is given as:
$$
\mathbb {P} \left(\epsilon_ {i} = 0 \mid \mathbf {v} _ {i} = 0\right) = 1, \mathbb {P} \left(\epsilon_ {i} = 0 \mid \mathbf {v} _ {i} = 1\right) = \beta , \text {a n d} \mathbb {P} \left(\epsilon_ {i} = 1 \mid \mathbf {v} _ {i} = 1\right) = 1 - \beta , \tag {6}
$$
where  $\mathbf{v}_i$  denotes the connection status of  $i$ -th entry of  $\mathbf{v}$ . Given a well-trained GNN encoder  $h$  via GCL and a noise  $\epsilon$  with the distribution in Eq. (6), the smoothed GNN encoder  $g$  is defined as:
$$
g (\mathbf {v}) = h (\mathbf {v} \oplus \epsilon), \text {w h e r e} \mathbf {v} \oplus \epsilon \in \mathbb {B} (\underset {\hat {\mathbf {v}} \in \mathbf {V}} {\arg \max } \mathbb {P} (\mathbf {v} \oplus \epsilon \in \mathbb {B} (\hat {\mathbf {v}}); h)). \tag {7}
$$