SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yue Yu $^{1}$ , Xiao Wang $^{2*}$ , Mengmei Zhang $^{1}$ , Nian Liu $^{1}$ , Chuan Shi $^{1*}$ $^{1}$ Beijing University of Posts and Telecommunications, China  
 $^{2}$ Beihang University, China  
yuyue1218@bupt.edu.cn, xiao_wang@buaa.edu.cn, {zhangmm, nianliu, shichuan}@bupt.edu.cn
Graph Contrastive Learning (GCL) has emerged as a popular training approach for learning node embeddings from augmented graphs without labels. Despite the key principle that maximizing the similarity between positive node pairs while minimizing it between negative node pairs is well established, some fundamental problems are still unclear. Considering the complex graph structure, are some nodes consistently well-trained and following this principle even with different graph augmentations? Or are there some nodes more likely to be untrained across graph augmentations and violate the principle? How to distinguish these nodes and further guide the training of GCL? To answer these questions, we first present experimental evidence showing that the training of GCL is indeed imbalanced across all nodes. To address this problem, we propose the metric "node compactness", which is the lower bound of how a node follows the GCL principle related to the range of augmentations. We further derive the form of node compactness theoretically through bound propagation, which can be integrated into binary cross-entropy as a regularization. To this end, we propose the PrOvable Training  $(\mathrm{POT}^1)$  for GCL, which regularizes the training of GCL to encode node embeddings that follows the GCL principle better. Through extensive experiments on various benchmarks, POT consistently improves the existing GCL approaches, serving as a friendly plugin.
Graph Neural Networks (GNNs) are successful in many graph-related applications, e.g., node classification [10, 20] and link prediction [33, 27]. Traditional GNNs follow a semi-supervised learning paradigm requiring task-specific high-quality labels, which are expensive to obtain [9]. To alleviate the dependence on labels, graph contrastive learning (GCL) is proposed and has received considerable attention recently [35, 31, 25, 21, 29]. As a typical graph self-supervised learning technique, GCL shows promising results on many downstream tasks [22, 30, 23].
The learning process of GCL usually consists of the following steps: generate two graph augmentations [35]; feed these augmented graphs into a graph encoder to learn the node embeddings, and then optimize the whole training process based on the InfoNCE principle [15, 35, 36], where the positive and negative node pairs are selected from the two augmentations. With this principle ("GCL principle" for short), GCL learns the node embeddings where the similarities between the positive node pairs are maximized while the similarities between the negative node pairs are minimized.
However, it is well known that the real-world graph structure is complex and different nodes may have different properties [1, 2, 11, 36]. When generating augmentations and training on the complex graph
$^{1}$ Code available at https://github.com/VoidHarui/POT-GCL
structure in GCL, it may be hard for all nodes to follow the GCL principle well enough. This naturally gives rise to the following questions: are some of the nodes always well-trained and following the principle of GCL given different graph augmentations? Or are there some nodes more likely not to be well trained and violate the principle? If so, it implies that each node should not be treated equally in the training process. Specifically, GCL should pay less attention to those well-trained nodes that already satisfy the InfoNCE principle. Instead, GCL should focus more on those nodes that are sensitive to the graph augmentations and hard to be well trained. The answer to these questions can not only deepen our understanding of the learning mechanism but also improve the current GCL. We start with an experimental analysis (Section 3) and find that the training of GCL is severely imbalanced, i.e. the averaged InfoNCE loss values across the nodes have a fairly high variance, especially for the not well-trained nodes, indicating that not all the nodes follow the GCL principle well enough.
Once the weakness of GCL is identified, another two questions naturally arise: how to distinguish these nodes given so many possible graph augmentations? How to utilize these findings to further improve the existing GCL? Usually, given a specific graph augmentation, it is easy to check how different nodes are trained to be. However, the result may change with respect to different graph augmentations, since the graph augmentations are proposed from different views and the graph structure can be changed in a very diverse manner. Therefore, it is technically challenging to discover the sensitivity of nodes to all these possible augmentations.
In this paper, we theoretically analyze the property of different nodes in GCL. We propose a novel concept "node compactness", measuring how worse different nodes follow the GCL principle in all possible graph augmentations. We use a bound propagation process [5, 32] to theoretically derive the node compactness, which depends on node embeddings and network parameters during training. We finally propose a novel PrOvable Training model for GCL (POT) to improve the training of GCL, which utilizes node compactness as a regularization term. The proposed POT encourages the nodes to follow the GCL principle better. Moreover, because our provable training model is not specifically designed for some graph augmentation, it can be used as a friendly plug-in to improve current different GCL methods. To conclude, our contributions are summarized as follows:
- We theoretically analyze the properties of different nodes in GCL given different graph augmentations, and discover that the training of GCL methods is severely imbalanced, i.e., not all the nodes follow the principle of GCL well enough and many nodes are always not well trained in GCL.
- We propose the concept of "node compactness" that measures how each node follows the GCL principle. We derive the node compactness as a regularization using the bound propagation method and propose PrOvable Training for GCL (POT), which improves the training process of GCL provably.
- POT is a general plug-in and can be easily combined with existing GCL methods. We evaluate our POT method on various benchmarks, well showing that POT consistently improves the current GCL baselines.
Let  $G = (\mathcal{V}, \mathcal{E})$  denote a graph, where  $\mathcal{V}$  is the set of nodes and  $\mathcal{E} \subseteq \mathcal{V} \times \mathcal{V}$  is the set of edges, respectively.  $\mathbf{X} \in \mathbb{R}^{N \times F}$  and  $\mathbf{A} = \{0,1\}^{N \times N}$  are the feature matrix and the adjacency matrix.  $\mathbf{D} = \text{diag}\{d_1,d_2,\dots,d_N\}$  is the degree matrix, where each element  $d_i = \| \mathbf{A}_i\|_0 = \sum_{j \in \mathcal{V}} a_{ij}$  denotes the degree of node  $i$ ,  $\mathbf{A}_i$  is the  $i$ -th row of  $\mathbf{A}$ . Additionally,  $\mathbf{A}_{sym} = \mathbf{D}^{-1/2}(\mathbf{A} + \mathbf{I})\mathbf{D}^{-1/2}$  is the degree normalized adjacency matrix with self-loop (also called the message-passing matrix).
GCN [10] is one of the most common encoders for graph data. A graph convolution layer can be described as  $\mathbf{H}^{(k)} = \sigma (\mathbf{A}_{sym}\mathbf{H}^{(k - 1)}\mathbf{W}^{(k)})$ , where  $\mathbf{H}^{(k)}$  and  $\mathbf{W}^{(k)}$  are the embedding and the weight matrix of the  $k$ -th layer respectively,  $\mathbf{H}^{(0)} = \mathbf{X}$ , and  $\sigma$  represents the non-linear activation function. A GCN encoder  $f_{\theta}$  with  $K$  layers takes a graph  $(\mathbf{X},\mathbf{A})$  as input and returns the embedding  $\mathbf{Z} = f_{\theta}(\mathbf{X},\mathbf{A})$ .
Graph Contrastive learning (GCL). GCL learns the embeddings of the nodes in a self-supervised manner. We consider a popular setting as follows. First, two graph augmentations  $G_{1} = (\mathbf{X},\tilde{\mathbf{A}}_{1})$  and  $G_{2} = (\mathbf{X},\tilde{\mathbf{A}}_{2})$  are sampled from  $\mathcal{G}$ , which is the set of all possible augmentations. In this paper,
we focus on the augmentations to graph topology (e.g. edge dropping [35]). Then the embeddings  $\mathbf{Z}_1 = f_{\theta}(\mathbf{X},\tilde{\mathbf{A}}_1)$  and  $\mathbf{Z}_2 = f_{\theta}(\mathbf{X},\tilde{\mathbf{A}}_2)$  are learned with a two-layer GCN [35, 36]. InfoNCE [15, 35] is the training objective as follows:
$$
l \left(\mathbf {z} _ {1, i}, \mathbf {z} _ {2, i}\right) = \frac {e ^ {\theta \left(\mathbf {z} _ {1 , i} , \mathbf {z} _ {2 , i}\right) / \tau}}{e ^ {\theta \left(\mathbf {z} _ {1 , i} , \mathbf {z} _ {2 , i}\right) / \tau} + \sum_ {j \neq i} e ^ {\theta \left(\mathbf {z} _ {1 , i} , \mathbf {z} _ {2 , j}\right) / \tau} + \sum_ {l \neq i} e ^ {\theta \left(\mathbf {z} _ {1 , i} , \mathbf {z} _ {1 , l}\right) / \tau}}, \tag {1}
$$
where  $\theta(a, b) = s(g(a), g(b))$ ,  $s$  is a similarity measure,  $g$  is a projector, and  $\tau$  is the temperature hyperparameter. For node  $i$ , the positive pair is  $(\mathbf{z}_{1,i}, \mathbf{z}_{2,i})$ , where  $\mathbf{z}_{1,i}$  and  $\mathbf{z}_{2,i}$  denote the embedding of node  $i$  in the two augmentations,  $\{((\mathbf{z}_{1,i}, \mathbf{z}_{2,j}) \mid j \neq i\}$  and  $\{((\mathbf{z}_{1,i}, \mathbf{z}_{1,l}) \mid l \neq i\}$  are the negative pairs. The overall loss function of the two augmentations is  $\mathcal{L}_{\mathrm{InfoNCE}}(\mathbf{Z}_1, \mathbf{Z}_2) = \frac{1}{2N} \sum_{i=1}^{N} (l(\mathbf{z}_{1,i}, \mathbf{z}_{2,i}) + l(\mathbf{z}_{2,i}, \mathbf{z}_{1,i}))$ .
3 The Imbalanced Training of GCL: an Experimental Study
As mentioned in Section 1, due to the complex graph structure, we aim to investigate whether all the nodes are trained to follow the principle of GCL. Specifically, we design an experiment by plotting the distribution of InfoNCE loss values of the nodes. To illustrate the result of a GCL method, we sample 500 pairs of augmented graphs with the same augmentation strategy in the GCL method and obtain the average values of In
(b) GCA
(c) ProGCL
Figure 1: The imbalance of GCL training
foNCE loss. We show the results of GRACE [35], GCA [36], and ProGCL [25] on WikiCS [14] in Figure 1 with histograms. The vertical red lines represent 1/4-quantile, median, and 3/4-quantile from left to right. The results imply that the nodes do not follow the GCL principle well enough in two aspects: 1) the InfoNCE loss of the nodes has a high variance; 2) It is shown that the untrained nodes (with higher InfoNCE loss values) are much further away from the median than the well-trained nodes (with lower InfoNCE loss values), indicating an imbalanced training in the untrained nodes. GCL methods should focus more on those untrained nodes. Therefore, it is highly desired that we should closely examine the property of different nodes so that the nodes can be trained as we expected in a provable way. Results on more datasets can be found in Appendix D.
4.1 Evaluating How the Nodes Follow the GCL Principle
In this section, we aim to define a metric that measures the extent of a node following the GCL principle. Since how the nodes follow the GCL principle relies on the complex graph structure generated by augmentation, the metric should be related to the set of all possible augmentations. Because InfoNCE loss can only measure the training of a node under two pre-determined augmentations, which does not meet the requirement, we propose a new metric as follows:
Definition 1 (Node Compactness). Let  $f$  denote the encoder of Graph Contrastive Learning. The compactness of node  $i$  is defined as
$$
\underline {{\tilde {f}}} _ {1, i} = \min  _ {\substack {G _ {1} = (\mathbf {X}, \mathbf {A} _ {1}) \in \mathcal {G}, \\ G _ {2} = (\mathbf {X}, \mathbf {A} _ {2}) \in \mathcal {G}}} \left\{\frac {1}{N - 1} \sum_ {j \neq i} \left(\mathbf {z} _ {1, i} \cdot \bar {\mathbf {z}} _ {2, i} - \mathbf {z} _ {1, i} \cdot \bar {\mathbf {z}} _ {2, j}\right) \mid \mathbf {Z} _ {1} = f _ {\theta} (\mathbf {X}, \mathbf {A} _ {1}), \mathbf {Z} _ {2} = f _ {\theta} (\mathbf {X}, \mathbf {A} _ {2}) \right\}. \tag{2}
$$
where  $\bar{\mathbf{z}} = \mathbf{z} / \| \mathbf{z}\| _2$  is the normalized vector.
Definition 1 measures how worse node  $i$  follows the GCL principle in all possible augmentations. If  $\underline{f}_i$  is greater than zero, on average, the node embedding  $\mathbf{z}_{1,i}$  is closer to the embedding of the positive sample  $\mathbf{z}_{2,i}$  than the embedding of the negative sample  $\mathbf{z}_{2,j}$  in the worst case of augmentation; otherwise, the embeddings of negative samples are closer. Note that normalization is used to eliminate
the effect of a vector's scale. In Definition 1, the anchor node is node  $i$  in  $G_{1}$ , so similarly we can define  $\underline{\tilde{f}}_{2,i}$  if the anchor node is node  $i$  in  $G_{2}$ .
We can just take one of  $G_{1}$  and  $G_{2}$  as the variable, for example, set  $G_{2}$  to be a specific sampled augmentation and  $G_{1}$  to be an augmentation within the set  $\mathcal{G}$ . From this perspective, based on Definition 1, we can also obtain a metric, which has only one variable:
Definition 2 ( $G_2$ -Node Compactness). Under Definition 1, if  $G_2$  is the augmentation sampled in an epoch of GCL training, then the  $G_2$ -Node Compactness is
$$
\underline {{\tilde {f}}} _ {G _ {2}, i} = \min  _ {G _ {1} = f _ {\theta} (\mathbf {X}, \mathbf {A} _ {1}) \in \mathcal {G}} \left\{\mathbf {z} _ {1, i} \cdot \mathbf {W} _ {2, i} ^ {C L} \mid \mathbf {Z} _ {1} = f _ {\theta} (\mathbf {X}, \mathbf {A} _ {1}) \right\}, \tag {3}
$$
where  $\mathbf{W}_{2,i}^{CL} = \bar{\mathbf{z}}_{2,i} - \frac{1}{N - 1}\sum_{j\neq i}\bar{\mathbf{z}}_{2,j}$  is a constant vector.
Similarly, we can define  $G_{1}$ -node compactness  $\underline{f}_{G_1,i}$ . Usually we have  $\underline{f}_{G_1,i} \geq \underline{f}_{2,i}$  and  $\underline{f}_{G_2,i} \geq \underline{f}_{1,i}$ , i.e.,  $G_{1}$ -node compactness and  $G_{2}$ -node compactness are upper bounds of node compactness.
4.2 Provable Training of GCL
To enforce the nodes to better follow the GCL principle, we optimize  $G_{1}$ -node compactness and  $G_{2}$ -node compactness in Definition 2. Designing the objective can be viewed as implicitly performing a "binary classification": the "prediction" is  $\min_{G_1}z_{1,i}\cdot W_{2,i}^{CL}$ , and the "label" is  $\mathbf{1}$ , which means that we expect the embedding of the anchor more likely to be classified as a positive sample in the worst case. To conclude, we propose the PrOvable Training (POT) for GCL, which provably enforces the nodes to be trained more aligned with the GCL principle. Specifically, we use binary cross-entropy as a regularization term:
$$
\mathcal {L} _ {\mathrm {P O T}} \left(G _ {1}, G _ {2}\right) = \frac {1}{2} \left(\mathcal {L} _ {\mathrm {B C E}} \left(\underline {{\tilde {f}}} _ {G _ {1}}, \mathbf {1} _ {n}\right) + \mathcal {L} _ {\mathrm {B C E}} \left(\underline {{\tilde {f}}} _ {G _ {2}}, \mathbf {1} _ {n}\right)\right), \tag {4}
$$
where  $\mathcal{L}_{\mathrm{BCE}}(x,y) = -\sum_{i=1}^{N}(y_i\log\sigma(x_i) + (1-y_i)\log(1-\sigma(x_i))$ ,  $\sigma$  is the sigmoid activation,  $\mathbf{1}_N = [1,1,\dots,1] \in \mathbb{R}^N$ , and  $\underline{\tilde{f}}_{G_2} = [\underline{\tilde{f}}_{G_2,1},\underline{\tilde{f}}_{G_2,2},\dots,\underline{\tilde{f}}_{G_2,N}]$  is the vector of the lower bounds in Eq. 3. Overall, we add this term to the original InfoNCE objective and the loss function is
$$
\mathcal {L} = (1 - \kappa) \mathcal {L} _ {\text {I n f o N C E}} \left(Z _ {1}, Z _ {2}\right) + \kappa \mathcal {L} _ {\text {P O T}} \left(G _ {1}, G _ {2}\right), \tag {5}
$$
where  $\kappa$  is the hyperparameter balancing the InfoNCE loss and the compactness loss. Since  $G_{1}$ -node compactness and  $G_{2}$ -node compactness are related to the network parameter  $\theta$ ,  $\mathcal{L}_{\mathrm{POT}}$  can be naturally integrated into the loss function, which can regularize the network parameters to encode node embeddings more likely to follow the GCL principle better.
4.3 Deriving the Lower Bounds
In Section 4.2, we have proposed the formation of our loss function, but the lower bounds  $\underline{\tilde{f}}_{G_1}$  and  $\underline{\tilde{f}}_{G_2}$  are still to be solved. In fact, to see Eq. 3 from another perspective,  $\underline{\tilde{f}}_{G_1,1}$  (or  $\underline{\tilde{f}}_{G_2,1}$ ) is the lower bound of the output of the neural network concatenating a GCN encoder and a linear projection with parameter  $\mathbf{W}_{1,i}^{CL}$  (or  $\mathbf{W}_{2,i}^{CL}$ ). It is difficult to find the particular  $G_1$  which minimizes Eq. 3, since finding that exact solution  $G_1$  can be an NP-hard problem considering the discreteness of graph augmentations. Therefore, inspired by the bound propagation methods [5, 32, 38], we derive the output lower bounds of that concatenated network in a bound propagation manner, i.e., we can derive the lower bounds  $\underline{\tilde{f}}_{G_1}$  and  $\underline{\tilde{f}}_{G_2}$  by propagating the bounds of the augmented adjacency matrix  $\mathbf{A}_1$ .
However, there are still two challenges when applying bound propagation to our problem: 1)  $\mathbf{A}_1$  in Eq. 3 is not defined as continuous values with element-wise bounds; 2) to do the propagation process, the nonlinearities in the GCN encoder  $f_{\theta}$  should be relaxed.
Defining the adjacency matrix with continuous values First, we define the set of augmented adjacency matrices as follows:
Definition 3 (Augmented adjacency matrix). The set of augmented adjacency matrices is
$$
\begin{array}{l} \mathcal {A} = \{\tilde {\mathbf {A}} \in \{0, 1 \} ^ {N \times N} | \tilde {a} _ {i j} \leq a _ {i j} \wedge \tilde {\mathbf {A}} = \tilde {\mathbf {A}} ^ {T} \\ \wedge \left\| \tilde {\mathbf {A}} - \mathbf {A} \right\| _ {0} \leq 2 Q \wedge \left\| \tilde {\mathbf {A}} _ {i} - \mathbf {A} _ {i} \right\| _ {0} \leq q _ {i} \forall 1 \leq i \leq N \}, \\ \end{array}
$$
where  $\tilde{\mathbf{A}}$  denotes the augmented adjacency matrix,  $Q$  and  $q_{i}$  are the global and local budgets of edge dropped, which are determined by the specific GCL model.
The first constraint restricts the augmentation to only edge dropping, which is a common setting in existing GCL models[35, 36, 8, 25] (see more discussions in Appendix B). The second constraint is the symmetric constraint of topology augmentations.
Since the message-passing matrix is used in GNN instead of the original adjacency matrix, we further relax Definition 3 and define the set of all possible message-passing matrices, which are matrices with continuous entries as expected, inspired by [38]:
Definition 4 (Augmented message-passing matrix). The set of augmented message-passing matrices is
$$
\begin{array}{l} \hat {\mathcal {A}} = \left\{\hat {\mathbf {A}} \in [ 0, 1 ] ^ {N \times N} \mid \hat {\mathbf {A}} = \hat {\mathbf {A}} ^ {T} \wedge \forall i, j: L _ {i j} \leq \hat {a} _ {i j} \leq U _ {i j} \right\}, \\ L _ {i j} = \left\{ \begin{array}{l l} \tilde {a} _ {i j} & , i = j \\ 0 & , i \neq j \end{array} \right., U _ {i j} = \left\{ \begin{array}{l l} ((d _ {i} - q _ {i}) (d _ {j} - q _ {j})) ^ {- \frac {1}{2}} & , i = j \\ \min  \{a _ {i j}, ((d _ {i} - q _ {i}) (d _ {j} - q _ {j})) ^ {- \frac {1}{2}} \} & , i \neq j \end{array} \right., \\ \end{array}
$$
where  $\hat{\mathbf{A}}$  denotes the augmented message-passing matrix.
Under Definition 4, the augmented message-passing matrices are non-discrete matrices with element-wise bounds  $L_{ij}$  and  $U_{ij}$ .  $L_{ij}$  is  $\tilde{a}_{ij}$  when  $i = j$ , since there is no edge added and the self-loop can not be dropped;  $L_{ij}$  is 0 when  $i \neq j$ , which implies the potential dropping of each edge.  $U_{ij}$  is set to  $((d_i - q_i)(d_j - q_j))^{-\frac{1}{2}}$  when  $i = j$  because  $\hat{a}_{ii}$  is the largest when its adjacent edges are dropped as much as possible, and  $U_{ij} = \min \{a_{ij}, ((d_i - q_i)(d_j - q_j))^{-\frac{1}{2}}\}$  when  $i \neq j$  since there is only edge dropping,  $U_{ij}$  cannot exceed  $a_{ij}$ .
Relaxing the nonlinearities in the GCN encoder The GCN encoder in our problem has non-linear activation functions as the nonlinearities in the network, which is not suitable for bound propagation. Therefore, we relax the activations with linear bounds [32]:
Definition 5 (Linear bounds of non-linear activation function). For node  $i$  and the  $m$ -th neuron in the  $k$ -th layer, the input of the neuron is  $p_{i,m}^{(k)}$ , and  $\sigma$  is the activation function of the neuron. With the lower bound and upper bound of the input denoted as  $p_{L,i,m}^{(k)}$  and  $p_{U,i,m}^{(k)}$ , the output of the neuron is bounded by linear functions with parameters  $\alpha_{L,i,m}^{(k)}, \alpha_{U,i,m}^{(k)}, \beta_{L,i,m}^{(k)}, \beta_{U,i,m}^{(k)} \in \mathbb{R}$ , i.e.
$$
\alpha_ {L, i, m} ^ {(k)} \left(p _ {i, m} ^ {(k)} + \beta_ {L, i, m} ^ {(k)}\right) \leq \sigma \left(p _ {i, m} ^ {(k)}\right) \leq \alpha_ {U, i, m} ^ {(k)} \left(p _ {i, m} ^ {(k)} + \beta_ {U, i, m} ^ {(k)}\right).
$$
The values of  $\alpha_{L,i,m}^{(k)},\alpha_{U,i,m}^{(k)},\beta_{L,i,m}^{(k)},\beta_{U,i,m}^{(k)}$  depend on the activation  $\sigma$  and the pre-activation bounds  $p_{L,i,m}^{(k)}$  and  $p_{U,i,m}^{(k)}$
To further obtain  $\alpha$  and  $\beta$ , we compute the pre-activation bounds  $\mathbf{P}_L^{(k)}$  and  $\mathbf{P}_U^{(k)}$  as follows, which is the matrix with elements  $p_{L,i,m}^{(k)}$  and  $p_{U,i,m}^{(k)}$ :
Theorem 1 (Pre-activation bounds of each layer [16]). If  $f = \sigma(\hat{\mathbf{A}}\sigma(\hat{\mathbf{A}}\mathbf{X}\mathbf{W}^{(1)} + \mathbf{b}^{(1)})\mathbf{W}^{(2)} + \mathbf{b}^{(2)})$  is the two-layer GCN encoder, given the element-wise bounds of  $\hat{\mathbf{A}}$  in Definition 4,  $\mathbf{H}^{(k)}$  is the input embedding of  $t$ -th layer, then the pre-activation bounds of  $t$ -th layer are
$$
\mathbf {P} _ {L} ^ {(k)} = \mathbf {L} [ \mathbf {H} ^ {(k)} \mathbf {W} ^ {(k)} ] _ {+} + \mathbf {U} [ \mathbf {H} ^ {(k)} \mathbf {W} ^ {(k)} ] _ {-} + \mathbf {b} ^ {(k)},
$$
$$
\mathbf {P} _ {U} ^ {(k)} = \mathbf {U} [ \mathbf {H} ^ {(k)} \mathbf {W} ^ {(k)} ] _ {+} + \mathbf {L} [ \mathbf {H} ^ {(k)} \mathbf {W} ^ {(k)} ] _ {-} + \mathbf {b} ^ {(k)},
$$
where  $[\mathbf{X}]_{+} = \max (\mathbf{X},0)$  and  $[\mathbf{X}]_{-} = \min (\mathbf{X},0)$ .
The detailed proof is provided in Appendix A.1. Therefore,  $p_{L,i,m}^{(k)}$  and  $p_{U,i,m}^{(k)}$  are known with Theorem 1 given, then the values of  $\alpha$  and  $\beta$  can be obtained by referring to Table 1. In the table, we denote  $p_{U,i,m}^{(k)}$  as  $u$  and  $p_{L,i,m}^{(k)}$  as  $l$  for short.  $\gamma$  is the slope parameter of ReLU-like activation functions that are usually used in GCL, such as ReLU, PReLU, and RReLU. For example,  $\gamma$  is zero for ReLU.
To this end, all the parameters to derive the lower bounds  $\underline{\tilde{f}}_{G_1}$  and  $\underline{\tilde{f}}_{G_2}$  are known: the bounds of the variable  $\hat{\mathbf{A}}$  in our problem, which are  $\mathbf{L}$  and  $\mathbf{U}$  defined in Definition 4; the parameters of the linear bounds for non-linear activations, which are  $\alpha$  and  $\beta$  defined in Definition 5 and obtained by Theorem 1 and Table 1. To conclude, the whole process is summarized in the following theorem, where we apply bound propagation to our problem with those parameters:
Theorem 2 (The lower bound of neural network output). If the encoder is defined as  $1$ ,  $\hat{\mathbf{A}}$  is the augmented message-passing matrix and  $\sigma$  is the non-linear activation function, for  $G_{1} = (\mathbf{X},\mathbf{A}_{1})\in \mathcal{G}$ ,  $\hat{\mathbf{A}} = \mathbf{D}^{-1 / 2}(\mathbf{A}_1 + \mathbf{I})\mathbf{D}^{-1 / 2}\in \hat{\mathcal{A}}$ , then
Table 1: Values of  $\alpha$  and  $\beta$
l≥0	u≤0	else
α( k )_{L,i,m}	1	γ	\(\frac{u-\gamma l}{u-l}\)
α( k )_{U,i,m}	1	γ	\(\frac{u-\gamma l}{u-l}\)
β( k )_{L,i,m}	0	0	0
β( k )_{U,i,m}	0	0	\(\frac{(\gamma-1)ul}{u-\gamma l}\)