SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\underline {{\tilde {f}}} _ {G _ {2}, i} = \sum_ {j _ {1} \in \mathcal {N} (i)} \hat {a} _ {j _ {1} i} \left[ \sum_ {j _ {2} \in \mathcal {N} (j _ {1})} \hat {a} _ {j _ {2} j _ {1}} x _ {j _ {2}} \tilde {\mathbf {W}} _ {i} ^ {(1)} + \tilde {\mathbf {b}} ^ {(1)} \right] + \tilde {\mathbf {b}} ^ {(2)} \tag {6}
$$
where  $\mathbf{W}_{2,i}^{CL} = \bar{\mathbf{z}}_{2,i} - \frac{1}{N - 1}\sum_{j\neq i}\bar{\mathbf{z}}_{2,j},\mathcal{N}(i) = \{j\mid a_{ij} > 0 or j = i\} ,$  (7)
$$
\lambda_ {i, m} ^ {(2)} = \left\{ \begin{array}{l l} \alpha_ {L, i, m} ^ {(2)} & \text {i f} W _ {2, i, m} ^ {C L} \geq 0 \\ \alpha_ {U, i, m} ^ {(2)} & \text {i f} W _ {2, i, m} ^ {C L} <   0 \end{array} , \Delta_ {i, m} ^ {(2)} = \left\{ \begin{array}{l l} \beta_ {L, i, m} ^ {(2)} & \text {i f} W _ {2, i, m} ^ {C L} \geq 0 \\ \beta_ {U, i, m} ^ {(2)} & \text {i f} W _ {2, i, m} ^ {C L} <   0 \end{array} \right. \right. \tag {8}
$$
$$
\Lambda_ {i} ^ {(2)} = W _ {2, i} ^ {C L} \odot \lambda_ {i} ^ {(2)}, \tilde {\mathbf {W}} _ {i} ^ {(\mathbf {2})} = \sum_ {m = 1} ^ {d _ {2}} \Lambda_ {i, m} ^ {(2)} W _ {:, m} ^ {(2)}, \tilde {b} _ {i} ^ {(2)} = \sum_ {m = 1} ^ {d _ {2}} \Lambda_ {i, m} ^ {(2)} \left(b _ {m} ^ {(2)} + \Delta_ {i, m} ^ {(2)}\right), \tag {9}
$$
$$
\lambda_ {i, m} ^ {(1)} = \left\{ \begin{array}{l l} \alpha_ {L, i, m} ^ {(1)} & i f \tilde {W} _ {i, m} ^ {(2)} \geq 0 \\ \alpha_ {U, i, m} ^ {(1)} & i f \tilde {W} _ {i, m} ^ {(2)} <   0 \end{array} , \Delta_ {i, l} ^ {(1)} = \left\{ \begin{array}{l l} \beta_ {L, i, l} ^ {(1)} & i f \tilde {W} _ {i, l} ^ {(2)} \geq 0 \\ \beta_ {U, i, l} ^ {(1)} & i f \tilde {W} _ {i, l} ^ {(2)} <   0 \end{array} \right. \right. \tag {10}
$$
$$
\boldsymbol {\Lambda} ^ {(1)} = \tilde {\mathbf {W}} _ {i} ^ {(2)} \odot \lambda^ {(1)}, \tilde {\mathbf {W}} _ {i} ^ {(1)} = \sum_ {l = 1} ^ {d _ {1}} \Lambda_ {i, l} ^ {(1)} \mathbf {W} _ {:, l} ^ {(1)}, \tilde {\mathbf {b}} _ {i} ^ {(1)} = \sum_ {l = 1} ^ {d _ {2}} \Lambda_ {i, l} ^ {(1)} \left(b _ {m} ^ {(1)} + \Delta_ {i, l} ^ {(1)}\right). \tag {11}
$$
$\lambda_{i}^{(2)}, \pmb{\Delta}_{i}^{(2)}, \pmb{\Lambda}_{i}^{(2)} \in \mathbb{R}^{d_{2}}; \lambda_{i}^{(1)}, \pmb{\Delta}_{i}^{(1)}, \pmb{\Lambda}_{i}^{(1)} \in \mathbb{R}^{d_{1}}; \tilde{\mathbf{W}}_{i}^{(2)}, \tilde{\mathbf{W}}_{i}^{(1)} \in \mathbb{R}^{F \times 1}; \tilde{\mathbf{b}}^{(2)}, \tilde{\mathbf{b}}^{(1)} \in \mathbb{R}; d_{1}$  and  $d_{2}$  are the dimension of the embedding of layer 1 and layer 2 respectively.
We provide detailed proof in Appendix A.2. Theorem 2 demonstrates the process of deriving the node compactness  $\underline{f}_{G_{2,i}}$  in a bound propagation manner. Before Theorem 2, we obtain all the parameters needed, and we apply the bound propagation method [32] to our problem, which is the concatenated network of the GCN encoder  $f_{\theta}$  and a linear transformation  $W_{2,i}^{CL}$ . As the form of Eq. 6, the node compactness  $\underline{f}_{G_{2,i}}$  can be also seen as the output of a new "GCN" encoder with network parameters  $\tilde{\mathbf{W}}_i^{(1)}, \tilde{\mathbf{b}}^{(1)}, \tilde{b}^{(2)}$ , which is an encoder constructed from the GCN encoder in GCL. Additionally, we present the whole process of our POT method in Algorithm 1.
Time Complexity The computation of the weight matrices in Theorem 2 dominates the time complexity of POT. Thus, the time complexity of our algorithm is  $O( N d_{k - 1}d_k + Ed_k)$  for the  $k$ -th layer, which is the same as the message-passing [4, 10] in the GNN encoders.
We choose four GCL baseline models for evaluation: GRACE [35], GCA [36], ProGCL [25], and COSTA [34]. Since POT is a plugin for InfoNCE loss, we integrate POT with the baselines and
Algorithm 1: Provable Training for GCL
Input :Graph  $G = (\mathbf{X},\mathbf{A})$  , the set of all possible augmentations  $\mathcal{G}$  Output:Node embeddings  $\mathbf{Z}_1$  and  $\mathbf{Z}_2$    
1 Initialize GNN encoder parameters  $\{\mathbf{W}^{(1)},\mathbf{b}^{(1)},\mathbf{W}^{(2)},\mathbf{b}^{(2)}\}$  .   
2 while not converge do   
3 Generate graph augmentations  $G_{1} = (\mathbf{X},\tilde{\mathbf{A}}_{1})$  and  $G_{2} = (\mathbf{X},\tilde{\mathbf{A}}_{2})$  .   
4 Do forward pass, obtain  $\mathbf{Z}_1 = f_\theta (\mathbf{X},\tilde{\mathbf{A}}_1)$  and  $\mathbf{Z}_2 = f_\theta (\mathbf{X},\tilde{\mathbf{A}}_2)$  , calculate  $\mathcal{L}_{\mathrm{InfoNCE}}$    
5 Obtain the pre-activation bounds  $\{\mathbf{P}_L^{(k)},\mathbf{P}_U^{(k)}\}$ $k = 1,2$  as in Theorem 1;   
6 Set  $\{\alpha_L^{(k)},\alpha_U^{(k)},\beta_L^{(k)},\beta_U^{(k)}\}$ $k = 1,2$  according to Table 1;   
7 Derive  $\underline{\tilde{f}}_{G_1,i},\underline{\tilde{f}}_{G_2,i}$  as in Theorem 2 for  $i = 1$  to  $N$  and calculate  $\mathcal{L}_{\mathrm{POT}}$  as in Equation 4;   
8 Do backward propagation with  $\mathcal{L} = (1 - \kappa)\mathcal{L}_{\mathrm{InfoNCE}}(\mathbf{Z}_1,\mathbf{Z}_2) + \kappa \mathcal{L}_{\mathrm{POT}}(G_1,G_2)$    
9 end   
0 return  $\mathbf{Z}_1,\mathbf{Z}_2$  for downstream tasks;
construct four models: GRACE-POT, GCA-POT, ProGCL-POT, and COSTA-POT. Additionally, we report the results of two supervised baselines: GCN and GAT [20]. For the benchmarks, we choose 8 common datasets: Cora, CiteSeer, PubMed [28], Flickr, BlogCatalog [13], Computers, Photo [18], and WikiCS [14]. Details of the datasets and baselines are presented in Appendix C.2. For datasets with a public split available [28], including Cora, CiteSeer, and PubMed, we follow the public split; For other datasets with no public split, we generate random splits, where each of the training set and validation set contains  $10\%$  nodes of the graph and the rest  $80\%$  nodes of the graph is used for testing.
5.2 Node Classification
Table 2: Performance  $\left( {\%  \pm  \sigma }\right)$  on Node Classification
Datasets	Metrics	GCN	GAT	GRACE	GCA	ProGCL	COSTA
w/o POT	w/ POT	w/o POT	w/ POT	w/o POT	w/ POT	w/o POT	w/ POT
Cora	Mi-F1	81.9 ± 1.1	82.8 ± 0.5	78.2 ± 0.6	79.2 ± 1.2	78.6 ± 0.2	79.6 ± 1.4	77.9 ± 1.4	79.9 ± 1.0	79.9 ± 0.7	80.5 ± 0.9
Ma-F1	80.6 ± 1.1	81.8 ± 0.3	76.8 ± 0.6	77.8 ± 1.4	77.7 ± 0.1	79.1 ± 1.4	77.3 ± 1.7	78.8 ± 0.9	79.5 ± 0.7	80.1 ± 0.8
CiteSeer	Mi-F1	71.6 ± 0.6	72.1 ± 1.0	66.8 ± 1.0	68.7 ± 0.6	65.0 ± 0.7	69.0 ± 0.8	65.9 ± 0.9	67.7 ± 0.9	66.8 ± 0.8	67.9 ± 0.7
Ma-F1	68.2 ± 0.5	67.1 ± 1.3	63.2 ± 1.3	64.0 ± 0.9	60.8 ± 0.8	65.5 ± 0.7	62.5 ± 0.9	63.6 ± 0.5	63.4 ± 1.4	64.0 ± 1.2
PubMed	Mi-F1	79.3 ± 0.1	78.3 ± 0.7	81.6 ± 0.5	82.0 ± 1.3	80.5 ± 2.3	82.8 ± 2.0	81.5 ± 1.1	81.9 ± 0.4	80.0 ± 1.5	80.6 ± 1.3
Ma-F1	79.0 ± 0.1	77.6 ± 0.8	81.7 ± 0.5	82.4 ± 1.2	80.5 ± 2.1	82.9 ± 2.0	81.3 ± 1.0	82.2 ± 0.4	79.3 ± 1.4	80.3 ± 1.5
Flickr	Mi-F1	48.8 ± 1.9	37.3 ± 0.5	46.5 ± 1.8	48.4 ± 1.3	47.4 ± 0.8	50.3 ± 0.3	46.4 ± 1.4	49.7 ± 0.5	52.6 ± 1.2	53.8 ± 1.2
Ma-F1	45.8 ± 2.4	35.2 ± 0.7	45.1 ± 1.4	47.0 ± 1.6	46.2 ± 0.8	49.3 ± 0.4	45.4 ± 1.1	48.9 ± 0.3	51.2 ± 0.9	52.6 ± 1.3
BlogCatalog	Mi-F1	72.4 ± 2.5	68.0 ± 2.1	70.5 ± 1.3	72.4 ± 1.4	76.2 ± 0.6	76.5 ± 0.5	74.1 ± 0.6	76.0 ± 0.7	73.3 ± 3.4	75.1 ± 1.5
Ma-F1	71.9 ± 2.4	67.6 ± 1.8	70.2 ± 1.3	71.8 ± 1.4	75.8 ± 0.7	76.1 ± 0.5	73.7 ± 0.6	75.7 ± 0.9	73.0 ± 3.6	74.9 ± 1.6
Computers	Mi-F1	87.6 ± 0.6	88.7 ± 1.3	85.9 ± 0.4	86.9 ± 0.7	88.7 ± 0.6	89.1 ± 0.3	88.6 ± 0.7	89.0 ± 0.5	82.5 ± 0.4	82.8 ± 0.7
Ma-F1	83.1 ± 2.3	79.3 ± 1.4	83.9 ± 0.8	84.5 ± 1.4	87.1 ± 0.7	87.6 ± 0.5	86.9 ± 1.1	87.4 ± 0.9	78.8 ± 1.7	79.0 ± 1.0
Photo	Mi-F1	91.7 ± 0.7	92.0 ± 0.5	91.2 ± 0.4	91.8 ± 0.3	91.5 ± 0.4	92.0 ± 0.3	92.1 ± 0.3	92.4 ± 0.3	91.5 ± 0.6	92.2 ± 1.0
Ma-F1	88.8 ± 2.0	88.9 ± 2.2	89.2 ± 0.7	90.0 ± 0.6	89.5 ± 0.5	90.0 ± 0.6	90.6 ± 0.6	90.9 ± 0.6	89.3 ± 1.3	90.1 ± 1.4
WikiCS	Mi-F1	81.2 ± 1.2	81.2 ± 0.8	80.3 ± 0.3	80.6 ± 0.3	78.7 ± 0.3	80.7 ± 0.3	78.8 ± 0.4	80.0 ± 0.5	76.2 ± 1.6	79.1 ± 0.9
Ma-F1	78.7 ± 1.9	78.5 ± 1.8	77.6 ± 0.3	78.0 ± 0.3	75.4 ± 0.1	78.0 ± 0.3	75.6 ± 0.3	77.2 ± 0.8	72.2 ± 1.6	75.8 ± 0.8
We test the performance of POT on node classification. For the evaluation process, we follow the setting in baselines [35, 36, 25, 34] as follows: the encoder is trained in a contrastive learning manner without supervision, then we obtain the node embedding and use the fixed embeddings to train and evaluate a logistic regression classifier on the downstream node classification task. We choose two measures: Micro-F1 score and Macro-F1 score. The results with error bars are reported in Table 2.
Bolded results mean that POT outperforms the baseline, with a t-test level of 0.1. As it shows, our POT method improves the performance of all the base models consistently on all datasets, which verifies the effectiveness of our model. Moreover, POT gains significant improvements over baseline models on BlogCatalog and Flickr. These two datasets are the graphs with the highest average node degree, therefore there are more possible graph augmentations in these datasets (the higher node degrees are, the more edges are dropped [35, 36]). This result illustrates our motivation that POT improves existing GCL methods when faced with complex graph augmentations.
5.3 Analyzing the Node Compactness
To illustrate how POT improves the training of GCL, we conduct experiments to show some interesting properties of POT by visualizing the node compactness in Definition 2. More experiments can be found in Appendix D.
POT Improves Node Compactness. We illustrate that our POT does improve the node compactness, which is the goal of InfoNCE. The result on Cora is shown in Figure 2. The x-axis is the epoch number, and the y-axis is the average compactness of all nodes. We can see that the node compactness of the model with POT is almost always higher than the model without POT, indicating that POT increases node compactness in the training of GCL, i.e. how the nodes follow the GCL principle is improved in the worst-case. Therefore, our POT promotes nodes to follow the GCL principle better. More results on BlogCatalog are shown in Appendix D.
(c) ProGCL
Figure 2: Node compactness in the training process
(d) COSTA
Node Compactness Reflects Different Properties of Different Augmentation Strategies. There are two types of augmentation strategies: GRACE adapts a uniform edge dropping rate, in other words, the dropping rate does not change among different nodes; GCA, however, adapts a larger dropping rate on edges with higher-degree nodes, then higher-degree nodes have a larger proportion of edges dropped. Since node compactness is how a node follows the GCL principle in the worst case, a higher degree results in larger node compactness, while a higher dropping rate results in lower node compactness because worse augmentations can be taken. To see whether node compactness can reflect those properties of different augmentation as expected, we plot the average node compactness of the nodes with the same degree as scatters in Figure 3 on BlogCatalog and Flickr. For GRACE, node degree and compactness are positively correlated, since the dropping rate does not change as the degree increases; For GCA, node compactness reflects a trade-off between dropping rate and degree: the former dominates in the low-degree stage then the latter dominates later, resulting the curve to drop then slightly rise.
(a) GRACE, BlogCatalog
(b) GRACE, Flickr
Figure 3: Degree and node compactness score
(c) GCA, BlogCatalog
(d) GCA, Flickr
5.4 Hyperparameter Analysis
In this subsection, we investigate the sensitivity of the hyperparameter in POT,  $\kappa$  in Eq. 5. In our experiments, we tune  $\kappa$  ranging from  $\{0.5, 0.4, 0.3, 0.2, 0.1, 0.05\}$ . The hyperparameter  $\kappa$  balances between InfoNCE loss and POT, i.e. the importance of POT is strengthened when  $\kappa > 0.5$  while the InfoNCE is more important when  $\kappa < 0.5$ . We report the Micro-F1 score on node classification of BlogCatalog in Figure 4. Although the performance is sensitive to the choice of  $\kappa$ , the models with POT can still outperform the corresponding base models in many cases, especially on BlogCatalog where POT outperforms the base models whichever  $\kappa$  is selected. The result on Cora is in Appendix D.
(a) GRACE-POT
(b) GCA-POT
(c) ProGCL-POT
(d) COSTA-POT
Figure 4: Hyperparameter analysis on  $\kappa$
Graph Contrastive Learning. Graph Contrastive Learning (GCL) methods are proposed to learn the node embeddings without the supervision of labels. GRACE [35] firstly proposes GCL with random edge dropping and feature masking as data augmentations and takes InfoNCE as the objective. Based on GRACE, GCA [36] improves the data augmentation with adaptive masking and dropping rate related to node centrality. COSTA [34] proposes feature augmentation in embedding space to reduce sampling bias. ProGCL [25] reweights the negatives and empowers GCL with hard negative mining. There are also node-to-graph GCL methods, like DGI [21] and MVGRL [8]. Inspired by BYOL [6], there are also several GCL methods that do not need negative samples [19] or use BCE objectives [31].
Bound Propagation. Bound Propagation methods derive the output bounds directly from input bounds. Interval Bound Propagation [5] defines the input adversarial polytope as a  $l_{\infty}$ -norm ball, and it deforms as being propagated through affine layers and monotonic activations. [16] proposes a relaxation on nonlinearity via Semidefinite Programming but is restricted to one hidden layer. [24] relaxes ReLU via convex outer adversarial polytope. [32] can be applied to general activations bounded by linear functions. For GNN, [37] and [38] study the bound propagation on GNN with feature and topology perturbations respectively. One step further, [17] proposes a Mixed Integer Linear Programming based method to relax GNN certification with a shared perturbation as input. However, almost all the previous works require solving an optimization problem with some specific solver, which is time-consuming and impossible to be integrated into a training process as we expected.
In this paper, we investigate whether the nodes follow the GCL principle well enough in GCL training, and the answer is no. Therefore, to address this issue, we design the metric "node compactness" to measure the training of each node, and we propose POT as a regularizer to optimize node compactness, which can be plugged into the existing InfoNCE objective of GCL. Additionally, we provide a bound propagation-inspired method to derive POT theoretically. Extensive experiments and insightful visualizations verify the effectiveness of POT on various GCL methods.
Limitations and broader impacts. One limitation of our work is the restriction on the settings of GCL. To extend our work to more types of GCL settings and graph-level GCL, one can define the set of augmentations and network structure properly, then follow the proof in Appendix A.2 to derive the node compactness in a new setting. The detailed discussion can be found in Appendix B. We leave them for future work. Moreover, there are no negative social impacts foreseen.
[1] Deyu Bo, Xiao Wang, Chuan Shi, and Huawei Shen. Beyond low-frequency information in graph convolutional networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3950-3957, 2021.
[2] Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. arXiv preprint arXiv:2006.07988, 2020.
[3] Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
[4] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263-1272. PMLR, 2017.
[5] Sven Gowal, Krishnamurthy Dj Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. Scalable verified training for provably robust image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4842-4851, 2019.
[6] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
[7] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
[8] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International conference on machine learning, pages 4116-4126. PMLR, 2020.
[9] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXiv preprint arXiv:1905.12265, 2019.
[10] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017.
[11] Xiaorui Liu, Jiayuan Ding, Wei Jin, Han Xu, Yao Ma, Zitao Liu, and Jiliang Tang. Graph neural networks with adaptive residual. Advances in Neural Information Processing Systems, 34:9720-9733, 2021.
[12] Yaoqi Liu, Cheng Yang, Tianyu Zhao, Hui Han, Siyuan Zhang, Jing Wu, Guangyu Zhou, Hai Huang, Hui Wang, and Chuan Shi. Gammagl: A multi-backend library for graph neural networks. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2861-2870. ACM, 2023.
[13] Zaiqiao Meng, Shangsong Liang, Hongyan Bao, and Xiangliang Zhang. Co-embedding attributed networks. In Proceedings of the twelfth ACM international conference on web search and data mining, pages 393–401, 2019.
[14] Péter Mernyei and Cătălina Cangea. Wiki-cs: A wikipedia-based benchmark for graph neural networks. arXiv preprint arXiv:2007.02901, 2020.
[15] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
[33] Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in neural information processing systems, 31, 2018.
[34] Yifei Zhang, Hao Zhu, Zixing Song, Piotr Koniusz, and Irwin King. Costa: Covariance-preserving feature augmentation for graph contrastive learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2524-2534, 2022.
[35] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.04131, 2020.
[36] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In Proceedings of the Web Conference 2021, pages 2069-2080, 2021.
[37] Daniel Zügner and Stephan Gunnemann. Certifiable robustness and robust training for graph convolutional networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 246-256, 2019.
[38] Daniel Zügner and Stephan Gunnemann. Certifiable robustness of graph convolutional networks under structure perturbations. In KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1656-1665. ACM, 2020.
Proof. We mainly make use of a conclusion in [16] and express it as the lemma below:
Lemma 1 (Bound propagation of affine layers [16]). In an affine layer  $\mathbf{x}^k = \mathbf{W}^{k-1}\mathbf{x}^{k-1} + \mathbf{b}^{k-1}$ , the lower bound  $\mathbf{l}^k$  and upper bound  $\mathbf{u}^k$  of  $k$ -th layer after affine transformation can be obtained by
$$
\mathbf {l} ^ {k} = [ \mathbf {W} ^ {k - 1} ] _ {+} \mathbf {l} ^ {k - 1} + [ \mathbf {W} ^ {k - 1} ] _ {-} \mathbf {u} ^ {k - 1} + \mathbf {b} ^ {k - 1}, \mathbf {u} ^ {k} = [ \mathbf {W} ^ {k - 1} ] _ {+} \mathbf {u} ^ {k - 1} + [ \mathbf {W} ^ {k - 1} ] _ {-} \mathbf {l} ^ {k - 1} + \mathbf {b} ^ {k - 1},
$$
where  $[\mathbf{W}^k]_+ = \max(\mathbf{W}^k, 0)$  and  $[\mathbf{W}^k]_- = \min(\mathbf{W}^k, 0)$ ,  $\mathbf{W}^k$  is the weights of affine transformation in the  $k$ -th layer.
In our problem, since  $\hat{\mathbf{A}}$  is variable, we take  $\mathbf{H}^{(k)}\mathbf{W}^{(k)}$  as the weights of the affine transformation and replace  $\mathbf{l}^k$  and  $\mathbf{u}^k$  with the bounds  $\mathbf{L}$  and  $\mathbf{U}$ . To this end, Theorem 1 is proved.
Proof. Since  $\mathbf{H}^{(k)}$  is the output embedding of the  $k$ -th layer, which implies that  $\mathbf{H}^{(k-1)}$  is the input embedding of  $k$ -th layer.  $\mathbf{P}^{(k)}$  is the pre-activation embedding in Theorem 1,  $f_{\theta}(\mathbf{X}, A_1) = \sigma(\mathbf{P}^{(2)})$ ,  $\mathbf{P}^{(2)} = \hat{\mathbf{A}} \sigma(\mathbf{P}^{(1)}) \mathbf{W}^{(2)} + \mathbf{b}^{(2)}$ ,  $\mathbf{P}^{(1)} = \hat{\mathbf{A}} \mathbf{X} \mathbf{W}^{(1)} + \mathbf{b}^{(1)}$ . With the parameters  $\alpha, \beta, \mathbf{L}$ ,  $\mathbf{U}$  obtained, as well as the parameters of the GCN encoder  $\{\mathbf{W}^{(1)}, \mathbf{b}^{(1)}, \mathbf{W}^{(2)}, \mathbf{b}^{(2)}\}$  given, we can derive the lower bound  $\underline{f}_{G_2,i}$  in a bound propagation [32] manner, by taking  $\hat{\mathbf{A}}$  as a variable with element-wise upper and lower bound in Definition 4:
$$
\begin{array}{l} \underline {{\tilde {f}}} _ {G _ {2}, i} = \min  _ {\mathbf {\hat {A}}} \left\{\sigma \left(\mathbf {P} _ {i} ^ {(2)}\right) \cdot \mathbf {W} _ {2, i} ^ {C L} \right\} (12) \\ = \min  _ {\mathbf {A}} \left\{\sum_ {m = 1} ^ {d _ {2}} \sigma \left(\mathbf {P} _ {i, m} ^ {(2)}\right) W _ {2, i, m} ^ {C L} \right\} (13) \\ = \sum_ {W _ {2, i, m} ^ {C L} \geq 0} \left(\alpha_ {L, i, m} ^ {(2)} \left(p _ {i, m} ^ {(2)} + \beta_ {L, i, m} ^ {(2)}\right)\right) \cdot W _ {2, i, m} ^ {C L} + \sum_ {W _ {2, i, m} ^ {C L} <   0} \left(\alpha_ {U, i, m} ^ {(2)} \left(p _ {i, m} ^ {(2)} + \beta_ {U, i, m} ^ {(2)}\right)\right) \cdot W _ {2, i, m} ^ {C L} (14) \\ = \sum_ {m = 1} ^ {d _ {2}} W _ {2, i, m} ^ {C L} \lambda_ {i, m} ^ {(2)} \left(p _ {i, m} ^ {(2)} + \Delta_ {i, m} ^ {(2)}\right) (15) \\ = \sum_ {m = 1} ^ {d _ {2}} \Lambda_ {i, m} ^ {(2)} \left(p _ {i, m} ^ {(2)} + \Delta_ {i, m} ^ {(2)}\right) (16) \\ = \sum_ {m = 1} ^ {d _ {2}} \Lambda_ {i, m} ^ {(2)} \left(\sum_ {j _ {1} \in \mathcal {N} (i)} \hat {a} _ {j 1 i} \sigma \left(\mathbf {P} _ {i} ^ {(1)}\right) \mathbf {W} _ {:, m} ^ {(2)} + b _ {m} ^ {(2)} + \Delta_ {i, m} ^ {(2)}\right) (17) \\ = \sum_ {j _ {1} \in \mathcal {N} (i)} \hat {a} _ {j _ {1} i} \sigma \left(\mathbf {P} _ {i} ^ {(1)}\right) \tilde {\mathbf {W}} _ {i} ^ {(2)} + \tilde {b} _ {i} ^ {(2)} (18) \\ = \sum_ {j _ {1} \in \mathcal {N} (i)} \hat {a} _ {j _ {1} i} \sum_ {l = 1} ^ {d _ {1}} \tilde {\mathbf {W}} _ {i, l} ^ {(2)} \lambda_ {j _ {1}, l} ^ {(1)} \left(p _ {j _ {1}, l} ^ {(1)} + \Delta_ {j _ {1}, l} ^ {(1)}\right) + \tilde {b} _ {i} ^ {(2)} (19) \\ = \sum_ {j _ {1} \in \mathcal {N} (i)} \hat {a} _ {j _ {1} i} \sum_ {l = 1} ^ {d _ {1}} \Lambda_ {i, l} ^ {(1)} \left(p _ {j _ {1}, l} ^ {(1)} + \Delta_ {j _ {1}, l} ^ {(1)}\right) + \tilde {b} _ {i} ^ {(2)} (20) \\ = \sum_ {j _ {1} \in \mathcal {N} (i)} \hat {a} _ {j _ {1} i} \left(\sum_ {j _ {2} \in \mathcal {N} (j _ {1})} \hat {a} _ {j _ {2} j _ {1}} x _ {j _ {2}} \tilde {W} _ {i} ^ {(1)} + \tilde {b} ^ {(1)}\right) + \tilde {b} _ {i} ^ {(2)}. (21) \\ \end{array}
$$
(13) is the equivalent form of (12), and (14) is because: since  $\alpha_{L,i,m}^{(k)}(p_{i,m}^{(k)} + \beta_{L,i,m}^{(k)}) \leq \sigma(p_{i,m}^{(k)}) \leq \alpha_{U,i,m}^{(k)}(p_{i,m}^{(k)} + \beta_{U,i,m}^{(k)})$ , then if  $W_{2,i,m}^{CL}$  is multiplied to the inequality, we have
$$
\alpha_ {L, i, m} ^ {(k)} \left(p _ {i, m} ^ {(k)} + \beta_ {L, i, m} ^ {(k)}\right) W _ {2, i, m} ^ {C L} \leq \sigma \left(p _ {i, m} ^ {(k)}\right) W _ {2, i, m} ^ {C L} \leq \alpha_ {U, i, m} ^ {(k)} \left(p _ {i, m} ^ {(k)} + \beta_ {U, i, m} ^ {(k)}\right) W _ {2, i, m} ^ {C L} \quad \text {i f} W _ {2, i, m} ^ {C L} \geq 0, \tag {22}
$$