SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\alpha_ {U, i, m} ^ {(k)} \left(p _ {i, m} ^ {(k)} + \beta_ {U, i, m} ^ {(k)}\right) W _ {2, i, m} ^ {C L} \leq \sigma \left(p _ {i, m} ^ {(k)}\right) W _ {2, i, m} ^ {C L} \leq \alpha_ {L, i, m} ^ {(k)} \left(p _ {i, m} ^ {(k)} + \beta_ {L, i, m} ^ {(k)}\right) W _ {2, i, m} ^ {C L} \quad \text {i f} W _ {2, i, m} ^ {C L} <   0. \tag {23}
$$
In (15), we combine  $\alpha_{L}^{(2)}$  and  $\alpha_{U}^{(2)}$  into a single parameter  $\lambda_{i}^{(2)}$ , and similarly for  $\Delta_{i}^{(2)}$ . In (16),  $\lambda_{i}^{(2)}$  and  $\mathbf{W}_{2,i}^{CL}$  are combined to be  $\Lambda_{i}^{(2)}$ . We expand the expression of  $p_{i,m}^{(2)}$  to get (17). Since  $m$  is not related to  $j_{1}$ , we move the outer sum into the inner sum, and the parameters can be further combined, therefore we get (18). With the ideas above, the following equations can be derived, and finally, we get (21) which is exactly the form in Theorem 2.
B A Detailed Discussion on the Limitations
The definition of augmented adjacency matrix. In Definition 3, we present the definitions and theorems under the condition of edge dropping since it is the most common topology augmentation in GCL [35, 31, 19]. The proposed POT is suitable for various augmentation strategies including edge dropping and adding. POT is applicable as long as the range of augmentation is well-defined in Definition 3. For example, if edge addition is allowed, the first constraint in Definition 3 can be removed, resulting in the change of the element-wise bound in Definition 5. After the element-wise bound is defined, the other computation steps are the same, then the POT loss with edge addition can be derived. Since edge addition is not a common practice and all the baselines use edge dropping only, we set the first constraint in Definition 3 to obtain a tighter bound.
Applying POT to graph-level GCL. If the downstream task requires graph embeddings, POT still stands. First, related concepts like "graph compactness" can be defined similarly to node compactness. Second, the pooling operators like "MEAN", "SUM", and "MAX" are linear, therefore the form of "graph compactness" can be derived with some modifications on the steps in Appendix A.2.
Generalizing to Other Types of Encoders. Definition 5 and Theorem 2 are based on the assumption that the encoder is GCN since GCN is the most common choice in GCL. However, other encoders are possible to be equipped with POT. GraphSAGE [7] has a similar form to GCN, then the derivation steps are similar. For Graph Isomorphism Network [26] and its form is
$$
h _ {v} ^ {(k + 1)} = \operatorname {M L P} ((1 + \epsilon) h _ {v} ^ {(k)} + \sum_ {u \in \mathcal {N} (v)} h _ {u} ^ {(k)}).
$$
We can first relax the nonlinear activation functions in MLP with Definition 5, then obtain the pre-activation bounds with Theorem 1, and finally follow the steps in Appendix A.2 to derive the form of node compactness when a GIN encoder is applied.
The provable training of feature augmentation In this paper, we focus on topology augmentations. However, as we have mentioned in the limitation, feature augmentation is also a common class of augmentations. We explore the provable training for feature augmentation as follows. Unlike topology augmentation, the augmented feature is only the input of the first layer, so it is easier to deal with. Inspired by [37], we can relax the discrete manipulation into  $l - 1$  constraints, then the node compactness is related to the feasible solution of the dual form. A similar binary cross-entropy loss can be designed. More detailed derivations are still working in progress.
However, more efforts are still to be made to deal with some subtle parts. Future works are in process.
C.1 Source Code
The complete implementation can be found at https://github.com/VoidHarui/POT-GCL. We also provide an implementation based on GammaGL [12] at https://github.com/BUPT-GAMMA/GammaGL.
C.2 Datasets and Baselines
We choose 8 common datasets for evaluations: Cora, CiteSeer, and PubMed[28], which are the most common graph datasets, where nodes are paper and edges are citations; BlogCatalog and Flickr, which are social network datasets proposed in [13], where nodes represent users and edges represent social connections; Computers and Photo [18], which are Amazon co-purchase datasets with products as nodes and co-purchase relations as edges; WikiCS [14], which is a graph dataset based on Wikipedia in Computer Science domain. The statistics of the datasets are listed in Table 3. A percentage in the last 3 columns means the percentage of nodes used in the training/validation/testing phase.
Table 3: The statistics of datasets
Datasets	Nodes	Edges	Features	Classes	Training	Validation	Test
Cora	2708	10556	1433	7	140	500	1000
CiteSeer	3327	9104	3703	6	120	500	1000
PubMed	19717	88648	500	3	60	500	1000
BlogCatalog	5196	343486	8189	6	10%	10%	80%
Flickr	7575	479476	12047	9	10%	10%	80%
Computers	13752	491722	767	10	10%	10%	80%
Photo	7650	238162	645	8	10%	10%	80%
WikiCS	11701	216123	300	10	10%	10%	80%
We obtain the datasets from PyG [3]. Although the datasets are available for public use, we cannot find their licenses. The datasets can be found in the URLs below:
- Cora, CiteSeer, PubMed: https://github.com/kimiyoung/planetoid/raw/master/data
- BlogCatalog: https://docs.google.com/uc?export  $\equiv$  download&id  $= 178$  PqGqh67RUYMMP6-SoRHDoIBh8ku5FS&confirm  $\equiv$  t
- Flickr: https://docs.google.com/uc?export  $\equiv$  download&id  $=$  1tZp3EB20fAC27SYWwax66_8uGsuU62X&confirm  $\equiv$  t
- Computers, Photo: https://github.com/shchur/gnn-benchmark/raw/master/data/npz/
- WikiCS: https://github.com/pmernyei/wiki-cs-dataset/raw/master/dataset
For the implementations of baselines, we use PyG to implement GCN and GAT. For other GCL baselines, we use their original code. The sources are listed as follows:
- GCN: https://github.com/pyg-team/pytorch.Geometryical/blob/master/examples/gcn.py
GAT: https://github.com/pyg-team/pytorch.Geometryical/blob/master/examples/gat.py
- GRACE: https://github.com/CRIPAC-DIG/GRACE
GCA: https://github.com/CRIPAC-DIG/GCA
- ProGCL: https://github.com/junxia97/ProGCL
COSTA: https://github.com/yifeiacc/COSTA
We implement our POT based on the GCL baselines with PyTorch.
To make a fair comparison, we set the drop rate to be consistent when testing the performance gained by our proposed POT. The drop rates of edges  $(p_e^1, p_e^2)$  are given in Table 4:
Table 4: Hyperparameters:  $\left( {{p}_{e}^{1},{p}_{e}^{2}}\right)$
Models	Cora	CiteSeer	PubMed	Flickr	BlogCatalog	Computers	Photo	WikiCS
GRACE	(0.4, 0.3)	(0.3, 0.1)	(0.4, 0.1)	(0.4, 0.1)	(0.4, 0.3)	(0.6, 0.4)	(0.5, 0.3)	(0.2, 0.3)
GCA	(0.3, 0.1)	(0.2, 0.2)	(0.4, 0.1)	(0.3, 0.1)	(0.3, 0.1)	(0.6, 0.3)	(0.3, 0.5)	(0.2, 0.3)
ProGCL	(0.3, 0.1)	(0.4, 0.3)	(0.4, 0.1)	(0.4, 0.1)	(0.4, 0.2)	(0.6, 0.3)	(0.6, 0.3)	(0.2, 0.3)
COSTA	(0.2, 0.4)	(0.3, 0.1)	(0.4, 0.1)	(0.4, 0.1)	(0.4, 0.2)	(0.6, 0.3)	(0.3, 0.5)	(0.2, 0.3)
Additionally, we provide the values of the hyperparameter  $\kappa$  in our POT, as well as the temperature  $\tau$ , in Table 5:
Table 5: Hyperparameters:  $\left( {\tau ,\kappa }\right)$
Models	Cora	CiteSeer	PubMed	Flickr	BlogCatalog	Computers	Photo	WikiCS
GRACE-POT	(0.7, 0.4)	(0.9, 0.5)	(0.5, 0.1)	(0.3, 0.1)	(0.5, 0.3)	(0.3, 0.1)	(0.2, 0.4)	(0.3, 0.05)
GCA-POT	(0.6, 0.3)	(0.7, 0.5)	(0.3, 0.1)	(0.2, 0.5)	(0.3, 0.3)	(0.2, 0.2)	(0.6, 0.2)	(0.3, 0.1)
ProGCL-POT	(0.4, 0.4)	(0.5, 0.2)	(0.5, 0.3)	(0.3, 0.5)	(0.3, 0.2)	(0.3, 0.2)	(0.2, 0.3)	(0.4, 0.1)
COSTA-POT	(0.4, 0.1)	(0.3, 0.05)	(0.4, 0.3)	(0.2, 0.1)	(0.05, 0.4)	(0.1, 0.1)	(0.2, 0.3)	(0.4, 0.3)
We provide more hyperparameters which may also have effects on the performance. In Table 6, "num_epochs" is the epoch number for training, and "pot_batch" is the batch size of POT, "-1" means full batch is used.
Table 6: Hyperparameters: (pot_batch, num_epochs)
Models	Cora	CiteSeer	PubMed	Flickr	BlogCatalog	Computers	Photo	WikiCS
GRACE-POT	(-1, 200)	(-1, 200)	(256, 2500)	(512, 1000)	(-1, 1000)	(512, 2500)	(512, 0.4)	(512, 4000)
GCA-POT	(-1, 700)	(-1, 1000)	(256, 2500)	(1024, 2500)	(1024, 1500)	(256, 2500)	(1024, 2500)	(1024, 3500)
ProGCL-POT	(-1, 1000)	(-1, 700)	(256, 2000)	(512, 2000)	(1024, 1500)	(256, 3000)	(1024, 2500)	(256, 3000)
COSTA-POT	(-1, 600)	(-1, 1500)	(512, 2000)	(1024, 2500)	(1024, 2500)	(256, 2500)	(512, 3500)	(256, 3000)
C.4 Environment
The environment where our codes run is as follows:
- OS: Linux 5.4.0-131-generic
- CPU: Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz
GPU: GeForce RTX 3090
D.1 More Results for Section 3
We present more results of the imbalance of GCL training on Cora and Flickr shown in Figure 5. The conclusion is similar: it can be observed that the training of GCL on more datasets is also imbalanced on the untrained nodes.
D.2 More Results for Section 5.3
We provide more results of the experiments in Section 5.3: the node compactness score during training on BlogCatalog in Figure 6. From the results, we can see that the conclusion in Section 5.3 is the same on more datasets: our POT can promote node compactness compared to the original GCL methods, which can enforce the nodes to follow the GCL principle better.
D.3 More Results of Hyperparameter Analysis
The result of the hyperparameter analysis of  $\kappa$  on Cora is shown in Figure 7. POT outperforms the baseline models with most of the choices of  $\kappa$ , which implies that the proposed POT is robust to the change of  $\kappa$ .
(a) GRACE-Cora
(b) GCA-Cora
(c) ProGCL-Cora
(d) GRACE-Flickr
(e) GCA-Flickr
(f) ProGCL-Flickr
Figure 5: More results showing the imbalance of GCL training
(a) GRACE
Figure 6: Node compactness in the training process on BlogCatalog
(c) ProGCL
(a) GRACE-POT
Figure 7: Hyperparameter analysis of  $\kappa$  on Cora
(d) COSTA-POT
D.4 More Insights and Explanations of Experiments on Node Compactness
Node Compactness during training. In Figure 2 and Figure 6, we observe that the node compactness of traditional GCL methods may decrease with the increase of epoch. We think the "false negative" issue may contribute to the drop in the curves in Figure 2. Since GCL uses all  $N - 1$  nodes as negative samples, there are many false negatives, i.e., some nodes with the same class as the anchor node are treated as negative samples in the contrastive loss. As in the definition, the node compactness measures the worst case of how well a node behaves across all possible augmentations. Therefore, as
the epoch increases and some false negative information is captured, that "worst case" value may decrease. As a result, we observe a drop in node compactness in traditional GCL methods. There are some existing methods alleviating this, called "hard negative mining", including the baseline method ProGCL. From Figure 2, we see that the curve of ProGCL almost does not drop, while the average node compactness of other methods decreases. Since our POT method explicitly optimizes the node compactness, the curve will continue to rise as the epoch increases. It is an open question and may be related to fundamental issues in contrastive learning.
Understanding the result in Figure 3. First, edges are randomly dropped with a uniform rate in GRACE. Since there are fewer unimportant edges around a low-degree node, those informative edges are more easily dropped, which prevents the contrastive loss from learning useful semantics, therefore those low-degree nodes are hard to be well-trained in GRACE. This is also the motivation of GCA. To alleviate this, GCA sets the dropping rate of edges by node centrality, keeping important edges around low-degree nodes. That's why we observe a high node compactness in low-degree nodes. However, the augmentation strategy of GCA puts a higher dropping rate for high-degree nodes. This may explain why node compactness drops as the degree increases at first. When the degree comes to relatively large, the node is "strong" enough for this augmentation strategy and is prone to be well-trained. To conclude, the node degree and node compactness are positively correlated in GRACE, however, there is a trade-off between degree and node compactness in GCA. This may explain why the result in Figure 3 is reasonable.
D.5 More Experimental Results
POT can improve the InfoNCE loss. In Section 2, we investigate the problem of existing GCL training with InfoNCE, since it is the only metric we have. After that, we propose "node compactness" as a more appropriate and intrinsic metric to evaluate the training of a node in GCL. Therefore, we conduct experiments in the Figures to show the validity of node compactness as well as some properties. That is the reason why we did not provide further analysis of InfoNCE. However, it is still inspiring to investigate whether POT can improve InfoNCE. Intuitively, POT can improve the InfoNCE loss. POT has a larger regularization on the nodes that are not well-trained across all possible augmentations, as a result, the training of nodes under two specific augmentations is also improved. To illustrate this, we conduct an experiment comparing the InfoNCE loss values of the nodes with/without using POT, similar to the experiment in Section 2. We choose GRACE and GCA as baselines and show the result of Cora in Figure 8. It can be seen that InfoNCE is improved by POT in two aspects: the average InfoNCE of the nodes is reduced, and the distribution of InfoNCE values becomes more centralized and balanced.
(b) GCA
Figure 8: POT improves InfoNCE. The average of InfoNCE values is reduced and the loss values are more balanced.
The applicability of POT to different augmentation strategies. To investigate whether our POT applies to various graph augmentations, we evaluate the performance of POT under four types of topology augmentation: random edge dropping proposed in GRACE; node centrality-based topology
augmentations proposed in GCA, including degree centrality, eigenvector centrality, and PageRank centrality. The backbone of the GCL baseline is GRACE. The result on Cora and Flickr is given in Table 7. POT consistently outperforms the baseline models with different graph augmentations. As stated in the Limitation section, since more theoretical verifications are needed, POT of other augmentations, including feature augmentations, are working in progress. We are looking forward to more capability of POT.
Table 7: POT improves existing GCL with different augmentations (Random: random edge dropping with a uniform rate; Degree: degree-centrality dropping rate; Eigenvector: eigenvector-centrality dropping rate; PageRank: PageRank-centrality dropping rate).
Dataset	Method	Random	Degree	Eigenvector	PageRank
F1Mi	F1Ma	F1Mi	F1Ma	F1Mi	F1Ma	F1Mi	F1Ma
Cora	GCL	78.2 ± 0.6	76.8 ± 0.6	78.6 ± 0.2	77.7 ± 0.1	78.0 ± 0.8	77.3 ± 0.6	77.8 ± 1.1	77.0 ± 1.2
GCL+POT	79.2 ± 1.2	77.8 ± 1.4	79.6 ± 1.4	79.1 ± 1.4	79.1 ± 1.2	78.3 ± 1.5	79.1 ± 1.0	77.8 ± 1.4
Flickr	GCL	46.5 ± 1.8	45.1 ± 1.4	47.4 ± 0.8	46.2 ± 0.8	47.7 ± 1.2	46.6 ± 1.3	48.4 ± 1.0	47.3 ± 1.2
GCL+POT	49.8 ± 1.0	48.7 ± 0.8	50.3 ± 0.3	49.3 ± 0.4	49.7 ± 1.0	48.8 ± 1.1	50.2 ± 0.8	49.3 ± 1.0
Node compactness can reflect the nodes' sensitivity to augmentations. We conduct an experiment to show the relationship between a node's compactness and the standard error of its InfoNCE loss under different graph augmentations. Specifically, we first train a GCL model, then fix the network parameters and sample 500 pairs of different augmentations. After encoding these augmentations, 500 InfoNCE loss values are obtained for each node. We choose to calculate the standard error of these 500 values of each node to reflect its sensitivity to different augmentations. We also compute each node's compactness as Theorem 2 in the paper, which is not related to specific augmentations. With the process above, we have a data sample (nodeCompactness, InfoNCE_std) for each node. We divide these samples into several bins by the value of node compactness and calculate the average of InfoNCE_std in each bin. Finally, we show the result as a scatter plot in Figure 9. It can be seen that less sensitive nodes have higher node compactness values. To conclude, this experiment shows that our proposed node compactness can be a proxy of the node's sensitivity to different graph augmentations, as the motivation stated.
Figure 9: Node compactness and the standard error of InfoNCE loss under different augmentations. More sensitive nodes have lower node compactness as expected.