SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 10: Performance (ACC) of unsupervised learning for Ablation study. The ablation indicates  $\eta = 0$  in Eq. (14). The bold numbers denote the better performances of the same method.
methods	ablation	MUTAG	PROTEINS	DD	NCII	COLLAB	IMDB-B	REDDIT-B	REDDIT-M5K
InfoGraph	✓	89.67±1.54	75.26±1.43	74.13±1.49	78.21±1.35	71.46±1.21	73.87±1.32	84.76±1.86	54.57±1.38
×	90.13±2.05	76.12±1.72	75.76±1.64	79.36±1.57	72.67±1.95	74.96±1.49	84.53±1.79	55.12±1.47
GraphCL	✓	87.24±1.96	75.87±2.17	79.14±1.67	79.13±1.27	72.52±1.37	72.44±1.46	89.87±2.13	56.12±1.73
×	87.93±2.42	76.82±1.34	77.35±1.95	80.11±1.47	74.16±1.37	73.87±1.52	90.23±1.87	56.83±1.35
AD-GCL	✓	87.44±2.13	74.29±2.80	76.25±1.48	75.12±2.13	73.85±1.05	73.02±1.35	87.11±1.95	54.61±2.35
×	88.50±1.82	75.22±1.93	76.14±1.21	78.15±1.81	74.57±1.98	73.48±1.41	88.16±1.37	55.64±1.63
JOAOv2	✓	87.19±1.92	73.15±1.46	73.15±2.17	74.15±1.67	72.62±1.43	72.18±1.72	84.19±1.67	53.74±1.70
×	88.76±1.43	75.27±1.61	74.62±2.58	76.23±1.75	72.85±1.73	72.97±1.37	85.31±1.48	54.68±1.48
AutoGCL	✓	89.02±1.47	76.23±1.25	78.95±1.39	82.63±2.12	71.31±1.72	73.95±1.36	89.41±1.81	57.28±1.62
×	89.87±1.85	76.03±1.37	79.31±1.27	82.95±1.26	72.23±1.52	74.52±1.44	90.65±1.46	57.93±1.72
E Strict Lovász Principle
We use regularization in Lovász principle (9) instead of constraint because its optimization is much easier and its performance is very close to the constrained optimization. In Algorithm 1, we propose a constrained optimization for the "strict Lovász principle" via projection.
Algorithm 1: Constrained optimization for "strict Lovász principle"
1: Initialization:  $\mu = 1$    
2: repeat   
3:  $\hat{H}^t = F(A,X;\theta^t)$  and  $\hat{z}^t = f(A,X;\phi^t)$    
4:  $H^{t} = \mathrm{Proj}_{U}(\hat{H}^{t})$  and  $z^t = \frac{\hat{z}^t}{\|\hat{z}^t\|}$    
5: obtain  $\theta^{t + 1},\phi^{t + 1}$  by SGD updating   
6: until Convergence
In Algorithm 1, the  $\mathrm{Proj}_U$  project  $\hat{H}^t$  to the orthonormal representation space, which is similar to the Gram-Schmidt process. We define  $\mathrm{proj}_w(h) := \frac{<h,w>}{<w,w >} w$ . Let  $W_{k}$  be the set that  $W_{k} := \{w_{1},w_{2},\dots ,w_{k}\}$ . For each vertex  $i\in V$ , we denote  $\Omega_{i}$  as the set of vector  $w_{j}$ s where  $j$  can be each vertex not adjacent to vertex  $i$ . Then the  $H^{t} = \mathrm{Proj}_{U}(\hat{H}^{t})$  is defined in Algorithm 2
Algorithm 2: The definition of projection function  $\mathrm{Proj}_U$
$$
\begin{array}{l} 1: w _ {1} = \hat {h} _ {1} ^ {t}, e _ {1} = \frac {w _ {1}}{\| w _ {1} \|} \\ 2: w _ {2} = \hat {h} _ {2} ^ {t} - \sum_ {w \in W _ {2 - 1} \cap \Omega_ {2}} \operatorname {p r o j} _ {w} (\hat {h} _ {2} ^ {t}), e _ {2} = \frac {w _ {2}}{\| w _ {2} \|} \\ 3: \dots \\ 4: w _ {k} = \hat {h} _ {k} ^ {t} - \sum_ {w \in W _ {k - 1} \cap \Omega_ {k}} \operatorname {p r o j} _ {w} \left(\hat {h} _ {k} ^ {t}\right), e _ {k} = \frac {w _ {k}}{\| w _ {k} \|} \\ 5: \dots \\ 6: w _ {n} = \hat {h} _ {n} ^ {t} - \sum_ {w \in W _ {n - 1} \cap \Omega_ {n}} \operatorname {p r o j} _ {w} (\hat {h} _ {n} ^ {t}), e _ {n} = \frac {w _ {n}}{\| w _ {n} \|} \\ 7: \text {O u t p u t} H ^ {t + 1} = \left[ e _ {1}, e _ {2}, \dots , e _ {n} \right] ^ {\top} \\ \end{array}
$$
The comparisons between the regularized  $(\mu = 1)$  optimization and the constrained optimization for two methods on four datasets are as follows.
Table 11: Comparison between regularized optimization and constrained optimization
method	MUTAG	PROTEINS	DD	NCI1
regularized opt.	InfoGraph	89.67±1.54	75.26±1.43	74.13±1.49	78.21±1.35
regularized opt.	GraphCL	87.24±1.96	75.87±2.17	79.14±1.67	79.13±1.27
constrained opt.	InfoGraph	86.12±2.32	75.49±1.52	76.42±1.56	77.80±1.24
constrained opt.	GraphCL	87.52±2.75	76.11±1.36	78.54±2.21	77.63±1.58
F Time cost comparison
We compare the time cost between the InfoMax principle and Lovász principle using the InfoGraph Sun et al. [2019] model on different datasets. We run the programming on a machine with Intel 7 CPU and RTX 3090 GPU. We repeat the experiment five times and report the results in Table 12. The Lovász principle is the fastest method among the three. The Lovász principle is the fastest method
Table 12: Time cost of InfoGraph. h stands for hour and m stands for minute. The brown value indicates the lowest time cost.
tasks	principle	MUTAG	PROTEINS	DD	NCI1	COLLAB	IMDB-B	REDDIT-B	REDDIT-M5K
unsupervised learning	InfoMax	2.3 m	12.6 m	1 h 39 m	36.5 m	1 h 50 m	5.8 m	3 h 14 m	7 h 31 m
Lovasz	1.8 m	11.7 m	1 h 25 m	33.2 m	1 h 46 m	5.1 m	3 h 9 m	7 h 26 m
Enhanced Lovasz	1.8 m	12.2 m	1 h 30 m	34.3 m	1 h 47 m	5.3 m	3 h 10 m	7 h 27 m
semi-supervised learning	InfoMax	2.3 m	13.1 m	1 h 47 m	46.3 m	2 h 21 m	9.6 m	3 h 47 m	8 h 52 m
Lovasz	2.0 m	12.7 m	1 h 39 m	43.1 m	2 h 15 m	8.1 m	3 h 27 m	8 h 16 m
Enhanced Lovasz	2.0 m	12.8 m	1 h 40 m	42.6 m	2 h 17 m	8.3 m	3 h 29 m	8 h 20 m
among the three, and the reasons are as follows:
- The Lovász principle is faster than the InfoMax principle because the former does not use the  $f$ -GAN [Nowozin et al., 2016] and the Jensen-Shannon MI estimator  $I_{\varphi}$  to evaluate the mutual information.
- Enhanced Lovász principle is slightly lower than Lovász principle because the computation of  $1 / (\pmb{z}_i^{(t - 1)\top}\pmb{u}_p^{(t - 1)})^2$  for every  $p\in V_i$  was already done when computing  $\mathcal{L}_{\mathrm{Lo}}$ .