SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This paper proposed a novel method called Lovász principle for unsupervised graph-level representation learning. An extension using the subgraph Lovász number was also presented. The numerical results of unsupervised learning, semi-supervised learning, and transfer learning showed that the proposed methods are more effective than graph kernels and InfoMax principle based representation learning methods. Besides unsupervised representation learning, it is possible to apply our methods to other tasks such as graph-level clustering and graph generation. For instance, we can add a clustering module (e.g. [Xie et al., 2016]) to  $\mathcal{L}_{\mathrm{Lo}}$  to construct an end-to-end clustering algorithm. We can combine  $\mathcal{L}_{\mathrm{Lo}}$  with variational autoencoder [Kingma and Welling, 2013] to train a model to generate new graphs. Nevertheless, the implementation of these methods is out of the scope of this paper.
The authors appreciate the reviewers' and AC's comments and time.
Bijaya Adhikari, Yao Zhang, Naren Ramakrishnan, and B Aditya Prakash. Sub2vec: Feature learning for subgraphs. In Advances in Knowledge Discovery and Data Mining: 22nd Pacific-Asia Conference, PAKDD 2018, Melbourne, VIC, Australia, June 3-6, 2018, Proceedings, Part II 22, pages 170-182. Springer, 2018.
Yunsheng Bai, Hao Ding, Yang Qiao, Agustin Marinovic, Ken Gu, Ting Chen, Yizhou Sun, and Wei Wang. Unsupervised inductive graph-level representation learning via graph-graph proximity. arXiv preprint arXiv:1904.01098, 2019.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning, pages 531-540. PMLR, 2018.
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps and spectral techniques for embedding and clustering. Advances in neural information processing systems, 14, 2001.
Karsten M Borgwardt, Cheng Soon Ong, Stefan Schonauer, SVN Vishwanathan, Alex J Smola, and Hans-Peter Kriegel. Protein function prediction via graph kernels. Bioinformatics, 21(suppl_1):i47-i56, 2005.
Jinyu Cai, Yunhe Zhang, and Jicong Fan. Self-discriminative modeling for anomalous graph detection. arXiv preprint arXiv:2310.06261, 2023.
Ting Chen, Song Bian, and Yizhou Sun. Are powerful graph neural nets necessary? a dissection on graph classification. arXiv preprint arXiv:1905.04579, 2019.
Ganqu Cui, Jie Zhou, Cheng Yang, and Zhiyuan Liu. Adaptive graph encoder for attributed graph embedding. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 976-985, 2020.
Asim Kumar Debnath, Rosa L Lopez de Compadre, Gargi Debnath, Alan J Shusterman, and Corwin Hansch. Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. correlation with molecular orbital energies and hydrophobicity. Journal of medicinal chemistry, 34(2):786-797, 1991.
Kaize Ding, Zhe Xu, Hanghang Tong, and Huan Liu. Data augmentation for deep graph learning: A survey. ACM SIGKDD Explorations Newsletter, 24(2):61-77, 2022.
Yin Fang, Qiang Zhang, Haihong Yang, Xiang Zhuang, Shumin Deng, Wen Zhang, Ming Qin, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Molecular contrastive learning with chemical element knowledge graph. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3968-3976, 2022.
Bent Fuglede and Flemming Topsoe. Jensen-shannon divergence and hilbert space embedding. In International symposium on Information theory, 2004. ISIT 2004. Proceedings., page 31. IEEE, 2004.
Laura Galli and Adam N. Letchford. On the lovász theta function and some variants. Discrete Optimization, 25:159-174, 2017.
Thomas Gartner, Peter Flach, and Stefan Wrobel. On graph kernels: Hardness results and efficient alternatives. In Learning theory and kernel machines, pages 129-143. Springer, 2003.
A Comaprision with the equivalent forms of the Lovász principles
In this section, we propose an equivalent form of the Lovász principle based on the complement graph of  $G$ . We first introduce an equivalent definition of Lovász number as follows.
Definition A.1 (Lovász number Lovász [1979]). Let  $\bar{G}$  be the complement graph of  $G$  and  $\bar{\mathcal{U}}$  be the orthonormal representations of  $\bar{G}$ , Lovász proposed an equivalent definition of Lovász number as
$$
\vartheta (G) := \max  _ {\boldsymbol {d}, \boldsymbol {U} \in \bar {\mathcal {U}}} \sum_ {p \in V} \left(\boldsymbol {d} ^ {\top} \boldsymbol {u} _ {p}\right) ^ {2}. \tag {19}
$$
where  $\pmb{d} \in \mathbb{R}^{d}$  ranges over all unit vectors.
The unit vector  $\pmb{d}$  in the Eq. (19) is also a suitable representation vector for graph  $G$ . Thus we can obtain a Lovász loss  $\bar{\mathcal{L}}_{\mathrm{Lo}}$  based on Definition A.1 as follows.
$$
\bar {\mathcal {L}} _ {\mathrm {L o}} := \sum_ {i = 1} ^ {| \mathcal {G} |} \left(- \sum_ {p \in V _ {i}} \left(\left(\boldsymbol {z} _ {i} ^ {\phi}\right) ^ {\top} \boldsymbol {h} _ {p} ^ {\theta}\right) ^ {2}\right) + \mu \left(\left\| \bar {\boldsymbol {M}} _ {i} \odot \left(\boldsymbol {H} _ {i} ^ {\theta} \left(\boldsymbol {H} _ {i} ^ {\theta}\right) ^ {\top} - \boldsymbol {I} _ {n}\right) \right\| _ {F} ^ {2} + \left(\left(\boldsymbol {z} _ {i} ^ {\phi}\right) ^ {\top} \boldsymbol {z} _ {i} ^ {\phi} - 1\right) ^ {2}\right). \tag {20}
$$
where  $\bar{M}_i = A_i$  is a mask matrix for complement graph  $\bar{G}$ . Similar to the main paper, an enhanced Lovász loss based on Definition A.1 at iteration  $t$  is defined as
$$
\bar {\mathcal {L}} _ {\mathrm {E L o}} ^ {(t)} := \bar {\mathcal {L}} _ {\mathrm {L o}} ^ {(t)} + \lambda \bar {\mathcal {L}} _ {\mathrm {S L N}} ^ {(t)}, \tag {21}
$$
where  $\bar{\mathcal{L}}_{\mathrm{SLN}}^{(t)}$  is the subgraph Lovász number (SLN) loss of complement graph  $\bar{G}$  based on Lovász number definition (19).
We compare the Lovász loss  $\bar{\mathcal{L}}_{\mathrm{Lo}}$  and enhanced Lovász loss  $\bar{\mathcal{L}}_{\mathrm{ELo}}^{(t)}$  (based on  $d$ ) with the  $\mathcal{L}_{\mathrm{Lo}}$  and  $\mathcal{L}_{\mathrm{ELo}}^{(t)}$  (based on handle vector  $c$ ). The results are shown in Tables 6, 7 and 8. For convenience, we show the average performance of all methods over all datasets in Figure 3. We see the two equivalent forms of Lovász principles have very similar performance in the three learning tasks and both of them outperform the InfoMax principle.
Table 6: Performance of unsupervised learning. The **bold**, blue and green numbers denote the best, second best, and third best performances respectively, which also applies to Tables 7 and 8.
methods	MUTAG	PROTEINS	DD	NCI1	COLLAB	IMDB-B	REDDIT-B	REDDIT-M5K
Lovász principle(use \( \mathcal{L}_{\text{Lo}} \) and c)	InfoGraph	89.67±1.54	75.26±1.43	74.13±1.49	78.21±1.35	71.46±1.21	73.87±1.32	84.76±1.86	54.57±1.38
GraphCL	87.24±1.96	75.87±2.17	79.14±1.67	79.13±1.27	72.52±1.37	72.44±1.46	89.87±2.13	56.12±1.73
AD-GCL	87.44±2.13	74.29±2.80	76.25±1.48	75.12±2.13	73.85±1.05	73.02±1.35	87.11±1.95	54.61±2.35
JOAOv2	87.19±1.92	73.15±1.46	73.15±2.17	74.15±1.67	72.62±1.43	72.18±1.72	84.19±1.67	53.74±1.70
AutoGCL	89.02±1.47	76.23±1.25	78.95±1.39	82.63±2.12	71.31±1.72	73.95±1.36	89.41±1.81	57.28±1.62
Lovász principle(use \( \mathcal{L}_{\text{ELo}} \) and c)	InfoGraph	90.13±2.05	76.12±1.72	75.76±1.64	79.36±1.57	72.67±1.95	74.96±1.49	84.53±1.79	55.12±1.47
GraphCL	87.93±2.42	76.82±1.34	77.35±1.95	80.11±1.47	74.16±1.37	73.87±1.52	90.23±1.87	56.83±1.35
AD-GCL	88.50±1.82	75.22±1.93	76.14±1.21	78.15±1.81	74.57±1.98	73.48±1.41	88.16±1.37	55.64±1.63
JOAOv2	88.76±1.43	75.27±1.61	74.62±2.58	76.23±1.75	72.85±1.73	72.97±1.37	85.31±1.48	54.68±1.48
AutoGCL	89.87±1.85	76.03±1.37	79.31±1.27	82.95±1.26	72.23±1.52	74.52±1.44	90.65±1.46	57.93±1.72
Lovász principle(use \( \mathcal{L}_{\text{Lo}} \) and d)	InfoGraph	88.41±2.47	74.37±2.53	76.26±1.57	77.16±2.43	72.51±1.93	74.12±1.65	86.30±1.92	55.18±2.28
GraphCL	86.59±1.82	76.42±1.97	78.35±1.42	78.24±1.68	71.46±2.02	73.81±1.92	88.16±1.35	54.75±2.41
AD-GCL	87.36±1.67	75.10±2.33	76.77±2.10	76.67±1.51	74.24±2.17	73.14±1.51	88.21±2.41	55.42±1.31
JOAOv2	88.58±2.30	74.26±1.52	75.01±1.65	77.28±1.92	73.21±1.81	73.21±1.95	85.47±2.02	55.09±2.46
AutoGCL	88.25±2.11	75.78±2.31	77.16±1.91	80.16±1.07	73.11±2.21	74.64±2.07	87.57±2.14	56.83±2.18
Lovász principle(use \( \mathcal{L}_{\text{ELo}} \) and d)	InfoGraph	89.92±1.26	75.37±1.78	76.12±1.81	80.47±1.99	73.18±1.64	73.11±1.38	85.16±2.31	54.87±2.35
GraphCL	88.93±2.51	75.13±1.26	77.52±2.18	81.35±1.62	73.26±2.01	74.21±1.45	89.47±2.23	55.21±1.92
AD-GCL	89.38±1.79	76.72±2.09	75.33±1.95	80.02±2.51	74.18±1.44	74.09±1.50	87.36±1.52	56.39±2.32
JOAOv2	87.30±1.25	74.62±1.30	76.12±1.12	79.14±2.50	73.11±2.30	73.16±1.42	87.69±2.20	55.67±1.83
AutoGCL	89.31±2.01	75.23±1.46	79.87±1.34	82.71±1.34	74.20±1.28	73.90±1.67	90.47±2.52	58.17±1.68
B Experimental setting of transfer learning
Following [Hu et al., 2019; You et al., 2021; Yin et al., 2022], we compare the performance of transfer learning of the Lovász principle with the InfoMax principle. We use the Pretrain-GNN method [Hu et al., 2019] as a baseline and employ the Infomax, EdgePred, AttrMasking, and ContextPred pre-training strategies. The experimental settings followed those of AutoGCL [Yin et al., 2022]. Specifically, we performed supervised pre-training for 100 epochs on the ChEMBL dataset [Mayr et al., 2018; Gaulton et al., 2012], and then fine-tuned the model for 30 epochs on 8 chemistry evaluation subsets, using a classifier with a hidden size of 300. Our training employed a batch size of 256 and a learning rate of 0.001. We substitute the InfoMax loss in the four contrastive learning methods (GraphCL, AD-GCL, JOAOv2, and AutoGCL) with the Lovász loss  $\mathcal{L}_{\mathrm{Lo}}$  or  $\mathcal{L}_{\mathrm{ELo}}$ . We repeat each experiment 10 times and report the average ROC-AUC scores in Table 3 of the main paper.
Table 7: Performance of semi-supervised learning.
methods	NCII	PROTEINS	DD	COLLAB	REDDIT-B	REDDIT-M5K	GITHUB
Lovász(use Lloand c)	GraphCL	75.46±1.53	75.12±1.87	77.46±1.52	76.12±1.15	89.87±1.68	53.69±1.68	66.72±1.53
AD-GCL	76.62±1.83	74.21±1.71	78.27±1.39	76.27±1.74	90.36±1.56	54.06±1.32	65.32±1.04
JOAOv2	76.13±1.76	73.73±1.86	76.27±1.48	77.35±1.27	89.31±1.85	53.17±1.76	66.35±1.96
AutoGCL	75.77±1.48	76.36±1.57	78.16±1.61	77.63±1.78	84.64±2.53	51.31±1.81	64.87±1.62
Lovász(use Lelloand c)	GraphCL	75.81±1.68	75.88±1.67	78.43±1.48	77.57±1.58	90.67±1.27	54.81±1.73	67.04±1.45
AD-GCL	77.28±1.04	75.43±1.58	78.67±1.64	76.98±1.87	91.54±1.39	55.46±1.59	66.87±1.25
JOAOv2	76.25±1.59	74.67±1.37	77.96±1.86	78.84±1.75	90.25±1.22	54.32±1.89	67.52±1.73
AutoGCL	76.53±1.92	76.89±1.55	78.82±1.90	78.46±1.39	87.31±1.57	53.17±1.50	66.47±1.26
Lovász(use Lloand d)	GraphCL	76.21±1.80	76.67±2.14	78.20±1.64	78.26±2.47	90.24±2.13	54.71±1.97	65.91±2.43
AD-GCL	75.71±1.46	75.39±1.57	77.86±1.77	77.53±1.89	91.14±2.63	54.17±2.75	66.14±1.71
JOAOv2	76.40±2.32	74.61±1.28	77.26±2.01	76.45±1.34	88.27±1.89	54.34±2.53	65.41±2.76
AutoGCL	76.84±1.53	76.49±2.03	78.35±1.75	78.02±2.41	86.17±1.61	52.26±1.76	64.26±2.41
Lovász(use Lelloand d)	GraphCL	76.28±1.59	76.21±2.30	79.34±2.51	78.10±1.76	91.21±2.04	55.62±2.68	66.59±1.62
AD-GCL	77.65±2.17	75.52±1.97	78.71±1.93	77.29±2.33	90.76±2.18	55.14±1.73	67.31±2.41
JOAOv2	76.19±2.30	75.41±2.24	78.03±2.58	79.31±1.49	91.34±2.11	53.70±2.41	66.35±2.28
AutoGCL	75.42±1.87	77.25±2.67	77.54±1.26	78.27±2.43	88.69±2.63	54.28±1.64	66.19±1.67
Table 8: Performance of transfer learning.
methods	BBBP	Tox21	ToxCast	SIDER	ClinTox	MUV	HIV	BACE
Lovász(use Lloand c)	GraphCL	71.37±1.74	75.66±1.82	63.35±1.47	62.11±1.35	77.02±1.67	72.25±1.42	79.23±1.43	78.51±1.58
AD-GCL	72.24±1.89	77.52±1.74	63.56±1.36	63.87±1.53	80.35±2.36	74.42±1.57	78.95±2.21	80.17±1.04
JOAOv2	72.16±1.35	75.86±1.21	63.92±1.52	62.56±1.12	81.26±2.37	75.94±1.38	79.01±2.68	79.82±1.39
AutoGCL	73.79±1.41	76.13±1.48	64.21±1.58	63.24±1.51	81.32±2.12	76.04±1.87	78.64±1.95	82.57±1.95
Lovász(use Lel0and c)	GraphCL	73.05±1.21	76.45±1.35	64.58±1.73	63.72±1.52	80.21±2.31	74.43±1.95	80.37±1.52	80.63±1.63
AD-GCL	72.48±1.59	77.96±1.71	64.27±1.68	63.91±1.74	81.76±2.01	75.88±1.48	81.08±2.35	82.21±1.49
JOAOv2	74.13±1.26	76.21±1.35	64.81±1.92	63.38±1.89	82.75±2.69	76.51±1.53	81.13±1.96	81.34±1.35
AutoGCL	74.67±1.81	76.97±1.76	65.36±1.45	64.13±1.48	82.13±2.41	76.93±1.62	79.56±1.41	83.57±1.30
Lovász(use L0and d)	GraphCL	72.53±2.16	75.27±1.35	64.20±1.82	63.16±2.25	79.17±1.63	74.53±2.28	78.61±1.55	80.43±1.77
AD-GCL	71.45±1.77	76.48±2.03	63.78±2.21	62.85±2.31	79.33±2.49	73.19±1.22	79.42±1.43	78.26±1.47
JOAOv2	71.78±2.27	75.19±1.87	64.26±1.65	63.27±1.80	81.46±1.83	74.39±2.48	79.59±2.17	80.14±2.51
AutoGCL	72.52±2.30	76.46±2.33	63.81±2.53	62.84±1.34	80.91±2.27	75.57±2.16	79.35±1.42	81.66±1.60
Lovász(use Lel0and d)	GraphCL	74.54±2.23	76.32±2.54	65.62±1.74	64.34±1.86	81.75±1.61	75.20±2.31	81.64±2.37	81.27±1.58
AD-GCL	71.74±2.81	77.85±1.95	64.81±2.52	64.25±2.57	80.54±1.32	75.34±1.76	80.56±1.64	81.65±2.48
JOAOv2	73.71±1.68	76.43±1.64	64.92±2.17	63.56±1.67	81.96±1.58	76.78±1.84	81.53±2.42	80.76±2.31
AutoGCL	74.69±2.10	76.21±1.81	65.13±1.83	63.77±2.52	82.47±2.52	75.42±2.25	80.74±1.56	82.89±2.68
(c) Transfer learning
Figure 3: The average performance of different types of methods
C Parameter sensitivity analysis
There are five hyperparameters need to be tuned in Lovász principle method: the dimension of representations  $d$ , the hyperparameter of orthonormal representation regularization  $\mu$ , the hyperparameter of orthogonal regularization in subgraph Lovász number (SLN) loss  $\gamma$ , hyperparameter of subgraph Lovász number (SLN) loss in enhanced Lovász loss  $\eta$ , the hyperparameter of graph-level representations  $\ell_2$ -norm regularization in semi-supervised Lovász loss  $\lambda$ . In this section, we analyze the parameter sensitivity on the InfoGraph Sun et al. [2019] with different hyperparameters. We repeat each experiment for ten times and plot the average accuracy on different datasets.
C.1  $d$  as the dimension of representations
In Lovász principle,  $d$  is the dimension of node-level representations  $\pmb{H} = [h_1,\dots,h_n]^\top \in \mathbb{R}^{n\times d}$  and graph-level representation of  $G$ ,  $\pmb{h}_v\in \mathbb{R}^d$ . In the definition of Lovász number, the node-level representations are orthonormal representations, i.e.,  $\pmb{H}^{\top}\in \mathcal{U}$ . Let  $\alpha (G)$  be the independent number of graph  $G$ , which is the size of the maximum independent set. If  $d\leq \alpha (G)$ , the node-level representations  $\pmb{H}$  are impossible to be orthonormal representations such that the Lovász number cannot be obtained. In Figure 4, we fix other hyperparameters and tune  $d$  from  $\{10,20,\dots,90,100\}$ .
(b) PROTEIN
(c) DD
(e) COLLAB
(f)IMDB-B
(g) REDDIT-B
Figure 4: The average ACC of different  $d$  on different dataset
(d) NCI1
(h) REDDIT-M5
The results show that a small  $d$  adversely affects the performance because  $d$  may be less than  $\alpha(G)$  on some graphs. When  $d$  is too large, the average accuracy decreases slightly because the representations with large  $d$  may capture some noisy information of a graph.
C.2  $\mu$  for orthonormal representation regularization
(e) COLLAB
Figure 5: The average ACC of different  $\mu$  on different data
(b) PROTEIN
(f)IMDB-B
(c) DD
(g) REDDIT-B
In Lovász principle,  $\mu$  is the hyperparameter for orthonormal representation regularization in Lovász loss  $\mathcal{L}_{\mathrm{Lo}}(9)$  as follows
$$
\mathcal {L} _ {\mathrm {L o}} := \sum_ {i = 1} ^ {| \mathcal {G} |} \max  _ {p \in V _ {i}} \frac {1}{\left(\left(\boldsymbol {z} _ {i} ^ {\phi}\right) ^ {\top} \boldsymbol {h} _ {p} ^ {\theta}\right) ^ {2}} + \mu \left(\left\| \boldsymbol {M} _ {i} \odot \left(\boldsymbol {H} _ {i} ^ {\theta} \left(\boldsymbol {H} _ {i} ^ {\theta}\right) ^ {\top} - \boldsymbol {I} _ {n}\right) \right\| _ {F} ^ {2} + \left(\left(\boldsymbol {z} _ {i} ^ {\phi}\right) ^ {\top} \boldsymbol {z} _ {i} ^ {\phi} - 1\right) ^ {2}\right). \tag {22}
$$
In Figure 5, we fix other hyperparameters and tune  $\mu$  from  $\{10^{-3}, 10^{-2}, \dots, 10^{5}, 10^{6}\}$ . The results show that  $\mu$  is not sensitive when  $0.1 \leq \mu \leq 1e3$ . If  $\mu$  is too small, the average accuracy decreases slightly because the node-level representations  $\pmb{H}$  may not be orthonormal representations. A very large  $\mu$  adversely affects the performance because the orthonormal representation regularization dominates the representation learning such that the Lovász principle fails.
(g) REDDIT-B
Figure 6: The average ACC of different  $\gamma$  on different data
C.3  $\gamma$  for orthogonal regularization in subgraph Lovász number (SLN) loss
In Lovász principle,  $\gamma$  is the hyperparameter for orthogonal regularization in subgraph Lovász number (SLN) loss  $\mathcal{L}_{\mathrm{SLN}}^{(t)}$  as follows
$$
\mathcal {L} _ {\mathrm {S L N}} ^ {(t)} := \sum_ {i = 1} ^ {| \mathcal {G} |} \sum_ {j = 1} ^ {| \mathcal {G} |} K _ {i j} ^ {(t - 1)} \| \boldsymbol {z} _ {i} ^ {\phi} - \boldsymbol {z} _ {j} ^ {\phi} \| _ {2} ^ {2} + \gamma (\| \boldsymbol {Z} _ {\phi} ^ {\top} \boldsymbol {Z} _ {\phi} - \boldsymbol {I} _ {d} \| _ {F} ^ {2} + \| \boldsymbol {Z} _ {\phi} ^ {\top} \mathbf {1} _ {N \times 1} \| _ {2} ^ {2}), \tag {23}
$$
In Figure 6, we fix other hyperparameters and tune  $\gamma$  from  $\{10^{-3}, 10^{-2}, \dots, 10^{5}, 10^{6}\}$ . The results show that  $\gamma$  is not sensitive when  $0.1 \leq \gamma \leq 1e3$ . If  $\gamma$  is too small, the average accuracy decreases slightly because the orthogonal constraints of spectral embedding may not hold. A very large  $\gamma$  adversely affects the performance because the orthogonal regularization of spectral embedding dominates the representation learning such that the Lovász principle fails.
C.4  $\eta$  for subgraph Lovász number (SLN) loss in enhanced Lovász loss
(d) NCI1
(e) COLLAB
Figure 7: The average ACC of different  $\eta$  on different data
(g) REDDIT-B
(h) REDDIT-M5
In Lovász principle,  $\eta$  is the hyperparameter for subgraph Lovász number (SLN) loss  $\mathcal{L}_{\mathrm{SLN}}^{(t)}$  in enhanced Lovász loss  $\mathcal{L}_{\mathrm{ELo}}^{(t)}$  as follows
$$
\mathcal {L} _ {\mathrm {E L o}} ^ {(t)} := \mathcal {L} _ {\mathrm {L o}} ^ {(t)} + \eta \mathcal {L} _ {\mathrm {S L N}} ^ {(t)}.
$$
In Figure 7, we fix other hyperparameters and tune  $\eta$  from  $\{10^{-3}, 10^{-2}, \dots, 10^{5}, 10^{6}\}$ . The results show that  $\eta$  is not sensitive when  $10^{-2} \leq \eta \leq 1e4$ . If  $\eta$  is a very small number, the enhanced Lovász loss  $\mathcal{L}_{\mathrm{ELo}}^{(t)}$  degenerates into the Lovász loss  $\mathcal{L}_{\mathrm{Lo}}$ , which also performs well in representation learning. A very large  $\eta$  adversely affects the performance because the subgraph Lovász number (SLN) loss  $\mathcal{L}_{\mathrm{SLN}}^{(t)}$  dominates the representation learning such that the Lovász principle may fail.
C.5  $\lambda$  for the  $\ell_2$ -norm regularization in semi-supervised Lovász loss
(h) REDDIT-M5
Figure 8: The average ACC of different  $\lambda$  on different data
In semi-supervised learning,  $\lambda$  the hyperparameter for the  $\ell_2$ -norm regularization in semi-supervised Lovász loss as follows
$$
\mathcal {L} _ {\text {L o - s e m i}} := \sum_ {l = 1} ^ {| \mathcal {G} ^ {L} |} \mathcal {L} _ {\text {s u p e r v i s e d}} \left(\hat {\boldsymbol {y}} _ {l} ^ {\psi}, \boldsymbol {y} _ {l}\right) + \mathcal {L} _ {\text {u n s u p e r v i s e d}} \left(\boldsymbol {H} _ {i} ^ {\theta}, \boldsymbol {z} _ {i} ^ {\phi}\right) + \lambda \sum_ {i = 1} ^ {| \mathcal {G} |} \| \boldsymbol {z} _ {i} ^ {\phi} - \boldsymbol {z} _ {i} ^ {\psi} \| _ {2} ^ {2}, \tag {24}
$$
In Figure 8, we fix other hyperparameters and tune  $\lambda$  from  $\{10^{-3}, 10^{-2}, \dots, 10^{5}, 10^{6}\}$ . The results show that  $\eta$  is not sensitive when  $0.1 \leq \gamma \leq 1e3$ . If  $\lambda$  is a very small number, the supervised encoder and unsupervised encoder may learn different information of a graph  $G$  such that the average accuracy slightly decreases. A very large  $\eta$  will cause the training to be trapped in early iterations such that the representation learning fails.
D Ablation study
In this section, we analyze the importance of the orthonormal representation regularization (Eq. (9)) and the subgraph Lovász number (SLN) loss (Eq. (14)) by ablation study on unsupervised learning.
D.1 Ablation study of orthonormal representation regularization
In the ablation study, we remove the orthonormal representation regularization in Eq. (9) by setting  $\mu = 0$ . The results in Table 9 show that the orthonormal representation regularization can improve the performance of graph representation learning for two reasons:
- The orthonormal representation constraint is part of the definition of Lovász number such that the Lovász principle may fail without the orthonormal representation regularization.
- The orthonormal representation regularization guides the method the learn the structure property of a graph.
Table 9: Performance (ACC) of unsupervised learning for Ablation study. The ablation indicates  $\mu = 0$  in Eq. (9). The **bold** numbers denote the better performances of the same method.
methods	ablation	MUTAG	PROTEINS	DD	NCII	COLLAB	IMDB-B	REDMIT-B	REDMIT-M5K
Lovász principle(use LLo)	InfoGraph	✓	88.25±1.63	74.38±1.52	73.05±1.77	76.89±1.42	72.23±1.35	72.09±1.41	83.56±1.22	53.21±1.90
×	89.67±1.54	75.26±1.43	74.13±1.49	78.21±1.35	71.46±1.21	73.87±1.32	84.76±1.86	54.57±1.38
GraphCL	✓	86.16±1.31	73.48±1.56	78.22±1.09	77.84±1.73	71.65±1.20	71.82±1.78	87.79±1.86	55.03±2.17
×	87.24±1.96	75.87±2.17	79.14±1.67	79.13±1.27	72.52±1.37	72.44±1.46	89.87±2.13	56.12±1.73
AD-GCL	✓	86.28±1.84	73.52±1.78	75.89±1.67	74.53±1.36	72.41±1.58	72.84±1.52	88.46±1.35	53.27±1.51
×	87.44±2.13	74.29±2.80	76.25±1.48	75.12±2.13	73.85±1.05	73.02±1.35	87.11±1.95	54.61±2.35
JOAOv2	✓	88.34±1.36	72.49±1.19	72.53±2.07	73.64±2.52	72.27±1.59	71.63±1.81	83.48±1.31	53.02±1.21
×	87.19±1.92	73.15±1.46	73.15±2.17	74.15±1.67	72.62±1.43	72.18±1.72	84.19±1.67	53.74±1.70
AutoGCL	✓	88.76±2.15	75.59±2.03	78.42±1.81	81.79±1.65	70.42±1.30	73.64±1.52	88.31±1.22	56.53±1.27
×	89.02±1.47	76.23±1.25	78.95±1.39	82.63±2.12	71.31±1.72	73.95±1.36	89.41±1.81	57.28±1.62
Lovász principle(use LELo)	InfoGraph	✓	89.51±1.63	75.18±1.67	76.29±1.52	78.24±1.61	71.22±1.43	74.35±1.07	83.87±2.39	56.23±1.15
×	90.13±2.05	76.12±1.72	75.76±1.64	79.36±1.57	72.67±1.95	74.96±1.49	84.53±1.79	55.12±1.47
GraphCL	✓	86.78±1.57	76.24±1.60	76.71±1.63	79.42±1.03	73.37±1.26	73.42±1.07	89.70±1.52	55.64±1.25
×	87.93±2.42	76.82±1.34	77.35±1.95	80.11±1.47	74.16±1.37	73.87±1.52	90.23±1.87	56.83±1.35
AD-GCL	✓	88.36±1.51	74.63±1.74	76.39±1.37	77.68±1.23	74.03±1.52	72.73±1.22	87.31±1.21	55.87±1.42
×	88.50±1.82	75.22±1.93	76.14±1.21	78.15±1.81	74.57±1.98	73.48±1.41	88.16±1.37	55.64±1.63
JOAOv2	✓	88.27±1.39	74.52±1.40	74.31±1.15	75.78±1.43	72.10±1.62	72.39±2.15	84.45±1.36	54.21±1.52
×	88.76±1.43	75.27±1.61	74.62±2.58	76.23±1.75	72.85±1.73	72.97±1.37	85.31±1.48	54.68±1.48
AutoGCL	✓	89.24±2.43	75.29±1.58	78.42±1.68	82.31±1.79	71.69±1.98	74.46±1.65	90.10±1.73	57.27±1.54
×	89.87±1.85	76.03±1.37	79.31±1.27	82.95±1.26	72.23±1.52	74.52±1.44	90.65±1.46	57.93±1.72
D.2 Ablation study of subgraph Lovász number (SLN) loss
If we remove the subgraph Lovász number (SLN) loss in Eq. (14) by setting  $\eta = 0$ , the enhanced Lovász loss (14) degenerates to simple Lovász loss (9). The results in Table 10 show that the subgraph Lovász number (SLN) loss can improve the performance of graph representation learning because similar graphs are guaranteed to be close to each other in the representation space.
D.3 Ablation study of orthonormal representation regularization