SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} _ {\text {L o - s e m i}} := \sum_ {l = 1} ^ {| \mathcal {G} ^ {L} |} \ell_ {\text {s u p e r v i s e d}} \left(\hat {\boldsymbol {y}} _ {l} ^ {\psi}, \boldsymbol {y} _ {l}\right) + \mathcal {L} _ {\text {u n s u p e r v i s e d}} (\mathcal {G}) + \lambda \sum_ {i = 1} ^ {| \mathcal {G} |} \left\| \boldsymbol {z} _ {i} ^ {\phi} - \boldsymbol {z} _ {i} ^ {\psi} \right\| _ {2} ^ {2}, \tag {10}
$$
where  $\lambda$  is a positive hyperparameter, the supervised loss  $\ell_{\mathrm{supervised}}$  is the cross-entropy loss, and the unsupervised loss  $\mathcal{L}_{\mathrm{unsupervised}}$  is the Lovász loss  $\mathcal{L}_{\mathrm{Lo}}$  (Eq. (9)) or the enhanced Lovász loss  $\mathcal{L}_{\mathrm{ELo}}$  (Eq. (14)). The last term encourages the representations learned by the two encoders to be similar.
4 Enhancing Lovász Principle with Subgraph Lovász Number
Lovász principle does not explicitly utilize the Lovász number in graph embedding, though the Lovász numbers of subgraphs can be useful in comparing graphs [Johansson et al., 2014]. Therefore, we propose to use subgraph Lovász number to enhance Lovász principle based graph representation learning. We may consider taking advantage of the Lovász-  $\vartheta$  kernel proposed by [Johansson et al., 2014]. However, we encounter the following two difficulties.
i. Computing the Lovász numbers (3) of subgraphs is time-consuming because we need to solve (2) for every graph and the number of subgraphs of each graph is often very large (up to  $2^{|V|}$ ). Hence, for large graph datasets, we cannot use (4) directly.
ii. The Lovász- $\vartheta$  kernel (4) is a pair-wise method and cannot effectively exploit the global structure of  $\mathcal{G}$ .
To solve the aforementioned problems, we present an iterative-refinement strategy that computes the subgraph Lovász numbers using the embeddings given by the Lovász principle. Specifically, at iteration  $t$ , we have the graph-level representations  $Z_{\phi}^{(t - 1)}$  and the node-level representations  $\mathcal{H}_{\theta}^{(t - 1)}$  given by iteration  $t - 1$ . Inspired by the Lovász-  $\vartheta$  kernel (4), we compute the similarity between graph  $G_{i}$  and  $G_{j}$  as
$$
K _ {i j} ^ {(t - 1)} = \sum_ {S _ {i} \subseteq V _ {i}} \sum_ {S _ {j} \subseteq V _ {j}} \frac {\delta (| S _ {i} | , | S _ {j} |)}{C _ {S _ {i} , S _ {j}}} k \left(\vartheta_ {S _ {i}} ^ {(t - 1)} \left(G _ {i}\right), \vartheta_ {S _ {j}} ^ {(t - 1)} \left(G _ {j}\right)\right), \tag {11}
$$
where  $C_{S_i, S_j} = \binom{|V_i|}{|S_i|} \binom{|V_j|}{|S_j|}$  and  $\vartheta_{S_i}^{(t-1)}(G_i)$  (similar for  $G_j$ ) is obtained by
$$
\vartheta_ {S _ {i}} ^ {(t - 1)} (G _ {i}) = \max  _ {p \in S _ {i}} \frac {1}{\left(\boldsymbol {z} _ {i} ^ {(t - 1) \top} \boldsymbol {h} _ {p} ^ {(t - 1)}\right) ^ {2}}. \tag {12}
$$
The computation of  $1 / (z_i^{(t - 1)\top}h_p^{(t - 1)})^2$  for every  $p\in V_{i}$  was already done when computing  $\mathcal{L}_{\mathrm{Lo}}$  via (9) at iteration  $t - 1$  and there is no need to solve (3). For (11), we do not need to consider all possible subgraphs and we can just randomly sample subgraphs with some fixed sizes (numbers of nodes), which is similar to the truncated Lovász-  $\vartheta$  kernel of [Johansson et al., 2014]. Thus we can obtain the similarity  $K_{ij}^{(t - 1)}$  efficiently. Adapting the idea of spectral embedding [Belkin and Niyogi, 2001], we propose the following subgraph Lovász number (SLN) loss (at iteration  $t$ )
$$
\mathcal {L} _ {\mathrm {S L N}} ^ {(t)} := \sum_ {i = 1} ^ {| \mathcal {G} |} \sum_ {j = 1} ^ {| \mathcal {G} |} K _ {i j} ^ {(t - 1)} \left\| \boldsymbol {z} _ {i} ^ {\phi} - \boldsymbol {z} _ {j} ^ {\phi} \right\| _ {2} ^ {2} + \gamma \left(\left\| \boldsymbol {Z} _ {\phi} ^ {\top} \boldsymbol {Z} _ {\phi} - \boldsymbol {I} _ {d} \right\| _ {F} ^ {2} + \left\| \boldsymbol {Z} _ {\phi} ^ {\top} \mathbf {1} _ {N \times 1} \right\| _ {2} ^ {2}\right), \tag {13}
$$
where  $\gamma > 0$ . The two regularization terms in  $\mathcal{L}_{\mathrm{SLN}}^{(t)}$  aim to make the graph-level representations orthonormal and centered, which is consistent with the constraints in spectral embedding. Minimizing  $\mathcal{L}_{\mathrm{SLN}}^{(t)}$  encourages that the graph-level representations of similar graphs (in the sense of subgraph Lovász numbers) are closer to each other at iteration  $t$ . Integrating (13) with (9), we obtain the following enhanced Lovász loss at iteration  $t$
$$
\mathcal {L} _ {\mathrm {E L o}} ^ {(t)} := \mathcal {L} _ {\mathrm {L o}} ^ {(t)} + \eta \mathcal {L} _ {\mathrm {S L N}} ^ {(t)}, \tag {14}
$$
where  $\eta > 0$  is a hyperparameter. It is worth noting that  $\mathcal{L}_{\mathrm{ELo}}$  as well as  $\mathcal{L}_{\mathrm{Lo}}$  can be implemented batch-wisely, via replacing  $\mathcal{G}$  with its subsets. Similar to  $\mathcal{L}_{\mathrm{Lo}}$ ,  $\mathcal{L}_{\mathrm{ELo}}$  can also be applied to semi-supervised graph classification, i.e., (10).
Besides the Lovász- $\vartheta$  introduced in Section 2.2, the closest work to our Lovász principle is the InfoMax principle. Following [Nowozin et al., 2016; Sun et al., 2019; Belghazi et al., 2018], suppose the node-level representation  $\pmb{h}_p(x)$  and the graph-level representation  $\pmb{z}(x)$  are depending on the input  $x$ ,  $T_{\varphi}$  is a discriminator parameterized by a neural network with parameters  $\varphi$ , the Jensen-Shannon mutual information (MI) estimator [Fuglede and Topsoe, 2004; Nowozin et al., 2016; Hjelm et al., 2019; Sun et al., 2019]  $I_{\varphi}$  between  $\pmb{h}_p$  and  $\pmb{z}$  is defined as
$$
I _ {\varphi} \left(\boldsymbol {h} _ {p}, \boldsymbol {z}\right) = \mathbb {E} _ {\mathbb {P}} \left[ - \operatorname {s p} \left(- T _ {\varphi} \left(\boldsymbol {h} _ {p} (x), \boldsymbol {z} (x)\right)\right) \right] - \mathbb {E} _ {\mathbb {P} \times \tilde {\mathbb {P}}} \left[ \operatorname {s p} \left(T _ {\varphi} \left(\boldsymbol {h} _ {p} \left(x ^ {\prime}\right), \boldsymbol {z} (x)\right)\right) \right], \tag {15}
$$
where  $x$  is the input sample from distribution  $\mathbb{P}$ ,  $x'$  is the negative sample from distribution  $\tilde{\mathbb{P}}$ , and  $\mathrm{sp}(a) = \log(1 + e^a)$  denotes the softplus function. Many recent graph-level representation learning methods [Sun et al., 2019; You et al., 2020; Yin et al., 2022] are based on the InfoMax principle, i.e., maximizing (15). For instance, the InfoGraph proposed by [Sun et al., 2019] obtains graph-level representations by maximizing the mutual information between the graph-level representation and the node-level representations as follows
$$
\phi^ {*}, \theta^ {*}, \varphi^ {*} = \underset {\phi , \theta , \varphi} {\arg \max } \sum_ {i = 1} ^ {| \mathcal {G} |} \frac {1}{| V _ {i} |} \sum_ {p \in V _ {i}} I _ {\varphi} \left(\boldsymbol {h} _ {p} ^ {\theta}, \boldsymbol {z} _ {i} ^ {\phi}\right) \triangleq - \mathcal {L} _ {\text {u n s u p e r v i s e d}} ^ {I _ {\varphi}} (\mathcal {G}). \tag {16}
$$
For semi-supervised learning, the dataset  $\mathcal{G}$  is split into labeled dataset  $\mathcal{G}^L$  and unlabeled dataset  $\mathcal{G}^U$ . They deploy another supervised encoder with parameter  $\psi$  and then generate the supervised node-level representations  $H_{i}^{\psi}$ , graph-level representations  $z_{i}^{\psi}$  and prediction  $\hat{y}_{i}^{\psi}$ . The loss function of InfoGraph for semi-supervised learning is defined as follows
$$
\mathcal {L} _ {\text {i n f o - s e m i}} = \sum_ {l = 1} ^ {| \mathcal {G} ^ {L} |} \ell_ {\text {s u p e r v i s e d}} \left(\hat {\boldsymbol {y}} _ {l} ^ {\psi}, \boldsymbol {y} _ {l}\right) + \mathcal {L} _ {\text {u n s u p e r v i s e d}} ^ {I _ {\varphi}} (\mathcal {G}) - \lambda \sum_ {i = 1} ^ {| \mathcal {G} |} \frac {1}{| V _ {i} |} I _ {\varphi} \left(\boldsymbol {z} _ {i} ^ {\phi}. \boldsymbol {z} _ {i} ^ {\psi}\right). \tag {17}
$$
The comparison between the InfoMax principle and our Lovász principle is as follows.
- The InfoMax principle focuses on the mutual information between graph-level representation and node-level representation, while our Lovász principle is derived from the Lovász number, a fundamental topological property of graph.
- Our Lovász principle only needs to optimize  $\phi$  and  $\theta$ . Differently, besides  $\phi$  and  $\theta$ , the InfoMax principle has to optimize an additional discriminator parameter  $\varphi$  for the Jensen-Shannon MI estimator. Thus, our Lovász principle is simpler than the InfoMax principle.
- Approximating mutual information using neural network is challenging [Nowozin et al., 2016] and the Jensen-Shannon MI estimator  $I_{\varphi}$  only provides an approximation by sampling rather than an exact computation. In contrast, our Lovász principle does not rely on mutual information and sampling.
It is worth noting that the Lovász convolutional networks (LCN) proposed by [Yadav et al., 2019] was motivated by the observation that removing certain vertices from a graph doesn't affect the graph's global properties such as the Lovász number. LCN does not involve any optimization related to the Lovász number and was designed as an alternative to GCN. Our Lovász principle is an optimization principle that can be used in any graph neural network (e.g. LCN). It is also useful in many applications such as graph prompt learning [Liu et al., 2023; Sun et al., 2022] and graph anomaly detection [Ma et al., 2021; Zhang et al., 2023; Cai et al., 2023].
In this section, we evaluate the effectiveness of the Lovász principle compared to the InfoMax principle in graph representation learning methods and a few other baselines such as graph kernels. The graph representation learning methods we considered in this paper include InfoGraph [Sun et al., 2019], GraphCL [You et al., 2020], AD-GCL [Suresh et al., 2021], JOAO [You et al., 2021], and AutoGCL [Yin et al., 2022], which are the most current and influential methods spanning from
2019 to 2022. Note that the graph-level representation learning principles (GIB) [Wu et al., 2020] and (SIB) [Yu et al., 2021] are not suitable for unsupervised graph learning and hence will not be compared in this work. To ensure fair comparisons, we follow the neural network architectures of those InfoMax based methods and replace the InfoMax loss (Eq. (16)) with the Lovász loss  $\mathcal{L}_{\mathrm{Lo}}$  (Eq. (9)) or enhanced Lovász loss  $\mathcal{L}_{\mathrm{ELo}}$  (Eq. (14)) while keeping the network structures and parameter settings unchanged. We conduct the experiments on TUD benchmark datasets [Morris et al., 2020] and ChEMBL benchmark datasets [Mayr et al., 2018; Gaulton et al., 2012].
6.1 Unsupervised Learning
The first category of compare methods is graph kernel-based methods such as graphlet kernel (GL) [Shervashidze et al., 2009], Weisfeiler-Lehman sub-tree kernel (WL) [Shervashidze et al., 2011], deep graph kernel (DGK) [Yanardag and Vishwanathan, 2015], and Lovász-  $\vartheta$  kernel [Johansson et al., 2014]. The second category is traditional unsupervised graph representation methods like node2vec [Grover and Leskovec, 2016], sub2vec Adhikari et al. [2018], and graph2vec [Narayanan et al., 2017]. The third category is the methods based on the InfoMax principle (Eq. (16)), including InfoGraph [Sun et al., 2019], GraphCL [You et al., 2020], AD-GCL [Suresh et al., 2021], JOAOv2 [You et al., 2021], and AutoGCL [Yin et al., 2022]. We also include the Lovász number method (LovászNum) [Lovász, 1979] as a baseline, where we solve the Lovász number (Eq. (2)) using semidefinite programming (SDP) [Wolkowicz et al., 2012] and then use the handle vector  $c^*$  as the graph-level representation.
Following [Sun et al., 2019; You et al., 2021; Yin et al., 2022], we train a graph representation model on unlabeled data to obtain graph representations and use these representations and graph labels to train a classifier. Our experimental setup is similar to that of AutoGCL [Yin et al., 2022]. Specifically, we use a 5-layer GIN [Xu et al., 2018] with hidden size 128 as the representation model and an SVM as the classifier. The model is trained with a batch size of 128 and a learning rate of 0.001. For those contrastive learning methods (e.g., JOJOv2 and AutoGCL), we use 30 epochs of contrastive pre-training under the naive strategy. We perform 10-fold cross-validation on each dataset and repeat 10 times with different random seeds and record the average accuracy (ACC) and standard deviation.
Table 1: Performance (ACC) of unsupervised learning. The baseline results are from AutoGCL [Yin et al., 2022] and JOAO [You et al., 2021]. The bold, blue and green numbers denote the best, second best and third best performances respectively, which also applies to Tables 2 and 3.
methods	MUTAG	PROTEINS	DD	NCII	COLLAB	IMDB-B	REDMIT-B	REDMIT-M5K
kernels	GL	81.66±2.11	-	-	-	-	65.87±0.98	77.34±0.18	41.01±0.17
WL	80.72±3.00	72.92±0.56	-	80.01±0.50	-	72.30±3.44	68.82±0.41	46.06±0.21
DGK	87.44±2.72	73.30±0.82	-	80.31±0.46	-	66.96±0.56	78.04±0.39	41.27±0.18
Lovasz-θ	82.57±1.68	71.86±1.41	-	75.90±1.33	-	67.26±1.85	76.03±1.87	43.57±1.79
vector embedding	node2vec	72.63±10.20	57.49±3.57	-	54.89±1.61	-	-	-	-
sub2vec	61.05±15.80	53.03±5.55	-	52.84±1.47	-	55.26±1.54	71.48±0.41	36.68±0.42
graph2vec	83.15±9.25	73.30±2.05	-	73.22±1.81	-	71.10±0.54	75.78±1.03	47.86±0.26
InfoMax principle	InfoGraph	89.01±1.13	74.44±0.31	72.85±1.78	76.20±1.06	70.65±1.13	73.03±0.87	82.50±1.42	53.46±1.03
GraphCL	86.80±1.34	74.39±0.45	78.62±0.40	77.87±0.41	71.36±1.15	71.14±0.44	89.53±0.84	55.99±0.28
AD-GCL	87.13±1.56	73.59±0.65	74.49±0.52	69.67±0.51	73.32±0.61	71.57±1.01	85.52±0.79	53.00±0.82
JOAOv2	86.91±1.01	71.25±0.85	66.91±1.75	72.99±0.75	70.40±2.21	71.60±0.86	78.35±1.38	55.57±2.86
AutoGCL	88.64±1.08	75.80±0.36	77.57±0.60	82.00±0.29	70.12±0.68	73.30±0.40	88.58±1.49	56.75±0.18
LovaszNum	81.24±1.59	62.46±1.31	67.65±2.31	74.73±1.88	72.47±1.83	70.57±1.73	71.25±1.59	43.24±1.72
Lovasz principle (use LLo)	InfoGraph	89.67±1.54	75.26±1.43	74.13±1.49	78.21±1.35	71.46±1.21	73.87±1.32	84.76±1.86	54.57±1.38
GraphCL	87.24±1.96	75.87±2.17	79.14±1.67	79.13±1.27	72.52±1.37	72.44±1.46	89.87±2.13	56.12±1.73
AD-GCL	87.44±2.13	74.29±2.80	76.25±1.48	75.12±2.13	73.85±1.05	73.02±1.35	87.11±1.95	54.61±2.35
JOAOv2	87.19±1.92	73.15±1.46	73.15±2.17	74.15±1.67	72.62±1.43	72.18±1.72	84.19±1.67	53.74±1.70
AutoGCL	89.02±1.47	76.23±1.25	78.95±1.39	82.63±2.12	71.31±1.72	73.95±1.36	89.41±1.81	57.28±1.62
Lovasz principle (use LELo)	InfoGraph	90.13±2.05	76.12±1.72	75.76±1.64	79.36±1.57	72.67±1.95	74.96±1.49	84.53±1.79	55.12±1.47
GraphCL	87.93±2.42	76.82±1.34	77.35±1.95	80.11±1.47	74.16±1.37	73.87±1.52	90.23±1.87	56.83±1.35
AD-GCL	88.50±1.82	75.22±1.93	76.14±1.21	78.15±1.81	74.57±1.98	73.48±1.41	88.16±1.37	55.64±1.63
JOAOv2	88.76±1.43	75.27±1.61	74.62±2.58	76.23±1.75	72.85±1.73	72.97±1.37	85.31±1.48	54.68±1.48
AutoGCL	89.87±1.85	76.03±1.37	79.31±1.27	82.95±1.26	72.23±1.52	74.52±1.44	90.65±1.46	57.93±1.72
As shown in Table 1, the enhanced Lovász loss  $\mathcal{L}_{\mathrm{ELo}}$  (Eq. (14)) achieves the best performance on all datasets. By replacing the InfoMax loss with the Lovász loss  $\mathcal{L}_{\mathrm{Lo}}$ , the performances of the five graph representation learning methods are improved, which demonstrates the effectiveness of the Lovász principle. Furthermore,  $\mathcal{L}_{\mathrm{ELo}}$  outperformed  $\mathcal{L}_{\mathrm{Lo}}$  in most cases, which verified the effectiveness of introducing subgraph Lovász numbers to the Lovász principle. It is worth noting that the LovászNum method performs worse than the Lovász principle based methods, which confirms the limitations we analyzed in Section 3 and verifies the necessity and significance of our proposed methods.
6.2 Semi-supervised Learning
Following [Hu et al., 2019; You et al., 2021; Yin et al., 2022], we compare Lovász principle with InfoMax principle in semi-supervised learning tasks. The semi-supervised losses of our Lovász principle based methods and InfoMax based methods  $\mathcal{L}_{\mathrm{info - semi}}$  were shown in (10) and (17) respectively. Following the settings of AutoGCL [Yin et al., 2022], we employ a 10-fold cross-validation on each dataset. For each fold, we use  $80\%$  of the total data as the unlabeled data,  $10\%$  as labeled training data, and  $10\%$  as labeled testing data. The classifier for labeled data is a ResGCN [Chen et al., 2019] with 5 layers and a hidden size of 128. We repeat each experiment 10 times and report the average accuracy in Table 2. We see that our Lovász loss  $\mathcal{L}_{\mathrm{Lo}}$  and the enhanced Lovász loss  $\mathcal{L}_{\mathrm{ELo}}$  outperformed InfoMax loss in all cases. Furthermore,  $\mathcal{L}_{\mathrm{ELo}}$  outperformed  $\mathcal{L}_{\mathrm{Lo}}$  in most cases. These results are consistent with those in Secion 6.1.
Table 2: Performance (ACC) of semi-supervised learning.
methods	NCII	PROTEINS	DD	COLLAB	REDDIT-B	REDDIT-M5K	GITHUB
no Pretrain	73.72±0.24	70.40±1.54	73.56±0.41	73.71±0.27	86.63±0.27	51.33±0.44	60.87±0.17
Pretrain-GNN	Infomax	74.86±0.26	72.27±0.40	75.78±0.34	73.76±0.29	88.66±0.95	53.61±0.31	65.21±0.88
ContextPred	73.00±0.30	70.23±0.63	74.66±0.51	73.69±0.37	84.76±0.52	51.23±0.84	62.35±0.73
InfoMax principle	GraphCL	74.63±0.25	74.17±0.34	76.17±1.37	74.23±0.21	89.11±0.19	52.55±0.45	65.81±0.79
AD-GCL	75.18±0.31	73.96±0.47	77.91±0.73	75.82±0.26	90.10±0.15	53.49±0.28	64.17±1.38
JOAOv2	74.86±0.39	73.31±0.48	75.81±0.73	75.53±0.18	88.79±0.65	52.71±0.28	66.66±0.60
AutoGCL	73.75±2.25	75.65±2.40	77.50±4.41	77.16±1.48	79.80±3.47	49.91±2.70	62.46±1.51
Lovász principle (use LLo)	GraphCL	75.46±1.53	75.12±1.87	77.46±1.52	76.12±1.15	89.87±1.68	53.69±1.68	66.72±1.53
AD-GCL	76.62±1.83	74.21±1.71	78.27±1.39	76.27±1.74	90.36±1.56	54.06±1.32	65.32±1.04
JOAOv2	76.13±1.76	73.73±1.86	76.27±1.48	77.35±1.27	89.31±1.85	53.17±1.76	66.35±1.96
AutoGCL	75.77±1.48	76.36±1.57	78.16±1.61	77.63±1.78	84.64±2.53	51.31±1.81	64.87±1.62
Lovász principle (use LELo)	GraphCL	75.81±1.68	75.88±1.67	78.43±1.48	77.57±1.58	90.67±1.27	54.81±1.73	67.04±1.45
AD-GCL	77.28±1.04	75.43±1.58	78.67±1.64	76.98±1.87	91.54±1.39	55.46±1.59	66.87±1.25
JOAOv2	76.25±1.59	74.67±1.37	77.96±1.86	78.84±1.75	90.25±1.22	54.32±1.89	67.52±1.73
AutoGCL	76.53±1.92	76.89±1.55	78.82±1.90	78.46±1.39	87.31±1.57	53.17±1.50	66.47±1.26
6.3 Transfer Learning
Following [Hu et al., 2019; You et al., 2021; Yin et al., 2022], we compare the performance of our Lovász principles with the InfoMax principle in the task of transfer learning. We use the PretrainGNN method [Hu et al., 2019] as a baseline and employ the Infomax, EdgePred, AttrMasking, and ContextPred pre-training strategies. The experimental settings follow those of AutoGCL [Yin et al., 2022]. More details are in the appendix. As shown in Table 3, the improved Lovász loss  $\mathcal{L}_{\mathrm{ELo}}$  performs the best on transfer learning tasks. In addition, the Lovász principle based methods generally outperform those based on the InfoMax principle in most cases.
Table 3: Performance (ROC-AUC score) of transfer learning.
methods	BBBP	Tox21	ToxCast	SIDER	ClinTox	MUV	HIV	BACE
no Pretrain	65.8±4.5	74.0±0.8	63.4±0.6	57.3±1.6	58.0±4.4	71.8±2.5	75.3±1.9	70.1±5.4
Pretrain-GNN's strategies	Infomax	68.8±0.8	75.3±0.5	62.7±0.4	58.4±0.8	69.9±3.0	75.3±2.5	76.0±0.7	75.9±1.6
EdgePred	67.3±2.4	76.0±0.6	64.1±0.6	60.4±0.7	64.1±3.7	74.1±2.1	76.3±1.0	79.9±0.9
AttrMasking	64.3±2.8	76.7±0.4	64.2±0.5	61.0±0.7	71.8±4.1	74.7±1.4	77.2±1.1	79.3±1.6
ContextPred	68.0±2.0	75.7±0.7	63.9±0.6	60.9±0.6	65.9±3.8	75.8±1.7	77.3±1.0	79.6±1.2
InfoMax principle	GraphCL	69.68±0.67	73.87±0.66	62.40±0.57	60.53±0.88	75.99±2.65	69.80±2.66	78.47±1.22	75.38±1.44
AD-GCL	70.01±1.07	76.54±0.82	63.07±0.72	63.28±0.79	79.78±3.52	72.30±1.61	78.28±0.97	78.51±0.80
JOAOv2	71.39±0.92	74.27±0.62	63.16±0.45	60.49±0.74	80.97±1.64	73.67±1.00	77.51±1.17	75.49±1.27
AutoGCL	73.36±0.77	75.69±0.29	63.47±0.38	62.51±0.63	80.99±3.38	75.83±1.30	78.35±0.64	83.26±1.13
Lovaszprinciple(use \( \mathcal{L}_{\text{Lo}} \))	GraphCL	71.37±1.74	75.66±1.82	63.35±1.47	62.11±1.35	77.02±1.67	72.25±1.42	79.23±1.43	78.51±1.58
AD-GCL	72.24±1.89	77.52±1.74	63.56±1.36	63.87±1.53	80.35±2.36	74.42±1.57	78.95±2.21	80.17±1.04
JOAOv2	72.16±1.35	75.86±1.21	63.92±1.52	62.56±1.12	81.26±2.37	75.94±1.38	79.01±2.68	79.82±1.39
AutoGCL	73.79±1.41	76.13±1.48	64.21±1.58	63.24±1.51	81.32±2.12	76.04±1.87	78.64±1.95	82.57±1.95
Lovaszprinciple(use \( \mathcal{L}_{\text{ELo}} \))	GraphCL	73.05±1.21	76.45±1.35	64.58±1.73	63.72±1.52	80.21±2.31	74.43±1.95	80.37±1.52	80.63±1.63
AD-GCL	72.48±1.59	77.96±1.71	64.27±1.68	63.91±1.74	81.76±2.01	75.88±1.48	81.08±2.35	82.21±1.49
JOAOv2	74.13±1.26	76.21±1.35	64.81±1.92	63.38±1.89	82.75±2.69	76.51±1.53	81.13±1.96	81.34±1.35
AutoGCL	74.67±1.81	76.97±1.76	65.36±1.45	64.13±1.48	82.13±2.41	76.93±1.62	79.56±1.41	83.57±1.30
6.4 Overall Performance and Significance Analysis
For convenience, we show the average performance of all methods over all datasets in Figure 2. We see our Lovász principles outperformed other methods in the three tasks.
To measure the significance of the improvement over the baselines, we implement paired t-tests on the mean scores obtained from the datasets. A p-value below 0.05 indicates a significant difference. The results presented in Table 4 demonstrate the statistical significance of the improvements achieved by our methods across all the datasets.
(a) Unsupervised learning
(b) Semi-supervised learning
(c) Transfer learning
Figure 2: The average performance of different types of methods
Table 4: Significance analysis ( $p$ -values) of improvement via the paired t-test. A  $p$ -value less than 0.05 indicates a significant improvement.
tasks	principles comparison	InfoGraph	GraphCL	AD-GCL	JOAOv2	AutoGCL
unsupervised	InfoMax vs Lovász (LLo)	0.00067	0.00286	0.02238	0.07347	0.00059
InfoMax vs Lovász (LEo)	0.00005	0.01626	0.01541	0.01319	0.00035
Lovász (LLo) vs Lovász (LEo)	0.00429	0.10925	0.01522	0.00079	0.00466
semi-supervised	InfoMax vs Lovász (LLo)	-	0.00028	0.01115	0.04290	0.02147
InfoMax vs Lovász (LEo)	-	0.00051	0.00051	0.00116	0.01129
Lovász (LLo) vs Lovász (LEo)	-	0.00169	0.00076	0.00133	0.00545
6.5 Measuring the Quality of Solver Approximation
Given a GNN model  $\mathcal{F}_W$  trained via the Lovász principle, the predicted Lovász number of a graph  $G$  is denoted as  $\hat{\vartheta}(G)$ , while the ground-truth Lovász number  $\vartheta(G)$  can be computed by SDP [Wolkowicz et al., 2012]. Then we define the relative prediction error for the Lovász number as
$$
e _ {\vartheta} = \left| \hat {\vartheta} (G) - \vartheta (G) \right| / \vartheta (G). \tag {18}
$$
Besides the regularized optimization of the Lovász principle in (9), we also propose a constrained optimization method in Appendix E. We select 50 graphs from each of the four datasets and report  $e_{\vartheta}$  given by both the regularized  $(\mu = 10)$  optimization and the constrained optimization for Lovász principle in Table 5. We can see that in almost all cases, the relative prediction errors are less than  $10\%$ . This indicates that the  $\mathcal{F}_W$  trained by the Lovász principle is a good and reliable approximator for the solver  $\mathcal{A}$  of the Lovász number. This is similar to the idea of learning to optimize.
Table 5: Relative prediction errors  ${e}_{\vartheta }$  given by regularized optimization and constrained optimization
eθ(%)	MUTAG	PROTEINS	DD	NCI1
regularized optimization	9.7±3.4	8.2±2.1	6.3±1.1	10.2±3.6
constrained optimization	6.5±2.4	7.3±1.6	6.1±1.2	8.5±2.3
6.6 More Numerical Results
The results of parameter sensitivity analysis, ablation study, and time cost comparison are in Appendix C, Appendix D, and Appendix F respectively.