SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ziheng Sun<sup>1,2</sup> Chris Ding<sup>1</sup> Jicong Fan<sup>1,2*</sup>
$^{1}$ School of Data Science, The Chinese University of Hong Kong, Shenzhen, China
zihengsun@link.cuhk.edu.cn {chrisding,fanjicong}@cuhk.edu.cn
This paper focuses on graph-level representation learning that aims to represent graphs as vectors that can be directly utilized in downstream tasks such as graph classification. We propose a novel graph-level representation learning principle called Lovász principle, which is motivated by the Lovász number in graph theory. The Lovász number of a graph is a real number that is an upper bound for graph Shannon capacity and is strongly connected with various global characteristics of the graph. Specifically, we show that the handle vector for computing the Lovász number is potentially a suitable choice for graph representation, as it captures a graph's global properties, though a direct application of the handle vector is difficult and problematic. We propose to use neural networks to address the problems and hence provide the Lovász principle. Moreover, we propose an enhanced Lovász principle that is able to exploit the subgraph Lovász numbers directly and efficiently. The experiments demonstrate that our Lovász principles achieve competitive performance compared to the baselines in unsupervised and semi-supervised graph-level representation learning tasks. The code of our Lovász principles is publicly available on GitHub $^{\dagger}$ .
Graphs, such as chemical compounds, protein structures, and social networks, are non-Euclidean data that represent the relationships between entities. There have been a large number of previous works studying many aspects of graphs, including mutagenicity prediction of chemical compounds [Debnath et al., 1991; Kriege and Mutzel, 2012], protein structure prediction [Borgwardt et al., 2005], and community analysis of social networks [Yanardag and Vishwanathan, 2015].
Graph-based learning problems can be organized into two categories: node-level learning and graph-level learning. In this paper, we will only focus on graph-level learning. It is known that in graph-level learning, one fundamental task or step is to measure the distance or similarity between graphs. An important class of methods comparing graphs is graph kernel and many graph kernels have been proposed in the past decades [Siglidis et al., 2020]. For instance, random walk kernels [Gartner et al., 2003] are the most widely-used and well-studied graph kernel family, which measure the graph similarity by counting the common random walks between graphs. The Weisfeiler-Lehman [Weisfeiler and Leman, 1968] family kernels are based on node label reassignment. Most graph kernels extract the similarity information between graphs by sampling sub-structures of graphs such as walks or reassigning the attributes of nodes with their neighborhoods. Note that graph kernels are implicit graph representation methods and hence their flexibilities are not high. In addition, the time and space complexities are quadratic with the number of graphs.
$\dagger$ https://github.com/SunZiheng0/Lovasz-Principle
Graph representation learning aims to convert data with graph structure into vector representations that can be applied to various downstream tasks, such as graph clustering and classification. Many studies have been conducted on graph-level representation learning, and some of them use neural message-passing algorithms [Kipf et al., 2018; Xie and Grossman, 2018; Gilmer et al., 2017]. For instance, the InfoGraph proposed by [Sun et al., 2019] achieves graph-level representations by maximizing the mutual information between the graph-level representation and the node-level representations. Graph contrastive learning (GraphCL) [You et al., 2020] and adversarial graph contrastive learning (AD-GCL) [Suresh et al., 2021] obtain graph-level representations by training graph neural networks (GNNs) to maximize the correspondence between the same graph's representations in its various augmented forms. JOint Augmentation Optimization (JOAO) [You et al., 2021] is a framework that automatically and adaptively selects data augmentations for GraphCL on specific graph data, using a unified bi-level min-max optimization approach. Automated Graph Contrastive Learning (AutoGCL) [Yin et al., 2022] uses learnable graph view generators and auto-augmentation strategy to generate contrastive samples while preserving the most representative structures of the original graph. These graph-level representation learning methods are all based on the InfoMax principle [Linsker, 1988]. Note that there are many other graph representation learning methods such as VGAE [Kipf and Welling, 2016; Hamilton et al., 2017; Cui et al., 2020], graph embedding [Wu et al., 2020; Yu et al., 2021; Bai et al., 2019; Verma and Zhang, 2019], self-supervised learning [Liu et al., 2022; Hou et al., 2022; Lee et al., 2022; Xie et al., 2022; Wu et al., 2021; Rong et al., 2020; Zhang et al., 2021b,a; Xiao et al., 2022], and contrastive learning [Le-Khac et al., 2020; Qiu et al., 2020; Ding et al., 2022; Xia et al., 2022; Fang et al., 2022; Trivedi et al., 2022; Han et al., 2022; Mo et al., 2022; Yin et al., 2022; Xu et al., 2021; Zhao et al., 2021; Zeng and Xie, 2021; Li et al., 2022a,b; Wei et al., 2022], which will not be detailed in this paper due to the page length limit.
The InfoMax principle [Linsker, 1988], which is very popular in graph-level representation learning, advocates maximizing the mutual information between the representations of entire graphs and the representations of substructures of varying sizes [Peng et al., 2020; Velickovic et al., 2019; Hassani and Khasahmadi, 2020; Xie et al., 2022; Qiu et al., 2020]. These InfoMax-based methods usually evaluate the mutual information (MI) between different representations using Jensen-Shannon MI estimator [Sun et al., 2019], following the formulations of  $f$ -GAN [Nowozin et al., 2016] and Mutual Information Neural Estimation (MINE) [Belghazi et al., 2018]. However, the Jensen-Shannon MI estimator necessitates the training of a neural network parameterized discriminator, which is overly complex. In addition, the estimator is based on sampling, which may not be accurate enough in exploiting the mutual information. As opposed to InfoMax, researchers proposed the graph information bottleneck (GIB) [Wu et al., 2020] and the subgraph information bottleneck (SIB) [Yu et al., 2021] that aim to learn the minimal sufficient representation for downstream tasks. But GIB [Wu et al., 2020] and SIB [Yu et al., 2021] may fail if the downstream tasks are not available in the representation learning stage.
In this work, we introduce a novel graph learning principle called Lovász principle, which is inspired by the Lovász number [Lovász, 1979] in graph theory. The Lovász number is an upper bound for a graph's Shannon capacity. It is closely associated with various global characteristics of a graph, such as the clique number and chromatic number of the complement graph. The handle vector for calculating the Lovász number is potentially a suitable choice for the graph-level representation, as it captures a graph's global features, though it suffers from a few difficulties. The contributions of this work are summarized as follows.
- We propose the Lovász principle, a novel framework for unsupervised graph representation learning. We show how to effectively and efficiently utilize the handle vectors to represent graphs. The Lovász principle exploits the topological structures of graphs globally via neural networks.
- We propose an enhanced Lovász principle via effectively incorporating subgraph Lovász numbers, while direct computation of subgraph Lovász numbers is extremely costly. The enhanced Lovász principle ensures similar graphs have similar representations.
- We extend the Lovász principles to semi-supervised representation learning. Note that it is possible to adapt the Lovász principles to more graph-based learning problems.
The experimental results of unsupervised learning, semi-supervised learning, and transfer learning on many benchmark graph datasets show that the proposed Lovász principles outperform graph kernels, classical graph embedding methods, and InfoMax principle based representation learning methods.
In this work, we use  $x$ ,  $\pmb{x}$ ,  $\mathcal{X}$  (or  $X$ ) to denote scalar, vector, matrix, and set respectively.  $\mathbf{1}_{a\times b}$  is a matrix of size  $a\times b$  consisting only ones.  $\pmb{I}_n$  denotes an identity matrix of size  $n\times n$ . Let  $G = (V,E)$  be a graph with  $n$  nodes and  $a$ -dimensional node features  $\{\pmb{x}_v\in \mathbb{R}^a |v\in V\}$ . We denote  $\pmb{A}\in \mathbb{R}^{n\times n}$  as the adjacency matrix and  $\pmb{X} = [\pmb{x}_1,\dots,\pmb{x}_n]^\top \in \mathbb{R}^{n\times a}$  as the node features matrix. Let  $\pmb{z}\in \mathbb{R}^d$  be the  $d$ -dimensional graph-level representation of  $G$ ,  $\pmb{h}_v\in \mathbb{R}^d$  be the  $d$ -dimensional node-level representation of node  $v$ , and  $\pmb{H} = [h_1,\dots,h_n]^\top \in \mathbb{R}^{n\times d}$  be the node-level representations matrix of  $G$ . We denote  $(p,q)$  as an edge between nodes  $p,q$  and  $(p,q)\in E$  if they are connected.
Let  $\mathcal{G} = \{G_1, \ldots, G_N\}$  be a dataset of  $N$  graphs with  $K$  classes, where  $G_i = (V_i, E_i)$ . For  $G_i$ , we denote its number of nodes as  $n_i$ , graph-level representation as  $\boldsymbol{z}_i$ , the adjacency matrix as  $\boldsymbol{A}_i$ , the node feature matrix as  $\boldsymbol{X}_i$ , and node-level representation matrix as  $\boldsymbol{H}_i$ . The graph-level representation matrix of dataset  $\mathcal{G}$  is denoted as  $\boldsymbol{Z} = [z_1, \dots, z_N]^\top \in \mathbb{R}^{N \times d}$ . The set of all node-level representations is denoted as  $\mathcal{H} = \{\boldsymbol{H}_1, \dots, \boldsymbol{H}_N\}$ .
2.1 Lovász number
The definition of Lovász number [Lovász, 1979] is based on orthonormal representations of a graph. Therefore we first introduce the definition of orthonormal representations.
Definition 2.1 (Orthonormal representations). Given a graph  $G = (V, E)$  with  $|V| = n$ . Let
$$
\mathcal {U} := \left\{\boldsymbol {U} \in \mathbb {R} ^ {d \times n}: \| \boldsymbol {u} _ {p} \| _ {2} = 1, p = 1, 2, \dots , n; \boldsymbol {u} _ {p} ^ {\top} \boldsymbol {u} _ {q} = 0, \forall (p, q) \notin E \right\}, \tag {1}
$$
where  $\pmb{u}_p$  is the  $p$ -th column of  $\pmb{U}$ . Then every  $\pmb{U} \in \mathcal{U}$  is an orthonormal representation of  $G$  in  $\mathbb{R}^d$ .
Clearly, every graph has at least one orthonormal representation. For example, a trivial representation is that each node  $p$  is represented by the standard basis vector  $e_p$ . Based on Definition 2.1, we introduce the Lovász number [Lovász, 1979] of a graph as follows.
Definition 2.2 (Lovász number). The Lovász number of a graph  $G = (V, E)$  is defined as
$$
\vartheta (G) := \min  _ {\boldsymbol {c}, \boldsymbol {U} \in \mathcal {U}} \max  _ {p \in V} \frac {1}{(\boldsymbol {c} ^ {\top} \boldsymbol {u} _ {p}) ^ {2}}, \tag {2}
$$
where  $\pmb{c} \in \mathbb{R}^d$  ranges over all unit vectors. The vector  $\pmb{c}$  yielding the minimum for (2), denoted by  $\pmb{c}^*$ , is called the handle of the representation, where the corresponding  $\pmb{U}$  is denoted as  $\pmb{U}^*$  for convenience.  $\pmb{U}^*$  is called the optimal representation of  $G$  in  $\mathbb{R}^d$ .
László Lovász provided a pentagon example, shown in Figure 1, to explain Lovász number defined by (2). The visualization of  $U^{*}$  and  $c^{*}$  of a pentagon is like an umbrella whose handle is  $c^{*}$  and the ribs are the five columns of  $U^{*}$ . These five disjoint node pairs, i.e.,  $(\pmb{u}_1^*,\pmb{u}_3^*),(\pmb{u}_1^*,\pmb{u}_4^*),(\pmb{u}_2^*,\pmb{u}_4^*),(\pmb{u}_2^*,\pmb{u}_5^*),(\pmb{u}_3^*,\pmb{u}_5^*)$ , are orthogonal to each other in visualization.
The Lovász number  $\vartheta(G)$  is an upper bound on the Shannon capacity of a graph  $G$  and is polynomial-time computable [Grötschel et al., 1981; Galli and Letchford, 2017] (e.g., using semidefinite programming (SDP)). Let  $\bar{G}$  be the complement of  $G$  with its clique number  $\omega(\bar{G})$  and chromatic number  $\chi(\bar{G})$ . The Lovász "sandwich theorem" shows that the Lovász number is bounded between the clique number and the chromatic number of  $\bar{G}$ , i.e.,  $\omega(\bar{G}) \leq \vartheta(G) \leq \chi(\bar{G})$ . Although computing  $\omega(\bar{G})$  and  $\chi(G)$  are
Figure 1: Pentagon example for Lovász number
both NP-hard, the "sandwich theorem" allows us to estimate the clique number and the chromatic number of  $\bar{G}$  using only  $\vartheta(G)$ . More detailed discussion can be found in [Knuth, 1993].
2.2 Lovász theta kernel
Johansson et al. [2014] defined the Lovász theta kernel to evaluate the similarity between graphs. Suppose  $S \subseteq V$  is a subset of the vertices of graph  $G$ , then the Lovász number of the subgraph
induced by  $S$  is defined as
$$
\vartheta_ {S} (G) := \min  _ {\boldsymbol {c}} \max  _ {p \in S} \frac {1}{\left(\boldsymbol {c} ^ {\top} \boldsymbol {u} _ {p}\right) ^ {2}}, \tag {3}
$$
where  $\mathbf{U}$  was pre-computed by Eq. (2) and  $c$  ranges over all unit vectors.
Definition 2.3 (Lovász-  $\vartheta$  kernel [Johansson et al., 2014]). Let  $k$  be a base kernel. The Lovász theta kernel between graphs  $G = (V, E)$  and  $G' = (V', E')$  is defined as
$$
k _ {\mathrm {L o}} (G, G ^ {\prime}) = \sum_ {S \subseteq V} \sum_ {S ^ {\prime} \subseteq V ^ {\prime}} \frac {\delta (| S | , | S ^ {\prime} |)}{C _ {S , S ^ {\prime}}} k \left(\vartheta_ {S} (G), \vartheta_ {S ^ {\prime}} \left(G ^ {\prime}\right)\right), \tag {4}
$$
where  $C_{S,S'} = \binom{|V|}{|S|}\binom{|V'|}{|S'|}$ ,  $\delta(|S|, |S'|) = 1$  if  $|S| = |S'$ , and  $\delta(|S|, |S'|) = 0$  otherwise.
$k_{\mathrm{Lo}}$  is a positive semi-definite kernel [Johansson et al., 2014]. It is able to capture global properties of graphs and has been shown useful in SVM-based graph classification [Johansson et al., 2014].
3 Lovász Principle for Graph Representation Learning
The Lovász number  $\vartheta(G)$  of a graph  $G$  provides an insight into the global property of the graph. It is a unique and deterministic value associated with an orthonormal representation  $U^{*}$  and a unit handle vector  $c^{*}$ . The umbrella example in Figure 1 explains how to compute the Lovász number: compacting the ribs (i.e.  $U^{*}$ ) as much as possible and using  $c^{*}$  as the handle. This example provides intuition that the handle vector  $c^{*}$  is a natural and suitable representation of the graph  $G$ .
Given  $\mathcal{G} = \{G_1, G_2, \ldots, G_N\}$  drawn from an unknown distribution  $\mathcal{D}_G$ , we want to represent each graph as a vector such that these vectors preserve some important information of  $\mathcal{D}_G$ . Suppose we have an algorithm  $\mathcal{A}$  such that
$$
\left(\boldsymbol {U} _ {i} ^ {*}, \boldsymbol {c} _ {i} ^ {*}\right) = \mathcal {A} \left(G _ {i}\right), \quad i = 1, 2, \dots , N, \tag {5}
$$
where  $\mathcal{A}$  is some solver for (2). It is natural to use  $c_{1}^{*}, c_{2}^{*}, \ldots, c_{N}^{*}$  as representations of  $G_{1}, G_{2}, \ldots, G_{N}$  respectively. However, this method has the following limitations $^{\ddagger}$ .
i. Non-uniqueness For any  $G_{i}$ , both  $\pmb{U}_{i}^{*}$  and  $\pmb{c}_i^*$  are not unique. For example, let  $\pmb{Q} \in \mathbb{R}^{d \times d}$  be an orthonormal matrix, i.e.,  $\pmb{Q}^{\top} \pmb{Q} = \pmb{Q} \pmb{Q}^{\top} = \pmb{I}_d$ , and let  $\pmb{c}_i' = \pmb{Q} \pmb{c}_i^*$  and  $\pmb{U}_i' = \pmb{Q} \pmb{U}_i^*$ . We have  $\| \pmb{c}' \|_2 = 1$ ,  $(\pmb{U}')^\top \pmb{U} = (\pmb{U}_i^*)^\top \pmb{U}_i^*$ , and  $(\pmb{c}')^\top \pmb{u}_p' = \vartheta(G_i)$ . This means  $(\pmb{c}_i^*, \pmb{U}_i^*)$  and  $(\pmb{c}_i', \pmb{U}')$  yield the same Lovász number for  $G_i$ , though they could be very different. Thus, for two graphs  $G_i$  and  $G_j$  in  $\mathcal{G}$ , even when they are isomorphic,  $\pmb{c}_i^*$  and  $\pmb{c}_j^*$  could be very different. However, for graph representation, we hope that similar graphs have similar representations. For two graphs  $G_i$  and  $G_j$ , one may align their orthonormal representations using  $\hat{\pmb{Q}} = \operatorname{argmin}_{\pmb{Q}^\top \pmb{Q} = I_d} \| \pmb{U}_i^* - \pmb{Q} \pmb{U}_j^* \|_F^2$  and compare them according to  $\| \pmb{c}_i^* - \pmb{c}_j^* \hat{\pmb{Q}} \|_2$ . This however only works when the  $n_i = n_j$  and  $G_i$  and  $G_j$  are matched.
ii. High computational cost For each  $G_{i}$  in  $\mathcal{G}$ , we need to solve the optimization problem (2), for which the time complexity of SDP is at least  $\mathcal{O}(|E_i|n_i^{2.5})$  [Jiang et al., 2020]. Thus the total time complexity for  $\mathcal{G}$  is  $\mathcal{O}(\sum_{i=1}^{N}|E_i|n_i^{2.5})$ . Therefore, this representation method is not scalable to large datasets.
iii. Ignorance of node features The computation of (5) solely relies on the graph structure and does not take advantage of the node feature matrix  $\mathbf{X}_i$  that is often available and informative.
iv. Non-generalization Suppose we have some new graphs and want to obtain their representations. We cannot utilize the representations of  $\mathcal{G}$  and we have to solve (2) again for each new graph.
v. Non-global sensing The computation of (5) treats each graph separately and cannot effectively take advantage of the global information or structure of  $\bar{\mathcal{G}}$ . Individual graphs may have noise or outliers, which cannot be handled by a local method.
$^{\ddagger}$ In our experiments (Table 1), this naive method, termed as LovászNum, is tested. In addition, the Lovász- $\vartheta$  kernel introduced in Section 2.2 is also tested.
To solve the aforementioned five issues, we present a machine learning method. We use a neural network  $\mathcal{F}_W$  (parameterized by  $W$ ) to approximate  $\mathcal{A}$ .  $\mathcal{F}_W$  can be learned from  $\mathcal{G}$  as well as some additional information such as the node feature matrices  $\{\mathbf{X}_1,\dots ,\mathbf{X}_N\}$ . Specifically, we hope that
$$
\left(\boldsymbol {U} _ {i} ^ {*}, \boldsymbol {c} _ {i} ^ {*}\right) \approx \mathcal {F} _ {W} \left(\boldsymbol {A} _ {i}, \boldsymbol {X} _ {i}\right), \quad i = 1, 2, \dots , N. \tag {6}
$$
Thus,  $\mathcal{F}_W$  plays a role representing a graph (drawn from  $\mathcal{D}_G$ ) to a matrix of nodes representation and a vector of graph representation. For new graphs sampled from  $\mathcal{D}_G$ ,  $\mathcal{F}_W$  should generalize well when the approximation errors in (6) are small enough and  $\mathcal{F}_W$  is not too complex. For convenience, we split  $\mathcal{F}_W$  into two parts, i.e.,  $\mathcal{F}_W(\cdot, \cdot) = (F(\cdot, \cdot; \theta), f(\cdot, \cdot; \phi))$ , though  $F$  and  $f$  can share some parameters. We let  $U_i^* \approx F(A_i, X_i; \theta)$  and  $c_i^* \approx f(A_i, X_i; \phi)$ .  $F(\cdot, \cdot; \theta)$  is the model of node-level representation learning while  $f(\cdot, \cdot; \phi)$  is the model of graph-level representation learning. We let
$$
\boldsymbol {H} _ {i} ^ {\theta} := F (\boldsymbol {A} _ {i}, \boldsymbol {X} _ {i}; \theta), \text {a n d} \boldsymbol {z} _ {i} ^ {\phi} := f (\boldsymbol {A} _ {i}, \boldsymbol {X} _ {i}; \phi), \forall i = 1, 2, \dots , N. \tag {7}
$$
We denote the graph-level representations matrix as  $\mathbf{Z}_{\phi} = [\mathbf{z}_1^{\phi}, \dots, \mathbf{z}_N^{\phi}]^\top$  and the node-level representations set as  $\mathcal{H}_{\theta} = \{\mathbf{H}_1^{\theta}, \dots, \mathbf{H}_N^{\theta}\}$ . To achieve (6), we propose to solve
$$
\underset {\phi , \theta} {\text {m i n i m i z e}} \sum_ {i = 1} ^ {N} \left\{\underbrace {\max  _ {p \in V _ {i}} \frac {1}{\left(\left(\boldsymbol {z} _ {i} ^ {\phi}\right) ^ {\top} \boldsymbol {h} _ {p} ^ {\theta}\right) ^ {2}}} _ {\ell_ {1}} + \mu \left(\underbrace {\left\| \boldsymbol {M} _ {i} \odot \left(\boldsymbol {H} _ {i} ^ {\theta} \left(\boldsymbol {H} _ {i} ^ {\theta}\right) ^ {\top} - \boldsymbol {I} _ {n _ {i}}\right) \right\| _ {F} ^ {2}} _ {\ell_ {2}} + \underbrace {\left(\left(\boldsymbol {z} _ {i} ^ {\phi}\right) ^ {\top} \boldsymbol {z} _ {i} ^ {\phi} - 1\right) ^ {2}} _ {\ell_ {3}}\right) \right\}, \tag {8}
$$
where  $M_{i} = \mathbf{1}_{n_{i}\times n_{i}} - \mathbf{A}_{i}$  is a mask matrix and  $\mu >0$  is a regularization parameter. The roles of  $\ell_1$ ,  $\ell_2$ , and  $\ell_3$  in (8) are explained as follows.
-  $\ell_1$  corresponds to the objective in the definition of Lovász number of  $G_i$ .
-  $\ell_2$  is to approximate the orthonormal representation for  $G_i$ , i.e.,  $(\pmb{h}_p^\theta)^\top \pmb{h}_q^\theta \approx 0$  if  $(p,q) \notin E_i$  and  $\| \pmb{h}_p \|_2 \approx 1 \, \forall p \in V_i$ .
-  $\ell_3$  corresponds to the unit-length requirement for the handle vector of  $G_i$ , i.e.,  $\|z_i^\phi\|_2 \approx 1$ .
We call (8) Lovász principle $^{\S}$ , since it aims to learn an  $\mathcal{F}_W$  to solve the optimization of Lovász number for the graphs drawn from  $\mathcal{D}_G$ . It is known that the Lovász number  $\vartheta(G)$  is an upper bound on the Shannon capacity of  $G = (V, E)$ . The Shannon capacity [Shannon, 1956] models the amount of information that can be transmitted across a noisy communication channel, where certain signal values can be confused with each other. Here, one signal value corresponds to one node of  $G$  and  $(p, q) \in E$  means that the corresponding two signals can be confused with each other. Therefore, the graph-level and node-level representations given by our Lovász principle correspond to the upper bound of the amount of information transmitted over the graph that is distinguishable between nodes.
Note that instead of the regularized unconstrained optimization (8), we can also use constrained optimization  $(\ell_2 = \ell_3 = 0)$ , which we call strict Lovász principle. We may use the Lagrange multipliers method, projected gradient descent, or exact (or inexact) penalty method to solve the constrained optimization. Take the inexact penalty method as an example, we just need to increase the  $\mu$  in (8) gradually in the optimization. The graph representation performance comparison between unconstrained and constrained optimizations will be shown in Section 6.5 and Appendix E.
For convenience, we let
$$
\mathcal {L} _ {\mathrm {L o}} := \sum_ {i = 1} ^ {| \mathcal {G} |} \max  _ {p \in V _ {i}} \frac {1}{\left(\left(\boldsymbol {z} _ {i} ^ {\phi}\right) ^ {\top} \boldsymbol {h} _ {p} ^ {\theta}\right) ^ {2}} + \mu \left(\left\| \boldsymbol {M} _ {i} \odot \left(\boldsymbol {H} _ {i} ^ {\theta} \left(\boldsymbol {H} _ {i} ^ {\theta}\right) ^ {\top} - \boldsymbol {I} _ {n _ {i}}\right) \right\| _ {F} ^ {2} + \left(\left(\boldsymbol {z} _ {i} ^ {\phi}\right) ^ {\top} \boldsymbol {z} _ {i} ^ {\phi} - 1\right) ^ {2}\right), \tag {9}
$$
and call it Lovász loss. The Lovász loss is mainly designed for unsupervised graph-level representation learning [Wu et al., 2022; Maron et al., 2019; Oono and Suzuki, 2019; Stahlberg et al., 2022], which can be used as an alternative to the popular InfoMax loss [Linsker, 1988] (see (16)).
Lovász principle for semi-supervised learning Inspired by InfoGraph [Sun et al., 2019] (see (17)), we propose a Lovász loss function for semi-supervised learning tasks. Suppose the dataset  $\mathcal{G}$  has
$^{\S}$ We also provide an equivalent formulation based on the complement graph of  $G$  in Appendix A.
two subsets: a labeled dataset  $\mathcal{G}^L$  and an unlabeled dataset  $\mathcal{G}^U$ . Then we deploy another supervised encoder with parameter  $\psi$  and generate the supervised node-level representations  $H_i^\psi$ , graph-level representations  $z_i^\psi$ , and then prediction  $\hat{\pmb{y}}_i^\psi$ . The overall loss function is