Title: Lovász Principle for Unsupervised Graph Representation Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Lovász Number

graph-level representation learning

unsupervised learning

semi-supervised learning
Abstract: This paper focuses on graph-level representation learning that aims to represent graphs as vectors that can be directly utilized in downstream tasks such as graph classification. 
We propose a novel graph-level representation learning principle called Lovász principle, which is motivated by the Lovász number in graph theory. The Lovász number of a graph is a real number that is an upper bound for graph Shannon capacity and is strongly connected with various global characteristics of the graph. Specifically, we show that the handle vector for computing the Lovász number is potentially a suitable choice for graph representation, as it captures a graph's global properties, though a direct application of the handle vector is difficult and problematic. We propose to use neural networks to address the problems and hence provide the Lovász principle. Moreover, we propose an enhanced Lovász principle that is able to exploit the subgraph Lovász numbers directly and efficiently. The experiments demonstrate that our Lovász principles achieve competitive performance compared to the baselines in unsupervised and semi-supervised graph-level representation learning tasks. The code of our Lovász principles is publicly available on GitHub.

Lovász Principle for Unsupervised Graph Representation Learning

Abstract

This paper focuses on graph-level representation learning that aims to represent graphs as vectors that can be directly utilized in downstream tasks such as graph classification. We propose a novel graph-level representation learning principle called Lovász principle, which is motivated by the Lovász number in graph theory. The Lovász number of a graph is a real number that is an upper bound for graph Shannon capacity and is strongly connected with various global characteristics of the graph. Specifically, we show that the handle vector for computing the Lovász number is potentially a suitable choice for graph representation, as it captures a graph's global properties, though a direct application of the handle vector is difficult and problematic. We propose to use neural networks to address the problems and hence provide the Lovász principle. Moreover, we propose an enhanced Lovász principle that is able to exploit the subgraph Lovász numbers directly and efficiently. The experiments demonstrate that our Lovász principles achieve competitive performance compared to the baselines in unsupervised and semi-supervised graph-level representation learning tasks. The code of our Lovász principles is publicly available on GitHub † .

1 Introduction

Graphs, such as chemical compounds, protein structures, and social networks, are non-Euclidean data that represent the relationships between entities. There have been a large number of previous works studying many aspects of graphs, including mutagenicity prediction of chemical compounds [Debnath et al. , 1991; Kriege and Mutzel, 2012], protein structure prediction [Borgwardt et al. , 2005], and community analysis of social networks [Yanardag and Vishwanathan, 2015].

Graph-based learning problems can be organized into two categories: node-level learning and graphlevel learning. In this paper, we will only focus on graph-level learning. It is known that in graph-level learning, one fundamental task or step is to measure the distance or similarity between graphs. An important class of methods comparing graphs is graph kernel and many graph kernels have been proposed in the past decades [Siglidis et al. , 2020]. For instance, random walk kernels [Gärtner et al. , 2003] are the most widely-used and well-studied graph kernel family, which measure the graph similarity by counting the common random walks between graphs. The Weisfeiler-Lehman [Weisfeiler and Leman, 1968] family kernels are based on node label reassignment. Most graph kernels extract the similarity information between graphs by sampling sub-structures of graphs such as walks or reassigning the attributes of nodes with their neighborhoods. Note that graph kernels are implicit graph representation methods and hence their flexibilities are not high. In addition, the time and space complexities are quadratic with the number of graphs.

Graph representation learning aims to convert data with graph structure into vector representations that can be applied to various downstream tasks, such as graph clustering and classification. Many studies have been conducted on graph-level representation learning, and some of them use neural messagepassing algorithms [Kipf et al. , 2018; Xie and Grossman, 2018; Gilmer et al. , 2017]. For instance, the InfoGraph proposed by [Sun et al. , 2019] achieves graph-level representations by maximizing the mutual information between the graph-level representation and the node-level representations. Graph contrastive learning (GraphCL) [You et al. , 2020] and adversarial graph contrastive learning (AD-GCL) [Suresh et al. , 2021] obtain graph-level representations by training graph neural networks (GNNs) to maximize the correspondence between the same graph's representations in its various augmented forms. JOint Augmentation Optimization (JOAO) [You et al. , 2021] is a framework that automatically and adaptively selects data augmentations for GraphCL on specific graph data, using a unified bi-level min-max optimization approach. Automated Graph Contrastive Learning (AutoGCL) [Yin et al. , 2022] uses learnable graph view generators and auto-augmentation strategy to generate contrastive samples while preserving the most representative structures of the original graph. These graph-level representation learning methods are all based on the InfoMax principle [Linsker, 1988]. Note that there are many other graph representation learning methods such as VGAE [Kipf and Welling, 2016; Hamilton et al. , 2017; Cui et al. , 2020], graph embedding [Wu et al. , 2020; Yu et al. , 2021; Bai et al. , 2019; Verma and Zhang, 2019], self-supervised learning [Liu et al. , 2022; Hou et al. , 2022; Lee et al. , 2022; Xie et al. , 2022; Wu et al. , 2021; Rong et al. , 2020; Zhang et al. , 2021b,a; Xiao et al. , 2022], and contrastive learning [Le-Khac et al. , 2020; Qiu et al. , 2020; Ding et al. , 2022; Xia et al. , 2022; Fang et al. , 2022; Trivedi et al. , 2022; Han et al. , 2022; Mo et al. , 2022; Yin et al. , 2022; Xu et al. , 2021; Zhao et al. , 2021; Zeng and Xie, 2021; Li et al. , 2022a,b; Wei et al. , 2022], which will not be detailed in this paper due to the page length limit.

The InfoMax principle [Linsker, 1988], which is very popular in graph-level representation learning, advocates maximizing the mutual information between the representations of entire graphs and the representations of substructures of varying sizes [Peng et al. , 2020; Velickovic et al. , 2019; Hassani and Khasahmadi, 2020; Xie et al. , 2022; Qiu et al. , 2020]. These InfoMax-based methods usually evaluate the mutual information (MI) between different representations using Jensen-Shannon MI estimator [Sun et al. , 2019], following the formulations of f -GAN [Nowozin et al. , 2016] and Mutual Information Neural Estimation (MINE) [Belghazi et al. , 2018]. However, the Jensen-Shannon MI estimator necessitates the training of a neural network parameterized discriminator, which is overly complex. In addition, the estimator is based on sampling, which may not be accurate enough in exploiting the mutual information. As opposed to InfoMax, researchers proposed the graph information bottleneck (GIB) [Wu et al. , 2020] and the subgraph information bottleneck (SIB) [Yu et al. , 2021] that aim to learn the minimal sufficient representation for downstream tasks. But GIB [Wu et al. , 2020] and SIB [Yu et al. , 2021] may fail if the downstream tasks are not available in the representation learning stage.

In this work, we introduce a novel graph learning principle called Lovász principle, which is inspired by the Lovász number [Lovász, 1979] in graph theory. The Lovász number is an upper bound for a graph's Shannon capacity. It is closely associated with various global characteristics of a graph, such as the clique number and chromatic number of the complement graph. The handle vector for calculating the Lovász number is potentially a suitable choice for the graph-level representation, as it captures a graph's global features, though it suffers from a few difficulties. The contributions of this work are summarized as follows.

We propose the Lovász principle, a novel framework for unsupervised graph representation learning. We show how to effectively and efficiently utilize the handle vectors to represent graphs. The Lovász principle exploits the topological structures of graphs globally via neural networks.

We propose an enhanced Lovász principle via effectively incorporating subgraph Lovász numbers, while direct computation of subgraph Lovász numbers is extremely costly. The enhanced Lovász principle ensures similar graphs have similar representations.

We extend the Lovász principles to semi-supervised representation learning. Note that it is possible to adapt the Lovász principles to more graph-based learning problems.

The experimental results of unsupervised learning, semi-supervised learning, and transfer learning on many benchmark graph datasets show that the proposed Lovász principles outperform graph kernels, classical graph embedding methods, and InfoMax principle based representation learning methods.

2 Notations and Preliminaries

In this work, we use x , x , X , X (or X ) to denote scalar, vector, matrix, and set respectively. 1 a × b is a matrix of size a × b consisting only ones. I n denotes an identity matrix of size n × n . Let G = ( V, E ) be a graph with n nodes and a -dimensional node features { x v ∈ R a | v ∈ V } . We denote A ∈ R n × n as the adjacency matrix and X = [ x 1 , ..., x n ] ⊤ ∈ R n × a as the node features matrix. Let z ∈ R d be the d -dimensional graph-level representation of G , h v ∈ R d be the d -dimensional node-level representation of node v , and H = [ h 1 , ..., h n ] ⊤ ∈ R n × d be the node-level representations matrix of G . We denote ( p, q ) as an edge between nodes p, q and ( p, q ) ∈ E if they are connected.

Let G = { G 1 , . . . G N } be a dataset of N graphs with K classes, where G i = ( V i , E i ) . For G i , we denote its number of nodes as n i , graph-level representation as z i , the adjacency matrix as A i , the node feature matrix as X i , and node-level representation matrix as H i . The graph-level representation matrix of dataset G is denoted as Z = [ z 1 , ..., z N ] ⊤ ∈ R N × d . The set of all node-level representations is denoted as H = { H 1 , . . . , H N } .

2.1 Lovász number

The definition of Lovász number [Lovász, 1979] is based on orthonormal representations of a graph. Therefore we first introduce the definition of orthonormal representations.

Definition 2.1 (Orthonormal representations) . Given a graph G = ( V, E ) with | V | = n . Let

where u p is the p -th column of U . Then every U ∈ U is an orthonormal representation of G in R d .

Clearly, every graph has at least one orthonormal representation. For example, a trivial representation is that each node p is represented by the standard basis vector e p . Based on Definition 2.1, we introduce the Lovász number [Lovász, 1979] of a graph as follows.

Definition 2.2 (Lovász number) . The Lovász number of a graph G = ( V, E ) is defined as

where c ∈ R d ranges over all unit vectors. The vector c yielding the minimum for (2), denoted by c ∗ , is called the handle of the representation, where the corresponding U is denoted as U ∗ for convenience. U ∗ is called the optimal representation of G in R d .

László Lovász provided a pentagon example, shown in Figure 1, to explain Lovász number defined by (2). The visualization of U ∗ and c ∗ of a pentagon is like an umbrella whose handle is c ∗ and the ribs are the five columns of U ∗ . These five disjoint node pairs, i.e., ( u ∗ 1 , u ∗ 3 ) , ( u ∗ 1 , u ∗ 4 ) , ( u ∗ 2 , u ∗ 4 ) , ( u ∗ 2 , u ∗ 5 ) , ( u ∗ 3 , u ∗ 5 ) , are orthogonal to each other in visualization.

The Lovász number ϑ ( G ) is an upper bound on the Shannon capacity of a graph G and is polynomial-time computable [Grötschel et al. , 1981; Galli and Letchford, 2017] (e.g., using semidefinite programming (SDP)). Let ¯ G be the complement of G with its clique number ω ( ¯ G ) and chromatic number χ ( ¯ G ) . The Lovász "sandwich theorem" shows that the Lovász number is bounded between the clique number and the chromatic number of ¯ G , i.e., ω ( ¯ G ) ≤ ϑ ( G ) ≤ χ ( ¯ G ) . Although computing ω ( ¯ G ) and χ ( ¯ G ) are

both NP-hard, the "sandwich theorem" allows us to estimate the clique number and the chromatic number of ¯ G using only ϑ ( G ) . More detailed discussion can be found in [Knuth, 1993].

2.2 Lovász theta kernel

Johansson et al. [2014] defined the Lovász theta kernel to evaluate the similarity between graphs. Suppose S ⊆ V is a subset of the vertices of graph G , then the Lovász number of the subgraph

induced by S is defined as

where U was pre-computed by Eq. (2) and c ranges over all unit vectors.

Definition 2.3 (Lovászϑ kernel [Johansson et al. , 2014]) . Let k be a base kernel. The Lovász theta kernel between graphs G = ( V, E ) and G ′ = ( V ′ , E ′ ) is defined as

where C S,S ′ = ( | V | | S | )( | V ′ | | S ′ | ) , δ ( | S | , | S ′ | ) = 1 if | S | = | S ′ | , and δ ( | S | , | S ′ | ) = 0 otherwise.

k Lo is a positive semi-definite kernel [Johansson et al. , 2014]. It is able to capture global properties of graphs and has been shown useful in SVM-based graph classification [Johansson et al. , 2014].

3 Lovász Principle for Graph Representation Learning

The Lovász number ϑ ( G ) of a graph G provides an insight into the global property of the graph. It is a unique and deterministic value associated with an orthonormal representation U ∗ and a unit handle vector c ∗ . The umbrella example in Figure 1 explains how to compute the Lovász number: compacting the ribs (i.e. U ∗ ) as much as possible and using c ∗ as the handle. This example provides intuition that the handle vector c ∗ is a natural and suitable representation of the graph G .

Given G = { G 1 , G 2 , . . . , G N } drawn from an unknown distribution D G , we want to represent each graph as a vector such that these vectors preserve some important information of D G . Suppose we have an algorithm A such that

where A is some solver for (2). It is natural to use c ∗ 1 , c ∗ 2 , . . . , c ∗ N as representations of G 1 , G 2 , . . . , G N respectively. However, this method has the following limitations ‡ .

Non-uniqueness For any G i , both U ∗ i and c ∗ i are not unique. For example, let Q ∈ R d × d be an orthonormal matrix, i.e., Q ⊤ Q = QQ ⊤ = I d , and let c ′ i = Qc ∗ i and U ′ i = QU ∗ i . We have ∥ c ′ ∥ 2 = 1 , ( U ′ ) ⊤ U = ( U ∗ i ) ⊤ U ∗ i , and ( c ′ ) ⊤ u ′ p = ϑ ( G i ) . This means ( c ∗ i , U ∗ i ) and ( c ′ i , U ′ ) yield the same Lovász number for G i , though they could be very different. Thus, for two graphs G i and G j in G , even when they are isomorphic, c ∗ i and c ∗ j could be very different. However, for graph representation, we hope that similar graphs have similar representations. For two graphs G i and G j , one may align their orthonormal representations using ˆ Q = argmin Q ⊤ Q = I d ∥ U ∗ i -QU ∗ j ∥ 2 F and compare them according to ∥ c ∗ i -c ∗ j ˆ Q ∥ 2 . This however only works when the n i = n j and G i and G j are matched.

High computational cost For each G i in G , we need to solve the optimization problem (2), for which the time complexity of SDP is at least O ( | E i | n 2 . 5 i ) [Jiang et al. , 2020]. Thus the total time complexity for G is O ( ∑ N i =1 | E i | n 2 . 5 i ) . Therefore, this representation method is not scalable to large datasets.

Ignorance of node features The computation of (5) solely relies on the graph structure and does not take advantage of the node feature matrix X i that is often available and informative.

Non-generalization Suppose we have some new graphs and want to obtain their representations. We cannot utilize the representations of G and we have to solve (2) again for each new graph.

Non-global sensing The computation of (5) treats each graph separately and cannot effectively take advantage of the global information or structure of G . Individual graphs may have noise or outliers, which cannot be handled by a local method.

To solve the aforementioned five issues, we present a machine learning method. We use a neural network F W (parameterized by W ) to approximate A . F W can be learned from G as well as some additional information such as the node feature matrices { X 1 , . . . , X N } . Specifically, we hope that

Thus, F W plays a role representing a graph (drawn from D G ) to a matrix of nodes representation and a vector of graph representation. For new graphs sampled from D G , F W should generalize well when the approximation errors in (6) are small enough and F W is not too complex. For convenience, we split F W into two parts, i.e., F W ( · , · ) = ( F ( · , · ; θ ) , f ( · , · ; ϕ )) , though F and f can share some parameters. We let U ∗ i ≈ F ( A i , X i ; θ ) and c ∗ i ≈ f ( A i , X i ; ϕ ) . F ( · , · ; θ ) is the model of node-level representation learning while f ( · , · ; ϕ ) is the model of graph-level representation learning. We let

We denote the graph-level representations matrix as Z ϕ = [ z ϕ 1 , ..., z ϕ N ] ⊤ and the node-level representations set as H θ = { H θ 1 , ..., H θ N } . To achieve (6), we propose to solve

where M i = 1 n i × n i -A i is a mask matrix and µ > 0 is a regularization parameter. The roles of ℓ 1 , ℓ 2 , and ℓ 3 in (8) are explained as follows.

ℓ 1 corresponds to the objective in the definition of Lovász number of G i .

ℓ 2 is to approximate the orthonormal representation for G i , i.e., ( h θ p ) ⊤ h θ q ≈ 0 if ( p, q ) / ∈ E i and ∥ h p ∥ 2 ≈ 1 ∀ p ∈ V i .

ℓ 3 corresponds to the unit-length requirement for the handle vector of G i , i.e., ∥ z ϕ i ∥ 2 ≈ 1 .

We call (8) Lovász principle § , since it aims to learn an F W to solve the optimization of Lovász number for the graphs drawn from D G . It is known that the Lovász number ϑ ( G ) is an upper bound on the Shannon capacity of G = ( V, E ) . The Shannon capacity [Shannon, 1956] models the amount of information that can be transmitted across a noisy communication channel, where certain signal values can be confused with each other. Here, one signal value corresponds to one node of G and ( p, q ) ∈ E means that the corresponding two signals can be confused with each other. Therefore, the graph-level and node-level representations given by our Lovász principle correspond to the upper bound of the amount of information transmitted over the graph that is distinguishable between nodes.

Note that instead of the regularized unconstrained optimization (8), we can also use constrained optimization ( ℓ 2 = ℓ 3 = 0 ), which we call strict Lovász principle. We may use the Lagrange multipliers method, projected gradient descent, or exact (or inexact) penalty method to solve the constrained optimization. Take the inexact penalty method as an example, we just need to increase the µ in (8) gradually in the optimization. The graph representation performance comparison between unconstrained and constrained optimizations will be shown in Section 6.5 and Appendix E.

For convenience, we let

and call it Lovász loss. The Lovász loss is mainly designed for unsupervised graph-level representation learning [Wu et al. , 2022; Maron et al. , 2019; Oono and Suzuki, 2019; Ståhlberg et al. , 2022], which can be used as an alternative to the popular InfoMax loss [Linsker, 1988] (see (16)).

Lovász principle for semi-supervised learning Inspired by InfoGraph [Sun et al. , 2019] (see (17)), we propose a Lovász loss function for semi-supervised learning tasks. Suppose the dataset G has

two subsets: a labeled dataset G L and an unlabeled dataset G U . Then we deploy another supervised encoder with parameter ψ and generate the supervised node-level representations H ψ i , graph-level representations z ψ i , and then prediction ˆ y ψ i . The overall loss function is

where λ is a positive hyperparameter, the supervised loss ℓ supervised is the cross-entropy loss, and the unsupervised loss L unsupervised is the Lovász loss L Lo (Eq. (9)) or the enhanced Lovász loss L ELo (Eq. (14)). The last term encourages the representations learned by the two encoders to be similar.

4 Enhancing Lovász Principle with Subgraph Lovász Number

Lovász principle does not explicitly utilize the Lovász number in graph embedding, though the Lovász numbers of subgraphs can be useful in comparing graphs [Johansson et al. , 2014]. Therefore, we propose to use subgraph Lovász number to enhance Lovász principle based graph representation learning. We may consider taking advantage of the Lovászϑ kernel proposed by [Johansson et al. , 2014]. However, we encounter the following two difficulties.

Computing the Lovász numbers (3) of subgraphs is time-consuming because we need to solve (2) for every graph and the number of subgraphs of each graph is often very large (up to 2 | V | ). Hence, for large graph dataset, we cannot use (4) directly.

The Lovászϑ kernel (4) is a pair-wise method and cannot effectively exploit the global structure of G .

To solve the aforementioned problems, we present an iterative-refinement strategy that computes the subgraph Lovász numbers using the embeddings given by the Lovász principle. Specifically, at iteration t , we have the graph-level representations Z ( t -1) ϕ and the node-level representations H ( t -1) θ given by iteration t -1 . Inspired by the Lovászϑ kernel (4), we compute the similarity between graph G i and G j as

where C S i ,S j = ( | V i | | S i | )( | V j | | S j | ) and ϑ ( t -1) S i ( G i ) (similar for G j ) is obtained by

The computation of 1 / ( z ( t -1) ⊤ i h ( t -1) p ) 2 for every p ∈ V i was already done when computing L Lo via (9) at iteration t -1 and there is no need to solve (3). For (11), we do not need to consider all possible subgraphs and we can just randomly sample subgraphs with some fixed sizes (numbers of nodes), which is similar to the truncated Lovászϑ kernel of [Johansson et al. , 2014]. Thus we can obtain the similarity K ( t -1) ij efficiently. Adapting the idea of spectral embedding [Belkin and Niyogi, 2001], we propose the following subgraph Lovász number (SLN) loss (at iteration t )

where γ > 0 . The two regularization terms in L ( t ) SLN aim to make the graph-level representations orthonormal and centered, which is consistent with the constraints in spectral embedding. Minimizing L ( t ) SLN encourages that the graph-level representations of similar graphs (in the sense of subgraph Lovász numbers) are closer to each other at iteration t . Integrating (13) with (9), we obtain the following enhanced Lovász loss at iteration t

where η > 0 is a hyperparameter. It is worth noting that L ELo as well as L Lo can be implemented batch-wisely, via replacing G with its subsets. Similar to L Lo, L ELo can also be applied to semisupervised graph classification, i.e., (10).

5 Related Work

Besides the Lovászϑ introduced in Section 2.2, the closest work to our Lovász principle is the InfoMax principle. Following [Nowozin et al. , 2016; Sun et al. , 2019; Belghazi et al. , 2018], suppose the node-level representation h p ( x ) and the graph-level representation z ( x ) are depending on the input x , T φ is a discriminator parameterized by a neural network with parameters φ , the JensenShannon mutual information (MI) estimator [Fuglede and Topsoe, 2004; Nowozin et al. , 2016; Hjelm et al. , 2019; Sun et al. , 2019] I φ between h p and z is defined as

where x is the input sample from distribution P , x ′ is the negative sample from distribution ˜ P , and sp ( a ) = log(1 + e a ) denotes the softplus function. Many recent graph-level representation learning methods [Sun et al. , 2019; You et al. , 2020; Yin et al. , 2022] are based on the InfoMax principle, i.e., maximizing (15). For instance, the InfoGraph proposed by [Sun et al. , 2019] obtains graph-level representations by maximizing the mutual information between the graph-level representation and the node-level representations as follows

For semi-supervised learning, the dataset G is split into labeled dataset G L and unlabeled dataset G U . They deploy another supervised encoder with parameter ψ and then generate the supervised node-level representations H ψ i , graph-level representations z ψ i and prediction ˆ y ψ i . The loss function of InfoGraph for semi-supervised learning is defined as follows

The comparison between the InfoMax principle and our Lovász principle is as follows.

The InfoMax principle focuses on the mutual information between graph-level representation and node-level representation, while our Lovász principle is derived from the Lovász number, a fundamental topological property of graph.

Our Lovász principle only needs to optimize ϕ and θ . Differently, besides ϕ and θ , the InfoMax principle has to optimize an additional discriminator parameter φ for the JensenShannon MI estimator. Thus, our Lovász principle is simpler than the InfoMax principle.

Approximating mutual information using neural network is challenging [Nowozin et al. , 2016] and the Jensen-Shannon MI estimator I φ only provides an approximation by sampling rather than an exact computation. In contrast, our Lovász principle does not rely on mutual information and sampling.

It is worth noting that the Lovász convolutional networks (LCN) proposed by [Yadav et al. , 2019] was motivated by the observation that removing certain vertices from a graph doesn't affect the graph's global properties such as the Lovász number. LCN does not involve any optimization related to the Lovász number and was designed as an alternative to GCN. Our Lovász principle is an optimization principle that can be used in any graph neural network (e.g. LCN). It is also useful in many applications such as graph prompt learning [Liu et al. , 2023; Sun et al. , 2022] and graph anomaly detection [Ma et al. , 2021; Zhang et al. , 2023; Cai et al. , 2023].

6 Experiments

In this section, we evaluate the effectiveness of the Lovász principle compared to the InfoMax principle in graph representation learning methods and a few other baselines such as graph kernels. The graph representation learning methods we considered in this paper include InfoGraph [Sun et al. , 2019], GraphCL [You et al. , 2020], AD-GCL [Suresh et al. , 2021], JOAO [You et al. , 2021], and AutoGCL [Yin et al. , 2022], which are the most current and influential methods spanning from

2019 to 2022. Note that the graph-level representation learning principles (GIB) [Wu et al. , 2020] and (SIB) [Yu et al. , 2021] are not suitable for unsupervised graph learning and hence will not be compared in this work. To ensure fair comparisons, we follow the neural network architectures of those InfoMax based methods and replace the InfoMax loss (Eq. (16)) with the Lovász loss L Lo (Eq. (9)) or enhanced Lovász loss L ELo (Eq. (14)) while keeping the network structures and parameter settings unchanged. We conduct the experiments on TUD benchmark datasets [Morris et al. , 2020] and ChEMBL benchmark datasets [Mayr et al. , 2018; Gaulton et al. , 2012].

6.1 Unsupervised Learning

The first category of compare methods is graph kernel-based methods such as graphlet kernel (GL) [Shervashidze et al. , 2009], Weisfeiler-Lehman sub-tree kernel (WL) [Shervashidze et al. , 2011], deep graph kernel (DGK) [Yanardag and Vishwanathan, 2015], and Lovászϑ kernel [Johansson et al. , 2014]. The second category is traditional unsupervised graph representation methods like node2vec [Grover and Leskovec, 2016], sub2vec Adhikari et al. [2018], and graph2vec [Narayanan et al. , 2017]. The third category is the methods based on the InfoMax principle (Eq. (16)), including InfoGraph [Sun et al. , 2019], GraphCL [You et al. , 2020], AD-GCL [Suresh et al. , 2021], JOAOv2 [You et al. , 2021], and AutoGCL [Yin et al. , 2022]. We also include the Lovász number method (LovászNum) [Lovász, 1979] as a baseline, where we solve the Lovász number (Eq. (2)) using semidefinite programming (SDP) [Wolkowicz et al. , 2012] and then use the handle vector c ∗ as the graph-level representation.

Following [Sun et al. , 2019; You et al. , 2021; Yin et al. , 2022], we train a graph representation model on unlabeled data to obtain graph representations and use these representations and graph labels to train a classifier. Our experimental setup is similar to that of AutoGCL [Yin et al. , 2022]. Specifically, we use a 5-layer GIN [Xu et al. , 2018] with hidden size 128 as the representation model and an SVM as the classifier. The model is trained with a batch size of 128 and a learning rate of 0.001. For those contrastive learning methods (e.g., JOJOv2 and AutoGCL), we use 30 epochs of contrastive pre-training under the naive strategy. We perform 10-fold cross-validation on each dataset and repeat 10 times with different random seeds and record the average accuracy (ACC) and standard deviation.

As shown in Table 1, the enhanced Lovász loss L ELo (Eq. (14)) achieves the best performance on all datasets. By replacing the InfoMax loss with the Lovász loss L Lo, the performances of the five graph representation learning methods are improved, which demonstrates the effectiveness of the Lovász principle. Furthermore, L ELo outperformed L Lo in most cases, which verified the effectiveness of introducing subgraph Lovász numbers to the Lovász principle. It is worth noting that the LovászNum method performs worse than the Lovász principle based methods, which confirms the limitations we analyzed in Section 3 and verifies the necessity and significance of our proposed methods.

6.2 Semi-supervised Learning

Following [Hu et al. , 2019; You et al. , 2021; Yin et al. , 2022], we compare Lovász principle with InfoMax principle in semi-supervised learning tasks. The semi-supervised losses of our Lovász principle based methods and InfoMax based methods L info-semi were shown in (10) and (17) respectively. Following the settings of AutoGCL [Yin et al. , 2022], we employ a 10-fold cross-validation on each dataset. For each fold, we use 80% of the total data as the unlabeled data, 10% as labeled training data, and 10% as labeled testing data. The classifier for labeled data is a ResGCN [Chen et al. , 2019] with 5 layers and a hidden size of 128. We repeat each experiment 10 times and report the average accuracy in Table 2. We see that our Lovász loss L Lo and the enhanced Lovász loss L ELo outperformed InfoMax loss in all cases. Furthermore, L ELo outperformed L Lo in most cases. These results are consistent with those in Secion 6.1.

6.3 Transfer Learning

Following [Hu et al. , 2019; You et al. , 2021; Yin et al. , 2022], we compare the performance of our Lovász principles with the InfoMax principle in the task of transfer learning. We use the PretrainGNN method [Hu et al. , 2019] as a baseline and employ the Infomax, EdgePred, AttrMasking, and ContextPred pre-training strategies. The experimental settings follow those of AutoGCL [Yin et al. , 2022]. More details are in the appendix. As shown in Table 3, the improved Lovász loss L ELo performs the best on transfer learning tasks. In addition, the Lovász principle based methods generally outperform those based on the InfoMax principle in most cases.

6.4 Overall Performance and Significance Analysis

For convenience, we show the average performance of all methods over all datasets in Figure 2. We see our Lovász principles outperformed other methods in the three tasks.

To measure the significance of the improvement over the baselines, we implement paired t-tests on the mean scores obtained from the datasets. A p-value below 0.05 indicates a significant difference. The results presented in Table 4 demonstrate the statistical significance of the improvements achieved by our methods across all the datasets.

80

Average Performance of Unsupervised Leaning

75

average ACC

70

65

F 09

55

50

kenels

embedding

Lovasz

Enhanced

Lovasz

(a) Unsupervised learning

(b) Semi-supervised learning

80.0

Average Performance of Semi-supervised Learning

77.5

75.0

average ACC

72.5

70.0

67.5

65.0

62.5

60.0

pretrain

no

pretrain

GNN

Lovasz

Enhanced

Lovasz

(c) Transfer learning

80

Average Performance of Transfer Leaming

75

average ROC-AUC score

70

65

60

55

50

pretrain

no

pretrain

GNN

Lovasz

Enhanced

Lovasz

6.5 Measuring the Quality of Solver Approximation

Given a GNN model F W trained via the Lovász principle, the predicted Lovász number of a graph G is denoted as ˆ ϑ ( G ) , while the ground-truth Lovász number ϑ ( G ) can be computed by SDP [Wolkowicz et al. , 2012]. Then we define the relative prediction error for the Lovász number as

Besides the regularized optimization of the Lovász principle in (9), we also propose a constrained optimization method in Appendix E. We select 50 graphs from each of the four datasets and report e ϑ given by both the regularized ( µ = 10 ) optimization and the constrained optimization for Lovász principle in Table 5. We can see that in almost all cases, the relative prediction errors are less than 10% . This indicates that the F W trained by the Lovász principle is a good and reliable approximator for the solver A of the Lovász number. This is similar to the idea of learning to optimize.

6.6 More Numerical Results

The results of parameter sensitivity analysis , ablation study , and time cost comparison are in Appendix C, Appendix D, and Appendix F respectively.

7 Conclusions

This paper proposed a novel method called Lovász principle for unsupervised graph-level representation learning. An extension using the subgraph Lovász number was also presented. The numerical results of unsupervised learning, semi-supervised learning, and transfer learning showed that the proposed methods are more effective than graph kernels and InfoMax principle based representation learning methods. Besides unsupervised representation learning, it is possible to apply our methods to other tasks such as graph-level clustering and graph generation. For instance, we can add a clustering module (e.g. [Xie et al. , 2016]) to L Lo to construct an end-to-end clustering algorithm. We can combine L Lo with variational autoencoder [Kingma and Welling, 2013] to train a model to generate new graphs. Nevertheless, the implementation of these methods is out of the scope of this paper.

Acknowledgments

This work was partially supported by the Youth program 62106211 of the National Natural Science Foundation of China, the General Program JCYJ20210324130208022 of Shenzhen Fundamental Research, the research funding T00120210002 of Shenzhen Research Institute of Big Data, the Guangdong Key Lab of Mathematical Foundations for Artificial Intelligence, and the funding UDF01001770 of The Chinese University of Hong Kong, Shenzhen.

The authors appreciate the reviewers' and AC's comments and time.

References

Bijaya Adhikari, Yao Zhang, Naren Ramakrishnan, and B Aditya Prakash. Sub2vec: Feature learning for subgraphs. In Advances in Knowledge Discovery and Data Mining: 22nd Pacific-Asia Conference, PAKDD 2018, Melbourne, VIC, Australia, June 3-6, 2018, Proceedings, Part II 22 , pages 170-182. Springer, 2018.

Yunsheng Bai, Hao Ding, Yang Qiao, Agustin Marinovic, Ken Gu, Ting Chen, Yizhou Sun, and Wei Wang. Unsupervised inductive graph-level representation learning via graph-graph proximity. arXiv preprint arXiv:1904.01098 , 2019.

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning , pages 531-540. PMLR, 2018.

Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps and spectral techniques for embedding and clustering. Advances in neural information processing systems , 14, 2001.

Karsten M Borgwardt, Cheng Soon Ong, Stefan Schönauer, SVN Vishwanathan, Alex J Smola, and Hans-Peter Kriegel. Protein function prediction via graph kernels. Bioinformatics , 21(suppl_1):i47i56, 2005.

Jinyu Cai, Yunhe Zhang, and Jicong Fan. Self-discriminative modeling for anomalous graph detection. arXiv preprint arXiv:2310.06261 , 2023.

Ting Chen, Song Bian, and Yizhou Sun. Are powerful graph neural nets necessary? a dissection on graph classification. arXiv preprint arXiv:1905.04579 , 2019.

Ganqu Cui, Jie Zhou, Cheng Yang, and Zhiyuan Liu. Adaptive graph encoder for attributed graph embedding. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 976-985, 2020.

Asim Kumar Debnath, Rosa L Lopez de Compadre, Gargi Debnath, Alan J Shusterman, and Corwin Hansch. Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. correlation with molecular orbital energies and hydrophobicity. Journal of medicinal chemistry , 34(2):786-797, 1991.

Kaize Ding, Zhe Xu, Hanghang Tong, and Huan Liu. Data augmentation for deep graph learning: A survey. ACM SIGKDD Explorations Newsletter , 24(2):61-77, 2022.

Yin Fang, Qiang Zhang, Haihong Yang, Xiang Zhuang, Shumin Deng, Wen Zhang, Ming Qin, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Molecular contrastive learning with chemical element knowledge graph. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 3968-3976, 2022.

Bent Fuglede and Flemming Topsoe. Jensen-shannon divergence and hilbert space embedding. In International symposium onInformation theory, 2004. ISIT 2004. Proceedings. , page 31. IEEE, 2004.

Laura Galli and Adam N. Letchford. On the lovász theta function and some variants. Discrete Optimization , 25:159-174, 2017.

Thomas Gärtner, Peter Flach, and Stefan Wrobel. On graph kernels: Hardness results and efficient alternatives. In Learning theory and kernel machines , pages 129-143. Springer, 2003.

Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, et al. Chembl: a large-scale bioactivity database for drug discovery. Nucleic acids research , 40(D1):D1100-D1107, 2012.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning , pages 1263-1272. PMLR, 2017.

Martin Grötschel, László Lovász, and Alexander Schrijver. The ellipsoid method and its consequences in combinatorial optimization. Combinatorica , 1:169-197, 1981.

Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining , pages 855-864, 2016.

William L Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. arXiv preprint arXiv:1709.05584 , 2017.

Xiaotian Han, Zhimeng Jiang, Ninghao Liu, and Xia Hu. G-mixup: Graph data augmentation for graph classification. In International Conference on Machine Learning , pages 8230-8248. PMLR, 2022.

Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International conference on machine learning , pages 4116-4126. PMLR, 2020.

R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations , 2019.

Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 594-604, 2022.

Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXiv preprint arXiv:1905.12265 , 2019.

Haotian Jiang, Tarun Kathuria, Yin Tat Lee, Swati Padmanabhan, and Zhao Song. A faster interior point method for semidefinite programming. In 2020 IEEE 61st annual symposium on foundations of computer science (FOCS) , pages 910-918. IEEE, 2020.

Fredrik Johansson, Vinay Jethava, Devdatt Dubhashi, and Chiranjib Bhattacharyya. Global graph kernels using geometric embeddings. In International Conference on Machine Learning , pages 694-702. PMLR, 2014.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 , 2016.

Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning , pages 2688-2697. PMLR, 2018.

Donald E Knuth. The sandwich theorem. arXiv preprint math/9312214 , 1993.

Nils Kriege and Petra Mutzel. Subgraph matching kernels for attributed graphs. arXiv preprint arXiv:1206.6483 , 2012.

Phuc H Le-Khac, Graham Healy, and Alan F Smeaton. Contrastive representation learning: A framework and review. Ieee Access , 8:193907-193934, 2020.

Namkyeong Lee, Junseok Lee, and Chanyoung Park. Augmentation-free self-supervised learning on graphs. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7372-7380, 2022.

Shuangli Li, Jingbo Zhou, Tong Xu, Dejing Dou, and Hui Xiong. Geomgcl: Geometric graph contrastive learning for molecular property prediction. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 4541-4549, 2022.

Yinqi Li, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Optimal positive generation via latent transformation for contrastive learning. Advances in Neural Information Processing Systems , 35:18327-18342, 2022.

Ralph Linsker. Self-organization in a perceptual network. Computer , 21(3):105-117, 1988.

Yixin Liu, Ming Jin, Shirui Pan, Chuan Zhou, Yu Zheng, Feng Xia, and S Yu Philip. Graph self-supervised learning: A survey. IEEE Transactions on Knowledge and Data Engineering , 35(6):5879-5900, 2022.

Zemin Liu, Xingtong Yu, Yuan Fang, and Xinming Zhang. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In Proceedings of the ACM Web Conference 2023 , pages 417-428, 2023.

László Lovász. On the shannon capacity of a graph. IEEE Transactions on Information theory , 25(1):1-7, 1979.

Xiaoxiao Ma, Jia Wu, Shan Xue, Jian Yang, Chuan Zhou, Quan Z Sheng, Hui Xiong, and Leman Akoglu. A comprehensive survey on graph anomaly detection with deep learning. IEEE Transactions on Knowledge and Data Engineering , 2021.

Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. Advances in neural information processing systems , 32, 2019.

Andreas Mayr, Günter Klambauer, Thomas Unterthiner, Marvin Steijaert, Jörg K Wegner, Hugo Ceulemans, Djork-Arné Clevert, and Sepp Hochreiter. Large-scale comparison of machine learning methods for drug target prediction on chembl. Chemical science , 9(24):5441-5451, 2018.

Yujie Mo, Liang Peng, Jie Xu, Xiaoshuang Shi, and Xiaofeng Zhu. Simple unsupervised graph representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7797-7805, 2022.

Christopher Morris, Nils M Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. arXiv preprint arXiv:2007.08663 , 2020.

Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. graph2vec: Learning distributed representations of graphs. arXiv preprint arXiv:1707.05005 , 2017.

Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems , 29, 2016.

Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. arXiv preprint arXiv:1905.10947 , 2019.

Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng, Yu Rong, Tingyang Xu, and Junzhou Huang. Graph representation learning via graphical mutual information maximization. In Proceedings of The Web Conference 2020 , pages 259-270, 2020.

Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gcc: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 1150-1160, 2020.

Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems , 33:12559-12571, 2020.

Shannon. The zero error capacity of a noisy channel. IRE Transactions on Information Theory , 2(3):8-19, 1956.

Nino Shervashidze, SVN Vishwanathan, Tobias Petri, Kurt Mehlhorn, and Karsten Borgwardt. Efficient graphlet kernels for large graph comparison. In Artificial intelligence and statistics , pages 488-495. PMLR, 2009.

Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research , 12(9), 2011.

Giannis Siglidis, Giannis Nikolentzos, Stratis Limnios, Christos Giatsidis, Konstantinos Skianis, and Michalis Vazirgiannis. Grakel: A graph kernel library in python. The Journal of Machine Learning Research , 21(1):1993-1997, 2020.

Simon Ståhlberg, Blai Bonet, and Hector Geffner. Learning general optimal policies with graph neural networks: Expressive power, transparency, and limits. In Proceedings of the International Conference on Automated Planning and Scheduling , volume 32, pages 629-637, 2022.

Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semisupervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv:1908.01000 , 2019.

Mingchen Sun, Kaixiong Zhou, Xin He, Ying Wang, and Xin Wang. Gppt: Graph pre-training and prompt tuning to generalize graph neural networks. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1717-1727, 2022.

Susheel Suresh, Pan Li, Cong Hao, and Jennifer Neville. Adversarial graph augmentation to improve graph contrastive learning. Advances in Neural Information Processing Systems , 34:15920-15933, 2021.

Puja Trivedi, Ekdeep Singh Lubana, Yujun Yan, Yaoqing Yang, and Danai Koutra. Augmentations in graph contrastive learning: Current methodological flaws & towards better practices. In Proceedings of the ACM Web Conference 2022 , pages 1538-1549, 2022.

Petar Velickovic, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. ICLR (Poster) , 2(3):4, 2019.

Saurabh Verma and Zhi-Li Zhang. Learning universal graph neural network embeddings with aid of transfer learning. arXiv preprint arXiv:1909.10086 , 2019.

Chunyu Wei, Jian Liang, Di Liu, and Fei Wang. Contrastive graph structure learning via information bottleneck for recommendation. Advances in Neural Information Processing Systems , 35:2040720420, 2022.

Boris Weisfeiler and Andrei Leman. The reduction of a graph to canonical form and the algebra which appears therein. NTI, Series , 2(9):12-16, 1968.

Henry Wolkowicz, Romesh Saigal, and Lieven Vandenberghe. Handbook of semidefinite programming: theory, algorithms, and applications , volume 27. Springer Science & Business Media, 2012.

Tailin Wu, Hongyu Ren, Pan Li, and Jure Leskovec. Graph information bottleneck. Advances in Neural Information Processing Systems , 33:20437-20448, 2020.

Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, and Xing Xie. Selfsupervised graph learning for recommendation. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval , pages 726-735, 2021.

Lingfei Wu, Peng Cui, Jian Pei, Liang Zhao, and Le Song. Graph neural networks . Springer, 2022.

Jun Xia, Lirong Wu, Jintao Chen, Bozhen Hu, and Stan Z Li. Simgrace: A simple framework for graph contrastive learning without data augmentation. In Proceedings of the ACM Web Conference 2022 , pages 1070-1079, 2022.

Teng Xiao, Zhengyu Chen, Zhimeng Guo, Zeyang Zhuang, and Suhang Wang. Decoupled selfsupervised learning for graphs. Advances in Neural Information Processing Systems , 35:620-634, 2022.

Tian Xie and Jeffrey C Grossman. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical review letters , 120(14):145301, 2018.

Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference on machine learning , pages 478-487. PMLR, 2016.

Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji. Self-supervised learning of graph neural networks: A unified review. IEEE transactions on pattern analysis and machine intelligence , 2022.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 , 2018.

Dongkuan Xu, Wei Cheng, Dongsheng Luo, Haifeng Chen, and Xiang Zhang. Infogcl: Informationaware graph contrastive learning. Advances in Neural Information Processing Systems , 34:3041430425, 2021.

Prateek Yadav, Madhav Nimishakavi, Naganand Yadati, Shikhar Vashishth, Arun Rajkumar, and Partha Talukdar. Lovasz convolutional networks. In The 22nd international conference on artificial intelligence and statistics , pages 1978-1987. PMLR, 2019.

Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining , pages 1365-1374, 2015.

Yihang Yin, Qingzhong Wang, Siyu Huang, Haoyi Xiong, and Xiang Zhang. Autogcl: Automated graph contrastive learning via learnable view generators. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8892-8900, 2022.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems , 33:5812-5823, 2020.

Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In International Conference on Machine Learning , pages 12121-12132. PMLR, 2021.

Junchi Yu, Tingyang Xu, Yu Rong, Yatao Bian, Junzhou Huang, and Ran He. Recognizing predictive substructures with subgraph information bottleneck. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

Jiaqi Zeng and Pengtao Xie. Contrastive self-supervised learning for graph classification. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 10824-10832, 2021.

Hengrui Zhang, Qitian Wu, Junchi Yan, David Wipf, and Philip S Yu. From canonical correlation analysis to self-supervised graph neural networks. Advances in Neural Information Processing Systems , 34:76-89, 2021.

Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph selfsupervised learning for molecular property prediction. Advances in Neural Information Processing Systems , 34:15870-15882, 2021.

Yunhe Zhang, Yan Sun, Jinyu Cai, and Jicong Fan. Deep graph-level orthogonal hypersphere compression for anomaly detection. arXiv preprint arXiv:2302.06430 , 2023.

Han Zhao, Xu Yang, Zhenru Wang, Erkun Yang, and Cheng Deng. Graph debiased contrastive learning with joint representation clustering. In IJCAI , pages 3434-3440, 2021.

A Comaprision with the equivalent forms of the Lovász principles

In this section, we propose an equivalent form of the Lovász principle based on the complement graph of G . We first introduce an equivalent definition of Lovász number as follows.

Definition A.1 (Lovász number Lovász [1979]) . Let ¯ G be the complement graph of G and ¯ U be the orthonormal representations of ¯ G , Lovász proposed an equivalent definition of Lovász number as

where d ∈ R d ranges over all unit vectors.

The unit vector d in the Eq. (19) is also a suitable representation vector for graph G . Thus we can obtain a Lovász loss ¯ L Lo based on Definition A.1 as follows.

where ¯ M i = A i is a mask matrix for complement graph ¯ G . Similar to the main paper, an enhanced Lovász loss based on Definition A.1 at iteration t is defined as

where ¯ L ( t ) SLN is the subgraph Lovász number (SLN) loss of complement graph ¯ G based on Lovász number definition (19).

We compare the Lovász loss ¯ L Lo and enhanced Lovász loss ¯ L ( t ) ELo (based on d ) with the L Lo and L ( t ) ELo (based on handdle vector c ). The results are shown in Tables 6, 7 and 8. For convenience, we show the average performance of all methods over all datasets in Figure 3. We see the two equivalent forms of Lovász principles have very similar performance in the three learning tasks and both of them outperform the InfoMax principle.

B Experimental setting of transfer learning

Following [Hu et al. , 2019; You et al. , 2021; Yin et al. , 2022], we compare the performance of transfer learning of the Lovász principle with the InfoMax principle. We use the Pretrain-GNN method [Hu et al. , 2019] as a baseline and employ the Infomax, EdgePred, AttrMasking, and ContextPred pre-training strategies. The experimental settings followed those of AutoGCL [Yin et al. , 2022]. Specifically, we performed supervised pre-training for 100 epochs on the ChEMBL dataset [Mayr et al. , 2018; Gaulton et al. , 2012], and then fine-tuned the model for 30 epochs on 8 chemistry evaluation subsets, using a classifier with a hidden size of 300. Our training employed a batch size of 256 and a learning rate of 0.001. We substitute the InfoMax loss in the four contrastive learning methods (GraphCL, AD-GCL, JOAOv2, and AutoGCL) with the Lovász loss L Lo or L ELo. We repeat each experiment 10 times and report the average ROC-AUC scores in Table 3 of the main paper.

(a) Unsupervised learning

(b) Semi-supervised learning

(c) Transfer learning

Average Performance of Unsupervised Learming

Average Performance of Semi-supervised Learming

Average Performance of Transfer Leaming

08

80.0

78

74

77.5

76

72

average ROC-AUC score

75.0

average ACC

74

average ACC

72.5

72

70

70.0

70

F 89

67.5

65.0

68 -

66

66

62.5

Lovasz

use C

Lovasz

use d

Enhanced

Lovasz

use C

Enhanced

Lovasz

use d

Lovasz

use C

Lovasz

use d

Enhanced

Lovasz

use c

Enhanced

Lovasz

use d

60.0

Lovasz

use C

Lovasz

use d

Enhanced

Lovasz

use C

Enhanced

Lovasz

use d

C Parameter sensitivity analysis

There are five hyperparameters need to be tuned in Lovász principle method: the dimension of representations d , the hyperparameter of orthonormal representation regularization µ , the hyperparameter of orthogonal regularization in subgraph Lovász number (SLN) loss γ , hyperparameter of subgraph Lovász number (SLN) loss in enhanced Lovász loss η , the hyperparameter of graph-level representations ℓ 2 -norm regularization in semi-supervised Lovász loss λ . In this section, we analyze the parameter sensitivity on the InfoGraph Sun et al. [2019] with different hyperparameters. We repeat each experiment for ten times and plot the average accuracy on different datasets.

C.1 d as the dimension of representations

In Lovász principle, d is the dimension of node-level representations H = [ h 1 , ..., h n ] ⊤ ∈ R n × d and graph-level representation of G , h v ∈ R d . In the definition of Lovász number, the node-level representations are orthonormal representations, i.e., H ⊤ ∈ U . Let α ( G ) be the independent number of graph G , which is the size of the maximum independent set. If d ≤ α ( G ) , the node-level representations H are impossible to be orthonormal representations such that the Lovász number cannot be obtained. In Figure 4, we fix other hyperparameters and tune d from { 10 , 20 , ..., 90 , 100 } .

(a) MUTAG

(b) PROTEIN

(c) DD

(d) NCI1

(e) COLLAB

(f) IMDB-B

(g) REDDIT-B

(h) REDDIT-M5

Average ACC of different d on MUTAG

Average ACC of different d on PROTEINS

Average ACC of different d on DD

Average ACC of different d on NCI1

ACC

aeane

88

72

9

4

72

100

Average ACC of different d on COLLAB

Average ACC of different d on IMDB-B

Average ACC of different d on REDDIT-B

Average ACC of different d on REDDIT-M5K

75

92

74

abeaae

72

52

58

71

82

81

80

The results show that a small d adversely affects the performance because d may be less than α ( G ) on some graphs. When d is too large, the average accuracy decreases slightly because the representations with large d may capture some noisy information of a graph.

C.2 µ for orthonormal representation regularization

(a) MUTAG

(b) PROTEIN

(c) DD

(d) NCI1

(e) COLLAB

(f) IMDB-B

(g) REDDIT-B

(h) REDDIT-M5

Average ACC of different mu on MUTAG

Average ACC of different mu on PROTEINS

Average ACC of different mu on DD

Average ACC of different mu on NCI1

06

78

88

72

70

le-3

se-1

le-3

le-3

1e-2

le-1

le3

le4

3e5

le-3

1-

1e0

Average ACC of different mu on COLLAB

Average ACC of different mu on IMDB-B

Average ACC of different mu on REDDIT-B

Average ACC of different mu on REDDIT-M5K

4

76

74

aeae

rage ACC

64

0

100

le3

100

le3

le4

10

100

le3

124

In Lovász principle, µ is the hyperparameter for orthonormal representation regularization in Lovász loss L Lo (9) as follows

In Figure 5, we fix other hyperparameters and tune µ from { 10 -3 , 10 -2 , ..., 10 5 , 10 6 } . The results show that µ is not sensitive when 0 . 1 ≤ µ ≤ 1 e 3 . If µ is too small, the average accuracy decreases slightly because the node-level representations H may not be orthonormal representations. A very large µ adversely affects the performance because the orthonormal representation regularization dominates the representation learning such that the Lovász principle fails.

(a) MUTAG

(b) PROTEIN

(c) DD

(d) NCI1

(e) COLLAB

(f) IMDB-B

(g) REDDIT-B

(h) REDDIT-M5

Average ACC of different gamma on MUTAG

Average ACC of different gamma on PROTEINS

Average ACC of different gamma on DD

Average ACC of different gamma on NCI1

6

82

92

91

an

76

abeuane

77

76

ee

80

78

74

73

77

73

72

76

86

le-1

le4

les

leb

Z-2T

100

le3

1e4

1e5

1e-1

le3

1e4

1e5

1e6

Average ACC of different gamma on COLLAB

Average ACC of different gamma on IMDB-B

Average ACC of different gamma on REDDIT-B

Average ACC of different gamma on REDDIT-M5K

98

75

87

57

74

86

73

54

71

73

53

72

81

52

71

51

le-3

le-2 1e-1

gamma

100

1e3le4

1e5

le-21e-1

10

gamma

100

1e-21e-1

10

eunueb

100

1e31e41e51e6

le-3

le-2 le-1

10

gamma

100

1e31e4le5le6

C.3 γ for orthogonal regularization in subgraph Lovász number (SLN) loss

In Lovász principle, γ is the hyperparameter for orthogonal regularization in subgraph Lovász number (SLN) loss L ( t ) SLN as follows

In Figure 6, we fix other hyperparameters and tune γ from { 10 -3 , 10 -2 , ..., 10 5 , 10 6 } . The results show that γ is not sensitive when 0 . 1 ≤ γ ≤ 1 e 3 . If γ is too small, the average accuracy decreases slightly because the orthogonal constraints of spectral embedding may not hold. A very large γ adversely affects the performance because the orthogonal regularization of spectral embedding dominates the representation learning such that the Lovász principle fails.

C.4 η for subgraph Lovász number (SLN) loss in enhanced Lovász loss

(a) MUTAG

(b) PROTEIN

(c) DD

(d) NCI1

(e) COLLAB

(f) IMDB-B

(g) REDDIT-B

(h) REDDIT-M5

Average ACC of different eta on MUTAG

Average ACC of different eta on PROTEINS

Average ACC of different eta on DD

Average ACC of different eta on NCI1

EE

78

28

16

77

81

90

76

verage ACC

9c

68

75

75

74

73

87

72

-a

1e-1

lef

le-3

-

cag

let

Average ACC of different eta on COLLAB

Average ACC of different eta on IMDB-B

Average ACC of different eta on REDDIT-B

Average ACC of different eta on REDDIT-M5K

75

77

.8

76

73

75

85

55

74

54

53

70

72

52

65

In Lovász principle, η is the hyperparameter for subgraph Lovász number (SLN) loss L ( t ) SLN in enhanced Lovász loss L ( t ) ELo as follows

In Figure 7, we fix other hyperparameters and tune η from { 10 -3 , 10 -2 , ..., 10 5 , 10 6 } . The results show that η is not sensitive when 10 -2 ≤ η ≤ 1 e 4 . If η is a very small number, the enhanced Lovász loss L ( t ) ELo degenerates into the Lovász loss L Lo, which also performs well in representation learning. A very large η adversely affects the performance because the subgraph Lovász number (SLN) loss L ( t ) SLN dominates the representation learning such that the Lovász principle may fail.

C.5 λ for the ℓ 2 -norm regularization in semi-supervised Lovász loss

(a) MUTAG

(b) PROTEIN

(c) DD

(d) NCI1

(e) COLLAB

(f) IMDB-B

(g) REDDIT-B

(h) REDDIT-M5

Average ACC of different lambda on MUTAG

Average ACC of different lambda on PROTEINS

Average ACC of different lambda on DD

Average ACC of different lambda on NCI1

78

78

80

9

L

79

76

76

78

adeJaMe

75

75

aa

7

afeane

74

73

73

75

74

71

72

72

1

74

73

72

T-at

100

le3

le4

IDD

le3

1e5

1e-2 1e-1

lambda

100

1e3

1e4

1e5

le-3

1e-1

100

1e3

1e4

1e6

E6

Average ACC of different lambda on COLLAB

Average ACC of different lambda on IMDB-B

Average ACC of different lambda on REDDIT-B

Average ACC of different lambda on REDDIT-M5K

92

68

78

90

ACC

53

66

IS

64

9

05

63

73

ambda

100

le3

1e5

leb

1e-1

10

lambda

100

le3

1e4

1e5

lambda

100

le3

1e4

1e5

let

10

lambda

100

1e3

1e4

1e5

In semi-supervised learning, λ the hyperparameter for the ℓ 2 -norm regularization in semi-supervised Lovász loss as follows

In Figure 8, we fix other hyperparameters and tune λ from { 10 -3 , 10 -2 , ..., 10 5 , 10 6 } . The results show that η is not sensitive when 0 . 1 ≤ γ ≤ 1 e 3 . If λ is a very small number, the supervised encoder and unsupervised encoder may learn different information of a graph G such that the average accuracy slightly decreases. A very large η will cause the training to be trapped in early iterations such that the representation learning fails.

D Ablation study

In this section, we analyze the importance of the orthonormal representation regularization (Eq. (9)) and the subgraph Lovász number (SLN) loss (Eq. (14)) by ablation study on unsupervised learning.

D.1 Ablation study of orthonormal representation regularization

In the ablation study, we remove the orthonormal representation regularization in Eq. (9) by setting µ = 0 . The results in Table 9 show that the orthonormal representation regularization can improve the performance of graph representation learning for two reasons:

The orthonormal representation constraint is part of the definition of Lovász number such that the Lovász principle may fail without the orthonormal representation regularization.

The orthonormal representation regularization guides the method the learn the structure property of a graph.

D.2 Ablation study of subgraph Lovász number (SLN) loss

If we remove the subgraph Lovász number (SLN) loss in Eq. (14) by setting η = 0 , the enhanced Lovász loss (14) degenerates to simple Lovász loss (9). The results in Table 10 show that the subgraph Lovász number (SLN) loss can improve the performance of graph representation learning because similar graphs are guaranteed to be close to each other in the representation space.

D.3 Ablation study of orthonormal representation regularization

E Strict Lovász Principle

We use regularization in Lovász principle (9) instead of constraint because its optimization is much easier and its performance is very close to the constrained optimization. In Algorithm 1, we propose a constrained optimization for the "strict Lovász principle" via projection.

1: Initialization: µ = 1

2: repeat

3: ˆ H t = F ( A,X ; θ t ) and ˆ z t = f ( A,X ; ϕ t )

4: H t = Proj U ( ˆ H t ) and z t = ˆ z t ∥ ˆ z t ∥

5: obtain θ t +1 , ϕ t +1 by SGD updating

6: until Convergence

In Algorithm 1, the Proj U project ˆ H t to the orthonormal representation space, which is similar to the Gram-Schmidt process. We define proj w ( h ) := <h,w> <w,w> w . Let W k be the set that W k := { w 1 , w 2 , ..., w k } . For each vertex i ∈ V , we denote Ω i as the set of vector w j s where j can be each vertex not adjacent to vertex i . Then the H t = Proj U ( ˆ H t ) is defined in Algorithm 2

Algorithm 2: The definition of projection function Proj U

1: w 1 = ˆ h t 1 , e 1 = w 1 ∥ w 1 ∥

3:

...

5: ...

6: w n = ˆ h t n -∑ w ∈ W n -1 ∩ Ω n proj w ( ˆ h t n ) , e n = w n ∥ w n ∥

7: Output H t +1 = [ e 1 , e 2 , ..., e n ] ⊤

The comparisons between the regularized ( µ = 1 ) optimization and the constrained optimization for two methods on four datasets are as follows.

F Time cost comparison

We compare the time cost between the InfoMax principle and Lovász principle using the InfoGraph Sun et al. [2019] model on different datasets. We run the programming on a machine with Intel 7 CPU and RTX 3090 GPU. We repeat the experiment five times and report the results in Table 12. The Lovász principle is the fastest method among the three. The Lovász principle is the fastest method

among the three, and the reasons are as follows:

The Lovász principle is faster than the InfoMax principle because the former does not use the f -GAN [Nowozin et al. , 2016] and the Jensen-Shannon MI estimator I φ to evaluate the mutual information.

Enhanced Lovász principle is slightly lower than Lovász principle because the computation of 1 / ( z ( t -1) ⊤ i u ( t -1) p ) 2 for every p ∈ V i was already done when computing L Lo.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the \"Lovasz Principle\", an unsupervised or semi-supervised graph representation learning approach inspired by the Lovasz number. The motivation for using the Lovasz Principle is well established, and the authors extensively discuss related work and similar approaches. The experimental setup is sound, and the Lovasz Principle is tested against many widely used graph representation learners, with better results in all experiments.\n\nSTRENGTHS:\n- The paper is well-written and easy to follow, with compelling arguments and intuitive explanations.\n\n- The steps taken to derive the Lovasz Principle are well laid on. \n\n- The experimental setup is sound and contains many different approaches to graph representation learning. Variants of the Lovasz Principle consistently obtain the best scores on all datasets. I appreciate the inclusion of Figure 2 for a quick visual summary.\n\n- Compared to InfoMax, the Lovasz Principle is simpler because it does not require a discriminator for the Jensen-Shannon MI estimator. The Lovasz Principle also has a faster runtime.\n\nWEAKNESSES:\n- The authors discuss the non-uniqueness of $U_i*$ and $c_i*$ and give some intuition regarding why this is the case, but mention that they \"hope that similar graphs have similar representations\". It would be interesting and make the argument more compelling if the authors computed $c*$ for various \"similar\" and \"different\" graphs and showed empirically that they are indeed similar or different.\n\n- Similarly, it would be interesting if the authors empirically showed that $\\mathcal{F}_W$ is actually a good approximator for $(U_i*, c_i*)$ by comparing them with $\\mathcal{A}(G_i)$, where $\\mathcal{A}$ is some solver.\n\n\n- It would make the paper more readable if the authors moved the footnote from page 4 to the caption of Tab. 1.\n\n- I appreciate that the authors go into great detail regarding hyperparameter sensitivity in the Supplementary material, but they don't mention the effect of the hyperparameters in the main paper. The reader should be aware that hyperparameters can significantly affect the final performance, and one should consider this when using the Lovasz Principle.\n\n- I could not find any code provided by the authors for reproducibility.\n\nQUESTIONS:\n- While Figure 1 and the description for the pentagon example are great, I was slightly confused when I initially read the part regarding the orthogonal vector pairs. This is because I've read $(u_i*, u_j*)$ as being a vector, not a pair. It would be great if the authors somehow clarified the intuition about the orthogonality.\n\n- I assume that $\\odot$ in Eq. 8 is the Hadamard product? It would be nice to state this somewhere explicitly.\n\n- The authors mention obtaining node-level representations via the $F(\\cdot, \\cdot, \\theta)$ function but are tackling only graph-level tasks. Have you tried node-level tasks? It would be interesting to see the approach tested on them.\n\n- Have you tried assessing if $\\mathcal{F}_W$ is a good approximator for $(U_i*, c_i*)$? Maybe the formulation helps only in a representation-learning sense and doesn't produce a good approximator.\n\n- Will the code be publicly available? I will update my score accordingly if this is the case.\n\nLIMITATIONS:\n- The Lovasz Principle adds some new hyperparameters that could affect the final performance if not correctly chosen. The authors have included an extensive discussion regarding this in the Supplementary to the paper, but I believe this should also be mentioned in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a technique for unsupervised graph-level (and potentially node-level) representation learning on graphs. The main idea behind the proposed method is the concept of the *Lovász number*, a graph invariant that is related to several graph properties and is computed by solving a min-max optimisation problem on the graph. In particular, to compute this, one needs to optimise a graph-level representation (named the *handle vector*), as well as a set of node-level representations (named the *optimal graph representation*), under certain constraints.\n\nAlthough the optimisation problem can be solved in polynomial time, it has several disadvantages (it is computationally expensive, it does not have a unique solution, it cannot generalise to unseen graphs, etc.). The authors propose to overcome them by replacing the conventional optimiser with a neural network-based one, i.e. they train a neural network to map each graph to the aforementioned representations, by optimising the Lovasz number objective (regularised by the constraints) with gradient descent. This idea is extended to a more fine-grained setup in order to incorporate subgraph-level information via the so-called Lovasz kernel (a kernel that compares subgraph-level Lovasz numbers). The method is evaluated on several scenarios (unsupervised, semi-supervised and transfer learning), showing consistently competitive performance, and ablated with regards to some algorithmic choices and hyperparameters.\n\nSTRENGTHS:\n**Presentation and motivation**: The paper is generally well-written (with the exception of some clarity issues – see weaknesses). The idea of the Lovasz number is well-explained and illustrated, as well as its role as a graph description is well-motivated. Moreover, the arguments in favour of devising a learning algorithm instead of a conventional optimiser are well-presented. \n\n**Originality and potential impact**. The concept of the Lovasz number is under-explored by the graph ML community (especially from the perspective of graph neural networks), therefore this paper brings a refreshing idea into the field, that might be useful in topics beyond unsupervised learning and pose some interesting new research questions (I mention some of them in the weaknesses section).  It may also incentivise more research exploring other graph properties and combinatorial optimisation problems for unsupervised learning.\n\n**Empirical results**. The method seems to provide consistent performance improvements over multiple unsupervised learning baselines, which indicates its practicality. Moreover, judging from the reported results in the appendix, it seems relatively robust to hyperparameter choices.\n\nWEAKNESSES:\n**Lack of theoretical justification & ambiguity regarding what the model is learning**. Given the empirical results, it seems that the method has some appealing property that makes it work well in practice. However, perhaps the biggest weakness of this paper is that it remains unclear what this property is and the authors have not discussed this adequately. In particular,\n- It is unclear if the GNN is actually learning the handle vector + optimal graph representations (when optimising Eq. (8)). To be more precise, it is unclear if a GNN *can* express this (mapping a graph to representations that optimise Eq. (2), using a GNN). Given the computational complexity of the problem, I suspect that a GNN (of linear complexity), probably will *not* be able to optimally succeed in this task, but perhaps maybe only approximate it? These are important theoretical questions that should be discussed by the authors.\n- Additionally, the constraints are not guaranteed to hold, which casts further doubt on the ability to converge to the desired representations. I think it would be useful to actually test (e.g. for the models whose performance is reported in the tables) how far are the resulting representations from satisfying the constraints, and how good are they in approximating the Lovasz number. Moreover, do the node features (that are taken into account by the GNN, but not in Eq. (2)) affect the resulting representations? Reporting these results will provide further insights into what these models are actually learning. \n- I am wondering if it is actually desired to learn to infer the handle vector since the results in Table 1 (reported as Lovasz number) are significantly worse than the proposed method (reported as Lovasz principle). Isn’t this contradictory to the initial motivation?\n- Eventually, I am wondering why is this method so good (and better than the Lovasz number) at solving the tested downstream tasks. What kind of information do the resulting representations contain? Maybe something related to substructures? Note that the arguments mentioned by the authors are attractive properties of the Lovasz number, but not the handle vector. \n\n\n**Clarity issues and inadequate justification of design choices**. Some of the choices made by the authors are not sufficiently justified and I’d like to encourage them to provide more details. In particular,\n- Section 3, Eq. (8):  Wouldn’t it be possible to guarantee the satisfaction of the 2nd constraint (unit-length constraint for the handle vector) by construction? For example, isn’t it possible to extract an unconstrained graph representation using a GNN and then divide it by its norm? It would be interesting to discuss if something similar could happen for the 1st constraint.\n- Section 3, Eq. (10): Similarly, here why does one need a second encoder (and therefore the third regularisation term)? Why not use the same encoder and compute both objectives with that?\n- Section 4. This section lacks the required level of detail and some parts are inadequately explained. \n  - I am a bit confused by the iterative nature of the algorithm here. Could the authors elaborate? Do the iterations refer to gradient iterations, i.e. is this akin to bi-level optimisation? If this is the case, why is the first term of Eq. (14) not indexed by $t$? \n  - How many subgraphs are needed to obtain a good estimate of the kernel?\n  - The role of the spectral embedding idea is unclear to me. Why are the handle vectors encouraged to be orthogonal in Eq. (13), and most importantly what is the role of the last term?\n\nQUESTIONS:\n**Experiments**.\n- Could the authors discuss how the loss terms behave in practice (for both Eq. (8) and Eq. (13))? Does one term dominate over the others, and how far are the constraints from being satisfied (this is also related to my first concern)?\n- More experimental/implementation details are needed regarding the transfer learning experiment. The authors mentioned that these can be found in the supplementary material, but I couldn't find enough details. For example, it was unclear how the pre-training is performed and where the Lovasz number loss is used.\n- Appendix Section 3.2.: I found it a bit puzzling that the performance does not deteriorate when $\\mu$ is too large (e.g. 1000). I would expect that these values would imply an unsuccessful optimisation of the first term of the objective. Have the authors tested this (e.g. by testing if the learned graph and node-based representations lead to a value close to the Lovasz number)?\n- Appendix section 4.1. is a bit confusing. It seems that in several cases, using the orthonormality constraint deteriorates performance. Could the authors comment on this?\n- Did the authors reimplement the baselines, in order to ensure fairness of comparison?\n\n\n**Minor**:\n- If I am not mistaken the Lovasz number is computed by also optimising over the dimension of the handle vector. Could the authors account for this as well? \n- Could the authors provide more details on the computational complexity of computing the Lovasz number (this is relevant to the question if a GNN can solve this problem to optimality)?\n- In the last two sections of Table 1, I assume that the authors simply used the neural network architectures of the mentioned papers, however the way the results are presented is a bit confusing. Maybe explicitly mentioning it in the caption would help. Moreover, the important element that should be highlighted in this table, is if the Lovasz optimisation objective improves against the baseline one, so maybe the authors would like to reorder the rows in the table to emphasise this comparison.\n- It might be also useful to add some results in Table 1 from end-to-end supervised learning methods, in order to obtain a more complete picture of the capabilities of the unsupervised methods.\n- Have the authors considered using the node-level representations for node-wise tasks?\n- L291: “For those contrastive…naïve strategy”. Do the authors refer to the baselines? This should be clarified to avoid confusion.\n- Typo: Section 4.3.\n\nLIMITATIONS:\nThe authors make a short discussion on the limitations in the conclusion, but I think more should be mentioned (especially regarding the ambiguity regarding the learned representations - see the weaknesses section).  No foreseeable negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper centers on graph-level representation learning, aimed at converting graphs into vectors useful for downstream tasks like graph classification. The authors propose a unique learning principle named the Lovász principle, inspired by the Lovász number in graph theory. The Lovász number, a real number serving as an upper bound for a graph's Shannon capacity, has strong ties to various global graph characteristics. The authors suggest that the handle vector, used for calculating the Lovász number, could be an effective choice for graph representation given its ability to capture global graph properties. However, its direct application poses challenges. To address these, the authors propose using neural networks to offer the Lovász principle. They also present an enhanced Lovász principle capable of directly and efficiently utilizing subgraph Lovász numbers. Experimental results demonstrate competitive performance of these Lovász principles in comparison to the baselines in both unsupervised and semi-supervised graph-level representation learning tasks.\n\nSTRENGTHS:\n1. The authors' proposal of Lovász theta kernels for graph representation learning shows great promise, thanks to its theoretical foundation.\n2. The paper includes thorough experiments conducted in various settings, including unsupervised, semi-supervised, and transfer learning. These experiments provide strong evidence of the effectiveness of the proposed methods.\n\nWEAKNESSES:\n1. Although the author proposed a new graph kernel, the contribution seems slightly small to me.\n2. The author did not demonstrate the framework of the model.\n3. Some classic baselines were not compared.\n\nQUESTIONS:\n1. I strongly suggest that the author discuss the application of graph kernel in updated scenarios, such as graph prompt learning. It is highly recommended that the authors consider citing the prompt learning on graph[1,2] paper and explore its application in conjunction with the proposed method in this paper, especially for unsupervised settings. This would enhance the discussion and potential application of prompt-based models.\n\n [1] Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. Z Liu, X Yu, Y Fang, X Zhang - Proceedings of the ACM Web Conference 2023, 2023\n\n [2] Sun, M., Zhou, K., He, X., Wang, Y. and Wang, X., 2022, August. Gppt: Graph pre-training and prompt tuning to generalize graph neural networks. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining.\n\n2. Providing a framework of the proposed model can indeed enhance the clarity of its functionality.\n3. Why semi-supervised learning baseliens such as GCN, GAT, GIN are not compared in the paper?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a new method for graph representation in supervised, semi-supervised and transfer learning based on the Lovasz theta function [Lov79]. They also incorporate local information using Lovasz subgraph numbers inspired by the work on Lovasz theta kernal [Joh+14]. They also present empirical evaluation and show that their method is competitive with state-of-the-art.\n\nSTRENGTHS:\n- The paper is easy to follow for the most part.  \n- The empirical evaluation shows the method performs well compared to existing methods.\n\nWEAKNESSES:\nSome rewriting needed on Section 4 and minor typos. Also ablation studies and final parameter selection still a bit unclear. \n\n**SLN** \n\n- Lines 209-211. Please define \"subgraph Lovasz number\" explicitly. \n\n- Lines 214-218. The description of Lovasz $\\vartheta$ kernel [Joh+14] is misleading. \n    - Lines 214-216. Possibly I misunderstood this, but subgraph lovasz value as defined in [Joh+14, Definition 2] can be well-approximated using $O(n \\log n)$ samples [Joh+14, Thm. 1].\n\n    - Lines 217-218. In so far as the kernel being a pair-wise method and not capturing the \"global structure\" of $\\cal G$, here is the trivial equivalent using the kernel:  $ \\sum_{i=1}^{\\cal G} \\sum_{j=1}^{\\cal G} \\hat{k}_{\\tt Lo}(G_i, G_j) $.\n\n- Lines 226-229. How is $K^{(t)}_{ij}$ computed, more precisely the number of  sampled subgraphs and how those are selected? \n\n**Ablation and default parameters** \n\n- The ablation study has explored parameter choices for $\\mu$, $\\eta$, etc. over log-ranges for the different graph datasets. However, what is not so clear is the influence of graph dataset properties (e.g., size of the graphs, density, node properties, etc.) \n\n- It would be great if the authors could present preferably in the main text or the appendix, a set of sane starting points. Of course, for each use-case, HPO should be done but often, the practitioner take is to go with defaults initially. \n\n**Minor typos**\n\n- Appendix. Section 4. lines 80-91 \"orthnormal\" -> \"orthonormal\"\n- Appendix. Section 5. line 95. \"guarantee\" -> \"guaranteed\"\n\nQUESTIONS:\n- It would be interesting to see in eqn (9) how the different loss terms during training corresponding to the theta-value ($l1$) and  orthonormal representations ($l2$ and $l3$) and similarly for SLN when thinking of the final READOUT as non-linear analogues of the Lovasz representation.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission presents a new approach to graph-level representation learning inspired by the Lovász number in graph theory. Specifically, the study proposes using the Lovász principle as a novel framework for unsupervised/semi-supervised graph representation learning. It offers a method to utilize handle vectors, which capture a graph's global features through neural networks. An enhanced Lovász principle is also proposed, which efficiently uses subgraph Lovász numbers, thereby ensuring similar graphs have similar representations. Experimental results show that the Lovász principles outperform other graph representation methods (such as InfoMax) in unsupervised learning, semi-supervised learning, and transfer learning. Therefore, the Lovász principles offer a promising new approach to graph-level representation learning.\n\nSTRENGTHS:\nThere are three main strengths:\n\n1. A new approach based on the Lovász principle is proposed for graph-level representation learning. This principle, inspired by the Lovász number in graph theory, adds a new perspective and strategy to graph learning, shifting from traditional methods.\n\n2. Some subgraph tricks are used to enhance the performance of Loasz-based GNN models.\n\n3. The Lovász principles outperform other methods in experiments on unsupervised learning, semi-supervised learning, and transfer learning.\n\nWEAKNESSES:\nWeaknesses:\n\n1. Compared with current GNN models, such as Infomax-based and kernel-based methods, there are indeed some gains in terms of ACC on the task of graph classification. However, the overall gained performance in Table 1,2, and 3 is not very significant, especially when you consider the variances. This may question whether Lovász Principle can provide more useful global information than InfoMax. The other concern is that it is unclear how the approximation affects the whole performance since the final vectors used in the proposed models are approximated.\n\n2. In terms of the novelty of using the Lovász Principle for designing GNN models, one of my main concerns is that this idea is not very novel. There is a lack of discussion on the difference between current work and previous works. See related works in [1]. In terms of run-time comparison, there is no significant reduction compared with InfoMax based on the run-time table in the appendix.\n\n[1] Yadav, P., Nimishakavi, M., Yadati, N., Vashishth, S., Rajkumar, A. and Talukdar, P., 2019, April. Lovasz convolutional networks. In The 22nd international conference on artificial intelligence and statistics (pp. 1978-1987). PMLR.\n\nQUESTIONS:\nOverall, this paper has a good idea of using the Lovasz Principle (the handle vector) to present the graph and incorporate this idea to design GNN models. However, there is some related work, and the authors should have discussed it in the submission.\n\nQ1: How could you deal with large-size graphs? For example, the datasets RDT-B and RDT-M5K were used in [2]. My concern is that it is unclear how you can estimate C_{S_i,S_j} accurately when n, the number of nodes in the graph, is large. \n\nQ2. Compared with InfoMax, why the performance of InfoMax-based (in terms of best ones) are very close to the proposed? More discussions are needed.\n\n[2] Sun, Fan-Yun, Jordan Hoffmann, Vikas Verma, and Jian Tang. \"Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization.\" arXiv preprint arXiv:1908.01000 (2019).\n\nSome minors:\n\n1. It is helpful to list out all dataset statistics for people who are unfamiliar with this area.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}