Title: Unsupervised Graph Neural Architecture Search with Disentangled Self-Supervision
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph Neural Architecture Search

Unsupervised Learning

Self-supervised Learning
Abstract: The existing graph neural architecture search (GNAS) methods heavily rely on supervised labels during the search process, failing to handle ubiquitous scenarios where supervisions are not available. In this paper, we study the problem of unsupervised graph neural architecture search, which remains unexplored in the literature. The key problem is to discover the latent graph factors that drive the formation of graph data as well as the underlying relations between the factors and the optimal neural architectures. Handling this problem is challenging given that the latent graph factors together with architectures are highly entangled due to the nature of the graph and the complexity of the neural architecture search process. To address the challenge, we propose a novel Disentangled Self-supervised Graph Neural Architecture Search (DSGAS) model, which is able to discover the optimal architectures capturing various latent graph factors in a self-supervised fashion based on unlabeled graph data. Specifically, we first design a disentangled graph super-network capable of incorporating multiple architectures with factor-wise disentanglement, which are optimized simultaneously. Then, we estimate the performance of architectures under different factors by our proposed self-supervised training with joint architecture-graph disentanglement. Finally, we propose a contrastive search with architecture augmentations to discover architectures with factor-specific expertise. Extensive experiments on 11 real-world datasets demonstrate that the proposed model is able to achieve state-of-the-art performance against several baseline methods in an unsupervised manner.

Unsupervised Graph Neural Architecture Search with Disentangled Self-supervision

Abstract

The existing graph neural architecture search (GNAS) methods heavily rely on supervised labels during the search process, failing to handle ubiquitous scenarios where supervisions are not available. In this paper, we study the problem of unsupervised graph neural architecture search, which remains unexplored in the literature. The key problem is to discover the latent graph factors that drive the formation of graph data as well as the underlying relations between the factors and the optimal neural architectures. Handling this problem is challenging given that the latent graph factors together with architectures are highly entangled due to the nature of the graph and the complexity of the neural architecture search process. To address the challenge, we propose a novel Disentangled Self-supervised Graph Neural Architecture Search ( DSGAS ) model, which is able to discover the optimal architectures capturing various latent graph factors in a self-supervised fashion based on unlabeled graph data. Specifically, we first design a disentangled graph super-network capable of incorporating multiple architectures with factor-wise disentanglement, which are optimized simultaneously. Then, we estimate the performance of architectures under different factors by our proposed self-supervised training with joint architecture-graph disentanglement. Finally, we propose a contrastive search with architecture augmentations to discover architectures with factor-specific expertise. Extensive experiments on 11 real-world datasets demonstrate that the proposed DSGAS model is able to achieve state-ofthe-art performance against several baseline methods in an unsupervised manner.

1 Introduction

Graph neural architecture search (GNAS), aiming to automatically discover the optimal architecture for graph neural network (GNN) based on graph-structured data and task, has shown remarkable progress in enhancing the predictive power and saving human endeavors for various graph applications [1]. The existing GNAS methods generally follow a supervised paradigm such that they optimize the weights within architectures given a training dataset with a supervised loss (e.g., the cross entropy loss of label predictions) and estimate the architecture performance based on the validation dataset with supervision signals. For example, the label prediction accuracy is adopted for architecture ranking during the neural architecture search process [2, 3, 4]. As a result, supervised labels become indispensable for applying the existing GNAS methods.

However, ground-truth labels in reality may be extremely scarce or hardly available in many graph applications. For example, a variety of biological problems require a significant amount of human labors and time costs in clinical tests to obtain labels for supervision [5, 6, 7]. As the existing

GNAS approaches heavily rely on supervised labels for weight training and architecture evaluation, they will suffer from performance deterioration in unsupervised settings, failing to discover optimal architectures in the scenarios where labels are scarce or not available.

In this paper, we study unsupervised graph neural architecture search, i.e., discovering optimal GNN architectures without labels for graph-structured data, which remains unexplored in the literature. The key problem lies in two important aspects: i) discover the latent graph factors that drive the formation process of graph data [8, 9, 10, 11, 12, 13, 14]; ii) capture the underlying relations between the factors and the optimal neural architectures. For instance, a molecular graph may consist of groups of atoms as well as bonds representing different functional units [15], requiring different optimal neural architectures to make accurate predictions.

Nevertheless, solving the problem is highly non-trivial and challenging given that the hidden factors are entangled in the graph and very difficult to capture, e.g., a social network may contain several communities originating from various interests (e.g., sports, games, etc.) [16, 10], with the nodes and edges belonging to different communities mixing together. Moreover, the architectures with different functional factors are also entangled within the weight-sharing super-network [17, 18, 19], resulting in inaccurate architecture performance estimations under different hidden factors.

To tackle the challenge, we propose a novel unsupervised graph neural architecture search method, i.e., Disentangled Self-supervised Graph Neural Architecture Search ( DSGAS ) 3 . Given graph data without supervised labels, our proposed DSGAS model can discover the optimal architectures capturing multiple latent factors in a self-supervised fashion. In particular, we first design a disentangled graph super-network, where multiple architectures are disentangled for simultaneous optimization w.r.t various latent factors. Then, we propose a self-supervised training with joint architecturegraph disentanglement, which disentangles architectures and graphs within a common latent space. The super-network is trained through a routing mechanism between architectures, graphs and selfsupervised tasks, to obtain an accurate estimation of the architecture performance under each latent factor. Finally, we propose a contrastive search with architecture augmentations, where a novel architecture-level instance discrimination task is introduced to discover architectures with distinct capabilities of capturing various factors in a self-supervised fashion. Extensive experiments show that the proposed DSGAS model is able to significantly outperform the state-of-the-art GNAS baselines under both unsupervised and semi-supervised settings. Detailed ablation studies and analyses also demonstrate that DSGAS is able to discover effective architectures with our proposed disentangled self-supervision designs. The contributions of this paper are summarized as follows:

We are the first to study the problem of unsupervised graph neural architecture search and propose the Disentangled Self-supervised Graph Neural Architecture Search ( DSGAS ) model capable of discovering the optimal architectures without labels, to the best of our knowledge.

We introduce three novel modules, i) disentangled graph architecture super-network, ii) selfsupervised training with joint architecture-graph disentanglement and iii) contrastive search with architecture augmentations, which can discover the optimal architectures capturing various graph latent factors with disentangled self-supervision.

Extensive experiments on 11 real-world graph datasets show that our proposed method DSGAS is able to discover effective graph neural architectures without supervised labels and significantly outperform the state-of-the-art baselines in both unsupervised and semi-supervised settings.

2 Preliminaries and Problem Formulation

Graph Neural Architecture Search Denote the graph space as G and the label space as Y . A graph neural network can be denoted as a function f α,w : G → Y , which is characterized by architecture parameters α ∈ A and learnable weights w ∈ W given an architecture space A and a weight space W . Graph neural architecture search (GNAS) aims at automating the design of graph neural architectures, i.e., obtaining the best-performed architectures by searching α . As α is usually instantiated as selecting GNN operations, e.g., GCN [20], GAT [21], GIN [22], we also call α as

Disentangled Self-Supervised Graph Neural Architecture Search

Self-supervised Training with Joint

Architecture-Graph Disentanglement

Disentangled Graph

Architecture Super-Network

Contrastive Search with

Architecture Augmentations

GCN

GAT

GIN

Sage

…

Disentangled

Architectures

𝛼1

𝛼2

𝛼𝑘

…

0

1

2

3

𝜶𝟏

𝜶𝒌

𝜶𝟐

𝒄𝟏

𝒄𝒌

𝒄𝟐

∇𝑤ℒ𝑤

Update

𝑤

and

𝛼

Latent

Factors

Pretext tasks

under Factors

Architecture Augmentations

𝑇 𝑓

Operation Choice

Perturbation

Weight

Perturbation

Embed

Dropout

𝑓 𝛼 1 ,𝑤

′

𝑓 𝛼 2 ,𝑤

′

𝑓 𝛼 𝑘 ,𝑤

′

…

𝑓 𝛼 1 ,𝑤

𝑓 𝛼 2 ,𝑤

𝑓 𝛼 𝑘 ,𝑤

…

…

Contrastive

Search

Mixed Operation

∇𝛼ℒ𝛼

Graph Data

operation choices for brevity. Generally, GNAS solves the bi-level optimization problem [23] :

where L train and L val denotes the loss of the predictions of the architecture f α,w ( · ) against supervised labels on training and validation datasets. The optimization problem can be viewed as having two objectives that Eq.(2) aims to obtain accurate architecture performance estimation, and Eq.(1) aims to search the best-performed architectures. To avoid the cost of training from scratch for each architecture, the super-network [24, 25] arises as a commonly adopted technique to obtain faster architecture performance estimation, where the architecture candidates are viewed as sub-networks of the super-network, and their weights are shared during the training process.

Unsupervised Graph Neural Architecture Search We consider the problem of unsupervised graph neural architecture where labels, which are adopted for the performance estimation and the search process in the supervised GNAS, are not accessible. The problem of unsupervised GNAS can be formulated as optimizing an architecture generator that is able to discover powerful architectures by exploiting inherent graph properties without labels, i.e., G ↦→ f α,w instead of ( G , Y ) ↦→ f α,w as done by supervised GNAS methods. Then, the discovered architectures f α,w ( · ) can be utilized in downstream tasks, e.g., finetuning the weights w or the operation choices α , or extra shallow classifiers for further prediction.

3 Disentangled Self-supervised Graph Neural Architecture Search

In this section, we introduce Disentangled Self-supervised Graph Neural Architecture Search ( DSGAS ) to search architectures without labels, by proposing three key components: disentangled graph architecture super-network, self-supervised training with joint architecture-graph disentanglement, and contrastive search with architecture augmentations.

3.1 Disentangled Graph Architecture Super-Network

To discover architectures that potentially have optimal performance, we resort to guiding the search towards architectures' capabilities of capturing the inherent graph factors, which are shown important in the graph formation [8, 9]. As architectures may expert in different graph factors, we propose a

disentangled graph architecture super-network to incorporate K different architectures to be estimated and searched w.r.t factors simultaneously, where the hyperparameter K denotes the number of factors.

Disentangled Super-Network Layer For each super-network layer, we adopt K mixed operations parameterized by different α to learn K -chunk graph representations:

where A is the adjacency matrix of the graph, H denotes the input graph representations, and GNN α k ( · ) denotes the mixed GNN operations parameterized by α k . For the convenience of differentiable optimization, we adopt continuous parameterization and weight-sharing mechanism [25] to implement the mixed operations:

where |O| is the number of GNN operation choices, α k,i = exp( θ α k,i ) ∑ j exp( θ α k,j ) denotes the probability of the i -th operation for the k -th architecture α k , and θ is learnable parameters.

Overall Disentangled Super-Network The overall super-network is constructed in the form of a directed acyclic graph (DAG) with an ordered sequence of disentangled super-network layers. More details about the DAG construction and the according GNN operations for each disentangled super-network layer are included in Appendix. The output of the last layer Z = [ H 1 , H 2 , . . . , H K ] describes the various aspects of the graphs and serves as the final graph representations, which can be utilized or finetuned in downstream tasks. In this way, the architectures' operation choices α = [ α 1 , α 2 , . . . , α K ] and weights w are incorporated in one super-network. For brevity, we use f α k ,w ( · ) to denote the k -th architecture induced from the super-network. Note that the design of K operation choices alleviates the entanglement of architectures by providing more flexible choices of paths [26] in the super-network. For instance, the 'mean' operation captures structural properties while the 'max' operation captures representative elements [22], and in this case, our design can capture both of them by choosing corresponding operations to learn respective representations instead of choosing only one of them which may conflict each other.

3.2 Self-supervised Training with Joint Architecture-Graph Disentanglement

Inspired by graph self-supervised learning, we utilize graph pretext tasks to measure the architectures' capabilities of capturing latent factors. Predictive pretext tasks [27], for example, design a pseudo label generator s ( · ) , and optimize the prediction probability 4 p ( s ( G i ) |G i ) . However, the tasks usually take holistic views of the graphs and neglect the entanglement of the latent factors, which may lead to suboptimal performance estimation. Therefore, to disentangle the factors, we transform the probability into the expectation of multiple subtasks under different latent factors by the Bayesian formula:

where p ( k |G i ) denotes the probability of latent factor k given the i -th graph instance G i , and p ( s ( G i ) |G i , k ) denotes the pretext task under k -th latent factor. An intuitive explanation of Eq. (5) is that it first infers the latent factors and then conducts factor-specific self-supervised training to capture the latent factors, which we describe in detail as follows.

Architecture-aware Latent Factor Inference Directly modeling p ( k |G i ) is difficult as we do not know prior what GNN encoders are suitable for inferring the latent factors. Intuitively, one solution is to get the architectures being searched involved in the inference stage. By the Bayesian formula, we factorize the probability w.r.t architecture choices:

where p ( α j |G i ) is a prior distribution, and we adopt a uniform distribution for simplicity. Then we can model the probability distributions of latent factors given the graph G i by utilizing the architectures being searched:

where c k is a learnable vector to represent the k -th latent factor, and z i,k = f α k ,w ( G i ) denotes the graph representations output by the k -th architecture for the graph G i . Enc ( · ) denotes architecture encoding techniques to obtain embeddings of operation choices α so that the structural properties and correlations of the neural architectures can be considered [28, 29].

Factor-aware Graph Self-Supervised Learning Under the k -th latent factor, we leverage the corresponding architectures f α k ,w ( · ) for conducting the factor-specific pretext tasks as p ( s ( G i ) |G i , k ) to estimate the capturing capabilities of architectures under various factors. The overall objective is to maximize Eq.(5), and the loss can be calculated by

where N is the number of samples and the upper bound is obtained by Jensen's Inequality. Then we can generalize our method to other graph self-supervised tasks with specially-designed task loss functions by defining -log p ( s ( G i ) |G i , k ) as the task loss function l ( f α k ,w , G i ) for the graph G i under the k -th factor, and calculate the loss by

In this way, the disentangled architectures in Sec. 3.1 coupled with factors disentangled from graph data can be routed pairwisely, and trained with factor-specific self-supervision to obtain more accurate performance estimation under each factor. Similar to [25], the super-network weights are updated with w = w -λ w ∇ w L w to obtain the weights that can represent the architectures' capabilities.

3.3 Contrastive Search with Architecture Augmentations

In this section, we focus on encouraging the disentanglement of architectures and searching architectures with distinct capabilities of capturing different factors. The main insight of our proposed search method is intuitively based on the following two observations shown in the literature: 1) As architectures similar in operation choices and topologies have similar capabilities of capturing semantics for downstream tasks [28, 30, 31], slight modifying the architecture will have a slight influence on its capability. 2) Since different GNN architectures expert in different downstream tasks [32], the architectures searched for different disentangled latent factors are expected to have dissimilar capabilities under different factors.

Contrastive Search Inspired by self-supervised contrastive learning [33, 34] that capture discriminative features by pulling similar instances together and pushing dissimilar instances away in the latent space, we propose an architecture-level instance discrimination task to encourage the architectures to capture various latent factors. The task is defined as

where s ( α k ) is assigned to k as surrogate labels for the architecture, ϕ ( · ) calculates the similarity of two embeddings and T f ( · ) denotes architecture augmentations that transform an architecture to another architecture with similar capabilities capturing factors, i.e., f α k ,w ↦→ f ′ α k ,w . Then the loss function can be calculated by

Similar to [25], the architecture parameters are updated with α = α -λ α ∇ α L α to search architectures with better capabilities of capturing factors.

Architecture Augmentations To create various views of architectures, we design three basic architecture augmentations from the perspectives of architecture operation choices α , architecture weights w and the internal embeddings H :

Operation Choice Perturbation. This augmentation randomly reshapes the distributions of the mixed operations by altering the temperature in the softmax function in Eq. (4) with

where the temperature τ is sampled from a uniform distribution U ([1 /r 1 , r 1 ]) , and r 1 ≥ 1 is a hyper-parameter controlling the perturbation degree.

Weight Perturbation. This augmentation randomly adds Gaussian noises ϵ ∼ N (0 , σ 2 ) to r 2 % of the architecture weights w , where r 2 controls the perturbation ratio, and σ 2 is the standard deviation of the weights.

Embedding Dropout. This augmentation randomly drops r 3 %of the embeddings H output from the mixed operations, where r 3 controls the dropout ratio.

Note that these architecture augmentations can further be randomly composed to generate mixed augmentations.

4 Experiments

In this section, we conduct experiments on 8 real-world datasets with unsupervised settings to verify the design of our method. We also include detailed ablation studies to analyze the effectiveness of each component, and 3 real-world datasets in semi-supervised settings to show that our method can alleviate the label scarcity issues by pretraining the super-network.

Baselines We compare our method with 11 baselines from the following two different categories.

Manually designed GNNs. We include five representative GNNs as our baselines, i.e., GCN [20], GAT [21], GIN [22], GraphSage [35], GraphConv [36] and a simple baseline MLP, which constitutes our search space. For graph-level classification tasks, we adopt global mean pooling for each layer and concatenation to obtain the graph representations for these baselines.

Graph neural architecture search. We include representative GNAS baselines GraphNAS [3], PAS [4] and GASSO [37], where PAS is specially designed for graph-level classification tasks by searching the pooling operations, and GASSO is specially designed for node-level classification tasks by searching the graph structures simultaneously. We also include two classical NAS baselines, random search and DARTS [25]. As these baselines are not specially designed for graphs, we adopt our search space for these baselines.

Datasets For unsupervised settings, we conduct experiments on four graph-level classification datasets including PROTEINS [38], DD [39], MUTAG [40], IMDB-B [41] from TUDataset [42] and four node-level classification datasets Coauthor CS, Coauthor Physics from the Microsoft Academic Graph [43], Amazon Computers, Amazon Photos from the Amazon Co-purchase Graph [44]. For semi-supervised settings, we adopt three real-world datasets, OGBG-Molhiv, OGBN-Arxiv [45]

and Wechat-Video 5 . The datasets cover various graph-related fields including small molecules, bioinformatics, social networks, e-commerce networks, and academic coauthorship networks. The statistics are summarized in Table 1.

Super-network We briefly introduce the super-network construction as follows. The super-network consists of two parts, the operation pool and the directed acyclic graph (DAG). The operation pool includes several node aggregation operations (e.g., GCN, GIN, GAT, etc.), graph pooling operations (e.g., SortPool, AttentionPool, etc.), and layer merging operations (e.g., MaxMerge, ConcatMerge, etc.). The DAG determines how the operations are connected to calculate the graph representations for the subsequent classification tasks.

More details of the experiments are provided in the Appendix, including additional experiments and analyses, experimental setups, configurations, and implementation details.

4.1 Main Results

Unsupervised Settings From the experimental results summarized in Table 2, we have the following observations: 1) GNNs perform differently on various datasets. The best GNN baselines for these datasets are GraphSage, GAT, GraphConv, GraphConv, GAT, GCN, GCN, GraphConv successively, and the performance varies greatly across datasets and GNNs. It verifies that no GNN architecture is dominant for all datasets, which is consistent with the literature [32] and shows the demand for automated GNN architecture designing based on the data characteristics to obtain the optimal representations. 2) Most GNAS baselines fail in the unsupervised setting. Since the existing GNAS baselines highly rely on supervised signals to search the architectures, they inherently do not suit the unsupervised settings. As an ad hoc remedy for the existing GNAS baselines in unsupervised settings, we substitute the supervised metrics with the self-supervised ones during the searching process as simple extensions. However, these GNAS baselines, contrary to supervised settings, do not guarantee better performance than manually designed GNNs for all datasets. For example, all GNAS baselines even perform worse than manually designed GNNs on DD dataset and do not have significant improvements on most datasets. The reasons behind might be that simply using graph self-supervised metrics does not consider the entanglement of architectures and factors, leading to inaccurate estimation of the architectures' capabilities. 3) Our method has significant improvements over the baselines on most datasets. Compared with manually designed GNN baselines, DSGAS has a performance improvement of 3.1% on PROTEINS and over 1% on most datasets. We contribute this to its ability of automatically tailoring GNNs for various datasets, showing its effectiveness of automatic GNN designing. DSGAS also significantly surpasses GNAS baselines, showing its superiority in graph neural architecture search in unsupervised settings, which benefits from the design of discovering architectures that can capture various graph factors in a self-supervised fashion.

10%

5%

1%

(OGBG-Molhiv)

53

57

61

65

69

Performance (%)

10%

5%

1%

(OGBN-Arxiv)

58

61

64

67

10%

5%

1%

(Wechat-Video)

74

77

80

83

86

DARTS

PAS

GASSO

DSGAS-P

DSGAS

PROTEINS

74

75

76

Accuracy (%)

DD

75

76

77

78

Computers

83

84

85

86

87

Photo

91

92

93

DSGAS

DSGAS-CONT

DSGAS-FA

DSGAS-DISEN

(a)

PROTEINS

74

75

76

Accuracy (%)

DD

75

76

77

78

79

Computers

84

85

86

87

Photo

91

92

93

Compose

Alpha

Weight

Embed

(b)

Semi-supervised Settings From Figure 2, we have the following observations: Compared with the baselines, DSGAS significantly alleviates the performance drop when the number of available supervised labels is fewer, which verifies that our method fully exploits latent factors inside graph data and boost the supervised architecture search stage by warming up the weights and architecture parameters of the super-network. Its significant improvement over the ablated version DSGAS-P also verifies the effectiveness of pretraining the super-network by the proposed modules of self-supervised training with joint architecture-graph disentanglement and contrastive search with architecture augmentations. For example, our model with the pretraining stage has an absolute improvement of 5% on Wechat-Video dataset with 1% labels compared with the ablated version without the pretraining stage, which shows the effects of pretraining the super-networks on alleviating the label scarcity issues. In comparison, the performance of other baselines decays significantly more than our method, showing that current GNAS can not tackle scenarios with scarce labels. For example, on OGBG-Molhiv, PAS is the best baseline while the worst with 10% and 1% labels respectively, which may due to the inaccurate performance estimation with scarce labels. The phenomenon further strengthens the necessity of designing effective unsupervised GNAS methods.

4.2 Additional Experiments

Ablation Studies We evaluate the effectiveness of each module of our framework by comparing the following ablated versions of our method: DSGAS-CONT removes our proposed contrastive search module and search architectures with the vanilla self-supervised loss. DSGAS-FA further replaces our proposed factor-aware training module with the vanilla self-supervised training. DSGAS-DISEN further replaces our proposed disentangled super-network with the vanilla super-network.

INPUT

GIN

GAT

GraphConv

CONCAT

GIN

GraphConv

GCN

(a) Computers

INPUT

GraphConv

GIN

CONCAT

GAT

SumPool

GraphConv

GCN

AttPool

MaxPool

GIN

GraphConv

SortPool

SumPool

GCN

MaxPool

MeanPool

MaxPool

SumPool

ConcatMerge

SumMerge

MaxMerge

(b) DD

INPUT

GraphConv

GIN

CONCAT

GIN

GraphConv

(c) Photo

INPUT

GIN

GraphConv

GraphSage

CONCAT

GCN

SortPool

GraphSage

MLP

SumPool

AttPool

GIN

GraphConv

MeanPool

SortPool

GraphSage

MLP

SortPool

AttPool

MeanPool

SumPool

SumMerge

ConcatMerge

MaxMerge

(d) PROTEINS

We compare the performance of the ablated versions and the best GNAS baselines on the real-world datasets under unsupervised settings. From Figure 3a, we have the following observations. First, our proposed DSGAS outperforms all the variants on all datasets, demonstrating the effectiveness of each component of our proposed framework in searching graph architectures under unsupervised settings. Second, DSGAS-CONT drops drastically in performance on all datasets compared to the full version, showing the superiority of our proposed disentangled contrastive architecture search module in searching architectures. Third, the performance also decays for DSGAS-FA and DSGAS-DISEN on most datasets, showing the necessity of capturing various latent factors with different architectures.

Architecture Visualizations We visualize the architectures searched on different unsupervised datasets in Figure 4. It shows that the searched architectures of different factors adopt quite different GNN operations while sometimes sharing the same operations, which leads to an overall architecture with complex internal connections between operations. This phenomenon implies that DSGAS can optimize the architecture operation choices as well as the operation connections for different factors to have a competitive performance on various graph datasets, which also verifies the superiority of DSGAS in automated architecture search to save human endeavors for architecture designs.

Effects of Architecture Augmentations In Figure 3b, we show the results of different architecture augmentation methods on different datasets compared with the best GNAS baseline. We find that though the best augmentations differ among datasets, they have similar performance improvements in most cases, which verifies the design of contrastive search with architecture augmentations.

5 Related Works

Graph Neural Architecture Search Instead of manually designing more sophisticated models for various scenarios, neural architecture search, aiming to automatically discover the optimal architectures for given tasks, emerges as a hot topic recently in computer vision [46, 23], natural language processing [47], graph representation learning [1, 48, 49], etc. In the field of graph representation learning with various applications [50, 51, 52, 53, 54], graph neural architecture search (GNAS) methods, as the most related to our works, can be roughly classified into reinforcementlearning-based methods [3, 55], evolutionary-based methods [56, 2, 57, 58], and differentiable methods [59, 60, 61, 62, 37, 63, 4, 64, 65]. However, supervised labels are indispensable for the existing GNAS methods to conduct neural architecture search, which limits their applications in widely-existed scenarios where labels are scarce or not available.

Unsupervised Neural Architecture Search In unsupervised settings, some neural architecture search methods replace supervised labels with self-supervised loss during searching [66, 67, 68, 69, 70, 71]. Another classic of related methods design special metrics, whose calculation does not depend on labels, as proxies for model performance estimation [72, 73]. For example, UnNAS [74] adopts pretext tasks like image rotation, coloring images, solving puzzles, etc. However, these methods are specially designed for computer vision, and can not be directly adopted to graph data. Some GNAS

works exploit self-supervised loss as auxiliaries to augment the supervised search process [63], but the supervised labels are still mandatory for its search. To the best of our knowledge, this is the first work on unsupervised graph neural architecture search.

Graph Self-supervised Learning Graph self-supervised learning [75, 76] is devoted to obtaining graph representations by extracting informative knowledge with well-designed pretext tasks without labels, which can be roughly classified into contrastive [77, 78, 79, 80, 81, 82, 83, 84, 85, 86] and generative [87, 88, 76, 89], and predictive methods [90, 91, 92]. The existing graph self-supervised learning methods usually focus on designing better pretext tasks with a fixed encoder such as GCN [20], GAT [21] and GIN [22]. Another class of related methods attempt to automate the choices of pretext tasks [93, 94, 95, 96, 97]. We mainly consider graph neural architecture in unsupervised settings, while other pretext tasks are orthogonal to our framework and can be incorporated.

Disentangled Representation Learning The primary objective of disentangled representation learning is to delineate and interpret the various latent factors which influence the data we encounter in an observable context, rendering each of these factors as unique vector representations [98, 99]. It has emerged to be a useful tool in various domains, including those of computer vision [100, 101, 102, 103, 104, 105], and recommendation systems [106, 107, 108, 109, 110, 111, 112, 113], graph representation learning [9, 114, 10, 14, 11, 115, 116, 117]. As the most related, GRACES [63] characterize the graph latent factors inside data by designing a self-supervised disentangled graph encoder, and conduct graph neural architecture search for each graph to handle graph distribution shifts, while the training and searching process still follows the supervised paradigm. In contrast, we focus on automating the GNN designs with disentangled self-supervision in this paper.

6 Conclusions

In this paper, we propose a novel Disentangled Self-Supervised Graph Neural Architecture Search ( DSGAS ) framework to automate the GNN designs with disentangled self-supervision, which includes disentangled graph architecture super-network, self-supervised training with joint architecture-graph disentanglement and contrastive search with architecture augmentations. Extensive experiments demonstrate that our proposed method can discover architectures with capabilities of capturing various graph latent factors and significantly outperform the state-of-the-art GNAS baselines. Detailed ablation studies and analyses show the effectiveness of our method design. One limitation is that in this paper we mainly focus on homogeneous graphs, and we leave extending our method to heterogeneous graphs for further explorations.

References

Ziwei Zhang, Xin Wang, and Wenwu Zhu. Automated machine learning on graphs: A survey. arXiv preprint arXiv:2103.00742 , 2021.

Yaoman Li and Irwin King. Autograph: Automated graph neural network. In International Conference on Neural Information Processing , pages 189-201, 2020.

Yang Gao, Hong Yang, Peng Zhang, Chuan Zhou, and Yue Hu. Graph neural architecture search. In IJCAI , volume 20, pages 1403-1409, 2020.

Lanning Wei, Huan Zhao, Quanming Yao, and Zhiqiang He. Pooling architecture search for graph classification. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 2091-2100, 2021.

Linda Martin, Melissa Hutchens, Conrad Hawkins, and Alaina Radnov. How much do clinical trials cost. Nat Rev Drug Discov , 16(6):381-382, 2017.

Debleena Paul, Gaurav Sanap, Snehal Shenoy, Dnyaneshwar Kalyane, Kiran Kalia, and Rakesh K Tekade. Artificial intelligence in drug discovery and development. Drug discovery today , 26(1):80, 2021.

Duxin Sun, Wei Gao, Hongxiang Hu, and Simon Zhou. Why 90% of clinical drug development fails and how to improve it? Acta Pharmaceutica Sinica B , 2022.

Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. Graph neural networks for social recommendation. In The world wide web conference , pages 417-426, 2019.

Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, and Wenwu Zhu. Disentangled graph convolutional networks. In International conference on machine learning , pages 4212-4221. PMLR, 2019.

Yiding Yang, Zunlei Feng, Mingli Song, and Xinchao Wang. Factorizable graph convolutional networks. Advances in Neural Information Processing Systems , 33:20286-20296, 2020.

Ansong Li, Zhiyong Cheng, Fan Liu, Zan Gao, Weili Guan, and Yuxin Peng. Disentangled graph neural networks for session-based recommendation. arXiv preprint arXiv:2201.03482 , 2022.

Thilini Cooray and Ngai-Man Cheung. Graph-wise common latent factor extraction for unsupervised graph representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 6420-6428, 2022.

Teng Xiao, Zhengyu Chen, Zhimeng Guo, Zeyang Zhuang, and Suhang Wang. Decoupled self-supervised learning for non-homophilous graphs. arXiv preprint arXiv:2206.03601 , 2022.

Haoyang Li, Xin Wang, Ziwei Zhang, Zehuan Yuan, Hang Li, and Wenwu Zhu. Disentangled contrastive learning on graphs. Advances in Neural Information Processing Systems , 34:21872-21884, 2021.

Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. Advances in neural information processing systems , 31, 2018.

Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnexplainer: Generating explanations for graph neural networks. Advances in neural information processing systems , 32, 2019.

Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. Fair darts: Eliminating unfair advantages in differentiable architecture search. In European conference on computer vision , pages 465-480. Springer, 2020.

Xiangxiang Chu, Bo Zhang, and Ruijun Xu. Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12239-12248, 2021.

Stephen Cha, Taehyeon Kim, Hayeon Lee, and Se-Young Yun. Supernet in neural architecture search: A taxonomic survey. arXiv preprint arXiv:2204.03916 , 2022.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations , 2018.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 , 2018.

Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research , 20(1):1997-2017, 2019.

Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International conference on machine learning , pages 4095-4104. PMLR, 2018.

Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 , 2018.

Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In European conference on computer vision , pages 544-560. Springer, 2020.

Lirong Wu, Haitao Lin, Cheng Tan, Zhangyang Gao, and Stan Z Li. Self-supervised learning on graphs: Contrastive, generative, or predictive. IEEE Transactions on Knowledge and Data Engineering , 2021.

Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, and Mi Zhang. Does unsupervised architecture representation learning help neural architecture search? Advances in Neural Information Processing Systems , 33:1248612498, 2020.

Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture optimization. Advances in neural information processing systems , 31, 2018.

Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas-bench101: Towards reproducible neural architecture search. In International Conference on Machine Learning , pages 7105-7114. PMLR, 2019.

Arber Zela, Julien Niklas Siems, Lucas Zimmer, Jovita Lukasik, Margret Keuper, and Frank Hutter. Surrogate nas benchmarks: Going beyond the limited search spaces of tabular nas benchmarks. In Tenth International Conference on Learning Representations , pages 1-36. OpenReview. net, 2022.

Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. Advances in Neural Information Processing Systems , 33:17009-17021, 2020.

Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering , 2021.

Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. arXiv preprint arXiv:2203.13457 , 2022.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference on artificial intelligence , pages 4602-4609, 2019.

Yijian Qin, Xin Wang, Zeyang Zhang, and Wenwu Zhu. Graph differentiable architecture search with structure learning. Advances in Neural Information Processing Systems , 34, 2021.

Paul D Dobson and Andrew J Doig. Distinguishing enzyme structures from non-enzymes without alignments. Journal of molecular biology , 330(4):771-783, 2003.

Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research , 12(9), 2011.

Asim Kumar Debnath, Rosa L Lopez de Compadre, Gargi Debnath, Alan J Shusterman, and Corwin Hansch. Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. correlation with molecular orbital energies and hydrophobicity. Journal of medicinal chemistry , 34(2):786797, 1991.

Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining , pages 1365-1374, 2015.

Christopher Morris, Nils M Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. arXiv preprint arXiv:2007.08663 , 2020.

Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Hsu, and Kuansan Wang. An overview of microsoft academic service (mas) and applications. In Proceedings of the 24th international conference on world wide web , pages 243-246, 2015.

Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based recommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval , pages 43-52, 2015.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems , 33:22118-22133, 2020.

Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys (CSUR) , 54(4):1-34, 2021.

Krishna Teja Chitty-Venkata, Murali Emani, Venkatram Vishwanath, and Arun K Somani. Neural architecture search for transformers: A survey. IEEE Access , 10:108374-108412, 2022.

Chaoyu Guan, Ziwei Zhang, Haoyang Li, Heng Chang, Zeyang Zhang, Yijian Qin, Jiyan Jiang, Xin Wang, and Wenwu Zhu. Autogl: A library for automated graph learning. In ICLR 2021 Workshop GTRL , 2021.

Yijian Qin, Ziwei Zhang, Xin Wang, Zeyang Zhang, and Wenwu Zhu. Nas-bench-graph: Benchmarking graph neural architecture search. arXiv preprint arXiv:2206.09166 , 2022.

Ziwei Zhang, Peng Cui, and Wenwu Zhu. Deep learning on graphs: A survey. IEEE Transactions on Knowledge and Data Engineering , 34(1):249-270, 2020.

Ziwei Zhang, Haoyang Li, Zeyang Zhang, Yijian Qin, Xin Wang, and Wenwu Zhu. Large graph models: A perspective. arXiv preprint arXiv:2308.14522 , 2023.

Zeyang Zhang, Ziwei Zhang, Xin Wang, and Wenwu Zhu. Learning to solve travelling salesman problem with hardness-adaptive curriculum. In Thirty-Fifth AAAI Conference on Artificial Intelligence , 2022.

Ziwei Zhang, Xin Wang, Zeyang Zhang, Peng Cui, and Wenwu Zhu. Revisiting transformation invariant geometric deep learning: Are initial representations all you need? arXiv preprint arXiv:2112.12345 , 2021.

Zeyang Zhang, Xin Wang, Ziwei Zhang, Haoyang Li, Yijian Qin, Simin Wu, and Wenwu Zhu. Llm4dyg: Can large language models solve problems on dynamic graphs? arXiv preprint , 2023.

Kaixiong Zhou, Qingquan Song, Xiao Huang, and Xia Hu. Auto-gnn: Neural architecture search of graph neural networks. arXiv:1909.03184 , 2019.

Matheus Nunes and Gisele L Pappa. Neural architecture search in graph neural networks. In Brazilian Conference on Intelligent Systems , pages 302-317, 2020.

Min Shi, David A Wilson, Xingquan Zhu, Yu Huang, Yuan Zhuang, Jianxun Liu, and Yufei Tang. Genetic-gnn: Evolutionary architecture search for graph neural networks. Knowledge-Based Systems , page 108752, 2022.

Wentao Zhang, Zheyu Lin, Yu Shen, Yang Li, Zhi Yang, and Bin Cui. Deep and flexible graph neural architecture search. In International Conference on Machine Learning , pages 26362-26374. PMLR, 2022.

Yuhui Ding, Quanming Yao, Huan Zhao, and Tong Zhang. Diffmg: Differentiable meta graph search for heterogeneous graph neural networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 279-288, 2021.

ZHAO Huan, YAO Quanming, and TU Weiwei. Search to aggregate neighborhood for graph neural network. In 2021 IEEE 37th International Conference on Data Engineering , pages 552-563, 2021.

Yanxi Li, Zean Wen, Yunhe Wang, and Chang Xu. One-shot graph neural architecture search with dynamic search space. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 8510-8517, 2021.

Shaofei Cai, Liang Li, Jincan Deng, Beichen Zhang, Zheng-Jun Zha, Li Su, and Qingming Huang. Rethinking graph neural architecture search from message-passing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6657-6666, 2021.

Yijian Qin, Xin Wang, Ziwei Zhang, Pengtao Xie, and Wenwu Zhu. Graph neural architecture search under distribution shifts. In International Conference on Machine Learning , pages 18083-18095. PMLR, 2022.

Zeyang Zhang, Ziwei Zhang, Xin Wang, Yijian Qin, Zhou Qin, and Wenwu Zhu. Dynamic heterogeneous graph attention neural architecture search. In Thirty-Seventh AAAI Conference on Artificial Intelligence , 2023.

Yijian Qin, Xin Wang, Ziwei Zhang, Hong Chen, and Wenwu Zhu. Multi-task graph neural architecture search with task-aware collaboration and curriculum. Advances in neural information processing systems , 2023.

Sapir Kaplan and Raja Giryes. Self-supervised neural architecture search. arXiv preprint arXiv:2007.01500 , 2020.

Aleksandr Timofeev, Grigorios G Chrysos, and Volkan Cevher. Self-supervised neural architecture search for imbalanced datasets. arXiv preprint arXiv:2109.08580 , 2021.

Nam Nguyen and J Morris Chang. Csnas: Contrastive self-supervised learning neural architecture search via sequential model-based optimization. IEEE Transactions on Artificial Intelligence , 3(4):609-624, 2021.

Changlin Li, Tao Tang, Guangrun Wang, Jiefeng Peng, Bing Wang, Xiaodan Liang, and Xiaojun Chang. Bossnas: Exploring hybrid cnn-transformers with block-wisely self-supervised neural architecture search. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12281-12291, 2021.

Zhuowei Li, Yibo Gao, Zhenzhou Zha, Zhiqiang Hu, Qing Xia, Shaoting Zhang, and Dimitris N Metaxas. Towards self-supervised and weight-preserving neural architecture search. arXiv preprint arXiv:2206.04125 , 2022.

Zhenyu Hou, Yukuo Cen, Yuxiao Dong, Jie Zhang, and Jie Tang. Automated unsupervised graph representation learning. IEEE Transactions on Knowledge and Data Engineering , 2021.

Xuanyang Zhang, Pengfei Hou, Xiangyu Zhang, and Jian Sun. Neural architecture search with random labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10907-10916, 2021.

Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley. Neural architecture search without training. In International Conference on Machine Learning , pages 7588-7598. PMLR, 2021.

Chenxi Liu, Piotr Dollár, Kaiming He, Ross Girshick, Alan Yuille, and Saining Xie. Are labels necessary for neural architecture search? In European Conference on Computer Vision , pages 798-813. Springer, 2020.

Yixin Liu, Ming Jin, Shirui Pan, Chuan Zhou, Yu Zheng, Feng Xia, and Philip Yu. Graph self-supervised learning: A survey. IEEE Transactions on Knowledge and Data Engineering , 2022.

Yaochen Xie, Zhao Xu, and Shuiwang Ji. Self-supervised representation learning via latent graph prediction. arXiv preprint arXiv:2202.08333 , 2022.

Petar Velickovic, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. ICLR (Poster) , 2(3):4, 2019.

Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semisupervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv:1908.01000 , 2019.

Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng, Yu Rong, Tingyang Xu, and Junzhou Huang. Graph representation learning via graphical mutual information maximization. In Proceedings of The Web Conference 2020 , pages 259-270, 2020.

Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph-level representation learning with local and global structure. In International Conference on Machine Learning , pages 11548-11558. PMLR, 2021.

Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gcc: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 1150-1160, 2020.

Yuehui Han, Le Hui, Haobo Jiang, Jianjun Qian, and Jin Xie. Generative subgraph contrast for selfsupervised graph representation learning. arXiv preprint arXiv:2207.11996 , 2022.

Sihang Li, Xiang Wang, An Zhang, Yingxin Wu, Xiangnan He, and Tat-Seng Chua. Let invariant rationale discovery inspire graph contrastive learning. In International Conference on Machine Learning , pages 13052-13065. PMLR, 2022.

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.04131 , 2020.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems , 33:58125823, 2020.

Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International Conference on Machine Learning , pages 4116-4126. PMLR, 2020.

Zhenyu Hou, Xiao Liu, Yuxiao Dong, Chunjie Wang, Jie Tang, et al. Graphmae: Self-supervised masked graph autoencoders. arXiv preprint arXiv:2205.10803 , 2022.

Jintang Li, Ruofan Wu, Wangbin Sun, Liang Chen, Sheng Tian, Liang Zhu, Changhua Meng, Zibin Zheng, and Weiqiang Wang. Maskgae: Masked graph modeling meets graph autoencoders. arXiv preprint arXiv:2205.10053 , 2022.

Zhenyu Hou, Yufei He, Yukuo Cen, Xiao Liu, Yuxiao Dong, Evgeny Kharlamov, and Jie Tang. Graphmae2: A decoding-enhanced masked self-supervised graph learner. In Proceedings of the ACM Web Conference 2023 , pages 737-746, 2023.

Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. When does self-supervision help graph convolutional networks? In international conference on machine learning , pages 10871-10880. PMLR, 2020.

Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Selfsupervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems , 33:12559-12571, 2020.

Wei Jin, Tyler Derr, Haochen Liu, Yiqi Wang, Suhang Wang, Zitao Liu, and Jiliang Tang. Self-supervised learning on graphs: Deep insights and new direction. arXiv preprint arXiv:2006.10141 , 2020.

Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In International Conference on Machine Learning , pages 12121-12132. PMLR, 2021.

Yihang Yin, Qingzhong Wang, Siyu Huang, Haoyi Xiong, and Xiang Zhang. Autogcl: Automated graph contrastive learning via learnable view generators. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 8892-8900, 2022.

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In Proceedings of the Web Conference 2021 , pages 2069-2080, 2021.

Wei Jin, Xiaorui Liu, Xiangyu Zhao, Yao Ma, Neil Shah, and Jiliang Tang. Automated self-supervised learning for graphs. arXiv preprint arXiv:2106.05470 , 2021.

Mingxuan Ju, Tong Zhao, Qianlong Wen, Wenhao Yu, Neil Shah, Yanfang Ye, and Chuxu Zhang. Multi-task self-supervised graph neural networks enable stronger task generalization. arXiv preprint arXiv:2210.02016 , 2022.

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798-1828, 2013.

Xin Wang, Hong Chen, Si'ao Tang, Zihao Wu, and Wenwu Zhu. Disentangled representation learning, 2023.

Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. Advances in neural information processing systems , 31, 2018.

Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool, Bernt Schiele, and Mario Fritz. Disentangled person image generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 99-108, 2018.

Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems , 29, 2016.

Emily L Denton et al. Unsupervised learning of disentangled representations from video. Advances in neural information processing systems , 30, 2017.

Luan Tran, Xi Yin, and Xiaoming Liu. Disentangled representation learning gan for pose-invariant face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1415-1424, 2017.

Xin Wang, Zihao Wu, Hong Chen, Xiaohan Lan, and Wenwu Zhu. Mixup-augmented temporally debiased video grounding with content-location disentanglement. 2023.

Xin Wang, Hong Chen, Yuwei Zhou, Jianxin Ma, and Wenwu Zhu. Disentangled representation learning for recommendation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Hong Chen, Yudong Chen, Xin Wang, Ruobing Xie, Rui Wang, Feng Xia, and Wenwu Zhu. Curriculum disentangled recommendation with noisy multi-feedback. Advances in Neural Information Processing Systems , 34:26924-26936, 2021.

Xin Wang, Hong Chen, and Wenwu Zhu. Multimodal disentangled representation for recommendation. In 2021 IEEE International Conference on Multimedia and Expo (ICME) , pages 1-6, 2021.

Jianxin Ma, Chang Zhou, Hongxia Yang, Peng Cui, Xin Wang, and Wenwu Zhu. Disentangled selfsupervision in sequential recommenders. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 483-491, 2020.

Jianxin Ma, Chang Zhou, Peng Cui, Hongxia Yang, and Wenwu Zhu. Learning disentangled representations for recommendation. Advances in neural information processing systems , 32, 2019.

Haoyang Li, Xin Wang, Ziwei Zhang, Jianxin Ma, Peng Cui, and Wenwu Zhu. Intention-aware sequential recommendation with structured intent transition. IEEE Transactions on Knowledge and Data Engineering , 2021.

Xin Wang, Zirui Pan, Yuwei Zhou, Hong Chen, Chendi Ge, and Wenwu Zhu. Curriculum co-disentangled representation learning across multiple environments for social recommendation. 2023.

Yipeng Zhang, Xin Wang, Hong Chen, and Wenwu Zhu. Adaptive disentangled transformer for sequential recommendation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 3434-3445, 2023.

Yanbei Liu, Xiao Wang, Shu Wu, and Zhitao Xiao. Independence promoted graph disentangled networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 4916-4923, 2020.

Zeyang Zhang, Xingwang Li, Fei Teng, Ning Lin, Xueling Zhu, Xin Wang, and Wenwu Zhu. Outof-distribution generalized dynamic graph neural network for human albumin prediction. In IEEE International Conference on Medical Artificial Intelligence , 2023.

Zeyang Zhang, Xin Wang, Ziwei Zhang, Zhou Qin, Weigao Wen, Hui Xue, Haoyang Li, and Wenwu Zhu. Spectral invariant learning for dynamic graphs under distribution shifts. In Advances in Neural Information Processing Systems , 2023.

Zeyang Zhang, Xin Wang, Ziwei Zhang, Haoyang Li, Zhou Qin, and Wenwu Zhu. Dynamic graph neural networks under spatio-temporal distribution shift. In Advances in Neural Information Processing Systems , 2022.

Peer reviews:
comments: SUMMARY:
This paper proposed a method DSGAS to automcatiicaly design the architectures in an unsupervised manner. It discovers the optimal  architectures by learning the latent graph factor.

STRENGTHS:
It is interesting and novel to automate the design of GNNs in an unsupervised manner, which holds significant potential.

WEAKNESSES:
1. The necessity of designing disentanglement is not clear. It is true that GNNs may preferred different graph factors. What is the difference between the proposed method and the baseline that: using one latent factor design one architectures, and then run multiple times and ensembles the search architectures? 
The motivation for disentangled architectures is not well justified. While the evaluations on K are provided, it would be more effective to achieve K=2 by ensemble two K=1 results.


2. The experiments are insufficient to justify the effectiveness of the proposed design method. Since it designs GNNs in an unsupervised manner, it would be better to conduct fair comparisons with existing unsupervised methods rather than methods designed under a supervised manner.

3. Apart from the comparisons of performance, the comparisons of search cost, the parameters of the searched architectures should be provided as well.

QUESTIONS:
Please check the weaknesses.

LIMITATIONS:
Please check the weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of unsupervised graph neural architecture search, which has received limited attention in existing literature. The authors propose a novel approach called Disentangled Self-supervised Graph Neural Architecture Search (DSGAS) to discover optimal architectures capturing latent graph factors in an unsupervised manner. The DSGAS model consists of a disentangled graph super-network that incorporates multiple architectures with factor-wise disentanglement, self-supervised training to estimate architecture performance under different factors, and a contrastive search method to discover architectures with factor-specific expertise. Extensive experiments on 11 real-world datasets demonstrate that the proposed model achieves state-of-the-art performance compared to several baseline methods.\n\nSTRENGTHS:\n- The paper addresses an important and underexplored problem in the field of graph neural architecture search, i.e., scenarios where supervised labels are not available.\n\n- The proposed DSGAS model introduces a novel approach to discovering optimal architectures by leveraging latent graph factors in a self-supervised fashion based on unlabeled graph data.\n\n- The disentangled graph super-network and the self-supervised training with joint architecture-graph disentanglement are novel contributions that enhance the understanding and performance of the proposed model.\n\n- The extensive experimental evaluation on 11 real-world datasets demonstrates the superiority of the DSGAS model over several baseline methods, showing its effectiveness in unsupervised graph neural architecture search.\n\nWEAKNESSES:\n- While the paper presents a novel approach, further details regarding the implementation of the proposed DSGAS model in the main paper would be beneficial for the readers to understand the design clearly. For example, it is better to illustrate the details of the supernet construction in the main paper.\n\n- Some typos should be corrected, e.g., heterogenous -> heterogeneous on line 323.\n\nQUESTIONS:\nPlease see Weaknesses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper gives a pioneer solution for graph neural architecture search with limited labels. The key idea is to train a super-network containing disentangled factor-wise architectures by a self-supervised learning specially designed for graph neural architecture search. In this way, the paper addresses the key problem of finding the optimal architectures capturing various graph factors in an unsupervised fashion. The extensive experiments show that the method achieves signifiant improvements over the state-of-the-art GraphNAS methods under unsupervised settings. Detailed ablation studies also show the effectiveness of each proposed components.\n\nSTRENGTHS:\nUnsupervised GraphNAS is an important and valuable problem which remains unexplored in the literature. This paper is a timely work to introduce GraphNAS in the scenarios where labels are scarce, and these scenarios are actually quite common in practice. The method design is novel, where the contrastive search module is especially interesting that it pushes together architectures with similar factor expertises and pull away unsimilar architectures by architecture augmentations to explore the search space. The extensive experiments verify its capability of automating the architecture design for the data from various fields.\n\nWEAKNESSES:\n1. Although it may fall outside the scope of this paper, i'm curious about whether the model can be applied into areas like AI4Science where labels are quite scarce?\n\n2. Pretraining techniques have shown promising applications in many scenarios like NLP and CV, and they usually have a close relationship with unsupervised or self-supervised techniques. It would better to discuss the relationship between this method and GraphNAS pretraining.\n\n3. The framework diagram (figure 1) is a bit small, especially that the font size which is too small to read clearly. It would be better to enlarge the diagram and increase the font size a bit.\n\nQUESTIONS:\n1. Although it may fall outside the scope of this paper, i'm curious about whether the model can be applied into areas like AI4Science where labels are quite scarce?\n\n2. Pretraining techniques have shown promising applications in many scenarios like NLP and CV, and they usually have a close relationship with unsupervised or self-supervised techniques. It would better to discuss the relationship between this method and GraphNAS pretraining.\n\nLIMITATIONS:\nI don't see any concerns regarding the societal impact in this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper mainly focuses on the problem of graph neural architecture search without labels. The authors find that the key problem is to discover the latent graph factors that drive the formation of graph data as well as the underlying relations between the factors and the optimal neural architectures. To this end, the authors propose the disentangled super-network, and design a self-supervised training and searching paradigm to automate the architecture design. The experiment results are significant and the deeper analyses are convincing.

STRENGTHS:
- This paper mainly focuses on the problem of graph neural architecture search without labels, which is important yet underexplored in the literature.

- The proposed methods sound reasonable. 

- The experiment results are significant and the deeper analyses are convincing.

WEAKNESSES:
- Can the authors explain the details of the intuition for the proposed contrastive search?

- The experiments include both node classification and graph classification tasks. Do these tasks share the same search space?
 
- As the existing graph NAS methods are in need of labels for training and searching, how do the authors modify them to compare as baselines?

- What about the complexity of the method?

QUESTIONS:
Please see the weaknesses.

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08

comments: SUMMARY:
In this paper, the authors study the problem of unsupervised graph neural architecture search, which remains unexplored in the literature. The authors propose a novel Disentangled Self-supervised Graph Neural Architecture Search (DSGAS) model, which is able to discover the optimal architectures capturing various latent graph factors in a self-supervised fashion based on unlabeled graph data. Extensive experiments on 11 real-world datasets demonstrate that the proposed DSGAS model is able to achieve state-of-the-art performance against several baseline methods in an unsupervised manner.

STRENGTHS:
1. I believe the problem of unsupervised graph neural architecture search is important, and this work is significant that it may unlock more NAS applications in practice. 

2. The writing is clear and well-organized in general. 

3. The experments and analyses are extensive.

WEAKNESSES:
Please see my comments in the questions part.

QUESTIONS:
1. What is the relationship between supervised and unsupervised nas paradigms in terms of the optimization problem, e.g., line 84 in the main paper?

2. In Figure 1, the super-network seems a simple DAG, why do the searched architectures in Figure 4 seem kind of complex in terms of the operation connections?

3. In Figure 1, it seems that the mixed operation has different colors for different $\alpha_i$, is it possible that some factor-wise architectures share the same operation?

4. In Figure 1, is it required that all the architectures should choose the same augmentations? In other words, is it possible that some architectures adopt operation choice perturbation, while others adopt weight perturbation in the same batch for contrastive learning?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08