Unsupervised Graph Neural Architecture Search with Disentangled Self-Supervision
Authors: Zeyang Zhang
Xin Wang
Ziwei Zhang
Guangyao Shen
Shiqi Shen
Wenwu Zhu
Venue: NeurIPS 2023 poster (2023)
Keywords: Graph Neural Architecture Search
Unsupervised Learning
Self-supervised Learning

Abstract: The existing graph neural architecture search (GNAS) methods heavily rely on supervised labels during the search process, failing to handle ubiquitous scenarios where supervisions are not available. In this paper, we study the problem of unsupervised graph neural architecture search, which remains unexplored in the literature. The key problem is to discover the latent graph factors that drive the formation of graph data as well as the underlying relations between the factors and the optimal neural architectures. Handling this problem is challenging given that the latent graph factors together with architectures are highly entangled due to the nature of the graph and the complexity of the neural architecture search process. To address the challenge, we propose a novel Disentangled Self-supervised Graph Neural Architecture Search (DSGAS) model, which is able to discover the optimal architectures capturing various latent graph factors in a self-supervised fashion based on unlabeled graph data. Specifically, we first design a disentangled graph super-network capable of incorporating multiple architectures with factor-wise disentanglement, which are optimized simultaneously. Then, we estimate the performance of architectures under different factors by our proposed self-supervised training with joint architecture-graph disentanglement. Finally, we propose a contrastive search with architecture augmentations to discover architectures with factor-specific expertise. Extensive experiments on 11 real-world datasets demonstrate that the proposed model is able to achieve state-of-the-art performance against several baseline methods in an unsupervised manner.

Unsupervised Graph Neural Architecture Search with Disentangled Self-supervision
Zeyang Zhang'; Xin Wang':
Ziwei Zhang', Guangyao Shen', Shiqi Shen', Wenwu Zhu't
Department of Computer Science and Technology. BNRist, Tsinghua University, Wechat, Tencent gy-zhang20tmails.. tsinghua edu. on, (xin_vang, zwzhang)@tsinghua. edu. flucasgyshen, shiqishen) com, wwzhu@tsinghua.c edu.cn
Abstract
The existing graph neural architecture search (GNAS) methods heavily rely on supervised labels during the search process, failing handle ubiquitous scenar- ios where supervisions are not available. In this paper, we study the problem of unsupervised graph neural architecture search, which remains unexplored in the literature. The key problem is to discover the latent graph factors that drive the formation of graph data as well as the underlying relations between the factors and the optimal neural architectures. Handling this problem is challenging given that the latent graph factors together with architectures are highly entangled due the nature of the graph and the complexity of the neural architecture search pro- cess. To address the challenge, we propose a novel Disentangled Self-supervised Graph Neural Architecture Search (DSGAS) model, which is able to discover the optimal architectures capturing various latent graph factors in a self-supervised fashion based unlabeled graph data. Specifically, first design disentan- gled graph super-network capable of incorporating multiple architectures with factor-wise disentanglement, which optimized simultaneously. Then, timate the performance of architectures under different factors by our proposed self-supervised training with joint architecture-grant disentanglement. Finally, we propose a contrastive search with architecture augmentations to discover archi- tectures with factor-specific expertise. Extensive experiments datasets demonstrate that the proposed DSGAS model is able to achieve state-of- the-art performance against several baseline methods in unsupervised manner.
11 real-world
Introduction
Graph neural architecture search (GNAS), aiming to automatically discover the optimal architecture for graph neural network (GNN) based on graph-structured data and task, has shown remarkable progress in enhancing the predictive power and saving human endeavors for various graph appli- cations (.. The existing GNAS methods generally follow a supervised paradigm such that they optimize the weights within architectures given training dataset with supervised loss (e.g., the cross entropy loss of label predictions) and estimate the architecture performance based on the validation dataset with supervision signals. For example, the label prediction accuracy is adopted for architecture ranking during the neural architecture search process 3 4]. As a result, supervised labels become indispensable for applying the existing GNAS methods.
However, ground-trun labels reality may extremely scarce or hardly available in many graph applications. For example, variety of biological problems require significant amount of human labors and time costs in clinical tests to obtain labels for supervision [5] 6 7]. As the existing
This work done during the author's internship at Wechat, Tencent i Corresponding authors
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

GNAS approaches heavily rely on supervised labels for weight training and architecture evaluation, they will suffer from performance deterioration in unsupervised settings, failing to discover optimal architectures in the scenarios where labels are scarce or not available.
In this paper, we study unsupervised graph neural architecture search, i.e., discovering optimal GNN architectures without labels for graph-structured data, which remains unexplored in the literature. The key problem lies in two important aspects: i) discover the latent graph factors that drive the formation process of graph data 8 9 TO 12 I3 14: ii) capture the underlying relations between the factors and the optimal neural architectures. For instance, molecular graph may consist of groups of atoms as well as bonds representing different functional units [5] requiring different optimal neural architectures make accurate predictions.
Nevertheless, solving the problem is highly non-trivial and challenging given that the hidden factors entangled in the graph and very difficult to capture, e.g., social network may contain several communities originating from various interests (e.g., sports, games, etc.) 10), with the nodes and edges belonging to different communities mixing together. Moreover, the architectures with different functional factors are also entangled within the weight-sharing super-network (17] [8] [9], resulting in inaccurate architecture performance estimations under different hidden factors.
To tackle the challenge, we propose a novel unsupervised graph neural architecture search method, i.e., Disentangled delf-superviss Graph Neural Architecture Search (DSGAS Given graph data without supervised labels, our proposed DSGAS model can discover the optimal architectures capturing multiple latent factors in self-supervised fashion. In particular, first design a disentangled graph super-network, where multiple architectures are disentangled for simultaneous optimization w.r.t various latent factors. Then, we propose a self-supervised training with joint architecture- graph disentanglement, which disentangles architectures and graphs within a common latent space. The super-network is trained through routing mechanism between architectures, graphs and self- supervised tasks, to obtain an accurate estimation of the architecture performance under each latent factor. Finally, we propose contrastive search with architecture augmentations, where novel architecture- instance discrimination task is introduced to discover architectures with distinct capabilities of capturing various factors in a self supervisdd fashion. Extensive experiments show that the proposed DSGAS model is able to significantly outperform the statc-of-ther of-tth-art art GNAS baselines under both unsupervised and semi-supervised settings. Detailed ablation studies and analyses also demonstrate that DSGAS is able to discover effective architectures with our proposed disentangled self-supervision designs. The contributions of this paper are summarized as follows:
We are the first to study the problem of unsupervised graph neural architecture search and propose the Disentangled Self-supervised Graph Neural Architecture Search (DSGAS) model capable of discovering the optimal architectures without labels, to the best of our knowledge.
We introduce three novel modules, i) disentangled graph architecture supervised training with joint architecture graph disentanglement and iii) contrastive search with architecture augmentations, which can discover the optimal architectures capturing various graph latent factors with disentangled self-suupervision
ii) self-
Extensive experiments on 11 real-world graph datasets show that our proposed method DSGAS is able to discover effective graph neural architectures without supervised labels and significantly outperform the oftteofftthe-ar baselines in both unsupervised and cup-svueee settings.
2 Preliminaries and Problem Formulation
Graph Neural Architecture Search Denote the graph space as G and the label space as ), A graph neural network can be denoted as a function fa,w : G -> V. which characterized by architecture parameters E A and learnable weights W given an architecture space A and weight space W. Graph neural architecture search (GNAS) aims at automating the design of graph neural architectures, i.c., obtaining the best-performed architectures by searching As is usually instantiated as selecting GNN operations, e.g., GCN 20) GAT 21], GIN 22
we also call a as
TThe codes are available Github

---

Disentangled Self- Supervised Graph Neural Architecture Search
Graph Data without Labels
Disentangled Graph Architecture Sxper-Network
Self supervised Training with joint Architecture Graph Disentanglement
Contrastive Search with Architecture Augmentations
0
Operation
fe
Martintattione T,
GAT
M iic ciniin
1
GIN
62
Latent Factors
Park
2
Sapa
Disentangled
Architectures
under inder
11
a2
0K
Seath
The framework of Disentangled Self-super Graph Neural Architecture Search (DSGAS), including the following three key components: 1) Disentangled graph architecture super- network enables multiple architectures to be disentangled and optimized simultaneously in an end-to- end manner. 2) Self-supervised training with joint architecture-grat -rrapp disentanglement estimates the performance of architectures under various latent factors by considering the relationship between architectures, graphs and factors. 3) Contrastive search with architecture augmentations encourages and discovers architectures with distinct capabilities of capturing factors. (Best viewed in color)
Figure 1:
operation choices for brevity. Generally, GNAS solves the bi-level optimization problem 2
o- = arg min Leal w (ax),,
a'A
s.t. w (a) arg min Cmain((x w),
(2)
wEW(a)
where Guain and Liva denotes the loss of the predictions of the architecture fa, (.) against supervised labels on training and validation datasets. The optimization problem can be viewed as having two objectives that (2) aims to obtain accurate architecture performance estimation, and Eq. aims to search the best-performed architectures. To avoid the cost of training from scratch for each architecture, the super-network [24 25 arises as a commonly adopted technique to obtain faster architecture performance estimation, where the architecture candidates are viewed as sub-networks of the super-network and their weights are shared during the training process.
Unsupervised Graph Neural Architecture Search We consider the problem of unsupervised graph neural architecture where labels, which are adopted for the performance estimation and the search process in the supervised GNAS, are not accessible. The problem of unsupervised GNAS can be formulated optimizing an architecture generator that is able to discover powerful architectures by exploiting inherent graph properties without labels, i.e., G + fa." instead of (G.V) + fo.. as done by supervised GNAS methods. Then, the discovered architectures sa, can be utilized in downstream tasks, e.g., finetuning the weights w or the operation choices extra shallow classifiers for further prediction.
Disentangled Self-supervised Graph Neural Architecture Search
In this section, introduce Disentangled Self-supervised Graph Neural Architecture Search (DSGAS) to search architectures without labels, by proposing three key components: disentangled graph architecture super-network self-supervised training with joint architecture-grat disentangle- ment, and contrastive search with architecture augmentations.
3.1 Disentangled Graph Architecture Super-Network
To discover architectures that potentially have optimal performance, we resort to guiding the search towards architectures' capabilities capturing the inherent graph factors, which are shown important in the graph formation g 9.. As architectures may expert in different graph factors, we propose

---

disentangled graph architecture super- upeernetworr to incorporate K different architectures to be estimated and searched w.r.t factors simultaneously, where the hyperparameter K denotes the number of factors.
Disentangled Super-Network Layer For each super-network. layer, we adopt K mixed operations parameterized by different a to learn K-chunk graph representations:
H₂ GNN AH,
where A is the adjacency matrix of the graph, H denotes the input graph representations, and GNN (.) denotes the mixed GNN operations parameterized by ag. For the convenience of differen- tiable optimization, we adopt continuous parameterization and weight-sharing mechanism 25 to
implement the mixed operations:
10
GNN... a2 (H, = a GNN,G,,,,HH)
(4)
exp(Ong.)) denotes the probability of where |O] is the number of GNN operation choices, 003y1 Σ, exp((....)) the i-th operation for the k-th architecture Og; and 0 is learnable parameters.
Overall Disentangled Super-Network The overall super-network is constructed in the form of directed acyclic graph (DAG) with an ordered sequence of disentangled super-network layers. More details about the DAG construction and the according GNN operations for each disentangled super-network layer are included in Appendíx. The output of the last layer Z [H, Hz,..., H describes the various aspects of the graphs and serves as the final graph representations, which can be utilized or finetuned in downstream tasks. In this way, the architectures' operation choices = [a1, .2...... [x] and weights w are incorporated in one super-network. For brevity, we use fa (-) denote the k-th architecture induced from the super-network. Note that the design of K operation choices alleviates the entanglement of architectures by providing more flexible choices of paths (26) in the super-network. For instance, the 'mean' operation captures structural properties while the 'max' operation captures representative elements [22]. and in this case, our design can capture both of them by choosing corresponding operations to learn respective representations instead of choosing only one of them which may conflict each other.
3.2 Self-supervised Training with Joint Grchitecture-Graph Disentanglement
Inspired by graph self-supervised learning, we utilize graph pretext tasks to measure the architectures' capabilities of capturing latent factors. Predictive pretext tasks [27). for example, design pseudo label generator s(-), and optimize the prediction probability 4 p(s(G.)]GG. However, the tasks usually take holistic views of the graphs and neglect the entanglement of the latent factors, which may lead to suboptimal performance estimation. Therefore, disentangle the factors, we transform the probability into the expectation of multiple subtasks under different latent factors by the Bayesian formula:
p(>(G))) = ((
where p(k(G.) denotes the probability of latent factor A: given the i-th graph instance G,, and p(((G,) G., k) denotes the pretext task under k-th latent factor. An intuitive explanation of Eq, (5) is that it first infers the latent factors and then conducts factor-specific self-supervised training to capture the latent factors, which we describe in detail as follows.
Architecture-aware Latent Factor Inference Directly modeling p(k G.) difficult as we do not know prior what GNN encoders suitable for inferring the latent factors. Intuitively, one solution to get the architectures being searched involved in the inference stage. By the Bayesian formula, we factorize the probability w.r.t architecture choices:
p(k(G,) Eppa, (..kk))))))(())))))))))
WWe take graph classification an example for simplicity, while the case of node classification can be easily extended.

---

where p(ayl);) is prior distribution, and we adopt uniform distribution for simplicity. Then we can model the probability distributions of latent factors given the graph G. by utilizing the architectures being searched:
exp (Z. [En((a)). c.)
p(k|gi, Em= exp ((z,,)) [En((a)), cm.)
where C₁ is a learnable vector to represent the k-th latent factor, and Z5,A = far, denotes the graph representations output by the k-th architecture for the graph 9s. Enc(-) denotes architecture encoding techniques to obtain embeddings of operation choices ar so that the structural properties and correlations of the neural architectures can be considered [28 29].
Factor-aware Graph Self-Supervised Learning Under the k-th latent factor, we leverage the corresponding architectures (.). for conducting the factor-specific pretext tasks as to estimate the capturing capabilities of architectures under various factors. The overall objective is maximize E5), and the loss can be calculated by
k)
N Σ
1 Togt
log Ep(rI.)) (p(s(G)))ss)k)
N Epcigs (-
(8)
where N is the number samples and the upper bound is obtained by Jensen's Inequality. Then we can generalize our method to other graph self- tasks with specially-designed task loss functions by defining log p( (G;) (..). k) as the task loss function [(far, ws 9i) for the graph G, under the k-th factor, and calculate the loss by
L N Σ PeCHIO.) (I(Sar.u wi))
In this way, the disentangled architectures in Sec. 3.1 coupled with factors disentangled from graph data can be routed pairwisely, and trained with factor-specific self-supervision to obtain more accurate performance estimation under each factor. Similar to [25] the super-network weights are updated with w w Aw wu to obtain the weights that can represent the architectures' capabilities.
3.3 Contrastive Search with Architecture Augmentations
In this section, we focus on encouraging the disentanglement of architectures and searching archi- tectures with distinet capabilities of capturing different factors. The main insight of our proposed search method is intuitively based on the following two observations shown in the literature: 1) As architectures similar in operation choices and topologies have similar capabilities of capturing semantics for downstream tasks [28] 30, 31). slight modifying the architecture will have a slight influence on its capability.
Since different GNN architectures expert in different downstream tasks (32, the architectures searched for different disentangled latent factors are expected to have dissimilar capabilities under different factors.
Contrastive Search Inspired by self-supervised contrastive learning 33 34
that capture dis-
criminative features by pulling similar instances together and pushing dissimilar instances away in the latent space, we propose an architecture-level instance discrimination task to encourage the architectures to capture various latent factors. The task is defined
p(s(aw)19,,a)) = exp 0 (Zi,k, 2.,)())))
(10)
Σn exp ((Zi.j j. 2 a(a))
Zi, = for (G.). (G.). = T, ( (fa
(1))
where s(ax) is assigned to k as surrogate labels for the architecture, o(-) calculates the similarity of embeddings and (-) denotes architecture augmentations that transform an architecture to
another architecture with similar capabilities capturing factors, i.e., far.w + San.w- Then the loss
function can be calculated by
La Σ log Eptan ((((a))(()))))))))))))))))))))() G. at).
(12)
Similar to 25), the architecture parameters are updated with a = a- do VaLa Co to search architectures with better capabilities of capturing factors.
5

---

Architecture Augmentations To create various views of architectures, we design three basic architecture augmentations from the perspectives of architecture operation choices a, architecture weights w and the internal embeddings H:
Operation Choice Perturbation. This augmentation randomly reshapes the distributions of the mixed operations by altering the temperature in the softmax function in Eq. (4) with
exp(0an. /T)
O.K E, exp(0ax.))
(13)
where the temperature 7 is sampled from uniform distribution U([L/r1,71)) and r1 2 is hyper-parameter controlling the perturbation degree.
Weight Perturbation. This augmentation randomly adds Gaussian noises ~ N(, o²) to % the architecture weights w, where r2 controls the perturbation ratio, and o² is the standard deviation of the weights.
Embedding Dropout. This augmentation randomly drops r3% of the embeddings H output from the mixed operations, where r3 controls the dropout ratio.
Note that these architecture augmentations can further be randomly composed to generate mixed augmentations.
4 Experiments
In this section, we conduct experiments on real-world datasets with unsupervised settings to verify the design of our method. We also include detailed ablation studies analyze the effectiveness of each component, and 3 real-world datasets in semi-supervia settings to show that our method can alleviate the label scarcity issues by pretraining the super-network.
Baselines
We compare method with 11 baselines from the following different categories.
Manually designed GNNs. We include five representative GNNs as our baselines, i.e., GCN 20
GAT 21,. GIN [22], GraphSage (35. GraphConv (6 and a simple baseline MLP, which constitutes our search space. For graph-level classification tasks, we adopt global mean pooling for each layer and concatenation to obtain the graph representations for these baselines.
Graph neural architecture search. We include representative GNAS baselines GraphNAS (3). PAS (4) and GASSO (37], where PAS is specially designed for graph-level classification tasks by searching the pooling operations, and GASSO is specially designed for node-level classification tasks by searching the graph structures simultaneously. We also include two classical NAS baselines, random search and DARTS (25). As these baselines are not specially designed for graphs, we adopt our search space for these baselines.
Table I: Summary of dataset statistics. Unsup./Semi. denotes Unsupervised and Semi-supervised settings, Graph/Node denotes graph and node classification tasks, ACC/AUC denotes Accuracy and ROC-AUC evaluation metrics.
MUTAG
PROTEINS
Cumputers
Plasics
OGEN-Arii Wechat-
Setting
Unsep.
Nudk
Ntal:
Evaluation Metrire
ACC
ACC
ACC
AUC
AIX
Grmphs
188
1,50
Nodex
17
13,252
7,600
18.33
$4.009
26
169 94%
60.774
Axg. Fdgess
56
211
183
1,714
505.424
245812
(80,22)
$90.417
28
2,484 (444941
3. 112, 155
+ Fentures
7
89
767
745
5,805
R,15
300
128
512
+ Clhsses
2
10
5
40
13
For unsupervised settings, we conduct experiments on four graph-level classification datasets including PROTEINS (38) DD 39), MUTAG [40], IMDB-B [41] from TUDataset [42 and four node-level classification datasets Coauthor CS, Coauthor Physics from the Microsoft Academic
Datasets
Graph 43 Amazon Computers, Amazon Photos from the Amazon Co-purchase Graph 44 For
semi-supervised settings, adopt three real-world datasets. OGBG-Molliv, OGBN-Arxi
45

---

Table 2: The results (aceuracy%) of all the methods on the world datasets in unsupervised settings. Numbers after the + signs represent standard deviations, The best results are in bold and the second-best results are underlined. As the search space of GASSO and PAS do not suit the graph-level and node-level tasks respectively, we omit their results.
Data
Graph Classification
Node Classification
Method
PROTEINS
DD
MUTAG
IMDB-B
CS
Computers
Physies
Photo
GCN
72.8=0.7
77.0=09
78.6=16
63.5=08
93.0+0.3
86.0+0.4
95.7=0.1
90.8
GAT
72.3
77.5
78.0=08
54.4=117
93,44:0.3
85.8
95.610.1
91.4106
GIN
72.6=0.4
77.3=07
86.3=17
70.7=005
93.1
76.7+005
95.3+0.1
91.1
GraphSage
72.9=0.7
77.1=0.1
78,3=16
53.0=21
93.2+00.3
78 78.4
95.4+0.11
89 99.2+0.7
GraphConv
72.
77.3=t06
87.2=14
71.1=06
93.1
74.74007
95.3:1011
91.54005
MLP
70.5
76.1
74.8=111
50,3
91.5..0:44
56.640.3
94.610.1
87.4
Random
74.5
L.
82.1
69.0=2.1
92.9
84.8
95,410.1
91.1
DARTS
73.6::09
75.7=00
86,5=23
70.4=06
92.8+0.3
79.7+:0.5
95.240.1
91.5++0
GraphNAS
73.6
75.2=009
77.5=00
62.7=113
91.61:0.3
69.010.6
94.5#0.1
89.3
PAS
74.6*m3
76.5=00
84,0=16
64.6213.8
GASSO
93.1+b0.3
84.9+0.
95.7=t011
92.0+0.3
DSGAS
76.0:0022
78.4
88.7=20.7
72.0=0.5
93.5.10.2
86.6:10.4
95.7:10.1
93.3+.0.3
The datasets cover various graph-related fields including small molecules, bioinformatics, social networks, e-commerce networks, and academic coauthorship networks. The statistics are summarized in Table
and Wechat- Wechat-Vided
Super-network We briefly introduce the super-network construction follows. The super-network consists of two parts, the operation pool and the directed acyclic graph (DAG). The operation pool includes several node aggregation operations (e.g., GCN, GIN, GAT, etc.), graph pooling operations (c.g., SortPool, AttentionPool, etc.), and layer merging operations (c.g., MaxMerge, ConcatMerge, etc.). The DAG determines how the operations connected calculate the graph representations for the subsequent classification tasks.
More details of the experiments are provided in the Appendix, including additional experiments and analyses, experimental setups, configurations, and implementation details.
4.1 Main Results
Unsupervised Settings From the experimental results summarized in Table we have the following observations: 1) GNNs perform differently on various datasets. The best GNN baselines for these datasets are GraphSage, GAT, GraphConv, GraphConv, GAT, GCN, GCN, GraphConv successively. and the performance varies greatly across datasets and GNNs. verifies that no GNN architecture is dominant for all datasets, which is consistent with the literature 332 and shows the demand for automated GNN architecture designing based on the data characteristics to obtain the optimal representations. 2) Most GNAS buselines fail in the unsupervised setting. Since the existing GNAS baselines highly rely on supervised signals to search the architectures, they inherently do not suit the unsupervised settings. As ad hoc remedy for the existing GNAS basclines in unsupervised settings, we substitute the supervised metrics with the self-supervised ones during the searching process simple extensions. However, these GNAS basclines, contrary to supervised settings, do not guarantee better performance than manually designed GNNs for all datasets. For example, all GNAS baselines even perform worse than manually designed GNNs DD dataset and do not have significant improvements on most datasets. The reasons behind might be that simply using graph self-supervised metrics does consider the entanglement of architectures and factors, leading to inaccurate estimation of the architectures' capabilities. 3) Our method has significant improvements over the buselines on mos! datasets. Compared with manually designed GNN baselines, DSGAS has a performance improvement of 3.1% % on PROTEINS and over 1% on most datasets. We contribute this its ability of automatically tailoring GNNs for various datasets, showing its effectiveness of automatic GNN designing. DSGAS also significantly surpasses GNAS baselines, showing its superiority in graph neural architecture search in unsupervised settings, which benefits from the design of discovering architectures that can capture various graph factors in self eef-suupervied fashion.
$ bbttps//algo.weixinngg.ok
7

---

DARTS
PAS
GASSO
DSGAS-P
DSGAS
Performance (%)
69
86
67-
65
83
64
61
80
57
61
77
53
58
74
10%
5%
1%
10%
5%
1%
10%
5%
1%
Label Rate (OGBG-Molhiv)
Label Rate (OGBN-Arxiv)
Label Rate
(Wechat-Video)
Figure 2: The performance of GNAS methods on real-world datasets under semi-supervised settings, where DSGAS-P denotes DSGAS without pretraining. The results are averaged by five random runs. (Best viewed in color)
DSGAS
DSGAS CONT
D59AS-SA
DSGAS-DSEN
Compese
Alpha
Weight
Embed
Accuracy (96)
Accuracy (96
PROTEINS
DD
Computers
Photo
PROTEINS
DD
Computers
Photo
(a)
(b)
Figure 3: (a) Comparisons different ablated variants of DSGAS on real-world datasets under unsupervised settings. The horizontal dashed line refers to the results of the best-performed GNAS baseline. (b) Comparisons of different architecture augmentations of DSGAS on real-world datasets under unsupervised settings, where 'Alpha', 'Weight' and 'Embed" denote the augmentations from perspective of operation choices, weight and embeddings. 'Compose' denotes uniformly choosing one of the three augmentations. The horizontal dashed line refers to the results of the best-performed GNAS baseline. (Best viewed in color)
Semi-supervised Settings From Figure 2 we have the following observations: Compared with the baselines, DSGAS significantly alleviates the performance drop when the number of available supervised labels is fewer, which verifies that our method fully exploits latent factors inside graph data and boost the supervised architecture search stage by warming up the weights and architecture parameters of the neterkn Its significant improvement over the ablated version DSGAS-P P also verifies the effectiveness pretraining the super-network by the proposed modules of self-supervised training with joint pehiitecture-grpph disentanglement and contrastive search with architecture augmentations. For example, our model with the pretraining stage has an absolute improvement of 5% Wechat Video dataset with 1% labels compared with the ablated version without the pretraining stage, which shows the effects pretraining the super-networks on alleviating the label scarcity issues. In comparison, the performance of other baselines decays significantly more than our method, showing that current GNAS can not tackle scenarios with scarce labels. For example, on OGBG-Molhiv, PAS is the best baseline while the worst with 10% and 1% labels respectively, which may due to the inaccurate performance estimation with scarce labels. The phenomenon further strengthens the necessity of designing effective unsupervised GNAS methods.
4.2 Additional Experiments
Ablation Studies We evaluate the effectiveness of each module of our framework by comparing the following ablated versions our method: DSGAS-CONT removes our proposed contrastive search module and search architectures with the vanilla self-supervised loss. DSGAS-FA further replaces our proposed factor-aware training module with the vanilla seff-super training. DSGAS-DISEN further replaces proposed disentangled super-network with the vanilla super-network.

---

INPUT
Crapa
GUN
Grapattove
Grapbtoon
CONCAT
(a) Computers
(b) DD
(c) Photo
(d) PROTEINS
Figure 4: Visualizations of the search architectures on different datasets. Nodes denote GNN operations except that 'INPUT' denotes the input graphs with structures and features. Directed edges denote calculation flows, where different colors denote the architecture operation choices under different factors. (Best viewed in color)
We compare the performance of the ablated versions and the best GNAS baselines on the real- world datasets under unsupervised settings. From Figure Ba we have the following observations. First, our proposed DSGAS outperforms all the variants on datasets, demonstrating the effectiveness of each component of our proposed framework in searching graph architectures under unsupervised settings. Second, DSGAS-CON drops drastically in performance on all datasets compared the full version, showing the superiority of proposed disentangled contrastive architecture search module in searching architectures. Third, the performance also decays for DSGAS- S-AA and OSGAS-DISHE on most datasets, showing the necessity of capturing various latent factors with different architectures.
Architecture Visualizations We visualize the architectures searched different unsupervised datasets Figure 4 It shows that the searched architectures different factors adopt quite different GNN operations while sometimes sharing the same operations, which leads to an overall architecture with complex internal connections between operations. This phenomenon implies that DSGAS can optimize the architecture operation choices well the operation connections for different factors to have competitive performance on various graph datasets, which also verifies the superiority of DSGAS in automated architecture search to save human endeavors for architecture designs.
Effects of Architecture Augmentations In Figure 3b we show the results of different architecture augmentation methods on different datasets compared with the best GNAS baseline. We find that though the best augmentations differ among datasets, they have similar performance improvements in most cases, which verifies the design contrastive search with architecture augmentations.
5 Related Works
Graph Neural Architecture Search Instead of manually designing more sophisticated models for various scenarios, neural architecture search, aiming to automatically discover the optimal architectures for given tasks, emerges hot topic recently in computer vision 446 23), natural language processing [77, graph representation learning
etc. In the field of graph graph neural architecture search (GNAS) methods, as the most related to our works, can be roughly classified into reinforcement-
48
representation learning with various applications
(50 51
52
53 54
methods [56] 2 57 58], and differentiable 64 65). However, supervised labels are indispensable for the existing GNAS methods to conduct neural architecture search, which limits their applications in widely-existed scenarios where labels are searce or not available.
bearning-basee methods 3. 55 evolutionary-
methods (59] 60, 6] 62
63
Unsupervised Neural Architecture Search In unsupervised settings, some neural architecture search methods replace supervised labels with self-supervised loss during searching [66] 67 68 69 70 Another classic of related methods design special metrics, whose calculation does not depend
For example, UnNAS 774 adopts pretext tasks like image rotation, coloring images, solving puzzles, etc. However, these methods are specially designed for computer vision, and can not be directly adopted to graph data. Some GNAS
on labels, as proxies for model performance estimation 72 73

---

works exploit self-supervised loss as auxiliaries to augment the supervised search process [63]] but the supervised labels are still mandatory for its search. To the best of our knowledge, this is the first work on unsupervised graph neural architecture search.
Graph Self-supervised Learning Graph self-supervised learning 75 76 is devoted to obtaining
graph representations by extracting informative knowledge with well dell-des pretext tasks without
labels, which can be roughly classified into contrastive [77] 778
79 80
81 82| [3 84 83 [6) and
generative [8 [88] 76 [9], and predictive methods (90] 91 92] The existing graph self-supervised
learning methods usually focus designing better pretext tasks with a fixed encoder such as
GCN 20 GAT [2 and GIN 222 Another class of related methods attempt to automate the choices
of pretext tasks 993 94 95
96 97 We mainly consider graph neural architecture in unsupervised settings, while other pretext tasks are orthogonal to our framework and can be incorporated.
Disentangled Representation Learning The primary objective of disentangled representation learning is to delineate and interpret the various latent factors which influence the data we encounter in observable context, rendering each of these factors unique vector representations [98 99).
It has emerged to be a useful tool in various domains, including those of computer vision 1100
101 [02 [003] [04 [05], and recommendation systems [106) [107, [08] [199] [10] 111
112
113]
graph representation learning 9 114
10 [4
[T5] 116 [17]. As the most related, GRACES
63
characterize the graph latent factors inside data by designing self-supervised disentangled graph encoder, and conduct graph neural architecture search for each graph to handle graph distribution shifts, while the training and searching process still follows the supervised paradigm. In contrast, focus on automating the GNN designs with disentangled self-supervision in this paper.
Conclusions
In this paper, we propose novel Disentangled celf-Super Graph Neural Architecture Search (DSGAS) framework to automate the GNN designs with disentangled self-supervision, which includes disentangled graph architecture super-network, self-supervised training with joint architecture-grat disentanglement and contrastive search with architecture augmentations. Extensive experiments demonstrate that our proposed method can discover architectures with capabilities of capturing various graph latent factors and significantly outperform the state-of-the-art GNAS baselines. Detailed ablation studies and analyses show the effectiveness of our method design. One limitation is that in this paper we mainly focus on homogeneous graphs, and we leave extending our method to heterogeneous graphs for further explorations.
Acknowledgements
This work supported by the National Key Research and Development Program of China No. 2020AAA0106300 National Natural Science Foundation of China (No. 62222209, 62250008, 62102222, 622061 186619999 49), Beijing National Research Center for Information Science and Technology under Grant No. BNR2023RC01005 BNR2023TD03006, China National Postdoctoral Program for Innovative Talents No. BX20220185. China Postdoctoral Science Foundation No. 2022M71 1813, and Beijing Key Lab of Networked Multimedia. All opinions, findings, conclusions, and recommendations in this paper are those of the authors and do not necessarily reflect the views of the funding agencies.
References
[1] Ziwei Zhang. Xin Wang, and Wenwu Zhu. Automated machine learning on graphs: A survey, arXiv preprint 1rXiv:2133.007. 2021.
[2] Yaoman Li and Irwin King. Autograph: Automated graph neural network. In International Conference Neural Information Processing, pages 189-201, 2020.
[3] Yang Gao, Hong Yang, Peng Zhang, Chuan Zhou, and Yue Hu. Graph neural architecture search. In IJCAL, volume 20, pages 1403-1409, 2020.
[4] Lanning Wei, Huan Zhao, Quanming Yao, and Zhiqiang He. Pooling architecture search for graph classification. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2091 1090- 2021.
10

---

[5] Linda Martin, Melissa Hutchens, Conrad Hawkins, and Alaina Radnov. How much do clinical trials cost. Nat Rev Drug Discov, 16(6):311332, 2017.
[6] Debleena Paul, Gaurav Sanap, Snehal Shenoy, Dnyaneshwar Kalyane, Kiran Kalia, and Rakesh K Tekade. Artificial intelligence in drug discovery and development. Drug discovery today, :600, 2021.
[7] Duxin Sun, Wei Gao, Hongxiang Hu, and Simon Zhou. Why 90% of clinical drug development fails and how to improve it? Acta Pharmaceutica Sinica B, 2022.
[8] Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. Graph neural networks for social recommendation. In The world wide web conference, pages 417-426, 2019.
[9] Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, and Wenwu Zhu. Disentangled graph convolutional networks. In International conference on machine learning, pages 4212-4221. PMI., 2019.
[10] Yiding Yang, Zunlei Feng. Mingli Song, and Xinchao Wang. Factorizable graph convolutional networks. Advances in Neural Information Processing Systems, 33:20286-2008 20296, 2020.
[11] Ansong Li, Zhiyong Cheng, Fan Liu, Zan Gao, Weili Guan, and Yuxin Peng. Disentangled graph neural networks for session-based recommendation. arXiv preprint arXiv:2201.03455 2022.
[12] Thilini Cooray and Ngai-Man Cheung. Graph-wise common latent factor extraction for unsupervised graph representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6420-6428, 2022.
[13] Teng Xiao, Zhengyu Chen, Zhimeng Guo, Zeyang Zhuang, and Suhang Wang. Decoupled self- self-supervised learning for non non-homophilous graphs. arXiv preprint arXiv:2206.0360 2022.
[14] Haoyang Li, Xin Wang. Ziwei Zhang, Zehuan Yuan, Hang Li, and Wenwu Zhu. Disentangled contrastive learning on graphs. Advances in Neural Information Prucessing Systems, 0921872-218885 2021.
[15] Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovee. Hicrarchical graph representation learning with differentiable pooling. Advances in neural information processing systems, 31, 2018.
[16] Zhitao Ying. Dylan Bourgeois. Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnexplainer: Gener- ating explanations for graph neural networks. Advances in neural information processing systems, 32, 2019.
[17] Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. Fair darts: Eliminating unfair advantages in differentiable architecture search. In European conference computer vision, pages 465-480. Springer, 2020.
[18] Xiangxiang Chu, Bo Zhang, and Ruijun Xu. Fairnas Rethinking evaluation fairness of weight sharing neural architecture search. Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 12239-1222 12248, 2021.
[19] Stephen Cha, Tachyeon Kim, Haycon Lec, and Se- Young Yun. Supemet in neural architecture search: A taxonomic survey, arXiv preprint arXiv:2204.0399 2022.
[20] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint XXXx:I609.02007. 2016.
[21] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
[22] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint axXiv:1810.0088 2018.
[23] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 20(1):1997-200 2019.
[24] Hicu Pham, Melody Guan, Barret Zoph, Quoc Lc, and Jeff Dean. Efficient neural architccture search via parameters sharing. In International conference on machine learning, pages 4095-4104. PMLR, 2018.
[25] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint
2018

---

[26] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In European conference on computer vision, pages 544-560. Springer, 2020.
[27] Lirong Wu, Haitao Lin, Cheng Tan, Zhangyang Gao, and Stan Zii. Self-supervised learning on graphs: Contrastive, generative, or predictive. IEEE Transactions on Knowledge and Data Engineering, 2021.
[28] Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, and Mi Zhang. Does unsupervised architecture representation learning help neural architecture search? Advances in Neural Information Processing Systems, 33: 1: 12486- 12498, 2020.
[29] Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture optimization. Advances in neural information processing systems, 31, 2018.
[30] Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas-bench- 101: Towards reproducible neural architecture scarch. In International Conference on Machine Learning, pages 7105-7114. PMLR, 2019.
[31] Arber Zela, Julien Niklas Siems, Lucas Zimmer, Jovita Lukasik, Margret Keuper, and Frank Hutter. Surrogate nas benchmarks: Going beyond the limited search spaces of tabular nas benchmarks. In Tenth International Conference on Learning Representations, pages 36. OpenReview. net, 2022.
[32] Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. Advances in Neural Information Processing Systems, 33:17009-17021 2020. 2020.
[33] Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering. 2021.
[34] Yifei Wang, Qi Zhang. Yisen Jiansheng Yang, and Zhouchen Lin. Chaos is ladder: A new theo- retical understanding of contrastive learning via augmentation overlap. arXiv preprint arXiv:2203. 13457, 2022.
[35] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
[36] Christopher Morris, Martin Ritzert, Matthias Fey. William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, pages 4602-4609, 2019.
[37] Yijian Qin, Xin Wang, Zeyang Zhang, and Wenwu Zhu. Graph diffcrentiable architecture search with structure learning, Advances in Neural Information Processing Systems, 34, 2021.
[38] Paul D Dobson and Andrew J Doig. Distinguishing enzyme structures from non-enzymes without alignments. Journal of molecular biology, 330(4):771-783, 2003.
[39] Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research, 12(9). 2011.
[40] Asim Kumar Dehnath, Rosa L Lopez de Compadre, Gargi Debnath, Alan Shusterman, and Corwin Hansch. Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. correlation with molecular orbital energies and hydrophobicity. Journal of medicinal chemistry, 34(2):786- 797, 1991.
[41] Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 1365-1374, 2015.
[42] Christopher Morris, Nils M Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of henchmark datasets for learning with graphs. arXiv preprint arXiv 2020.
[43] Amab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Hsu, and Kuansan Wang. An overview of microsoft academic service (mas) and applications. In Proceedings of the 24th international conference on world wide web, pages 243-246, 2015.
[44] Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based recommen- dations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pages 43 52, 2015.
12

---

[45] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118-22133, 2020.
[46] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po- Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. comprehensive survey of neural architecture search; Challenges and solutions. ACM Computing Surveys (CSUR), 54(4):1-34 2021.
[47] Krishna Teja Chitty-Venkata, Venkata, Murali Emani, Venkatram Vishwanath, and Arun Somani. Neural architecture search for transformers: A survey. IEEE Access, 10:108374 108412, 2022.
[48] Chaoyu Guan, Ziwei Zhang, Haoyang Li, Heng Chang, Zeyang Zhang. Yijian Qin, Jiyan Jiang, Xin Wang, and Wenwu Zhu, Autogl: A library for automated graph learning. In ICLR 2021 Workshop GTRL, 2021.
[49] Yijian Qin, Ziwei Zhang. Xin Wang, Zeyang Zhang, and Wenwu Zhu. Nas-bench graph: Benchmarking graph neural architecture search. arXiv preprint arXiv:2206.09) 19660, 2022.
[50] Ziwei Zhang, Peng Cui, and Wenwu Zhu. Deep learning on graphs: A survey, IEEE Transactions on Knowledge and Data Engineering, 34(1):249-270, 2020.
[51] Ziwei Zhang, Haoyang Li, Zeyang Zhang, Yijian Qin, Xin Wang, and Wenwu Zhu. Large graph models: A perspective. arXiv preprint 2i3: 14522, 2023.
[52] Zeyang Zhang, Ziwei Zhang, Xin Wang. and Wenwu Zhu. Learning to solve travelling salesman problem with hardness-adaptine curriculum. In Thirty-Fitth AAAI Conference on Artificial Intelligence, 2022.
[53] Ziwei Zhang, Xin Wang, Zeyang Zhang, Peng Cui, and Wenwu Zhu. Revisiting transformation invariant geometrie deep learning: Are initial representations all you need? arXiv preprint 1rX1v:2112.12345. 2021.
[54] Zeyang Zhang, Xin Wang, Ziwei Zhang, Haoyang Li, Yijian Qin, Simin Wu, and Wenwn Zhu, Llm4dyg: Can large language models solve problems on dynamic graphs? arXiv preprint, 2023.
[55] Kaixiong Zhou, Qingquan Song, Xiao Huang, and Xia Hu, Auto-gnn: Neural architecture search of graph neural networks. 144iv:990900311 2019.
[56] Matheus Nunes and Gisele Pappa. Neural architecture search graph neural networks. Bruzilian Conference on Intelligent Syxtems, pages 302-317, 2020.
[57] Min Shi, David Wilson, Xingquan Zhu, Yu Huang, Yuan Zhuang, Jianxun Liu, and Yufei Tang. Genetic-gnn. Evolutionary architecture search for graph neural networks. Knowledge- Based Systems. page 108752, 2022.
[58] Wentao Zhang. Zheyu Lin, Yu Shen, Yang Li, Zhi Yang. and Bin Cui. Deep and flexible graph neural architecture search. In International Conference on Machine Learning, pages 26362-26374, PMLR, 2022.
[59] Yuhui Ding, Quamming Yao, Huan Zhao, and Tong Zhang. Diffng: Differentiable meta graph search for heterogeneous graph neural networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, pages 279 779-288,77 2021.
[60] ZHAO Huan, YAO Quanming, and TU Weiwei. Search aggregate neighborhood for graph neural network. In 202/ IEEE 37th International Conference Data Engineering, pages 552-563, 2021.
[61] Yanxi Li, Zean Wen, Yunhe Wang, and Chang Xu. One-shot graph neural architecture search with dynamic search space. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 8510-8517, 2021.
[62] Shaofei Cai. Liang Li, Jinean Deng, Beichen Zhang, Zheng-Jun Zha, Li Su, and Qingming Huang. Rethinking graph neural architecture search from pressage-passing Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6657-6666, 2021.
[63] Yijian Qin, Xin Wang, Ziwei Zhang, Pengtao Xie, and Wenwu Zhu. Graph neural architecture search under distribution shifts. In International Conference on Machine Learning, pages 18083-18095. PMLR, 2022.
[64] Zeyang Zhang, Ziwei Zhang, Xin Wang. Yijian Qin, Zhou Qin, and Weawu Zhu. Dynamic heterogeneous graph attention neural architecture search. In Thirty- Sevemth AAAI Conference on Artificial Intelligence. 2023.
13

---

[65] Yijian Qin, Xin Wang, Ziwei Zhang, Hong Chen, and Wenwu Zhu. Multi-task graph neural architecture search with task-aware collaboration and curriculum, Advances in neural information processing systems, 2023.
[66] Sapir Kaplan and Raja Giryes. arXiv:2007.01500, 2020.
Self-supervised neural architecture search.
arXiv preprint
[67] Aleksandr Timofeev, Grigorios G Chrysos, and Volkan Cevher. Self selfrsuue neural architecture search for imbalanced datasets. arXiv preprint arXiv:210.008580, 2021.
[68] Nam Nguyen and J Morris Chang. Csnas: Contrastive self-supervised learning neural architecture search via sequential model-based optimization. IEEE Transactions on Artificial Intelligence, (24,6609624, 2021.
[69] Changlin Li. Tao Tang. Guangrun Wang, Jiefeng Peng, Bing Wang. Xiaodan Liang, and Xiaojun Chang. Bossnas: Exploring hybrid cuntrransformers with block-wisely sule-super neural architecture search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12281-12291, 2021.
[70] Zhuowei Li. Yibo Gao, Zhenzhou Zha, Zhiqiang Hu, Qing Xia, Shaoting Zhang, and Dimitris N Metaxas. Towards self-supervised and weight-preserving neural architecture search. arXiv preprint arXiv:2206.04 04/5, 2022.
[71] Zhenyu Hou, Yukuo Cen, Yuxiao Dong, Jie Zhang, and Jie Tang. Automated unsupervised graph representation learning. IEEE Transactions on Knowledge and Data Engineering, 2021.
[72] Xuanyang Zhang, Pengfei Hou, Xiangyu Zhang, and Jian Sun. Neural architecture search with random labels. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 10907-10916, 2021.
[73] Joe Mellor, Jack Tumnrr, Amos Storkey, and Elliot Crowley, Neural architecture search without training. In International Conference on Machine Learning, pages 7588-7598. PMLR, 2021.
[74] Chenxi Liu, Piotr Dollár, Kaiming He, Ross Girshick, Alan Yuille, and Saining Xie. Are labels necessary for neural architecture search? In European Conference on Computer Vision, pages 798-813. Springer, 2020.
[75] Yixin Liu, Ming Jin, Shirui Pan, Chuan Zhou, Yu Zheng, Feng Xia, and Philip Yu. Graph self-supervised learning: survey. IEEE Transactions Knowledge and Data Engineering. 2022.
[76] Yaochen Xie, Zhao Xu, and Shuiwang Ji. Sule-super representation learning via latent graph prediction. arXiv preprint arXiv:2202. 220X::2202208333,, 2022.
[77] Petar Velickovic, William Fedus, William L Hamilton, Pictro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. ICLR (Poster), 2(3)44 2019.
[78] Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv:1908.01000 2019.
[79] Zhen Peng, Wenhing Huang, Minnan Luo, Qinghua Zheng, Yu Rong. Tingyang Xu, and Junzhou Huang. Graph representation learning via graphical mutual information maximization. In Proceedings of The Web Conference 2020, pages 270-270, 2020.
[80] Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph- level representation learning with local and global structure. In International Conference on Machine Learning, pages 11548 1 11558. PMLR, 2021.
[81] Jiezhong Qiu, Qibin Chen, Yuxiao Dong. Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gee: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1150-1160, 2020.
[82] Yuehui Han, Le Hui, Haobo Jiang. Jianjun Qian, and Jin Xie. Generative subgraph contrast for self- supervised graph representation learning. arXiv preprint arXiv:2207. 11996, 2022.
[83] Sihang Li, Xiang Wang. An Zhang, Yingxin Wu, Xiangnan He, and Tat- Chua. Let invariant rationale discovery inspire graph contrastive learning. In International Conference on Machine Learning, pages 13052- 13065. PMLR, 2022.
14

---

[84] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.0413: 04131, 2020.
[85] Yuning You, Tianlong Chen. Yongduo Sui. Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems, 33:5812- 5823, 2020.
[86] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi- view representation lcarning graphs International Conference on Machine Learning, pages 4116 4126. PMLR, 2020.
[87] Zhenyu Hou, Xiao Liu, Yuxiao Dong, Chunjie Wang, Jie Tang, et al. Graphmae: Self-supervised masked graph autocncoders. arXiv preprint arXiv:2205. 10803, 2022.
[88] Jintang Li, Ruofan Wu, Wangbin Sun, Liang Chen, Sheng Tian, Liang Zhu, Changhua Meng. Zibin Zheng, and Weigiang Wang. Maskgae: Masked graph modeling meets graph autoencoders. arXiv preprint arXiv:2205. 10053, 2022.
[89] Zhenyu Hou, Yufci He, Yukuo Cen, Xiao Liu, Yuxiao Dong, Evgeny Kharlamov, and Jie Tang. Graphmae2: A decoddnng-enaanced masked self-supervised graph learner. In Proceedings of the ACM Web Conference 2023, pages 737-746, 2023.
[90] Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. When does self-supervision help graph convolutional networks? In international conference on machine learning, pages 10871-10880. PMLR, 2020.
[91] Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self- supervised graph transformer on large-scale molecular data, Advances in Neural Information Processing Systems, 33:12559-12577 2020.
[92] Wei Jin, Tyler Derr, Haochen Liu, Yiqi Wang, Suhang Wang, Zitao Liu, and Jiliang Tang. Self-supervised learning on graphs: Deep insights and new direction. arXiv preprint arXiv:2006.10144 2020.
[93] Yuning You. Tranlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In International Conference on Machine Learning, pages 12121-12132. PMLR, 2021.
[94] Yihang Yin, Qingzhong Wang. Siyu Huang, Haoyi Xiong, and Xiang Zhang, Autogel: Automated graph contrastive learning via learnable view generators. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8892-8900, 2022.
[95] Yanqiao Zhu Yichen Xu, Feng Yu. Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In Proceedings of the Web Conference 2021. pages 206922000,200 2021.
[96] Wei Jin, Xiaorui Liu, Xiangyu Zhao, Yao Ma, Neil Shah, and Jiliang Tang. Automated self sulf-supe lcarning for graphs. arXiv preprint arXiv: 05470, 2021.
[97] Mingxuan Ju. Tong Zhao, Qianlong Wen, Wenhao Yu, Neil Shah, Yanfang Ye, and Chuxu Zhang. Multi-task self-supervised graph neural networks enable stronger task generalization. arXiv preprint 10Xiv:2100.00016, 2022.
[98] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: review and new perspectives. IEEE transactions on pattern amalysis and machine intelligence, 35(8): 1798-18898 2013.
[99] Xin Wang, Hong Chen, Si'an Tang, Zihao Wu, and Wenwu Zhu. Disentangled representation learning, 2023.
[100] Jun-Ting Hsich, Binghin Liu, De-An Huang. Li F Fei-Fei, and Juan Carlos Niebles. Learning decom- pose and disentangle representations for video prediction, Advances in neural information processing systems, 31, 2018
[101] Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool, Bernt Schiele, and Mario Fritz. Disentan- gled person image generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 99 108, 2018
[102] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets, Advances in neural information processing systems, 29, 2016.
[103] Emily L Denton et al. Unsupervised learning of disentangled representations from video. Advances in neural information processing systems, 30, 2017.
15

---

[104] Luan Tran, Xi Yin, and Xiaoming Liu. Disentangled representation learning for pose-invariant face recognition. In Proceedings of the IEEE conference on computer vision and patiern recognition, pages 1415-1424, 2017.
[105] Xin Wang, Zihao Wu, Hong Chen, Xiaohan Lan, and Weawu Zhu. Mixup-augmented temporally debiased video grounding with content- locetton disentanglement. 2023.
[106] Xin Wang, Hong Chen, Yuwei Zhou, Jianxin Ma, and Wenwu Zhu. Disentangled representation learning for recommendation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[107] Hong Chen, Yudong Chen, Xin Wang, Ruobing Xie, Rui Wang, Feng Xia, and Weawu Zhu. Curriculum disentangled recommendation with noisy multi-feedback. Advances in Neural Information Processing Systems, 34:26924-26996 2021.
[108] Wang. Hong Chen, Wenwu Zhu. Multimodal disentangled representation for recommendation. In 2021 IEEE International Conference on Multimedia and Expo (ICME), pages 1 1-6, 2021.
[109] Jianxin Ma, Chang Zhou, Hongxia Yang, Peng Cui, Wang, and Wenwu Zhu. Disentangled self- supervision in sequential recommenders. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 483 491, 2020.
[110] Jianxin Ma, Chang Zhou, Peng Cui, Hongxia Yang. and Wenwu Zhu. Learning disentangled representa- tions for recommendation. Advances in neural information processing systems, 32, 2019.
[111] Haoyang Li, Xin Wang, Ziwei Zhang, Jianxin Ma, Peng Cui, and Wenwu Zhu. Intention-awe aware se- quential recommendation with structured intent transition. IEEE Transactions on Knowledge and Data Engineering. 2021.
[112] Xin Wang, Zirui Pan, Yuwei Zhou. Hong Chen, Chendi Ge, and Wenwu Zhu. Curriculum co-disentangled representation learning across multiple environments for social recommendation. 2023.
[113] Yipeng Zhang, Xin Wang. Hong Chen, and Wenwu Zhu. Adaptive disentangled transformer for sequential recommendation. In Proceedings the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pages 3434-344000 3445, 2023.
[114] Yanbei Liu, Xiao Wang. Shu Wu, and Zhitao Xiao. Independence promoted graph disentangled networks. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4916 142244 2020.
[115] Zeyang Zhang, Xingwang Li, Fei Teng, Ning Lin, Xueling Zhu, Xin Wang, and Wenwu Zhu. Out- of-distribution generalized dynamic graph neural network for human albumin prediction. IEEE International Conference on Medical Artificial Intelligence, 2023.
[116] Zeyang Zhang. Xin Wang, Ziwei Zhang, Zhou Qin, Weigao Wen, Hui Xue, Haoyang Li, and Wenwu Zhu. Spectral invariant learning for dynamic graphs under distribution shifts. In Advances in Neural Information Processing Systems, 2023.
[117) Zeyang Zhang, Xin Wang, Ziwei Zhang, Haoyang Li, Zhou Qin, and Wenwu Zhu. Dynamic graph neural networks under apatio-tempora distribution shift. In Advances in Neural Information Processing Systems, 2022.
16


Reviews:
[{"review_id": "cArkQzkJ2H", "reviewer": "Reviewer_bqLE", "summary": "In this paper, the authors study the problem of unsupervised graph neural architecture search, which remains unexplored in the literature. The authors propose a novel Disentangled Self-supervised Graph Neural Architecture Search (DSGAS) model, which is able to discover the optimal architectures capturing various latent graph factors in a self-supervised fashion based on unlabeled graph data. Extensive experiments on 11 real-world datasets demonstrate that the proposed DSGAS model is able to achieve state-of-the-art performance against several baseline methods in an unsupervised manner.", "questions": "1. What is the relationship between supervised and unsupervised nas paradigms in terms of the optimization problem, e.g., line 84 in the main paper?\n\n2. In Figure 1, the super-network seems a simple DAG, why do the searched architectures in Figure 4 seem kind of complex in terms of the operation connections?\n\n3. In Figure 1, it seems that the mixed operation has different colors for different $\\alpha_i$, is it possible that some factor-wise architectures share the same operation?\n\n4. In Figure 1, is it required that all the architectures should choose the same augmentations? In other words, is it possible that some architectures adopt operation choice perturbation, while others adopt weight perturbation in the same batch for contrastive learning?", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. I believe the problem of unsupervised graph neural architecture search is important, and this work is significant that it may unlock more NAS applications in practice. \n\n2. The writing is clear and well-organized in general. \n\n3. The experments and analyses are extensive.", "weaknesses": "Please see my comments in the questions part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the detailed comments and insightful questions. We make responses to the reviewer’s comments as follows.\n\n> What is the relationship between supervised and unsupervised nas paradigms in terms of the optimization problem, e.g., line 84 in the main paper?\n\nThank you for your comment. The problem of unsupervised GNAS can be formulated as optimizing an architecture generator that is able to discover powerful architectures by exploiting inherent graph properties without labels. Since the labels are not available under unsupervised settings, the validation and training metrics in Line 84 can not be calculated to measure the architecture performance and search architectures as in supervised NAS methods. \n\n> In Figure 1, the super-network seems a simple DAG, why do the searched architectures in Figure 4 seem kind of complex in terms of the operation connections? \n\nThank you for your question. The proposed disentangled super-network can flexibly incorporate multiple architectures to be searched with regard to various graph factors. This design alleviates the entanglement of architectures by providing more flexible choices of paths in the super-network, which also results in complex yet competitive architectures. \n\n> In Figure 1, it seems that the mixed operation has different colors for different $\\alpha_i$, is it possible that some factor-wise architectures share the same operation?\n\nThank you for your question. Yes, some factor-wise architectures can share the same operation.  During the search process, the architectures are encouraged to capture different graph factors but are not prohibited from sharing the same operation. This enables the factor-wise architectures to share some common knowledge in learning the graph properties.\n\n> In Figure 1, is it required that all the architectures should choose the same augmentations? \n\nThank you for your question. No, the architectures can choose different augmentations. Like the data augmentations in contrastive learning, all the architecture argumentations act as a transformation of an architecture to a similar architecture. In Figure 3(b), 'Compose' randomly chooses different architecture augmentations for each architecture, and it is shown to be effective on various datasets. "}, {"review_id": "8PH3JChT1i", "reviewer": "Reviewer_5ige", "summary": "This paper mainly focuses on the problem of graph neural architecture search without labels. The authors find that the key problem is to discover the latent graph factors that drive the formation of graph data as well as the underlying relations between the factors and the optimal neural architectures. To this end, the authors propose the disentangled super-network, and design a self-supervised training and searching paradigm to automate the architecture design. The experiment results are significant and the deeper analyses are convincing.", "questions": "Please see the weaknesses.", "limitations": "None.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper mainly focuses on the problem of graph neural architecture search without labels, which is important yet underexplored in the literature.\n\n- The proposed methods sound reasonable. \n\n- The experiment results are significant and the deeper analyses are convincing.", "weaknesses": "- Can the authors explain the details of the intuition for the proposed contrastive search?\n\n- The experiments include both node classification and graph classification tasks. Do these tasks share the same search space?\n \n- As the existing graph NAS methods are in need of labels for training and searching, how do the authors modify them to compare as baselines?\n\n- What about the complexity of the method?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the detailed comments and suggestions. We respond to the reviewer’s comments point by point as follows.\n\n> Can the authors explain the details of the intuition for the proposed contrastive search?\n\nThank you for your suggestion. We provide the details of the intuition for the proposed contrastive search as follows. As architectures similar in operation choices and topologies have similar capabilities of capturing semantics for downstream tasks, slightly modifying the architecture will have a slight influence on its capability. Moreover, since different GNN architectures are experts in different downstream tasks, the architectures searched for different disentangled latent factors are expected to have dissimilar capabilities under different factors. For these two reasons, we propose the contrastive architecture search to capture discriminative features by pulling similar architectures together and pushing dissimilar architectures away in the latent space. \n\n> The experiments include both node classification and graph classification tasks. Do these tasks share the same search space?\n\nThank you for your comment. In comparison with the search space for node classification tasks, the search space for graph classification tasks has two extra kinds of tailorable operations, i.e., graph pooling operations to capture global representations and layer merging operations to provide jumping knowledge. We provided the details of the search space in Appendix D.1. \n\n> As the existing graph NAS methods are in need of labels for training and searching, how do the authors modify them to compare as baselines?\n\nThank you for your question. We first pre-train the models by self-supervised loss and then evaluate the models by finetuning an extra classifier. For the graph NAS baselines, the self-supervised loss is utilized to substitute the supervised loss to train the model parameters as well as select the architectures. \n\n> What about the complexity of the method?\n\nThank you for your suggestion. We provide the complexity analysis as follows. Denote the number of nodes and edges in the graph as $N$ and $E$, the number of latent factors as $K$, the number of operation choices as $|\\mathcal{O}|$, the dimensionality of hidden representations as $d$. The time complexity of the disentangled super-network is $O(K|E|d + K|V|d^2)$, where the computation for each factor is fully parallelizable and amenable to GPU acceleration, and $K$ is usually a small constant. The time complexity of the self-supervised training and contrastive search modules is both $O(K^2d^2)$. As architectures under different factors share the parameters, the number of learnable parameters is the same as classical graph super-network, i.e., $O(|\\mathcal{O}|d^2)$. Therefore, the complexity of our method is comparable to classical GNAS methods."}, {"review_id": "0SMLqDWHYB", "reviewer": "Reviewer_Hgo9", "summary": "This paper gives a pioneer solution for graph neural architecture search with limited labels. The key idea is to train a super-network containing disentangled factor-wise architectures by a self-supervised learning specially designed for graph neural architecture search. In this way, the paper addresses the key problem of finding the optimal architectures capturing various graph factors in an unsupervised fashion. The extensive experiments show that the method achieves signifiant improvements over the state-of-the-art GraphNAS methods under unsupervised settings. Detailed ablation studies also show the effectiveness of each proposed components. \n", "questions": "\n1. Although it may fall outside the scope of this paper, i'm curious about whether the model can be applied into areas like AI4Science where labels are quite scarce?\n\n2. Pretraining techniques have shown promising applications in many scenarios like NLP and CV, and they usually have a close relationship with unsupervised or self-supervised techniques. It would better to discuss the relationship between this method and GraphNAS pretraining.\n", "limitations": "I don't see any concerns regarding the societal impact in this work.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Unsupervised GraphNAS is an important and valuable problem which remains unexplored in the literature. This paper is a timely work to introduce GraphNAS in the scenarios where labels are scarce, and these scenarios are actually quite common in practice. The method design is novel, where the contrastive search module is especially interesting that it pushes together architectures with similar factor expertises and pull away unsimilar architectures by architecture augmentations to explore the search space. The extensive experiments verify its capability of automating the architecture design for the data from various fields.\n", "weaknesses": "\n1. Although it may fall outside the scope of this paper, i'm curious about whether the model can be applied into areas like AI4Science where labels are quite scarce?\n\n2. Pretraining techniques have shown promising applications in many scenarios like NLP and CV, and they usually have a close relationship with unsupervised or self-supervised techniques. It would better to discuss the relationship between this method and GraphNAS pretraining.\n\n3. The framework diagram (figure 1) is a bit small, especially that the font size which is too small to read clearly. It would be better to enlarge the diagram and increase the font size a bit. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the valuable suggestions, which are helpful for the improvement of the paper. We respond to the reviewer’s comments point by point as follows.\n\n> Although it may fall outside the scope of this paper, i'm curious about whether the model can be applied into areas like AI4Science where labels are quite scarce?\n\nThanks for your inspiring comment. We agree that there exist some trending areas like AI for Science that are close to the problem we focus on, i.e., graph neural architecture search with limited or even no labels. For instance, we notice that some recent works attempt to comprehend the relationship between metabolic pathways and molecular pathways for synthesizing new molecules by leveraging the power of graph neural networks. In these scenarios, labels, e.g., the molecular properties or subtypes, are limited, posing great challenges for graph neural architecture search to automatically enhance the GNN architectures. Since our method is specially designed for GNAS with limited or even no labels, these scenarios are natural and promising applications, which we leave to future works for explorations.\n\n> Pretraining techniques have shown promising applications in many scenarios like NLP and CV, and they usually have a close relationship with unsupervised or self-supervised techniques. It would better to discuss the relationship between this method and GraphNAS pretraining.\n\nThanks for your comment. To the best of our knowledge, it remains unexplored for GraphNAS pretraining, which could simultaneously contain multiple goals like self-supervised learning, transfer learning, multi-task learning, etc. In this paper, we mainly focus on unsupervised graph neural architecture search, i.e., to discover graph architectures without labels. In semi-supervised experiments, we also show some progress in pretraining the super-networks to alleviate the label scarcity issues. For example, our model with the pretraining stage has an absolute improvement of 5% on one dataset compared with the ablated version without the pretraining stage. We will leave exploring more aspects of GraphNAS pretraining in future works.\n\n> The framework diagram (figure 1) is a bit small, especially that the font size which is too small to read clearly. It would be better to enlarge the diagram and increase the font size a bit.\n\nThank you for your suggestions. We will improve the diagram in the revised main paper."}, {"review_id": "CGjN1BwLrP", "reviewer": "Reviewer_GehM", "summary": "This paper addresses the problem of unsupervised graph neural architecture search, which has received limited attention in existing literature. The authors propose a novel approach called Disentangled Self-supervised Graph Neural Architecture Search (DSGAS) to discover optimal architectures capturing latent graph factors in an unsupervised manner. The DSGAS model consists of a disentangled graph super-network that incorporates multiple architectures with factor-wise disentanglement, self-supervised training to estimate architecture performance under different factors, and a contrastive search method to discover architectures with factor-specific expertise. Extensive experiments on 11 real-world datasets demonstrate that the proposed model achieves state-of-the-art performance compared to several baseline methods.\n", "questions": "Please see Weaknesses.\n", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper addresses an important and underexplored problem in the field of graph neural architecture search, i.e., scenarios where supervised labels are not available.\n\n- The proposed DSGAS model introduces a novel approach to discovering optimal architectures by leveraging latent graph factors in a self-supervised fashion based on unlabeled graph data.\n\n- The disentangled graph super-network and the self-supervised training with joint architecture-graph disentanglement are novel contributions that enhance the understanding and performance of the proposed model.\n\n- The extensive experimental evaluation on 11 real-world datasets demonstrates the superiority of the DSGAS model over several baseline methods, showing its effectiveness in unsupervised graph neural architecture search.\n", "weaknesses": "- While the paper presents a novel approach, further details regarding the implementation of the proposed DSGAS model in the main paper would be beneficial for the readers to understand the design clearly. For example, it is better to illustrate the details of the supernet construction in the main paper.\n\n- Some typos should be corrected, e.g., heterogenous -> heterogeneous on line 323.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the valuable comments. We respond to the reviewer’s comments point by point as follows.\n\n> While the paper presents a novel approach, further details regarding the implementation of the proposed DSGAS model in the main paper would be beneficial for the readers to understand the design clearly. For example, it is better to illustrate the details of the supernet construction in the main paper.\n\nThank you for your suggestion. We described the details of the super-network configuration in the appendix due to the page limit. Here, we briefly describe the super-network. The super-network consists of two parts, the operation pool and the directed acyclic graph (DAG). The operation pool includes several node aggregation operations (e.g., GCN, GIN, GAT, etc.), graph pooling operations (e.g., SortPool, AttentionPool, etc.), and layer merging operations (e.g., MaxMerge, ConcatMerge, etc.). The DAG determines how the operations are connected to calculate the graph representations for the subsequent classification tasks. We will add the illustrations in the revised main paper.  \n\n> Some typos should be corrected, e.g., heterogenous -> heterogeneous on line 323.\n\nThank you for your suggestions. We will correct the typos in the revised main paper."}, {"review_id": "LKhy3U1WeO", "reviewer": "Reviewer_JBpb", "summary": "This paper proposed a method DSGAS to automcatiicaly design the architectures in an unsupervised manner. It discovers the optimal  architectures by learning the latent graph factor.", "questions": "Please check the weaknesses.", "limitations": "Please check the weaknesses.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "It is interesting and novel to automate the design of GNNs in an unsupervised manner, which holds significant potential.\n", "weaknesses": "1. The necessity of designing disentanglement is not clear. It is true that GNNs may preferred different graph factors. What is the difference between the proposed method and the baseline that: using one latent factor design one architectures, and then run multiple times and ensembles the search architectures? \nThe motivation for disentangled architectures is not well justified. While the evaluations on K are provided, it would be more effective to achieve K=2 by ensemble two K=1 results.\n\n\n2. The experiments are insufficient to justify the effectiveness of the proposed design method. Since it designs GNNs in an unsupervised manner, it would be better to conduct fair comparisons with existing unsupervised methods rather than methods designed under a supervised manner.\n\n3. Apart from the comparisons of performance, the comparisons of search cost, the parameters of the searched architectures should be provided as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the comments and suggestions. We have carefully reviewed each point raised and make responses to the reviewer point by point as follows.\n\n> The necessity of designing disentanglement is not clear. It is true that GNNs may preferred different graph factors. What is the difference between the proposed method and the baseline that: using one latent factor design one architectures, and then run multiple times and ensembles the search architectures? The motivation for disentangled architectures is not well justified. While the evaluations on K are provided, it would be more effective to achieve K=2 by ensemble two K=1 results.\n\nThank you for your comment. Following your suggestion, we compare a baseline that ensembles K searched architectures by averaging their outputs. The results on the dataset Computers are shown in the following table. \n\n| K        | 2          | 3          | 4          | 5          |\n| :------- | :--------- | :--------- | :--------- | :--------- |\n| Ours     | 85\\.1+-0.4 | 86\\.7+-0.4 | 87\\.3+-0.4 | 86\\.5+-0.4 |\n| Ensemble | 84\\.0+-0.3 | 84\\.2+-0.4 | 84\\.0+-0.5 | 83\\.2+-0.4 |\n\nAs shown in the table, our method has a significant performance improvement over the ensemble baseline. The reason might be that the ensemble baseline can not learn to capture various graph factors, and accordingly can not search the optimal architectures with regard to different graph factors. In contrast, our method is an end-to-end manner, which is learned to jointly discover the latent factors and search multiple factor-wise expert architectures to achieve state-of-the-art performance on various datasets.\n\n> It would be better to conduct comparisons with existing unsupervised methods rather than methods designed under a supervised manner.\n\nThank you for your comment. To the best of our knowledge, the study of unsupervised graph neural architecture search remains unexplored in the literature, and our method is the first GNAS method designed under unsupervised settings. The existing GNAS methods are designed in a supervised manner, and for fair comparisons, we extend these methods as baselines by replacing the supervised loss with the self-supervised loss to suit the unsupervised NAS settings.\n\n> Apart from the comparisons of performance, the comparisons of search cost, the parameters of the searched architectures should be provided as well.\n\nThank you for your suggestion. Following your suggestion, we provide the comparisons of GNAS methods in terms of performance, search cost, and architecture parameters in the following table. The time is tested with one NVIDIA 3090 GPU.\n\n| Data | CS |  |  | Computers |  |  | Physics |  |  | Photo |  |  |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| Metric | ACC(%) | Time(s) | #Params(K) | ACC(%) | Time(s) | #Params(K) | ACC(%) | Time(s) | #Params(K) | ACC(%) | Time(s) | #Params(K) |\n| Random | 92.9+-0.3 | 1071 | 899 | 84.8+-0.4 | 3605 | 144 | 95.4+-0.1 | 2095 | 1096 | 91.1+-0.6 | 522 | 142 |\n| DARTS | 92.8+-0.3 | 34 | 915 | 79.7+-0.5 | 79 | 144 | 95.2+-0.1 | 75 | 1116 | 91.5+-0.6 | 13 | 126 |\n| GraphNAS | 91.6+-0.3 | 647 | 1011 | 69.0+-0.6 | 5295 | 372 | 94.5+-0.1 | 2268 | 1198 | 89.3+-0.7 | 435 | 238 |\n| GASSO | 93.1+-0.3 | 34 | 2632 | 84.9+-0.4 | 69 | 370 | 95.7+-0.1 | 75 | 3236 | 92.0+-0.3 | 13 | 361 |\n| Ours | 93.5+-0.2 | 49 | 1013 | 86.6+-0.4 | 201 | 259 | 95.7+-0.1 | 99 | 1250 | 93.3+-0.3 | 20 | 288 |\n\n As shown in the table above, the training time of our method is on par with the state-of-the-art one-shot NAS methods (e.g.,  DARTS, GASSO,), which is much more efficient than the multi-trial NAS methods (e.g., random and GraphNAS). The numbers of parameters of the searched architectures are comparable for different methods. While being competitive in efficiency, our method has significant performance improvements over the baselines."}]

Author rebuttal:

Meta-review: This work proposes a novel, disentanglement-based approach to graph neural architecture search. All reviewers assess that the work is a significant contribution, with very interesting insight---many of the reviewers strongly argued for acceptance. While some concerns were raised about the method's motivation and reproducibility, the Authors provided a convincing rebuttal that clears the residual doubt. I strongly support this work for acceptance at NeurIPS.
Decision: Accept (poster)