SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Summary of dataset statistics. Unsup./Semi. denotes Unsupervised and Semi-supervised settings. Graph/Node denotes graph and node classification tasks. ACC/AUC denotes Accuracy and ROC-AUC evaluation metrics.
Datasets	MUTAG	IMDB-B	PROTEINS	DD	Computers	Photos	CS	Physics	OGBG-Molhiv	OGBN-Arxiv	Wechat-Video
Setting	Unsup.	Unsup.	Unsup.	Unsup.	Unsup.	Unsup.	Unsup.	Unsup.	Semi.	Semi.	Semi.
Task	Graph	Graph	Graph	Graph	Node	Node	Node	Node	Graph	Node	Node
Evaluation Metric	ACC	ACC	ACC	ACC	ACC	ACC	ACC	ACC	AUC	ACC	AUC
# Graphs	188	1,000	1,113	1,178	1	1	1	1	41,127	1	1
# Avg. Nodes	17	19	39	284	13,752	7,650	18,333	34,493	26	169,343	60,774
# Avg. Edges	56	211	183	1,714	505,474	245,812	182,121	530,417	28	2,484,941	3,182,156
# Features	7	1	3	89	767	745	6,805	8,415	300	128	512
# Classes	2	2	2	2	10	8	15	5	2	40	13
Datasets For unsupervised settings, we conduct experiments on four graph-level classification datasets including PROTEINS [38], DD [39], MUTAG [40], IMDB-B [41] from TUDataset [42] and four node-level classification datasets Coauthor CS, Coauthor Physics from the Microsoft Academic Graph [43], Amazon Computers, Amazon Photos from the Amazon Co-purchase Graph [44]. For semi-supervised settings, we adopt three real-world datasets, OGBG-Molhiv, OGBN-Arxiv [45]
Table 2: The results (accuracy%) of all the methods on the real-world datasets in unsupervised settings. Numbers after the  $\pm$  signs represent standard deviations. The best results are in bold and the second-best results are underlined. As the search space of GASSO and PAS do not suit the graph-level and node-level tasks respectively, we omit their results.
Data Method	Graph Classification	Node Classification
PROTEINS	DD	MUTAG	IMDB-B	CS	Computers	Physics	Photo
GCN	72.8±0.7	77.0±0.9	78.6±1.6	63.5±0.8	93.0±0.3	86.0±0.4	95.7±0.1	90.8±0.6
GAT	72.3±0.9	77.5±0.7	78.0±0.8	54.4±1.7	93.4±0.3	85.8±0.3	95.6±0.1	91.4±0.6
GIN	72.6±0.4	77.3±0.7	86.3±1.7	70.7±0.5	93.1±0.3	76.7±0.5	95.3±0.1	91.1±0.7
GraphSage	72.9±0.7	77.1±0.4	78.3±1.6	53.0±2.1	93.2±0.3	78.4±0.4	95.4±0.1	89.2±0.7
GraphConv	72.1±0.6	77.3±0.6	87.2±1.4	71.1±0.6	93.1±0.3	74.7±0.7	95.3±0.1	91.5±0.5
MLP	70.5±0.4	76.1±0.7	74.8±1.1	50.3±0.6	91.5±0.4	56.6±0.3	94.6±0.1	87.4±0.8
Random	74.5±0.9	74.8±1.3	82.1±2.8	69.0±2.1	92.9±0.3	84.8±0.4	95.4±0.1	91.1±0.6
DARTS	73.6±0.9	75.7±0.9	86.5±2.3	70.4±0.6	92.8±0.3	79.7±0.5	95.2±0.1	91.5±0.6
GraphNAS	73.6±0.7	75.2±0.9	77.5±0.7	62.7±1.3	91.6±0.3	69.0±0.6	94.5±0.1	89.3±0.7
PAS	74.6±0.3	76.5±0.9	84.0±1.6	64.6±13.8	-	-	-	-
GASSO	-	-	-	-	93.1±0.3	84.9±0.4	95.7±0.1	92.0±0.3
DSGAS	76.0±0.2	78.4±0.7	88.7±0.7	72.0±0.5	93.5±0.2	86.6±0.4	95.7±0.1	93.3±0.3
and Wechat-Video<sup>5</sup>. The datasets cover various graph-related fields including small molecules, bioinformatics, social networks, e-commerce networks, and academic coauthorship networks. The statistics are summarized in Table 1.
Super-network We briefly introduce the super-network construction as follows. The super-network consists of two parts, the operation pool and the directed acyclic graph (DAG). The operation pool includes several node aggregation operations (e.g., GCN, GIN, GAT, etc.), graph pooling operations (e.g., SortPool, AttentionPool, etc.), and layer merging operations (e.g., MaxMerge, ConcatMerge, etc.). The DAG determines how the operations are connected to calculate the graph representations for the subsequent classification tasks.
More details of the experiments are provided in the Appendix, including additional experiments and analyses, experimental setups, configurations, and implementation details.
Unsupervised Settings From the experimental results summarized in Table 2, we have the following observations: 1) GNNs perform differently on various datasets. The best GNN baselines for these datasets are GraphSage, GAT, GraphConv, GraphConv, GAT, GCN, GCN, GraphConv successively, and the performance varies greatly across datasets and GNNs. It verifies that no GNN architecture is dominant for all datasets, which is consistent with the literature [32] and shows the demand for automated GNN architecture designing based on the data characteristics to obtain the optimal representations. 2) Most GNAS baselines fail in the unsupervised setting. Since the existing GNAS baselines highly rely on supervised signals to search the architectures, they inherently do not suit the unsupervised settings. As an ad hoc remedy for the existing GNAS baselines in unsupervised settings, we substitute the supervised metrics with the self-supervised ones during the searching process as simple extensions. However, these GNAS baselines, contrary to supervised settings, do not guarantee better performance than manually designed GNNs for all datasets. For example, all GNAS baselines even perform worse than manually designed GNNs on DD dataset and do not have significant improvements on most datasets. The reasons behind might be that simply using graph self-supervised metrics does not consider the entanglement of architectures and factors, leading to inaccurate estimation of the architectures' capabilities. 3) Our method has significant improvements over the baselines on most datasets. Compared with manually designed GNN baselines, DSGAS has a performance improvement of  $3.1\%$  on PROTEINS and over  $1\%$  on most datasets. We contribute this to its ability of automatically tailoring GNNs for various datasets, showing its effectiveness of automatic GNN designing. DSGAS also significantly surpasses GNAS baselines, showing its superiority in graph neural architecture search in unsupervised settings, which benefits from the design of discovering architectures that can capture various graph factors in a self-supervised fashion.
<sup>5</sup>https://algo.weixin.qq.com/
Figure 2: The performance of GNAS methods on real-world datasets under semi-supervised settings, where DSGAS-P denotes DSGAS without pretraining. The results are averaged by five random runs. (Best viewed in color)
Figure 3: (a) Comparisons of different ablated variants of DSGAS on real-world datasets under unsupervised settings. The horizontal dashed line refers to the results of the best-performed GNAS baseline. (b) Comparisons of different architecture augmentations of DSGAS on real-world datasets under unsupervised settings, where 'Alpha', 'Weight' and 'Embed' denote the augmentations from perspective of operation choices, weight and embeddings. 'Compose' denotes uniformly choosing one of the three augmentations. The horizontal dashed line refers to the results of the best-performed GNAS baseline. (Best viewed in color)
Semi-supervised Settings From Figure 2, we have the following observations: Compared with the baselines, DSGAS significantly alleviates the performance drop when the number of available supervised labels is fewer, which verifies that our method fully exploits latent factors inside graph data and boost the supervised architecture search stage by warming up the weights and architecture parameters of the super-network. Its significant improvement over the ablated version DSGAS-P also verifies the effectiveness of pretraining the super-network by the proposed modules of self-supervised training with joint architecture-graph disentanglement and contrastive search with architecture augmentations. For example, our model with the pretraining stage has an absolute improvement of  $5\%$  on Wechat-Video dataset with  $1\%$  labels compared with the ablated version without the pretraining stage, which shows the effects of pretraining the super-networks on alleviating the label scarcity issues. In comparison, the performance of other baselines decays significantly more than our method, showing that current GNAS can not tackle scenarios with scarce labels. For example, on OGBG-Molhiv, PAS is the best baseline while the worst with  $10\%$  and  $1\%$  labels respectively, which may due to the inaccurate performance estimation with scarce labels. The phenomenon further strengthens the necessity of designing effective unsupervised GNAS methods.
4.2 Additional Experiments
Ablation Studies We evaluate the effectiveness of each module of our framework by comparing the following ablated versions of our method: DSGAS-CONT removes our proposed contrastive search module and search architectures with the vanilla self-supervised loss. DSGAS-FA further replaces our proposed factor-aware training module with the vanilla self-supervised training. DSGAS-DISEN further replaces our proposed disentangled super-network with the vanilla super-network.
(a) Computers
(b) DD
(c) Photo
(d) PROTEINS
Figure 4: Visualizations of the search architectures on different datasets. Nodes denote GNN operations except that 'INPUT' denotes the input graphs with structures and features. Directed edges denote calculation flows, where different colors denote the architecture operation choices under different factors. (Best viewed in color)
We compare the performance of the ablated versions and the best GNAS baselines on the real-world datasets under unsupervised settings. From Figure 3a, we have the following observations. First, our proposed DSGAS outperforms all the variants on all datasets, demonstrating the effectiveness of each component of our proposed framework in searching graph architectures under unsupervised settings. Second, DSGAS-CONT drops drastically in performance on all datasets compared to the full version, showing the superiority of our proposed disentangled contrastive architecture search module in searching architectures. Third, the performance also decays for DSGAS-FA and DSGAS-DISEN on most datasets, showing the necessity of capturing various latent factors with different architectures.
Architecture Visualizations We visualize the architectures searched on different unsupervised datasets in Figure 4. It shows that the searched architectures of different factors adopt quite different GNN operations while sometimes sharing the same operations, which leads to an overall architecture with complex internal connections between operations. This phenomenon implies that DSGAS can optimize the architecture operation choices as well as the operation connections for different factors to have a competitive performance on various graph datasets, which also verifies the superiority of DSGAS in automated architecture search to save human endeavors for architecture designs.
Effects of Architecture Augmentations In Figure 3b, we show the results of different architecture augmentation methods on different datasets compared with the best GNAS baseline. We find that though the best augmentations differ among datasets, they have similar performance improvements in most cases, which verifies the design of contrastive search with architecture augmentations.
Graph Neural Architecture Search Instead of manually designing more sophisticated models for various scenarios, neural architecture search, aiming to automatically discover the optimal architectures for given tasks, emerges as a hot topic recently in computer vision [46, 23], natural language processing [47], graph representation learning [1, 48, 49], etc. In the field of graph representation learning with various applications [50, 51, 52, 53, 54], graph neural architecture search (GNAS) methods, as the most related to our works, can be roughly classified into reinforcement-learning-based methods [3, 55], evolutionary-based methods [56, 2, 57, 58], and differentiable methods [59, 60, 61, 62, 37, 63, 4, 64, 65]. However, supervised labels are indispensable for the existing GNAS methods to conduct neural architecture search, which limits their applications in widely-existed scenarios where labels are scarce or not available.
Unsupervised Neural Architecture Search In unsupervised settings, some neural architecture search methods replace supervised labels with self-supervised loss during searching [66, 67, 68, 69, 70, 71]. Another classic of related methods design special metrics, whose calculation does not depend on labels, as proxies for model performance estimation [72, 73]. For example, UnNAS [74] adopts pretext tasks like image rotation, coloring images, solving puzzles, etc. However, these methods are specially designed for computer vision, and can not be directly adopted to graph data. Some GNAS
works exploit self-supervised loss as auxiliaries to augment the supervised search process [63], but the supervised labels are still mandatory for its search. To the best of our knowledge, this is the first work on unsupervised graph neural architecture search.
Graph Self-supervised Learning Graph self-supervised learning [75, 76] is devoted to obtaining graph representations by extracting informative knowledge with well-designed pretext tasks without labels, which can be roughly classified into contrastive [77, 78, 79, 80, 81, 82, 83, 84, 85, 86] and generative [87, 88, 76, 89], and predictive methods [90, 91, 92]. The existing graph self-supervised learning methods usually focus on designing better pretext tasks with a fixed encoder such as GCN [20], GAT [21] and GIN [22]. Another class of related methods attempt to automate the choices of pretext tasks [93, 94, 95, 96, 97]. We mainly consider graph neural architecture in unsupervised settings, while other pretext tasks are orthogonal to our framework and can be incorporated.
Disentangled Representation Learning The primary objective of disentangled representation learning is to delineate and interpret the various latent factors which influence the data we encounter in an observable context, rendering each of these factors as unique vector representations [98, 99]. It has emerged to be a useful tool in various domains, including those of computer vision [100, 101, 102, 103, 104, 105], and recommendation systems [106, 107, 108, 109, 110, 111, 112, 113], graph representation learning [9, 114, 10, 14, 11, 115, 116, 117]. As the most related, GRACES [63] characterize the graph latent factors inside data by designing a self-supervised disentangled graph encoder, and conduct graph neural architecture search for each graph to handle graph distribution shifts, while the training and searching process still follows the supervised paradigm. In contrast, we focus on automating the GNN designs with disentangled self-supervision in this paper.
In this paper, we propose a novel Disentangled Self-Supervised Graph Neural Architecture Search (DSGAS) framework to automate the GNN designs with disentangled self-supervision, which includes disentangled graph architecture super-network, self-supervised training with joint architecture-graph disentanglement and contrastive search with architecture augmentations. Extensive experiments demonstrate that our proposed method can discover architectures with capabilities of capturing various graph latent factors and significantly outperform the state-of-the-art GNAS baselines. Detailed ablation studies and analyses show the effectiveness of our method design. One limitation is that in this paper we mainly focus on homogeneous graphs, and we leave extending our method to heterogeneous graphs for further explorations.
[1] Ziwei Zhang, Xin Wang, and Wenwu Zhu. Automated machine learning on graphs: A survey. arXiv preprint arXiv:2103.00742, 2021.
[2] Yaoman Li and Irwin King. Autograph: Automated graph neural network. In International Conference on Neural Information Processing, pages 189-201, 2020.
[3] Yang Gao, Hong Yang, Peng Zhang, Chuan Zhou, and Yue Hu. Graph neural architecture search. In IJCAI, volume 20, pages 1403-1409, 2020.
[4] Lanning Wei, Huan Zhao, Quanming Yao, and Zhiqiang He. Pooling architecture search for graph classification. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2091-2100, 2021.
[104] Luan Tran, Xi Yin, and Xiaoming Liu. Disentangled representation learning gan for pose-invariant face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1415-1424, 2017.
[105] Xin Wang, Zihao Wu, Hong Chen, Xiaohan Lan, and Wenwu Zhu. Mixup-augmented temporally debiased video grounding with content-location disentanglement. 2023.
[106] Xin Wang, Hong Chen, Yuwei Zhou, Jianxin Ma, and Wenwu Zhu. Disentangled representation learning for recommendation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[107] Hong Chen, Yudong Chen, Xin Wang, Ruobing Xie, Rui Wang, Feng Xia, and Wenwu Zhu. Curriculum disentangled recommendation with noisy multi-feedback. Advances in Neural Information Processing Systems, 34:26924-26936, 2021.
[108] Xin Wang, Hong Chen, and Wenwu Zhu. Multimodal disentangled representation for recommendation. In 2021 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6, 2021.
[109] Jianxin Ma, Chang Zhou, Hongxia Yang, Peng Cui, Xin Wang, and Wenwu Zhu. Disentangled self-supervision in sequential recommenders. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 483-491, 2020.
[110] Jianxin Ma, Chang Zhou, Peng Cui, Hongxia Yang, and Wenwu Zhu. Learning disentangled representations for recommendation. Advances in neural information processing systems, 32, 2019.
[111] Haoyang Li, Xin Wang, Ziwei Zhang, Jianxin Ma, Peng Cui, and Wenwu Zhu. Intention-aware sequential recommendation with structured intent transition. IEEE Transactions on Knowledge and Data Engineering, 2021.
[112] Xin Wang, Zirui Pan, Yuwei Zhou, Hong Chen, Chendi Ge, and Wenwu Zhu. Curriculum co-disentangled representation learning across multiple environments for social recommendation. 2023.
[113] Yipeng Zhang, Xin Wang, Hong Chen, and Wenwu Zhu. Adaptive disentangled transformer for sequential recommendation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3434-3445, 2023.
[114] Yanbei Liu, Xiao Wang, Shu Wu, and Zhitao Xiao. Independence promoted graph disentangled networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4916-4923, 2020.
[115] Zeyang Zhang, Xingwang Li, Fei Teng, Ning Lin, Xueling Zhu, Xin Wang, and Wenwu Zhu. Out-of-distribution generalized dynamic graph neural network for human albumin prediction. In IEEE International Conference on Medical Artificial Intelligence, 2023.
[116] Zeyang Zhang, Xin Wang, Ziwei Zhang, Zhou Qin, Weigao Wen, Hui Xue, Haoyang Li, and Wenwu Zhu. Spectral invariant learning for dynamic graphs under distribution shifts. In Advances in Neural Information Processing Systems, 2023.
[117] Zeyang Zhang, Xin Wang, Ziwei Zhang, Haoyang Li, Zhou Qin, and Wenwu Zhu. Dynamic graph neural networks under spatio-temporal distribution shift. In Advances in Neural Information Processing Systems, 2022.