SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Synthetic Datasets. We construct a synthetic multi-task node classification dataset named Multi-GNN Label (MGL). We generate several MGL graphs using different graph generation methods, e.g., the Erdős-Rényi model (MGL-ER), the Watt-Strogatz small-world model (MGL-WS), and the Barabási-Albert preferential attachment (MGL-BA). The node features are sampled from a Gaussian distribution. To generate labels with diverse characteristics, we adopt different GNNs with random parameters to generate multiple hidden node representations. For each hidden representation, we use the KNN algorithm to group nodes into 5 classes. As such, each GNN can generate node labels corresponding to a node classification task. Specifically, we use two random MLP, GCN, GAT, and SAGE to generate 8 labels.
Table 1: The test accuracy of all the methods on the synthetic dataset MGL and the test ROC-AUC of all the methods on the real-world datasets OGBG. We run all experiments 10 times with different random seeds and report the average results with standard deviations. The best results are in bold.
Dataset	MGL-ER	MGL-WS	MGL-BA	Tox21	ToxCast	Sider
GCN	51.50±0.37	50.17±0.14	50.41±0.21	76.41±0.60	64.91±0.53	62.02±1.71
GAT	50.95±0.17	49.91±0.19	47.77±0.52	75.62±0.89	63.52±0.57	58.57±1.98
GIN	52.40±0.41	47.67±0.28	50.48±0.45	77.20±0.85	64.58±0.49	58.48±1.63
SAGE	62.58±0.13	64.77±0.14	66.15±0.25	75.87±0.49	64.08±0.65	60.12±0.97
k-GNN	56.49±0.29	58.80±0.14	62.19±0.47	76.87±0.51	63.86±0.59	60.07±1.14
ARMA	63.90±0.29	64.20±0.28	65.05±0.37	75.93±0.87	64.36±0.83	60.79±1.24
MLP	50.08±0.13	40.93±0.13	43.63±0.20	74.47±0.75	62.67±0.61	60.70±1.29
Random	57.40±0.22	60.80±0.19	59.15±0.17	76.20±0.26	64.89±0.60	58.87±1.38
DARTS	64.94±0.20	64.16±0.29	62.39±0.23	76.96±0.57	65.23±0.60	60.64±1.37
GNAS	49.76±2.95	61.17±0.00	64.28±0.95	74.97±0.41	61.85±1.07	57.11±1.31
PAS	-	-	-	75.45±0.47	63.85±0.35	59.31±1.48
GRACES	-	-	-	74.82±0.85	65.77±0.53	61.85±2.56
MTGC3	66.33±0.34	67.39±0.42	68.36±0.22	77.99±0.42	66.36±0.26	62.08±1.76
- Real-world Datasets. We choose three widely-used multi-task graph classification datasets included OGB [13]: OGBG-Tox21 [14], OGBG-ToxCast [35], and OGBG-Sider [18]. These datasets contain a set of properties of toxicological assays or adverse drug reactions of drug molecules, where molecules can be represented as graphs and each task corresponds to predicting a property of drug molecules.
Baselines. We compare our model with 12 baselines from the following three different categories.
- Manually Design GNNs: we include the GNNs in our search space (refer to Section 2.2) as our baselines, i.e., GCN, GAT, GIN, SAGE, k-GNN, and ARMA. We also include MLP as a baseline.
- Graph Neural Architecture Search: We consider three representative GraphNAS baselines: GNAS [7], an reinforcement learning based method; PAS [42] and GRACES [32], two recent differentiable GraphNAS methods. Note that since GNAS and GRACES are specifically designed for graph classification tasks. We also consider two classic NAS baselines, random search and DARTS [24] combined with our search space.
Experimental Details. We set the number of layers as 3 for synthetic datasets, and 5 for real-world datasets. For all datasets except ToxCast, we use the task-separate head. For ToxCast, we use the cross-mixed head with 16 chunks.
Number of Learnable Parameters. Denote  $|V|, |E|$  as the number of nodes and edges in the graph, respectively, and  $d$  as the dimensionality of hidden representations. We denote  $d_S$  as the dimensionality of the structure generation part, i.e.,  $\mathbf{S}$  is a matrix with the shape  $d_S \times d$ . The number of learnable parameters of a typical message-passing GNN is  $O(d^2)$ . In our framework,  $w, \alpha, \theta$ , and  $\mathbf{S}$  has  $O(|\mathcal{O}|d^2)$ ,  $O(N|\mathcal{O}|)$ ,  $O(d^2)$ , and  $O(Nd_s d)$  parameters, respectively. The total number of learnable parameters is  $O(N(|\mathcal{O}| + d_S d) + |\mathcal{O}|d^2)$ . To guarantee fair comparison, we use small  $d_S$  and  $d$  in the experiments so that the different models are comparable in terms of the number of learnable parameters.
We summarize the experimental results in Table 1. For the results on synthetic datasets, our model outperforms all baselines in all three settings. Specifically, we find that most GNNs perform poorly, indicating that they cannot well handle multiple tasks. The existing GraphNAS methods fail to outperform some manually designed GNNs on these graphs. In contrast,  $\mathrm{MTGC}^3$  shows much better results by searching multiple architectures and capturing collaborative patterns between tasks.
As for the results on the three real-world multi-label graph classification benchmarks, we find that some NAS methods, e.g., DARTS and GRACES, achieve slightly better results than manually designed GNNs in some cases, demonstrating the importance of automating architectures. Nevertheless,
Figure 3: Visualizations of architecture parameters of different chunks in MGL-ER.
(a) The 1st layer
(b) The last layer
Figure 4: Visualizations of the absolute values of  $p_{ij}$  of different layers in MGL-ER.
Figure 5: Visualizations of architecture parameters of different chunks at the 3rd layer in ToxCast.
Variant	MGL-WS	Tox21	ToxCast
NoStru	67.17±0.60	77.42±1.00	66.30±0.30
Separate	65.35±0.86	76.20±0.98	66.12±0.80
FullCollab	67.21±0.55	77.83±0.71	66.01±0.50
MLPHead	64.47±1.16	77.62±0.57	64.61±0.53
NoCL	67.15±0.52	77.68±0.48	66.00±1.07
MTGC³	67.39±0.42	77.99±0.42	66.36±0.26
Table 2: The performance of different variants of  $\mathrm{MTGC}^3$  on different datasets.
these methods still fail to design suitable architectures for multiple tasks. Our proposed MTGC $^3$  model again outperforms all the baselines on the three datasets, demonstrating that our model is able to search for proper architectures in real-world multi-task scenarios.
4.3 Visualization and Analysis
To gain deeper insights of our method, we further conduct some analyses. Specifically, we generate four labels by a random MLP, GCN, GAT, and SAGE respectively in MGL, and denote the labels as 1, 3, 5, 7. Then, we add Gaussian noises to the node representations and generate four additional labels denoted as 2, 4, 6, 8. As a result, we generate four pairs of highly correlated tasks, i.e., label 1 and 2 are related, etc. Then, we visualize the architecture parameters, i.e.,  $\alpha_{ik}$ , at the first layer. The results on MGL-ER are shown in Figure 3. The results show that  $\mathrm{MTGC}^3$  can search for proper architectures for different tasks. For the tasks directly generated by random GNNs (see chunks 1, 3, 5, 7),  $\mathrm{MTGC}^3$  has the highest  $\alpha$  on the corresponding operation.
In addition, we visualize the absolute values of  $p_{ij}$  in the soft task-collaborative module in Figure 4. Large absolute values indicate that knowledge can be transferred between tasks. For the last layer, we can observe that more knowledge can be transferred between our constructed task pairs, e.g., chunk 1 and 2, chunk 5 and 6. The results demonstrate that the soft task-collaborative module can accurately capture the transferability between tasks. Meanwhile, for the 1st layer, the absolute values exhibit more complex patterns, indicating that some complex knowledge sharing between chunks may exist. It is also worth noting that the absolute values of  $p_{ij}$  at the 1st layer are generally larger than that at the last layer. This phenomenon indicates that there is more shared information at the shallow layers than at the deep layers, which is consistent with the branched sharing architecture in multi-task learning [6].
The above analyses are for the task-separated head case. We further analyze the cross-mixed head case. We visualize the architecture parameters  $\alpha_{ik}$  on ToxCast dataset in Figure 5. We can observe that different chunks choose different operations, indicating that different chunks still need to find the proper operations to fit the tasks assigned through the cross-mixed head.
In this section, we evaluate the effectiveness of each module of our framework by conducting ablation studies. We compare the following variants of our model:
- MTGC³-NoStru: we use the original graph structure for all chunks.
- MTGC<sup>3</sup>-Separate: we disable the collaboration between chunks, i.e.,  $\forall i \neq j, p_{ij} = 0$ , so that different tasks search architectures separately.
- MTGC<sup>3</sup>-FullCollab: we let  $p_{ij} = 1$ , i.e., full collaboration, thus the independence of each chunk is not guaranteed.
- MTGC<sup>3</sup>MLPHead: we use an MLP as the task head.
- MTGC<sup>3</sup>-NoCL: we remove the task-wise curriculum training and use standard gradient descents.
We report the results of these variants on MGL-ER, Tox21, and ToxCast in Table 2. We have the following observations. Overall, our proposed full MTGC $^3$  model outperforms all the variants under all three settings, demonstrating that each component of our method is indispensable to achieve satisfactory performance in multi-task graph learning. The performance margin above MTGC $^3$ -NoStru illustrates the validity of learning diverse graph structures for different chunks. Either separating all chunks by setting  $p_{ij} = 0$  or allowing maximum collaboration by setting  $p_{ij} = 1$  causes performance degradation, demonstrating that our soft task-collaborative module is essential to capture the transferability between tasks. In addition, if we use an MLP as the task head, the performance will drop severely, indicating the importance of the task-separated head and cross-mixed head in multi-task learning. Moreover, the task-wise curriculum training also has a consistently positive impact on the model performance.
5.1 Graph neural network and multi-task learning
Message-passing GNNs [17, 39, 47, 11, 23, 22, 21, 20, 50, 51] have been proposed as an effective framework for graph machine learning following the neighborhood aggregation scheme. At each layer, nodes learn representations by aggregating their neighbors' representations. Then, the representation of the whole graph is learned by pooling all node representations [17, 47]. Graph structure learning technique [15, 44] is also applied in GNNs, which learns a better graph structure for message passing during training procedure. Graph structure learning is beneficial for enhancing the robustness of graph embedding, especially in noisy scenarios.
Multi-task learning [3, 16, 26] aims to jointly learn a set of tasks with shared parameters. Recently, some works learn to exploit similarities between tasks in GNNs to benefit multi-task graph learning. New et al. [28] explore the influence of the Hessians of each task's loss function in multi-task graph learning. SGNN-EBM [25] learns task relationships in the latent space by knowledge extracted from external datasets. MetaLink [2] uses graphs to model the relationship between data and tasks in the relational multi-task setting. Nevertheless, these works do not explore automatic architecture design in multi-task graph learning.
5.2 Neural architecture search
Recent years have witnessed a surge of research interest in NAS methods, which aim at designing neural architectures automatically for given tasks. Since the architecture search space is discrete, reinforcement learning (RL) [56, 29] and evolution algorithm (EA) [45?] are often used in NAS methods. Besides, another strategy is transferring the discrete architecture search space into a differentiable space, e.g., DARTS [24] and SNAS [46] construct a supernet where all candidate operations are mixed to update the architecture as well as the weights simultaneously through the classical gradient descent method.
GraphNAS is gaining increasing attention from the research community [19, 7, 10, 31, 52, 33, 54, 52], including RL [7, 30], EA [49, 9], and differentiable NAS [19, 31, 32] algorithms. GraphNAS works also consider searching specific modules in graph representation learning, such as message passing layers [1], attention layers [10], and pooling operations [42]. However, existing GraphNAS methods only consider a single task and neglect the multi-task scenarios.
Outside of graphs, some multi-task NAS searches for a unified architecture for multiple tasks. However, most existing works only focus on searching the interaction operations between tasks but ignore designing different task backbones [36, 8, 37, 38, 43, 34, 48]. More importantly, all the above works focus on computer vision task but do not consider graph tasks.
In this paper, we propose a novel  $\mathrm{MTGC}^3$  method to tackle GraphNAS problem in multi-task scenarios. Our method searches for multiple architectures for different tasks while considering task collaboration by designing the structurally diverse supernet, the soft task-collaborative module, and the task-wise curriculum training. Extensive experiments on both synthetic and real-world datasets demonstrate that  $\mathrm{MTGC}^3$  can achieve state-of-the-art performance for multi-task graph learning. One possible limitation of this work is that our method only considers static and homogenous graphs and does not account for more complex graphs, such as dynamic or heterogenous graphs. Future research should explore methods that can effectively handle these complex graph types, in order to broaden the usage of the method in real-world applications.
This work was supported by the National Key Research and Development Program of China No. 2020AAA0106300, National Natural Science Foundation of China (No. 62222209, 62250008, 62102222, 62206149), Beijing National Research Center for Information Science and Technology under Grant No. BNR2023RC01003, BNR2023TD03006, Beijing Key Lab of Networked Multimedia, China National Postdoctoral Program for Innovative Talents No. BX20220185, and China Postdoctoral Science Foundation No. 2022M711813.
[1] Shaofei Cai, Liang Li, Jincan Deng, Beichen Zhang, Zheng-Jun Zha, Li Su, and Qingming Huang. Rethinking graph neural architecture search from message-passing. In IEEE Conference on Computer Vision and Pattern Recognition, pages 6657-6666, 2021.
[2] K Cao, J You, and J Leskovec. Relational multi-task learning: Modeling relations between data and tasks. In International Conference on Representation Learning (ICLR), 2022.
[3] Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
[4] Hong Chen, Yudong Chen, Xin Wang, Ruobing Xie, Rui Wang, Feng Xia, and Wenwu Zhu. Curriculum disentangled recommendation with noisy multi-feedback. Advances in Neural Information Processing Systems, 34, 2021.
[5] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pages 794-803. PMLR, 2018.
[6] Michael Crawshaw. Multi-task learning with deep neural networks: A survey. arXiv preprint arXiv:2009.09796, 2020.
[7] Yang Gao, Hong Yang, Peng Zhang, Chuan Zhou, and Yue Hu. Graph neural architecture search. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, 2020.
[8] Yuan Gao, Haoping Bai, Zequn Jie, Jiayi Ma, Kui Jia, and Wei Liu. Mtl-nas: Task-agnostic neural architecture search towards general-purpose multi-task learning. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 11543–11552, 2020.
[9] Chaoyu Guan, Xin Wang, Hong Chen, Ziwei Zhang, and Wenwu Zhu. Large-scale graph neural architecture search. In International Conference on Machine Learning, pages 7968-7981. PMLR, 2022.
[10] Chaoyu Guan, Xin Wang, and Wenwu Zhu. Autoattend: Automated attention representation search. In Proceedings of the 38th International Conference on Machine Learning, 2021.
[11] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in neural information processing systems, 2017.
[45] Lingxi Xie and Alan Yuille. Genetic cnn. In Proceedings of the IEEE international conference on computer vision, 2017.
[46] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. In Proceedings of the 7th International Conference on Learning Representations, 2019.
[47] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In 7th International Conference on Learning Representations, 2019.
[48] Lijun Zhang, Xiao Liu, and Hui Guan. Automtl: A programming framework for automating efficient multi-task learning. Advances in Neural Information Processing Systems, 35:34216-34228, 2022.
[49] Wentao Zhang, Zheyu Lin, Yu Shen, Yang Li, Zhi Yang, and Bin Cui. Deep and flexible graph neural architecture search. In International Conference on Machine Learning, pages 26362-26374. PMLR, 2022.
[50] Zeyang Zhang, Xingwang Li, Fei Teng, Ning Lin, Xueling Zhu, Xin Wang, and Wenwu Zhu. Out-of-distribution generalized dynamic graph neural network for human albumin prediction. In IEEE International Conference on Medical Artificial Intelligence, 2023.
[51] Zeyang Zhang, Xin Wang, Ziwei Zhang, Haoyang Li, Zhou Qin, and Wenwu Zhu. Dynamic graph neural networks under spatio-temporal distribution shift. In Advances in Neural Information Processing Systems, 2022.
[52] Zeyang Zhang, Ziwei Zhang, Xin Wang, Yijian Qin, Zhou Qin, and Wenwu Zhu. Dynamic heterogeneous graph attention neural architecture search. In Thirty-Seventh AAAI Conference on Artificial Intelligence, 2023.
[53] Zeyang Zhang, Ziwei Zhang, Xin Wang, and Wenwu Zhu. Learning to solve travelling salesman problem with hardness-adaptive curriculum. Thirty-Fifth AAAI Conference on Artificial Intelligence, 2022.
[54] Zizhao Zhang, Xin Wang, Chaoyu Guan, Ziwei Zhang, Haoyang Li, and Wenwu Zhu. Autogt: Automated graph transformer architecture search. In The Eleventh International Conference on Learning Representations, 2022.
[55] Yuwei Zhou, Xin Wang, Hong Chen, Xuguang Duan, Chaoyu Guan, and Wenwu Zhu. Curriculum-nas: Curriculum weight-sharing neural architecture search. In Proceedings of the 30th ACM International Conference on Multimedia, pages 6792-6801, 2022.
[56] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In Proceedings of the 5th International Conference on Learning Representations, 2017.