Multi-task Graph Neural Architecture Search with Task-aware Collaboration and Curriculum
Authors: Yijian Qin
Xin Wang
Ziwei Zhang
Hong Chen
Wenwu Zhu
Venue: NeurIPS 2023 poster (2023)
Keywords: graph neural network
neural architecture search
multi-task learning

Abstract: Graph neural architecture search (GraphNAS) has shown great potential for automatically designing graph neural architectures for graph related tasks. However, multi-task GraphNAS capable of handling multiple tasks simultaneously has been largely unexplored in literature, posing great challenges to capture the complex relations and influences among different tasks. To tackle this problem, we propose a novel multi-task graph neural architecture search with task-aware collaboration and curriculum (MTGC3), which is able to simultaneously discover optimal architectures for different tasks and learn the collaborative relationships among different tasks in a joint manner. Specifically, we design the layer-wise disentangled supernet capable of managing multiple architectures in a unified framework, which combines with our proposed soft task-collaborative module to learn the transferability relationships between tasks. We further develop the task-wise curriculum training strategy to improve the architecture search procedure via reweighing the influence of different tasks based on task difficulties. Extensive experiments show that our proposed MTGC3 model achieves state-of-the-art performance against several baselines in multi-task scenarios, demonstrating its ability to discover effective architectures and capture the collaborative relationships for multiple tasks.

Multi-task Graph Neural Architecture Search with Task-aware Collaboration and Curriculum
Yijian Qin Xin Wang Ziwei Zhang', Hong Chen', Wenwu Zhu 1.2- "Department of Computer Science and Technology, Tsinghua University, B qiny ginyj19@mails wttaaaiissssssssiieeee tsinghua. edu. cn, Sxin_wang,,zee wozhanww (wwzhang)@tsingeee @tsinghua. edu. cn h- b-chen200mails. tsinghua.edis edu.. wwzhu@tsinghua. edu.
Abstract
Graph neural architecture search (GraphNAS) has shown great potential for auto- matically designing graph neural architectures for graph related tasks. However, multi-task GraphNAS, capable of handling multiple tasks simultaneously and cap- turing the complex relationships and dependencies between them, has been largely unexplored in literature. To tackle this problem, we propose novel multi-task graph neural arghitecture search with task-aware collaboration and curriculum (MTGC*), which is able to simultancously discover optimal architectures for dif- ferent tasks and learn the collaborative relationships among different tasks in a joint manner. Specifically, we design the structurally diverse supernet to manage multiple architectures and graph structures a unified framework, which combines with our proposed soft lask-collaoorative module to learn the transferabiliity rela- tionships between tasks. To further improve the architecture search procedure, we develop the task-wise curriculum training strategy that reweighs the influence of different tasks based on their relative difficulties. Extensive experiments show that our proposed MTGC model achieves state-of-the-art performance against several baselines in multi-task scenarios, demonstrating its ability to discover effective architectures and capture collaborative relationships for multiple tasks.
Introduction
Graph-structured data has attracted lots attentions in recent years for its flexible representation ability in various domains. Graph neural networks (GNNs) such GCN [17], GAT [39] and GIN 47
have been proposed and achieved great success in many graph applications. To save human efforts on designing GNN architectures for different tasks, graph neural architecture search (GraphNAS) 19
42] has been proposed to search for optimal GNN architectures. These automatically designed GNN architectures have achieved competitive better performances compared with manually designed GNNs for single task.
On the other hand, multi-task scenarios ubiquitous for graph data. For example, in drug discovery, predicting multiple properties of molecules can be regarded as multiple tasks. Properly capturing the relationship between these tasks allows for better learning of them. Nevertheless, there has been work reported on automatically searching GNN architectures for multi-task scenarios in the literature.
In this paper, we study the important yet unexplored problem of multi-task GraphNAS for the first time. Given that different GNNs employ different architectures for adaption to various graph tasks, we tackle this problem through jointly customizing architecture for each individual task and considering the task collaborations to share useful information among different tasks. However, discovering
Corresponding Authors. BNRist is the abbreviation of Beijing National Research Center for Information Science and Technology.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

the optimal GNN architectures for multiple tasks poses several challenges: (1) It is non-trivial to manage multiple architectures within a supernet, which is crucial to guarantee the effectiveness of the neural architecture search; (2) It is challenging to capture the complex relationships among different tasks for information sharing during the searching process; (3) remains a challenge to balance the influence of different tasks on architecture optimization since different tasks may show diverse difficulties and optimization patterns.
To address these challenges, we propose the multi-task graph neural arghitecture search with task-
aware collaboration and curriculum (MTGC3 in this paper. Our proposed MTGC3 model is able to discover multiple optimal architectures within a unified supernet and simultaneously optimizes them through learning the collaborative relationships of different tasks in joint manner. Specifically, design the structurally diverse supernet to manage multiple architectures and graph structures within supernet for effective architecture search. Then, we introduce the soft tusk-collaborative module to capture the complex relationships among tasks, enabling sharing useful information during the searching process. We further develop the task-wise curriculum training strategy to better optimize the supernet via reweighing the influence of different tasks on architecture optimization based on task difficulties. Extensive experiments on both synthetic and real-world datasets validate the superiority of our proposed MTGC" model over existing baselines via customizing the optimal architecture for each task and sharing useful information among them. Detailed ablation studies further verify the effective designs of MTGC3
Our contributions are summarized follows.
We are the first to investigate the problem of multi-task graph neural architecture search via proposing the multi-task graph neural architecture search with task-aware collaboration and curriculum (MTGC*), to the best of our knowledge.
We propose to jointly discover optimal architectures for multiple tasks and capture their complex relations within a unified framework by designing i) the structurally diverse supernet, ii) the soft task mosk-collaborative module, and iii) the task-wise curriculum training strategy.
Extensive experimental results demonstrate that our proposed MTGC model outperforms state- of-the-art baselines on both synthetic and real-world datasets.
2 Problem Formulation and Preliminaries
2.1 Message-passing graph neural network
GNNs are state-of-the the-art art models for graph machine learning, which typically follow message passing scheme where nodes aggregate information from their neighbors in each layer formulated as:
m (1) ( (11 E NG)
h ((+1) Upd)te(m!"),
(2)
where h.()) is the representation of node i at the 1-th layer, N; denotes the neighbors of node i derived from the adjacent matrix, Agg(.) is the aggregation function, Update(-) is an updating function between two node representations. Different GNNs have different aggregation and updating functions, which is the main search objective for GraphNAS.
2.2 Multi-task graph neural architecture search
Given set of graph tasks (T'm 1M m= where each task has a set of graphs Gm (gtt I in shared graph space G and the labels Ym. = {vim y denote y as the composition of all label spaces, i.e., y U(v 1mmmm m== The goal of multi-task graph learning is to design a model F : G + y to map the graphs to the label space while minimizing the loss function, i.c..
mm in its own label space ym We
arg min (((F(Gmm), Ym )
where L is the multi-task loss function. Here we use multi-task graph classification as an example, but the definition can be easily extended to multi-task node classification problems. In GraphNAS,

---

Soft Task collabbolabrr Module
1.0
P
Architecture
-0.
0.5
Task A
(9)] otherwise
1.
matro
b
(243)
perameter training
-0.5
Structurally Diverse Supernet
Task
(a) Tox21
Q
Loss
O2
Task C
Task-wise Curriculum Learning
-0.5
1
3y
Oc
lz
Ply
0T,
ll
Model weights training
task
(1 y)
BC tanh
ac
(b) ToxCast
5%
la
3f,
Figure 2: An overview of our proposed MTGC² method. Step 1: Perform- ing forward propagation through the structurally diverse supernet and the soft task-collaborative module. Step 2: Updating architecture parameters and the soft task-collaborative module. Step 3: Updating model weights through the task-wise curriculum learning.
Figure I: Kendall corre- lation coefficient of the performance ranks of different architectures on different tasks.
we mainly focus on the function f being GNNs. A typical GNN consists of two parts: an architecture E AA and learnable weights weight space, respectively. Therefore, we denote GNNs in Eq. as the following mapping function: Fo,w: 9 y. In this paper, we focus the aggregation functions Agg(-). Specifically, we consider a search space of standard layer-by-layer architectures without sophisticated connections such as residual or jumping connections, though our proposed method can easily generalized. We choose five widely used message-
W, where and W denotes the architecture space and the
GNN layers as our operation candidate set O, including GCN [17). GAT [39], GIN [47). SAGE (T], k-GNN (27). and ARMA [12. Besides, also adopt MLP, which does not consider graph structures.
The Proposed Method
To develop GraphNAS method for multi-task learning, we first consider what designs should be included that benefit multi-task learning. We propose an assumption on which our method is based.
Assumption 3.1. Different GNN architectures are suitable for learning different graph tasks. For ex- ample, if we search architectures separately for different tasks, i.e., the following bi-level optimization problem:
Om arg min L( (aFa Yn))
w(a) arg min L(F (( (Gm), (Gm) Ym),
where am the most suitable architecture for task Tm. Then, the optimal architecture O.y can be different for different tasks.
Assumption 3.] limplies that different tasks have different characteristics, while these characteristics require different architectures. To validate this assumption, we randomly choose several architectures and evaluate them different tasks individually on two graph datasets: Tox21 and ToxCast (refer to Section [4.] for details). We rank these architectures according their performance for each task. Then we use Kendall rank correlation to compute the similarity score between rankings of the architectures for pairs tasks. The result shown Figure [
We can find that many tasks have negative correlation between different architectures, indicating that different architectures behave very differently on different tasks. The result illustrates the necessity customize an architecture for different tasks instead of using shared architecture.

---

On the other hand, it is also important to share useful information between different tasks, which is common and basic assumption multi-task learning [3] Combining these two aspects, our goal is to simultaneously search multiple architectures for different tasks while learning the collaborative relationships of them. Figure 2 shows the overall framework of our method. In the following subsections, we present our proposed method in detail.
3.1 Structurally diverse supernet
need to search multiple architectures for different tasks simultaneously. To effectively manage multiple architectures, we propose the structurally diverse supernet, following the widely adopted supernets in NAS [29]. The design principle is that, although different tasks require different architectures, we cannot completely separate these architectures since they need to share useful knowledge. We propose to balance this tradeoff by disentangling the supernet. In our structurally diverse supernet, the node representation at each layer is divided into N chunks with different GNN operations. In addition, we make the graph structure be diverse different chunks to further disentangle the hidden features and help different chunks to learn more specific knowledge for the corresponding tasks. disentangled layer can formulated as follows:
Based Assumption 3.1
(5) where f one of the layers in the mapping function F, x is the hidden features given by the last layer, and A is the adjacent matrix. Moreover, represents concatenation, A; is the adjacent matrix of the graph used in the i-th chunk. f. denotes the GNN operation of the i-th chunk. By stacking the disentangled layers into a GNN, we can capture variety of latent factors of the graphs by different types of GNN operations and graph structures, which is beneficial for multi-task learning.
A) = fi A.),
Following the NAS literature 24 differentiable way. Specifically, in the supernet, all possible operations are jointly considered by mixing different operations into mixed operation, i.c., fxx) Eo, EO 0x0x(x) where oş is candidate operation and O is the operation search space, a denotes the architecture parameters to be optimized. We combine the DARTS manner with our structurally diverse supernet enable differentiable optimization, which can be formulated as follows:
we adopt supernets to search for the optimal architecture in
A,)
Σ au-ou(x, Ai),
(6)
OREO
where ouk is the architecture parameter of the k-th operation the i-th chunk. As result, multiple architectures in different chunks are contained in the supernet and they can also transfer knowledge to each other.
To adopt diverse structures for different chunks, we use a parameter-efficient and differentiable manner to generate the weights of edges for the graph structures, which can be formulated follows:
As,(u,) 8A.,))) (1 8)A (u.v) sigmoid((S)).., (S),.)),
where (a,( the weight of edge between node u and in A; A(a,o) indicates if there is an edge between node and in the original graph, S and Sy are learnable parameters to generate edge weights of the i-th chunk, is the hyper-paramen Consequently, we generate an adjacent matrix A, in continuous space. We can conveniently use gradient based methods to optimize it.
3.2 Soft cask-collaborative module
Using the structurally diverse supernet, different chunks can exchange information to share knowledge. However, the independence of chunks is not encouraged, and therefore representations could be potentially mixed together. To further ensure that different chunks of the supernet correspond to relatively independent architectures, we propose soft colaboollae module. First, we rewrite Eq. (6) as follows:
Ax) = Σ aakk Oo A.).
(8)
OLEO
where x, is the output of the j-th chunk of the previous layer, and 0ijk is the GNN operation only
containing the corresponding parameters. For example, many types of GNN operations Eq. (6)
contain parameter matrix with the shape d de in the node updating function in Eq. 22, where

---

de is the dimensionality of each chunk and d = N x de is the overall dimensionality. For these operations, we can split the parameter matrix into N matrices with the shape de de, where the j-th matrix corresponds to the part of Xj. We denote Oijk as the GNN operation only containing this part of parameters.
Using Eq. [8, we introduce our proposed soft calk-collaoorativ module. Our goal is to keep the independence of each chunk, i... the i-th chunk of node representation at the previous layer mainly contributes to the i-th chunk of node representation at the next layer. We formulate it as follows:
fi(x, A.) = Σ Oskk Σ Pijoisi(xs, Ai), Pij = { ttan Oyy otherwise, OLEO
if = j.
where # is learnable parameters. Pij represents the impact of the j-th chunk of node representation at the previous layer to the i-th chunk of node representation at the next layer. We remark that the absolute value of tanh 9 indicates the amount of contribution while the sign of tanh indicates the positivity negativity of the correlation. Since lanh 0 (-1, 1), ensure that node representation from the same chunk has the largest impact on the next layer. Therefore, the architectures of different chunks are relatively independent, which helps to make the hidden representation disentangled and contain more useful information for different tasks. Besides, we initialize & as O so that the initial structurally diverse supernet simply consists of several separated GNN architectures at the beginning. By updating / in the training procedure, our proposed method can automatically learn the transferability between architectures.
After learning representations containing diverse information by the supernet, our model makes predictions for multiple tasks. We propose two approaches to building the task head.
sesa-sepa head: We manually pair each chunk of the supernet to a task, i.e., the node representation of the last layer of the supernet is split according to the chunks, and each chunk connected to the head the corresponding task. In this case, the architecture of each chunk can be regarded as the customized architecture for the corresponding task, and Pij represents the collaborative pattern between different tasks, i.e., whether to share useful information between them. Since the number of chunks needs to be equal to the number of tasks, this approach is only applicable when the number of tasks is not too large.
Cross-mixed head: When the number of tasks is large, using the task separaep head will result in the dimensionality of the representation growing too large. Therefore, we further propose the cross-mixed head by randomly pre-assigning a few dimensions to each task, i.e., the head of each practice, we overlay a mask
task is only connected to the representations of these dimensions. tensor sampled from Bernoulli distribution the weights of the classification heads. This approach allows each task to acquire different hidden features while preventing some tasks from dominating all the architectures in the supernet.
3.3 Tisk-wise curriculum training
Our framework is end-to-end and can be optimized by the multi-task loss. But using multi-task loss may cause difficulties in training due to task imbalances [5]. At first we give the proposition below to analyze the gradient of different chunks in our model. We denote wy the parameters that generate xj in the last layer.
When using the overall loss function of multiple tasks L for gradient back- propagation in our framework, the partial derivative of L with respect to wy is:
Proposition 3.2.
0C
OL
doiyn(Xp,, As)
Σ
Σ OGik OLEO
Pij 0f:
(10)
Dwy
Ouy
For the proof of Proposition of the right side En,eo Otik J0,y' (..) as the partial derivative of the mixed operation in classical ac supernet in Eq. (6. and Sc is the sum of these partial derivatives weighted by score Piy af, aw,
3.2 please refer to Appendix B. In Eq. (10). we can regard the latter part
Bw,y
However, different tasks may have different difficulties and show diverse patterns in the optimization 8C large score Pig' 87. may dominate the training process. Besides, in our model, the optimization
process, e.g., tasks unbalanced in terms of backpropagated gradient scales, some tasks with
5

---

Algorithm 1: MTGC3
) Input: Training dataset ) and {Y;) learning rates T/a, 7/8 71 and The Initialize learnable parameters w, a, S and 8. Set = 0;
while not converge do
Forward propagate by Eq. and Eq. to calculate loss L; I/s VsC(w, S, 0), 0 0 a,
< na V V.L(w, a, S. S S
Using ac to calculate p' by Eq
Øfy
Forward propagate by Eq. 5 and Eq 11 to calculate loss L'; w T/ V w Cw,, oz, 0)
Increase 7
end
curves of different chunks can also differ greatly, which makes the optimization of our model further challenging.
To tackle these problems, we propose the task wise curriculum training strategy. Specifically, we modify Eq.
(9) using curriculum strategy as:
0C ll2 A,) E Oak Ephoosk(s, Ar), 14j = PPs -1) aʃ; tanh
Psj ||
ac lla 8f,
0f: ll2
0f; l/2
(11)
where 7 is a parameter that increases from to during the training process, and || * 12 is the ly-normalization of the target partial derivative for measuring its scale. We use Eq. (II) instead of Eq. (9) while calculating the gradients of w. Using this curriculum strategy benefits the optimization
1 ll thus the contribution of tasks with
process of our framework. When ? O, we have ply
57, la"
large partial derivatives to the loss function is suppressed. For tasks with small || ac ||2, the tanh a, function tends to be linear function, thus ply R Pij and their contributions are not affected. As y increases, Eq. [1 gradually converges to Eq. (9), keeping Eq. (TT) an unbiased estimate of the original loss function. Note that gradients are often used in curriculum learning to measure the difficulty of data [40 55 41 533, but our method introduce this idea measure the difficulty of tasks by gradients to implement task-aware curriculum learning and adjust the training process accordingly.
The overall training procedure is shown in Algorithm [ The entire model can be learned end end-to-end by gradient based approaches. We update the learnable parameters a, 6, and S in iterative way. After training, we keep the architecture and parameters in the supernet for evaluation without the architecture discretization step, enhancing flexibility architecture search and simplifying the optimization strategy.
4 Experiments
4.1 Experiment Setting
Datasets. We adopt both synthetic and real-world multi-task graph datasets.
Synthetic Datasets. We construct synthetic multi-task node classification dataset named Multi- GNN Label (MGL). We generate several MGL graphs using different graph generation methods, e.g., the Erdös-Rényi model (MGL-ER), the Watt-Strogatz small-world model (MGL-WS), and the Barabási-Alberi preferential attachment (MGL-BA). The node features are sampled from Gaussian distribution. To generate labels with diverse characteristics, we adopt different GNNs with random parameters to generate multiple hidden node representations. For each hidden representation, we use the KNN algorithm to group nodes into classes. As such, each GNN can generate node labels corresponding to a node classification task. Specifically, we use two random MLP, GCN, GAT, and SAGE to generate labels.

---

Table 1: The test accuracy of all the methods on the synthetic dataset MGL and the test ROC-AUC of all the methods on the real-world datasets OGBG. We run all experiments 10 times with different random seeds and report the average results with standard deviations. The best results are bold.
Dataset
MGL-ER
MGL-WS
MGL-BA
Tox21
ToxCast
Sider
GCN
20.30
50.1720.14
50.41
76.41 10.60
61.91
62.0221.71
GAT
10.95
10.11
17.77=0.52
75.62
63.5240.57
58.57
GIN
52.4010.41
17.67
50.4840.45
77.20
61.5840..4
58.4811.63
SAGE
62.58go.13
64.7720.14
66. 66.1544..4
75.87 49.49
64.08 64.08440.6
60.12
K-GNN
56.49
58. 58.80
62.19 19.477
76.87
63.8640.59
60.07+1.14
ARMA
60.90
64.20=a.22
65.05440.5
75.93
64.3640.88
60.79+1.24
MI.P
50.08
40.93
43.63
74.47
62.67
60.70
Random
57.40
60.80
59. 19.1570.17
76. 76.20=0.20
64.8940.60
58.87=1.38
DARTS
64.9440.20
64. 1620.29
62.39
76.96=0.67
65. (5.23440.00
60.6441.37
GNAS
49.7622.95
61.1720.00
64.28
74.97 20.41
61.8541.0 1.07
57.1141.31
PAS
75.45
63.85++0.35
59.31+1148
GRACES
74.82
60.07
61.85
MTGC
66.33
67. 67.3920.42 68.36_0.22
77.99
66. 66.36 20.3660..2
62.08
Real-world Datasets. We choose three widely-used multi-task graph classification datasets included OGB [13]: OGBG-Tox21 [14). TGxGssoxCast [35]. and OGBG-Sider [18]. These datasets contain a set of properties of toxicological assays or adverse drug reactions of drug molecules, where molecules can be represented as graphs and each task corresponds to predicting a property of drug molecules.
Baselines. We compare our model with 12 baselines from the following three different categories.
Manually Design GNNs:
we include the GNNs in our search space (refer to Section 2.2 as our baselines, i.e., GCN, GAT, GIN, SAGE, k-GNN, and ARMA. We also include MLP baseline.
Graph Neural Architecture Search: We consider three representative GraphNAS baselines: GNAS 77]. an reinforcement learning based method; PAS 42] and GRACES [3). two recent differentiable GraphNAS methods. Note that since GNAS and GRACES are specifically designed for graph classification tasks. We also consider two classic NAS baselines, random search and DARTS 24 combined with our search space.
Experimental Details. We set the number of layers 3 for synthetic datasets, and 5 for real-world datasets. For all datasets except ToxCast, we use the task-separate head. For ToxCast, we use the cross-mixed head with 16 chunks.
Number of Learnable Parameters. Denote [V]. E as the number of nodes and edges in the graph, respectively, and d as the dimensionality of hidden representations. We denote ds as the dimensionality of the structure generation part, i.e., S is matrix with the shape dg x d. The number of learnable parameters of a typical message-passing GNN is O(d). In our framework, w, a, 6, and S has ((()))) ON)O)), O(d?), ), and O(Nd,d) parameters, respectively. The total number of learnable parameters is ((O( + dgd) Olld). To guarantee fair comparison, ds and d in the experiments so that the different models are comparable in terms of the number of learnable parameters.
small
4.2 Qualitative Results
We summarize the experimental results in Table For the results on synthetic datasets, our model outperforms all baselines in all three settings. Specifically, we find that most GNNs perform poorly, indicating that they cannot well handle multiple tasks. The existing GraphNAS methods fail to outperform some manually designed GNNs these graphs. In contrast, MTGC² shows much better results by searching multiple architectures and capturing collaborative patterns between tasks.
As for the results on the three real-world multi-label graph classification benchmarks, we find that some NAS methods, e.g., DARTS and GRACES, achieve slightly better results than manually de- signed GNNs in some cases, demonstrating the importance of automating architectures. Nevertheless,
7

---

0.5
MLP
0.4
operation
GAT
0.3
chunk
SAGE
0.2
GN
0.1
ARMA
2
0.0
3 4 5 chunk
1 2 chunk
chunk
(a) The 1st layer
(b) The last layer
Figure 3: Visualizations of architec-
ture parameters of different chunks in MGL-ER.
Figure 4: Visualizations of the absolute values of Piş of different
layers in MGL-ER.
0.5
Variant
MGI-WS
Tox21
ToxCast
operation
GCN SAGE
0.3
NoStru
67.17
77.42+1.00
66.30+0.30
Separate
65.35
76.20
66. 12.1200.80
GIN
0.2
FullCollab
67.21
.7.83 40.71
66.01 +0.50
K-GNN ARMA
MLPHead
64.47
77.62
64.61
0.0
NoCL
67.1520.52
77.6810.48
66.00+1.0r
chunk
MTGC
67.39
77.99+0.42
66.36
Figure 5: Visualizations of architecture param- eters of different chunks at the 3rd layer in ToxCast.
Table 2:
The performance of different variants of MTGC on different datasets.
these methods still fail design suitable architectures for multiple tasks. Our proposed MTGC3 model again outperforms all the baselines on the three datasets, demonstrating that model is able to search for proper architectures in real-world multi-task scenarios.
4.3 Visualization and Analysis
To gain deeper insights of our method, we further conduct some analyses. Specifically, we generate four labels by a random MLP, GCN, GAT, and SAGE respectively MGL, and denote the labels as 3, 5, 7. Then, we add Gaussian noises to the node representations and generate four additional labels denoted as 4, 6, 8. result, we generate four pairs of highly correlated tasks, i.e., label 1 and 2 are related, etc. Then, we visualize the architecture parameters, i.e., Oikk, at the first layer. The results on MGL-ER are shown Figure The results show that MTGC can search for proper architectures for different tasks. For the tasks directly generated by random GNNs (see chunks 1, 3, 5, 7), 3TGC has the highest on the corresponding operation.
In addition, we visualize the absolute values of Pij in the soft task-collaborative module in Figure 4 Large absolute values indicate that knowledge be transferred between tasks. For the last layer, can observe that more knowledge can be transferred between our constructed task pairs, e.g., chunk and 2, chunk and 6. The results demonstrate that the soft task-collaborative module accurately capture the transferability between tasks. Meanwhile, for the 1st layer. the absolute values exhibit more complex patterns, indicating that some complex knowledge sharing between chunks may exist. It is also worth noting that the absolute values of P.j at the 1st layer are generally larger than that the last layer. This phenomenon indicates that there more shared information the shallow layers than at the deep layers, which is consistent with the branched sharing architecture in multi-task learning (6)
The above analyses are for the task-separated head case. We further analyze the cross-mixed head We visualize the architecture parameters ark on ToxCast dataset in Figure We can observe that different chunks choose different operations, indicating that different chunks still need to find the proper operations to the tasks assigned through the pross-mixed head.
4.4 Ablation Study
In this section, we evaluate the effectiveness of each module of our framework by conducting ablation studies. We compare the following variants of model:

---

MTGC -NoStuu we use the original graph structure for all chunks.
ATGGC-SSepaaan we disable the collaboration between chunks, i.e., vi + J.p = 0, so that different tasks search architectures separatedly.
FuulColbl we let Pij = 1, i..., full collaboration, thus the independence of each chunk is not guaranteed.
MTGCC*MMPPHes we use an MLP the task head.
MTGC -TCCC--NCLL: we remove the wise curriculum training and use standard gradient descents.
We report the results of these variants on MGL-ER, Tox21, and ToxCast in Table 2 We have the following observations. Overall, our proposed full MTGC model outperforms all the variants under all three settings, demonstrating that each component of our method is indispensable to achieve satisfactory performance in multi-task graph learning. The performance margin above MTGCs. NoStru illustrates the validity of learning diverse graph structures for different chunks. Either separating all chunks by setting P.j causes performance degradation, demonstrating that our soft calaboollaoo module is essential to capture the transferability between tasks. In addition, if use an MLP as the task head, the performance will drop severely, indicating the importance of the task-separated head and cross-mixed head in multi-task learning. Moreover, the tisk-wise curriculum training also has a consistently positive impact on the model performance.
allowing maximum collaboration by setting Pij 1
5
Related Works
5.1 Graph neural network and multi-task learning
Message-passing GNNs 17 39
23 22 21 20 50 51] have been proposed as an effective framework for graph machine learning following the neighborhood aggregation scheme. At each layer, nodes learn representations by aggregating their neighbors' representations. Then, the representation of the whole graph learned by pooling all node representations [17 47). Graph structure learning technique [15 |4] is also applied in GNNs, which learns a better graph structure for message passing during training procedure. Graph structure learning is beneficial for enhancing the robustness of graph embedding, especially in noisy scenarios.
aims jointly learn set of tasks with shared parameters. Recently, some works learn to exploit similarities between tasks in GNNs to benefit multi-task graph learning. New al. (28) explore the influence of the Hessians of each task's loss function in multi-task graph learning. SGNN-EBM (25) learns task relationships in the latent space by knowledge extracted from external datasets. MetaLink (2) uses graphs to model the relationship between data and tasks in the relational multi-task setting. Nevertheless, these works do not explore automatic architecture design in multi-task graph learning.
Multi-task learning 3
16 26
5.2 Neural architecture search
Recent years have witnessed surge of research interest in NAS methods, which aim at designing neural architectures automatically for given tasks. Since the architecture search space is discrete,
reinforcement learning (RL) [56 (29] and evolution algorithm (EA) [45P are often used in NAS methods. Besides, another strategy is transferring the discrete architecture search space into differentiable space, e.g., DARTS (24) and SNAS (46) construct supernet where all candidate operations are mixed to update the architecture well the weights simultaneously through the classical gradient descent method.
GraphNAS is gaining increasing attention from the research community 19 including RL
31 52 33 54 52)
algorithms. GraphNAS works also consider searching specific modules in graph representation learning, such as message passing layers [1, attention layers [10], and pooling operations 42 However, existing GraphNAS methods only consider single task and neglect the multi-task scenarios.
7 30) EA [49 9] and differentiable NAS [9
31
32
Outside of graphs, some multi-task NAS searches for a unified architecture for multiple tasks. However, most existing works only focus on searching the interaction operations between tasks but ignore designing different task backhones [36
48). More importantly, all the above works focus computer vision task but do not consider graph tasks.

---

Conclusion
In this paper, we propose a novel MTGC method to tackle GraphNAS problem in multi-task scenarios. Our method searches for multiple architectures for different tasks while considering task collaboration by designing the structurally diverse supernet, the soft task-collaborative module, and the task-wise curriculum training. Extensive experiments on both synthetic and real-world datasets demonstrate that MTGC can achieve state-of-the-aa performance for multi-task graph learning. One possible limitation of this work is that method only considers static and homogenous graphs and does not account for more complex graphs, such as dynamic or heterogenous graphs. Future research should explore methods that can effectively handle these complex graph types, in order to broaden the usage of the method in real-world applications.
Acknowledgments and Disclosure of Funding
This work was supported by the National Key Research and Development Program of China No. 2020AAA0106300,, National Natural Science Foundation of China (No. 62222209, 62250008, 62102222, 62206149), Beijing National Research Center for Information Science and Technology under Grant No. BNR2023RC0100: BNR2023TD03006, Beijing Key Lab of Networked Multi- media, China National Postdoctoral Program for Innovative Talents No. BX20220185, and China Postdoctoral Science Foundation No. 2022M711813.
References
[1] Shaofei Cai, Liang Li, Jincan Deng, Beichen Zhang. Zun Zha, Li Su, and Qingming Huang. Rethinking graph neural architecture search from peassage-passing In IEEE Conference on Computer Vision and Pattern Recognition, pages 6657-6666, 2021.
[2] K Cao, J You, and J Leskovec. Relational multi-task learning: Modeling relations between data and tasks. International Conference on Representation Learning (ICLR). 2022.
[3] Rich Caruana. Multitask learning, Machine learning, 28( 28(1):41-75, 1997.
[4] Hong Chen, Yudong Chen, Xin Wang, Ruobing Xie, Rui Wang. Feng Xia, and Wenwu Zhu. Curriculum disentangled recommendation with noisy multi-feedback. Advances in Neural Information Processing Systems, 34, 2021.
[5] Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pages 794-803. PMLR, 2018.
[6] Michael Crawshaw. Multi-task learning with deep neural networks: survey. arXiv preprint arXiv:2009.0999. 2020.
[7] Yang Gao, Hong Yang, Peng Zhang, Chuan Zhou, and Yue Hu. Graph neural architecture search. In Proceedings of the Twenty- International Joint Conference on Artificial Intelligence, 2020.
[8] Yuan Gao, Haoping Bai. Zequn Jie, Jiayi Ma, Kui Jia, and Wei Liu. Mtl-nas: Task-agnostic neural architecture search towards general-purpose multi-task learning, In Proceedings the IEEE/CVF Conference on computer vision and pattern recognition, pages 11543-11552, 2020.
[9] Chaoyu Guan, Xin Wang, Hong Chen, Ziwei Zhang, and Wenwu Zhu. Large-scale graph neural architecture search. In International Conference on Machine Learning, pages 7968-7981. PMLR, 2022.
[10] Chaoyu Guan, Xin Wang. and Wenwu Zhu. Autoattend: Automated attention representation search. In Proceedings of the 38th International Conference Machine Learning, 2021.
[11] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in neural information processing systems, 2017.
10

---

[12] William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, 2017.
[13] Weihua Hu, Matthias Fey, Marinka Zitiik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs In Advances in neural information processing systems, 2020.
[14] Ruili Huang, Menghang Xia, Dac- Trung Nguyen, Tongan Zhao, Srilatha Sakamuru, Jinghua Zhao, Sampada A Shahane, Anna Rossoshek, and Anton Simeonov. Tox2lchallenge build predictive models of nuclear receptor and stress response pathways as mediated by exposure to environmental chemicals and drugs. Frontiers in Environmental Science, 3:85, 2016.
[15] Wei Jin, Yao Ma, Xiaorui Liu, Xianfeng Tang, Suhang Wang, and Jiliang Tang. Graph structure learning for robust graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference knowledge discovery & data mining, pages 66-74, 2020.
[16] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7482-7491, 2018.
[17] Thomas N. Kipf' and Max Welling, Semi-supervised classification with graph convolutional networks. In Proceedings of the Sth International Conference on Learning Representations. 2017.
[18] Michael Kuhn, Ivica Letunic, Lars Juhl Jensen, and Peer Bork. The sider database of drugs and side effects. Nucleic acids research, 41(D1):D1075-D100 2016.
[19] Guohao Li, Guocheng Qian, Itzel C Delgadillo, Matthias Muller, Ali Thabet, and Bernard Ghanem. Sgas: Sequential greedy architecture search. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
[20] Haoyang Li, Xin Wang. Ziwei Zhang, Zehuan Yuan, Hang Li, and Wenwu Zhu. Disentangled contrastive learning on graphs. Advances in Neural Information Processing Systems, 34:21872- 2 21884, 2021.
[21] Haoyang Li, Ziwei Zhang. Xin Wang, and Wenwu Zhu. Disentangled graph contrastive learning with independence promotion. IEEE Transactions on Knowledge and Data Engineering, 2022.
[22] Haoyang Li, Ziwei Zhang, Xin Wang, and Wenwu Zhu. Learning invariant graph representations for out-of-distribution generalization. Advances in Neural Information Processing Systems, 35:1 35:11828-118 11841, 2022.
[23] Haoyang Li, Ziwei Zhang, Xin Wang, and Wenwu Zhu. Invariant node representation learning under distribution shifts with multiple latent environments. ACM Transactions on Information Systems, 42(1): 2023.
[24] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In Proceedings of the 7th International Conference on Learning Representations, 2019.
[25] Shengchao Liu, Meng Qu, Zuobai Zhang, Huiyu Cai, and Jian Tang. Structured multi-task learning for molecular property prediction. In International Conference on Artificial Intelligence and Statistics, pages 8906-8920 PMLR, 2022.
[26] Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 1871-1880, 2019.
[27] Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
[28] Alexander New, Michael J Pekala, Nam Q Le, Janna Domenico, Christine D Piatko, and Christopher D Stiles. Curvatureiiformeed multi-task learning for graph networks. In ICMI. 2022 2nd Al for Science Workshop. 2022.

---

[29] Hieu Pham, Melody Y Guan, Barret Zoph. Quoe V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In Proceedings of the 35th International Conference on Machine Learning, 2018
[30] Yijian Qin, Xin Wang. Peng Cui, and Wenwu Zhu. Gqnas: Graph q network for neural architecture search. In 2 st IEEE International Conference on Data Mining, 2021.
[31] Yijian Qin, Xin Wang, Zeyang Zhang, and Wenwu Zhu. Graph differenniablee architecture search with structure learning. In Advances in neural information processing systems, 2021.
[32] Yijian Qin, Xin Wang. Ziwei Zhang, Pengtao Xie, and Wenwu Zhu. Graph neural architecture search under distribution shifts, In International Conference on Machine Learning, pages 18083-18095. PMLR, 2022.
[33] Yijian Qin, Ziwei Zhang, Xin Wang. Zeyang Zhang, and Wenwu Zhu. Nas-bench-gre graph: Benchmarking graph neural architecture search, Advances in Neural Information Processing Systems, 35:54 69, 2022.
[34] Dripta S Raychaudhuri, Yumin Suh, Samuel Schulter, Xiang Yu, Masoud Faraki, Amit K Roy-Choon hhy-Chowdhury, and Manmohan Chandraker. Controllable dynamic multi-task architectures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10955 2022.
[35] Ann Richard, Richard Judson, Keith A Houck, Christopher M Grulke, Patra Volarath, Inthirany Thillainadarajah, Chihae Yang, James Rathman, Matthew T Martin, John F Wambaugh, al. Toxcast chemical landscape: paving the road to 21 century toxicology. Chemical research 2016.
in toxicology, 29(8): 1225-1251, 2016.
[36] Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. Latent multi- task architecture learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4822-4829, 2019.
[37] Tianxiang Sun, Yunfan Shao, Xiaonan Li, Pengfei Liu, Hang Yan, Xipeng Qiu, and Xuanjing Huang. Learning sparse sharing architectures for multiple tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8936-8943, 2020.
[38] Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adashare: Learning what to share for efficient deep multi-task learning. Advances in Neural Information Processing Systems, 13:8778-8740, 2020.
[39] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In Proceedings of the 6th International Conference on Learning Representations, 2018
[40] Xin Wang, Yudong Chen, and Wenwu Zhu, survey on curriculum learning. IEEE Transactions on Patlern Analysis and Machine Intelligence, 44(9):4555-4576 2021.
[41] Xin Wang, Zirui Pan, Yuwei Zhou, Hong Chen, Chendi Ge, and Wenwu Zhu. Curriculum co- disentangled representation learning across multiple environments for social recommendation. 2023.
[42] Lanning Wei, Huan Zhao, Quanming Yao, and Zhiqiang He. Pooling architecture search for graph classification. In CIKM '21: The 30th ACM International Conference on Information and Knowledge Management, 2021.
[43] Jie Wu and Yanan Sun. Evolving deep parallel neural networks for multi-task learning. In International Conference on Algorithms and Architectures for Parallel Processing, pages 517- 531. Springer, 2021.
[44] Qitian Wu, Wentao Zhao, Zenan Li, David P Wipf, and Junchi Yan. Nodeformer: A scalable graph structure learning transformer for node classification. Advances in Neural Information Processing Systems, 35:27387-27401, 2022.
12

---

[45] Lingxi Xie and Alan Yuille. Genetic emn. In Proceedings of the IEEE international conference on computer vision, 2017.
[46] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. In Proceedings of the 7th International Conference on Learning Representations, 2019.
[47] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In 7th International Conference on Learning Representations, 2019.
[48] Lijun Zhang, Xiao Liu, and Hui Guan. Automtl: A programming framework for automating efficient multi-task learning. Advances in Neural Information Processing Systems, 35:34216- 34228, 2022.
[49] Wentao Zhang, Zheyu Lin, Yu Shen, Yang Li. Zhi Yang, and Bin Cui. Deep and flexible graph neural architecture search. In International Conference on Machine Learning, pages PMLR, 2022.
[50] Zeyang Zhang, Xingwang Li, Fei Teng, Ning Lin, Xueling Zhu, Xin Wang, and Wenwu Zhu. Out-of-distribution generalized dynamic graph neural network for human albumin prediction. In IEEE International Conference Medical Artificial Intelligence, 2023.
[51] Zeyang Zhang, Xin Wang. Ziwei Zhang, Haoyang Li, Zhou Qin, and Wenwu Zhu. Dynamic graph neural networks under spatio-temporal distribution shift. In Advances in Neural Informa- lion Processing Systems, 2022.
[52] Zeyang Zhang, Ziwei Zhang. Xin Wang, Yijian Qin, Zhou Qin, and Wenwu Zhu. Dynamic heterogeneous graph attention neural architecture search. In Seirtt-Seventh AAAI Conference on Artificial Intelligence, 2023.
[53] Zeyang Zhang. Ziwei Zhang, Xin Wang, and Wenwu Zhu, Learning to solve travelling sales- man problem with hardness-adaptive curriculum. Thirty-Fifth AAAI Conference Artificial Intelligence, 2022.
[54] Zizhao Zhang, Xin Wang, Chaoyu Guan, Ziwei Zhang, Haoyang Li, and Wenwu Zhu. Autogt: Automated graph transformer architecture search. In The Eleventh International Conference Learning Representations, 2022.
[55] Yuwei Zhou, Xin Wang, Hong Chen, Xuguang Duan, Chaoyu Guan, and Wenwu Zhu. Curriculum-nass Curriculum weight-sharing neural architecture search. In Proceedings of the 30th ACM International Conference on Multimedia, pages 6792-6801, 2022.
(56) Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. In Proceedings of the 5th International Conference on Learning Representations, 2017.
13


Reviews:
[{"review_id": "fKTdytlgQs", "reviewer": "Reviewer_xv8u", "summary": "This paper proposes a multi-task graph NAS approach by learning the relationships between tasks. It uses the structurally diverse supernet to learn multiple architectures and structures together, the soft task-collaborative module to learn task relationships to exchange information, and then use task-wise curriculum training to balance task difficulties. Empirical results on OGB datasets are strong.", "questions": "Check the weaknesses parts.", "limitations": "N/A", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-organized and easy to follow.\n\n2. This paper introduces the multi-task graph NAS problem, which is important in graph learning, yet unexplored in previous works.\n\n3. The motivations of the proposed three designs are good. They are reasonable for solving the problem.\n\n4. The results on OGB datasets are good compare with SOTA baselines.", "weaknesses": "1. After calculating the edge weights, how do you perform the architectures using these edge weights in the continuous space? The implementation of this part is missing in the paper.\n\n2. Lack of detailed motivation for some parts of the methods. In Equation 9, there are other functions that can also be chosen to represent the relationship between tasks such as sigmoid, why tanh function is used here?\n\n3. For the soft task-collaborative module, the parameters $\\theta$ are learned in a continuous space. How do you keep them in the final architecture?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your reviewing efforts and constructive comments. We address your comments point by point.\n\n*Q1: After calculating the edge weights, how do you perform the architectures using these edge weights in the continuous space? The implementation of this part is missing in the paper.*\n\nResponse 1: We appreciate the reviewer's comments. The edge weights represent the importance of different weights. Once the edge weights are calculated, we multiply the weights on the message passed by this edge. If we denote $w_{ij}$ as the edge weights between node $i$ and $j$, then the message passing process shown in Equation (1) can be changed to:\n\n$$\\mathbf{m_i}^{(l)} = \\text{Agg}(w_{ij}\\mathbf{h}_j^{(l)}|j\\in \\mathcal{N}_i)$$\n\nIn this way, we can use edge weights for all GNN operations. We will add a detailed explanation of how to use the edge weights in the revised version.\n\n*Q2: Lack of detailed motivation for some parts of the methods. In Equation 9, there are other functions that can also be chosen to represent the relationship between tasks such as sigmoid, why $\\tanh$ function is used here?*\n\nResponse 2: We appreciate the reviewer's feedback. In Equation 9, the choice of the $\\tanh$ function is motivated by its desirable properties for capturing task relationships within our framework. We have the reasons as follows:\n\n(1)\tSymmetry around the origin: The $\\tanh$ function is symmetric around the origin, which allows it to model both positive and negative relationships between tasks. This is particularly important as tasks can exhibit different types of relationships, including positive correlations, negative correlations, or no correlations at all.\n\n(2)\tBounded output range: The $\\tanh$ function outputs values between -1 and 1, which provides a bounded range for representing the strength of task relationships. This range can be interpreted as the degree of collaboration or interdependence between tasks, with values closer to 0 indicating weaker relationships and values closer to -1 or 1 indicating stronger relationships.\n\nIn summary, $\\tanh$ function has great properties which exactly match the demand of task relationship representation in our framework.\n\n*Q3: For the soft task-collaborative module, the parameters $\\theta$ are learned in a continuous space. How do you keep them in the final architecture?*\n\nResponse 3: We appreciate the reviewer's question. In our soft task-collaborative module, the parameters $\\theta$ are indeed learned in a continuous space. After the whole training procedure, we keep the continuous values of $\\theta$ in the final architecture, and we evaluate the performance directly with the parameters in the supernet.\n"}, {"review_id": "yPVR6dbYui", "reviewer": "Reviewer_sAxj", "summary": "This paper proposes a graph multi-task neural architecture search technique, which is a new scene in graph NAS. This paper takes some reasonable measures to handle the problem, including structurally diverse supernet, soft task-collaborative module, and task-wise curriculum training. Its performance is worthy of recognition.", "questions": "Please see the Weaknesses.", "limitations": "None.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed supernet and collaborative module of this article are well-motivated.\n\n- This article introduces task-wise curriculum learning, which make sense in the multi-task NAS problem.\n\n- The experimental results in this paper are very good, which show the mechanism of the method clearly.\n\n- This article performs a full ablation analysis.", "weaknesses": "- Multi-task NAS is not a new technique in other NAS area. Prior arts have had some exploration. It is better to provide more comparison with those methods, especially the issue of how to exchange information between different tasks.\n\n- In the article, the definition of search space is vague. Please give more details of it.\n\n- Figure 2 is pleasing but hard to understand. It proves to be rather challenging to comprehend the inner workings of the framework and what the input-output formats are for each module.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your reviewing efforts and constructive comments. We address your comments point by point.\n\n*Q1: Multi-task NAS is not a new technique in other NAS area. Prior arts have had some exploration. It is better to provide more comparison with those methods, especially the issue of how to exchange information between different tasks.*\n\nResponse 1: Thank you for the comment. Multi-NAS has been indeed explored in areas like CV and NLP. However, we designed our method based on the specific requirements of graph multi-task learning scenarios. \n\n(1) Our approach considers different graph structures for different tasks, which is a critical consideration in graph multi-task settings. Our structurally diverse supernet enables different tasks to learn with different graph structures, which is not considered in existing general multi-task NAS methods that mainly focus on searching for sharing parts.\n\n(2) Our approach contains practical considerations for graph scenarios, such as backbone search and handling a large number of tasks. The semantic information of the multi-tasks in the graph domain is diverse and complex, leading to significant performance differences across GNNs. Figure 1 in our paper demonstrates these disparities. Existing general multi-task NAS methods[1][2][3][4][5] mainly focus on searching for sharing parts while manually fixing the backbone. In contrast, our method combines backbone searching and shared parameter searching to effectively address the unique challenges of graph multi-task problems. We also incorporate a cross-mixed head design that significantly reduces the number of parameters required for scenarios with an extremely large number of tasks, which is a practical consideration that is not addressed in existing general multi-task NAS methods.\n\n(3) Our design of information exchange between different tasks is new. Our proposed soft task-collaborative module captures the complex relationships between tasks and is specifically designed for the supernet, allowing for simultaneous optimization. Furthermore, our task-wise curriculum training strategy is tailored to our layer-wise disentangle network, and our re-weighing technique rebalances the partial derivatives from different tasks within our framework. Extensive experiments further support the effectiveness and superiority of our method.\n\nBesides, we have compared our method with several representative general multi-task methods such as MTL-NAS[1], Sparse Sharing[2], Raychaudhuri et al.[3], AdaShare[4] and AutoMTL[5]. The results are shown in Appendix D due to the space limit. The results demonstrate the effectiveness of our methods on multi-task graph learning.\n\nIn the revised version, we will include the discussion that addresses the unique challenges in graph multi-tasks NAS, which motivates the need for a specialized approach. We also acknowledge that our method can be extended to other domains by considering more domain-specific priors. By providing this analysis, we aim to strengthen the justification for our proposed approach and clarify the distinct contributions of our work.\n\n[1] MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning. CVPR 2020.\n\n[2] Learning Sparse Sharing Architectures for Multiple Tasks. AAAI 2020.\n\n[3] Controllable Dynamic Multi-Task Architectures. CVPR 2022.\n\n[4] Adashare: Learning what to share for efficient deep multi-task learning. NIPS 2020.\n\n[5] AutoMTL: A Programming Framework for Automating Efficient Multi-Task Learning. NIPS 2022.\n\n*Q2: In the article, the definition of search space is vague. Please give more details of it.*\n\nResponse 2: Thank you for the comments, we introduce the candidate operators in Section 2.2, we contain GCN, GAT, GIN, SAGE, k-GNN, ARMA, and MLP in our search space. The entire GNN backbone is a layer-by-layer architecture without sophisticated connections. \n\n*Q3: Figure 2 is pleasing but hard to understand. It proves to be rather challenging to comprehend the inner workings of the framework and what the input-output formats are for each module.*\n\nResponse 3: We appreciate the reviewer's feedback regarding Figure 2. The structurally diverse supernet is the backbone of the framework, the input is the graphs of different tasks, and the output is the prediction for the target tasks. The soft-collaborative module is a part that contains learnable parameters in the supernet. The task-wise curriculum learning strategy is the optimization method that controls the calculation of gradients. We will ensure that the revised manuscript includes an improved Figure 2 with accompanying descriptions for better clarity and understanding.\n"}, {"review_id": "96cvgfUDK8", "reviewer": "Reviewer_Eks4", "summary": "Existing GraphNAS algorithms search for well-performing architectures for a single task, this paper searches architectures for multiple graph task at the same time to share common knowledges. Specifically, this paper use structurally diverse supernet and soft task-collaborative module to search for the optimal architectures and the collaborative pattern of different tasks. The paper further proposes to leverage curriculum learning to balance gradient scales of different tasks during searching. Empirical results show that this method can improve GraphNAS in multi-task scenarios.\n", "questions": "See weakness.", "limitations": "No.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Different from existing GraphNAS papers, this paper introduces a complete framework to search for multiple architectures for multi-tasks at the same time.\n2. This paper introduces curriculum learning on multi-task scene to balance task gradients in searching phase.\n3. The experiments on different datasets and ablation study are sufficient to show how the designed method works.\n\n", "weaknesses": "1. Some designs of this paper are not well supported. At the beginning of Section 3 the authors present Assumption 1, which is the basis of the structurally diverse supernet. The authors also verified their assumption through experiments that different tasks require different architectures. However, the graph structure differences for different tasks are proposed in the supernet and are not experimentally verified.\n2. In Section 4.3, the absolute value of p is shown in Figure 4. Do positive and negative values have the same meaning of transferred knowledge? Please give more explanation of the meaning.\n3. In the algorithm proposed by the authors there are many hyperparameters that need to be optimized, such as the learning rates of different parts. This may lead to difficulties in tuning when switching between different datasets.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your reviewing efforts and constructive comments. We address your concerns point by point.\n\n*Q1: The graph structure differences for different tasks are proposed in the supernet and are not experimentally verified.*\n\nResponse 1: Thank you for the comment. Following your suggestion, we validate the necessity of graph structure differences for different tasks as we did for architecture differences. To maintain consistency with the process in our model, we first randomly sampled 12 pairs of different $S_u, S_v$. We train all these tasks separated with the best architecture searched by DARTS algorithm and these pairs of $S_u, S_v$. These $S_u, S_v$ are fixed during training, and we use them to generate edge weights as in Equation (7). We rank these pairs according to their performance and calculate the Kendall rank correlation between rankings of different tasks as well. We find the Kendall correlation values are very low. In Tox21, 73.6% values are less than 0.2, 47.2% values are less than 0. In ToxCast, 78.6% values are less than 0.2, and 47.0% values are less than 0. The results indicate that different graph structures also behave differently on different tasks, illustrating the necessity of graph structure differences. We will add the experiment and results in the revised version.\n\n*Q2: In Section 4.3, the absolute value of p is shown in Figure 4. Do positive and negative values have the same meaning of transferred knowledge? Please give more explanation of the meaning.*\n\nResponse 2: We appreciate the reviewer's comment regarding the interpretation of positive and negative values in Figure 4. We can consider $p_{ij}$ as a part inside the operation $o_{ijk}$. Since the parameters in $o_{ijk}$ can be negative, $p_{ij}$ can also be negative. Positive or negative $p_{ij}$ represents the embedding is positively or negatively correlated with downstream parameters While positive and negative values of $p_{ij}$ represent different magnitudes, they both indicate the presence of transferred knowledge. The absolute value is used to emphasize the strength or magnitude of the transferred information, regardless of its direction (positive or negative). The sign of $p_{ij}$ only represents the direction of it. We will provide a detailed explanation in the revised version to clarify the meaning of the absolute value of $p_{ij}$.\n\n*Q3: Difficulties in tuning when switching between different datasets.*\n\nResponse 3: Thank you for the comment. We have provided the typical hyper-parameter settings, including the learning rates of different parts, in Appendix C. Besides, we explore the sensitivity of these hyper-parameters. The results are shown below.\n\n| $\\eta_S$ | 0.0008 | 0.001 | 0.0012 |\n| --- | --- | --- | --- |\n| MGL-WS | $67.01_{0.66}$ | $67.39_{0.42}$ | $67.13_{0.48}$ |\n\n| $\\eta_w$ | 0.004 | 0.005 | 0.006 |\n| --- | --- | --- | --- |\n| MGL-WS | $66.74_{0.52}$ | $67.39_{0.42}$ | $67.40_{0.63}$ |\n\n| $\\eta_\\alpha$ | 0.01 | 0.012 | 0.014 |\n| --- | --- | --- | --- |\n| MGL-WS | $67.54_{0.52}$ | $67.39_{0.42}$ | $66.83_{0.47}$ |\n\nThe results demonstrate that our method is not very sensitive to these hyper-parameters, indicating that they can be easily tuned.\n \n"}, {"review_id": "rndXSugLfE", "reviewer": "Reviewer_cm5U", "summary": "This paper proposes a method called MTGC3 for searching GNNs in multi-task learning. Firstly, it highlights the importance of designing different GNNs for different tasks. Then, it introduces the Structurally Diverse Supernet and Soft Task-Collaborative modules, which enable the generation of task-specific architectures that can operate separately or collaboratively. The task-wise training strategy is employed to address the task-imbalance problem.\n", "questions": "Please check the weakness.", "limitations": "Please check the weakness.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper presents an interesting and novel approach that jointly searches for different architectures for different tasks, considering task-specific and shared information.\n", "weaknesses": "Method design:\n\nThe relationship with GNNs is not clear, and further comparisons with general multi-task+NAS methods should be discussed. \n While this paper seems to be the first to propose a method for searching multi-task GNNs, the designed method (i.e., separated trunk, soft collaboration module, and task-wise learning) appears to have weak connections with GNNs. Why can't existing NAS+MTL methods be applied? From this perspective, the first contribution of this paper appears to be weakened.\n\nIn terms of experiments, the evaluations of the proposed method could be further improved:\n\n1. Performance comparisons with general multi-task methods should be included, such as in Line 316.\n2. In Table 2, the results of the ablation study are remarkably similar to each other, indicating that they represent the basic functionality of the designed modules in this paper.\n3. Evaluation of the Cross-mix head. This module is proposed to allocate sufficient hidden units for each task. When evaluating this module, it seems necessary to remove the masked tensors and utilize only a few units for each task instead of showing the α_{ik} values?\n\nIn summary, the key contribution of this paper is not adequately justified. The paper overlooks MTL+GNN baselines, which are essential for demonstrating the effectiveness of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your reviewing efforts and constructive comments. We address your comments point by point.\n\n*Q1: The relationship with GNNs.*\n\nResponse 1: Thank you for the comment. Here we would like to furtherly clarify that we designed our method based on the specific requirements of graph multi-task learning scenarios.\n\n(1) Our approach considers different graph structures for different tasks, which is a critical consideration in graph multi-task settings. Our structurally diverse supernet enables different tasks to learn with different graph structures, which is not considered in existing general multi-task NAS methods that mainly focus on searching for sharing parts.\n\n(2) Our approach contains practical considerations for graph scenarios, such as backbone search and handling a large number of tasks. The semantic information of the multi-tasks in the graph domain is diverse and complex, leading to significant performance differences across GNNs. Figure 1 in our paper demonstrates these disparities. Existing general multi-task NAS methods[1][2][3][4][5] mainly focus on searching for sharing parts while manually fixing the backbone. In contrast, our method combines backbone searching and shared parameter searching to effectively address the unique challenges of graph multi-task problems. We also incorporate a cross-mixed head design that significantly reduces the number of parameters required for scenarios with an extremely large number of tasks, which is a practical consideration that is not addressed in existing general multi-task NAS methods.\n\nIn the revised version, we will include the discussion that addresses the unique challenges in graph multi-tasks NAS, which motivates the need for a specialized approach. We also acknowledge that our method can be extended to other domains by considering more domain-specific priors. By providing this analysis, we aim to strengthen the justification for our proposed approach and clarify the distinct contributions of our work.\n\n*Q2: Performance comparisons with general multi-task methods.*\n\nResponse 2: We appreciate the reviewer's comment. Besides, we have compared our method with several representative general multi-task methods such as MTL-NAS[1], Sparse Sharing[2], Raychaudhuri et al.[3], AdaShare[4] and AutoMTL[5]. The results are shown in Appendix D due to the space limit. \n\n| Method | Tox21 | ToxCast | Sider |\n| --- | --- | --- | --- |\n| MTL-NAS | $74.77_{0.24}$ | $63.14_{0.52}$ | $55.31_{0.64}$ |\n| AdaShare | $67.34_{1.08}$ | $62.91_{0.41}$ | $60.41_{0.46}$ |\n| AutoMTL | $73.02_{0.90}$ | $62.69_{0.39}$ | $53.94_{1.87}$ |\n| Sparse Sharing | $75.17_{1.26}$ | $64.10_{0.70}$ | $57.65_{1.15}$ |\n| Raychaudhuri et al.[5] | $75.86_{0.55}$ | $62.85_{0.24}$ | $55.90_{1.25}$ |\n| MTGC3 | $78.01_{0.68}$ | $66.74_{0.57}$ | $62.26_{1.42}$ |\n\nThe results demonstrate the effectiveness of our methods on multi-task graph learning. We will revise this section in the revision.\n\n[1] MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning. CVPR 2020.\n\n[2] Learning Sparse Sharing Architectures for Multiple Tasks. AAAI 2020.\n\n[3] Controllable Dynamic Multi-Task Architectures. CVPR 2022.\n\n[4] Adashare: Learning what to share for efficient deep multi-task learning. NIPS 2020.\n\n[5] AutoMTL: A Programming Framework for Automating Efficient Multi-Task Learning. NIPS 2022.\n\n*Q3: The results of the ablation study are remarkably similar to each other, indicating that they represent the basic functionality of the designed modules in this paper.*\n\nResponse 3: We appreciate the reviewer's comments. In order to provide a clearer illustration of the contributions of the modules. We conducted another experiment with the variant model MTGC^3-NoAll (NoStru+FullCollab+MLPHead+NoCL). This variant model only keeps the separate chunks with different architectures in our proposed design. The results are as follows:\n\n| Method | MGL-WS | Tox21 | ToxCast |\n| --- | --- | --- | --- |\n| DARTS| $64.16_{0.29}$ | $76.96_{0.57}$ | $65.23_{0.60}$ |\n| MTGC^3-NoAll | $66.68_{0.32}$ | $77.39_{0.20}$ | $65.62_{0.61}$ |\n| MTGC^3-NoStru | $67.17_{0.60}$ | $77.42_{1.00}$ | $66.30_{0.30}$ |\n| MTGC^3-FullCollab | $67.21_{0.55}$ | $77.83_{0.71}$ | $66.01_{0.50}$ |\n| MTGC^3-MLPHead | $64.47_{1.16}$ | $77.62_{0.57}$ | $64.61_{0.53}$ |\n| MTGC^3-NoCL | $67.15_{0.52}$ | $77.68_{0.48}$ | $66.00_{1.07}$ |\n| MTGC3 | $67.39_{0.42}$ | $77.99_{0.42}$ | $66.36_{0.26}$ |\n\nCompared with DARTS, we find this variant does achieve a stable improvement in performance, indicating that the proposed key idea (using different architectures in different chunks for different tasks) contributes a lot. All other designs in our paper are based on this key idea. And the performance can still be improved by these designed modules. Although the improvements they bring may not be as significant as the main contributing factor, they are still valuable for the overall framework. Besides, we find that in some cases MTGC^3-MLPHead behaves even worse than MTGC^3-NoAll, this may be due to that our designed modules are led to poorer learning when receiving more mixed gradients with the MLP classification head. We will add the experiments and discussions in the revised version.\n\n*Q4: Evaluation of the Cross-mix head.*\n\nResponse 4: We appreciate the reviewer's suggestion regarding the evaluation of the Cross-mix head module and the allocation of hidden units for each task. In our implementation, we use 128 dimensions for ToxCast and $p=1/16$ in the Bernoulli distribution. Therefore, 8 units are assigned to each task in expectation. Following your suggestion, we conduct experiments with a few units for each task. We pre-assigned only 2 units for each task. The evaluation metric on ToxCast is $61.70_{0.58}$, while that of our method is $66.36_{0.26}$, indicating that only using very few units for each task does not work well in this case. Our cross-mixed head is a reasonable solution to this situation.\n"}, {"review_id": "b06apXXuzb", "reviewer": "Reviewer_a5eh", "summary": "This paper puts forth an intriguing and innovative research problem: how to effectively search for graph neural architectures within multi-task domains. In addressing this question, the authors introduce MTGC, a methodology involving three primary stages. Firstly, forward propagation is carried out through the structurally diverse supernet in combination with the soft task-collaborative module. Secondly, both the architecture parameters and the soft task-collaborative module are updated. Finally, model weights are modified through task-wise curriculum learning. The aforementioned steps seem practical and are validated by their effectiveness in the experimental contexts. ", "questions": "As stated in weakness. ", "limitations": "No limitation part is included. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The work presented in this paper is well articulated and comprehensible apart from a few minor presentation issues. \n\n(2) The paper adds a novel angle to the existing discourse by applying GNAS to a multi-task setting, which is a noteworthy approach. \n\n(3) The research problem has been meticulously defined and the related challenges have been effectively pinpointed. \n\n(4) The proposed methods come across as plausible. I like the manner in which the graph structure has been disentangled. It is generalizable. \n\n(5) The experiment stage of the research is replete with ample benchmarks, which efficiently validate the effectiveness of the proposed method. ", "weaknesses": "(1) The left part (three graphs) in Figure 2 is confusing. Are there three different input graphs? or just three disentangled graphs from the same input graph? It should be clarified. \n\n(2) What kind of knowledge should different GNN architectures share in the multi-task setting? Can you present more discussions about this? or raise some examples? \n\nThere are some related works that should be cited. \n\n[1] Factorizable graph convolutional networks. \n\n[2] Automatic relation-aware graph network proliferation. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your reviewing efforts and constructive comments. We address your comments point by point.\n\n*Q1: The left part (three graphs) in Figure 2 is confusing. Are there three different input graphs? or just three disentangled graphs from the same input graph? It should be clarified.*\n\nResponse 1: We appreciate the reviewer's feedback regarding Figure 2. There are indeed three different input graphs in the left part of Figure 2, and each of these graphs represents the input of a task. We will make sure to provide additional clarification in the revised version to avoid any ambiguity.\n\n*Q2: What kind of knowledge should different GNN architectures share in the multi-task setting? Can you present more discussions about this? or raise some examples?*\n\nResponse 2: We appreciate the reviewer's value question about the shared knowledge in our multi-task architecture. If the labels of two tasks have a high correlation, they will share more information inside. For example, if the hidden feature of a chunk's output in a layer is useful for the inference of two different tasks, the supervised signals of both tasks will further optimize the learning of the parameters of this chunk. Although the form of the shared knowledge is parameter that cannot be intuitively understood. The strength of the sharing can be estimated by $p_{ij}$ in our framework, as shown in Figure 4.\n\n*Q3: There are some related works that should be cited.*\n\nResponse 3: Thank you for your suggestion, we will ensure that the revised manuscript includes the references of these papers to address the reviewer's concern.\n"}]

Author rebuttal:

Meta-review: This paper proposes a new multi-task graph neural architecture search method. 

Four reviewers gave acceptance scores while satisfied with its task definition, method design, and thorough experimental results. 
On the contrary, one reviewer raised the insufficient evidence of the proposed contributions. 

During the rebuttal, the authors provided additional results on the concerns, and AC judged that the authors successfully addressed the issues. 

Considering the scores and the author response, AC recommends accepting this paper.
Decision: Accept (poster)