Decoupling Weighing and Selecting for Integrating Multiple Graph Pre-training Tasks
Authors: Tianyu Fan
Lirong Wu
Yufei Huang
Haitao Lin
Cheng Tan
Zhangyang Gao
Stan Z. Li
Venue: ICLR 2024 poster (2024)
Keywords: Multi-tasking learning
Graph Neural Networks
AutoML
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: A Decoupled Framework for Integrating Multiple Graph Pre-training Tasks

Abstract: Recent years have witnessed the great success of graph pre-training for graph representation learning. With hundreds of graph pre-training tasks proposed, integrating knowledge acquired from multiple pre-training tasks has become a popular research topic. In this paper, we identify two important collaborative processes for this topic: (1) select: how to select an optimal task combination from a given task pool based on their compatibility, and (2) weigh: how to weigh the selected tasks based on their importance. While there currently has been a lot of work focused on weighing, comparatively little effort has been devoted to selecting. This paper proposes a novel instance-level framework for integrating multiple graph pre-training tasks,
Weigh And Select (WAS), where the two collaborative processes, weighing and selecting, are combined by decoupled siamese networks. Specifically, it first adaptively learns an optimal combination of tasks for each instance from a given task pool, based on which a customized instance-level task weighing strategy is learned. Extensive experiments on 16 graph datasets across node-level and graph-level downstream tasks have demonstrated that by combining a few simple but classical tasks, WAS can achieve comparable performance to other leading counterparts.
The code is available at https://github.com/TianyuFan0504/WAS.

Published as a conference paper at ICLR 2024
DECOUPLING WEIGHING AND SELECTING FOR INTEGRATING MULTIPLE GRAPH PRE- PREIINAINING TASKS
Tianyu Fan 1.2.,, Lirong Wu 12. Yufei Huang 122 Haitao Lin 12, Cheng Tan 1,2 Zhangyang Gao Stan Z. Li
Westlake University, *Zhejiang University
{fantianyu, wuli wulirong, huangyufei, 1 Anhaitta]eweestttee... lake. edu. cn; (tancheng, gaozangyan..ggng.g.gg.... stan. zq. 1 1 @westlake, cn
ABSTRACT
Recent years have witnessed the great success of graph pre-training for graph representation learning. With hundreds of graph pre-training tasks proposed, integrating knowledge acquired from multiple pre-training tasks has become a popular research topic. In this paper, identify two important collaborative processes for this topic: (1) select: how to select an optimal task combination from given task pool based on their compatibility, and (2) weigh: how to weigh the selected tasks based on their importance. While there currently has been a lot of work focused weighing, comparatively little effort has been devoted to selecting. This paper proposes novel instance-level framework for integrating multiple graph pre-training tasks, Weigh And Select (WAS), where the two collaborative processes, weighing and selecting, are combined by decoupled siamese networks. Specifically, it first adaptively learns an optimal combination of tasks for each instance from a given task pool, based on which a customized instance-level task weighing strategy is learned. Extensive experiments on 16 graph datasets across node-level and graph-level downstream tasks have demonstrated that by combining a few simple but classical tasks, WAS can achieve comparable performance to other leading counterparts. The code is available at https://gitthbb. com/ www./ianynoooooooeeeee FanonuuuaO000/WA.....
INTRODUCTION
Relationships between entities in various real-world applications, such social media, molecules, and transportation, can be naturally modeled as graphs. Graph Neural Networks (GNNs) (Hamilton et 2017; Veličković et al., 2017; Wu et al., 2023age; 2022c) have demonstrated their powerful capabilities to handle ollatin--depender tasks. However, most of the existing work in GNNs is focused on supervised or semi-supervised settings, which require labeled data and hence are expensive and limited. Recent advances in graph pre-training (Wu et al., 2021; Xie et al., 2021; Liu et al., 2021c) are aimed to reduce the need for annotated labels and enable training on massive unlabeled data. The primary purpose of graph pre- rre-training is to extract informative knowledge from massive unlabeled data and the learned knowledge can then be transferred to some specific downstream tasks. While hundreds of graph pre-training tasks have been proposed in this regard (Sun al., 2019; Hu et al., 2020b; Zhu et al., 2020b; You et al., 2020a; Zhang et al., 2020; Wu et al., 2023d), as shown in Fig. 1(a), there no single pre-training task that performs best for all datasets.
Therefore, integrating or more specifically linearly weighing multiple tasks, has emerged as a more effective approach than designing more complex tasks. For example, AutoSSI. (Jin et al., 2021) combines the weights of task losses based on a pscudo-homophily measure, and ParetoGNN (Ju al., 2022) reconeiles pre-training tasks by dynamically assigning weights that promote the Pareto optimality. Another related work is AUX-TS (Han et al., 2021), which also adaptively combines different tasks, but this combination appears in the fine-tuning stage. However, all three works perform task weighing in a global manner, ignoring the fact that different instances (e.g., nodes in a social network or graphs in a molecular dataset) may have their own specificities. To solve this, AGSSL (Wu et al., 2022a) has been proposed to learn instance-level task weighing strategies during the fine-tuning stage. Nonetheless, these works to weigh all tasks focus only on the importance issue, but ignore the compatibility issue, i.e., the possible conflicts between different tasks, which cannot be resolved by simply weighing all tasks. More seriously, as the task pool expands, compatibility issue
Equal Contribution. t: Corresponding Author.

---

Published as a conference paper at ICLR 2024
0.79
(LAAOO OnlyAM
CP
TQC-AUCI9I
0.78
GP-517
5
0.77
EP
GPTGNN
0.76
GraphCL
GraphCL
0.75
GraphLoG
GraphiLoG
16
Bace
Bear
City
Sider
NN
0.0
6.2
0.4
0.6
0.8
10
A4g
Cp
89
GFT
AM's weight A
(a) Dataset Performance
(h) Task Importance
(c) Task Compatibility
Figure 1: (a) Performance ranking (1: best, 7: poorest) of seven pre-training tasks (rows) on eight datasets (columns). (b) Performance fluctuation on Bace (molecule dataset) when combining two tasks, AM and CP, with different task weight X. (c) Performance gains or drops over that without pre-training when combining two tasks (diagonal represents only a single task) on Bace.
Table A comprehensive comparison between previous methods and ours. Stage indicates at what stage the method is applied. Task Type represents the levels of tasks that the method can handle.
Instance-level
Weighing
Selecting
Task Type
Stage
AutoSSL (Tin et al,, 2021)
Node
pre-training
ParetoGNN (Ju
2022)
X
X
Node
pre-training
AUX-TS (Han ct al., 2021)
Node
fine-tuning
AGSSL (Wu al., 2U22a)
X
Node&Graph
tine-tuning
WAS (ours)
Node&Graph
fine-tuning
becomes more severe, which deprives existing methods of the ability to keep gaining performance growth. Therefore, it is necessary select some tasks solve it. In addition, previous works have only evaluated their effectiveness on node-level tasks but neglected graph-level tasks. We have summarized the properties of these methods in Table. 1 and compared them with WAS.
We would like to raise several issues through investigation on several classical graph pre-training tasks, including AttrMask (AM), ContextPred (CP) (Hu et al., 2020b), EdgePred (EP) (Hamilton et al., 2017), GPT-GNN (Hu et al., 2020c), GraphCL (You et al., 2020a), GraphLoG (Xu et al., 2021), and InfoGraph (IG) (Sun et al., 2019), as shown Fig. Let us first draw an interesting conclusion: both importance weighing and task selecting of tasks are quite important, where the former addresses the issue of task importance, while the latter addresses the issue of task compatibility.
Fig. (bb) compares the performance under varying importance weights À when combining AM and CP. We can see that the best performance is achieved at X == 0.4. While task weighing focuses on the importance of each task, it neglects compatibility between different tasks, which limits existing methods from achieving higher performance. The results in Ii). illustrate the huge impact of the compatibility issue when integrating multiple tasks. It can be seen that not all combinations can bring performance improvements, and some of them even degrade performance (combination of AM and GraphCL brings -7.08 drop), which highlights the necessity of selecting suitable task combinations, Note that Fig. (c) shows only the combination between two tasks. As the task pool expands, such complex, and conflicts between different tasks may become more severe.
combinations become
Based on the above investigations, here we would like to ask: How to address the importance and compatibility issues between different tasks when integrating them during the fine-tuning stage? We identified two key issues in the combining process: (1) task selecting - how to select an optimal combination from given task pool based on the task compatibility, (2) importance weighing how to weigh the importance of the selected tasks. These two are obviously related or collaborative. More important tasks should be selected more, but selecting tasks based solely on importance can lead to severe task conflicts. While previous works, AutoSSI., ParetoGNN, AUX-TS, and AGSSL, have focused on importance weighing, they have all overlooked task selecting, which has deprived them of the ability to keep gaining performance growth as the task pool grows larger.
In this paper, we propose a novel framework, Weigh And Select (WAS) for task selecting and impor- tance weighing. The two collaborative processes are combined in decoupled siamese networks, where (1) an optimal combination tasks is selected for each instance based on a sampling distribution calculated based the task compatibility, and (2) task weights then calculated for the selected tasks according to their importance. To the best of our knowledge, this work the first attempt to use the weighing& & selecting strategy for integrating multiple graph pre-training tasks, Extensive
2

---

Published as a conference paper at ICLR 2024
experiments on 16 datasets show that WAS can achieve comparable performance to other leading counterparts for both node-level and graph-level tasks. Our contributions are summarized as follows: (I) We show the limitations of current weighing-one -onliiinnnnnnnnnnnnn schemes and demonstrate the importance of task selecting process. (2) To the best of our knowledge, we are the first to identify two important collaborative processes: select and weigh; provide extensive experiments to explain in detail why the collaboration of the two is important, how it differs from the weighing-onke based methods and why do the two processes need to be decoupled. (3) We propose novel framework adaptively, dynamically, and compatibly select and weigh multiple pre- wre-training tasks for each instance separately.
2 RELATED WORKS
Graph Pre- Training. Graph neural networks (GNNs) are powerful tools to capture useful information from graph data (Zügner & Günnemann, 2019; Wu et al., 2022d;b; Hu et al., 2020a). Recently, there have been lots of efforts in pre-training GNNs to alleviate the need for expensive labeled data and improve the generalization ability. These methods usually use various well-designed pre- training tasks to pre-train GNNs on large unlabeled datasets. Generally, mainstream pre-training tasks can be divided into three categories: generative, contrastive, and predictive. The generative methods, such as EdgePred (Hamilton et al., 2017), AttrMask (Hu et al., 2020b), and GPT-GNN (Hu et al., 2020c), focus on reconstructing important information for each graph at the intra-data level. Besides. the contrastive methods, such as ContextPred (Hu et al., 2020b), GraphLoG (Xu et al., 2021). GraphCL (You et al., 2020a), and JOAO (You et al., 2021), apply data transformations construct different views for each graph, aiming to learn representations to distinguish the positive views from the negative views. The predictive methods, such as G-Motif (Rong et al., 2020), CLU (You et al., 2020b), and PAIRDIS (Jin et al., 2020), generally self- self-generate labels by some simple statistical analysis and then perform prediction-style tasks. More details can be found in Appendix A.
Multi-Tasking Learning. There are many works on multi-tasking learning (Doersch & Zisserman, 2017; Ren Lee, 2017; Zamir et al., 2018; Wu et al., 2020; Yu et al., 2020; Wang et al., 2022) on CV or NLP domains. Due to the immense success of Transformer (Vaswani et al., 2017), many methods from non-graph domains are applicable only to the transformer architecture (He et al., 2022; Zhu et al., 2023; Wang al., 2022). In addition, lots of tasks focus on designing methods to combine losses (Yu et al., 2020; Liu et al., a), ignoring the powerful potential of instance-level design. The existing methods on graphs can be broadly classified into two categories, global-level and instance-level. For global-level, AUX-TS (Han et al., 2021) combines different tasks to promote a target pre-training task's performance. AutoSSL (Jin al., 2021) combines the weights of losses on tasks by measuring pseudo- hreudo-homphhily.. and ParetoGNN (Ju et al., 2022) reconcile tasks by dynamically assigning weights that promote the Pareto optimality. They trained the model by combining losses, ignoring the different needs of different instances. For instance-leve level, AGSSL (Wu et al., 2022a), which is closest to us, designs its weighing function to approach an ideal Bayesian teacher for each instance. Despite their great success, all the above methods focus only on importance weighing but ignore task selecting,
3 PRELIMINARY
Notations. Let G EV.E) denote a graph, where V {V1, u2..... .U.] and E C V denote
the node set of [V] = N nodes and the edge set. Given set of graphs G (9, 92. GM }. graph classification aims to learn a GNN encoder ho) : g -> TR and an additional projection head (o(+): R + RC to adapt to downstream tasks, where C' is the number of category.
Pre-training and fine-tuning on Graphs. Graph pre-training aims to extract informative knowledge from massive unlabeled data Prre through a pre-training task (f) and then transfer the learned knowledge to a downstream task Ldown (0, w) on the labeled data Ddown, which includes two steps:
(I) Pre-training GNN encoder he (-) on an unlabeled dataset D Prre. with the objective formlulated as
0 pre arg min Cpre Dpre).
(I)
0
(2) Fine-tuning the pre-trained GNN encoder ha (-) with prediction head g.(+) the labeled dataset Ddown: defined as
min Ldown (ho Swi D.own). (0,20)
Multi-teacher Knowledge Distillation. Given K teacher models (., (),.... FK()) and
student model ($),, the multi-teacher knowledge distillation extracts knowledge from multiple
3

---

Published as a conference paper at ICLR 2024
Pre-training lask
Encoder (5
Prediction Head O
Teacher
Downstream
Pre-training Task
Eneader [
8.y0
Teacher 2
Pre-training Tisk K
Encoder h,
Prediction Head Boy
Teacher
Graph g.
slop-gradient
selection cf.i))
p?
A(-aj
1011100
KL ergencc
MLP Mammmntum updattig
valimax
P
Student
Weighing
Dymamic Quii
neight (v)
Figure 2: Overall workflow of WAS. Firstly, we train multiple teachers with different pre-training tasks. Secondly, we pass the teacher's 's representations to two modules (Selecting and Weighing) to get the selecting results x(- i) and initial weights w(, i) for each instance G. Finally, we weigh only those selected teachers to get weights X(.,i) and distill the integrated distributions into the student.
teachers and then distills the extracted knowledge into student, which can be formulated as follows
Σ (NNT(()),(S())
LMKD -LK
where AE is the weight of k-th teacher that satisfies Σ A 1, and LxI(-) denotes the Kull- back-Leibler divergence that measures the distribution differences.
METHODOLOGY
The graph pre-training tasks are designed to provide more information to the model during pre- training, and in this sense, using more tasks provide richer information. However, there two issues to be addressed when combining tasks, namely, the importance of each task and the compatibility between tasks. Since different tasks exhibit different importance and compatibility on different data, in this paper, we propose a novel instance-level framework, Weigh And Select (WAS), as shown in Fig. 2, which can be divided into two steps: knowledge extraction and knowledge transfer. In the knowledge extraction step, we train multiple teachers with different tasks to extract different levels of knowledge. In the knowledge transfer step, we integrate those knowledge for each instance by weighing and selecting, and then distill the integrated knowledge into the student model. To deal with the importance stop-grade -rrddinat and momentum updating, which assigns different weights to each teacher and determine whether they should be selected, and then only weigh those selected teachers, i.c., weigh and select. In this section, we delve into the details of our framework WAS by answering the following questions:
compatibility issues, we design decoupled siamese networks with
QL (Instance): How to design a framework that enables customized schemes for each instance?
Q2. (Importance): How to design the weighing module to address the importance issue?
Q3. (Decouple): How to decouple selecting from weighing to prevent their results from overlapping?
Q4. (Compatibility): How to design the selecting module to address the compatibility issue?
4.1 INSTANCE-LEVEL MULTI-TEACHER KNOWLEDGE DISTILLATION
To answer Q1 (Instance), adopt multi-teacher knowledge distillation (MKD) achieve knowledge transfer in our framework. MKD was first proposed to obtain a model with fewer parameters, i.e., model compression (Wu et al., 2023b;c), but here we use it for distilling the knowledge of different teachers into one student model. As opposed to the method of combining different losses, used in A (Jin et al., 2021) and ParetoGNN (Ju al., 2022), MKD can get the distribution of each teacher on each individual instance. This means that the teacher's impact on students is independent at the instance-level. so we are able to learn customized teacher combinations for each instance to enable the student model to learn a better representation. Furthermore, since we directly weigh the output distributions of teachers rather than the task losses, the learned weights can truly reflect the importance of different teachers, because their output distributions are at almost the same level for given instance. However, different losses may be on different orders magnitude, so the weights of different losses cannot directly reflect the importance of different graph pre-training tasks.
4

---

Published as a conference paper at ICLR 2024
To combine different teachers at the instance-level, we learn eustomized weighing and selecting strategies for each instance separately. Specifically, given K teacher models ff ()), f2 ()..... fK() trained by different pre-training tasks, our goal is to obtain an ideal combination of them for each instance Ge. The whole framework can be divided into three key steps: (1) Get the label distribution
Pf 520(, (9.) of the k-th teacher f2 (.) on i-th instance G:- (2) Pass the obtained distributions through two mutually independent modules W and S, i.e., weighing (introduced in 4.2.1) and selecting (introduced in 4.2.3), to obtain initial weight (kk, (0, 1] and selecting results <(k, i) E {0, 1}. and then weigh only those selected teachers by softmax to output teacher weight A(k, a) E (0. 1]. (3) Integrate the outputs of different teachers to obtain an integrated teacher distribution P.7 as follows,
Pf Σ k<kk, ()Ak, i)
(4)
where Ex [kk, i) 11, and then distill the integrated distribution Df" into the student ()) via multi-teacher knowledge distillation, with the learning objective formulated as
min Ldown (0.w) + E CKL (P.,
(5)
6 A,
4.2 STAMESE NETWORKS FOR TASK WEIGHING AND SELECTING
An instance-level framework has been presented in Sec. 4.1 to transfer knowledge from multiple teachers to a student, but doesn't 't tell us how to weigh and select. In this subsection, we propose siamese networks that consist of two modules dealing with importance and compatibility.
4.2.1 TASK WEIGHING BY IMPORTANCE MODELING
Here, to answer Q2 (Importance), we design a weighing module W to adaptively learn suitable task weights for each instance. The first question is, how to model the importance issue. some current multi-task learning methods (Jin et al., 2021; Ju et al., 2022), they directly optimize the weights X of different tasks (usually in the form of the coefficient of losses). However, this is only useful at the global-level as different tasks have different forms of loss. For example, GraphCL performs contrastive learning between graphs (graphs as instances), while AM masks node attributes on single graph (nodes instances). To solve this, we use latent space described by the variable { ) and associate each teacher with a latent factor μa E RC that captures the local importance of different teachers. The importance weight of the k-th teacher to graph Gi can be calculated as follows:
exp (Sk.z)
ik, Ek exp (Sx.))
where Cu.i = T (Jes f (GG))
(6)
where v E RC is a vector of global parameters to be learned, which determines whether the value of each dimension in (Ms @ (G.)) has positive impact on the importance.
After modeling the importance issue, the second question is, how to optimize the learning of w(.-). Inspired by AGSSL (Wu et al., 2022a), an optimal weighing scheme should make the integrated teacher distribution p²)() close to the Bayesian teacher (G), which provides true class probabilities but is often unknown. Luckily, Menon et al. (2021) demonstrate that P'(G) can be estimated using the ground-truth label ey. Therefore, we approximately treat Pt (G) ~ Cy. optimize w(.-)) -) by minimizing the binary cross entropy loss Lw Daowwl EiEdaom. ((P')(G) P PG,)) on the downstream labeled set, and estimate weights {)) for any unlabeled instance Gj-
4.2.2 DECOUPLE MODULES BY SIAMESE NETWORKS
Before going deep into the selecting module, we must first clarify one question: what kind of selecting results do we want? As mentioned earlier, the weight (kkk i) can
Comparison between three selecting methods
Random Importance based Dacoupie
AUC(%)
characterize the importance of each task, so is natural to
select those tasks that are important and remove those that are not, i.e., using the weight as the selecting probability. However, this scheme is equivalent to explaining compat- ibility in terms of importance and completely confuses the two, which defies expectations. Fig. 3 shows the results of three selecting methods on four datasets. It can be seen that the completely importance- based selection
60
50
BACE
HIV
SIDER
BBBP
Figure 3: A detailed comparison of three task selecting schemes on four datasets.
5

---

Published as a conference paper at ICLR 2024
method neglects compatibility and is sometimes worse than random selection. However, it would be a pity to discard the task importance altogether, as it undoubtedly provides rich information wanted to design selecting module to utilize this
that can be used to guide task selection, information but not rely on it. That is, we want the selecting results and weights to be related but not coupled with each other. Then, the two modules should be architected in such way that the selecting results can make use of the weights but are not directly derived from them. We call this process decouple. As shown in Fig. 3, the decoupled selection achieves the best performance.
To answer Q3 (Decouple), we construct the selecting module and the weighing module in the form of siamese networks inspired by Nair & Hinton (2010) to get weights and selecting results simultaneously, and then weigh only those selected teachers. Such a construction can strictly distinguish between the two modules, allowing the selecting module to focus on solving the task compatibility problem and the weighing module to focus on the task importance problem.
4.2.3 TASK SELECTING BY QUIT MECHANISM
To answer Q4 (Compatibility), propose novel selecting module 3 adaptively resolve the compatibility between tasks and select the most suitable task combination for each instance separately.
As mentioned in Sec. 4.2.2, we construct the selecting module and the weighing module the form of siamese networks. However, since the two modules have the same architecture, if we update them in the same way (c.g., both by back-pre pooek--ppgggtion) their outputs are highly likely to be re-coupled together. Therefore, momentum updating to optimize the parameters of the selecting module S. Let tweighing denote the parameters of weighing module W and selecting denote the parameters of selecting module S, the process of momentum updating can be written as follows:
cut off the selecting module's gradient back- mnok-propagation and adopt the
Oselecting = Oselecting + (1 -m) @weighhngs
where m is the momentum updating rate, which controls the similarity of S and W. This setup enables the selecting module to acquire enough knowledge from the historical weights of different teachers to guide the selecting results because fselecttng is based on historical wweighing
However, since O #xelecting are all superimposed by tweighing at different epochs, the output of the selecting module is still likely to be close to the weighing module if W converges earlier. Therefore, we add additional projection head (implemented as MLP) after S to enhance the discrepancy between the selecting and weighing results. Such an organization enables S to still utilize the information of weighing but be thoroughly decoupled from the weighing module, allowing the two modules to do their work more independently. Given the output kg(k, i) of the selecting module (just like the output of the weighing module (k:, i)), we feed Kig (R;, i) into the MLP and then normalize the result (M (k: i)) to ensure that at least one teacher is selected for each instance, defined as
(K,(Ns((k,)) i)) Knorm (k; max j=1 (LP((s((,,))) i))
(8)
Note that Knorm (k: i) [0, 1]. so every teacher has a chance to be sampled. In addition, a teacher not sampled in the previous epoch does not mean that it will not be sampled in the next epoch, which allows our model to try a sufficient number of task combinations. At the convergence stage, the probability of some teachers will be close to 0 so that they hardly participate in knowledge distillation, which we call dynamic quit. helps us select the most suitable teachers for each instance because the model can try enough task combinations and finally discard those useless tasks. To differentiably select k-th teacher for i-th instance according to the normalized sampling probability Knorm (k, i), we use the reparameterization trick, which is commonly used in previous works (Huang et al., 20; Liu et al., 2021b). Specifically, we adopt Ctmbel-Soffman sampling (Maddison et al., 2016), which samples x(k, i) from the Bernoulli distribution with probability Knorm (k, i). defined as
<<kk i) =
11+ exp (log Koorm (G.1)+))
2
where 7 1.0 is the temperature of gumbel-softmax distribution, and G ~ Gumbel 1) is a Gumbel random variable. Finally, we weigh only those selected teachers by re-softmax, follows <(k, i) exp([(k) i))
i(k, ΣK ro(ks i) exp(((k,
(10)

---

Published as a conference paper at ICLR 2024
So far, our WAS framework, including weigh and select, has been built. Due to space constraints, pseudo-code and complexity analysis are placed in Appendix B and Appendix C.
EXPERIMENTAL EVALUATION
In this section, we evaluate WAS on 16 public graph datasets for both node-level and graph-level downstream tasks. To be specific, we would like to answer the following five questions: Q1. (Improvement) Does WAS achieve better performance than its corresponding teachers? Q2. (Effec- tiveness) How does WAS compare to other leading baselines? Can WAS's performance continue to grow as the task pool expands? Q3. (Localization) Can WAS learn customized task combinations for different instances? Q4. (Decouple) Whether the selecting results are decoupled from the weighing results? Q5. (Integrity) How does each component contribute to the performance of WAS?
Datasets & Implementation Details. For graph-level tasks, following Liu et al. (2022), we use 5-layers GIN (Xu et al., 2018) as our backbone. We randomly select 50k qualified molecules from GEOM (Axelrod & Gomez-Bombardle Bombarelli, 2020) for the pre-training and then fine-tune on 8 graph datasets, including BACE, BBBP, ClinTox, SIDER, Tox21, Toxcast, MUV, and HIV. For node-level tasks, following Wu et al. (2022a), we use 2-layers GCN (Kipf & Welling, 2016) as our backbone. We conduct experiments on real-world datasets widely used in the literature (Yang et al., 2016; Hassani & Khasahmadi, 2020a; Thakoor et al., 2021), i... Physics, CS, Photo, Computers, WikiCS, Citeseer, Cora, and ogbn-arxiv. A statistical overview of these datasets placed in Appendix D. The implementation details and hyperparameter settings for each dataset are available in Appendix E.
Task Pool. There are a total of 12 classic graph pre-training tasks that make up our task pool. For graph-level tasks, we consider 7 pre-training tasks, including AttrMask (Hu et al., 2020b), ContextPred (Hu et al., 2020b) EdgePred (Hamilton et al., 2017). GPT-GNN (Hu et al., 2020c), GraphLoG (Xu et al., 2021), GraphCL (You et al., 2020a), and InfoGraph (Sun et al., 2019). For node-level tasks, we follow AutoSSL (Jin et al., 2021) to consider five tasks: DGI (Velickovic al., 2019), CLU (You et al., 2020b), PAR (You et al., 2020b), PAIRSIM (Jin et al., 2020), and PAIRDIS (Peng et al., 2020). Detailed description of these tasks has been placed in Appendix A.
Table 2: Results of seven baseline teachers and WAS for molecular property prediction. For each dataset, we report the mean and standard deviation of ROC-AUC(%) with scaffold splitting. The best and second-best results are marked in bold and underlined, respectively. The arrows indicate whether the multi-task method has improved relative to the average performance of the seven teacher tasks.
BACE
BARP
C CnTox
SIDER
Tox21
Texcast
NUV
HCV
Avg. Rank
AttrMask
77.4=0.2
65.3+1.6
70.3=7.5
55.1
74.4=0.5
62.640.1
75.4+2.7
75.9+0.4
5.3
ContextPred
77.3+110
69.0+2..
66.97.6
58.7++1.6
72.9-08
61.7+0077
73.++0.3
76.14244
5.6
EdgePred
66.11.26
68.6166.7
67.3122
58.911.3
68.219.1
59.0.0.9
73.5119
7388-144
7.5
GPT-GNN
78.6229
65.311..
56.1289
57.910.22
74.310.7
63.310.3
75.6118
74.81114
5.8
GraphCL
77.5=166
69.9+11.6
72.1=477
19.9+:15
75.1
62,8=007
15.1
74.5+00.6
3.4
GraphLoo
18.10
66.4+288
14.1
59.5+24
739=114
62.3++0.6
73.++10
75.5+0.5
5.5
InfoGraph
71.4:2.4
6.12+111
7..5:-6.3
58.240.44
7..0+0.5
60.7:0.9
7321-0.8
74.340.5
7.5
Average-Woigh
75.3+1.21
66.640.21
62.6+4.41
569=15
67.8+088
62.2+181
74.2+0.71
74.8-031
7.5
Randomn Select
77.211.71
67.2:10.5L
58.111.51
59.2-1.71
74,3110.67
59.310071
71.6114
76.1-121
6.3
WAS
80.7+0.51
70.5+1.07
73.1+0.51
60.4-0.4"
75.4+0.71
63.7+155
76.3+1.01
78.4-1.91
1.0
5.1 PERFORMANCE COMPARISION
A1: Performance comparision with Teachers. To answer Q1 (Improvement), we report the results for seven individual tasks and the model trained by WAS using the seven tasks as its teach- ers in Table. 2. Besides, we add two additional combination methods, Average-Weine -eight ght and Random-Se Selnnt
better compare the performance. Average-Weigaa assigns equal weights to each teacher, and Random-Sel lct randomly select teachers for each instance. From Table. 2, we can make the following observations: (1) There no one optimal pre-training task that works for all datasets, which means we need to choose the most suitable pre-training tasks for each dataset separately. (2) Most of the results (9 out of 16) of Average average-Weight and Random-Select are worse than the average performance, indicating that importance and compatibility issues are quite important and challenging.
(3) WAS achieves the best performance on all the datasets, which means that it can handle the compatibility problem between tasks effectively in the vast majority of cases.
A2: Performance comparison with other baselines. For graph-level tasks (Table. 3), we compare WAS with other state-of-the graph pre-training baselines, including AD-GCL (Suresh et al., 2021),
7

---

Published as a conference paper at ICLR 2024
Table 3: Comparison with leading graph pre-training basclines for the task of molecular property prediction, where the best and second marked bold and underlined, respectively.
BACE
BBBF
ClinTox
SIDER
Tox21
Toxcaat
MUV
EIV
Avg. Rank
AD-GCL
76.3110.6
70.1=12
78.1
59.410.77
74.74:11
62.840.5
71.6=0.8
77.7+11.2
4.4
R-GCL
73.6+12
69.3=0.4
78.9=4.1
59.910.9
75.240.3
62.440.6
74.7=1.2
16.1
4.3
Co
76.3+119
64.0+1.5
67.6+2.9
61.1+22.5
74,9+0.22
61.740.1
74.2+1.4
7,.3+1.4
5.6
G-Motif
71.54109
70.3=1.5
81.4=2.2
60.1
73.1404
61,440.6
74.5=00.6
76.0+113
5,4
J0A0
73.4++1.2
64.7=0.8
66.1+3.7
60.7++10
74.8+00.
62.440.4
75.5+009
76.9+-111
5.0
GraphMAE
75.0:10.7
6..3=0.7
65.3+399
57.1
68.810.5
61.540.55
718=0.3
73.51:.88
8.3
SimGRRCE
7..8:10.4
6..7=0.6
66.8+:2.0
5844111
74.31.02
62.1+0.1
74.3=0.8
74.4+1.0
GraphMVP
80.3+144
67.9=10
76.42111
60.21116
74.61:0.
63,540.2
15.5=116
76.24:0.8
3.8
WAS
80.7+10.5
70.5=110
73.1=0.5
60.410.4
75,410.77
63.711.5
76.3=110
78,4
1.8
Table 4: Performance comparison with baseline teachers and other methods for node classification. The baseline tasks form the task pool of the multi-task methods, while single-task represents other complex designed tasks. The Blue in line WAS indicates that it beats all multi-task methods. The results of AGSSL are from its original paper, and the others are reproduced on our platform.
Pubmad
Paysics
Comput.rrs
ogon-arxiv
Avg Rank
CLU
$1.410.2
71.820.5
79.3:00.6
91.21:0.6
93.520.55
92.310.4
$7.6:10.6
71.52:66
11.1
buxelines
82,410.3
71.610.6
79.6:10.4
91.3::0.5
71.3:044
11.4
PAIRSIM
82.410.4
71882066
79.320.6
91.6:00.5
93.320.3
92.510.6
$7.110.9
71.220.3
9.9
PAIRDIS
81,9206
72.2:0.5
79.6::0.5
9144=0.5
94.020..6
92.1::0.5
86.520.7
714=0.3
10.6
DGI
82.11004
72.520.4
79.9:20.6
918810.7
93.910.6
92.010.6
$7.410.7
71.920.3
8.9
GRLC
$2.810.1
73.21:0
80.82002
89.920.2
93.2100.5
7102:0.7
single task
CG3
83.410.4
73.6:00.2
81.110.11
91.7.10.4
93.61:0.6
92.1:10.5
87,8.10.9
71,41.99
7.1
GCA
83,910.4
80.12:0.4
94.110.5
MVGRL
83.4110.4
72.7:00.6
8..6:1.8
91.7:10.2
0OM
93.1110.4
87,910.22
00M
6.3
GRACE
79.71:0.6
91.520044
94.520.4
OOM
mulli task
ParetoGNN
83.5++0.6
73.9-0.5
80.1+-02
92.3+-0.4
94.4+-0.6
93.44-0.2
87.9+0.3
71.0+0011
5.5
AutoSSL
93.9100.6
72.31:0.4
AGSSL
$4.2:10.33
73.620..6
80.6:20.3
92.410.5
94.820.3
93.32004
$8.710.4
72,5-0.3
3.1
WAS
84.4200.4
7..1::055
80.920.3
92.6=003
94.9:0.3
93.6:10.7
88.9100.3
71.7=0.3
1.8
80
Cintox
SIDER
70-
Perforaance
Pefformance
78-
75-
74.
#GCmmeestual
+JDAD
+GraphMVP
+AD-CCL
+84G11
*GConsectull
+JDND
+Grephetpp
+AD--CC
+84GL
+G-Molt
(a) Performance comparison on ClinTox
(b) Performance comparison on SIDER
Figure 4: Evaluation on whether the performance of WAS can improve as the task pool expands.
R-GCL (Li et al., 2022), GraphMVP (Liu et al., 2022), JOAO (You et al., 2021), GraphMAE (Hou al., 2022), SinGRACE (Xia al., 2022),G-Caa G-Contextual and G-Motif (Rong et al., 2020). On out of 8 datasets, WAS performs better than the other state-of-the-art methods, even though it combines only several simple but classical tasks. For node-level tasks (Table. 4), we compare WAS with other single-task methods, including GRLC (Peng et al., 2023), GCA (Zhu al., 2020c), MVGRL (Hassani Khasahmadi, 2020b), GRACE (Zhu et al., 2020a) and CG3 (Wan et al., 2020), and three multi-task methods, e.g., ParetoGNN (Ju al., 2022), AutoSSL (Jin et al., 2021) and AGSSL (Wu et al., 2022a). Here we would like to emphasize that the most important contribution of WAS is the handling of the compatibility, i.e., the continuous performance improvement that WAS achieve the number of tasks increases. To demonstrate this, we extend AGSSL to the graph-level tasks and compare it with WAS. As shown in Fig. 4, the performance of WAS improves consistently as the number of tasks gradually increases, but the performance of AGSSL hardly benefits from more tasks.
5.2 EVALUATION ON INSTANCE-LEVEI. RESULTS
A3: Customized combinations for different instances. To show the different selecting results for different instances, chose 4 molecular graphs from Bace and visualized the probability of being selected over them for each task in Fig. 5. We can see that the selecting module can select customized combinations for each instance even if we do pre-specify the number of selected teachers.
8

---

Published as a conference paper at ICLR 2024
Ax
Scleet Pobbbi by
9.0
38.
92
25 325 350
272 200
100 225
126
335
23
30
12
389 Fporn
22
20
25
150
372
230
Epoch
Eoech
Epoch
(a) Select IG
(b) Select AM
(c) Select GM&GraphLoG (d) Select three teachers
Figure 5: Probability of being selected for different teachers (tasks) on different instances from Bace
lt.
Sdoot 8
81:
500
223
150 373 348
37
72
389
0
25
350 372 250
50 15
100 12
Epech
Spocn
spoch
Spoch
(a) Selecting on Bace
(b) Weighing on Bace
(c) Selecting on Toxcast (d) Weighing on Toxcast
Figure 6: Comparison of the selecting probabilities and importance weights for different instances. (a)(b) selecting IG (rangking third importance) as the teacher for an instance from Bace, (c) selecting GraphCL. and CP (ranking fourth importance) as teachers for an instance from Toxcast.
A4: Evolution process of selecting and weighing. Here, we select 2 molecular graphs from Bace and Toxcast to visualize the results of selecting and weighing. It can be seen from Fig. 6 that the selected teacher not necessarily the teacher with the highest weight, which confirms our earlier statement that two modules need to be decoupled to deal with two issues of the importance and compatibility separately, and also demonstrates the effectiveness of our decoupled siamese networks.
5.3
ABLATION STUDY
A5: Ablation study on each compo- nent. The results in Table. 5 provide some constructive insights: (1) The performance of Importance-based se- lecting can be even lower than that of Random selecting. This confirms our claim that the compatibility is- sue needs to be considered separately from the importance issue. (2) Re- moving the projection head (imple- mented as MLP) would be detrimental to performance,
Table 5: Mean and standard deviation of ROC-AUC(%). Ran- dom means randomly selecting teachers. All means selecting all teachers. Importance-base based means selecting the top-3 important teachers. All of them use learned weights.
BACE
EIV
Tox21
BBBP
Random All Importance-bused
77.2+1.7
76.1=1.2
74.3++0.6
67.2+0.5
75.2111.2
0.7
73.61009
6..9:0.4
75.410.7
73.5=1.2
74.01:0.4
64.410.9
WAS w/o MLP
1..4+1.1
75.4=099
73.9=:0.8
69.6:112
WAS w/o re-weighing
65.1+3.7
63.3-2.4
64.1+3.00
60.7=5.1
it would increase the risk of coupling the result of se- lecting with weighing, which highlights the role of the projection head in the selection module. (3) WAS outperforms all other selecting strategies, which indicates that proper selecting strategy can greatly improve performance. (4) Re-weighing after selection is very important. If we remove the re- weighiiin the final sum of the teacher weights will not equal 1, which may result in the number of selected teachers having significant impact on performance. Due to space limitations, some more related experiments on hyperparameter sensitivity and teacher selection are placed in Appendix F.
WAS
80.7+0.55
78.4=1.9
75.4:00.7
70.5=10
CONCLUSION
With hundreds of graph pre- pre-training tasks proposed, combining the information drawn by multiple pre- training tasks has become a hot research topic. In this paper, we identify two important collaborative processes for this combination operation: weigh and select; we provide extensive experiments to explain why the collaboration of the two is important, how differs from the weighing-only based methods and why do the two need to be decoupled. Furthermore, we propose a novel decoupled framework WAS that is capable of customizing a selecting & weighing strategy for each downstream instance separately. Extensive experiments on 16 graph datasets show that WAS can not only achieve comparable performance to other leading counterparts by selecting and weighing several simple but classical tasks, but also achieve consistent performance improvements the task pool expands.

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGMENTS
This work was supported by National Key R&D Program of China (No. 20222D0115100, National Natural Science Foundation of China Project (No. U21A20427), and Project (No. WU2022A009) from the Center of Synthetic Biology and Integrated Bioengineering of Westlake University.
REFERENCES
Simon Axelrod and Rafael Koomez-oomarrelli Geom: Energy-annotace molecular conformations for property prediction and molecular generation. arXiv preprint arXiv:2006.05531, 2020.
Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2070-2079, 2017.
Kaitlyn M Gayvert, Neel Madhukar, and Olivier Elemento. A data-driven approach to predicting successes and failures of clinical trials. Cell chemical biology, 23(10) 2016.
Lee Giles, Kurt Bollacker, and Steve Lawrence. Citeseer: automatic citation indexing system. In Proceedings of the third ACM conference on Digital libraries, pp. 89-98, 1998.
William I. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning large graphs. arXiv preprint arXiv: 02216.02666... 2017.
Xucting Han, Zhenhuan Huang, Bang An, and Jing Bai. Adaptive transfer learning graph neural networks. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 565-5775777 574,,,44225 2021.
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International Conference Machine Learning, pp. 4116-4126. PMLR, 2020a.
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In Proceedings of International Conference on Machine Learning, pp. 3451-3461. 2020b.
Yun He, Steven Zheng, Yi Tay, Jai Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, Yaguang Li, Zhao Chen, Donald Metzler, Heng-Tze Tze Cheng, and Ed H. Chi. HyperPrompt: Prompt-based candicondiiinning of transformers. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 8678-8690. PMLR, 17-23 Jul 2022. URL httphttss//pootet pproceed wegeediiggg.....o ngs. mllr, www..ppessss.... / vv 1 62 22 html.
Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang. Chunjje Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 594-604, 2022.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitmik, Percy Liang, Vijay Pande, and Jure Leskovee. Strategies for pre-training graph neural networks. arXiv preprint arXiv: 10:00...2000 12265, 2019.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint wrXiv:2005.00687 2020a.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Conference Learning Representations, 2020b. URL https: /oppnrpprrv////e/eeeee ew. net forum? d=HJ] WHJSFDH.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 1857-1867, 2020c.
Po Yao Huang, Frederick Liu, Sz Rung Shiang, Jean Oh, and Chris Dyer. Attention- based multimodal neural machine translation. Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, 2016a
10

---

Published as a conference paper at ICLR 2024
Ruili Huang, Menghang Xia, Dac-Trung Nguyen, Tongan Zhao, Srilatha Sakamuru, Jinghua Zhao, Sampada A Shahane, Anna Rossoshek, and Anton Simeonov. Tox2 Nox2Ichallenge to build predictive models of nuclear receptor and stress response pathways mediated by exposure environmental chemicals and drugs. Frontiers in Environmental Science, 3:85, 2016b.
Wei Jin, Tyler Derr, Haochen Liu, Yiqi Wang, Suhang Wang, Zitao Liu, and Jiliang Tang. Self- supervised learning on graphs: Deep insights and new direction. arXiv preprint arXiv:2006.1014. 2020.
Wei Jin, Xiaorui Liu, Xiangyu Zhao, Yao Ma, Neil Shah, and Jiliang Tang. Automated self-supervised learning for graphs. arXiv preprint arXiv:2106.054700 2021.
Mingxuan Ju, Tong Zhao, Qianlong Wen, Wenhao Yu, Neil Shah, Yanfang Ye, and Chuxu Zhang Multi-task self-supervised graph neural networks enable stronger task generalization. arXiv preprint arXiv:2210.02016. 2022.
George Karypis and Vipin Kumar. A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM Journal on scientific Computing, 20(1):359-392, 1998.
Sunghwan Kim, Paul A Thiessen, Evan E Bolton, Jie Chen, Gang Fu, Asta Gindulyte, Lianyi Han, Jane He, Siqian He, Benjamin A Shoemaker, al. Pubchem substance and compound databases. Nucleic acids research, (((D1::11222-112133 2016.
Thomas N Kipf and Max Welling. Semi semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: htt:::669.0000 009002297 2016.
Michael Kuhn, Ivica Letunic, Lars Juhl Jensen, and Peer Bork. The sider database of drugs and side effects. Nucleic acids research, 44(D1):D105-D1009, 2016.
Sihang Li, Xiang Wang, An Zhang, Xiangnan He, and Tat-Seng Chua. Let invariant rationale discovery inspire graph contrastive learning. In ICML, 2022.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 14:18888-1889 2021a.
Pengbo Liu, Hailong Cao, and Tiejun Zhao. Gumbel-attention for multi-modal machine translation. arXiv preprint 103iv:2103.0886, 2021b.
Shengehao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre- training molecular graph representation with 3d geometry. In International Conference on Learning Representations, 2022. URL https: / hwtp///ipppeeeeeeeeeeeeeeeeeeeee
fooem/formmmm xQUelpOKE am.
Yixin Liu, Shirui Pan, Ming Jin, Chuan Zhou, Feng Xia, and Philip Yu. Graph self-supervised learning: A survey. arXiv preprint 2031v22033.00111, 2021c
Yi Luo, Aiguo Chen, Ke Yan, and Ling Tian. Distilling self- self-knowledge from contrastive links to classify graph nodes without passing messages. arXiv preprint arXiv:2/06.0084/, 2021.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint (XXi::6611.007/1 2016.
Ines Filipa Martins, Ana L Teixeira, Luis Pinheiro, and Andre O Falcao. A bayesian approach to in silico blood-brain barrier penetration modeling. Journal of chemical information and modeling, 52 1111686-1697,, 2012.
Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. Automating the construction internet portals with machine learning. Information Retrieval, 3(2):127-163, 2000.
Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. statistical perspective on distillation. In International Conference on Machine Learning, pp. 7632-7642. PMLR, 2021.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In leml, 2010.
11

---

Published as a conference paper at ICLR 2024
Liang Peng, Yujie Mo, Jie Xu, Jialie Shen, Xiaoshuang Shi, Xiaoxiao Li, Heng Tao Shen, and Xiaofeng Zhu. Grle: Graph representation learning with constraints. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-14, 2023. doi: N0.1109//NNLSS22222..2222
Zhen Peng, Yixiang Dong, Minnan Luo, Xiao-Ming Wu, and Qinghua Zheng. Self-supervised graph representation learning via global context prediction. arXiv preprint arXiv:2003.00 01604, 2020.
Zhongzheng Ren and Yong Jae Lee. Cross-domain self-supervised multi-task feature learning using synthetic imagery. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 762-771, 2017.
Ann M Richard, Richard S Judson, Keith A Houck, Christopher M Grulke, Patra Volarath, Inthirany Thillainadarajah, Chihae Yang, James Rathman, Matthew T Martin, John F Wambaugh, et al. Toxcast chemical landscape: paving the road to 21st century toxicology. Chemical research in toxicology, (28)::125-1251,
Sebastian G Rohrer and Knut Baumann. Maximum unbiased validation (muv) data sets for virtual screening based on pubchem bioactivity data. Journal of chemicul information and modeling, 49 (2):169-184) 2009.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. Advances Neural Information Processing Systems, 33, 2020.
Prithviraj Sen, Galilco Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rat Collective classification in network data. Al magazine, 29(3):93 93,9999999990 2008
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv: /8/ 1.05868,, 2018.
Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv: 2019.
Susheel Suresh, Pan Li, Cong Hao, and Jennifer Neville. Adversarial graph augmentation to improve graph contrastive learning. NeurlPS, 2021.
Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Rémi Munos, Petar Veličković, and Michal Valko. Bootstrapped representation learning on graphs. arXiv preprint 4rXiv:2102.06514, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua 2017.
Bengio. Graph attention networks. arXiv preprint
Petar Velickovic, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In ICLR (Poster), 2019.
Sheng Wan, Shirui Pan, Jian Yang, and Chen Gong. Contrastive and generative graph convolutional networks for graph-based semi supervieee learning. arXiv preprint 0xXx:22009.07111, 2020.
Zeyuan Wang, Qiang Zhang, HU Shuang-Wai Haoran Yu, Xurui Jin, Zhichen Gong, and Huajun Chen. Multi-level protein structure pre-training via prompt learning. In The Eleventh International Conference on Learning Representations, 2022.
Lirong Wu, Haitao Lin, Cheng Tan, Zhangyang Gao, and Stan 7. Li. Self-supervised learning on graphs: Contrastive, generative, or predictive. IEEE Transactions on Knowledge and Data Engineering, 2021.
Lirong Wu, Yufei Huang, Haitao Lin, Zicheng Liu, Tianyu Fan, and Stan Z. Li. Automated graph supe-supe learning via multi-teacher knowledge distillation, arXiv preprint arXiv:2210.02099, 2022a.
12

---

Published as a conference paper at ICLR 2024
Lirong Wu, Haitao Lin, Yufei Huang, and Stan Li. Knowledge distillation improves graph structure augmentation for graph neural networks. Advances in Neural Information Processing Systems, 35: 11815-11827, 2022b.
L Lirong Wu, Jun Xia, Zhangyang Gao, Haitao Lin, Cheng Tan, and Stan Li. Graphmixup: Improving daas-iibalaanee node classification by reinforcement mixup and self-supervised context prediction. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 519 535. Springer, 2022c.
Lirong Wu, Jun Xia, Haitao Lin, Zhangyang Gao, Zicheng Liu, Guojiang Zhao, and Stan Z Li. Teaching yourself: Graph self-distillation on neighborhood for node classification. arXiv preprint arXiv:2210.00 10X0002200.02097, 2022d.
Lirong Wu, Haitao Lin, Bozhen Hu, Cheng Tan, Zhangyang Gao, Zicheng Liu, and Stan Z Li. Beyond homophily and homogeneity assumption: Relation-baser based frequency adaptive graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2023a.
Lirong Wu, Haitao Lin, Yufei Huang. Tianyu Fan, and Stan Z Li. Extracting htwr/highh-feqqeenyy knowledge from graph neural networks and injecting it into mlps: An effective gnn-to-mip distillation framework. arXiv preprint arXiv:2305. 10758, 2023b.
Lirong Wu, Haitao Lin, Yufei Huang, and Stan Z Li. Quantifying the knowledge in gnns for reliable distillation into mlps. arXiv preprint arXivv2306.05668, 2023c.
Lirong Wu, Haitao Lin, Zihan Liu, Zicheng Liu, Yufei Huang, and Stan 7. Li. Homophily-enhance self-supervision for graph structure learning: Insights and directions. IEEE Transactions Neural Networks and Learming Systems, 2023d.
Lirong Wu, Cheng Tan, Zihan Liu, Zhangyang Gao, Haitao Lin, and Stan Li. Learning augment graph structure for both homophily and heterophily graphs. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 18. Springer, 2023e.
Sen Wu, Hongyang R Zhang, and Christopher Ré. Understanding and improving information transfer in multi-task learning. arXiv preprint arXiv:2005.00944 2020.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning.
Chemical science, 9(2):513-530, 2018
Jun Xia, Lirong Wu, Jintao Chen, Bozhen Hu, and Stan Z Li. Simgrace: A simple framework for graph contrastive learning without data augmentation. In Proceedings of the ACM Web Conference 2022, pp. 1070-1079, 2022.
Yaochen Xie, Zhao Xu, Zhengyang Wang, and Shuiwang Ji. Self-supervised learning of graph neural networks: unified review. arXiv preprint 102. 2021.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv: 100.::.110000000 2018.
Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph-level representation learning with local and global structure. In International Conference on Machine Learning, pp. 11548 11558.1588... PMLR, 2021.
Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning. pp. 40-48. PMLR, 2016.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Bal- can, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5812-5823, Curran Associates, Inc., 2020a. URL https: http::/psss/p...oppeeec....ee/ee paper/2020/F11 ottp::/e........2222222200030110
cc/
13

---

Published as a conference paper at ICLR 2024
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. When does self-super help graph convolutional networks? In International Conference on Machine Learning, pp. 10871- 10880. PMLR, 2020b.
Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In International Conference on Machine Learning, pp. 12121-12132. PMLR, 2021.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances Neural Information Processing Systems, 33: 5824-5836, 2020.
Daniel Zaharevitz. Aids antiviral screen data, 2015.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3712-3722, 2018.
Hanlin Zhang, Shuai Lin, Weiyang Liu, Pan Zhou, Jian Tang, Xiaodan Liang, and Eric Xing. Iterative graph self-distillation. arXiv preprint arXiv:2010. /2609, 2020.
Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. Graph-less neural networks: Teaching old mlps new tricks via distillation. arXiv preprint arXiv:211100872,,022 2021.
Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15659-15669, October 2023.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep Graph Contrastive Representation Learning. In ICML Workshop on Graph Representation Learning and Beyond, 2020a. /ttp://t:/
http/aaxiivv..a///////ee//aaaala org/ abs /000 04131.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint 20Xv:2006.044333 2020b.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. arXiv preprint 149445,,00.14 2020c.
Daniel Zügner and Stephan Günnemann. Adversarial attacks on graph neural networks via meta learning. arXiv preprini arXiv: 190.00842, 2019.
14

---

Published as a conference paper at ICLR 2024
Appendix
A. DETAILS ON PRE-TRAINING TASKS
Here, we provide a high-level comparison among existing graph pre-training tasks in Table. Al.
Generative tasks: The generative tasks focus on the intra-data information embedded in the graph and aim to reconstruct the important structures or features for each graph. By doing so, learns better representation that encodes the key components of the data.
The contrastive tasks handle inter-data information by first applying transfor- mations to construct different views for each graph. Each view contains information at different granularity. The training objective is to align the representations of views from the same data (positive pairs) and pull apart the representations of views from different data (negative pairs).
Contrastive tasks:
The predictive methods generally self-generate labels by some simple statistical analysis or expert knowledge and then perform prediction-style tasks based on self-generated labels.
Predictive tasks:
Table Al: Comparison between existing pre-training tasks.
Pre-training Tasks
Generative
Contrastive
Predictive
EdgePred (Hamilton et al., 2017)
AttrMask (Hu et al., 2019)
GPT-GNN (Hu et al., 2020c)
GraphMVP-C G (Liu et al., 2022)
DGI (Velickovic et al., 2019)
InfoGraph (Sun et al., 2019)
ContextPred (Hu et al., 2019)
G-Contextual (Rong et al., 2020)
GraphCL (You et al., 2020a)
GraphLoG (Xu et al., 2021)
AD-GCL (Suresh et al., 2021)
JOAO (You al., 2021)
SimGRACE (Xia et al., 2022)
R-GCL (Li al., 2022)
GraphMVP-C et al., 2022)
GraphMAE (Hou et al,, 2022)
PAIRSIM (Jin et 2020)
PAIRDIS (Peng et al., 2020)
CLU (You et al., 2020b)
PAR (You et al., 2020b)
G-Motif (Rong al., 2020)
Baseline. For graph-level classification tasks, we take seven classic graph pre-training tasks to make up task pool. (1) AttrMask (Hu et al., 2020b), which learns the regularities of node/edge attributes. (2) ContextPred (Hu et al., 2020b), which explores graph structures by predicting the contexts. (3) EdgePred (Hamilton et al., 2017), which predicts the connectivity of node pairs. (4) GPTGNN (Hu et al., 2020c), which introduces an attributed graph generation task to pre-train GNNs. (5) GraphLoG (Xu et al., 2021), which introduces a hierarchical prototype capture the global semantic clusters. (6) GraphCL (You et al., 2020a), which constructs specific contrastive views of graph data. (7) InfoGraph (Sun et al., 2019), which maximizes the mutual information between the representations of the graph and substructures. For node-level classification tasks, we follow AutoSSL (Jin et al., 2021) to adopt five classic tasks. (1) DGI (Velickovic et al., 2019), which maximizes the mutual information between graph representation and node representation. (2) CLU (You et al., 2020b), which predicts pseudo-labeld from K -menss clustering on node features. (3) PAR (You et al., 2020b), which predicts lseudo-labels from Metis graph partition (Karypis & Kumar, 1998). (4) PAIRSIM (Jin et al., 2020), which predicts pairwise feature similarity between nodes. (5) PAIRDIS (Peng et al., 2020), which predicts the shortest path length between nodes.
15

---

Published as a conference paper at ICLR 2024
B. PSEUDO CODE
The pscudo-code of WAS is summarized in Algorithm (node-level) and Algorithm 2 (graph-level).
Algorithm 1 Algorithm of the WAS framework for node-level tasks
Input: K Pre-training Tasks Ti, T2, .. TR, Graph G = (A,X) with N nodes, and Number of Epochs:
Iteration.
Output: GNN Encoder h-(-) and Projection Head (.)..) Use K pre-training tasks to pre-train GNN encoder to get teacher models' parameters (0 Wamr
for iter 1,
Iteration do
for 1, ... do
Save the output {P(k, 2-; (hog (x,)) on from the pre-trained teachers and then freeze them.
Input (kk, a) into Weighing and Selecting module to obtain the weights Arnis (k, i) and selecting results <(k;,)) i) of each teacher.
Weigh those selected teachers with weights A(k, i) a(k.,)rar(()nneeee.)))
=i== mik.))esp(Nnnstkkk.)))
Integrate the knowledge of different teachers by p" (x:) Ek
D==1 ((K,
Distill the integrated knowledge to the student model by Eq. (5).
end for
end for
Return: Student's Enocder ho(-) and Prediction Head g-((-).
Algorithm 2 Algorithm of the WAS framework for graph-level tasks
Input: K Pre-training Tasks Ti, T2, ... Tk. a set of graph (9, " GMs J. and Number of Epochs:
Iteration.
Output: GNN Enoeder he(-) and Projection Head (-).
Use K pre-trainne training tasks to pre-train GNN encoder to get teacher models' parameters (0) W
for iter = 1, 2, ... Ittration do for i 1, ... .M do
Save the output (P(k. r) ght (heg (G:))) from the pre-trained teachers and then freeze them. Input P(k, i) into Weighing and Selecting module to obtain the weights Asnit (k, i) and selecting results <(k, of each teacher.
Weigh those selected teachers with weights A(k; i) = edb..)esp((Anss ΣR (1
Integrate the knowledge of different teachers by (G.) Ek ((K,,
Distill the integrated knowledge to the student model by Eq. (5).
end for
for
Return: Student's Enocder ho(-) and Prediction Head (-)()).
C. COMPLEXITY ANALYSIS
(1) [Few additional parameters] Despite the additional parameters introduced, WAS achieves similar computational efficiency as previous works, because the training with multiple tasks is more hard to optimize than the training with one single task. The two processes, weigh and select, actually only 2KT(C' 1) + 2 more parameters (I is teacher numbers, K is the dimension of the label distribution, and is the dimension of latent space), which acceptable, as T is usually small.
(2) [Few additional memory] Since we adopt multi-teacher architecture, we can not only train multiple teachers in parallel, but also save only their outputs instead of the full model parameters.
(3) [Few additional time consumption] In Table. A2, demonstrate the training time (in the pre-training and fine-tuning setting) single method (Clu and DGI) and WAS (train parallel) here. Specifically, to illustrate that the two processes, weigh and select, will not cause unacceptable time consumption, we additionally compare the case without them.
(4) [Few additional Time complexity] The time complexity of WAS is based on: 1. Training teachers: [[[[][]]]= (FFNNdF) N the number of instances, ||A||] is the number of non-zeru values in the adjacency matrix, and and F are the dimensions of input and hidden spaces. 2. Knowledge
16

---

Published as a conference paper at ICLR 2024
integrating: O((N + 1)FT). The overall complexity is still determined by the first term. Moreover, we can reduce the complexity of training teachers to O(||A|FF - _NdF) by parallel training.
Table A2: Training time of single method (Clu and DGI) and WAS (train in parallel).
Cora
Citeseer
Photo Pumbed
6.75s
9.57s
28.39s
11.95s
DGI
9.18s
9.266
26.23s
12.06s
WAS (w/o weight&weigh) WAS (w/o select) WAS
11.73s
12.19s
32.17s
15.79s
12,04s
12.61s
32.82s
16.23s
12.97s
12.76s
34.07s
16.78s
D. DATASET STATISTICS
In this section, we summarize 16 datasets used in this paper.
Dataset about Pharmacology: The Blood-Brain Barrier Penetration (BBBP) (Martins al., 2012) dataset measures the penetration properties of molecule into the central nervous system. The Side Effect Resource (SIDER) (Kuhn et al., 2016) dataset represents the adverse drug reactions. Tox21 (Huang et al., 201 6b), ToxCast (Richard et al., 2016), and C1 inTox (Gayvert et al., 2016) are related to the toxicity of molecular compounds.
Dataset about Biophysics: Maximum Unbiased Validation (MUV) (Rohrer & Baumann, 2009) using refined nearest neighbor analysis to construct based on PubChem (Kim et al., 2016). HIV (Zaharevitz, 2015) aims predicting inhibit HIV replication. BACE, which is gathered in MoleculeNet (Wu et al., 2018), measures the binding results for inhibitors 3-secretase 1. Following the setting of (Liu et al., 2022), scaffold splitting to split datasets into train/val/test sets by ratios of 0.7/0.2/0.1.
Table A3: Statistical information of the graph-level datasets.
Dataset
BACE
BARP
CI inTTo
S1 Sider
Tox21
ToxCast
MUV
RTV
# Tasks
12
617
17
1
# Molecules
1,513
2,039
1,478
1,427
7,831
8,576
93,087
41,127
Dataset about Citation network: Cora (Sen et al., 2008), Citeseer (Giles al., 1998), -gxnn (Hu et al., 2020a) and Pubmed (McCallum et al., 2000) consists of papers from different fields and each node a scientific paper.
Dataset about Co-authorship network: Coauthor-CS (CS) and Coauthor-Physies (Physics) (Shchur et al., 2018) are Co-authorship network datasets consisting of undi- rected graphs, where nodes represent authors and are connected by edges if they are co-authors. Different authors have different research areas.
Amazon-Phote (Photo) and Amazon Compuess (Computers) (Shchur et al., 2018) are datasets consisting of an undirected graph, where the nodes represent goods and the edges represent goods that often purchased at the same time. When we evaluate the performance of node classification, we need to use the training and testing data. For Cora, Citeseer, and Pubmed, follow the data splitting strategy in (Kipf & Welling, 2016). For CS, Physics, Photo, and Computers, we follow (Zhang et al., 2021; Luo et al., 2021) to randomly split the data into train/val/test sets, and each random seed corresponds to a different splitting. For ogbn-arxiv, we use the public data splits provided by the authors (Hu et al., 2020a).
Dataset about Co-purchase network:
E. DETAILS ON EXPERIMENTAL SETUP
Experimental settings. For graph-level tasks, we follow the setting of GraphMVP (Liu et al., 2022). We are-trained all methods on the same dataset based on GEOM (Axelrod Komez-Bombarellle 2020) using the GIN (Xu al., 2018) the backbone model, and then fine-tune them on
17

---

Published as a conference paper at ICLR 2024
Table A4: Statistical information of the node-level datasets.
Dataset
Citeseer
Pubmed
Photo
Phys Computers
ogbn-a
# Nodes
2,708
3,327
19,717
7,650
18,333
34,493
13,752
169,343
# Edges
5,278
4,614
44,324
119,081
81,894
247,962
245.861
1,166,243
1 Features
1,433
3,703
745
6,805
8,415
128
# Classes
8
classical molecular property prediction datasets. For fair comparison, the model with the highest validation accuracy selected for testing. For node-level tasks, we follow the settings in AGSSL. (Wu al., 2022a) exactly (search space of hyperparameter, model architecture, experiment setting, etc.) for fair comparison. For pre-training, we pre-train the GNN encoder (using GCN (Kipf & Welling, 2016)) with pre-training tasks and for fine-tuning, we use the pre-trained GNN encoder with projection head under the supervision of specific downstream task. We conduct our experiments on NVIDIA Tesla V100 GPU and we use Intel(R) Xeon(R) Gold 6240R @ 2.40GHz CPU.
Hyperparameter Settings. The following hyperparameters are set the same for all molecular datasets: Adam optimizer with weight decay Toxcast). The other hyperparameters settings (including loss weight temperature momentum rate m, and learning rate Ir) are included in Table. A5. The following hyperparameters are set the same for node classification tasks: Adam optimizer with weight decay = 5e 4; Epoch E = 500. The other hyperparameter settings (including hidden dimension F)) are included in Table. A6.
Epoch E 250 (E 500 for ClinTox and
Table A5:
Hyperparameters of graph-level tasks.
BACE
BB3?
ClinTox
SIDER
Tox21
Toxcast
3.6
3.3
4.2
2.7
4,2
3.5
1.1
4.9
1.3
1.6
1.5
1.5
1.5
0.9
0.9
0.9
0.5
0.1
0.5
0.5
0.3
Ir
0.01
0.01
0.001
0.01
0.01
0.001
0.01
0.01
Table A6: Hyperparameters of node-level tasks.
Cora
Cf teseer
Pismmed
CS
Physica
Photoo
Computers
ogbn-arxiv
F
128
256
512
64
256
64
128
512
20
0.5
1
1.2
1
1
1.5
1.2
2
1.2
m0
0.5
0.3
0.7
0.7
0.7
b-
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.001
18

---

Published as a conference paper at ICLR 2024
F. MORE RELATED EXPERIMENTS
EVALUATION ON HYPERPARAMETER SENSITIVITY
The hyperparameter sensitivity on the loss weight a is provided in Fig. A1, from which we can make two observations that (1) As the loss weight a mncrease, the model performance first improves and then decreases, which suggests that knowledge transferring helps improve performance, but too large can cause the loss directly associated with the downstream task to be masked, causing performance degradation. (2) Larger a leads to smaller variance, probably because combining the knowledge of multiple pre- rre-training tasks effectively improve the stability of the student model.
Bace
ВВВР
69
ROC-AUC (%)
ROC-AUC (%6
68
78
67
66
65
64
0.1
0.5
1.0
10
50
0.1
0.5
1.0
10
50
Hyperparameter
Hyperparameter
(a) Hyperparameter sensitivity Bace
(b) Hyperparameter sensitivity on BBBP
Figure Al: Hyperparameter sensitivity analysis on the loss weight a.
EVALUATION ON TEACHER SELECTION
In Table. A7, we count the average number selected teachers, and then count the ratio of selecting lel///ooonuumblll important teacher. From the results reported in Table. A7, we can draw the following conclusions: (1) teachers with the highest weights have higher probability of being selected, and (2) the selection module does not necessarily always select the higher-weight teachers.
Table A7:
Average number of teachers being selected, and the ratios Hoop-//Tp--uee (numberr important teachers. The selection module does not necessarily always select the higher-importance teachers.
Cora
Citeseer
BACE
B3BP
average number of selected teachers ratios selecting Top-1 important teacher (%)
3.9
3.1
0.63
0.71
0.68
0.55
ratios of selecting [opp-number] important teacher (%)
0.26
0.34
0.41
0.10
19


Reviews:
[{"review_id": "EgO6oTgxBY", "reviewer": "Reviewer_fcHp", "summary": "Since the roles of pre-training tasks vary with downstream tasks and different pre-training tasks may not be compatible, the authors point out that selecting and weighting tasks are key parts of the pre-training. Moreover, the existing methods that select tasks based on learned weights would confuse the roles of selection and weighting. To address these limitations, they propose WAS to decouple these two processes.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed WAS decouples the selecting and weighting to avoid performance reduction caused by task conflicts. They calculate weights based on output distributions instead of losses to address the non-comparability between different loss functions.\n2. The proposed method can automatically select the number and type of suitable pre-training tasks for different downstream instances.\n3. The authors conduct extensive comparison experiments, and the proposed WAS performs better on both node-level and graph-level tasks. In addition, they visualized the evolution processes of selecting tasks and updating weights, which proves the decoupling of selection and importance weighting.", "weaknesses": "1. The authors should add more ablation experiments to demonstrate the effectiveness of the proposed model, such as the role of different updating methods in decoupling the selecting and weighting outputs, and the role of reweighting after selection. \n2. The comparison model in A5 needs more description to distinguish it from A1, such as whether Random-Select and ALL use learned weights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PNO67APYD9", "reviewer": "Reviewer_C6hu", "summary": "In recent times, graph pre-training for graph representation learning has gained prominence, with numerous pre-training tasks emerging. Integrating knowledge from various pre-training tasks is now a focal research area. Two critical collaborative processes for integration are: (1) selecting the best task combination considering their compatibility and (2) determining the importance of the chosen tasks. While much research has been on weighing tasks, selection has received lesser attention. This paper introduces a new instance-level framework named Weigh And Select (WAS) that merges both processes using decoupled Siamese networks. WAS adaptively determines the best task combination for individual instances, leading to a tailored instance-level task weighing strategy. Experiments across 16 graph datasets reveal WAS's efficacy, producing results comparable to top-performing methods by merging several basic tasks.", "questions": "Please see weaknesses", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. The authors of the paper have done a good job in providing a compelling motivation for their research. Their thorough analysis of different pre-training tasks combined with a detailed examination of several datasets showcases their comprehensive approach. Furthermore, their assessment of task importance and compatibility provides valuable insights, shedding light on the central issue at hand.\n\nS2. In terms of addressing the research concerns, the authors have carefully identified and highlighted them. These concerns have been formulated into four well-defined research questions that give readers a clear roadmap of the study's objectives. On a broader note, the manuscript is well-organized, with a structured flow that facilitates easy comprehension, making the writing lucid and clear to the audience.\n\nS3. With respect to the empirical aspect of the study, the authors have presented an exhaustive set of experimental results. These results showcase outcomes that are indeed promising.", "weaknesses": "W1. The concept of instance is not well defined. Different pre-training tasks may have different definitions of instances fundamentally (eg. at a node-level, subgraph/graph-level, edge-level etc). Having a common and converged definition may not work well. Furthermore, downstream task could also have different definitions of instances.\n\nW2. While 4 questions are clear in the identification of the 4 steps/issues to address, the solutions are quite standard. E.g. using Gumbel-Softmax sampling for Bernoulli distribution,  the weighting scheme and the Siamese network architecture are all well known tools.\n\nW3. The compatibility issue, or interferences among tasks have been observed in previous work in other areas or problem settings [a,b,c]. Some discussion on this aspect, and its particular challenges in graph context, would further strengthen the motivation of the paper. \n\n[a] Zeyuan Wang, Qiang Zhang, HU Shuang-Wei, Haoran Yu, Xurui Jin, Zhichen Gong, and Huajun Chen. 2022. Multi-level Protein Structure Pre-training via Prompt Learning. In The Eleventh International Conference on Learning Representations.\n[b] Sen Wu, Hongyang R Zhang, and Christopher Ré. 2020. Understanding and improving information transfer in multi-task learning. arXiv preprint arXiv:2005.00944 (2020).\n[c] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems 33 (2020), 5824–5836.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cxxGWOmNmj", "reviewer": "Reviewer_pHBL", "summary": "In this paper, the authors have studied how to effectively integrate multiple graph pre-training tasks. They have identified two important collaborative processes, i.e., selecting and weighing.  They propose a new instance-level framework for integrating multiple graph pre-training tasks, named WAS (Weigh And Select), where the weighing and selecting processes are combined by decoupled siamese networks. Extensive experiments on 16 graph datasets have been performed to demonstrate the effectiveness of the proposed method.", "questions": "1. On average, how many tasks are usually selected for each instance?\n\n2. From Figure 6(d), we can observe that the following 4 tasks have the largest weights, i.e, EP, AM, IG, and CP. How about we only use these 4 tasks and learning the their weights for prediction?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. In this paper, the authors introduce a new framework, namely WAS, for task selecting and importance weighing to integrate multiple graph pre-training tasks.\n\n2. The authors show the limitations of existing weighing-only schemes and demonstrate the importance of task selecting process.\n\n3. The authors have performed extensive experiments to demonstrate the effectiveness of the proposed method. \n\n4. This paper is clearly written and easy to follow.", "weaknesses": "1. In Table 1, 2, and 3, the authors do not compare the proposed WAS framework with existing frameworks that only focus on weighing (i.e., AutoSSL, ParetoGNN, AUX-TS, and AGSSL). However, in Table 4, WAS is compared with these methods. This makes the experimental settings not consistent.\n\n2. The complexity of the proposed has not been studied. Although the proposed method can achieve some performance improvement, it may take more training/inference time, due to combining multiple pre-training tasks. The authors need to have some analysis about the complexity of the proposed method.\n\n3. The authors also need to perform experiments to analyse how many tasks are selected for each instance on average.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g9827mwDlo", "reviewer": "Reviewer_66Ho", "summary": "This paper focus on multi-tasking graph pre-training and proposes a weighting and selecting network to model the compatibility and importance of tasks. The proposed WAS consists of knowledge extraction and transfer step, and it is powered by the decoupled siamese networks to assign weights to each teacher and do selection. Experiments are conducted on different benchmark datasets to showcase its effectiveness.", "questions": "1. The proposed method does not treat graph-structured data specially, nor is it tailored for graph modeling. Why is it positioned for pre-training on graphs, instead of for general pre-training?\n\n2. Any in-depth understanding of the relationship for the selected tasks? Are they independent?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n\n2. The evaluation is conducted on both graph-level and instance-level tasks.\n\n3. The idea on solving the compatibility issue of multiple tasks is new and insightful.", "weaknesses": "1. The motivation of focusing on graph-structured data is unclear.\n\n2. Compared to the baseline methods for multi-task learning on instance-level, the improvement from WAS is marginal. On the largest graphs in the experiment (ogbn-arxiv), it achieves worse performance compared to baseline method.\n\n3. Besides the empirical results, the theoretical analysis is still needed to answer why both selecting and weighing is needed,  and why there is compatibility issue (i.e., why some tasks shouldn't be selected).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)