Learning to Transfer via Modelling Multi-level Task Dependency
Authors: Haonan Wang
Zhenbang Wu
Ziniu Hu
Yizhou Sun
Venue:  (2020)
Keywords: multi-task learning
attention mechanism

Abstract: Multi-task learning has been successful in modeling multiple related tasks with large, carefully curated labeled datasets. By leveraging the relationships among different tasks, multi-task learning framework can improve the performance significantly. However, most of the existing works are under the assumption that the predefined tasks are related to each other. Thus, their applications on real-world are limited, because rare real-world problems are closely related. Besides, the understanding of relationships among tasks has been ignored by most of the current methods. Along this line, we propose a novel multi-task learning framework - Learning To Transfer Via Modelling Multi-level Task Dependency, which constructed attention based dependency relationships among different tasks. At the same time, the dependency relationship can be used to guide what knowledge should be transferred, thus the performance of our model also be improved. To show the effectiveness of our model and the importance of considering multi-level dependency relationship, we conduct experiments on several public datasets, on which we obtain significant improvements over current methods.

Under review as a conference paper at ICLR 2020
LEARNING TO TRANSFER VIA MODELLING MULTI- LEVEL TASK DEPENDENCY
Anonymous authors Paper under double-blind review
ABSTRACT
Multi-task learning has been successful in leveraging the dependency among dif- ferent tasks to improve general performance. However, when applying 'dis- crete' data (graph/text), most of the existing frameworks only leverage the general task dependency with the assumption that the task dependency remains the same for (I) different data samples; and (2) different sub-structures (node/word) in data sample (graph/text). Yet, this assumption may not be true for real-world 'discrete" datasets. Thus, we propose novel multi-task learning framework Learning To Transfer Via Modelling Multi-level Task Dependency, which lever- ages both general task dependency and data-specific lask dependency to guide the knowledge transfer among tasks. We also propose decomposition method to reduce the space complexity needed by transfer functions from quadratic to lin- car. To show the effectiveness of our framework and the importance of modelling multi-level task dependency, we conduct experiments on several public graph/text datasets, which we obtain significant improvements over current frameworks.
INTRODUCTION
Multi-task learning Caruana 1997) aims to train a single model on multiple related tasks jointly, so that useful knowledge learned from one task can be transferred to enhance the generalization performance of other tasks. Over the last few years, different types of multi-task learning mecha- nisms (Sener & Koltun 2018 Guo & Farooq 2018 Ish 2016 Lon 2015) have been proposed and proved better than single-task learning methods from natural language processing (Palmer et al. 2017) and computer vision (Cortes et al.
2015 chemical study (Ramsundar al. 2015).
Despite the success multi-task learning, when applying to 'discrete' data (graph/text), most of the current multi-task learning frameworks (Zamir et al. 2018 Ish 2016) only leverage the general task dependency with the assumption that the task dependency remains the same for (1) different data samples; and (2) different sub-structures (node/word) in one data sample (graph/text). However, this assumption is not always true many real-world problems.
Different data samples may have different task dependency, For example, when we want predict the chemical properties of a particular toxic molecule, despite the general task dependency, its representations learned from toxicity prediction tasks should be more significant than the other tasks.
(2) Even for the same data sample, different sub-structures may have different task depen- dency. Take sentence classification as example. Words like 'good' or 'bad' may transfer more knowledge from sentiment analysis tasks, while words like 'because' or "so" may transfer more from discourse relation identification tasks.
In this work, to accurately learn the task dependency in both general level and data-specific level, we propose novel framework, 'Learning to Transfer via Modelling mulTi-level Task dEpeNdency' (L2T-MITTEN). The general task dependency is learned graph. And the data-specific task dependency is learned with the position-wist -ist mutual attention mechanism. The two-level task dependency can be used by framework to improve the perfor- mance on multiple tasks. And the objective function multi-task task learning can further enhance the quality of the learned task dependency. By iteratively mutual enhancement, our framework can not
parameterized weighted dependency

---

Under review as a conference paper at ICLR 2020
only perform better multiple tasks, but also extract high-quality dependency structures at different levels, which can reveal some hidden knowledge of the datasets.
Another problem is that to transfer task-specific representations between every task pair, the number of transfer functions will grow quadratically as the number of tasks increases, which is unaffordable. To solve this, we develop universal representation space where all task-specific representations get mapped to and all target tasks can be inferred from. This decomposition method reduces the space complexity from quadratic to linear.
We validate our multi-task learning framework extensively on different tasks, including graph classi- cation, node classification, and text classification. Our framework outperforms all the other state-of- the-art (SOTA) multi-task methods. Besides, we show that L2T-MITTEN can be used as an analytic tool extract interpretable task dependency structures at different levels on real-world datasets.
Our contributions in this work are threefold:
We propose novel multi-task learning framework to learn to both general task dependency and data-specific task dependency. The learned task dependency structures can be mutually enhanced with the objective function of multi-task learning.
We develop a decomposition method to reduce the space complexity needed by transfer functions from quadratic to linear.
We conduct extensive experiments on different real-world datasets to show the effectiveness of our framework and the importance of modelling multi-level task dependency.
2 RELATED WORK
According to recent survey (Ruder 2017), existing multi-task learning methods can be categorized by whether they share the parameters hardly or softly.
2.1 HARD PARAMETER SHARING
For hard parameter sharing, a bottom network will be shared among all the tasks, and each individual task will have its own task-specific output network. The parameter sharing of the bottom network reduces the parameter needed to be learned and thus can avoid over-fitting to a specific task. How- ever, when the tasks are not relevant enough (Sha 2002 Baxter 2011), the shared-botton layers will suffer from optimization conflicts caused by mutually contradicted tasks. If the bottom model is not capable enough to encode all the necessary knowledge from different tasks, this method will fail to correctly capture all the tasks.
Besides, Dy & Krause 2018) points that the gradients of some dominant task will be relatively larger than gradients of other tasks. This dominant phenomenon will be more obvious when the proportions of labeled data between tasks are uneven, in which case the model will be majorly optimized on data-rich tasks. To alleviate this problem, some recent works (Sener & Koltun 2018 Dy & Krause 2018 try to dynamically adjust the task weight during the training stage. Sener & Koltun (2018) casts the multi-task learning to multi-objective optimization problem, and they use gradient-baser optimization method find Pareto optimal solution. Dy Krause (2018) proposes a new normalization method on gradients, which attempts to balance the influences of different tasks. Recently, [uuo & Farooq (2018) proposes to apply Mixture-of -x-eets on multi-task learning, which linearly combines different experts (bottoms) by learnable gates. Because different experts can capture different knowledge among tasks, this model can, to some extent, model the dependency relationship among tasks.
2.2 SOFT PARAMETER SHARING
Methods using soft parameter sharing (Lon 2015 Ish 2016 Dai et al. 2015 Yan 2017) do not
keep the shared bottom layers. Instead, for soft-parameter models, most of the model parameters tasc-fpe Lon 2015) focuses reducing the annotation effort of the dependency parser tree. By combining two networks with a L2 normalization mechanism, knowledge from a different source language can be used reduce the requirement of the amount of annotation. Further, in some
2

---

Under review as a conference paper at ICLR 2020
existing works, the shallow layers of the model will be separated from other layers, and be used as the feature encoders to extract task-specific representations. For example, Ish (2016) proposes a Cross- Stitch model which is typical separate bottom model. Cross-Stitch model will be trained on different tasks separately to encode the task-speeific representations from different bottom layers. Then, cross-stitch unit is used introduces the tensor factorization model to allow common knowledge to be shared at each layer in the network. By the strategy proposed in their work, parameters are softly shared across the corresponding layers of the deep learning network and the parameter sharing ratio will be determined by the model itself.
gate combine those separately trained layers. Yan (2017)
We also note that some recent works (Zamir et al. 2018 Lan et al. 2017 Liu et al., 2018) can learn to capture task dependency. Zamir al. (2018) computes an affinity matrix among tasks based on whether the solution for one task can be sufficiently easily read out of the representation trained for another task. However, can only capture the general task dependency. [Lan et al. (017) uses sigmoid gated interaction module between two tasks to model their relation. But will suffer from quadratic growth in space as the number of tasks increases. Liu et al. (2018) utilizes shared network to share features across different tasks and uses the attention mechanism to automatically determine the importance of the shared features for the respective task. However, there is knowledge transferred interaction between tasks.
3 APPROACH
In this section, propose our framework L2T-MITTEN, which can end-to-end learn the task de- pendency in both general and data-specific level, and help to assemble multi-task representations from different task-specific encoders.
3.1 PROBLEM FORMULATION
To formulate our framework, we first start by briefly introducing a general setting of multi-task
learning For each task (I. we have corresponding dataset {(x, (1) (.) (( with (1) data samples, where x represent the feature vector of k-th data sample and Vı is its label. We would like train T models for these tasks, and each model has its own parameter W Note that for different multi-task learning frameworks, these parameters can be shared hardly or softly. The goal of multi-task learning to improve general performance by sharing information among related tasks. The total loss of multi-task learning is calculated as:
(t)
T
(2 E L(W(O) x("), Uk (1) N(0
min
(1)
3.2
ARCHITECTURE OVERVIEW
Tisk: Eacoden(E.)
Taak E. (XQ)
Thol Internction
Taok
Trsk: RendOStt
Labeli::
Asszmbial Repreertation
Toxicity
Tank Enosden
Tasks Eg (C,)
Task: Intercction
Thok ReaeOue
Labeh: Ouudoobilky
Assebiled Rey csotttiion
E,(X,)
Tacck (X,)
Tiaks Intenstion Unit
Toskr BaatOut
Labelr: Solubiitt
FinadenT.)
Aoambled exmmatii
task-ppeclii
Eneaders
Transfer Bleck
Readout Bleck
Figure 1: The overall architecture of our multi-task learning framework (122T-MTTTE), For each input data, will transfer its task- representations among different tasks, and assemble the transferred representations via a task -seciici Interaction Unit to get the final representation for each task.

---

Under review as a conference paper at ICLR 2020
As is shown in Figure framework consists of three components: Task-specific Encoders, Transfer Block, and Readout Block.
The Task-specific Encoders consists Tf separate feature encoders, which can be any type of feed-forward networks based on specific data. Unlike hard parameter sharing methods that tie the bottom encoders' parameters together, keep each feature encoder separate efficiently extract task-specific knowledge. In this way, for given data sample Xs, we can use these encoders to get task-specific representations {E, (Xx)))=> where E, (Xx) is the representation of X₁ for task t.
To conduct multi-task learning, one can simply use the representation of each task alone to predict labels without sharing any parameters. However, this model will suffer for tasks without sufficient labeled data. Therefore, we would like to (1) transfer the knowledge among these I tasks and (2) assemble the transferred representations, which is what we do in the Transfer Block.
The Readout Block also consists of I separate readout modules depending on the specific data. The detailed architecture for different tasks can be found Appendix B
3.3
TRANSFER BLOCK
3.3.1
TASK-SPECIFIC REPRESENTATIONS TRANSFER
In the Transfer Block, the first step is to transfer the task-specific representations from source to target tasks. A naive way is to use transfer function F--j(() to transfer the task-specific repre- sentation from the space of task to task j for every task pair:
(X))(X;) = o (F:().)W..)),
(2)
where Wi+s Raxd and is the dimension of the task-specific representation.
However, this will result in total number of 7 transfer functions. Thus, to prevent the quadratic growth, we develop a universal representation space where all task-specific representations get mapped to and all target tasks can be inferred from. More specifically, we decompose each transfer function (.) to Fry o (.s(.). Assume that we are trying to transfer the task-specific represen- tation E3 (x(xx)) from task i to task j. where i, E {1,2,-.,,
..., T). We decompose the transfer matrix Win; to S; and T,. In this way, we only need 27 transfer functions in total for F.x(.) and (.) The space complexity is reduced from O(T2) to O(T). Here we denote the transferred representation from task i to task as:
Has Fr, (X, (Ex
where S,, Ti E Bdxd and d' is hyper-parameter.
3.3.2 REPRESENTATIONS ASSEMBLY VIA MULTI-LEVEL TASK DEPENDENCY
Torget
(1) Interaction between task pair
Taga Tact (We)
Saial Sofmax
Sofmus
Update Attention
Torget Key
p1))
Namriiad Pasiiiicelloinnnie Powiist-wis Mituel-aa Attention
Prnifionwiis Mutusi- Talk Atteztion
Traget Thaie
Sounce Task 0M Reprosmtation (BR-A1AAD5) hn
Tranferee
Seatoe Talk
Searce
Seance Valae
(2) Incorparate task depundency prior
Oilitir (We))
Element-wise Product
Figure 2: Position-wiis mutual attention mechanism.
With the transferred representations, the next step of the Transfer Block is to assemble the transferred representations with respect to the multi-level task dependency. Here, the multi-level task dependency consists of two parts: (1) the general task dependency and (2) the data-specific task dependency. The multi-level task dependency is modelled by the position-wii wise mutual attention mechanism shown in Figure
4

---

Under review as a conference paper at ICLR 2020
To model the general task dependency, represent it by parameterized weighted dependency graph E ?7*T" The learnable weight of this parameterized dependency graph represents the transferable weight between any task pair. Note that the dependency graph is asymmetrical. In this way, the negative influence of irrelevant tasks can be reduced as much as possible.
Further, even for the same task pair, the transferable weight (dependency) may be different for (1) different data samples; (2) different sub-structure (node/word) in one data sample (graph/text). Therefore, we study the data-specific task dependency in depth. To efficiently model the data- specific task dependency, we consider the mutual attention between representations of the same data sample under source and target tasks.
Given Hijj, the transferred representation from the task i to task j. and E, the original representation from the target task, we get the position mutual attention by:
A = SUM (H, Woi) @ (X,(x,) Wkj) DIM=-1] 1/,
(4)
Vd"
where Woi, WK; Rdxd" are the query and key projection matrices, d" is hhper-paramett @ the Hadamard product, and SUM used to eliminate the last dimension (d"). We use Hadamard product instead of matrix multiplication because we only want the sub-structure in given data sample to interact with its counterpart under other tasks. Take graph data as an example, a certain node of one graph will only give attention to the same node of that graph under other tasks.
Then, for a target task j. we obtain (1) a set of general task dependency D, {Di+j}m= and (2) a set of data-specific task dependency A, {A,,))) To integrate them, we first seale sata-specific task dependency A, by the general task dependency D,. And then, caleulate the weighted sum of the transferred representations according to the multi-level task dependency. The final assembled representation X (for data sample and task j) is as follow:
Σ (
T (n Asn)) @ (Hi-) Wv:) W
(5)
k
where D is the normalized version of D. Wv. E ddddd is the value projection matrix, Wo; Raxd" is the output projection matrix. Note that r here is scalar parameter used prevent the vanish gradient problem of two normalization operations.
4 EXPERIMENT
In this section, we evaluate the performance our proposed L2T-MITTEN approach against several classical and SOTA approaches two application domains: graph and text. In graph domain, train a multitask Graph Convolutional Network (GCN) for both graph-level and node-level classifi- cation. And in text domain, we train a multitask Recurrent Neural Network (RNN) for text classifi- cation. Further, we provide visualization and analysis on the learned hidden dependency structure. Codes and datasets will be released.
4.1 DATASETS
For graph-level classification, we use Tox21 (Wu et al. 2017) and SIDER (Kuhn et al.) 2010).
Toxicology in the 21st Century (Tox21) is a database measuring the toxicity of chemical compounds. This dataset contains qualitative toxicity assays for 8014 organic molecules on 12 different targets including nuclear receptors and stress response pathways. our experiment, we treat each molecule graph and each toxicity assay binary graph-level classification task (for 12 tasks in total).
Tox21:
The Side Effect Resource (SIDER) is a database of marketed drugs and adverse drug actions. This dataset contains qualitative drug side-effects measurements for 1427 drugs on 27 side-effects. In experiment, treat each drug (organic molecule) as graph and the problem of predicting whether a given drug induces a side effect as a individual graph-level classification tasks (for 27 tasks in total).
SIDER:
5

---

Under review as a conference paper at ICLR 2020
For node-level classification, use DBLP (Tang al. 2008 and BlogCatalog (IV al. 2009)
DBLP: In the dataset, authors are represented by nodes and its feature is generated by titles of their papers. Two authors are linked together if they have co-authored least two papers in 2014-2019. We use 18 representative conferences as labels. An author is assigned to multiple labels if he/she has published papers in some conferences. The processed DBLP dataset is also published in our repository.
The BlogCatalog is collection of bloggers. In the dataset, bloggers represented as nodes, and there is link between two bloggers if they are friends. The interests of each blogger be tagged according the categories that he/she published blogs in. This dataset uses 39 categories as labels. Each blogger is assigned to multiple labels if he/she published a blog in some categories.
BlogCatalog:
For text classification, we use TMD dataset.
TMDb:
The Movie Database (TMDb) dataset is collection of information for 4803 movies. For each movie, the dataset includes information ranging from the production company, production country, release date to plot, genre. popularity, etc. In our experiment, we select plots the input with genres the label. We treat the problem of predicting whether given plot belongs to genre as an individual text-level classification tasks (for 20 tasks total). summary of the five datasets is provided in Appendix A
4.2 BASELINE METHODS
We compare our L.2T-MITTEN approach with both classical and SOTA approaches. The details are given as follows:
Single-task method:
Single-Task: Simply train network consists of encoder block and readout block for each task separately.
Classical multi-task method:
Shared-Bottom (Caruana 1997): ) This is a widely adopted multi-task learning framework which
consists of shared-bottons network (encoder block in our case) shared by all tasks, and separate tower network (readout block in our case) for each specific task. The input is fed into the shared- bottom network, and the tower networks are built upon the output of the shared-bottom. Each tower will then produce the task-specific outputs.
SOTA multi-task methods:
Cross-Stitch [Ish] 2016): This method uses "cross-stitch" unit learn the combination of shared and task-specific representation. The "cross-stitch" unit is k x k trainable matrix (k: is the number of tasks) which will transfer and fuse the representation among tasks by the following equation:
i1
an
a1k
Ø1
Orjsk
TT
where v; is the output of the lower level layer for task auy is the transfer weight from task task i, and 2; is the input of the higher level layer for task i.
MMoE (Guo Farooq 2018): This method adopts the Multi-gate Mixture-of-Exper structure. This structure consists of multiple bottom networks (experts), and multiple gating networks which take the input features and output softmax gates assembling the experts with different weights. The assembled features are then passed into the task-specific tower networks.
All the baseline models use the same encoder and readout block for each task. The architecture details are provided in Appendix B
'The Movie Database Wehsite: hutps//www..hhemvvichh...gg

---

Under review as a conference paper at ICLR 2020
4.3 EXPERIMENTAL SET-UP
We partition the datasets into 80:20 training/testing sets (i.e. each data sample can either appear in the training testing set) and evaluate our approach under multiple setting (1) Sufficient setting: all tasks have sufficient labeled training data; (2) Imbalanced setting: some tasks have more labeled training data than others; (3) Deficient setting: all tasks have deficient labeled training data. Models are trained for 100 epochs using the ADAM optimizer.
Table Experiment result for graph datasets
Dataset
Labeled Data Ratio
Single-Task Shared-Bottone
Cross-Stitch
MMoE
Our
All 80%
0.8063
0.8171
0.8204
0.8049
0.8333
Tox21
Partially 10%
0.7138
0.7309
0.7128
0.7331
0.7410
All 10%
0.7719
0.7934
0.7823
0.7848
0.8033
All 80%
0.6458
0.6484
0.6676
0.6406
0.6701
SIDER
Partially 10% All 10%
0.5682
0.5534
0.5504
0.5377
0.5741
0.6277
0.6290
0.6285
0.6261
0.6363
Partially 1 %
0.8069
0,8056
0.5148
0.7930
0.8241
DBLP
All 10%
0.8232
0.8077
0.5150
0.8177
0.8367
Partially 5% All 20%
0.5154 0.6100
0.6521 0.6667
0.5259
0.6720
0.6769
BlogCatalog
0.5272
0.6850
0.6861
Table Experiment result for text dataset
Dataset
Labeled Data Ratio
Single-Task
Shared-Bottoe
Cross-Stitch
MMoE
Our
All 80%
0.8172
0.8272
0.8543
0.8198
0.8484
TMDb
Partially 10%
0.7324 0.8033
0.7293
0.7234
0.6590
0.7404
All 10%
0.8227
0.8452
0.7869
0.8480
4.4 RESULT
We report the performance of our approach and baselines on graph classification, node classification and text classification tasks in terms of AUC-ROC score in Table and respectively
From the above result, first of all, we can see that the multi-tast methods outperform the single-task method in most cases which shows the effectiveness of knowledge transfer and multi-task learning. Further, we can see that our proposed 1.2T-MITTEN approach outperforms both classical and SOTA in most tasks. Finally, approach shows significant improvement under deficient labeled training data setting, since our approach leverages the structure of the data sample itself to guide the transfer among tasks.
Secondly, we found that in the real-world dataset, like DBLP dataset, our model can outperform other SOTA methods significantly, which demonstrate the importance of taking multi-level depen- dency into consideration. Note that the Single-Task can achieve the second-bes, best result. This fact indicates that in the real-world dataset, tasks may be irrelevant each other. Our multi-level task dependency can be more effective to prevent the influence of other irrelevant tasks.
Furthermore, conduct experiments the text classification dataset, TMDb. The Cross-Stitch model achieves the best result when the label ratio for every task is 80%. However, our task can achieve the best result for partially labeled setting (partially 10%) and few labeled setting (all 10%).
"The settings built by applying different masks training sets, e.g. in imbalanced setting, we randomly training set. PIn the tables, "All p% means all tasks' labeled training data ratios masked to 5%, while "Partially %% means only randomly selected tasks masked 9% labeled training data.
mask some data samples

---

Under review as a conference paper at ICLR 2020
This fact demonstrates that directed task dependency graph can effectively prevent the negative knowledge be transferred among different tasks when the training label is few.
4.3 VISUALIZATION AND ANALYSIS
For visualization and analysis of the learned multi-level task dependency structure, we will take DBLP as an example here due to its simplicity in interpreting and understanding.
Figure 3: Visualization of the learned multi-level task dependency structure for DBLP dataset
(a) General tasks (conferences) dependency structure
(b) Nodes (authors) cluster
(c) Data-specific tasks (conferences) dependency structure for each cluster (Left: cluster A, Right: cluster F)
First, in Figure 3a where we directly visualize the learned general task dependency matrix, we can approach indeed captures the task dependency structure in general, i.e. conferences from the same domain are more likely to be in the same sub-tree. Moreover, in Figure 3b we plot the authors (nodes) according the learned data-specific task dependency matrix and can that there are some clusters formed by authors. Further, we visualize the mean value of the -ata-specifii task dependency for each cluster, as shown in Figure 3c We can see that different cluster does have different task dependency. This is desirable since when predicting if an author has published papers in some conferences, authors from different domains should have different transfer weight among conferences (tasks). As summary, it is demonstrated that our approach can capture the task dependency at multiple levels according to specific data.
CONCLUSION
We propose 1.2T-MITTEN, a novel multi-task learning framework that (1) employs the position- wise mutual attention mechanism to learn the multi-level task dependency; (2) transfers the task- specific representations between tasks with linear space-efficiency; and (3) uses the learned multi- level task dependency to guide the inference. We design three experimental settings where training data is sufficient, imbalanced or deficient, with multiple graph/text datasets. Experimental results demonstrate the superiority of our method against both classical and SOTA baselines. We also show that our framework can be used as an analytical tool to extract the task dependency structures at different levels, which can reveal some hidden knowledge of tasks and of datasets
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, July 23-26, 2002, Edmonton, Alberta, Canada, 2002. ACM. ISBN 1-58113-567-X.
Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federa- tion of Natural Language Processing. ACL 2015, July 26-31, 2015. Beijing, China, Volume 2: Short Papers. 2015. The Association for Computer Linguistics. ISBN 978-1-941643-73-0. URL
https:: www. acl www.waalwweb.o.. www.wrrggg.o.o org hwwpo//oooooooooooooooo/////////
2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV,
USA, June 27-30, 2016, 2016. IEEE Computer Society. ISBN 978- 978-1-4673-8851-11 URL http:
// feeeepploree.. see. org/ /xpeg//pplic htpp///mttteeeennee Tssue. jtw.:sssseessspppppppp 18p7pummmber::77777
5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26 2017, Conference Track Proceedings, 2017. OpenReview.net. URL [RRLitttps////peeevviiee
net/ net/group dimemiidd=iiii wtt.:l/.cc//11/cooneeee/
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.066 1607.06450 2016.
Jonathan Baxter. A model of inductive bias learning. CoRR, abs/1 106.0245, 2011. URL http:
0245
Rich Caruana. Multitask learning, Machine Learning, 28(1):41-75, 1997. doi: 10.1023/A:
1007379606734. URLtatts:://// :/sttps::/do... htt::://doooooooo..o .ar./org/10.. 10. 1023/A: 1233AA:0003376061133
Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Gar- nett (eds.). Advances in Neural Information Processing Systems 28: Annual Con- ference Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, 2015. URL httt::/pppers. wwtp:/sssppp.cccc.ooooo.c.ca
in www.oonnprrossoooooooo ing--snnnnn--- ng- hyr. 28-2005
Jifeng Dai, Kaiming He, and Jian Sun. Instance- semantic semaaneeennmmmniiicmmeennnnnnni milii cascades. CoRR, abs/15 1t//1/122.441,, 2015. URL /Rl.:htpp://aaxii.orr//bbbb// 04412
via multi network
Jennifer G. Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, 2018. PMLR. URL http:// proceedings mlr wtt.mmrrprrrsssee prw.ss/80//
Yike Guo and Faisal Farooq (eds.). Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018, 2018. ACM. doi: 10.1145/3219819 URL https:: doi http:://///o...0/010111111..11/0 11445/218818
John Elder IV, Françoise Fogelman-Soulis Peter A. Flach, and Mohammed Javeed Zaki (eds.). Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Paris, France, June 28 July 1, 2009, 2009. ACM. ISBN 978- 16855-69055-4
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. CoRR, 100.06009, 2016. http:t
abs 02907
Michael Kuhn, Monica Campillos, Ivica Letunic, Lars Juhl Jensen, and Peer Bork. A side effect resource to capture phenotypic effects of drugs. Molecular Systems Biology, 6(1):343, 2010. doi: 10. 10.11088/mbb22009900 URL hntps: / http://wwwwtttttt www.wewwbpress.ommmmm www./oig//o/ooooooooo/oooo//oo.o
wwt.ddci////////cc/////e /bss/////////// 10. 100.1038
msb. 2009.009990 98.
Man Lan, Jianxiang Wang, Yuanbin Wu, Zheng-Yu Niu, and Haifeng Wang. Multi-task attention- based neural networks for implicit discourse relationship representation and identification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1299-13900 1308, Copenhagen, Denmark, September 2017. Association for Computational Lin- guistics. doi: 10.18653//1/DI17-1134 URL https: /www.w......m. aclweb. wwt. ozg/antholoqy/
D17-1134

---

Under review as a conference paper at ICLR 2020
Shikun Liu, Edward Johns, and Andrew J. Davison. End-to-end multi-task learning with attention. org/ abs/ 10704
CoRR, abs/ hts//803.10 183.117044 2018. URL
Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.). Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, 2017. Association for Computational Linguisties. ISBN 978-1-945622 83-8. URL attps: www wwwwwb.wwa........ owg. www.gg/annooogg//nnneoooo htw.hhhoooy/ooolmm////e1.
Bharath Ramsundar, Steven M. Kearnes, Patrick Riley, Dale Webster, David E. Konerding, and Vijay S. Pande, Massively multitask networks for drug discovery. CoRR, abs/1502.02077 2015.
URL
org/ wttp::/gggg........./. 02072
Sebastian Ruder. An overview of multi-task learning in deep neural networks.
CoRR,
abs/1706.05098, 2017. http:////aax
abaaabss/0000 05098
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. CoRR, abs/1810.04650, 2018. URL http::httttttttt org/ abs/ / 1 1810. 04650
Jie Tang, Jing Zhang. Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. Ametminer: Extraction and mining of academic social networks. In KDD '08, pp. 990-998, 2008
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence to sequence for sets. arXiv preprint arXiv: 1rXXiv:511.0665 1511.06391,,1, 2015.
Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande. MoleculeNet: A Benchmark for Molecular Machine Learning. arXiv e-prints, art. arXiv:1703.00564 Mar 2017.
Amir Roshan Zamir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. CoRR, ftss/8804008322 2018.
URL http::tttt//ittt/ org/ abs / /10//1114... 08328
A
DATASET SUMMARY
Table 3: Graph datasets summary
Dataset
Source
Graphs
Nodes Avg.
Edges Avg.
Graph Labels
Node Labels
Tox21
Bio
8014
18
48
12
SIDER
Bio
1427
33
105
27
DBLP
Citation
14704
24778
18
BlogCatalog
Social
10312
333983
39
Table 4: Text dataset summary
Dataset
Movies
Words Avg.
Genres
TMDb
8014
59
20
B ARCHITECTURE DETAILS
B. ARCHITECTURE DETAILS FOR GRAPH MODEL
As shown in Figure 4 in the Encoder Block, several layers of graph convolutional lay- ers (Kipf & Welling 2016) followed by the layer normalization (Ba et al. 2016). In the Readout
Block, for geaeh task, set-to-set (Vinyals al. 2015) the global pooling operator
10

---

Under review as a conference paper at ICLR 2020
to extract the graph-level representation which is later fed classifier; while for node-level task, we simply eliminate the global pooling layer and feed the node-level representation directly to the classifier.
GCN ReLU LayerNorm
GCN ReLU
Dense Tanh LayerNorm
Set2Set or None
Classifier
LayerNorm
Task-specific Encoder
Readout Block
Figure 4: Graph convolutional networks architecture. Note that in node-level task, the Set2Set layer (global pooling) is eliminated.
B.2 ARCHITECTURE DETAILS FOR TEXT MODEL.
The text model uses long short-term memory (LSTM) architecture in their Encoder Block, and the dot-product attention in the Readout Block, shown in Figure 5 get the text-level representation is as follows:
The dot-product attention used to
Softmax
(O (OH)); Ō=
where O E Baxd is the output of the LSTM, Hn E Rlxdd is the hidden state for the last word, a E R is attention weight for each word, and E RRxdd is the text-level representation (n the number words, d the feature dimension for each word).
LSTMs
Attention
Classifier
Task-specific Encoder
Readout Block
Figure 5: Text classification network architecture.


Reviews:
[{"review_id": "r1xqvc3uqr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is on an improvement of multi-task learning by considering the input tasks at two levels: (1) at task level, i.e. the relationship between the tasks and (2) by the data associated with each task. Their major argument is that most current methods hold the assumption that the tasks are correlated with each other but they conjecture that in the real-world this is not necessarily true and try to model the relationship between the input tasks at these two levels and incorporate that in the learning framework. To show effectiveness of their approach they test their method on differently oriented public datasets representing graphs, nodes and text and compare performance with some of the recent approaches to multi-task learning.\n\nComments to authors\n1. Overall while one could get the gist of the arguments in the paper, it was not thoroughly reviewed by the authors for grammar, so it was hard to follow the finer points of the arguments. There are several grammatical mistakes and errors, on every page, it'd be too cumbersome to point them all out.\n2. The distinction between the \"general task dependency\" and the \"data dependency\" does not seem significant enough. The data-dependent task dependency actually depends on the \"general task dependency\" as stated in the paper. This is probably manifested in the relatively slight improvement of the method compared with the SOTA. Perhaps more clarity on the difference and contribution of each \"level\" would make the significance stand out  clearer.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SyluxUeL5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a ‘Learning to Transfer via Modeling Multi-level Task Dependency’ for multi-task learning’, which uses the attention mechanism to learn task dependency.\n\nIn the introduction, authors claim ‘most of the current multitask learning framework rely on the assumption that all the tasks are highly correlated’. I don’t think this claim is correct. In fact, most state-of-the-art multi-task learning models can learn task dependency via different forms.\n\nIn the proposed network, different tasks have their own encoder, which leads to a large number of model parameters especially when there are a large number of tasks. This situation becomes even worse when each task has a limited number of labeled samples.\n\nThe attention has been used in multi-task learning. Authors can google ‘multi-task learning attention’ to find related works. Of course authors need to compare with those related works.\n\nA typo: “is theposition-wise mutual attention between”", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJxJYVNg9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission argues for the modeling the relationships between different tasks and incorporating such relationships when training multi-task frameworks. Though the basic concept (usefulness of modeling and incorporating the relationships among tasks) is valid, the submission has a number of critical issues, namely missing prior work that did that that already, missing critical specifics of the method, and unnecessary mix of different concepts. \n\nElaborated comments: \n\nA) Authors seem to be unaware of critically related prior work that specially modeled task relationships and did much of what's proposed in this submission, especially \"Taskonomy: Disentangling task transfer learning\". The \"relationship among tasks\" that this submission frequently talks about is the main concept in taskonomy 2018 paper (see their abstract). Besides the apparent similarities (eg the fig 1 of this submission vs fig 1&2 of taskonomy or fig 4 of this submission vs fig 13&7 of taskonomy), the formulation has strong similarities too (transferring from \"task-specific\" encoders of source tasks to target tasks using transfer readout functions, or ensembling multiple task-specific representations which seem to be the same as taskonomy's higher order transfer). This submission should be majorly revised in light of prior work and the critically relevant ones should be discussed and experimentally compared to. \n\nB) The presentation suffers from missing critical specifics. For instance, the \"general task dependency\" matrix  shown in Fig 3 and mentioned in page 4, which seem to be the same concept as taskonomy's task affinity matrix, is only mentioned in passing. While that seem to be one of the most important components of the method and its definition and extraction method should be discussed. \n\nC) Inline with the point B above, the presentation of the \"Transfer Block\" and what the authors refer to as \"Point-wise Mutual Attention Mechanism\" has issues and missing details. This block could potentially have new points in it, but it's not feasible to judge that and its technical correctness given the current disposition. For instance eq 2 seem to suggest the authors develop a universal representation space where all task-specific representations get mapped to and all target tasks can be inferred from (to reduce T^2 complexity to 2T). The rest of the section does not provide a clear implementation of this and add mathematical/notation confusions. Eg H_i_j is defined to be the task-specific representation of the source task i but is indexed over both tasks i and j where j is the target, or there is a E_j(X_j) where both indexes are j while E's index is over tasks and X's index is over datapoints. \n\nSimilarly the submission seem to jump over certain concepts/terms e.g. \"multi-view task dependency\" in page 4 vs\"multi-level task dependency\" in the title, etc. What exactly \"view\" or \"level\" mean here? Are those phrases really needed? Dropping any loosely grounded phrase would be a useful practice toward a clearer presentation.\n\nOverall, unfortunately the submission suffers from serious issues in its current shape. \n\n\n----\nComments after rebuttal stage: \n\nThanks to the authors for the rebuttal. It provided some help, but unfortunately it doesn't resolve the majority of the issues, as most of them are too major. A clear discussion on how the proposal is different and why it is better than the recent works that were not cited would be needed, and likely authors needed strong experimental comparison with some of them, eg to prove both general and data-specific task dependency is needed. \n\nI also didn't find the hierarchical justification clear or convincing \"The reason is that different from the image which [1] focuses on, text and graph data are hierarchical: word -> sentence and node -> graph. The task dependency at the basic level (word and node) may be different from the general task dependency\". \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "ByxJhPKOqS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a multi-task learning method that uses attention mechanism to identify relations between the tasks.  \n\nMethod:\n\n- The authors motivate the use of attention mechanism for identifying a sample-dependent measure  of task relatedness by that \"task dependency can be different for different data samples..\" At the introduction stage this argument was not clear to me. I would suggest to expand this part of the paper by providing a stronger motivation for the proposed approach.\n- what kind of mappings are used in (2)?\n\nExperiments:\n\n- it seems that all the datasets used are for multi-label learning. Thus, in the evaluation procedure, could the same input X appear both in the training and the test data sets (but in different tasks)? If yes, I believe it might make the evaluation less thorough. In either case it would be helpful to have this information in the description of the setting\n- since use of attention is the main contribution of this work, but not the only part of the method,  I would recommend adding to the evaluation a method which is equivalent to the proposed one, but doesn't involve attention (i.e. only uses D). \n\nAdditional comments:\n\n- in its current form the manuscript is rather hard to follow, it requires a thorough proof-reading\n- it is unclear what Figure 1 on page 2 is for\n- on page 2 phrase \"... the label ratio is imbalanced.\" is confusing. I believe the authors meant that the data (not label) proportions between the tasks are uneven\n- on page 3 the authors say that minimisation of the empirical risk (eq. (1)) is \"the goal of multi-task learning\". This sentence needs rephrasing, because from the point of view of empirical risk minimisation any multi-task approach is worse than the corresponding single-task version (i.e. its empirical risk is higher). Only in terms of the generalization performance one can argue that information sharing is beneficial.\n- notation in (3) is confusing - index i is used in two meanings\n- it's unclear what k in eq. (4) is\n- it seems that a few references are broken\n\nTo my knowledge the idea of making amount of transfer between tasks dependent on the particular sample at hand is new. Therefore, in my opinion, with improved presentation (and in particular motivation at the beginning of the manuscript) and additional evaluation demonstrating effects of the attention component the manuscript could be recommended for acceptance.\n\n---------------------------------------------------------------\n\nI thank the authors for their comments. The quality fo the manuscript has indeed improved and the differences with the existing methods are clearer. However, in light of the reviewers' comments, I agree that at least the experimental section needs to be extended by adding relevant baselines. In particular, comparison to \"End-to-end multi-task learning with attention\" is needed to demonstrate importance of the task-level dependence measure. If direct comparison is not possible (or in addition to it) I would suggest to evaluate a modification of L2MITTEN with matrix D being equal to all 1s.  ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this work, the authors address a multi-task learning setting and propose to enhance the estimation of task dependency with an attention mechanism capturing sample-dependant measure of task relatedness. All reviewers and AC agree that the current manuscript lacks clarity and convincing empirical evaluations that clearly show the benefits of the proposed approach w.r.t. state-of-the-art methods. Specifically, the reviewers raised several important concerns that were viewed by AC as critical issues:
(1) the empirical evaluations need to be significantly strengthened to show the benefits of the proposed methods over SOTA -- see R2’s request to empirically compare with the related recent work [Taskonomy, 2018] and R4’s request to compare with the work [End-to-end multi-task learning with attention, 2018]. R4 also suggested to include an ablation study to assess the benefits of the attention mechanism. Pleased to report that the authors addressed the ablation study in their rebuttal and confirmed that the proposed attention mechanism plays an important role in the performance of the proposed method. 
(2) All reviewers see an issue with the presentation clarity of the conceptual and technical contributions  -- see R4’s and R2’s detailed comments and questions regarding technical contributions; see R3’s and R4’s comments that the distinction between the general task dependency and the data-driven dependency is either not significant or is not clearly articulated; finding better examples to illustrate the difference (instead of reiterating the current ones) would strengthen the clarity and conceptual contributions.  
A general consensus among reviewers and AC suggests, in its current state the manuscript is not ready for a publication. It needs more clarifications, empirical studies and polish to achieve the desired goal.


Decision: Reject