Learning to Transfer via Modelling Multi-level Task Dependency

1 INTRODUCTION . Multi-task learning ( Caruana , 1997 ) aims to train a single model on multiple related tasks jointly , so that useful knowledge learned from one task can be transferred to enhance the generalization performance of other tasks . Over the last few years , different types of multi-task learning mechanisms ( Sener & Koltun , 2018 ; Guo & Farooq , 2018 ; Ish , 2016 ; Lon , 2015 ) have been proposed and proved better than single-task learning methods from natural language processing ( Palmer et al. , 2017 ) and computer vision ( Cortes et al. , 2015 ) to chemical study ( Ramsundar et al. , 2015 ) . Despite the success of multi-task learning , when applying to ‘ discrete ’ data ( graph/text ) , most of the current multi-task learning frameworks ( Zamir et al. , 2018 ; Ish , 2016 ) only leverage the general task dependency with the assumption that the task dependency remains the same for ( 1 ) different data samples ; and ( 2 ) different sub-structures ( node/word ) in one data sample ( graph/text ) . However , this assumption is not always true in many real-world problems . ( 1 ) Different data samples may have different task dependency . For example , when we want to predict the chemical properties of a particular toxic molecule , despite the general task dependency , its representations learned from toxicity prediction tasks should be more significant than the other tasks . ( 2 ) Even for the same data sample , different sub-structures may have different task dependency . Take sentence classification as an example . Words like ‘ good ’ or ‘ bad ’ may transfer more knowledge from sentiment analysis tasks , while words like ‘ because ’ or ‘ so ’ may transfer more from discourse relation identification tasks . In this work , to accurately learn the task dependency in both general level and data-specific level , we propose a novel framework , ‘ Learning to Transfer via ModellIng mulTi-level Task dEpeNdency ’ ( L2T-MITTEN ) . The general task dependency is learned as a parameterized weighted dependency graph . And the data-specific task dependency is learned with the position-wise mutual attention mechanism . The two-level task dependency can be used by our framework to improve the performance on multiple tasks . And the objective function of multi-task learning can further enhance the quality of the learned task dependency . By iteratively mutual enhancement , our framework can not only perform better on multiple tasks , but also can extract high-quality dependency structures at different levels , which can reveal some hidden knowledge of the datasets . Another problem is that to transfer task-specific representations between every task pair , the number of transfer functions will grow quadratically as the number of tasks increases , which is unaffordable . To solve this , we develop a universal representation space where all task-specific representations get mapped to and all target tasks can be inferred from . This decomposition method reduces the space complexity from quadratic to linear . We validate our multi-task learning framework extensively on different tasks , including graph classication , node classification , and text classification . Our framework outperforms all the other state-ofthe-art ( SOTA ) multi-task methods . Besides , we show that L2T-MITTEN can be used as an analytic tool to extract interpretable task dependency structures at different levels on real-world datasets . Our contributions in this work are threefold : • We propose a novel multi-task learning framework to learn to both general task dependency and data-specific task dependency . The learned task dependency structures can be mutually enhanced with the objective function of multi-task learning . • We develop a decomposition method to reduce the space complexity needed by transfer functions from quadratic to linear . • We conduct extensive experiments on different real-world datasets to show the effectiveness of our framework and the importance of modelling multi-level task dependency . 2 RELATED WORK . According to a recent survey ( Ruder , 2017 ) , existing multi-task learning methods can be categorized by whether they share the parameters hardly or softly . 2.1 HARD PARAMETER SHARING . For hard parameter sharing , a bottom network will be shared among all the tasks , and each individual task will have its own task-specific output network . The parameter sharing of the bottom network reduces the parameter needed to be learned and thus can avoid over-fitting to a specific task . However , when the tasks are not relevant enough ( Sha , 2002 ; Baxter , 2011 ) , the shared-bottom layers will suffer from optimization conflicts caused by mutually contradicted tasks . If the bottom model is not capable enough to encode all the necessary knowledge from different tasks , this method will fail to correctly capture all the tasks . Besides , Dy & Krause ( 2018 ) points out that the gradients of some dominant task will be relatively larger than gradients of other tasks . This dominant phenomenon will be more obvious when the proportions of labeled data between tasks are uneven , in which case the model will be majorly optimized on data-rich tasks . To alleviate this problem , some recent works ( Sener & Koltun , 2018 ; Dy & Krause , 2018 ) try to dynamically adjust the task weight during the training stage . Sener & Koltun ( 2018 ) casts the multi-task learning to a multi-objective optimization problem , and they use a gradient-based optimization method to find a Pareto optimal solution . Dy & Krause ( 2018 ) proposes a new normalization method on gradients , which attempts to balance the influences of different tasks . Recently , Guo & Farooq ( 2018 ) proposes to apply Mixture-of-Experts on multi-task learning , which linearly combines different experts ( bottoms ) by learnable gates . Because different experts can capture different knowledge among tasks , this model can , to some extent , model the dependency relationship among tasks . 2.2 SOFT PARAMETER SHARING . Methods using soft parameter sharing ( Lon , 2015 ; Ish , 2016 ; Dai et al. , 2015 ; Yan , 2017 ) do not keep the shared bottom layers . Instead , for soft-parameter models , most of the model parameters are task-specific . Lon ( 2015 ) focuses on reducing the annotation effort of the dependency parser tree . By combining two networks with a L2 normalization mechanism , knowledge from a different source language can be used to reduce the requirement of the amount of annotation . Further , in some existing works , the shallow layers of the model will be separated from other layers , and be used as the feature encoders to extract task-specific representations . For example , Ish ( 2016 ) proposes a Cross-Stitch model which is a typical separate bottom model . Cross-Stitch model will be trained on different tasks separately to encode the task-specific representations from different bottom layers . Then , a cross-stitch unit is used a as a gate to combine those separately trained layers . Yan ( 2017 ) introduces the tensor factorization model to allow common knowledge to be shared at each layer in the network . By the strategy proposed in their work , parameters are softly shared across the corresponding layers of the deep learning network and the parameter sharing ratio will be determined by the model itself . We also note that some recent works ( Zamir et al. , 2018 ; Lan et al. , 2017 ; Liu et al. , 2018 ) can learn to capture task dependency . Zamir et al . ( 2018 ) computes an affinity matrix among tasks based on whether the solution for one task can be sufficiently easily read out of the representation trained for another task . However , it can only capture the general task dependency . Lan et al . ( 2017 ) uses a sigmoid gated interaction module between two tasks to model their relation . But it will suffer from quadratic growth in space as the number of tasks increases . Liu et al . ( 2018 ) utilizes a shared network to share features across different tasks and uses the attention mechanism to automatically determine the importance of the shared features for the respective task . However , there is no knowledge transferred or interaction between tasks . 3 APPROACH . In this section , we propose our framework L2T-MITTEN , which can end-to-end learn the task dependency in both general and data-specific level , and help to assemble multi-task representations from different task-specific encoders . 3.1 PROBLEM FORMULATION . To formulate our framework , we first start by briefly introducing a general setting of multi-task learning . For each task t ∈ { 1 , ... , T } , we have a corresponding dataset { ( X ( t ) k , y ( t ) k ) } N ( t ) k=1 with N ( t ) data samples , where X ( t ) k represent the feature vector of k-th data sample and y ( t ) k is its label . We would like to train T models for these tasks , and each model has its own parameterW ( t ) . Note that for different multi-task learning frameworks , these parameters can be shared hardly or softly . The goal of multi-task learning is to improve general performance by sharing information among related tasks . The total loss of multi-task learning is calculated as : min W T∑ t=1 ∑N ( t ) k=1 L ( W ( t ) ; X ( t ) k , y ( t ) k ) N ( t ) ( 1 ) 3.2 ARCHITECTURE OVERVIEW . As is shown in Figure 1 , our framework consists of three components : Task-specific Encoders , Transfer Block , and Readout Block . The Task-specific Encoders consists of T separate feature encoders , which can be any type of feed-forward networks based on specific data . Unlike hard parameter sharing methods that tie the bottom encoders ’ parameters together , we keep each feature encoder separate to efficiently extract task-specific knowledge . In this way , for a given data sample Xk , we can use these encoders to get task-specific representations { Et ( Xk ) } Tt=1 , where Et ( Xk ) is the representation ofXk for task t. To conduct multi-task learning , one can simply use the representation of each task alone to predict labels without sharing any parameters . However , this model will suffer for tasks without sufficient labeled data . Therefore , we would like to ( 1 ) transfer the knowledge among these T tasks and ( 2 ) assemble the transferred representations , which is what we do in the Transfer Block . The Readout Block also consists of T separate readout modules depending on the specific data . The detailed architecture for different tasks can be found in Appendix B . 3.3 TRANSFER BLOCK . 3.3.1 TASK-SPECIFIC REPRESENTATIONS TRANSFER . In the Transfer Block , the first step is to transfer the task-specific representations from source to target tasks . A naive way is to use a transfer function Fi→j ( · ) to transfer the task-specific representation from the space of task i to task j for every task pair : Fi→j ( Xk ) = σ ( Ei ( Xk ) Wi→j ) , ( 2 ) whereWi→j ∈ Rd×d , and d is the dimension of the task-specific representation . However , this will result in a total number of T 2 transfer functions . Thus , to prevent the quadratic growth , we develop a universal representation space where all task-specific representations get mapped to and all target tasks can be inferred from . More specifically , we decompose each transfer function Fi→j ( · ) to FTj ◦ FSi ( · ) . Assume that we are trying to transfer the task-specific representation Ei ( Xk ) from task i to task j , where i , j ∈ { 1 , 2 , ... , T } . We can decompose the transfer matrix Wi→j to Si and Tj . In this way , we only need 2T transfer functions in total for FSi ( · ) and FTj ( · ) . The space complexity is reduced from O ( T 2 ) to O ( T ) . Here we denote the transferred representation from task i to task j as : Hi→j = FTj ◦ FSi ( Xi ) = σ ( Ei ( Xk ) Si ) T Tj , ( 3 ) where Si , Tj ∈ Rd×d ′ and d′ is a hyper-parameter .

Summary: The paper is on an improvement of multi-task learning by considering the input tasks at two levels: (1) at task level, i.e. the relationship between the tasks and (2) by the data associated with each task. Their major argument is that most current methods hold the assumption that the tasks are correlated with each other but they conjecture that in the real-world this is not necessarily true and try to model the relationship between the input tasks at these two levels and incorporate that in the learning framework. To show effectiveness of their approach they test their method on differently oriented public datasets representing graphs, nodes and text and compare performance with some of the recent approaches to multi-task learning.