Measuring and Harnessing Transference in Multi-Task Learning

1 INTRODUCTION . Deciding if two or more objectives should be trained together in a multi-task model , as well as choosing how that model ’ s parameters should be shared , is an inherently complex issue often left to human experts ( Zhang & Yang , 2017 ) . However , a human ’ s understanding of similarity is motivated by their intuition and experience rather than a prescient knowledge of the underlying structures learned by a neural network . To further complicate matters , the benefit or detriment induced from co-training relies on many non-trivial decisions including , but not limited to , dataset characteristics , model architecture , hyperparameters , capacity , and convergence ( Wu et al. , 2020 ; Vandenhende et al. , 2019 ; Standley et al. , 2019 ; Sun et al. , 2019 ) . As a result , a quantifiable measure which conveys the effect of information transfer in a neural network would be valuable to practitioners and researchers alike to better construct or understand multi-task learning paradigms ( Baxter , 2000 ; Ben-David & Schuller , 2003 ) . The training dynamics specific to multitask neural networks , namely cross-task interactions at the shared parameters ( Zhao et al. , 2018 ) , are difficult to predict and only fully manifest at the completion of training . Given the cost , both with regards to time and resources , of fully training a deep neural network , an exhaustive search over the 2m−1 possible combinations ofm tasks to determine ideal task groupings can be infeasible . This search is only complicated by the irreproducibility inherent in traversing a loss landscape with high curvature ; an effect which appears especially pronounced in multi-task learning paradigms ( Yu et al. , 2020 ; Standley et al. , 2019 ) . In this paper , we aim to take a step towards quantifying transference , or the dynamics of information transfer , and understanding its effect on multi-task training efficiency . As both the input data and state of model convergence are fundamental to transference ( Wu et al. , 2020 ) , we develop a parameter-free approach to measure this effect at a per-minibatch level of granularity . Moreover , our quantity makes no assumptions regarding model architecture , and is applicable to any paradigm in which shared parameters are updated with respect to multiple task losses . By analyzing multi-task training dynamics through the lens of transference , we present the following observations . First , information transfer is highly dependent on model convergence and varies significantly throughout training . Second , and perhaps surprisingly , excluding certain task gradients from the multi-task gradient update for select minibatches can improve learning efficiency . Our Under review as a conference paper at ICLR 2021 Transference in CelebA Transference in MetaWorld A0 A8 A7 A6 A5 A4 A3 A2 A1 -0.4 -0.3 -0.2 -0.1 1.0 Transference in CelebA Gradient Update A0 A1 A2 A3 A4 A5 A6 A7 A8 ( a ) reach push press button top open window close window -0.8 -0.6 -0.4 -0.2 1.0 Transference in MetaWorld Gradient Update close window open window press button top push reach ( b ) Figure 1 : Transference in ( a ) CelebA for a subset of 9 attributes ; ( b ) Meta-World for “ push ” , “ reach ” , “ press button top ” , and “ open window ” . To determine task groupings , we compute the transference of each task i on all other tasks j , i.e . Zt { i } →j and average over time . For the purpose of illustration , we normalize the transference along each axis . Notice the majority of the tasks in ( a ) concentrate around a single value for each attribute . Tasks which exhibit transference above this value are considered to have relatively high transference . For instance , A3 exhibits higher-than-average transference on A0 , A4 , and A5 . A similar effect is observed in ( b ) , with “ close window ” manifesting high transference on “ push ” and “ reach ” . analysis suggests this is due to large variation in loss landscapes for different tasks as illustrated in Figure 4 . Building on these observations , we propose two methods to utilize transference in multitask learning algorithms – to choose which tasks to train together as well as determining which gradients to apply at each minibatch . Our experiments indicate the former can identify promising task groupings , while the latter can improve learning performance over prior methods . In summary , our main contributions are three-fold : we ( 1 ) propose the first measure ( to our knowledge ) which quantifies information transfer among tasks in multi-task learning ; ( 2 ) demonstrate how transference can be used as a heuristic to select task groupings ; ( 3 ) present a method which leverages minibatch-level transference to augment network performance . 2 RELATED WORK . Multi-Task Formulation . The most prevalent formulation of MTL is hard parameter sharing of hidden layers ( Ruder , 2017 ; Caruana , 1993 ) . In this design , a subset of the hidden layers are typically shared among all tasks , and task-specific layers are stacked on top of the shared base to output a prediction value . Each task is assigned a weight , and the loss of the entire model is a linear combination of each task ’ s loss multiplied by its respective loss weight . This particular design enables parameter efficiency by sharing hidden layers across tasks , reduces overfitting , and can facilitate transfer learning effects among tasks ( Ruder , 2017 ; Baxter , 2000 ; Zhang & Yang , 2017 ) . Information Sharing . Prevailing wisdom suggests tasks which are similar or share a similar underlying structure may benefit from co-training in a multi-task system ( Caruana , 1993 ; 1997 ) . A plethora of multi-task methods addressing what to share have been developed , such as Neural Architecture Search ( Guo et al. , 2020 ; Sun et al. , 2019 ; Vandenhende et al. , 2019 ; Rusu et al. , 2016 ; Huang et al. , 2018 ; Lu et al. , 2017 ) and Soft-Parameter Sharing ( Misra et al. , 2016 ; Duong et al. , 2015 ; Yang & Hospedales , 2016 ) , to improve multi-task performance . Though our measure of transference is complementary with these methods , we direct our focus towards which tasks should be trained together rather than architecture modifications to maximize the benefits of co-training . While deciding which tasks to train together has traditionally been addressed with costly crossvalidation techniques or high variance human intuition , recent advances have developed increasingly efficient algorithms to assess co-training performance . Swirszcz & Lozano ( 2012 ) and Bingel & Søgaard ( 2017 ) approximate multi-task performance by analyzing single-task learning characteristics . An altogether different approach may leverage recent advances in transfer learning focused on understanding task relationships ( Zamir et al. , 2018 ; Achille et al. , 2019b ; Dwivedi & Roig , 2019 ; Zhuang et al. , 2020 ; Achille et al. , 2019a ) ; however , Standley et al . ( 2019 ) show transfer learning algorithms which determine task similarity do not carry over to the multi-task learning domain and instead propose a multi-task specific framework . Training Dynamics . Significant effort has also been invested to improve the training dynamics of MTL systems . In particular , dynamic loss reweighing has achieved performance superior to using fixed loss weights found with extensive hyperparameter search ( Kendall et al. , 2018 ; Guo et al. , 2018 ; Liu et al. , 2019 ; Chen et al. , 2017 ; Sener & Koltun , 2018 ; Lin et al. , 2019 ) . Another set of methods seeks to mitigate the optimization challenges in multi-task learning by manipulating the task gradients in a number of ways such as ( 1 ) modifying the direction of task gradients with the underlying assumption that directional inconsistency of gradients on the shared parameters are detrimental to model convergence and performance ( Zhao et al. , 2018 ; Suteu & Guo , 2019 ) , and ( 2 ) altering both the direction and the magnitude of the task gradients ( Yu et al. , 2020 ; Chen et al. , 2020 ; Wang et al. , 2020b ) . Instead of directly modifying the task gradients during optimization , our work builds upon these approaches by quantifying how a gradient update to the shared parameters would affect training loss and choosing the combination of gradients which maximizes positive information transfer . Looking into the Future . Looking at what could happen to determine what should happen has been used extensively in both the meta-learning ( Finn et al. , 2017 ; Nichol et al. , 2018 ; Brinkmeyer et al. , 2019 ; Grant et al. , 2018 ; Kim et al. , 2018 ) as well as optimization domains ( Nesterov , 1983 ; Hinton & Plaut , 1987 ; Zhang et al. , 2019 ; Izmailov et al. , 2018 ; Johnson & Zhang , 2013 ) . Lookahead metalearning algorithms focusing on validation loss have also been used to improve generalization in multi-task learning systems ( Wang et al. , 2020a ) , and our work further adapts this central concept to multi-task learning to both quantify and improve information transfer . 3 TRANSFERENCE IN MULTI-TASK LEARNING . Within the context of a hard-parameter sharing paradigm , tasks collaborate to build a shared feature representation which is then specialized by individual task-specific heads to output a prediction . Accordingly , tasks implicitly transfer information to each other by updating this shared feature representation with successive gradient updates . We can then view transference , or information transfer in multi-task learning , as the effect of a task ’ s gradient update to the shared parameters on another task ’ s loss during training . While the the shared parameter update using a task ’ s gradient , often but not always , increases the losses of the other tasks in the network , we find the extent to which these losses change to be highly task specific . This indicates certain tasks interact more constructively than others . Further , we notice this effect to be reproducible and nearly unchanged across successive training runs with varying parameter initializations . Motivated by these observations , we derive a quantitative measure of transference , describe how it can be used to determine which tasks should be trained together , and provide empirical analysis of these claims . Later , we will build upon these ideas to derive a new multi-task learning algorithm . 3.1 MEASURING TRANSFERENCE . Consider an m-multitask loss function parameterized by { θs } ∪ { θi| i ∈ [ m ] } where θs represents the shared parameters and θi represents the task i ∈ [ m ] specific parameters . Let Ltotal ( X , θs , { θi } ) = ∑ i∈ [ m ] Li ( X , θs , θi ) , denote the total loss where Li represents the non-negative loss of task i . For simplicity of notation , we set the loss weight of each task to be equal to 1 , though our construction generalizes to arbitrary weightings . For a given training batch X t at time-step t , we can first update the task specific parameters { θt+1i } using standard gradient updates . We can now define the quantity θt+1s|ξ to represent the updated shared parameters after a gradient step with respect to the tasks in the non-empty subset ∅ ⊂ ξ ⊆ [ m ] . Assuming SGD for simplicity , we have1 θt+1s|ξ : = θ t s − η ∑ i∈ξ ∇θsLi ( X t , θts , θti ) . This quantity allows us to calculate a lookahead loss using the updated shared parameters while keeping the task-specific parameters as well as the input batch unchanged across different subsets of task gradients . That is , in order to assess the effect of the gradient update of tasks in ξ on a given task j , we can compare the loss of task j before and after applying the gradient update on the shared parameters with respect to ξ . In order to eliminate the scale discrepancy among different task losses , we consider the ratio of a task ’ s loss before and after the gradient step on the shared parameters as a scale invariant measure of relative progress . We can then define an asymmetric measure for calculating the transference of the tasks in ξ at a given time-step t on a single task j as Ztξ j = 1− Lj ( X t , θt+1s|ξ , θ t+1 j ) Lj ( X t , θts , θtj ) . ( 1 ) Notice that a positive value of Ztξ j indicates that the update on the shared parameters results in a lower loss on task j than the original parameter values , while a negative value of Ztξ j indicates that the shared parameter update is antagonistic for this task ’ s performance . Also , note that for ξ = { j } , our definition of transference encompasses a notion of self-transference , i.e . the effect of a task ’ s gradient update on its own loss . This quantity is particularly useful as a baseline to determine whether a subset of gradient updates can result in improved performance when compared with a task ’ s own self-transference . As we discuss in the next section , transference provides an effective guideline for choosing the subset of tasks to train together in a multi-task setting .

Summary: [Summary] This paper studies the problem of task relationship/transference in multi-task learning, by introducing a quantifiable measurement based on relative loss updates.  A (nonsymmetric) task transference between task $i$ and task $j$ then can be computed by measuring the relative change of training loss of task $j$, with the updated shared parameters from the training loss of task $i$. By finding a subset of tasks that achieves maximal total transference over every single task, multi-task learning performance can be further improved.