Regularizing Deep Multi-Task Networks using Orthogonal Gradients

1 INTRODUCTION . Deep neural networks have proven to be very successful at solving isolated tasks in a variety of fields ranging from computer vision to NLP . In contrast to this single task setup , multi-task learning aims to train one model on several problems simultaneously . This approach would incentivize it to transfer knowledge between tasks and obtain multi-purpose representations that are less likely to overfit to an individual problem . Apart from potentially achieving better overall performance ( Caruana , 1997 ) , using a multi-task approach offers the additional benefit of being more efficient in memory usage and inference speed than training several single-task models ( Teichmann et al. , 2018 ) . A popular design for deep multi-task networks involves hard parameter sharing ( Ruder , 2017 ) , where a model contains a common encoder , which is shared across all tasks and several problem specific decoders . Given a single input each of the decoders is then trained for a distinct task using a different objective function and evaluation metric . This approach allows the network to learn multi-purpose representations through the shared encoder which every decoder will then use differently according to the requirements of its task . Although this architecture has been successfully applied to multi-task learning ( Kendall et al. , 2018 ; Chen et al. , 2017 ) it also faces some challenges . From an architectural point of view it is unclear how to choose the task specific network capacity ( Vandenhende et al. , 2019 ; Misra et al. , 2016 ) as well as the complexity of representations to share between tasks . Additionally , optimizing multiple objectives simultaneously introduces difficulties based on the nature of those tasks and the way their gradients interact with each other ( Sener & Koltun , 2018 ) . The dissimilarity between tasks could cause negative transfer of knowledge ( Long et al. , 2017 ; Zhao et al. , 2018 ; Zamir et al. , 2018 ) or having task losses of different magnitudes might bias the network in favor of a subset of tasks ( Chen et al. , 2017 ; Kendall et al. , 2018 ) . It becomes clear that the overall success of multi-task learning is reliant on managing the interaction between tasks , and implicitly their gradients with respect to the shared parameters of the model . This work focuses on the second category of challenges facing networks that employ hard parameter sharing , namely the interaction between tasks when being jointly optimized . We concentrate on reducing task interference by regularizing the angle between gradients rather than their magni- tudes . Based on our empirical findings unregularized multi-task networks have high variation in the angles between task gradients , meaning gradients frequently point in similar or opposite directions . Additionally , well-performing models share the property that their distribution of cosines between task gradients is zero-centered and low in variance . Nearly orthogonal gradients will reduce task competition as individual task decoders learn to use different features of the encoder , thus not interfering with each other . Furthermore , we discover that popular regularization methods such as Dropout ( Srivastava et al. , 2014 ) and Batchnorm ( Ioffe & Szegedy , 2015 ) implicitly orthogonalize the task gradients . We propose a new gradient regularization term to the multi-task objective that explicitly minimizes the squared cosine between task gradients and show that our method obtains competitive results on the NYUv2 ( Nathan Silberman & Fergus , 2012 ) and SUN RGB-D ( Song et al. , 2015 ) datasets . 2 RELATED WORK . Multi-task learning is a sub-field of transfer learning ( Pan & Yang , 2009 ) and encompasses a variety of methods ( Caruana , 1997 ) . The recent focus on deep multi-task learning can be attributed to the neural network ’ s unparalleled success in computer vision ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2014 ; He et al. , 2016 ) and its capability to create hierarchical , multi-purpose representations ( Bengio et al. , 2013 ; Yosinski et al. , 2014 ) . Deep multi-task learning is commonly divided into hard or soft parameter sharing methods ( Caruana , 1997 ; Ruder , 2017 ) . Soft parameter sharing maintains separate models for each task but enforces constraints on the joint parameter set ( Yang & Hospedales , 2016 ) . In this work we focus solely on hard parameter sharing methods , which maintain a common encoder for all tasks but also contain task-specific decoders that use the learned generic representations . We further split deep multi-task approaches into architecture and loss focused methods . Architecture based methods aim at finding a network structure that allows optimal knowledge sharing between tasks by balancing the capacities of the shared encoder and the task specific decoders . Most multitask related work chooses the architecture on an ad hoc basis ( Teichmann et al. , 2018 ; Neven et al. , 2017 ) , but recent research looks to answer the question of how much and where to optimally share knowledge . Cross-stitch networks maintain separate models for all tasks but allow communication between arbitrary layers through specialized cross-stitch units ( Misra et al. , 2016 ) . Branched multitask networks allow for the decoders to also be shared by computing a task affinity matrix that indicates the usefulness of features at arbitrary depths and for different problems ( Vandenhende et al. , 2019 ) . Liu et al . ( 2019b ) introduces attention modules allowing task specific networks to learn which features from the shared feature network to use at distinct layers . Loss focused methods try to balance the impact of individual tasks on the training of the network by adaptively weighting the task specific losses and gradients . Certain tasks might have a disproportionate impact on the joint objective function forcing the shared encoder to be optimized entirely for a subset of problems , effectively starving other tasks of resources . Kendall et al . ( 2018 ) devise a weighting method dependent on the homoscedastic uncertainty inherently linked to each task while Chen et al . ( 2017 ) reduce the task imbalances by weighting task losses such that their gradients are similar in magnitude . Sener & Koltun ( 2018 ) cast multi-task learning as a multi-objective optimization problem and aim to find a Pareto optimal solution . They also analyze gradients but with the goal to then scale these such that their convex combination will satisfy the necessary conditions to reach the desired solution . In contrast to these approaches our method does not seek to scale gradients , neither directly nor via task weights , but conditions the optimization trajectory towards solutions that have orthogonal task gradients . The problem of conflicting gradients or task interference has been previously explored in multi-task learning as well as continual learning . Zhao et al . ( 2018 ) introduce a modulation module that reduces destructive gradient interference between tasks that are unrelated . Du et al . ( 2018 ) choose to ignore the gradients of auxiliary tasks if they are not sharing a similar direction with the main task . Riemer et al . ( 2018 ) maximize the dot product between task gradients in order to overcome catastrophic forgetting . These methods have in common the interpretation that two tasks are in conflict if the cosine between their gradients is negative , while alignment should be encouraged . Our work differs from this perspective by additionally penalizing task gradients that have a similar direction , arguing that by decorrelating updates the shared encoder is able to maximize its representational capacity . A similar observation regarding orthogonal parameters is made by Rodrı́guez et al . ( 2016 ) who propose a weight regularization term for single task learning that decorrelates filters in convolutional neural networks . Another way to minimize interference is to encourage sparsity ( French , 1991 ; Javed & White , 2019 ) , although doing so directly might interfere with the network ’ s ability to learn shared representations ( Ruder , 2017 ) . Finally our work is in line with recent research ( Liu et al. , 2019a ; Santurkar et al. , 2018 ) that emphasizes the benefit of analyzing gradients to understand neural networks and devise potential improvements to their training . We share elements with Drucker & Le Cun ( 1991 ) and more recently Varga et al . ( 2017 ) in that we propose explicit regularization methods for gradients . 3 ORTHOGONAL TASK GRADIENTS . In this work we present a novel gradient based regularization term that orthogonalizes the interaction between multiple tasks . We define a multi-task neural network as a shared encoder fθsh and a set of task-specific decoders fθti , for each of the T tasks T = { t1 , ... , tT } . The encoder creates a mapping between the input space X and a latent feature space Rd that is used by each of the decoders to predict the task specific labels Yti . Each of the inputs in X is associated to a set of labels for the tasks in T , forming the dataset D = { xi , yt1i , ... , y tT i } i∈N of N observations . For task t ∈ T we define the empirical loss as Lt ∆ = 1N ∑ i∈N Lt ( fθt ( fθsh ( xi ) ) , yi ) . The multitask objective can be then constructed as a convex combination of individual task losses using the weights wt ∈ R : LT = ∑ t∈T wtLt ( 1 ) Using gradient descent to minimize the multi-task loss in Equation 1 , we obtain the following update rule for the parameters θsh : θsh = θsh − γ ∑ t∈T wt ∂Lt ∂θsh ( 2 ) It becomes clear that the overall success of a multi-task network is dependent on the individual task gradients and their relationship to each other . Task gradients might cancel each other out or a certain task might dominate the direction of the encoder ’ s parameters . We further examine the interaction between two tasks ti and tj by looking at the cosine of their gradients with respect to the encoder : cos ( ti , tj ) = cos ( ∂Lti ∂θsh , ∂Ltj ∂θsh ) ( 3 ) Previous work argues that negative transfer , task interference or competition ( Du et al. , 2018 ; Sener & Koltun , 2018 ; Zhao et al. , 2018 ) happens when this cosine is negative , leading to tasks with smaller gradient magnitudes in fact increasing their error during training . The interference between tasks lies in the competition for resources in the shared encoder fθsh . Based on empirical observations we argue that multi-task networks not only benefit when the cosine is non-negative but more so when task gradients are close to orthogonal . In a continual learning setting it makes sense for task gradients to be as aligned as possible in order to avoid catastrophic forgetting ( Riemer et al. , 2018 ) . In a multi-task setting it is however unclear whether maximizing the transfer between tasks also leads to a superior solution for all objectives , especially since there is no risk of forgetting . In our experiments we observe a higher performance when orthogonalizing correlated tasks on the SUN RGB-D dataset . Standley et al . ( 2019 ) make a similar finding that learning related tasks does not necessarily improve the multi-task optimization . In our approach we minimize the squared cosine during training which diminishes competition as each task will be able to optimize different parameters of the encoder . By also orthogonalizing positive transfer the encoder will produce a richer feature space and multi-purpose representations . To minimize the cosine between two task gradients we simply add the squared cosine to the multitask objective function from Equation 1 with an additional hyper-parameter α ∈ R to adjust the penalty weight : Ltitj = wtiLti + wtjLtj + α cos2 ( ti , tj ) ( 4 ) We can generalize Equation 4 to T tasks by taking the squared Frobenius norm of the cosine distance matrix between gradients . We define∇θsh as the column vector of unit normalized partial derivatives of the task losses with respect to θsh . The distance matrix for T tasks ( cos2 ( ti , tj ) ) 1≤i , j≤T can be then efficiently computed by taking the outer product of ∇θsh with itself . Subsequently we subtract the constant distance between identical tasks and normalize by accounting for the matrix symmetry as well as the number of task pairs in order to ensure the same bounds for the penalty term . ∇θsh = ( ˆ∂Lt1 ∂θsh , ˆ∂Lt2 ∂θsh , ... , ˆ∂LtT ∂θsh ) LT = ∑ t∈T wtLt + α T ( T − 1 ) ‖∇ᵀθsh∇θsh − IT ‖ 2 F ( 5 ) The above equation generalizes the gradient regularization term for T tasks and maintains its range within [ 0 , 1 ] . Even though wt are hyper-parameters , our focus is solely on the cosine regularization and will therefore treat them as constants . In all the experiments we add our penalty term to the naive approach of having all tasks equally weighted . Computing the regularization term for each layer in the shared encoder is computationally prohibitive , so in practice we restrict ourselves to computing the loss with respect to only the last layer of the encoder . Finally , we will refer to Φ ( ti , tj ) as the distribution of cosines between the gradients of ti and tj throughout training , having mean µ ( ti , tj ) and standard deviation σ ( ti , tj ) . Our gradient regularization method minimizes σ ( ti , tj ) , which will be empirically shown later on .

Summary: This paper embraces the idea that better multi-task/lifelong learning can be achieved if tasks produce gradients that are orthogonal to the gradients produced by other tasks. The authors propose an approach to regularizing learning in order to incentivize this to happen. However, as they mention themselves, the regularized loss is computationally intractable in general and they only apply it to a subset of their network as a result. Given the computational scalability concerns, it is natural to wonder why researchers in the community would adopt this approach rather than other approaches that also aim to make gradients orthogonal.