Task-Based Top-Down Modulation Network for Multi-Task-Learning Applications

1 INTRODUCTION . The goal of multi-task learning is to improve the learning efficiency and increase the prediction accuracy of multiple tasks learned and performed together in a shared network . Over the years , several types of architectures have been proposed to combine multiple tasks training and evaluation . Most current schemes assume task-specific branches , on top of a shared backbone ( Figure 1a ) and use a weighted sum of tasks losses , fixed or dynamically tuned , to train them ( Chen et al. , 2017 ; Kendall et al. , 2018 ; Sener & Koltun , 2018 ) . Having a shared representation is more efficient from the standpoint of memory and sample complexity and can also be beneficial in cases where the tasks are correlated to each other ( Maninis et al. , 2019 ) . However , in many other cases , the shared representation can also result in worse performance due to the limited capacity of the shared backbone and interference between conflicting gradients of uncorrelated tasks ( Zhao et al. , 2018 ) . The performance of the multi-branch architecture is highly dependent on the relative losses weights and the task correlations , and can not be easily determined without a ” trial and error ” phase search ( Kendall et al. , 2018 ) . Another type of architecture ( Maninis et al. , 2019 ) that has been recently proposed uses task specific modules , integrated along a feed-forward backbone and producing task-specific vectors to modulate the feature-maps along it ( Figure 1b ) . Here , both training and evaluation use a single tasking paradigm : executing one task at a time , rather than getting all the task responses in a single forward pass of the network . A possible disadvantage of using task-specific modules and of using a fixed number of branches , is that it may become difficult to add additional tasks at a later time during the system life-time . Modulation-based architectures have been also proposed by Strezoski et al . ( 2019 ) and Zhao et al . ( 2018 ) ( Figure 1c ) . However , all of these works modulate the recognition network channel-wise , using the same modulation vector for all the spatial dimension of the feature-maps . We propose a new type of architecture with no branching , which performs single task at a time but with no task-specific modules ( Figure 1d ) . The core component of our approach is a top-down ( TD ) modulation network , which carries the task information in combination with the image information , obtained from a first bottom-up ( BU1 ) network , and modulates a second bottom-up ( BU2 ) network common for all the tasks . In our approach , the modulation is channel-wise as well as spatial-wise ( a full tensor modulation ) , calculated sequentially along the TD stream . This allows us , for example , to modulate only specific spatial locations of the image depending on the current task , and get interpretability properties by visualizing the activations in the lowest feature-map of the TD stream . In contrast to previous works , our modulation mechanism is also ” image-aware ” in the sense that information from the image , extracted by the BU1 stream , is accumulated by the TD stream , and affects the modulation process . The main differences between our approach and previous approaches are the following : First , as mentioned , our approach does not use multiple branches or task-specific modules . We can scale the number of tasks with no additional layers . Second , our modulation scheme includes a spatial component , which allows attention to specific locations in the image , as illustrated in figure 2a for the Multi-MNIST tasks ( Sabour et al. , 2017 ) . Third , the modulation in our scheme is also image dependent and can modulate regions of the image based on their content rather than location ( relevant examples are demonstrated in figures 2b and 2c ) . We empirically evaluated the proposed approach on three different datasets . First , we demonstrated on par accuracies with the single task baseline on an uncorrelated set of tasks with MultiMNIST while using less parameters . Second , we examined the case of correlated tasks and outperformed all baselines on the CLEVR ( Johnson et al. , 2017 ) dataset . Third , we scaled the number of tasks and demonstrated our inherent attention mechanism on the CUB200 ( Welinder et al. , 2010 ) dataset . The choice of datasets includes cases where the tasks are uncorrelated ( Multi-MNIST ) and cases where the tasks are relatively correlated ( CLEVR and CUB200 ) . The results demonstrate that our proposed scheme can successfully handle both cases and shows distinct advantages over the channel-wise modulation approach . 2 RELATED WORK . Our work draw ideas from the following research lines : Multiple Task Learning ( MTL ) Multi task learning has been used in machine learning well before the revival of deep networks ( Caruana , 1997 ) . The success of deep neural networks in single task performance ( e.g . in classification , detection and segmentation ) has renewed the interests of the computer vision community in the field ( Kokkinos , 2017 ; He et al. , 2017 ; Redmon & Farhadi , 2017 ) . Although our primary application area is computer vision , multi task learning has also many application in other fields like natural language processing ( Hashimoto et al. , 2016 ; Collobert & Weston , 2008 ) and even across modalities ( Bilen & Vedaldi , 2016 ) . We further refer the interested reader to a review that summarizes recent work in the field ( Ruder , 2017 ) . Over the years , several types of architectures have been proposed in computer vision to combine the training and evaluation of multiple tasks . First works used several duplications ( as many as the tasks ) of the base network , with connections between them to pass useful information between the tasks ( Misra et al. , 2016 ; Rusu et al. , 2016 ) . These works do not share computations and can not scale with the tasks . More recent architectures , which are in common practice these days , assume task-specific branches on a top of a shared backbone , and use a weighted sum of losses to train them . The joint learning of several tasks has proven to be beneficial in several cases ( He et al. , 2017 ) but can also decrease the results of some of the tasks due to a limited network capacity , uncorrelated gradients from the different tasks ( sometimes called destructive inference ) and different learning rates ( Kirillov et al. , 2019 ) . A naive implementation of multi-task learning requires careful calibration of relative losses of the different tasks . To address these problem several methods have been proposed : ” Grad norm ” ( Chen et al. , 2017 ) dynamically tunes gradients magnitudes over time , to obtain similar learning rates of the different tasks . Kendall et al . ( 2018 ) uses a joint likelihood formulation to derive task weights based on the intrinsic uncertainty in each task . Sener & Koltun ( 2018 ) applies an adaptive weighting of the different tasks , to force a pareto optimal solution to the multi task problem . Along an orthogonal line of research , other works suggested to add task-specific modules to be activated or deactivated during training and evaluation , depending on the task at hand . Liu et al . ( 2019b ) suggests task specific attention networks in parallel to a shared recognition network . Maninis et al . ( 2019 ) suggests adding several types of low-weight task-specific modules ( e.g . residual convolutional layers , squeeze and excitation ( SE ) blocks and batch normalization layers ) along the recognition network . Note that the SE block essentially creates a modulation vector , to be channelwise multiplied with a feature-map . Modulation vectors have been further used in Strezoski et al . ( 2019 ) for a recognition application , in Cheung et al . ( 2019 ) for continual learning applications and in Zhao et al . ( 2018 ) for a retrieval application and proved to decrease the destructive interference between tasks and the effect of the catastrophic forgetting phenomena . Our design , in contrast , does not use multi-branch architecture , nor task-specific modules . Our network is fully-shared between the different tasks . Compared to Zhao et al . ( 2018 ) , we modulate the feature-maps in the recognition network channel-wise as well as spatial-wise , depending on both the task and the specific image at hand . Top-Down Modulation Networks Neuroscience research provides evidence for a top-down context , feedback and lateral processing in the primate visual pathway ( Gazzaley & Nobre , 2012 ; Gilbert & Sigman , 2007 ; Lamme et al. , 1998 ; Hopfinger et al. , 2000 ; Piëch et al. , 2013 ; Zanto et al. , 2010 ) where top-down signals modulate the neural activity of neurons in lower-order sensory or motor areas based on the current goals . This may involve enhancement of task-relevant representations or suppression for task-irrelevant representations . This mechanism underlies humans ability to focus attention on task-relevant stimuli and ignore irrelevant distractions ( Hopfinger et al. , 2000 ; Piëch et al. , 2013 ; Zanto et al. , 2010 ) . In this work , consistent with this general scheme , we suggest a model that uses top-down modulation in the scope of multi-task learning . Top down modulation networks with feedback , implemented as conv-nets , have been suggested by the computer vision community for some high level tasks ( e.g . re-classification ( Cao et al. , 2015 ) , keypoints detection ( Carreira et al. , 2016 ; Newell et al. , 2016 ) , crowd counting ( Sam & Babu , 2018 ) , curriculum learning ( Zamir et al. , 2017 ) etc . ) and here we apply them to multi-task learning applications . 3 APPROACH . We will first describe the task-dependent vector-modulation mechanism , as proposed in Zhao et al . ( 2018 ) , illustrated in figure 3a , and then describe our architecture ( figure 3c ) in detail . 3.1 VECTOR-MODULATION MODULE . A vector-modulation ( a channel-wise modulation ) of a given tensor X by a vector z is defined as the product of the elements of the vector z and the corresponding channels of the tensor X . Each element in the output tensor Y is calculated by : Y ( y , x , ch ) = X ( y , x , ch ) ∗ z ( ch ) ( 1 ) where X is the tensor to be modulated and Y is the modulated tensor , both in the form ( H×W ×C ) where H , W , are the spatial dimensions of the tensors and C their channel dimension . The vector z ∈ RC has dimension equal to the number of channels of X , Y. x , y , ch are the column , row , and channel number and indicate a specific element in a tensor . In training the network , the elements of z are considered as parameters and are directly optimized ( C additional parameters ) . In the scope of Multi Task Learning , where several tasks co-exist , the network switches on the fly between several modulation vectors { zk } K k=1 where K is the number of tasks , see figure 3a for illustration . The network performs one task at a time , and the modulated tensor Y depends on the selected task . This vector-modulation module has been used in Zhao et al . ( 2018 ) separately for every stage in the recognition network ( with additional CK parameters in every stage ) . Two limitations of this module are that it ignores the spatial dimensions of the image , and the lack of information from the image itself . The possible use of the same strategy to explicitly optimize spatial-aware modulation tensors { Zk ( y , x , ch ) } K k=1 ( figure 3b ) was discussed in Zhao et al . ( 2018 ) but was deemed infeasible due to the large amount of added parameters ( HWCK additional parameters in every stage ) . Our method addresses both of these issues in an efficient manner and demonstrates better accuracy , showing that spatial-wise modulation and the use of image information are beneficial to many kinds of tasks .

Summary: This paper introduces a new light-weight framework for multi-task learning. In this method, the combination of extracted image features and task are fed into a top-down network which is responsible for generating a task-specific weight matrix. The weights are next convolved with the input image as an input to the task-agnostic bottom-up network that generates the labels.