Multi-Task Learning by a Top-Down Control Network

1 INTRODUCTION . The goal of multi-task learning is to improve the learning efficiency and increase the prediction accuracy of multiple tasks learned and performed in a shared network . In recent years , several types of architectures have been proposed to combine multiple tasks training and evaluation . Most current schemes assume task-specific branches , on top of a shared backbone ( Figure 1a ) and use a weighted sum of tasks losses for training ( Chen et al. , 2017 ; Sener & Koltun , 2018 ) . Having a shared representation is more efficient from the standpoint of memory and sample complexity ( Zhao et al. , 2018 ) , but the performance of such schemes is highly dependent on the relative losses weights that can not be easily determined without a “ trial and error ” search phase ( Kendall et al. , 2018 ) . Another type of architecture ( Zhao et al. , 2018 ; Strezoski et al. , 2019 ) uses task-specific vectors to modulate the feature-maps along a feed-forward network , in a channel-wise manner ( Figure 1b ) . Channel-wise modulation based architecture has been shown to decrease the destructive interference between conflicting gradients of different tasks ( Zhao et al. , 2018 ) and allowed Strezoski et al . ( 2019 ) to scale the number of tasks without changing the network . Here , both training and evaluation use the single tasking paradigm : executing one task at a time , rather than getting responses to all the tasks in a single forward pass . Executing one task at a time is also possible by integrating task-specific modules along the network ( Maninis et al. , 2019 ) . A limitation of using task-specific modules ( Maninis et al. , 2019 ) or of using a fixed number of branches ( Strezoski et al. , 2019 ) , is that it may become difficult to add additional tasks at a later time during the system life-time . We propose a new type of architecture with no branching , which performs a single task at a time with no task-specific modules . Our model is trained to perform a set of tasks ( { ti } Ti=1 ) one task at a time . The model receives two inputs : the input image , and a learned vector that specifies the selected task tk to perform . It is constructed from two main parts ( Figure 1c ) : a main recognition network that is common to all tasks , termed below BU2 ( BU for bottom-up ) , and a control network that modifies the feature-maps along BU2 in a manner that will compute a close approximation to the selected task tk . As detailed below , the control network itself is built from two components ( Figure 1d ) : a top-down ( TD ) network that receives as inputs both a task vector as well as image information from a bottom-up stream termed BU1 ( Figure 1d ) . As a result , the TD stream combines task information with image information , to control the individual units of the feature-maps along BU2 . The modification of units activity in BU2 therefore depends on the task to perform , the spatial location , and the image content extracted by BU1 . As shown later , the task control by our approach becomes highly efficient in the sense that the recognition network becomes tuned with high specificity to the selected task tk . Our contributions are as follow : a . Our new architecture is the first to modulate a multi-task network as a function of the task , location ( spatial-aware ) and image content ( image-aware ) . All this is achieved by a top-down stream propagating task , image and location information to lower levels of the bottom-up network . b . Our scheme provides scalability with the number of tasks ( no additional modules / branches per task ) and interpretability ( Localization of relevant objects at the end of the top-down stream ) . c. We show significantly better results than other state-of-the-art methods on four datasets : MultiMNIST ( Sener & Koltun , 2018 ) , CLEVR ( Johnson et al. , 2017 ) , CELEB-A ( Liu et al. , 2015 ) and CUB-200 ( Welinder et al. , 2010 ) . Advantages are shown in both accuracy and effective learning . d. We introduce a new measure of task specificity , crucial for multi-tasking , and show the high task-selectivity of our scheme compared with alternatives . 2 RELATED WORK . Our work draws ideas from the following research lines : Multiple Task Learning ( MTL ) Multi-task learning has been used in machine learning well before the revival of deep networks ( Caruana , 1997 ) . The success of deep neural networks in the performance of single tasks ( e.g. , in classification , detection and segmentation ) has revived the interest of the computer vision community in the subject ( Kokkinos , 2017 ; He et al. , 2017 ; Redmon & Farhadi , 2017 ) . Although our primary application area is computer vision , multi-task learning has also many applications in other fields like natural language processing ( Hashimoto et al. , 2016 ; Collobert & Weston , 2008 ) and even across modalities ( Bilen & Vedaldi , 2016 ) . Over the years , several types of architectures have been proposed in computer vision to combine the training and evaluation of multiple tasks . First works used several duplications ( as many as the tasks ) of the base network , with connections between them to pass useful information between the tasks ( Misra et al. , 2016 ; Rusu et al. , 2016 ) . These works do not share computations and can not scale with the number of tasks . More recent architectures , which are in common practice these days , assume task-specific branches on top of a shared backbone , and use a weighted sum of losses to train them . The joint learning of several tasks has proven beneficial in several cases ( He et al. , 2017 ) , but can also decrease the accuracy of some of the tasks due to limited network capacity , the presence of uncorrelated gradients from the different tasks and different rates of learning ( Kirillov et al. , 2019 ) . A naive implementation of multi-task learning requires careful calibration of the relative losses of the different tasks . To address these problem several methods have been proposed : ‘ Grad norm ’ ( Chen et al. , 2017 ) dynamically tunes gradient magnitudes over time to obtain similar rates of learning for the different tasks . Kendall et al . ( 2018 ) uses a joint likelihood formulation to derive task weights based on the intrinsic uncertainty in each task . Sener & Koltun ( 2018 ) applies an adaptive weighting of the different tasks , to force a pareto optimal solution on the multi-task problem . Along an orthogonal line of research , other works suggested to add task-specific modules to be activated or deactivated during training and evaluation , depending on the task at hand . Liu et al . ( 2019 ) suggests task specific attention networks in parallel to a shared recognition network . Maninis et al . ( 2019 ) suggests adding several types of low-weight task-specific modules ( e.g. , residual convolutional layers , squeeze and excitation ( SE ) blocks and batch normalization layers ) along the recognition network . Note that the SE block essentially creates a modulation vector , to be channelwise multiplied with a feature-map . Modulation vectors have been further used in Strezoski et al . ( 2019 ) for a recognition application , in Cheung et al . ( 2019 ) for continual learning applications and in Zhao et al . ( 2018 ) for a retrieval application and proved to decrease the destructive interference between tasks and the effect of catastrophic forgetting . Our design , in contrast , does not use a multi-branch architecture , nor task-specific modules . Our network is fully-shared between the different tasks . Compared to Zhao et al . ( 2018 ) , we modulate the feature-maps in the recognition network both channel-wise and spatial-wise , also depending on the specific image at hand . Top-Down Modulation Networks Neuroscience research provides evidence for a top-down context , feedback and lateral processing in the primate visual pathway ( Gazzaley & Nobre , 2012 ; Gilbert & Sigman , 2007 ; Lamme et al. , 1998 ; Hopfinger et al. , 2000 ; Piëch et al. , 2013 ; Zanto et al. , 2010 ) where top-down signals modulate the neural activity of neurons in lower-order sensory or motor areas based on the current goals . This may involve enhancement of task-relevant representations or suppression for task-irrelevant representations . This mechanism underlies humans ability to focus attention on task-relevant stimuli and ignore irrelevant distractions ( Hopfinger et al. , 2000 ; Piëch et al. , 2013 ; Zanto et al. , 2010 ) . In this work , consistent with this general scheme , we suggest a model that uses top-down modulation in the scope of multi-task learning . Top down modulation networks with feedback , implemented as conv-nets , have been suggested by the computer vision community for some high level tasks ( e.g. , re-classification ( Cao et al. , 2015 ) , keypoints detection ( Carreira et al. , 2016 ; Newell et al. , 2016 ) , crowd counting ( Sam & Babu , 2018 ) , curriculum learning ( Zamir et al. , 2017 ) , etc . ) and here we apply them to multi-task learning applications . 3 APPROACH . A schematic illustration of our network is shown in Figures 1c and 2a , and explained further below . A control network is used in this scheme to control each of the units in the main recognition network ( BU2 ) given the task and the current image . In practice , this scheme is implemented by using three separate sub-networks ( two of them identical ) with lateral inter-connections . We next describe the network architecture and implementation in detail . Overall structure and information flow Our model contains three essentially identical subnetworks ( BU1 , TD , BU2 ) , with added lateral connections between them . The networks BU1 , BU2 share identical weights . The network receives two inputs : an input image ( to both BU1 , BU2 ) , and a task specification provided at the top of the TD network by a one-hot vector selecting one of k possible tasks . The processing flows sequentially through BU1 , TD ( in a top-down direction ) , BU2 , and the final output is produced at the top of BU2 . In some cases , discussed below , we used an additional output ( object locations ) at the bottom of TD . During the sequential processing BU1 first creates an initial image representation . The TD creates a learned representation of the selected task that converts the one-hot vector to a new form ( task embedding ) , and propagates it down the layers . On the way down the TD stream also extracts relevant image information from the BU1 representation via the BU1-TD lateral connections . Finally , the TD stream controls the BU2 network , to apply the selected task to the input image via the TD-BU2 lateral connections . Bottom-Up streams The BU streams use a standard backbone ( such as Resnet , VGG , LeNet , etc . ) , which is usually subdivided into several stages followed by one or more fully-connected layers , including the final classifier . The lateral connections between streams are placed at the end of each stage , connecting between tensors of the same sizes , allowing element-wise modifications . Top-down stream The TD stream we use ( unless stated otherwise ) is a replica of the BU stream in terms of number of layers , type of layers ( convolutional / residual ) and number of channels in each layer . The downsampling layers ( used in the BU stream ) are replaced with upsampling layers ( nearest neighbour interpolation layers ) . This design allows us to immediately extend any given BU backbone to our scheme , and it gives good results in our comparison , but the optimal TD structure is subject to future studies . The TD stream has two inputs : the selected task at its top , and inputs from BU1 via lateral connections . The selected task is usually specified by a one-hot vector , which is transformed via learnable weights , into a learned task-representation tensor ( called the task embedding , ‘ Emb ’ in Figure 2a ) that serves as an input to the TD stream . lateral connections For the lateral connections , we experimented extensively with different types , and based on the results we selected two types of connections , one for BU1-to-TD , which is additive , the second for TD-to-BU2 , which is multiplicative , shown in Figures 2b and 2c . More details and ablation studies of the lateral connections are given in the Supplementary . Auxiliary losses The use of three sub-networks ( BU1 , TD , BU2 ) suggest the natural use of auxiliary losses at the end of the BU1 or TD streams . In the scope of multi-task learning , the TD auxiliary loss can be used to train the extraction of useful spatial information such as the detection of task-relevant objects . This issue is further discussed in Section 4.2 where we demonstrate the use of a localization loss in the last TD feature-map . We show that applying a localization loss allows us to obtain a task-dependent spatial map in inference time , helping interpretability by locating objects of interest . Training & evaluation During training , the learning optimizes all the weights along the BU and TD streams , shared by all tasks , as well as the task specific embedding parameters . Learning uses a standard backpropagation , as the full model forms an end-to-end trainable model . In training time , the network is supplied with an input image and a selected task , drawn at random from the different tasks . During testing , the different tasks are applied sequentially to each test image . We used in our implementation shared weights between BU1 and BU2 streams . The main motivation for this design was to allow in future applications a multi-cycle use of the model , by using the BU and TD streams iteratively . With this broader goal in mind , our scheme can also be seen as an unfolded version of a BU-TD recurrent network for one and a half cycles , which is the minimal cycles that allow an image-aware modification process .

Summary: In this paper a novel top-down control network is introduced for multi-task learning. Different from the traditional bottom-up attention models, the authors introduce a top-down module to modify the activation of recognition network based on different tasks. Specifically，the proposed module consists of three identical networks, which are BU1, TD, BU2 streams. Given the input, the BU1 is firstly trained, and then the TD streams is trained by assigning the specific labels. After that, the BU2 is updated with the top-down parameters. Experimental results demonstrate the effectiveness of proposed model.