Multi-Task Learning by a Top-Down Control Network
Authors: Hila Levi
Shimon Ullman
Venue:  (2021)
Keywords: multi task learning
computer vision

Abstract: As the range of tasks performed by a general vision system expands, executing multiple tasks accurately and efﬁciently in  a single network has become an important and still open problem. Recent computer vision approaches address this problem by branching networks, or by a channel-wise modulation of the network feature-maps with task speciﬁc vectors. We present a novel architecture that uses a dedicated top-down control network to modify the activation of all the units in the main recognition network in a manner that depends on the selected task, image content, and spatial location. We show the effectiveness of our scheme by achieving signiﬁcantly better results than alternative state-of-the-art approaches on four datasets. We further demonstrate our advantages in terms of task selectivity, scaling the number of tasks and interpretability. 
Code is supplied in the supplementary materials and will be publicly available.

Under review as a conference paper at ICLR 2021
MULTI-TASK LEARNING BY A TOP-DOWN CONTROL NETWORK
Anonymous authors Paper under double-blind review
ABSTRACT
As the range of tasks performed by general vision system expands, executing multiple tasks accurately and efficiently in a single network has become an im- portant and still open problem. Recent computer vision approaches address this problem by branching networks, or by a channel-wise modulation of the network maaus with task specifie vectors. We present novel architecture that uses a dedicated top-down control network to modify the activation of all the units in the main recognition network in a manner that depends on the selected task, image content, and spatial location. We show the effectiveness of our scheme by achieving significantly better results than alternative state-of of-the-art approaches on four datasets. We further demonstrate our advantages in terms of task selectivity, scaling the number of tasks and interpretability.
Code is supplied in the Supplementary material and will be publicly available.
INTRODUCTION
The goal of multi-task learning is improve the learning efficiency and increase the prediction accuracy of multiple tasks learned and performed in a shared network.
In recent years, several types of architectures have been proposed to combine multiple tasks training and evaluation. Most current schemes assume task-specific branches, on top of a shared backbone (Figure [a) and use a weighted sum of tasks losses for training (Chen al. 2017; [Sener & Koltun 2018 Having a shared representation is more efficient from the standpoint of memory and sample complexity (Zhao et al.) 2018), but the performance of such schemes is highly dependent on the relative losses weights that cannot be easily determined without "trial and error" search phase
(Kendall et al. 2018).
Another type of architecture (Zhao et al. 2018: Strezoski et al., 2019) uses task-specific vectors to modulate the feature-maps along a feed-forware network, in a channel-wise manner (Figure [bb.
Channel-wiss modulation based architccture has been shown to decrease the destructive interference between conflicting gradients of different tasks (Zhao et al. 2018) and allowed Strezoski et al.) (2019) the
scale the number of tasks without changing the network. Here, both training and evaluation single tasking paradigm: executing one task at a time, rather than getting responses to all the tasks in a single forward pass. Executing one task a time is also possible by integrating task-specific modules along the network (Maninis et al. 2019). A limitation of using task-specific modules (Maninis et al.) 2019) or of using a fixed number of branches (Strezoski al. 2019), is that may become difficult to add additional tasks at a later time during the system life-time.
We propose a new type of architecture with no branching. which performs single task at a time with no task-specific modules. Our model is trained to perform set of tasks (t,)=) one task at a time. The model receives two inputs: the input image, and learned vector that specifies the selected task I to perform. It is constructed from two main parts (Figure Ic]: a main recognition network that is common to all tasks, termed below BU2 (BU for bottom-up), and a control network that modifies the feature-maps along BU2 in manner that will compute close approximation to the selected task ty- As detailed below, the control network itself is built from two components (Figure Idh: top-down (TD) network that receives as inputs both task vector well as image information from bottom-up stream termed BUI (Figure Id). As result, the TD stream combines task information with image information, to control the individual units of the feature-maps along BU2. The modification of units

---

Under review as a conference paper at ICLR 2021
TASK
TASK
TASK
TD
Condrol
(a)
(b)
(c)
(d)
architecture, task-specific branches on top of shared backbone. (b) Channel- wise modulation architecture, task vectors modulate the feature maps along the main network. (c) Our architecture uses top-down (TD) control network and modifies the featuremaps along the recognition net (BU2) dement-wise. according to the image and to the current task. (d) The internal structure of the control network: image information, extracted by BU1, is combined with task information and accumulated by the TD stream, to control the units along BU2. (c) Images examples with their corresponding tasks. upper part: M-MNIST, the task to recognize all the digits, CELEB-A, example tasks are the classification smile, sunglasses or earrings, lower part, left: CLEVR, example task to recognize the material of the cylinder to the right of the blue cube. right: CUB200, an example task is to identify the color of the bird's neck.
Figure 1: Schemes and Datasets: (a)
activity in BU2 therefore depends on the task perform, the spatial location, and the image content extracted by BU1. As shown later, the task control by our approach becomes highly efficient in the sense that the recognition network becomes tuned with high specificity to the selected task 4x-
Our contributions are as follow:
Our new architecture is the first modulate multi-task network function of the task, location (spatial-aware) and image content (image-aware). All this is achieved by top-down stream propagating task, image and location information to lower levels of the bottom-up network.
b. Our scheme provides scalability with the number of tasks (no additional modules branches per task) and interpretability (Localization of relevant objects the end of the top-down stream).
c. We show significantly better results than other state-of- the-art methods on four datasets: Multi- MNIST (Sener & Koltun 2018), CLEVR (Johnson et al. 2017). CELEB-A (Liu al. 2015 and
CUB-200 (Welinder et al., 2010). Advantages are shown in both accuracy and effective learning.
d. We introduce a new measure of task specificity, crucial for multi-tasking, and show the high talk-seleytivity of our scheme compared with alternatives.
RELATED WORK
Our work draws ideas from the following research lines:
Multiple Task Learning (MTL) Multi-task learning has been used in machine learning well before the revival of deep networks (Caruana 1997). The success deep neural networks in the performance of single tasks (e.g., in classification, detection and segmentation) has revived the interest of the computer vision community in the subject (Kokkinos) 2017 He et al., 2017 Redmon & Farhadi 2017). Although our primary application area is computer vision, multi-task learning has also many applications in other fields like natural language processing (Hashimoto et al. 2016 Collobert & Weston 2008) and even across modalities (Bilen & Vedaldi. 2016).
Over the years, several types of architectures have been proposed in computer vision to combine the training and evaluation of multiple tasks. First works used several duplications (as many as the tasks) of the base network, with connections between them to pass useful information between the tasks (Misra et al. 2016, Rusu et al., 2016). These works do not share computations and cannot scale with the number of tasks. More recent architectures, which are in common practice these days, assume task-specific branches on top of a shared backbone, and use a weighted sum of losses to train them. The joint learning of several tasks has proven beneficial in several cases (He et al.) 2017), but can also decrease the accuracy of some of the tasks due to limited network capacity, the presence of uncorrelated gradients from the different tasks and different rates of learning (Kirillov al. 2019). A naive implementation of different tasks. To address these problem several methods have been proposed: 'Grad norm' (Chen
learning requires careful calibration of the relative losses of the
2

---

Under review as a conference paper at ICLR 2021
et al. 2017) dynamically tunes gradient magnitudes over time obtain similar rates of learning for the different tasks. Kendall et al. (2018) uses a joint likelihood formulation to derive task weights based on the intrinsic uncertainty in each task. [Sener & Koltun (2018) applies an adaptive weighting of the different tasks, to force a pareto optimal solution on the multi-task problem.
Along orthogonal line of research, other works suggested to add task-specific modules to be activated or deactivated during training and evaluation, depending on the task at hand. Liu et al. (2019) suggests task specific attention networks in parallel to a shared recognition network. Maninis et al. (2019) suggests adding several types of low-weight task-specific modules (e.g., residual convolutional layers, squeeze and excitation (SE) blocks and batch normalization layers) along the recognition network. Note that the SE block essentially creates a modulation vector, to be channel- wise multiplied with a feature-map. Modulation vectors have been further used in [Strezoski al. (2019) for a recognition application, in [Cheung et al. (2019) for continual learning applications and in Zhao et al. (2018) for retrieval application and proved decrease the destructive interference between tasks and the effect of catastrophic forgetting.
Our design, in contrast, does not use a multi-branch architecture, nor task-specific modules. Our network is fully-shared between the different tasks. Compared to Zhao et al. (2018), we modulate the feature-maps in the recognition network both channel-wist and spatial-wise wise, also depending on the specific image at hand.
Top-Down Modulation Networks Neuroscience research provides evidence for top-down con-
text, feedback and lateral processing in the primate visual pathway (Gazzaley & Nobre 2012 Gilbert
& Sigman 2007 Lamme al. 1998; Hopfinger al. 2000, Piëch et al. [013 Zanto
2010)
where tow- signals modul ate the neural activity of neurons in lower-order sensory or motor areas based on the current goals. This may involve enhancement of task-relevant representations or
suppression for task-irrelevant representations. This mechanism underlies humans ability to focus
attention on rele-relevant stimuli and ignore irrelevant distractions (Hoptinger al. 2000 Piech
et al. 2013 Zanto et al. (2010).
In this work, consistent with this general scheme, we suggest a model that uses top-down modulation in the scope of multi-task learning. Top down modulation networks with feedback, implemented as conv-nets, have been suggested by the computer vision community for some high level tasks (e.g..
re-classification (Cao et al.] (013)) keypoints detection (Carreira et al., 2016 Newell et al.] (2016). crowd counting (Sam & Babu] 2018), curriculum learning (Zamir et al. 2017 ) etc.) and here we apply them to multi-task learning applications.
APPROACH
A schematic illustration of our network is shown in Figures Ic and 2a and explained further below. control network is used in this scheme to control each of the units in the main recognition network (BU2) given the task and the current image. In practice, this scheme is implemented by using three separate sub-networks (two of them identical) with lateral inter-connections. We next describe the network architecture and implementation in detail.
Overall structure and information flow Our model contains three essentially identical subnet- works (BU1, TD, BU2), with added lateral connections between them. The networks BUI, BU2 share identical weights. The network receives two inputs: input image (to both BUI, BU2), and task specification provided at the top of the TD network by a one-hot vector selecting one of k possible tasks. The processing flows sequentially through BU1, TD (in top-down direction). BU2, and the final output is produced at the top of BU2. In some cases, discussed below, we used an additional output (object locations) at the bottom of TD. During the sequential processing BUI first creates an initial image representation. The TD creates learned representation of the selected task that converts the one-hot vector to a new form (task embedding), and propagates down the layers. On the way down the TD stream also extracts relevant image information from the BUI representation via the BU1-TD lateral connections. Finally, the TD stream controls the BU2 network, apply the selected task to the input image via the TD-BU2 lateral connections.
Bottom-Up streams The BU streams use a standard backbone (such as Resnet, VGG, LeNet, etc.), which is usually subdivided into several stages followed by more fully-connected layers,
3

---

Under review as a conference paper at ICLR 2021
Optional audiiary
Emb Task
Prediction
loyers
TD connectivity ity
BU2 connectivity
down
SuB CoRM
BU1
come layers
1xt comm
3x3
Image
Optional auxiliary localization loss
Image
(a)
(b)
(c)
Figure 2: (a) The model architecture. The model consists of three identical networks, BUI, TD, BU2 (e.g.. each one a Resent-188 18). BUI and TD together constitute the TD-control network in figure Tc The imput consists of the input image (to BU1, BU2), and task specification the top of TD. The flow in TD goes from top to bottom; the processing flows sequentially through BUI, TD, BU2, with final output the top of BU2. (b.c) The lateral connections implemented as Ixl convolutions. Connections from BUI to (b) are additive, from to BU2 (c) are multiplicative.
including the final classifier. The lateral connections between streams placed the end of each stage, connecting between tensors of the same sizes, allowing element-wise modifications.
Top-down stream The TD stream we use (unless stated otherwise) is a replica of the BU stream in terms of number of layers, type of layers (convolutional / residual) and number of channels in each layer. The downsampling layers (used in the BU stream) are replaced with upsampling layers (nearest immediately extend any given BU backbone
neighbour interpolation layers). This design allows to our scheme, and it gives good results in our comparison, but the optimal TD structure is subject to future studies. The TD stream has two inputs: the selected task its top, and inputs from BU1 via lateral connections. The selected task is usually specified by one-hot vector, which is transformed via learnable weights, into learned reskssepresnnttioon tensor (called the task embedding, 'Emb' in Figure 2a) that serves as an input to the TD stream.
lateral connections For the lateral connections, we experimented extensively with different types, and based the results selected types of connections, for BU1-to-Til which is additive, the second for TD-to-BE BU2, which is multiplicative, shown in Figures 2b and 2c More details and ablation studies of the lateral connections are given in the Supplementary.
Auxiliary losses The use of three sub-networks (BU1, TD, BU2) suggest the natural use of auxiliary losses at the end of the BUI or TD streams. In the scope of multi-task learning, the TD auxiliary loss can be used to train the extraction of useful spatial information such the detection of task-relevant objects. This issue is further discussed Section 4.2 where we demonstrate the use of a localization loss in the last TD feature-map. We show that applying localization loss allows vask-dependent spatial map inference time, helping interpretability by locating objects interest.
obtain
Training evaluation During training, the learning optimizes all the weights along the BU and TD streams, shared by all tasks, well as the task specific embedding parameters. Learning uses standard backpropagation, as the full model forms an end-to to-enn trainable model.
In training time, the network is supplied with input image and selected task, drawn random from the different tasks. During testing, the different tasks are applied sequentially to each test image.
We used in our implementation shared weights between BUI and BU2 streams. The main motivation for this design was to allow in future applications a multi-cycle use the model, by using the BU and TD streams iteratively. With this broader goal in mind, scheme also be seen unfolded version of a BU-TD recurrent network for one and half cycles, which is the minimal cycles that allow image-aware modification process.
4

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTS
We validated our approach on four datasets (Multi-MNIST, CLEVR, CELEB-A and CUB-200) with tasks ranging from low-level (e.g., colors) to higher-level (e.g., CLEVR configurations, facial features) recognition, and from simple (e.g.. classification by location) more complex tasks (e.g.. classification by combined attributes and spatial relations). We compared our approach to alternatives in terms of prediction accuracy, scaling to larger number of tasks and task selectivity.
4.1 DATASETS & TASKS
Multi-MNIST (Sabour et al., 2017) is a version of the MNIST dataset in which multiple MNIST images are placed on a grid with some spatial overlaps, previously used in Sener & Koltun (2018). We used 2x1, 2x2, 3x3 grids; several training examples are shown in Figure Te We used Multi-MNIST to test performance on two sets of tasks: (a) recognizing digit at selected location ( ("by loc', e.g.. to recognize the digit in the upper-right location) and (b) recognizing digit which is right to another digit ("by ref", e.g., to recognize the digit to the right of the digit '7").
is synthetic dataset, consisting of 70K training images and 15K validation images. The dataset includes images of simple 3D objects, with multiple attributes (shape, size, color and material) pairs. We used the CLEVR dataset to test performance
CLEVR
together with corresponding (question-answer on sets of 'by ref' tasks, scaling the number of tasks up to 1645, with a fixed model size. We created multiple tasks by randomly choosing 40, 80, 160 and 1645 queries about an attribute of an object to the (left, right, up, down) of referred object. An example task is: "What is the color of the object to the left of the metal cylinder?" (metallic cylinder the referred object).
CELEB-A is a of real-world celeb face images, intensively used in the scope of MTL (e.g.. Sener & Koltun (2018), Strezoski et al. (2019)) on attribute classification tasks. The dataset consists of 200K images with binary annotations on 40 face attributes related to expression, facial parts, etc.
CUB-200 is a fine-grained recognition dataset that provides 11,788 bird images of 200 bird species, previously used (Strezoski et al., 2019). We used CUB-200 to test performance on real-world images with low-level features, and to demonstrate our use of interpretability. An example task is to recognize the color of the bird's crown.
The datasets and corresponding tasks are illustrated in Figure Teland further discussed in the Supple- mentary material.
4.2 IMPLEMENTATION DETAILS
We performed five randomly initialized training runs for each of our experiments, and present average accuracy and standard deviation.
Table I: Main Hyperparameters
We used LeNet, VGG-11, VGG-7 and resnet- 18, as BU backbone architectures for the Multi-MNIST, CLEVR, CELEB-A and CUB- 200 experiments respectively. We used the Adam optimizer, and performed lcarning rate search over {1e-5, _ small validation set; the main hyperparameters (number of epochs, learning rate and batch size) are shown in Table [
dataset
epochs
M-NNIST CLEVR CELER
1em3
512
100
le-4
128
- 1} on a
1e-3
512
CUB
1c
128
We trained the full model end-to-end, using cross entropy loss at the end of BU2, In some of the experiments of CLEVR and CUB-200 datasets we added an auxiliary loss at the end of the TD stream. The target in this case a 224x224 mask, where a single pixel, blurred by Gaussian kernel (s.d. 3 pixels) was labeled as the target location. Training one task at a time, we minimized the cross-entropy loss over the 224x224 image at the end of the TD softmax output (which encourages a small detected area), for each visible ground-truth annotated object or part. This auxiliary loss, allows us, at inference, to create task-dependent spatial maps of detected objects; examples of interest are shown in figure and in the Supplementary material. For fair comparison, also trained another version of the channel modulation architecture with an additional regression loss, calculated by a FC layer at the top of the network, using the same ground truth annotations.
Database details, full architecture description, more hyperparameters and an analysis of the number of parameters in the architectures be found in the Supplementary material.
5

---

Under review as a conference paper at ICLR 2021
Table 2: Mean of repetitions M-MNIST. Our architecture consistently achieves highest accuracies.
digits
digits #P
9 digits P
by loc
9 digits F.
by ref Av. Are
ALG
Av. Ave
As. Ace
As. Age
Single mak
x2
90.46
x4
94.15
86.02
x10
50.33
scaling
grad-norm
x1.12
x1.37
91.39
x1.99
74.66
x2.11
kendall
x1.12
95.51
x1.37
91.48
x1.99
74.56
v2.11
20.36
mull-ubj-rp
x1.12
95.81
x1.37
91.24
x1.99
75.13
x2.11
29.82
task-romning
x1.004
95.12
x1.008
92.09
x1.018
80.52
x1.021
40.00
ch-mod
X1.004
95.87
x1.008
91.38
x1.018
76.56
x1.021
32.69
(extended)
x1.35
ControlNet (ours)
x1.29
96.67
x1.32
94.64
x1.39
88.07
x1.40
72.25
4.3 COMPARISONS:
We compared our method with existing alternatives from three main approaches (listed in Table 2 (i) A 'Single task' approach, where each task is performed by its own network, (ii) 'Multi-branched approach, where the sum of the individual losses is minimized (in 'Uniform scaling' the losses are equally weighted, whereas in 'mult-obj-opt" (Sener & Koltun 2018), 'kendall' (Kendall al.) 2018) and 'grad-norm' (Chen et al., 2017) the weights are dynamically tuned) and (iii) a 'Modulation' approach, where channel- -wise vectors modulate the recognition network in several of the net's ' stages ('ch-mod' (Zhao et al. 2018) uses learnable weights and 'task-routing'' (Strezoski et al. 2019) uses constant binary weights). For fair comparison with approach placed the task embeddings ("modulation aattonaaioo approach) and the lateral connections from the TD ("ControlNet, ours' approach) in the same recognition network locations followed by single branch for final recognition (the original implementation of "task-routing" uses multiple branches).
4.4 RESULTS
Table summarizes our results on the Multi-MNIST experiment with 2, 4 and digits. We show the average accuracy of all tasks based on 5 experiments for each row. The '# Pr column shows the number of parameters as a multiplier of the number of parameter a standard LeNet architecture. specified in the Supplementary
Detailed results with standard deviations and number of parameters material. Our method achieves significantly better results than the other approaches, even compared with the single-task baseline. Scaling the number of tasks increases the accuracy gap almost without additional parameters. The 'by ref" test (10 tasks) proved to be more difficult (lower accuracies). and shows significant gap between approach and other methods. Extending the codnnel-modulation scheme to a wider backbone (' (cch-mod(cxteeded) in the table, with 15 and 25 channels featuremaps) with roughly the same number of parameters as our scheme, maintains a large accuracy gap.
Our quantitative results for the CLEVR, CELEB-A and CUB-200 experiments are summarized in Table 3 The "#P' column shows the number of parameters as multiplier of the number of parameter used in the corresponding BU architecture, Experiments that used localization ground -ruuhd-truth data are indicated with / on the 'loc' column. The results show better accuracy of our scheme compared to all baselines. Table Table shows that our results on CLEVR, 40 tasks setting, surpass other methods by significant margin. Our advantage the CELEB-A and CUB-200 experiments smaller than in the former tests, possibly due to database biases (e.g., color of different bird's parts are highly correlated) and the diverse appearance of relevant attributes in these databases.
Table Performance on CLEVR, CELEB-A and CUB200. Our approach yields better accuracy than alternatives
(a) CLEVR
(b) CELEB-A
(c) CUB-200
+
#
Single task
x40
Single task
x40
Single task
x
74.34 +00.07
Uni-se
Uni-se
x31.04
10.31
ch-mod
x1.013
00.06
02.02
ch-mod
70 1:0.14
cb mod
x1.002
$7.05
10..9
task-routing
x1.013
$9.30
1.0.04
ch mod
79.9 1:0.18
ch-mod
x1.003
89.87
1:0. 40
ControlNet
x1.15
90.46
::0.03
lask noo-routing
80.04 1:0.22
Insk-routing
ControlNot
80399 1:0.00
ControlNet
.1..6
96.83
1.0.08

---

Under review as a conference paper at ICLR 2021
4.5 EXPERIMENTS DISCUSSION
100
We discuss below additional aspects of the experiments and general conclusions.
ww.ccurccy
80
Single tast Mutti betti-brncceed
Accuracy vs. model size tradeoff Figure demonstrates the av- erage accuracy of the 9-class experiment as function of the number of parameters in four types of architectures; top-left better. Large markers in the figure correspond to the fixed architectures used in the experiments above. Within each family, parameters can be changed by changing (uniformly) the number of channels along the network. Small markers correspond to modified network sizes: wider LeNet architectures (termed above (tt-mod(extended)) ), or reduced TD implementations (using TD streams with I, 4 6 channels along its meature-maps). Exact design choices are summarized in the Supplementary material. Our control network architectures correspond to the highest (red) curve in the plot, indicating higher performance for a similar number of parameters, A similar comparison for the CLEVR dataset is reported in the Supplementary material.
50
Our architecture achieves higher performance for a similar number of parameters.
Figure
Scaling the number of tasks a shows the accuracies of the Multi-MNIST experiment with the 2. and tasks datasets. Note that as the number of digits increases, the task also increases in difficulty due to the digits overlap. Increasing the number of tasks increases the accuracy gap compared with alternative models with a similar or even larger number of parameters.
Table 6a below shows the accuracies of gradually increasing the number of tasks up to 1645 on the CLEVR dataset. Compared with alternatives, our results are scalable with the number of tasks, with a smaller decrease performance for 1645 tasks. The uniform scaling approach cannot scale ahove 80 tasks due to memory restrictions, and did not take part in this experiment. Overall, the results show that scheme deals better in performing multiple tasks in limited capacity, the performance gap increases with the number of tasks and complexity and the scheme benefits from the contribution of spatial information and image content.
Adding Tasks A general question of interest in mutli-task learn- ing whether the tasks need to be pre-defined, compared with the possibility of adding at least some tasks at later stage to an already trained model. We used in this task an extension of the M-mnist 'by- ref" experiment. Specifically, our architecture has been trained and evaluated on the defined 'by ref" tasks, excluding the task involving the digit "9" (9 tasks in all). We then extended the embedding layer and trained it, while keeping the shows the results. The obtained accuracy accuracy remains unchanged (74.9%%).
Table 4: Adding task (Ace%)
before
after
existing tasks
74.89
74.89
added
16.30
64.68
rest of the model fixed, on the new task examples. Table I for the added digit '9' task are 64.68%, other tasks The new task accuracy is lower than the mean, but shows significant learning, compared with the pre-trained accuracy of 16.3%. The accuracy of all other tasks unaffected (avoiding 'catastrophic forgetting') without requiring any further training of the previous tasks.
Tasks heterogeneity Our model performs multiple tasks in the same network by applying didivaationmodiicaatti according to the task and to the image at hand. A question of interest whether the of tasks is limited to a homogenous set, such as similar classification tasks (as in the M-MNIST experiment), or extends to more heterogencous tasks. In particular, performing both recognition and pixel-labeling tasks in the same network using task selection schemes (e.g., channel modulation, routing, ours) has not been studied in the past.
Table S:
Hetrogenous tasks: executing recognition and segmentation with / without tasks selection.
diglts by CLS (Ace)
digits CLS (Ace)
loc inst
clalseg insl.
branches
SEG (JOU)
# branches
SEG (IOU)
ControlNet
x
$8.40
65.34
73.83
46.73
ch-mnd
2
76.67
61.16
46.96
38.97
ControlNet
$8.53
7

---

Under review as a conference paper at ICLR 2021
We extended our proposed architecture perform both recognition tasks (producing class labels) and segmentation tasks (producing a spatial map), guided by the TD instruction provided to the network. The instruction is composed of two parts: selecting digit, either by location or by reference (the "by loc' and the "by ref" vector in the M-MNIST experiments above), and then applying either recognition or segmentation, using two slots one-hot-vector (results in Table [ row 4). We compared this task selection with three alternatives. The first includes no instruction, implemented with an individual branch for each task (top row, multi-branched architecture). The second performs both segmentation and recognition together, implemented with two simultaneously executed branches (second row in the table). The final compares with channel modulation (row 3). To deal with both recognition and segmentation tasks we used two separate output branches (one producing one of 10 class labels, the other producing a 28*28 map) rather than a single one. The branches were used according to the task instruction: using one at a time when Mlassification/segeente
instruction is given (rows 3, 4), or using both branches simultancously when no such instruction is provided (row 2). Results show that our model with the selected branch (row 4) performs better than all alternatives. A direct comparison with the alternative of using the two branches together, as in standard (un-instructed) branching models (second row), reveals that the selected branch increases in performance, and the same time, the results of the un-selected branch significantly decreases in performance, by 10-30%.
Ablations and the use of image content We compared our scheme, which uses image information (via BUI) and performs full tensor clement-wise modification of the feature maps (of BU2) to the channel modulation scheme (which performs channel modulation with no image content information) with roughly the same number of parameters, shown in Table 6b top row (ch-mod (extended)). We also tested two ablations of our network: one without the BUT stream, removing the image-content contribution (TD, second row) and second without the TD stream, where the task is supplied to both BU streams, concatenated to the image data (BU, third row, details Supplementary). We conducted the experiments on two sets of tasks applied to the 9-location MNIST: classification by location ("by loc') ) and classification by reference ( ('by ref"), ) and on two sets of tasks used in the CLEVR dataset (40 and 1645 tasks), which are inherently a 'by ref" tasks, using reference objects. Table 66b below shows that ControlNet achieves significantly hetter results than its ablations, with gap that increases as the complexity or the number of tasks increase. The comparison between our model (4th row) and our ablated model (2nd row), which does not use the image content in the modification process, shows the significant advantage of using image content information. The addition of BUI (which shares weights with BU2) improves the accuracy substantially with almost no additional parameters.
Task Selectivity TTas--selectivee is likely to be a crucial property of architectures that execute one task at time (such as ch-mod, ours) in order to fully utilize the network resources for the selected task. We defined a sask-sclectivity measure by comparing the prediction accuracy of the model for the selected task vs. non-selected tasks. To make this comparison, we trained readout heads to predict from the final representation produced by the BU2 stream not just the selected task, but all possible tasks. For example, in the 4-digits MNIST, we trained four readout branches the top of the final representation (trained to produce the selected location) to predict the digit identities at all four locations. We define talk-seley by the ratio between the accuracy for the selected task (above chance level) and the average accuracy of the non-selected tasks (above chance level). Detailed results are shown in Figure 4a for the multi-MNIST, 4 tasks setting (top) and 9 tasks-setting (bottom). The results show over 90% accuracy for the selected task branch (along the diagonal), and close to chance-level accuracies for all other branches. Figure 4b) summarizes the results of the same experiment for the channel modulation architecture, which shows less selectivity. The corresponding selectivity indices for the 4-class case are 26.5 and 8.25 and for the 9-class case 37.23 and 7.97 for our model and poannel-modulation respectively. The higher selectivity index of our method is likely
Table 6: (a) Performance on CLEVR, Our approach is scalable with the number of tasks with an inercasing gap over ch.mod. (b) Test of ablations and image content. 'im': image contribution.
(a) Scaling the number of tasks
(b) Test of ablations and image content
number tasks
mnist loc
mn mnist by
cleve (40) ref
cleve
40
160
1645
method
im
by
uni-sc
68. 10
70.42
ch-mod (extended)
X
70.81
38.57
91.45
73.95
db-med
89.97
87.50
38
82.32
66.86
98.10
68.12
Comtoolees
96.85
95.60
$3.83
BU
55.77
93.21
73.77
ContnolNet
$8.07
72.25
96.85
88.83
8

---

Under review as a conference paper at ICLR 2021
14
15
grey
shape objes
63
67
down snall retal abject
14
14
20
21
Fh
cylisdee
13
14
94
1G
black bilu
4
Scares:
0. 246 0. 635 9.875
eylinder
(a) Ours
(b) ch-mod
Figure 4: (a.b) Task selectivity. The columns each plot are the selected task locations, rows are the readout locations. With high selectivity, off-diagonal recognition will be at chance level. Our scheme (a, selectivity indices 26.5, 37.2) shows higher selectivity than the Ghannel-modulatiin scheme (b, selectivity indices 8.3, 8.0). (c,d) Attention maps, obtained the end of the TD stream, highlighted with turquoise; intermediate objects of interest are well localized: (c) CUB-200: What is the color of the bird's crown? (d) CLEVR: What the shape of the object below the small metal object? Best viewed in color while zoomed-in.
be related the increased gap in performance with respect to alternative models in Tables 2 and
3 Results in the section on tasks heterogeneity further show that even for heterogeneous tasks, task
selection increases the performance of the selected task at the expense of the non non-selected tasks.
Task-dependent spatial maps Using task dependent localization loss at the end of the TD stream in train time allows us to obtain task-dependent spatial maps in inference time, helping interpretability of the result by locating intermediate objects of interest. Figures and 4d demonstrates the location maps produced by our architecture at inference time. In both examples, the predicted mask/object is well localized (on the crown of the bird or on the object below the small metal object) and the attribute (color or shape) is correctly predicted.
case of wrong result, it is possible to examine whether the error in the attribute was associated with mis-detecting the relevant part. Additional examples of interest and failure cases are shown Supplementary material.
SUMMARY
We described an architecture for multi-task learning, which is qualitatively different from previous models, primarily by the use of the BUI stream and a TD convolutional stream, which controls the final BU2 stream as function of the selected task, location and image content. We tested our network on four different datasets, showing improvements in accuracy compared with other schemes, along with scaling the number of tasks with minimal effect on performance, and helping interpretability by pointing to relevant image locations. Comparisons show higher task-selectivity our scheme, which may explain at least in part its improved performance.
More generally, multiple-task learning algorithms are likely to become increasingly relevant, since general vision systems need to deal with broad range of tasks, and executing them efficiently in a single network is still an open problem. Our task-dependent TD control network is a promising direction in this field in terms of accuracy and scalability. In future work plan adapt architecture wider range of applications (e.g., scene understanding, images generation), to wider range of architectures with higher capacity and to examine the possible combination of branching strategy with our TD task selection approach, extending our heterogeneous tasks example. Our architecture is also potentially useful for problems such as online learning, domain adaptation and catastrophic forgetting. demonstrated in part by our example of adding new task to an already trained model, and we plan to further explore these central problems in the future.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Hakan Bilen and Andrea Vedaldi. Integrated perception with recurrent multi-task neural networks. In Advances in neural information processing systems, pp. 235-243, 2016.
Chunshui Cao, Xianming Liu, Yi Yang, Yinan Yu, Jiang Wang, Zilei Wang, Yongzhen Huang, Liang Wang, Chang Huang, Wei Xu, et al. Look and think twice: Capturing top-down visual attention with feedback convolutional neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2956-2966 2015.
Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, and Jitendra Malik. Human pose estimation with iterative error feedback. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4733-4742, 2016.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient nor- malization for adaptive loss balancing deep multitask networks. arXiv preprint arX1v:171..02257. 2017.
Brian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Superposition of many models into one. arXiv preprint wrXiv:1902.05300 2019.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pp. 160-167, ACM, 2008
Adam Gazzaley and Anna C Nobre. Top-down modulation: bridging selective attention and working memory. Trends in cognitive sciences, 16(2):129-13) 135, 2012.
Charles D Gilbert and Mariano Sigman. Brain states; top-down influences in sensory processing. Neuron, 54(5):677-666, 2007.
Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. joint many-tast model: Growing neural network for multiple nip tasks. arXiv preprini arXiv: 1.01587, 2016 6.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Joseph Hopfinger, Michael Buonocore, and George Mangun. The neural mechanisms of top-down attentional control. Nature neuroscience, 3(3):844, 2000.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEFF Conference Computer Vision and Pattern Recognition, pp. 2901-2910, 2017.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7482-7491, 2018.
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmen- tation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2019.
2019.
lasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6129-6138, 2017.
Victor AF Lamme, Hans Super, and Henk Spekreijse. Feedforward, horizontal, and feedback processing in the visual cortex. Current opinion neurobiology, 8(4):529-5355 1998.
Shikun Liu, Edward Johns, and Andrew Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1871-1880, 2019.
10

---

Under review as a conference paper at ICLR 2021
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Devis-Kokitssi Maninis, 1liaa Radosavovic, and lasonas Kokkinos. Attentive single-tasking of multiple tasks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1851-1860, 2019.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994-4003, 2016.
Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. Springer, 2016.
In European conference on computer vision, pp.
Valentin Piech, Wu Li, George N Reeke, and Charles D Gilbert. Network model of top-down influences on local gain and contextual interactions in visual cortex. Proceedings of the National Academy of Sciences,
(0143))E4108-E41 17, 2013.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7263-7271, 2017.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 10xv::6606046671 2016.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in neural information processing systems, pp. 3856 33666, 2017.
Deepak Babu Sam and Venkatesh Babu. Top-down feedback for crowd counting convolutional neural network. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Ozan Sener and Vladlen Koltun. Multi-task learning multi-objective optimization. In Advances in Neural Information Processing Systems, pp. 527-538, 2018.
Gjorgji Strezoski, Nanne van Noord, and Marcel Worring. Many task learning with task routing. arXiv preprint 1903. 12117, 2019.
Welinder, Branson, Mita, C. Wah, Schroff, Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-00) Californi Institute of Technology, 2010.
Amir R Zamir, Te-Lin Wu, Lin Sun, William B Shen, Bertram E Shi, Jitendra Malik, and Silvio Savarese. Feedback networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1308-131 1317, 2017.
Theodore Zanto, Michael Rubens, Jacob Bollinger, and Adam Gazzaley. Top-down modulation of visual feature processing: the role of the inferior frontal junction. Neuroimage, 53(2):736-745, 2010.
Xiangyun Zhao, Haoxiang Li, Xiaohui Shen, Xiaodan Liang, and Ying Wu. A modulation module for multi-task learning with applications in image retrieval. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 401-416, 2018


Reviews:
[{"review_id": "zV6WAIo4qcI", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper propose a way to combine image information and task information as controllers for multi-task learning. In this way, the authors expect to extract more task/image specific features in a shared backbone.\n\nThe proposed method seems novel and intuitive. Though there are some typos and grammar mistakes, overall the paper is easy to follow. \n\nMy major concerns are the followings:\n1. the claim that the proposed scheme provides scalability with the number of tasks is stretching: the number of tasks still need to predefine and the number of task heads are not optimized compared to previous algorithms. \n2. While there is improvement in performance, but it is not clear what factors causes the improvement. As compared to previous schemes the proposed model needs much more computationally (two bottom up runs and a top-down run) and utilizes more information.\n3. All the tasks are classification tasks /discriminative models. It is not demonstrated if this would be working with a mixture of generative/discriminative tasks.  \n\nIf possible, as the TD and BU's are sharing the same structures, it would be interesting to explore what are learned by visualizing the weights in each channels and layers. and further explore which feature map is heavily used for which specific task. These weights/activation distributions will help us to better understand what is actually learned in the proposed scheme.  The task spatial maps seems to be a good start, but it would be better if the author provide more analysis on intermediate layer activations ", "extra_scores": {}, "extra_text": {}}, {"review_id": "YHBpmHsCN4J", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper a novel top-down control network is introduced for multi-task learning. Different from the traditional bottom-up attention models, the authors introduce a top-down module to modify the activation of recognition network based on different tasks. Specifically，the proposed module consists of three identical networks, which are BU1, TD, BU2 streams. Given the input, the BU1 is firstly trained, and then the TD streams is trained by assigning the specific labels. After that, the BU2 is updated with the top-down parameters. Experimental results demonstrate the effectiveness of proposed model.\n\nStrength:\n1. It is interesting to introduce the semantic information from the top layer to guide the feature representation learning.\n\nWeakness:\n Although the experimental results show the better performance on the image classification, there are exist several unclear parts:\n\n1. The definition of multi-task in this paper refers to the different dataset’s classification? Or referring to the different tasks, e.g. localization, classification, and attributes predication. In my opinion, the authors should provide more details on designing the validation experiments. And the proposed model should be tested on different tasks instead of only on the task of visual recognition.\n2. Some bottom-up based model e.g. FiLM should be used as the baselines to validate the advantage of introducing the top-down stream.\n3. If the top-down stream would make the recognition be sensitive for the visual variations? Or the classification results may be dependent on the number of training samples?\n\n4. If the proposed model would be helpful for transfer learning?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2Im2GKhT5un", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new architecture for multi-task learning that uses a top-down control network to modulate the activations of the main (bottom-up) recognition network. The model is applied to four datasets/tasks: multi-MNIST, CLEVR, CELEB-A, and CUB-200, demonstrating good performance compared with baselines. I have the following comments and questions:\n\n- The proposed architecture requires the equivalent of three forward/backward passes: BU1, TD, and BU2. The number of parameters is used gauge complexity in Table 2 and Figure 3 but FLOPs might be a better metric here since BU1 and BU2 share parameters. How do the models compare in terms of FLOPs to baselines? \n- The datasets/tasks used herein are homogenous and therefore straightforward for multi-task learning. How does the proposed architecture fair in a more challenging setting involving heterogeneous tasks, e.g., Misra et al., 2016?\n- Re \"In some of the experiments of CLEVR and CUB-200 datasets we added an auxiliary loss at the end of the TD stream. The target in this case is a 224x224 mask, where a single pixel, blurred by a Gaussian kernel (s.d. 3 pixels) was labeled as the target location.\", How was this mask obtained? How do the models perform without the use of this auxiliary loss? Why was this loss only used in CLEVR and CUB-200?\n\nMinor:\n- Add references to first column in Table 2.\n\nTo conclude, the proposed architecture is novel, the paper is clear, but the experimental work leaves some questions unanswered.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is very interesting and novel, and all reviewers are of the same opinion. 
The main concern, however, is on the experimental section that is limited to image classification benchmarks and that some critical comparisons are missing (e.g. clarify factors that play key role in improvement, more computation and therefore more free parameters, how about non discriminative tasks, etc). 
The heterogeneity question is in my opinion only partially answered by the authors but I also feel proper handling of this matter would require a proper multi-task setup and different target for the work.
I also personally find applicability of the approach quite limited, I encourage the authors to further improve their work as I feel that with a proper revision would make a nice contribution for the community.
Decision: Reject