Task-Based Top-Down Modulation Network for Multi-Task-Learning Applications
Authors: Hila Levi
Shimon Ullman
Venue:  (2020)
Keywords: deep learning
multi-task learning

Abstract: A general problem that received considerable recent attention is how to perform multiple tasks in the same network, maximizing both efficiency and prediction accuracy. A popular approach consists of a multi-branch architecture on top of a
shared backbone, jointly trained on a weighted sum of losses. However, in many cases, the shared representation results in non-optimal performance, mainly due to an interference between conflicting gradients of uncorrelated tasks. Recent approaches address this problem by a channel-wise modulation of the feature-maps along the shared backbone, with task specific vectors, manually or dynamically tuned. Taking this approach a step further, we propose a novel architecture which
modulate the recognition network channel-wise, as well as spatial-wise, with an efficient top-down image-dependent computation scheme. Our architecture uses no task-specific branches, nor task specific modules. Instead, it uses a top-down modulation network that is shared between all of the tasks. We show the effectiveness of our scheme by achieving on par or better results than alternative approaches on both correlated and uncorrelated sets of tasks. We also demonstrate our advantages in terms of model size, the addition of novel tasks and interpretability. 
Code will be released.

Under review as a conference paper at ICLR 2020
TASK-BASED TOP-DowN MODULATION NETWORK FOR MULL--TASK--EARNING APPLICATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
A general problem that received considerable recent attention is how to perform multiple tasks in the same network, maximizing both efficiency and prediction accuracy. A popular approach consists of multi-branch architecture on top of shared backbone, jointly trained on a weighted sum of losses. However, in many cases, the shared representation results in non-optimal performance, mainly due to an interference between conflicting gradients of uncorrelated tasks. Recent ap- proaches address this problem by a channel-wise modulation of the feature-map along the shared backbone, with task specific vectors, manually or dynamically tuned. Taking this approach step further, we propose novel architecture which modulate the recogmition network channel-wise, as well as spatial-wise, with an efficient top-down image-dependent computation scheme. Our architecture uses no task-specific branches, nor task specific modules. Instead, it uses a top-down modulation network that is shared between all of the tasks. We show the ef- fectiveness of our scheme by achieving on par or better results than alternative approaches on both correlated and uncorrelated sets of tasks. We also demon- strate our advantages in terms of model size, the addition of novel tasks and inter- pretability.
Code will be released.
INTRODUCTION
The goal of multi-task learning is to improve the learning efficiency and increase the prediction accuracy of multiple tasks learned and performed together a shared network.
Over the years, several types of architectures have been proposed to combine multiple tasks training and evaluation. Most current schemes assume task-specific branches, on top of a shared backbone (Figure Tap and use weighted sum of tasks losses, fixed or dynamically tuned, to train them (Chen) jt al. 2017 Kendall et al., 2018 Sener & Koltun 2018). Having a shared representation is more efficient from the standpoint of memory and sample complexity and can also be beneficial in cases where the tasks are correlated to each other (Maninis et al., 2201)). However, in many other cases, also result in worse performance due to the limited capacity of the shared backbone and interference between conflicting gradients of uncorrelated tasks (Zhao et al., 2018). The performance of the multi-branch architecture is highly dependent on the relative losses weights and the task correlations, and cannot be easily determined without a "trial and error" phase search (Kendall al. 2018).
the shared representation
Another type of architecture (Maninis et al. 2019) that has been recently proposed uses task specific modules, integrated along a fordarorward backbone and producing task-specific vectors modu- late the feature-maps along it (Figure lb Here, both training and evaluation use single tasking paradigm: executing one task at a time, rather than getting all the task responses a single forward pass of the network. A possible disadvantage of using task-specific modules and of using fixed number of branches, is that it may become difficult to add additional tasks at later time during the system life-time Modulation-basee architectures have been also proposed by (Strezoski et al. (2019) and Zhao et al. (2018) (Figure [T].. However, all of these works modulate the recognition network channel- wise, using the same modulation vector for all the spatial dimension of the feature-maps. maps.
We propose a new type of architecture with no branching, which performs single task at a time but with no task-specific modules (Figure Td). The core component of approach is top-down (TD)

---

Under review as a conference paper at ICLR 2020
XX
X
XX
TASK
(a)
(b)
(c)
(d)
Figure 1: (a) Multi branched architecture, task specific branches top of shared backbone, induces capacity and destructive interference problems, force careful tuning. Recently proposed architectures: (b) using tasks specific modules and (c) using channel-wise modulation modules. (d) Our architecture: top-down image-aware full tensor modulation network with no task specific modules.
modulation network, which carries the task information in combination with the image information, obtained from first bottom-up (BU1) network, and modulates second bottom-up (BU2) network common for all the tasks. In our approach, the modulation is channel-wise as well as spatial-wis (a full tensor modulation), calculated sequentially along the TD stream. This allows us, for example, to modulate only specific spatial locations the image depending on the current task, and get interpretability properties by visualizing the activations the lowest feature-map of the TD stream. In contrast to previous works, our modulation mechanism is also "image-aware" in the sense that information from the image, extracted by the BU1 stream, is accumulated by the TD stream, and affects the modulation process.
The main differences between approach and previous approaches the following: First, as mentioned, our approach does not use multiple branches or task-specific modules. We can scale the number of tasks with no additional layers. Second, our modulation scheme includes spatial component, which allows attention to specific locations in the image, as illustrated in figure 2a for the Multi-MNIST tasks (Sabour al. 2017). Third, the modulation in scheme is also image dependent and can modulate regions of the image based on their content rather than location (relevant examples are demonstrated figures 26 and 2c].
We empirically evaluated the proposed approach on three different datasets. First, we demonstrated on par accuracies with the single task baseline on an uncorrelated set of tasks with MultiMNIST while using less parameters. Second, we examined the case of correlated tasks and outperformed all baselines on the CLEVR (Johnson al. 2017) dataset. Third, we scaled the number of tasks and demonstrated our inherent attention mechanism on the CUB200 (Welinder et al. (0100) dataset. The choice of datasets includes cases where the tasks are uncorrelated (Multi- (NNUt) and cases where the tasks are relatively correlated (CLEVR and CUB200). The results demonstrate that our proposed scheme can successfully handle both cases and shows distinct advantages over the channel-wise modulation approach.
RELATED WORK
Our work draw ideas from the following research lines:
Multiple Task Learning (MTL) Multi task learning has been used machine learning well be- fore the revival of deep networks (Caruana 1997). The success of deep neural networks in single task performance (e.g. in classification, detection and segmentation) has renewed the interests of the computer vision community in the field (Kokkinos 2017 He et al. 2017 Redmon & Farhadi 2017]. Although our primary application area is computer vision. multi task learning has also many application in other fields like natural language processing (Hashimoto et al., 2016; Collobert & Weston 2008) and even across modalities (Bilen & Vedaldi (016)). We further refer the interested reader review that summarizes recent work in the field (Ruder 2017)
2

---

Under review as a conference paper at ICLR 2020
(a) M-MNIST examples
(b) CLEVR example
(c) CUB200 example
Figure 2: Images examples with their corresponding tasks. Our architecture benefits from its built- in image-aware, task dependent, localized attention mechanism. (a) M-MNIST examples, the tasks are to recognize the digits by their location. (b) CLEVR example. an example task is to determine whether there sphere to the right of cylinder. (c) CUB200 example, an example task to classify the bird's neck color.
Over the years, several types of architectures have been proposed in computer vision to combine the training and evaluation of multiple tasks. First works used several duplications (as many the tasks) of the base network, with connections between them to pass useful information between the tasks (Misra et al. 2016, Rusu al. 2016). These works do not share computations and cannot scale with the tasks. More recent architectures, which are in common practice these days, assume task-specific branches on top of a shared backbone, and use a weighted sum of losses to train them. The joint learning of several tasks has proven to be beneficial in several cases (He et al.) 2017) but can also decrease the results of some of the tasks due to a limited network capacity, uncorrelated gradients from the different tasks (sometimes called destructive inference) and different learning rates (Kirillov et al. 2019). A naive implementation of multi-task learning requires careful calibration of relative losses of the different tasks. To address these problem several methods have been proposed: "Grad norm" " (Chen et al. 2017) dynamically tunes gradients magnitudes over time, to obtain similar learning rates of the different tasks. [Kendall al. (2018) uses joint likelihood formulation to derive task weights based on the intrinsic uncertainty in each task. [Sener & Koltun (2018) applies adaptive weighting of the different tasks, force pareto optimal solution the multi task problem.
Along an orthogonal line of research, other works suggested to add task-specific modules to be activated or deactivated during training and evaluation, depending on the task at hand. Liu et al. (20196) suggests task specific attention networks parallel a shared recognition network. Mani- nis et al. (2019) suggests adding several types of low-weight task-specift modules (e.g. residual convolutional layers, squeeze and excitation (SE) blocks and batch normalization layers) along the recognition network. Note that the SE block essentially creates modulation vector, to be channel- wise multiplied with a feature-map. Modulation vectors have been further used in [Strezoski et al. (0199) for recognition application, in Cheung et al. (2019) for continual learning applications and in Zhao et al. (2018) for a retrieval application and proved to decrease the destructive interference between tasks and the effect of the catastrophic forgetting phenomena.
Our design, in contrast, does not use network is fully-shared between the different tasks. Compared to Zhao et al. (2018), modulate the feature-mapp in the recognition network channel-wise as well as spatial-wisee depending on both the task and the specific image at hand.
architecture, nor task- modules. Our
Top-Down Modulation Networks Neuroscience research provides evidence for a top-down con-
text, feedback and lateral processing the primate visual pathway (Gazzaley & Nobre 2012 Gilbert & Sigman| 2007 Lamme et al.) 1998; Hopfinger et al.) 2000; Piech et al. [013)] Zanto et al. 2010 where top-down signals modulate the neural activity of neurons lower-order sensory or motor areas based on the current goals. This may involve enhancement of task-relevant representations or suppression for fask-irrelevant representations. This mechanism underlies humans ability to focus
attention task-relevant stimuli and ignore irrelevant distractions (Hopfinger al. 2000 Piëch et al. 2013 Zanto et al., 2010).
In this work, consistent with this general scheme, suggest model that uses top-down modulation in the scope of multi-task learning. Top down modulation networks with feedback, implemented as conv-nets, have been suggested by the computer vision community for some high level tasks (e.g.
3

---

Under review as a conference paper at ICLR 2020
BU1
Z
Z'
H
Z
Corv layer
(M(b)
Task
M
HQ
x
Y
(a)
(b)
(c)
BU2 stream: The recognition network
H X
Task Prediction
W
(d)
Figure 3: Several types modulation modules, the trainable parameters are illustrated in yellow. (d) For simplicity show only one modulation stage, where X is the input tensor to be modulated and Y is the output tensor. (a) task-dependent vector-modulation architecture (Zhao et al. (018)), the modulation vectors switched by the task and explicitly being optimized. (b) Hypothetical exten- sion of (a) to spatial-wise modulation tensors, cannot be done in practice due to the huge number of parameters optimize. (c) Our approach: We optimize the parameters in the convolutional layers along the top-down network and use the created featuremaps as the modulation tensors.
re-classification (Cao et al. 2015), keypoints detection (Carreira et al. 2016 Newell et al. 2016), crowd counting (Sam & Babu] (018), curriculum learning (Zamir et al. 2017) etc.) and here apply them to multi-task learning applications.
APPROACH
We will first describe the task-dependent vector-modulation mechanism, proposed Zhao al. 2018), illustrated in figure 3al and then describe our architecture (figure 3c in detail.
3.1
RECTOR-MODULLAN MODULE
A moduuatoool (a channel-wise modulation) of given tensor X by vector z is defined as product the elements of the vector z and the corresponding channels the tensor X. Each element in the output tensor Y is calculated by:
Y(y, x, ch) X(y, ch) z(ch)
where X is the tensor to be modulated and Y is the modulated tensor, both the form (H x W x C) where H, W, are the spatial dimensions of the tensors and C their channel dimension. The vector RC has dimension equal to the number of channels of X, Y. Ir, y. ch are the column, row, and channel number and indicate a specific element a tensor. In training the network, the elements of are considered as parameters and are directly optimized (C additional parameters).
the scope of Multi Task Learning, where several tasks co-exist, network switches on the fly K illustration. The network performs one task JK= where K is the number of tasks, figure 3a for time, and the modulated tensor Y depends the selected task. This vector vector-oddlltton module has been used in Zhao et al. (2018) separately for every stage in the recognition network (with additional CK parameters in every stage).
between several modulation vectors {2²)

---

Under review as a conference paper at ICLR 2020
Two limitations of this module that it ignores the spatial dimensions of the image, and the lack of information from the image itself. The possible use of the same strategy to explicitly optimize
K
spatial-aware modulation tensors {z 2, ch) k=1
(figure 3b) was discussed in Zhao et al. (2018)
but was deemed infeasible due to the large amount of added parameters (HWCK additional param- eters in every stage). Our method addresses both of these issues in an efficient manner and demon- strates better accuracy, showing that spatial-wise modulation and the use of image information are beneficial to many kinds of tasks.
3.2
TENSOR MONDULAITNNA MODULE
tensor modulation is defined by:
Yy,,,, X(y, ch) Z(y, 2h,
(2)
Where Z E Hx is a modulation tensor.
To avoid the infeasible computation of directly optimizing Z, propose the use of created fea- turemaps as the modulation tensors. Practically, we use a dedicated top-down (TD) convolutional stream, shared between the tasks, create the modulation featuremaps, and optimize the weights of the convolutional layers instead of directly optimizing the modulation tensors (figure 3c). The depends
number of added parameters in this can be approximately estimated by 3 x 3 x C² parameters for each convolutional layer (several convolutional layers may be used in one stage). Avoiding the dependency of the number of added parameters on H, W and K allows us apply the proposed architecture to large images and to scale the number of tasks, as illustrated our experiments.
the precise architecture of the TD stream but
A gated modulation module with residual connection We further define gated modulation module with a residual connection as:
Y(y, I. ch) X(y. x, ch) X(y, ch) o (Z, X(y, 2, ch) @ Z(y, x. ch)
(3)
where the modulation tensor Z is gated with sigmoid tanh function before the multiplication
and then added to the input tensor through a residual connection. The residual gated modulation with Z is equivalent to the modulation with Z = (Z)) Motivated by our ablation studies 4.3.2
unless stated otherwise, we use the gated modulation as defined in Eq. B in all of our experiments. For simplicity we denote this operation by the symbol @.
3.3 IMAGE-AWARD TASK-DEPENDENT ToP-DowN MODULATION NETWORK
An illustration of our network design is shown in figure ld In our design, bottom-up (BU2) recognition network is modulated by a top-down (TD) modulation stream. The inputs to the TD stream are the current task k, and the feature- maps along the first bottom-up stream (BU1, where BUI and BU2 share the same weights), added to the TD stream via lateral connections. The outputs of the TD stream its feature-maps, which sequentially modulate the tensors along the recognition network (BU2). Figure 3c illustrate our architecture for one modulation step.
Auxiliary losses Our architecture can be naturally decomposed into three sub-networks (BU1, TD, BU2), allowing the structural advantage of adding auxiliary losses at the end of the BU1 or TD streams. This possibility is application-dependenn. In the scope of multi-task learning, the TD auxiliary loss might be beneficial because it allows the use of spatial information in a task- dependent manner. This issue is further discussed in section 443.4] where we demonstrate the use of localization loss in the last TD featuremap. Applying the localization loss in train time allows us to obtain an attention map in inference time, which illustrates the relative weights assigned by the network to different locations in the image.
5

---

Under review as a conference paper at ICLR 2020
mat.
CaW
giyy
ZASA JE
303
KAPM 0.453 2,03
AEEOM K,007
(a)
(b)
(c)
(d)
Figure 4: (a) Multi-MNIST accuracy scatter plot, top-right is better. We show higher accuracies
than the single-task baseline while using much less parameters (b) Multi-MNIST: Typical training curves, we show better accuracy than the single-task baseline along the training (c), (d) A close look into the network decision making process; (b) good example: the crown area is well localized and the prediction follows the ground truth, (c) error example: the breast isn't 't localized well enough.
DATABASES AND EXPERIMENTS
4.1 DATABASES
We validate our approch on three different datasets:
MultiMNIST MultiMNIST (Sabour al.) 2017) is a multi-task learning version of the MNIST
dataset in which multiple MNIST images are placed the same image. We use 2, 3 and 4 classes experiments built as suggested by Sener Koltun (0118). Several examples are demonstrated in Figure 2a In the 2-classes experiment the tasks are: classifying the digit the top-left (task-LU) and classifying the digit on the bottom-right (task-RL). RL). We corespondently add (task-LL) and (task- RU) for classifying the digits on the bottom-left and top-right the 3 and 4-classes experiments. The digits are independently chosen and the tasks are considered to be uncorrelated. We use 60K examples and directly apply LeNet (LeCun al. 1998) the underlying backbone in experi- ments.
CLEVR CLEVR synthetic dataset, consists of 70K training images and 15K validation im- ages, mainly used as a diagnostic dataset for VQA. The dataset includes images of 3D primitives, with multiple attributes (shape, size, color and material) and set of corresponding (question- answer) tuples. We followed the work Liu et al. (2019a), which suggested to use CLEVR not as a VQA dataset, but rather as a referring expression dataset, and further adapt it to multi-task learn- ing methodology. The tasks in our setup consist of questions ("Are there exactly two cylinders in the image?", "Is there a cube right to a sphere?", "Is there a red sphere?" and "Is the lefumost sphere in the image large?"), arbitrarily chosen, with various compositionary properties.
CUB200 is a fine grained recognition dataset that provides 11,788 bird images (equally divided for training and testing) over 200 bird species with 312 binary attribute annotations, most of them referring to the colors of specific birds' parts. In contrast to other work (Strezoski et al.) 2019) that used all of the 312 attributes as yes/no question, we regoniani the attributes as multi-task problem of 12 tasks (for 12 annotated bird's parts) each with 16 classes (the annotated colors + an unknown class) and train using a multi-class cross-entropy loss. To demonstrate our interpretability capability, we further used the parts' location, annotated by single point to each seen part, as an auxiliary target at the end the TD stream.
4.2 EXPERIMENTS
architecture We use LeNet, VGG-11 and resnet-18 as our backbone BU architectures for the Multi-MNIST. CLEVR and CUB-200 experiments correspondingly. Each of the backbones has been divided to two parts; a first part that consists mainly of the convolutional layers the backbone and second part with the fully connected layers (the classifier).

---

Under review as a conference paper at ICLR 2020
Performance (mean + std of 5 repetitions) on Multi-MNIST, uncorrelated tasks, higher is better. Our architecture achieves consistently better accuracies than the single task baseline while using much less parameters (the third column shows the ratio between the number of parameters and standard LeNet architecture). Scaling the number of tasks in our architecture costs no additional hardware.
Table 1:
ALG
#P
LU accuracy
RL accuracy
LL accuracy
RU accuracy
tasks
Single task
x2
96.99
40.07
95.93
Uniform scaling
x1.12
95.86
+0.08
94.75
#0.16
MOO
x1.12
96.25
#0.34
95.38 #0.16
ch-mod
x1.002
96.53
10.04
95,21 20009
Ours
x1.29
97.16
1.005
96.19
20.14
tasks
Single task
x3
90.84
+.003
90.64
0..03
83.22
4:0.06
Uniform scaling
x1.25
87.92
87.55
78.36
4:0.7
MOO
x1.25
87.38
40.31
86.82
#0.38
79.42
+0.44
ch-mod
x1.005
88.42 10.53
88.70
20 27
80.27
10.42
Ours
x1.31
90.83
10.18
90.67
20.24
83.45
10.38
tasks
Single task
x4
95.73 1:.14
92.97
20.06
93.1
10.21
94.81 10.11
Uniform scaling
x1.37
92.86
+0.30
89,30
=0.57
88.88
10.69
91.80 10.32
MOO
x1.37
92.96
10.10
90.04
#0.46
89.87
20.26
92.08
+0.27
ch-mod
x1.007
93.29
10.17
90.05 =0.38
90.09 10.18
92.11
Ours
x1.32
95.76
#0.07
93.89 =0
93.81 #0.27
95.11
20.10
In our architecture, both BU streams consist of the first part of the backbone and share their weights. The TD stream, unless specified otherwise, is a replica of the BU stream, in terms of layers structure and number of channels, combined with bilinear upsampling layers. The classifier is only attached to the BU2 stream. Information is passed between the BU1, TD and BU2 streams using lateral connections implemented
Ixl convolutions. A task embedding layer (a fully connected layer) added on the top of the TD stream. See an illustration of the full scheme in figure [d and detailed architecture description of the MulTI-MNIST experiments the supplementary materials.
baselines We compare our method both to "single task" approach, where each task is indepen- dently solved and to a "uniform scaling" approach, where uniformly weighted sum of the indi- vidual losses is being minimized. We have also compared our architecture to "ch-mod", a channel- wised vector modulation architecture (Zhao et al. 2018) and to MOO (multi objective optimization approach) where the weights of loss items are dynamically tuned as suggested by /Sener & Koltun (018).
4.3 RESULTS
4.3. MULTIMNIST
We use the Multi-MNIST dataset to demonstrate our performance in case of uncorrelated tasks for 2, 3, and 4 tasks recognition problems with additional hardware. All models trained using standard LeNet architecture. We used batch size of 512 images trained on 1 GPU with learning rate of le using the Adam optimizer. Training curves are presented in figure [].
Figure 4b visualize the performance profile the 2-classes experiment as seatter plot of accuracies on task-LU and task-RL for the single task approach (vertical and horizontal lines correspondingly) and the multi-branched approach for several manually tuned loss weights (the blue dots). The seatter plot demonstrate capacity problem, where better accuracies (above certain limit) in one task cannot be achieved without being reflected as lower accuracies on another task. Our results are marked red star, showing better accuracies than the single-task case with much less parameters.

---

Under review as a conference paper at ICLR 2020
Table 2: Ablations on Multi-MNIST.
(a) spatial-win wise image-awar modulation
ALG
image
spatial
LU accuracy
RL accuracy
LL accuracy
RU accuracy
mean
ch-mod
93.29 10.17
90.05
10.38
90.09 10.18
92.11 10.27
V
93.52 10.18
90.62
#0.21
90.64 10.18
92.39
10.16
+0.41
X
95.45 #0.11
93.29
20.17
93.19
20.24
94.60
10.15
+2.75
Ours
95.76 #0.07
93.89 20.28
93.81 40.27
95.11 #0.10
+3.26
(b) number of channels
(c) connectivity type
(d) auxiliary losses
#ch
# P
LU ac
RL ac
td,bu2
LU ac
RL ac
LU ac
RL ac
dup
x1.29
97.09
96.11
+.
96.55
95.60
bu2
97.09
96.11
10
x1.25
97.20
96.18
97.06 0.10
96.15 0.06
bu2+bul
96.75
95.71
6
x1.10
96.88
96.05
97.16 0.05
96.19 0.14
bu2+td
96,82
95.56
2
x1.02
96.59
95.55
96.43
95.39
bu2+td+bul
96.84
95.53
x1.01
96.30
95.59
8,
96.63
96.07
Table summarizes our results on the Multi-MNIST experiment while sequentially enlarging the number of tasks. We show mean-sst based on 5 experiments for each row. Our method achieves better results than the single-task baseline while using much less parameters (the third column shows the number parameters as a multiplier of the number of parameter a standard Lenet architec- ture). Other approaches, including the channel-wise modulation approach, achieve lower accuracy rates. Scaling the number tasks keeps the accuracy gap almost without additional parameters.
4.3.2 ABLATIONS ON MULTI-MNIST
We further conducted ablation studies on Multi-MNIST, examine several aspects our proposed architecture. Table 2 shows the ablation results, analyzed as follows:
Using spatial-wise and image-aware modulation modules. Our experiments show that extend- ing the existing channel-wise modulation architecture to an image-aware spatial-wise modulation architecture improves the results. Table 2a quantify the improvement the results compared to the channel-wise modulation baseline (Zhao et al. (2018), first row in the table). We show mean + std based on repetitions of the full training pipeline for each row. Using channel-wise image-awan aware modulation architecture by sequentially integrating information from the featuremaps in BU1 (sec- ond row) improves the accuracies by ~ 0.4%. Using a spatial-wist modulation without using the in- formation from BUT stream (third row) improve the accuracies by 2.7%. Our approach, that both image-aware and apatial-wise modulation, improves the accuracies by solid gap of ~ 3.3%.
Number of channels in the TD stream. Table compares the results accuracies of our proposed architecture (first line, where the TD stream is a replica of the BU stream which has 1, 10 and 20 channels in its feature-maps) with cheaper architectures which use reduced number of channels along the layers in the TD stream. Our experiments show trend line (the accuracies decrease when the number of channels in the TD stream decreases) and that optimizing the number of channels along the TD stream in terms of efficiency-accueeee accuracies tradeoff can be done (demonstrated by the second row in the table where higher accuracy achieved while using less parameters).
Connectivity type. Our architecture uses two sets of lateral connections; the first set passes in- formation from the BUI stream to the TD stream, and the second passes information from the TD stream to the BU2 stream. Table(2ccomppres the results accuracy of our proposed architecture when using different connectivity types to the TD stream (first column) and to the BU2 stream (second column). Here + is addition connectivity, modulation with residual connection as described in Equation 3 The table shows higher accuracy when using addition connectivity along the TD stream and small preference for the gated mod-
is multiplication connectivity and & is gated
8

---

Under review as a conference paper at ICLR 2020
Table 3: Performance on CLEVR, higher is better. Our approach yields better accuracies also on correlated set of tasks with no additional hardware tasks are added. Better accuracies are demon- strated both compared the single task and uniform scaling approaches while using less parameters.
ALG
#P
quel accuracy
que2 accuracy
que3 accuracy
que4 accuracy
2 tasks
Single task
97.81
97.95
Uniform scaling
x1.5
98.09
98.19
ch-mod
x1.001
97.91
97.45
Ours
x1.56
98.17
98.19
3 tasks
Single task Uniform scaling
x3
96.92
97.81
99.93
x2
97.67
97.73
99.89
ch-mod
x1.001
97.01
97.55
99.90
Ours
x1.56
98.25
97.92
99.93
4 tasks
Single task
96.73
97.93
99.94
Uniform scaling
x2.5
97.47
97.93
99.92
98.
ch-mod
x1.001
97.67
97.77
99.91
98.62
Ours
x1.56
98.43
98.16
99.93
98.66
ulation connectivity over the multiplication connectivity along the BU2 stream. better compare between the two connectivity types we carried 5 experiments and report mean + std. We used the gated modulation connectivity type in our experiments due to its slightly higher results.
Auxiliary losses. Our architecture, although usually uses only one classification loss at the end of the BU2 stream, can be easily adapted to integrate two auxiliary losses, one the end of the BUI stream (same classification loss) and the other the image plane at the end of the TD stream (segmentation loss). Table 2d shows no additional improvement when using these auxiliary losses on the Multi-MNIST experiment. Note that TD auxiliary segmentation loss (here, a binary cross entropy loss between the predicted digit and a zero-one map of the target digit) can also be used to add interpretability to scheme. Examples are shown in the CUB200 experiment, section 4.3.4
4.3.3 CLEVR
We used the CLEVR dataset to show our performance in case of correlated tasks (the questions on CLEVR correlated) and demonstrate ability enlarge the number of tasks with extra hardware while keeping the targets accuracies.
Our results are summarized in Table 3 We trained all models using VGG-11 architecture but decreased the number of channels in the output of the last convolutional layer from 512 to 128 to allow training with larger batch size. A detailed analysis of the number of parameters can found in the supplementary materials. We used a batch size of 128 images trained on GPUs with learning rate 1e using the Adam optimizer.
Table 3 shows that our results are better than both single task and uniform scaling approach while using much less parameters (the third columns shows number of parameters of each architecture as multiplier of the number of parameters in single task VGG-11 backbone). Here, the channel- wise modulation approach uses the smallest number of parameters but also gets the worst results. The table also shows that scaling the number of tasks (with no additional hardware) is not only feasible but also may improve the results of each task separately. We further note that we used TD layers that are replica of the VGG-11 BU layers. Further reducing the number of parameters by decreasing the channel dimensions in the TD stream can be easily done but is not our main scope in this work.

---

Under review as a conference paper at ICLR 2020
Table 4: Performance on CUB200, higher is better. Our architecture is scalable with the number of tasks and outperforms other methods. All models trained for 200 epochs with Ir le-4 using resnet- 18 backbone.
wing
uppertail
threat
msgpe
leg
eye
breast
forchrad
bally
CRowT
mncam
Single
77.91
15.09
13,46
70.99
62.82
89.66
75.54
15.30
71.07
77.63
21.82
70.26
74.30
Uniform scaling
77.55
74.58
75.25
77.46
MDO
K2:78
82.17
77.37
75.66
64.84
91.39
81.15
79.63
75.03
81.57
75.44
73.80
78. 78.40
cb mod
$2.07
91.72
80.033
77.20
68.61
91.30
81.29
$1.22
76.91
82.29
77.87
76.23
79.72
cb-mod
78.84
76.79
75.18
72.09
58.75
90.35
75.99
77.17
71.30
76.74
72.54
67.07
74.61
$4.90
31.77
81.01
67.60
91.08
83.64
79.55
79.96
4.3.4 CUB 200
We used the CUB-200 dataset to further demonstrate our performance on correlated tasks in real- world images, scaling the number of tasks and using another type of backbone architecture (a Resnet backbone). In contrast to previous experiments, we did not aim at reducing the number of parameters (since we are using a Resnet backbone); rather we demonstrate better performance, and our built-in capability to visualize the attention maps at the end of the TD stream.
We trained all models using Resnet-18 architecture. We used batch size of 128 images trained on 2 GPUs with learning rate of 1e using the Adam optimizer for 200 epochs. While training our architecture we add an auxiliary loss at the end of the TD stream. The target in this case a one-hot 224x224 mask, where only single pixel is labeled a standard deviation of 3 pixels. Training one task at a time, we minimize the cross-ent cross-entropy loss over the 224x224 image at the end of the TD softmax output (which encourages small detected area) for each visible ground-trut truth annotated task/part. For a fair comparison, we also compared our results the channel-wised modulation architecture trained with the same localization auxiliary loss (on the coarse map at the end of the BU2 stream, fifth line in the table).
foreground, blurred by Gaussian kernel with
Figures 4c and 4d demonstrate the attention maps produced by our architecture in inference time. Figure 4c is an example where the predicted mask is well localized on the crown of the bird (the task) and the color is correctly predicted. Figure 4d demonstrate an error case where the breast of the bird is not well localized by the mask and as a consequence the color is wrongly predicted. More examples of interest are shown in the supplementary materials.
Our quantitative results are summarized in Table The results show better accuracy of our scheme compared to all baselines. We specifically show better accuracy compared to the channel-wist mod- ulation scheme, indicating the preference of our image-dependent spatial-wise modulation process on the CUB200 database.
SUMMARY
We proposed a novel architecture for multi-task learning using a top-down modulation network. Compared with current approaches, our scheme does not use task-dependent branches task- dependent modules, and the modulation process is executed spatial-wise as well as channel-wise, guided by the task and by the information from the image itself. We tested our network on three different datasets, achieving on par or better accuracies on both correlated and uncorrelated sets of tasks. We have also demonstrated inherent advantages of our scheme: adding tasks with no extra hardware that result in a decrease in the total number of parameters while scaling the number of tasks, and allowing interpretability by pointing to relevant image locations.
More generally, multiple-task learning algorithms are likely to become increasingly relevant, since general vision systems need to deal with a broad range tasks, and executing them efficiently in single network is still open problem. In future work we plan to adapt our described architecture to wider range of applications (e.g. segmentation, images generation) and examine possible com- binations of approaches such combining branchl-branching strategy with TD approach. We also plan to study additional aspects multi-task learning such as scaling the number of tasks and tackling the catastrophic forgetting problem.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Hakan Bilen and Andrea Vedaldi. Integrated pereeption with recurrent multi-task neural networks. In Advances neural information processing systems, pp. 235-243, 2016.
Chunshui Cao, Xianming Liu, Yi Yang, Yinan Yu, Jiang Wang, Zilei Wang, Yongzhen Huang, Liang Wang. Chang Huang, Wei Xu, et al. Look and think twice: Capturing top-down visual attention with feedback convolutional neural networks. In Proceedings of the IEEE International Conference Computer Vision, pp. 2956-2964, 2015.
Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, and Jitendra Malik. Human pose estimation with iterative error feedback. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4733-4742, 2016.
Rich Caruana. Multitask learning. Machine learning, 28(1))41-75, 1997.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks. arXiv preprint 1rXiv:1711.0220 2017.
Brian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Superposition of many models into one. arXiv preprint wrXiv:/902.055222 2019.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference Machine learning, pp. 160-167. ACM, 2008.
Adam Gazzaley and Anna C Nobre. Top-down modulation: bridging selective attention and working memory. Trends in cognitive sciences, 16(2):129-133) 2012.
Charles Gilbert and Mariano Sigman. Brain states: top-down influences in sensory processing. Neuron, 54(5):677-666, 2007.
Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. A joint many-task model: Growing neural network for multiple nlp tasks. arXiv preprint arXiv:1611.01588 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Joseph B Hopfinger, Michael H Buonocore, and George R Mangun. The neural mechanisms of top-down attentional control. Nature neuroscience, 3(3):284, 2000.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2901-2910, 2017.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7482-7491, 2018.
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmen- tation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 9404-9413, 2019.
lasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid- and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pp. 6129-6138, 2017.
Victor AF Lamme, Hans Super, and Henk Spekreijse. Feedforward, horizontal, and feedback pro- cessing in the visual cortex. Current opinion in neurobiology, 8(4):529-535, 1998.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (1(11)2728-2324 1998.
11

---

Under review as a conference paper at ICLR 2020
Runtao Liu, Chenxi Liu, Yutong Bai, and Alan L Yuille. Clevr-ref+: Diagnosing visual reasoning with referring expressions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4185-4194, 2019a.
Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1871- 1880, 2019b.
Kevis-Kokitsi Maninis, Ilia Radosavovic, and lasonas Kokkinos. Attentive single-tasking of multi- ple tasks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1851-1860, 2019.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994 4003, 2016.
Alejandro Newell, Kaiyu Yang. and Jia Deng. Stacked hourglass networks for human pose estima- tion. In European conference computer vision, pp. 483-499, Springer, 2016.
Valentin Piech, Wu Li, George N Reeke, and Charles D Gilbert. Network model of top-down influences on local gain and contextual interactions visual cortex. Proceedings of the National Academy of Sciences, 110(43):E4108 1111---1111--
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7263-7271, 2017.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint 17Xiv:1706.05098, 2017.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1606v 2016.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in neural information processing systems, pp. 3856-3866, 2017.
Deepak Babu Sam and R Venkatesh Babu. Top-down feedback for crowd counting convolutional neural network. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objectivve optimization. In Advances in Neural Information Processing Systems, pp. 527-538, 2018.
Gjorgji Strezoski, Nanne van Noord, and Marcel Worring. Many task learning with task routing. arXiv preprint arXiv: 1219903..22/1. 2019.
Welinder, Branson, Mita, C. Wah, Schroff, Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, Californa Institute of Technology, 2010.
Amir R Zamir, Te-Lin Wu, Lin Sun, William B Shen, Bertram E Shi, Jitendra Malik, and Silvio Savarese. Feedback networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1308-1317, 2017.
Theodore P Zanto, Michael T Rubens, Jacob Bollinger, and Adam Gazzaley. Top-down modulation of visual feature processing: the role the inferior frontal junction. Neuroimage, 53(2):736-745, 2010.
Xiangyun Zhao, Haoxiang Li, Xiaohui Shen, Xiaodan Liang, and Ying Wu. A modulation module for multi-task learning with applications in image retrieval. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 401-416, 2018.
12

---

Under review as a conference paper at ICLR 2020
Table 5: Number of parameters in the M-MNIST and CLEVR architectures
(a) M-MNIST experiment
(b) CLEVR experiment
Modale Anchitecture
jarams
Architecture
# paramn
Recogattion backhone
21,250
Recognition backbone
7,448,256
Each branch
3,000
7,473,152
TD with laterals
8,306,688
taek embedding
320
task enabediing
1568
anchitecture
Single archtilecture
14,921,408
Mulli-branched architecture
21, 250
3000
Malti branched architecture
14. 921, 408 + 7, 478. 152 K
TD modulationn anchitectire
21, 250 + 9651 320
modulation architecture
14. 921, 408 306, 688 1566.
A IMPLEMENTATION DETAILS
For the MultiMNIST experiments, we use an architecture based on LeNet (LeCun et al., 1998), )
We followed Sener& & Koltun (2018) and use 5x5 convolutional layers and one fully-connected layer as the shared backbone and two other fully-connected layers as task specific branches for the multi-branched architecture (See figure Sa for details).
Our architecture is illustrated in figure 5b We use the shared backbone and a single branch as the recognition network (BU2). On the TD stream we use an embedding layer followed by two 5x5 convolutional layers. BUT share the same weights with BU2. the three subnetworks are combined together using lateral connections, implemented as convolutions. The networks for the CLEVR and CUB200 experiments where similarly implemented using VGG-11 and ResNet-18 18 backbones correspondently. The exact number of parameters used by the Multi-MNIST and by the CLEVR architectures summarized in table 5
B ADDITIONAL QUALITATIVE EXAMPLES ON CUB200
To demonstrate interpretability capabilities trained proposed network with auxiliary localization cross entropy loss on the last layer of the TD stream (details in section 4.3.4). Here we present several more examples of interest we did not include in the main text.
FC 50-350
digit
Image
Coew Sacs 1D
Cont tack, 20
RC 520
Come SX5, 20
FC 320 200-88
FC
(10 classes)
Coew Se5, 1
Conv back 30
PC 2x40e
Task
50-550
digit classes]
Image
520-32
Fr $8.50
50
w10 classes)
target dight
(a)
(b)
experiment: (a) hulti-branched archi- tecture (b) Ours top-down modulation architecture. The number of parameters in each architecture summarized in table 5a
Figure 5: Architectures used in the two-classes
13

---

Under review as a conference paper at ICLR 2020
otey
buff
facoryal
2.304
sape
gtrees
gmay
belly
yallov
SEN
stoy yellow
SITy
Figure 6: More qualitative examples to demonstrate our ability to identify the relevant regions that affected the network prediction. In all of these images the target part (the task, shown in the upper part of each image), is precisely localized and the prediction (shown in the lower part of each image) follows the ground truth. Best viewed in color while zoomed in.
FANK:
sage
a1.ive
TANK forehead
white
33/dy
Figure 7: Error cases. Left images demonstrate good examples, counted as failure cases due to annotations errors. Our network successfully localize the asked part and correctly predict its color. Right images demonstrate bad localization examples. Ground truth classes were still predicted, with a very high score, maybe due to the correlated nature of the tasks. Best viewed in color while zoomed in.
14


Reviews:
[{"review_id": "rJe7ZT5l9S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the multi-task setting by using modulation connections between three network pipelines, i.e., one bottom-up network to contextualize the problem, one top-bottom network that is conditioned on the tasks, on a last bottom-up network that solves the task. \nThe key contribution of the paper is to introduce a feature-wise and spatial-wise tensor to modulate the different neural pipelines better. Finally, they assess the proposed method on three datasets: Multi-MNIST, a yes/no CLEVR, and CUB-200.\n\nThe abstract, introduction, and related works are pretty clear. Figure 1 is also a nice summary that puts the paper architecture in perspective with another approach, and it is a very insightful sketch. I appreciate the effort of the authors to release the code with several baselines. I also acknowledge the diversity of tasks that are studied.\n\nHowever, I have three concerns that I am willing with the authors.\n\nMy first concern deals with the method description, which I found a bit misleading. Thus, I not sure that I fully got all the subtleties of the proposed method. The mathematical notation is misleading: Are Y, and X function of (x,y,ch) or are tensors over x, y, z. Later in the text, W is defined over (ch, t), but it is also mentioned that t is an input. Thus, is W \\in R^(CxT) or W(t) \\in R^(C). Besides, the implicit tilling with * makes things even harder to follow. On a different side, what do you mean by training the convolution network instead of optimizing W. Is W fixed? Do you use simply the feature map after 1x1 conv as mentioned in 4.2? How do you embed the task t in general, how do you append it to the feature map of BU1.\nIn the current paper state, I would not be able to reproduce the experiments.\n \nThe second concern relies on the results. The gap between the methods is tiny, e.g. max 0.5 in 2-MNIST, and may fluctuate a lot from one experiment to another, e.g., it is weird that 3-MNIST is harder than 4-MNIST. Note that the same observations can be applied to the CLEVR. Therefore, it is hard to assess the method without the std over at least 5 seeds. The result only convinces me regarding 4-MNIST without such std.\n\nMy third concern is about CUB200 experiments. The authors used an auxiliary loss on top of TD to help to visualize the network decision. As such auxiliary losses provide additional information, I have the following question: did you add the same losses to other baselines? Did you use a stop-gradient before decoding the feature-maps? Otherwise, the comparison between methods may not be fair\n\n\nRemarks:\n - I am missing some results from external literature. For instance, even if I prefer your setting CUB300  over 312 questions, it would have be nice to add such experiments in the appendix. (or literature baselines on N-MNIST)\n - Please report the original MOO too results in addition to your experiments\n - Why ch-mod is missing in 4-CLEVR?\n - Can you describe how did you pick the CLEVR questions (before/after computing the results)? It would have been nice to have experiments that randomly pick K questions (and repeat the process N time + report std), or even dynamically condition on the question at hand.\n - it took me quite some time to understand the meaning of #P, please make it explicit from the beginning, or add in the caption!\n - Although releasing the code is good, I also encourage you to put a table in the appendix with the hyper-parameters. The paper should be as much self-content as possible. It is also hard to evaluate the quality of the training time during the review\n - typo: extra parenthesis in Eq 4\n\nIn conclusion, the authors give some good intuition about promising methods, but I had some difficulties in understanding all the details of their approaches. Besides, I am missing both std and external references to assess the quality of the methods. In this current state, I cannot recommend paper acceptance even if I acknowledge several qualities of the paper, but I am open to discussion.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BkgBBW0RcH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "> What is the specific question/problem tackled by the paper?\nThis paper tackles a restricted multi-task setting where the task is known. The main contribution is a new architecture for training a task conditional model. The new architecture is reminiscent of an encoder-decoder-classifier with ladder (latent) connections, the decoder is conditioned on task ID. The claim is this is a type of modulation, it is unclear. Results on three multi-task datasets show that the proposed method is slightly better than compared methods and single task learning. There is no theory or loss function to analyze.\n\n> Is the approach well motivated, including being well-placed in the literature?\nIn my opinion, this is lacking. The assumption that task ID is known is fairly severe. Unfortunately the prior works cited also have this restriction, whereas few papers under the topic of continual learning have removed this limitation. This assumption/drawback needs to be clearly mentioned in the paper and discussed if it is realistic? A related shortcoming is that the training data simultaneously comes from all the tasks, whereas prior work has looked at the more interesting setup where tasks arrive sequentially and incrementally. \n- Reference [1] seems relevant and should be cited as it shows context dependent gating of tasks / modulation as well. Other missing references e.g. learning without forgetting (LwF) [2] and [3]. \n- There is not a clear explanation to think that this is modulation since the result is only passed through a residual connection. More importantly there is no discussion on these important issues. I found the writing to be brief and sketched. \n- in the introduction, it would be good to define multi-task learning with the assumption made clear. It would be good to introduce what you mean by TD and BU clearly\n- Another drawback is assuming the tasks being encoded as integers, whereas there might be a continuous task space with interpolation, or hierarchical task structure.\n- \"However, all of these works modulate the recognition network\nchannel-wise, using the same modulation vector for all the spatial dimension of the feature-maps.\" - why is this not enough? A nontrivial explanation or discussion is needed. Simply extending to W(t, y, x, ch) would increase performance by a little.\n- how is the proposed model different from a conditional model like a task conditional classifier? Also in experiments.\n- How is the proposed model different from an encoder-decoder? The impact of \"modulation\" is not clear.\n- \"We can scale the number of tasks with no additional layers.\" - task conditional classifier can also scale in this way to the number of tasks. This claim is not valid.\n- Page 3: \"uncorrelated gradients from the different tasks\" - need not be uncorrelated, but still can be interfering\n- next about Kendall (2018) and Sener (2018) - need to compare and contrast to them.\n- Last para on page 3 seems not relevant.\n- Modulation equations: this seems specific to CNNs. How would you extend this technique to beyond CNNs to recurrent units or even simpler MLPs? Modulation as a technique has been successfully applied in these architectures as well.\n- \"added to the input tensor X through a residual connection\" - this is not clear at all. Are the residual connections not shown in Fig 1(d)? \n- \"it to be unfeasible due to their large dimensions\" - can you explain please? later you say \"To avoid the unfeasible computation burden of directly optimizing W\"\n- Fig 1d, would be good to mark the modulation arrows in a different color\n\n> Does the paper support the claims? This includes determining if results, whether theoretical or empirical, are correct and if they are scientifically rigorous.\nHaving said that, I like the experimental section even in the restricted setup. But a lot of details are missing. It is not surprising that there is slight increase in performance over channel modulation due to the increase expressivity. \n- table 1: why is there degradation in the LL task across all methods? the introduction of an additional task seems to bring the performance back up. It seems to be a weakness of your method. Please improve the discussion. I'm inclined to think that the tasks are not uncorrelated, as claimed by the authors. \n- table 1: how did you arrive at the number of parameters like 1.12x? Doesn't the separate BU and TD nets mean you have at least 2x parameters compared to single task? It seems the larger number of params in single task is mainly coming from the hidden layers?\n- table 1: it would be fair for the comparison methods to have equal number of parameters as the proposed method.\n- Missing experimental comparison to Kendall et. al. 2018\n- Missing details about reproduction of results from Sener (2018)\n- An important baseline would be to show image sensitive full tensor modulation without the new architecture. Similar to XdG.\n- Another baseline should be a task-conditional classifier that takes task as input along with the image. \n- ablation study: what are the auxiliary losses? I could not find any details.\n- The third experiment with CUB seems to use a different loss function that the other methods. This is somewhat hard to evaluate.\n- number of parameters are not reported for the CUB experiment\n- \"where only a single pixel is labeled as foreground, blurred by a Gaussian kernel\" needs more details about the smoothing\n- In the CUB experiment, due to lateral connections, the top-down result 224x224 image is not the only input to the BU2 classifier, the interpretability argument is weak.\n- Why did you choose these 4 questions from CLEVR? There are many interesting types of questions that can be handled.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BkgA0-5RqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new light-weight framework for multi-task learning. In this method, the combination of extracted image features and task are fed into a top-down network which is responsible for generating a task-specific weight matrix. The weights are next convolved with the input image as an input to the task-agnostic bottom-up network that generates the labels.\n\nThe idea of the paper interesting. The main shortcoming of the paper in my point of view is that all the numbers are reported as a single number, so they are prone to be changed by using different initial networks or optimizers. Here are some more comments:\n\n1) One limitation of the result section is that all the numbers are reported as static numbers. I am interested to see the training curves, either in using the wall-clock time or iteration in the x-axis and testing accuracy in y.\n\n2) Sections 3.1 and  3.2 as the main parts are not well-written. The shapes of the tensors are vague. What is the y,x in the parentheses? What does ch stand for? (defined?) I think that this part of the paper requires significant improvement.\n\n3) One valid question is how the proposed method is scalable. For example, can a model trained for 3 tasks used for 4 tasks? How hard is adding a new task? Also, worths comparing it with the learning from scratch. \n\n4) In Section 3, the discussion about the loss function is missing. I believe that the explanation of how to choose a loss function as well as auxiliary losses should be move there. Also, I didn't find the current explanation of BU1 and TD auxiliary loss for Multi-MNIST very clear.\n\n5) Why the results of your method is better than the single model? This behavior should be justified. My impression is that each task trained independently should outperform any multi-training method. Your results seem counter-intuitive in this respect.\n\n6) I am not able to make any strong conclusions from Section 4.3.2. It is really hard to tell which connection is better based on a single number. I would suggest providing confidence intervals for making such kind of arguments. For example, you may train from 10 different network initializations and use them to construct more reliable estimates. I also believe that more reliable estimations are required for Table 3. \n\n\nMinor:\n* In paragraph 2 of pages 2, you mention \"as illustrated in Figure 2a\". I do not see the attention to a part of the image. Am I missing something? A similar issue exists in the next sentence: I don't see any content-related modulization in Figures 2b and 2c. Please clarify.\n* use comma after equations if the equations are not ending the sentences. For example, add a comma after eq (1), (2) and (3). Also on page 4, \"Where $W$\" -> \"where $W$\".\n* Page 4, \"Our method address\" -> \"Our model addresses\"\n* Where the third column of Table 1 is defined? On page 8. Move it to earlier sections.\n* In Table 2b, you have used +x, but the notation for gated modulation is something else in the text.\n* Are LL and RU used in Table 1 defined in the text?\n* The bold numbers in Figure 2b seem wrong. If you are bolding the large accuracies, be consistent in all tables.\n* Font of table 4 can be larger", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is interested in multi-task learning. It introduces a new architecture which condition the model in a particular manner: images features and task ID features are fed to a top-down network which generates task-specific weights, which are then used in a bottom-up network to produce final labels. The paper is experimental, and the contribution rather incremental, considering existing work in the area. Experimental section is currently not convincing enough, given marginal improvements over existing approaches - multiple runs as well as confidence intervals would help in that respect.

Decision: Reject