Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: > What is the specific question/problem tackled by the paper?
This paper tackles a restricted multi-task setting where the task is known. The main contribution is a new architecture for training a task conditional model. The new architecture is reminiscent of an encoder-decoder-classifier with ladder (latent) connections, the decoder is conditioned on task ID. The claim is this is a type of modulation, it is unclear. Results on three multi-task datasets show that the proposed method is slightly better than compared methods and single task learning. There is no theory or loss function to analyze.

> Is the approach well motivated, including being well-placed in the literature?
In my opinion, this is lacking. The assumption that task ID is known is fairly severe. Unfortunately the prior works cited also have this restriction, whereas few papers under the topic of continual learning have removed this limitation. This assumption/drawback needs to be clearly mentioned in the paper and discussed if it is realistic? A related shortcoming is that the training data simultaneously comes from all the tasks, whereas prior work has looked at the more interesting setup where tasks arrive sequentially and incrementally. 
- Reference [1] seems relevant and should be cited as it shows context dependent gating of tasks / modulation as well. Other missing references e.g. learning without forgetting (LwF) [2] and [3]. 
- There is not a clear explanation to think that this is modulation since the result is only passed through a residual connection. More importantly there is no discussion on these important issues. I found the writing to be brief and sketched. 
- in the introduction, it would be good to define multi-task learning with the assumption made clear. It would be good to introduce what you mean by TD and BU clearly
- Another drawback is assuming the tasks being encoded as integers, whereas there might be a continuous task space with interpolation, or hierarchical task structure.
- "However, all of these works modulate the recognition network
channel-wise, using the same modulation vector for all the spatial dimension of the feature-maps." - why is this not enough? A nontrivial explanation or discussion is needed. Simply extending to W(t, y, x, ch) would increase performance by a little.
- how is the proposed model different from a conditional model like a task conditional classifier? Also in experiments.
- How is the proposed model different from an encoder-decoder? The impact of "modulation" is not clear.
- "We can scale the number of tasks with no additional layers." - task conditional classifier can also scale in this way to the number of tasks. This claim is not valid.
- Page 3: "uncorrelated gradients from the different tasks" - need not be uncorrelated, but still can be interfering
- next about Kendall (2018) and Sener (2018) - need to compare and contrast to them.
- Last para on page 3 seems not relevant.
- Modulation equations: this seems specific to CNNs. How would you extend this technique to beyond CNNs to recurrent units or even simpler MLPs? Modulation as a technique has been successfully applied in these architectures as well.
- "added to the input tensor X through a residual connection" - this is not clear at all. Are the residual connections not shown in Fig 1(d)? 
- "it to be unfeasible due to their large dimensions" - can you explain please? later you say "To avoid the unfeasible computation burden of directly optimizing W"
- Fig 1d, would be good to mark the modulation arrows in a different color

> Does the paper support the claims? This includes determining if results, whether theoretical or empirical, are correct and if they are scientifically rigorous.
Having said that, I like the experimental section even in the restricted setup. But a lot of details are missing. It is not surprising that there is slight increase in performance over channel modulation due to the increase expressivity. 
- table 1: why is there degradation in the LL task across all methods? the introduction of an additional task seems to bring the performance back up. It seems to be a weakness of your method. Please improve the discussion. I'm inclined to think that the tasks are not uncorrelated, as claimed by the authors. 
- table 1: how did you arrive at the number of parameters like 1.12x? Doesn't the separate BU and TD nets mean you have at least 2x parameters compared to single task? It seems the larger number of params in single task is mainly coming from the hidden layers?
- table 1: it would be fair for the comparison methods to have equal number of parameters as the proposed method.
- Missing experimental comparison to Kendall et. al. 2018
- Missing details about reproduction of results from Sener (2018)
- An important baseline would be to show image sensitive full tensor modulation without the new architecture. Similar to XdG.
- Another baseline should be a task-conditional classifier that takes task as input along with the image. 
- ablation study: what are the auxiliary losses? I could not find any details.
- The third experiment with CUB seems to use a different loss function that the other methods. This is somewhat hard to evaluate.
- number of parameters are not reported for the CUB experiment
- "where only a single pixel is labeled as foreground, blurred by a Gaussian kernel" needs more details about the smoothing
- In the CUB experiment, due to lateral connections, the top-down result 224x224 image is not the only input to the BU2 classifier, the interpretability argument is weak.
- Why did you choose these 4 questions from CLEVR? There are many interesting types of questions that can be handled.


review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper introduces a new light-weight framework for multi-task learning. In this method, the combination of extracted image features and task are fed into a top-down network which is responsible for generating a task-specific weight matrix. The weights are next convolved with the input image as an input to the task-agnostic bottom-up network that generates the labels.

The idea of the paper interesting. The main shortcoming of the paper in my point of view is that all the numbers are reported as a single number, so they are prone to be changed by using different initial networks or optimizers. Here are some more comments:

1) One limitation of the result section is that all the numbers are reported as static numbers. I am interested to see the training curves, either in using the wall-clock time or iteration in the x-axis and testing accuracy in y.

2) Sections 3.1 and  3.2 as the main parts are not well-written. The shapes of the tensors are vague. What is the y,x in the parentheses? What does ch stand for? (defined?) I think that this part of the paper requires significant improvement.

3) One valid question is how the proposed method is scalable. For example, can a model trained for 3 tasks used for 4 tasks? How hard is adding a new task? Also, worths comparing it with the learning from scratch. 

4) In Section 3, the discussion about the loss function is missing. I believe that the explanation of how to choose a loss function as well as auxiliary losses should be move there. Also, I didn't find the current explanation of BU1 and TD auxiliary loss for Multi-MNIST very clear.

5) Why the results of your method is better than the single model? This behavior should be justified. My impression is that each task trained independently should outperform any multi-training method. Your results seem counter-intuitive in this respect.

6) I am not able to make any strong conclusions from Section 4.3.2. It is really hard to tell which connection is better based on a single number. I would suggest providing confidence intervals for making such kind of arguments. For example, you may train from 10 different network initializations and use them to construct more reliable estimates. I also believe that more reliable estimations are required for Table 3. 


Minor:
* In paragraph 2 of pages 2, you mention "as illustrated in Figure 2a". I do not see the attention to a part of the image. Am I missing something? A similar issue exists in the next sentence: I don't see any content-related modulization in Figures 2b and 2c. Please clarify.
* use comma after equations if the equations are not ending the sentences. For example, add a comma after eq (1), (2) and (3). Also on page 4, "Where $W$" -> "where $W$".
* Page 4, "Our method address" -> "Our model addresses"
* Where the third column of Table 1 is defined? On page 8. Move it to earlier sections.
* In Table 2b, you have used +x, but the notation for gated modulation is something else in the text.
* Are LL and RU used in Table 1 defined in the text?
* The bold numbers in Figure 2b seem wrong. If you are bolding the large accuracies, be consistent in all tables.
* Font of table 4 can be larger
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper tackles the multi-task setting by using modulation connections between three network pipelines, i.e., one bottom-up network to contextualize the problem, one top-bottom network that is conditioned on the tasks, on a last bottom-up network that solves the task. 
The key contribution of the paper is to introduce a feature-wise and spatial-wise tensor to modulate the different neural pipelines better. Finally, they assess the proposed method on three datasets: Multi-MNIST, a yes/no CLEVR, and CUB-200.

The abstract, introduction, and related works are pretty clear. Figure 1 is also a nice summary that puts the paper architecture in perspective with another approach, and it is a very insightful sketch. I appreciate the effort of the authors to release the code with several baselines. I also acknowledge the diversity of tasks that are studied.

However, I have three concerns that I am willing with the authors.

My first concern deals with the method description, which I found a bit misleading. Thus, I not sure that I fully got all the subtleties of the proposed method. The mathematical notation is misleading: Are Y, and X function of (x,y,ch) or are tensors over x, y, z. Later in the text, W is defined over (ch, t), but it is also mentioned that t is an input. Thus, is W \in R^(CxT) or W(t) \in R^(C). Besides, the implicit tilling with * makes things even harder to follow. On a different side, what do you mean by training the convolution network instead of optimizing W. Is W fixed? Do you use simply the feature map after 1x1 conv as mentioned in 4.2? How do you embed the task t in general, how do you append it to the feature map of BU1.
In the current paper state, I would not be able to reproduce the experiments.
 
The second concern relies on the results. The gap between the methods is tiny, e.g. max 0.5 in 2-MNIST, and may fluctuate a lot from one experiment to another, e.g., it is weird that 3-MNIST is harder than 4-MNIST. Note that the same observations can be applied to the CLEVR. Therefore, it is hard to assess the method without the std over at least 5 seeds. The result only convinces me regarding 4-MNIST without such std.

My third concern is about CUB200 experiments. The authors used an auxiliary loss on top of TD to help to visualize the network decision. As such auxiliary losses provide additional information, I have the following question: did you add the same losses to other baselines? Did you use a stop-gradient before decoding the feature-maps? Otherwise, the comparison between methods may not be fair


Remarks:
 - I am missing some results from external literature. For instance, even if I prefer your setting CUB300  over 312 questions, it would have be nice to add such experiments in the appendix. (or literature baselines on N-MNIST)
 - Please report the original MOO too results in addition to your experiments
 - Why ch-mod is missing in 4-CLEVR?
 - Can you describe how did you pick the CLEVR questions (before/after computing the results)? It would have been nice to have experiments that randomly pick K questions (and repeat the process N time + report std), or even dynamically condition on the question at hand.
 - it took me quite some time to understand the meaning of #P, please make it explicit from the beginning, or add in the caption!
 - Although releasing the code is good, I also encourage you to put a table in the appendix with the hyper-parameters. The paper should be as much self-content as possible. It is also hard to evaluate the quality of the training time during the review
 - typo: extra parenthesis in Eq 4

In conclusion, the authors give some good intuition about promising methods, but I had some difficulties in understanding all the details of their approaches. Besides, I am missing both std and external references to assess the quality of the methods. In this current state, I cannot recommend paper acceptance even if I acknowledge several qualities of the paper, but I am open to discussion.


review_title: Official Blind Review #3

Meta-review: The paper is interested in multi-task learning. It introduces a new architecture which condition the model in a particular manner: images features and task ID features are fed to a top-down network which generates task-specific weights, which are then used in a bottom-up network to produce final labels. The paper is experimental, and the contribution rather incremental, considering existing work in the area. Experimental section is currently not convincing enough, given marginal improvements over existing approaches - multiple runs as well as confidence intervals would help in that respect.

Decision: Reject