Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: In this paper a novel top-down control network is introduced for multi-task learning. Different from the traditional bottom-up attention models, the authors introduce a top-down module to modify the activation of recognition network based on different tasks. Specifically，the proposed module consists of three identical networks, which are BU1, TD, BU2 streams. Given the input, the BU1 is firstly trained, and then the TD streams is trained by assigning the specific labels. After that, the BU2 is updated with the top-down parameters. Experimental results demonstrate the effectiveness of proposed model.

Strength:
1. It is interesting to introduce the semantic information from the top layer to guide the feature representation learning.

Weakness:
 Although the experimental results show the better performance on the image classification, there are exist several unclear parts:

1. The definition of multi-task in this paper refers to the different dataset’s classification? Or referring to the different tasks, e.g. localization, classification, and attributes predication. In my opinion, the authors should provide more details on designing the validation experiments. And the proposed model should be tested on different tasks instead of only on the task of visual recognition.
2. Some bottom-up based model e.g. FiLM should be used as the baselines to validate the advantage of introducing the top-down stream.
3. If the top-down stream would make the recognition be sensitive for the visual variations? Or the classification results may be dependent on the number of training samples?

4. If the proposed model would be helpful for transfer learning?

review_title: An interesing paper on introducing the top-down information as the supervision for multi-task neural network learning.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper propose a way to combine image information and task information as controllers for multi-task learning. In this way, the authors expect to extract more task/image specific features in a shared backbone.

The proposed method seems novel and intuitive. Though there are some typos and grammar mistakes, overall the paper is easy to follow. 

My major concerns are the followings:
1. the claim that the proposed scheme provides scalability with the number of tasks is stretching: the number of tasks still need to predefine and the number of task heads are not optimized compared to previous algorithms. 
2. While there is improvement in performance, but it is not clear what factors causes the improvement. As compared to previous schemes the proposed model needs much more computationally (two bottom up runs and a top-down run) and utilizes more information.
3. All the tasks are classification tasks /discriminative models. It is not demonstrated if this would be working with a mixture of generative/discriminative tasks.  

If possible, as the TD and BU's are sharing the same structures, it would be interesting to explore what are learned by visualizing the weights in each channels and layers. and further explore which feature map is heavily used for which specific task. These weights/activation distributions will help us to better understand what is actually learned in the proposed scheme.  The task spatial maps seems to be a good start, but it would be better if the author provide more analysis on intermediate layer activations 
review_title: More analysis would be appreciated.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a new architecture for multi-task learning that uses a top-down control network to modulate the activations of the main (bottom-up) recognition network. The model is applied to four datasets/tasks: multi-MNIST, CLEVR, CELEB-A, and CUB-200, demonstrating good performance compared with baselines. I have the following comments and questions:

- The proposed architecture requires the equivalent of three forward/backward passes: BU1, TD, and BU2. The number of parameters is used gauge complexity in Table 2 and Figure 3 but FLOPs might be a better metric here since BU1 and BU2 share parameters. How do the models compare in terms of FLOPs to baselines? 
- The datasets/tasks used herein are homogenous and therefore straightforward for multi-task learning. How does the proposed architecture fair in a more challenging setting involving heterogeneous tasks, e.g., Misra et al., 2016?
- Re "In some of the experiments of CLEVR and CUB-200 datasets we added an auxiliary loss at the end of the TD stream. The target in this case is a 224x224 mask, where a single pixel, blurred by a Gaussian kernel (s.d. 3 pixels) was labeled as the target location.", How was this mask obtained? How do the models perform without the use of this auxiliary loss? Why was this loss only used in CLEVR and CUB-200?

Minor:
- Add references to first column in Table 2.

To conclude, the proposed architecture is novel, the paper is clear, but the experimental work leaves some questions unanswered.
review_title: BU-TD Network for Multi-task Learning

Meta-review: The paper is very interesting and novel, and all reviewers are of the same opinion. 
The main concern, however, is on the experimental section that is limited to image classification benchmarks and that some critical comparisons are missing (e.g. clarify factors that play key role in improvement, more computation and therefore more free parameters, how about non discriminative tasks, etc). 
The heterogeneity question is in my opinion only partially answered by the authors but I also feel proper handling of this matter would require a proper multi-task setup and different target for the work.
I also personally find applicability of the approach quite limited, I encourage the authors to further improve their work as I feel that with a proper revision would make a nice contribution for the community.
Decision: Reject