Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes an way to incorporate task information into multi-task learning (MTL). The hope is that more explicit knowledge of task information will improve MTL, especially in cases where the model seems to confuse tasks (as shown in the example of speech translation in Table 1). 

Pros:
- This is a timely contribution that addresses an important problem in MTL
- The proposed method is reasonable and relatively easy to implement: it basically involves adding a network that characterizes the task, and using its outputs to modify the MTL model's parameters. 

Cons:
- While the results in Table 2 and Table 3 look promising, the presentation would benefit from more analyses experiments (see below). For example,  investigating whether the number of confused cases in Table 1 actually reduces, comparing whether the added parameters of the task-characterization network requires more data, exploring other ways to modulate the MTL model. Since the method is relatively straightforward, the paper would look stronger if it has more empirical analyses. 

Additional questions and comments: 
- Some claims need to be explained or clarified. For example in introduction the authors write: "However, providing task information is not straightforward for certain modalities such as speech and image. Also, providing explicit task information might not always be desirable, e.g., automatic multilingual speech recognition task." Why is task information not straightforward for speech? I understand it is harder to integrate the discrete label with the speech signal but it has been done before and the reason needs to be spelled out (i.e. don't assume the reader works in ASR.) 
- Can you list out the same examples in Table 1 for the proposed systems? Did they solve the problem of confused tasks? 
- Can you clarify how many new parameters are added for the different task-characterization networks, relative to the full network? It was also be interesting to fix to random weights in the task-characterization networks to see if the results are expected. 
- Please discuss other methods for modulating the weights of the model, besides the one used. I think it's not necessarily to do experiments in those, but mentioning other potential techniques for modulation would make the idea more comprehensive, perhaps suggesting it as future work. 
review_title: Proposed method is reasonable but more analyses would strengthen the paper

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
This paper proposes a new framework that computes the task-specific representations to modulate the model parameters during the multi-task learning (MTL). This framework uses a single model with shared representations for learning multiple tasks together. Also, explicit task information may not be always available, in such cases the proposed framework is useful. The proposed framework is evaluated on various datasets spanning multiple modalities, where the MTL model even achieves state-of-the-art results on some datasets. 

Pros:
1) The paper is well written and easy to follow. 

2) The proposed framework achieves state-of-the-art results on some tasks/datasets. 

Cons: 
1) Seems like the implicit approach does perform more or less the same as the explicit approach. I believe the explicit approach is not something new and has been tried in various works. In that sense, the implicit approach is not exciting. I agree that the proposed framework would be able to learn task specific information without any explicit supervision, but this use case is not well explored in this paper. Would your approach scale to N (> 5-10) tasks? 

2) From Table-2, OHV performs better than explicit TCN. The paper argues that providing OHV text labels is not always possible for tasks involving non-text modalities such as speech and images. But, I think it is possible to learn simple joint space task embeddings presentations for tasks from multiple modalities. 

3)  From Table-3, the difference between explicit and implicit TCN methods is small. Statistical significance scores are not provided to know whether the improvements are real. I believe the explicit TCN approach is not something new so it would be better to know if implicit learning is strictly better than explicit.

Overall: 
The paper is well written and easy to follow with SOTA results. However, I am concerned about the novelty aspect of the proposed framework. It is not clear from the current experiments whether the proposed task-specific representation learning is better than a simple OHV or simple multimodal joint-space task embeddings. 

Questions:
1) Is implicit TCN better than explicit TCN in Table-3? Can you provide the statistical significance scores?

2) Instead of OHV, can you learn a join space task embedding representation? This paper: https://arxiv.org/pdf/1611.04558.pdf used special tokens for multilingual translation, can you think of extending to the multi-modal setup with speech and image? If not possible, why?

3)  Would your approach of learning task-specific representation learning scale to more task i.e., > 10 tasks?

Other comments: 
1) It would have been easier to understand the implicit task information section if there is a corresponding pictorial presentation of the approach. 


review_title: Automatic learning of task-specific representations without any explicit representation in an MTL setup

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a multi-task Transformer that does speech and machine translation within the same framework. It proposes task adaptive parameters by modulating model parameters to account for differences in tasks. The authors conduct experiments on speech and machine translation and show reasonable performance.

Honestly, this paper is lacking because this is just too incremental over the T5 model (which can train over 20 tasks without any task specific parameters). It feels quite lacklustre comparing to the many exciting multi-task or transfer learning works out there. The notion of training speech and language adds a little to the paper but I think it is insufficient to feel excited about.

The results in the paper don't look terribly good either. I'm also perplexed why the performance is even outperformed by the baseline. I also don't think it is really a good idea to reduce T5 to "OHV" (one hot vector) and completely remove the pretraining completely. Namely, "Unlike T5, we don’t initialize the models with the text embeddings from large pretrained language model (Devlin et al., 2018). Instead, we focus on establishing the importance of task information during MTL and having a single model for all the tasks." this is way too convenient and unconvincing. I think the authors should compare their model with T5 to have an apples to apples comparison. 

I have also concerns about the practicality of modulating parameters and how this would cope over N task when N is large. The number of tasks in this paper is very small (compared to T5) and this is not well ablated in the paper. What about other adapter based approaches like projected attention layers (https://arxiv.org/abs/1902.02671) or hypergrid (https://arxiv.org/abs/2007.05891)? I think it's worth to compare the modulated parameters with other forms of multi-task adaptations of these transformer models. 

Overall I recommend rejection. While this is a promising direction, this is poor execution. The proposed novelty of modulated parameters is also insufficient. Results are also not very exciting. 


review_title: Room for improvement (good direction but poor execution)

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper combines Feature-wise Linear Modulation (FiLM) with a single/multi-modal transformer for a joint multi-task neural network and applies the proposed model to two tasks, which are single-modal machine translation and multi-modal (speech/text) machine translation, speech translation, and speech recognition.

I have the following comments and reservations:
+ The paper is well-written and easy to follow.
- There is a plethora of closely related work that was not mentioned or discussed or compared to, e.g., Zhao et al.. 2018; Strezoski et al., 2019; Cheung et al., 2019, to name a few.
- The paper lacks comparison to strong multi-task baselines and compares the proposed model only to standard multi-task, transfer learning, and meta learning approaches and single-task state-of-the-art approaches that don’t have access to the additional data from the other tasks.
- The main title reads: “Learning without Forgetting”, yet there is no discussion or experiments with other methods that address the forgetting problem [Li & Hoiem, 2018].

As such, the work lacks sufficient novelty, is not well-placed within existing related prior work, and does not compare to adequate baselines.

References:

Brian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Superposition of many models into one. arXiv preprint arXiv:1902.05522, 2019.

Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell., 40(12):2935–2947, 2018. doi: 10.1109/TPAMI.2017.2773081.

Gjorgji Strezoski, Nanne van Noord, and Marcel Worring. Many task learning with task routing. arXiv preprint arXiv:1903.12117, 2019.

Xiangyun Zhao, Haoxiang Li, Xiaohui Shen, Xiaodan Liang, and Ying Wu. A modulation module for multi-task learning with applications in image retrieval. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 401–416, 2018.
review_title: Multi-Task + FiLM + Transformers + Multi-Modal

Meta-review: This paper proposes an approach for improving MultiTaskLearning by providing a way of incorporating task specific information.

Pros:
1) All reviewers agreed that the paper is clearly written
2) Interesting to see a single model for AST, STS (speech-to-speech translation) and MT 

Cons:
1) The work is not adequately compared with related work (some important references are also missing) - The authors did perform some additional experiments with T5  and pointed out some drawbacks but this needs to be explored a bit more.
2) The answers about scalability are not very convincing and need more empirical results. 

Overall, none of the reviewers were very positive about the paper and felt that while this is a good first attempt, more work is needed to make it suitable for acceptance. 
Decision: Reject