Learning without Forgetting: Task Aware Multitask Learning for Multi-Modality Tasks
Authors: Sathish Reddy Indurthi
Mohd Abbas Zaidi
Nikhil Kumar Lakumarapu
Beomseok Lee
HyoJung Han
Sangha Kim
Inchul Hwang
Venue:  (2021)
Keywords: Deep Learning
Joint Learning
Meta Learning
Multi-task Learning

Abstract: Existing joint learning strategies like multi-task or meta-learning focus more on shared learning and have little to no scope for task-specific learning. This creates the need for a distinct shared pretraining phase and a task-specific finetuning phase. The fine-tuning phase creates separate systems for each task, where improving the performance of a particular task necessitates forgetting some of the knowledge garnered in other tasks. Humans, on the other hand, perform task-specific learning in synergy with general domain-based learning. Inspired by these learning patterns in humans, we suggest a simple yet generic task aware framework to incorporate into existing joint learning strategies. The proposed framework computes task-specific representations to modulate model parameters during joint learning. Hence, it performs both shared and task-specific learning in a single-phase resulting in a single model for all the tasks. The single model itself achieves significant performance gains over the existing joint learning strategies.  For example, we train a model on Speech Translation (ST), Automatic Speech Recognition (ASR), and Machine Translation (MT) tasks using the proposed task aware joint learning approach. This single model achieves a performance of 28.64 BLEU score on ST MuST-C English-German, WER of 11.61 on ASR TEDLium v3, and BLEU score of 23.35 on MT WMT14 English-German tasks. This sets a new state-of-the-art performance (SOTA) on the ST task while outperforming the existing end-to-end ASR systems with a competitive performance on the MT task. 

Under review as a conference paper at ICLR 2021
LEARNING WITHOUT FORGETTING: TASK AWARE MULTI-TASK LEARNING FOR MULTI-MODALITTY TASKS
Anonymous authors Paper under double-bling review
ABSTRACT
Existing Multi-Task (MTL))ng(MTL) strategies like joint or meta-learning focus more on shared learning and have little to no scope for task-specific learning. This creates the need for distinct shared pretraining phase and a task-specific finetuning phase. The finetuning phase creates separate models for each task, where improving the performance of a particular task necessitates forgetting some of the knowledge garnered in other tasks. Humans, the other hand, perform task-specific learning in synergy with general domain-based learning. Inspired by these learning patterns humans, we suggest a simple yet generic task aware framework to incorporate into existing MTL strategies. The proposed framework computes task-specific representations modulate the model parameters during MTL. Hence, it performs both shared and task-specifee learning in a single phase resulting in a single model for all the tasks. The single model itself achieves significant performance gains over the existing MTI. strategies. For example, we train model on Speech Translation (ST), Automatic Speech Recognition (ASR), and Machine Translation (MT) tasks using the proposed task aware multitask learning approach. This single model achieves a performance WER of 11.61 on ASR TEDLium v3, and BLEU score of 23.35 MT WMT14 English-Germnn tasks. This sets a new state-of-the-an performance (SOTA) on the ST task while outperforming the existing end-to-end ASR systems with competitive performance on the MT task.
28.6c BLEU score on ST MuST-C English- English-German,
INTRODUCTION
The process of Multi-Task Learning (MTL) on set of related tasks is inspired by the patterns displayed by human learning. involves pretraining phase over all the tasks, followed by a finetuning phase. During pretraining, the model tries to grasp the shared knowledge of all the tasks involved, while in the finetuning phase, task-specific learning is performed to improve the performance. However, the other tasks that it learnt during pretraining. Humans, on the other hand, are less susceptible forgetfulness and retain existing knowledge/skills while mastering a new task. For example, a polyglot who masters a new language learns to translate from this language without losing the ability to translate other languages. Moreover, the lack of task-based flexibility and having different finetuning/pretrainii phases cause gaps in the learning process due to the following reasons:
result of the finetuning phase, the model forgets the information about
Role Mismatch: Consider the MTL system being trained to perform the Speech Translation(ST), Automatic Speech RecognitionnASRR and Machine (ranslation(M'T) tasks. The Encoder block has very different role in the standalone ASR, MT and ST models and hence we cannot expect a single encoder perform well all the tasks without any cues to identify/use task information. Moreover, there is a discrepancy between pretraining and finetuning hampering the MTL objective.
Task Awareness: At each step in the MTL, the model tries to optimize over the task at hand. For tasks like ST and ASR with the same source language, is impossible for the model to identify the task and alter its parameters accordingly. hence necessitating finetuning phase. A few such examples have been provided Table Humans, on the other hand, grasp the task they have to perform by means of context explicit cues.

---

Under review as a conference paper at ICLR 2021
Task
Confounding Task
MTL Prediction
Task Aware Output
sogar ihre eigenen Eltern
even their own parents Mobility opportunity to
En ASR
En-De ST
Mobilität opportunity to..
En-De MT
En-Ro MT
Situatia este alarmantă
Die Situation ist alarmierend Das verstehe ich nicht war ein Erfolg oder nicht.
don't 't understand it.
En-De ST
En ASR
was success or not.
Table 1: Issues due to the lack of task information in MTL strategies. The MTL model, unable to identify the task, produces the output corresponding to another task either completely or par- (En:Eiihn,,gglihh. De:German, Ro:Romanian), Task aware output is the output obtained from our proposed approach.
Although MTL strategies help the finetuned models perform better than the models directly trained on those tasks, their applicability is limited to finding a good initialization point for the finetuning phase. Moreover, having separate model for each task increases the memory requirements, which detrimental in low resource settings.
In order to achieve the goal of jointly learning all the tasks, similar to humans, we need to perform shared learning in synergy with task-specific learning. Previous approaches such as Raffel et al. (2019) trained a joint model for a set of related text-to-text tasks by providing the task information along with the inputs during the joint learning phase. However, providing explicit task information is not always desirable, c.g., consider the automatic multilingual speech translation task. In order to ensure seamless user experience, it is expected that the model extracts the task information implicitly.
Thus, a holistic joint learning strategy requires a generic framework which learns task-specific information without any explicit supervision.
In this work, we propose a generic framework which can be easily integrated into the MTL strategies which can extract task-based characteristics. The proposed approach helps align existing MTL approaches with human learning processes by incorporating task information into the learning process and getting rid of the issues related forgetfulness. We design modulation network for learning the task characteristics and modulating the parameters of the model during MTL. As discussed above, the task information may or may not be explicitly available during the training. Hence, we propose two different designs of task modulation network to learn the task characteristics; one uses explicit task identities while the other uses the examples from the task as input. The model, coupled with the modulation network, jointly learns on all the tasks and at the same time, performs the task task-specific learning. The proposed approach tackles issues related to forgetfulness by keeping a single model for all the tasks, and hence avoiding the expensive finetuning phase. Having a single model for all the tasks also reduces memory constraints, improving suitability for low resource devices.
To evaluate the proposed framework, we conduct two sets of experiments. First, we include the task information during MTL on text-to-text tasks to show the effect of task information. Secondly, we train a model on tasks with different modalities and end goals, with highly confounding tasks. Our proposed framework allows the model to learn the task characteristics without any explicit supervision, and hence train a single model which performs well on all the tasks. The main contributions of this work are follows:
We propose approach to tackle the issue of forgetfulness which occurs during the finetuning phase of existing MTL strategies.
Our model, without any finetuning, achieves superior performance on all the tasks which alleviates the need to keep separate task-specific models.
Our proposed framework is generic enough be used with any MTL strategy involving tasks with multiple modalities.
2 TASK-AWARE MULTITASK LEARNING
An overview of our proposed approach is shown in Figure 1
2

---

Under review as a conference paper at ICLR 2021
Ouipuks
Transformar Network
Decoder
Torget
Prproceesing Layar
Decoder
Masked Mut-- lead Attertion
Mutt-Heed Altertion
Feed Forward
Task IDs
Charactaristic Natwork
FILM
Enerierr FILM
Mulii Head Attotiion
Food Forward
Source
Preprocaeeing
Luyar
Encoder
NX
Figure I: Overview of task aware multi-tast task learning. Task Characteristics Network (TCN) produces task embedding to modulate the parameters of base network. Inputs the TCN chosen based on type of TCN.
2.1 BASE MODEL
In general, the sequence- soreencett--eqeenee architecture consists of two components: (1) an encoder which computes set of representations X (....... Im Raxd corresponding æ, and decoder coupled with attention mechanism (Bahdanau et al. 2015) dynamically reads encoder's output and
predicts target language sequence Y 9/. ... yn Raxd It trained on a dataset to maximize the (Y 0)),, where 0 are parameters of the model.
We use the Transformer Vaswani al. (2017) as our base model. Based on the task modalities, we choose the preprocessing layer in the Transformer, i.e., speech or the text (text-embedding) preprocessing layer. The speech preprocessing layer consists of a stack of & CNN layers with stride for both time and frequency dimensions. This layer compresses the speech sequence and produces the output sequence such that input sequences corresponding to all the tasks have similar dimensions, model is shown in the rightmost part of Figure
d. The overview of the base
2.2 TASK MODULATION NETWORK
The task modulation network performs two operations. In the first step, it computes the task characteristics (t,z) using the task characteristics layer. It then modulates the model parameters O using be in the second step.
2.2.1 TASK CHARACTERISTICS NETWORK:
We propose two types of Task Characteristics Networks(TCN) to learn the task characteristics, where one uses explicit task identities while the other uses source-target sequences as input.
Explicit Task Information: In this approach, the tasks involved are represented using different task identities and fed as input to this TCN as one hot vectors. This network consists of a forwafd layer which produces the task embedding used for modulating the model parameters.
t, FF)(e).
(1)
where RS is one-hot encoding of tasks used during joint learning.
Implicit Task Information: The Implicit TCN computes the task embeddings using example sequences from the tasks without any external supervision. It consists of four sub-layers: (1) Sequence Representation Layer, (2) Bi-directional Attention Layer, (3) Sequence Summary Layer, and (4) Task Embedding Layer.
The sequence representation sub-layer consists of uni-directional Transformer Encoder (TE) blocks Vaswani et al. (007)). It takes the source and target sequences from the tasks input and produces
3

---

Under review as a conference paper at ICLR 2021
self-attended source and target sequences.
X
TE) == TE(Y).
(2)
where X E RRMxd, you E Rx This sub-layer computes the contextual representation of the sequences.
The Bi-directional Attention (BiA) sub-layer takes the self-attended source and target sequences from the previous laver as input and computes the relation between them using Dot-Product Attention Luong et al. (2015). As a result, we get target aware source (X"tt E IR RMMxa and source aware target representations as outputs.
X BiA(X" Yo8), y BiA X*
The sequence summary sub-layer is similar to the sequence representation sub layer and summarizes sequences. The sequence summaries are given by:
(4) where X E ERWxd Y Nxd xxd The Equation 4 summarizes the sequences X and Y which
X X (X" Y TE,(Y
contain the contextual and attention information. We take the last tokens from both the x E Rd and y E Rdd since the last token can see the whole sequence and acts summary of the sequence.
The task embedding layer computes te by taking the outputs the sequence summary sub-layer and applying feed-forward network:
te = FFF y)"").
2.2.2 MODULATING MODEL PARAMETERS
We modulate the parameters (0) of the network (Section 2.1 to account for the task-specific variation during MTL over set of tasks. We achieve this by scaling (7) and shifting (G) the outputs of each layer (e.g., transformer block) including any preprocessing layers in the model adopted based on the Reature-wise Linear Modulation (FiLM: Perezz et al. (2018)) The y and (3 parameters are obtained from the task embedding te either by using equation or
7= Lel: d],
te [d:] :].
where te E R2d, and d is the hidden dimension of the model.
Once we have y and B, we apply the feature-wise linear modulation (Perez et al. 2018 to compute the modulated output (0₂) for each block of the model.
O: =y (1 0:) B, 1 = L,
where L is the total number of blocks in the model and fi represents the Le block of the model with parameters 9 E and inputs ty.
2.3 TRAINING
MTL has been successfully applied across different applications of machine learning such natural language processing (Hashimoto et al. 2016 Collobert & Weston 2008). speech recognition (Liu
al., 2019 Deng et al.] al.] (203)), computer vision (Zhang et al.J 2014 Liu al. 2015 Girshick 2015), and drug discovery (Ramsundar et al. 2015). It comes in many forms: joint learning, learning
to learn, and learning with auxiliary tasks. We consider two MTL strategies: (1) joint learning
and (2) learning learn to train set of S tasks, = (7),... .......S) with corresponding datasets
D= = (D', (DS
DS}.
As our first training strategy, use Joint Learning (JL) (Caruana 1997 which is the most commonly used training strategy for MTL. In JL, the model parameters, including the output layer, are shared across all the tasks involved in the training. For the second training strategy under the learning-to- learn approach, we use a variant meta-learning, Modality Agnostic Meta Learning (MAML) (Finn) et al. 2017a). Even though MAML is mostly used in sho-shot learning settings,
it since
4

---

Under review as a conference paper at ICLR 2021
No.
MTL Strategy
Training Approach
En-De (f)
En- En-TT((T)
En-Ro((1)
Vanilla
7.65
3.59
5.55
Joint Learning
OHV(Raffel et al.
2019
16.77
6.10
11.79
3
Meta Learning
MAML (Finn et al.
201
7.74
2.93
5.78
OHV et al.
2019
16.08
5.93
12.43
This Work
5
Joint Learning
Implicit TCN
14.18
4.38
13.47
Meta Learning
Implicit TCN
14.50
4.43
14.32
Table Performance(BL.F5 of the models trained on the Single Modality for the En-De, En-Ro and En-Tr Machine Translation task. *Models based on OHV and MAMI are inspired from Raffel et al. (2019) and Finn et al. (2017a).
allows for task-specific learning during the meta-train step and it has also been shown to provide improvements in the field of speech conslation(Indurthh et al. 2020).
We resolve the source-target vocabulary mismatch across different tasks in MTL by using vocabulary of subwords (Sennrich al. 2016) computed from all the tasks. We sample batch of examples from D and use this as input to the TCN and the Transformer model. To ensure that each training example uses the task embedding computed using another example, we randomly shuffle this batch while using them as input to the TCN. This random shuffling improves the generalization performance by foreing the network learn task specifc characteristics (f₂)) Equation [ or 5 We compute the task embedding in the meta-train step well; however, the parameters of the TCN are updated only during the meta-test step. During inference time, we use the precomputed task embeddings using a batch of examples randomly sampled from the training set.
3 EXPERIMENTS
3.1 TASKS AND DATASETS
We conduct sets of experiments, with the tasks having the input modality, i.e., text and another over tasks having different input modalities, i.c., speech and text. The main motivation behind the text-based experiments to establish the importance of providing task information in MTL. Our main experiments, containing different input modalities involve highly confusing tasks. These experiments help us demonstrate the effectiveness of our approach generic setup. We incorporate the proposed task modulation framework into joint and meta-learning strategies and analyze effects.
3.1.1 SINGLE MODALITY EXPERIMENTS
We perform the small scale text-to-text machine translation task over three language pairs English- German/Romanian/ Turrkia (En-De/Ro/Tr). We keep English as the source language, which makes it crucial use task information and produce different outputs from the same input. Since it is easier to provide task identity through one-hot vectors in text, we provide the task information by simply prepending the task identity the source sequence of each task, e.g.. "translate from English to German", "translate from English to Turkish" similar to Raffel et al. (2019). We also train models using our proposed framework to learn the task information and shared knowledge jointly.
For En-De, we use 1.9M training examples from the Europarl v7 dataset. Europarl dev2006 and News Commentary nc-dev2009 are used as the dev and Europarl devtest200 6, Europarl test200 and News Commentary nc-devtest2007 as the test sets. For En-Tr we train using 200k training examples from the set imes2 2 dataset. We use newsdev201. as the dev and newstest ing examples from Europarl v8 and set imes2 datasets. We use newsde deved 2011 116 as dev and newstest 2016 6 the test set.
the test set. For En-Ro, use 600k train-
5

---

Under review as a conference paper at ICLR 2021
No.
MTL Strategy
Training
ST
ASR())
MT
N/A
Direct Learning
22.14
25.25
27.93
2
Pretraining
0.81
14.83
23.62
Joint Learning
3
+ Finetuning
25.99
12.95
22.63
4
Meta Learning
Pretraining
0.51
32.93
11.40
5
(Indurthi et al. 2020)
+ Finetuning
26.03
12.83
22.15
This Work
6
Explicit TCN
28.34
11.99
23.15
7
Joint Learning
Implicit TCN
28.64
11.61
23.35
Taskwise Best Models
28.88
11.36
23.71
Explicit TCN
28.16
11.68
23.13
10
Meta Learning
Implicit TCN
28.55
11.65
23,41
11
Taskwise Best Models
28.59
11.28
23.57
Table 3: Performance of the models trained on the Multiple Modality tasks, i.e., ST (MuST-C En-De), ASR (TED-LIUM 3), and MT (WMT'15 15 En-De).
3.1.2 MULTIPLE MODALITY EXPERIMENTS
To alleviate the data scarcity issue in Speech Translation (ST), several MTL strategies have been proposed to jointly train the ST task with Automatic Speech Recognition (ASR) and Machine Translation (MT) tasks. These MTL approaches lead to significant performance gains on both ST and ASR tasks after the finetuning phase. We evaluate proposed framework based on this multimodal MTL setting since passing the task information explicitly via prepending labels(like the text-to-text case) in the source sequence is not possible. We the following datasets for ST English-German, tasks:
ASR English, MT
MT En-De: We the Open Subtitles (Lison al. 2019) and WMT 19 corpora. WMT 19 consists of Common Crawl, Europarl v9, and News Commentary v 14 datasets(22M training examples).
ASR English: We used five different datasets namely LibriSpeech (Panayotov et al. 20155. MuST-C Di Gangi et al. 2019), TED-LIUM (Hernandez et al. 2018), Common Voice (Ardila et al. 2020)
and filtered IWSLT 19 IWS 2019
train the English ASR task.
ST Task: We use the Europarl ST (Iranzo-Sánchez et al. 2019), IWSLT 2019 (IWS) 2019) and
MuST-C (Di Gangi et al., 2019) datasets. Since ST task has lesser training examples, we use data
augmentation techniques (Lakumarapu et al. 2020)) to increase the number of training examples.
Please refer to the appendix for more details about the data statistics and data augmentation techniques used. All the models reported in this work use the same data settings for training and evaluation.
3.2 IMPLEMENTATION DETAILS AND METRICS
We implemented all the models using Tensorflow 2.2 framework. For all our experiments, we use the Transformer( Vaswani et al. 2017) as our base model. The hyperparameter settings such as learning rate, scheduler, optimization algorithm, and dropout have been kept similar to the Transformer, other than the ones explicitly stated to be different. The ASR performance is measured using Word Error Rate (WER) while ST and MT performances are calculated using the detokenized cased BLEU score (Post) 2018) We generate word-piece based universal vocabulary (Gu et al. 2018a) of size 32k using source and target text sequences of all the tasks. For the task aware MTL strategies, we choose a single model to report the results rather than finding the best model for each task separately.
We train the text-to-text translation models using Encoder and Decoder layers with batch size of 2048 text tokens. The training is performed using NVIDIA P40 GPU for 400k steps.
In multi- modalit experiments, the speech signals are represented using 50-dimensional log-Mel features and use 3 CNN layers in the preprocessing layer described in Section 2.1 We use 12 Encoder and Decoder layers and train for 600k steps using 8 NVIDIA V100 GPUs. For the systems without TCN, perform finetuning for 10k steps on each task.

---

Under review as a conference paper at ICLR 2021
S. No.
Task
Approach
(BLEU(()WER())
1
Liu et al. (6020)60 Encoders)
30.10
MT (WMT" 15 En-De)
2
Our Approach(11 Encoders)
23.71
3
Indurthi et al. (2020)
22.11
4
Pino al. 2020)
25.99
ST (MuSTC En-De)
5
Lakumarapu et al. 2020
27.51
6
Our Approach
28.88
7
Pham et al.
2019 (36 Encoders)
10.20
8
Pham et al. 2019 (12 Encoders)
12.40
ASR (TED-LIUM 3)
9
Our Approach 12 Encoders)
10.01
Table Performance comparison with the existing works in ASR, ST and MT.
plc
-3
-30
115
Fookd Germon
(a) Language pairs for MT task
(b) ASR, MT and ST tasks
D-SNE plct
ASh:
ASR: Tediimm
ASR: Librspecch EuroPart
MUSTIC
IYSUT
-20
Opensurortes
WHI
(c) Dataset-wise plots for ASR, MT and tasks
Figure 2: t-SNE plot of the task modulation output shows clear demarcation between various tasks and datasets.
3.3 RESULTS
3.3.1 SINGLE MODALITY EXPERIMENTS
The results for the text-to-text translation models trained with different MTL strategies have been provided in Table 2 The MTL models with prepended task label (Raffel et al. 2019) are referred to as OHV (One Hot Vector). Unlike T5, we don't initialize the models with the text embeddings from large pretrained language model (Devlin et al.) 2018). Instead, we focus on establishing the importance of task information during MTL and having a single model for all the tasks. As we can see from the results, providing the task information via text labels or implicitly using the proposed task aware MTL leads to significant performance improvements compared to the MTL without the task information. The models trained using OHV have better performance than those trained using implicit TCN. However, providing OHV via text labels is not always possible for tasks involving non-text modalities such speech and images.
7

---

Under review as a conference paper at ICLR 2021
3.3.2 MULTI MODALITY EXPERIMENTS
We evaluate the proposed two TCNs and compare them with the vanilla MTL strategies. The performance of all the models is reported in Table We also extended the T5 (Raffel et al. 2019) approach to the multi modality experiments and compare it with our approach.
The models trained using task aware MTL achieve significant per- formance gains over the models trained using vanilla MTL approach. Our single model achieves superior performance compared the vanilla MTL models even after the finetuning. This shows that not only is the task information essential to identify the task, but also helps to extract the shared knowledge better. Our JL and MAML models trained with task aware MTL achieve improvements of (+2.65, +2.52) for ST, (-1.34, -1.18) for ASR, and (+0.72, +1.26) for the MT task. MAML has some scope for task-specific learning during its meta train step, which explains why the improvements for MAMI are slightly lesser than JI. for ST and ASR tasks.
Effect of Task Information:
We also report results using Direct Learning (DL) approach, where separate models are trained for each task, to compare with MTL models. All the MTL models outperform the DL models on ST and ASR tasks have comparable performance on MT task.
Our proposed implicit TCN learns the task characteristics directly from the examples of each task and achieves a performance comparable to the models trained using explicit TCN. This indicates that it is better to learn the task information implicitly, specifically for tasks having overlapping characteristics. Figure contains the iSNE plots for task embeddings obtained from the implicit TCN for single and multi modality experiments. We can observe that the implicit TCN is also able to separate all the three tasks effectively without any external supervision.
Explicit v/s Implicit TCN:
Single model for all tasks: We select one single model for reporting the results for our approach, since, having a single model for multiple tasks is favourable in low resource settings. However, we also report the best models corresponding each task (row 8 and 11 of Table 3). We observe that choosing single model over task-specific models did not result in any significant performance loss.
Feature-wise v/s Input based modulation: We also implemented the input based conditioning (Toshniwal et al., 20188) Raffel et al. 2019) where we prepend the TCN output, i.e., task information to the source and target sequences. As compared to our approach, this approach provides comparable performance on the ASR task. However, the ST performance is erratic and the output is mixed between ST and ASR tasks. This shows that the feature- reature-wise modulation is more efficient way to carry out task-based conditioning for highly confusing tasks like ST and ASR.
Number of parameters added: The Explicit TCN, which is a dense layer, roughly 1500 new parameters are added. For the Implicit TCN, roughly 1 million new additional parameters are added. However, simply increasing the number of parameters is not sufficient to to improve the performance. For e.g., we trained several models by increasing the number of layers for encoder and decoder upto 16. However, these models gave inferior performance as compared to the reported models with 12 encoder and decoder layers.
Scaling with large number of tasks: The t-sne plots in the Figure 2b are drawn using the three test datasets. However, we used multiple datasets for each of the A.Rss.irrsseecch Common voice, TEDLIUM, MuSTC-ASR), ST (MuSTC, IWSLT20, Europarl), and MT (WMT19, OpenSubtitles) tasks in the multi-modality experiments. We analyze whether or not our proposed approach is able to separate the data coming from these different distributions. As compared to data coming from different tasks, separating the data coming from the same task(generated from different distributions) is more difficult. Earlier, in Figure 2b we observed that the output is clustered based on the tasks. Figure 2e shows that within these base- clusters, there are sub-clusters based on the source dataset. Hence, the model is able to identify each sub-task based on the source dataset. The model also gives decent performances on all of them. For example, the single model achieves a WER of 7.5 on the Librispeech tst-clean, 10.35 on MuSTC, 1 11.65 on the TEDLIUM v3 and 20.36 on the commonvoice test set. For the ST task, the same model gives a BLEU score of 28.64 on the MuSTC test set, 27.61 on the IWSLT tst-2010, and 27.57 on the Europarl test set. This shows that our proposed approach scales well with the total number of tasks.
Comparison with existing works: The design of our system, i.e., the parameters and the related tasks were fixed keeping the ST task in mind. We compare the results of our best systems(aftee checkpoint averaging) with the recent works in Table 4 We set new state-of-the-art (SOTA) on the
8

---

Under review as a conference paper at ICLR 2021
ST En-De MuST-C task. For the ASR task, we outperform the very deep Transformer based model Pham et al. (2019). We achieve a 19.2% improvement in the WER as compared to the model with MT model achieves
the same number of Encoder and Decoder blocks. The best transformer- BLEU score of 30.10, however, it uses 60 Encoder blocks. The performance drop on the MTT task is attributed to simply training bigger model without using any additional initialization techniques proposed in Liu et al. (2015); Wu et al. (2019). However, the MT task helps the other tasks and improves the overall performance of the system.
RELATED WORK
Various MTL techniques have been widely used improve the performance of end-to-end neural networks. These techniques are known to solve issues like overfitting and data scarcity. Joint learning (Caruana 1997) improves the generalization by leveraging the shared information contained in the training signals of related tasks. MAMI. (Finn et al. 201 was proposed for training joint model on a variety of tasks, such that it can quickly adapt to new tasks. Both the learning approaches require a finetuning phase resulting in different models for each task. Moreover, during finetuning phase the model substantially forgets the knowledge acquired during the large-scale pretraining.
One of the original solutions to this problem is pseudo-rehearsal, which involves learning the new task while rehearsing generated items representative of the previous task. This has been investigated and addressed to certain extent in AAtiinson al. (2018) and Li & Hoiem (2018). 8) He et al. (2020) address this by using a mix-review finetuning strategy, where they include the pretraining objective during the finetuning phase. |Raffel et al. (2019) take different approach by providing the task information to the model and achieve performance improvements on different text-to-text tasks. Although this alleviates the need for finetuning, it cannot be extended to the tasks involving complex modalities. In our work, we propose a generic framework on top of MTL to provide task information to the model which can be applied irrespective of the task modalities. It also removes the need for finetuning. tackling the issue of forgetfulness at its root cause.
few approaches have also tried to train multiple tasks with single model, Cheung et al. (2019) project the input to orthogonal sub-spaces based on the task information. In the approach proposed by & Hoiem (2018), the model trained on various image classification tasks having the same input modality. They preserve the output of the model on the training example such that the parameters don't deviate much from the original tasks. This is useful when the tasks share the same goal, e.g. classification. However, we train on much more varied set of tasks, which might also have the same inputs with different end goals. Strezoski et al. (2019) propose to apply a fixed mask based on the task identity. Our work these approaches, our model is capable of performing both task identification and the corresponding task learning simultaneously. It learns to control the interactions among various tasks based on the inter-task similarity without any explicit supervision.
model,,Cheung
be seen as a generalization of this work. As compared to all
In the domain of neural machine translation, several MTL approaches have been proposed (Gu et al. 2018alb) b). Similarly, recent works have shown that jointly training ST, ASR, and MT tasks improved the overall performance (Liu et al. 2019 Indurthi et al. 2020). However, all these require separate finetuning phase.
5 CONCLUSION
This work proposes task-aware framework which helps to improve the learning ability of the existing multitask learning strategies. It addresses the issues faced during vanilla multitask learning, which includes forgetfulness during finetuning and the problems associated with having separate models for each task. The proposed approach helps to align better the existing multitask learning strategies with human learning. It achieves significant performance improvements with single model on a variety of tasks which is favouralle in low resource settings.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. Common voice: A massively- multilingual speech corpus, 2020.
Craig Atkinson, Brendan McCane, Lech Szymanski, and Anthony V. Robins. Pseudo-recursal: Solving the catastrophic forgetting problem deep neural networks. CoRR, abs/1802.03875, 2018. URL http:::/aaxxi..rrg///./
03875
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Computer Science Mathematics CoRR, 2015.
Rich Caruana. Multitask learning, Muchine learning, 28(1):41-75, 1997.
Brian Cheung, Alexander Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Superposition of many models into one. In H. Wallach, H. Larochelle, A. Beygelzimer, d Alché-Buc, -uc, E. Fox, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 10868- 10877. Curran Associates, Inc., 2019. URL 2019/fi1e/4c7a 17bb1676632222 www.773329999999 0tt.:/00220000000222222662222000
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, ICML '08, pp. 160-167, New York, NY, USA, 2008. Association for doi: 10. 10.1145/1390156.13900 13901777 URL attps
Computing Machinery. ISBN 9781605582054.
http:/orr/..144/133388 1390177
L. Deng, G. Hinton, and B. Kingsbury. New types of deep neural network learning for speech recognition and related applications: an overview. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 8599-8603, 2013.
Jacob Devlin, Ming-Wes Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiiv1880.. 04805, 2018.
Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. MuST- C: Multilingual Speech Translation Corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2012-2017, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 1..18653/v//11.. 1202. URL https
www.walll..oowwww....w....wwwwww www.wreb.gg... www.bbbhagaaaaaaaaaaaaaaaaa Noachhaoooy//1333 1202
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 489-500, Brussels, Belgium, doi: 10.18853/v1/D1.... 1045. URL htt.t::tt/s//tt.t..
2018. Association for Computational Linguistics.
ac wwt.:/ewb.ooogo..oooooooo..o.. hww.gggggaanlaoaaooeoooooooo cgy/ 045
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1 1116-1122 135. JMLR. org, 2017a
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leant leatnllenning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 1126-1135. JMLR. org, 2017b
R. Girshick. Fast r-enn. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2015.
Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor O.K. Li. Universal neural machine trans- lation for extremely low resource languages. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long Papers), pp. 344-354, New Orleans, Louisiana, June 2018a. Association for Computational Linguistics. doi: 10.18653/vI/N18- 1032. URL attps
7/www. acl web. hwt.://www....r///aaoooooo//////
10

---

Under review as a conference paper at ICLR 2021
Jiatao Gu, Yong Wang, Yun Chen, Victor O. K. Li, and Kyunghyun Cho. Meta-learning for low-resource neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3622-3631, Brussels, Belgium, October-November 2018b. Association for Computational Linguistics. doi: 10. 10.18633/1/D11--1 1399.. URL |
www.wacllxae. http:///w.eb..rrr/aaoooooo//////
Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. A joint many-task model: Growing a neural network for multiple NLP tasks. CoRR, abs/161 a 2016. URL
Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott. Bing Liu, James Glass, and Fuchun Peng. of dialogue response models, 2020.
Analyzing the forgetting problem in the
François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Estève. Ted-lium 3: Twice as much data and corpus repartition for experiments on speaker adapta- tion. Lecture Notes in Computer Science, pp- 198 208, 2018. ISSN 1611-3349, doi: 10.1007/ 978-3-319- 3-1-3-999999999--- 2.21.. URL /tt.t:/t/////. dx wwg. 1007000077770 3- 3-9-999------------------------- 21
S. Indurthi, H. Han, N. K. Lakumarapu, B. Lee, I. Chung, S. Kim, and C. Kim. End-end speech-to-text translation with modality agnostic meta-learning. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7904-7908, 2020.
Sathish Reddy Indurthi, Insoo Chung, and Sangha Kim. Look harder: A neural machine translation model with hard attention. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3037-3044 Florence, Italy, July 2019. Association for Computa- tional Linguistics. doi: 10. 10.18653/w1//19-1111 12900 URL httpp:::p...... / www. aclweb. wrtg/ottooooaoiooeoooooo anthologyy/
P19- 1290
Javier Iranzo-Sánchez. Joan Albert Silvestre-Cerdà, Javier Jorge, Nahuel Roselló, Adrià Giménez, Albert Sanchis, Jorge Civera, and Alfons Juan. Europarl-st: A multilingual corpus for speech translation of parliamentary debates. arXiv preprint ArXiv::911.031167 2019.
The IWSLT 2019 Evaluation Campaign, November 2019. IWSLT, Zenodo. doi: 0.5228/7enodd.. 3525578. URL htttps://////
Nikhil Kumar I Lakumarapu, Beomseok Lee, Sathish Reddy Indurthi, Hou Jeung Han, Mohd Abbas Zaidi, and Sangha Kim. End-to-end offline speech translation system for IWSLT 2020 using modality agnostic meta-lcarning, In Proceedings of the 17th International Conference on Spoken Language Translation, pp. 73-79, Online, July 2020. Association for Computational Linguis- tics. doi: 10.18653/112022..vsstt..
URL https: //t::///wwwwww acl web...e.b... www.w/wwwwww....weeeeoe.eeeeeeee wwt.:r///oooogg///....o.
iwslt-1. 1.77
Li and D. Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935-2944 Dec 2018 ISSN 1939-3539, doi: 10.1109/PAM...211277777
Zhizhong Li and Derek Hoiem. Learning without forgettingg IEEE Trans. Pattern Anal. Mach. Intell., 40 40(12):2935 2947, 2018. doi: 10.100/TPAM12211.2773368 URL httpp:://bi..rgg/// 1109/TPAM 2017.2773081
Pierre Lison, Jörg Tiedemann, Milen Kouylekov, et al. Open subtitles 2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In LREC 2018, Eleventh International Conference on Language Resources and Evaluation. European Language Resources Association (ELRA), 2019.
Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-yi Wang. Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In Proceedings of the 2015 Conference of the North Americun Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 912-921, Denver, Colorado, May-June 2015, Association for Computational Linguistics. doi: 1/111/////N11111 1092. URL https: /wwp:://www... lwwb orgg/anttooooggy/IS--OO0
Xiaodong Liu, Kevin Duh, Liyuan Liu, and Jianfeng Gao. Very deep transformers for neural machine translation. arXiv preprint arXiv:2008.07771 2020.

---

Under review as a conference paper at ICLR 2021
Yuchen Liu, Hao Xiong, Zhongjun He, Jiajun Zhang, Hua Wu, Haifeng Wang. and Chengqing Zong. End-to-end speech translation with knowledge distillation. CoRR, abs/1904.08075, 2019. URL
http:://axivv wttpe/iirrrreeeo abs /tt..///9...... 38075
Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention- based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp.
Lisbon, Portugal, September 2015. Association for Computational Linguisties. doi: 10. 10.18663/1/DD551166 URL httts:://wwwt wwwwwbboor.
antho Sotthology/11-111
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-5210, 2015.
Ethan Perez, Floriam Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with general conditioning layer. In AAAI, 2018.
Ngoc-Quan Pham, Thai-Son Nguyen, Jan Nichues, Markus Müller, and Alex Waibel. Very deep self-attention networks for end-to-end speech recognition. CoRR, abs/1904, 13377, 2019. URL
hitt:://arxiv org. httggg/bb/19999 http://aaax....o////////.0..11. 13377
Juan Pino, Qiantong Xu, Xutai Ma, Mohammad Javad Dousti, and Yun Tang. Self-training for
end-to-end speech translation. arXiv preprint arXiv:2006. 02490, 2020.
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186-191, Belgium, Brussels. October 2018.
Association for Computational Linguistics. URL https: /www acl ort.//thclonnggy/////
w18-6319
Tomasz Potapezyk, Pawel Przybysz, Marein Chochowski, and Artur Szumaczuk. Samsung's system for the iwslt 2019 end-to-end speech translation task. In 16th International Workshop on Spoken Language Translation (IWSLT). Zenodo, November 2019, doi: 10.528 //zzeoooo.35254999 URL
5281 httpss:/ http://ccc..rggg//// 5t1.1100.2211eeeeeeeeee zenddo/ 3525498
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint Arxiv: 1060, 2019.
Bharath Ramsundar, Steven Kearnes, Patrick Riley, Dale Webster, David Konerding, and Vijay Pande. Massively multitask networks for drug discovery, 2015.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1725-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 1118866/11/1161111 URL /ttp:p:::/wwwww... www.w/wwwwww.w.......o.
lanthology/P1611
Gjorgji Strezoski, Nanne van Noord, and Marcel Worring. Many task learning with task routing.
CoRR, abs/1903, 12111 2019. URL/attp:/// httt::://aaxiiv http:///wwwww..o//a///////////// aalaabs1990 .2117
Shubham Toshniwal, Tara N Sainath, Ron J Weiss, Bo Li, Pedro Moreno, Eugene Weinstein, and Kanishka Rao. Multilingual speech recognition with a single end-to-end model. In ICASSP, pp. 4904-4908. IEEE, 2018
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-60088 6008, 2017.
Lijun Wu, Yiren Wang, Yingce Xia, Fei Tian, Fei Gao, Tao Qin, Jianhuang Lai, and Tie- Re-Yan Liu. Depth growing for neural machine translation. arXiv preprint 19X.vv190..01968 2019.
Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-task learning. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision - ECCV 2014, pp. 94-108, Cham, 2014, Springer International Publishing. ISBN 978-3-319-10599-44
12

---

Under review as a conference paper at ICLR 2021
6 APPENDIX
6.1 DATASETS
6.1.1 DATA AUGMENTATION FOR SPEECH TRANSLATION
Table provides details about the datasets used for the multi- modality experiments. Since En-De ST task has relatively fewer training examples compared to ASR and MT tasks, augment the ST dataset with synthetic training examples. We generate the synthetic speech sequence and pair it with the synthetic German text sequences. obtained by using the top two beam search results of the two trained English-to-Genn to-German NMT models. For speech sequence, we use the Sox library to generate the speech signal using different values of speed, echo, and tempo parameters similar to (Potapczyk et al., 2019). The parameter values are uniformly sampled using these ranges tempo (0.85, 1.3), speed E (0.95, 1.05), echo_delay E (20, 200), and echo _decay E (0.05, 0.2). We also train two NMT models on EN-De language pair to generate synthetic German sequence. The first model based on Edunov et al. (018) and the second model (Indurthi al. 2019) is trained on WMT 18 En-De and OpenSubtiles datsets. We increase the size of the IWSLT 19 (filterd)) ST dataset to five times of the original size by augmenting 4x data four text sequences using the top two beam results from each EN-De NMT model and four speech signals using the Sox parameter ranges. For the Europarl-ST, we augment 2x examples to triple the size. The TED-LIUM dataset does not contain speech-to-text translation examples originally; hence, create 2x synthetic speech-to texttt translations using speech-to- te-text tramscripts. Finally, for the MuST-C dataset, we only create synthetic speech and pair it with the original translation increase the dataset size 4x. The Overall, created the synthetic training data of size approximately equal to four times of the original data for the ST task.
6.1.2 TASK IDENTIFICATION WITHOUT TASK INFORMATION
Under the multi-modality setting, conducted smaller scale experiments using only one dataset for each ST, ASR, and ST tasks. The details of the datasets used have been provided in Table 7 We trained single p40 GPU for 400k steps. The corresponding results have been reported in Table 6 All the results have been obtained without any finetuning. Even though our task-aware MTL model achieves significant performance improvement over vanilla MTL models, vanilla MTL models are also able to give decent performance on all tasks without any finetuning. An explanation for this is that we used MuST-C dataset for the En-De ST task and TEDLium v3 for the ASR task, which means that the source speech is coming from 2 different sources. However, if we use the same datasets for both the tasks(after data augmentation), the MTL models get confused and the ST, ASR outputs are mixed. The MTL models might be able to learn the task identities simply based on the source speech sequences, since these sequence are coming from different datasets for each task type-MuST-C for ST and TED-LIUM v3 for ASR. However, this does not mean that vanilla MTL models perform joint learning effectively. A human who can perform multiple tasks from the same input is aware of the task he has to perform beforehand. Similarly, it is unreasonable to expect different outputs (translation, transcription) from a model to the same type of input (English speech) without any explicit task information.
can observe that the
6.3.3 IMPLEMENTATION DETAILS
The detailed hyperparameters settings used for the single modality and multi modality experiments have been provided the Table g
13

---

Under review as a conference paper at ICLR 2021
Task
Corpus
# hours
# #Eammples
MT
Open Subtitles
N/A
22,512,639
MT
WMT 19
N/A
4,592,289
ASR
LibriSpeech
982
232.958
ASR
IWSLT 19 ST(fittree)
220
145,372
ASR
MuST-C
400
229,702
ASR
Common Voice
1469
232,958
ASR
TED-LIUM 3
452
286,263
ST
Europarl-ST
89
32,628
ST
IWSLT 19 ST(filtered)
220
145,372
ST
MuST-C
400
229,703
Table 5:
Number of original training examples in each dataset.
MT BLEU (T)
ASR(WER (4)
ST(BEEU (1)
No.
MTL Strategy
Test
Dev
Test
Dev
Test
1
Joint Learning
14.77
29.56
30.87
13.10
12.70
2
Meta Learning
14.74
28.58
29.92
13.89
13.67
This Work
Task Aware Meta Learning (with implicit TCN)
3
18.84
21.29
23.44
17.77
17.51
Table 6: Performance of models trained using different approaches on the ASR, MT and ST tasks using different datasets
Train
Dev Examples
Test Examples
Task
Corpus
hours
Examples
MT
WMT
N/A
4,592,289
3,000
3,003
ASR
TED-LIUM 3
452
286,263
1,469
591
MuST-C
400
229,703
1,423
2,641
ST
Synthetic
N/A
689, 103
N/A
N/A
Table The data statisties of ASR, MT and ST tasks used in experiments.
Hyperparameter
Single Modality
Multi Modality
ba batching batch size
dynamic
static
2048 (tokens) adam
16 (examples)
optimizer adam.betas
adam
(0.9,0.997)
(0.9,0.997)
1r_scheduler
inverse_sqrt
inverse_sqir
1r
2.0
2.0
n..warmppsstep
16000
16000
label_smoothing
0.1
0.1
dropout
0.1
0.1
decay rate hidden size encoder_layers
1.0
1.0
512
512
12
decoder.embed.dim
512
512
decoder.layers
12
num_heads
8
8
filter (ffn layers)
1024
1024
Table 8: Hyperparameter details for the experiments
14


Reviews:
[{"review_id": "cuPlztTJGq8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a new framework that computes the task-specific representations to modulate the model parameters during the multi-task learning (MTL). This framework uses a single model with shared representations for learning multiple tasks together. Also, explicit task information may not be always available, in such cases the proposed framework is useful. The proposed framework is evaluated on various datasets spanning multiple modalities, where the MTL model even achieves state-of-the-art results on some datasets. \n\nPros:\n1) The paper is well written and easy to follow. \n\n2) The proposed framework achieves state-of-the-art results on some tasks/datasets. \n\nCons: \n1) Seems like the implicit approach does perform more or less the same as the explicit approach. I believe the explicit approach is not something new and has been tried in various works. In that sense, the implicit approach is not exciting. I agree that the proposed framework would be able to learn task specific information without any explicit supervision, but this use case is not well explored in this paper. Would your approach scale to N (> 5-10) tasks? \n\n2) From Table-2, OHV performs better than explicit TCN. The paper argues that providing OHV text labels is not always possible for tasks involving non-text modalities such as speech and images. But, I think it is possible to learn simple joint space task embeddings presentations for tasks from multiple modalities. \n\n3)  From Table-3, the difference between explicit and implicit TCN methods is small. Statistical significance scores are not provided to know whether the improvements are real. I believe the explicit TCN approach is not something new so it would be better to know if implicit learning is strictly better than explicit.\n\nOverall: \nThe paper is well written and easy to follow with SOTA results. However, I am concerned about the novelty aspect of the proposed framework. It is not clear from the current experiments whether the proposed task-specific representation learning is better than a simple OHV or simple multimodal joint-space task embeddings. \n\nQuestions:\n1) Is implicit TCN better than explicit TCN in Table-3? Can you provide the statistical significance scores?\n\n2) Instead of OHV, can you learn a join space task embedding representation? This paper: https://arxiv.org/pdf/1611.04558.pdf used special tokens for multilingual translation, can you think of extending to the multi-modal setup with speech and image? If not possible, why?\n\n3)  Would your approach of learning task-specific representation learning scale to more task i.e., > 10 tasks?\n\nOther comments: \n1) It would have been easier to understand the implicit task information section if there is a corresponding pictorial presentation of the approach. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "V6XDgekk3TH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper combines Feature-wise Linear Modulation (FiLM) with a single/multi-modal transformer for a joint multi-task neural network and applies the proposed model to two tasks, which are single-modal machine translation and multi-modal (speech/text) machine translation, speech translation, and speech recognition.\n\nI have the following comments and reservations:\n+ The paper is well-written and easy to follow.\n- There is a plethora of closely related work that was not mentioned or discussed or compared to, e.g., Zhao et al.. 2018; Strezoski et al., 2019; Cheung et al., 2019, to name a few.\n- The paper lacks comparison to strong multi-task baselines and compares the proposed model only to standard multi-task, transfer learning, and meta learning approaches and single-task state-of-the-art approaches that don’t have access to the additional data from the other tasks.\n- The main title reads: “Learning without Forgetting”, yet there is no discussion or experiments with other methods that address the forgetting problem [Li & Hoiem, 2018].\n\nAs such, the work lacks sufficient novelty, is not well-placed within existing related prior work, and does not compare to adequate baselines.\n\nReferences:\n\nBrian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Superposition of many models into one. arXiv preprint arXiv:1902.05522, 2019.\n\nZhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell., 40(12):2935–2947, 2018. doi: 10.1109/TPAMI.2017.2773081.\n\nGjorgji Strezoski, Nanne van Noord, and Marcel Worring. Many task learning with task routing. arXiv preprint arXiv:1903.12117, 2019.\n\nXiangyun Zhao, Haoxiang Li, Xiaohui Shen, Xiaodan Liang, and Ying Wu. A modulation module for multi-task learning with applications in image retrieval. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 401–416, 2018.", "extra_scores": {}, "extra_text": {}}, {"review_id": "HTT2DYN3fRh", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an way to incorporate task information into multi-task learning (MTL). The hope is that more explicit knowledge of task information will improve MTL, especially in cases where the model seems to confuse tasks (as shown in the example of speech translation in Table 1). \n\nPros:\n- This is a timely contribution that addresses an important problem in MTL\n- The proposed method is reasonable and relatively easy to implement: it basically involves adding a network that characterizes the task, and using its outputs to modify the MTL model's parameters. \n\nCons:\n- While the results in Table 2 and Table 3 look promising, the presentation would benefit from more analyses experiments (see below). For example,  investigating whether the number of confused cases in Table 1 actually reduces, comparing whether the added parameters of the task-characterization network requires more data, exploring other ways to modulate the MTL model. Since the method is relatively straightforward, the paper would look stronger if it has more empirical analyses. \n\nAdditional questions and comments: \n- Some claims need to be explained or clarified. For example in introduction the authors write: \"However, providing task information is not straightforward for certain modalities such as speech and image. Also, providing explicit task information might not always be desirable, e.g., automatic multilingual speech recognition task.\" Why is task information not straightforward for speech? I understand it is harder to integrate the discrete label with the speech signal but it has been done before and the reason needs to be spelled out (i.e. don't assume the reader works in ASR.) \n- Can you list out the same examples in Table 1 for the proposed systems? Did they solve the problem of confused tasks? \n- Can you clarify how many new parameters are added for the different task-characterization networks, relative to the full network? It was also be interesting to fix to random weights in the task-characterization networks to see if the results are expected. \n- Please discuss other methods for modulating the weights of the model, besides the one used. I think it's not necessarily to do experiments in those, but mentioning other potential techniques for modulation would make the idea more comprehensive, perhaps suggesting it as future work. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "FkxFGrYEj7F", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a multi-task Transformer that does speech and machine translation within the same framework. It proposes task adaptive parameters by modulating model parameters to account for differences in tasks. The authors conduct experiments on speech and machine translation and show reasonable performance.\n\nHonestly, this paper is lacking because this is just too incremental over the T5 model (which can train over 20 tasks without any task specific parameters). It feels quite lacklustre comparing to the many exciting multi-task or transfer learning works out there. The notion of training speech and language adds a little to the paper but I think it is insufficient to feel excited about.\n\nThe results in the paper don't look terribly good either. I'm also perplexed why the performance is even outperformed by the baseline. I also don't think it is really a good idea to reduce T5 to \"OHV\" (one hot vector) and completely remove the pretraining completely. Namely, \"Unlike T5, we don’t initialize the models with the text embeddings from large pretrained language model (Devlin et al., 2018). Instead, we focus on establishing the importance of task information during MTL and having a single model for all the tasks.\" this is way too convenient and unconvincing. I think the authors should compare their model with T5 to have an apples to apples comparison. \n\nI have also concerns about the practicality of modulating parameters and how this would cope over N task when N is large. The number of tasks in this paper is very small (compared to T5) and this is not well ablated in the paper. What about other adapter based approaches like projected attention layers (https://arxiv.org/abs/1902.02671) or hypergrid (https://arxiv.org/abs/2007.05891)? I think it's worth to compare the modulated parameters with other forms of multi-task adaptations of these transformer models. \n\nOverall I recommend rejection. While this is a promising direction, this is poor execution. The proposed novelty of modulated parameters is also insufficient. Results are also not very exciting. \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach for improving MultiTaskLearning by providing a way of incorporating task specific information.

Pros:
1) All reviewers agreed that the paper is clearly written
2) Interesting to see a single model for AST, STS (speech-to-speech translation) and MT 

Cons:
1) The work is not adequately compared with related work (some important references are also missing) - The authors did perform some additional experiments with T5  and pointed out some drawbacks but this needs to be explored a bit more.
2) The answers about scalability are not very convincing and need more empirical results. 

Overall, none of the reviewers were very positive about the paper and felt that while this is a good first attempt, more work is needed to make it suitable for acceptance. 
Decision: Reject