Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modeling
Authors: Hongyu Gong
Yun Tang
Juan Pino
Xian Li
Venue: NeurIPS 2021 Poster (2021)
Keywords: Multi-head attention
parameter sharing
speech
translation

Abstract: Multi-head attention has each of the attention heads collect salient information from different parts of an input sequence, making it a powerful mechanism for sequence modeling. Multilingual and multi-domain learning are common scenarios for sequence modeling, where the key challenge is to maximize positive transfer and mitigate negative interference across languages and domains. In this paper, we find that non-selective attention sharing is sub-optimal for achieving good generalization across all languages and domains. We further propose attention sharing strategies to facilitate parameter sharing and specialization in multilingual and multi-domain sequence modeling. Our approach automatically learns shared and specialized attention heads for different languages and domains. Evaluated in various tasks including speech recognition, text-to-text and speech-to-text translation, the proposed attention sharing strategies consistently bring gains to sequence models built upon multi-head attention. For speech-to-text translation, our approach yields an average of $+2.0$ BLEU over $13$ language directions in multilingual setting and $+2.0$ BLEU over $3$ domains in multi-domain setting.

Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modeling
Hongyu Gong, Yun Tang, Juan Miguel Pino, Xian Li
Facebook Al Research
jyyyong..mtaagg..jaaaabbabbiaa jua.aaaaaaaaanaaaaaaaaaa (iaaraaaiiaaiaanalreeear com
Abstract
Multi-head attention has each of the attention heads collect salient information from different parts of an input sequence, making it powerful mechanism for sequence modeling. Multilingual and multi-domain learning are common scenarios for sequence modeling, where the key challenge is to maximize positive transfer and mitigate negative interference across languages and domains. In this paper, we find that non-selective attention sharing is sub-optimal for achieving good gener- alization across all languages and domains. We further propose attention sharing strategies to facilitate parameter sharing and specialization in multilingual and dolti- sequence modeling. Our approach automatically learns shared and specialized attention heads for different languages and domains. Evaluated in vari- ous tasks including speech recognition, text-to-text and speech-to-tess translation, the proposed attention sharing strategies consistently bring gains to sequence mod- els built upon multi-head attention. For speech-to-text translation, our approach yields average of +2.0 BLEU over 13 language directions in multilingual setting and +2.0 BLEU over domains in multi-domain setting.
1 Introduction
Recent progress on deep learning models, in particular multi-head attention, has brought significant gains sequence modeling tasks including speech recognition (Moritz et al., 2020), text-to-text translation (Vaswani et al. 2017). and speech-to-text translation (Vila et al. (0018] Gangi et al.) 2019). Attention mechanism allows model to focus informative parts of the inputs, and multi head attention computes attention over inputs by multiple heads independently. With each head attending to different information, multi-head attention potentially captures more complicated data patterns and extracts sophisticated knowledge.
Sequence modeling has attracted a lot of research interest in multilingual and multi-domain settings, where a model is trained on data in multiple language directions and data from different domains respectively. Key advantages of these settings are better data efficiency and the support of knowledge transfer among languages or domains. This is critical for resource- limited scenarios. For example, multilingual translation enhances the performance of low-resource languages via knowledge transfer from high- high-resource languages (Gu et al./ 2018 Inaguma et al. 2019b). Given the data scarcity in individual domains, common practice is to combine the data from various domains to augment the training set WWang et al. 2020d). Another appealing aspect of multilingual or multi-domain models is their low deployment and maintenance costs compared with numerous models trained for individual language pairs or domains.
Despite the positive knowledge transfer, negative interference has also been observed in multilingual (or multi-domain) training especially when languages (or domains) are dissimilar. Recent studies reveal from the optimization perspective that conflicting gradients in shared parameters is one cause of interference between languages (or domains) (Yu et al. 2020). A promising direction for interference
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

mitigation is to design better strategies of parameter sharing. In some previous works, sharing is based on the similarity between languages (or domains), which require expert knowledge or pre-computed relatedness (Wu et al. 2019). Recent studies also propose branches and components specific to languages (or domains) addition to shared modules (Bapna and Firat 2019 Guo et al.) 2020).
In this work, we bring the mitigation of language and domain interference under common umbrella, and tackle it by improving parameter sharing within multi-head attention. We propose strategies to select attention heads for different languages or domains, Instead of sharing everything across languages or domains, our model automatically learns to share heads among subset of languages or domains. It encourages positive transfer within the subset and preserves their specificity without interference from outside the subset. The major contributions of this work are summarized below:
1. We propose attention head selection to mitigate language or domain interference;
2. The parameter sharing strategies are lightweight and preserve inference efficiency;
3. We extensively evaluate attention sharing strategies on various sequence modeling tasks including speech recognition, texi-to-text and speech-to-text translation. Consistent gains are achieved across multiple benchmark datasets.
The paper is structured as follows. Section 2 discusses related works on sequence modeling in multilingual and multi-domain setting. In Section we introduce the proposed strategies of head selection in multi-head attention. Section 4 describes the empirical evaluation, followed by a discussion in Section 5 We conclude this paper in Section
2
Related Work
Multilingual learning. Multilingual modeling has the potential improve low-resource language performance through knowledge transfer from high-resource languages, and it draws great interest from researchers in speech recognition and translation (Pratap al., 2020 Heigold et al. 2013
Johnson et al. 2017; Dabre et al. 2020; Liu et al., 2020 Inaguma et al. 2019a Li et al., 2020). Although impressive progress has been made for low-resource or zero-shot tasks, also found the multilingual model has inferior performance on high-resource tasks due to multilingual interference. In order address this issue, some works focus on multilingual models with task-specific parameters. Different parameter sharing strategies are examined on Transformer (Sachan and Neubig 2018 1).. Attention dependent on target languages proposed to enhance multilingual translation (Blackwood et al. 2018). Treating multilingual modeling as an adaptation problem. (Bapna and Firat (2019) first build a universal multilingual model for all languages and then finetune newly added adapters for each language pair. Another thread of work is to increase the model capacity to compensate the performance loss in high-resource languages (Pratap et al.) 2020). Shazeer et al. (017)) propose mixture-of-ceee efpertsss and select RNN cells based on imput tokens. Lepikhin et al. (2020) integrate mixture of FFN experts in the GShard model, and later Fedus et al. (2021) propose Switch Transformer to route tokens to different FFN sub-layers. Different from previous works, propose strategies of attention sharing among languages in the level of attention heads for multilingual modeling.
Multi-domain learning. Similar to multilingual learning, multi dolti-domain learning (MDL) can ef- fectively utilize data from different domains but also suffers from interference due inter-domain heterogeneity (Saunders) 2021 Pham et al., (021). Previous works address this issue from two perspectives: optimization and model architecture. In the optimization aspect, attempts have been made to synchronize the learning speed of different tasks (Chen et al.) 2018). adjust the gradients of individual tasks to alleviate gradient conflicts (Yu et al. 2020) and apply regularization to achieve better generalization in different domains (Dakwale and Monz 2017 7 Khayrallah et al. 2018: Thomp son et al., 2019). terms of model architecture, domain-
labels (Kobus et al. (217). word embedding Zeng al. 8a), sub-networks (Wang et al. 2020d) are adopted to address the issue of domain divergence. The architecture can be specified during the general training with the mixed data from multiple domains (Wang al. 2020dd or during the finetuning in individual domains (Bapnal and Firat 2019). In this work, we deal with domain interference by leveraging domain-specific attention heads in multi-head attention.
Attention selection. Selective self-attention networks propose to apply masking to the inputs and pay more attention to content words (Geng et al. 2020). Liu et al. (2021) select text-related image regions

---

with attention in multi-modality translation. Compared to these methods, we conduct automatic attention head selection for different tasks and focus on mitigating task interference.
3 Model
In this section, we start with preliminaries of multi-head attention, and introduce our approach to attention interference mitigation. We put multilingual and multi-domain sequence modeling under the same umbrella in this study. For the simplicity of the following discussions, we refer to the Two settings multi-task modeling, where task is one language from the standard multi-head attention, our model provides more attention heads than those used in computation. Different subsets of heads are assigned each task so that partial attention sharing enables knowledge transfer and meanwhile mitigates interference. We introduce latent variables to modulate head selection, and propose strategies learn the head assignment different tasks.
domain. Different
3.1 Preliminary
Multi-head attention. As a core component of Transformer, multi-head attention parameterizes each head with key, query and value transformation matrices (Vaswani et al. 2017). The token representation is transformed into key, query and value vectors via these transformations. Each head assigns the attention of this token over the input sequence based on the matching between its query vector and key vectors of other tokens. The value vectors are weighted by the attention as the contextualized token representation. It is passed through linear projection the output of the attention head. Suppose that head h has output (*),, Multi-head attention with H heads yields an output for the given token, which is the concatenation of all head outputs.
x(2) + ... $ (6) 2 x (H)
where @ is the vector concatenation.
Interference. Maximal parameter sharing aims learn universal knowledge across languages (Wang et al. 2020e) and domains (Zeng et al. 2018b). To capture the task specificity, different languages or domains compete for model capacity, which is observed the interference in previous studies. The interference results in degraded performance in jointly trained models. However, few works look into the improvement of parameter sharing within multi-head attention. This study explores head selection strategies to mitigate the interference in multilingual and multi-domain models.
3.2 Latent Variable for Head Selection
First, we outline approach to learn more general-purpose multi-head attention in Transformer from the Bayesian neural network perspective. Suppose that the input sequence is x and the output sequence is y. For conditional sequence modeling tasks such as machine translation, the posterior of p(yl z) can be computed by marginalizing over the posterior of latent variable 2, which modulates parameters O in the standard Transformer architecture:
ply x, θ) = Ep(=)0) jpy xx,)] ff ptu x, 2))((()))) dz
(2)
Parameterization of 21. In this work, we define 24 as modulating the selection of attention heads (h) by task t. We have =, =- ja where z(a) is discrete latent variable from Bernoulli distribution indicating whether task selects attention head This modeling choice allows us to prune attention heads, which preserves computation efficiency well regularizes training.
Marginalizing over 24 intractable given numerous heads in neural models. Therefore, we use variational inference to derive an approximate solution. Suppose that (zt, Bt) is from task t. Specifi- cally, we learn an inference network qs (2t), which is paramterized with o, to approximate the true distribution p(z,) and optimize the evidence lower bound (ELBO) of p(yla)):
log xx)
Σ (Eng(e) It, [x, Po
(L(gg(2) ll p(z)))
(3)
where KI. is the KL-divergence between two distributions. In our work, we assume identical if probability of each head being selected. Therefore, we have p(z: = 1) = H', where H and H' are numbers of selected attention heads and all head candidates.
3

---

Training and interference. We use the Gumbel-Softmar reparameterization Jang et al. 2017) (h) from the posterior (s ))) ). makes the model end-to-end differentiable,
draw samples of 2
while learning discrete policies of head selection without resorting to policy gradients. We adopt
lightweight estimator of qo(=z))) (=) by directly learning the logit parameters {Ø:"):
exp((o, (1) ((1))/) <> 9(0,1)
( (1))
Σje(o.1; exp((o) () +())//)))))
where (O0, 1) is the Gumbel distribution, and T is temperature hyperparameter which increases the discreteness of samples when 7 - 0.
We will discuss different head selection strategies in Section 3.3 which make binary selection
decisions based on real-valued posterior (x(z))
3.3 Attention Selection Strategies
H heads
heads
H' heads
Group
Group
Group H
(a) Subset strategy.
Group strategy.
Figure Attention sharing strategies. The blue heads are selected while the grey heads are not.
Suppose that the output dimension of multi-head attention is d, and the dimension of each attention d We provide large pool of H' (H'' H) attention head candidates in every Transformer head is H.' layer, and H' is hyperparameter controlling the search space size of attention selection strategies. The model requires attention outputs have a consistent dimension d, so each task needs to select exact H heads among H' candidates. We introduce two strategies for the attention head selection: subset strategy and group strategy.
Subset strategy. The subset strategy is straightforward, and we compare the posterior (2) (*) h [1, H'] of all H' heads given task t. A subset of И heads with the highest posterior selected by the task, and there are CHH'' subset choices. The subset strategy described in Fig. a).
The binary mask sf indicates whether an attention head h is assigned to task t.
[1, 1, 0, otherwise,
h TopH((gg(2 (M)).
(h)
(5)
where TopH(.) returns the top H heads with the highest values.
The outputs of the selected heads are concatenated as the attention output. Note that the subset strategy does not consider the order of the attention heads. For example, when head 2 and 3 are selected, head 2 contributes the beginning part of attention output. With head and 2 selected, the output of head 2 goes to the last part of the attention output.
Group strategy. We further propose group strategy to preserve the order of attention heads during head selection. Different from the subset strategy, the group strategy first divides H' heads into Н groups. As is shown in Fig. bb), each group contains = ग candidates. Each task could choose one attention head from each group, and has access to H heads per layer. There are y. possible combinations of heads. The group strategy keeps the head order in that heads from group only contribute to g's corresponding dimensions in the attention output. The head with the highest

---

posterior in its group would be selected by given task L. We use the binary mask i9 to indicate the selection of head á in group g.
(h)
(h)
1,
= ((as((x( :'' E E)))
(6)
0,
otherwise.
The output of group g is:
x(9)
(h) (M)
kEg
The outputs of H groups are concatenated as the output of the attention module for task t.
With either subset or group strategy, the sequence model is trained to assign attention heads to different tasks to maximize the lower bound in inequality (3). The number of additional parameters ( introduced by our attention selection only O(T H' L.), where T the number of tasks, H' is the number of head candidates per layer, and J. is the number of layers. It is small compared with the total parameter size of the model, and head selection is thus lightweight and memory efficient. Moreover, the head selection is inherently pruning process. Regardless of the size of head candidates, only a fixed number of attention heads are involved in computation for a given task. Hence approach is also computationally efficient in model inference.
4 Experiments
We evaluate sequence models in multilingual and multi-domain settings respectively. Various applications are considered including multilingual machine translation (MT), automatic speech recognition (ASR) and speech translation (ST) in both multilingual and multi-domain settings. We integrate attention selection strategies into the self-attention" module. Our implementation is based on the FAIRSEQ toolkit (Ott et al., 2019 Wang et al. 2020b). We include widely used sequence models built on multi-head attention strong baselines below.
T.ransformer (Vaswani et al.) 2017). It is a state-of-the-art model in machine translation, which takes texts in source languages as inputs and generates texts in target languages.
2. S2T Transformer (Wang et al. 2020a). As variant of Transformer for speech processing, S2T Transformer is stack of a convolutional subsampler and Transformer, where the subsampler processes audio log mel-filter features and sends them to Transformer for text generation.
3. Adapter model (Bapna and Firat 2019). Adapters have been shown as an effective approach to language and domain adaptation. Task-specific layers are added on top of each Transformer layer in a well-trained (S2T) Transformer. A typical adapter layer consists of two feed-forward sub-layers.
4. Static strategy of head selection. A static strategy assigns each task with fixed subset of attention heads based on the task similarity (Standley et al.) [2200) [Sen al. 2019). In the multilingual setting, we group languages into linguistic families, and each family is assigned with an exclusive set of heads. As for the multi-domain setting, each domain has its own set of attention heads.
We report parameter size and decoding speed as memory and computation efficiency metrics respec- tively. Decoding speed is measured by the number of tokens decoded per second by one GPU.
4.1 Machine Translation
The task of machine translation is to translate text from one language to another. The metric BLEU measures the overlap between model translations and the ground truth (Papineni et al. 2002).
Dataset. We experiment with public multilingual machine translation datasets collected by WMT shared tasks as used by (Liu et al. 2020). The dataset consists of parallel sentences between English and other 14 languages Its data statistics are summarized in Appendix A. We evaluate models on
WW also tried head selection in the encoder-decoder attention but did not observe big improvements when using alone or in combination with self-attention head selection.
'The 14 languages are: Chinese (zh). Czech (cs), Estonian (et). Finnish (fi). French (fr). German (de). Gujarati (gu), Kazakh (kk), Latvian (lv), Lithuanian (It), Romanian (ro), Russian (ru), Spanish (es), Turkish (tr).
5

---

both one-to-mant (02M) and many-to-one (M20) translations, which are translation from English to 14 languages and from 14 languages English respectively.
Model configurations. The attention selection is based on the source language on the encoder side for M20 translation, and is based the target language in the decoder part for O2M translation. For both subset and group strategies, the number of attention head candidates is set as 8 in each layer (i.e., H))8), and only 4 heads (i.e., H=4) are selected for computation. We will discuss how the hyperparameter H' affects model performance in Section S For static strategy, we group languages into 5 linguistic familie Each family is assigned with attention heads which are shared by all languages in this family. Therefore, a total of 20 attention heads are used in the static strategy.
Other baselines have attention heads in each Transformer layer. We also include Transformer baseline with 8 attention heads, which measures the effect of increased attention heads. All models have 6 encoder layers and 6 decoder layers, the embedding dimension is 512 and the feed-forward dimension is 1024. They are trained with a batch size of 131k tokens and a learning rate of 0.0007. For O2M translation, attention selection models and Transformer are trained for 140k steps. As for M20 translation, they are trained for 100k steps. The adapter model is initialized with parameters from the trained Transformer, and tunes adapter layer parameters for 40k steps with Transformer parameters frozen. Adapter layers are added to Transformer for each language direction, and they have an intermediate dimension of 256. The dimension is selected so that the number of parameters (460M) in the adapter model is close to the parameter size (420M) in attention selection models.
Table I: BLEU (1) of Machine Translation on WMT Datasets (#Params: the number (Million) of model parameters. Speed: the number of tokens decoded per second. AVG-A: average BLEU over 14 directions, High and Low are average BLEU over high- and low-resource languages respectively.)
02M
M20
#Params
Speed
BLEU
#Params
Speed
BLEU
(M)
(tok/s)
AVG-A
High
Low
(M)
(tok/s)
AVG-A
High
Low
Transformer (M=4)
416
1140
20.1
25.7
16,0
416
1252
22.8
27.9
19.0
Transformer (H=8)
416
1089
20.6
26.7
16.1
416
1156
23.7
29.0
19.7
Adapter
460
1021
20.9
26.7
16.6
460
1117
23.3
28.7
19.3
Static strategy
434
1133
20.9
27.1
16.3
434
1250
23.6
29.0
19.5
Group strutegy
420
1137
21.0
27.1
16.4
420
1245
23.5
28.8
19.6
Subset strategy
420
1133
20.9
27.0
16.4
420
1250
23.3
28.7
19.4
Results. We group 14 language directions based on their amount of training data. We have 6 high-resource languages with more than 10M parallel sentences, and 8 low-resource languages with fewer than 10M sentence pairs. Table [ shows model performance on WMT datasets. More attention heads improve Transformer performance while hurting the decoding speed. In comparison with Transformer with heads, group strategy achieves +0.9 and +0.7 BLEU on average of 14 language directions in O2M and M20 translations respectively at comparable decoding speed. Transformer with 8 heads and adapter achieve BLEU scores comparable to both group and subset strategies but fall behind in inference efficiency. Static strategy demonstrates comparable performance group and subset strategies in all metrics except the parameter size.
4.2 Speech Recognition
The task of Automatic Speech Recognition (ASR) is transcribe source audios in the same language. Word error rate (WER) ASR evaluation metric, which measures the difference of model outputs from the ground truth (Klakow and Peters 2002). Lower WER indicates better recognition.
Model configuration. Models included in the experiments of speech recognition are S2T Trans- former, S2T Transformer with adapter layers, S2T Transformer with static, group and subset strategies. With static strategy, we group 8 languages into 2 families and each family has an exclusive set of 4 attention heads. Following the setup of (Salesky et al. 2021). all models have 1024 channels in the input convolutional subsampler, 12 encoder layers and 6 decoder layers with 4 attention heads per layer. Again we include the S2T Transformer baseline with 8 heads. The embedding dimension 256 and the feed-forward dimension is 2048. We set a batch size of 320k tokens and a learning
((I Indo- European family: es, de, es, fr. gu. lt, Iv. ro and ru: (2) Estonian family: et: (3) Uralic family: fi: (4) Turkic family: kk and tr, (5) Sino-Tibetan family: zh.
(1) Atro-Asiatic family: ar; (2) Indo- dur-pen family: de, el, es, fr. it, pt and ru.

---

rate of 0.0005 during training. Attention selection models and S2T Transformer are trained for 250 epochs. Adapter model is initialized with parameters of the trained S2T Transformer, and is then trained for another 200 epochs with only adapter layer parameters tuned. The intermediate dimension of adapter layers is again set as 256. To prevent over-fitting, we stop the model training when the model does not improve on the validation set for 10 epochs. To reduce the performance variance, we average checkpoints of the last 10 epochs, and use the averaged model for evaluation.
4.2.1 Multilingual Speech Recognition
Dataset. We use the multilingual TEDx (mTEDx) dataset for speech recognition (Salesky et al. 2021). It collects audio recordings from TEDx talks. Eight languages are covered including Arabic (ar), German (de), Greek (el), Spanish (es), French (fr), Italian (it), Portuguese (pt) and Russian (ru).
Table 2: WER (L) of Speech Recognition m TEDx Dataset
#Params
Speed (tok/s)
BLEU
(M)
AVG
ar
de
el
es
pt
ru
S2T Transformer (H=4)
31M
1118
49.0
109.3
72.3
43.3
23.9
27.8
28.6
31.0
55.3
S2T Transformer (H=8)
31M
1052
46.0
103.3
69.7
40.5
21.5
25.4
25.7
28.0
53.5
Adapter
50M
1016
41.1
93.4
57.2
33.0
21.4
25.3
24.3
27.2
46.7
Static strategy
35M
1108
49.4
110.1
72.9
43.8
24.1
27.9
29.4
31.4
55.8
Group strategy
35M
1107
40.0
94.2
59.8
33.5
18.2
22.0
21.9
24.6
45.5
Subset strategy
35M
1114
44.7
97.3
65.3
38.7
22.4
25.8
26.4
29.0
52.4
Results. S2T transformer share all parameters among languages. Attention selection models select attention heads based on the source and target languages. Adapter adds adapter layers based on the language directions. We report the ASR results in Table 2 It brings down 6. % WER for S2T Transformer to increase from adapter model reduces the WER by 16.1% and subset strategy by 8.8%, while static strategy does not change the performance too much. Group strategy achieves the largest drop of 18.4% in WER of S2T Transformer (H=4) with comparable decoding speed. Moreover, it outperforms S2T Transformer with heads in both speed and WER.
attention heads. Compared to S2T Transformer with 4 heads,
4.2.2 Multi-Domain Speech Recognition
Dataset. Besides mTEDx data, we include two other public datasets, CoVoST 2 and EuroParl, which are commonly used for speech translation. Since source audios are accompanied by transcripts, we could use their source audio-text data for speech recognition tasks. We investigate multi-domate modeling with these three datasets.
1. CoVoST 2 (Wang et al., 2020c). With Common Voice as the audio source, CoVoST 2 covers speech-to-text translations from 21 languages to English and from English to 15 languages.
2. EuroParl Strazoo-Sánchan et al. 20200. It provides paired audio-text instances from and into 6 European languages, which are compiled from the debates in European Parliament.
Table 3: WER (4) of Speech Recognition on mTEDx, CoVoST and EuroParl Dataset
#Params
Speed
BLEU
(M)
(tok/s)
m mTEDx
CoVoST
EuroParl
Separate S2T Transformers (H=4)
32M
1378
49.0
41.9
115.0
Sepurate S2T Transformers (H=8)
32M
1280
46.0
40.7
94.0
Joint S2T Transformer (H=4)
32M
1409
42.7
38.3
25.6
Joint S2T Transformer (H=8)
32M
1299
43.3
38.6
25.9
Adapter
39M
1274
41.7
37.0
24.0
Static strategy
39M
46.3
30.0
Group strategy
36M
1400
41.0
36.4
24.3
Subset strategy
36M
1403
41.8
37.0
25.0
Results. In the multi-domain setting, attention selection models assign different heads to each domain. The static selection strategy provides each domain with an exclusive subset of attention heads. Adapter model adds domain-specifie adapter layers to S2T Transformer. 3 reports WER of models trained for 400 epochs three domains: mTHDx, CoVoST 2 and EuroParl respectively. The S2T Transformer jointly trained on multi-domain data (in the row of "Joint S2T Transformer
7

---

(H=4)") reduces WER by 12.9%, 8.6% and 77.7% in three domains respectively, when compared with the models separately trained in individual domains (in the row of "Separate S2T Transformer (H=4)"). This demonstrates the benefits of positive transfer between domains.
The performance of speech recognition could be further improved by the mitigation of the domain interference. Attention selection with group and subset strategies outperform that with the static strat- egy. Both attention selection and adapter model achieve lower WER than the joint S2T Transformer with both and attention heads. Attention selection with group strategy has the lowest WER on both mTEDx and Co datasets, decreasing WER by 4. 4.0% and 5.0% respectively in comparison with joint S2T Transformer (H=4). The best system on EuroParl is adapter model, yielding WER reduction by 6.3% than the joint S2T Transformer (H=4).
4.3 Speech Translation
Now with focus the task of speech translation, again design experiments in multilingual and multi-domain settings. In the multilingual setup, we train translation models with samples in multiple languages to investigate language interference. As for the multi-domain setup, the models are trained with data from multiple domains so that we could look into the domain interference. BLEU serves as the evaluation metric of speech translation systems.
Baselines. We use the same baselines as in speech recognition. As recommended by (Salesky et al. 2021). initialize the encoders in speech translation with the encoders trained in the task of speech recognition in Section 4.2 for the purpose of improving training efficiency and performance.
Model configurations. All models are trained for up to 400 epochs. Other model configurations in ST are the same as those in ASR.
4.3.1 Multilingual Speech Translation
To explore language interference, we perform experiments on multilingual speech translation.
Dataset. We again use mTEDx dataset for multilingual speech translation. Besides speech recognition data, mTEDx also collects speech translation data from TEDx talks. Its test set covers 13 language directions. The training data is provided in 10 of these directions, so there are 3 zero-shot directions.
Table BLEU (f) of Speech Translation on mTEDx (AVG-A: average over all directions, AVG-T:
average of 10 training directions, and AVG-Z: average of 3 zero-shot directions)
#Params
Speed
BLEU
(M)
(tok/s)
AVG-A
AVG-T
AVG-Z
S2T Transformer (H=4)
31M
1038
13.2
14.6
8.5
S2T Transformer (H=8)
31M
968
13.7
15.1
9.0
Adapter
55M
826
14.8
Statie strategy
33M
1030
13.2
14.5
Group strategy
35M
1024
15.2
16.7
Subset strategy
35M
1033
13.3
14.7
8.5
Results. Here the static strategy of head selection groups source languages into two families as in multilingual ASR task, and all target languages fall into the same Indo-European family. Table 4 summarizes the multilingual speech translation results on mTEDx. S2T Transformer has +0.4 BLEU with heads increased to Since adapter model brings language-specifie layers, cannot deal with zero-shot translations. Group strategy, subset strategy and adapter model bring improvements over S2T Transformer (H=4) which are jointly trained 13 language directions. multiple languages interfere within S2T Transformer whose parameters are shared by all languages. Attention selection with group strategy achieves the best translation performance. In comparison with S2T Transformer (H=4), group strategy achieves an average of +2.1 and +1.9 BLEU in training and zero-shot directions respectively. It leads +2.0 BLEU on average of all directions.
suggests that
4.3.2 Multi-Domain Speech Translation
In this experiment, we investigate interference across domains the task of speech translation, and evaluate the effectiveness of different models in multi-domain training. The attention selection now is

---

based on the data domain instead of languages, i.e., samples in different domains would choose their own attention heads. Similarly for adapter model, its adapter layers are domain-specific in this setup.
We again use CoVoST 2 and EuroParl as additional domains. We focus on the 13 language directions in mTTEDx test set, and use the subset of CoVoST 2 and EuroParl corpora in the same directions. Co 2 has 5 common directions and EuroParl has 11 common directions as mTEDx. Details about these datasets are included in Appendix A.1
Table 5: BLEU (1) of Speech Translation on mTEDx, Co 2 and EuroParl Dataset
#Params
Speed (tok/s)
TEDx
CoVoST
EuroPari
(M)
AVG-A
AVG-T
AVG-Z
AVG
AVG
Separate S2T Transformer (H=4)
32M
1105
13.2
14.6
8.5
17.6
19.1
Separate S2T Transformer (H=8)
32M
957
13.7
15.1
9.0
19.0
19.0
Joint S2T Transformer (H=4)
32M
1148
13.7
13.6
13.9
17.3
19.0
Joint S2T Transformer (H=8)
32M
1062
13.7
13.6
13.9
17.4
19.1
Adapter
39M
940
14.0
14.3
13.2
17.9
20.0
Static strategy
39M
1138
11.5
12.0
9.7
14.5
16.9
Group strategy
36M
1141
15.6
15.9
14.8
19.6
20.8
Suhset strategy
36M
1145
13.8
14.3
13.1
17.9
19.2
Results. Table 5 shows the average BLEU of speech translation in mTEDx, CoVoST 2 and EuroParl. We report results in rows of "Joint S2T Transformer" when S2T Transformers are trained with the mixture of three datasets. The results included in rows "Separate S2T Transformer" when S2T Transformers are trained on each dataset independently. Zero-shot translations in mTEEx benefit a lot from additional data of CoVoST and EuroParl, the joint S2T Transformer (H=4) shows an average of +5.4 BLEU over separate S2T Transformer (H=4). However, there is drop of 1.0 BLEU in its training directions, brought by the interference from CoVoST 2 and EuroParl domains.
Again we observe that static strategy falls behind group and subset strategies. Attention selection with learned strategies and adapter model bring gains to the joint model in individual domains. Compared with the joint S2T Transformer (H=4), adapter model improves m TEDx translation by 0.3 BLEU, CoVoST 2 translation by 0.6 BLEU and EuroParl by 1.0 BLEU average. The attention selection with group strategy outperforms all other models. Its average BLEU gain over adapter model 1.6 BLEU in mTEDx, 1.7 BLEU in CoVoST 2 and 0.8 in EuroParl.
Discussion
Hyperparameterr H' The attention selection mod- els set hyperparameter H' as the total number of multi-head attention,
attention head candidates which controls the search space of attention shar- ing strategies. We now explore how the performance varies with H' for group and subset strategies.
50
Group strategy
Subset strategy
48
Word error rate
46
Evaluated on the task of multilingual speech recog- nition, models have the same hyperparameters as those in multilingual ASR experiments except for H'. Attention selection models are configured with H' = 4, 8, 12, 16 respectively, and Figure shows the change of WER with H'.
44
42
40
When H' 4, there no attention selection and all attention heads are shared by different languages. We observe large drop of error rate as H' increases from 4 to 8. For the subset strategy, WER keeps decreasing when the number of head candidates grows from 4 to 16. As for group strategy, H' 8 is the optimal hyperparameter on the ASR task. As we continue
Total number of attention heads
Figure 2: WER of speech recognition on mT- EDx with different HI'
5 $(es, fr, it, pt, ru]-en "es-(en, ft. it, pt). fr- fr-(en, es. pt). it-(en, es), pt-len, es)

---

increasing H' to 12 and 16, the error rate increases a bit. The performances of subset and group strategies are close when H' 16.
The search space of group strategy is a strict subset of the space of subset strategy. However, we observe that group strategy shows comparable better performance than subset strategy across tasks, including MT, ASR and ST. One possible explanation is that group strategy keeps the head order information while subset strategy does not. With a larger pool of head candidates, there is less sharing among tasks. The performance of the group strategy degrades bit due to less positive transfer dependent on attention sharing. As for the subset strategy, better head assignments are learned in the enlarged search space.
0
8
ar
de
pt
ar
de
pt
ru
(a) Encoder with group strategy.
(b) Decoder with group strategy.
Figure 3: Heatmap visualize the sharing between languages in multilingual ASR (The darker language pair is, the more attention heads they share.)
Attention Sharing among Languages. now analyze the attention sharing pattern among lan- guages. Take the multilingual model on mTEDx speech recognition selection learned with group strategy. We count the number heads shared by each language pair in the model, and visualize it with a heatmap in Fig. [] where the darkness reflects the amount of sharing. The diagonal cells the heatmap corresponds to the number of attention heads used by each language, i.e., the total number of attention heads in all layers.
example, whose head
For European languages including Spanish (es), French (fr), Italian (it) and Portuguese (pt), their shared attention heads are fewer in decoder than in encoder. This seems contradicted with previous findings that parameter sharing is beneficial for languages with high linguistic proximity. We note that they are high-resource languages in mTEDx corpus, which is also justified by their relatively lower WER. Their data sufficient to learn good speech recognition, and sharing parameters with other languages hurt the preservation of the language specificity. This explains why the high-resource European languages do not share too many heads in the learned group strategy.
Another pattern we observe from Fig. is that low-resource languages tend to share more attention heads with high-resource languages. For example, Arabie (ar) and Russian (ru) have relatively more sharing with Italian (it) than other languages. Low-resource languages benefit from the knowledge transfer from high-resource ones. Due to page limit, we include more discussions in Appendix A.3
6 Conclusion
Research efforts in multilingual and multi-domain modeling have been driven by the increasing need to improve data efficiency and model performance. In this work, propose head selection strategies to allow attention heads to shared or specialized for different languages or domains. effectively mitigates interference within multi-head attention which is a part of strong sequence models, and demonstrates good empirical gains in various text generation tasks.
This work has several limitations left for future research. We did not explore head selection based on both language and data domain. We did not analyze model fairness and robustness. technology used for text generation, the model might have systemic bias produce inappropriate outputs.
10

---

References
Ankur Bapna and Orhan Firat. Simple, scalable adaptation for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLPP), pages 1538 1548, 2019.
Graeme W. Blackwood, Miguel Ballesteros, and Todd Ward. Multilingual neural machine translation with task-specific attention. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pages 31 12-3122. Association for Computational Linguistics, 2018
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stuckholmsmässun, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 793-802. PMLR, 2018.
Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan. A survey of multilingual neural machine translation. ACM Computing Surveys (CSUR), 53:1 38, 2020. URL https://dl..... acm. org/ doi/pdf 1145/3406095
Praveen Dakwale and Christof Monz. Finetuning for neural machine translation with limited degradation across in-and out-of-domain data. Proceedings of the XVI Machine Translation Summit, 117, 2017.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, arXiv preprint arXiv:2101.3961, 2021.
Mattia Antonino Di Gangi, Matteo Negri, and Mareo Turchi. Adapting transformer to end-to-end spoken language translation. In Gernot Kubin and Zdravko Kacic, editors, Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pages 1133 1137. ISCA, 2019.
Xinwei Geng, Longyue Wang, Xing Wang, Bing Qin, Ting Liu, and Zhaopeng Tu. How does selective mechanism improve self-attentioo networks? In Dan Jurafsky, Joyce Chai, Natalic Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 9986-2995 Association for Computational Linguistics, 2020.
Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor OK Li. Universal neural machine translation for extremely low resource languages. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume
1 (Long Papers), pages 344-354, 2018.
Pengsheng Guo, Chen- Yu Lee, and Daniel Ulbricht. Learning to branch for multi-task learning. In International Conference Machine Learning, pages 3854-3863, PMLR, 2020.
Georg Heigold, Vincent Vanhoucke, Andrew W. Senior, Patrick Nguyen, Marc' Aurelio Ranzato, Matthieu Devin, and Jeffrey Dean. Multilingual acoustic models using distributed deep neural networks. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 8619-8623, 2013.
H. Inaguma, Kevin Duh, Tatsuya Kawahara, and Shinji Watanabe. Multilingual end-to-end speech translation. 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 570-577, 2019a.
Hirofumi Inaguma, Kevin Duh, Tatsuya Kawahara, and Shinji Watanabe. Multilingual end-to-end speech translation. In IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2019, Singapore, December 14-18, 2019, pages 570-577. IEEE, 2019b.
Javier Iranzo-Sánchez, Joan Albert Silvestre-Cerda, Javier Jorge, Nahuel Roselló, Adrià Giménez, Albert Sanchis, Jorge Civera, and Alfons Juan. Europarl-st: A multilingual corpus for speech translation of parliamentary debates. ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8229-8233 IEEE, 2020.
11

---

Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmat In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
M. Johnson, M. Schuster, Quoe V. Le, M. Krikun, Y. Wu, Z. Chen, Nikhil Thorat, Fernanda B. Viégas, M. Wattenberg, G. Corrado, Macduff Hughes, and Dean. Google's 's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguisticy, 5:339-351, 2017.
Huda Khayrallah, Brian Thompson, Kevin Duh, and Philipp Koehn. Regularized training objective for continued training for domain adaptation in neural machine translation. In Alexandra Birch, Andrew M. Finch, Minh-Thae Maah-Thag Luong, Graham Neubig, and Yusuke Oda, editors, Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, NMT@ACL 2018, Melbourne, Australia, July 20, 2018, pages 36-44. Association for Computational Linguisties, 2018.
Dietrich Klakow and Jochen Peters. Testing the correlation of word error rate and perplexity. Speech Communication,
28, 2002.
Catherine Kobus. Josep Maria Crego. and Jean Senellart. Domain control for neural machine translation. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, Varna, Bulgaria, September 2 8, 2017, pages 372-378. INCOMA Ltd., 2017.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. CoRR, abs/2006. 16668, 2020.
X. Li, Changhan Wang, Y. Tang, C. Tran, Yuqing Tang, Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. Multilingual speech translation with efficient finetuning of pretrained models. arXiv: Computation and Language, 2020.
Pengbo Liu, Hailong Cao,, and Tiejun Zhao. Gumbel-attention for multi- modal machine translation. CoRR, 2as/2103.08866/ 2021. URL httppp://aaiii..orgg//////////// /tabs///100. 08862
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742, 2020.
Niko Moritz, Takaaki Hori, and Jonathan Le. Streaming automatic speech recognition with the transformer model. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6074-6078. IEEE, 2020.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Waleed Ammar, Annie Louis, and Nasrin Mostafazadeh, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Demonstrations, pages 48-53. Association for Computational Linguistics, 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-33 311-318. 2002.
MinhQuang Pham, Josep Maria Crego, and François Yvon. Revisiting multi-domain machine translation. Transactions of the Association for Computational Linguistics, 9:17-35, 2021.
Vineel Pratap, Anuroop Sriram, Paden Tomasello, Awni Y. Hannun, Vitaliy Liptchinsky, Gabriel Synnaeve, and Ronan Collobert. Massively multilingual asr: 50 languages, model, 1 billion parameters. In INTERSPEECH, 2020.
Devendra Singh Sachan and Graham Neubig. Parameter sharing methods for multilingual self- attentional translation models. In Proceedings of the Third Conference on Machine Translation: Research Papers, WMT 2018, Belgium, Brussels, October 31 November 1, 2018 pages 261-271. Association for Computational Linguistics, 2018
12

---

Elizabeth Salesky, Matthew Wiesner, Jacob Bremerman, Roldano Cattoni, Matteo Negri, Marco Turchi, Douglas W. Oard, and Matt Post. The multilingual tedx corpus for speech recognition and translation. CoRR, abs/2102.01757/ 2021. URL.
Danielle Saunders. Domain adaptation and multi- adaptation for neural machine translation: A survey. CoRR, abs/2104.06951, 2021. URL
Sukanta Sen, Kamal Kumar Gupta, Asif Ekbal, and Pushpak Bhattacharyya. Multilingual unsuper- vised nmt using shared encoder and language-specific decoders. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3083-3089 2019.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gate mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
T. Standley, A. Zamir, Dawn Chen, L. Guibas, Jitendra Malik, and S. Savarese. Which tasks should be learned together in multi-task learning? In ICML, 2020.
Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh, and Philipp Koehn. Overcoming catastrophic forgetting during domain adaptation of neural machine translation. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 2062-2068. Association for Computational Linguistics, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998 08886608 2017.
Laura Cross Vila, Carlos Escolano, José A. R. Fonollosa, and Marta R. Costa-jussà. End-to-end speech translation with the transformer. In Jordi Luque, Antonio Bonafonte, Francese Alías Pujol, and António Teixeira, editors, Fourth International Conference, IberSPEECH 2018, Barcelona, Spain, 21-23 November 2018, Proceedings, pages 60-63. 63. ISCA, 2018
Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. Fairseq s2t: Fast speech-to-text modeling with fairseq. In Proceedings of the 1st Conference of the Asia- Pucific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: System Demonstrations, pages 33-39, 2020a.
Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. fairseq s2t: Fast speech-to-text modeling with fairseq. In Proceedings of the 2020 Conference of the Asian Chapter of the Association for Computational Linguistics (AACL): System Demonstrations, 2020b.
Changhan Wang, Anne Wu, and Juan Pino. Covost 2: A massively multilingual speech-to-text trans- lation corpus. CoRR, abs/2007. 10310, 2020c. URL httpp::///s:riiiiie www/obsg////.oooooooo...oo..eea/ htwpaab//.0/2////
Yong Wang, Longyue Wang, Shuming Shi, Victor OK Li, and Zhaopeng Tu. Go from the general to the particular: Multi-domain translation with domain transformation networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages
2020d.
Zirui Wang, Zachary C Lipton, and Yulia Tsvetkov. On negative interference in multilingual language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4438 4450, 2020e.
Sen Wu, Hongyang R Zhang, and Christopher Ré. Understanding and improving information transfer in multi-task learning. In International Conference on Learning Representations, 2019.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33, 2020.
13

---

Jiali Zeng, Jinsong Su, Huating Wen, Yang Liu, Jun Xie, Yongjing Yin, and Jianqiang Zhao. Multi- domain neural machine translation with word-level domain context discrimination. Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31- November 4, 2018, pages 447-457. Association for Computational Linguisties, 201 Sa.
Jiali Zeng, Jinsong Su, Huating Wen, Yang Liu, Jun Xie, Yonging Yin, and Jianqiang Zhao. Multi- domain neural machine translation with word-level domain context discrimination. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 447-457, 2018b.
14


Reviews:
[{"review_id": "piKy9G9mtg", "reviewer": "Reviewer_NgLe", "summary": "A neat paper with an interesting idea for multi-task/multi-domain sequence learning.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a new approach for multi-task/multi-domain sequence learning where domain/task specific knowledge is represented via different head sets. Empirical results on multiple benchmarks of machine translation, speech recognition, and speech translation demonstrate the effectiveness of the proposed method. \n\nThe paper is very well written and the proposed method, at least to me, contains something new. Empirical studies are fairly rich with a few interesting insights disclosed. \n\nSome of my concerns:\n(1) If the method also works on tasks other than sequence learning, for example, classification and matching. \n(2) On machine translation, the method is more useful on high resource languages, any reasons? If we still expect good performance on low resource languages, what can we do?\n(3) Improvement over Adapter on MT seems not that significant, though the authors explain that their models are advantageous on efficiency. It would be great if the authors could show some quantitative results on efficiency comparison. That would be more convincing. \n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I expect to see more discussions on limitations. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "aw8GTDU9jV", "reviewer": "Reviewer_WC28", "summary": "The paper proposes a model of latent selection of attention heads to improve the performance of sequence modelling in multilingual and multi-domain NMT, speech recognition, and speech-to-text translation.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "_Originality_: The methods proposed in the paper are novel and well distinguished from related work.\n\n_Quality_: The method proposed is interesting. Experiments were nicely conducted on multiple tasks to show the better performance of the model on different settings when compared to the baselines. However, all the experiments lack error bars over different random seeds (which is different from averaging checkpoints of the same run, or showing the average of the results over all languages). Since oftentimes the difference between the model proposed and the baseline is small, it would be important to show these error bars to better understand how much the proposed method improves upon the baselines. Furthermore, the method itself could be better explained and I wish more space was given to §3. Particularly, what is the Variational NMT model used? Is there a learnable prior?\n\n_Clarity_: The paper is well-written and well-structured, albeit explanations of the methods could be more detailed as mentioned above.\n\n_Significance_: The ideas and results shown in this work are relevant to researchers and practioners that study multilingual and multi-domain models. The paper is extensive in the number of tasks in which they test their model and baselines in and that can provide insight into future ideas.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2h"}, "extra_text": {}}, {"review_id": "OkVKgZkXGNv", "reviewer": "Reviewer_zLUP", "summary": "This paper aims to mitigate the problem of negative interference when learning models across different languages and domains. In particular, it proposes an approach that learns to share or specialize attention heads based on two different strategies, namely subset, and group selection.  The evaluation of multi-lingual/domain settings shows that this approach outperforms standard transformers and adapters without increasing the number of parameters. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a simple approach that is effective in multilingual and multidomain settings.  These are challenging settings due to the problem of negative interference which has been observed in past studies; mitigating this issue has a big potential impact and should be of interest to the community. \n\nLearning to share or specialize components of neural networks for multilingual tasks has been proposed in the past [1]. The proposed idea is somewhat less general since its novelty lies specifically in the latent variable model with the two selection strategies for attention heads in transformers. Another limitation of the idea is that the selection model is independent of the context since the estimator used for the selection process learns directly the logit parameters. This makes it potentially less flexible and more data-dependent/specific. \n\nThe technical exposition was mostly clear and easy to follow but the positioning with respect to prior was missing emphasis on studies that learn to share components for multilingual and multidomain settings.  This is also reflected in the evaluation which lacks comparison with such methods. Other than that the evaluation provides evidence about the superiority of the approach against standard transformers and adapter baselines.  One potential issue is that the reporting is not thorough and some details for judging the results are missing e.g. showing the number of parameters for each model, detailed scores, and standard deviation when averaging.  One interesting result is that the proposed approach outperforms adapters without requiring their additional parameters and the finetuning step. This makes the adaptation process easier. \n\nQuestions/comments:\n- In line 97, it is unclear what \"our model provides more attention heads\" means. It looks like that the model selects a subset of the heads for each language or domain. Could the authors provide some clarification?\n\n- In the subset strategy, if the model assigns non-zero values to the logits of less than H heads Eq. 5 could lead to a smaller dimension than expecting after concatenation. How do the authors deal with this potential issue?\n \n- For the multilingual experiment in Table 1, could the authors report the scores for the individual languages? The differences look small and it is unclear if the improvement comes from a single language or if it is present in all examined pairs. \n\n- I would suggest a more clear reporting of the number of parameters that are trainable for each model and task (including the baseline transformer).  Currently, only the parameters of adapters and the proposed model are reported for the MT task (lines 214-215). \n\n- This notation used for the logit parameters \\phi^{(h)}_t is a bit counterintuitive. The function takes as argument 0 or 1 which is strange for indicating that the logits are specific to the outcome 0 or 1 of the target variable. Also, there is a subscript missing from the denominator in Eq. 4: \\phi^{(h)} should be \\phi^{(h)}_t. \n\n[1] Share or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation, ICLR 2021", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors refer to some of the limitations of the proposed approach by exploring how the hyperparameter affects performance and mentioning aspects that were not explored in this study. I would suggest that the authors discuss the lack of conditioning on the context representations for their selection variables and previous work that has focused on learning to share components for multilingual settings to better position the work with respect to prior work. The discussion regarding potential negative societal impact is generic and could have been a bit more elaborate.   ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "ETO8qNdLVLK", "reviewer": "Reviewer_HGp8", "summary": "Multi-head attention is an essential component for popular Transformer models. This paper proposes to learn shared and specialized attention heads for different languages and domains. The authors formulate attention selection as latent variables and adopt Gumbel softmax to select attention heads.  Experiments on text-to-text and speech-to-text translation verify the effectiveness of the proposed approach. ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The key idea is simple and intuitive. The writing is clear, and experiment settings are described in detail. \n\nThe main formulation in Section 3.2 is not very rigorous, even wrong. In Equation 2, the authors say that \"can be computed by marginalizing over the posterior of latent variable z\". According to line 122, $\\Theta$ is parameters is Transformer.  It is important to note that p(z|\\Theta)  is not posterior distribution. p(z|x,y) is the posterior distribution.  Also, p(z|\\Theta) is usually written in p_{\\theta}(z).   In lines 128-130, the authors say that \"Specifically, we learn an inference network q(z) to approximate the true distribution p(z) and optimize the evidence lower bound (ELBO)\". In the standard variational bound solution, q(z) here should be the posterior distribution q(z|x,y). However, the authors do not include $y$ in the implementation to compute $q(z|x,y)$. \n\nThe proposed approach achieves promising results on speech recognition.  However, on multilingual Machine Translation, the improvements achieved by the proposed approach are marginal.  Table 1 shows that the proposed approach only outperforms Adapter with 0.1 and 0.2 average  BLEU improvements. \n\nMany related studies are missing. The proposed approach is highly related to dynamic networks and conditional computation.  Dynamic networks are networks that only use a part of architecture for inference. One representative model is Switch-Transformer. The authors should include these studies in the new version and add more comparisons on experiments. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It would be better to carefully polish the formulation and add more comparisons with dynamic networks, like Sparse-MoE. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes multi-head attention in multilingual and multi-domain sequence modeling tasks. The paper claims that non-selective attention sharing is sub-optimal for achieving good generalization across all languages and domains, and proposes new attention sharing strategies for different languages and domains to mitigate interference. Experiments are reported in speech recognition, text-to-text and speech-to-text translation, with consistent gains.

Most reviewers agree that the proposed method is novel and interesting and the paper is clear and well written, though some experiments need to be clarified (addressed in the rebuttal). The main weaknesses pointed out by the reviewers is a non-standard conditional VAE formulation, which the authors clarified I their rebuttal, and the lack of discussion with and positioning with respect to studies that learn to share components for multilingual and multidomain settings (including adaptors). In a future iteration, I urge the authors to take into account the detailed comments made by the reviewers when preparing a new version of their paper.
Decision: Accept (Poster)