Rethinking and Improving Multi-task Learning for End-to-end Speech Translation
Authors: Yuhao Zhang
Chen Xu
Bei Li
Hao Chen
Tong Xiao
Chunliang Zhang
JingBo Zhu
Venue: EMNLP 2023 Main (2023)
Keywords: Speech translation
multi-modal translation
machine translation

Abstract: Significant improvements in end-to-end speech translation (ST) have been achieved through the application of multi-task learning. However, the extent to which auxiliary tasks are highly consistent with the ST task, and how much this approach truly helps, have not been thoroughly studied. In this paper, we investigate the consistency between different tasks, considering different times and modules. We find that the textual encoder primarily facilitates cross-modal conversion, but the presence of noise in speech impedes the consistency between text and speech representations. Furthermore, we propose an improved multi-task learning (IMTL) approach for the ST task, which bridges the modal gap by mitigating the difference in length and representation. We conduct experiments on the MuST-C dataset. The results demonstrate that our method attains state-of-the-art results. Moreover, when additional data is used, we achieve the new SOTA result on MuST-C English to Spanish task with 20.8\% of the training time required by the current SOTA method.

Rethinking and Improving Multi-task Learning for End-to-end Speech Translation
Yuhao Zhang', Chen Xu', Bei Li", Hao Chen', Tong Xiao 1.2- Chunliang Zhang and Jingho Zhu 1.2
'School of Computer Science and Engineering, Northeastern University, Shenyang, China TriuTrans Research, Shenyang, China yoohao. zhang@gmail com, { (xuchenneu, wwbei_ @out com (xiaotong, zhangel, zhujingbo @maee.mml............ee.eeeeeele neu. edu. cn
Abstract
ASR/MT7
Auxiliary loss
ST loss
Significant improvements in end-to-end speech translation (ST) have been achieved through the application of multi-task learning. How- ever, the extent to which auxiliary tasks are highly consistent with the ST task, and how much this approach truly helps, have not been thoroughly studied. In this paper, we investi- gate the consistency between different tasks, considering different times and modules. We find the textual encoder primarily facili- tates cross modsl conversion, but the presence of noise in speech impedes the consistency be- tween text speech representations. Further- more, propose improved multi-task learn- ing (IMTL) approach for the ST task, which bridges the modal gap by mitigating the differ- ence in length and representation. We conduct experiments on the MuST-C dataset. The re- sults demonstrate that method attains state- of-the-art results. Moreover, when additional data is used, we achieve new SOTA result MuST-C English Spanish task with 20.8% of the training time required by the current SOTA method.
+
Acoustic encoder
Textual encoder
Decoder
T Speech
1 Target text
SSuuree text
Figure I: Multi-task training architecture for Speech translation. The dashed line part will be removed in fine-tune stage.
training (Tang et al., 2021: Ye al., 2021; Dong et al., 2021b; Han al., 2021). This technique as- sures good convergence of the models and empha- sizes the importance of the auxiliary loss, offering brand-new perspective for further advancements in E2E ST (Tang et al., 2022).
However, further exploration is necessary to de- termine how and to what extent these auxiliary tasks aid the final ST model. Notably, not all aux- iliary tasks in the fine-tuming stage are beneficial. This inconsistency arises because MTL is typically viewed multi-objective optimization problem, often resulting in training trade-off's when objec- tives conflict (Désidéri, 2012). The ideal MTL outcome to achieve Pareto optimality (Sener and Koltun, 2018), indicating solutions are superior to any alternatives. Since MTL does not ensure op- timal performance for the ST task, fine-tuning is crucial to overcome this shortcoming. Some stud- ies even underscore critical conflict between the ST task and ASR and MT tasks (Xu et al., 2021; Tang et al., 2022), necessitating a fine-tuning strat- egy. So answering these questions crucial designing an optimal MTL strategy for the ST task.
1 Introduction
End-to-end (E2E) models have made significant strides in the artificial intelligence realm, especially in speech translation (ST). These models have low latency and less error propagation by providing di- rect translations from speech inputs (Bérard et al., 2016; Duong al., 2016). This approach contrasts with traditional pipeline models that rely on sepa- rate automatic speech recognition (ASR) and ma- chine translation (MT) systems. However, the E2E model's single-model design poses new challenges due to its need for cross modal and cross- lingual transfers (Zheng et al., 2021; Xu et al., 2021). 1). To address this, recent studies have utilized multi-task learning (MTL), leveraging cross cross-modal cross- lingual training objectives for pre-training or joint
In this paper, we rethink task consistency in MTL and introduce gradient-based consistency metric, which denotes the consistency of the gradient direc- tion between the ST task and other auxiliary tasks. Our analysis shows that 1) ASR aids the acoustic
Conresponding author.

---

encoder and MT facilitates the textual encoder in audio-to-text transfer, 2) length inconsistency hin- ders aligning the representations of the two modal- ities, 3) disparity between noisy speech features and clean text embeddings as considerable obsta- cles, and 4) the timing and degree of task influence exhibit significant variation.
x
CASR =
log p 1,
(2)
yl LMT = Σ
log ((u1
(3)
where |- denotes length of the sequence. The model contains three modules: the acoustic en- coder (A-Enc), the textual encoder (T-Enc), and the decoder. The MT task shares the T-Enc and the decoder with the ST while the ASR shares all pa- rameters. Thus we can quantify the consistency of different tasks from the perspective of the gradient.
Inspired by the aforementioned observations, relax the ASR task that only uses it to help the acoustic encoder. We propose the Looking Back Mechanism (LBM) to overcome length consistency. It can significantly shrink the speech length with- out information loss. To bridge the modality gap, we introduce the Local-to-Global (L2G) training strategy. By incorporating speech-like noise into the text and utilizing an L2G extractor, we enhance contextual information each layer. This method effectively guides the interaction between audio and text sequences and aligns with audio process- ing requirements. We further propose a task-norm- based weight decrease method to speed up train- ing, adjusting task weights based on auxiliary task influence and timing, thus avoiding unnecessary training costs.
We employ cosine similarity as the metric for gradient direction, where higher values indicate stronger consistency between tasks within sin- gle model. To calculate the similarity, we flatten the gradient matrix into vector, providing accurate assessment of task consistency, despite yielding lower values. We focus on evaluating the gradient of the feed-forward (FFN) sub-layer and self-attention (ATTEN) sub-layer as representative parameters of the entire model. Our backbone model is the robust ConST (Ye et al., 2022). In our experiments, we set n 200. We sample five times and use average values to obtain solid results.
We test our method on MuST-C 8 language data- sets. The results show method achieve comparable performance with SOTA work without fine-tuning on ST task. We then set the new SOTA by utilizing the knowledge distillation method (Liu et al., 2019). With additional data, we achieve comparable performance with that of using only 12.5% ~ 33.3% training cost compared with the SOTA work !
2.1 Consistency in Different Modules
The consistency between the ST task and the other two tasks (ASR and MT) within these modules is shown in Figure 2. Although the ASR task shares all the parameters with the ST task, only the A-Enc exhibits high consistency with the ST task. This indicates that modeling speech the A-Enc serves the same purpose for both ASR and ST tasks, which aligns with the conclusions of Anastasopoulos and Chiang (201 8) and Bahar et al. (2019). However, the consistency between the two tasks decreases sharply after the textual modal processing in the T-Enc and decoder. The decoder's divergence is expected due to generating texts in different lan- guages. The T-Enc converts the acoustic feature to a textual feature for both tasks, but Figure 2 re- veals lower consistency. It suggests specific need for semantic-level representation in the ST task to achieve the cross-lingual goal.
2 Task Consistency Quantification
this section we investigate the consistency issue of multi-task learning on three tasks. We randomly sample samples as analysis set D ((s, x)) Here 8, x, y denote the speech, transcription and translation sequences respectively. We then build the ASR set DASR (s) and MT DMT ((xyy) }) By inputting the same data (s, y) into the model, we observe that different training tasks yield distinct parameter gradients. The losses of different tasks are given by:
The decoder exhibits higher consistency com- pared the encoder for the MT task, suggesting that the behavior of the ST decoder leans towards cross-language processing. However, the T-Enc still exhibits low consistency. Taking into account the above analysis on the ASR task, the T-Enc plays
yy Σ
LSt
log p(uxlyin-1,s)
(1)
OOur code is available at IMTL.

---

A Enc
T-Enc
Decoder
ASR
MT
Consitenny
0.6
Conststency
Consisenes
Comsistency
0.2
0.4
0.4
0.1
0.1
0.2
0.2
0.0
ASR
ASR
MT
6
2
(a) ATTEN sublayer
(b) sublayer
(a) ATTEN sublayer
(b) FFN sublayer
Figure 2: Consistency of different tasks in different modules.
Figure 3: Consistency of different tasks in different layer of T-Enc.
a unique role in MTL and does not align closely with either the other two tasks (Xu et al., 2021). Therefore, our subsequent analysis will focus on the T-Enc.
limited. Thus we further investigate other factors that impede the consistency between the MT and ST tasks.
2.3 Diserepancy between MT and ST
2.2 Impact of T-Ene
We focus two main differences between speech and text features: length and representation space. The length disparity arises from modeling granu- larity (frames for speech while sub-words for text) (Xu et al., 2023b). The representation space dis- crepancy is due to acoustic features extracted by the acoustic encoder lacking text-based information (Li et al., 2021; Fang et al., 2022). We implement the shrinking method (Liu et al., 2020; Dong et al., 2021a) and contrastive learning (CL) loss (Ye al.,
We conducted a comparison the consistency be- tween the ST task and the other two tasks within each layer of the T-Enc. Figure 3 illustrates that the bottom layer of the T-Enc demonstrates a stronger resemblance to the ASR task. The discrepancy arises the feature extraction process diverges fur- ther between the ASR and ST tasks. This suggests that the moosl transformation of A-Enc not achieved, then the T-Enc begins extracting textual information which required by ST to adequately address the cross-lingual objective. We also no- ticed that the ST task gradually aligns with the MT task, but the consistency between them is still low. This raises the question of whether the ASR task leads to insufficient alignment between speech and text in the T-Enc.
2022; Zhang et al., 2023) at the top of T-Enc re- spectively to investigate "Length" and "Rep" to represent the shrinking and CL methods respectively in Figure 5 and 6.
two issues. we employ
To remove the influence of the ASR task, the experiments conducted with the MT and ST tasks. Figure 5 illustrates the shrinking method effectively increases consistency in the decoder, as a more compact sequence casier to extract information from during cross-attention. However, this approach also results in the loss of original information, leading to significant degradation in the consistency between the two tasks in the T-Enc. On the other hand, when incorporating additional alignment loss, the changes in consistency within both modules are minimal.
We conducted further investigations into the im- pact of the ASR task on the T-Enc. We introduced two widely used ASR losses: 1) the Connectionist. Temporal Classification (CTC) loss (Graves et al.. 2006) after the A-Enc, and 2) the cross-entropy (CE) loss after the decoder. The former updates only the A-Enc (Bahar al., 2019), while the lat- ter updates the entire model. By exploring various MTL combinations, the changes consistency be- tween the MT and ST tasks are depicted in Fig- ure 4. We discovered that as the ASR training be- comes more intensive, the decrease in consistency between the MT and ST tasks becomes more pro- nounced the top layers. Although increasing the ASR training workload burdens the T-Enc, research indicates that the ASR task is crucial in helping the acoustic encoder model speech (Le et al., 2023). We find the impact of the ASR task on the T-Enc is
To explore why CL loss does not work, we con- duet an in-depth analysis from the perspective of information entropy (IE). Higher entropy implies greater outcome uncertainty. We compute the IE of each T-Ene's self-attention weights, as shown in Figure 6. The MT task shows lower IE compared to the ST task, indicating reduced noise sources in its representation. When we shrink the length the speech, the IE is noticeably reduced. This can

---

Information entropp:
Cf CTC
wio ASH losn
Imgg
6.0
0.08
4.0
2.0
Figure 4: Influence of textual encoder by ASR training strategy.
Figure 6: Information entropy of T-encoder attention weight.
Baseline
Longth
Rep
MI tadk - ASR tank
0.3
Consitency
0.3
Consittncy
Consisincc
Consistoncy
0.5
0.2
0.2
0.2
0.1
0.1
0.1
0.0
T-Eae
Decader
0.0 T-Fac
Decnder
10
(a) ATTEN sublayer
(b) FFN sublayer
(a) T-Fin
(b) Decoder
Figure 5: Consistency of different modules.
Figure 7: Changes consistency along training epochs.
Method
explain why the decoder exhibits higher gradient consistency. But if we use the CL loss, does not have impact on the IE. The noisy speech sequence makes it difficult to learn more textual information. This finding can explain the CL. on sentence-leved representations performs worse than token-level ap- proaches (Ye al., 2022). However, when the CL Joss is introduced based the compressed speech, we observe that additional information is learned in the middle layers. Thus we can find that shrinking is necessary and information gap between ST and MT tasks still needs to be mitigated.
We propose the improved MTL (called IMTL) method from three perspectives: 1) denoising the ST sequence, 2) adding noising to MT, and 3) and improving training efficiency. The overall training objective is:
LST + W.CASR Wm. + woLcL (4)
The CL denotes contrastive learning and set the we 0.3. Based on the above analysis, we use the CTC loss as the ASR task.
3.1 Stable Shrinking
2.4 Time of Taking Effect
From the previous analysis, we find the de- coder benefits from the decrease in speech length. Though Liu et al. (2020) and Dong et al. (202 have proposed methods to figure the issue out, there are two main problems still need to be improved.
We have identified the discrepancies between dif- ferent tasks, but the interplay of these tasks dur- ing the training process has not been thoroughly studied. Figure illustrates the changes in con- sistency among the different modules throughout training. We observe that the assistance provided by the ASR task primarily occurs the early stage and becomes less significant later on. On the other hand, the impact of the MT task on the ST task is more complex, with a gradual decrease in con- sistency over time, indicating slower assistance to the ST task. Additionally, the behaviors of the T-Enc and decoder differ significantly. These ob- servations highlight the diversity in the timing and effects of different tasks, underscoring the need for a careful strategy to optimize their training effects and timing.
Instability
Once tokens are removed, the gradi- from the decoder can not guide the acoustic encoder. Especially when the prediction of speech not accurate at the earlier training stage, this will cause the unstable training.
Information loss If tokens are wrongly removed by method, information loss will happen. The blank token also contains pause information which can help the model understand the sentence.
To address the aforementioned issues, we pro- pose the Looking- back mechanism (LBM), as de- picted in Figure 8. Given a sequence of speech

---

Consistency loss
Gradient truncation Information loss
Share
Transformer
T-Enc layer
layer
loss
L2G extractor
Nice
<blank>
to
(a) Unstable shrinking
Acoustic encoder
like noise
Looking-back mechanism (I.BM)
"How are you"
"How" "are"
"you" "?"
Figure 8: (a) Unstable shrinking and (b) LBM. The red line (h) can supply additional information. Thus avoids the problems (a).
Figure 9: Encoder of to-gobaa (L2G) training. One T-Enc layer consists of Transformer layer and L2G extractor. The light pink parts are shared.
the 5/ and &y:
features s = (s1, 8n), we first calculate the probabilities of CTC paths and extract the posi- tion with the highest value as the decoding result T -4,,..,, 4x). Since T is generated through monotonic decoding, adjacent positions may tain repeated tokens in the result. For each repeated segment in the sequence, we select the token with the highest confidence within that segment to form a new unique result T' (t,....,+*") t.m)) Additionally, the corresponding new features s' = (s/,,.., k/m can be generated. Note that we do not filter out the blank positions to prevent eтror propagation. There- fore, the key to resolving the mentioned problems lies in effectively transferring all the information from the compressed features s'. The LBM method utilizes features from s' retrospectively retrieve information from s and extract the missing information.
FFN (Norm(s/ &:)
(6)
here Norm() denotes the normalization layer, and FFN() denotes feed-forward network used fil- redundant information. The LBM method can automatically learn the weights of each repeated frame, ensuring that the obtained sf does not in- troduce additional noise. Even when the length is significantly reduced, the LBM can preserve all the original information and avoid gradient truncation, thereby promoting stable training.
3.2 Local-to-global Training
We have observed an information difference in rep- resentation between the MT and ST tasks in the T-Enc. The main reason that the speech fea- ture undergoes high-level abstraction by the acous- tic encoder, while the text embedding remains un- processed and devoid of any noise. This inherent difference causes the model classify these tasks differently, resulting in inconsistent gradi- ents. Wang et al. (2020b) injects some audio-like tokens into the MT sequence, while we propose the local-to-global (L2G) training strategy to bridge the information gap.
Formally, for an arbitrary feature s/ in s', can determine its index in s based on t and T. We set a boundary b for looking back, ensuring that [j - b.j + b] contains all the repeated tokens. We construct the search matrix A by including all features from max(1,5 b) to min(j + b, n), ex- cluding the feature at index j. We then search and aggregate information in A using the following formula:
We first introduce noise to the clean text embed- ding. Taking into account the characteristics of repeated information and blank positions in speech sequences, we randomly add blanks or duplicate certain tokens with probability of 0.2 for each position. Our goal is to facilitate the learning consistent representations for the two tasks. To
3; Softmax (R(s)) R(A)")
(5)
where the R denotes the linear transfer, Softmax() normalizes the correlation between s/ and to 0 ~ We final use the fusion module to integrate

---

Models
En-De
En-Es
En-Fr
En-It
En-NI
En-Pt
En-Ro
En-Ru
Avg.
Fairseq ST (Wang al., 2020a)
22.7
27.2
32.9
27.3
28.1
21.9
15.3
24.8
Revisit ST (Zhang et al., 2022a)
v
23.0
28.0
33.5
23.5
27.1
28.2
23.0
15.6
25.2
STEMM (Fang et al., 2022)
25.6
30.3
36.1
25.6
30.1
31.0
24.3
17.1
27.5
ConST (Ye et al., 2022)
25.7
30.4
36.8
26.3
30.6
32.0
24.8
17.3
28.0
ST (Cheng et al., 2022)
26.4
31.0
37.2
26.6
30.9
32.8
25.4
18.3
28.6
CMOT (Zhou al., 2023)
27.0
31.1
37.3
26.9
31.2
32.7
25.3
17.9
28.7
CRESS (Fang and Feng, 2023)
27.2
31.9
37.8
27.3
31.6
33.0
25,9
18.7
29.2
Baseline
25.8
30.4
36.7
26.1
30.5
32.0
24.7
17.3
28.0
IMTI.
26.9
31.5
37.7
27.3
31.3
33.0
25.5
18.3
28.9
IMTL-KD
27.5
31.8
38.2
27.7
32.0
33.4
25.9
18.6
29.4
Table I: Performance different data set. FT denotes the model needs fine-tuning stage. i means the work does not use the unlabeled speech data.
Models
FT
En-De
En-Fr En-Es
local information. To address this, proposc increasing window approach to assist the acoustic representation in capturing global textual informa- tion. Specifically, we introduce an increasing stride
ConST (Ye et al., 2022)
<
28.3
38.3
32,0
STPT (Tang et al., 2022)
39.7
33.1
S (Cheng et al., 2022)
v
29.3
38.5
32.4
CMOT (Zhou al., 2023)
29.0
39.5
32.8
CRESS (Fang and Feng. 2023)
v
29.4
40.1
33.2
for the convolution field, where each layer's 's field increases by d. Therefore, the kernel size of the
SpeechUT (Zhang al., 2022b)
v
30.1
41.4
33.6
Baseline
28.4
39.1
32.4
i-th T-Enc layer is c + d i.
IMTL
29.3
40,6
33.4
IMTL-KD
29.7
41.1
33.9
3.3
MTL Based on Task Impact
Table 2: Performance different data with addi- tional training data.
Our previous analysis reveals that the impact of different tasks and modules varies over time. This insight has inspired us develop a new training strategy that gradually eliminates the auxiliary task, rather than relying on an additional fine-tuning stage. This approach simplifies and streamlines the entire training process. To achieve this objec- tive, we need to determine whether the auxiliary task is beneficial each training step and assess its level of impact. We can examine the change in task consistency address the first question. When the task consistency stabilizes and different tasks reach a balanced state, we can reduce the training weight assigned to the auxiliary task. However, to effectively decrease the weight, we must quantify the influence of the auxiliary task.
achieve this, we propose the L2G feature extrac- for. We aim to use the interaction window size to limit the positions from which information ex- tracted. Convolution networks are well-suited for this purpose, and we implement the L2G extractor using:
x Conv (Nor((N)m(x))
(7)
where Conv() denotes the depthwise separable con- volution (Chollet, 2016). We add the extractor in front of each Transformer layer in T-Enc. This ex- tractor can learn relevant information from given window which is determined by the convolution kernel size. Unlike the self-attention mechanism that learns from the entire sequence, this window focuses on specific region, aiding the two tasks in learning the same information. However, also in- troduces additional information for MT task, which necessitates the text's ability to enhance its denois- ing capabilities. Finally, we utilize the consistency loss to align the representations extracted by the extractor and attention mechanisms.
In multi-task learning, the use of norms has been extensively studied (Argyriou et al., 2008; Maurer et al., 2013). Norms can evaluate the sparsity a matrix and are commonly employed to enhance the information in network parameters, thereby im- proving the effectiveness of MTL. Consequently, gradient norms have been successfully utilized in computer vision (Chen et al., 2018) to balance the impact of different tasks. Taking inspiration from this, we propose task impact metric for auxiliary tasks based on gradient norms. We sample k in- stances from the training set to create D', which we then feed into the model to obtain gradients for the
The study conducted by Xu al. (2021) demon- strates that the MT task requires more global understanding to form a semantic-level representa- tion, whereas the acoustic task primarily relies on

---

Length ratio(%)
Length
Models
En-De
En-Fr
Models
En-De
En-Fr
ratio(%)
Baseline
25.8
36.7
Bascline
25.8
100.00
36.7
100.00
+Fixed window
26.2
37.3
Shrinking
25.7
53.97
36.8
57.72
+L2G
26.4
37.5
+LBM
26.3
55.67
37.2
60.13
1.BM
26.3
37.2
+Fixed window
26.6
37.5
Table 3: Ablation study on shrinking method.
+1.2G
26.9
37.7
Table 4: Ablation study on L2G training.
various tasks. The task impact m of auxiliary task can be calculated using the following formula:
dataset. The Dev set was used for validation, and tst-COMMON set served the test set for all tasks. SentenccPiece segmentation with a vocabulary size of 10,000 was applied all training datasets, The detail of the data shown in Appendix A.
| l2 JED 118 12
Σ
mi h:
where 5/ is the ATTEN (self-attention sub-layer) gradient of data j for task i, || 2l2 denotes the 2- norm of the matrix. The higher shows updating the gradient will have greater impact on the ST task. Containing the change of different tasks, we give the weight of the different task at I-th update as follows:
4.2 Model settings
We used the Fairseq toolkit (Ott et al., 2019; Wang et al., 2020a) to implement our methods. The Transforme
BASE configurations were cho- sen as the baseline settings, with approximately 150M parameters. We reproduced the ConST method to establish strong baseline (Ye et al., 2022). The acoustic encoder was initialized with the audio-only pre-trained HuBert (Hsu al., 2021). In the presence of additional data, we fol- lowed the setup of SpeechUT (Zhang et al., 2022b), which utilized a hidden size of 768, 12 attention heads, and 3072 FFN dimension. Each training batch contained 20M audio frames. We set the training steps to 80K. When using additional MT data, the data size for different tasks becomes ex- tremely unbalanced. Therefore, MT task for 15 epochs with 8192 tokens per batch and then sampled 3 million sentences as MT data for MTL. We change the updated frequency to 4 and the training step to 40K. The kernel size and the increased stride d for the L2G extractor was set to 5 and 3, respectively. The value of s was set to 5000 for the ASR task and 10,000 for the MT task. The initial weights of ASR and MT tasks 1.0. We updated the task weight every 5000 training steps and removed the task when the weight fell below 0.1. During inference, we average the last 10 checkpoints for evaluation. The other decoding settings are the same as those in CRESS (Fang and Feng, 2023). We use ScareBLEEU (Post, 201 8) the metric for ST performance. The experiments were conducted eight NVIDIA GeForce RTX 3090 GPUs.
i = - (m:)
(9)
where u represents the current training step, and denotes the smoothing coefficient. The impact of these two hyper-pane syper-pprameters can be likened to tem- perature coefficients and we can set appropriate and values to ensure that changes in task weights correspond to changes in task consistency. Since the weight between T-Enc and Decoder differr, we select the maximum value w for the MT task. The design of w takes into account the consistency and impact of different tasks, thus avoiding necessary computational resources when auxiliary tasks are not beneficial. Furthermore, this training strategy allows us to remove the other task time and achieve optimal performance without the need for tedious fine-tuning stages.
first trained the
4 Experiments
4.1 Data
We conducted experiments on the multilingual MuST-C dataset (Di Gangi et al., 2019). The dataset consists of eight language pairs: English (En) to German (De), French (Fr), Spanish (Es), Ro- manian (Ro), Russian (Ru), Italian (It), Portuguese (Pt), and Dutch (NI). For the En-De, En-Fr, and En-Es MT tasks, we collected external training data from WMT16, WMT14, and WMT13 respec- tively. As additional ASR data, we utilized the LibriSpeech (Panayotov et al., 2015) 100
Sttp:///sshhbh.oomgggggssssstttt

---

MT
L2G
Training time
Models
En-De
En-Fr
En-Es
0.8
ASR 12sk
SpeechUT
96 Gd 80k tuning steps
Weight
0.6
IMTL
12 Gd
32 Gd
20Gd
0.4
0.2
Table 5: A comparison of training cost with additional MT data. 1 Gd indicates that using one GPU training one day. The SpeechUT and IMTI. the V V100 and 3090 GPU respectively.
Training steps(K)
Figure 10: IE of different methods at the T-Enc layers
(left). Task weights along training steps (right).
not benefit the model, although it significantly re- duces the length of the sequence. However, after applying the LBM method, the model achieves a 0.5 BLEU improvement while maintaining a low length ratio. This phenomenon demonstrates that shrinking alone is not stable, and the loss of infor- mation can lead to performance degradation. We find the average length of En-De audio is about times the length of En-Fr audio, thus the shrinking effect is better.
4.3 Results
The comparison of our IMTL and other works un- der the circumstance of no additional data is shown in Table 1. We find that the work utilizing the pre- training and fine-tuning paradigm achieves a sig- nificant improvement compared to the vanilla train- ing strategy. ST even designs two-stage fine- tuning method. However, few works have explored the extent of improvement gained by pre-training (Le al., 2023), which a high-cost method. Our IMTL, which dynamically decreases the weight of the auxiliary task and does not rely on fine-tuning, still achieves state-of-the-arr (SOTA) performance. This proves that our method fixes the consistency during multi-task learning and further improves training efficiency. We have noticed that the newly proposed SOTA work implements teacher-foreing bridge the modal gap, known as the knowledge distillation (KD) method. We further incorporate the KD method (Liu et al., 2019; Xu et al., 2021) into our IMTL, resulting in IMTL-KD. This demon- strates that our method is complementary to the KD method and achieves new SOTA performance.
4.5 Effect of L2G
We conducted an ablation study on L2G training. and the results are presented in Table 4. It shows that adding noise and constraining the field of in- formation interaction significantly improve the per- formance compared to the baseline. Furthermore, the method still performs well based on the LBM, which confirms the conclusion that compressed quences can learn additional information. When we apply the local-to-global strategy, the performance gains further improvement, which demonstrates that increasing the field size is more suitable for the goal of modal transformation.
We also analyzed the changes in information entropy (IE) when applying different methods in Figure 10. We observed that the IE of the first MT layer is the highest since we add some noise to the embedding. Compared to the fixed method, the L2G method can learn more information the middle layers of the model, indicating that a fixed size hinders the extraction of more global information. After employing the KD method, the IEs of all layers become more consistent with MT, except for the first noisy layer.
We also compare our method with other works that utilize extra training data. The SOTA work SpeechUT aims to cover all speech-to-ted tasks, thus it requires a significant amount of training resources (pre-training for 3 days with 32 GPUs) and complicated training strategy. In contrast, our model achieves comparable or better performance with much fewer training resources (e.g., 1.5 days with 8 GPUs for the En-De task) and does not require fine-tuning. The building process is much simpler and more efficient.
4.6 Change of Task Weight
We display the changes in task weights in Fig- ure 10. The weight of the ASR task decreases rapidly, while the weight of the MT task gradually decreases, slowly eliminating its impact on the ST task. This also aligns with the observed pattern of
4.4 Effect of LBM
We compare the effects of the shrinking (Liu et al., 2020; Dong et al., 2021a) and LBM methods in Table 3. Directly using the shrinking method does

---

gradient consistency in our analysis. We compare the training time in Table and find that our method requires about 12.5% ~ 33.3% of the training cost of SpeechUT on three MuST-C tasks. Additionally, our method does not require alignment with the fine-tuning stage on the ST task. This demonstrates the efficiency of our method.
niques are also employed achieve this objective (Fang et al., 2022; Zhou et al., 2023: Fang and Feng, 2023). Furthermore, the mixing-up of two modal representations has been found to be effec- tive (Cheng et al., 2022). The inclusion of blank tokens (Wang et al., 2020b; Zhang et al., 2023) can improve denoising capabilities. To address length inconsistencies, shrinking based ASR prediction or cluster methods have been utilized (Dong et al., 202 a; Zhang et al., 2022b).
5 Related Work
E2E ST has gained attention for its advantages over cascade systems terms of reduced latency and error propagation (Bérard et al., 2016; Duong et al., 2016; Weiss et al., 2017; Xu et al., 2023a). How- ever, two main challenges hinder the adoption of E2E ST: 1) limited ST training data and 2) difficul- ties in modeling the modality gap. To address these challenges, pre-training strategies have emerged, including audio-only self-learning (Baevski et al., 2020; Hsu et al., 2021), joint transiitaaniiiiioonn en- coding (Ao et al., 2022; Zhang et al., 2022b; Chen et al., 2022), and combining MT and ASR data for pre-training (Wang et al., 2020c; Zheng et al.. 2021). These approaches have shown significant improvements in ST performance.
Conclusion
Most advanced ST methods heavily rely on multi- task learning, but few studies focus on the rela- tionship between auxiliary tasks and the ST task itself. In this study, design a gradient consis- tency metric to analyze the impact of other tasks on the ST task during the multi-tast learning pro- cess. Based on our analysis, we propose improved methods that address three key aspects: length, rep- resentation, and training efficiency. Experimental results on the MuST-C dataset demonstrate that our approach achieves state-of-the-at performance and significantly improves training efficiency.
Pre-training methods are also combined with multi-stage and multi-task strategies. The multi- stage method involves pre-training all modules with auxiliary tasks, followed by integration and fine-tuning for the ST task (Xu et al., 2021; Li et al., 2021; Zhang et al., 2023). On the other hand, multi- task training utilizes multiple training objectives within single model, eventually fine-tuning with the ST loss (Wang al., 2020b; Le et al., 2020; Vy- dana et al., 2021; Tang et al., 2021; Ye et al., 2021). While most SOTA methods employ the pre-training and fine-tuning paradigm, few studies have inves- tigated the impact of other tasks on boosting the ST task, considering the time-consuming nature of pre-training. Tang et al. (2022) provided a simple analysis that showed gradient interference is not serious and the effectiveness of MTL. In this paper, we conduct comprehensive experiment explore the impact and time efficiency of other tasks.
Acknowledgement
This work was supported in part by the National Science Foundation of China (N..62260566, the National Key R&D Program of China, the China HTRD Center Project N...020AAA01079000 the Natural Science Foundation of Liaoning Province of China (2022-KF-16-01) the Yunnan Provin- cial Major Science and Technology Special Plan Projects (N0.202103AAA00011, the Fundamental Research Funds for the Central Universitics (Nos. N2216016, N2216001, and N221 N2216002), and the Program of Introducing Talents of Discipline to Universities, Plan 111 (No. 6009). The authors would like thank anonymous reviewers for their insightful comments.
Limitations
There are some limitations that our work has not figured out. The analysis is mainly carried out on the MuST-C dataset, where the training data size is not large. We did not apply the statc-of-the- art knowledge distillation (KD) method to further improve performance. The effect of knowledge distillation based on IMTL has not been sufficiently investigated.
Mitigating differences in representation and ad- dressing variations in sequence lengths are two ways used to bridge the modality gap between text and speech. Some work proposes the use of adapters to reduce differences in pre-trained modules (Bahar et al., 2019; Li et al., 2021; Xu et al., 2021). Contrastive learning (Ye et al., 2022; Zhang et al., 2023) and knowledge distillation tech-

---

References
Jean-Antoine Désidéri. 2012. Multiple-gradion descent algorithm (mgda) for multiobjectivv optimization. Comptes Rendus Mathematique, (50(5-6):313-3)
Antonios Anastasopoulos and David Chiang. 2018.
Tied multitask learning for neural speech translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, Volume (Long Papers), pages 82-91, New Orleans, Louisiana. Association for Computational Linguis- tics.
Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. MuST-C: a Multilingual Speech Translation Corpus. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies. Volume 1 (Long and Short Papers), pages 2012-2017, Min- ncapolis, Minnesota, Association for Computational Linguistics.
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang. Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, and Furu Wei. 2022. SpeechT5: Unified-modal encoder-decoder pre pre-traiming for spoken language processing. Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 5723-5738, Dublin, Ireland. Associa- tion for Computational Linguistics.
Qianqian Dong, Mingxuan Wang. Hao Zhou, Shuang Xu, Bo Xu, and Lei Li. 2021a. Consecutive decod- ing for speech-to-text translation. In The Thirty-fitth AAAI Conference on Artificial Intelligence, AAAI.
Qianqian Dong, Rong Ye, Mingxuan Wang, Hao Zhou, Shuang Xu, Bo Xu, and Lei Li. 2021b. "listen, un- derstand and translate": Triple supervision decouples end-to-end speech-to-text translation. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pages 12749-12759.
Andreas Argyriou, Theodoros Evgeniou, and Massimil- iano Pontil. 2008. Convex multi-task feature learning. Machine learning, 73:243-272.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances Neural Information Processing Sys- tems, volume 33, pages -2440.
Long Duong, Antonios Anastasopoulos, David Chiang, Steven Bird, and Trevor Cohn. 2016. An attentional model for speech translation without transcription. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 949-959.
Parnia Bahar, Tobias Bieschke, and Hermann Ney. 2019. A comparative study on end-to-end speech to text translation. 2019 IEEE Automatic Speech Recog- nition and Understanding Workshop (ASRU), pages 792-799. IEEE.
Qingkai Fang and Yang Feng. 2023. Understanding and bridging the modality gap for speech translation. arXiv preprint arXiv:2305.088 087066
Alexandre Bérard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. 2016. Listen and translate: proof of concept for end-to-end speech-to-text trans- lation. arXiv preprint arXiv: 1612.01744.
Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. 2022. STEMM: Self-learning with speech-text manifold mixup for speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7050-7062, Dublin, Ireland. Association for Computational Linguistics.
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEFF Journal of Se- lected Topics in Signal Processing, (666):500-1115
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber: 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning. pages 369-376.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. 2018. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pages 794-803. PMLR.
Chi Han, Mingxuan Wang. Heng Ji, and Lei Li. 2021. Learning shared semantic space for speech-to-text translation. In Findings of the Association for Com- putational Linguisticx: ACL-IJCNLP 2021, pages 2214-2225, Online. Association for Computational Linguistics.
Xuxin Cheng, Qianqian Dong, Fengpeng Yue, Tom Ko, Mingxuan Wang, and Yuexian Zou. 2022. M3st: Mix at three levels for speech translation. ArXiv, abs/2212.03657.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdel- rahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. arXiv preprint 2006v:2106.07444/
François Chollet. 2016. Xception: Deep learning with depthwise separable convolutions. 2017 IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 1800-1807.

---

Hang Le, Juan Pino, Changhan Wang. Jiatao Gu, Didier Schwab, and Laurent Besacier. 2020. Dual-decoder transformer for joint automatic speech recognition and multilingual speech translation. In Proceed- ings of the 28th International Conference on Com- putational Linguistics, pages 3520-3533, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
speech translation and recognition. In Proceedings of the 60th Annual Meeting of the Association for Computational I Linguistics (Volume I: Long Papers), pages 1488-1499, Dublin, Ireland. Association for Computational Linguistics.
Yun Tang, Juan Pino, Changhan Wang, Xutai Ma, and Dmitriy Genzel. 2021. A general multi-task learning framework to leverage text data for speech to text tasks. In ICASSP 2021 2021 IEEE International Conference Acoustics, Speech and Signal Process- ing (ICASSP), pages 6209-6213.
Phuong-Hang Lc, Hongyu Gong, Changhan Wang. Juan Pino, Benjamin Lecouteux, and Didier Schwab. 2023. Pre-training for speech translation: Cte meets opti- mal transport. In 40th International Conference on Machine Learning.
Hari Krishna Vydana, Martin Karafiát, Katerina Zmo-
likova, Lukáš Burget, and Honza Čemnckk.. 2021. Jointly trained transformers models for spoken lan- guage translation. In ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7513-7517.
Xian Li, Changhan Wang. Yun Tang, Chau Tran, Yuqing Tang. Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. 2021, Multilingual speech trans- lation from efficient finetuning of pretrained models. In Proceedings of the 59th Annual Meeting of the As- sociation for Computational Linguistics and the 1 1tth International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 827-838, Online, Association for Computational I. Linguistics.
Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. 2020a. Fairseq S2T: Fast speech-to-text modeling with fairseq. Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Lin- guistics and the 10th International Joint Conference on Natural Language Processing: System Demon- strations, pages 33-39, Suzhou, China. Association for Computational Linguistics.
Yuchen Liu, Hao Xiong, Jiajun Zhang. Zhongjun He, Hua Wu, Haifeng Wang, and Chengging Zong. 2019, End-to-End Speech Translation with Knowledge Dis- tillation. In Proc. Interspeech 2019, pages 128- 1132.
Chengyi Wang. Yu Wu, Shujic Liu, Zhenglu Yang, and Ming Zhou. 2020b. Bridging the gap between pre- training and fine-tuning for end-to-end speech trans- lation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9161-9168.
Yuchen Liu, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2020. Bridging the modality gap for speech- to-text translation. arXiv preprint arXiv:2010.144 14920.
Andreas Maurer, Massi Pontil, and Bernardino Romera- Paredes. 2013. Sparse coding for multitask and trans- fer learning. In International conference on machine learning, pages 343-351. PMLR.
Chengyi Wang, Yu Wa, Shujie Liu, Ming Zhou, and Zhenglu Yang. 2020c. Curriculum pre-training for end-to-end speech translation. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 3728 03788, Online. Asso- ciation for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli, 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HIT 2019: Demonstrations.
Ron J Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-sequene models can directly translate foreign speech. arXiv preprint arXiv: 473.v:1703.08581.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and San- jeev Khudanpur. 2015, Librispeech: An asr corpus based on public domain audio books. In 2015 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages $206-5210
Chen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang, Shen Huang, Qi Ju, Tong Xiao, and Jingho Zhu. 2021. Stacked ccoustic-and-textie the pre-trained models into speech translation en- coders. In Proceedings of the 59th Anmual Meet- ing of the Association for Computational Linguistics and the I 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 2619-2630, Online. Association for Computa- tional Linguisties.
encoding: Integrating
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191. Brussels, Belgium. Association for Computa- tional Linguistics.
Ozan Sener and Vladlen Koltun. 2018. Multi-task learn- ing as multi-objective optimization. Advances in neural information processing systems, 31.
Chen Xu. Rong Ye, Qianqian Dong, Chengqi Zhao, Tom Ko. Mingxuan Wang. Tong Xiao, and Jingbo Zhu. 2023a. Recent advances in direct speech-to- text translation. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intel- ligence, IJCAI-23, pages 6796-6804. International
Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, and Juan Pino. 2022. Unified speech-text pre-training for

---

Joint Conferences on Artificial Intelligence Organi- zation. Survey Track.
Appendix
A Data Details
Chen Xu, Yuhao 7hang, Chengbo Jiao, Xiaoqian Liu, Chi Hu, Xin Zeng, Tong Xiao, Anxiang Ma, Huizhen Wang, and JingBo Zhu. 2023b. Bridging the gran- ularity gap for acoustic modeling. arXiv preprint arXiv:2305. 17356.
We conducted experiments the multilingual MuST-C dataset (Di Gangi al., 2019). The de- tail of the data is shown in Table 6. The detail of additional data is shown in Table 7.
Rong Ye, Mingxuan Wang, and Lei Li. 2021. End-to- End Speech Translation via Cross-Modal Progressive Training. In Proc. Interspeech 2021, pages 2267- 2271.
Language
Hours(h)
Sentence(K)
En-De
408
234
Rong Ye. Mingxuan Wang, and Lei Li. 2022. Cross- modal contrastive learning for speech translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5099-5113, Seattle, United States. Association for Computational Linguistics.
En-Es
504
270
En-Fr
492
280
En-It
465
258
En-NI
442
253
En-Pt
385
211
En-Ro
432
240
En-Ru
489
207
Biao Zhang, Barry Haddow, and Rico Sennrich. 2022a. Revisiting end-to-end speech-to-text translation from scratch. In International Conference on Machine Learning, pages 26193-26205. PMLR.
Table 6: Training data size of the MuST-C 8 languages.
Yuhao Zhang, Chen Xu, Bojie Hu, Chunliang Zhang, Tong Xiao, Jingho Zhu. 2023. Improving end-to- end speech translation by leveraging auxiliary speech and text data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13984- 13992.
Dateset
Language
Sentence
WMT16
En-De
3.9M
WMT13
En-Es
14.2M
WMT14
En-Fr
31.2M
LibriSpeeh 100h
En
28.5K
Ziqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, and Furu Wei. 2022b. SpeechUT: Bridging speech and text with hidden- unit for encoder-decoder based speech-text pre- training. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing. pages 1676, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Table Training data size of additional MT and ASR data.
B Contrastive Loss
Renjie Zheng, Junkun Chen, Mingbo Ma, and Liang Huang. 2021. Fused acoustic and text encoding for multimodal bilingual pretraining and speech transla- tion. arXiv preprini arXiv:2102.0577
In this paragraph, we introduce the notation and define the loss function for contrastive training. We start by defining two outputs: A(s) represents the output of the ST encoder when given the speech input s, and M(x) represents the output of the pre- trained text encoder when given the transcription x. We then consider set of training samples denoted as (st, 2;).
Yan Zhou, Qingkai Fang, and Yang Feng, 2023. Cmot: Cross-modal mixup via optimal transport for speech translation. arXiv preprint ar 23055 14635.
The loss function for contrastive training, de-
noted as LCL. is defined as follows:
Σ log (((A()))
LCI. =
(M))))) Emjitt (((l))))
(10)
In this equation, ((. ) is function that com- putes the similarity between the input vectors. For our purposes, we choose the cosine function as

---

Training task(s)
Speed (Seconds/Epoch)
6.3
Braclinc IMTL
ST, MT, ASR
118
PPL
ST, MT
~93
5.0
ST
~675
Table 8: Training data size of additional MT and ASR data.
Traiing stepa(K)
Figure 11: PPL Dev set during training.
ASR and MT tasks. Specifically, any auxiliary task whose training weight diminishes below thresh- old of 0.1 will be effectively halted to optimize the training process. As bonus, subsequent training phases are computationally more efficient than the standard approach, given that both the forward and backward computations are integrated components of the overall training pipeline. Table 8 shows a rough estimate of the training speed of our IMTL approach on the MuST-C dataset with different training tasks.
T(..)) and apply average pooling to the two se- quence representations. The variable is a scaler that controls the sharpness of the function output, and in this case, set to 0.1.
For each speech input 8;, have its correspond- ing labeled transcription 24, which forms a positive sample (s,, r;). Additionally, we utilize transcrip- tions other than x, (denoted as zj for j + i) to create negative samples.
C Information Entropy
Information entropy is concept from informa- tion theory that measures the average amount of information contained a set of data or the un- certainty associated with the data. In the context of information theory, entropy is calculated using the probabilities of different outcomes events occurring within system. The higher the entropy, the greater the uncertainty or lack of information about the outcomes. Conversely, lower entropy indicates a higher degree of predictability or knowl- edge about the outcomes. The formula is given by:
H(X)= Σx(x) p(x) * (x(x))(x))
(11)
where (X) represents the entropy of a random
variable X, Pa:) is the probability of each possible outcome æ, and the sum is taken over all possible outcomes.
D Coverage Speed
Figure 11 shows the coverage speeds of the base- line and our IMTL. We can find the IMTL is better in terms of convergence speed and effect.
E Training Speed
There are mainly three tasks (ASR, MT, and ST) during the training strategy. Our Improved Multi- Task Learning (IMTL) algorithm dynamically ad- justs the training weights assigned to the auxiliary


Reviews:
[{"review_id": "i3oHELIvMu", "reviewer": "Reviewer_XztX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you sincerely for your valuable comments, for which we are deeply grateful. Here is our response to them. Here is our response to them. \n\n### Response to the weakness:\n\nWe appreciate your suggestion regarding the incorporation of an organized model diagram to convey our approach more effectively. In the current version of our paper, Figure 1 offers a concise overview of a widely-used multi-task training architecture. This architecture comprises an acoustic encoder designed to process speech input, followed by a textual encoder that takes both the output from the acoustic encoder and the source text. The architecture culminates in a decoder that computes three task-specific losses: for Automatic Speech Recognition (ASR), Machine Translation (MT), and Speech Translation (ST), respectively.\n\nWhile Figure 1 serves as a foundational framework, we recognize the importance of a more holistic and clearer diagram to elucidate our specific contributions. Specifically, our Shrinking Mechanism and LBM module, as illustrated in Figure 8, are injected between the two encoders. LBM is also based on the computation of the auxiliary loss in the original architecture. Additionally, our Local-to-Global Training Strategy, presented in Figure 9, serves as an enhancement to the textual encoder originally depicted in Figure 1. In the revised version of the paper, we intend to integrate a more encompassing architecture overview that synthesizes all these elements for clearer, more intuitive understanding.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1 I really like the deep analysis on Task Consistency. \n\n2 Problem formulation is well defined and the results seems to perform better.", "reasons_to_reject": "This paper lacks an organized model diagram to help readers better understand the entire algorithmic process", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper conducts a comprehensively the impact of other tasks on the ST task during the multi-task learning process. This paper propose an improved multi-task learning (IMTL) approach for the ST task, which bridges the modal gap by mitigating the difference in length and representation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 292: Figure 3--> Figure 8"}, "extra_text": {}}, {"review_id": "SzEdY9f47f", "reviewer": "Reviewer_SFpv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks sincerely for your thorough comments, we think that we can address them all in the improved version of the paper. Here is our response to each of the concerns. \n\n### Response to the weaknesses:\n1. **The instability training of CTC shrink mechanism can be solved by pre-training the ASR task.**\n\n    We appreciate the reviewer's insightful comment on the training instability of the CTC Shrinking mechanism and its potential resolution through pre-training on the ASR task. We acknowledge the merit in this approach and would like to highlight that our \"Shrinking\" method is indeed congruent with the strategy presented by Liu et al. In the improved version, we will add this reference and make a more comprehensive comparison with our method. Here, we would like to highlight the differences between Liu et al. 2020’s work with ours:\n    - Our primary motivation is to simplify the often complicated training process. As a result, our novel LBM method successfully achieves training stability via multi-task learning, obviating the need for a pre-trained ASR model. We think this is an important contribution, in line with our aims to streamline the training process.\n    - Furthermore, we'd like to draw attention to the empirical results presented in Table 3, which underscore the superior performance of our LBM approach. Unlike the \"Shrinking\" method, which only attains a performance similar to the baseline, our LBM retains more essential information in the sequence, leading to a more robust performance, e.g. 26.3. In addition to resolving the issue of training instability, our LBM also effectively mitigates the loss of important sequence information, a problem that remains unaddressed by the \"Shrinking\" method alone.\n\n2. **Limiting the visible context of attention can also solve this problem.**\n\n    Thank you for pointing out the alternative approach of limiting the visible context of attention as a potential solution. To provide a rigorous comparison, we implemented this strategy using the cross-attention module, as you suggested. Specifically, we modified the temperature parameter in the softmax function within the attention operation to narrow the focus of the distribution, thereby \"limiting the visible context of attention.\"\n\n    We evaluated this approach on the MuST-C En-De dataset alongside our LBM and the baseline. Contrary to expectations, the method of limiting attention did not yield a significant improvement in performance. This led us to explore other avenues, culminating in the development of our LBM method. By integrating additional information into the shrunken sequence, our LBM method achieved much better performance, underscoring its simplicity and effectiveness. In light of your suggestion, we will include a more comprehensive comparison between the method of limiting attention and our LBM in the updated version of the paper.\n\n    | System | MuST-C En-De |\n    | ------ | --------------- |\n    | Baseline (Ye et al., 2022) |     25.8      |\n    | Shrinking (Liu et al., 2020)  |     25.7       |\n    | Limiting attention weight|25.9|\n    | LBM (Ours)                  |26.3|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method attains state-of-the-art results.\n2. The problem is well analyzed.\n3. Experiments are abundant", "reasons_to_reject": "1. The instability training of the ctc shrink mechanism can actually be solved by simple pre-training on the ASR task [1]. LBM doesn't seem necessary. \n2. LBG focuses on a specific region. Limiting the visible context of attention can also solve this problem.\n\n[1] Bridging the Modality Gap for Speech-to-Text Translation. arXiv:2010.14920v1", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, auhtors investigate the consistency between different tasks, considering different times and modules. They find that the textual encoder primarily facilitates cross-modal conversion, but the presence of noise in speech impedes the consistency between text and speech representations. Furthermore, auhtors propose an improved multi-task learning (IMTL) approach for the ST task, which bridges the modal gap by mitigating the difference in length and representation.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fl0NfxLVeY", "reviewer": "Reviewer_uUBZ", "summary": "", "questions": "- Please refer to the first point in \"Reasons to reject\". Do the reported result uses any form of (both supervised and self-supervised) pre-training? If it does use some form of pre-training, why do you specify \"no fine-tuning\" (Table 1) to characterize the proposed method?\n- L119: Do you mean that the ASR encoder is shared with ST encoder, while the three decoders of the ASR, MT, and ST tasks are shared?\n- There is no embedding layer for text in Figure 9. Is the text input directly fed to the Transformer decoder? This is a bit unusual in my opinion. Could you explain this choice of configuration?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks sincerely for your thorough and valuable feedback for further improving our paper. We think that we can address them all in the improved version of the paper. \n\nFirst and foremost, the reviewer expressed concern about why we still report certain well-known findings. Therefore, we reiterate our motivation here. The field of end-to-end ST has been progressing rapidly, with the application of self-supervised pre-trained models gaining traction. State-of-the-art (SOTA) performance is also advancing swiftly, leading to more intricate training strategies and higher training costs simultaneously. Given this context, we aim to reconsider whether the prevailing multi-task learning approach still aligns with previous conclusions. Furthermore, we aim to propose solutions that enhance the training strategy.\n\nFor instance, what is the optimal way to employ the ASR task? While certain prior studies have demonstrated the usefulness of the ASR task in the encoder, recent SOTA research has yielded contrasting outcomes. For instance, progressive training methodologies such as XSTNet (Ye et al., 2021) and ConST (Ye et al., 2022) relocate the ASR task to the decoder, achieving state-of-the-art (SOTA) performance. Conversely, some studies have even omitted the ASR task altogether, with methodologies like STEMM (Fang et al., 2022), CMOT (Zhou et al., 2023), and CRESS (Fang et al., 2023) still managing to secure SOTA results. These conflicting results raise intriguing questions, motivating us to undertake a comprehensive analysis to clarify the role and optimal positioning of the ASR task in modern multimodal systems.\n\nSecondly, we must acknowledge that certain experimental settings are not included in the main content due to page limitations. However, we have presented our training configuration in the Appendix and have included the training details in the code submitted alongside this paper.\n\nThe followings are our response to all you concerned.\n\n### Response to Reasons To Reject:\n1. Some details are not clear enough.\n    - **Does it mean that no pre-training in any form is used in this work?** \n    \n        In our experiments, as specified in line 859, we employ Hubert, a self-supervised speech feature extractor, for the acoustic encoding. It's noteworthy that we do not consider the use of this pre-trained model as part of our multi-task learning (MTL) framework. This follows the convention in existing literature where such self-supervised models are treated solely as feature extractors and are not counted in the pre-training phase.\n\n       We wish to clarify that our decision to employ Hubert arises from the challenges posed by the substantial noise in raw audio data. Preliminary attempts to train an end-to-end Speech-to-Text (ST) model from scratch were not successful. Nonetheless, we recognize the potential of using Mel filterbanks as an alternative speech feature. We remain committed to exploring this avenue in future work.\n   \n   - **The two cited papers are not appropriate enough, because they used pre-training while here is multi-task learning.** \n      \n        We think that the paper [1] is more suitable to be referenced in this context. Their work demonstrates the utility of incorporating CTC during the pre-training phase. Additionally, in our perspective, STPT (Tang et al., 2022), and SpeechUT (Zhang et al., 2022c) have also employed multi-task learning during their pre-training stages. Concretly, STPT (Tang et al., 2022) utilizes Speech to Phoneme classification task and SpeechUT (Zhang et al.,2022c) uses Speech-to-Unit task the as the ASR task respectively. We cite these studies to substantiate that ASR remains beneficial, particularly within the context of multi-task learning, as some work has removed the ASR task from their strategy.\n   \n     [1] Pre-training for Speech Translation: CTC Meets Optimal Transport. Le et al., 2023. ICML 2023\n      \n    - **Some important details on experiments are not included in the main content.** \n      \n         Due to page limitations, we were unable to include experimental details in the main content. However, as described in the Appendix, our baseline is identical to that of ConST, and the training configuration and data used in conjunction with external MT data match those of SpeechUT, ensuring a fair comparison. Our code will be made available as open-source, and comprehensive training details can be found in the Supplementary Materials submitted alongside this paper.\n    \n2. **Certain previous work is missing when introducing the proposed method.** \n   \n    We will revise the sections that are not adequately cited. Thanks for these suggestions to make our paper clear.\n\n3. **Several findings are not novel enough compared with existing work.**\n   - **Regarding the finding that \"ASR aids the acoustic encoder\"**, we have explained the rationale at the beginning of this response. Some studies have omitted the ASR task, prompting us to explore the most effective utilization of the ASR task. \n   - **Regarding the finding that \"MT facilitates the textual encoder in audio-to-text transfer\"**, we perceive this discovery as an explanation for the phenomenon depicted in Figure 3 and as the basis for the subsequent analysis.\n   - **Regarding the finding that \"length inconsistency hinders aligning the representations of the two modalities\"**, this paper seeks to underscore the significance of length inconsistency in influencing the alignment of representations between the two modalities. This observation is often overlooked in works aiming to bridge the gap between the two modalities.\n\n### Response to Questions:\n- **Do you use any form of (both supervised and self-supervised) pre-training?**\n\n    As stated in Appendix A.2, we utilize the pre-trained HuBert model. Many works consider it as a feature extractor and do not incorporate it into the pre-training stage. The reason we emphasize that our approach does not necessitate a fine-tuning stage is that we have observed several other methods requiring additional pre-training strategies (e.g., SpeechUT, Progressive training), followed by fine-tuning, apart from using a pre-trained acoustic encoder. These approaches are complex and time-consuming. Therefore, we introduce an improved MTL approach that yields the ST model directly, eliminating the need for intricate pre-training processes. Furthermore, previous studies also overlook accounting for the pre-training time of HuBert, making the comparison of training costs with their methods equitable.\n- **Do you mean that the ASR encoder is shared with ST encoder, while the three decoders of the ASR, MT, and ST tasks are shared?**\n  \n    Regarding the baseline architecture, the answer is yes. The ASR task shares all three modules—acoustic encoder, textual encoder, and decoder—with the ST task. Similarly, MT also shares two modules—textual encoder and decoder—with the ST task. The baseline architecture is the same as ConST (Ye et al., 2022).\n    In contrast, for our IMTL approach, the ASR task only comprises a single acoustic encoder that is shared with the ST task. We arrived at this configuration based on our analysis, as we determined that the ASR component is unnecessary for the decoder. \n\n- **The description of Figure 9 is not clear.**\n\n    Figure 9 depicts the architecture of the encoder components. The left part represents the two encoders for the ST task, while the right part corresponds to the textual encoder for the MT task, with the source text serving as input. The textual encoder is shared between the two tasks, and both of their outputs are subsequently fed into the decoder (which is shared too and not illustrated in this figure) through cross-attention modules. Additional description will be incorporated to clarify these details.\n\n### Response to Typos Grammar Style And Presentation Improvements:\n**What do the legends, such as \"Baseline,\" \"Length,\" and \"Rep,\" mean in Figures 5 and 6?**\n\n\"Length\" refers to the model utilizing the shrinking method (L209). On the other hand, \"Rep\" pertains to the model incorporating the contrastive learning (CL) loss positioned at the top of the T-Enc method (L211). The CL loss serves as an alignment mechanism operating on the representation (Rep).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written in general. The structure is clear and easy to understand.\n- The work is well-motivated as MTL is currently a dominant approach in building strong ST systems, hence a good understanding of the interplay between the auxiliary tasks with the main ST task would facilitate researches in this direction.\n- While many of the findings/observations from this paper have been known in previous work, the paper offers an interesting quantitative analysis based on module gradients and information entropy to systematically demonstrate the benefits and/or downsides of each auxiliary task (ASR/MT) to ST.", "reasons_to_reject": "1. The scope of work is confusing in several parts and some key details are missing.\n\t- As I understand, L57-62 discusses MTL in (both self-supervised and supervised) pre-training, but the paper focuses on MTL in ST training which usually requires no additional fine-tuning. Does it mean that no pre-training in any form is used in this work?\n\t- This paper examines ASR's impact on the speech encoder (L67, 195) within the ST joint training framework. Yet, the references to Tang et al. 2022 and Zhang et al. 2022c (L195) pertain to pre-training.\n\t- The model architecture used as well as other important details on experiments are not included in the main content for easier comparison with previous work.\n\n2. Previous work that the proposed method built upon is not clearly specified in the method description.\n\t- Previous CTC-based shrinking method, which is referred to as unstable shrinking in the paper and is the first step in the proposed LBM method, should be mentioned in related text to enhance clarity.\n\t- Likewise, proper credit to prior research like Wang et al. 2020b for the idea of introducing noise to text input should be included in the relevant section (description of the L2G approach).\n\n3. Several findings are a bit weakly positioned with existing work; additional references are needed to enhance transparency. In particular, it is not clear from the Introduction (and in later text) that which findings have already been known and the differences in contributions this paper makes compared to previous work should be emphasized.\n\t- Regarding \"ASR aids the acoustic encoder\", this is a well-established fact and previous works that proved this result in a MTL setup (which is the main topic of this paper) should be appreciated such as Weiss et al. 2017, Anastasopoulos et al. 2018 (Tied Multitask Learning for Neural Speech Translation), and Bahar et al. 2019 (A Comparative Study on End-To-End Speech to Text Translation).\n\t- Regarding \"MT facilitates the textual encoder in audio-to-text transfer\", this is the motivation behind the increasingly common cascaded encoders approach to decouple the speech encoder into cascaded speech-then-text encoders. Initial work in this direction should be mentioned such as Liu et al. 2020 (Bridging the Modality Gap for Speech-to-Text Translation), Dong et al. 2021 (Listen, Understand and Translate\": Triple Supervision Decouples End-to-end Speech-to-text Translation), and Xu et al. 2021.\n\t- \"length inconsistency hinders aligning the representations of the two modalities\". The length inconsistency issue has been raised in previous work such as Wang et al. 2020b  and Liu et al. 2020 (Bridging the Modality Gap for Speech-to-Text Translation).", "questions_to_authors": "- Please refer to the first point in \"Reasons to reject\". Do the reported result uses any form of (both supervised and self-supervised) pre-training? If it does use some form of pre-training, why do you specify \"no fine-tuning\" (Table 1) to characterize the proposed method?\n- L119: Do you mean that the ASR encoder is shared with ST encoder, while the three decoders of the ASR, MT, and ST tasks are shared?\n- There is no embedding layer for text in Figure 9. Is the text input directly fed to the Transformer decoder? This is a bit unusual in my opinion. Could you explain this choice of configuration?", "paper_topic_and_main_contributions": "This paper performs a comprehensive study of the effects of auxiliary tasks (namely ASR and MT) to the ST task in a multi-task learning (MTL) setup. Based on the observations from the analyses, the authors propose an MTL approach that specifically improves the consistency between the tasks, mitigates the length mismatch, and bridge the representation gap due to different modalities. The proposed method consists of three main parts:\n- A shrinking-based method called Looking-back mechanism (LBM) based on CTC outputs to minimize information loss\n- Local-to-global Training (L2G) strategy that inserts audio-like noises into the text input sequence, following by a CNN extractor to enable the consistency of the representation learning between the two tasks\n- MTL Based on Task Impact training strategy that adjusts the task impacts as training progresses.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "In addition to the important references cited in \"Reasons to reject\", I think the literature review could be improved by the following additional references:\n- For shrinking method, should cite the first work (as far as I know) proposing this, which is Liu et al. 2020 \"Consecutive decoding for speech-to-text translation\". \n- L208: Previous work on ST earlier than Fang et al., 2022 already highlighted this and proposed different methods to remedy this issue such as Li et al., 2020 (\"Multilingual speech translation with efficient finetuning of pre-trained models\"). \n- Landmark papers on multi-task learning such as Bahar et al. 2019 (A Comparative Study on End-To-End Speech to Text Translation) who used CTC loss for ASR task in joint training should be appreciated.\n- L446: Le et al., 2023 (Pre-training for Speech Translation: CTC Meets Optimal Transport) studied the effects of different losses in supervised pre-training.\n- L560:  Bahar et al. 2019 (A Comparative Study on End-To-End Speech to Text Translation), Li et al., 2020 (\"Multilingual speech translation with efficient finetuning of pre-trained models\") proposed using adapters to connect the pre-trained modules while Le et al., 2021 studied the use of adapters within every Transformer layers.", "Typos_Grammar_Style_and_Presentation_Improvements": "- It would facilitate the reading to briefly explain \"task consistency\" in the Introduction in my opinion.\n- L316: $w$ is not previously defined. Similar for ATTN in L418.\n- As a minor suggestion, previous work such as Tang et al. 2022 that also adopts the same analysis method using gradient similarity between subtasks should be cited in the relevant part as well.\n- Figure 5-6, what do legends such as \"Baseline\", \"Length\", and \"Rep\" mean?\n- L173: typo in the full stop in the middle of a sentence.\n- L246: \"needs mitigate\" -> \"needs to be mitigated\"\n- L279: \"two main problems stall need improve\" -> \"two main problems still need to be improved\"?\n- L283: \"guide the acoustic encode\" -> \"guide the acoustic encoder\"\n- L292: link to Figure 3 needs to be fixed (it leads to Table 3 instead). \n- L336: \"a information difference\" -> \"an information difference\"\n- Eq 8: as a minor suggestion, putting 1/k outside of the sum is more visual in my opinion.\n- Table 2: \"STFT\" -> \"STPT\""}}, {"review_id": "I75vOerRKf", "reviewer": "Reviewer_idg1", "summary": "", "questions": "I recommend providing more descriptive explanations for the Figures so that they are self-contained and easier to read.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough comments sincerely. Overall, the reviewer stated that our paper is not self-contained enough due to the writing. This is due to page limitations, which prompted us to relocate the experiment details to the Appendix. We believe that most of your concerns can be addressed by referring to the Appendix directly. We will carefully revise the paper and add the essential details into experiments settings. Reviewer #3 commented that the paper is generally well-written and we suppose he has referred to the Appendix. Furthermore, we have submitted the codes for review and plan to open-source them. This will enable access to all relevant details and ensure the reproducibility of our results.\n\n### The followings are response to Reasons To Reject:\n1. **How do the authors introduce the L2G extractor and what is its configuration? What are \"u\" and \"s\" in equation 9?**\n\n    - Equation (7) details our L2G method, which serves as a key feature of our textual encoder. In an effort to enhance the encoder's capability for processing local information, we employ depthwise separable convolutions integrated within a residual network fashion. This design allows the model to transition seamlessly from capturing local patterns to understanding global context. The unique aspect of our L2G approach lies in the gradual expansion of the convolutional kernel size. This strategy facilitates the model's transition from focusing on local features to incorporating a more global field of view, thereby enriching the encoder's representational power. For a detailed configuration of our L2G method, including kernel sizes and other hyperparameters, we refer the reader to Appendix A.2, Line 872. The kernel size c and the increased stride d for the L2G extractor was set to 5 and 3 respectively.\n    \n        Moreover, we have submitted our code as Supplementary Materials. The architecture of our model can be found at IMTL/fairseq/models/speech_to_text/s2t_joint.py, and the implementation of LBM is available here as the LookBackModule class on line 736. The implementation of L2G is located in IMTL/fairseq/models/transformer/transformer_encoder.py, spanning from line 100 to line 119.\n\n   - The coefficients 'u' and 's' in Equation (9) serve as parameters to regulate task weights. We emphasize our consideration of both task impact and consistency. Consequently, we select appropriate 'u' and 's' values to ensure that changes in task weights correspond to changes in task consistency. The impact of these two hyperparameters can be likened to temperature coefficients. Higher values of 'u' and lower values of 's' signify a faster decrease in task weight. Detailed configurations for 'u' and 's' are provided in Appendices L874 and L877, respectively. Specifically, 's' was set to 5000 for the ASR task and 10,000 for the MT task. 'u' was updated every 5000 steps to adjust the weights.\n\n2. **The effect of proposed method on the consistency is not shown and the authors do not explain why their method could work.**\n\n    We appreciate the opportunity to elucidate how our proposed methods affect consistency and why they yield effective results. Note that our proposed method is inspired by our observation and the subsequent analyses, thus some changes in consistency have shown in the analysis section, such as the impact of removing the ASR task from the decoder, as depicted in Figure 4. Regarding why our method effectively resolves the identified problem, we have indeed provided explanations:\n\n   - **Impact of LBM on Consistency**: As shown in Table 3, the introduction of the LBM enhances the frame length by approximately 3%, leading to an improvement in performance. Specifically addressing consistency, Figure 5, labeled as \"Length,\" illustrates that the shrinking method displays better consistency in the decoder.\n\n   - **Role of L2G in Task Representation**: The Local-to-Global (L2G) method aims to bridge the gap in task representations between ST and MT. While the impact on consistency was initially elusive, we applied an information entropy metric for a more nuanced evaluation. Figure 10 (left) reveals that the L2G and L2G-KD methods result in less disparity in task representations compared to traditional fixed window methods.\n\n   - **Impact of Task Weight Reduction**: We have provided the training perplexity (PPL) as a measure of model performance after applying task weight adjustments. These findings could be found in Appendix D, confirm that our method achieves a faster convergence rate.\n\n    We believe the aforementioned explanations adequately elucidate the efficacy of our method, and we will supply changes in consistency after employing the complete method in the subsequent version.\n\n3. Some of the claims are not fully supported by their approach.\n\n    - **The approach (equations 8 and 9) seems to leave task consistency unaddressed**: Equation 9 takes consistency into account, and in L425, 'u' and 's' represent the coefficients which are formulated based on the consistency alteration. Additionally, we can adjust the 'u' and 's' values to control the rate of task weight reduction. For instance, Figure 7 illustrates the rapid decrease in consistency for the ASR task. In response, we will enhance the update frequency accordingly. Consequently, the weight of the ASR task adheres to the evolving consistency patterns of the ASR task.\n\n    - **Adjusting the weights of different auxiliary tasks can not reduce computation**: Regarding the reason for the efficacy of weight adjustment in expediting the training process, we specify in L879 that we cease training a task if its weight drops below 0.1. This approach prevents needless training. As illustrated in Figure 10 (right), we halt training the ASR task after 35,000 steps. Further, Figure 11 in Appendix D shows a faster convergence speed.\n\n    We suppose that these issues might have arisen due to moving the experiment settings to the Appendix section. To address this, we intend to provide additional descriptions in the main content to ensure our claims are self-contained.\n\n### Response to Questions:\n\nWe will add sufficient explanations and make the Figures easy to understand for readers. If you have any further questions about our work, please feel free to contact us during the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The approach to investigate consistency between different tasks provides insights into the varied utilities of auxiliary tasks and helps identify potential areas of interest.\n- Their proposed approaches, including the improved shrinking method, the local-to-global training method, and consistency/impact-based weighting, appear to be reasonable extensions and have been found to improve performance.\n- The overall performance on MuST-C is decent.", "reasons_to_reject": "- The writing needs improvement to provide a clear description of their proposed approaches. In its current form, it is not evident how their approach works and how to replicate the results. For example, how does the L2G extractor integrate with the Transformer and what are the configurations? What are \"u\" and \"s\" in equation 9?\n- While the investigation of consistency motivates most of the proposed improvements, it is not clear how these methods affect or improve consistency, as no results were presented. Partially due to the writing, it is also not apparent why these proposed methods can address the identified problems.\n- Some of the claims are not fully supported by their approach, partially due to writing. For instance, the authors suggest that it is necessary to consider both consistency and task impact in determining the weighting for the auxiliary task. However, their approach (equations 8 and 9) seems to only take into account the task impact, leaving task consistency unaddressed. Additionally, the authors claim that their approach can avoid unnecessary computation resources when the auxiliary tasks are not beneficial; however, their weight approach appears only to adjust the weights of different auxiliary tasks and does not reduce computation on these tasks.", "questions_to_authors": "I recommend providing more descriptive explanations for the Figures so that they are self-contained and easier to read.", "paper_topic_and_main_contributions": "This paper investigates methods to enhance multi-task learning for end-to-end speech translation. The authors propose using cosine similarity between the gradient vectors computed from different tasks as a measure of consistency between tasks and present the following findings:\n\na) The ASR and ST tasks show high consistency with respect to the gradients at the acoustic encoder, but exhibit little consistency at the text encoder and decoder. Consequently, the authors suggest employing only the ASR task for training the acoustic encoder.\n\nb) The consistency level between MT and ST is relatively low in both the text encoder and decoder, with the discrepancy being more pronounced at the text encoder. The authors hypothesize that the length discrepancy between audio features and text token sequences, along with the representation gap between the two, are sources of concern. They propose two approaches: one is a modified version of shrinking, aimed not to lose information, and the other involves adding speech-like noise (blanks and repetitions) to the text sequence, along with a local-to-global convolution approach to progressively utilize larger context windows.\n\nc) The consistency between ST and the auxiliary tasks changes over the training process. The authors propose a task weighting strategy to adjust the importance of auxiliary tasks based on their consistency with the ST task, as well as the impact of the training gradient, removing the necessity of a separate fine-tuning step.\n\nExperiments conducted on MuST-C demonstrate that the proposed approach achieves a 0.9 BLEU improvement over the ConST baseline, averaged over the 8 language pairs in MuST-C.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the impact of auxiliary tasks (ASR, MT) on speech translation in the multi-task learning (MTL) setup. The authors presents in-depth analysis on the interaction between tasks and identifies issues causing degradation. The authors propose an improved MTL approach with a number of techniques (looking-back mechanism based on CTC output, local-to-global strategy to insert audio-like noises, dynamic adjustment based on task impact).

The paper is well positioned, and all reviewers enjoy the comprehensive and systematic analyses for deeper understanding of how each auxiliary task benefits or harms the primary task. The proposed improved MTL approach are also evaluated extensively and show favorable performance over many baselines. Authors have addressed most of the comments regarding experimental setups and positioning of the paper compared to the literature. I recommend the authors to incorporate the feedback from the reviewers into the manuscript.
Decision: Accept-Main