Objective Soups: Multilingual Multi-Task Modeling for Speech Processing
Authors: A F M Saif
Lisha Chen
Xiaodong Cui
Songtao Lu
Brian Kingsbury
Tianyi Chen
Venue: NeurIPS 2025 poster ()
Keywords: Multilingual speech recognition
automatic speech translation
multi-objective optimization
conflict-aware training
Primary area: applications
TL;DR: This paper proposes conflict-aware multi-objective training strategies for multilingual automatic speech recognition and speech translation by selectively aligning gradients from conflicting layers to enhance efficiency and performance.

Abstract: The need for training multilingual multi-task speech processing (MSP) models that perform both automatic speech recognition and speech-to-text translation is increasingly evident. However, a significant challenge arises from the conflicts among multiple objectives when using a single model. Multi-objective optimization can address this challenge by facilitating the optimization of multiple conflicting objectives and aligning the gradient updates in a common descent direction. While multi-objective optimization helps avoid conflicting gradient updates, a critical issue is that when there are many objectives, such as in MSP, it is often {\em difficult to find} a common descent direction. This leads to an important question: Is it more effective to separate highly conflicting objectives into different optimization levels or to keep them in a single level? To address this question, this paper investigates three multi-objective MSP formulations, which we refer to as \textbf{objective soup recipes}. These formulations apply multi-objective optimization at different optimization levels to mitigate potential conflicts among all objectives. To keep computation and memory overhead low, we incorporate a lightweight layer‑selection strategy that detects the most conflicting layers and uses only their gradients when computing the conflict‑avoidance direction. We conduct an extensive investigation using the CoVoST v2 dataset for combined multilingual ASR and ST tasks, along with the LibriSpeech and AISHELL-1 datasets for multilingual ASR, to identify highly conflicting objectives and determine the most effective training recipe among the three proposed multi-objective optimization algorithms.

Objective Soups: Multilingual Multi-Task Modeling for Speech Processing
M Saif!
Lisha Chen
Xiaodong Cui Songtao Lu
Brian Kingsbury² Tianyi Chen 1,5
Rensselaer Polytechnic Institute IBB Research
The Chinese University of Hong Kong Cornell Tech
University of Rochester saiiafrri.eer edu, i.iish wwwn@rochester.cooooo edu,
"[cuix, bedk) ibm. com, 3 "stluCcse, cuhk, edu. hk, 1,5 @heetiaany118@maill com
Abstract
Training single model for multilingual, multi-task speech processing (MSP) is severely hampered by conflicting objectives between tasks like speech recogni- tion and translation. While multi-objective optimization (MOO) aims to align gradient updates, its effectiveness diminishes as the number of tasks grows, mak- ing it difficult to find common descent direction. This raises fundamental question: should highly conflicting objectives be optimized jointly or separated into a hierarchical structure? To address this question, this paper investigates three multi-objective MSP formulations, which we refer to as objective soup recipes. These formulations apply multi-objective optimization different op- timization levels to mitigate potential conflicts among all objectives. To ensure efficiency, we introduce a lightweight layer-selection mechanism that computes the conflict-avoiding gradient using only the most problematic layers. minimiz- ing computational and memory overhead. Extensive experiments on CoVoST v2, LibriSpeech, and AISHELL-- reveal that a bi-level recipe separating recog- nition and translation tasks consistently outperforms standard flat optimization. Our work demonstrates that hierarchical MOO is a more effective and scalable approach for building state-of-the-art MSP models. Our code has been released at https://githine gigtth........ com http://aaaaaffboccaiaaas Soups
Introduction
Multilingual speech processing the backbone of voice-driven applications, from virtual assistants to voice search [25 Modern deployments increasingly demand a single model that can (1) transcribe
speech in many languages, and (2) perform an auxiliary speech-to-texx translation task [55] 8).
Unifying these capabilities simplifies maintenance and reduces inference costs, yet joint training remains difficult due to language diversity, task heterogeneity, and model complexity [30 21 46
A common approach is to introduce one loss term for each task or requirement, for example, self-supervised learning loss to learn robust representations across languages [41] 48
29].
language-specific Connectionist Temporal Classification (CTC) loss for transcription [34] cross- entropy for translation (60), and fairness constraints for underrepresented languages 100 [1] and then optimize them simultaneously. We refer to this methodology objective soups, where the idea is to leverage multiple objectives to learn a single model that satisfies the above requirements. A related line of work is rewarded soups 44, where the goal is to achieve multi-reward alignment for model by interpolating weights fine-tuned on diverse reward objectives. Although simple and efficient, such
"his work was conducted when L. Chen and T. Chen were at Rensselaer Polytechnic Institute, and when Lu was at IBM Research, United States. The work was supported by IBM through the IBM Rensselaer Future of Computing Research Collaboration.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

soups could suffer from gradient confticts: improving one objective can actively harm the others, stalling overall training. oulttvobjective optimization provides a principled approach for handling conflicting training objectives 20
37 36 19] We propose a hybrid multi-level and multi-objective optimization algorithm tailored to MSP: in our formulation, the self-supervised loss serves as lower- level constraint that drives the model toward language-agnonn aggnosicc acoustic representations. We first optimize this lower level, then feed its solution into the upper-level supervised snrectvv-coomining CTC for transcription and cross-entropy for translation- by adding a penalty term. Allowing the representation-orienn
self-supervised objective to converge first yields a more stable optimization path than a flat objective soup, and this bilevel structure naturally extends to additional hierarchies (e.g., grouping objectives by tasks or languages) [49]
Our findings and contributions. We propose three optimization recipes: i) VS-MSP: a single-level vecto-optimizztiion approach; ii) VC-MSP: a bilevel approach that places self-supervised learning in the lower level and supervised fine-tuning in the upper level; and, iii) VM-MSP: a multilevel approach that organizes objectives hierarchically by tasks or by languages, so that the most conflicting objectives are separated across levels. Experiments on Co v2, LibriSpeech, and AISHELL datasets, covering model sizes from 58M to 150M parameters, reveal the following patterns:
Fl: Multi-objective optimization reduces gradient conflicts and boosts performance. VC- MSP outperforms the classic pre-training and fine-tuning baseline by 9.8% in terms of WER and 8.6% in terms of BLEU.
F2: Introducing hierarchies across objectives further improves accuracy. The multi-level approach VM-MSP outperforms VC-MSP, delivering 4.2% WER and 2.8% BLEU gains.
F3: Task-based hierarchies outperform language-based ones. Separating speech recognition and translation at different levels yields larger gains than separating by language, yielding up to 7.4% in terms of WER and 8.5% in terms of BLEU improvements over baselines.
F4: Layer-wise conffict pruning reduces overhead without losing accuracy. We select layers whose gradients have negative cosine similarity. and compute the conflict-avoidant dynamic update directior oonly on this subset of gradients. By pruning layers with aligned gradients, we reduce up to 17% in time and 1 18% in memory, without degrading WER or BLEU.
From these findings, we conclude that multiohjective optimization, with carefully defined optimiza- tion hierarchy and a lightweight layer-selection mechanism, offers an effective and practical recipe for multilingual multi-task speech recognition and translation.
2 Related Work
In this section, we briefly review related work on multi-objectivee optimization, multilingual speech recognition, and translation; see also Appendix [ for additional related work.
Multi-objective optimization. Classical multi- mujti-objeectiv optimization methods focus on optimizing multiple conflicting objectives, with approaches like scalarization and lexicographic ordering B9 9 19]. In speech processing, multi-objective optimization has been applied at the system level to balance accuracy and model size [40]. In contrast, our work addresses objective conflict at the training level, optimizing multiple loss functions jointly within shared multilingual model.
38 51 36
Multilingual speech recognition and translation. Earlier approaches in multilingual speech recognition relied on deep neural networks, hidden Markov models, and LSTM models, gradually progressing to Seq2Seq and transformer-based architectures [27] 24 58 6). While significant advancements have been made, challenges remain in enabling mulliitasking and reducing conflicts across objectives-a gap that this paper addresses.
Multi-task learning for speech. Multi-task learning for speech recognition and translation tasks has seen limited exploration but includes methods such as dual encoder architectures (33) and large-scale multitask models like Whisper [43] Recent work, such as Mu² SLAM [3), incorporates cross-modality learning, while others use joint pre-training and fine-tuning 55) 47). However, most methods rely on static weighting strategies, which do not adequately address confficting objectives.
This paper investigates conflicting objectives in MSP and proposes three algorithms to mitigate these conflicts. Our approach demonstrates a significant improvement over baseline methods, highlighting the effécctivnness of multi-objective optimization in multilingual MSP tasks.
Definition of the conflict-avoidant dynamic update direction is provided in Section

---

3 Unifying Multi-Objective Optimization Training Methods
In this section, we introduce multi-objective optimization, its optimality condition, discuss three problem formulations, and present the corresponding algorithms to solve these problems.
3.1 Multi-objective optimization: primer
Multi-objective optimization aims to learn a model that simultaneously optimizes multiple conflicting
objectives, which can represent different tasks or learning metrics [36 9 Let Q E IR' denote the model parameter. Given M objectives with each denoted fn (+), for E [M], the general multi-objective optimization problem is to solve
min BE3*
(()) = (0) (x)
Since optimizing multiple objectives simultancoosly is often challenging, understanding the notion of conflict is crucial, which is defined below.
Definition 3.1 (Gradient conflict). For any pair i, j E [M], let (+))) and ()(@) be the loss functions for two different languages or MSP tasks, parameterized by θ. We say gradient conflict exists cos( Vel, Vel,) < O, where cos(.) is the cosine similarity function.
The definition of gradient conflict indicates that improving one objective along its gradient degrades the other, necessitating trade-offs for balanced optimization. Next we introduce the necessary optimality condition for multi-objective optimization.
Definition 3.2 (Parcto stationary). A model θ is Pareto stationary if there exists À E A = {)
RM 1 x 1, À 2 0} such that VL(@)> = 0, which is equivalent to minxea |V = O.
We denote the model parameters by @ == {8, Φ), where 0 is the pa- rameter of the backbone and ø is for a seguuage/tss-deeendded see an illustration in Figure 2 Be- fore formulating the multi-objective problems, we specify the objectives.
Objective given L: (vector of objectives)
Recipe VS-MSP: min (supervised objectives, self- objective)
layer;
min (supervised objectives)
Recipe 2: VC-MSP:
self-supervised objective
Objectives of self-supervised and su- pervised training. For pre-training shared backbone parameters 0, we em- ploy the Contrastive Predictive Cod- ing (CPC) loss (o) [11] for self- supervised training to learn good language-agnostic acoustic features. For language and task specific parameters di,n, use su-
min (supervised muiti-lingua objectives for task 1) min (supervised multi lingual objectives for task self-sueeviiee objective
Recipe 3: VM-MSP:
Figure 1: An overview of the objective soup and the three recipes.
pervised loss, (0, di,n), where E [T] and E [N] represent different languages and tasks,
respectively. This can be either the CTC loss (ete(O, ④) 2 for transcription and the cross-entropy
loss (0, Φ) for translation.
A set of self-supervised losses and multiple supervised losses form the objectives for multilingual MSP. The use of eef-ssueervieed loss during the supervised training restricts the search region to a neighborhood of models with good representation capability that continues to intersect the Pareto set [6). This neighborhood refers to the sublevel set Rs := {0 (, < +0) } of the self-supervised loss around its minimum t := ming (0). which keeps the encoder's invariances intact. Opera- tionally, we not project onto R.s. Instead, our update uses a penalty term v (l.. - -tt) to enforce the constraint and bias the iterates to remain near R.s. This corresponds to the e-constraint viewpoint in multi-objective optimization (39] $3.2], where increasing approximates restricting the feasible set to Rs. We then jointly optimize the self-supervised and supervised multi-lingua multi-task losses
Confonmer
Clasiifcation
English language
$
output
SETT
Labeled
earpet
S2TT
Unlabeled data
enrpet
Shared model parometers g
Taskoipecfic parameters dy.
Catalon lang-gge
Figure 2: Multi-head conformer model.
3

---

using the proposed objective soup approach. Specifically, we propose three different recipes that use different hierarchical modeling of the objectives, placing different objectives at different optimization levels. See Figure 1 and a detailed discussion thereafter.
Note that for MSP, we can represent all the objectives vector ((O) containing both self-supervised loss and supervised losses from different languages and tasks such as multilingual speech recognition and translation, where θ == {0, $1.1, @T.N J: that is, [(O) =- [La (0, $1.1)..... (0, OT, }].
To encourage Pareto stationarity for objectives different MSP variants, we can employ either static or dynamic weighting multi-objective optimization methods.
Limitation of static weighting, In static weighting, we optimize the (weighted) average of the multiple objectives [32 [59]. This method is simple but may suffer from conflicting objectives where gradients have conflicting directions. For instance, considering (+) = (0, or,,) + ((, and Le ( L,(0, (e) ) + nln(() (0) two objectives having conflicting directions, (t, t') E (T) and (n,n) E [ then (Vole, (O), Voly < O.
Proposed dynamic weighting, To avoid conflicting directions, we can employ dynamic weighting method that uses dynamically weighted gradients from individual objectives to avoid conflict and enables optimization in conflict-vvoiding (CA) direction 19]. Specifically, a CA direction d is the steepest common descent direction that maximizes the worst descent, given by
d()) arg max min - (VL( d) d ΛεΛNT
||d|?
By reformulation, such a direction is equal to dynamically weighted gradients of different objec- tives ()) given by d(O)= VZ(()) (0) with weight (0) computed by
(*)))= = arg min
||V[())1]
(3)
AEANT
However, finding the true gradients of V (()) is computationally expensive. Hence, we employ stochastic variant of MGDA, the MoDo algorithm (9), which obtains an unbiased stochastic gradient estimate for (3) via a double sampling technique.
At each iteration k, denote tt and & as two independent samples from the labeled dataset D, and Ve(gh; @*) and VE(S,; O*) as the stochastic gradients. We leverage the MoDo update in [9] by
h 1 1ANNT NT - >>*~>*(V @*) T VV A*))
where * is the step size, IA denotes the projection to A see a summary Table
Having introduced the basics of multi-objective optimization, we formulate the three multi-objectivee MSP problems and define their corresponding parameter update rules. For brevity, we consider only the three objectives mentioned above; the generalized formulation is provided in the Appendix B
3.2 Vectorized single-level MSP (VS-MSP)
In this formulation, treat all objectives as a single-level vectorized objective without any lower- level constraints. Hence, the problem formulation is:
min ((0)= : (e 11,1). Lce (0, 212)) (ot OT, 1), Lce Lce (1 (n)))
(5)
@ER
language
Thth language tasks
For single vectorized objective training, we can optimize the vectorized objectives using Algorithm | in Appendix B where the shared backbone parameters are updated using
g - gl Σ Viceteoolcst (oletc ot.)) Σ X Volca 0i.2) ax Volu(())
(6)
where a>> O is the learning rate for the backbone parameters, and X and X Xr.ce. which represent the dynamic weights for speech recognition and translation objectives, are computed using the MoDo algorithm [9] Here, Xe is the dynamic update direction for the self-supervised objective, L. with
EL Me + ΣT Xf. + X 1. Similarly, by taking the gradients of each supervised objective with respect to task speciffc output heads, the task-specific parameters are updated via
01,1
c+1 ot,, - V fetee 8*)) and o k++1 = oi - SV (0)
(7)
where 8> 0 is the learning rate of the task task-specific parameter.

---

3.3 Vectorized objectives with lower-level constraint for MSP (VC-MSP)
To mitigate the challenge of conflicting objectives and reduce the search space for an optimal Pareto stationary point, incorporating a suitable lower-level constraint, (f), can be beneficial [39]. However, (,( must satisfy an essential property: its gradient update direction should exhibit minimal conflict with the upper-level objectives. When such alignment holds, the constraint restricts neighborhood that still intersects the Pareto set
the feasible region to a representation-preseeeee and admits a common descent direction for gradient-based solvers. If the lower-level objectives are strongly conflicting, the constraint does not aid optimization. In this context, we incorporate the supe-suee loss a lower-level constraint, it exhibits this desirable property (see Appendix E). This approach helps align the gradient directions and maintain a feasible optimization region, ultimately enhancing overall performance. By constraining the self self-supervised loss to be smaller than a threshold E, our VC-MSP method can be formulated as
min 8E3
((0)= == lete (0, (0, (1.2) (oc
(0r,)))) y leo (0, 2122)
(8a)
1sst language with 2 tasks
T. language with 2 tasks
s.t. (0) min (u))") Se
(8b)
This formulation minimizes the vector of supervised losses L()) subject to the constraint & (0) S where f := ming (0). The e-constraint defines the feasible region for the supervised objectives. We optimize this constrained problem via its penalized first-order realization (see Eq. (15). which targets Pareto-stationan solutions for the constrained setting- -i.., limit points at which no feasible common descent direction exists that decreases all supervised objectives simultaneously 39). Em- pirically, placing the self supervise objective the lower level and ASR/tranllttion at the upper level yields the best validation performance in our setting. This observation motivates the VC-MSP algorithm: separate the self self-supervised objective from the supervised ones and optimize them at the lower and upper levels, respectively, using a single training loop with a penalty schedule.
To train model using the VC-MSP algorithm, the backbone 0 is updated via
g++1 = 8h
Volcte oi
Σ Mi.co Volco ((t2) - on Voln
where Σ 1 f,00
Et= 1.00 = 1. To update task-specific heads, we use 7 see Algorithm 2
3.4 Vectorized multilevel MSP (VM-MSP)
Building upon the VC-MSP formulation, we introduce VM-MSP, multilevel MSP algorithm. With VM-MSP, we aim to explore whether extending our VC-MSP algo- rithm into multi-level opti- mization based on tasks and languages offers advantages and mitigates the risk of being trapped in sub-optimal Pareto stationary points. In multi- level optimization, it follows hierarchical structure, with decisions made at different levels within the hierarchy. The problem formulation for multilevel MSP optimization can be expressed follows:
Method
Backhone update role
VS-MSP gh++1 gk ca
Valn
(8) Volu
VC-MSP gh+1 00
Voln
VMLMSP ghl -k
m Σ T.EE
X Vol.
1
Table 1: The red boxes represent updates for the objectives with gradient conflict mitigation, the hlue box represents the update for the supervised objective as the penalty terms, the teal boxes represent updates for the self- supervised objective as penalty term.
min
Lave (11 $2 0)
4 E 45.0
8.L. $
argmin Loo(th Φ2 0) 0
(10)
s.t. (u(0)
min L, (0)) S 0'
5

---

where 11 := .r.1) and Φ2 := 0r,2)
In VM-MSP, training is performed multiple levels with feedback across different levels. Specifically, we update the backbone parameters f via
T
gk+l 6 a Σ N M,ere Volcte(0* of da + m Σ
Me Volce
&i + +VVolu(0%) (11)
where 77, and are penalty parameters, with 7/2 controlling the relative influence of the translation loss with respect to the ASR loss, and 7) controlling the influence of the self- self-supervised loss lower-level constraint in the multilevel optimization. Here, E At,ete
and Σ Ar,c = 1. We update task-specific classification parameters using (7): see a summary in Algorithm of Appendix
Note that in this formulation, the unsupervised loss is optimized first, followed by the translation objective, and then the ASR objective. We can also change the optimization order; the unsupervised objective is optimized first, followed by the ASR objective, and then translation.
Remark 3.3. For multilevel optimization, objectives are prioritized based on their importance. In VM- MSP, this includes task-based and language-bass optimization experiments with speech recognition and translation, alternating their primary and secondary levels. Language-based multilevel optimization involves English (LibriSpeech) and Chinese (AISHELL), alternating their primary and secondary levels.
multilevel optimization. Task-base multilevel
To update the backbone parameters, 0, and task-specific parameters, Φ, in the three algorithms, we use first-order gradient-based updates. A summary is provided in Table [ with detailed descriptions and derivations deferred to Appendix B
4 Experimental Results and Findings
In this section, compare our three proposed MSP algorithms with several baselines: two-stage training (pre-training then fine-tuning), static weighting (fine-tuning with tuned loss weights), and joint (bilevel) training. Our goal is to identify the most effective method for resolving conflicting objectives, thereby avoiding the risk of the model getting stuck in a suboptimal Pareto stationary point. We analyze speech recognition and translation performance in multilingual setup using the CoVoST 2 dataset, selecting five languages for speech recognition (English (En), French (Fr), German (De), Spanish (Es), Catalan (Ca)) and four for translation (Fr, De, Es, Ca). Additionally, we conduct experiments with combination of the LibriSpeech and AISHELL datasets. Our results consistently demonstrate that our approaches outperform the baselines, confirming their effectiveness in achieving superior speech recognition and translation performance.
Models and hyper-parameters: In our experiments, we evaluate three encoder-decoder architec- tures: Conformer + Transformer decoder. Conformer 266 A uses 12 Conformer blocks with a model dimension of d 612 and H 12 attention heads (head size 51); Conformer B uses Conformer blocks with d = 512 and H = 8 heads (head size 64). Both employ convolutional kernel of size 31. Each model includes a speech recognition classification head consisting of a dropout layer (rate 6 = 0.1) followed by linear projection to the speech recognition vocabulary size VASR, producing logits that are trained with the CTC loss. The translation component is a standard Transformer decoder with L. = 3 encoder layers and La = 3 decoder layers, model dimension d matching the Conformer encoder (612 for A, 512 for B),
12 or attention heads, forward dimension dir = 2048, dropout rate 8 = 0.1, and learned token embeddings of size d. The output logits over the translation vocabulary VSST are trained with cross-entropy loss.
Whisper-medium, We adopt the medium-sized Whisper model 43 (24 encoder layers, 12 decoder layers, = 1024, H 16, and feed-forward dimension of 4096). Its encoder processes input audio into hidden representations, while its decoder performs autoregressive decoding for both speech recognition and translation tasks, trained with cross-entropy loss over the output vocabulary.
All models are trained with the W optimizer, using a backbone learning rate of a = 5 x 10-5 and head/decoder learning rate of B 5 x 10-4. For VC-MSP, the bilevel penalty is initialized to = 0 and increased by 0.02 each epoch. For VM-MSP, we set 77 = 0.1 and 172 = 0, with each being incremented by 0.02 per epoch.
Training time and memory complexity: Table 5 reports the GPU memory footprint and per- epoch training time for the baseline two-stage pipeline and our MSP variants. Introducing dynamic

---

Table 2: Speech recognition and translation on Vo using Conformer models comparing two-stage, two-stage static, joint training, VS-MSP, VC-MSP, and VM-MSP.
Param
Lang
Two-stage training
Twoostage stalic
training
MSP
VS-
MSP
VM-MSSP UAS
USA
VM-MSP (Eificsent)
22.2%
22.2%
21.3%
21.15
20.6%
19.7%
19.9%
19.6%
Fr
23.8%
23.01%
22.9%
23.11%
22.6%
21.8%
22.1%
21.7%
Speec necegnition (WFR () (150 M)
16.7%
16.0%
16.015
15.3%
14.66%
14,9%
14.7%
Ca
21.2%
21.0%
20.5%
20.31
19.8%
19. 19.1%
19.3%
19.3%
19.75
188.4%
27.9%
Fr
26.11
20.0%%
25.3%
25.11%
24.3%
2.4.1%
24.35%
24.35
Speech recognition (WER 4)
De
22.5%
22.65%
22.0%
22.25%
21.65%
21. 22%
21,59
996
22.6%%
25.2%
24.8%
233.
27.2
28.3
29.4
26.9
27.9
27.7
28.2
28.5
28.5
Trandlation (150
Ea-
79.1
20.3
30.2
30.0
50.9
31.3
31.6
31.5
Ca-+En
22.7
22.6
23.8
23.6
24.5
25.4
25.8
25.6
Ave.
26.5
26.6
27.5
27.4
28.1
28.7
28.9
28.8
Fr-+Ein
24.2
23.5
24.1
23.9
25.8
26.5
36.6
26.4
De-+En En
22.4
22.6
23.3
23.0
23.9
24.5
24.7
26.4
Translation (BLEU EU (83 M)
Ca-+En
19.7
19.5
20.2
20.5
20.8
21.3
21.6
21.4
Table 3: Speech recognition and translation CoVoST-2 and Whisper-medium model comparing two-stage. two-stage static, joint training, VS-MSP, VC-MSP, and VM-MSP.
Tisk
Lang/
Two-stage tmining
Two stage
Joint training
VM-MSP UAS
VM-MSP
Lang Eag
stati
MSP
MSP
USA
(Emicicut)
En
15.9%
15.25
15.6%
14.9%
14.3%
14.6%
14.3%
Fr
21.7%
21.7%
21.11
21.5%
20.6%
20.1%
20.4%
20.2%
Spezch recognition (WER))
De
10.4%
10.2%
9.3%
10.1%
8.9%
8.2%
3.5%
8.0%
13.5%
12.8%
12.2%
Ca
17.5%
17.3%
16.7%
16.9%
10.0%
15.9%
16.2%
16.0%
14.7%
De
33.2
33.4
24.2
34.0
34.7
35.1
35.4
35.2
Translation (BLBU 1>
Bs-+ En
37.3
37.4
37.9
38.2
38.6
39.0
39.1
39.0
29.5
30.3
30.9
Ave.
33.1
33.3
33.9
34.0
34.5
34.9
35.2
35.1
weighting increases memory usage from 12.5 GB 14.7 GB (~18%) and extends each epoch by 0.6 hours, from 3.5 to 4.1 h (~17%), due to additional gradient computations. contrast, the efficient MSP variant-which restricts dynamic weight calculations
only the layers exhibiting gradient conflicts-ling overhead to just 0.3 GB and 0.2 h per epoch while preserving performance (see Appendix D). Moreover, our MSP framework scales gracefully with more tasks, reducing deployment resource demands by consolidating all objectives into a single model (see Appendix IT).
Based experiments, summarize observations in the following sections.
4.1 Conflicting speech recognition and translation objectives
Presence of multiple conflicting objectives degrades the model's performance. In this section, we investigate the effect of conflicting objectives on model performance using two algorithmic settings mne-training-fie-ttniin present the cosine similarities between the supervised gradient vectors for five languages and the self-supervised gradient used in speech recognition and translation. The accompanying heat map visualizes these similarity scores, highlighting which objectives align strongly (high similarity) and which diverge. As shown, objectives with lower similarity measures tend to conflict more intensely, whereas the self-supervised gradients exhibit markedly higher alignment with the other objectives. We further investigate the presence of conflicting objectives in MSP in Appendix B
and VS-MSP across different model sizes. In Figure 3 we
7

---

Table 4: Speech recognition and translation VoS-2 using Conformer model with penalty increase rates of 0.002 0.02 per epoch.
Task
Lang Lang Eing
Two-stage training
UAS (UR=.02)
VM-MSP USA (IR=002)
UAS (IR=0002)
USA (IR-0002)
En
28.6%
26.9%
27.2%
28.9%
26.3%
Speech recognitio M) (WER L)
26.1%
24.1%
24.3%
26.4%
23.8%
Do
22.5%
21.2%
21.5%
22.8%
20.5%
Ca
25.2%
23.8%
24.198
25.3%
23.2%
Ave.
23.6%
23.9%
23.1%
Pr-+Bi
Transhaiion (BLEU 1)
De
22.4
24.5
24.7
25.3
22.8
Ea
24.6
36.1
26.5
26.7
25.3
21.3
21.6
20.1
Ave. 22.7
24.6
24.9
25.3
23.3
As shown in Table Table the VS-MSP method consistently outperforms the two-stage method. The key difference between these two approaches the use of multi-objective optimization. This result suggests the presence of conflicts among the speech recognition and translation objectives and highlights the effectiveness of object-b optimization in addressing these conflicts.
4.2 Enhancing performance with multilevel optimization
Multilevel optimization significantly improves MSP performance by effectively balancing learning objectives and narrowing the search for optimal Pareto stationary points. This section examines the impact of multilevel optimization MSP performance. We tested two optimization sequences: UAS (uple-sue - speech recognition - translation) and USA selff-uuerrvied - translation recognition). In the UAS sequence, the unsupervised loss is optimized first, followed by the ASR objective, and then the translation objective. Conversely, for the USA sequence, the unsupervised objective is optimized first, followed by the translation objective and then the ASR one.
SPEECH RECOGNITION: Tables and show that VM-MSP consistently achieves the low- est WER across languages and model sizes. On the 150 M-parameter Conformer, VM-MSP (USA) reduces WER by 10.2% versus two-stage training and by 4.2% versus VC-MSP. Likewise, on Whisper, VM-MSP (UAS) outperforms two- stage training by 1.3% and VC-MSP by 4.1% These gains confirm the effectiveness of VM- MSP's multilevel optimization.
En
S.COSS 0.03
5,033
0.11
3.23
0.14
0.17
De
4.01
4.314
4:071
4.97
4.25
36
0.033
4.72
0.513
6.1
4.23
ProtEn
6.11
0.90
4:25
4.14
4.077
A.R1
4.044
0
TRANSLATION: Tables and VM-MSP has the highest BLEU scores. On the 150 M- parameter Conformer, VM-MSP (USA) boosts bLEU 9.1 % over two-stage training and 2.8% over VC-MSP. Whisper, gains 6.4% over two-stage and 2.0% over VC-MSP. These results underscore VM-MSP's robustness in translation tasks. We observe similar performance gains with the smaller Conformer model and other model
Es->En
0.29
4:20
4.22
0.01
4.12
4.010
4.027
0.91
621
6402
6425
8427
641
6444
Uroup
di
Figure 3: Heat-map of Cosine similarities among ASR and speech translation objectives.
4.3
Optimization order multilevel optimization
The order of optimization impacts MSP accuracy Multilevel optimization. this section, we investigate the significance of optimization order in multilevel optimization for MSP. By comparing the performance of different MSP algorithms under varying optimization sequences (UAS and USA), we aim to elucidate how the order of optimization affects MSP performance.
From the results in Table and Table 3 yields superior recognition performance compared to USA. This finding indicates the importance of
we observe that the UAS optimization sequence consistently
Additional results using BEST-RQ and Wav2Vec2 presented in Appendix
H (Tbles

---

Cosine Value
Cosine Value
LA
Cosine
Encodertayyo
Encadertayes [yssertayys
Encudertayel
Encodertaye
Encodentayed Layee
Encedartayye Layee
Escedertayee
Encolerlayee
Encodertayyee
Encodertaye
Encodertay Racoderlayr
Encodent Laaoeer--arrr Lanu3
dncodertaye Lave
Recoderlaye Layee
Encedertayee Layeer
Enceddrt.ayee Layer
Incoderlayed Layyyyy
Encodert.sy Lyye
Encodestayel Layer
Encodertayel Loreeetaayees
Encedentayet
Encedert.ayer
Model Layer
Model Layer
Model Layer
(a) Epoch
Epoch
Epoch
Figure 4: During two-stage training (without multi-objective optimization), the Conformer model exhibits persistent gradient conflicts in the same layers.
prioritizing certain objectives during the training process and highlights the erucial role of optimization order when designing multilevel optimization algorithms for MSP.
4.4 Effect of penalty parameter
Penalty parameters play a crucial role in multilevel MSP training. In penalty- based multilevel problems, selecting the appropriate penalty parameter is crucial. These methods prioritize upper-level ob- jectives while controlling lower-level ob- jectives through penalty term. Using a smaller penalty parameter can weaken con- straint enforcement, causing suboptimal lower-level performance, slower convergence, and imbalanced optimization [4] This is evident in our MSP experiments. We further conducted experiments following the same training procedure other simulations, using 83M parameter model with two different penalty parameter increase rates. A lower increase rate of 0.002, capped at 1.5, resulted in worse WER for lower-level tasks, as shown in Table 4 Given the equal importance of speech recognition and translation objectives in our study, we applied a larger penalty parameter increase rate of 0.02 for the lower levels, with a final value capped at 1.5. This adjustment improved lower-level performance but slightly degraded upper-level performance. Therefore, selecting the penalty parameter requires careful consideration of the trade-offs between upper- and lower-level priorities. A detailed explanation of this selection process is provided in Appendix H
Table 5: Resource comparison per epoch
GPU Memory (GB)
Time (h/epoch)
Model
Two-stage
12.5
3.5
MSP (dynamic weighting)
14.7
4.1
MSP (Efficient)
12.8
3.7
4.5 Our observations across different model sizes
Our observations are consistent across different model sizes. We assess the consistency of our observations across different model sizes. Results from Tables 2 and confirm the reliability and generalizability of our findings, offering insights for scalable ASR system design.
SPEECH RECOGNITION, From Tables and B observed that the two-stage approach achieved competitive performance across all languages. The VS-MSP method consistently outperformed the two-stage method, and the VC-MSP model demonstrated even better performance. The most notable finding, however, is the performance of the VM-MSP, which exhibited significant improvements. Specifically, the VM-MSP model optimized with the UAS objective sequence achieved the lowest average WER, demonstrating its effectiveness in leveraging unlabeled data. These observations hold true for both the Whisper and Wav2Vec2 models.
TRANSLATION. Tables 2 and 3 illustrate the translation comparison for different models. Similar to speech recognition findings, the two-stage approach demonstrated competitive performance across all language pairs. The VS-MSP model consistently outperformed other algorithms in the translation task. Interestingly, the VM-MSP model optimized with the USA objective sequence achieved the highest average BLEU, outperforming other algorithms.
4.6 Consistent observations in language- multilevel optimization
To verify our findings, we conducted experiments in both task-based and language- multilevel settings using Conformer models (100 M and 58 M parameters) with comparable hyperparameters

---

on the LibriSpeech and AISHELL datasets. For the language- based multilevel MSP evaluation, we combined both datasets focus exclusively on speech recognition. The results (Appendix Table 7 exhibit the same pattern as our task-based optimization experiments, further confirming the effectiveness of the proposed algorithms.
4.7 Efficient calculation of the dynamic update direction
Focusing only on the gradients of conflicting lay-
ers when computing the update direction im- proves efficiency without degrading performance.
50
Percentage of Confiicting Layers (%)
Our layer-wise analysis (Figure 5) reveals a stable pattern: only 20-30% of backbone layers- the early encoder blocks that encode low-level acous- tic cues-connistently display large negative cosine similarities, whereas deeper blocks, which model higher-level and more sunguage-agnosti abstractions, exhibit no conflict. Leveraging this observation- and in line with [50] apply lightweight conflict- based pruning strategy that, after detecting the con- flicting layers in the first 20 epochs, limits the dy- namic update-direction computation to their gradi- ents. This selective scheme shrinks GPU memory from 14.7 GB to 12.8 GB and shortens training time from 4.1 h to 3.7 per epoch (Table while preserving final performance (see VM-MSP (Efficient)
45
40
25
10
15
20
25
30
Epoch
Figure 5: Percentage of conflicting layers.
in Tables and 3. Full implementation details are provided in Appendix D
Conclusions and Limitations
In conclusion, our study highlights the substantial advantages of integrating self-supervised loss as constraining objective within multilevel multi-objective optimization structure for multilingual multi-task speech processing. Our findings indicate that segregating highly conflicting objectives into different optimization levels yields significant benefits for speech recognition and translation tasks. This approach not only enhances the effectiveness multi- oojti--b optimization but also underscores its potential for optimizing complex tasks across diverse linguístic boundaries.
Our evaluation focused on architectures that exhibit clear separation between shared and task- specific parameters-foo example, shared encoder followed by distinct classification heads for ASR and translation. This architectural pattern is common in speech processing, but how the method generalizes to more tightly coupled architectures such models with unified decoder shared across rasks-remains an open question. In such cases, task-specific conflicts may propagate more deeply through shared components, potentially requiring alternative optimization strategies or additional regularization. Investigating these scenarios a promising direction for future research.
10

---

References
[1] Antonios Anastasopoulos and David Chiang. Tied multitask learning for neural speech translation. arXiv preprint arXiv: 002506655, 2018.
[2] Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick Von Platen, Yatharth Saraf, Juan Pino, Alexei Bacvski, Alexis Conneau, and Michael Auli. Xls-r: Self-supervised cross-lingual speech representation learning at scale. arXiv preprint S.kiv22/11009225 2021.
[3] Alexei Baevski, Steffen Schneider, and Michael Auli. wq-wav2vee: Self-supervised learning discrete speech representations. arXiv preprint arXiv:/910.05455 2019.
[4] Alexei Bacvski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for sell-supervised learning of speech representations. Advances in neural information processing systems. 33:12449 12460, 2020.
[5] Junwen Bai, Bo Li, Yu Zhang, Ankur Bapna, Nikhil Siddhartha. Khe Chai Sim, and Tara N Sainath. Joint unsupervised and supervised training for multilingual asr. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 6402 6406, 2022.
[6] Ankur Bapna, Colin Cherry. Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, and Alexis Conneau. mslam: Massively multilingual joint pre-training for speech and text. arXiv preprint XxXi:2220200137 374, 2022.
[7] Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. Aishell-1: An open-source mandarin speech corpus and speech recognition baseline. conference of the oriental chapter of the international coordinating committee on speech databases and speech I/O systems and assessment, pages 1-5, 2017.
[8] Dongpeng Chen and Brian Kan-Wing Mak. Multitask learning of deep neural networks for low-resource speech recognition. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 23(7):1172- 1183, 2015.
[9] Lisha Chen, Heshan Fernando, Yiming Ying, and Tianyi Chen. Three-way trade-off multi-objecttiv learning: Optimization, generalization and conffict-avoidance. Journal of Machine Learning Research, 25(193):1 53, 5 2024.
[10] Lisha Chen, AFM Saif, Yanning Shen, and Tianyi Chen. FERERO: A flexible framework for preference- guided multi-objective learning. In Advances in Neural Information Processing Systems, 2024.
[11] Lisha Chen, Quan Xiao, Ellen Hidemi Fukuda, Xinyi Chen, Kun Yuan, and Tianyi Chen, Efficient first- order optimization on the Pareto set for multi-objective learning under preference guidance. In P Proceedings of the International Conference Machine Learning, 2025.
[12] Franklin Y Cheng. Multiobjective optimum design of structures with genetic algorithm and game theory: Application to life-cycle cost design. In Computational mechanics in structural engineering, pages 1 1-16. 1999.
[13] Yong Cheng, Yu Zhang, Melvin Johnson, Wolfgang Macherey, and Ankur Bapna. Mu slam: Multitask, multilingual speech and language models. In International Conference on Machine Learning, pages 5504-5520, 2023.
[14] Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self-supervised learning with pandon-projection quantizer for specch recognition. In International Conference on Machine Learning, pages 3915-3924, 2022.
[15] Shun-Po Chuang, Tzu- Wei Sung, Alexander H Liu, and Hung-yi Lee. Worse wer, but better bleu? leveraging word embedding as intermediate in multitask and-to-end speech translation. arXiv preprint arXiv:2005.1066 2020.
[16] Xiaodong Cui, AFM Saif, Songtao Lu, Lisha Chen, Tianyi Chen, Brian Kingsbury, and George Saon. Bilevel joint unsupervised and supervised training for automatic speech recognition. JEEE/ACM Trans. Audio Speech Lang. Process., 33:286 296, 12 2024.
[17] Kalyanmoy Deb, Karthik Sindhya, and Jussi Hakanen. Multi-objective optimization. In Decision sciences. pages 161-200. 2016.
[18] Michalis Doumpos and Constantin Zopounidis. oblti-objective optimization models in finance and investments. Journal of Global Optimization, 76(2):243 (644::243224,.20 2020.

---

[19] Heshan Fernando, Han Shen, Miao Liu, Subhajit Chaudhury, Keerthiram Murugesan, and Tianyi Chen. Mitigating gradient bias in multi-bjjective learning: A provably convergent stochastic approach. In Proc. International Conference on Learning Representations, 2023.
[20] Jörg Fliege and Benar Fux Svaiter. Steepest descent methods for multicriteria optimization. Mathematical methods of operations research, 51:479-494, 2000.
[21] Jinlan Fu, SienKiong Ng. and Pengfei Liu. Polyglot prompt: Multilingual multitask promptraining. arXiv preprint or Xi1: 2022.
[22] Amab Ghoshal, Pawel Swietojanski. and Steve Renals. Multillinuual training of deep neural networks. In IEEE international conference on acoustics, speech and signal processing, pages 7319-7323, 2013.
[23] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Conncctionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the international conference Machine learning, pages 369-376, 2006.
[24] Alex Graves, Navdeep Jaitly. and Abdel-rahman Mohamed. Hybrid speech recognition with deep bidi- rectional Istm. In IEEE workshop on automatic speech recognition and understanding, pages 273-278, 2013.
[25] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. IEEE international conference on acousties, speech and signal processing, pages 6645-6649, 2013.
[26] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, and Yonghui Wu. Conformer: Convolution-angmented transformer for speech recognition. arXiv preprint 20X00:2005.01100, 2020.
[27] Georg Heigold, Vincent Vanhoucke, Alan Senior, Patrick Nguyen, Mare' Aurelio Ranzato, Matthicu Devin, and Jeffrey Dean. Multilingual acoustic models using distributed deep neural networks. In IEEE international conference on acoustics, speech and signal processing, pages 8619-6623, 2013.
[28] Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The magazine, 29- 2012.
views of four research groups. IEFE Signal processing
[29] Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, and Abdelrahman Mo- hamed. Hubert: How much can a bad teacher benefit asr pre-training? In IEEE International Conference Acoustics, Speech and Signal Processing, pages
2021.
[30] Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andres Felipe Cruz Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, and Hany Hassan Awadalla. Scalable and efficient moe training for multitask multilingual models. arXiv preprint arXiv:2109.1044: 2021.
[31] Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, 2018.
[32] Vitaly Kurin, Alessandro De Palma, llya Kostrikov, Shimon Whiteson, and Pawan K Mudigonda. In defense of the unitary scalarization for deep multi-task learning. Advances in Neural Information Processing Systems, 35: 15::1-191213,,, 2022.
[33] Hang Le, Juan Pino, Changhan Wang. Jiatao Gu, Didier Schwab, and Laurent Besacier. Dual-decoder transformer for joint automatic speech recognition and multilingual speech translation. arXiv preprint arXiv:2011.0077 000747. 2020.
[34] Jaesong Lee and Shinji Watanabe. Intermediate loss regularization for ctc-based speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6224-6228, 2021.
[35] Xian Li, Changhan Wang, Yun Tang, Chau Tran, Yuqing Tang, Juan Pino, Alexci Baevski, Alexis Conncau, and Michael Auli. Multilingual specch translation with efficient finctuning of pretrained models. arXiv preprint 2010 12829, 2020.
[36] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-avers averre gradient descent for multi-tas task learning. In Advances in Neural Information Processing Systems, 2021.
12

---

[37] Suyun Liu and Luis Nunes Vicente. The stochastic multi-gradient algorithm for multi-objective optimization and its application to supervised machine learning. Annals of Operations Research, pages 1-30, 2021.
[38] R Timothy Marler and Jasbir S Arora. Survey of multi-objecciiv optimization methods for engineering. Structural and multidisciplinary optimization, 26(6):369-395, 2004.
[39] Kaisa Miettinen. Nonlinear multiobjective optimization, volume 12. Springer Science & Business Media, 1999.
[40] Takafumi Moriya, Tomohiro Tanaka, Takahiro Shinozaki, Shinji Watanabe, and Kevin Duh. Evolution- strategy- based automation of system development for high-perteen high-perfrrmace speech recognition. /EEE/ACM Transactions on Audio, Speech, and Language Processing, 27(1):77-88 88, 2018.
[41] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.03742 2018.
[42] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In IEEE international conference on acoustics, speech and signal processing, pages 5206 5210, 2015.
[43] Alee Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcl Mcl.avvy, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. International conference on machine learning. pages 28492-2855 18,588.22 2023.
[44] Alexandre Rame, Guillaume Couairon, Corentin Dancelte, Jean- BrannBBptistte Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36, 2024.
[45] Yi Ren, Jinglin Liu, Xu Tan, Chen Zhang, Tao Qin, Zhou Zhao, and Yan Liu. Simulspeech: End- to-end simultancous speech to text translation. Annual Meeting of the Association for Computational Linguistics, pages 3787-3796, 2020.
[46] AFM Saif, Lisha Chen, Xiaodong Cui, Songtao Lu, Brian Kingsbury, and Tianyi Chen. M2ASR: Multilin- gual multi-task automatic speech recognition via multi-objective optimization. In Proc. Interspeech 2024, pages 1 1240-1244, 2024.
[47] AFM Saif, Xiaodong Cui, Han Shen, Songtao Lu, Brian Kingshury, and Tianyi Chen. Joint unsuper- vised and supervised training for automatic speech recognition via bilevel optimization. arXiv preprint arXiv:2401.0698) 2024.
[48] Steffen Schneider, Alexci Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre- training for specch recognition. arXiv preprint arXiv:1944.05866 2019.
[49] Han Shen, Quan Xiao, and Tianyi Chen. On penalty- penalty-based bilevel gradient descent method. Mathematical Programming. 2025.
[50] Guangyuan Shi, Qimai Li, Wenlong Zhang, Jiaxin Chen, and Xiao-Ming Wu. Recon: Reducing conflicting gradients from the root for multi-task learning. URL hilps://axxiv. orglubs/230.11228 2023.
[51] Joar Skalse, Lewis Hammond, Charlie Griffin, and Alessandro Abate. Lexicographic multi-objective reinforcement learning. arXiv preprint arXiv:2212. 13,7369,,,,, 2022.
[52] Matthias Sperber, Graham Neubig, Jan Nichues, and Alex Waibel. Attention- passing models for robust and data-efficient end-to-end speech translation. Transactions of the Association for Computational Linguistics, 7:313-325, 2019.
[53] Chaitanya Talnikar, Tatiana Likhomanenko. Ronan Collobert, and Gabriel Synnaeve. Joint masked epc and ctc training for asr. In JEEE International Conference on Acoustics, Speech and Signal Processing, pages 9049-3049, 2021.
[54] Samuel Thomas, Sriram Ganapathy, and Hynek Hermansky. Cross- Cross-lingual and multi-stream posterior features for low resource lvesr systems. In Annual Conference of the International Speech Communication Association, 2010.
[55] Shubham Toshniwal, Tara N Sainath, Ron J Weiss, Bo Li, Pedro Moreno, Eugene Weinstein, and Kanishka Rao. Multilingual speech recognition with single end-to-end end model. In IEEE international conference on acoustics, speech and signal processing, pages 4904-4908, 2018
13

---

[56] Zoltán Tüske, Joel Pinto. Daniel Willett, and Ralf Schlüter. Investigation cross-and multilingual mip features under matched and mismatched acoustical conditions. IEEE international conference acoustics, speech and signal processing, pages 7349-7353, IEEE, 2013.
[57] Changhan Wang, Anne Wu, and Juan Pino. Covost and massively multilingual speech-to-text translation. arXiv preprint arXiv:2007.13310. 2020.
[58] Shinji Watanabe, Takaaki Hori, and John Hershey. Language independent end-to-end architecture joint language identification and speech recognition. In IEEE Automatic Speech Recognition and Understanding Workshop, pages 265 225-271,22 2017.
[59] Derrick Xin, Behrooz Ghorbani, Justin Gilmer, Ankush Garg, and Orhan Firat. current multi-task optimization methods in deep learning even help? Advances in Neural Information Processing Systems, 35:13597-13609, 2022.
[60] Hao Zhang, Nianwen Si, Yaqi Chen, Wenlin Zhang, Xukui Yang, Dan Qu, and Wei-Qiang Zhang Improving speech translation by cross-modal multi graite contrastive learning. IEEE/ACM Transactions Audio, Speech, and Language Processing, 31:1075-1086, 2023.
[61] Shiyu Zhou, Shuang Xu, and Bo Xu. Multilingual end-to end speech recognition with single transformer low-resouree languages. arXiv preprint arXiv: 1806.8866.000 050600 2018.
[62] Shiyu Zhou, Yuanyuan Zhao, Shuang Xu, Bo Xu, et al. Multilingual recurrent neural networks with residual lcarning for low-resource speech recognition. In INTERSPEECH, pages 704-708, 2017.
14

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
Codes are included with the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipps.ccppullc//uuuii wwwiiicgguie//eeeeeeeeeis CoodeSubeiiie ssi ssionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification:
We included WERs and BLEU scores.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper discusses the effect of conflicting objectives during the training of multilingual, multitask speech-processing model, which has no direct societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper has no such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Codes included with the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

Supplementary Materials
Table of Contents
Detailed Related Work
22
Algorithm Development
23
Task Specifie Formulation and Update Rule
25
C.I
VS-MSP for single vectorized objectives
25
VC-MSP for vectorized objectives with constraint lower level
.........
25
VM-MSP for multilevel ASR optimization
25
Efficient Training
27
E
Gradient Conffict
27
F
Baseline Training Methods
28
F.1
Two-stage training (Pre (Pre-training + Fine- Pine-tuning)
28
F2
Static Weighting
29
F3 Joint two-stage training without multi oujettobbeectve optimization
................................
29
Experimental setup
29
H
Ablation Study
30
H.1
Impact of Pre-traing tr-iirainnng Method
31
H.2 Impact of VM-MSP on line-tuning speech loundation model
................................
31
H.3 Impact of penalty parameter
32
Resource Efficiency of the multi-objective optimization Model
32
A Detailed Related Work
In this section, provide comprehensive review of existing works on multi-objective optimization, multilingual speech recognition, translation, and multi-task learning for speech processing.
Multi-objective optimization aims to optimize multiple, often conflicting, objectives simultancously and has been applied in various domains including enginecring [12), finance [18), and decision making [17. Multi-objective optimization has also been used for solving lexicographic multi- objective problems [3 51] and speech processing. In previous work, multi-objective optimization has been applied at the system development level, using evolutionary strategies jointly optimize recognition accuracy and model size by tuning meta- meta-paaameters such as model topology and training work focuses loss-level conflicts during the training of multilingual and multitask models, where multiple learning signals interact within shared backbone.
configuration [40]. In contrast,
Multilingual speech recognition and translation. Early studies in multilingual ASR employed
deep neural networks, hidden Markov models, and multilayer perceptrons [27 54 56 22). The
introduction of LSTM models brought significant improvements multilingual speech recognition Subsequently, Seq2Seq models with hybrid attention/CTC algorithms and transformer-ban
24 622
architectures emerged state-of-the-an [58 55 611. For multilingual translation, transformer-base
models with self-super pre-training have been widely adopted 35 6 45
However, existing
approaches often lack the capability effectively perform multi-taskin across diverse specch-related tasks. These efforts remain orthogonal to our proposed multi multi-objecctiv optimization algorithms and potentially benefit from their integration.
Multi-task Learning for Speech Recognition. Multi-task learning for joint speech recognition and translation has been explored with limited success. Early attempts introduced algorithms for
22

---

Notation
Description
+ E liv
Model parameter including backbone and classification head parameter.
0 E R
Backbone parameter.
8 E I²
Backbone parameter at k-tth iteration.
0" E IR*
Optimum backbone parameter.
Ø E 1""
Parameter of the task-specific classification head.
in E ""
Classification head parameter of 7-th task and t-th language.
0 E R"
Classification head parameter of n-th task and t-th language k-th iteration.
Ø E R'
A group of all classification head parameters of level p.
6² E 1"
Optimum parameter of the task-specific classification head.
L
Vector of all objectives.
Cy
Vector of all objectives with penalized lower-level constrained objective
used for VC-MSP method.
Cp
Vector of all objectives in level p used for VM-MSP method.
em E [M]
m-th objective.
Eg
supervised loss with supervised data.
En
self-supervised loss.
E [T]
Represents a specific language (For example: English, German, etc.). Represents a specific task (For example: speech recognition or transla- tion.).
11 [N]
1 E K
Current iteration number.
p E
Optimization level.
Constraint defines the feasible region for the upper-level objectives
Condiict-voodding update direction.
7
Learning rate of À update.
0
Learning rate of backbone parameter.
8
Learning rate of task task-specific classification parameter.
A
Dynamic weight to combine the gradient.
Dynamic weight at k-th iteration.
Dynamic weight of self-supervised objective at k-th iteration.
X
Dynamic weight of nt-th task and t-th language at k-th iteration.
X
Optimum dynamic weight to combine the gradient.
7p-1,P 2
Penalty parameter of p-th level of multilevel optimization (VM-MSP).
1/p X 1p-11
Combined penalty constant for the lowest level (VM-MSP).
Stochastic unlabeled sample during training at iteration k.
gh
Stochastic labeled sample during training at iteration k.
D
Labeled dataset.
Table List of notations used this paper
joint speech recognition and translation decoding 1 and intermediate word embeddings with two- stage models [5 52 Transformer- dual encoder-decoder architectures were also developed,
featuring separate decoders for speech recognition and translation tasks B3 Whisper (43 demon-
strated the potential of large-scale multitask models, while Mu SLAM [3] leveraged cross-modality
learning across multilingual and supervised subtasks. Joint pre-training and fine-tuning approaches have also been proposed to simplify training [5] 47 53 Despite these advancements, most methods rely on static weighting strategies or constrained optimization, which do not explicitly address the challenges posed by conflicting objectives. This lead to suboptimal performance when tasks inherently conflict.
B Algorithm Development
After formalizing three training algorithms in Section S our subsequent objective is to devise gradient-based algorithm capable of addressing large-scale, high-dimensional multilingual multi-task
23

---

challenges by providing an update rule that converges to Pareto-stationar solutions. We will focus on the algorithm development of VC-MSP. as this can be easily extended the other two methods (VS-MSP, VM-MSP). To achieve gradient-based algorithm for VC-MSP that can avoid conflicting update directions, we leverage recent advances in unconstrained multi-objective optimization [9] and employ penalty-based approach to convert the constrained multi-objective optimization problem in [ into an unconstrained multi-objective optimization problem. This approach simultancously conducts self-supervised pre-training and supervised multi-objective learning, as in (8); that is,
min BER MEE"
(0) :0(C,0,, 11.1) + (0). (0, vI, (0.
(12)
Eg (0, 1 + (0), ... _, (r,00,TTN) 7(.
where 7) is a penalty parameter. This penalty parameter integrates the self-supervised constrained objective with the supervised objectives and ensures that the feasible region of the supervised objective remains within certain bounds.
Parameters update. Using the dynamic weighting and penalization method, we update the backbone parameters, 0, of the MSP model. Next, we describe the backbone parameters and task -seciiccc classification parameters update rules for VS-MSP, VC-MSP, and VM-MSP.
VS-MSP. For single vectorized objective training, we only need consider if the objectives have conflicting update directions. As multilingual multi-task training, we are using separate language datasets; we can assume that the objectives have conflicting update directions. We can also prove this assumption by calculating (Vel,, (O), Volr,, < O. We optimize these vectorized objectives using the algorithm: where the shared backbone parameters updated via
T
N
g++l 8
a
N (ol of,,) - adu (olu(
(13)
In this context, a > 0 denotes the learning rate specifically assigned to the backbone parameters. Moreover, Mr and Au represent the dynamic update directions for supervised and self-supervised objectives, respectively, which are computed using the MoDo algorithm. Similarly, taking the gradients of each of the supervised objective functions with respect to the parameters of task-specifit output heads, task-specific output layers are updated via,
6++1 Lin
oi. BV &e (ot
(14)
where B> 0 is the learning rate for the task- parameter.
VC-MSP. To train model using the VC-MSP algorithm, the backbone parameters are updated via
T N
gk+1 8h a
Vola of,m) an/Volu((h)
(15)
To update task-specific classification heads, we employ see a summary in Algorithm
VM-MSP. In VM-MSP, we separate highly conflicting objectives into distinct optimization levels. Here, we assume that all objectives at level p function as lower-level objectives for those at level - 1. Consequently, we can update the backbone parameters using the penalize method, that is
Ni Ma Vola(O" (t
T2
Na Vola(O", )tg )
g++l 0t t
Σ
Niging
(16)
b2 n===
Tp Mpnng Vola(0", of mp) 0mP-1
Np
Tp
NP
Σ
Aip.op Vola(8", of ppp.mp) + on coVVol((0)))
amp--
Update task-specific classification parameters using
x+1 Otp =0 of, BV pl gh)
(17)
where Np and Tp represent the total number of tasks and languages at level p. respectively. We
represent the penalty parameter at level p Tpp and that for the self-supervised objective as 7).
24

---

C Task Specific Formulation and Update Rule
In this section, we will explore in detail the three multi-objective optimization setups in speech recognition and translation tasks and establish the parameter update rules for each of them.
C.1 VS-MSP for single vectorized objectives
For single vectorized objective training, we only need to consider the objectives have conflicting update directions. As in the multilingual multi-task training, can assume that the objectives have conflicting update directions. We can also verify this assumption by calculating (Vel,, (9), Vele (()) < 0. We can formulate this single vectorized objective for
separate language datasets,
ASR and translation tasks following 5 as
(L(0,01.)): (0, 1I,N), (.,.,)))..,,,.,.. (.0,)))).... (T), BeR
min
(18)
As there is no lower-level constraint, we optimize this vectorized objective using Algorithm the shared backbone parameters updated using the following equations
where
T Σ Vola -ata))-(VVVollla Vot (19)
g++1= - 0²
k Vols of
where Ae,1 and dς,2 are dynamic update directions for ASR and translation tasks, respectively, and X is the dynamic update direction for the self- supe--upereieed objective calculated using the MoDo algorithm. update the classification heads using
P1,1
k+1
0i,1 - -V 0%).
(20a)
212
A+ o²,2 VVolo 0*).
(20b)
C.2 VC-MSP for vectorized objectives with constraint lower level
self-supervised CPC loss, (0),,, a lower-level constraint to guide optimization toward a region with good representation and at the same time admits common descent directions for supervised loss, (0, Φ). The problem formulation for VC-MSP in speech recognition and translation tasks can written as follows:
In this setup,
min (C, 01.1. (.. 1T.1). (0 2T222) BER
s.t. (0) min (0) Se.
(21)
The backbone parameters 0 is updated using,
T
g++1 = 8¹ a te Vot of,)
k (o
- - Vnla (22)
The task specific classification parameters are updated using
20a and 20b
C.3 VM-MSP for multilevel ASR optimization
In multilevel optimization problem, there is hierarchy of objectives. We can reformulate the multilingual multi-task ASR optimization task into different multilevel optimization problems based the tasks. languages, language families which they belong. We study these set-ups and solve these optimization problems using a penalty-based gradient descent method.
Multilevel optimization based on tasks. We can extend the MSP optimization problem into three levels based on the tasks: speech recognition, translation, and self-supervised task. We always place the sele-super objective at the lowest level and optimize it first, as the optimization of all other
25

---

objectives directly depends on the optimization of the self sulf-supervised objective.
argmin 01 17,1 ER"
(0 22.1.
7,28
s.t. 01.21" .. -7.2
argmin 01.2. 2R
(0 02.1. 21,2 22,,
8.1. =² argmin Lu (0). DER
(23)
We apply penalty-based method to convert this multilevel multi-objective optimization problem into a single-level optimization problem and apply dynamic multi-objective optimization to update the parameters in conflict-avoiding direction.
Σ X Vol( 2i.2) am2 Volo(((()))) ) (24)
gh+z 8h Σ
M M. Volz(0 of, omn
Here, 77 and are penalty parameters. We can combine and and get = m x 172 for self-supervised loss.
T g++l = 0h a M Vola(O" am Σ 2 Vola(0 - Vol((0)) (25)
Next, update the classification heads via
k+1 0
-V
VVoll VV 84)
(26a)
k+1
k
(26b)
We provide a detailed algorithm of multilevel ASR optimization the optimization order of ASR and translation tasks.
We also do experiments altering
Multilevel optimization based language. We can also extend the ASR optimization problem
multiple levels based on languages.
argmin 2 2Rt.eRRR..,.0. 10. 25,
(, 0221 ,0))
La .T................ (27)
s.b. P7.1
argmin
0T.. 8-
s.t. 0" = argmin (0)
deR=
In this setup, we optimize all the objectives of one language in one optimization level and optimize other languages' objectives in other optimization levels. For simplicity of implementation, we will consider two languages. We can update the model parameters using the following penalty-based update rules.
N
N
gh+ 8h a Σ Xt, (olete( (ff. om Σ 2,m Volete Φb..) ) Vol (28)
1=1
n=1
In this equation, 77, and 7/2 are penalty parameters. We can combine 7h and 7/2 to obtain 7 = 7/ x 72- which is used for the self-supervised loss. The parameter N 2 represents the total number of tasks (in this experiment, ASR and translation). The terms X and >ς,n, represent the dynamic update directions for languages 1 and 2, respectively, during the k-th iteration for task
Next, we update the classification heads via
k++
of,1 3V VVololot
(29a)
0.2
k+1 6i.2 3V a
(29b)
In both task-based and language-based MLO, we alter the order of objectives at the optimization level to examine the effects of their arrangement. By doing so, we can better understand how the sequence of objectives influences the optimization process and outcomes.
26

---

D Efficient Training
To reduce computational overhead during dynamic update-direction computation, we introduce lightweight method for identifying conflicting layers. For each shared encoder layer I, we compute task-specific gradients g.6) and evaluate their pairwise cosine similarities. The cosine similarity between gradients of tasks i and at layer / is defined as:
(0 ( 11 II 119)
cos A
We denote the set of conflicting task pairs at layer as p( := {(i.)) j) cos f < 0}. A layer is My
considered be conflicting if the average cosine similarity across all such pairs is negative:
1
cos 8 <.
|P(
(KJ))PP))
Only layers meeting this criterion selected for computing the conflict-avoidance update direction This targeted layer selection substantially reduces training time and memory usage (see Section 4.7 and Table D while maintaining the performance gains of multilevel optimization.
E
Gradient Conflict
In this setup, aim to separate highly conflicting objectives into upper and lower optimization levels. However, a sub-question arises within this setup: which objectives are highly conflicting? To address this question, need to establish boundary threshold that distinguishes objectives with significant conflicts. We can create such a threshold by calculating the degree of conflict using the cosine similarity the gradients of the objectives. If the cosine similarity of two objectives is smaller than a certain threshold, they are optimized at different levels. If Vee,,..0 and Vede, are gradients of two objectives, then we can calculate the cosine similarity using
(V)),,(()), (
cOS Volu, (9) V Ve (9)
(30)
where is the angle between the gra-
dients of two different objectives. To
calculate the similarity between up-
82
date directions, we use the same con- former model and train using two different languages and objectives si- multaneously. We train the model for 20 epochs using both objectives and then average the gradients of their updates separately. We follow the same process for all languages and record their average gradients for 20 epochs. We can now calculate cosine similarity between the gradi- ent update direction of two objectives from these recorded gradients. We also compare the cosine similarity be- tween sclf-supervised and supervised losses.
Fr-SEn
t-+en
Ca-SEn
Unoup
Deata
Ba
Figure 6: Scatter plot of cosine similarities between MSP objectives.
In Figure and we depict the cosine similarity of supervised objective gradients across five languages, along with the self-supervised objective gradient for speech recognition and translation. The heat map displays the similarity values, while the scatter plot, with points colored by their cluster assignments, helps visualize which objectives are closely related (high similarity) and which are not. The size and color of the points represent the similarity values and cluster assignments, respectively.
27

---

Algorithm 1 VS-MSP for multilingual multi-task MSP.
Input: Labeled data (xr y). unlaheled data X, f x =x)) learning rates and
for k = 1 to K do
sample çî of 2
2u. + v and & (x v²) (su 0²), Ves(Et; 0h, .²)), Vls(22; 8h, xt))
compute vlu (Cf:
)*),
update X++1 by
update gh+1 by
13
update Pt,n by
1++1
14
vA E [T), Vn [N]
end for
Output: 0* (
Algorithm 2 VC-MSP for multilingual multi-task MSP
Input: Labeled data y), unlabeled data X, { vu.
m. learning rates a, B, and
penalty parameter 7]; for & to K do
sample <* = x g
(of, vtt and E = U$) 8%), (E, 0 )²), Ve, x6))
compute Vl (S
update X by
update gh+11 by
15
update Pt,
k++ by
14
14 E (7) [N]
end for
Output: 0K (o
Our analysis of these figures reveals that tasks with lower cosine similarity exhibit higher conflicts. Notably, the self-supprviied gradients show significantly higher alignment with the other objectives. We found that segregating the highly conflicting speech recognition and translation tasks into different optimization levels reduced the overall conflict among the objective gradients, which consequently improved the overall performance.
Baseline Training Methods
In this section, we outline the baseline methods used to compare against our multi-objective opti- mization algorithms.
F1 Two-stage training (Pre-training + Fine-tuning)
This method involves two sequential steps:
1. Pre-training: The model is first pre-trained on self-supervised learning objective, such as CPC, Whisper Wav2Vec2, to learn general- purpose representations from unlabeled speech data. During this stage, the backbone parameters are updated using:
ceneral-purpose
g++1_ = 8 a ook,(
(31)
where 4, represents the self-supervised loss, and a is the learning rate.
2. Fine-tuning: After pre-training, the model is fine-tuned supervised task (c.g., speech recognition and translation) using the CTC or cross-entropy loss to adapt the learned representations task-specific objectives. During fine-tuning:
The backbone parameters are updated using:
g++1 0h
Vols
(32)
NT
==
where 3 is the learning rate, N and T denote the number of tasks and languages, respectively.
28

---

Algorithm 3 VM-MSP for multilingual multi-task MSP.
Input: Labeled data (I,y), unlabeled data X, { mu w learning rates o, B, and
penalty "/.-...0 ,7P; for to K do
sample < = x
S1 = (17.v)) and E = (22.
compute 1 8%), VL,(f; 0h :²)), VL,(22 (64)
update X++1 by
update @h+11 by (16)
update o by [1 Vt, E [T,], E [M,]]
end for
Output: 0* (
The parameters of the individual classification heads are updated using:
inn
k+1
oi.n oos 0²),
(33)
where Ur,n denotes the parameters for task and language t.
F.2 Static Weighting
This method follows the same process pre-training fine-tuning but introduces static weighting during fine-tuning. Instead of using equal weights for all supervised objectives, a grid search is performed to assign suitable weights each objective. The backbone parameters updated using:
T D
gh+1 8h 5
Me. Vot of
(34)
where Men, represents the static weight assigned to the supervised objective for task and language t. For our experiments, the following language-specific weights were used:
En, Fr, De, Es, Ca] = [0.18, 0.19, 0.27, 0.16 0.20).
F.3 Joint two-stage training without multi-objective optimization
This method follows the same process as VC-MSP but does not incorporate multi-objective opti- mization 47). Instead, all supervised objectives are optimized jointly without dynamic weighting or conflict-aware gradient alignment, resulting in simpler optimization process.
G Experimental setup
In this section, we outline the dataset, models, hyperparameters, and data pre-processing techniques employed in evaluating VS-MSP, VC-MSP, and VM-MSP algorithms.
Dataset. We evaluate our training algorithms on a combined dataset of LibriSpeech 42 AISHELL
vI [7], and CoVoST v2 [57]. LibriSpeech is an English speech dataset consisting of 960 hours of data along with transcripts. AISHELL v1 a 178-hour multi-channel Mandarin speech corpus designed for various speech/speaker processing tasks. We have combined these two datasets to create single multilingual dataset. Our approach involved splitting the LibriSpeech dataset, allocating 860 hours for self-supervised pre-training and using the 100-hour train-clean- 100 subset for supervised training. The trained models are tested on the AISHELL test dataset and the LibriSpeech test-clean dataset. During training using CoVoST dataset, ensure balanced training. For high-resource En, we use the ful data without up-sampling, while applying up-san sampling for low-resource languages- 4x for Ca and Es and 2x for Fr and De.
equal batch sizes across all languages and tasks to
In the first experiment, we use combined LibriSpeech and AISHELL multilingual dataset and train a multi-head conformer for multilingual MSP tasks. In the second experiment, we use the CoVoST v2 training dataset for multilingual speech recognition and translation training. The CoVoST v2 test set is used to evaluate the trained models. CoVoST v2 is widely used benchmark multilingual
29

---

Table 7: ASR WERs (LibriSpeech) and CERs (AISHELI) for two-stage training. Joint training. VS-MSP. VC-MSP, and VM-MSP, with VM-MSP using UEC (self-supervised English Chinese) and UCE (self- supervised - Chinese English) optimization sequences.
Parua
Lang
Two-stige training
Jount training
VS-MSP
VC-MSP
VM-MSP UEC
VM-MSP UCE
En (lest (lest-cleun) cleun)
6,2%
5.9%
6.1%
5.7%
5.2%
5.4%
100M
En (texi-ocher)
17.0%
16.8%
17.1%
16.7%
16.3%
16.5%
5.8%
5.5%
5.3%
9.7%
9.6%
9.3%
8.9%
(lest
7,8%
7.1%6
7.3%%
6.8%
6.5%
6.6%
(test
17.8%
17.7%
17.3%
17.0%
17.1%
Zh
7.4%
6.8%
7.0%
6.5%
6.1%
5.8%
Ave
11.0%
10.4%
10.8%
10.2%
9.9%
9.8%
translation corpus covering translations from 21 languages into English and from English into 15 languages.
Models and hyper-parameters. For additional simulations, we employ the wav2vec2 model.
wav2vec2-large + Transformer decoder. We initialize from XI.SR-33 (24 layers, = 1024, 16, H = 16. FF dim 4096). ASR is handled by linear CTC head over the transcription vocabulary. For translation, we append a Transformer decoder with three encoder layers and three decoder layers (dim 1024, // 16, FF dim 4096), outpulting via a linear projection to the translation vocabulary.
Hyper-parameters. We use grid search to optimize hyperparameters, including learning rate, batch size, step size of MoDo, and penalty parameter increasing rate. For both SSL pre-training and supervised fine-tuning, the backbone learning rate is consistently set higher than the classification parameter learning rate. The SSL pre-training phase starts with learning rate of a 5 x 10 for 100 epochs, annealed by factor of 0. every 20 epochs. Fine-tuning uses a maximum learning rate of B = 5 x 10-5, with scheduler reducing the learning rate by factor of 0.1 if the test loss does not improve within 10 epochs. All multi-objective models (VS-MSP, VC-MSP, and VM-MSP) and joint PT+FT models are trained for 200 epochs. For PT+FT, we pre-train the model for 200 epochs and fine-tune it for an additional 100 epochs. A batch size of 256 and Adam optimizer are used for both self-supervised and supervised training. The same hyperparameter settings applied across all training methods to ensure consistency and comparability.
Penalty parameter for ASR and translation. For VC-MSP, the initial penalty parameter 1) is set to and increases by 0.02 per epoch. The increase stops once the penalty reaches a maximum value of 1.5. For VM-MSP, the second-level penalty parameter is initially set to 0.1 and increases by 0.02 per epoch, while the lower-level penalty constant 7/2 starts at and also increases by 0.02 per epoch. The increase for both penalty constants stops once they reach 1.5. A higher increase rate for the lower level ensures equal importance of both upper-level and lower-level objectives.
Data weepprocessing. Our experiments cover both supervised and self-supervised regimes and share common log-Mei preprocessing pipeline. Raw audio is converted 80-dimensional log-Mel spectrograms and normalized to zero mean and unit variance. In the SELF-SUPERVIES setup, the model receives a 2 context window and is trained to predict the following 1 segment without any data augmentation. In the SUPERVISED setup, we apply SpecAugment to the normalized features to improve robustness. For CONFORMER-BASFR MODELS, transcripts are tokenized with SentencePiece [31] using 1,000-token word-level vocabulary for every language except Chinese, where we employ a 4,930-token character-level model. For WAv2VEC 2.0, we use the character-level CTC vocabulary bundled with the official checkpoints, and for WHISPER we adopt the byte-pair- encoding tokenizer released with its original implementation.
Computational Resources. All simulations were run on two NVIDIA A5000 GPUs and two NVIDIA A4500 GPUs, with Intel 19-7920X CPU and 128 GB of DDR4 memory.
H Ablation Study
In this section, we study the impact of different pre-training methods and provide detailed explana- tion of the effect of the penalty parameter the overall training process.
30

---

Table 8: Speech recognition WERs and translation BLEU score comparison between CPC and BEST-RQ pre-training methods. For translation do Lang - En translation.
VMLMSP-UAS
VMLMSP-USS (CPC: BLEU)
VM-MSP-LLS (BESTRO
VM-MSP-UAS (BESTRQ
Param
I Lang
(CPC-
WER)
WEX)
BLEU)
En
19.7%
19.1%
21.3%
29.5
20.9%
30.1
17.0%
31.3
16.5%
31.9
Ca
19.1%
25.4
18.6%
26.2
Are.
18.4%
28.7
17.8%
29.4
Table Speech recognition WERs and translation BLEU scores between Wav2Vec2 with and without VM-MSP methods. For translation, we perform translation from Lang - En.
Joint training
Joint training
Vee2 (WER)
Waw2Vec2 (BLEU) Without VM-MSP
Way2Necd (WER) With VM-MSP
War2Vec2 (BLBU) With VM-MSP
Pumm
Lang
Without VM-MSP
Fa
18.15
16.3%
31.2
18.2%
32,4
300M
16.1%
29.9
34.1
15.2%
35.1
30.8
Ca
18.9%
20.2
18.0%
27.2
Ave.
17.2%
30.4
16.1%
31.4
H.1 Impact of Pre-training Method
In this ablation study, we assess the impact two different pre-training techniques- CPC and BEST-RQ [14]-on the performance of our VM-MSP method. The purpose of this ablation is to isolate the contribution of the pre-training method to the overall performance of the MSP tasks. We keep the settings consistent across both methods, with the model containing 150 million parameters in all cases. The tasks evaluated include speech recognition in various languages and translation for translating from different source languages into English.
The results in Table g compare CPC and BEST-RQ across five languages. The results indicate a consistent improvement when using the BEST-RQ pre-training method. Specifically, BEST-RQ leads to 3.3% absolute improvement in the average WER compared to CPC across all languages.
On the translation task, BEST-RQ also outperforms CPC, resulting in a 2.4% absolute increase in the average BLEU score across the evaluated languages. This indicates that BEST-RQ not only improves the speech recognition task but also enhances the downstream translation quality, likely due to the richer representations learned during pre-training.
Overall, these results suggest that the pre-training method plays a crucial role in enhancing both speech recognition and translation performance. The BEST-RQ approach, with its enhanced capability to model complex speech patterns. proves to be more effective than CPC, thus making it the more suitable choice for the VM-MSP algorithm.
H.2 Impact of VM-MSP on fine-tuning speech foundation model
We evaluate VM-MSP (UAS) method using the pre-trained Way2Vec2-XI.S-F model 2 In this approach, we use the pre-trained model as the backbone and add task- and language-specific linear layers and decoder to predict the output vocabulary, training with the CTC and cross-entropy loss. All hyperparameters remain consistent with our previous training protocols, except for the tokenizer. Specifically, we use the same tokenizer as the pre-trained Wav2Vec2 model maintain consistency in subword segmentation and ensure compatibility with the model's learned representations.
In the first experiment, adopt joint pre- training feanninng--ntttuute fine-tuning approach, training the Wav2 model across all languages for speech recognition and translation tasks while following the same procedure earlier joint pre-training+fine -unnngg training. In the second experiment, joint two-stage with multi-objective optimization, similar to the VM-MSP (USA) training approach. For both experiments, the model is trained for 50 epochs. The results, summarized in Table 9 show
extend
31

---

Table 10: Comparison of resource requirements between a single multi-objeccivve model and multiple single-
objective models during deployment.
Encoder Param
Classification Heads
Total Param
Storage Size
Loading Time (s)
Model
Inference time (ms)
MSP Model
-58
-25M
-83M
-55.,4 MB
-0.10
-11.4
Five Single-
-290M
-125M
411.0M
-792 MB
-0.5
~55.9
Objective Models
that, on average, the Wav2Vee2 model trained with VM-MSP outperforms the standard Wav2Vec2 model by 6.4% in the speech recognition task and by 3.3% in the translation task.
H.3 Impact of penalty parameter
In our multilingual multi-task ASR experiments, we investigated the effects of different penalty parameter increase rates to balance the ASR and translation tasks. We tested two configurations:
A lower increase rate of 0.002, which led to worse WER/BLEU score for lower-level tasks, shown in Tables
A higher increase rate of 0.02, which improved lower-level performance but slightly degraded upper-level performance.
Choice capped value for the penalty parameter: We capped the penalty parameter at 1.5 based on our observed trade-off between upper- and lower-level tasks. A penalty higher than 1.5 could have improved lower-level performance further, but it would have significantly degraded upper-level metrics. Thus, 1.5 was chosen an optimal balance point.
Post-Maximum Penalty Effects: The penalty parameter reached its maximum value of 1.5 after 75 epochs, but training continued for another 25 epochs. During this time, we observed further improvements in lower-level WER/BLEU scores, while upper-level performance deteriorated. This reinforces the critical role that penalty parameter selection plays in balancing competing objectives.
I Resource Efficiency of the multi-objective optimization Model
This section addresses the question: How does a single multi-objective optimization model reduce resource demands during deployment, making it more efficient solution overall?
Reduced Storage Requirements: A single multi-objective model is highly memory- efficient due to parameter sharing across tasks, see Table: [O] For example, the 83 M multi-objective model used in experiments has size of 158.4 MB, comprising encoder (-58M parameters) shared across all objectives and classification heads (-25M parameters). In contrast, deploying five separate models for these tasks would require 5x MORE BACKBONE PARAMETERS, resulting in significantly higher storage demands. Assuming each single-objective model uses an encoder of similar size, the total storage requirement for separate models would reach approximately 792 MB.
Efficient Inference: The multi-objective model also minimizes latency and computational overhead during inference. In our system, it takes only 0. 0.10s load the single multi- objective model, whereas loading five separate models takes 0.5s. This reduction in loading time directly translates to faster response times and improved computational efficiency.
By consolidating multiple objectives into single model, the multi-objective optimization approach not only achieves significant memory savings but also ensures faster deployment and reduced computational demands, making it scalable and efficient solution for world applications.
32


Reviews:
[{"review_id": "QW8fQdqYKV", "reviewer": "Reviewer_mmrV", "summary": "Multi-language and Multi-task speech processing remains as a very challenging issue. In this paper, author cope with this challenge via Multi-objective optimization.  Typically in the literature researchers tend to use one loss term for each task, for instance, a prediction loss to learn a good representation, language-specific CTC for transcription, cross-entropy for translation and fairness constraints for underrepresented languages.   Multi-objective optimization  provides a principled approach for handling conflicting training objectives. More specifically, this paper proposes to first optimize the lower level loss , the self-supervised loss l_u. Second, it feed its solution into upper level losses such as combining CTC and cross entropy. This hierarchical approach can keep moving on . Experimental results show its effectiveness.", "questions": "Is this approach applicable to broad speech tasks such as speaker verfication and speech enhancement? How reboust is it?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \nProposed a workable solution to hand the key problem facing speech processing.  \n\nWeaknesses: \nThe proposed solution looks fragile in real scenarios since it require carefully designing the hierarchical combination.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the strengths of our paper and noting that the proposed solutions can effectively address the key problem in speech processing. We hope our responses below to your comments address your concerns. \n\n**Q1: The proposed solution looks fragile in real scenarios since it requires careful design of the hierarchical structure.**\n\nWe propose and investigate a principled and practically deployable recipe: (i) measure conflict, (ii) place objectives across levels accordingly, (iii) enforce the lower‑level via a penalty schedule, and (iv) restrict dynamic weighting to conflicting layers.\n\nOur core empirical finding is that **the order** in which conflicting objectives are optimized—the **hierarchy**—influences model performance in multilingual multi‑task speech processing. Rather than relying on ad-hoc heuristics, we use **gradient cosine similarity** (Figure 5) to identify strongly conflicting objective pairs (e.g., ASR CTC vs. translation CE). This conflict analysis directly informs our grouping of objectives into separate optimization levels.\n\nLeveraging this insight, we develop algorithms through a **multilevel penalty‑based formulation**, where a time‑varying penalty parameter $\\eta$ dynamically controls the influence of lower‑level objectives. As shown in **Table 5** of the paper, tuning $\\eta$ via standard hyperparameter‑search procedures yields stable progress across levels **without manual intervention**. This dynamic control makes our scheme robust across penalty schedules.\n\nWe have extensively evaluated this hierarchical structuring of conflict objectives under a wide variety of model architectures, sizes, languages, and penalty schedules for multilingual speech recognition and translation, and observed consistent performance improvements. Therefore, we believe this would be a robust recipe in practice.\n\n**Q2: Is this approach applicable to broad speech tasks such as speaker verification and speech enhancement? How robust is it?**\n\nYes, our algorithms extend naturally to additional speech tasks with minimal modifications. Once a new task is defined by its corresponding loss with the task-specific dataset, it can be incorporated into the existing multilevel structure without altering the core optimization algorithms. The shared parameters will continue to be updated using the same conflict-aware optimization procedure already described. This flexibility reflects one of the strengths of our algorithm—it scales to more tasks without requiring architectural changes or task-specific adaptation.\n\nFor example, to add the following tasks:\n\n- **Speaker Verification**  \n  Add a cross-entropy loss: $\\ell_{\\text{sv}}(\\theta, \\phi_{\\text{sv}})$\n  where $\\phi_{\\text{sv}}$ are the speaker-verification head parameters.\n- **Speech Enhancement**  \n  Add a mean-squared error loss: $\\ell_{\\text{se}}(\\theta, \\phi_{\\text{se}})$\n  where $\\phi_{\\text{se}}$ are the enhancement-decoder parameters.\n\nThe overall shared-parameter update (cf. Eq. 8) incorporates these new losses:\n\n$$\\theta^{k+1} = \\theta^k - \\alpha \\Bigl( L_{\\mathrm{ASR}} + L_{\\mathrm{ST}} + L_{\\mathrm{SVE}} + L_{\\mathrm{Unsupervised}} \\Bigr),$$\nwhere\n$$L_{\\mathrm{ASR}} = \\sum_{t=1}^{T} \\lambda^k_{t,\\mathrm{ctc}} \\nabla_\\theta \\ell_{\\mathrm{ctc}},$$\n$$L_{\\mathrm{ST}} = \\eta_1 \\sum_{t=1}^{T} \\lambda^k_{t,\\mathrm{ce}} \\nabla_\\theta \\ell_{\\mathrm{ce}},$$\n$$L_{\\mathrm{SVE}} = \\eta_2 \\bigl( \\lambda^k_{\\mathrm{sv}} \\nabla_\\theta \\ell_{\\mathrm{sv}} + \\lambda^k_{\\mathrm{se}} \\nabla_\\theta \\ell_{\\mathrm{se}} \\bigr),$$\n$$L_{\\mathrm{Unsupervised}} = \\eta \\nabla_\\theta \\ell_{\\mathrm{u}}.$$\n\nOur simulations confirm that with the existing CTC, CE, and CPC objectives for ASR and ST tasks, our algorithm successfully mitigates conflicts among these tasks—demonstrating its robustness and broad applicability.\n\nTo further evaluate its extensibility to broader speech tasks, we conducted an additional experiment by adding a gender-classification task alongside multilingual ASR, using both the LibriSpeech train-clean-100 and AISHELL datasets. For gender classification, we trained our model on the LibriSpeech train-clean-100 split.\n\nOur results demonstrate that our proposed algorithms—VM-ASR variants **UECG** and **UCEG**—consistently outperform baseline methods across both ASR and gender classification tasks. In the **UECG** variant, we first optimize the unsupervised loss, followed by the English ASR loss, then the Chinese ASR loss, and finally the gender classification loss. In contrast, the **UCEG** variant follows a different optimization order: we first optimize the unsupervised loss, then the Chinese ASR loss, followed by the English ASR loss, and finally the gender classification loss. \n\n\n| Param | Task | Lang | Two-stage (PT+FT) | Joint PT+FT W/O MOO | VS-ASR | VC-ASR | VM-ASR (UECG) | VM-ASR (UCEG) |\n|-------|------|------|-------------------|--------------------|--------|--------|--------------|--------------|\n| **58M** | **ASR (WER)** | En (test-clean) | 8.0% | 7.3% | 7.7% | 7.2% | 6.9% | **6.8%** |\n|         |                 | En (test-other) | 18.3% | 17.8% | 18.1% | 17.6% | 17.5% | **17.3%** |\n|         |        **ASR (CER)**         | Zh              | 7.9%  | 7.4%  | 7.6%  | 7.1% | **6.6%** | 6.7% |\n|         |                 | **Average**     | 11.4% | 10.8% | 11.1% | 10.6% | 10.3% | **10.2%** |\n| **58M** | **Gender Classification (Accuracy)** | En (test-clean) | 97.3% | 97.4% | 97.6% | 97.7% | 98.1% | **98.2%** |\n|         |                 | En (test-other) | 90.8% | 91.4% | 92.3% | 92.8% | 93.2% | **93.4%** |\n\n**Q3: fold--> feed and other typos**\n\nGood catch! We have corrected the typos in the latest version of the paper."}, {"review_id": "qZXoMoXuPH", "reviewer": "Reviewer_95Wr", "summary": "In this work, the authors focus on the training interference among self-supervised training, speech recognition with CTC and speech translation with cross entropy criteria. They propose to consider the multi-objection optimization problem as multi-level multi-task optimization problem and penalize the minor level tasks, such as unsupervised and cross entropy for the translation task. The proposed training strategy shows good improvement for both multilingual  ASR and ST tasks at the cost of more memory and training time. A simple and light weight layer selection strategy is also examined and reduce the training memory and time significantly. However, there are many details are missing in the main paper, which are included in the appendix. Also, the experiment should include baselines from other research work for a comprehensive comparison.", "questions": "- two optimization sequences: USA and UAS, are discussed in the Table 3 and 4. They are not explained properly. Do ``USA'' mean the model is optimized according equ. 8 (Translation as minor task) while ``UAS'' chooses the ASR CTC task as secondary task and optimized with penalty parameter?\n- Do you try to use cross entropy to optimize the ASR task too? It may increase the backbone model shared between ASR and ST.\n\nTypo:\nline 240: \"VC-MSP, the bilevel penalty is initialized to η0 = 0 and increased by 0.02 each epoch\" -> \"VC-MSP, the bilevel penalty is initialized to η = 0 and increased by 0.02 each epoch\"", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- the multi-objection optimization problem is considered as a multi-level multi-task optimization problem with penalization applied to minor level tasks. \n- the proposed method achieves good improvement compared with the baselines\n- an efficient method is proposed to reduce the computation cost during training.\n\nWeaknesses:\n- There are many details are missing in the main paper, such as the efficient method MSP method. Thought those information is included in the appendix.\n-  The experiment should include baselines from other research work for a comprehensive comparison.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the strengths of our paper, which include achieving significant improvements and efficiently reducing computational cost. We hope our responses to your comments below address your concerns.\n\n\n**Q1: Details are missing in the main paper.**\n\nIn the revised manuscript, we have incorporated key content from the appendix into the main body of the paper to improve clarity and self-containment. Specifically:\n\n1. The technical discussion of **static vs. dynamic weighting** has been moved from Appendix B into **Section 3** of the main text.\n2. We have **merged Sections 3 and 4** to improve narrative flow and better integrate the core optimization formulation with its motivation.\n3. Full descriptions of the **Conformer** and **Whisper** models have been moved from Appendix G into **Section 5**, ensuring architectural details are accessible.\n4. **Figure 5** has been relocated from Appendix E into **Section 5.1**, where it now serves as a key component in motivating our method.\n\nThese changes aim to make the paper easier to follow and ensure that all essential methodological and experimental components are included in the main text.\n\n**Q2: The experiments should include baselines from other research work for a comprehensive comparison.**\n\nWe agree that comprehensive baselines are essential for evaluating generalization. In addition to using the Conformer model, our experiments include two **widely recognized and competitive models** from prior work: **Wav2Vec2** and **Whisper**. These models are strong baselines with proven effectiveness in multilingual and multi-task speech processing tasks [1,2,3]. To ensure fairness, we evaluated all models under the **same experimental conditions**, including training data and optimization setup.\n\nIn future work, we plan to extend our study to larger datasets, more tasks, and additional model architectures to further evaluate the generality of our approach.\n\n> 1. Radford, A. et al. \"Robust speech recognition via large-scale weak supervision.\" \n> 2. Pratap, V. et al. \"Scaling speech technology to 1,000+ languages.\" \n> 3. Team, Gemini, et al. \"Gemini: a family of highly capable multimodal models.\"\n\n**Q3: two optimization sequences: USA and UAS, are discussed in Tables 3 and 4. They are not explained properly.**\n\nWe thank the reviewer for this observation.  The descriptions of **USA** and **UAS** are originally provided in **Section 5.2, line 267**, and follow the optimization ordering principle used in **penalty-based optimization**. Specifically:\n\n- **UAS** refers to the optimization sequence: *Unsupervised → ASR → Translation*. That is, we first optimize the self-supervised loss, then the ASR objective (CTC loss), and finally the translation objective (cross-entropy loss).\n- **USA** refers to the sequence: *Unsupervised → Translation → ASR*. That is, we first optimize the self-supervised loss, then the translation objective, and finally the ASR objective.\n\nThese sequences reflect different priority orderings in the multilevel structure (Equation 8), where upper-level objectives are weighted more heavily, and lower-level objectives are included as penalty terms.\n\nTo avoid confusion, we have **explicitly added** the following clarification to the updated version of the paper (in Section 5.2):\n\n> “We tested two optimization sequences: UAS (self-supervised → speech recognition → translation) and USA (self-supervised → translation → recognition). UAS means the unsupervised loss is optimized first, followed by the ASR objective (CTC loss), and then the translation objective (cross-entropy loss). For USA, the unsupervised objective is optimized first, followed by the translation objective and then ASR.”\n\nWe hope this resolves the concern and makes the optimization orderings clearer.\n\n**Q4: Do you try to use cross entropy to optimize the ASR task too? It may increase the backbone model shared between ASR and ST.**\n\nWe appreciate the reviewer’s suggestion. In our experiments, we intentionally use **CTC** as the objective for ASR, rather than cross-entropy. CTC is a **sequence alignment–based criterion** that is specifically designed to handle input-output length mismatches without requiring frame-level alignment or teacher forcing, making it particularly well-suited for ASR [1,2,3]. \n\nWhile using cross-entropy for ASR could lead to tighter coupling between ASR and ST objectives by sharing a decoder, it would also require architectural changes (e.g., adding an autoregressive decoder for ASR). Therefore, we follow standard practice by using **CTC for ASR** and **cross-entropy for ST**, which also aligns well with our goal of analyzing and mitigating gradient conflicts between distinct objectives.\n\n> 1. Watanabe, S. et al. \"Hybrid CTC/attention architecture for end-to-end speech recognition.\"\n> 2. Baevski, A. et al. \"wav2vec 2.0: A framework for self-supervised learning of speech representations.\"\n> 3. Gulati, A. et al. \"Conformer: Convolution-augmented Transformer for Speech Recognition.\"\n\n**Q5: Typo: line 240**\n\nThanks a lot for catching the typos! We have corrected the typo."}, {"review_id": "UXc2XmgTWf", "reviewer": "Reviewer_Ljor", "summary": "This paper focuses on the training of multilingual speech processing models that perform both transcription and translation for multiple languages. The challenge is that each task has its own objective function, and therefore this can be viewed as a multi-objective optimization problem. The standard approach is to either (1) do multistep pre-training and fine-tuning phases with different objectives on different data or (2) linearly combine all objectives into a single objective. The authors argue that a more nuanced multi-objective approach is better. \n\nThey propose three training methods variants (VS-MSP, VC-MSP, VM-MSP) and perform experiments on CoVOST2 and LibriSpeech/AIShell. Results in WER and BLEU are generally promising.", "questions": "**W1.** The models used here have a large set of what authors called backbone parameters \\theta that are shared across language/tasks and another collection of language/task-dependent parameters \\phi. The \\theta parameter is involved in the unsupervised loss term, whereas \\phi are involved in the various supervised loss terms. I believe the fact that there are independent \\phi parameters for different objective means that there is not so much \"gradient conflict\" in the objectives that are most likely to have conflict. \n\nFor example, suppose I have a single decoder that performs both translation and transcription, then it is likely the two different objectives will cause conflict. However, in the conformer model for example, the translation is performed by the decoder while the transcription is performed with CTC on the encoder. Another example: if transcription for Chinese and English is part of the same CTC with a shared vocabulary, then there is higher chance of gradient conflict. However, if the two languages are handled by different heads, then there will be less conflict. \n\nIn other words, I think the proposed methods and results are tied to a certain type of architecture that has a nice division of shared and task-dependent parameters. If the authors agree with this assessment, I would recommend the following changes to the paper: \n\n - Discuss the model architecture in more detail (and include Figure 7) in the main text\n - Add the disclaimer that the proposed methods are most suited for this kind of architecture. Add in the Limitation how things might differ for different architectures. \n\n\n**W2.** The appendix is quite long. I usually do not need to pay too much attention to the appendix in reviewing but I find that for this paper I needed to read the appendix in detail to understand what is being done. Below are some parts that I think deserve to be in the main text:\n - Technical discussion of static vs dynamic weighting \n - How Equations 5-8 actually optimizes the objectives that are defined (especially it is hard to see how the constraints are handled just from the main text)\n - Figure 5 motivating some amount of gradient conflict in practice. \n - Figure 7 and details of the model architecture as mentioned previously \n\nTo reclaim space, I think the following parts of the main text can be reduced: \n - I don't think Fig 1 actually reflects what is happening in practice and can be removed. For example, do we really know that the unsupervised objective has broader support like shown in the green curve? If not, it is best not to include it. \n - There is significant repetition in Sec 3 and Sec 4 first trying to motivate methods in the abstract then adapting it to the specific setting of speech processing. I think it is best just to explain everything in the specific setting considering W1. \n\n**W3.** I think the language needs to be rigorous when talking about optimization. Here are some examples that bothered me a bit: \n - Line 565 \"ensuring guaranteed convergence to Pareto stationary solutions\"; Line 156 \"ensuring the attainment of a Pareto stationary point.\" -> Is there really guarantees in convergence or attainment? If not, better to say \"encouraging convergence\" \n - Line 120: \"while uniquely narrowing the search region of the optimal solutions to a representation preserving subspace.\" First what is a representation preserving subspace? Second, does your update rule actually narrow the search region? \n - Line 148: \"A higher degree of conflict would undermine its role in narrowing the search space for a common optimal point\". I think this doesn't fit with the definition of pareto optimality, which by construction is about defining what counts as \"optimal\" despite conflicts in objectives.\n - Eq 4. has multiple nested \"such that\" constraints. It's understandable what is meant, but is there a better way to write this?\n\n**W4.** Currently the related work sections in the main text and appendix covers \"Multilingual speech recognition and translation\" and \"Multi-task Learning for Speech Recognition\" nicely, but I would also recommend adding a paragraph on \"Multi-objective Optimization\".  First, there have been some work that explored multi-objective optimization for speech processing which may be worth mentioning. For example, [1] performs multi-objective optimization of hyperparameters on accuracy and size/speed objectives. The actual method is very different from yours, but it may be worth mentioning how the problem differs. Second, your proposed method may be viewed as a form of multi-objective optimization called lexicographic method [2]. So it will be good to cite that. Another related citation is [3] which is also similarly motivated, though it is focused on RL. \n\n - [1] Evolution-Strategy-Based Automation of System Development for High-Performance Speech Recognition https://ieeexplore.ieee.org/abstract/document/8470178\n - [2] Survey of multi-objective optimization methods for engineering https://link.springer.com/article/10.1007/s00158-003-0368-6\n - [3] Lexicographic Multi-Objective Reinforcement Learning https://arxiv.org/pdf/2212.13769", "limitations": "Currently there is not much discussion of Limitation (besides \" further theoretical analysis would be an\ninteresting direction for future research\".) As mentioned in W1, talking about the results and methods being possibly dependent on the architecture would be worth mentioning as a Limitation (assuming the authors agree).", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n - S1. Tackles a well-known challenge in multilingual speech processing \n - S2. Performs a comprehensive set of experiments using different Conformer and Whisper models\n\nWeakness:\n - W1. The positive results are probably tied to the architecture used. I am not sure if the VS-MSP, VC-MSP, VM-MSP methods make sense if the architecture does not assume a backbone and different heads for different objectives. \n - W2. Many of the important and interesting details are in the appendix. The main text itself was not as insightful as it could be. \n - W3. Some of the description/language relating to optimization is not rigorous. \n - W4. Related work in multi-objective optimization can be made more detailed. \n\nAll these are points that can be addressed with a writing revision. Please see below for suggestions.", "was_revised": true, "final_justification": "I've raised the score from 3 to 4 reflecting author's satisfactory response.", "rebuttal": "We thank the reviewer for recognizing the strengths of our paper, which include addressing a well-known challenge in multilingual speech processing. We also appreciate the detailed review that will further strengthen our work. We hope that our responses to your comments below fully address your concerns.\n\n**Q1: Results are tied to a certain type of architecture.**\n\nWe appreciate the reviewer’s thoughtful question. In our study, we evaluated the proposed method across three speech models—**Conformer**, **Wav2Vec2**, and **Whisper**—all of which are widely employed in speech processing [1,2].\n\nWhile the use of separate heads may reduce gradient conflict at the output level, it does **not eliminate conflicts within the shared encoder**, which contains the majority of the model parameters. As shown in **Figure 3**, we observe significant gradient conflicts even when task-specific heads are used, highlighting that the shared representation learning remains a critical bottleneck. Our multi-objective optimization algorithm is specifically designed to **address these conflicts in the shared layers**.\n\nWe agree that our current results and analysis are grounded in architectures with a modular separation between shared and task-specific parameters. However, we note that this architectural pattern is **prevalent in modern multilingual and multitask speech systems**. Investigating how our method behaves under alternative designs—such as models with a **shared decoder** for both transcription and translation—is beyond the scope of this paper, and is a valuable direction for future work.\n\nFollowing the reviewer’s suggestion, we have added the following into the **Conclusions and Limitations** section:\n\n>**Our evaluation focused on architectures that exhibit a clear separation between shared and task-specific parameters—for example, a shared encoder followed by distinct classification heads for ASR and translation. This architectural pattern is common in speech processing, but how the method generalizes to more tightly coupled architectures—such as models with a unified decoder shared across tasks—remains an open question. In such cases, task-specific conflicts may propagate more deeply through shared components, potentially requiring alternative optimization strategies or additional regularization. Investigating these scenarios is a promising direction for future research.** \n\n> 1. Burchi, M. et al. \"Multilingual audio-visual speech recognition with hybrid ctc/rnn-t fast conformer.\"\n> 2. De Zuazo, X. et al. \"Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languages.\"\n\n\n**Q2: The appendix is quite long.**\n\nFollowing the reviewer’s suggestion, we have moved the following parts into the main text. \n\n**i) Static vs. dynamic weighting:** We moved the discussion of static vs. dynamic weighting from Appendix B to Section 3.\n\n**ii) Equations 5-8:** We moved the explanation of Equations 5–8 from the appendix to Section 3 for clarity.\n\nSpecifically, we clarify that Equations 5–8 describe a **penalty-based multilevel optimization**, where each objective is optimized sequentially while incorporating lower-priority objectives as penalties, scaled by a time-varying parameter $\\eta$. This mechanism encourages the model to prioritize upper-level objectives while still accounting for progress on lower levels.\n\n**iii) Fig 1** is intended as a **conceptual illustration** adapted from Fig 3.2.1 in [1]. In that figure, the geometry of constrained multi-objective optimization is used to show how varying the constraint bound affects the feasible set of the primary objective.\n\nIn our adaptation, we use the CPC loss as a constraint-like term that influences the optimization landscape of the main tasks. The figure does **not represent actual empirical distributions**, but rather **visualizes the intuition** that if the unsupervised constraint is too tight, it may overly restrict the optimization; whereas if it is too loose, it may have negligible influence. As this figure is a conceptual illustration, we have removed the figure and directly cited [1].\n \n**iv) Fig 5 and 7:** We moved these figures to Section 5 (Experimental Results and Findings) and added corresponding descriptions.\n\n**v) Sec 3 and 4:** In the revised manuscript, we have **merged Sec 3 and 4 into a single unified section** to streamline the presentation and eliminate redundancy. Repeated explanations and motivating sentences have been removed or consolidated for clarity.\n\nHope our revision will address your concern on the writing. \n\n**Q3: Language needs to be rigorous.**\n\nTo make the language more rigorous we have made the following changes:\n\n**Line 565:** We have replaced \"ensuring the attainment...\" with \"encouraging convergence\". \n\n**Line 120:** We appreciate the clarification request. We have (i) replaced **“subspace”** with **“representation‑preserving neighborhood”** and (ii) clarified how “narrowing” is implemented.\n\n- **“Representation‑preserving neighborhood.”**  \n  We define it as the **sublevel set** of the self‑supervised loss around its minimum:\n\n$$\\mathcal{R}_\\delta := \\\\{\\theta: \\ell_u(\\theta) \\le \\ell_u^\\star + \\delta\\\\}, \\quad \\ell^*_u := \\min\\_{\\theta} \\ell_u(\\theta).$$\n\nStaying in $\\mathcal{R}_\\delta$ keeps the encoder representation close to the invariances encouraged by the self‑supervised objective.\n\n- **Does the update actually “narrow” the search region?**  \n  We do **not** claim a projection onto $\\mathcal{R}_\\delta$. Instead, our training uses a **penalized surrogate** of the $\\varepsilon$‑constraint method ([1], 3.2): we add the penalty term $\\eta\\left(\\ell_u(\\theta)-\\ell_u^\\star\\right)$ to the vector objective. Increasing $\\eta$ **restricts** the iterates to remain near $\\mathcal{R}\\_\\delta$. In the limit of large $\\eta$ (or with an explicit $\\varepsilon$‑constraint), this corresponds to solving the $\\varepsilon$‑constraint formulation, which **does** reduce the feasible set. Our revised text therefore states:  \n  **“restricting the search region to a representation‑preserving neighborhood that continues to intersect the Pareto set,”** emphasizing a restriction rather than a literal subspace or a guaranteed projection.\n\n> 1. Miettinen, K. \"Nonlinear multiobjective optimization.\" 1999.\n\n**Line 148:** We agree that Pareto optimality is defined independently of the degree of conflict among objectives. Our statement is not about the existence of the Pareto set but about the algorithmic effect of gradient conflict on gradient‑based multi‑objective optimization. In methods that seek a common descent direction (e.g., MGDA\\MoDo‑type updates), the set of feasible common descent directions\n$$\\mathcal{C}(\\Theta):=\\\\{d:\\left<\\nabla\\ell_m(\\Theta),d\\right>\\leq0, \\forall m\\\\}$$\n\n**shrinks as gradient conflict increases** (i.e., as pairwise cosine similarities become more negative). When this cone is small or degenerate, it becomes harder to make simultaneous progress on all objectives, even though Pareto‑optimal solutions still exist.\n\nOur lower‑level constraint is introduced precisely because—empirically—it is mostly aligned with the supervised objectives (Appendix E, Fig. 5). Under such alignment, constraining it guides the optimization to a “representation‑preserving” region that still intersects the Pareto set while increasing the chance that a computable common descent direction exists. If, counterfactually, the lower‑level objective was strongly conflicting, then constraining it would indeed be unhelpful for the solver—this is exactly the point we intended to convey.\n\nTo avoid any misunderstanding, we have revised the manuscript to make clear that our claim concerns optimization dynamics and the size of the common‑descent cone, not the definition or existence of Pareto‑optimal solutions.\n\n>**its gradient update direction should exhibit minimal conflict with the upper‑level objectives. When such alignment holds, the constraint restricts the feasible region to a representation‑preserving neighborhood that still intersects the Pareto set and admits a common descent direction for gradient‑based solvers; if the lower‑level objective were strongly conflicting, the constraint would not aid optimization.**\n\n**Eq 4:** Thank you for the suggestion. We agree that alternative displays can be clearer. For consistency with the rest of our notation and to avoid ripple changes across sections, we keep Eq. (4) as written in this revision. As the reviewer notes, the current form is understandable.\n\n**Q4: Related work section.**\n\nWe have added the following in the related work section:\n\n**Multi-objective optimization** aims to optimize multiple, often conflicting, objectives simultaneously and has been applied in various domains including engineering [1], finance [2], and decision making [3]. Multi-objective optimization has also been used for solving\nlexicographic multi-objective problems [4,5]. It has also seen applications in speech processing. In previous work, multi-objective optimization has been applied at the system development level, using evolutionary strategies to jointly optimize recognition accuracy and model size by tuning meta-parameters such as model topology and training configuration [6]. In contrast, our work focuses on loss-level conflicts during the training of multilingual and multitask models, where multiple learning signals interact within a shared backbone. \n\n> 1. Cheng, Y. \"Multiobjective optimum design of structures with genetic algorithm and game theory: Application to life-cycle cost design.\" \n> 2. Doumpos, M. et al. \"Multi-objective optimization models in finance and investments.\"\n> 3. Deb, K. et al. \"Multi-objective optimization.\"\n> 4. Marler, R. el al. \"Survey of multi-objective optimization methods for engineering.\"\n> 5. Skalse, J. et al. \"Lexicographic multi-objective reinforcement learning.\"\n> 6. Moriya, T. et al. \"Evolution-strategy-based automation of system development for high-performance speech recognition.\""}, {"review_id": "Un1Z48TCfx", "reviewer": "Reviewer_UXqi", "summary": "The paper proposes a method that addresses the multiple objectives when training a speech translation and speech recognition model for a handful of languages. These training objectives are in conflict with each other, the authors propose to remedy this by task/language specific augmentation parameters and alternating between pretraining (language model objective) and task-specific training (ST and ASR).", "questions": "N/A", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "There are several points where the paper is hard to understand and first (to me) suggested to go into different directions than it ultimately did. The paper has a long discussion of conflicting gradients, a definition of \"Pareto stationary\" solutions. The method is first described by nested minimization (which is hard to understand since after minimization to obtain parameter values, you cannot change them again). But the solution is the introduction of task-specific parameters but otherwise relies on task/language specific weights. The nested minimization in the Figure 2 leads ultimately to the introduction of additional weight (\"penalty\") parameters. Their role is not very well described, although a key feature is that they increasingly downweight the secondary/tertiary objectives.\n\nThere is a very long appendix (which I did not read in detail) that may clear up this confusion - but the main body of the paper should be more clear about these points.\n\nThe method has a lot of hyper parameters, such as task/language specific weights, the penalty parameters. However, the selection of these hyper parameters is not described in the main body of the paper.\n\nBased on what the paper ultimately does, it shows clear empirical evidence that the proposed added parameters and training scheme works well. The novelty of the approach, as I understand it, is limited (task-specific augmentation parameters and curriculum training are common techniques), but there is still sufficient substance here for acceptance of the paper.\n\nThere is some nice analysis how gradient conflicts evolve over time (Figure 3), although they do not seem to change much during training.", "was_revised": true, "final_justification": "The feedback by the authors cleared up some of my confusion, and I would be happy to see the paper accepted.", "rebuttal": "We thank the reviewer for appreciating our work. We hope our responses to your comments below address your concerns. \n\n**Q1: Nested minimization is hard to understand; after minimization you cannot change parameters again.**\n\nIn the paper, the nested formulation is a conceptual statement of priorities. Operationally, we implement it via a standard penalty‑based conversion to an unconstrained vector optimization (Eq. 9) and then use dynamic multi‑objective updates (MoDo) to obtain a conflict‑avoiding common‑descent direction (Eqs. (10)–(12)), with explicit first‑order update rules summarized in Table 1 and Algorithms 2–3. Thus, no literal “re‑minimization after minimization” occurs during training—the solver performs single‑loop gradient updates with penalties that encode the nesting.\n\n\n**Q2: Role of the penalty (‘additional weight’) parameters is not well described; they increasingly down‑weight lower levels.**\n\nIn the updated version of the paper, we clarify the role of the penalties in two places in the main text. First, we explain the $\\varepsilon$-constraint perspective: increasing $\\eta$ restricts iterates to a representation‑preserving neighborhood $\\mathcal{R}_{\\delta}$ around the self‑supervised minimizer, which encodes the lower‑level constraint during upper‑level optimization. Second, we give the concrete backbone update rules showing exactly where $\\eta$ (and $\\eta_1$ at the intermediate level) enters.\n\n**More explanation is provided in reviewer Ljor Q:3 (question regarding Line 120) section** \n\n**Q3: Many hyperparameters; selection not described in the main body.**\n\nWe report the core settings—optimizer, learning rates, and (crucially) the penalty schedule $\\eta$—directly in Section 4. The detailed hyperparameter‑selection process has been added to Appendix G (Experiment Setup): \n\n**Hyper-parameters.** We use grid search to optimize hyperparameters, including learning rate, batch size, step size of MoDo, and penalty parameter increasing rate. For both SSL pre-training and supervised fine-tuning, the backbone learning rate is consistently set higher than the classification parameter learning rate. The SSL pre-training phase starts with a learning rate of $\\alpha = 5 \\times 10^{-4}$ for 100 epochs, annealed by a factor of 0.1 every 20 epochs. Fine-tuning uses a maximum learning rate of $\\beta = 5 \\times 10^{-5}$, with a scheduler reducing the learning rate by a factor of 0.1 if the test loss does not improve within 10 epochs. All multi-objective models (VS-MSP, VC-MSP, and VM-MSP) and joint PT+FT models are trained for 200 epochs. For PT+FT, we pre-train the model for 200 epochs and fine-tune it for an additional 100 epochs. A batch size of 256 and AdamW optimizer are used for both self-supervised and supervised training. The same hyperparameter settings are applied across all training methods to ensure consistency and comparability.\n\n\n**Q4: The appendix may clear up confusion; main body should be clearer.**\n\nWe moved essential definitions and mechanisms to the main body: (i) conflict‑avoiding direction and its MoDo realization, (ii) the constrained formulation and its penalty‑based counterpart, and (iii) explicit per‑method backbone updates (Table 1) so readers need not consult the appendix to follow the training procedure. Appendix B provides derivations; the main text already contains the practical rules and schedules.\n\n\n**Q5: Novelty is limited but the empirical evidence is clear.**\n\nThank you for the assessment of the empirical gains. While our system leverages familiar components (shared encoder, task‑specific heads), our contribution is **not** a re‑branding of curriculum or static weighting. What is novel is how we **structure, operationalize, and measure** conflict in multilingual ASR–ST using a multilevel multi‑objective lens:\n\n1. **Formulation‑level novelty.**  \n   We cast multilingual ASR–ST as a **multilevel (lexicographic)** MOO problem in which the self‑supervised objective serves as an **ε‑constraint**, and supervised tasks are **separated across levels**. Unlike single‑level multi‑tasking or curricula with time‑varying weights, our **penalty‑based realization** yields **single‑loop** gradient updates that *implement* the hierarchy without nested solves (see Table 1).\n\n2. **Actionable hierarchy selection & diagnostics.**  \n   We show that **which objective resides at which level matters**, and we provide a **data‑driven rule**: measure gradient conflict, place objectives accordingly, and **anneal penalties** to encode priorities. Ablations justify the chosen hierarchy and schedule (Sec. 4).\n\n3. **Efficiency via targeted conflict handling.**  \n   We introduce **layer‑wise conflict selection** that computes dynamic weights **only where conflict is observed**, reducing compute/memory overhead **without** degrading WER/BLEU (Sec. 4.7).\n\n4. **Systematic conflict analysis for multilingual ASR–ST.**  \n   We provide a **comprehensive characterization** of conflict across languages, tasks, and layers, explaining *why* multilevel separation helps here and when it should transfer to other architectures (Sec. 4; Limitations).\n\nTogether, these elements form a principled, practically deployable recipe: (i) measure conflict, (ii) place objectives across levels accordingly, (iii) enforce the lower‑level via a penalty schedule, and (iv) restrict dynamic weighting to conflicting layers. We believe this constitutes a substantive methodological contribution beyond conventional task‑specific heads or generic curricula.\n\n**Q6: Gradient conflicts don’t change much during training.**\n\nGreat observation! We observe a similar phenomenon under two‑stage training (without multi-objective optimization): conflicts persist in the same layers (Figure 3). Our method addresses this by (i) separating highly conflicting objectives across levels (VC/VM‑MSP), and (ii) targeting the conflicting layers when computing the update direction, which is precisely where conflicts concentrate."}]

Author rebuttal:

Meta-review: **Summary**

This paper identifies the challenges of optimizing for multiple speech tasks simultaneously.  These tasks have distinct objective functions and the authors explore different approaches to combining these.

**Reasons to accept**

The reviewers were consistently positive on this work. They identified this as an important task, and the proposed analyses and solutions to be well motivated.  Ljor summarized this as follows “the contribution are not a new exciting concepts, but the experiments are sound and can serve as good reference for people working on this problem.”.

**Reasons not to accept**

General lack of enthusiasm, and a concern that the findings of this work won’t scale sufficiently well to influence other work. This leads to limitations of both novelty and significance.   

**Decision rationale**

The reviewers are quite positive on this work.  The limitations of the work stem primarily from the scope of the problem that’s being addressed.  There is good analysis and well motivated experiments that have been brought to bear on a particular problem.  These are sufficient enough reasons to accept.  To be more convinced in this decision, I would look for a more obvious statement of significance on this work addressing the question “why will a future reader cite this work?”  While I believe there are answers to this question, they are not centered by the authors themselves sufficiently to make it obvious to the reviewers or to me.
Decision: Accept (poster)