Title: Objective Soups: Multilingual Multi-Task Modeling for Speech Processing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multilingual speech recognition

automatic speech translation

multi-objective optimization

conflict-aware training
Abstract: The need for training multilingual multi-task speech processing (MSP) models that perform both automatic speech recognition and speech-to-text translation is increasingly evident. However, a significant challenge arises from the conflicts among multiple objectives when using a single model. Multi-objective optimization can address this challenge by facilitating the optimization of multiple conflicting objectives and aligning the gradient updates in a common descent direction. While multi-objective optimization helps avoid conflicting gradient updates, a critical issue is that when there are many objectives, such as in MSP, it is often {\em difficult to find} a common descent direction. This leads to an important question: Is it more effective to separate highly conflicting objectives into different optimization levels or to keep them in a single level? To address this question, this paper investigates three multi-objective MSP formulations, which we refer to as \textbf{objective soup recipes}. These formulations apply multi-objective optimization at different optimization levels to mitigate potential conflicts among all objectives. To keep computation and memory overhead low, we incorporate a lightweight layer‑selection strategy that detects the most conflicting layers and uses only their gradients when computing the conflict‑avoidance direction. We conduct an extensive investigation using the CoVoST v2 dataset for combined multilingual ASR and ST tasks, along with the LibriSpeech and AISHELL-1 datasets for multilingual ASR, to identify highly conflicting objectives and determine the most effective training recipe among the three proposed multi-objective optimization algorithms.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method that addresses the multiple objectives when training a speech translation and speech recognition model for a handful of languages. These training objectives are in conflict with each other, the authors propose to remedy this by task/language specific augmentation parameters and alternating between pretraining (language model objective) and task-specific training (ST and ASR).\n\nSTRENGTHS AND WEAKNESSES:\nThere are several points where the paper is hard to understand and first (to me) suggested to go into different directions than it ultimately did. The paper has a long discussion of conflicting gradients, a definition of \"Pareto stationary\" solutions. The method is first described by nested minimization (which is hard to understand since after minimization to obtain parameter values, you cannot change them again). But the solution is the introduction of task-specific parameters but otherwise relies on task/language specific weights. The nested minimization in the Figure 2 leads ultimately to the introduction of additional weight (\"penalty\") parameters. Their role is not very well described, although a key feature is that they increasingly downweight the secondary/tertiary objectives.\n\nThere is a very long appendix (which I did not read in detail) that may clear up this confusion - but the main body of the paper should be more clear about these points.\n\nThe method has a lot of hyper parameters, such as task/language specific weights, the penalty parameters. However, the selection of these hyper parameters is not described in the main body of the paper.\n\nBased on what the paper ultimately does, it shows clear empirical evidence that the proposed added parameters and training scheme works well. The novelty of the approach, as I understand it, is limited (task-specific augmentation parameters and curriculum training are common techniques), but there is still sufficient substance here for acceptance of the paper.\n\nThere is some nice analysis how gradient conflicts evolve over time (Figure 3), although they do not seem to change much during training.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the training of multilingual speech processing models that perform both transcription and translation for multiple languages. The challenge is that each task has its own objective function, and therefore this can be viewed as a multi-objective optimization problem. The standard approach is to either (1) do multistep pre-training and fine-tuning phases with different objectives on different data or (2) linearly combine all objectives into a single objective. The authors argue that a more nuanced multi-objective approach is better. \n\nThey propose three training methods variants (VS-MSP, VC-MSP, VM-MSP) and perform experiments on CoVOST2 and LibriSpeech/AIShell. Results in WER and BLEU are generally promising.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n - S1. Tackles a well-known challenge in multilingual speech processing \n - S2. Performs a comprehensive set of experiments using different Conformer and Whisper models\n\nWeakness:\n - W1. The positive results are probably tied to the architecture used. I am not sure if the VS-MSP, VC-MSP, VM-MSP methods make sense if the architecture does not assume a backbone and different heads for different objectives. \n - W2. Many of the important and interesting details are in the appendix. The main text itself was not as insightful as it could be. \n - W3. Some of the description/language relating to optimization is not rigorous. \n - W4. Related work in multi-objective optimization can be made more detailed. \n\nAll these are points that can be addressed with a writing revision. Please see below for suggestions.\n\nQUESTIONS:\n**W1.** The models used here have a large set of what authors called backbone parameters \\theta that are shared across language/tasks and another collection of language/task-dependent parameters \\phi. The \\theta parameter is involved in the unsupervised loss term, whereas \\phi are involved in the various supervised loss terms. I believe the fact that there are independent \\phi parameters for different objective means that there is not so much \"gradient conflict\" in the objectives that are most likely to have conflict. \n\nFor example, suppose I have a single decoder that performs both translation and transcription, then it is likely the two different objectives will cause conflict. However, in the conformer model for example, the translation is performed by the decoder while the transcription is performed with CTC on the encoder. Another example: if transcription for Chinese and English is part of the same CTC with a shared vocabulary, then there is higher chance of gradient conflict. However, if the two languages are handled by different heads, then there will be less conflict. \n\nIn other words, I think the proposed methods and results are tied to a certain type of architecture that has a nice division of shared and task-dependent parameters. If the authors agree with this assessment, I would recommend the following changes to the paper: \n\n - Discuss the model architecture in more detail (and include Figure 7) in the main text\n - Add the disclaimer that the proposed methods are most suited for this kind of architecture. Add in the Limitation how things might differ for different architectures. \n\n\n**W2.** The appendix is quite long. I usually do not need to pay too much attention to the appendix in reviewing but I find that for this paper I needed to read the appendix in detail to understand what is being done. Below are some parts that I think deserve to be in the main text:\n - Technical discussion of static vs dynamic weighting \n - How Equations 5-8 actually optimizes the objectives that are defined (especially it is hard to see how the constraints are handled just from the main text)\n - Figure 5 motivating some amount of gradient conflict in practice. \n - Figure 7 and details of the model architecture as mentioned previously \n\nTo reclaim space, I think the following parts of the main text can be reduced: \n - I don't think Fig 1 actually reflects what is happening in practice and can be removed. For example, do we really know that the unsupervised objective has broader support like shown in the green curve? If not, it is best not to include it. \n - There is significant repetition in Sec 3 and Sec 4 first trying to motivate methods in the abstract then adapting it to the specific setting of speech processing. I think it is best just to explain everything in the specific setting considering W1. \n\n**W3.** I think the language needs to be rigorous when talking about optimization. Here are some examples that bothered me a bit: \n - Line 565 \"ensuring guaranteed convergence to Pareto stationary solutions\"; Line 156 \"ensuring the attainment of a Pareto stationary point.\" -> Is there really guarantees in convergence or attainment? If not, better to say \"encouraging convergence\" \n - Line 120: \"while uniquely narrowing the search region of the optimal solutions to a representation preserving subspace.\" First what is a representation preserving subspace? Second, does your update rule actually narrow the search region? \n - Line 148: \"A higher degree of conflict would undermine its role in narrowing the search space for a common optimal point\". I think this doesn't fit with the definition of pareto optimality, which by construction is about defining what counts as \"optimal\" despite conflicts in objectives.\n - Eq 4. has multiple nested \"such that\" constraints. It's understandable what is meant, but is there a better way to write this?\n\n**W4.** Currently the related work sections in the main text and appendix covers \"Multilingual speech recognition and translation\" and \"Multi-task Learning for Speech Recognition\" nicely, but I would also recommend adding a paragraph on \"Multi-objective Optimization\".  First, there have been some work that explored multi-objective optimization for speech processing which may be worth mentioning. For example, [1] performs multi-objective optimization of hyperparameters on accuracy and size/speed objectives. The actual method is very different from yours, but it may be worth mentioning how the problem differs. Second, your proposed method may be viewed as a form of multi-objective optimization called lexicographic method [2]. So it will be good to cite that. Another related citation is [3] which is also similarly motivated, though it is focused on RL. \n\n - [1] Evolution-Strategy-Based Automation of System Development for High-Performance Speech Recognition https://ieeexplore.ieee.org/abstract/document/8470178\n - [2] Survey of multi-objective optimization methods for engineering https://link.springer.com/article/10.1007/s00158-003-0368-6\n - [3] Lexicographic Multi-Objective Reinforcement Learning https://arxiv.org/pdf/2212.13769\n\nLIMITATIONS:\nCurrently there is not much discussion of Limitation (besides \" further theoretical analysis would be an\ninteresting direction for future research\".) As mentioned in W1, talking about the results and methods being possibly dependent on the architecture would be worth mentioning as a Limitation (assuming the authors agree).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors focus on the training interference among self-supervised training, speech recognition with CTC and speech translation with cross entropy criteria. They propose to consider the multi-objection optimization problem as multi-level multi-task optimization problem and penalize the minor level tasks, such as unsupervised and cross entropy for the translation task. The proposed training strategy shows good improvement for both multilingual  ASR and ST tasks at the cost of more memory and training time. A simple and light weight layer selection strategy is also examined and reduce the training memory and time significantly. However, there are many details are missing in the main paper, which are included in the appendix. Also, the experiment should include baselines from other research work for a comprehensive comparison.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- the multi-objection optimization problem is considered as a multi-level multi-task optimization problem with penalization applied to minor level tasks. \n- the proposed method achieves good improvement compared with the baselines\n- an efficient method is proposed to reduce the computation cost during training.\n\nWeaknesses:\n- There are many details are missing in the main paper, such as the efficient method MSP method. Thought those information is included in the appendix.\n-  The experiment should include baselines from other research work for a comprehensive comparison.\n\nQUESTIONS:\n- two optimization sequences: USA and UAS, are discussed in the Table 3 and 4. They are not explained properly. Do ``USA'' mean the model is optimized according equ. 8 (Translation as minor task) while ``UAS'' chooses the ASR CTC task as secondary task and optimized with penalty parameter?\n- Do you try to use cross entropy to optimize the ASR task too? It may increase the backbone model shared between ASR and ST.\n\nTypo:\nline 240: \"VC-MSP, the bilevel penalty is initialized to η0 = 0 and increased by 0.02 each epoch\" -> \"VC-MSP, the bilevel penalty is initialized to η = 0 and increased by 0.02 each epoch\"\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
Multi-language and Multi-task speech processing remains as a very challenging issue. In this paper, author cope with this challenge via Multi-objective optimization.  Typically in the literature researchers tend to use one loss term for each task, for instance, a prediction loss to learn a good representation, language-specific CTC for transcription, cross-entropy for translation and fairness constraints for underrepresented languages.   Multi-objective optimization  provides a principled approach for handling conflicting training objectives. More specifically, this paper proposes to first optimize the lower level loss , the self-supervised loss l_u. Second, it feed its solution into upper level losses such as combining CTC and cross entropy. This hierarchical approach can keep moving on . Experimental results show its effectiveness.

STRENGTHS AND WEAKNESSES:
Strengths: 
Proposed a workable solution to hand the key problem facing speech processing.  

Weaknesses: 
The proposed solution looks fragile in real scenarios since it require carefully designing the hierarchical combination.

QUESTIONS:
Is this approach applicable to broad speech tasks such as speaker verfication and speech enhancement? How reboust is it?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-06