Kati8/ICLR_2023_2025 huggingface dataset:
authors: A F M Saif
authors: Lisha Chen
authors: Xiaodong Cui
authors: Songtao Lu
authors: Brian Kingsbury
authors: Tianyi Chen
reviews: 
Comments: Multitask multilingual ASR consists of several objectives that may conflict with each other, and existing multiobjective optimization (MOO) methods can train models in a conflicting-objective-aware manner to resolve the conflicts to some extent. This paper provides a comprehensive comparative analysis of 3 MOO methods on 2 tasks (ASR and S2TT) and a suite of languages. The paper shows consistent improvements of MOO-based methods over non-MOO-based baselines and ablates several design choices.
Strengths: 1. The paper shows that training with an conflicting-objective-aware optimization technique (e.g. based on MOO) improves performance on ASR and S2TT across a bunch of languages, which was a nice finding.
2. The paper performs comprehensive analyses for each of their algorithm’s components e.g. the ordering of the tasks, the choice of the penalty hyperparameter, comparisons for 2 model sizes, etc.
3. The performance improvements are nice and consistent across settings.
Weaknesses: 1. The paper emphasizes multitask and multilingual learning but only considers 2 tasks (ASR and S2TT). It was unclear to me whether the findings of the paper would generalize to a true multitask setup with > 2 tasks. It does already consider several languages, which is great.
2. There are some missing baselines that I believe are needed to ensure that it is indeed conflicting objectives that cause degraded performance, and that MOO solves them. See the ‘Questions’ section for a description of these baselines.
3. Since the proposed methods are more resource-heavy than baseline, it would be fairer to report a memory-/speed-matched comparison between baselines and the proposed methods.
4. The related work section mentions that multi-task learning for joint ASR-S2TT has seen limited exploration but cites a range of papers that indeed do tackle joint ASR-S2TT; I recommend removing the ‘limited exploration’ phrase. Also, worth citing the Whisper paper (https://arxiv.org/abs/2212.04356) as an example of a model that does joint ASR and S2TT>
5. Writing: The writing can be made crisper as it was a little confusing at times. Concretely, throughout the paper you refer to ASR and S2TT tasks as ‘multitask ASR’; but S2TT is not an ASR task. I recommend calling them ‘multitask speech-to-text’ tasks for example. Also, the equations in Sections 4.1 - 4.3 consist of equations that are difficult to parse; I recommend simplifying them a little. The discussion in Section 5 can be made significantly crisper; there is no need to repeat many of the same numbers from the table again in the text.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 2
reviews: 
Comments: The paper shows advantages of integrating unsupervised loss as constraining objectives within a multilevel multi-objective optimization. The paper experimented with LibriSpeech and AISHELL and CoVoST v2 dataset for both ASR and speech-to-text translation tasks and has shown detailed results indicating segregating highly conflicting objectives into different optimization levels.
Strengths: Novel approach in optimizing the multi-objective for ASR, S2TT tasks yielding improved results for Speech recognition, S2TT. The experiments and approach have been well-documented 
It explained the reasoning of optimization order in MLO yielding different outcomes where UAS optimization sequence yields superior performance
It also shows that choosing right penalty parameter playing a crucial role in MLO based ASR training
Weaknesses: The experimental results provided a stronger base for the results, however it could be helpful to have more theoretical reasoning for the same
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4
reviews: 
Comments: This work explores the paradigm of multi-objective optimization (MOO) for multitask multilingual speech processing and proposes three multi-objective training objectives to improve ASR and speech-to-text translation (S2TT) across multiple languages. These objectives are designed by separating out the supervised/unsupervised/task-specific/language-specific losses into different optimization levels. The authors demonstrate that such multi-level optimization (MLO) is effective and provides significant performance gains compared to other baselines across multiple languages for both ASR and S2TT.
Strengths: * The authors are the first to explore the use of multi-objective optimization at different optimization levels for ASR and S2TT across multiple languages.
* The paper is structured well and was easy to read.
* The authors derive consistent gains across both tasks and multiple languages using the proposed multi-level optimization technique.
Weaknesses: * The reported baseline WERs and BLEU scores on CoVoST 2 appear to be significantly weaker than the state-of-the-art. The authors should clarify this further.  

* The intuition for the proposed optimization levels could be motivated further. I elaborate on this point further in my questions to the authors below.  

* The experimental section has many missing details that I elaborate in my questions below.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3