ARECHO: Autoregressive Evaluation via Chain-Based Hypothesis Optimization for Speech Multi-Metric Estimation
Authors: Jiatong Shi
Yifan Cheng
Bo-Hao Su
Hye-jin Shim
Jinchuan Tian
Samuele Cornell
Yiwen Zhao
Siddhant Arora
Shinji Watanabe
Venue: NeurIPS 2025 spotlight ()
Keywords: Speech Evaluation
Speech Assessment
Speech Profiling
Dynamic Classifier Chain
Primary area: applications
TL;DR: We present ARECHO, a token-based autoregressive speech assessment framework that models inter-metric dependencies to enable accurate, interpretable, and flexible evaluation across diverse speech tasks.

Abstract: Speech signal analysis poses significant challenges, particularly in tasks such as speech quality evaluation and profiling, where the goal is to predict multiple perceptual and objective metrics. For instance, metrics like PESQ (Perceptual Evaluation of Speech Quality), STOI (Short-Time Objective Intelligibility), and MOS (Mean Opinion Score) each capture different aspects of speech quality. However, these metrics often have different scales, assumptions, and dependencies, making joint estimation non-trivial. To address these issues, we introduce ARECHO (Autoregressive Evaluation via Chain-based Hypothesis Optimization), a chain-based, versatile evaluation system for speech assessment grounded in autoregressive dependency modeling. ARECHO is distinguished by three key innovations: (1) a comprehensive speech information tokenization pipeline; (2) a dynamic classifier chain that explicitly captures inter-metric dependencies; and (3) a two-step confidence-oriented decoding algorithm that enhances inference reliability. Experiments demonstrate that ARECHO significantly outperforms the baseline framework across diverse evaluation scenarios, including enhanced speech analysis, speech generation evaluation, and noisy speech evaluation. Furthermore, its dynamic dependency modeling improves interpretability by capturing inter-metric relationships. Across tasks, ARECHO offers reference-free evaluation using its dynamic classifier chain to support subset queries (single or multiple metrics) and reduces error propagation via confidence-oriented decoding.

ARECHO: Autoregressive Evaluation via Chain-Based Hypothesis Optimization for Speech Multi-Metric Estimation
Jiatong Shi
Yifan Cheng*
Bo-Hao Su Hye-jin Shim! Jinchuan Tian
Samuele Cornell
Yiwen Zhao Siddhant Arora Shinji Watanabe
Carnegie Mellon University
2 Huazhong University of Science and Technology j iiatongsscs. cmu.edu
Abstract
Speech signal analysis poses significant challenges, particularly in tasks such speech quality evaluation and profiling, where the goal is to predict multiple percep- tual and objective metrics. For instance, metrics like PESQ (Perceptual Evaluation of Speech Quality), STOI (Short-Time) Objective Intelligibility, and MOS (Mean Opinion Score) each capture different aspects of speech quality. However, these metrics often have different scales, assumptions, and dependencies, making joint estimation non-trivial. To address these issues, introduce ARECHO (Autore- gressive Evaluation via Chain-based Hypothesis Optimization), a chain-based versatile evaluation system for speech assessment grounded in autoregressive de- pendency modeling. ARECHO distinguished by three key innovations: (1) a comprehensive speech information tokenization pipeline; (2) dynamic classi- fier chain that explicitly captures inter-metric dependencies; and (3) a two-step confidence-oriented decoding algorithm that enhances inference reliability. Experi- ments demonstrate that ARECHO significantly outperforms the bascline framework across diverse evaluation scenarios, including enhanced speech analysis sis. speech generation evaluation, and, noisy speech evaluation. Furthermore, its dynamic dependency modeling improves interpretabilityy by capturing inter-metric relation- ships. Across tasks, ARECHO offers reference-free evaluation using its dynamic classifier chain support subset queries (single or multiple metries) and reduces етror propagation via confidence-oriented decoding.
1 Introduction
Speech assessment and profiling are essential components in the speech processing community, owing the inherently complex and multidimensional nature of speech signals (Huang et al. [022 Yi et al. 2022; Shi et al.) 2024; Torcoli et al. 2021). These signals typically encompass various attributes. such clarity, naturalness, emotional expressiveness, and acoustic quality, which challenging to characterize comprehensively. While subjective evaluation remains the gold standard for assessing speech quality due its capacity for nuanced and context-aware judgments, it suffers from several limitations, including inter-rater variability, limited scalability, and insufficient coverage of diverse evaluation dimensions (Huang al. 2022 Cooper et al.) 2024, Zielinski et al. 2008 [oizou 201 Jiménez et al. 2021 Naderi et al. (2200). Consequently, objective methods have emerged as scalable and consistent alternatives that aim approximate subjective judgments (Cooper et al.) 2024 Torcoli
et al. 2021 Shi et al. 2025a).
Prior work has introduced numerous specialized metrics targeting specific speech characteristics, including speech perceived quality, naturalness in synthesized speech, and paralinguistic features
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

such as emotion and speaker identity (Reddy et al. 2021; Sacki et al. 2022, Yi et al. 2022, Huang al. 2024c, Wu al. 2024; Goncalves et al., 2024] Dung al., 2024). Although these metrics provide valuable insights individually, they are often analyzed in isolation, overlooking potential inter-dependencies among them. Comprehensive profiling through multi-metric evaluation offers notable advantages, including greater efficiency, more holistic analysis, and the potential to leverage shared information across metrics (Zhang et al. 2024). In light of these benefits, recent research has explored unified frameworks that predict multiple speech assessment metrics concurrentyy (Kumar Tjandra et al. 2025 Shi et al. 2025b).
ct al. 2023
Despite their promise, unified multi-metric prediction systems present several key challenges stem- ming from the heterogeneity of speech metries, limited supervision, and the lack of inter-metric reasoning:
Challenge I: Diverse Scale Issues. Speech metrics vary significantly in scale and type, complicating joint modeling and optimization. For example, MOS (Mean Opinion Score) ranges from to 5 and SI-SNR unbounded over (-00, oo). Optimizing such diverse metrics with uniform loss functions L.) can lead to biased learning, overemphasizing metrics with larger numerical ranges while under-representing perceptually salient ones like MOS.
Challenge II: Limited Data Availability. Many metrics rely on auxiliary references that are often unavailable in practical scenarios. PESQ requires clean reference audio, WER depends transcripts, and speaker similarity requires paired utterances. As a result, real-world datasets are often partially la- beled, demanding systems that can flexibly adapt semi-supervised or weakly supervised conditions and support arbitrary subsets of supervision targets during both training and inference.
Challenge III: Dependency Modeling with Flexible Control. Existing frameworks such as Uni- VERSA (Shi et al.) 2025b) and TorchSquim (Kumar et al., 2023) typically predict all metrics independently and in parallel. This limits their ability to leverage the inherent dependencies among metrics, for instance, the natural correlation between intelligibility and naturalness, which could otherwise inform both reasoning and prediction. Moreover, in the presence of incomplete labeling, parallel prediction becomes inefficient and less effective in generalizing from available cues.
To address these challenges. we propose ARECHO (Autoregressive Evaluation via Chain-based Hypothesis Optimization), a flexible and dependency-awee awaare evaluation framework for speech assess- ment. ARECHO formulates speech evaluation as a chain-based prediction task, utilizing a dynamic classifier chain to model inter-metric relationships in an autoregressive fashion. To handle the diverse scales and types of speech metrics (e.g., categorical vs. continuous, bounded vs. unbounded), we design a speech tokenization pipeline that robustly encodes these heterogeneous metric values into a unified token space. This formulation not only improves predictive performance but also improves interpretability through structured dependency reasoning
The key contributions of this work are as follows:
(1) We design a comprehensive speech tokenization pipeline that explicitly handles the diversity of metric types and scales by encoding them into a consistent and learnable token representation.
(2) We introduce dynamic classifier chain algorithm that explicitly captures and exploits dependen- cies among evaluation metrics.
(3) We propose a two-step confidence-oriented decoding strategy that improves robustness and prediction reliability by dynamically adjusting the inference trajectory.
(4) We conduct extensive experiments demonstrating that ARECHO consistently outperforms existing frameworks across multiple speech domains, including synthesized, enhanced, and corrupted speech analysis, while providing improved interpretability and flexibility.
2 Related Works
Speech Assessment. Speech evaluation has usually been studied in the context of speech generation tasks, particularly speech synthesis and speech enhancement (SE). In these domains, assessment efforts are typically oriented toward task-specific goals, for instance, naturalness in speech synthesis and noise reduction in SE (Cooper et al. 2024 Hu & Loizou 2007b). In speech synthesis, subjective
'The code is open-sourced at http:////tttu....aaaaaa////////r

---

human evaluation remains the gold standard, with evaluations ranging from general assessments such as overall naturalness to more fine-grained dimensions such as speaking style and expressiveness (Li)
al. 2024 Yang et al. 2024 Shimizu et al. 2024 Feng & Yoshimoto 2024). However, such human
evaluations face well-known limitations, including challenges scaling, score inconsistency. and
limited coverage of diverse perceptual factors (Huang et al.] [022]] ZZelinski et al.] 2008 Loizou
et al., 2021 Naderi et al. 2020). To overcome these limitations, recent works have introduced objective evaluation systems that are trained to predict human perceptual scores, such as
mean opinion scores (MOS), using supervised learning frameworks (Falk et al.] 2008 Yoshimura
et al. 2016; Lo et al. 2019: Saeki et al. 2022: Huang et al., 2024c). These models have gained popularity in speech synthesis due to their scalability and ability to generalize to unseen systems, A similar trend is observed in SE, where classical signal- based metrics (c.g., hignal-to-noiss ratio)
are widely used with simulated data experiments (Perlmutter et al. 1977 Hansen & Pellom 1998)
Xu et al. 2013). Yet, they often fail to reflect human perception accurately, especially in roal-world,
noisy scenarios (Hu & Loizou, 2007b). As a result, perceptually aligned objective metrics have been
proposed to bridge the gap between computational evaluation and subjective judgment (Hu & Loizou
2007a Reddy et al. 2021 Rao et al. 2021 Yi et al. 2022).
While the above metrics are directly tied to task-specific outcomes, there is growing interest in broader frameworks for general speech assessment or profiling (Zezario et al. 2022a 2024; Chen &
2022 Close et al. 2024 Zezario et al. 2022b Kumar al.) |003] Tjandra et al. 2025 Shi ct al.
Tsao
2025a b). Such frameworks aim to extract a rich set of meta- iecta-information that spans across multiple dimensions of the speech signal. This can include not only quality and intelligibility, but also speaker traits, emotional content, and environmental context. In this work, we align with this broader vision and propose system that moves beyond task-specific metrics to support general purpose,, multi-faceted speech evaluation
Multi-Metric Evaluation and Dependency Modeling. Speech evaluation inherently involves multiple dimensions, such as naturalness, intelligibility, and emotional expression, that are often correlated through shared acoustic and prosodic cues. Effectively capturing the dependencies among these metrics is therefore essential for producing faithful and insightful assessments of speech quality.
general-purpose purpose evaluation frameworks that unify diverse metrics and applications (Zezario et al., 2022a b 2024; Chen & Tsao 2022; Close et al., 2024 Kumar et al.) 2023] Tjandra et al. 2025; Shi et al., 2025a b). These systems aim to provide scalable, task-agnostic profiling across various speech properties, including quality, speaker traits, emotional content, and environmental conditions. Frameworks such Uni (Shi et al. (025b) and TorchSquim (Kumar et al., 2023) support multi-metric prediction, but typically treat each metric independently, overlooking the latent correlations that can enhance performance and interpretability. For instance, improvements in naturalness often align with gains in intelligibility or perceived emotion due to overlapping signal characteristics.
Recent work has increasingly focused on general-
In parallel. captioning- baaed approaches have explored generating natural language rationales to explain metric predictions (Ghosh et al. |2024| Xie et al., 2025 Ghosh et al. [2025] Wang et al.) 2025b
Deshmukh et al. 2025 Ma et al. 2025; Kuan & Lee, 2025 Wen et al.] 2025 Wang et al., 2025a
Huang et al. 2022aa 2025; Chen et al. (025). While these methods offer interpretability, they often rely on pretrained LLMs and may struggle with precision or non-textual metrics. ARECHO instead
focuses on structured, metric-token-basse based modeling, enabling exact scoring and more fine-grained control.
3 ARECHO
3.1
Base Task Formulation
We adopt the general task formulation from (Shi et al. 2025b Kumar et al. 2023) for speech multi- metric estimation, with an extension to explicitly handle categorical metrics in addition numerical ones. Let th paired sample from dataset D be represented as (S', Y*). where S denotes a single- channel speech signal and y denotes the set of associated evaluation metrics. Each Y {vd bes consists of multiple metric values, where b is the index of a metric. Here, B is a set of indices, which
We adopt the term "metric" broadly in this work; see Appendix Afor detailed discussion.
3

---

can be partitioned into indices for numerical metrics Bum and indices for categorical metrics Beat. i... B
mum cat
The core model predicts all metrics directly from the input signal:
Y = f(S';
where f()) denotes the base prediction model and Y (DETes represents the predicted metrics
The training objective for the multi-metric estimation minimizes the prediction error across all metrics using a regression (n-norm) and cross-entropy losses for Brom and Beat, respectively:
Tig
Bun + TiB..
Σ
llei - 16lll
CR Mb.),
DE
MEB
where n = 1 in our experiments and CE(-) is the cross-entropy loss function.
Building on the task formulation introduced above and the challenges outlined in Sec. including heterogeneous metric scales, partial supervision, and the lack of inter-metric reasoning. we propose ARECHO, flexible and robust framework for multi-metric speech evaluation. ARECHO addresses these limitations through three key algorithmic components: (1) comprehensive tokenization framework that standardizes diverse metric types into a unified representation space; (2) dynamic classifier chain that captures inter-metric dependencies via flexible, data-driven sequencing; and (3) two-step confidence-oriee -rrientnce-oreetedd decoding strategy that enhances prediction reliability under uncertainty. Each of these components is described in detail the following subsections.
3.2 Tokenizing Everything
To address the above Challenge of heterogeneity across evaluation metrics, ranging from unbounded continuous scores discrete categorical labels, we introduce a unified tokenization framework that transforms all metric values into shared discrete representation space. This formulation enables ARECHO model metric prediction as sequence generation task over tokens, allowing consistent treatment of diverse metric types and facilitating autoregressive dependency modeling.
Given sample (S', Y'), where Y = {ut,bee lee includes both numerical and categorical metrics, define set of tokenization functions T = Ty. where each To maps a ground-truth value v to discrete token =i, from finite vocabulary V/:
* = To(b),
=f E V.,
where the total vocabulary is V = (be) (V)).
The full tokenized label sequence for sample i becomes Z = {28,)EB, which serves as the target sequence for autoregressive prediction. For numerical metrics (i.e., E Baun), we apply quantization- based tokenization by partitioning the value range into uniformly adaptively spaced bins | For categorical metrics (i.e., b Bex), the tokenization direct by mapping each class label to a unique token.
The inverse function To 14 + or V. C. (where C. is the label for categorical metric b E Bea) is used to reconstruct predictions:
d T
1 (2)).
2 Vi-
This formulation allows unified modeling across metric types, improves dependency learning, and enables flexible inference (see Sec. 3.3
3.3 Dynamic Classifier Chain
To address Challenge III: Dependency Modeling with Flexible Control, introduce dynamic classifier chain architecture that models inter-metric dependencies while allowing flexible prediction
Pleasse refer to Appendix F for examples of metrics and their types.
'Unlike (hh et al. 2025b) we omit these modalities and leave them for future work.
'Plesse refer to Appendix B for more discussion.). While Linear works well overall, Normal and Log-based
schemes can be beneficial for metrics with skewed distributions (e.g.. SNR, MOS).

---

orders. Our design is inspired by the multi-labe label classifier chain model (Read et al. 2011 2021 but generalizes token-level formulation suitable for autoregressive sequence modeling.
Motivating Example. Consider a case where we want to predict three metrics: Gender, Emotion, and MOS. Instead of predicting these values independently, model constructs target sequence:
full = [<Gender>; Male, <Emotion?, Happy, <MOS>, 78]
Here, tokens like <Gender> and <MOS> are metadata tokens that serve prompts, indicating which metric the model should predict next. The following tokens, Male, Happy. 3.78, are the corresponding value tokens, representing the model's predictions for each metric
Formal Definition. For each metric b B, we define metadata token my .M drawn from finite vocabulary M. These tokens act as queries for predicting the associated values. Note that M is separate set of metadata tokens and is disjoint from V. the value token vocabulary.
The full target sequence for training a flat, interleaved sequence of metadata and value tokens:
T full
[mb,> zbyy Mabs -hy..... mbx 40g
where (b1...., bx] is a permutation of selected metries for given input, and zi is the value token for metric by in sample i.
Training Objective. This sequence is modeled autoregressively:
2K-1 P(at T
P(T SS)
P(a) S")
(6)
where x/ E EUVV either metadata value token, and S the input speech representation.
Randomizing the metric order during training exposes the model to diverse conditioning patterns, helping it generalize across different evaluation needs.
The advantages of the use of metadata tokens include: Metric Identification: Metadata tokens specify which metric to predict, even when value formats overlap (e.g., many metrics return scores in similar ranges). Dependency Control: Prior metrie predictions are part of the sequence history, enabling context-aware inference of subsequent metrics.
Flexible Inference. During inference, the model begins with metadata token (e.g., <Emotion>) and autoregressively generates its corresponding value token (e.g.. Sad). This procedure iterates for any subset of requested metrics, allowing dynamic chain-style reasoning in arbitrary query orders. Users can either (i) specify a fixed prediction order or (ii) rely on our two-step
decoding in Sec. 3.4 to automatically determine an effective sequence. Such flexibility is particularly advantagcous when metric availability or user interests differ across samples.
Support for Partial Supervision. The autoregressive formulation also helps address Challenge II: Limited Label Availability. For samples with only a few known metrics, the training target simply omits the unavailable ones. For instance:
T partiall [mag =6,, 1/ig =
This allows the model to learn from partially labeled data without masking or imputation.
Compared conventional classifier chains that assume fixed label spaces and static prediction orders (Chen et al.) 2018 Gerych et al. 22021), our token-level formulation generalizes the idea for more expressive and modular metric modeling. We elaborate how this supports efficient decoding in Section
3.4 Two-step Confidence-oriented Decoding
While the dynamic classifier chain enables flexible and dependency-awar prediction of multiple metrics, it introduces challenges at inference time, particularly due to the presence of metadata tokens. Since the order of metric prompts is randomly optimized during training, the model's confidence in predicting metadata tokens can be unreliable, making difficult to guide decoding based on their probabilities.
We provide detailed running example in Appendix
5

---

To mitigate the decoding instability introduced by metadata tokens in the dynamic classifier chain, we adopt rwo-step senfidence-orienter decoding strategy that guides inference using the confidence of value tokens instead of the less reliable metadata scores.
Let BK be the set of all metrics and T prev the current decoded prefix after i metrics, corresponding to the frready-predicter subset By S Bg. With K - k metrics still to predict, the decoding for each remaining metric b BR') BBx proceeds in two phases:
Step 1: Preliminary Prediction. Append the metadata token my to the prefix,
T = T Tpw + mb:
and obtain provisional value token &o together with its softmax- based confidence Conf(
Step 2: Confidence-driven Candidate Search. Using the intermediate prefix T Tre T prev + ris extract the B most probable candidate values
z Top- ( (( SS, Tpee)]
For every 3n E 2 form the candidate sequence
T pre
+ Eb.
compute its likeliiol log P(T) S), and retain the highest-scoring sequence as the final
prediction for metric
After all candidates for every E Bx BB are evaluated, we keep the top-B partial hypotheses (by log-likelhhod) to serve as prefixes for the next metric. By revisiting low-confidence predictions through this confidence-awar beam search, the proposed strategy substantially improves the stability and accuracy of autoregressive multi-metric inference.
Experimental Setup
4.1 Datasets
To comprehensively evaluate the generalization ability, robustness, and versatility of ARECHO, conduct experiments across diverse set of datasets spanning multiple speech domains. These datasets are selected to represent key practical scenarios that require distinct types of evaluation metrics and pose varying challenges in terms of signal complexity, perceptual quality, and annotation availability. Specifically, we include (1) basic speech data to provide foundational coverage of speech variability across domains, (2) simulated corrupted speech datasets test objective quality prediction under controlled noise conditions, (3) enhanced speech recordings to assess robustness to natural distortions and variability, and (4) synthesized speech datasets evaluate naturalness and expressiveness of generated speech. We briefly describe each dataset below; additional details, including licensing and preprocessing procedures, are provided in Appendix E
Basic Speech Data. We include data sampled from the OWSM-V3 corpus (Peng et al. 2023). large-scale aggregation of speech recognition and translation datasets. This collection provides general-purpose coverage over wide range of speakers, styles, and domains.
Corrupted Speech Data (Simulation). We used simulated SE data generated via the URGENT2024 challenge ZZhagg al. 2024) training data generation script | Furthermore, we also include, for training purposes only, the w Voice Bank+DEMANND Veaux et al. (2013); Thiemann et al. (2013) benchmark dataset for speech denoising.
Enhanced Speech Data. Together with the simulated data described previously, we used the blind test set from the URGENT2024 challenge. It includes both simulated and real-world recordings with speech corrupted by one or more of the distortions in the same way as mentioned above. These recordings were enhanced by various participants' submitted systems in the URGENT2024 challenge.
Synthesized Speech Data. We utilize two benchmark datasets: VoiceMOS 2022 Challenge (Huang et al. 2022) and NISQA (Mittag et al. 2021). VoiceMOS 2022 challenge was developed for the
Available guthbbccoomuggggncccllleeggggggg cchalenne

---

VoiceMOS Challenge 2022, comprising diverse collection of synthetic speech generated by various TTS systems. NISQA (Mittag et al. 202T) dataset also contains synthesized speech samples with corresponding quality ratings across multiple perceptual dimensions.
Given these sources, construct unified training, development, and test sets by carefully sampling across all domains. This curation ensures that the model is exposed to balanced mixture of evaluation metrics and acoustic conditions, promoting generalization across tasks.
To further examine the effect training data scale, we prepare two training configurations: smaller Base set (308.77 hours) and a larger Scale set (2137.74 hours). Each is paired with shared development set (18.65 hours). Evaluation performed using four domain-specific test sets corresponding to the main application areas: simulated enhancement (4.51 hours), enhanced speech data (30. 12 hours), and speech synthesis-related data (3.46 hours). Full statistical details are available in the Appendix E.5
4.2 Model Setups
Metries in ARECHO. To ensure broad coverage and comprehensive speech profiling, ARECHO incorporates a diverse set of evaluation metrics from two main sources: (I) automatically computed metrics derived from existing models algorithms, and (2) pre-annotated information extracted from dataset metadata or human subjective evaluations.
For the first category, we employ the VERSA toolkit (Shi ct al. 2025a) to estimate 47 independent metries, 25 dependent metrics, and 7 non-mateinn non-matching metrics truth metrics derived from dataset annotations, such as language labels, emotional categories, and human-annotunl annotated MOS scores. In total, ARECHO models 87 metrics, comprising 65 numerical and 22 categorical metrics?" [
S For For the second category, we include
Baseline Setup. We adopt the UniVERSA model (Shi 2025b) as our baseline. It uses
Transformer-based audio encoder built on WavLM representations (Chen et al. (022) to extract shared speech embeddings. Each metric then independently predicted using a metric-specific pooling layer followed by an iivector-based prediction head (Snyder et al. 2018). Regression targets are predicted as scalars, whereas classification targets use softmax output corresponding to the number of classes.
To study the effect of discrete metric modeling, implement variant called UniVERSA-T, where all numerical metrics are converted into classification tasks via uniform quanti- zation tokenization, using the same architecture as the original UniVERSA.
Tokenization.
Proposed Model Setup. For ARECHO, we retain the same audio encoder as UniVERSA but replace the prediction heads with Transformer- booooeeeer--rddddd decoder that autoregressively generates the full token sequence Ti as defined in Sec. 3.3
All models are trained under both the Base and Scale training configurations. Detailed architecture specifications, training procedures, and decoding hyperparameters are provided in Appendix G Additional ablation studies exploring the effects of model components and training configurations can be found in Appendix
4.3 Evaluation
We adopt standard evaluation metries for regression and classification tasks. For each numerical metric. we compute the mean squared error (MSE), linear correlation coefficient (LCC), and Kendall's (KTAU). For each categorical metric, report accuracy (ACC) and Fl score. Evaluation scores are averaged across all numerical and categorical metrics, respectively, to provide overall regression and classification performance
*The complete configuration VERSA to calculate related metries detailed in Appendix F
"A complete breakdown of all supported metrics, including their types, sources, and usage, is provided in the Appendix[F
II appendices, we additionally report root mean squared error (RMSE), mean absolute error (MAE), and Spearman's 's rank correlation coefficient (SRCC) for regression metrics; precision, and recall for classification metrics. The complete experimental result with all metrics presented in Appendix H
7

---

Table Main experimental results for comparison between baseline and ARECHO. The "Domain" indicates the evaluation set used for the model assessment.
Regression Metries
Classificatin Metrics
Data
Domain
Model
Token
Chain
MSF ())
LCC (1)
KTAU (1)
Acc (1)
(1)
UnIVERSA
160.06
0.69
0.53
0.68
0.42
Dev.
UnIVERSA-T
40.95
0.78
0.68
0.70
0,46
ARECHO
25.73
0.86
0.72
0.71
0.51
61.54
0.71
0.54
Enhanced
UniVERSA-T
X
27.34
0.81
0.68
0.70
0.47
ARECHO
0.51
UniVERSA
170.65
0.61
0.48
0.70
0.46
Corrupted
UnIVERSA-T
77.72
0.74
0.67
0.71
0.50
ARECHO
44.22
0.82
0.70
0.72
0.55
UnIVERSA DTiVERSA-T
58.79
0.76
0.54
0.69
0.45
Synthesized
0.84
0.68
0.72
0.50
ARECHIO
4.99
0.91
0.78
0.79
0.65
Un VERSA
96.99
0.69
0.52
Avg. Test
UniVERSA-T
37,72
0.79
0.68
0.71
0.49
ARECHO
23.26
0.86
0.72
0.74
0.57
UniVERSA
116.01
0.74
0.73
0.49
Dev.
UniVERSA-T
27.98
0.86
0.75
0.74
0.52
ARECHO
29.61
0.75
UnIVERSA
43.05
0.84
0.67
0.72
0.47
Enhanced
UniVERSA-T
69.94
0.80
0.71
0.74
0.50
ARECHO
32.63
0.83
0.73
0.75
0.53
Scale
UnIVERSA
X
151.97
0.88
0.75
0.75
0.54
Corrupted
DEiVERAATT
0.74
0.54
ARECHO
34.37
0.84
0.76
0.77
0.56
UnIVERSA UniVERSA-T
0.84
0.65
0.71
0.47
Synthosized
8.23
0.84
0.68
0.73
0.49
ARECHO
0.72
0.75
0.54
UnIVERSA
67.16
0.86
0.70
0.73
0.50
Avg. Test
UnIVERSA-T
39.32
0.82
0.72
0.74
0.51
ARECHO
25.21
0.85
0.74
0.76
0.54
Experimental Results
Overall Performance.
summarrzes the overall performance of the proposed ARECHO model compared to baselines. Across both the Base and Scale training configurations, ARECHO consistently and significantly outperforms the Uni VERVA and Un VERSA-T baselines the majority of evaluation metries [1] These results highlight the effectiveness of ARECHO's dynamic classifier chain and confidence-orientedd decoding strategy in capturing inter-metric dependencies and improving predic- tion robustness. Notably, the improvements observed with Uni VERSA-T over UniVERSA demonstrate the benefit of tokenizing numerical metrics, validating our unified representation approach. Building upon this, ARECHO achieves further gains by leveraging structured autoregressive modeling, which enables more informed and context-aware metric prediction.
Effects of Data Scaling. The Scale training set introduces greater domain imbalance, with a heavier emphasis corrupted speech compared to other scenarios Under this condition, ARECHO delivers substantial improvements on corrupted speech evaluation, showcasing its strong modeling capacity in data-rich domains. However, the gains are comparatively smaller on synthesized and enhanced speech, suggesting that domain balance remains an important factor for achieving broad generalization.
Observations Baseline Behavior. While the original UniVERSA model underperforms in most settings, still shows relative strength modeling fine-grained numerical metrics, particularly reflected in its high LCC scores under the Scale configuration. This indicates that tokenization, while beneficial overall, may introduce granularity loss for certain regression tasks. Slight performance degradation in Uni VERSA-T and ARECHO specific numerical metrics highlights trade-off between discrete modeling and numerical precision, an open challenge we aim to address in future work.
Statistical significance evaluated permutation testing. See Appendix E for detailed statistics domain distribution.

---

Ablation and Extended Analysis. We conduct set of ablations and diagnostic studies to assess robustness, efficiency, and practical behavior; full results are reported in Aapendix[aandll [ These studies examine (i) tokenization resolution, (ii) decoding strategy, (iii) MOS-style perceptual metrics, and (iv) task-oriented metric subsets. Across all settings, ARECHO remains stable: performs well with compact tokenizations, achieves near-optimal accuracy with simple greedy decoding, and shows strong gains on perceptual quality metrics (e.g., MOS). When trained on either task-specific metric subsets or the full union of metrics, ARECHO shows evidence of negative transfer from "irrelevant" metrics and often improves on core targets (e.g. SRMR, SDR, human MOS). Together, these results indicate that ARECHO is both effective and adaptive: it can leverage cross-metric structure when helpful while preserving efficiency and specialization. We additionally report efficiency analysis in Appendix M showing that ARECHO attains these benefits with substantially reduced training and inference cost relative to prior multi-metric systems.
Further Discussion on Dependency Modeling. The proposed ARECHO framework learns to adaptively control the inference order of metrics through its dynamic classifier chain and two- step decoding strategy, introduced in Sec. 3.3 and Sec. 3.4 This design enables the model to prioritize more informative or stable metries early in the prediction sequence, providing con- textual cues that improve downstream metric predictions.
Table 2: Top-5 and Bottom-5 metrics ranked by average position (Avg. Pos.) across three test sets. Please refer Appendix F for more details about the metrics.
Test Set
Rank
Metric Name
Avg. Pos.
Top-1
Q-SpeakerGender
16.50
Top-2
Q Q-Speechmmpirmeen
20.35
Top-3
SpeecchStyle
21.47
Enhanced
Bun-3
SNR Simulation
163.52
Table 2 highlights how ARECHO internally discovers and exploits an ordering ratio- nale 13 Across different test sets, metrics related to structured annotations or acoustic scene characteristics (e.g., Q-SpeakerGender, Q-Speechlmpaiment, RIR Room Size) consis- tently appear early in the prediction sequence. These metrics are arguably easier to estimate and provide strong prior information for subsequent metries. In contrast, more subjective and unsta- ble metrics, such MOS scores from Voice- MOS and NISQA, tend to appear later in the sequence. This suggests that ARECHO defers harder or noisier predictions until more context available from previously decoded metrics. Depending on the context, some metrics are also inherently harder to predict e.g.. SNR simulation for enhanced and synthesized test sets. For enhanced speech data, the noise was removed by an SE model, while for the synthesized speech, the residual noise in the text-to- to-eeeech generated speech signal usually very low.
Btm-2
NISQA Real MOS
167,91
Btn-1
VoiceMOS Real MOS
171.58
Top-1
RIR Room Size
1.82
Top-2
Q-Spechhimpimeen
12.65
T
S-eccchiellivery
Corrupted
Bun-3
CER
167.26
Btm-2
NISQA Real MOS
170.62
Bin-1
VoiceMOS Real MOS
171.38
Top-1
Q- Q-Background
12.09
Top-2
NISQA Coloration
27.51
Synthesized
Top-3
Q-Purpose
27.95
Rtm-3
Cya
154.75
Bun-2
SNR Simulation
158.64
Rtm-1
CER
161.61
This emergent ordering aligns with human intuition and supports the benefit of structured inter-metric reasoning. It also allows the system to maintain robustness when certain labels are missing or unreliable, by leveraging earlier predictions to guide later stages.
As discussed in Sec. 3.3 our proposed dynamic classifier chain supports both flexible order search and inference under arbitrary query sets. In particular, it also allows for inference with a fixed, static order of metrics when such an order is used during training. To investigate how different static orderings affect the performance of ARECHO, we provide additional analyses in the Appendix []]
Complexity and Efficiency. While ARECHO provides several advantages as discussed above, it also maintains modest training cost and strong computational efficiency. By effectively utilizing partially labeled data, ARECHO achieves shorter training time comparable to standard multi-task learning with masking (e.g., Uni Uni VERSA) Despite its sequential decoding process, ARECHO remains practical: computing the same set of metrics through their original estimators (e.g.. LLM-based evaluators) is over x00x x slower than a single forward pass of ARECHO.
Phiile Table 2 presents part of the resulting orders, we further claborate the details in Appendix "We present detailed analysis in Appendix M

---

6 Conclusion
We introduce ARECHO (Autoregressive Evaluation via Chain-based Hypothesis Optimization), versatile and interpretable framework for multi-metric speech assessment. By unifying diverse metric types through tokenization and modeling inter-metric dependencies via a dynamic classifier chain, ARECHO offers flexible and scalable alternative traditional parallel prediction models. Backed with our proposed two-step confidence-orientee decoding, ARECHO maintains the flexibility in decoding, which can support diverse challenging evaluation settings. Extensive experiments across corrupted, enhanced, and synthesized speech signals demonstrate that ARECHO consistently outperforms strong baselines while enabling more structured and adaptable evaluation. We believe ARECHO provides a step toward general-purpose, dependency-awant modeling for speech and potentially broader machine learning evaluation tasks.
References
Jont B Allen and David A Berkley. Image method for efficiently simulating small-room acoustics. The Journal of the Acoustical Society of America, 65(4)9943-950, 1979.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weher. Common Voice: A massively-multiinguaa speech corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4218-4224 2020.
Kaito Baba, Wataru Nakata, Yuki Saito, and Hiroshi Saruwatari. The TO5 system for the voicemos challenge 2024: Transfer learning from deep image classificr to naturalness mos prediction of high-quality synthetic speech. In 2024 IEEE Spoken Language Technology Workshop (SLT). pp. 818 824. IEEE, 2024.
Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. Evaluation measures for ordinal regression. In 2009 Ninth international conference on intelligent systems design and applications, pp. 283 287. JEEE, 2009.
Thomas P Barnwell III. Objective measures for speech quality testing. The Journal of the Acoustical Society of America, (6(6):1558-11 1663, 1979.
Christoph Boeddeker. Wangyou Zhang, Tomohiro Nakatani, Keisuke Kinoshita, Tsubasa Ochiai, Marc Delcroix. Naoyuki Kamo, Yanmin Qian, and Reinhold Haeb-Umbach. Convolutive transfer function invariant SDR training criteria for multi-channel reverberant speech separation. In ICASSP 2021-2021 IELE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8428 8432. IEEE, 2021.
Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. Alshell-1: open-source mandarin speech corpus and speech recognition baseline. In 2017 20th conference of the oriental chapter of the international coordinating committee on speech databases and speech 1/O systems and assessment (O-COCOSDA), pp- 1-5. JEEE, 2017.
Wenzhi Cao, Vahid Mirjalili, and Sebastian Raschka. Rank consistent ordinal regression for neural networks with application to age estimation. Puttern Recognition Letters, 140:325-331, 2020.
Chen Chen, Yuchen Hu, Siyin Wang, Helin Wang, Zhehuai Chen, Chao Zhang, Chao-Han Han Huck Yang, and Eng Siong Chng. Audio large language models can be descriptive speech quality evaluators. arXiv preprint 251.:22501.17202. 2025.
Fei Chen and Philipos C Loizou. Analysis of a simplified normalized covariance measure based on binary weighting functions for predicting the intelligibility of noise-suppressed speech. The Journal of the Acoustical Society of America, 128())3715-3722 2010.
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. WavLM: Large-scale self-super pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 1666):500--1588 2022.
Shang-Fu Fu Chen, Yi-Chen Chen, Chih-Kuan Ych, and Yu-Chiang Wang. Order-free RNN with visual attention for multi-label classification. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Yu-Wen Chen and Yu Tsao. InQSS: speech intelligibility and quality assessment model using a multi-tan task learning network. In Proc. Interspeech 2022, pp. 3088 3092, 2022.
10

---

Michael Chinen, Felicia SC Lim, Jan Skoglund, Nikita Gureev, Feargus O'Gorman, and Andrew Hines. Visgol v3: An open source production ready objective speech and audio metrie. In 2020 swelfth international conference on quality of multimedia experience (QoMEX), pp. 1-6. IEEE, 2020.
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report, 2024. URL. https
//arxiv. org/ abs/ abs/2407 22022 10759
George Close, William Ravenscroft, Thomas Hain, and Stefan Goetze. Multi- Multi-CMGAN+/+ Leveraging multi- objective speech quality metric prediction for speech cnhancement. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 351-355. IEEE, 2024.
Erica Cooper. Wen-Chin Huang, Yu Tsao, Hsin-Min Wang, Tomoki Toda, and Junichi Yamagishi. A review on subjective and objective evaluation of synthetic speech. Acoustical Science and Technology, pp. 124-12, 2024.
Soham Deshmukh, Dareen Alharthi, Benjamin Elizalde, Hannes Gamper, Mahmoud Al Ismail, Rita Singh, Bhiksha Raj, and Huaming Wang. PAM: Prompting language models for audio quality assessment. In Proc. Interspeech 2024, pp. 3320-3324, 2024.
Soham Deshmukh, Satvik Dixit, Rita Singh, and Bhiksha Raj. Mellow: a small audio language model for reasoning. arXiv preprint arXiv:2503. 08540, 2025.
Lorenz Diener, Marju Purin, Sten Sootla, Ando Saabas, Robert Aichner, and Ross Cutler. Plemos - a data data-driven non-intrusive metric for the evaluation of packet loss concealment algorithms. In Proc. Interspeech 2023, pp- 2533-2537, 2023. doi: 10.21 00.21457/mterspecchh222.. 201.ppeccc....-115....
Harishchandra Dubey, Ashkan Aazami, Vishak Gopal, Babak Naderi, Sebastian Braun, Ross Cutler, Alex Ju, Mchdi Zohourian, Min Tang, Mehrsa Golestaneh, et al. ICASSP 2023 deep noise suppression challenge. IEEE Open Journal of Signal Processing. 2024.
Tiago H Falk, Schastian Möller, Vasilis Karaiskos, and Simon King. Improving instrumental quality prediction performance for the Blizzard Challenge. Proc. Blizzard Challenge Workshop. 2008.
Tiago Falk, Chenxi Zheng, and Wai Wai-Yip Chan. non- non-intrusive quality and intelligibility measure of reverherant and dereverberated speech. IEEE Transactions on Audio, Speech, and Language Processing, 18 (7):176617774, 2010.
Xincan Feng and Akifumi Yoshimoto. Llama-VITS: Enhancing uts synthesis with semantic awareness. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC- CORCCC 2024), pp. 10642- 10656, 2024.
Cédric Févotte, Rémi Gribonval, and Emmanuel Vincent. BSS_EVAL toolbox user guide-revision 2.0, 2005.
Eduardo Fonseca, Jordi Pons, Xavier Favory, Frederie Font, Dmitry Bogdanov, Andres Ferraro, Sergio Oramas, Alastair Porter, and Xavier Serra. Freesound datasets: A platform for the creation of open audio datasets. In Proc. ISMIR pp. 486-493, 2017.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: ontology and human--labeled dataset audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780, IEEE, 2017.
Walter Gerych, Tom Hartvigsen, Luke Buquicchio, Emmanuel Agu, and Elke Rundensteiner. Recurrent Bayesian classifier chains for exact multi-label classification. Advances in Neural Information Processing Systems, 144:15981-199999 2021.
Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi. S Sakshi, Oriol Nicto, Ramani Duraiswami, and Dinesh Manocha. GAMA: A large audio-language model with advanced audio understanding and complex reasoning abilities. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 6288-6313, 2024.
Sreyan Ghosh, Zhifeng Kong, Sonal Kumar, S Sakshi, Jachyeon Kim, Wei Ping, Rafacl Valle, Dinesh Manocha, and Bryan Catanzaro. Audio Flamingo 2: An audio-language model with long-audio understanding and expert reasoning abilities. arXiv preprint N.Ki::250330398. 2025.
Lucas Goncalves, Ali N Salman, Abinay R Naini, Laureano Moro Velazquez. Thomas Thebaud. Leibny Paola Garcia, Najim Dehak, Berrak Sisman, and Carlos Busso. Odyssey 2024-speech emotion recognition challenge: Dataset, baseline framework, and results. Development, 10(9,290):4 54, 2024.

---

Pedro Antonio Gutiérrez, Maria Perez-Ortiz, Javier Sanchez-Monedero, Francisco Fernandez-Navaroe and Cesar Mervas-Martinee Ordinal regression methods: survey and experimental study. IEEE Trunsactions on Knowledge and Data Engineering, 1871): 2015.
John HL Hansen and Bryan Pellom. An effective quality evaluation protocol for speech enhancement algorithms. In Proc. ICSLP, volume 7. pp. 2819-282222 Citeseer, 1998.
Tomoki Hayashi, Ryuichi Yamamoto, Takenori Yoshimura, Peter Wu, Jiatong Shi, Takaaki Sacki, Yooncheol Ju, Yusuke Yasuda, Shinnosuke Takamichi, and Shinji Watanabe. ESPnet2-TTS: Extending the edge of TTS 2021.
research. arXiv preprint
Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position emhedding for vision transformer. In European Conference on Computer Vision, pp. 289-305. Springer, 2024.
Yi Hu and Philipos C Loizou. Evaluation of objective quality measures for speech enhancement. IEEE Transactions on Audio, Speech, and Language Processing, 16(1):229-22) 2007a.
Yi and Philipos C Loizou. Subjective comparison and evaluation speech enhancement algorithms. Speech communication, 19(7-8):588-60 601, 2007b.
Chien- Yu Huang, Ke-Han Lu, Shih-Heng Wang. Chi- Yuan Hsiao, Chun- Yi Kuan, Haibin Wu, Siddhant Arora, Kai-Wei Chang, Jiatong Shi, Yifan Peng, et al. Dynamic-SuPERI Towards a dynamic, collaborative, and comprchensive instruction-tuning benchmark for speech. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 12136-12144 12140. IEEE, 2024a.
Chien-Yu Huang, Wei-Chih Chen, Shu wen Yang, Andy T. Liu, Chen-An Li, Yu-Xiang Lin, Wei-Cheng Tseng, Anuj Diwan, Yi-Jen Shih, Jiatong Shi, William Chen, Xuanjun Chen, Chi- Yuan Hsiao, Puyuan Peng, Shih-Heng Wang. Chun-Yi Yi Kuan, Ke-Han Lu, Kai- Wei Chang, Chih-Kai Yang, Fabian Alejandro Ritter Gutierrez. Huang Kuan-Po. Siddhant Arora, You-Kuan Lin, CHUANG Ming To, Eunjung Yeo. Kalvin Chang. Chung-Ming Chien, Kwanghee Choi. Cheng- -lsu Hsieh, Yi-Cheng Lin, Chee-En Yu, I-Hsiang Chiu, Heitor Guimarães, Jionghao Han, Tzu-Quan Lin. Tzu- Yuan Lin, Homu Chang. Ting-Wo Wu Chang, Chun Wei Chen, Shou-Jen Chen, fu-Hua Chen, Hsi-Chun Cheng, Kunal Dhawan, Jia-Lin Fang, Shi-Xin Fang, KUAN YU FANG CHIANG, Chi An Fu, FuscnnFFulle Hsiao, Ching Yu Hsu, Shao-Syuan Huang, Lee Chen Wei, Hsi-Che Lin, Hauan-Hao Lin, Hsuan-Ting Lin, Jian-Ren Lin, Ting-Chun Liu, Li-Chun Lu,, Tsung-Min Pai, Ankita Pasad, Shih-Yun Shan Kuan, Suwon Shon, Yuxun Tang, Yun-Shao Tsai, Wei Jui Chiang. Tzu- Chich Wei, Chengxi Wu, Dien-Ruei Wu, Chao-Han Huck Yang, Chich-Chi Yang. Jia Qi Yip. Shao-Xiang Yuan, Haibin Wu, Karen Livescu, David Harwath, Shinji Watanabe, and Hung yi Lee. Dynamic-SUPP SUPERB phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks. In The Thirteenth International Conference on Learning Representations, 2025. URL http:ppwwwweeeeeeeeeeeerrr/raaaa
Wen Chin Huang, Erica Cooper, Yu Tsao, Hsin-Min Wang, Tomoki Toda, and Junichi Yamagishi. The VoiceMOS challenge 2022. In Proc. Interspeeck, pp. 4536-455555 4540,,444400 2022.
Wen-Chin Huang, Erica Cooper, and Tomoki Toda. MOS-Bench: Benchmarking generalization abilities of subjective speech quality assessment models. arXiv preprint 2:x:i::2411.37//1 2024b.
Wen-Chin Huang, Szu-Wei Fu, Erica Cooper, Ryandhimas E Zezario, Tomoki Toda, Hsin-Min Wang, Junichi Yamagishi, and Yu Tsao. The VoiceMOS challenge 2024: Beyond speech quality prediction. In Proc. IEEE SLT, pp. 803-810. IEEE, 2024c.
LARPA. Babel 03-13.
2011. Accessed: 2025-
Rafael Zequeira Jiménez, Gabriel Mittag, and Sebastian Möller. Removing the bias in speech quality scores collected in noisy crowdsourcing environments. In 2021 13th International Conference on Quality of Multimedia Experience (QoMEX), pp. 49-54. IEEE, 2021.
Jec-weon Jung, Hee-Soo Heo, Hemlata Tak, Hye-jin Shim, Joon Son Chung, Bong-Jin Lee, Ha-Jin Yu, and Nicholas Evans. AASIST: Audio anti-spoofing using integrated spectro-temporal graph attention networks. In ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP). pp. 6367-6371. IEEE, 2022.
Jee-weon Jung. Wangyou Zhang, Jiatong Shi. Zakaria Aldeneh, Takuya Higuchi, Alex Gichamba, Barry-John Theobald, Ahmed Hussen Abdelaziz, and Shinji Watanabe. ESPnet-SPP full pipeline speaker embedding toolkit with reproducible recipes, self-superrised front-ends, and off fif-the-shell models. In Proc. Interspeech 2024, pp. 4278 4282, 2024.
12

---

James M Kates and Kathryn Н. Arehart. Coherence and the speech intelligibility index. The journal of the acoustical society of America, 117(4)2224-22377 2005.
Dennis Klatt. Prediction of perceived phonetic distance from eritical-band spectra: A first step. In ICASSP'82. IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 7. pp. 1278-1281. IEEE, 1982.
Wessel Kraaij, Thomas Hain, Mike Lincoln, and Wilfried Post. The AMI meeting corpus, In Proc. International Conference on Methods and Techniques in Behaviorul Research, pp. 1-4, 2005.
Chun-Yi Yi Kuan and Hung-yi Lee. Can large audio-language models truly hear? tackling hallucinations with multi-task assessment and stepwise audio reasoning. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025.
Robert Kubichek. Mel-cepstral distance measure for objective speech quality assessment. In Proc. IEEE Pacific Rim conference on Communications Computers and Signal Processing, volume 1, pp. 125-128, 1993.
Anurag Kumar, Ke Tan, Zhaoheng Ni, Pranay Manocha, Xiaohui Zhang, Ethan Henderson, and Buye Xu. Torchaudio-Squim Reference-less speech quality and intelligibility measures in torchaudio. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Shuhua Li, Qirong Mao, and Jiatong Shi. PL-TTS: A generalizable prompt-basee based diffusion tts augmented by large language model. In Proc. Interspeech 2024, pp. 4888-4892 2024.
Chen Chen-Chou Lo, Szu-Wei Fu, Wen-Chin Huang, Xin Wang. Junichi Yamagishi, Yu Tsao, and Hsin-Min Wang. MOSNet: Deep learning-based objective assessment for voice conversion. In Proc. Interspeech 20/9, pp. 1541-1545, 2019, doi: 10.21 /t5/://terppeech22119999
Philipos C Loizou. Speech quality assessment. In Multimedia analysis, processing and communications, pp. 623 633-654.656 Springer, 2011.
Ziyang Ma. Zhisheng Zheng, Jiaxin Yc, Jinchao Li, Zhifu Gao, ShiLiang Zhang, and Xic Chen. emotion2vec: Self- supervised pre-training for speech emotion representation. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics: ACL. 2024, pp. 15747-15760, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/v1/2024. Andings-acll9311 URL htt.jaats://acaaaatii wwwpaatttoooooooooooooeooooooeee 2tt..22...........eee. findings- 931
Ziyang Ma, Zhuo Chen, Yuping Wang, Eng Siong Chng, and Xie Chen. Audio-Coll Exploring chain-of- of-thhug-tt reasoning in large audio language model. arXiv preprint ar arXiv:2501.07260 2025.
Pranay Manocha, Buye Xu, and Anurag Kumar. NORESQA: A framework for speech quality assessment using non-matching references. In Advances in Neural Information Processing Systems, volume 34, pp. 22363-22378, 2021.
Gabriel Mittag, Babak Naderi, Assmaa Chehadi, and Schastian Möller. NISQA: A deep enn-self-attention model for multidimensional speech quality prediction with crowdsoureed datasets. In Proc. Interspeech 2021, I, pp- 2127-211 2131, 2021. doi: 10.2 20.21433//nerrppccch.200 2022eeccc...29999999 1-99.
Babak Naderi, Rafael Zequeira Jiménez, Matthias Hirth, Sebastian Möller, Florian Metzger, and Tobias Hoßfeld. Towards speech quality assessment using a crowdsourcing approach: evaluation of standardized methods. Quality and User Experience, 6:1-21, 2020.
Douglas Paul and Janet M Baker. The design for the wall street journal-based CSR corpus. In Proceedings of the workshop on Speech and Natural Language, pp. 357-362, 1992.
Yifan Peng. Jinchuan Tian, Brian Yan, Dan Berrebbi, Xuankai Chang. Xinjian Li, Jiatong Shi, Siddhant Arora, William Chen, Roshan Sharma, et al. Reproducing whisper-style training using an open source toolkit and publicly available data. In 2023 IEFE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1-8. IEEE, 2023.
Perlmutter, L Braids, R Frazier, and Oppenheim. Evaluation of speech enhancement system. In IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 2, pp. 212-215. IEEE, 1977.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via scale weak supervision. In International conference on machine learning, pp. 28492- 28518 18. PMLR, 2023.
13

---

Alessandro Ragano, Jan Skoglund, and Andrew Hines. NOMAD: Unsupervised learning of perceptual embed- dings for speech enhancement and non-matching reference audio quality assessment. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1011-1015. IEBE, 2024a.
Alessandro Ragano, Jan Skoglund, and Andrew Hines. SCOREQ: Speech quality assessment with contrastive regression. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b.
Wei Rao, Yihui Fu, Yanxin Hu, Xin Xu, Yvkai Jv, Jiangyu Han, Zhongjie Jiang, Lei Xie, Yannan Wang. Shinji Watanabe, al. ConferencingSpeech Challenge: Towards far-ficld multi-channel speech enhancement for video conferencing. 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 679-686, IEEE, 2021.
Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Fibe Frank. Classifier chains for multi-label classification. Machine learning, 85:333-359, 2011.
Jesse Read, Bernhard Pfahringer, Geoffrey Holmes, and Eibe Frank. Classificr chains: a review and perspectives. Journal of Artificial Intelligence Research, 70:683-718, 2021.
Chandan KA Reddy, Vishak Gopal, and Ross Cutler. DNSMOS: A non-intrusive perceptual objective speech quality metric evaluate noise suppressors. In ICASSP 202/-2021 IEEF International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6493-6497. IEEE, 2021.
Chandan KA Reddy, Vishak Gopal, and Ross Cutler. DNSMOS 835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressons. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 886-888688888 890, IEEE, 2022.
Antony W Rix, John G Beerends. Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE international conference on acoustics, speech, and signal processing. Proceedings (Cat. No. O1C332221), volume 2, pp. 749-752, IEBE, 2001.
Takaaki Sacki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. UTMOS: Utokyo-sarulab sarulab system for voicemos challenge 2022. In Proc. Interspeech, pp. 4521 44211,5255555 2022.
Takaaki Sacki, Soumi Maiti, Shinnosuke Takamichi, Shinji Watanabe, and Hiroshi Saruwatari. Speech- BERTScore: Reference-aware automatic evaluation of speech generation leveraging nip evaluation metrics. In Proc. Interspeech, pp. 4943-4947, 2024.
Jiatong Shi, Jinchuan Tian, Yihan Wa, Jee-weon Jung, Jia Qi Yip, Yoshiki Masuyama, William Chen, Yuning Wu, Yuxun Tang, Massa Baali, et al. ESPnet-Codec: Comprehensive training and evaluation of neural codecs for audio, music, and speech. In 2024 IEEE Spoken Language Technology Workshop (SLT). pp. 562-569. IEEE, 2024.
Jaaton Shi, Hye-jin Shim, Jinchuan Tian, Siddhant Arora, Haibin Wu, Darius Petermann, Jia Qi Yip, You Zhang, Yuxun Tang, Wangyou Zhang, Dareen Safar Alharthi. Yichen Huang, Koichi Saito, Jionghao Han. Yiwen Zhao, Chris Donahue, and Shinji Watanabe. VERSA: versatile evaluation toolkit for speech, audio, and music. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies (System Demonstrations), pp. 191-209, Albuquerque, New Mexico, April 2025a. Association for Computational Linguisties. ISBN 979-8-89176-199-99
www.o//.........//o/ 2025. naacl-demo. 19/
Jiatong Shi, Hye-Jin Shim, and Shinji Watanabe. Uni-VERSA Versatile speech assessment with unified network, 2025b. URL /tttjatts:///xxitt wrg.aaawwii....oaaeaooa
20741
Reo Shimizu, Ryuichi Yamamoto, Masaya Kawamura, Yuma Shirahata, Hironori Doi, Tatsuya Komatsu, and Kentaro Tachibana. PromptTTS+: Controlling speaker identity in prompt-based text-to-speech using natural language descriptions. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 12672-12676. IEEE, 2024.
David Snyder, Daniel Gameia-Romero Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. X-vectors: Robust DNN embedddings for speaker recognition. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. $329-5333, IEEE, 2018
Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. An algorithm for intelligibility prediction of time- frequency weighted noisy speech. IEEE Transactions on audio, speech, and language processing, 19(7):2125 122)22125- 2011.
14

---

Yuxun Tang, Jiatong Shi, Yuning Wu. and Qin Jin. SingMOS: An extensive open-source singing voice dataset for MOS prediction. arXiv preprini arXiv:2406.109) 2024.
Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent. The diverse enviromments multi-channel acoustic noise database (demand): A database of multichannel environmental noise recordings. In Proceedings of Meetings on Acoustics, volume 19. AIP Publishing. 2013.
Andros Tjandra, Yi-Chiao Wu, Baishan Guo, John Hoffman, Brian Ellis, Apoorv Vyas, Bowen Shi, Sanyuan Chen, Matt Lc, Nick Zacharov, et al. Meta Audiobox Aestheties: Unified automatic quality assessment for speech, music, and sound. arXiv preprint arXiv:2502.05139 2025.
Matteo Torcoli, Thorsten Kastner, and Jürgen Herre. Objective measures of perceptual audio quality reviewed: An evaluation of their application domain dependence. IEEE/ACM Transactions on Audio. Speech, and Language Processing, 19:1530-1541) 2021.
JM Tribolet, Peter Noll, B McDermott, and R Crochiere. A study of complexity and quality of speech waveform coders. In ICASSP'78. IEEE International Conference Acoustics, Speech, and Signal Processing, volume 3, pp. IEEE, 1978.
Christophe Vcaux, Junichi Yamagishi, and Simon King. The voice bank corpus: Design, collection and data analysis of a large regional accent speech databasc. 2013 international conference oriental COCOSDA held jointly with 2013 conference on Asian spoken language research and evaluation (O-COCOSDACASLRE)) pp- 1-4. IEEE, 2013.
Siyin Wang, Wenyi Yu, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Yu Tsao, Junichi Yamagishi, Yuxuan Wang, and Chao Zhang. QualiSpeech; A speech quality assessment datasct with natural language reasoning and descriptions. arXiv preprint arXiv:2503..2222 202990, 2025a.
Yingzhi Wang, Pooneh Mousavi. Artem Ploujnikov, and Mirco Ravanelli. What are they doing? joint audio- speech co co-reasoning. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025b.
Cheng Wen, Tingwei Guo. Shuaijiang Zhao, Wei Zou, and Xiangang Li. SARI: Structured audio reasoning via curriculum-guiled reinforcement learning. arXiv preprint arXiv:2504.1500 2025.
Shu wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Ting Lin, et al. SUPERB: Speech Processing Universal PFRformance Benchmark. In Proc. Interspeech, pp. 1194-1 1194-1198, 2021.
Gordon Wichern, Joe Antognini, Michael Flynn, Licheng Richard Zhu, Emmett McQuinn, Dwight Crow, Ethan Manilow, and Jonathan Le Roux. WHAM!: Extending speech separation noisy environments. In Proc. Interspeech 2019. pp. 1368 1378--132,200 2019.
Haibin Wu, Huang- Cheng Chou, Kai- Kai-Wei Chang, Lucas Goncalves, Jiawei Du, Jyh-Shing Roger Jang, Chi-Chun Lee, and Hung- Yi Lee. EMO-SUPERB: in-depth look at speech emotion recognition. arXiv preprint arXiv:2202.1 30018 2024.
Zhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, and Chunyan Miao. Audio-reasoner: Improving reasoning capability in large audio language models. arXiv preprint arXiv:2503.02314 2025.
Yong Xu, Jun Du, Li-Rong Dai, and Chin- Lee. An experimental study on speech enhancement based on deep neural networks. IEEE Signal processing letters, 21(1):65-66) 68, 2013.
Dongchao Yang, Songxiang Liu, Rongjie Huang, Chao Weng, and Helen Meng. InstruciTTS: Modelling expressive its in discrete latent space with natural language style prompt. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
Gaoxiong Yi, Wei Xiao, Yiming Xiao, Babak Naderi, Sebastian Möller, Wafaa Wardah, Gabriel Mittag, Ross Culter, Zhuohuang Zhang, Donald S. Williamson, Fei Chen, Fuzheng Yang, and Shidong Shang. ConferencingSpeechh 2022 challenge: Non-intrusive objective speech quality assessment (nisqa) challenge for online conferencing applications. In Proc. Interspeech, pp. 3308 3308-3312, 2022.
Takenori Yoshimura, Gustav Eje Henter, Oliver Watts, Mirjam Wester, Junichi Yamagishi, and Keiichi Tokuda. A hicrarchical predictor of synthetic speech naturalness using neural networks. In Interspeech 2016, pp. 342-346, 2016. doi: 10.2 002144/7/nerrpeecch20---
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron Weiss, Ye Jia, Zhiffeng Chen, and Yonghui Wu. LibriTTS: A corpus derived from LibriSpeech for text-to-speech. In Proc. Interspeech 2019, pp. 1526-1530, 2019.
15

---

Ryandhimas E Zezario, Szu-Wei Fu, Fei Chen, Chiou-Shana Fuh, Msin-Min Wang, and Yu Tsao. Deep learning- based non-intrusive multi-ojjective speech assessment model with cross-domain features. IEEE/ACM Trunsactions on Audio, Speech, and Language Processing, 31:54-70, 2022a.
Ryandhimas E Zezario, Bo-Ren Brian Bai, Chiou-Shann Fuh, Hsin-Min Wang, and Yu Tsao. Multi-task pseudo- label learning for non-intrusive speech quality assessment model. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 831-835. IEEE, 2024.
Ryandhimas Edo Zezario, Szu wei Fu, Fei Chen, Chiou-Shann Fuh, Hsin-Min Wang, and Tsao. MTT-Net: A multi-target speech intelligibility prediction model. In Interspeech 2022, pp. 5463-5467, 2022b. doi: 10.21457/lmlerspeechh00 2022. 10828.
Wangyou Zhang, Robin Scheibler, Kohci Saijo, Samuele Cornell, Chenda Li, Zhaoheng Ni, Jan Pirklbauer, Marvin Sach, Shinji Watanabe, Tim Fingschcidt, and Yanmin Qian. URGENT challenge: Universality, robustness, and generalizability for speech enhancement. In Proc. Interspeech, pp. 4868 4872, 2024.
Slawomir Zielinski, Francis Rumsey, and Søren Bech. On some biases encountered in modern audio quality listening tests-a review. Journal of the Audio Engineering Sociery, 56(6))427-451, 2008.
A Terminology: Definition and use of "Metric"
In this work, we adopt the term "metric" to broadly refer to any form of meta-information that serves as measurable characterization of speech signal. This includes both conventional objective evaluation scores (e.g.. PESQ. STOI, SNR) and more abstract attributes (e.g., emotion category, speaker identity, language, or environment type) that contribute to comprehensive understanding of the signal.
Although some of these attributes may be considered "metrics" in the traditional mathematical sense, such as binary tags or categorical labels, we use the term uniformly to emphasize their role in systematic evaluation and profiling. This unified terminology supports our goal of building versatile and extensible framework that can evaluate various aspects of speech within a consistent and interpretable structure.
By adopting this general definition, we align with recent trends in universal speech evaluation (Shi al., 2024), where wide range of measurements treated under common umbrella facilitate joint modeling, dependency reasoning, and scalable system benchmarking.
B Tokenization with Probability Density Functions
To accommodate heterogeneous evaluation metries under a unified modeling interface, we trans- form all metric values into discrete tokens using either categorical mapping quantization-ban tokenization. This appendix elaborates on the numerical metrrcc tokenization strategies, explains the difference between uniform-over-vvauee and percentile-busel binning, and reports a reconstruction study comparing the two.
B.1 Quantization-aan Based Tokenization for Numerical Metries
For numerical metric bE Brum- we partition its continuous value domain into T bins, each mapped token in the vocabulary V,. Two general binning paradigms considered:
Uniform-over-valuee binning. The raw range [Mmin Ymax] is divided into equal-width value intervals. This simple strategy disregards the data distribution and can lead to highly imbalanced token frequencies when the metric values are skewed.
Percentile-based binning. Instead of equal value widths, we choose T equally spaced points on the empirical cumulative distribution of training data. Each bin therefore contains approximately the same number of samples, producing balanced token priors. This strategy (our default) preserves fine detail in dense regions of the data while avoiding sparse bins.
16

---

Table Quantize Quantize-vdequantin reconstruction error (RMSE/MAE) on held-out test sets. Percentile- based binning consistently outperforms uniform-ove corommovve--vlue binning across all domains. While 1000
bins improve reconstruction fidelity, they introduce more difficult prediction targets, so 7 500
adopted by default.
Domain
Tokenization
Bins
RMSE
MAE
Percentile
500
0.056
0.014
Uniform
500
0.100
0.025
Synthesized
Percentile
1000
0.034
0.007
Uniform
1000
0.097
0,020
Percentile
500
0.139
0.023
Uniform
500
0.223
0,045
Enhanced
Percentile
1000
0.103
0,013
Uniform
1000
0.220
0.039
Percentile
500
0.261
0.024
Uniform
500
0.512
0.045
Corrupted
Percentile
1000
0.165
0.012
Uniform
1000
0.507
0.040
Formally, for percentile-based binning compute empirical percentiles {4ps 17= with 1001/T and assign each sample value
2 = To (v6) = bucket {qp. H),
(7)
where bucket(-) returns the index of the interval [9p-11 qp.) containing v/. The inverse mapping
Ty decodes tokens to their bin centroids.
To evaluate discretization fidelity, conducted a quantize -deuuanzz--euuantiiee reconstruction experiment. Percentile boundaries were learned from the Base set described in Sec. [] ensuring that the bin definitions reflect the empirical distribution used for model training. We then applied both the percentile-based and uniform-over-value tokenizations to three held-out test sets, Synthesized, En- hanced, and Corrupted speech, examine robustness across domains. Two bin counts (T 500 and 1-0000) were compared.
According to Table 3 percentile- based tokenization achieves markedly lower reconstruction error than uniform-over-vallle binning, confirming that balancing token frequencies improves discretization fidelity and generalizes well across domains. Increasing the number of bins from 500 to 1000 further reduces RMSE and MAE, yet we found that such fine granularity makes the autoregressive prediction problem harder (more classes and sparser supervision). Hence, adopt 7-500 as practical trade-off between reconstruction quality and modeling stability.
B.2 Direct Mapping for Categorical Metries
For categorical metrics b E Beat the tokenization function T,, maps each ground-truth label vî directly unique discrete token a pre-defined label set Cy:
of
(o()) E V, {cls), cls,. cls
(8)
This tokenization is lossless and preserves the label semantics in the discrete representation.
C Illustrative Example of Dynamic Classifier Chain
To concretely demonstrate the operation of the proposed dynamic classifier chain model, we present running example based on three representative metrics from our evaluation framework:
Q Emotion: speech emotion quality
Q- Background: background environment condition
Q Clarity: perceptual clarity of the spoken content
17

---

Each of these metrics corresponds to a metadata token in the model's vocabulary. denoted as Mq--nnotion Backgendd- and m- respectively. These tokens act as prompts, instructing the model on which metric to predict next.
Training Phase
During training, the model is exposed sequences of interleaved metadata and value tokens, Suppose the ground-truth metric values for sample are:
Q Emotion "neutral", "
Q-Background = "indoor", Q Q-Clarity = " "clear".
possible training target sequence (after random permutation the metric order) could be:
mg Background "i "indoor", Mq-clarity: " "clear", Mg-mnotion "neutral") ").
The model learns generate this sequence autoregressively:
P(T S)
II P(z S);
where S the input speech representation, and be denotes each token in the sequence (either metadata value).
Inference Phase
At inference time, the user may wish evaluate only subset of metrics, in any order. For instance, if the user queries Q Q-Clarity first, followed by Q-Emotion, the model proceeds as:
1. Input mq-ccarity - predict 2q-ciarity "clear"
2. Input 1/lq-Bration - predict 2q-Bnation "neutral"
The autoregressive context includes both the input speech and all previously predicted tokens, i.e.,
P( P(zq-Enotion
Mig--iaatty:: #q-clarity: Mig-inntioo; S).
This allows the model leverage contextual signals from earlier metric predictions when estimating subsequent ones.
Partially Labeled Training
If only some metrics are annotated in the training data (e.g., only Q Q-Background is known), the model can still be trained with the partial sequence:
T partial = [/ " "indoor"].
This design obviates the need for label imputation or masking, and allows full exploitation of partially labeled datasets.
Key Takeaways
This example illustrates how metadata tokens serve as interpretable and flexible prompts for metric prediction. The dynamic classifier chain enables:
Contextual Reasoning: Predictions conditioned both input speech and prior metric-
value pairs.
Flexible Querying: Arbitrary subsets of metrics can be queried at time, in any order.
Efficient Supervision: The model can be trained on partially labeled samples without architectural changes.
In practice, this flexibility and generalization capability makes the dynamic classifier chain particularly well-suited for large-scale, real-world evaluation scenarios where annotations may be sparse, user goals diverse, and metric interdependencies significant.
18

---

D Detailed Procedure for Two-step Confidence-oriented Decoding
To enhance robustness autoregressive multi-metric inference under uncertain or ambiguous acoustic conditions, we introduce a two-step confidence-orinnter decoding strategy. This method explicitly incorporates token-level confidence when predicting each metric's 's value, focusing on value tokens rather than the less reliable metadata tokens.
For decoded prefix sequence T pevv assume that k metrics have already been predicted, forming the set By C Bg, where Bx is the complete set of K metrics. For each remaining metric b E Bx \ BBx- the decoding proceeds as follows.
We define the confidence of a predicted value token 2ς at decoding step as the maximum softmax probability over the token vocabulary:
Conf(z+) = max P( v: (,,T))
vEV
where S is the input speech signal and Tet is the sequence of previously decoded tokens.
The decoding process is then carried out two steps, as outlined in Algorithm 1
Step 1:
Preliminary Prediction. Append the metadata token m, corresponding to metric
to the current prefix:
T T prev my. Using this updated prefix, the model generates a provisional value token &s and computes its softmax confidence.
Step 2: Confidence-driven Candidate Search. To mitigate uncertainty in the preliminary prediction, extract the top-B candidate values:
z Top-B B P(e S, Tmo where T pre T prev mis-
Each candidate z appended the prefix, forming full candidate sequence T = T pre + 2.. We then evaluate the full-sequence log-likelihood log P(T S), selecting the candidate with the highest score the final prediction for metric b.
After decoding all remaining metrics in Bx B, we retain the top- B partial hypotheses for the next decoding step. This strategy ensures that low-confidence predictions can be revised through targeted, confidence-aweee aware re-ranking, thereby improving the accuracy and stability of metric prediction.
E
Dataset Details
E.1
Basic Speech Data
We use the OWSM data collection for the training, which includes a large variety of speech datasets.
Specifically, in this study, we use a subset of up to 1,000 hours of the OWSM V3 data, including datasets from AISHELL-I (Bu et al, 2017), AMI (Kraaij et al. 2005), BABEL [IARPA 20IT), and Common Voice (Ardila et al. 2020), ranging from 48 languages. The licenses of the datasets are Apache 2.0. CC-BY 4.0, JARPA Babel License, and CCO-1.0, respectively.
E.2 Corrupted Speech Data (Simulation)
The simulated corrupted speech data from the URGENT Challenge contains source speech from five public corpora, including DNS5 LibriVox data (Dubey et al. 2024) LibriTTS (Zen et al. 2019).
Common Voice 11.0 English portion (Ardila et al. 2020), VCTK, and WSJ (Paul & Baker 1992).
Noises are taken from the DNS5 challenge corpora (which collected them from Audioset (Gemmeke)
et al. 2017) and Freesound (Fonseca et al. 2017)) as well as from WHAM! (Wichern et al. 2019). Artificial reverberation uses room impulse responses from the DNS challenge that have been sim- ulated based on the image method (Allen & Berkley 1979 Dubey et al., 2024h. We use the same configuration as described Zhang et al. al. (2024): SNR ratio from Ui (7 20) dB, reverberation and clipping probability of 25% and bandwidth limitations sampled from {8, 16, 22.05, 24, 32) kHz.
19

---

Algorithm 1: Two-step Confidence-oricntte Decoding (Single Metric Case)
Input: Speech input S; model f: current prefix T previ target metric b; beam width B Output: Updated prefix T = T prev + [ris; 26]
Step I: Preliminary Prediction
Let rhg be the metadata token for metric
Append m prefix: T < T pee + ri66
Step 2: Confidence-driver Search
Let prefix to current metric: T pre + T prev + reb Retrieve top-F B alternative values:
2
Top-B B P(u S, T p
Initialize best score s DO, best sequence T T foreach 2, E (
Construct candidate prefix: T + T pre + 2b.
Compute log-probability score + log P(T ( S)
if then T T,
8" 8
return T
All data sampled at 48 kHz. total, we generated in this way 1300 hours of clean and corrupted speech pairs.
This dataset is split into 808,181 training, 3k validation, and test examples, respectively. Since these data are derived through simulation from existing datasets, the licenses of such datasets are inherited.
For additional training data we use also the Voice Bank+DEMAND Veaux et al. (2013) 3 Thiemann eet al. (2013) dataset. The Voice Bank+DEMAND Veaux et al. (201)); Thiemann et al. (2013) dataset widely-used benchmark for SE and noise suppression research. This dataset combines clean speech recordings from the Voice Bank corpus with noise samples from the Diverse Environments Multichannel Acoustic Noise Database (DEMAND). The dataset provides paired clean and noisy utterances and is sampled at 48 kHz. The clean speech component consists of recordings from 30 speakers (15 male, 15 female) from the Voice Bank corpus, sampled 48 kHz. For training purposes, 28 speakers (14 male, 14 female) contribute approximately 11,572 utterances, while the test set contains around 824 utterances from 2 speakers (I male, 1 female)
included in the training set. The noise samples from the DEMAND database, which includes 16 diverse real-world environments. For creating the noisy mixtures, the training set incorporates 10 noise types (8 from DEMAND plus 2 artificially generated noises) at four different signal-to-novl ratios (SNRs): 0, 5, 10, and 15 dB. The license of the dataset is Creative Commons Attribution 4.0 International (CC-BY 4.0).
E.3 Enhanced Speech Data
For enhanced speech evaluation, we leverage data released as part of the Universality, Robustness, and Generalizability for EnhancemeNT (URGENT) 2024 challenge, hosted at NeurlPS 2024 (Zhang et al.
2024). The dataset includes blind test set consisting of 1, 1,000 noisy utterances, 500 from simulated conditions with available clean references, and 500 from real-world recordings without references. These real-world samples reflect wide range of acoustic challenges, including environmental noise, reverberation, and device artifacts. All recordings are provided at a 16 kHz sampling rate.
The noisy inputs were processed by 1 114 enhancement systems (the official baseline and 113 participant submissions), yielding a total 114,000 enhanced utterances (approximately 293 hours of speech). Human evaluation was also conducted: 300 utterances per system rated using mean opinion scores for each of the 23 final-round systems, with an equal number drawn from simulated and real-world scenarios.
Following (hi al. (2025b), the noisy utterances are partitioned into training, development, and test subsets in an 85:5:10 ratio. These splits correspond to approximately 249, 14, and 30 hours of speech,
20

---

respectively. Half of the data is paired with reference signals, allowing both reference-basee and reference-free assessments.
This diverse and high-quality dataset enables us to rigorously assess ARECHO's ability to generalize across enhancement algorithms, acoustic conditions, and evaluation configurations in real-world SE applications.
E.4 Synthesized Speech Dataset
For synthesized speech evaluation, we utilize two complementary datasets: the VoiceMOS 2022 Challenge dataset (Huang et al.) (022) and the NISQA dataset (Mittag et al. (02T).
The VoiceMOS 2022 Challenge dataset (Huang et al. 2022) was developed for a community-driver initiative advance speech quality assessment research. contains speech samples from 187 distinct text-to-speech (TTS) systems sampled at 24 kHz. Each utterance receives quality ratings from at least 8 different human listeners using a standardized 5 5-point scale, providing reliable mean opinion scores for training and evaluation.
The NISQA dataset (Mittag al. 202T) offers a different perspective with its multi-dimensionn quality annotations. Beyond overall MOS ratings, provides fine-grained assessments of specific quality attributes: coloration, discontinuity, noisiness, and loudness. Sampled at 48 kHz, this dataset comprises over 14,000 audio samples created under diverse conditions, both simulated (including various codecs, packet-loss scenarios, and background noise levels) and live environments (such as mobile phone calls, Zoom, Skype, and WhatsApp communications). This comprehensive approach enables detailed analysis of quality perception factors across different communication contexts.
The licenses of the VoiceMOS 2022 challenge dataset scripts for downloading, processing, and listening test results are based on BSD 3-Clause License. The audio samples retain the licenses from their original sources. The Voice Conversion Challenge (VCC) datasets have varying licenses: VCC2016 and VCC2018 use Creative Commons Attribution 4.0 International (CC-BY 4.0), while VCC2020 audio samples are licensed under Open Data Commons Open Database License (ODbL) v1.0 with database contents under Database Contents License (DbCL) v1.0. The BLIZZARD dataset is available under a research license agreement requiring individual approval. The ESPNET-TTS dataset uses Libri LibriVox recordings, which in the public domain in the USA. The NISQA dataset is licensed under Creative Commons Attribution 4.0 International (CC-BY 4.0).
E.5 Experimental Dataset Details
Here, we provide a detailed breakdown of the datasets used in our experiments. The data spans four major domains: simulated corrupted speech, enhanced speech, synthesized speech, and emotion- labeled speech. All audio is standardized a 16 kHz sampling rate, and where applicable, samples are paired with reference signals (e.g.. for simulated corrupted speech). The datasets were preprocessed and partitioned to support both multi-domain training and domain-specific evaluation.
Training and Development Sets. To investigate the effect of data scale model performance, we prepare two training configurations with varying coverage across domains. The following table summarizes the number of hours contributed by each domain to the Base and Scale training sets:
Table 4: Training data composition across domains.
Domain
Base (hrs)
Scale (hrs)
Enhanced Speech
50.00
100.00
Simulated Corrupted Speech
99.91
999.00
Basic Speech Data
97.91
977.80
Synthesized Speech
60.94
60.94
Total
308.77
2137.74
In both configurations, we use a shared development set of 18.65 hours for validation and early stopping.
21

---

Test Sets.
Each domain is evaluated using a held-out test set to enable domain-specifie benchmark
ing:
Simulated Corrupted Speech: 4.51 hours
Enhanced Speech (URGENT2024); 30.12 hours
Synthesized Speech: 3.46 hours
For all domains except simulated corrupted speech data, preserve the original test split provided by the corresponding source dataset to ensure comparability with prior work. For corrupted simulated speech, where no canonical split was available, we randomly sample 3,000 utterances from the training data and reserve them for development and test purposes, respectively.
To facilitate reproducibility, we will release the detailed utterance IDs used in each split, along with the corresponding evaluation metries used during training and testing.
Metrics in ARECHO
As discussed in Sec. ARECHO incorporates two types of metrics: (1) those computed using VERSA, standardized evaluation toolkit that integrates range of open-source expert models for speech assessment; and (2) those derived from pre-annotated information provided within the datasets. In this appendix, we first detail the VERSA configuration used to compute the various metrics, followed by complete list of all metrics included in ARECHO modeling.
F1 VERSA Configuration
The VERSA configuration used for ARECHO's metric computation is shown in Listing In practice, some metrics are CPU-only, while others can benefit from GPU acceleration. To optimize efficiency, we divide the configuration into two subsets, one for CPU-based metrics and another for GPU- supported metrics, and submit them parallel jobs using SLURM to accelerate the overall evaluation process.
We apply a post-filtering step to remove NaN values from the predicted metrics. For metrics derived from Qwen2-Audio, we additionally discard any predictions that do fall within the predefined category sets.
This file contains the configuration for various universal metrics used in speech quality assessment for ARECHO.
visgol metric
visgol: visual quality of speech
name: visqol
model: default
Word error rate with ESPnet-OV8 model Hore model_ tag can be from the ESPnet huggingface https:// cog
The default model is 'espnet/owam, v3 1tte///w --lid: the nbest language tag lid
model_tag: default nbest: 1
nomad (reference (eseencce-dsedd) metric nomad: nomad reference-bar model
name: nomad cerra_cacheemmma.pt model_cache:
sTmr related metrics
srar: speech reverberation modulation energy ratio name: srmr
s_cochlear_fitent 23
low_freq: 125
22

---

min_cf: 4
cfx_cf: 128
fast: True
norm: False
Emotion simslarity calculated based emolvec --
the emotion similarity with emo2vec
-name: emolvec_eimilarity
noresqa related metrics
noresga: con-matching reference based speech guality assessment
name: noresqa
metric_type: #0: SRESQA-scorm I: NORESQA NOS
pysepm related metrics
pysepm.fusegsnr: seegunny--eeighed segmental suR
pysepm_llm: Log likelthood ratio
pysepm_uss: weighted spectral slope
pysepm_cd: cepstral distance objective speech quality measure Sysepm_Csig. Cbak, pysepm_Covl: composite objective speech quality
susepp_cati.hipee pysepm, mids_mid, cysepa low: coherence speech intelligibility indea pysepm_ncm ceatlized-covariann measure pysepm
nasqa score for speech quality assessment
nisga_mos mos_pred:: NISQA MOS prediction
noipe. NISQA noise prediction
nisga_ NISQA distortion prediction
-nisqa_col_pred: NISQA color prediction
--nisqa_loud_pred NISQA loudness prediction
NOTE: pretrain model can be downloaded with </tools/setup. sh
name: nisqa
nisqa_model_path: /bools//ISSA//veiihh////as/
discrete speech metrics
speech bert: speech bert score
speech. speech bleu
speech
token_ speech_ distancdistance. speech token distance score
name: discrete_speech
med fo related metrics med: mel cepstral distortion
.. cor:: fo correlation
.. f0srmse: fo root mean square error
name: mcd_f0
f0min: 40
f0max: : 800
mcep_shift:
mcep_fftl: 1024
acep_dim: 39
mcep_alpha: 0.466
taraatcchtoleeaaee: 0.1
pover thee_threehold: -20
dtw: false
An overall model on MOS- bench from Sheet toolkit s- the mos prediction from sage name: sheet_ssqa
peaq related metrics
peag: perceptual evaluation of speech quality name: pesq
23

---

stos related metrics
stos: short-time objective entelligibility
name: stoi
pseudo subjective metrics
utmos: UT- score
dnsmos: DNS NOS
picmos: PLC-NOS score
aecmos: AEC-NOS score
name: pseudo_mos
predictor__ypes: ["utmos", "dnsmos", "plcmos", "singmos", "utmosv2") predictor_args:
utmos:
fa: 16000
dnsmos:
fs: 16000
plemos:
fa: 16000
s singmos:
fs: 16000
utmosv2:
fe: 16000
Ford error rate with
model
whisper_ peper_wer:: word error rate openai- wphsper
name: whisper_wer
model_tag: default
beam_ text_cleaner: whisper
scoreq (reference-based) metric
--
ref:: scoreq baference-based model
name: scoreq_ref
data_domain: natural
model_cache: versa_ stteeaaacacctoorro.----
scoreg (non-reference-bees
metric
-- scoreg. nr: scoreq non non ree-rreeeeneebase model
scoreq_nr
data_domain: natural
model_cache: verra_cache/sooreept-tooe
Speech Enhancement based Netrics
model tag can any ESPnet- huggingface repo
se_ st_ss_arr the SN- from a reference speech enhancement model
name: se_snr
model_tag: default
PAН: Prompting Languaee Models for Audio Quality Assessment https://github.co http://ww..ccmmmooaaa/////////e/
name: pam repro: true
cache_dir: versa_cache/pam
io: soundfile TEXT ENCODER COMPIG text_model- 'gpt2>
text_len: 77
transformer_ dim: 768
freeze_tee tet__tee______e_ wwweennddeeeeihheeeeeeee True
AUDIO ENCODER CONPIG
audioenc_name: "TTSAT>
out_emb: 768
sampling_ras
rate: :ate::
44100
duration: 7
24

---

fmin: 50
fmax: 8000 #14000
n_fft: 1024 1028
hop_size: 320
mel_bins: 64
vineow_size: 1024
PROJECTION SPACE CONPIG
d_proj: 1024
temperature: 0.003
TRAINING AND EVALUATION CONPIG
classes: 527
batch_size: 1024
demo: False
Speaking rate calculating
ra correct matching words/character counts
name: speaking_rate
model_tag: default
beam_size: 1
text_cleaner: whisper_basic
Audioboz desthetics (Unified automatic quality assessment for speech music, and sound.)
name: audiebor.aatteetics
batch_size:
cache_dir: versa_cacce/audiooo..
match calculating
--asr_ --ass_match, inror_rater correct matching words/character counts
name: asr_match
model_tag: default beam_size: 1
text_cleaner: whisper_basiz
speaker related metrics
sph sph_simsl speaker cosine similarity
name: speaker
model_tag: default
asuspoof related metrics
asuspoof. evaluate how the generated speech is likely to be classifeied by deepfake classifier name: asvepoof_score
--
signal related metrics
str: signal to interference ratio
sar: signal to artifact ratio
sdr: signal
distortion ratio ci-sdr: scole-invariant signal to distortion ratio
si-snré: invar-invariant signal to noise ratio improvement name: mitric
Ketrics with Quen2Audio
Inference pipeline follow the quen2, audio release https https://github.co com / Suvell//Que2-Audss. Audio
ezmaple of using customized prompt (we offer default ones)
1. Speaker Cheracteristtes
name: gpen2_audlo_sppkkrr.com
name: gven2_audio_speaeeee
name:
To add customized prompt, use following item:
25

---

name: qwen2. audio_ spe.ker_akeaappppppe age
prompt: What the age of the speaker? Please in 'child', '20s', '30s', '40s', '50'', 260'', 770'', 'sentor'.
name:
impaireenn
2.
Foice Properties
name:
name:
qwon2_audio_pitchh..rrm
name:
qven2_audio__olce__7777
name:
quen2_ au___aadd__sspeechhee volume, _evem level
3. Speech Content
name:
qven2_audio__aaggue
name:
quon2_audio.specch.rrgeeeee
name:
name:
Speech Delivery
name: gren2_audioo.ppeeceenoooo
name:
awen2_audio.specchh.caa.
name:
qwen2_audie4 spw.audchooeeeehratttee.a
name:
wwen2_audio__aughterro
6. Interaction Patterns
name: aeano_aaddooooeelppiing.....
6. Recording Environment
name:
name:
qwen2_
name:
gren2_auudi__chameee...7777.
Listing 1: VERSA configuration for ARECHO.
F.2
Metrics Information
The metries information is detailed in Table and
Table lists the 47 supported independent metries in VERSA. We provide a summary of their key characteristics below:
Metric Types. Among the 47 metrics, 25 are numerical and 22 are categorical.
Model Dependency. All 47 metrics require pre-trained models for inference (model-based),
Value Ranges. The supported numerical metrics
diverse range of value scales: 11 metrics bounded in [1, (e.g., DNSMOS, NISQA, UTMOS).
metrics are bounded in [1, 10] (Audiobox Aesthetics metrics).
- 2 metrics are bounded in [0, 1] (e.g., PAM, SpoofS).
4 metrics have unbounded ranges (-o0, 2o) (c.g.. SE-SI-SNR, SE-SDR).
26

---

Table 5: List of supported independent metrics in VERSA. The "Model Based" column represents metrics that need pre-trained models. The "Target Direction" column indicates which direction is desirable for each metric without being overly technical.
Name
Type
Range
Model Basod
Target Direction
Reference
Deep Nošse Suppression MOS Score P.855 (DNSMOS P.855)
numericaa
[1.5]
Reddy
2002
Deep Noise Suppression MOS Score of P.808 (DNSMOS P:S0S)
numerical
11.5]
V
Konw ZC
Speich Quality and Naturalness Assessment Culoration (NISQA-COL.)
numerical
[1.5]
Mirat OF
Speech Quality and Naturalness Astosement Discvernmiiy (NISQA-DIS)
sumerical
[1.5]
Mistr
2011
Speech Quality and Nuturalness Assesamest Londness (NUS
sumerical
Specch Quality Naturalness Assessment
(NISQA-MOS)
numerical
[1.5]
Speech Quality and Naturalness Assesement Noisinesa (NISQA-NOI)
numerical
01.5]
U'Takyo-Saril ab System for ValeeMOS 2022 (UTMOS)
numerical
[1.5]
Niskee 1D
41244
Packet Lass Concealment-ooeu MOS (PLCMOS)
numerical
[1.5]
21125
10
Singing MOS (SingMOS)
numerical
[1.5]
11
Subjective Speech Quality Assessment (SSQA) SHEFT Toolkit (UTMOSV2)
nunerical
[1.5]
UTDokyo-Sarul.ab System VaizeMOS
sumerical
[1.5]
13
Speech Quality with Contrattvve Regression (SCOREQ) wo. Ref.
numerical
[1.5]
Rapato 21
14
Speech Enhaneement based SI-SNR (SE-SI SI--SRR
pumerical
[-inf, inf)
Zharz 21
2024
15
Spech Enhunemment based C1- SDR (SE-C)-SDR)
sumerical
(-inf, inl)
Speech Enhancemennt-eee
(SE)AA)
(-inf,
Speech Enhancenennt base
(SE-SDR)
pumerical
(-inf,
18
Prompting Langua-L Models (PAM) metric
numerical
[0. 11
Bestmni 20
19
Spech--t--cevrree reverrrteeevoaaaaaaiii Modulation Energy Ratio (SRMR)
numerical
[0, in
Thic 2
X010
20
Speaking Wonil/Caracter Rate (SWR/SCR)
sumerical
inf)
Railerd
7173
21
Aari-spoofin Score (SpoefS)
nunerical
Langrage Mlestiiicatonn
Calegorical
23
Audishhox Aesthetics Content Enjoyment (AA-CE)
numerical
[1. 10J
24
Audiohox Aesthelics Content Usefulness (AA-CU)
sunerical
[1. 10]
[pnnl 21 a1
3325
25
Aadishox Aesthetics Production Complexity (AA-PC)
sumerical
[1. 101
2395
26
Audiobox Aesthetics Production Quality (AA-PQ)
sumerical
Quen2 Receeding Envinoement
Type (Q-Chunnellype)
calegorical
28
Qwen2 Speech Content Language (Q-Lang)
categorical
29
Qwen2 Speech Delivery- Emotional Vocalizations (Q EmnoNoolllzztionn
categonical
Lage 4:
21,25
30
Qwen2 Voice Properties - Pichh Range (Q PPPchhangg)
calegonical
L
4024
31
Qwen2 Recording Envinonment Quality Qu-Eriualltt))
calegonical
32
Qwen2 Speaker Characteristies Age (Q-Age)
categoncaa
Qwen2 Speaker Characteristies Count (Q SpeakerCount)
categonical
34
Qwen2 Speaker Charactristiess Gender (Q- Gender)
callgooriaal
C2g e1
2022
35
Qwen2 Speech Delivery - Style (Q-Speskinggtylle
calegonical
110 41 31
2024
36
Qwen2 Receding Favince ment- grond (Q--Racgrrond)
categorical
37
Qwen2 Speech Delivery- Clarity (Q Claitty
calegonical
Spesch Delivery- Emotion (Q-Emution)
calegorical
3V
Qwen2 Speaker Characteristies Spoech Inquirment (Q (-Specchmmppmmmen)
calegorical
40
Qwen2 Speech Content- Purpase (-Purpase)
calegorical
7 750
31.41
41
Qwen2 Speech Delivery Rate (Q- (--SpecchRate)
categanical
42
Qwen2 Speech Conteni- Register (Q-Content Ragister)
catogorical
Voice Propasties- Volume Level (Q-Volumel
categorical
Qwen2 Speech Content- Vocabulary Complexity (Q-NeeCompleit)))
categorical
45
Owen2 Voice Properties Pitch (Q-Pitch)
categanical
46
Qwen2 Voice Propertics Voice Type (oiiceyTy)))
categonical
L
47
Prodicted Test Length
numerical
[0, inf
3 metrics are semi-bounded in [0, oc) (e.g., SRMR, SWR/SCR, Predicted Text Length).
Coverage Domains. The metrics span a wide set of evaluation domains:
Perceptual speech quality: DNSMOS, NISQA, UTMOS, SSQA, SCOREQ.
Speech enhancement: SE-SI-SNR, SE-CI-SDR, SE-SAR, SE-SDR.
Speech generation and profiling: Audiobox Aesthetics, PAM, Predicted Text Length.
Security and robustness: SpoofS (anti-spoofing)
Speech metadata analysis: Qwen2 suite, covering 22 distinct categorical dimensions.
Several metrics are provided for overlapping domains to improve robustness and account for domain- specific modeling biases. For instance, multiple metrics exist for speech quality prediction (e.g.. DNSMOS, NISQA, UTMOS, SSQA, SCOREQ). These metrics differ in terms of training data, model architecture, and target annotations, leading varied sensitivities across distortion types and speaker/content conditions. By including multiple predictors within the same domain, VERSA enables volisvvaiiotion of quality assessments and mitigates the risk of relying single potentially biased estimator. This redundancy also supports ensemble or consensus based evaluations, which are critical when deploying models across diverse real-world scenarios.
Table presents 25 supported dependent metries in VERSA. These metrics require auxiliary ref- erences, such as clean speech, transcripts, or pitch tracks, and are primarily used in settings where such ground-truth information is available (e.g., supervised evaluation of synthesis, enhancement, or recognition systems). Below we summarize their characteristics:
27

---

Table 6: List of supported dependent metrics in VERSA. The "Model Based" column represents metrics that need pre-trained models. The "Target Direction" column indicates which direction is desirable for each metric without being overly technical.
Model Based
Target Direction
Type
Range
Reference
Mel Copstral Distortion (MCD)
numerical
int)
X
Kubichek
1993
Correlation (F0-CORR)
numerical
[-1.
Hovsth
3
FO Root Mcan Square Error (FU-RMSE)
numerical
[0, int)
X
Davart ei
Signal-to-attffct Ratio (SAR)
inf)
5
seuualtoodiisortion Ratio (SDR)
numerical
(-inf. inf)
Fevelte 0
300
6
Pereoprual Evaluation of Speech Quality (PESQ)
namerical
[1.5]
Hime Objective Intelligbbility (STOI)
numerical
8
Speech BERT Score (D-BERT)
numerical
[-1, 1]
Sockt 6 31
9
Discrete Speech BLEU Score (D-BLBU)
numerical
[0, 11
Discrete Speoch Token Edit Distance (D-Distancc)
numericall
[0. 1]
Socld 6
A021
11
Speech Quality with Contrastive Regression (SCOREQ) w. Ref.
numetical
[1.5] 51
Al1201
ASR Mismatch
(ASR-Mismatch)
in[]
Virual Speech Quality Objective Listener (VISQOL)
numetical
[1.5]
Iid 2i me
2140
Frequency- Weighted SEGmental SNR (FWSEGSNR)
(-inf, inf)
X
Inblle
Weighted Spectral Slope (WSS)
KBM [762]
16
Cepsirum Distance (CD)
numetical
[0, int)
Banwel III
1979
Composite Objective Speach Quality Signal (Csig)
[1.5]
Composite Objective Speech Quality Background (Cbak)
numerical
11. 51
Ha & Loizou
307g
19
Composite Objective Spesch Quality Overall (Cavl)
numerical
[1.5] 5]
Hi K Tamai
Coherence Speech Intelligibility Coherence and Speech Intelligihility Index Low (CSII-LOW)
High (CSI--HIGII)
[0,
21
numerical
[0, 1]
Kala & Arblun
Coherence and Speech Intelligibility Index Mid (CSII-MID)
numerical
11
23
Normalized- Covariance Measure (NCM)
numerical
[-1, 1]
Chea & Lesze
2010
24
Convolutive invariant Speech distortion Ratio (CI-SDR)
numetical
(-inf, inf)
Bocileter el
2021
Scale isvee-iavariant& Specch-lo-noise (SI-SNR)
inf)
Booddeker
Model Dependency. Among the 25 dependent metrics, 10 are model-based (e.g., PESQ, VISQOL, D-BERT, D-BLEU), while the remaining 15 are traditional signal- or statis- tical metrics that operate without pretrained models.
Target Direction.
21 metrics have 1 as the preferred direction, indicating better performance with higher values.
metrics are better when minimized (1), including Mel Cepstral Distortion (MCD), FO-RMSE, Weighted Spectral Slope (WSS), and Cepstrum Distance (CD).
Value Ranges. The dependent metrics span various value scales:
metrics are bounded within fixed ranges, such as [o, 1] (e.g., STOI, CSII variants, D-BLEU).
metrics use perceptual MOS-like scales (e.g., PESQ, Csig, Covl) within [1,5]. metrics span |-1, 1] (e.g., FO-CORR, D-BERT, NCM).
Several metrics are unbounded or semi-bounded in [0, oo) or (-o0, xo), particularly SNR- and SDR-based metrics.
Metric Coverage and Redundancy. The dependent metrics are intentionally diverse to capture distinct aspects of speech fidelity and intelligibility. For instance:
Pitch-aware metrics: MCD, FO-CORR, and FO-RMSE quantify pitch and spectral
similarity.
SNR/SDR variants: Multiple versions (SAR, SDR, SI-SNR, CI-SDR, FWSEGSNR) are included assess distortions under different assumptions (e.g.. scale- convolution- invariance).
Perceptual quality: PESQ. VISQOL, Csig/Cbak/Covl provide subjective approxima- tions of human ratings.
Token-buul based semantic fidelity: D-BERT, D-BLEU, and D-Distance evaluate similarity discrete or latent representation spaces.
Including multiple metrics within the same subdomain (e.g., both PESQ and VISQOL perceptual quality, or both SDR and SI-SNR for distortion) enhances robustness and allows for comprehensive evaluation across system types and data conditions. This redundancy particularly important when no single metric reliably aligns with human perception in all scenarios.
28

---

Table 7: List of supported non-matching metries in VERSA. The "Model Based" column represents metrics that need pre-traie pre-trained models. The "Target Direction" column indicates which direction is desirable for each metric without being overly technical.
Model
Target Direction
No.
Name
Type
Range
Reference
Based
Non-matching Reference Speech Quality Assessment (Noresqa)
nomerical
[1.
Manocha
2
OpenAl Whisper Model Word Firor Rate (WER)
numerical
[0. inf)
Kivlood or
2023
3
OpenAl Whisper Model Character Ervor Rate (CER)
numerical
[0, inl
Kactord
2023
4
Emotion Similarity (EMO-SIM)
numerical
1-1.11
Ma al 2029
5
Speaker Similarity (SPK-SIM)
numerical
[-1. 1]
Jung 2024
Non-Matching Reference Quality Assessment (NOMAD)
numerical
[1.5]
Rigain 01744
7
Log Likelihood Ratio (LLR)
numerical
[0, inf)
Hi & Leazol
20774
Table 7 summarizes the 7 supported non- mantmaihiing metrics in VERSA. These metrics are designed to operate in scenarios where the reference not a direct ground-truth pair (e.g., a sample from the same speaker, emotion class, or general quality distribution), enabling broader evaluation capabilitiess such as speaker consistency or semantic fidelity under more relaxed constraints.
Model Dependency. All but one metric (LLR) rely on pre-trained models for their prediction. These include advanced encoders for speaker, emotion, or language content, reflecting recent trends toward model-basee alignment and comparison.
Target Direction.
metrics prefer higher values (1), including Noresqa, NOMAD, and similarity- based measures like EMO-SIM and SPK-SIM.
2 metrics, Whisper-based WER and CER, evaluated with lower-is-better semantics (L), indicating reduced transcription error.
Value Ranges. These metrics span various scales:
3 metrics are bounded in [1, 5] (Noresqa, NOMAD, LLR).
2 metrics range from [-1, 1] (EMO-SIM, SPK-SIM), reflecting cosine similarity scales.
2 metrics are semi-unbounded in [0, oo) (WER, CER).
Use Case Rationale. Non-matching metrics are particularly valuable in settings where exact pairwise references are unavailable or inappropriate. For example:
Speaker and emotion similarity (SPK-SIM, EMO-SIM) assess style preservation or consistency across generated outputs, even if reference content is not identical.
Quality predictors such as Noresqa and NOMAD estimate subjective quality with a flexible reference that may differ in content length.
Whisper-based WER/CER offer standardized ASRRorientee evaluation interface with- out needing paired transcriptions from a target reference set.
Together, these metries expand the evaluation scope beyond classical paired setups, allowing for more generalizable and accessible assessments across real-world conditions.
Table 8 summariize the 8 supported ground-truth metrics in VERSA. These metrics represent either directly observed attributes (e.g., simulation configurations, annotation- ratings) or oracle-level information that serves as the ultimate reference for evaluating predictive models.
Metric Types. The set includes both numerical and categorical metrics:
metrics are numerical (e.g., RT60, SNR, MOS scores).
metrics are categorical (e.g., language identity, room size).
Target Direction. Most ground-truth metries do not define direction of improvement, as they serve as reference labels rather than performance indicators. However, numerical metrics (SNR Simulation, URGENT MOS, VoiceMOS Real MOS, and NISQA Real MOS) are directionally desirable with higher values (t), indicating better signal quality or subjective perception.
Value Ranges.
metrics use a MOS-like scale bounded in [1,5] (VoiceMOS, URGENT MOS, NISQA).
29

---

Table 8: List of supported ground-truth metrics in VERSA. The "Model Based" column represents metrics that need pre-trat pretrrined models. The "Target Direction" column indicates which direction is desirable for each metric without being overly technical.
Target Direction
No.
Name
Type
Range
Reference
1
Real Language
categorical numerical
2
Reference Text Length
[0. inf)
3
RIR Room Size
categorical
4
RT60
numerical
[0. inf)
5
SNR Simulation
numerical
(-inf, inf)
6
URGENT MOS
numerical
[1.
Zhang al. 2024
7
VoiceMOS Real MOS
numerical
[1.
Huang et 2024c
8
NISQA Real MOS
numerical
[1.5]
Mittgg 2021
RT60 and Reference Text Length are semi-bounded in [0, oo).
-NNR Simulation is unbounded (-o0, oo), representing yeal-world variability in noise conditions.
Use Case and Role. Ground-truth metrics serve three primary purposes:
Evaluation targets: MOS ratings (e.g., VoiceMOS, NISQA, URGENT) used to supervise or validate quality prediction models.
Auxiliary context: Variables such as RT60 or SNR are useful for conditioning or interpreting model behavior in specific acoustic environments.
Oracle supervision: Some categorical features like language room size are ground- truth labels used in training or stratified evaluation.
These metries are typically unavailable in fully automatic pipelines but are crucial during dataset construction, model validation, and controlled benchmarking.
F.3
Metric Coverage
Due to failure metrics' calculation, missing reference information, or missing annotation, it common have a incomplete metric set for each sample. To this end, we provide the coverage of each metrics for the Base and Scale set in Table which helps to understand the metric imbalance issue existing the multi-metric estimation.
G Detailed Experimental Setup
G.1 Model Architecture
Baseline. We adopt the UniVERSA architecture with support for both numerical and categorical
metrics. The model leverages pretrained La-LM- encoder (Chen et al. 2022) as the audio frontend, extracted via the S3PRL interface with multilayer features enabled (wen Yang et 2021). To retain pretrained knowledge, we freeze all parameters in the upstream encoder during training. The frontend outputs are passed attention heads per layer, a hidden dimension of 1024, and dropout regularization applied at various levels (general: 0, attention: 0. positional: 0.1). The encoder uses a convolutional input layer and applies layer normalization before self-attention blocks, adopting linear position-wis layers and lightweight kernel size of
Transformer-bast based audio encoder composed of 4 layers with
Mean pooling is used to aggregate encoded sequences, followed by metric-specific projection head implemented as X-vector-based prediction head. The model set the prediction head for each metric individually to accommodate multiple simultaneous prediction heads (e.g.. for different metrics). The total parameter size for the baseline VniVERSA 604.38M with 288. 93M learnable parameters.
Tokenization. We use a token size of 500 for the default numerical tokenization. Linear tokenization applied to all numerical metrics.
ARECHO. For the ARECHO model, we use the same audio frontend and encoder as the VEIVERSA model. The final metric decoder is Transformer-base module designed to handle the diverse
30

---

Table 9:
Metrics with Percentage of Occurrences in the Base and Scale training sets.
Base Training Set
Scale Training Set
Occ. (%)
Metrics
Occ. (%)
Metrics
98.49
Q-PitchRange, Q-Background, Q- Q-NoiceType, Q-
98.24
Speakking Q- Q-VoicePitch, Q-SpecchRate, Q-
EmoVocalizaiio, Q-Geader, Q-SpeakerCount. Q-
ChannelType, Q-Gender, Q-Emo Vocalization.
Q-
SpeakingStyle, Q-Emotoon Q-Pich, Q- Q-Purpose, Q- VolumeLLeeel, Q EnvQuality. Q Sppeechlmarrmmen, Q Age. Q- VoeComplexity, Q-Clarity. Q-ContenrReiistrr Q-SpechRate, Q Q-ChannelType
Emotion, Q-Age, O-SpeakerCount, Q-Clarity.
Q-
VucCompeexity, Q ConiemRReiiterr Q Purpose,
Q-
Speschlmpariment, Q-PitchRange. Q-VoiceType,
VolumeLovel, Q-Background. Q-EnvQuality
98.41 Q-Lang
98.16 Q-Lang
86.55
SE-CI-SDR, LID, PAM. NISQA-MOS, UTMOS,
83.28
UTMOS+2, DNSMOS P.835, Real Language, NISQA-
SingMOS, SCOREQ. AA-PC, Real Language, NISQA NOL, NISQA- LOUD, SAR,AR, NISQA DIS, SWR/SCR, PLCMOS, NISQA-COL, DNSMOS P835, SpoofS, UTMOSv2, SE-SDR, AA-CU, AA-CE, DNS-
DIS, SpootS, NISQA-LOUD, AA-PQ, AA-CE, CU, SingMOS, NISQA-COO NISQA MOS, PAM, SSQA, SpeakingRate, SE-SAR, AA-PC, SE-SDR, CI-SDR, SCOREQ. NISQA-NOI, UTMOS, DNSMOS PS08, PLCMOS
MOS P808, AA-PQ. SSQA
86.54
SE SNR
83.27
SE SNR
80.44
SRMR
75.17
SRMR
38.44
RIR Room Size, SNR Simulation
50.45
SNR Simulation, RIR Room Size
35.65
SPK-SIM, D-BLEU, D-Distance, D-BERT
40.63
D-BLEU, D-BERT, SPK-SIM, D- D-Distance
29.78
Reference Text Length, ASR--Mismate NOMAD, EMO-SIM, Noresga, SCOREQ Ref., Predicted Text Length
39.11
NOMAD. Reference Text Length, EMO-SIM, Noresqa, ASR-Mismatch, SCOREQ Length
Ref., Predicted Text
26.79
STOI, FORMSE, MCD, SAR, PESQ. SDR, CI-SDR
29.04
FORMSE, PESQ, STOL, SAR, CI CL-SDR, MCD, SDR, FDCorr, SI.SNR
26.78 78.78
FOCom, SI-SNR
27.51
Cbak, Covl, LLR, WSS, CD, VISQOL, FWSEGSNR. Csig. NCM
20.92
Chak, FWSEGSNR, LLR,, Covl, WSS, VISQOL, Csig,
CD, NCM
27.50
CSII-MID, CSU-HIGH
20.91
CSII-HIGH, CSU--MID
26.61
CSII-LW
20.24
CSHI-LOW
25.19
RT60
19.18
RT60
3.01
WER, CER
13.22
NISQA Real MOS
1.74
NISQA Real MOS
11.46
WER, CER
1.52
SIR
2.93
VoioeMOS Real MOS
0.38
VoiceMOS Real MOS
0.65
URGENT MOS
0.16
URGENT MOS
space of evaluation targets. It comprises 4 self-attention blocks, each with 4 attention heads and 1024-dimensional feedforward layers. Regularization is applied with dropout rates of 0.1 for general, positional, source-attention, and self-attention components. The decoder adopts embedding-based input layer and applies layer normalization before each attention block. Similar the encoder, uses concatenation after self-attention and supports stochastic layer dropping with rate of 0.1.
To enhance the modeling of metric sequences, rotary position embeddings (RoPE) are enabled (Heo) et al., 2024), allowing better generalization to longer sequences. The decoder is trained with label smoothing improve generalization and reduce overconfidence. Standard start-of- of-sequence and end-of--equeme tokens (<sos>, <eos>) are used for autoregressive decoding when applicable. The total parameter size of the proposed ARECHO model 581.21M with 265. 76M. learnable parameters.
G.2 Training and Decoding Setup
All models in our experiments use Xavier uniform initialization and optimize using the AdamW optimizer with a learning rate of 0.001, scheduled via a warm-up mechanism over 25k steps. Training is conducted with gradient accumulation of 2 and batch size of 16, sorted by descending sequence length for efficiency. All the experiments are trained with GH200 for up to 5 days, with 100 epochs at maximum. It is worth noting that ARECHO uses significantly less GPU memory (50GB vs. 85GB) than Uni VERSA in a similar parameter size.
During decoding,
beam size of to conduct greedy search if not specified.
31

---

Table 10: Main experimental results with complete evaluation metrics for comparison between baseline and ARECHO. The "Domain" indicates the evaluation set used for the model assessment.
Regresden den Metries
Cluciiication Meirin
SRCC
UNSVIRRA IN
18005
5.17
4.13
5.28
0.69
0.65
0.13
0.65
0.41
0.47
0:41
Des
2.11
0:20
11:00
0.45
UNIVTASA
X
61.5 54
0.20
3.88
1.63
0.71
0.7
0.51
0.08
0.41
0.18
0.41
UNIVTASL-T
27
3.55
1.00
068
0.70
0151
0.47
UNSVIAAAA
12065
0.80
3.24
4.10
061
061
0.00
0.67
035
0.4h
Camppidd
UN
70:22
1.10
067
0.11
055
6.30
BNSVTASA
56.79
1.00
3.70
3.76
0.76
0.71
0.51
0.08
0:41
0.19
0.41
Syntheino
IN
X
1.52
0.91
0.5
6.50
UnSVIAAA
06.99
0.20
3.30
4.21
0.6
0.6
D.3
0.89
0/43
0.49
644
Ang.
32-12
2.36
0.79
068
0:71
0.2
6449
UNIVERRA UNIVERSA-T
X
116:03
3.56
2.25
0:59
0.89
0.14
0.73
0.51
0.50
0.49
UNSVIRRIA AARISOO
3.49
0.76
0.75
0.54
0.52
2.55
1.81
0.7
0:1
6/11
ARRITTO
32.63
2.86
1.53
0,57
6.73
0.25
0.55
0.51
Zcole
UNIVTASA ARETSIO
2
151.97
3.60
2.76
0.75
0:25
0.57
0:51
34.37
2.33
1.10
1.35
0.94
0.87
6.76
0.77
0.99
0.58
8.56
UNSVIRSAA
X
0131
6/41
Systhonod
3
AREITIO
1.49
0.90
0.35
0.85
6.72
0.25
0.58
8.34
UGAVERBA
3
0.4
031
AREISSO
25.31
332
118
1.38
0.85
0,87
6.74
0.76
0.97
0.56
8:34
H
Complete Main Experimental Table
To provide more comprehensive assessment of model performance, we expand our evaluation with additional regression metrics (i.e., Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), Balanced Mean Absolute Error (BMAE) in Baccianella et al. (2009) and Spearman' Rank Correlation Coefficient (SRCC)) as well as classification metrics (i.e., Precision and Recall) complementing the original metrics presented Table
Across all domains and data conditions, ARECHO consistently outperforms both baseline models UURSV and Uni VERSA-T) in the majority of regression and classification metrics. Specifically, the lower RMSE and MAE values indicate that ARECHO yields more stable and accurate point-wise predictions, with marked reduction in prediction variance and absolute deviation. For instance, on the Base-AS AvgTest condition, ARECHO achieves an RMSE of 1,86 and MAE of 1.06, compared to 2.36/1.40 1.0 for Uni UniVERSA-T -T and 4. 4.29/3.50 for UniVERSA, respectively.
Furthermore, the improvement in SRCC suggests stronger alignment with ranking orders. In both Base and Scale conditions, ARECHO attains the highest SRCC values, especially in acoustically challenging domains such as Corrupted and Synthesized, where robustness to distortions is crucial. These results reflect the model's enhanced capability to preserve ordinal consistency across perceptual quality metrics.
From classification perspective, ARECHO demonstrates consistent superiority in Fl scores, primarily due to a balanced improvement in both precision and recall. For example, in the Scale caall--ooruutee domain, ARECHO achieves an Fl score of 0.56, with precision of 0.59 and recall of 0.58, which is substantially higher than the corresponding metrics from baseline models. This indicates better dis- criminative performance in multi-metre classification, which is critical for downstream applications such as quality control and speech diagnostics.
Taken together, the augmented evaluation substantiates that the proposed chain-based approach not only improves overall prediction accuracy but also enhances ranking reliability and classification robustness across diverse testing scenarios.
32

---

Table 11: Ablation study: the effect of token size for ARECHO.
Domain
Model
(4)
RMSEE (1)
(AE((1
Mitries LCC
(RCC
(1AU
(t)
Precidon (?)
Chesification
Recall (1)
Usi V
9.21
2.39
0.86
0.54
A39CHO
1,000
30.86
2.53
2.49
1.32
0.54
Enhonced
Un: VERSA T
1,000
59.53
68.94
3,80
3.99
2.07
2.17
0.51
0.86
0.85
0.70
0.73
0.74
0.53
0.53
0.55
0.50
2,86
A383H0
1,000
3.17
1.70
Comppttd
Uni V
1,000
500
50:73
39.80
2,66
2,42
1.29
1.18
0.52
0.77
0.85
0.56
0.73
0.74
0.75
0.76
0.57
0.58
0.56
0.57
0.54
0.54
34.37
0.33
1.10
0.84
0.87
0.76
0.77
0.89
0.58
0.36
A38ONO
1,000
31,68
2,02
1.07
0.54
0.85
0.74
0.76
0.59
0L.8
0.36
VELVER
1.49
0.94
0.53
0.52
Symthesized
ARROCHO
1,000
7.31
1.41
1.49
0,85
0.85
0.71
0.72
0.75
0.75
0.57
0.56
0.55
0.55
I Ablation Experiments and Model Analysis
I.1 The Effect of Token Size
Previous experiments utilized a default token size of 500, which represents a balance between compu- tational efficiency and model performance. However, to gain more comprehensive understanding of the model's capacity to handle longer contexts, we extend our experiments to include token size of 1,000.
Table 11 presents the results of this ablation study across different domains, comparing both models token sizes of 500 and 1,000. Several interesting patterns emerge from this analysis. First, observe that the effect of increasing token size is not uniform across domains and models, suggesting a complex interaction between input characteristies and model architecture.
The enhanced speech test set reveals a contrasting trend for UniVERSA-T, where increasing token size actually improves performance (MSE decreases from 69.94 to 59.33). However, ARECHO still outperforms the baseline at both token sizes, achieving an MSE of 32.63 at 500 tokens compared to 43.32 1,000 tokens. This suggests that while UniVERSA-T benefits from increased context length in enhanced inputs, ARECHO achieves optimal performance with more concise representations.
Perhaps the most interesting results come from the corrupted speech test, where ARECHO shows unique pattern: is the only configuration where increasing the token size to 1,000 yields substantial improvements across all regression metrics (MSE decreases from 34.37 to 31.68. RMSE from 2.32 to 2.02, and MAE from 1.10 to 1.07). This suggests that when dealing with corrupted inputs, the additional context provided by longer token sequences allows ARECHO to better identify and mitigate noise through its confidence-orient significant performance degradation with increased token size in this domain, with MSE increasing from 39.80 to 60.73.
factorization mechanism. In contrast, Uni VERSA-T shows
For the synthesized speech domain, both models show relatively stable performance across token sizes, with ARECHO achieving modest improvements at the larger token size (MSE decreases from 8.63 to ). This stability in the most controlled experimental setting suggests that both models have sufficient capacity to capture the underlying patterns in synthetic data, though ARECHO consistently outperforms the baseline in classification metries.
From practical standpoint, these results have important implications for deploying ARECHO in real-world applications. First, the default token size of 500 appears to provide a good balance between performance and computational efficiency for most domains, particularly for Dev. and Enhanced inputs. Second, when dealing with corrupted inputs, increasing the token size to 1,000 can yield meaningful improvements, suggesting that adaptive token sizing based on input characteristics could be valuable strategy. Finally, the consistent outperformance of ARECHO over UEiVERSTT across most metries and domains, regardless of token size, underscores the robust advantages of our confidence-oriente factorization approach.
33

---

Table 12:
Ablation study; the effect of beam size for ARECHO.
Domain
Beam Size
MSE (1)
RMSE (1)
MAE (.)
Regression Metrics
LOC(TT
SRCC (*)
KTAU (1)
Aec (1)
Precision (1)
Classifictioon Metrics
Recall (1)
F1 (T)
25.73
2.16
1.27
0.36
0.96
0.72
0.71
0.52
0.53
0.51
Dev.
25.12
0.72
0.50
25.52
1.26
0.56
0.72
0.71
0.53
0.50
26.07
2:22
1.28
0.85
0.86
0.72
0.71
0.52
0.53
0.50
I
20.58
2,09
1.32
0.84
0.85
0.69
0.72
0.52
0.54
0.51
Enhanced
22.98
22.67
2.19
1,36
0.84
0.85
0.69
0.69
0.72
0.52
0.54
0.52
22.63
2,18
1,36
0.84
0.85
0.69
0.72
0.52
0.54
0.52
44.
2.57
1,29
0.82
0.84
0.70
0.72
0.16
0,56
0.55
Comppted
41.02
2.26
1,23
0.83
0.84
0.70
0.73
0.56
0.56
0.55
41.10
2:26
1,23
0.83
0.84
0.70
0.73
0.56
0.56
0.55
4,99
1.13
0.58
0.91
0.91
0.78
0.79
0.67
0.66
0.65
Synthesized
4.68
1.11
0.58
0.91
0.91
0.78
0.79
0.67
0.67
0.65
4.85
0.64
4.95
0.58
0.91
0.91
0.78
0.79
0.67
0.66
0.64
is worth noting that the classification metrics remain relatively stable across token sizes for both models, with ARECHO consistently achieving higher accuracy, precision, recall, and F1 scores. This indicates that while regression performance may be more sensitive to token size variations, the models' discriminative capabilities are more robust such changes.
I.2 The Effect of Beam Search
In our main experiments, we mainly use greedy search to conduct the two-step decoding. However, given the search space in the factorization space, we can also conduct the beam search on the problem. We use the ARECHO model trained on Base set as the candidate test different beam sizes across various domains.
Our experimental results, as shown in Table 12 illustrate several important findings regarding the impact of beam search on model performance. First, we observe that increasing the beam size does consistently lead to performance improvements across all domains and metries. For instance, in the development set, beam size 2 yields the lowest MSE (25. 12) and RMSE (2.14), while beam size achieves the lowest MAE (1.26). However, these improvements over greedy search (beam size decoding approach is already effective at
relatively marginal, suggesting that the confidence-oriente identifying high-quality factorizations even with a simple greedy search strategy.
Interestingly, the Enhanced domain shows a contrasting pattern, where greedy search (beam size 1) outperforms larger beam sizes regression metrics such MSE (20.58 22.67+ for larger beams). This counterintuitive result may stem from the nature of our confidence estimation mechanism, which might be optimized for the scoring function used in greedy search rather than the more complex search patterns in beam search. The classification metrics, however, remain consistent across beam sizes, indicating that the model's 's discriminative capabilities are robust variations in the search strategy.
For the Corrupted domain, observe that beam sizes 2-4 yield notable improvements over greedy search in regression metrics, with approximately 7% reduction MSE (from 44.22 to around 41). This suggests that when dealing with noisy corrupted inputs, the expanded search space provided by beam search allows the model to identify more reliable factorizations, potentially avoiding local optima that might trap the greedy approach.
The Synthesized domain, which represents our most controlled experimental setting, shows the best overall performance across all metrics. Here. beam size 2 achieves the optimal results with the lowest MSE (4.68) and RMSE (1.11), while beam size 3 yields the lowest MAE (0.57). The high correlation coefficients (LCC=0.91) across all beam sizes indicate that the model effectively captures the underlying relationships in this domain, regardless of the search strategy employed.
These findings have several implications for the deployment of ARECHO in practical applications. First, the choice of beam size should context-dependent, with smaller beam sizes (1-2) preferable for standard and enhanced domains to balance computational efficiency and performance. For corrupted inputs, larger beam sizes may provide more robust results, albeit with diminishing returns beyond beam size 3. Second, the relatively consistent performance across beam sizes suggests that our
34

---

Table 13: Teacher-foreed decoding analysis. We compare the original baseline, autoregressive ARECHO, and ceacher-forced decoding (which injects ground-truth values at each decoding step to suppress error propagation). We use (T.F.) to note model use teacher- teacher-foreing decoding.
Regression Metries
Classification Metries
Domain
Model
MSE (4)
RMSE ())
MAE (4)
LCC
SRCC (1)
KTAU (f)
Acc
(t)
ARECHO
40.95
2.16
1.27
0.86
0.86
0.72
0.71
0.51
Dev.
ARRCHO (T.F.
25.08
1.80
0.94
0.88
0.89
0.77
0.71
0.50
Enhanced
ARRCRO
20.58
2.09
1.32
0.84
0.85
0.69
0.72
0.51
ARECHO (T.F.)
3.50
1.11
0.43
0.88
0.91
0.76
0.72
0.51
Corpuptd
ARRCHO
44.22
2.37
1.29
0.82
0.84
0.70
0.72
0.55
ARECHO (T.F.)
8.58
1.22
0.56
0.85
0.70
0.72
ARECHO
4.99
1.13
0.58
0.91
0.78
0.72
0.55
Synthesized
ARRCHO (T.F.)
2,00
0.80
0.30
0.92
0.93
0.79
0.79
0.64
Table 14: Comparison on MOS-related metrics across four public datasets. ARECHO shows consistent improvements in both regression and rank correlation measures over Uni VERSA-T,
Metric
Regression Metrics
Correlation Metrics
MSE (1)
RMSE ())
MAE (L)
LCC
SRCC (1)
KTAU (1)
UniVERSA-T
0.83
NISQA-MOS
ARECHO
0.30
0.55
0.37
0.89
0.88
0,72
VERSA-T
0.06
UTMOS
ARBCHO
0.05
0.22
0.16
0.97
0.97
0.87
PLCMOS
Ubi VERVERSA-T
0.37
0.86
0.09
ARECHO
0.32
0.56
0.38
0.89
0.89
0.72
URGENT-MOS
Uni VERVERRA-T
0.20
0.45
0.35
0.78
0.78
0.62
AROCHO
0,26
0.51
0.40
0.74
0.75
0.61
confidence-oriented decoding approach effectively identifies high-quality factorizations regardless of the exact search strategy, speaking to the robustness of our proposed methodology.
is worth noting that while beam search expands the exploration of the factorization space, comes with increased computational costs. The marginal improvements observed in most domains may not justify the additional computational burden resource-constrained environments. Nevertheless, in critical applications where even small improvements in accuracy are valuable, selective application of beam search may warranted, particularly when dealing with corrupted or noisy inputs.
1.3 Error Propagation Analyais/ Hybrid Decoding
To better understand the effect error propagation autoregressive decoding, we investigate the decoding with macher-forcing where ground-truth metric values are injected at each step during inference. This removes the accumulation of decoding errors and provides an upper bound on model performance under ideal conditioning.
As shown in Table 13 teacher-forced decoding consistently improves over autoregressive ARECHO, yielding a 50-50% reduction in prediction error. This quantifies the impact of error propagation in sequential metric estimation. The effect most pronounced the corrupted and synthesized conditions, where domain shifts amplify early mistakes.
These findings highlight practical trade-off: while teacher-forcing suppresses propagation and approximates an upper bound, it relies on ground-truth access and thus forfeits the fully reference-free nature of ARECHO.
I.4 Analysis MOS Prediction
To further analyze per-metric performance, we conduct a detailed comparison to investigate how inter-metric dependencies affect the modeling of MOS scores. As shown below, ARECHO consistently outperforms Uni VERSA-T across multiple datasets, highlighting the benefit of inter-metric prediction and autoregressive dependency modeling.
35

---

Table 15: DNSMOS ablation on ARECHO. "Full" denotes multi--mttre training, while "Only" refers to single-metric prediction.
Regression Metries
Correlation Metries
Metric
Model
MSE (1)
RMSE (J)
MAE (4)
LCC (t)
SRCC (t)
KTAU (f)
DNSMOS
ARECHO-Full only
0.04
0.21
0.16
0.85
0.81
0.65
0.20
0.15
0.86
0.83
0.66
Overall, ARECHO demonstrates notable improvements in MSE and MAE, particularly for NISQA-MOS, UTMOS, and PLCMOS. The correlation gains (LCC/SRCC/KTAU) further suggest that ARECHO effectively captures cross-metric dependencies that are beneficial for reliable MOS prediction.
Interestingly, shown in Table 15| the difference between ARECHO-full (multi-metric prediction) and ARECHO Malc (single-metric prediction) is marginal for DNSMOS, This suggests that the benefit of inter-metrie conditioning varies by dataset complexity and metric interdependence: tasks with stronger perceptual correlation (e.g., MOS-related metrics) benefit more from autoregressive chaining, whereas narrowly defined metrics like DNSMOS be learned independently.
J Task-Oriented Metric Subset Analysis
To further examine whether training with all metrics is always optimal and whether unrelated metrics might negatively affect task-relevant ones, conduct targeted experiments comparing subset and fullset training configurations across two representative domains: speech synthesis and speech enhancement. The results show that ARECHO remains robust under both settings, with diverse metric supervision generally improving or maintaining performance.
The dynamic classifier chain design enables each metric to be predicted at random position within the autoregressive chain, conditioned on a variable subset of preceding metries. This stochastic ordering allows ARECHO to capture informative inter-metric dependencies while factorizing away irrelevant information, preventing negative transfer even under mixed supervision.
All experiments (UniVERSA baseline, Subset, and Fullset) are trained on the same Base dataset described Sec. and detailed in Appendix E. The only difference lies in the set of target metries:
Speech Synthesis Subset: UTMOS, FO correlation, WER, Language, etc.
Speech Enhancement Subset: DNSMOS, PESQ, RIR Room Size, etc.
Fullset: Union of all metrics from both domains.
No additional data introduced for Fullset training: any performance differences therefore reflect the effect of training objectives rather than data volume.
Not all utterances contain all metries in the base dataset due data-source variability. For example:
Only 13.22% of samples contain NISQA Real MOS (human-annotatee
Nearly all have UTMOS and Language (non-intrusive model-based).
For enhancement data, 26.779% include intrusive metrics (PESQ/STOI//DR)) 80.44% have SRMR, and all have DNSMOS and Qwen-Recording Quality.
ARECHO naturally handles partial labels via its dynamic chain formulation, learning from whatever subset of metrics is available per sample. In contrast, Uni VERSA relies on masking- -based multi-task training. which reduces both data efficiency and inter-metric contextualization.
The results shown in Table 16 19 confirm three main trends:
Fullset training outperforms Subset training for most metrics (NISQA MOS, SRMR, SDR) and performs par for metrics such DNSMOS and STOI.
No negative transfer observed: unrelated metrics not degrade the performance of task-relevant ones.
36

---

Table 16: Subset vs. fullset training comparison for speech synthesis metrics (Synthesized test set is used for evaluation).
Metric
MSE (.)
RMSE
MAR
UniVERSA
1.22
NISQA Real MOS
Subset
0.43
0.65
0,41
UnIVERSA
0.25
0.50
0.39
UTMOS
Subset
0.09
0.30
0.22
Table 17: Subset vs. fullset training comparison for speech synthesis classification metrics (Synthe- sized test set is used for evaluation).
Metric
Setup
Ace. (f)
F1 (7)
UnIVERSA
0.90
0.88
Language
Fulleet
0.98
Table 18: Subset fullset training comparison for speech enhancement metrics (Enhanced test is used for evaluation).
Metric
Setup
(.)
())
(U)
UniVEASA
73.73
8.59
7.86
SRMR
Subset
4.02
2.01
1.26
Fullset
1.83
1.35
0.96
UniVERSA
5.50
2.34
2.29
DNSMOS
Subset
0.05
0.22
0.16
Fullset
0.05
0,22
0.16
UniVERSA
0.16
0.41
0.39
STOI
Subset
0.04
0.06
0.03
Fullset
0.00
0.05
0.03
UniVERSA
151.75
12.32
10.70
SDR
Subset
69.43
8.33
4.02
Fullset
19.83
4,45
2.93
UniVERSA
0.46
0.68
0.53
PESQ
Subset
0.19
Fullset
0.17
0.41
0.31
Table 19: Subset fullset training comparison for speech enhancement classification metrics (Enhanced test set used for evaluation).
Metric
Setup
Ace. (t)
(1)
Un1
0.97
Qwen-Recording Quality
Subset
0.97
0.95
Fullset
0.97
0.95
Dynamic classifier chain improves stability: randomizing dependency order during training allows decoupling of irrelevant signals while leveraging cross-metric correlations when beneficial.
Performance gains differ based on metric availability and difficulty:
Label Availability and Dependency Opportunity. Metries with limited supervision but strong inter-metric correlation (e.g., NISQA MOS) benefit the most from joint training. Densely available metrics (UTMOS, DNSMOS) show smaller improvements.
Metric Type and Modeling Difficulty. Subjective and high- vigiincr metrics gain more from multi-metric context, while objective or low-variance ones are largely self-sufficient.
Examples. NISQA MOS exhibits large improvements due to its sparse labels and correlation with perceptual quality scores. SDR benefits from complementary cues from PESQ, DNSMOS, and STOI, while UTMOS and DNSMOS remain stable due to high label coverage.
37

---

For further analysis of task-specific metric sensitivity, metrics can be grouped into:
Easy-dependent metrics:
high correlation and availability (e.g.. UTMOS, DNSMOS) that perform well without additional context.
Hard-dependent metrics: sparse or subjective metrics (e.g., NISQA MOS, STOI) that benefit strongly from auxiliary conditioning.
Overall, ARECHO demonstrates consistent performance across both homogeneous and heterogeneous metric sets. Joint training with diverse metric objectives does not harm, and often enhances, task- specific predictions. This stability stems from the dynamic classifier chain's 's ability to balance inter- metric dependency and independence, enabling scalable and interpretable multi-domain evaluation.
K Dependency Analysis- Dynamic Dependency Analysis
In this appendix, we show the complete order sequence of different metrics in Figure 1 Based the detailed orders, analysis of the autoregressive prediction sequences reveals several important patterns:
2 and 3
1. Foundational-oo-Dervvee Metric Flow: Across all speech types, we observe a consistent pattern where foundational characteristics (speaker, environment) predict derived measures (quality, intelligibility). This suggests the model has learned that basic physical properties constrain possible values of perceptual qualities, reflecting an implicit understanding of the causal structure in speech quality assessment.
2. Context-Specifie Dependency Anchors: Each speech type exhibits distinct "anchor metries" that appear early the sequence:
Enhanced speech anchors on speaker identity (Q-Gender)
Corrupted speech anchors on environmental acoustics (RIR Room Size)
Synthesized speech anchors on background conditions (Q-Background)
This reveals that the primary determinants of quality differ fundamentally based on the speech processing context.
3. Categorical- Bategorical-Befor--Niiie Bww.ore-ooooerriiiee Pattern: The consistent positioning of categorical metrics before numerical metrics suggests that discrete classifications provide efficient information compression that enables more accurate prediction of continuous measures. This aligns with efficient coding principles where high-level abstractions enable more precise low-level predictions.
4. Signal Processing Metries as Terminal Nodes:
Technical metries (MCD, SI-SNR, PESQ) consistently appear in later positions, indicating they represent terminal nodes in the de- pendency graph that are influenced by multiple upstream factors rather than serving as predictive foundations.
5. Non- Uniform Metric Inter-dependencies: The varied positioning of some metrics across speech types (e.g., Q-Gender position 0 in enhanced speech but position 12 synthesized speech) suggests that inter-dependencies are not fixed but are highly context-dependent, challenging universal models of speech quality assessment.
These findings suggest that the model has learned a contextually adaptive compression of the speech quality space, where prediction sequences are optimized minimize uncertainty hierarchical fashion. The emergent structures appear to reflect not just statistical correlations but meaningful organization of speech quality dimensions that aligns with human perceptual hierarchies.
L Dependency Analysis Static Dependency Analysis
To investigate the effectiveness of dependency, we conduct static dependency analysis in the inference and fine-tuning. Here, we mainly focus on two static order discussions, including (1) an order of matching-required metrics comes first (order-mr) and (2) coarse-to-fine conceptual order (order-c2f), Firstly, for the matching-require order, we follow the definition in VERSA Shh et al. (2025b), and create static ordinary in both matching-require and non-matching-requice Secondly, further
38

---

① Gouda
O Apedllapaie
Q Spuilagftyle
0 Rarually
@ RilRap
Redlagapi
SANHI
5po45
MEGAANK
Q Quacke
M.P
AX-I
Ub
QcClaty
S5-43 Al
O
Raui
smheng
N500-3006
A9gA
INCMUS
AM-V
CB
Migg006
C1-0X
STY
$8.5A4 SAB
S0 SAR
WXD
D HRAT
PoWe
RAVVR
1P6 SMM
13 HIRE
Relinacc Lantaggh
Newoaaa
MASRDSN
KTM
Shil Simulittie
NSSQN
Venchitts Hoal
MDS
Figure 1: Visualization of metric order for enhanced speech test set via color blocks (Red = Early, Blue Late).
HB 3o.3 Soz
Apesláequie
@ Popee
SaCrepieriy
@ CaundRgiioe
Gedder
O Redproall
O Spe Sprkeertwa
OcCheedfipe
Saalake
G.PAkking
Qpee kngglee
U-hkk
QutarQuatyy
AA-CC
B-Dusacz
Langagz
MBDA-M0
Lengh
PEC3005
Rest
NOMAD
SPX SM
aWEISRR
AV-N
AV
140
Se/MDR
RAM
Spout
NDGA-LOCD
SE 508
D-0ITT
DNSMRGEPa5
SC00RQ Ref
S90A
SESAR
S2-515NB
52-41-50
Nogg.08
NSS XXS
DELL
ST0E
CSULOW
MDD
P550
CSE MBb
S15NR
URGENT M0S
WER
CBR
NSRAA M6S5
NaunMOS Reil
Figure 2: Visualization of metric order for corrupted speech test set via color blocks (Red = Early, Blue Late).
NBRGN ZYE
@uuppe
AA-Q
@ Feeelllalle alosi
Warefore
SRGA
Godder
MSSA MiS
AACU
52-5145
SugttS
N504-M0
AA-P:
UTMU8s2
Radl agaage
3E-DK
5K3H
UN5M002235
SB-C1-50X
IL:305
AACt
Spork
NBQMMev MDS
D-Dutacc
SPORRO
Rakenac Baula
MDD
STCE
WiS
(
4188M
R190
Onl
Cik
NCM
CSI--ECH
UIGEENT MIS
CSBAUW
LEA
visgol
USB-DD
Cha
330
Figure 3: Visualization of metric order for synthesized speech test set via color blocks (Red Early, Blue Late).
explore the effectiveness of metric granularity, we then design a coarse-to-fine order, from the general audio quality and perceptual metrics, distortion and noise quality metries, speech enhancement quality metrics, acoustic and prosodic characteristics and then down to speaker information, more complicated speech content, and final parts emotion, and environmental contextualization Both order-mr and order-c2f are detailed below:
39

---

order-mr Metries
SE-SDR, SE-SAR, SE-SI-SNR, SE-CI-SDR, SDR, SAR, SI-SNR, CI-SDR, VISQOL, PESQ, STOI, FWSEGSNR, LLR, wSS, CD, Csig, Cbak, Covl, CSII-HIGH, CSII-MID, CSII-LOW, NCM, MCD, FORMSE, FOCorr, SPK-SIM, D-BERT, D-BLEU, D-Distance,
SCOREQ Ref ASR-Mismatch, WER, CER, Reference Text Length, Predicted
Text Length, SRMR, LID, RealLanguage, Q-Lang, NISQA-MOS, NISQA-NOI, NISQA-DIS MISQA-COL, NISQA-LOUD, SSQA, DNSMOSP. 835, DNSMOSP 808, SCOREQ, PAM, SWR/SCR, AA-CE, AA-CU, AA-PC, AA-PQ, SpoofS, Noreaqa, NOMAD, q q-SpeakerCount, R-Gender, Q-Age, D-SpeechImpariment, Q-Pitch, q-PitchRange, Q-VoiceType, q-VolumeLevel, g-ContentRegister, Q Q-VocComplexity, Q-Purpose, Q-Emotion, Q-Clarity, q-SpeechRate, Q-Speakingttyle, D-Bmovocalization, Q-Background, Q-EnvQuality, q-ChannelType, SNR Simulation, RIR Room Size, RT60, EMO-SIM, NISQA Real MOS, UTMOS, UTMOSv2, PLCMOS, SingMOS, URGENT MOS, VoiceMOS Real MOS
order-c2f Metrics
MISQA-MOS, NISQAReal MOS, VoiceMOSReal MOS, UTMOS, UTMOSv2, PLCMOS, SingMOS, URGENT MOS, SSQA, SCOREQ, SCOREQ Ref., NISQA-NOI, NISQA-DIS, NISQA-COL, NISQA-LOUD, DNSMOSP 835, DNSMOSP 808, Simulation, RIR Room Size, Noresqa, NOMad, Spoots, SE-SDR, SE-SAR, PESQ, STOI, SE-SI-SNR, SE-CI-SDR, SDR, SAR, SI-SNR, CI-SDR, FWSEGSNR, LLR, WSS, CD, Csig. Cbak, Covl, CSII-HIGH, CSII-MID, CSII-LOW, NCM, SWR/SCR, D-BERT, D-ELEU, D-Distance, MCD, FORMSE, FOCorr, PAM, RT60, Q-Clarity, Q-SpeechRate, VISQOL, SPK-SIM, q-SpeakerCount, Q-Gender, Q-Age, Q-Pitch, Q-PitchRange, o-VoiceType, Q-VolumeLevel, LID, RealLanguage, Q-Lang, Reference Text Length, Predicted Text Length, ASR-Mismatch, WER, CER, Q- Q-VocComplexity, Q- R.ContentRegister, Q-Purpose, EMO-SIM, Q-Emotion, q-SpeakingStyle, D-SpeechImparimmntt, A.Emotionallocalizaatii AA-CE, AA-CU, AA-PC, AA-PQ, q-Background, q-EnvQuality, q-ChannelType, SAMR
The results, shown in Table 20 reveal that static ordering can outperform dynamic beam search when the beam size is limited to I. In the inference stage, order-c2f achieves notably better regression performance in the Enhanced and Synthesized domains. This suggests that gradually increasing the complexity of predicted metrics helps the model build robust internal representations, especially for perceptually grounded or synthetic speech signals. contrast, order-mr yields improved regression metrics in the Corrupted domain and stronger classification metrics across most domains. This indicates that prioritizing aatching-requirer metrics can help the model more effectively attend to severe distortions, particularly in noisy or degraded speech, which benefits tasks involving subjective perceptual scores (e.g., MOS).
In the fine-tuning stage, we initialize from the best-performing random-order sampled model and continue training using the static orders. However, we observe that performance tends to degrade compared to the pre-trained baseline. We attribute this to the reduced exploration capability of model under a fixed decoding order, which may limit flexibility to adapt across domains. Additionally, the divergent behavior across domains suggests that the optimal decoding order may be comain-specifee further highlighting the potential limitations of a static ordering scheme. In such cases, dynamic decoding strategies like beam search may offer better adaptability and robustness by allowing the model to flexibly determine the optimal metric prediction sequence based on context.
M Efficiency Analysis
To complement the performance evaluation of different models, provide analysis of their training efficiency in terms of average time per training epoch. Figure compares the epsch-wise training time of UniVERSA, UniVERSA-T, and the proposed ARECHO model.
As shown the figure, ARECHO achieves a notable reduction in training time, averaging 6632.09 seconds per epoch, compared to Uni UniVERSA (7668.49 s) and Uni (7923.34 s). This improve- ment reflects the design efficiency the ARECHO architecture, which maintains strong performance while accelerating the training process.
40

---

Table 20: Static dependency results on ARECHO.
Regresion Metties
Metrics
(1)
(1)
MAE(())
LCC((1)
(t)
(1)
(?)
andase2r
0.70
Corupted
anda-e2r
1.26
0.70
0.55
Symthesized
andes-c2r
0.78
Lahanced
onder c2r
order-mr
25.40
22.05
2.25
2.41
1.44
1.41
0.82
0.83
0.84
0.84
0.67
0.67
0.72
0.72
0.54
0.53
0.56
0.55
0.53
0.53
Fine-tune
Corupted
order-me
unle--c2f
$2.72
39.79
2:30
2.49
1.29
1.36
0.82
0.82
0.84
0.84
0.70
0.70
0.72
0.73
0.56
0.55
0.56
0,57
0.54
0.55
Synthesizal
order-aas
482
1.13
0.60
0.90
0.90
0.76
0.78
0.65
0.65
063
unfer-22f
5.36
1.19
0.65
0.90
0.90
0.76
0.77
0.63
0.62
0.40
Avg. Training Time epocch)
8,000
7,668.49 49
7,923.34
6,632.09
6,000
4,000
2,000
0
UniVERSA
UniVERSA-T
ARECHO
Figure 4: Average training time per epoch for Uni Uni UniVERSA-T and ARECHO the Base training set.. The proposed ARECHO model demonstrates improved training efficiency.
Such efficiency is particularly advantageous for large-scale training scenarios requiring frequent model updates. The reduction computational cost also supports the scalability of ARECHO in practical deployment.
N Limitations
While ARECHO demonstrates strong performance and versatility across multiple speech evaluation tasks, several limitations remain:
Tokenization Granularity. The conversion of continuous metrics into discrete tokens introduces a trade-off between resolution and model complexity. Although configurable, coarse quantization may lose subtle perceptual differences, while fine- fine-grained tokenization increases sequence length and decoding burden.
Autoregressive Inference Overhead. The sequential nature of the dynamic classifier chain, while beneficial for modeling inter-metric dependencies, results higher inference latency compared to parallel prediction frameworks, potentially limiting real-time applicability.
Metric Order Sensitivity. Despite randomized ordering during training, the model may still exhibit sensitivity decoding order during inference, especially under domain shift or when limited context is available in early steps.
Partial Label Generalization. Although ARECHO supports partially labeled supervision, its effectiveness under extreme label sparsity or comain-mismatched metric availability has not been extensively evaluated.
Dependence on Predefined Metadata. The current system relies on manually defined metadata tokens for each metrie. Scaling hundreds of fine-gra one-grained evaluation metrics may require automated schema learning ontology- modeling.
41

---

While ARECHO reduces reliance on expensive estimators, it does not replace human listening in high- stakes settings. Automated scores can be misapplied if used out of context (e.g.. clinical decisions). We therefore recommend numan-in-the-loow
verification, transparent reporting of uncertainty, and restricted deployment. Finally, several targets are ordinal; integrating ordinal-aware objectives (e.g.. Cao et al. 2020); Gutiérrez et al. (2015): Baccianella et al. 22099) remains promising future work.
We leave addressing these limitations to future work, partieularly exploring hybrid decoding strategies, more efficient dependency modeling, and extension to open-vocabulary or structured metric spaces.
O Broder Impact
Positive Impacts. ARECHO has the potential to improve the transparency, accessibility, and scal- ability of speech evaluation, particularly in applications involving speech synthesis, enhancement, and emotional communication. By providing a unified and interpretable multi-metric evaluation framework, this work facilitate more equitable benchmarking of speech technologies across languages, devices, and acoustic conditions. Moreover, ARECHO's support for partial supervision and reference-free evaluation makes especially valuable in low-resource or real-world deployment settings, where traditional evaluation pipelines may be infeasible. This could help advance assis- tive technologies. conversational AI, and accessibility tools for underrepresented communities and individuals with speech impairments.
Negative Impacts. At the same time, automated speech evaluation systems carry certain risks. First, deployed naively or trained on biased datasets, they may reflect or amplify social, demographic, or linguistic biases, leading unfair assessments of speech quality, intelligibility, across speaker groups. Second, while ARECHO supports partial supervision and reference- evaluation, improper use in sensitive domains (e-g., hiring, education, or healthcare) could result in over-reliance on automated metrics without adequate human oversight. Lastly, as the system models multi-metric dependencies, interpretability claims must be contextualized carefully avoid misleading conclusions about causality or human perception.
expressiveness
We encourage responsible use of this framework, particularly in human-facine applications, and emphasize the importance of representative training data, transparency in metric selection, and human-in-the- validation to ensure fairness and reliability.
P ARECHO's Logo Design
1|
ARECHO
Figure 5: The ARECHO logo: a visual representation of chain-based, autoregressive speech evalua- tion.
The ARECHO logo was designed to reflect the system's core principles, autoregessive dependency modeling, multi-metric speech evaluation, and structured reasoning. consists two key elements:
Waveform Bars: A stylized set vertical lines representing an audio waveform, symboliz- ing the raw speech signal input.
Interlocked Chain Link: Depicts ARECHO's ' classifier chain architecture. It highlights the autoregressive inference procedure and the inter-metric dependencies leveraged during prediction.
These elements are enclosed in a minimal circular boundary to suggest cohesive and holistic system. Alongside, the bold, uppercase logotype "ARECHO" uses a geometric sans-serif font to convey precision, clarity, and technical strength.
42

---

Two visual variants of the logo were created:
A light-on-dark version for slides and visual presentations.
A symbol-only version for compact branding (e.g., repository icons badges).
The design encapsulates ARECHO's mission: to unify diverse speech evaluation metrics under
structured, dependency-awwan modeling framework.
Acknowledgment
This work is supported by the Defence Science and Technology Agency (DSTA) in Singapore. We would like to thank Daniel Leong and Megan Choo for their valuable comments. Experiments of this work used the Bridges2 at PSC and Delta/DeltaAl NCSA computing systems through allocation CIS210014 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, supported by National Science Foundation grants 2138259, 2138286, 2138307, 2137603, and 2138296.
43

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper generally follows the guidelines.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We present the limitations of the paper section on limitations in the appendix (Appendix N
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All experiments are conducted on open-source frameworks, together with a careful guideline for data preparation, model configuration, and any specific parameters to be used. Experimental details are discussed in Appendix G
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The code for conducting experiments included the supplementary materi- als, where the users can conduct the experiments with well-written shell interface. For each dataset used, in the Appendix, we also report their licenses when possible. For URGENT2024 dataset, since it is derived from existing datasets through simulation, the license of the datasets that were used inherited. Please refer to Appendix for details.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All experiments carefully discussed in related sections in the main content and appendices. Please refer to Appendix G for details.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The metrics are calculated with statistical significance. Due to the overall large-scale test set, most results appeared be statistically significant, so we do not expand the discussion in the main content but put the discussion in the appendices only.
44

---

8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computing information discussed together with the experimental setup
in the Appendix G
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:///rw.ppp.ccpcpee//teeeiiae
Answer: [Yes]
Justification: The paper is carefully checked to follow the NeurIPS code of ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include section in the appendix specifically related to this.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not think the paper will pose such risks, would like to investigate more upon discussion if certain risks are raised by the reviewers or the NeurlPS committee.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have carefully checked include all citations for the assets used and include their license in the corresponding sections.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We have included the assets in the supplementary materials, and the corre- sponding information is discussed in the readme file within the supplementary materials.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects. 15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
45

---

Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
16. Declaration LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: The core method development this research does not involve LLMs as any important, original, or non-standard components.
46


Reviews:
[{"review_id": "y4thKyFPZY", "reviewer": "Reviewer_DC5A", "summary": "ARECHO is a framework designed for multi-metric speech evaluation, aiming to predict various speech quality and profiling metrics (e.g., PESQ, STOI, MOS) jointly rather than independently. The main contributions of ARECHO are (1) converting all metric values (numerical/categorical, with different scales) into a unified token space, enabling sequence modeling; (2) using an autoregressive model to capture dependencies between metrics, allowing the model to predict metrics in a flexible, context-aware order rather than in parallel; (3) enhancing prediction robustness by refining metric prediction based on confidence in the output, addressing instability in dynamic prediction order.", "questions": "1. How sensitive is ARECHO’s performance to the choice of quantization (e.g., bin size, uniform vs. adaptive) for different metrics? Is there a systematic way to choose tokenization strategies?\n2. As the mentioned error accumulation issue, how does the model mitigate the risk of early prediction errors cascading through the chain?\n3. How much does the two-step confidence-oriented decoding contribute relative to a standard greedy or beam search? Any quantitative ablation study will be better.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n1. The approach to encode all types/scales of metrics into tokens enables joint modeling and paves the way for autoregressive multi-task prediction. Unlike prior methods that predict each metric independently, ARECHO explicitly models the relationships between metrics, leading to improved accuracy and interpretability.\n2. The two-step decoding mechanism allows the model to dynamically adjust the prediction order and recover from uncertain predictions, leading to higher reliability.\n\n## Weaknesses\n1. Converting numerical metrics to discrete tokens (quantization) may result in a loss of precision for fine-grained regression tasks. Is there any way to measure this loss?\n2. The error accumulation is a potential issue. The autoregressive, chain-based prediction means errors in early metrics could affect downstream metric predictions.\n3. Another concern is the complexity in training and inference, as the dynamic chain and two-step decoding add computational and implementation complexity compared to simple parallel predictors.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer DC5A for the detailed and constructive feedback. Your comments helped us strengthen the clarity, technical analysis, and experimental validation of our work. Below, we address each point and outline the corresponding revisions.\n\n\n## W1 and Q1: Measure Loss of the Quantization\n\nWe thank the reviewer for raising this important concern. Since our tokenization bins are learned from the training set, we evaluated their generalization to test data via a reconstruction analysis. Specifically, we measured how well the quantized–dequantized values match the original ground-truth metrics on the test set.\n\n| Domain            | Tokenization Type | Bins | RMSE  | MAE   |\n|-------------------|--------------------|------|-------|-------|\n| Synthesized speech| Percentile         | 500  | 0.056 | 0.014 |\n| Synthesized speech| Uniform            | 500  | 0.100 | 0.025 |\n| Synthesized speech| Percentile         | 1000 | **0.034** | **0.007** |\n| Synthesized speech| Uniform            | 1000 | 0.097 | 0.020 |\n| Enhanced speech   | Percentile         | 500  | 0.139 | 0.023 |\n| Enhanced speech   | Uniform            | 500  | 0.223 | 0.045 |\n| Enhanced speech   | Percentile         | 1000 | **0.103** | **0.013** |\n| Enhanced speech   | Uniform            | 1000 | 0.220 | 0.039 |\n| Corrupted speech  | Percentile         | 500  | 0.261 | 0.024 |\n| Corrupted speech  | Uniform            | 500  | 0.512 | 0.045 |\n| Corrupted speech  | Percentile         | 1000 | **0.165** | **0.012** |\n| Corrupted speech  | Uniform            | 1000 | 0.507 | 0.040 |\n\nThese results confirm that our quantization strategy yields low reconstruction error across domains. Notably, percentile-based tokenization significantly outperforms uniform binning, demonstrating that data-aware strategies preserve finer-grained information crucial for accurate metric prediction.\n\n**Revision**: We will expand **Appendix A** to include these findings.\n\nNoted that even though with 1000 bin size, the reconstruction performance is better. It cannot guarantee better performance in ARECHO as it also introduce more difficult prediction targets. Practically, we find 500 bin size is a good tradeoff of effectiveness between better reconstruction and easier prediction. We have discussed the results in Table 10.\n\n\n## W2 & Q2: Error Accumulation and Metric Dependency\n\nARECHO’s autoregressive design explicitly models inter-metric dependencies, which improves both prediction quality and interpretability (as demonstrated in Table 1 and Appendix J).\n\nTo alleviate concerns about error propagation, ARECHO supports **flexible inference**, users can query **only a subset of metrics**, bypassing previously predicted values. In such cases, ARECHO operates like an independent predictor **without chaining**.\n\nWe here examplified experiments comparing full-chain vs. single-metric (no chaining) inference for **DNSMOS** (on generation test set):\n\n**DNSMOS Performance Comparison:**\n\n| Model         | MSE  | RMSE | MAE  | LCC  | SRCC | KTAU |\n|---------------|------|------|------|------|------|------|\n| ARECHO-full   | 0.04 | 0.21 | 0.16 | 0.85 | 0.81 | 0.65 |\n| ARECHO-only   | 0.04 | 0.20 | 0.15 | 0.86 | 0.83 | 0.66 |\n\nThese results confirm that ARECHO is resilient to error accumulation while benefiting from dependency modeling when enabled.\n\n**Revision**: We will clarify this flexibility in the main text and highlight these results in **Appendix L**.\n\n\n## W3: Training and Inference Complexity\n\nARECHO is trained using random order sampling, which is comparable in cost to standard multi-task models as evidenced in Appendix L.\n\nAlthough inference is sequential, the process is highly efficient: ARECHO achieves **over 100× speedup** compared to computing ground-truth metrics using model-based estimators (e.g., Qwen-based metrics and models with pre-trained encoders like UTMOS, Noresqa, ScoreQ).\n\nWhile parallel predictors are simpler per step, they **scale poorly with large metric sets** in terms of memory and compute. In contrast, ARECHO provides a scalable, accurate, and efficient alternative.\n\n**Revision**: We will add a complexity analysis and comparison in **Appendix L**.\n\n\n\n## Q3: Confidence-Oriented vs. Greedy Decoding\n\nWe conducted an ablation comparing **two-step confidence-oriented decoding** versus **greedy decoding** on enhanced speech test set.\n\n| Decoding         | MSE  | RMSE | MAE  | \n|---------------|------|------|------|\n| Confidence   | **20.58** | **2.09** | **1.32** | \n| Greedy   | 22.31 | 2.31 | 1.40 | \n\nThis supports our claim that naive greedy decoding, while simple, is suboptimal in the presence of inter-metric dependencies and dataset frequency imbalance. The confidence-based approach provides more principled, performance-driven metric ordering.\n\n**Revision**: We will include this ablation in **Appendix K** and add a summary pointer in the main text.\n\n\nWe thank the reviewer again for the thoughtful suggestions, which have significantly improved both the clarity and technical depth of our work."}, {"review_id": "5H5747wMKv", "reviewer": "Reviewer_z2T7", "summary": "The paper proposes a classifier chain approach based on a WavLM backbone model to evaluate speech signals (multi-metric evaluation), following a multi-task approach that is able to model inter-metric dependencies. The classifier chain is implemented through a transformer-based autoregressive model. At inference time, the paper proposes a metric-prediction-order approach based on the model's confidence in predicting each metric. To deal with multiple metric scales, the paper proposes a tokenization-based approach. The experimental setup comprises speech signals from various domains and contexts, including up to 2138 hours of speech for training and 18.65 hours for testing, as well as a total of 87 different metrics (most of which are model-based, but also include classical signal processing-based and subjective metrics). The entire set of experiments encompasses analyses of various tokenization methods, token size, beam size during inference, and an examination of the order in which metrics are predicted across different speech domains.", "questions": "- Appendix D outlines the procedure for the two-step confidence-oriented decoding in detail. However, it does not specify how parameter B was selected, nor does it provide empirical results demonstrating the influence of B on the model’s performance.\n\n- A notable limitation of the results section is the absence of specific findings related to subjective metrics. While the performance on model-based metrics may suffice in many cases, the extent to which the proposed approach improves the prediction of subjective metrics remains unclear. The comparison between order-c2f and order-mr offers an interesting perspective, yet it fails to provide concrete evidence regarding the method’s effectiveness on subjectively annotated evaluations.\n\n- Given that many of the target metrics are themselves generated by models, it is worth questioning the added value of using the proposed approach over directly using the ground-truth models. Additionally, although the autoregressive strategy is valuable, it inherently introduces error propagation. In this context, for model-based metrics, would it not be more appropriate to inject the actual values from the ground-truth models into the autoregressive loop, instead of relying on ARECHO predictions?\n\n- Could the authors provide statistics on the distribution and balance of the target values and ordinal ranks within the training set? If the data is found to be imbalanced, I strongly recommend using a balanced Mean Absolute Error (MAE) or a similar metric, rather than relying on the standard MAE or classification-based metrics.", "limitations": "The manuscript does not raise any concern about the potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "The paper tackles a fundamental challenge in speech quality evaluation through an innovative and promising approach that lays the groundwork for future research in the field. The manuscript is well-structured and clearly presents the proposed method in most parts. The authors support their proposal with a comprehensive experimental evaluation, which enables a robust assessment of its performance and lends credibility to the reported results.\n\nThe core contribution—the prediction of quality metrics via sequential tokenization—represents a novel and original direction. This approach opens the possibility for more general prompt-based speech quality evaluation, potentially marking a significant advance for various established speech tasks such as audio generation, speaker diarization, and source separation. Moreover, it could benefit biomedical applications, where different dimensions of speech can serve as indicators of a patient's health status and where subjective evaluations are often costly and difficult to obtain. In this context, the use of inter-metric modeling via classifier chains is a promising strategy.\n\nHowever, a major concern lies in how the approach handles the prediction of quality metrics. All metrics are encoded as categorical variables, including those that are inherently continuous, and the models are trained using standard classification techniques. Yet, the problem at hand (for several metrics) is more accurately characterized as an ordinal regression task. This constitutes a fundamental limitation, as the commonly used categorical cross-entropy loss does not account for the ordinal nature of the target variables.\n\nA substantial body of literature has proposed surrogate loss functions that are better suited for ordinal regression, demonstrating improved performance over naïve classification-based approaches. Relevant references include:\n\nFathony, R., Bashiri, M. A., & Ziebart, B. (2017). Adversarial surrogate losses for ordinal regression. Advances in Neural Information Processing Systems, 30.\n\nNazabal, A., Olmos, P. M., Ghahramani, Z., & Valera, I. (2020). Handling incomplete heterogeneous data using VAEs. Pattern Recognition, 107, 107501.\n\nNachmani, I., Genossar, B., Scharf, C., Shraga, R., & Gal, A. (2025, April). SLACE: A Monotone and Balance-Sensitive Loss Function for Ordinal Regression. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 39, No. 18, pp. 19598–19606).\n\nGiven this, regression-based evaluation metrics would be more appropriate than classification-based ones. It is also essential to account for the distribution of quality levels across the dataset. In this regard, a balanced version of the Mean Absolute Error (MAE) is recommended, as discussed in:\n\nBaccianella, S., Esuli, A., & Sebastiani, F. (2009, November). Evaluation measures for ordinal regression. In 2009 Ninth International Conference on Intelligent Systems Design and Applications (pp. 283–287).\n\nNote that they have been previously used for voice quality evaluation using the GRBAS scale in biomedical contexts.", "was_revised": true, "final_justification": "I consider the paper to tackle a fundamental challenge in speech quality evaluation through an innovative and promising approach. During the rebuttal phase, the authors provided solid experimentation and paid more attention to metrics that reflected the ordinal nature of the variables they were predicting. They have also included experimentation that injects original ground truth values obtained from reference models into the inference phase, which helps understand the approach's limits and establish benchmarks for further comparison. I consider this a solid piece of work, and I will keep my score as it was initially.", "rebuttal": "We sincerely thank Reviewer z2T7 for their thoughtful and constructive feedback. Your comments helped us further clarify the motivation and scope of our work, and we greatly appreciate the opportunity to address them in detail. Below, we respond point by point and outline the revisions we will make to strengthen the manuscript.\n\n\n## W1: Ordinal Regression Loss\n\nOur core focus in this work is to develop a unified framework capable of modeling both regression and categorical metrics via a tokenized sequence prediction paradigm. This tokenization strategy enables consistent modeling of heterogeneous evaluation targets and facilitates scalable autoregressive inference.\n\nWe fully acknoweldge that many metrics, such as MOS and SNR, are ordinal in nature and that modeling them using ordinal-specific surrogate losses is likely to improve performance and fidelity. We appreciate the reviewer’s references to SLACE and adversarial ordinal loss as concrete and compelling alternatives. While our current formulation focuses on a unified classification-based modeling pipeline to maintain consistency across all metric types, we view integrating ordinal-aware objectives as a high-impact direction for enhancing metric-specific accuracy. We will explicitly incorporate this direction in our Future Work section, acknowledge the previous works, and plan follow-up experiments to validate this hybrid design.\n\n\n## W2 & Q4: Use of Macro-averaged MAE\n\nWe are grateful for the emphasis on macro-averaged MAE. This perspective aligns with our intention to **equitably reflect performance across frequent and rare classes**, particularly given the diversity of our evaluation metrics. We clarify that our current evaluation already follows the macro-averaged MAE formulation as discussed in [Gutiérrez et al., 2016], but we acknowledge the need to make this design choice more visible.\n\nWe will update both the main text and Appendix D to clearly state this design choice and its rationale.\n\n## Q1: Effect of Beam Size B\n\nWe thank the reviewer for raising this clarification. The impact of beam size B is empirically evaluated in **Ablation I.2**, where we demonstrate that a small beam size (B=2) achieves an effective trade-off between prediction accuracy and decoding efficiency.\n\nTo improve accessibility, we will revise **Appendix D** to include a direct pointer to this analysis and briefly summarize the key findings to guide readers.\n\n\n## Q2: Subjective Metric Evaluation\n\nWe agree with the reviewer that evaluating subjective metrics is essential for understanding the practical utility of the system. In response, we have expanded our experimental results to include detailed evaluations on subjective metrics such as **NISQARealMOS** and **URGENTMOS**.\n\nThese results demonstrate the **metric-specific nature** of performance. For **NISQARealMOS**, which captures perceptual quality, our proposed autoregressive model (**ARECHO base**) achieves substantial improvements across all metrics, **MAE: 0.12**, **RMSE: 0.23**, **SRCC: 0.98**, and **Kendall’s Tau: 0.92**, compared to the non-token UniVERSA base baseline (**MAE: 0.85**, **SRCC: 0.81**) and token baseline UniVERSA-T base (**MAE: 0.30**, **SRCC: 0.94**). This highlights strong alignment between the predicted and ground-truth perceptual scores.\n\nIn contrast, for **URGENT_MOS**, which reflects speech enhancement subjective evaluation, the improvements are less pronounced. While **UniVERSA-T** base performs best (**MAE: 0.33**, **SRCC: 0.92**), the ARECHO model shows moderate gains over the UniVERSA base baseline (**MAE: 0.41 vs. 0.58**, **SRCC: 0.71 vs. 0.67**), indicating room for further improvement on certain subjective metrics.\n\nWe will report both **ranking-based** (e.g., SRCC, Kendall’s Tau) and **error-based** (e.g., MAE, RMSE) metrics, and include the full results in **Appendix L**, with key summary findings highlighted in **Section 5**.\n\n\n## Q3: Use of Ground-Truth Values During Inference\n\nWe appreciate the reviewer’s thoughtful question regarding the role of ground-truth values in inference. ARECHO is specifically designed for efficient and generalizable inference, especially in settings where ground-truth computation is impractical, such as **real-time evaluation**, **black-box systems**, **reference-dependent metrics**, or **large-scale deployment**.\n\nOur experiments show that ARECHO achieves **over 100× speedup** compared to computing all metrics using their original estimators (e.g., VERSA). This efficiency is critical for applications requiring rapid and scalable evaluation.\n\nInjecting ground-truth at inference would limit applicability, but we recognize that hybrid decoding strategies may promote a useful trade-off between performance and realism. We will include this hybrid strategy as a promising future direction and add a brief feasibility discussion in Appendix L.\n\n\n## Limitations: Potential Societal Impacts\n\nWe thank the reviewer for encouraging reflection on broader implications. While automated multi-metric estimators can substantially accelerate system development and iteration, they should not be viewed as replacements for human listening tests, especially in high-stakes or perceptually nuanced applications. Final evaluation should always include human oversight to ensure alignment with subjective experience.\n\nWe acknowledge the potential societal risks that may arise if these automated predictions are misinterpreted without appropriate context or used in sensitive domains such as healthcare or biometric identification. To mitigate such risks, we advocate for: **transparent model design**, **human-in-the-loop evaluation**, and **restricted deployment protocols**. We will include this discussion in the main manuscript (\"limitation section\") to promote responsible development and deployment.\n\n## Final Remarks\n\nWe are deeply grateful for Reviewer z2T7’s valuable feedback. The issues you raised have significantly improved the clarity, robustness, and broader framing of our work. At the same time, we are also excited to see that the proposed ARECHO framework can serve as a promising anchor for multiple future directions, including integrating ordinal regression losses, exploring hybrid models that incorporate ground-truth supervision, and enabling practical uses such as reward modeling for speech generation and foundational evaluation in broader applications."}, {"review_id": "0xOdhsEeww", "reviewer": "Reviewer_XEXh", "summary": "This paper proposed a new autoregressive evaluation toolkit for speech signal, which uniforms the scales and dependendices of different metrics. The paper aimmed to improve the interpretability between metrics as well.", "questions": "1. In section 3.2, the authors applied quantization-based tokenization by partitioning the value range into uniformly or adaptively spaced bins. This is not clear how they do this, e.g. difference and scale between different metrics. Also, the quantization maybe helpful for training, but will the orignial regression task can also help improve the performance? How did these tasks be performed is not clear to me.\n2. In section 3.3, the authors use a random order the metrics but the inference is with arbitrary query orders. What's the influence of these orders? \n3. I think my most concern about this work is **whether there are such relationships between metrics in many speech tasks**. For instance, if we want to do the age estimation task, maybe gender and pitch is important, but others maybe less important. In this case, such method maybe too heavy and cannot improve the overall performance too much. I think the authors should also provide **examples about how to use ARECHO in certain tasks**, like speech enhancement, asr, tts.., and how the metric links help each other. Moreover, I think the current method is better for **understanding a long audio**, instead of such simple evaluations.\n4. The authors should analyze whether some of the metric estimation may hurt others. For example, there are many MOS metrics with different aspects. They should be independant but are considered dependant in this paper.", "limitations": "The main limitation is that the method is lack of examples to be used in certain tasks, and a more detailed analysis of relationships and influence between specific metrics.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. The quality of the paper is good. It points out the problems of existing metrc frameworks and studies a new solution.\n2. The originity of the paper is good. This is the first paper to study the relationship between speech signal metrics.\n3. From the Table 1, we can find that the proposed method significantly outperforms the previous baselines.\n\n# Weaknesses\nI have some concerns regarding the clarify and significance. If the authors could answer them well, I will consider raise my scores.\n1. The clarify is somehow not clear to me. See # Questions below.\n2. The significance is to be discussed. See # Questions below.", "was_revised": true, "final_justification": "After discussing with the author, I found it is interesting to investigate more experiments and the clarification needs to be improved. However, the overall contribution and novelty is good enough. I raised the score from 3 to 4.", "rebuttal": "We sincerely thank Reviewer XEXh for the thoughtful, constructive, and detailed feedback. Your comments have been instrumental in helping us clarify technical aspects, highlight the strengths of ARECHO, and identify areas for improvement. We deeply appreciate your time and effort in reviewing our work. Below, we carefully address each point raised and outline corresponding clarifications and revisions.\n\n## Q1: Clarification on Quantization Details\n\nWe appreciate your request to clarify our quantization strategy. In response, we will revise Appendix A to include concrete illustrations of our percentile-based tokenization and its impact on model behavior.\n\nRather than performing an exhaustive search per metric, we use a consistent percentile-based tokenization strategy with 500 bins for all continuous-valued metrics. This enables uniform treatment across diverse metrics and simplifies model design. In Table 10, we conducted an ablation study on bin size, showing that 500 bins present a good tradeoff between reconstruction quality and ARECHO prediction difficulty. \nMore importantly, we would like to reinforce why this design matters. ARECHO’s tokenization strategy unifies both regression and classification metrics under a single modeling framework, this is a **key architectural innovation**. Unlike UniVERSA, which relies on direct regression, our token-based method ensures consistent modeling across heterogeneous metrics, paving the way for robust scaling and modular extension.\n\nWhile ordinal regression is indeed a viable alternative, we deliberately opted for token-based quantization to support generalization, controllable sampling, and eventual integration with language-driven generation. We fully agree this is a rich area for future exploration, and will add a discussion on hybrid modeling as a natural next step in evolving ARECHO.\n\nWe will expand Appendix A with visualizations and deeper explanation of this strategy, and we will highlight this design motivation in Section 3.2.\n\n## Q2: Order Effect on Metric Prediction\n\nThank you for raising the important question regarding the ordering of metrics during inference. We appreciate the opportunity to clarify that ARECHO supports two inference modes:\n\n- In the fixed order mode, users manually define the sequence of metrics to be queried.\n- In the flexible mode, our proposed two-step confidence-oriented decoding enables the model to dynamically determine the most effective order.\n\nThis flexibility is not an afterthought, it is central to ARECHO’s ability to adapt to incomplete annotations and metric sparsity. Our results in Appendix J and K demonstrate that flexible decoding not only preserves performance but also improves interpretability by aligning prediction order with metric dependencies. This makes ARECHO uniquely positioned among multi-metric estimators to handle real-world uncertainty in deployment scenarios.\n\nWe will revise the text to more clearly present this distinction between training and inference order and explain the robustness benefits enabled by dynamic ordering.\n\n## Q3.1: Importance of Inter-metric Relationships\n\nWe strongly agree that inter-metric relationships are crucial, and ARECHO places them at the heart of its design. Unlike UniVERSA and TorchSquim, which treat metrics independently, ARECHO models their **semantic and statistical dependencies** directly through chain-based autoregression.\n\nThis leads to two significant benefits:\n- **Performance gains** across all major metrics and datasets (Table 1),\n- **Interpretability**, as demonstrated in our dependency analysis (Appendix J).\n\nWe emphasize that ARECHO is **not limited** by this coupling: when needed, it can revert to single-metric inference, operating efficiently and independently. This optionality makes ARECHO not only accurate but **controllable and practical** in real-world settings.\n\n## Q3.2: Practical Usage of ARECHO\n\nWe appreciate the reviewer’s interest in the real-world applicability of ARECHO. As discussed in Section 5, ARECHO is applicable to a wide range of scenarios, including:\n\n- Evaluation of corrupted speech signals\n- Text-to-speech (TTS) systems\n- Speech enhancement pipelines\n\nWe thank you for highlighting the opportunity to make this clearer in the paper. We will revise the section to better emphasize this versatility.\n\nMore importantly, ARECHO is **ready to be used** beyond evaluation. It can serve as:\n\n- A reward model for reinforcement learning setups\n- A component in multi-metric loss functions during training of other systems\n\nThese are exciting directions that we aim to explore in future work. We would like to provide some references where previous studies utilize different metrics to enhance their speech generation systems (e.g., TTS systems or enhancement systems)\n- Preference alignment improves language model-based TTS (Tian et al. 2025)\n- Aligning Generative Speech Enhancement with Human Preferences via Direct Preference Optimization (Li et al. 2025)\n- Multi-CMGAN+/+: Leveraging multi-objective speech quality metric prediction for speech enhancement (Close et al. 2024)\n- Improving Speech Enhancement with Multi-Metric Supervision from Learned Quality Assessment (Wang et al. 2025)\n\nLastly, we would like to highlight that ARECHO achieves over 100× speedup compared to computing ground-truth metrics. This efficiency is critical for real-time and large-scale deployment. To better reflect this advantage, we will revise the manuscript to include a detailed comparison between ARECHO and ground-truth metric computation, which is currently underexplored in the text.\n\n## Q3.3: Usage for Long Audio\n\nWe thank the reviewer for the thoughtful suggestion to consider long-form audio. We fully agree that temporal dependency modeling across extended audio contexts is an important next step. Accordingly, we will explicitly include this point as a direction for future work in the revised manuscript.\n\n## Q4: Analysis on MOS Data Evaluation\n\nWe sincerely appreciate the reviewer’s suggestion to conduct deeper analysis on MOS metrics. In response, we conducted a detailed evaluation to examine how inter-metric relationships impact the modeling of MOS-related metrics. As shown below, ARECHO generally outperforms UniVERSA-T across multiple datasets.\n\nMain metrics comparison (demonstrating benefits of inter-metric prediction):\n\n| Metric     | Model        | MSE  | RMSE | MAE  | LCC  | SRCC | KTAU |\n|------------|--------------|------|------|------|------|------|------|\n| NisqaMOS   | UniVERSA-T   | 0.41 | 0.64 | 0.43 | 0.86 | 0.83 | 0.69 |\n| NisqaMOS   | ARECHO       | 0.30 | 0.55 | 0.37 | 0.89 | 0.88 | 0.72 |\n| UTMOS      | UniVERSA-T   | 0.06 | 0.25 | 0.19 | 0.97 | 0.97 | 0.85 |\n| UTMOS      | ARECHO       | 0.05 | 0.22 | 0.16 | 0.97 | 0.97 | 0.87 |\n| PLCMOS     | UniVERSA-T   | 0.37 | 0.61 | 0.41 | 0.86 | 0.86 | 0.69 |\n| PLCMOS     | ARECHO       | 0.32 | 0.56 | 0.38 | 0.89 | 0.89 | 0.72 |\n| URGENTMOS  | UniVERSA-T   | 0.20 | 0.45 | 0.35 | 0.78 | 0.78 | 0.62 |\n| URGENTMOS  | ARECHO       | 0.26 | 0.51 | 0.40 | 0.74 | 0.75 | 0.61 |\n\nDNSMOS ablation (ARECHO-full: multi-metric prediction, ARECHO-only: single-metric prediction):\n\n| Metric  | Model         | MSE  | RMSE | MAE  | LCC  | SRCC | KTAU |\n|---------|---------------|------|------|------|------|------|------|\n| DNSMOS  | ARECHO-full   | 0.04 | 0.21 | 0.16 | 0.85 | 0.81 | 0.65 |\n| DNSMOS  | ARECHO-only   | 0.04 | 0.20 | 0.15 | 0.86 | 0.83 | 0.66 |\n\nWe thank the reviewer again for prompting this analysis. Due to space constraints, we will include extended results and deeper analysis in the revised manuscript and appendix.\n\n## Final Remarks\n\nWe sincerely thank Reviewer XEXh once again for their constructive and detailed feedback. Your questions helped us clarify architectural motivations, improve empirical presentation, and strengthen the practical framing of ARECHO. We are confident the rebuttal and the scheduled presentation-related revision will address your concerns and more clearly convey the significance and applicability of our contributions."}]

Author rebuttal:

Meta-review: Summary
This paper presents ARECHO a structure for calculating multiple speech metrics simultaneously.  The advantage is the modeling of dependencies between metrics, and an innovative curriculum to handle uncertainty.

Reasons to accept
There is broad consensus from the reviewers that this paper is technically sound and well motivated.  The writing is mostly clear, though reviewer XEXh required some clarification through reviewer discussion, which was mostly addressed.

Reasons not to accept
Reviewer DC5A had some comments about the complexity of the framework, and the fidelity of the scores through error accumulation and discretization loss.  These reasons are relatively minor.

Decision rationale
Recommend acceptance.  The reviewers were positive, with minimal criticism of the work.  This framework has a potential to be taken up by the broader community. The rebuttal process brought this paper up from a positive, but borderline paper.
Decision: Accept (spotlight)