Title: ARECHO: Autoregressive Evaluation via Chain-Based Hypothesis Optimization for Speech Multi-Metric Estimation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Speech Evaluation

Speech Assessment

Speech Profiling

Dynamic Classifier Chain
Abstract: Speech signal analysis poses significant challenges, particularly in tasks such as speech quality evaluation and profiling, where the goal is to predict multiple perceptual and objective metrics. For instance, metrics like PESQ (Perceptual Evaluation of Speech Quality), STOI (Short-Time Objective Intelligibility), and MOS (Mean Opinion Score) each capture different aspects of speech quality. However, these metrics often have different scales, assumptions, and dependencies, making joint estimation non-trivial. To address these issues, we introduce ARECHO (Autoregressive Evaluation via Chain-based Hypothesis Optimization), a chain-based, versatile evaluation system for speech assessment grounded in autoregressive dependency modeling. ARECHO is distinguished by three key innovations: (1) a comprehensive speech information tokenization pipeline; (2) a dynamic classifier chain that explicitly captures inter-metric dependencies; and (3) a two-step confidence-oriented decoding algorithm that enhances inference reliability. Experiments demonstrate that ARECHO significantly outperforms the baseline framework across diverse evaluation scenarios, including enhanced speech analysis, speech generation evaluation, and noisy speech evaluation. Furthermore, its dynamic dependency modeling improves interpretability by capturing inter-metric relationships. Across tasks, ARECHO offers reference-free evaluation using its dynamic classifier chain to support subset queries (single or multiple metrics) and reduces error propagation via confidence-oriented decoding.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a new autoregressive evaluation toolkit for speech signal, which uniforms the scales and dependendices of different metrics. The paper aimmed to improve the interpretability between metrics as well.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. The quality of the paper is good. It points out the problems of existing metrc frameworks and studies a new solution.\n2. The originity of the paper is good. This is the first paper to study the relationship between speech signal metrics.\n3. From the Table 1, we can find that the proposed method significantly outperforms the previous baselines.\n\n# Weaknesses\nI have some concerns regarding the clarify and significance. If the authors could answer them well, I will consider raise my scores.\n1. The clarify is somehow not clear to me. See # Questions below.\n2. The significance is to be discussed. See # Questions below.\n\nQUESTIONS:\n1. In section 3.2, the authors applied quantization-based tokenization by partitioning the value range into uniformly or adaptively spaced bins. This is not clear how they do this, e.g. difference and scale between different metrics. Also, the quantization maybe helpful for training, but will the orignial regression task can also help improve the performance? How did these tasks be performed is not clear to me.\n2. In section 3.3, the authors use a random order the metrics but the inference is with arbitrary query orders. What's the influence of these orders? \n3. I think my most concern about this work is **whether there are such relationships between metrics in many speech tasks**. For instance, if we want to do the age estimation task, maybe gender and pitch is important, but others maybe less important. In this case, such method maybe too heavy and cannot improve the overall performance too much. I think the authors should also provide **examples about how to use ARECHO in certain tasks**, like speech enhancement, asr, tts.., and how the metric links help each other. Moreover, I think the current method is better for **understanding a long audio**, instead of such simple evaluations.\n4. The authors should analyze whether some of the metric estimation may hurt others. For example, there are many MOS metrics with different aspects. They should be independant but are considered dependant in this paper.\n\nLIMITATIONS:\nThe main limitation is that the method is lack of examples to be used in certain tasks, and a more detailed analysis of relationships and influence between specific metrics.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a classifier chain approach based on a WavLM backbone model to evaluate speech signals (multi-metric evaluation), following a multi-task approach that is able to model inter-metric dependencies. The classifier chain is implemented through a transformer-based autoregressive model. At inference time, the paper proposes a metric-prediction-order approach based on the model's confidence in predicting each metric. To deal with multiple metric scales, the paper proposes a tokenization-based approach. The experimental setup comprises speech signals from various domains and contexts, including up to 2138 hours of speech for training and 18.65 hours for testing, as well as a total of 87 different metrics (most of which are model-based, but also include classical signal processing-based and subjective metrics). The entire set of experiments encompasses analyses of various tokenization methods, token size, beam size during inference, and an examination of the order in which metrics are predicted across different speech domains.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper tackles a fundamental challenge in speech quality evaluation through an innovative and promising approach that lays the groundwork for future research in the field. The manuscript is well-structured and clearly presents the proposed method in most parts. The authors support their proposal with a comprehensive experimental evaluation, which enables a robust assessment of its performance and lends credibility to the reported results.\n\nThe core contribution—the prediction of quality metrics via sequential tokenization—represents a novel and original direction. This approach opens the possibility for more general prompt-based speech quality evaluation, potentially marking a significant advance for various established speech tasks such as audio generation, speaker diarization, and source separation. Moreover, it could benefit biomedical applications, where different dimensions of speech can serve as indicators of a patient's health status and where subjective evaluations are often costly and difficult to obtain. In this context, the use of inter-metric modeling via classifier chains is a promising strategy.\n\nHowever, a major concern lies in how the approach handles the prediction of quality metrics. All metrics are encoded as categorical variables, including those that are inherently continuous, and the models are trained using standard classification techniques. Yet, the problem at hand (for several metrics) is more accurately characterized as an ordinal regression task. This constitutes a fundamental limitation, as the commonly used categorical cross-entropy loss does not account for the ordinal nature of the target variables.\n\nA substantial body of literature has proposed surrogate loss functions that are better suited for ordinal regression, demonstrating improved performance over naïve classification-based approaches. Relevant references include:\n\nFathony, R., Bashiri, M. A., & Ziebart, B. (2017). Adversarial surrogate losses for ordinal regression. Advances in Neural Information Processing Systems, 30.\n\nNazabal, A., Olmos, P. M., Ghahramani, Z., & Valera, I. (2020). Handling incomplete heterogeneous data using VAEs. Pattern Recognition, 107, 107501.\n\nNachmani, I., Genossar, B., Scharf, C., Shraga, R., & Gal, A. (2025, April). SLACE: A Monotone and Balance-Sensitive Loss Function for Ordinal Regression. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 39, No. 18, pp. 19598–19606).\n\nGiven this, regression-based evaluation metrics would be more appropriate than classification-based ones. It is also essential to account for the distribution of quality levels across the dataset. In this regard, a balanced version of the Mean Absolute Error (MAE) is recommended, as discussed in:\n\nBaccianella, S., Esuli, A., & Sebastiani, F. (2009, November). Evaluation measures for ordinal regression. In 2009 Ninth International Conference on Intelligent Systems Design and Applications (pp. 283–287).\n\nNote that they have been previously used for voice quality evaluation using the GRBAS scale in biomedical contexts.\n\nQUESTIONS:\n- Appendix D outlines the procedure for the two-step confidence-oriented decoding in detail. However, it does not specify how parameter B was selected, nor does it provide empirical results demonstrating the influence of B on the model’s performance.\n\n- A notable limitation of the results section is the absence of specific findings related to subjective metrics. While the performance on model-based metrics may suffice in many cases, the extent to which the proposed approach improves the prediction of subjective metrics remains unclear. The comparison between order-c2f and order-mr offers an interesting perspective, yet it fails to provide concrete evidence regarding the method’s effectiveness on subjectively annotated evaluations.\n\n- Given that many of the target metrics are themselves generated by models, it is worth questioning the added value of using the proposed approach over directly using the ground-truth models. Additionally, although the autoregressive strategy is valuable, it inherently introduces error propagation. In this context, for model-based metrics, would it not be more appropriate to inject the actual values from the ground-truth models into the autoregressive loop, instead of relying on ARECHO predictions?\n\n- Could the authors provide statistics on the distribution and balance of the target values and ordinal ranks within the training set? If the data is found to be imbalanced, I strongly recommend using a balanced Mean Absolute Error (MAE) or a similar metric, rather than relying on the standard MAE or classification-based metrics.\n\nLIMITATIONS:\nThe manuscript does not raise any concern about the potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nARECHO is a framework designed for multi-metric speech evaluation, aiming to predict various speech quality and profiling metrics (e.g., PESQ, STOI, MOS) jointly rather than independently. The main contributions of ARECHO are (1) converting all metric values (numerical/categorical, with different scales) into a unified token space, enabling sequence modeling; (2) using an autoregressive model to capture dependencies between metrics, allowing the model to predict metrics in a flexible, context-aware order rather than in parallel; (3) enhancing prediction robustness by refining metric prediction based on confidence in the output, addressing instability in dynamic prediction order.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The approach to encode all types/scales of metrics into tokens enables joint modeling and paves the way for autoregressive multi-task prediction. Unlike prior methods that predict each metric independently, ARECHO explicitly models the relationships between metrics, leading to improved accuracy and interpretability.\n2. The two-step decoding mechanism allows the model to dynamically adjust the prediction order and recover from uncertain predictions, leading to higher reliability.\n\n## Weaknesses\n1. Converting numerical metrics to discrete tokens (quantization) may result in a loss of precision for fine-grained regression tasks. Is there any way to measure this loss?\n2. The error accumulation is a potential issue. The autoregressive, chain-based prediction means errors in early metrics could affect downstream metric predictions.\n3. Another concern is the complexity in training and inference, as the dynamic chain and two-step decoding add computational and implementation complexity compared to simple parallel predictors.\n\nQUESTIONS:\n1. How sensitive is ARECHO’s performance to the choice of quantization (e.g., bin size, uniform vs. adaptive) for different metrics? Is there a systematic way to choose tokenization strategies?\n2. As the mentioned error accumulation issue, how does the model mitigate the risk of early prediction errors cascading through the chain?\n3. How much does the two-step confidence-oriented decoding contribute relative to a standard greedy or beam search? Any quantitative ablation study will be better.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}