Better Late Than Never: Evaluation of Latency Metrics for Simultaneous
  Speech-to-Text Translation
Authors: name: Peter Polák; name: Sara Papi; name: Luisa Bentivogli; name: Ondřej Bojar
Published: 2025-09-22 04:21:19

Abstract: Simultaneous speech-to-text translation (SimulST) systems have to balance
translation quality with latency--the delay between speech input and the
translated output. While quality evaluation is well established, accurate
latency measurement remains a challenge. Existing metrics often produce
inconsistent or misleading results, especially in the widely used short-form
setting, where speech is artificially presegmented. In this paper, we present
the first comprehensive analysis of SimulST latency metrics across language
pairs, systems, and both short- and long-form regimes. We uncover a structural
bias in current metrics related to segmentation that undermines fair and
meaningful comparisons. To address this, we introduce YAAL (Yet Another Average
Lagging), a refined latency metric that delivers more accurate evaluations in
the short-form regime. We extend YAAL to LongYAAL for unsegmented audio and
propose SoftSegmenter, a novel resegmentation tool based on word-level
alignment. Our experiments show that YAAL and LongYAAL outperform popular
latency metrics, while SoftSegmenter enhances alignment quality in long-form
evaluation, together enabling more reliable assessments of SimulST systems.