StressTest: Can YOUR Speech LM Handle the Stress?
Authors: name: Iddo Yosha; name: Gallil Maimon; name: Yossi Adi
Published: 2025-05-28 18:32:56

Abstract: Sentence stress refers to emphasis, placed on specific words within a spoken
utterance to highlight or contrast an idea, or to introduce new information. It
is often used to imply an underlying intention that is not explicitly stated.
Recent advances in speech-aware language models (SLMs) have enabled direct
processing of audio, allowing models to bypass transcription and access the
full richness of the speech signal and perform audio reasoning tasks such as
spoken question answering. Despite the crucial role of sentence stress in
shaping meaning and speaker intent, it remains largely overlooked in evaluation
and development of such models. In this work, we address this gap by
introducing StressTest, a benchmark specifically designed to evaluate a model's
ability to distinguish between interpretations of spoken sentences based on the
stress pattern. We assess the performance of several leading SLMs and find
that, despite their overall capabilities, they perform poorly on such tasks. To
overcome this limitation, we propose a novel synthetic data generation
pipeline, and create Stress17k, a training set that simulates change of meaning
implied by stress variation. Then, we empirically show that optimizing models
with this synthetic dataset aligns well with real-world recordings and enables
effective finetuning of SLMs. Results suggest, that our finetuned model,
StresSLM, significantly outperforms existing models on both sentence stress
reasoning and detection tasks. Code, models, data, and audio samples -
pages.cs.huji.ac.il/adiyoss-lab/stresstest.