In-Context Learning Boosts Speech Recognition via Human-like Adaptation
  to Speakers and Language Varieties
Authors: name: Nathan Roll, name: Calbert Graham, name: Yuka Tatsumi, name: Kim Tien Nguyen, name: Meghan Sumner, name: Dan Jurafsky

Human listeners readily adjust to unfamiliar speakers and language varieties
through exposure, but do these adaptation benefits extend to state-of-the-art
spoken language models? We introduce a scalable framework that allows for
in-context learning (ICL) in Phi-4 Multimodal using interleaved task prompts
and audio-text pairs, and find that as few as 12 example utterances (~50
seconds) at inference time reduce word error rates by a relative 19.7% (1.2
pp.) on average across diverse English corpora. These improvements are most
pronounced in low-resource varieties, when the context and target speaker
match, and when more examples are provided--though scaling our procedure yields
diminishing marginal returns to context length. Overall, we find that our novel
ICL adaptation scheme (1) reveals a similar performance profile to human
listeners, and (2) demonstrates consistent improvements to automatic speech
recognition (ASR) robustness across diverse speakers and language backgrounds.
While adaptation succeeds broadly, significant gaps remain for certain
varieties, revealing where current models still fall short of human
flexibility. We release our prompts and code on GitHub.