Advancing Speech Understanding in Speech-Aware Language Models with GRPO
Authors: name: Avishai Elmakies, name: Hagai Aronowitz, name: Nimrod Shabtay, name: Eli Schwartz, name: Ron Hoory, name: Avihu Dekel

In this paper, we introduce a Group Relative Policy Optimization (GRPO)-based
method for training Speech-Aware Large Language Models (SALLMs) on open-format
speech understanding tasks, such as Spoken Question Answering and Automatic
Speech Translation. SALLMs have proven highly effective for speech
understanding tasks. GRPO has recently gained traction for its efficiency in
training LLMs, and prior work has explored its application to SALLMs, primarily
in multiple-choice tasks. Building on this, we focus on open-format tasks that
better reflect the generative abilities of the models. Our approach leverages
GRPO with BLEU as the reward signal to optimize SALLMs, and we demonstrate
empirically that it surpasses standard SFT across several key metrics. Finally,
we explore the potential of incorporating off-policy samples within GRPO for
these tasks, highlighting avenues for further improvement and further research.