Learning from Failures in Multi-Attempt Reinforcement Learning
Authors: name: Stephen Chung; name: Wenyu Du; name: Jie Fu
Published: 2025-03-04 02:53:39

Abstract: Recent advancements in reinforcement learning (RL) for large language models
(LLMs), exemplified by DeepSeek R1, have shown that even a simple
question-answering task can substantially improve an LLM's reasoning
capabilities. In this work, we extend this approach by modifying the task into
a multi-attempt setting. Instead of generating a single response per question,
the model is given multiple attempts, with feedback provided after incorrect
responses. The multi-attempt task encourages the model to refine its previous
attempts and improve search efficiency. Experimental results show that even a
small LLM trained on a multi-attempt task achieves significantly higher
accuracy when evaluated with more attempts, improving from 45.6% with 1 attempt
to 52.5% with 2 attempts on the math benchmark. In contrast, the same LLM
trained on a standard single-turn task exhibits only a marginal improvement,
increasing from 42.3% to 43.2% when given more attempts during evaluation. The
results indicate that, compared to the standard single-turn task, an LLM
trained on a multi-attempt task achieves slightly better performance on math
benchmarks while also learning to refine its responses more effectively based
on user feedback. Full code is available at
https://github.com/DualityRL/multi-attempt