Replacing thinking with tool usage enables reasoning in small language
  models
Authors: name: Corrado Rainone, name: Tim Bakker, name: Roland Memisevic

Recent advances have established a new machine learning paradigm based on
scaling up compute at inference time as well as at training time. In that line
of work, a combination of Supervised Fine-Tuning (SFT) on synthetic
demonstrations and Reinforcement Learning with Verifiable Rewards (RLVR) is
used for training Large Language Models to expend extra compute during
inference in the form of "thoughts" expressed in natural language. In this
paper, we propose to instead format these tokens as a multi-turn interaction
trace with a stateful tool. At each turn, the new state of the tool is appended
to the context of the model, whose job is to generate the tokens necessary to
control the tool via a custom DSL. We benchmark this approach on the problem of
repairing malfunctioning Python code, and show that this constrained setup
allows for faster sampling of experience and a denser reward signal, allowing
even models of size up to 3B parameters to learn how to proficiently expend
additional compute on the task.