Beyond Markovian: Reflective Exploration via Bayes-Adaptive RL for LLM
  Reasoning
Authors: name: Shenao Zhang, name: Yaqing Wang, name: Yinxiao Liu, name: Tianqi Liu, name: Peter Grabowski, name: Eugene Ie, name: Zhaoran Wang, name: Yunxuan Li

Large Language Models (LLMs) trained via Reinforcement Learning (RL) have
exhibited strong reasoning capabilities and emergent reflective behaviors, such
as backtracking and error correction. However, conventional Markovian RL
confines exploration to the training phase to learn an optimal deterministic
policy and depends on the history contexts only through the current state.
Therefore, it remains unclear whether reflective reasoning will emerge during
Markovian RL training, or why they are beneficial at test time. To remedy this,
we recast reflective exploration within the Bayes-Adaptive RL framework, which
explicitly optimizes the expected return under a posterior distribution over
Markov decision processes. This Bayesian formulation inherently incentivizes
both reward-maximizing exploitation and information-gathering exploration via
belief updates. Our resulting algorithm, BARL, instructs the LLM to stitch and
switch strategies based on the observed outcomes, offering principled guidance
on when and how the model should reflectively explore. Empirical results on
both synthetic and mathematical reasoning tasks demonstrate that BARL
outperforms standard Markovian RL approaches at test time, achieving superior
token efficiency with improved exploration effectiveness. Our code is available
at https://github.com/shenao-zhang/BARL.