Genius: A Generalizable and Purely Unsupervised Self-Training Framework
  For Advanced Reasoning
Authors: name: Fangzhi Xu, name: Hang Yan, name: Chang Ma, name: Haiteng Zhao, name: Qiushi Sun, name: Kanzhi Cheng, name: Junxian He, name: Jun Liu, name: Zhiyong Wu

Advancing LLM reasoning skills has captivated wide interest. However, current
post-training techniques rely heavily on supervisory signals, such as outcome
supervision or auxiliary reward models, which face the problem of scalability
and high annotation costs. This motivates us to enhance LLM reasoning without
the need for external supervision. We introduce a generalizable and purely
unsupervised self-training framework, named Genius. Without external auxiliary,
Genius requires to seek the optimal response sequence in a stepwise manner and
optimize the LLM. To explore the potential steps and exploit the optimal ones,
Genius introduces a stepwise foresight re-sampling strategy to sample and
estimate the step value by simulating future outcomes. Further, we recognize
that the unsupervised setting inevitably induces the intrinsic noise and
uncertainty. To provide a robust optimization, we propose an
advantage-calibrated optimization (ACO) loss function to mitigate estimation
inconsistencies. Combining these techniques together, Genius provides an
advanced initial step towards self-improve LLM reasoning with general queries
and without supervision, revolutionizing reasoning scaling laws given the vast
availability of general queries. The code will be released at
https://github.com/xufangzhi/Genius.