LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level
  Mathematical Reasoning
Authors: name: Di Zhang; name: Jianbo Wu; name: Jingdi Lei; name: Tong Che; name: Jiatong Li; name: Tong Xie; name: Xiaoshui Huang; name: Shufei Zhang; name: Marco Pavone; name: Yuqiang Li; name: Wanli Ouyang; name: Dongzhan Zhou
Published: 2024-10-03 18:12:29

Abstract: This paper presents an advanced mathematical problem-solving framework,
LLaMA-Berry, for enhancing the mathematical reasoning ability of Large Language
Models (LLMs). The framework combines Monte Carlo Tree Search (MCTS) with
iterative Self-Refine to optimize the reasoning path and utilizes a pairwise
reward model to evaluate different paths globally. By leveraging the
self-critic and rewriting capabilities of LLMs, Self-Refine applied to MCTS
(SR-MCTS) overcomes the inefficiencies and limitations of conventional
step-wise and greedy search algorithms by fostering a more efficient
exploration of solution spaces. Pairwise Preference Reward Model~(PPRM),
inspired by Reinforcement Learning from Human Feedback (RLHF), is then used to
model pairwise preferences between solutions, utilizing an Enhanced Borda Count
(EBC) method to synthesize these preferences into a global ranking score to
find better answers. This approach addresses the challenges of scoring
variability and non-independent distributions in mathematical reasoning tasks.
The framework has been tested on general and advanced benchmarks, showing
superior performance in terms of search efficiency and problem-solving
capability compared to existing methods like ToT and rStar, particularly in
complex Olympiad-level benchmarks, including GPQA, AIME24 and AMC23.