Diverse Inference and Verification for Advanced Reasoning
Authors: name: Iddo Drori; name: Gaston Longhitano; name: Mao Mao; name: Seunghwan Hyun; name: Yuke Zhang; name: Sungjun Park; name: Zachary Meeks; name: Xin-Yu Zhang; name: Ben Segev; name: Howard Yong; name: Nakul Verma; name: Avi Shporer; name: Alon Amit; name: Madeleine Udell
Published: 2025-02-14 07:22:25

Abstract: Reasoning LLMs such as OpenAI o1, o3 and DeepSeek R1 have made significant
progress in mathematics and coding, yet find challenging advanced tasks such as
International Mathematical Olympiad (IMO) combinatorics problems, Abstraction
and Reasoning Corpus (ARC) puzzles, and Humanity's Last Exam (HLE) questions.
We use a diverse inference approach that combines multiple models and methods
at test time. We find that verifying mathematics and code problems, and
rejection sampling on other problems is simple and effective. We automatically
verify correctness of solutions to IMO problems by Lean, and ARC puzzles by
code, and find that best-of-N effectively answers HLE questions. Our approach
increases answer accuracy on IMO combinatorics problems from 33.3% to 77.8%,
accuracy on HLE questions from 8% to 37%, and solves 80% of ARC puzzles that
948 humans could not and 26.5% of ARC puzzles that o3 high compute does not.
Test-time simulations, reinforcement learning, and meta-learning with inference
feedback improve generalization by adapting agent graph representations and
varying prompts, code, and datasets. Our approach is reliable, robust, and
scalable, and in the spirit of reproducible research, we will make it publicly
available upon publication.