Iterative Reasoning Preference Optimization
Authors: name: Richard Yuanzhe Pang, name: Weizhe Yuan, name: Kyunghyun Cho, name: He He, name: Sainbayar Sukhbaatar, name: Jason Weston

Iterative preference optimization methods have recently been shown to perform
well for general instruction tuning tasks, but typically make little
improvement on reasoning tasks (Yuan et al., 2024, Chen et al., 2024). In this
work we develop an iterative approach that optimizes the preference between
competing generated Chain-of-Thought (CoT) candidates by optimizing for winning
vs. losing reasoning steps that lead to the correct answer. We train using a
modified DPO loss (Rafailov et al., 2023) with an additional negative
log-likelihood term, which we find to be crucial. We show reasoning improves
across repeated iterations of this scheme. While only relying on examples in
the training set, our approach results in increasing accuracy for
Llama-2-70B-Chat from 55.6% to 81.6% on GSM8K (and 88.7% with majority voting
out of 32 samples), from 12.5% to 20.8% on MATH, and from 77.8% to 86.7% on
ARC-Challenge, which outperforms other Llama-2-based models not relying on
additionally sourced datasets.