Self-Consistency Preference Optimization
Authors: Archiki Prasad
Weizhe Yuan
Richard Yuanzhe Pang
Jing Xu
Maryam Fazel-Zarandi
Mohit Bansal
Sainbayar Sukhbaatar
Jason E Weston
Jane Yu
Venue: ICML 2025 poster (2025)
Keywords: Self-Alignment
Unsupervised and Semi-supervised Reasoning
Iterative Training
Primary area: deep_learning->large_language_models

Abstract: Self-alignment, whereby models learn to improve themselves without human annotation, is a rapidly growing research area. However, existing techniques often fail to improve complex reasoning tasks due to the difficulty of assigning correct rewards. An orthogonal approach that is known to improve correctness is self-consistency, a method applied at inference time based on multiple sampling in order to find the most consistent answer. In this work, we extend the self-consistency concept to help train models. We thus introduce self-consistency preference optimization (ScPO), which iteratively trains consistent answers to be preferred over inconsistent ones on unsupervised new problems. We show ScPO leads to large improvements over conventional reward model training on reasoning tasks such as GSM8K and MATH, closing the gap with supervised training with gold answers or preferences, and that combining ScPO with standard supervised learning improves results even further. On ZebraLogic, ScPO finetunes Llama-3 8B to be superior to Llama-3 70B, Gemma-2 27B, and Claude-3 Haiku.

Self- Celf-Consistency Preference Optimization
Archiki Prasad
Weizhe Yuan 13 Richard Yuanzhe Pang'
Jing Xu
Maryam Fazel-Zarandi Jane Yu'
Mohit Bansal
Sainbayar Sukhbaatar'
Jason Weston
1
Abstract
propose "sselfrewarddng"" training pipeline for instruction- following, comprising two steps: (i) using the LLM to gener- ate new queries and for each query; and (ii) building preference pairs and train- ing the LLM using iterative direct preference optimization loss (DPO; Rafailov al., 2024; Xu et al., 2023). However, Huang al. (2024) demonstrate that LLMs struggle at eval- uating the correctness of their own responses on complex problem-solving tasks which have an unambiguous correct answer, thereby rendering Yuan et al. 's self-evaluation ap- proach ineffective. Using an external reward model (RM) to rank responses can have similar problems; even if such models are trained on reasoning tasks they may still suffer on oft-o--disttibuutin problems (Casper al., 2023; Zhang et al., 2024; Mahan et al., 2024).
Self-alignment, whereby models learn to improve themselves without human annotation, is rapidly growing research area. However, existing tech- niques often fail to improve complex reason- ing tasks due to the difficulty of assigning cor- rect rewards. An orthogonal approach that is known to improve correctness is self-consistency, a method applied at inference time based on mul- tiple sampling in order to find the most consis- tent answer. In this work, we extend the self- consistency concept to help train models. We thus introduce self-consistency preference opti- mization (ScPO), which iteratively trains con- sistent answers to be preferred over inconsistent ones on unsupervised new problems. We show ScPO leads to large improvements over conven- tional reward model training on reasoning tasks such as GSM8K and MATH, closing the gap with supervised training with gold answers or prefer- ences, and that combining ScPO with standard supervised learning improves results even further. On ZograLogic, ScPO finetunes Llama-3 8B to be superior to Llama-3 70B, Gemma-2 27B, and Claude-3 Haiku.
the generated responses
To address this, we introduce Self-consistency Preference Optimization (ScPO). ScPO is an approach to train LLMS for complex problem-solving tasks without access to gold solutions or final answers in the training data. Our approach leverages the concept of self-consistency (Wang et al., 2023), an inference-time only approach that improves performance reasoning tasks by generating multiple so- lutions using the LLM and choosing the most frequent final answer. More consistent answers are more likely to be cor- rect because mistakes made by the model are often random, so incorrect solutions are unlikely to lead to the same answer multiple times (Fischler Bolles, 1981; ; Chen et al., 2023). In ScPO, the self-consistency concept is instead applied daring unsupervised self-training. The method consists of (i) selecting
1. Introduction
Training large language models (LLMs) on human- annotated data has improved their performance on a wide array of tasks (Bai et al., 2022; Touvron et al., 2023). How- ever, the size and quality of human data remains a major bottleneck as the data collection process is often resource- intensive in terms of cost, time, and expertise. To address this challenge, recent works focus on iteratively training from model-generated data via self-training (Yuan et al., 2024; Chen al., 2024b). Notably, Yuan al. (2024)
queries, (ii) annotating prefer- ence pairs using the most self-consistent response (winner) and least self-consistent response (loser), and (iii) optimizing a loss function that is weighted for each instance depending on the model's confidence in the preference pair. Addition- ally, we propose semi- remi-supervised variant of ScPO that jointly trains L.MMs on labeled and unlabeled instances, tak- ing advantage of human annotations whenever available. Unlike self-consistency applied during inference, SCPO does not increase inference-time compute, but they can also be combined together for better performance.
Mett FAIR FNC Chapel Hill 'New York University. Corre- spondence to: Archiki Prasad <archiki@cs.unc.cda udu..
In our experiments using Llama-3 8B models (Dubey et al., 2024), we show that even without access to any gold an- swers during training, two iterations of unsupervised ScPO
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Self- Self-Consistency Preference Optimization
Model at Iteration t
Seff-consistency Preference Optimization (ScPO)
M₁
Optimize with ScPO loss (Csuro) obtain M,+ and iterate
Generate problems
Seed Generated
Rachel has $120 to spend on new bike that costs $80, and she has to pay a 5% sales tax on the bike. she also wants to buy helmet that costs $15, how much money will she have left?
Problems
Q: Rachel has $120 costs
D
pair's weight
Compute
$15, will she have left?
D
Sample responses
M
Chosen
Rejected
Votes
Sample k responses
M
Add to train data If mox(votes) 2 r
A: Let's think step by step Then ... the enswer
Votes([C] Votes([R)
w
A: Let's think step by step... ... the answer
Vores
Answers
Vexaa
ScPO Loss wfDPO NLL]
Chosen
Rejected
D pairs
Answers
25
Answers
Generating New Problems
Building Preference Pairs
Training w/ ScPO Loss
Figure 1. Self-consistency Preference Optimization (ScPO). Given a query, we sample multiple responses from the current model M: and count the frequency of each answer (i.e., votes). We select the highest and lowest votes chosen and rejected responses (middle), and use these preference pairs to train the model with weighted Cs.ro loss (right). We employ a similar pipeline for generating new queries from the model itself (left), filtering out data where self-consistency is low.
improves zero-shot accuracy of the base model by 22.74% and 5.26% (absolute) on GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) respectively, closely matching the performance (< 1% difference) of the super- vised baseline from Pang et al. (2024). Moreover, when supplied with the gold labels in the training set and addi- tional model-generated problems, remi-supervisee ScPO improves GSM8K accuracy over the supervised baseline by 2.35%. On challenging logical puzzles in Zebral (Dziri al., 2024)- - where only test puzzles (without solutions) publicly available training Llama-3 8B with ScPO improves puzzle accuracy by 6.5%, outperforming larger LLMs such as Llama-3 70B, 2 27B (Team et al., 2024), and Claude-3 Haiku (Anthropic, 2024).
swers) for responses, ScPO uses the consistency of the model Mr, as measured by a real-valued vote function V(.) defined below, to rate and rank the quality of each response. Our vote function is based on self-consistency (Wang et al., 2023) of the model. In fact, ScPO can also be used with any measure of model consistency such as internal consis- tency (Liang et al., 2024) or universal consistency (Chen et al., 2024a).
Generating New Problems. Following other self- aligmment methods (Yuan et al., 2024; Yu et al., 2024), we use few-shot prompting to self-generate additional problems from the model. Using the seed set, multiple example prob- lems are chosen at random and placed in context to generate new problem. Note that some prior works are constrained to simultaneously generating both new query along with its corresponding correct answer (Yu et al., 2024). In contrast, with ScPO, we do not rely on accurately generating the corresponding answer, allowing the model to generate more diverse problems as long as the problems are well-formed and at least some are answerable. While the model may generate some unanswerable queries, these can be filtered out using the vote function (-)). Specifically, we filter out query if none of the responses generated by M, have vote 2 r (shown in Figure I; left). At each iteration t, we aug- ment the seed queries with the problems generated from M, to obtain the training problems for the next iteration D(+1-
2. Self-consistency Preference Optimization
As depicted in Figure 1, ScPO is an unsupervised iterative training method that starts with base language model. Each iteration makes use of existing training problems/queries (without labels) as well as newly generated problems. The self-consistency metric is used in both generating new prob- lems and building preference pairs. We describe each step of ScPO's iterative training setup below. All prompts for so- lution generation and new problem generation can be found in Appendix D.
Initialization. ScPO assumes access to an initial base model Mo and a small amount of (seed) high-quality unla- beled queries, which are typically complex reasoning prob- lems. The model will be trained and updated at each training iteration resulting in models M1, M2. ... ,M,, where T is the total number of iterations. Instead of gold labels (an-
Building Self-Consistence Preference Pairs. For each problem x in the training data Dr, we use temperature- based sampling with the current model M, to generate responses y = {y1: y2, ... yx) sampled from M2(+2) in- cluding any rationales, e.g., chain-of-though (Wei al.,
2

---

Self-Consistency Preference Optimization
2022), followed by the final answer. Following Wang et al. (2023), the vote function V()) extracts the final an- swer corresponding to each response y E Ye via ans(-) and returns the relative frequency of the final answer, i.e, ((y) -
model M, as the reference model in the DPO loss (Rafailov et al., 2024). After training on this loss, the trained model is used to initialize the next iteration, i.., Mc+1 t- Me.
> 2m= (ans((ym)= = ay)( As illustrated in Fig- ure I (middle), using the vote function, we create preference pairs Di by selecting the most consistent response as the chosen (winning) response and selecting the least consistent one as the rejected (losing) response, provided that the vote of the chosen response is greater than a threshold r.' In other words,
Iterative Training. Starting with an initial seed model Mo. we train a series of models M1, M2, i.. for T 2 iterations (we justify this choice in Appendix B). Each model M4+1 is trained using LsCro Dpairs, the data generated by the 1tth model, defined as follows:
Mg: Seed LLM, initialized with pretrained LLM (need not be instruction-finetn finetuned).
Dpairs
1 E Dr, = arg max V(y), yEB, y arr min V(v), and V(y*) 2 rJ.
Mi: Initialized with Mo to generate Dpaira from Do (+
new problems) and trained using LSePo-
yEn.
My: Initialized with Mj to generate Di from D2 (+ new problems) and trained using LScPo-
SCPO Loss Function. ScPO operates under the assump- tion that when multiple responses sampled for problem i
This approach is similar to the Self-Rewarding LM training loop (Yuan et al., 2024) except for the fact that we use the model's self-consistency to score responses instead of using the same model which Huang et al. (2024) show is often challenging. In contrast to other iterative bootstrapping techniques for rea- soning (Zelikman et al., 2022; Pang et al., 2024), ScPO does not require access to gold labels such as gold responses or final answers, allowing ScPO to scale beyond the problems from an existing training dataset.
map to the same answer, then the predicted answer is likely
to be correct, the same assumption as in Wang et al. (2023). Consequently, we use consistency via a vote function V(-) as a proxy to create preference pairs. However, at the same time, the number of votes attained by a response can also reflect the model's confidence in the response (Xiong et al., 2024; Kabra al., 2024), implying that pairs where the vote margin the difference in votes attained by the chosen vs. the rejected response - is larger, are of higher quality and vice-versa (refer to Appendix A). We model this in ScPO's training by using an instance-level weight w(x) to the loss, i.e., for the preference pair (r. ,)" E Dpairs w(x)= = (V(++)- - VVy /k, where k is the total number of responses generated for each question (total number of votes cast).? 2 We thus use the following loss function:
judge to verify its own correctness,
Semi-Supervised Training with SCPO. Although ScPO does not require access to gold labels, we can easily incorpo- rate datasets with gold labels in conjunction with unlabeled datasets during ScPO training. To this end, we alter the pref- erence pair creation strategy described in that case. When gold labels are available for query Tgold. sample k responses, and create pairs such that the chosen response y+ is correct and the rejected response y is incorrect (dis- carding queries where such pairs cannot be created). Since we already know these pairs are of high quality, we set the weight of annotated instances w(Zgota) = For queries that do not have gold labels, we use our self-consisteney criterion for pair creation and compute the weighted loss for those examples as before. A special case is that if all data is labeled, the loss reduces to the IRPO loss.
Lycpoly ,y [x))
-w(x) logo Bogg Mo xx)
Ma(y (x)
Mx x) B log
M1 x))
Weighted DPO Loss
aw(xx
log Mo(y+1
y+|
Weighted NLL Loss
The loss includes a DPO and NLL term similar to the re- cently introduced supervised IRPO (Pang et al., 2024) loss but in our case we have an unsupervised objective and use our introduced weighted loss. Here (-) denotes the sigmoid function, and a. B are hyperparameters of the loss function, and / represents the LLM parameters being trained in the current iteration. At the tt iteration, we use the initialized
3. Experimental Setup
Datasets and Metries. We evaluate the effectiveness of ScPO range of math and logical reasoning datasets:
GSM8K (Cobbe et al., 2021) contains train/test split of 7.5K/1..3KK 1.3 grade school math word problems. For the purpose of this work, we split the train set into a train/dev split with 6.7K/O.8K problems respectively. We use the dev split for hyperparameter tuning and checkpoint se-
By design, several responses can share final answer (but for example, their chain-of-though may he different). So, we cluster the responses by final answer and pick response at random. "This normalization ensures that weights w()) E [0, 1].
3

---

Self-Consistency Preference Optimization
lection. The overall data split becomes 6.7K/0.8K/1.3K in the train/dev/test set, respectively. We report perfor- mance based on exact match accuracy of the final numeric answer on the test set.
Baselines. We compare models trained with ScPO in un- supervised (denoted as ScPOUnsup. and semi-supervised (denoted as SCPOsemi-Sup.) settings against the following:
Seed model (Zero-shot CoT). We compare against the seed model (Mo) using zero-shot chain-of-thought prompting (Kojima et al., 2022) generated with greedy decoding and report results with or without inference-time self-consistency (SC; Wang et al., 2023).
MATH (Hendrycks et al., 2021) is a dataset of chal- lenging high-school math competitions that contains a train/test split of 7.5K/5K problems, respectively. Similar GSM8K, we reserve 10% of samples from the train set to create held-out dev set for model selection and hyper- parameter tuning, resulting in our final train/dew/test splits with 6.7K/0.8K/5K problems, respectively. We report the accuracy of the final answer on the test set.
Supervised Training with Gold Answers (IRPOGola).
We use a strong supervised preference optimization method for reasoning tasks (Pang et al., 2024), to serve as an upper- bound on performance for unsupervised train- ing as this uses gold data from the train set, which we compare to unsupervised and semi-supervised ScPO. For each query x. preference pairs are constructed such that chosen responses are correct and rejected responses incorrect with w(x)= 1.
ZebraL.ogic (Dziri et al., 2024) is a logical reasoning benchmark. It is a test set of IK logic grid puzzles (or Einstein's ' puzzles) designed as constraint satisfaction problem (Prosser, 1993). Each puzzle is comprised of n houses with m unique features, resulting in an n X m table. Given a list of clues, solving the puzzle requires deducing the correct (unique) assignment of values in the table, i.c., a unique value for each feature and house. Evaluation metries for this dataset are: puzzle accuracy (overall, easy, and hard puzzles) as well as cell accuracy.
Unsupervised Training with External RM (IRPORM) We propose a new variant of IRPO that we also ex- pect to be a strong baseline. Given the plethora of publicly-available reward models (RMs; Lambert et al., 2024), in the absence of gold labels, off-the-shelf RMs can be used to score a set of responses y Mc(-1z) and create preference pairs such that chosen and re- jected responses have the maximum and minimum re- ward, respectively, i.e., y+ =arg arg maxyeg RM(y(z) and y = arg mingeg RM(y2z)) with w(x)= =1. 1. strongly performing EemoRM-Llamas et al., 2024a) as reward model.3
Base Models. For GSMSK and MATH, we use Llama-3 Base 8B (Dubey et al., 2024) as the seed model Mo. We note that the instruction-tune -unned version may have already been fine-tuned the gold data from these tasks, new experimental settings cannot be reliably tested in that case. For Zebral Logic, we use Llama-3 Instruct 8B (Dubey et al., 2024) as the seed model.
We use the model (Wang
Language Models Self-Improved (LMSI). Following Huang et al. (2023), we implement LMSI, another un- supervised bascline that uses LLM self-consistency to generate target CoT solutions for problems and iteratively trains the LLM via supervised finetuning, i.e., the NLL loss, differing from ScPO's weighted preference- loss. Similar to ScPO, we generate additional reasoning problems using the LLM followed by consistency-base filtering (detailed in Section 2).
Preference Training Data. We use the Llama-3 Instruct 8B model to generate additional problems (queries). For GSM8K and MATH, we prompt the model to generate a problem similar to 4-shot examples of problems from the train set. Note that the prompt only requires valid human- written problems and not their corresponding answers. We filter out problems where max,5 V(yi < 0.5k (or, r 0.5k) where k is the number of responses sampled or votes cast for each query. That is, where less than half of the votes go towards the majority answer, which we found to be good threshold based on the dev set accuracy (see Section 5). Since Mj models tend to be more consistent than Mo (cf. Section 5), for M2 training data, we increase the filtering threshold r to 0.7k and 0.6k on GSM8K and MATH, respectively. For ZebraLogic, we prompt the model to rephrase or perturb features of puzzle from the dataset in one-shot manner. Then, we use the underlying model Mr to generate k=16 responses for each question and filter out questions where none of the responses accrue т = 2 or more votes (exactly matching solutions) for M2 and set r 0.5k for training M2.
When generating multiple response or new problems from the LLM, we sample with temper- ature of 0.7 and top-p 0.9. For GSM8K and MATH, we set & With every iteration of training, the models become more consistent due to the training objective (see Section 5), thereby, making picking the rejected response harder, i.e., none of the responses are incorrect or all the responses share the same final answer. Therefore, to sample rejected responses, we further generate 8 responses sam-
Hyperparameters.
'Wang et al. (2024a) use training splits of GSM8K and MATH to train ArmoRM, rendering these datasets highly in-distribution for the RM while Zebral is -uttof-distriuution (further dis- cussed Section 5).
4

---

Self- Self-Consistency Preference Optimization
Table GSM8K zero-shot accuracy after training Llama-3 Base 8B with SCPO and baselines, using greedy or 8-way self- consistency (SC)-based inference. The best performance is in bold, and second-best is underlined. We list train set sizes for each method: "Seed" corresponds to seed problems in the train set, whereas "Gen." indicates additional problems generated by the model (without answers). IRPOGold, and ScPOSent-Sup. high- lighted in green. the gold answers create preference pairs (when available, indicated with ').
Table 2. MATH zero-shot accuracy after training Llama-3 Base 8B with ScPO and baselines, using greedy or 8-way self- consistency (SC)-based inference. "Seed" corresponds to seed queries in the train set, "Gen." are additional model-generated problems (without answers). IRPO and ScPOsemi-SSug high- lighted in green, use gold answers train (indicated with 1)
Method
Iter. Train Data (K) Test Acc. (%)
# Seed / Gen. Greedy SC
Method
Iter. Train Data (K) Test Acc. (%)
without access to gold labels Seed model (zero-shot) Mo
-1-
14.46
18,20
# Seed/ Gen.
Greedy
SC
IRPO
Mi
6.4/-
18.06
24.20
without access to gold labels Sced model (zero-shot) Mo
M2
6.5/-
18.08
22.64
41.17
51.80
LMSI
Mi
0.6/1.2
16.78
22.92
IRPORM
Mi
5.5
48.67
69.98
M2
1.1/2.0
16.96
20.20
M2
4.4/-
50.11
61.25
ScPO Unsupp
M1
0.6/
17.36
25.70
LMSI
Mi
5.3/-
53.53
63.91
M2
1.2/2.5
19.72
24.58
M2
1.1/5.2
56.71
62.55
with access to gold labels
ScPO Unsupp.
M1
5.3/-
61.03
71.49
IRPO
Mi
2.7¹/-
18.64
26.88
M2
1.4/5.1
63.91
71.11
M2
3.0 1-
20.32
26.88
with access to gold labels
ScPOsemi-Sup
Mi
2.7
/12
19.88
27.35
IRPOGold
M1
4.4% /-
61.41
72.93
M2
3.0
20.48
26.92
M2
5.7 /-
64.29
72.56
ScPO Semi-Sup.
M1
41.9
63.61
74.30
M2
/4.5
66.64
74.75
based on MATH's train set (Lightman et al., 2024; Wang et al., 2024a). Hence, ScPO's 's improvement over IRPORM would likely be larger if the RM had not used in-domain gold labels during training. Overall, we find SCPO has the ability to outperform RMs, especially in -fittriibiitttionn settings. Lastly, in comparison to LMSI, another iterative and unsupervised baseline, two iterations of ScPO UnsOp. outperform that of LMSI by 7.20% and 2.76% on GSM8K and MATH, respectively, when using greedy decoding. This highlights the importance of weighted preference objective in training LLMs effectively using self ceff-consistency.
pled with higher temperature of 1.2 to encourage more diverse answers. On ZebraLogic, due to the complex nature of the response (an n X m table), we find that sampling a response that gets multiple votes is relatively infrequent, so we set k 16 for this task. All models are trained for 10 epochs with learning rate of 5e-6 (cosine scheduling), and effective batch size of 16. Lastly, we set DPO loss term hyperparameter B = 0.5 and NLL regularization coefficient == 1. When a dev set is available (e.g., GSM8K and MATH), we use accuracy on the dev set for checkpoint se- lection (at every epoch). For ZebraLogic, which is similarly challenging to MATH and does not have a train or dev set, for each iteration, we train for the same number of epochs that performed best during MATH training.
Iterations of ScPO improve reasoning. From Tables I and 2, we observe that two iterations of ScPO consistently improves the LLM's performance when using greedy de- coding in both unsupervised and semi-supervised settings compared to one iteration. On GSM8K, greedy test accu- racy improves by 2.88%, and 3.03% when using ScPO for unsupervised and semi-supervised training, respectively, Similarly, on MATH, in Table 2, we find that M2 models with ScPO outperforms their M1 counterparts by up to C 2.36% in greedy accuracy. This can be explained by models becoming more accurate and consistent after one round of ScPO training (shown in Section 5). Consequently, this allows us to bootstrap from additional problems in the origi- nal and generated training data, for which the Mo model did not have consistent response. However, we find that the accuracy computed using 8-way self-consistency (SC) satu- rates after the first iteration, sometimes even resulting in a slight decrease compared to M1. This may happen because now that the model trained to be more consistent there
4. Main Results
4.1. Math Reasoning
ScPO outperforms unsupervised baselines. Comparing methods on GSM8K, in Table 1, we observe that training with only one iteration of ScPO outperforms the zero-shot seed model and IRPORM. by 22.7 22.74% and 12.36%, respec- tively, using greedy decoding. Similarly, on MATH (cf. Ta- ble 2), two iterations of ScPOunsup. yields an improvement of 5.26% and 1.64% respectively compared to the same two baselines. We further note that while IRPORM is not given direct access to the gold labels, it uses the ArmoRM, which has been trained on human- -nnotttn step-level data
5

---

Self- Celf-Consistency Preference Optimization
Table 'ogi test performance after unsupervised training of Llama-3 Instruct 8B with ScPO, compared baselines. "Seed"
corresponds to original puzzles in the test set, whereas "Gen." indicates additional puzzles generated. "Taken from the Leaderboard.
Method
Train Data (K)
Puzzle Ace. (%)
Cell Acc.
# Seed / Gen.
Overall
Easy
Hard
(%) )
Llama-3 Instruct 70B
17.2
52.1
3.6
42.9
Gemma-2 27B IT" Claude-3 Haiku"
-/-
16.3
50.7
2.9
41.2
-/-
14.3
47.9
1.2
37.9
Mo (Llama-3 Instruct 8B)
-/-
11.6
40.0
0.4
39.1
Mi w/ IRPORM
1.0/-
11.3
37.9
1.0
42.1
M. w/ LMSI
0.4
16.2
51.1
2.6
45.8
M2 w/ LMST
0.4
16.8
53.6
2.5
46.9
M w/ ScPOUasup. Unsup.
0.4
17.0
54.3
2.5
47.6
M2 w/ ScPOUasup.
0.5 2.2
18.1
58.2
2.5
45.2
less benefit from applying self-consistency at inference time (see analysis in Section 5). We find that a third iteration of training also shows minimal gains, however we utilize the (unlabeled) problems from the test set build preference pairs, we find that can obtain additional performance boosts on top of Mg. discussed in Appendix B.
In Appendix C, we repeat the math reasoning experiments with Llama-3.1 Base 8B and find that while the absolute per- formance increases, the relative trends among the baselines remain the same with two iterations of ScPOsemi-Sup improving the greedy test accuracy of the seed model by 25.32% and 8.66% on GSM8K and MATH, respectively.
4.2. ZebraLogic: A Challenging Logical Reasoning Task
Unsupervised ScPO is comparable to IRPO training with gold labels. We can compare the unsupervised train- ing of ScPO with the supervised training using gold la- bels of IRPO in Tables 1 and 2. The results show that ScPOunsup. without using any gold labels can yield com- parable accuracy to IRPO on GSM8K and MATH with < 1% gap in greedy performance and < 2% gap in accuracy using 8-way self-consistency after two iterations of train- ing (M2). This comparable performance of ScPOUnsup is likely due high correlation (0.8 across the datasets) between the vote shares and accuracy on the test set, further discussed in Appendix A. Note that on tasks that are challenging for the seed model Mo, such as MATH, we can only bootstrap a small set of examples from the original set of training problem as compared to IRPO (i.c., only around a quarter of examples obtain a clear majority answer). How- offset this gap in training data by generating new problems using fhw-shot prompting (cf. Section 2) and creating preference pairs using our self-consistency method. This yields improvements during the second iteration.
ScPO outperforms unsupervised baselines. Table 3 re- ports performance on ZebraLogic of ScPO and various baselines, using greedy decoding. We observe large im- provements over the seed model, Llama-3 Instruct 8B (Mo) with one iteration of unsupervised ScPO (M1). improving performance by 5.4% and 8.5% in overall puzzle accuracy (exact match of tables) and cell accuracy (match of each cell in the table), respectively, In contrast, unsupervised training of IRPORM yields only mild gains over the seed model by 3% in cell accuracy and even slight drop in puzzle accuracy (11.6% to 11.3%). This can be attributed to ZebraLogic puzzles being moRM (cf. Section 5), thus trailing behind one iteration of ScPO by 5.7% in puzzle accuracy and 5.5% in cell accu- racy, Moreover, two iterations of ScPO outperform that of LMSI by 4.6% on easy puzzles and 1.3% overall accu- racy. Taken together, training with ScPO for two iterations improves the performance of the seed model by 8 positions on the leaderboard (from 38th to 30th with a 6.5% boost in puzzle accuracy and, to the best of our knowledge, is the best 8B-scale L.MM on ZebraLogic.
for the Ar-
ever,
Semi-supervised training with ScPO outperforms IRPO. Lastly, in Tables and 2, we evaluate the semi-supervised version of ScPO combined with using gold labels. We find that on GSMSK, SePOsemi-Sup. improves the greedy accu- racy by 2.35% and SC accuracy by 2.19% in comparison to IRPO Similar trends hold on the MATH dataset, where one iteration of ScPOsemi-Sup, outperforms IRPO by 1.24% using greedy decoding. These results show the utility of using ScPO to bootstrap from model -eeerrage problems even with access to a labeled training set.
8B LLM trained with ScPO outperforms larger models. Comparison of ScPO-trained models to other models in Table 3 demonstrates that ScPO-training after two iterations (M₂) outperforms significantly larger models such as Llama- Instruct 70B, Gemma-2 27B, and Claude-3 Haiku by 0.9%, 188%, and 3.8% in overall puzzle accuracy, respectively. Additionally, we find that models trained using ScPO also yield the highest cell accuracy. We attribute these gains over
6

---

Self-Consistency Preference Optimization
Table Ablation comparing unweighted loss (u((2)= = 1) to the proposed weighted loss used in ScPO. ScPO outperforms the unweighted loss in all cases.
Table 5. Impact of using different thresholds majority vote filter training data on MATH. Margin (%) denotes the difference in accuracy of the chosen and rejected response.
Method
Train (K)
Test Ace. ( (%)
Setting
Margin # Train Test Acc.
Seed Gen.
Greedy SC (8-way)
Mo
14.46
Mi (T = -.11k
18%
6.7K
15.44
Mi w/ w())=1
5.3/-
58.53
69.07
MATH GSM8K
Mi (T = -0.k)
44%
2.4K
16,34
M2 w/ w()==1
1.4/5.1
62.62
69.90
M1 (T -0.5k)
57%
1.8K
17.36
MI w/ ScPOUnsup
5.3/-
61.03
71.49
M1 (T -0.7k)
68%
0.7K
14.76
M2 w/ SCPO Unsup.
1.4/5.1
63.91
71.11
MI w/ (x(x)=
0.6/1.2
15.92
25.34
M2 w/ w(x)= =1
1.2/2.5
18.74
25,58
GSM8K the LLM is most consistent and accurate whereas on ZebraLogic it is the least consistent and accurate.
Mi w/ ScPOUnang
0.2/
17.36
25.70
M2 w/ ScPO
1.2/2.5
19.72
24.58
Impact of consistency-based filtering on construct- ing preferences. In Section 3, when generating self- consistency preference data for GSM8K and MATH, we filter out instances where fewer than half of the votes go towards the majority answer, i.e., T = 0.5k. The choice of this threshold presents a trade-off between the number of preference pairs available for training and the quality of the training data, and affects the difference (margin) in accuracy of the chosen and the rejected response. Assuming access to the gold answers to measure quality of preference data, in Table 5, we analyze this trade-off on MATH. As the vote threshold increases from T = 0.1 to t = 0.7k, the quality of training preference pairs increases, with the accuracy margin increasing from 18% to 68%. On the other hand, the size of the training data decreases from 6.7K pairs to fewer that 700 pairs. Interestingly, Table 5 shows that as we vary the threshold, the performance of the trained model increases till == 0.5k and then decreases. In other words, from T T 0.1k to r = =0.5k the quality of the preference data (or the accuracy margin) takes precedence over the quantity, improving downstream performance by 1.92%. However, when we set T 0. 7k, we end up with fewer than 700 pairs to train which we suspect is insufficient (in terms of both data size and diversity) to train model with 8B parameters.
larger models the substantial improvement in solving easy puzzles with SCPO (up to 10.3%).
5. Ablations and Analysis
Importance of weighted ScPO loss. While the results in Section& 4 are obtained using the weighted Lseoo loss that a function of consistency, here we compare SCPO using an unweighted loss. More specifically,
train using the same preference dataset created based on self-consistency of responses, but with =(x)=1 in the Lsero loss. In Table 4, we observe that across datasets and iterations, the weighted loss consistently outperforms the unweighted version. The improvement in accuracy is even more pronounced for the first iteration of training Mi, yielding an improvement of 2.5% in accuracy on GSM8K and 1. 1.44% on MATH with greedy inference. Even in the second iteration, M2 models trained with ScPO outperform their unweighted counter- parts by roughly 1% on both GSM8K and MATH. This indicates that it is better to take the amount of votes into account when optimizing for consistency, as this indicates confidence in the chosen and rejected labeling.
Models become more consistent across iterations. In Figure 2, we analyze how the degree of model consistency varies across iterations. To this end, we measure the vote share V(y+)/ )/k of the most consistent response, i.e., chosen response in self-consistency of models trained using unsu- pervised ScPO. From Figure 2, we conclude that ScPO training increases the consistency of models with each train- ing iteration across different tasks. We suspect this finding stems from three contributing factors: (i) with increasing iterations models become more accurate (Section 4); (ii) ad- ditional rounds of meferencce-ppimizatie diversity (Kirk et al., 2024); and (iii) training with ScPO effectively distills the SC distribution into the model's 's single- sample distribution, Additionally, we find that models are more consistent on tasks with higher test accuracy, i.e., on
Majority Vote Share (%)
70
Mo
60
0
Mi
M2
50
40
30
20
decreases model
0
GSM8K
MATH
ZebraLogic
Figure 2. Vote share (%) of the most consistent response: P(y+)/k
increases with iterations across all datasets.
7

---

Self- Celf-Consistency Preference Optimization
GSMBK
quires human judgements of LLM generations (Cobbe et al., 2021; UJsato et al., 2022; Lightman et al., 2024), making it similarly expensive. Our work focuses on the setting with- out access to gold solutions or final answers, which remains largely unaddressed. While other works such as She et al. (2024); Yuan et al. (2024); Rosset et al. (2024); Tran et al. (2023) geared towards general instruction following tasks (as opposed to reasoning tasks specifically) circumvent the need for human-annotater labels in the dataset by using the model itself to score the responses, these works demonstrate only modest gains in the context of reasoning tasks.
SC
7.8%
25.9%
66.3%
Ammo
19.1%
80.3%
MATH
8%
3%
49.8%
Ammo
6%
ZebreLogic
SC
16.0%
17.8%
66. 2%
Armo
10.5%
53,9%
Metric(Correct)- < Metric/WWong) Metric(Correct) Metric(Wrong)
Tied Metrics
Figure 3. Comparing the quality of metrics: self-consistency (SC) and ArmoRM to distinguish between correct and incorrect re- sponses on all datascts.
Consistency in LLMs. Self-consistency (Wang et al., 2023) relies upon the intuition that sampling several re- sponses, some of which lead to the same answer, lends higher certainty that the consistent answer is the correct one. Application of self-consistency at inference time has enabled performance improvements
Comparison of self-consistency to RMs. Our results in Section 4 show that models trained with unsupervised ScPO outperform models trained with IRPO using ArmoRM to build preference pairs. To study this further, we conduct additional analysis by measuring the ability of the two meth- ods to distinguish between correct and incorrect responses, comparing the methods to gold labels in Figure 3. We find that ArmoRM consistently has more incorrect orderings of pairwise preferences (the chosen is incorrect and the rejected correct) than ScPO across all three datasets (shown in red). This added noise in training may be major factor as to why IRPORM performs poorly compared to ScPO UPPOUnsup Unsup- On the other hand, self-consistens ber of ties, i.e., when the chosen and rejected answers get the same number of votes; these are ignored in ScPO's loss since w(x) (zz===)))) =0. Lastly, we find in the outoo--distriuution set- ting of ZegraLogic,, self-consistency outperforms ArmoRM with 12.3% more correct orderings of pairwise preferences (shown in green in Figure 3).
a number of domains like math (Wang al., 2023), code generation (Shi et al., 2022; Li et al., 2022; Chen et al., 2018), and even open-ended tasks like summarization and question answering (Chen et al., 2024a). In this work, we explore using self-consistency at training time for reasoning tasks, constructing preference pairs according to the self-consistent final answer. While Huang et al. (2023) also use self-consistency to finetune models without access to gold labels via NLL loss, we em- ploy a preference optimization loss function that is weighted according to the consistency of an answer. Intuitively, the consistency of an answer is reflection of the model con- fidence, and several prior works have demonstrated that leveraging model uncertainty can lead to faster convergence and improved performance (Gal & Ghahramani, 2016; Kr- ishnan & Tickoo, 2020; Corbière al., 2019). Concurrently with this work, Jiao et al. (2025) propose training models on freebakkfe from test cases, wherein they employ self-consistency to construct the test cases itself. However, we note that our work additionally shows the utility of self- consistency in generating new problems to augment the seed data (Section 4) as well as in our weighted loss function (Table in Section 5).
results in a greater num-
6. Related Work
Iterative Training of LLMs. Iterative training or self- training has shown meaningful improvements in a number of domains such as safety (Bai al., 2022), multilingual reasoning (She al., 2024), and evaluation (Wang al., 2024b). Because LLMs often struggle with both generating and validating solutions to complex reasoning tasks, prior works on training LLMs for complex problem-solving tasks largely rely on human-annotater (gold) final answers (Ze- likman et al., 2022; Chen et al., 2024b; Pang et al., 2024) or access to an external reward model that performs well on the underlying task (Singh et al., 2024: Dong et al., 2023). However, both these classes of approaches suffer from their own shortcomings. Firstly, manually annotating or verifying the final answer requires working through the solution step- by-step, making it especially resource-intensive for complex mtlti-step problems. Training strong reward models for such reasoning and problem-solving tasks also often re-
7. Conclusion
In this paper, we introduced Self-Consistenge Preference Optimization (ScPO). ScPO leverages the concept of self- consistency, usually employed only at inference time, to improve the self-training of large language models. By iteratively optimizing to prefer consistent answers to in- consistent ones, ScPO achieves significant improvements over traditional reward model training without the need for additional gold labels. Our experiments demonstrate the efficacy of ScPO on various reasoning tasks, including GSM8K, MATH, and ZebraLogic, where in the latter it outperforms several larger state-of-the-an language models.
8

---

Self-Consistency Preference Optimization
We also showed that ScPO works well in semi-supervised setups with access to some gold labels, in addition to labeled inputs improving performance further. These results highlight the potential of ScPO to improve self- alignment across reasoning tasks - domain that prior self- alignment methods still struggle with. Future work could extend SCPO to tasks where a single final answer cannot be easily parsed (e.g., summarization) through universal self-consistency (Chen et al., 2024a). While we explore consistency according to several models (Llama-3 and 3.1 8B, Base and Instruct), future work could also investigate consistency according to a suite of other models and tasks.
ated tests. In The Eleventh International Conference on Learning Representations, 2023.
Chen, X., Liu, C.,, and Song, D. Execution-guided neu- ral program synthesis. In International Conference on Learning Representations, 2018.
Chen, X., Aksitov, R., Alon, U., Ren, J., Xiao, K., Yin, P., Prakash, S., Sutton, C., Wang. X., and Zhou, D. Universal self-consistency for large language models. In ICMI, 2024 Workshop on In-Context Learning, 2024a. URL https: / //wpnreeveew... net forum?rrummmm ?irrummmiiiissss jHi.isssHHFFnn JHF7nAN.
Chen, Z., Deng, Y., Yuan, H., Ji, K., and Gu, Q. Self- play fine-tuning converts weak language models to strong language models. In Forty-first International Confer- ence on Machine Learning, 2024b. URL https: openreview net forum? www.iddo44.c........
Acknowledgements
We sincerely thank Ilia Kulikov, other members of the RAM team at FAIR, well the anonymous reviewers for their valuable feedback on the paper. Part of this work was done during an internship at Meta FAIR and was partially sup- ported at UNC by NSF-CAREER Award 1846185, NSF-AI Engage Institute DRL-2112635, DARPA Machine Com- The views contained in this article are those of the authors and of the funding agencies.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
monsense (MCS) Grant
Corbière, C., Thome, N., Bar-Hen, A., Cord, M., and Pérez, P. Addressing failure prediction by learning model con- fidence. Advances in Neural Information Processing Systems, 32, 2019.
Impact Statement
This work presents a new training algorithm that uses self- consistency for training large language models on math and logical reasoning tasks without the need for gold labels. The outputs produced by models trained with ScPO may exhibit undesirable behavior similar the base model and have the same potential for misuse as other fine-tuned LLMs (Wei- dinger et al., 2021). Hence, more studies are needed to evaluate and mitigate such biases in LLMs.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. RAFT: Reward ranked finetuning for generative founda- tion model alignment. Transactions on Muchine Learn- ing Research, 2023. ISSN 2835-8856. URL https: / //openreviee net forum? d=m ddddd1mm0mmmmmmmm
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A., al. The llama 3 herd of models. arXiv preprint arXiv:2407.2118.1 2024.
References
Anthropic. The claude model family: Opus, sonnet, haiku. 2024.
Dziri, N., Lu, X., Sclar, M., Li, L.,, Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Le Bras, R., et al. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36, 2024.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.0586.. 2022.
Fischler, M. A. and Bolles, R. C. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381-395, 1981
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P.. et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307. 15217, 2023.
Gal, Y. and Ghahramani, Z. Dropout bayesian ap- proximation: Representing model uncertainty in deep learning. In Balcan, M. F. and Weinberger, K. Q. (eds.), Proceedings of The 33rd International Conference on Ma- chine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1050-1059, New York, New
Chen, B., Zhang, F., Nguyen, A., Zan, D.,. Lin, Z., Lou, J.-G., and Chen, W. Codet: Code generation with gener-

---

Self-Consistency Preference Optimization
York, USA, 20-22 Jun 2016. PMLR. URL URL https:/: proceedi ngs mlr www..press/v....m / htmml.
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. Competition-level code generation with alpha- code. Science, 378(6624):1092-18
Hendrycks, D., Burns, C.,, Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
2022.
Liang, X., Song, S., Zheng, Z.,, Wang, H., Yu, Q., Li, X., Li, R.-H., Xiong, F., and Li, Z. Internal consistency and self-feedback in large language models: A survey. arXiv preprint arXiv:2407. 14507, 2024.
Huang, J., Gu, S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. Large language models can self-improve. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods Nat- ural Language Processing, pp. 1051-1068, December 2023. URL https: / www.aaaalatoooooooooooooo wrg.o//....gg/....... emnl p-ma inninn.. 67.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024.
Mahan, D., Van Phung, D., Rafailov, R., Blagden, C., Lile, N., Castricato, L., Fränken, J.-P., Finn, C., and Albalak, A. Generative reward models. arXiv preprint arXiv:2410. 12832, 2024.
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., and Zhou, D. Large language mod- els cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttppneeeeiw..../////////// net id=IkmD3 EKBPO.
Pang, R. Y., Yuan, W., He, H., Cho, K., Sukhbaatar, S., and Weston, J. Iterative reasoning preference optimization. Advances in Neural Information Processing Systems, 37: 116617-116637. 2024.
Jiao, F., Guo, G., Zhang, X., Chen, N. F., Joty, S., and Wei, F. Preference optimization for reasoning with pseudo feedback. In International Conference Learning Rep- resentations, 2025.
Prosser, P. Hybrid algorithms for the constraint satisfac- tion problem. Computational intelligence, 9(3):268-29) 1993.
Kabra, A., Rangreji, S., Mathur, Y., Madaan, A., Liu, E., and Neubig, G. Program-aided reasoners (better) know what they know. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies ( (Volume 1: Long Papers), pp. 2262-2278, 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36. 2024.
Rosset, C., Cheng, C.-A., Mitra, A., Santacroce, M., Awadal- lah, A., and Xie, T. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03711 2024.
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Ham- bro, E.,, Grefenstette, E., and Raileanu, R. Understanding the effects of rlhf Ilm generalisation and diversity. In The Twelfth International Conference Learning Repre- sentations, 2024.
She, S., Zou, W., Huang, S., Zhu, W., Liu, X., Geng, X., and Chen, Mapo: Advancing multilingual reasoning through aatitiinguallliiggmene.-e.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Ad- vances in neural information processing systems, 35: 22199-22213, 2022.
optimiza- tion. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 10015-10027, 2024.
Krishman. R. and Tickoo, O. Improving model calibration with accuracy versus uncertainty optimization. Advances in Neural Information Processing Systems, 33:18237- 18248, 2020.
Shi, F., Fried, D.,, Ghazvininejad, M., Zettlemoyer, L., and Wang, S. I. Natural language to code translation with execution. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3533-3546, 2022.
I Lambert, N., Pyatkin, V., Morrison, J., Miranda, I.., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint 4731v 2024.
Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P., Garcia, X., Liu, P. J., Harrison, J., Lee, J., Xu, K., Parisi, A. T., Kumar, A., Alemi, A. A., Rizkowsky, A., Nova, A., Adlam, B., Bohnet, B., Elsayed, G. F,, Sedghi,
10

---

Self-Consistency Preference Optimization
H., Mordatch, I., Simpson, I., Gur, Snoek, J., Pen- nington, J., Hron, J., Kenealy, K., Swersky, K., Mahajan, K., Culp, L. A., Xiao, L., Bileschi, M., Constant, N., Novak, R., Liu, R., Warkentin, T., Bansal, Y., Dyer, E., Neyshabur, B., Sohl-Dickstein, J., and Fiedel, N. Beyond human data: Scaling self-training for problem-solving with language models. Transactions on Machine Learn- ing Research, 2024. ISSN 2835-8856. URL https: :// openreview net. forum? d=yNNgGG Ex- pert Certification.
Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self- consistency improves chain of thought reasoning in lan- guage models. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: / //openreview.oo http://www.e/foooummiieee F
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F,, Chi, E., Le, Q. V., Zhou, D., et al. Chain-of of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems. 35:24824-24837, 2022.
Somers, R. H. A new asymmetric measure of association for ordinal variables. American sociological review, pp- 799-811, 1962.
Weidinger, L.., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-.., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359 2021. URL https:: hatp:://arxiv.ooo wtgg/aaie abs/ 2112. 04359.
Team, G., Riviere M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, I.., Mesnard, TT,, Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.001 18, 2024.
Xiong, M., Hu, Z., Lu, X., LI, Fu, J., He, J., and Hooi, B. Can LLMs express their uncertainty? an empirical evalu- ation of confidence elicitation in LLMs. In The Twelfth International Conference on Learning Representations, 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Moly- bog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subra- manian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Sto- jnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models, 2023. URL org/ abs 2007.00300 09288.
Xu, J., Lee, A., Sukhbaatar, S., and Weston, J. Some things are more cringe than others: Preference opti- mization with the pairwise cringe loss. arXiv preprint arXiv:2312. 16682, 2023.
Yu, L., Jiang, W., Shi, H., YU, J., Liu, Z., Zhang, Y., Kwok, J., Li, Z., Weller, A., and Liu, W. MetaMath: Bootstrap your own mathematical questions for large language mod- els. In The Twelfth International Conference on Learning Representations, 2024.
Yuan, W., Pang, R. Y., Cho, K., Li, X., Sukhbaatar, S., Xu. J., and Weston, J. Self-rewarding language mod- els. In Forty-first International Conference Machine Learning, 2024. URL https: openreview. net/ forum? O1p
Tran, H., Glaze, C., and Hancock, B. Iterative DPO align- ment. Technical report, Snorkel A1, 2023.
Uesato, J., Kushman, N., Kumar, R., Song, F,, Siegel, N., Wang, L., Creswell, A., Irving, G., and Higgins, Solv- ing math word problems with and outcome-base based feedback. arXiv preprint arXiv:2211.14277 2022.
Zelikman, E., Wu, Y., Mu, J., and Goodman, N. STaR: Boot- strapping reasoning with reasoning. Advances in Neural Information Processing Systems, 15:15576-55888 2022.
Zhang, L., Hosseini, A., Bansal, H., Kazemi, M., Ku- mar, A., and Agarwal, R. Generative verifiers: Re- ward modeling as next-token prediction. arXiv preprint 0rXhv:2408.15240. 2024.
Wang, H., Xiong, W. Xie, T., Zhao, H., and Zhang, T. Inter- pretable preferences via multi-objective reward modeling and mixture- of experts. In Findings of the Association for Computational linguistics: EMNLP 2024, pp. 10582- 10592, 2024a.
A. Relationship between Consistency and Accuracy
Wang, T., Kulikov, I., Golovneva, O., Yu, P., Yuan, W., Dwivedi- Yu, J., Pang, R. Y., Fazel-Zarandi, M., Weston, J., and Li, X. telf-taught evaluators. arXiv preprint arXiv:2408. 02666. 2024b.
Level of consistency or vote share correlates with accu- racy. We observe that the degree of consistency, or vote

---

Self- Self-Consistency Preference Optimization
share, is positively and strongly correlated with accuracy. This relationship is evidenced in Table by high rank order correlation for all three datasets, determined by Somer's s D (Somers, 1962). which measures the degree of association between two possibly dependent variables. This association is lowest for MATH, likely because the chal- lenging nature of this task makes difficult for the model to produce consistent answers.
Table 7. Somers' computed between Acc(y) and V(y) for y {yt. L... the most and least consistent responses, test set for different values of k.
Dataset / Somer's D k = 2
4
k 16
GSM-8K
0.39
0.65
0.80
0.89
ZebraLogic
0.66
0.82
0.92
0.93
Table 8, GSM8K zero-shot accuracy after training Llama-3.1 Base 8B with ScPO and baselines, using greedy self- consistency (SC)-based inference.
Table Somers' D computed between Acc(y) and Vyy) for E {v" } on test set.
Dataset
Somers' D
Method
Iter. Train Data (K)
Test Acc. (%)
GSM8K
0.80
Seed / Gen.
Greedy SC (8-way)
MATH 0.68
ZebraLogic 0.92
withour access to gold labels Seed model (zero-shot) Mo
/
-/-
43.14
59.59
Furthermore, we measure the impact of the number of sam- ples used measure consisteenc (k) on its Somer's D correlation with correctness in Table 7. The results indi- cate that (i) lower values of k (e.g. k 2/4) have lower correlation with correctness accuracy which we find is because of fewer instances where any answer gets multiple votes; (ii) while larger values of l: = 16 yield slightly higher correlations, we prioritize computational efficiency in the data generation phase, and use sufficiently large value of k- in addition filtering and weighted loss in ScPO.
IRPORM
M1
6.5
6.5/-
58.60
73.01
M2
6.7
60.04
72.19
LMSI
M1
6.7/5.7
48.75
65.71
M
6.3/4.8
52.39
60.42
ScPOUmepp.
Mi
6.7/5.7
61.64
71.95
M2
5.5/4.9
64.22
75.13
with access gold labels
/
IRPOcold
M1
5,6 /-
60.05
76.04
M2
5.8 /-
65.50
79.61
SCPO
Mi
5.6"
5.4
65.60
79.08
M2
5.21
/4.9
68.46
79.75
Table 9. MATH zero-shot accuracy after training Llama-3.1 Base 8B with ScPO and baselines, using greedy self-consistency (SC)- inference.
B. Transduction During Inference
Bootstrapping preference pairs from test queries further boosts performance. In our primary experiments, we re- port results for two rounds of iterative training. However, as shown in Table 10, introducing third round of ScPO yields only marginal improvements, with gains of less than 1 5% over the second round. To address this saturation, we explore generating new problems and building preference pairs using the queries from test split as exemplars instead of the train split. This strategy results in more substan- tial improvements (+1, (+1.44% for GSM8K), as it enables the model to better adapt to the unique characteristics of the more substantial improvements
Method
Iter., Train Data (K)
Test Acc. (%)
Seed Gen.
Greedy SC (8-way)
without access gold labels Seed model (zero-shot)
Mo
/
15.70
24.62
IRPORM
M1
6.2/
20.68
27.32
M2
6.6/-
20.74
25.88
LMSI
M1
0.9/0.9
16.26
24.38
M2
1.0/13
15.94
22.60
ScPO Unaup.
Mi
0.9/0.9
19.38
27.74
M2
1.41.77
23.20
30.10
with access to gold labels
test set. For MATH, when using SC accuracy, resulting in an improvement bump of 1 1.26%. We note that ZebraLogic is excluded from this analysis, as it only provides test samples.
IRPOGold
M1
2.71
22.40
31.64
M2
3,2 /-
22.86
32.30
ScPOsami-Sup.
Mi
2.7
2.7
22.98
32.18
M2
3,22
24.36
32.64
of SCPOSemi improve the greedy test accuracy of the seed model by 25.32% and 8.66% on GSM8K and MATH, respectively; while two iterations ScPOUnsup. boost the greedy accuracy of the seed model by 21.08% on GSM8K and 7.5% MATH dataset.
C. Results on Math Reasoning with Llama-3.1
We now repeat the math reasoning experiments in Sec- tion 4.1 with Llama-3.1 Base 8B and find that while the ab- solute performance increases, the relative trends among the baselines remain the same - with ScPO as the most performant unsupervised technique and ScPOsemi-Sup. yielding the overall highest accuracy on GSM8K and MATH. In Tables and 9, we observe that two iterations
12

---

Self- Self-Consistency Preference Optimization
Table 10. Training Ma by bootrapping from questions in the train and set. On GSM8K, bnotstrap 8.7K, 5.8K pairs using train, and test problems, respectively. On MATH, we build 4.4K, and 4.2K preference pairs using train and test problems, respectively.
GSM8K Ace.
MATH Acc.
Method
Greedy
SC (8-way)
Greedy
SC (8-way)
Mo
41.17
58.80
14.46
18.20
M1 w/ ScPOUnews Uneup.
61.03
71.49
17.36
25.70
M2 w/ ScPO Uneup.
63.91
71.11
19.72
24.58
Ms w/ ScPOUnews Uneup.
64.21 65.35
70.81
19.76
24.66
M3 w/ ScPOussu. on test queries
70.96
20.00
25.84
D. Prompts
We provide task-specific prompts used for both generating new problems and for generating candidate solutions.
Response Generation: ZebraLogic
Example Puzzle:
There are 3 houses, numbered to from left right, from across the street. Each house is occupied different person. Each house has unique attribute for each of the following characteristics:
Each person has unique name: 'Peter', 'Eric", 'Amoold' Each person has unique favorite drink: 'tea', 'water', 'milk"
## Clues:
Peter is in the second house.
2. Amold is directly left of the one who only drinks water.
3. The one who only drinks water is directly left of the person who likes milk.
Answer to the Example Puzzle:
"Given Clue 1, we know Peter is in House According to Clue 2, Arnold directly left of the one who only drinks water. The person in House cannot be on the left of anyone, so Arnold must in House 1. Thus, Peter drinks water, and Eric lives in House 3. Then, according to Clue 3, Eric drinks milk. Therefore, Arnold drinks tea.", "solution": { "House I": { "Name": "Arnold", "Drink": "tea" }. "House 2": "Name": "Peter", "Drink": "water" }, "House 3"; "Name": "Eric", "Drink"; "milk" }
"reasoning":
Puzzle Solve: {puzzle}
Prompt: Now please solve the above puzzle. Present your reasoning and solution in the following json format: [json template)
Response Generation: GSM8K
Prompt: Answer the following question step-by-step. When you are ready, place the final answer in new line as #UUH < number >,
Q: {question}
A: Let's think step by step.
13

---

Self- Belf-Consistency Preference Optimization
Response Generation: MATH
Prompt: Answer the following question step-by-step. When you are ready, place the final answer in a new line as: The final answer is (boxed{( < your answer>>$ Q: {question}
A: Let's think step by step.
Query Generation:
ZebraLogic
Example Puzzle:
Attributes to Change: I"Name", "Drink"]
There 3 houses, numbered I from left right, as seen from across the street. Each house is occupied by different person. Each house has unique attribute for each of the following characteristics: Each person has unique name: 'Peter', 'Eric', 'Arnold', Each person has unique favorite drink: 'tea", 'water', 'milk"
## Clues:
Peter is in the second house.
Arnold is directly left of the one who only drinks water.
The one who only drinks water is directly left of the person who likes milk.
Answer:
Let's change the "Name" and "Drink" attributes of the given puzzle to create a new puzzle. There are 3 names and drinks involved Mentions of "Name" changes from 'Peter', 'Eric', 'Aroold' to mentions of "Name": 'Molly', 'Shannon', 'Kelly" respectively. Instead of "Drink" as the attribute, let's their "Food" preferences as the attribute. So mentions of "Drink" changes from 'tea', 'water', 'milk' mentions of "Food": 'pizza', 'burgers', "fries" respectively. Now, changing the language of the puzzle and clues we get,
New Atribute Map: ("Name": "Name", "Drink": "Food")
Puzzle:
"* There are 3 houses, numbered to 3 from left to right, as seen from across the street. Each house is occupied by different person. Each house has a unique attribute for each of the following characteristics:
Each person has unique name: 'Molly", 'Shannon', 'Kelly'.
Each person has unique favorite food: 'pizza', 'burgers', 'fries'
## Clues:
1. Molly is in the second house.
2. Kelly is directly left of the one who only eats burgers.
3. The one who only eats burgers is directly left of the person who likes fries.
Puzzle rephrase:
Auributes to Change: (attributes dict)
(input puzzle]
Rephrase the above puzzle by changing only the attributes above. ALWAYS mention the "New Attribute Map" and enclose the new puzzle within " ". Aside from these attributes keep the logic of the puzzle as similar as possible. Similar to the example above, give your reasoning before rephrasing the puzzle.
Prompt:
14

---

Self- Self-Consistency Preference Optimization
Query Generation: GSM8K and MATH
Q: shot question }
Q: few-shot question 2}
Q: shot question }
Q: (few-shot question 4]
Prompt: Based the examples above, generate ONE solvable math word problem with similar diffi culty. Note that all the information needed to solve the problem should be included in the question. Output the question and nothing else.
Q:
15


Reviews:
[{"review_id": "RtJBV5nBhV", "reviewer": "Reviewer_XcBF", "summary": "The paper introduces Self-Consistency Preference Optimization (ScPO), an unsupervised method for training LLMs to improve reasoning tasks. ScPO leverages the concept of self-consistency—traditionally used at inference—to iteratively optimize models by preferring answers with high consensus over inconsistent ones. \n\n- A weighted loss function that prioritizes high-confidence preference pairs based on vote margins.\n\n- Semi-supervised extensions combining labeled and unlabeled data.\n\n- Experiments on GSM8K, MATH, and ZebraLogic showing ScPO outperforms supervised baselines (e.g., +22.74% on GSM8K) and larger models (e.g., Llama-3 8B trained with ScPO surpasses Llama-3 70B on ZebraLogic).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your detailed review and questions. Please find our response below your comments:\n\n> Significance of Results\n\nOur test sets include \\>= 1K samples for GSM8K and ZebraLogic, and 5K problems for MATH. In our primary unsupervised paradigm with greedy decoding, ScPO **consistently** outperforms the IRPO (RM) and LMSI baselines across **three datasets and two base models** *(Tables 1-3, 8-9)*. The unsupervised baselines exhibit high variance across different datasets and models. For example, while LMSI is the second-best unsupervised method behind ScPO (by 7.2%) with Llama-3 8B in Table 1, it performs significantly worse with Llama-3.1 8B on the same GSM8K dataset in Table 8, trailing ScPO by 11.83% and IRPO RM by 7.6%. Additionally, IRPO RM is the least effective on GSM8K and ZebraLogic with Llama-3 8B. Given that all methods have similar computational and data budgets (see Sec 4), this strongly supports the effectiveness of ScPO for reasoning tasks.\n\n> Configuration of Baselines\n\nWe conduct zero-shot evaluation to assess the improvement in the model's reasoning abilities from zero-shot instruction training. As shown in Tables 1-3, 8-9, using 8-way self-consistency (SC) on the test set, SC improves the performance of all baselines. However, models trained with ScPO still achieve the highest performance (after SC) in both supervised and semi-supervised settings across two datasets and two model families. Therefore, we expect similar results for other inference-time variants, such as weighted SC or Best-of-N sampling.\n\n> Vote-margin and Correctness\n\nWe reiterate that the intuition behind self-consistency is that model errors are generally random, making it unlikely to repeat the same incorrect answer across different samples (L 28-36). This concept, effective in various domains and predating its use in LLMs (e.g., RANSAC by Fischler & Bolles, 1981), is supported by popular LLMs showing improved accuracy with majority voting in reasoning tasks (e.g., e.g. DeepSeekMath, Gemini1.5). Empirically, our results (Appendix A, Table 6\\) demonstrate a strong correlation between consistency and correctness across datasets, indicating LLMs are not widely over-confident in math and logical reasoning tasks. This aligns with prior findings that LLMs are well-calibrated ([Kadavath et al. 2022](https://arxiv.org/abs/2207.05221)) and that self-consistency reliably estimates model confidence ([Xiong et al. 2023](https://arxiv.org/abs/2306.13063), [Kabra et al. 2023](https://arxiv.org/pdf/2311.09553)). In domains with prevalent overconfidence, self-consistency could be combined with inference-time calibration techniques ([Zhao et al. 2021](https://arxiv.org/abs/2102.09690)).\n\n> Number of Iterations\n\nRefer to Table 7 in Appendix B, where we observe that math reasoning performance largely plateaus after the second iteration, with less than a 1-point gain in the third iteration. However, the same table shows that sampling questions from a different distribution—using the distributions of questions (*without answers*) from the test set to sample additional related problems (L81-98)—yields further improvements in the third iteration on top of the M2 models (L270-274).\n\n> Sensitivity to Threshold τ\n\nAs noted in L352-378 and Table 5, the initial threshold is based on the training data quality (Margin: Acc(preferred) \\- Acc(rejected)) and the number of instances meeting the cutoff, i.e., Vote(preferred) \\>= τ. Even at lower thresholds, Table 5 shows we can improve the base model's performance. While the training data quality and sample size depend on the LLM's inherent consistency for a specific domain/dataset, we applied the same method to train the Llama-3.1 8B model in Appendix C and achieved significant gains without tuning hyperparameters for the new base model.\n\n> Dealing with Ambiguous Answers\n\nWe reiterate that ScPO is designed to improve model’s reasoning performance with unsupervised or semi-supervised training. Different from general instruction following tasks such as creative writing with subtle human preferences, **on reasoning tasks we desire to prefer correct solutions and disprefer incorrect ones, making the domain relatively less ambiguous** (please let us know if you have any specific dataset in mind). Nevertheless, ScPO can be combined with techniques to measure consistency in generative settings, such as universal self-consistency (L432-436) or executable programs for correctness ([Lambert et al. 2024](https://arxiv.org/abs/2411.15124)). For example, in the ZebraLogic benchmark, we use a programmatic approach to measure consistency, where answers in complex \"json\" format require a symbolic function for comparison and multiple votes.\n\nWe hope our response has addressed all of your questions and will allow you to revisit your score. We are happy to answer any followup questions and requests you may have.", "claims_and_evidence": "Most claims are supported by empirical results. ScPO outperforms IRPO and LMSI baselines (Tables 1–3) but the improvement is not significant. But the evaluation just considers zero-shot accuracy, and I'm not sure whether other baseline models are using the best configurations, or if other models will perform better using inference-time weighted voting/reward model.", "methods_and_evaluation_criteria": "Make sense. The benchmark datasets GSM8K and Math are adequate.", "theoretical_claims": "No theoretical proofs are provided. The loss function is empirically justified but lacks formal analysis (e.g., convergence guarantees or why vote margins correlate with correctness).", "experimental_designs_or_analyses": "Strengths: Ablation studies (Tables 4–5) and correlation analysis (Appendix A) strengthen validity.\n\nWeaknesses:\n- The choice of 2 iterations is under-explained (mentions some related work but lacks systematic analysis).\n- Threshold τ is tuned on dev sets, but sensitivity to this hyperparameter is not thoroughly tested.", "relation_to_broader_scientific_literature": "ScPO builds on self-consistency and preference optimization, and may contribute to improve LLM's self-consistency for math reasoning.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "see above", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. How does ScPO generalize to tasks with ambiguous final answers?\n2. Could higher vote shares reflect over-confidence rather than accuracy? How is this risk mitigated?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TkG9zYs9Ox", "reviewer": "Reviewer_R9NR", "summary": "The paper introduces Self-Consistency Preference Optimization (SCPO), a novel approach to self-training large language models (LLMs) for complex reasoning tasks without requiring gold labels/solutions. SCPO extends the concept of self-consistency (typically used only at inference time) to create preference pairs during training by sampling multiple responses for each problem and identifying the most consistent vs. least consistent answers. The key innovation is a weighted preference optimization objective where weights depend on the vote margin between chosen and rejected responses. The paper also presents a semi-supervised variant that incorporates gold labels when available, further improving performance.\n\nThe authors validate the method on reasoning (GSM8K, MATH) and logical reasoning (ZebraLogic) benchmarks, showing that unsupervised SCPO nearly matches supervised preference optimization with gold labels, while semi-supervised SCPO outperforms fully supervised baselines. Most notably, on ZebraLogic, SCPO helps Llama-3 8B outperform significantly larger models like Llama-3 70B and Claude-3 Haiku.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your extensive review and comments and are glad to see you appreciate the “creative application of self-consistency”, “impressive results”, and “thoughtful ablations”. Please find our detailed response to your comments below and let us know if you have any follow up questions.\n\n> Impact of number of samples (K)\n  \n**We measure the impact of the number of samples used to measure self-consistency on its Somer’s D correlation with correctness (as done in Table 6\\) for K=2,4,8,16.**\n\n| Dataset | K=2 | K=4 | K=8 | K=16 |\n| :---- | :---- | :---- | :---- | :---- |\n| GSM-8K | 0.39 | 0.65 | 0.80 | 0.89 |\n| ZebraLogic | 0.66 | 0.82 | 0.92 | 0.93 |\n\nThe results indicate that (i) lower values of K (e.g. K=2/4) have lower correlation with correctness or accuracy which we find is because of fewer instances where any answer gets multiple votes; (ii) while larger values of K=16 yield slightly higher correlations, we prioritize computational efficiency in the data generation phase (L100-122), and use a sufficiently large value of K=8 in addition to filtering (L118-121) and a weighted loss (L141-149) for ScPO training.\n\n> Computational Overhead of ScPO: \n\nWe reiterate that **all our baselines including IRPO, LMSI, as well as ScPO have the same computational overhead by design** as we use similar size training datasets, same number of samples (K), and the same training hyperparameters. Specifically, to your point on generating training data, we note that this process is done once at the start of each iteration when using the LLM at inference-time, and *not during training*. Therefore, we can make use of popular strategies for speeding up LLM inference such as the [vLLM library](https://github.com/vllm-project/vllm), increasing batch-size, and utilizing multiple GPUs in parallel, making the data-generation process far less computationally demanding than the training itself. We will include this discussion in the paper.\n\n> Impact of ScPO on model diversity: \n \n**In Fig 2 of Sec 5, we visualize the vote share of the model's responses across iterations, and find that it increases across iterations for all datasets**, indicating a decrease in the number of unique answers (diversity). We suspect this is a consequence of RLHF training that has been well-documented in prior work ([Kirk et al. 202](https://arxiv.org/abs/2310.06452)4, [Murthy et al. 2024](https://arxiv.org/abs/2411.04427), Murthy et al. 2024\\) and is outside the scope of our study. At the same time, in Sec 4 (Tables 1-3 and 8-9), we report test accuracy after 8-way self-consistency and find that models trained with ScPO continue to benefit from diversity in generations via SC at test-time. \n\n> Number of Iterations: \n\n We refer you to Table 7 in Appendix B where we find that performance on math reasoning largely plateaus after the second iteration, with \\<1 point gain with a third iteration on training. However, in the same table we find that sampling questions from a different distribution, i.e., the distributions of questions (*without using the answers*) from the test set and using it to sample additional related problems (L81-98) yields additional improvements in the third iteration on top of the M\\_2 models (L270-274). Therefore, we believe that increasing the diversity of the seed problems, or after each iteration can be an effective way to delay performance saturation. Also, developing a RLHF method that does not diminish diversity can be a solution.\n\n> Open Ended Reasoning Tasks:  \n\nWe reiterate that ScPO is designed to improve model’s reasoning performance with unsupervised or semi-supervised training. Different from general instruction following tasks such as creative writing with subtle human preferences, on reasoning tasks we desire to prefer correct solutions and disprefer incorrect ones. Nevertheless, in such scenarios, we believe ScPO can be combined with techniques to measure consistency in more generative settings such as universal self-consistency (L432-436) or using executable programs to measure correctness ([Lambert et al. 2024](https://arxiv.org/abs/2411.15124)). Note that we use a similar programmatic or symbolic approach to measure consistency **for the ZebraLogic benchmark where the answer is in a complex “json” format that cannot be directly compared via exact string match and requires a symbolic function to measure equivalent answers and multiple votes**. Measuring consistency in more open-ended tasks (e.g. creative writing) is indeed under explored, and likely to be first worked out in inference time uses, which we leave for future work. We will expand on the final paper.", "claims_and_evidence": "The claims in this paper are supported by empirical evidence. The primary claim that SCPO improves reasoning abilities without access to gold solutions is demonstrated through significant performance gains on GSM8K, MATH, and ZebraLogic. The claim that unsupervised SCPO approaches the performance of supervised training is supported by results showing <1% gap in performance. The authors also provide evidence for the superiority of their weighted loss function through ablation studies that show 1-2.5% improvements over unweighted alternatives.\n\nThe paper includes analyses showing the correlation between vote share and accuracy, which validates the main assumption that consistency is a good proxy for correctness although this result was relegated to Appendix A.", "methods_and_evaluation_criteria": "The use of GSM8K and MATH as benchmarks for mathematical reasoning and ZebraLogic for logical reasoning is a sensible choice. The authors evaluate using both greedy decoding and self-consistency inference, showing improvements in both settings. The comparison against multiple baselines (zero-shot CoT, IRPO_RM, LMSI, IRPOGold) is broad and provides enough evidence for the effectivenetss of SCPO.\n\nIt would be good to use benchmarks thay go beyond math and logical reasoning to show the versitility of SCPO but those are notoriously difficult to come by. The focus on GSM8K, MATH, and ZebraLogic is understandable.", "theoretical_claims": "The paper doesn't present formal theoretical claims or proofs. It focuses on the presentation and empirical validation of their new method.", "experimental_designs_or_analyses": "I examined the core experiments in the paper and found no issues:\n1. Baseline comparisons on reasoning datasets (Tables 1-3): sound evaluation of the core points of the paper.\n2. Weighted loss ablation (Table 4): Direct comparison between weighted and unweighted versions.\n3. Consistency analysis (Figure 2): Measurement of vote share increases across iterations.\n4. Threshold filtering experiment (Table 5): Testing different consistency thresholds.\n5. Preference accuracy analysis (Figure 3): Comparison of self-consistency vs. reward models in correctly ordering preferences.\n\nThe hyperparameters ar well-documented (learning rate, epochs, temperature settings). The only limitation is the lack of ablation on the number of samples (k) used for voting, which would help understand computational efficiency tradeoffs. Otherwise, the experiments provide strong evidence for the paper's claims.", "relation_to_broader_scientific_literature": "SCPO extends Wang et al.'s (2023) self-consistency concept from inference to training, presenting a clean and practical strategy for self-improvement in LLMs. While methods like LMSI (Huang et al., 2023) also use self-consistency for unsupervised training, SCPO builds on them improving the end performance. The approach relates to self-alignment work like Yuan et al. (2024), but addresses their limitation in reasoning tasks identified by Huang et al. (2024).", "essential_references_not_discussed": "The paper covers the most relevant literature well. I don't see any critical omissions that would significantly impact the understanding of the work's context or contributions. The authors cite key papers on self-consistency, preference optimization, and self-training approaches.", "other_strengths_and_weaknesses": "Strengths:\n- Clear presentation of the work makes understanding the paper very easy\n- Creative application of self-consistency to training rather than just inference\n- Impressive results on ZebraLogic, showing a smaller model can outperform much larger ones\n- Strong empirical validation across multiple datasets and model scales\n- Thoughtful ablation studies that justify design choices\n\nWeaknesses:\n- Limited discussion of computational overhead from generating multiple responses per query during training\n- Current evaluation limited to math and logic tasks with definitive answers; unclear generalizability to more open-ended reasoning tasks\n- While the paper shows that consistency correlates with correctness, deeper analysis of when/why this relationship might break down would strengthen the work", "other_comments_or_suggestions": "- It would be valuable to analyze how SCPO affects the diversity of reasoning paths across iterations\n- A more detailed analysis of computational requirements compared to baselines would help readers understand practical tradeoffs", "questions_for_authors": "1. Have you investigated the trade-off between computation costs and performance gains in SCPO? Specifically, how does the computational overhead of generating multiple responses during training compare to other methods, and how might this scale with larger models?\n\n2. The results show diminishing returns after the second iteration of SCPO. Do you have insights into whether this is due to fundamental limitations of self-consistency as a training signal, or might there be ways to continue improving with more iterations?\n\n3. For real-world applications, have you considered how SCPO might perform on problems where there isn't a single definitive correct answer, or where answer extraction is more challenging than in the studied tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JK2mvwY4BX", "reviewer": "Reviewer_Z9n6", "summary": "This paper considers self-alignment of LLMs, where the data consist of only prompts but not the ground truth. The authors propose to use the self-consistency to choose winning and losing samples, where the responses corresponding to the most/least final answers are considered as the winning/losing samples.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your review and for appreciating the “intuitive and simple” design of our method as well as our evaluation setup. \n\n> The choices are to certain extent reasonable, one concern here is that the self-improvement baseline, LSMI, is not quite up-to-date. Considering self-alignment as a quite active area, there should be more follow-ups suitable for serving as a baseline.\n\nWe are not aware of a more up-to-date follow-up of the LMSI baseline that is applicable to our unsupervised setting but please let us know if we missed something you had in mind. Furthermore, we note that two variants of IRPO as well as the 8B RM are fairly up to date since **IRPO was published in Neurips 2024, and the Armo RM (also released in mid 2024\\)** was among the best performing 8B reward model as per the RewardBench leaderboard at the time of development. \n\n> Another concern is that this work is very similar to \\[2\\]. See their Figure 2 for their method with self-consistency feedback. Since \\[2\\] is made public last Nov, I consider this as a concurrent work to this paper.\n\nWhile the “pseudo feedback from self-consistency” idea in \\[2\\] is concurrent and related to ScPO, we would like to identify the following key differences:\n\n- As shown in Fig 1 (left) and Sec 2 (L 81-99), we show that ScPO can be used to augment the initial seed training data with additional questions sampled from the same model. Furthermore, we show that self-consistency plays a crucial role in filtering for well-formed and answerable questions, which \\[2\\] lacks. The effectiveness of generating additional data can be seen from not only the unsupervised results, but also our results from the semi-supervised setting where ScPO outperforms the supervised IRPO (gold) baseline by up to 2.35% on GSM8K.  \n- Next, we also incorporate self-consistency in our weighted loss that adjusts the weight of a training instance based on the relative confidence, i.e. difference in vote share of chosen and rejected responses (L141-149). We demonstrate the importance of this weighted training loss in Table 4 of Sec 5\\. Thus, we argue that \\[2\\] is equivalent to the w(x)=1 baseline (only trained on seed data) which ScPO outperforms.\n\n That being said, we thank you for pointing out this concurrent work and we will cite it in future versions of our paper.", "claims_and_evidence": "The major claim is that self-consistency preference optimization improves self-alignment for reasoning task. This claim is quite intuitive and plausible. Their results served as empirical evidence for this claim.", "methods_and_evaluation_criteria": "The method itself is quite intuitive and simple, which is likely to be effective for unsupervised self-alignment, as self-consistency is well-recongnized for filtering samples.\n\nThe method and baselines are evaluated on popular reasoning datasets, GSM8K and MATH, which is quite standard hence I have no question about the evaluation.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The baselines are reward-model fine-tuning, and language model self-improving (LMSI) [1]. The choices are to certain extent reasonable, one concern here is that the self-improvement baseline, LSMI, is not quite up-to-date. Considering self-alignment as a quite active area, there should be more follow-ups suitable for serving as a baseline.\n\n[1] Large Language Models Can Self-Improve. https://arxiv.org/abs/2210.11610", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "Another concern is that this work is very similar to [2]. See their Figure 2 for their method with self-consistency feedback. Since [2] is made public last Nov, I consider this as a concurrent work to this paper. My positive evaluation is based on the consideration that both works are concurrent (so it did not hurt the score of my evaluation).\n\n[2] Preference Optimization for Reasoning with Pseudo Feedback. https://arxiv.org/abs/2411.16345v1", "other_strengths_and_weaknesses": "see above", "other_comments_or_suggestions": "see above", "questions_for_authors": "see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bNlFmtAewi", "reviewer": "Reviewer_rpC3", "summary": "This paper introduces SCPO, a novel training method for large language models (LLMs) that leverages self-consistency to improve performance on complex reasoning tasks without requiring gold labels. SCPO iteratively trains models to prefer consistent answers over inconsistent ones by generating multiple responses, selecting the most and least consistent ones as preference pairs, and optimizing a weighted loss function based on the model's confidence in these pairs. The method is evaluated on GSM8K, MATH, and ZebraLogic datasets, showing significant improvements over existing unsupervised baselines and closing the gap with supervised training methods. Key findings include that SCPO outperforms models trained with external reward models and self-improvement methods, and it can further enhance results when combined with supervised learning. The approach demonstrates the effectiveness of using self-consistency as a training signal for improving both the accuracy and consistency of LLMs on reasoning tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your detailed review and comments. We are also glad to see you found our paper to have “intuitive motivation” as well as “extensive experimental results”. Please find our in-depth response to your comments below.\n\n> Additional relevant baselines:\n\n Thanks for pointing these papers out to us, we contrast ScPO to these works as:\n\n- **Chen et al. 2024**: We argue that they are focused on **general instruction following, and their method is not proven to work in reasoning tasks** (as we do in ScPO) like math etc. Also, note that their main technical contribution is to add a length control factor to DPO’s implicit reward term. While “length bias” has been an issue with general instruction following tasks [(Dubois et al 2024](https://arxiv.org/abs/2404.04475)) as models spuriously prefer longer generations, preference evaluation for reasoning domains is more straightforward – preferring correct answers and rejecting the rest, so we expect length-controlled DPO to perform similarly to the standard DPO algorithm used in ScPO.  \n- **Kim et al. 2025**: At its core, this paper uses the **model's confidence on two generations corresponding to the same prompt, to construct preference pairs** with additional preference label smoothening for the bottom 10% annotations that are expected to be noisy. This stands in contrast to ScPO which uses self-consistency of predictions to generate preference annotations, and we compare the quality of the two metrics (confidence vs self-consistency) to curate preference data below to demonstrate the superiority of ScPO on reasoning tasks:  \n  - *Setup:* We incorporate Kim et al.’s strategy for preference data creation by using the model’s confidence. Specifically, we compute the confidence for each generation (from the same sample set used to compute SC) and create preference pairs by choosing the most confident response and rejecting the least confident response. Further, following Kim et al. (2025), we filter out the last 10% of training instances with the lowest difference in confidence of chosen and rejected responses (indicative of noise in annotation), and compare its Somer’s D correlation with accuracy with that of self-consistency (Table 6). \n\n| Dataset/Metric | Confidence (Kim et al. 2025\\) | Self-Consistency (ours) |\n| :---- | :---- | :---- |\n| GSM8K | 0.11 | 0.80 |\n| ZebraLogic | \\- | 0.93 |\n\n  - *Results:* On ZebraLogic, we find that the accuracy of the most confident answer is in fact 0 rendering Kim et al.’s method completely ineffective, whereas SC on the other hand correlates strongly with correctness. Similarly, on GSM8K we find that confidence score of generation shows little correlation with correctness, and is most likely not effective to generate preference training data. **The results indicate that across datasets self-consistency offers a better estimate of correctness of an answer, and therefore yields higher quality preference for training.** These findings are consistent with prior work that uses self-consistency to estimate model confidence for reasoning tasks ([Xiong et al. 2023](https://arxiv.org/abs/2306.13063), [Kabra et al. 2023](https://arxiv.org/pdf/2311.09553)).\n\n> Ablation with number of samples (K)\n\n**We measure the impact of the number of samples used to measure self-consistency on its Somer’s D correlation with correctness (as done in Table 6\\) for K=2,4,8,16.**\n\n| Dataset | K=2 | K=4 | K=8 | K=16 |\n| :---- | :---- | :---- | :---- | :---- |\n| GSM-8K | 0.39 | 0.65 | 0.80 | 0.89 |\n| ZebraLogic | 0.66 | 0.82 | 0.92 | 0.93 |\n\nThe results indicate that (i) lower values of K (e.g. K=2/4) have lower correlation with correctness or accuracy which we find is because of fewer instances where any answer gets multiple votes; (ii) while larger values of K=16 yield slightly higher correlations, we prioritize computational efficiency in the data generation phase (L100-122), and use a sufficiently large value of K=8 in addition to filtering (L118-121) and a weighted loss (L141-149) for ScPO training.\n\n> related works like \\[1, 2, 3, 4, 5\\] where selecting high-quality data or weighting techniques are already common in DPO-based methods\n\nThank you for pointing out these works. We haven’t discussed them in our paper because they are about weighting prompts based on their quality, removing noisy or unhelpful questions. In contrast, our work is about weighing solutions, giving less importance to solutions that are likely to be wrong. However, it could be helpful to discuss those works and their differences, which we will do in the final revision. \n\nWe hope these additional results and explanations address your questions and will allow you to revisit your score.", "claims_and_evidence": "The claims made in the submission are generally supported by clear and convincing evidence. The authors provide extensive experimental results across multiple datasets (GSM8K, MATH, ZebraLogic) demonstrating performance improvements over baselines.", "methods_and_evaluation_criteria": "The proposed methods (SCPO) and evaluation criteria make sense for the problem of improving LLM reasoning without gold labels. SCPO's approach of using self-consistency to generate training signals aligns well with the challenge of lacking labeled data.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental designs are mostly sound but could benefit from including additional relevant baselines such as [1, 2]. These would provide more comprehensive comparisons and strengthen the validation of SCPO's effectiveness.\n\n\n1. Chen, Changyu, et al. \"Bootstrapping language models with dpo implicit rewards.\" arXiv preprint arXiv:2406.09760 (2024).\n2. Kim, Dongyoung, et al. \"Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment.\" The Thirteenth International Conference on Learning Representations. 2025.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "Some recent work on self-improving/self-annotating alignment also utilizes LLMs for preference annotation, similar to the approach. These references could support the rationale behind this work. For example:\n\n1. Chen, Changyu, et al. \"Bootstrapping language models with dpo implicit rewards.\" arXiv preprint arXiv:2406.09760 (2024).\n2. Kim, Dongyoung, et al. \"Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment.\" The Thirteenth International Conference on Learning Representations. 2025.", "other_strengths_and_weaknesses": "Strengths:\n- The paper has intuitive motivation and relatively solid experimental validation.\n\nWeaknesses:\n- Limited baselines in experiments; should include more relevant comparison methods.\n- No ablation studies on the number of responses k, which is crucial for the reweighting term w(x).\n- The importance of the weighted SCPO loss is demonstrated in Table 5, but the discussion lacks references to specific related works like [1, 2, 3, 4, 5] where selecting high-quality data or weighting techniques are already common in DPO-based methods.\n\n[1] $\\beta$-dpo: Direct preference optimization with dynamic $\\beta$. NeurIPS, 2024.  \n[2] Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682.  \n[3] Reward difference optimization for sample reweighting in offline rlhf. EMNLP 2024.  \n[4] Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models. EMNLP 2024.   \n[5] Filtered direct preference optimization. EMNLP 2024.", "other_comments_or_suggestions": "see `Strengths And Weaknesses`", "questions_for_authors": "see `Strengths And Weaknesses`", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes an unsupervised method named ScPO, which can improve LLMs' ability of complex reasoning without requiring gold labels. 

Most reviewers acknowledged the motivation, the intuitiveness of the method, the rationality of the experiments, the validity of the results and writing quality of the submission. The authors make great efforts and address most of the concerns of the reviewers during the rebuttal phase. All the reviews tend to accept this paper during the reviewer-author discussion phase. 

Overall, this paper is valuable for the community. Therefore, I recommend accepting this paper.
Decision: Accept (poster)