Varying Shades of Wrong: Aligning LLMs with Wrong Answers Only
Authors: Jihan Yao
Wenxuan Ding
Shangbin Feng
Lucy Lu Wang
Yulia Tsvetkov
Venue: ICLR 2025 Poster (2025)
Keywords: alignment
preference learning
reasoning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: In the absence of abundant reliable annotations for challenging tasks and contexts, how can we expand the frontier of LLM capabilities with potentially wrong answers? We focus on two research questions: (1) Can LLMs generate reliable preferences among wrong options? And if so, (2) Would alignment with such wrong-over-wrong preferences be helpful? We employ methods based on self-consistency, token probabilities, and LLM-as-a-judge to elicit wrong-over-wrong preferences, and fine-tune language models with preference optimization approaches using these synthesized preferences. Extensive experiments with seven LLMs and eight datasets demonstrate that (1) LLMs do have preliminary capability in distinguishing various shades of wrong, achieving up to 20.9% higher performance than random guess; (2) Alignment with wrong-over-wrong preferences helps LLMs to produce less wrong and sometimes even outright correct answers, while improving overall model calibration. Code and data are publicly available at https://github.com/yaojh18/Varying-Shades-of-Wrong.

Published as a conference paper at ICLR 2025
VARYING SHADES OF WRONG: ALIGNING LLMs
WITH WRONG ANSWERS ONLY
Jihan Yao*! Wenxuan Ding Shangbin Feng"! Lucy Lu Wang Yulia Tsvetkov!
University of Washington Thhe University of Texas at Austin Allen Institute for Al (jihany2, shangbin] @ecss @cs. washington. wenxuandeutexas edu
ABSTRACT
In the absence of abundant reliable annotations for challenging tasks and con- texts, how can we expand the frontier of LLM capabilities with potentially wrong answers? We focus on two research questions: (1) Can LLMs generate reliable preferences among wrong options? And if so, (2) Would alignment with such wrong-over-wrong preferences be helpful? We employ methods based on self- consistency, token probabilities, and LLM-as-a-judge to elicit wrong-over-wrong preferences, and fine-tune language models with preference optimization ap- proaches using these synthesized preferences. Extensive experiments with seven LLMs and eight datasets demonstrate that (1) LLMs do have preliminary capa- bility in distinguishing various shades of wrong, achieving up to 20.9% higher performance than random guess; (2) Alignment with wrong-over-wrong prefer- ences helps LLMs to produce less wrong and sometimes even outright correct answers, while improving overall model calibration. Code and data are publicly available at http:://gituubb.oo/aaog/ 8/ //Vaaing--Saaaaeee Shades-of-Wromm
INTRODUCTION
Post-training with preference optimization, a.k.a. alignment, has become crucial part of the de- velopment large language models (LLMs) (Touvron et al., 2023). From online alignment with PPO (Schulman et al., 2017), to recent developments in offline alignment with DPO (Rafailov al., 2023), LLMS improve by learning nuanced distinctions and separability between answers: a correct answer is preferred over incorrect answer in reasoning problems (Wang et al., 2023c), or a re- sponse that adequately completes a user instruction is preferred over failing one (Dubois et al., 2023), etc. This response "correctness" in alignment procedures and implementations typically comes from datasets with ground truth answers (Wang et al., 2023c), human annotation (Ouyang al., 2022), or state-of-the-art L.LMs (Lee al., 2023), which are employed to generate high- quality reference answers to construct preference pairs and/or reward model training data.
However, in this work, we ask an emerging problem: what if there is no correct answer in the align- ment process? What if the task comes without annotated ground-truths. is prohibitively expensive and time-intensive for human annotation, and even state-of-the-arr LLMs are too poor to consistently provide correct answers? We see an increasing number of challenging benchmarks such as theorem proving (Welleck et al., 2021; Zhou et al., 2024) and structured reasoning (Ding et al., 2023; Fang et al., 2024), where absolute correct answers difficult impossible to collect. As a result, exist- ing alignment procedures relying on high-quality expert-annotated data might struggle to expand the frontier of model capabilities. In response, we focus on the massive low-quality alignment, where LLMs align
completely
wrong answers generated by LLMs and propose wrong-over-wer by learning to prefer wesswwrogg answers over more-wrong ones given the spectrum of wrongness. We present the outline of our work in Figure 1:
RQI: Can LLMs discriminate between varying shades of wrong and produce wrong-over-wron preferences? We experiment on four tasks with clear wrongness distinctions, e.g., to find the shortest path in network with a ground truth path length of finding path of length is "less wrong" than a path of length 11 (Wang et al., 2023a). For each task, we sample multiple solutions to
"qqull contribution

---

Published as a conference paper at ICLR 2025
Phase 1 Eliciting Wronng over Wrong Preferences
Phase 2 Wrong-over ooroog--ve---lroo Aliigment
Less wrong + ->
A: 11
More correct
11->3 5
Daciele preferances....
LLM (penerstor
In calibrated
LLMaigned
Q: Give the shortest path from node to node
8
DPO Alignment
Consistency
Logits
Pairwise
Score
preferred model's output thee given instruction based on thair corractiiiss
Grada the scale to
O,
confidenc +0.8
Mosastor
Ioquenoftin3 11
Xr0.8 1H0.6
preferred over 11
"Score(B): 4; Score/11)
288x 11
Figure 1: Two phases of aligning LLMs with wrong answers: eliciting wrong woeng-ooer-woong prefer- ences and wrong-over-wrong alignment. Phase we employ four methods to elicit wrong-over. wrong preferences, based on answer consistency, logits-based confidence, and LL-M-a----juee ap- proaches. In Phase 2, we align LLMs with wrong-overs serg--ovewwoongggg preferences using DPO and expect to have less wrong, more correct, and better-calibrated answers.
problem, filter for incorrect answers, and employ LLMs to elicit wrong-over-wrong preferences via colfisonnsst (Wang et al., 2023d), token probabilities, pairwise comparison (Sun et al., 2024a), and LLM-as-a-judge task to evaluate the quality of generated wrong-over-wrong preferences.
al., 2023). We employ quantitative "proxy"! of wrongness in each
RQ2: Is LLM alignment with trong-over-wrong preferences helpfid? We fine-tune language models to align them using the L.M--geneaatee wrong-over-wee oven-wrong preferences. We hypothesize that wrong- wver-wrong alignment will make models better calibrated, learning a better representation space for correct and incorrect responses. The models might produce more correct answers as determined by ground truth?, even if the answers still incorrect, they will be less wrong based on the proxy.
Extensive experiments with seven open and proprietary models across eight datasets demonstrate the potential of wrong alignment: (I) LLMs do have preliminary capabilities provide wrong-over-wrong judgements, with an accuracy of up to 70.9% across the four datasets spanning knowledge, factuality, commonsense, and structured data. We further identify that score-based meth- ods together with margin filtering might be the best approach for preference elicitation and pairs of wrong answers with the larger gaps in scores should be retained. (2) Alignment with the elicited wrong-over-wrong preferences confirms the above hypotheses, demonstrating improvements across all three objectives: aligned LLMS are on average across datasets up to 9.0% less wrong as evaluated by the proxy. up to 7.0% of answers become correct post-alignment, while the Estimated Calibration Error (ECE) is reduced by up to 9.4% across the four tasks. Wrong-overs soong-over-wron alignment focuses on generating and selecting alignment data, functioning ment algorithms. demonstrates the unique potential to expand the frontier of model capabilities for tasks with clear wrongness distinctions and unavailable ground-truths.
orthogonal complement to existing align-
METHODOLOGY
2.1 ELICITING WRONG-OVER. -EEE--VVRR----GGGG--- PREFERENCES
When LLM-generated answers are incorrect, it is often possible that some answers are less wrong than others. We this "varying shades of wrong" and investigate whether LLMs could provide reliable wrong-over- wrong-over-wrong preferences if no reliable ground truths are available.
Formally, given a question q and pair of wrong answers (a1,a2), we aim employ LLMs to ap- proximate a "ground-truth" wrong-over-wre preference function f(a; > a2 q) + {1,0, -1}, where 1 indicates that a1 less wrong and should preferred over a2, 1 vice versa, and 0
Wrongness proxies are by no means perfect; they merely serve as objective and quantitative measures.
While wrong-over-wrong alignment aims improve models without ground-truths, ground-truths are still required to evaluate its effectiveness. Thus, we datasets with ground-truths examples in this paper.
2

---

Published as a conference paper at ICLR 2025
dicates the two answers not separable due to ambiguity uncertainty. Since the ground-truth preference function f(a1 > ag q) is usually unavailable, we propose using a silver function f(a; > a2 q) to estimate f and employ f for evaluation (e.g., in the shortest path problem, a path of length less wrong than a path of length 11, when is correct).
Although using proxy functions to directly construct wrong-over wrong preferences may seem more ideal, these proxies often rely on the availability of ground-truth or models trained on ground truth. Moreover, not all datasets have well-defined proxies, and those that do vary vastly in format. Thus, we propose to elicit wrong-over-wrong preferences from an L.M,, JuuM(a > a2 q) - (1,0.-1) Given that LLMs are known to suffer from poor calibration in evaluation (Liu et al., 2024), our first research question is: Can LLMs provide reliable can any fuM be good estimator of f? We explore two typical LLLM-as---judee methods:
preferences, or
Pairwise comparison As increasingly employed in LLM-as-a-judge research (Dubois al., 2023; Bai et al., 2023), an LLM can directly compare two answers. We prompt an LLM to reason about which answer in (a1, a2) less wrong (e.g., "select the preferred Al model's output for given instruction based on their correctness" ). The result is denoted as PC(a1 > a2 q. LLM) + (1,0,-1 - and the full prompt text is in Table 13.
However, LLMs could be sensitive to answer order and suffer from positional biases (Wang et al., 2023b; Zheng et al., 2023). To mitigate this limitation, we employ consistency checks by flipping the order of answers, prompt the LLM to judge again, and disregard any inconsistent results. The final pairwise comparison preference function after filtering is:
1 de) fau > a2 q) (PC(a) > a2 /11 LLM) PC(a2 > a1 | LLM))
Score-based Another strategy is to let an LLM score the wrongness of each answer then compare the scores (Bansal et al., 2024). We employ the prompt "grade the correctness of each response on 5-point scale from 0 to 5" to obtain the LLM's score for each answer score(a full prompt text is in Table 14. This scoring mechanism allows for a more fine- mee-graindd measure than pairwise comparison. We identify two important factors in this scoring- -bsse approach:
LLM). The
Batch size: We simultaneously include b answers {a, (a to the same question q for the LLM to score in one prompt. Increasing b allows the L.LM to compare and contrast multiple answers simultaneously, potentially improving its ability to discern relative answer reliability. There is trade-off between efficieney/calliratiinn and context length.
an- swer pair. Specifically, we use the m-percentile Mm of all score margins to filter out pairs with smaller differences, as larger score margins are more trustworthy indicators of wrongness dispar- ity (Li et al., 2024b; Yang et al., 2024; Yuan et al., 2024) (e.g., employing M5o indicates that only the wrong-over-wrong pairs with the top 50% score gaps are retained.).
Score margin: Margin indicates the absolute score difference between
To sum up, the score-based preference function is defined as:
(1) fa + a2 q) sgn (scorcaag) 1( scorcag |> Mn,))
where scoreag score(a; q, LLM) score(a2 q. LLM) and 1(-) is the indicator function. For comparison, we also include three baseline methods:
Heuristic Research suggests that response length correlates with answer quality (Zhao et al., 2024), i.e., longer answers with detailed reasoning steps may be closer to a correct solution. We define this trivial heuristic preference function as: I'aliusaa a2 q) = =gn((en((a)) len(az)),
where sgn is the sign function.
Consistency- based For multiple sampled responses, the frequency of answers can be interpreted as a measure of model confidence (Wang et al., 2023d; Manakul et al., 2023; Miao et al., 2024). Given m sampled answers {a.) the sampling repetition score is calculated as sr(a,)
m Σラ ((a; az), representing how frequently an answer is repeated across multiple samples.
3

---

Published as a conference paper at ICLR 2025
Based on the consistency scores, the preference function is defined as: S [noLLa(aa > 02 q)
sgn(sr(a1)) - sr (r(2))), assuming that the more frequent answers are more likely to be correct (Cole et al., 2023). Note that flo might not be applicable to open-ended generations where there is no fixed of candidate answers.
Logits-based A well-calibrated LLM's token probabilities should reflect the reliability of re- sponses (Geng al., 2024), and more probable answer is more likely be correct. We employ Negative log-Liielihood (NLL) to estimate the LLM's confidence in each wrong answer, specifi- cally NLL q) -log Pt) - log(Ps) where & is token in answer and Pt is
ell) its token probability. The rogits-based preference function is then defined as: (a1 > a2 q)) sgn(NLL((2) q) NLL(a, q)).
2.2 WRONG-OVER WRONG ALIGNMENT
We hypothesize that there are valuable signals in wrong-over-wrong preferences, that LLMs could learn to distinguish answer reliability, improve model calibration, and more. Strong-over-wer alignment is a set of synthetic experiments which provide controlled environment to study under the worst scenario with no ground-truths available, whether we can still push the boundaries of model capabilities using only low-quality answers. Thus in practice, it is unnecessary avoid right- over-wrong alignment. Instead, we show that wrong-over-wrong alignment is good supplement to nigt-over-wrong alignment in Table 3.
To this end, we first synthesize a dataset Dwow through the following pipeline. First, given an original (1)
dataset Do ( ggt)) ) we sample m answers from the LLM and synthesize scenarios with
LLM(q")). Then we iterate over every combination (a." dx of {a (1 aj )- and judge the answers with JILM SMOLAM- The one considered less wrong is the chosen response a, and the other is the rejected response a,. We ignore the cases where the L.MM gives a tie (FLM = 0). Full details of wrong-over-wrong dataset construction are available in Algorithm 1.
wrong answers only by filtering out correct ones: {a)) a' (0) * a d
With this dataset, we fine-tune an LLM using any preference optimization method (Azar et al., 2024; Hong et al., 2024; Wu et al., 2024; Calandriello et al., 2024). For example, the training objective of DPO can be expressed as:
To (a ② [oro(To: Tre)) = -. log a Bllgg The a. q) T ②
no 9) log
where Ty is the policy model and Th is the reference policy. To evaluate hypothesis about the benefits of wrong-over-wrong alignment, we measure model performance from three aspects:
Less wrong We evaluate the wrongness of answers that still wrong aost-alignment based on f(a, > a2 q). A reduction in wrongness indicates that LLMs are producing better attempt at challenging problem. As LLMs learn to separate varying shades of wrong in the alignment tuning process, we expect the answer distribution of LLMs move towards less-wrong direction.
More correct We also evaluate the accuracy of models producing correct answers. An increase in accuracy demonstrates that surprisingly, training on wrong answers only can guide models to pro- duce correct answers. As the model moves its output distribution to "less wrong" direction, answers that were previously close-to-correct may be adjusted enough to align with the correct solution, even though the training process only employs incorrect answers.
Better calibrated By training the model prefer less-wrong answers, learns make finer distinctions between varying shades of wrongness and its confidence estimation should become more reliable. We evaluate the calibration of models based token probabilities.
3 EXPERIMENT SETTINGS
Models We employ three open and proprictary LLMs for experiments spanning different scales and access levels. First, we ALLMMA3-8B (Dubey et al., 2024), GPT-3.5, and GPT-4o (Achiam
4

---

Published as a conference paper at ICLR 2025
et al., 2023) to sample 10 answers per problem with random option orders for multiple-choiee ques- tions to increase the robustness of evaluation. We employ a temperature of 1.0 and a max generation length of 1024. LLMs employed in this stage are called generators. We then elicit wrong-over wrong preferences using the same LLMs with each of flai J f S fiL f(r) and fJiM- LLMs employed in this stage are called evaluators. In Appendix A, we also investigate the performance of GEMINI-FLASH, GEMINI-PRO (Team et al., 2023), MISTRAL-7B (Jiang et al., 2023), GEMMA-7B (Team et al., 2024) and aggregated evaluators.
Finally, we conduct QLoRA fine-tuning (Dettmers et al., 2023) on LLAMA3-8B using the collected wrong-over-wrong preferences through DPO. More preference optimization methods are experi- mented with in Appendix A. We experiment on preferences elicited from methods, respectively <(p)
c(x) GPT--0 with consistency checks, S with Mso and M,o. and f an oracle reference. We sample 4000 wrong-over-wrong pairs for each experiment.
Data We employ datasets with carefully selected but not necessarily perfect proxy correctness function p(a q). Specifically, the proxy function gives a heuristic- -assed or model-based correct- ness score to each answer based on ground-truth and by comparing p(a q) we can know which answer is "less wrong". To have more accurate evaluation and mitigate the bias and subjectivity brought by a single proxy, we include wide range of proxies across various tasks:
Knowledge Crosswords (KC) (Ding et al., 2023) is a multiple-choice structured knowledge rea- soning benchmark where LLMs are tasked with filling three blanks with entities and satisfy given factual constraints. The correctness proxy is pl (a
oo correctly filled blanks
of all hlanks(3)
NLGraph (NLG) (Wang et al., 2023a) is a graph reasoning benchmark and we employ the short- est path subset, where LLMs find the shortest path between nodes in weighted undirected graph. The correctness proxy is p( a () = Wenur
Im, 1wo-wal with Wa being the weight of the
shortest path in a, Wgr being ground truth, and Wwwrrs being the weight of the longest simple path.
Bio Generation (BG) LLMs are asked to generate a biography a named individual, and the cor- rectness of the answer is evaluated via FActScore (Min et al., 2023), which computes the percent- age of atomic facts supported by retrieved passages. The names and retrieved passages are sourced from Wikipedia. The proxy function is defined as p (a q)) FActScore(a q) E [0, 1].
COM2 (Fang et al., 2024) is multiple-choice commonsense reasoning benchmark. We Vera (Liu et al., 2023) to obtain the commonsense plausibility scores for each option. The proxy func- tion is defined as (
q) = Vera(a q) E [0, 1].
We sample 625, 625, 625, and 380 questions from each dataset, each split into training sets D validation sets Dvar. and test sets Drem with an approximately 8:1:1 ratio. We then generate Dwow from Diaan employing the pipeline in $2.2. Drem is used for evaluation and Diai for hyperparameter tuning. We also experiment on datasets without clear right- wrong distinctions in Appendix A. More detailed experiment settings be found in Appendix B.
Evaluation and Metrics For eliciting wrong-over wrong preferences, we evaluate by the proxy ground-trutt truth preferences f(a > q). The accuracy of LLM-generated wrong-over-wrong preferences is defined as: Accиow Paase a. [[f(a, > ar q) fiMo)LLM(( > a, q)). For wrong-over-wrong alignment, we compare WoW-aligned models with unaligned base models and evaluate improvements across three objectives: less wrong, more correct, and better calibration.
Less Wrong: We employ the average proxy function scores of all wrong answers: Pwrrng Newn Eoun Easza, p(a; q), where Nwming is the number of wrong answers and ag indicates the correct answer.
More Correct: We employ task accuracy: Acc = + E Ea, 1 1(a; = =agt). where N is the number of all answers.
Better Calibration: We extract the confidence of LLM answers with exL(NL q)) E [0, 1], create 10 bins of [0.1i, 0.1(i + 1)], and employ Expected Calibration Error (ECE) (Guo et al., 2017) measure the calibration quality.
5

---

Published as a conference paper at ICLR 2025
LLAMA3-8B
GPT-3.5
GPT-40
Method
Margin
KC BG COM NLG
BG COM NLG KC BG COM' NLG Overall
EVALUATOR- EVALUATORR-DDEPENDDIEE INDEENDEET
Msn
.83
.480
.498
.492
.546
.519
.482
.43
.91
.459
.444
.503
.488
HEURISTIC
M
.502
425
.492
.514
.899
.568
.489
.420
.500
.408
.380
.533
.474
Ms0
.500
.505
.582
.266
.470
.605
.44
.565
.548
.359
CONSISTENCY
Mi0
447
.441
.578
.506
.556
.695
.600
.42
.494
.566
LLAMA3-SB AS EVALUATOR
all
.498
.492
.455
486
481
.488
.530
.533
.509
.499
.468
.503
.496
PAIRWISE
filter
-518
.566
.437
.510
.528
.479
.549
.582
.564
.419
.491
.525
.533
Ms0
.541
.568
.524
.505
.570
.531
.482
.464
.566
.491
.419
.577
.532
LOGITS
Mj0
.559
.669
.432
.528
.571 .649
.496
.427
.400
.444
.310
.630
.582
Ms
.621
.552
.493
.554
1643
.502
.574
.580
.604
.514
.626
.424
.546
SCORE
M10
.654
.551
.458
.579
.701
.485
.659
.524
.800
.632
.662
.500
.558
GPT-3.5 AS EVALUATOR
all
.512
.433
.472
.500
.504 .474
.531
.512
463
.492
.466
.502
.444
PAIRWISE
filter
.531
.631
.502
.231
.520 .500
.493
.531
.387
.437
.433
.400
.536
Ms0
.548
.511
.570
.475
.507
LOGITS
Mι0
.541 .538
.570
.430
.505
M50
.244
566
.473
.552
.578 .522
.608
.600
264
502
.503
.551
.547
SCORE
M10
.585
.632
.517
.555
.583 .550
.718
.575
.00
.566
.662
.573
.590
GPT-40
EVALUATOR
all
.605
.593
.507
.551
.646
.512
.515
.577
.434
.501
.526
.537
.562
PAIRWISE
filter
.691
.689
.533
.602
.712 .536
.558
.661
.477
.490
.604
.549
.624
My0
.91
.539
.486
.572
.544
LOGITS
.584
.507
.591
.574
Mso
.733
.677
.544
.605
793
.591
.617
.661
.500
.639
.641
SCORE
Mj0
.793
.75
.534
.652
.335
.655
.711
.684
.400
5886
.520
.578
.709
Table 1: Accuracy of LLM-generated wrong-over-wront preferences Accwow: The three LLMs across on top are employed to generate answers (generator). Best results are bold, second best are in underline, and incompatible unavailable results are denoted as "-" For pairwise comparison, margin "all" without consistency checks and margin "filter" with consistency checks. Overall accuracy is weighted average across all datasets by the number of wrong-over oovrrwrong pairs in each dataset. LLMs do have preliminary capabilities to distinguish varying shades wrong with up to Acowsw 0.709 achieved by the score-based approach, specifically f with Mj0-
4 RESULTS
Eliciting Wrong-over-Wromg Preferences We present the accuracy of LLM-generated wrong- over-wrong preferences in Table I.
Overall: Feasible elicit wrong-over-wrong preference. Most (approach, LLM) combinations yield wrong-over-wee
preferences that are significantly better than random guess: the average Accwow is 0.553 across datasets, and the best Acewow 0.709 achieved by fort.4 with margin of Mj0. This suggests that LLMs, with the right approach, do possess preliminary capabilities to distinguish various shades of wrong.
JGPT40 with M achieves the best Accwww The average Acowow across LLMs for consistency-base with Mio. logits-based with Mi0. pairwise compari- son with consistency checks, score-based with M10, and heuristic are 0.554, 0.566, 0.564, 0.619, and 0.474, indicating that the score- score-based method outperforms all other approaches by at least 9.4% and provides clearly non-random judgments. This suggests that the token probabilities of aligned LLMs might not be well-calibrated (Sorensen et al., 2024; Feng et al., 2024) for wrong- over-wrong contexts, while LLM-as-a-jule methodologies (Li et al., 2024a) with score-based prompting offer the most promising solution.
Best eliciting method: Scored- based. ((]
Answers for Bio Generation questions open-ended and cannot compute repetition; GPT-3.5 and GPT-40 open models for caleulating logits and NI.I..

---

Published as a conference paper at ICLR 2025
KC
BG
COM
NLG
Method
Dornut
Acc-
ECE, Purmurt
Ace;
ECE₁ Pumnet Ace, ECE, Pornut Ace; ECE;
ORIGINAL
466
.555
.235
.532
.027
.576
.312
.669
.053
.750
.142
.649
SELF-GENERATOR
PAIRWISE filter
.475
.627
.096
.670
.059
.500
.326
.690
.049
.806
.179
.493
SCORE M50
.529
.597
.251
.661
.043
.580
.325
.660
.039
.800
.203
.551
SCORE M10
.532
.584
.315
.682
.075
.561
.357
.681
.020
.877
.292
.578
ORACLE
.529
.576
.279
.695
.108
.440
.330
.689
.064
.846
.182
.596
MIX-GENERE MIX-GENERATOR
PAIRWISE filter
.533
.574
.201
.634
.075
.535
.355
.698
.048
.832
.192
.538
SCORE Ms0
.528
.590
.15
.619
.065
.523
.329
.669
.067
.27
.221
.585
SCORE M10
.520
.565
.273
.687
.129
.560
.346
.677
.065
.843
.303
.522
ORACLE
.537
.581
.185
.691
.086
.472
.328
.697
.067
.832
.266
.474
Table 2: Evaluation of wrong-over-wrong alignment on less wrong (Dunag): more correct (Acc), and better calibration (ECE). The best results are in bold, second best are in underline, and green background indicates improvement dicates that wrong-over-wroog pairs are generated from only LLAMA3-8B while "Mix-Generator" uses all 3 LLMs answers. "Oracle" means aligning with proxy "ground- truth" wrong-over-we preference f. Wrong-over-wrone alignment is helpful across the board, with up to 0.163, 0.161, and 0. 175 improvement reducing wrongness, increasing correct answers, and improving calibration.
the original LLAMA3-8B. "Self-Geenratoor" in-
Improve upon original eliciting method: wise comparison method, we notice that LLMs are sensitive to response order and applying con- sistency checks by removing inconsistent judgments due order-flipping improves 9.0% Accwow on average. Employing score margins to only consider the most separable cases also improves Accwow for logits- based, consistency-ben ments across datasets from Ms0 to M0 being 2.1%% 11, 1.3%, and 6.9%, and from 00 to M being 5.8%, 72.4%, and 65.3%. The significant improvement from M to Mi0 due to prevalent cases where the L.M gives the same score for wrong answers with different levels of wrongness. Appendix A also illustrates the trade-off effect of batch size.
consistency checks and score margins. For the pair-
and soorebbaedd methods, with average improve-
Failed eliciting method: self-evaluation. LLMs may not be good at producing accurate wrong- over-wrong preferences for their own generation. While GPT-40 provides good wrong-over- wrong preferences on generation from LLAMA3-SB and GPT-3.5, accuracy suffers for its own answers, with 21.1% and 19.7% drop for score-based with M10 and pairwise comparison with consistency checks on average across datasets. This echos findings that LLM self-critique and self-correction might not be satisfactory (Valmeekam et al., 2023; West et al., 2024; Huang al., 2024a). More interestingly, employing weak LLMs to evaluate strong LLMs may be helpful (Khan al., 2024), with LLAMA3-8B achieving 0.649 Acc on answers generated by GPT-4o.
Alignment with Wrong-over-Wrong Preferences We present the evaluation of wrong-over. wrong alignment in Table
Overall: Wrong-over-wrom, wrong alignment helps to reduce wrongness, produce correct answers, and improve calibration. After wrong-over-wrong alignment, LLAMA3--88 improves aver- age across datasets on Apwnngg AAce, and AFCE by 0.074, 0.045, and 0.044. We also report the precision, recall and Fl improvement on Knowledge Crosswords and COM2 datasets in Table 12. This validates that wrong-over ovon-ooge--roog alignment is moderately helpful across the three ob- jectives. Even though we only align on wrong answers, we still magically end up making LLM generate 4.5% with wrong answers generated by L.MMs that come in large quantities.
correct answers average. This finding highlights the potential of alignment
Best preference data: Score-based achieves the least wrong and most correct answers while pairwise comparison achieves the best calibration. Score-based preferences with Mio on self- generated data achieve the best average improvement of Apwnngg = 0.090 across datasets. ((s) with Myp on gexerated data has the best average improvement of AAcc 0.070. This sug-
JCPT-4
7

---

Published as a conference paper at ICLR 2025
gests that the score-based method has the best Accwow and is most helpful in reducing wrongness and producing correct answers. However, fGPT-0 with consistency checks has the best average calibration improvement of -ACCE 0.094. We hypothesize that the margin used in score- based methods improves Acewow by avoiding cases with similar degrees of wrongness. This also indicates the models' tendency to avoid challenging distinctions, which can harm calibration.
Varied improvement across datasets: open-ended questions benefit more from wrong-over- wrong alignment compared to multiple-choice questions. We observe substantial improve- ment on Bio Generation and NLGraph with up to 0.163, 0.161, and 0.175 for Apwrong. AAcc and AECE. However, relatively less improvement is observed on Knowledge Crosswords and COM?, with up to 0.071, 0.073, and 0.138 for Apuning AAce, and AECE. This may be because Bio Generation and NLGraph are open-ended questions and there is more space for improvement compared to maltiple-choice questions in Knowledge Crosswords and COM2 where wrong op- tions are limited and LLMs are confined to pre-defined options.
5 ANALYSIS
KC
COMP
BG
NLG
Task Utility and Preference Accuracy We explore the impact of two factors on the reliability of wrong-over- wrong preference: accuracy of the evaluator's s generated answers (lask accuracy) and the evaluator's confidence in its generated answers (lask confidence). We visualize the correlations and present Pearson correlation coefficients (Sedgwick, 2012) in Figure 2. We observe positive cor- relation between task accuracy and Accwow, but neg- ative correlation between task confidence and Accwow. This suggests that models that perform well on the task are also good at distinguishing various shades of wrong, while over-confident and under-calibrated models harm wrong-over soag--ove-wooongg preferences. We also find that LLM struggles to differentiate two answers with close wrong- ness levels in Appendix A.
ACCwow
0.6
0.4
all 0.280
P.3: -.173
P.8G 0.542
P_0OMP-
0.2
0.2
0.4
0.6
0.8
task accuracy
0.8
ACCyow
0.6
0.4
P-all .0.172
P3RC
Preference Accuracy and Alignment Improvement
P_BG
-9.434
We examine how the quality of wrong-over-wrong judgements Accwow is related to improvements through wrong-overrwrone alignment. Figure 3 demonstrates a weak positive relationship between Accwa and improve- ment in Apwnng: and no significant correlation between Accwow and improvement in Ace - AFCF. Sur- prisingly, this suggests that improvement resulting from rrong-vver-wrong aligmment is nuanced and is not sensi- tive to the absolute accuracy of wrong- -ver-wroog pref- erence data. There is also clear positive relationship be- tween Accwaw and Apwrong AAcc and ECE on the Bio Generation dataset, indicating that sensitivity to wrong- over-wrong judgment quality varies by dataset.
0.2
0.5
0.6
0.7
0.8
0.9
task confidence
Figure 2: Correlation between task accuracy, confidence and Acewow of score-based eliciting with M Data points from all 3 LLMs we used to elicit wroog--over-woom preferences. P stands for Pearson correlation coeffi- cient. The ability to elicit wrong-over- wrong preferences is positively corre- lated with task ability but negatively correlated with confidence.
Right-over- Wrone Alignment We conduct experiments on right-over- preferences and a 50:50 mix of right-over-wrong and wrong-over-wrom preferences. The results in Table reveal that: (1) Right-over-wrong alignment has the best average improvement in more correct Apwnng 0.090 on mix-generated data, while on self-generated data, wrong-over-wrong alignment yields the best Apwrngg = 0.068. Mixing generators is especially helpful to right-over- alignment, with average AAce improved by 55.7%. (2) Right-over- alignment has the best average improve- ment in more correct AAcc 0.134, while a mixture of wight-over-wrong and wrong-over-wwie wrong preferences achieves the best average improvement in calibration - -AECE = 0.107. This partly resolves the previous finding that alignment hurts calibration (Sorensen al., 2024; Feng et al.,
8

---

Published as a conference paper at ICLR 2025
KC
BG
COM
NLG
0.15
0.15
0.15
Perovo
0.10
AcC
0.10
0.10
0.05
0.05
0.00
0.05
0.00
0.00
0.4
0.6
0.8
1.0
0.4
0.6
0.8
1.0
0.4
0.6
0.8
1.0
ACCwow
ACCwow
AcCwow
Trarson=0.2600
R²00.088
Pearson=0.002. R0000000
Pearson 0000110000........... R²=0.000
Figure 3: Correlation between Acewow and improvement after Brong-over-wree alignment in less wrong Apwrong. more correct AAcc, and better calibration - AECE.. Data points sourced from all methods (/GPT40 with consistency checks, (( SGPT.40 with M50 and Mí0, and oracle î), and the oracle method is considered as Acewow 1.0. The black line is the linear regression on all four datasets while the green line is the linear regression on Bio Generation dataset. Wrong-over over-wrong alignment is not sensitive to the accuracy of moong-over-wrole preference.
2024): we hypothesize that this is due to missing prong-over-wwre preferences in alignment data that make LLMs unable to disentangle varying shades of wrong for fine-graincd calibration.
Generalization to Unseen Data We exam- ine how wrong-over-wrong alignment can gen- eralize to other unseen tasks in the same do- main. We employ Hellaswag (Zellers et al., 2019) and the maximum flow subtask in NL- Graph (Wang et al., 2023a) two unseen datasets while being in the same domain as the datasets employed for alignment tuning, 2OM2 and shortest path. Results in Table 5 demon- strate that wrong-over-wer wrong alignment could indeed generalize to unseen data, with an aver- age improvement of O. 118, 0.002, and 0.029 on Apwrngg AAce, and AECE.
KC
COM'
NLG
Pawwy
Accy ECE,
Priest Ase
Peonet
Avcy ECE,
ORIGINAL.
466
235
.750
BESI W
529
591
251
.355
1088
D48
847
.292
.588
SELP-GUNERATOR
R+W (PAIRWISE)
.933
.706
065
.359
.005
.057
8066
.179
.193
R4W (ScORF
665
174
R+W (ORACLE)
.607
775
060
.357
7760
.158
.519
.579
.005
079
.33
892
.029
777
.153
.652
MIX-GENERATOR
.530
.008
053
825
New
Map)
.94
.548
R+W (SCORE Mao)
.559
.711
.104
.343
.655
.034
881
.08
.496
R4W (ORACTL)
.567
740
.625
.344
748
111
826
.214
.490
.588
787
.110
.374
711
.83
8866
268
.337
Qualitative Examples We present qualita- tive examples in Table 4, showing that WoW- aligned models could generate less wrong, more correct, and better calibrated answers. More successful and failed examples can be found in Appendix A.
Table 3: Evaluation of right-over- align- ment less wrong (Punong), more correct (Acc), and better calibration (FCF). "w" and "R" stand for wrong-over-wroogg and right-over wrong data. Wrong-over-we
alignment is a good supple- ment to existing wight-over-wrong alignment and best employed in conjunction.
RELATED WORK
LLM Alignment In recent years, aligning LLMs with human preferences has become impor- tant research question, driven by the need for safety (Dai et al., 2024; Qi et al., 2024; Huang et al., 2024b), honesty (Yang et al., 2023; Wen et al., 2024), factuality(Liang diversity (Ding et al., 2024), etc. The initial methods, such as Reinforcement Learning from Hu- man Feedback (RLHF) (Ziegler et al., 2019: Stiennon et al., 2020; Ouyang al., 2022: Munos et al., 2024; Chakraborty et al., 2024), leverage human preferences to train reward models and em- ploy algorithms such as Proximal Policy Optimization (PPO) (Schulman al., 2017) for alignment. Recent alignment methods move away from explicit reward models, as seen in Direct Preference Optimization (DPO) (Rafailov et al., 2023). Alongside this, Reinforcement Learning from AI Feed- back (RLAIF) (Bai et al., 2022; Lee et al., 2023) and LLM-as-a-Judge (Dubois et al., 2023; Bai et al., 2023) frameworks have introduced AI-driven feedback systems, allowing LLMs to give feed- back and improve themselves with minimal human intervention. Techniques such Self-Alignment (Li et al., 2024b; Wang et al., 2024c; Yuan et al., 2024: Sun et al., 2024a), which involves models evaluating and aligning with their own outputs, represent a further step towards scalable oversight.
al., 2024; Lin al., 2024),

---

Published as a conference paper at ICLR 2025
Question
In undirected graph, the nodes numbered from
and the edges are rep-
resented by (start node, end node, weight): (0, 2. 2), (0. 5, 4), (0. 3. 1). (0, 3), (l, 2. 4). (1. 5. 1). (1. 4, 4). (2. 4. 3). (2. 3. 1). (3, 5. 2). (3, 4. 3). (4, 5. 2). Q: Give the
shortest path from node 0 to node 1. Please also give the total weight of the shortest
path. [Correct answer: (0 3 4x.]
Original Answer
[CoT steps]. The final unswer is: shortest path: + 3
total weight: 6. [CoT steps]. Therefore, the answer The shortest path from node to node is 3 x 4, with total weight of 5. (The path found shorter: >.5]
Less Wrong
More Correct
stepsJ. Final answer: The shortest path from node to node Node -> Node
3 Node with total weight of [Successfully find the shortest path.]
Better Calibrated
[CaT steps]. The total weight of the shortest path is: + (weight 2). + (weight
1), - (weight 3). Total weight: + 6. [Even though the answer equally wrong compared to original answer, NLL increases from 0.255 - 0.373.]
Qualitative examples on NLGraph where WoW-aligned models become less wrong, more correct, and better calibrated on NLGraph. We provide explanations in [red].
Table 4:
Instead of alignment objective, we investigate an orthogonal research question to the improvement in alignment approaches and focus on the preference data: in challenging tasks and contexts when correct and high-quality answers are not readily available (Baheti et al., 2024; Agarwal al., 2024), could LLMs distinguish various shades of wrong and align with wrong-over-wront preferences? Prior work has also established the utility and representativeness of fine-grained and continuous preferences over categorical labels in uncertain (Lin et al., 2022; Tian et al., 2023) and subjective contexts (Sorensen et al., 2024; Feng et al., 2024).
Easy-to-Hard Generalization Most of these advancements alignment benefit from the availability of "correct" answers. In contrast, our work posits that "correct" answers are in- evitably unavailable unreliable for emerging and challenging tasks. Bowman et al. (2022) points out advanced models may engage in tasks that are beyond current evaluation ca- pabilities. Easy-to-hard generalization (Burns et al., 2024; Hase et al., 2024; Xu et al., 2024b; Sun et al., 2024b) introduces series of solu- tions hoping to improve LLMs on hard tasks by finetuning on easy in-domain tasks. Our work uniquely points out new path for training fu- ture LLMs on hard tasks when correct answers are unavailable, arguing for wrong-over- aligument based on synthetic preference data as complementary training objective.
HellaSwag
Maximum Flow
Method
Pereyt
Acc;
ECE,
Porset
Aoe;
ECE,
ORIGINAL.
.230
.737
.089
.112
.069
.663
SELF-GENERATOR
PAIRWISE
.243
.729
.098
.344
.066
.567
SCORE Me
.200
.679
.047
.330
.083
.637
SCORE Mj0
264
.719
.068
.342
.109
.673
ORACLE
.227
.729
.023
.151
.049
.659
MIX-GENERATOR
PAIRWISC
.247
.706
.082
.348
.094
.533
SCORE M50
2004
.729
.090
336
.083
.621
SCORE M,
.50
.771
.117
.326
.089
.627
ORACLE .267
.753
.099
.202
.074
.582
Table 5: Generalization to unseen datasets, from
COM2 to HellaSwag and from shortest path to maximum flow. While tuned only on COM2 and shortest path, the aligned models improve on the two unseen datasets well, with average im- provement of 0. 0.118, 0.002, and 0.029 on Apwrngg. AAcc, and -AECE.
7 CONCLUSION
With the growing race towards bigger, better LLMs capable of solving a wider range of tasks, it becomes evident that the availability of carefully curated data is a major bottleneck. Our work inves- tigates the potential to alleviate this limitation by eliciting preferences among wrong answers with an LLM and aligning with these grong-over-wrong preferences. We empirically investigate wrong- over-wrong alignment with seven L.MSs and eight datasets. We find that LLMS do have preliminary capability to rank wrong answers and produce reliable wrong- wrong preferences. The strongest approach such as score-based L-Ma-saau achieves up to 70.9% accuracy across datasets. In ad- dition, alignment with such wrong-over-wrong preferences is helpful in calibrating the models, e.g., reducing answer "wrongness" " by 9.0%, producing 7.0% more correct answers, and improving over- all model calibration by 9.4% average. We envision future wrong-overs over-wrong alignment methods as an important addition to advance LLM capabilities on challenging tasks where correct answers or rich annotations are not readily available.
10

---

Published as a conference paper at ICLR 2025
LIMITATIONS AND ETHICS STATEMENT
Imperfect proxies. The proxy functions we use to measure correctness are inherently imperfect. While they serve reasonable approximations for distinguishing varying degrees of wrongness, they are not definitive, e.g. heuristic-based proxies may merely focus on the final answer and over- look the reasoning steps and model-based proxies are inaccurate and may suffer from domain shift problem. As a result, some wrong-over-wrong preferences may be misjudged, potentially affecting the evaluation of Accwow and analysis results.
Sensitivity to hyperparameters. The alignment process we employ is sensitive to hyperparameter selection. This dependence can lead to variability in results across different models and datasets, making it challenging to guarantee consistent performance improvements. Further exploration of more robust hyperparameter configurations (Falkner et al., 2018; Arango et al., 2024) is necessary.
Limited scope of experimental datasets. Experiments truth answers exist, and proxy functions can be employed to approximate wrongness. However, this setup doesn't fully reflect the application scenarios we aim to address tasks where clear ground-truth answers are available (Wang et al., 2024a), or the problems are so challenging that even expert human annotators or LLMs might struggle (Welleck al., 2021). Evaluating the generaliz- ability of our framework in these conditions remains an important avenue for future work.
conducted on datasets where ground-
Unique focus on knowledge and reasoning problems. We focus on tasks and problems with absolute and indisputable correct answers in this work, such as multi-hop QA (Ding et al., 2023) and graph reasoning (Wang et al., 2023a). Consequently, existing alignment works in these domains often assume access to correct answers and construct wron pairs for preference learning (Wang et al., 2023c; Cheng et al., 2024; Lin et al., 2024). However, objective "correct answers" often not feasible in general instruction following tasks. thus they employ human preference data between a pair of responses that is not necessarily right-over-wrong. Our scope is to inves- tigate wrong-over-wrong alignment specifically focusing on the first type of knowledge/reasoning problems with absolute correctness while we leave general instruction following as future work.
Incomparable wrongness in social contexts. In social contexts, wrongness often subjective and potentially incomparuble, which introduces biases and faimess concerns in wrong-over sorng-over-rrong alignment. For instance, when judging the wrongness of statements related to sensitive topics such as politics or identity, cultural biases may influence the model's preferences (Xu et al., 2021; Bender et al., 2021; Feng et al., 2023). A statement like "People from certain neighborhoods are more likely to commit crimes" might be judged less wrong than "All people from certain neighborhoods are criminals" " but these evaluations reflect specific perspectives that may not be universally shared or appropriate in all contexts. Furthermore, aligning models with such judgments risks reinforcing harmful stereotypes or systemic biases. Ensuring fairness and transparency in these judgments is critical, and models should be designed to recognize when wrongness is subjective and abstain from making harmful comparisons.
REPRODUCIBILITY STATEMENT
We provide all necessary details for the implementation and evaluation of our proposed wrong- over-wrong alignment approach in Appendix B. Specifically, Appendix B includes information of g datasets splits and preprocessing steps, 7 LLM checkpoints, hyperparameters selection and detailed configuration for each table. Moreover, the prompts for preference elicitation methods are explained in Table 13 and Table 14. Our codes, including scripts for dataset generation, preference eliciting, preference accuracy evaluation, alignment, and evaluation of alignment are available publicly in repo https://gttub..om//ypjjjj 1a V/Var/ing-- Shadess-ffh Wrong to facilitate easy verification of our results.
ACKNOWLEDGMENTS
This research was developed with funding from the Defense Advanced Research Projects Agency's 's (DARPA) SeiFy program (Agreement No. HR001 12520300). The views expressed are those of the
11

---

Published as a conference paper at ICLR 2025
author and do not reflect the official policy or position of the Department of Defense or the U.S. Gov- emmment. We also gratefully acknowledge gift funding from Google, OpenAI, Microsoft Research, and the Allen Institute for AI (Ai2).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08 08774, 2023.
Rishabh Agarwal, Nino Vieillard, Yongehao Zhou, Piotr Stanezyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. In The Tweifth International Conference on Learning Representations, 2024.
Sebastian Pineda Arango, Fabio Ferreira, Arlind Kadra, Frank Hutter, and Josif Grabocka. Quick- tune: Quickly learning which pretrained model to finetune and how. In The Twelfth International Conference on Learning Representations, 2024.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Ashutosh Baheti, Ximing Lu, Faeze Brahman, Ronan Le Bras, Maarten Sap, and Mark Riedl. Left- over lunch: Advantage-baser based offline reinforcement learning for language models. In The Twelfth International Conference Learning Representations, 2024.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08073. 2022.
Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. Benchmarking foundation models with language-model- model-as-an an-aa-a------aa-- an-examinerr In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 78142-78167, 2023.
Hritik Bansal, John Dang, and Aditya Grover. Peering through preferences: Unraveling feedback acquisition for aligning large language models. In The Twelfth International Conference on Learn- ing Representations, 2024.
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 610-623, 2021.
Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilè Lukošiūté, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring progress on scalable over- sight for large language models. arXiv preprini (tf:::22110034444 2022.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbren- ner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generaliza- tion: Eliciting strong capabilities with weak supervision. In Forty-first International Conference on Machine Learning. 2024.
Daniele Calandriello, Zhaohan Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, et al. Human alignment of large language models through online preference optimisation. In Forty-first International Conference on Machine Learning. 2024.
Souradip Chakraborty, Amrit Bedi, Alec Koppel, Huazheng Wang. Dinesh Manocha, Mengdi Wang, and Furong Huang. PARL: A unified framework for policy alignment in reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024.
12

---

Published as a conference paper at ICLR 2025
Sahil Chaudhary. Code alpaca: An instruction-followiin llama model for code generation. GitHub repository, 2023.
Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu, Wenwei Zhang, Zhangyue Yin, Shimin Li, Linyang Li, Zhengfu He, Kai Chen, and Xipeng Qiu. Can ai assistants know what they don't 't know? In Forty-firs International Conference on Machine Learning. 2024.
Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust dpo: Aligning lan- guage models with noisy feedback. In Forty-first International Conference on Machine Learning, 2024.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. Scaling Instruction-finttree lan- guage models. Journal of Machine Learning Research, 25(70):1 53, 2024.
Jeremy Cole, Michael Zhang. Dan Gillick, Julian Eisenschlos, Bhuwan Dhingra, and Jacob Eisen- stein. Selectively answering ambiguous questions. In Proceedings of the 2023 Conference Empirical Methods Natural Language Processing, pp. 530-543, 2023.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024.
Tim Dettmees, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: efficient finetuning of quantized llms. Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 10088-10115, 2023.
Li Ding, Jenny Zhang, Jeff Clune, Lee Spector, and Joel Lehman. Quality diversity through human feedback: Towards open-ended iiveerity-driverr optimization. In Forty-first International Confer- ence on Machine Learning, 2024.
Wenxuan Ding, Shangbin Feng, Yuhan Liu, Zhaoxuan Tan, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. Knowledge crosswords: Geometric reasoning over structured knowledge with large language models. arXiv preprint arXiv:2310.01290, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: a simulation framework for methods that learn from human feedback. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 30039-30069, 2023.
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter op-
timization at scale. In International conference on machine learning, pp. 1437-1446. PMLR, 2018.
Tianqing Fang, Zeming Chen, Yangqiu Song, and Antoine Bosselut. Complex reasoning over logical queries on commonsense knowledge graphs. arXiv preprint arXiv:2403.07396 2024.
Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair nlp models. In ACL: Annual Meeting of the Association for Computational Linguistics, 2023.
Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov. Modular pluralism: Pluralistic alignment via multi-llm collaboration. arXiv preprint 1.Xiv:2406.15511 2024.
Jiahui Geng, Fengyu Cai, Yuxia Wang. Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pupers), pp. 6577-6595, 2024.
13

---

Published as a conference paper at ICLR 2025
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, Pp. 1321-1330. PMLR, 2017.
Peter Hase, Mohit Bansal, Peter Clark, and Sarah Wiegreffe. The unreasonable effectiveness of easy training data for hard tasks. arXiv preprint arXiv:2401.0675/ 2024.
Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without ref- erence model. arXiv preprint arXiv:2403.076%/ 2,4):5
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference Learning Representations, 2024a.
Weidong Huang, Jiaming Ji, Chunhe Xia, Borong Zhang, and Yaodong Yang. Safedreamer: Safe reinforcement learning with world models. In The Twelfth International Conference on Learning Representations, 2024b.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah Smith, Iz Beltagy, et al. Camels in changing cli- mate: Enhancing Im adaptation with tulu 2. arXiv preprint arXiv:231 1. 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel. Guillaume Lample, Lucile Saulnier, al. Mistral 7b. arXiv preprint arXiv:2310.06825. 2023.
Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. Debating with more per- suasive ilms leads to more truthful answers. In Forty-first International Conference on Machine Learning, 2024.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richard Nagyfi, et al. Openassistant conversations coneerstioon-democcniinnngn large language model alignment. In Proceedings the 37th Inter- national Conference on Neural Information Processing Systems, pp. 777669476637 2023.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.0020 00w.0,026770 2023.
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, hai zhao, and Pengfei Liu. Generative judge for evaluating alignment. In The Twelfth International Conference on Learning Representations, 2024a.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. Self-alignment with instruction backtranslation. In The Twelfth International Conference on Learning Representations, 2024b.
Yuxin Liang, Zhuoyang Song, Hao Wang, and Jiaxing Zhang. Learning trust your feelings: Leveraging self-awareness in Ilms for hallucination mitigation. arXiv preprint 2rXiv:2401.15449) 2024.
Lichess Team. Lichess database. Accessed: 2023.
2023.
Sheng-Chich Lin, Luyu Gao, Barlas Oguz, Wenhan Xiong, Jimmy Lin, Wen-tau Yih, and Xilun Chen. Flame: Factuality- alignment for large language models. arXiv preprint arXiv:2405.01525. 2024.
Stephanie Lin, Jacoh Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205. 14334, 2022.
Jiacheng Liu, Wenya Wang, Dianzhuo Wang, Noah A Smith, Yejin Choi, and Hannaneh Hajishirzi. Vera: A general purppoe plausibility estimation model for commonsense statements. In Pro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp- 1264-1287, 2023.
14

---

Published as a conference paper at ICLR 2025
Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. Calibrating llm-based evaluator. In Proceedings of the 2024 Joint Inter- national Conference Computational Linguistics, Language Resources and Evaluation (LREC- COLING 2024), pp. 2638-2656, 2024.
Potsawee Manakul, Adian Liusie, and Mark Gales. Selfcheckgpt: Zero-resource black-box halluci- nation detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9004-9017, 2023.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference- free reward. arXiv preprint arXiv:2405. 14734, 2024.
Ning Miao, Yee Whye Teh, and Tom Rainforth. Selfcheck: Using L.LSs zero-shot check their own step-by-step reasoning. In The Twelfth International Conference on Learning Representations, 2024.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Tyyer, Luke Zenlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual pre- cision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12076-12100, 2023.
Remi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang. Matthieu Geist, Thomas Mesnard, Côme Fiegel, al. Nash learning from human feedback. In Forty-first International Conference on Machine Learning, 2024.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal. Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pp. 27730-27744, 2022.
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. Conference on health, inference, and learning, pp. 248-260 PMLR, 2022.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference Learning Representations, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: your language model is secretly a reward model. In Pro- ceedings of the 37th International Conference Neural Information Processing Systems, pp. 53728-53741, 2023.
Romstad, M. Costalba, J. Kiiski, G. Linscott, S. Nicolet, S. Geschwentner, and J. VandeVondeles Stockfish, 2021. Version 14. Available online: https: htts//sttoffischhhsssssess org.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Philip Sedgwick. Pearson's correlation coefficient. Bmj, 345, 2012.
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christo- pher Michael Rytting, Andre Yc, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. A roadmap to pluralistic aligmment. arXiv preprint arXiv:2402.050076 2024.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Rad- ford, Dario Amodei, and Paul Christiano. Learning summarize from human feedback. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 3008-3021, 2020.
Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Daniel Cox, Yiming Yang, and Chuang Gan. Salmon: Self-alignment with instructable reward models. In The Twelfth International Conference Learning Representations, 2024a.
15

---

Published as a conference paper at ICLR 2025
Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. Easy-to-hard generalization: Scalable alignment beyond human supervision. arXiv preprint arXiv:2403.09472, 2024b. 2024b.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Julictte Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint wxXv:2403.082295 2024.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5433-543340 5442, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1 13971, 2023.
Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. Can large language mod- els really improve by self-critiquing their own plans? In NeurlPS 2023 Foundation Models for Decision Making Workshop, 2023.
Heng Wang. Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov.
Can language models solve graph problems natural language? In Proceedings of the 37th International Conference Neural Information Processing Systems, pp. 00840-30861, 2023a.
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305. 17926, 2023b.
Peiyi Wang, Lei Li, Liang Chen, Feifan Song, Binghuai Lin, Yunbo Cao, Tianyu Liu, and Zhi- fang Sui. Making large language models better reasoners with alignment. arXiv preprint NXXiv2330900214: 2023c.
Tiannan Wang. Jiamin Chen, Qingrui Jia, Shuai Wang, Ruoyu Fang, Huilin Wang, Zhaowei Gao, Chunzhao Xie, Chuou Xu, Jihong Dai, et al. Weaver: Foundation models for creative writing. arXiv preprint arXh::2401.17268 2024a.
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scien- tific problem-solving abilities of large language models. In Forty-first International Conference on Machine Learning. 2024b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023d.
Ziqi Wang, Le Hou, Tianjian Lu, Yuexin Wu, Yunxuan Li, Hongkun Yu, and Heng Ji. Enabling lanuguage models to implicitly learn self-improvement, In The Twelfth International Conference on Learning Representations, 2024c.
Sean Welleck, Jiacheng Liu, Ronan Le Bras, Hannaneh Hajishirzi, Yejin Choi, and Kyunghyun Cho. Naturalproofs: Mathematical theorem proving in natural language. In Thirty-fifth Conference Neurul Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Bingbing Wen, Jihan Yao. Shangbin Feng, Chenjun Xu, Yulia Tsvetkov, Bill Howe, and Lucy Lu Wang. The art of refusal: A survey of abstention in large language models. arXiv preprint arXiv:2407. 18418, 2024.
16

---

Published as a conference paper at ICLR 2025
Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, et al. The generative ai paradox: "what it can create, it may not understand". In The Twelfth International Conference on Learning Representa- lions, 2024.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: Statc-of-the-an natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38 45, 2020.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.0067. 2024.
Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein. Detox- ifying language models risks marginalizing minority voices. In Proceedings of the 2021 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2390-2397, 2021.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng. Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint ar arXiv:2304.122:4 2023.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for Ilm alignment? comprehensive study. In Forty-first International Conference on Machine Learning, 2024a.
Zhuoyan Xu, Zhenmei Shi, Junyi Wei, Fangzhou Mu, Yin Li, and Yingyu Liang. Towards few-shot adaptation of foundation models via multitask finetuning, In The Twelfth International Conference on Learning Representations, 2024b.
Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. Rled: Reinforcement learning from contrastive distillation for Im alignment. In The Twelfth International Conference on Learning Representations, 2024.
Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig, and Pengfei Liu. Alignment for honesty. arXiv preprint arXiv:2312.07000, 2023.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. relarrew language models. In Forty-first International Conference on Machine Learning. 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019.
Yizhuo Zhang, Heng Wang, Shangbin Feng, Zhaoxuan Tan, Xiaochuang Han, Tianxing He, and Yulia Tsvetkov. Can 1lm graph reasoning generalize beyond pattern memorization? arXiv preprint arXiv:2406. 15992, 2024.
Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning. In Forty-first Interna- tional Conference on Machine Learning, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li. Dacheng Li, Eric P Xing, et al. Judging 1lm as-jdge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Pro- cessing Systems, pp. 46595-46622 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: less is more for alignment. In Proceedings of the 37th International Conference Neural Information Processing Systems, pp. 55006-55021, 2023.
17

---

Published as a conference paper at ICLR 2025
Jin Peng Zhou, Yuhuai Wu, Qiyang Li, and Roger Baker Grosse. REFACTOR: Learning extract theorems from proofs. In The Tweifth International Conference on Learning Representatiins, 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909008599 2019.
18

---

Published as a conference paper at ICLR 2025
A ANALYSIS (CONT.)
More general LLMs evaluators We present the rest of elicited wrong-over-wrong preferences, including evaluators being GEMINI-FLASHI GEMINI-PRO, and MISTRAL- 7B. The results in Table 9 demonstrate that GPT-40 is still the best LLM at evaluating wrong-over-wrong preference and surpasses other LLMs by at least 12.0% on overall Accwow- We can also see leveraging GEMINI- FLASII, GEMINI-PRO to evaluate answers generated by GPT-40 successfully mitigate the poor accuracy of self-evaluation.
Margin KC BG COM2 overall
Code-Alpaca
M
.183 258
.211
.174
.220
M50
.367
.516
.442
.348
.439
Mi0
.425 .544
.466
.482
.505
FLAN
M
.000 .098
.000
.096
.080
M50
.007197
.000
.193
.160
M
.585
.000
.610
.487
Gemini-Alpaca
Domain-specific LLMs as evaluators To better figure out how Acewo is relate to eval- uators, we instruction finetune GFMMA-7B on 7 different datasets (Ivison et al., 2023), namely Code-Alpaca (Chaudhary, 2023), FLAN (Chung et al., 2024), Gemini-Alpaca, LIMA (Zhou et al., 2023), Open Assistant 1 (Köpf et al., 2023), Science literature and WizardLM Evol-Instruct V2 (Xu et al., 2023).
M00
.297 .356
.396
.347
.349
M50
.512.499
.555
.527
.510
M10
.534 .458
.679
.575
.519
LIMA
M,00
.110 .079
.215
.099
.098
Ms0 Mi0
.221.159
.428
.19
.19
.610
.508
.343
.466
.501
Open Assist
Then elicit wrong-over-wrong preference from them on Knowledge Crosswords, Bio Generation, COM2 and NLGraph datasets with GPT-3.5 being generator. We only experiment on score-based eliciting. Results in Table demonstrate that domain-specific LLMs as evaluators show considerable variance in their wrong-over-wrong preference elicitation accu- racy across datasets. For example, GEMMA-7B fintuned on FLAN achieves the best wrong- over-wrong accuracy on Bio Generation and NL.Graph while unable to follow instructions on Knowledge Crosswords and COM?. This suggests finetuning way improve LLM- based wrong-over-wrong accuracy in certain domains.
M
.240 .15
.220
.180
.187
M
.481 .351
.441
.360
.375
M10
.500 ,563
.602
.553
.555
Science
M
.241 .236
.557
.248
.262
M50
.482
.42.
.561
.482
.482
Mi0
.555
.574
.470
.525
WizardLM
M
.194 .278
.322
.239
.259
M50
.388
.508
.474
.478
.482
M10
.579 .551
.523
.479
.529
Table 6: Accuracy of wrong-over-wrong prefer- ence elicited from GEMMA-7B fintuened on dif- ferent instruction datasets. Finetuned LLMs have very different wrong-over-wrong preference aceu- racy across domains.
More challenging datasets We investigate LLMs' capability to give wrong-over-wrone preferences on more challenging datasets. NL- Graph (full) and ChessPuzzle (CP). NLGraph (full) employs all min/max questions in NLGraph, including very challenging questions like "finding the maximum flow between two nodes a weighted undirected graph" and "finding assignment of jobs to applicants such that the maximum number of applicants find the job they are interested in"" Chess Puzzle contains chess puzzles from the lichess.org website (Lichess Team, 2023), where the inputs are board state represented in Forsyth-Edwarce Notation (FEN) and LLMs are asked to generate the optimal next move for current player. However, we find that LLMs tend to give in- valid moves, which makes it hard to evaluate wrongness. Thus employ Stockfish (Romstad et al., 2021), 1), SOTA chess engine, to generate the best moves for the current player and ask an LLM to choice from given options. Evaluators' task accuracy is 0.204 (random guess 0.000) on NLGraph (full) and 0.461 (random guess 0,250) on Chess Puzzle. The correctness proxy for NLGrraph (full)
(a q) = -lva Verl. where va the extracted final answer and Ugr is the ground-truth an- swer. The correctness proxy for Chess Puzzle is p(nll( (a q) = WT STnaaas q). where wT Stoeffahs is the win rate predicted by Stockfish. Experiment results in Table 10 demonstrate that the evaluator's
19

---

Published as a conference paper at ICLR 2025
Acowow drops significant challenging tasks, e-g- JGTT40 with Mio drops by 23.7%. The average A Acowow across datasets for pairwise comparison with consistency check, logits-based with Mn and scored-based 1o are 0.585, 0.546, 0.550, indicating pairwise comparison may be more suitable for challenging tasks.
(4)
Batch size for score-based eliciting We also investigate the effect of batch size on score-based preference eliciting. We em- ploy LLAMA3-8B as generator and GPT-40 as evaluator. For simplicity, only experi- ment on Knowledge Crosswords and NLGraph (shortest path) dataset. Experiment results in Table 7 indicate that 5 is the best batch size for our experiment setup, which verifies sumption that batch size has a trade-off effect on scored-based -ased eliciting.
Margin
KC
NLG
Overall
batch size
M
0.447
0.318
0.383
M50 M10
0.643
0.596
0.620
0.719
0.701
0.710
batch size 2
M
0.482
0.364
0,423
M30
0.654
0.575
0.615
M10
0.708
0.623
0.666
Consistency of pairwise comparison Pair- wise comparison eliciting displays high sensi- tivity to positional bias. We report the propor- tions of consistent rrong-over-wrong prefer- ence after flipping for LLAMA3-8B, GPT-3.5 and GPT-4o 0.286, 0.128, 0.598. Stronger LLMs can generate more consistent preference, and small LLMs not reliable evaluators.
batch size 5
M
0.479
0.383
0.431
10
0.733
0.605
0.669
M10
0.793
0.652
0.723
batch size 10
M,00
0.404
0.299
0.352
M00
0.723
0.598
0.661
M10
0.804
0.634
0.719
Wrongness margin affecting Accwow The accuracy of wrong-over-wrong judgement can also be affected by "wrongness margin" of wrong-over-wrong pair. For example, close- to-correct answer (e.g. "the sun almost all rises in the east") is obviously better than abso- lutely wrong answer (e.g. "the sun rises in the west"), while two wrong answers with close wrongness (e.g. "the sun sometimes rises the east" and "the sun sometimes perhaps rises in the east") are hard differentiate. We experiment Knowledge Crosswords dataset and evaluate the wrongness margin is defined as the the absolute difference correctly filled blanks. The results in Table 11 demonstrate that larger wrongness margin means easier evaluation, with average Acewow 0.696, 0.636 and 0.518 for wrongness margin being 3, 2, 1. This trend is particularly evident for highly wrong answer comparison, with having an accuracy of 0.533 while > 0 only has an accuracy of 0.463.
Table 7: Accuracy of scored-based wrong-over- wrong preference on Knowledge Crosswords and NLGraph (shortest path) dataset. The generator is LLAMA3-80 8B and the evaluator is GPT-4o. The number of answers generated to each question is 10. b 55 yields the best overall Accwow-
No or negative effect of mixing generators is ob- served. Experiments Table 2 demonstrate no ef- fect from mixing generators for ((p) 1 with consis- tency checks, for with Mi and oracle, with little change in average proxy scores discovered. And Ta- ble 2 also indicates an negative effect on fGPT440 with Ms0 with average improvement on AAcc across datasets drops by 28.9%. This suggests learning from the model's own mistakes is perhaps more im- portant for wrong-over-wrong ulignment.
IPO
SPPO
DRPO
DPO
.057
.006
0.00
-0.05
027.
.068
f({)
Pwrong
Acc
ECE
Figure 4: Evaluation different of prefer- ence optimization methods on less wrong, more correct and better calibration. The number is averaged over six experiment setups ((pairwise comparison with consis- tency check, score-based with Mio- score- based with Ms -enerr,e,,,,,or, Mix- Generator)) on NLGraph dataset.
Preference Optimization Approach To investi- gate the influence of different preference opti- mization methods, we also experiment wrong-over- wrong alignment with IPO (Azar al., 2024),
20

---

Published as a conference paper at ICLR 2025
ORPO (Hong et al., 2024) SimPO (Meng al., 2024) and SPPO (Wu al., 2024) on the NL- Graph dataset. Results in Figure demonstrate that preference optimization methods other than DPO sensitive noise and could not be employed for soenn--ver-wrooee aligmment without hy- perparameter tuning. SimPO collapses on all 6 experiment setups, IPO collapses on of them, and SPPO and ORPO show negative improvement compared original LLAMA3-8B. This suggests wrong-over-wrong alignment could be noisy and we should adopt stable preference optimization methods like DPO (Xu et al., 2024a; Chowdhury al., 2024).
Alignment domains without clear wrong- ness distinctions We wonder the effective- ness of wrong-over-wrong alignment is uni- versal and doesn't rely on clear wrongness distinctions within the task domain. To better prove this, we implement wrong-over-wrong alignment on SciBench (Wang et al., 2024b) and MedMCQA (Pal et al., 2022) where no such proxy to evaluate the wrongness of an- swers exists. SciBench is college-level scien- tific benchmark emphasizing scientific theorem application and numerical computation, while MedMCQA is a multiple-choice QA dataset in the medical domain. Experiment results in Ta- ble 8 indicate that wrong-over-wran wrong alignment is not universally helpful in domains without clear right-wrong distinctions. This suggests we should investigate more fine-grained separabil- ity among wrong answers and design a more sophisticated LLM-as-aajudge prompt.
MedMCQA
SeiBench
Method
Acc₁
ECE2
Acc₁
ECE
ORIGINAL
0.566
0.083
0.105
0.634
DELF-GENERATUR
SCORE M50
0.565
0.111
0.095
0.607
SCORE M10
0.550
0.082
0.068
0.660
MIX-GENERATOR
SCORE M50 SCORE M10
0.548
0.083
0.108
0.591
0.553
0.098
0.095
0.688
Table 8: Generalization datasets without distinc- tive wrong-over wrong separability. Wrong-over- wrong alignment is not helpful for domains with- out clear right-wrong distinctions.
Qualitative examples We present qualitative examples demonstrating the improvement of wrong- werrwwoog alignment on Knowledge Crosswords, Bio Generation, COM?, NL.Graph from Table 15 to Table 25.
B EXPERIMENT DETAILS
Model Details We employ LLMS in the experiments, the detailed information of LLMs used in this paper is follows: 1) LLAMA3-8B. through the META-LLAMA/METAA LLAMA-3-8B checkpoint on Huggingface (Wolf et al., 2020); 2) MISTRAL-7B, through the checkpoint on Huggingface; 3) GEMMA-7B, we di- rectly employ 7 instruction finetuned GEMMA-77 7Bs from Ivison et al. (2023); 4) GPT-3.5, through checkpoint on OpenAI API; 5) GPT-40, through the GPT-40-2024-0 by the
the DEITRALA/MIISRRA------SSTE
13 checkpoint on OpenAl API; 6) DEMINI-FLASHE through the GEMINI- 1.5-INI-..--LASH--- Vertex Al API; 7) GEMINI-PRG through the GEMINI -5-PP00001 by the Vertex AI API.
Dataset Details We employ 8 datasets in the experiments, for the main experiments and 4 left in the appendix. The detailed information is:
Knowledge Crosswords (KC) (Ding et al., 2023): We downsample questions containing exactly three blanks, and we create four answer options by randomly choosing 1, 2, or 3 correctly filled blanks. We sample 625 questions from the official dataset and split them into train, validation, test sets with an 8:1:1 ratio.
NLGraph (NLG) (Wang et al., 2023a; Zhang et al., 2024) We employ the "shortest path" subset for main experiments denoted as NL.rraph (shortest path) and "shortest path", "maximum flow" and "matching" subsets for experiments in the appendix denoted as NL.Graph (full). Correctness proxy for "maximum flow" and "matching" subset is: pmm@ (a q) = fx fal with fa being value given in and fgr being the ground-truth. We sample 380, 350 and 600 questions in "shortest path", "maximum flow" and "matching" subsets and split them into train, validation, and test sets with an 8:1:1 ratio.
21

---

Published as a conference paper at ICLR 2025
Bio Generation (BG) We employ the bio generation dataset and retrieved documents provided in FActScore (Min et al., 2023). We sample 625 questions from the official dataset and split them into train, validation, test sets with an 8:1:1 ratio.
COM (Fang et al., 2024): We employ the "2i" and "3i" question subsets of COM where we empirically find that Vera could generate more accurate scores. sample 625 questions from the official dataset and split them into train, validation, test sets with 8:1:1 ratio.
Hellaswag (Zellers al., 2019): Correctness proxy function for HellaSwag is: pl (a q) Vera(a a q) E [0, 1]. We sample 125 questions from the official validation split and split them sets with 1:1 ratio.
into validation,
Chess Puzzle (Lichess Team, 2023): We downsample the beginner-level chess puzzles with ELO ranging from 600 to 1000. We sample 625 questions from the official dataset and split them into ratio.
train, validation, test sets with an 8:1:1
SciBench (Wang et al., 2024b): We sample 625 questions from the official dataset and split them into train, validation, and test sets with an 8:1:1 ratio.
MedMCQA (Pal et al., 2022): We sample 500 questions from the official train split as train set. We sample 125 questions from the official validation split into validation, test sets with 1:1 ratio.
Implementation Details
Pairwise comparison.
The exact prompt we use for pairwise comparison is in Table 13.
Score-based The exact prompt we for score-based eliciting is in Table 14.
Generation of wrong-over-wrong dataset The pipeline used to generate Dwaw is described in
Algorithm
QLora finetuning We employ Unsloth and Transformers libraries for preference optimization.
We apply grid search on learning rate (1e-4, 5e-5, le-5), learning rate scheduler (cosine, cosine with restart and reduce on plateau), weight decay (0, le-5, Ie-3) and number of train epochs (1, 3, 5) for main experiments and right-over-wrong alignment experiments. We use random seed 42 for all of our experiments.
Experiment Details In Table 1, batch size 5 for all score methods due optimal empir- ical results. In Table 2, we define FActScore > 0.9 as "correct" on Bio Generation dataset; all the preferences are elicited from GPT-4o. Table the best wroogg-ove-wrooe alignment setup for Knowledge Crosswords, Bio Generation, COM2 and NLGraph are score-based with M55 on self- generated data, pairwise comparison with consistency check on mix-generated data and score-based with M on self-generated data; Bio Generation dataset is omitted because there is not well-defined ground-truth on this dataset. Correlations between task accuracy, confidence and other wrong-ove over- wrong preference eliciting methods are displayed in Figure 5 and Figure 6.
22

---

Published as a conference paper at ICLR 2025
LLAMA3-8B
GPT-3.5
GPT-40
Method
Margin
KC BG COM NLG KC BG COM' NLG KC
BG COM NI.G Overall
MODEL-INDEPENDINT
HEURISTIC
M100
.91
.866
.888
498
.519
5044
.43
466
547
441
.43
.502
.489
CONSISTENCY Mi00
.296
.287
360
.318
.279
336
311
.325
.313
.328
LLAMA3-SB Evaluator
LOGITS
M100
.527
.540
.525
.508
.546
.166
.506
.477
.528
.991
.440
.546
.519
SCORE
M100
.499
.36
.385
.455
,496 .376
.88
434
.38
.37
.525
.273
.400
GPT--355 1..5a. Evaluator
LOGITS
Ma00
528
508
.552
.479
.044
SCORE
M
212 .352
.287
.323
,289 .327 .458
.22
.132
.273
.269
.307
.335
GPT-4o Evaluator
LOGITS
Mon
.566
.478
.540
.526
all
.593
.507
.51
512
.515
.577
.501
.526
.537
.562
PAIRWISE
filter
.533
.566
.558
.661
.490
.604
.549
.624
M
.385
.450
.383
540
351
.484
.408
.481
.353
.467
.369
.393
SCORE
Mso
.677
.544
591
.617
.661
.54
.50
.581
.639
Mi0
.793
.795
.534
.652
.655
.711
.684
.586
.520
.578
.709
MISTRAI--7B8 as Evaluator
all
.433
.471
.462
.492
.44
.482
.524
.501
.453
.503
.471
.514
.482
PAIRWISE
filter
.551
.427
.451
.505
.531
.491
.574
.538
.542
.365
.542
.560
.514
Mj00
331
.356
.370
.389
.37
348
.431
.453
.051
.305
.354
.337
.366
SCORE
M50
.41
.538
.505
.527
6622
512
.584
.599
.102
.514
.632
.576
.549
Mi0
.683
5558
.486
.558
.73
.513
.626
.565
.22
.557
.677
.565
.565
GEMINI-FLASH as Evaluator
all
.31
.505
.5000
.509
.5119
.473
.538
.574
.415
.471
.519
.531
.509
PAIRWISE
filter
.574
.540
.517
.520
.557
.444
.561
.629
.410
.373
.556
.565
.529
Mj00
482
.433
.433
.62
.521
.368
.497
.455
.453
.377
.560
.429
.420
SCORE
Myp
.642
.584
.510
.517
.661
.497
.630
.616
.288
.510
.674
.600
.567
Mι0
746
.606
.504
.551
.719
.483
.756
.665
.800
.593
.817
.576
.599
GEMINI-PRO as Evaluator
all
.539
.538
.493
.513
.558
.435
.409
.554
.396
.437
.522
.530
.509
PAIRWISE
filter
.598
5888
.547
.564
628
.471
.557
.658
.43
.412
.560
.569
.557
Mj00
.494
.442
.450
.397
.555
.361
.483
.476
.344
.363
.460
.417
.428
SCORE
Mso
.653
.612
.551
.50
662
.200
.600
.631
.66
511
.567
.660
.811
Mj0
.742
.695
.573
.33
.662
.524
.681
.662
.50 .600
.732
.572
.633
Aggregated Evaluator
all
.255
.155
.482
.509
.226 .477
.508
.542
.445
.484
.495
.520
.502
PAIRWISE
filter
.577
.574
.498
.489
.579
.487
.549
.600
.457
.416
.531
.528
.524
M
.527
.540
.525
.588
.537
.512
.529
.478
.500
.509
.459
.543
.144
LOGITS
Mso
.41 .568
.524
.505
.559
.521
.526
.470
.299
.515
.453
.575
.54
Mio
.599
.669
.42
.588
.556
.594
.533
.429
.300
5544
.409
.611
.511
Ma
.416
.392
.396
.378
.463
.355
.472
.441
.323
.341
.439
.355
.398
SCORE
Msn
.619
.588
.513
.546
.670
.524
.602
.615
435
512
.597
.588
.665
Mj0
.701
.400
.512
.571
.721
.535
.692
.613
504
584
.678
.561
.609
Table 9: Accuracy of wrong-over-wrong preference elicited from MISTRAI--7B. GEMINI-FLASH and GEMINI-PRO, and also the accuracy of heuristic, consistency- based, logits-based and score- z(4)
based methods with margin of M100 that are not displayed in the main paper. JCPT-40 with My still achieves the best weighted average Accwow = 0.709 across all datasets.
23

---

Published as a conference paper at ICLR 2025
LLAMA3-8B
GPT-3.5
GPT-40
Method
Margin
Overall
NLG
CP
NLG
CP
NLG
CP
Moden-indeenndnn
M ro0
0.510
0.506
0.472
0.504
0.535
01,481
0.501
HEURISTIC
M50
0.509
0.506
0.468
0.519
0.547
0.464
0.502
M10
0.457
0.512
0.476
0.501
0.486
0.475
0.478
Mou
0.359
0.281
0.323
0.300
0.269
0.301
0.318
CONSISTENCY
Mso
0.544
0.478
0.514
0.499
0.432
0.498
0,505
Mio
0.509
0.535
0,506
0.492
0,302
0.468
0.478
LLaMA3-8B as Evaluator
all
0.501
0,516
0.477
0.493
0.551
0.504
0.503
PAIRWISE
filter
0.586
0.560
0.538
0.504
0.500
0.510
0.545
M00
0.527
0.495
0.469
0.507
0.574
0.525
0.511
LOGITS
M50
0.539
0.488
0.456
0.508
0.625
0.532
0.518
M
0.562
0.512
0.453
0.485
0.656
0.525
0.530
M you
0.426
0,411
0.421
0,406
0.412
0.372
0.417
SCORE
Ms0
0.547
0.524
0.536
0.534
0.593
0.529
0.546
0.529
0.564
0.552
0.545
0.636
0.511
0.557
GPT-3.5
Evaluator
all
0.505
0,485
0.514
0.469
0.507
0.499
0.502
PAIRWISE
filter
0.802
0.547
0.590
0.585
0.882
0.535
0.642
M000
0.479
0.518
0.489
LOGITS
M50
0.472
0.521
0.484
M 0.368
0.492
0.515
0.498
jou
0.311
0.401
0.355
0.333
0.219
0.359
SCORE
M30
0.579
0.510
0.540
0.538
0.593
0.438
0.553
M10
0.578
0.496
0.527
0.501
0.614
0.635
0.553
GPT-4o as Evaluator
all
0.560
0.554
0.512
0.575
0.503
0.546
PAIRWISE
filter
0.586
0.519
0.569
0.520
0.613
0.512
0.568
M
0.546
0.525
0.564
LOGITS
Ms
0.577
0.532
0.594
M
0.630
0.617
M00
0.352
0.346
0.344
0.402
0.399
0.381
0.361
SCORE
M50
0.573
0,538
0.513
0.554
0.487
0,546
Mj0
0.551
0.535
0.490
0.529
0.397
0.541
Table 10: Accuracy of wrong-over-wrom preference on NLGraph (full) and ChessPuzzle dataset.
JGrras with consistency check achieve the best weighted average Accwo 0.642 across all datasets.
(p)
Algorithm Dwow generation pipeline
Input: Original dataset Do ((g d
Dwow =
for i in 2, do
Sample m answers from L.M:: {a (01m
Filter out correct answers: {a!) a (1) 1==1 ~ "L
# "
for aj (i) (i)
.a in combination(11 (1) pm
J)=1) do
if fuuM(a)) > 0g
(I)
q (
then
ac: a, = a (0)
a
else if fLuM a
a
(i)
(
then
Oc: a 0 0
end if
Append (q ace ar) to D How
end for
end for
return Dwaw
24

---

Published as a conference paper at ICLR 2025
Margin
3 0
3 - 1
2 0
3 2
2 1
1 1 0
logits (self)
Ma0
0.571
0.543
0.557
0.500
0.520
0.502
M50
0.590
0.553
0,551
0.514
0.533
0,531
M10
0.649
0.576
0.625
0.540
0.464
0.629
score (L..MAA3-BB)
M100
0.574
0.527
0,512
0.423
0.467
0,459
Mso
0.749
0.711
0,658
0.541
0.610
0.587
M
0.758
0.774
0.714
0.607
0.667
0.645
score (GPT-3.5)
1
0.308
0.238
0.233
0.189
0.188
0.236
Mso
0.463
0.454
0.463
0.399
0.392
0.431
Mo
0.780
0.776
0.670
0.617
0.585
0.434
score (GPT-40)
M,00
0.753
0.668
0,575
0.509
0.481
0.339
Mso
0.917
0.895
0,812
0.724
0.764
0.508
Mj0
0.958
0.940
0.829
0.792
0.867
0.529
average
0.696
0.665
0.607
0.533
0.558
0.463
Table 11: Preference accuracy for wrong-over- pair with different wrongness margins. a > means preferring answer filling in blanks correctly than answer filling in b blanks correctly. To have an overview, we also include correct answers which fill in 3 blanks correctly. We experiment on score-based eliciting with 3 LLM evaluators respectively and logits-based with self-evaluation. The results are an average on generators. The largest wrongness margin 3 > accurate preference.
yields the most
BG
NLG
KC
COMP
BG
NLG
0.7
0.6
0.7
AcCwow
0.5
ACCwaN
0.6
0.4
Pllll 0.g03
0.5
P.KC 0.440
0.254
0.3
P_0G = 6.123
P.COM = 6.732
0.4
P.BG
0.505
0.491
P.KLG - 979
PLCONN
P.NLG 0.730
0.2
0.4
0.6
0.8
task accuracy
0.2
0.6
task accuracy
0.7
0.6
AcCwow
0.5
ACCwow
0.6
0.4
Pl -0.205
0.5
P2 4646
+0.077
0.3
P.CO 0.723
PBBG 0.325
0.4
Pikcc -0.760
P.BG 00.49
P.NLG 0.691
P.CON 0.374
0.3
P.NLG 0.553
0.5
0.6
0.7
0.8
0.9
task confidence
0.7
0.9
task confidence
Figure 5: Correlation between task accuracy, confidence and Accwow for pairwise compar- ison with consistency check. Data points from all LLMs we used to elicit wong-over- wrong preferences. P. stands for Pearson relation coefficient.
Figure 6: Correlation between task accuracy, confidence and Acc for score-based with Mso. Data points are from all 3 L.MSs we used elicit wrong-over-wrong preferences. P stands for Pearson correlation coefficient.
25

---

Published as a conference paper at ICLR 2025
KC
COM2
Method
Pmicro Pmacro Rumtero Fliccoo Pmicro P macco Ramccoo F micro
ORIGINAL
0.564 0.561 0.559 0.559 0.689 0.679 0.682 0.676
SELF-GENERATOR
PAIRWISE FILTER 0.627 0.622 0.622 0.621
0.699 0.692 0.697
0.689
SCORE Ms0
0.597 0.598 0.591
0.592
0.675 0.663 0.666
0.659
SCORE M10
0.584 0.581 0.579
0.579
0.683
0.674 0.677
0.666
ORACLE
0.580 0.576 0.578
0.576
0.700 0.688 0.689
0.682
MIX-GENERATOR
PAIRWISE FILTER 0.574 0.572 0.569 0.570
0.698 0.687 0.692
0.686
SCORE Mso
0.590 0.592 0.587
0.588
0.674
0.658 0.665
0.659
SCORE Mj0
0.565 0.563 0.562
0.562
0.680 0.671 0.676 0.666
ORACLE
0.581 0.579 0.580 0.577 0.699 0.684 0.691 0.684
Table 12: Precision, recall and Fl on Knowledge Crosswords and COM Wrong-over wrong align- ment indeed can make LLMs generate more correct answers regardless of label distribution.
Prompt: You are helpful assistant whose goal is to select the preferred Al model's s output for given instruction based on their correctness.
Instruction:
[question here]
Output 1:
[answer d1 here]
Output 2:
[answer 0g here]
Please first briefly describe your reasoning (in less than 100 words), and then conclude with "Preferred output: 2." in the last line. Let's think step-by-step.
Table 13: Prompt for pairwise comparison wrong-over-wrong preference eliciting.
Below is an instruction from a user and five candidate responses. Please grade the correctness of each response on S-point scale from 0 to where 0 is the lowest correctness, and 5 the highest correctness. Only use integers for the ratings.
Prompt:
Instruction:
[question here]
Response 1:
[answer here]
Response 2:
[answer 02 here]
Response 3:
[answer dg here]
Response 4:
[answer d4 here]
Response 5:
[answer ag here]
Please first briefly describe your reasoning (in less than 100 words for each response), followed by new line in the format of "Score: <rating>" Let's think step-by-step.
Table 14: Prompt for score-based maagg-over-wroogg preference eliciting.
26

---

Published as a conference paper at ICLR 2025
Question:
Instruction: Pick the correct answer for each blank that satisfies all the given constraints. Con- straints: (Miles_Goodmin) Goodman, WroteMusicFor, blank 3); (Julien_Temple, directed, blank 1); (blank 2, actedIn, blank 3); (blank actedIn. blank 1); Alexandr,,,, actedIn, blank 3);
Correct answer:
blank I: Earth. Girls Are Easy, blank 2: Damon_Wayans,
Wayans, blank 3: Blankman
Original answer:
Let's analyze the question step by step: 1. Julien Temple directed blank 1. From the options, conclude that blank 1 is either Miles Goodman wrote music for blank 3. From the options, we can conclude that blank 3 is either "Blankman" or "On Guard (1997 film)". 3. blank 2 acted in blank 1 and blank 3. From the options, we can conclude that blank is the same person both cases. The only options that satisfy this constraint we can conclude that blank is either "Blankman" or "On ConnGGard_.CC (awrdd(199__fmm)) previous conclusion, blank 3 must be "Blankman". 5. Putting all together, we can conclude that the correct answer is B. blank 1: Burn_ Ep Barnes, blank 2: Theroux, blank 3: Blankman.
Easy" "Burn'Emm_ "aann'Em_Up_Bunni _p__ennes".........ae 2.
B and D. 4. Jason Alexander acted in blank 3. From the options, Given the
Final answer: B.
Less wrong answer:
Let's analyze the question step by step: 1. Jumien.Temple, directed, blank 1: Julien Temple directed Earth. Girls Are Easy, blank 1 must be Earth. Girls-Are Eas. 2. blank 2, acte- dln, blank 1: From step 1, blank is Sarrth.GrrlsMAae
Easy. So, blank 2 must have acted in Earth, Girls_Are Easy. Let's call blank 2 X. 3. blank 2, actedIn, blank 3: From step 2, X acted in Girls Are Easy and also acted in blank 3. 4. Jason Alexander, actedIn, blank 3: Jason Alexander acted in blank 5. blank 2, actedIn, blank 1: From step 2, X acted in Earth_Girls_4 Girls Easy again (because it's the same film). 6. Goodman, WroteMusicFor, blank 3: Miles Goodman wrote music for blank 7. Julien. directed, blank 1: We al- ready knew this, but it's repeated to ensure consistency.
Now, let's analyze the options: blank 1: Burn 'Em Up Barnes, blank Justin Theroux, blank 3: On OnarG (oaudd(199)) (197_fflm) Burn"'Em_Upp 'Em_Up__meee Barnes is not directed by Julien Tem- ple, and Justin Theroux didn't act in On Gun_Guar_ (1997 film). Discarded. B. blank 1: Earth_Girts Girls-rrerrreere.....ee Easy, blank 2: Damon_Wayane that blank is Earth.Girls
blank 3: Blankman: We already know Easy. Damon Wayans didn't act in Garth_Girls.ae Easy, and Blankman isn't the same film as On Guard (1997 film). Discarded. C. blank 1: Burn 'Em Up Barnes, blank 2: Justin Theroux, blank 3: Blankman: Burn 'Em Up Barnes is not directed by Julien Temple, and Justin Theroux didn't 't act in Blankman. Discarded. D.. blank I: Earth_ E
Arres A blank 2: T blank 3: Blankman: This option fits all the
constraints: blank 1 is Eirth Are Easy (step 1) blank 2 is Justin_Therouu (step 2) blank is Blankman (works with steps 3, 4, and 6)
So, the final answer D. blank 1: Earth Girls Are Easy, blank Justin Theroux, blank 3: Blankman
(Correctly filled blanks 2)
More correct answer:
Let's break down the constraints step by step: 1. Goodmaa, WroteMusicoorr, blank 3) This constraint tells us that Miles Goodman wrote music for a movie, and the title of that movie directed, blank 1) This constraint tells us that Julien Temple directed a movie, and the title of that movie is blank 1. 3. (blank 2, actedIn, blank 3) This constraint tells us that blank acted in movie, and the title of that movie is blank 3. 4. (blank 2, actedIn, blank 1) This constraint tells us that blank 2 acted in another movie, and the title of that movie is blank 1. (Jason Alexander, actedIn, blank 3) This constraint tells us that Jason Alexander acted in movie, and the title of that movie is blank 3.
is blank 3. 2.
Table 15: Examples showing the effect of wrong-over-were wrong alignment on Knowledge Crosswords dataset (part) 1).
27

---

Published as a conference paper at ICLR 2025
Now, let's analyze the options: A. blank 1: Burn_ Burn_"Em_Upp 'Em_Up_Barnes, blank 2: Justin_Theroux, blank 3: Blankman Burn" "Em_Up_ Barnes is not a well-known movie, and Blankman is not a movie that was directed by Julien Temple or written music for by Miles Goodman. B. blank 1: Earth Girls Easy, blank 2: Justin Theroux, blank On Guard (1997 film) Earth.Girls Easy is real movie, but On Guard film) is not movie that was directed by Julien Temple or written music for by Miles Goodman. Additionally, Justin Theroux did not act in On (1997_
C. blank I: Earth. Girhh Are Easy, blank 2: Hamon_Wayane blank 3: Blankman Damon Wayans is an actor, but he did not act in
Gre
The only option that satisfies all the constraints is: C. blank 1: Earth Girls Are Easy, blank 2: Damon Wayans, blank 3: Earth Girls Are Easy This option fills in the blanks fol- lows: Julien Temple directed Earth_ _ (constraint 2) Damon Wayans acted in Baath_Girrs_Arr Easy (constraint 4) Damon Wayans acted in SeardhGGrls_Maee Erayy (con- straint 3) Jason Alexander acted in Sarth.GGrls_Ai Easy (constraint 5) Miles Goodman wrote music for Earth. Girls Are Easy (constraint 1)
The movie Earth.Girls and Damon Wayans and Jason Alexander both appeared in the film. Miles Goodman wrote the original songs for the movie. (Correctly filled blanks ++ 3)
Easy is 1988 musical comedy film directed by Martin Davidson,
Better calibrated answer:
Let's analyze the question step by step: 1. (Julien. Temple, directed, blank 1) This constraint states that Julien Temple directed a movie, and the title of that movie is blank 1. 2. (blank 2, actedIn, blank 1) This constraint states that blank 2 acted in the same movie in step 1. 3. (blank 2, actedIn, blank 3) - This constraint states that blank 2 acted in another movie, and the title of that movie is blank 3. 4. Moodma, WroteMusicFor, blank 3) - This constraint states that Miles Goodman wrote music for the same movic as in step 3. 5. (Jason_Alexand)] actedIn, blank This constraint states that Jason Alexander acted in the same movie as in step 3.
Now, let's analyze the options: A. blank 1: Earth Girls Easy, blank 2: Damon Wayans, blank 3: Blankman This option is incorrect because Earth_Girls, Are Easy is 1984 film directed by Sidney Lumet, not Julien Temple. Additionally, Justin Theroux was not in- volved in that film. B. blank 1: Burn_ "urn_'E__U_ Saam-Up_Barnee blank 2: Theroox,, blank 3: Blankman This option is incorrect because Justin Theroux was not involved in the 1984 film Burn 'Em Up Barnes, which is TV movie. Additionally, Jason Alexander was not involved in that film. C. blank 1: Burn 'Em Up Barnes, blank 2: Justin Theroux, blank 3: On_ In_Guard_(11) (1997d_ film) This option is incorrect because Jason Alexander
not involved in the 1997 film On_ Guard_ which is a TV movie. Additionally, Miles Goodman was not involved in that film. D. blank 1: Marth_GirlsAAr Are blank 2: Justin Theroox, blank 3: Blankman This option is incorrect because Justin Theroux was not involved in the 1984 film Easy, Easy and Jason Alexander was not involved in that film.
After analyzing the options, the only remaining possibility is: B. blank 1: Burn 'Em Up Barnes, blank Justin Theroux, blank 3: Blankman This option satisfies all the constraints: - Julien Temple directed the movie Burn"'mm_Up 'ee______Baaneee (1990), which is blank I. Justin Theroux acted in Burn_ Burn.'Em_Up EEm_Up_BBaaeee. also acted in Blankman (1994), which is blank 3, along with Jason Alexander. Miles Goodman wrote music for Blankman, which is blank 3.
which is blank 2. Justin Theroux
Therefore, the final answer is: B. blank 1: Burn_ 'Em Up Barnes, blank 2: Theroux, blank 3: Blankman
(Even the answers are equally wrong, NLL increases 0.129 - 0.185)
Table 16: Examples showing the effect of wrong-over-wrong alignment on Knowledge Croswords dataset (part 2).
28

---

Published as a conference paper at ICLR 2025
Failed answer:
Let's break down the constraints step by step: 1. Gooddan, wroteMusicFor, blank 3): This constraint tells us that Miles Goodman wrote music for movie, and the title that movie that only options C and D mention Miles Goodman
is blank 3. From the options, writing music. 2. (Julien_Temple, directed, blank 1): This constraint tells us that Julien Temple directed movie, and the title of that movie is blank 1. From the options, options and mention Julien Temple directing a movie. The title of that movie either Burn_" 'Em_ 1Em__p_Barne Barnes Earth, Marth_Girls_Aii Girls_Are_Fasyy 3. (blank 2, actedIn, blank 3): This constraint tells us that blank 2 acted in movie, and the title of that movie is blank 3. From the previous step, we know that blank is either Burn 'Em Up Barnes or Earth Girls Are Easy. Since blank is the same in both options B and C, we can conclude that blank acted in movie with the title blank 3, which either Burn. Boum_'mm_U] EEm_.eeBarnees or Girtth Easy. 4. (blank 2, actedIn, blank 1): This constraint is redundant, as it's already known that blank 2 acted in movie with the title blank I, which either Burn 'Em Up Barnes or Earth Girls Are Easy. 5. Jason_Alexander
see that only
actedIn, blank 3): This constraint tells us that Jason Alexander acted in movie, and the title that movie is blank 3. From the previous steps, we know that blank is either Burn 'Em Up Barnes or Earth Girls Are Easy. Since Jason Alexander acted in of those movies, we can conclude that Jason Alexander acted in a movie with the title blank 3. Now, let's analyze the remaining constraints: Justin Theroux needs to be blank 2, it's mentioned in constraints 3 and 4. Blank 3 needs to be either Burn_'Em__ Em__pp Saa.UUpBBaar Barnes or Farth_Girls_Ars LeessAre__aaa Easy, it's mentioned in constraints 1, 3, and Blank needs be ei- Easy, as it's mentioned in constraints 2 and 4.
ther Burn_ 'Em_Up Samm-U_Baaann Barnes or Earth_Girls The only option that satisfies all the constraints is: B. blank 1: Burn_ 'Em.Up Sam_Up_Baann Barnes, blank 2: Justin Theroux, blank 3: Earth Girls Are Easy This option satisfies all the con- straints: Miles Goodman wrote music for Earth Girls Are Easy. Julien Temple di- rected Burn_ "Em_Up__aant Garlh. Are Easy Jason Alexander acted in Earth. MaathhGirrs.cc Are Easy.. (Correctly filled blanks 1 + 0)
Burm_'Em_UU
Justin Theroux acted in both Burn Baaee'Em__ppBrrre and
Table 17: Examples showing the effect of wrong-over srong-over-wrong wrong alignment on Knowledge Crosswords dataset (part 3).
29

---

Published as a conference paper at ICLR 2025
Question:
Tell me a bio of Luiz Inácio Lula da Silva.
Original answer:
Luiz Inácio Lula da Silva, commonly known as Lula, is a Brazilian politician who served as the 35th President of Brazil from 2003 to 2011. He one of the most popular and influential politicians in Brazil's recent history. Early Life: Lula was born October 27, 1945, in Garan- huns, a small town in the northeastern state of Pernambuco, Brazil. His father was a coffee farmer and his mother was seamstress. He was the fourth of seven children, and his family struggled financially throughout his childhood. Lula grew up in a poor and rural area, where he learned the value of hard work and the importance of education. He dropped out of school at the age of 12 to work in a textile factory, where he eventually became a union leader. This early experience in labor organizing would shape his future political career. Political Career: Lula's political career began in the late 1970s, when he became involved with the Brazilian Workers' Party (PT), a left-wing party founded in the 1980s. He quickly rose through the ranks of the party, becoming its presidential candidate in 1989. Although Lula lost the election to Fernando Collor de Mello, he continued to build his political base and gained popularity for his fight against corruption and social inequality. In 1994, he was accused of corruption and money laundering, which led to a conviction and a four-year prison sentence. However, in 1998, the Supreme Court overturned Lula's conviction, citing lack of evidence. This cleared the way for him to run for president again in 2002. He won the election in a landslide, receiving nearly 65Presidency: During his presidency, Lula implemented a series of reforms aimed at reducing poverty and inequality. He increased social spending, increased the minimum wage, and ex- panded access to education and healthcare. He also launched a number of economic initiatives, including a debt reduction plan and an infrastructure development program. Lula's presidency was marked by strong economic growth, which lifted millions of Brazilians out of poverty. He also gained international attention for his leadership on issues such as climate change and so- cial justice. Presidential Terms: Lula served two consecutive terms as president, from 2003 to 2011 1. During his first term, he focused on addressing poverty and inequality, and implemented range of social welfare programs. During his second term, Lula emphasized economic de- velopment and infrastructure projects. He invested heavily in public transportation, energy, and telecommunications, and launched a series of initiatives to promote sustainable development. Investigation and Charges: After leaving office in 2011, Lula was investigated for corruption and money laundering. He was accused of receiving bribes from a construction company, OAS, in exchange for favors and contracts. He denied the allegations, but was arrested and charged in 2018. In 2019, Lula was convicted and sentenced to nine years and six months in prison. He maintained his innocence and appealed the conviction, questioning the faimesss of the trial. Return to Politics: Despite his imprisonment, Lula remained a beloved figure in Brazil, and his popularity continued to grow. In 2021, the Supreme Court ordered his release from prison, citing the unfairness of the trial. Lula was subsequently reinstated as a senator and began to reorganize the Workers' Party. 2022, Lula announced his candidacy for the 2022 Brazilian presidential election, and won the election in landslide. He is currently serving as the 39th President of Brazil, having taken office on January 1, 2023. Legacy: Luiz Inácio Lula da Silva's legacy is complex and multifaceted. On the one hand, he is credited with reducing poverty and inequality, and promoting social justice and economic development in Brazil. On the other hand, his presidency was marked by corruption scandals, and his own conviction and imprison- ment have raised questions about his integrity and accountability. Despite these controversies, Lula remains towering figure in Brazilian politics, and his commitment to social justice and poverty reduction has inspired generations of Brazilians. His political career is a testament to the power of grassroots organizing and the importance of fighting for social justice, even in the face of significant challenges and setbacks.
Table 18: Examples showing the effect of wrong-over-wrong alignment on Bio Generation dataset (part 1).
30

---

Published as a conference paper at ICLR 2025
Less wrong/ More correct / Better calibrated answer:
Luiz Inácio Lula da Silva, commonly known as Lula, is Brazilian politician and former trade unionist who served as the 34th President of Brazil from 2003 to 2011. He was born on October 27, 1945, in Garanhuns, small town in the state of Pernambuco, Brazil. Lula's early life was marked by poverty and hardship. His parents were both poor and separated when Lula was young, and he was forced to work at a young age to support his family. He eventually moved to São Paulo and became involved in trade union activism, eventually becoming one of the key figures in the Brazilian Workers' Central Union (CUT), Lula's political career began in the 1980s, when he was elected the leader of the Metalworkers' Union and became prominent figure in the Brazilian labor movement. He was one of the founders of the Workers' Party (PT) in 1980, and he quickly became the party's s most prominent figure. In 2002, Lula became the first worker and the first non-PSDB (Brazilian Social Democracy Party) candidate to run for President of Brazil. Despite initial skepticism about his candidacy, he won the election in landslide, receiving over 61During his two terms in office, Lula implemented a number of significant reforms and policies. He implemented a series of social and economic programs aimed at reducing poverty and income inequality, including the Bolsa Família program, which provided financial assistance to low-income families, and the Fome Zero program, which aimed to eradicate hunger and poverty. Lula also played key role in shaping Brazil's foreign policy, particularly in the areas of trade and international relations. He was one of the key figures in the formation of the BRICS (Brazil, Russia, India, China, and South Africa) alliance, and he played a significant role in shaping the policy of the G20 and the United Nations. After leaving office in 2011, Lula remained a prominent figure in Brazilian politics and continued to play significant role in shaping his party's 's policies and candidates. In 2018, he was elected as the Chair of the PT, and he became the leading figure in the country's 's political opposition, advocating for changes to the country's political system and advocating for the nomination of his chosen successor, Fernando Haddad, in the 2018 presidential election. In 2018, Lula's candidacy in the presidential election was barred due to a corruption conviction, and he was replaced as the PT's presidential candidate by Fernando Haddad. However, Haddad ultimately lost the election to Jair Bolsonaro. Lula's 's legacy in Brazil is complex and controversial. On one hand, he widely credited with improving the lives of millions of Brazilians, particularly the poor and the working class, through his social and economic policies. On the other hand, he was also accused of corruption and was ultimately convicted of accepting bribes and ordered to serve prison sentence. He is currently appealing his conviction and is being held in federal prison in Brazil. Despite his current legal situation, Lula's 's political career and legacy continue to be subject to significant debate and discussion in Brazil and around the world. His presidency marked a significant shift in the country's 's political landscape, and his policies and legacy continue to have significant impact Brazilian politics and society.
(FActScore improves 0.836 + 0.925, drops 0.512 + 0.416)
Table 19: Examples showing the effect of wrong-over-wrong alignment on Bio Generation dataset (part 2).
31

---

Published as a conference paper at ICLR 2025
Failed answer:
Luiz Inácio Lula da Silva, commonly known as Lula, is Brazilian politician and former trade unionist who served as the 34th President of Brazil from 2003 to 2011 and again from 2016 to 2018. Early Life and Career: Lula was born October 11, 1945, in Garanhuns, Pernambuco, Brazil, to Armando da Silva and Hermilda da Silva. His early life was marked by poverty and hardship, with his family struggling make ends meet. Lula's mother died of complications related to a gas explosion when he was just 10 years old, and he was subsequently raised by his aunt. Lula's s early career was marked by his involvement in the trade union movement, He became involved with the Metal Workers Union, where he quickly rose through the ranks and became one of the most prominent union leaders in Brazil. During this time, he became known for his charismatic leadership style. his ability to connect with ordinary people, and his advo- cacy for workers' rights. Political Career: In 1980, Lula co-founded the Workers' Party (PT), left-wing political party that sought represent the interests of working-class Brazilians. He became the party's first president and held the position until 2002. In 2002, Lula became the first trade unionist to run for and win the presidency of Brazil, defeating the incumbent Fer- nando Henrique Cardoso. He was re-elected in 2006 and served two terms as president until 2011. During his first tenure president, Lula implemented a range of policies aimed at reduc- ing poverty and income inequality in Brazil. He increased access to education and healthcare, and implemented policies stimulate economic growth and job creation. In 2010, Lula was succeeded by his former minister and protégé, Dilma Rousseff, who became the first female president of Brazil. However, Rousseff's presidency was later marred by corruption scandals and political instability, culminating in her impeachment and removal from office in 2016. Lula returned polities and ran again for the presidency in 2014, but was not eligible to run due to a constitutional ban on former presidents from holding the office again. He subsequently threw his support behind Rousseff's re-election bid, but she was ultimately impeached and re- moved from office. In 2018, Lula ran for the presidency again and won the election, becoming the first and only person to be re-elected as President Brazil. However, his presidency was short-lived, as he was impeached and removed from office in August 2016, citing corruption allegations and constitutional violations. Lula was subsequently replaced by his vice president, Michel Temer. Personal Life: Lula has been married to Maria de Deus Inácio da Silva (known as Marinha) since 1970, and they have four sons together. In 2018, Lula was sentenced to 12 years and month in prison for corruption and money laundering, stemming from the "Opera- tion Car Wash" corruption scandal. He has since remained in jail, and his legacy and political future remain uncertain. Legacy: Lula's presidency was marked by significant economic and social changes in Brazil, including a reduction in poverty and income inequality, as well as an unprecedented level of social mobility. He is widely regarded one of the most important fig- ures in Brazil's 's recent history, and his legacy continues to shape Brazilian politics and society. Despite his political setbacks and legal troubles, Lula remains powerful and influential figure in Brazilian politics, and his name continues to be mentioned as potential candidate for future elections.
(FActScore drops 0.836 - 0.770)
Table 20: Examples showing the effect of wrong weee-wrogg alignment on Bio Generation dataset (part 3).
32

---

Published as a conference paper at ICLR 2025
Question:
Instruction: As an expert in commonsense reasoning, your task is to provide a concise response to question based on the given context. The question focuses studying the causes, effects, or attributes of personas related to the given context. Context: Noah had been planning surprise visit for Everly. As part of the surprise, Noah buys gift for Everly. When Everly arrives at the airport, Noah picks Everly up. Question: What event or state is both what Everly wants to do after Noah buys gift for Everly and also what Everly wants to do after Noah picks Everly up at the airport?
Correct answer:
Ranking by commonsense plausibility: Everly says thank you > Everly shocked > Evvrrly go out to eat > Everly rest
(The commonsense plausibility is given by Vera.)
Original answer:
Here's ' the step-by-step analysis: 1. The question asks about event state that is common to both scenarios: after Noah buys a gift for Everly and after Noah picks Everly up at the airport. 2. The context suggests that Noah is planning surprise visit for Everly, and buying gift is part of the surprise. 3. When Noah buys a gift, it implies that Everly didn't 't expect the gift, and therefore, will likely be surprised when she receives it. When Noah picks Everly up at the airport, it's likely that Everly didn't expect the pick-up, and therefore, will also be surprised. 5. The common event or state that arises from both scenarios Everly's surprise. Therefore, the correct answer is B. Everly is shocked.
Less wrong answer:
Let's break down the question step by step: 1. The question asks about an event or state that Everly wants to do after Noah buys a gift for her and also after Noah picks her up at the airport. 2. We know that Everly has been anticipating a surprise visit from Noah, which suggests that she's looking forward to seeing Noah. 3. When Noah buys a gift for Everly, it's likely that she wants to see or meet Noah to receive the gift, which means she wants to be with Noah. 4. Similarly, when Noah picks Everly up the airport, she likely wants to be with Noah, she was expecting a visit from him. 5. Therefore, the event or state that Everly wants to do after Noah buys a gift for her and after Noah picks her up the airport is likely to be: * **Being with Noah** Since options B-D do not seem related to this event, the correct answer is not present in the options. If had to choose, would say that the correct answer not present in the options (**N/A**) **).
(Instead of giving a wrong answer, LLM choose to abstain.)
More correct answer:
Let's 's' break down the question step by step: 1. The question asks about an event state that is common to both situations: after Noah buys a gift for Everly and after Noah picks Everly up at the airport. 2. In both situations, Everly is receiving service action from Noah (a gift or transportation). 3. Given the context, it's likely that Everly wants to reciprocate or respond positively Noah's is actions. 4. Among the options, only one relates to Everly's response to Noah's 's actions: A. Everly says thank you. 5. Therefore, the most likely answer is A. Everly says thank you, as it is common outcome for Everly after receiving a gift being picked up by Noah. Final answer: A. Everly says thank you.
(LLM gives right answer.)
Table 21: Examples showing the effect of wrong-over-wrong alignment on 2OM2 dataset (part 1).
33

---

Published as a conference paper at ICLR 2025
Better calibrated answer:
Let's break down the question step by step: 1. The question asks about an event or state that is common to scenarios: after Noah buys gift for Everly and after Noah picks Everly up at the airport. 2. In both scenarios, Everly is expected to be pleased or surprised, given the context of the surprise visit and gift. The correct answer must capture this idea of Everly's positive reaction. 4. Options A, B, and D) are not directly related to Everly's ' actions reactions. 5. Only option C. "Everly rests", too vague be relevant to the context. Final answer: None of the options fit the description. The correct answer is not among the provided options.
(Even the answers are equally wrong, NLL increases 0.412 - 0.509)
Failed answer: Let's break down the question step by step: 1. The question asks about an event state that Everly wants to do after specific events: Noah buying gift and Noah picking Everly up at the airport. 2. The question stem "What event or state..." suggests that we're looking for common action experience that Everly desires in both scenarios. 3. From the context, we know that Noah is planning a surprise visit and has bought a gift for Everly. This suggests that Everly is likely expecting pleasant experience or a reunion with Noah. 4. After Noah picks Everly up at the airport, it's natural to infer that Everly wants to reunite with or see Noah, which is a pleasing experience. likely that Everly wants to see or have the gift, which is also pleasing experience. Given this analysis, the correct answer is: Everly visits/see Noah. both scenarios, Everly wants to experience the pleasure of being with or sceing Noah, which is the most likely common action or state that Everly desires.
Similarly, after Noah buys a gift for Everly, it's
(LLM gives answer that doesn't 't appear in options)
Table 22: Examples showing the effect of wrong-over alignment on COM2 dataset (part 2).
34

---

4. Consider the edges from node 3: (3, 4) with weight 3, (3, 5) with weight 2, (3, 2) with weight Published as a conference paper at ICLR 2025
Question:
In undirected graph, the nodes are numbered from t and the edges are: edge between node 0 and node 2 with weight 2, an edge between node 0 and node 5 with weight 4, an edge between node 0 and node with weight an edge between node 0 and node I with weight 3, an edge between node and node 2 with weight 4, an edge between node and node 5 with weight an edge between node and node 4 with weight 4, an edge between node 2 and node with weight 3, edge between node 2 and node 3 with weight an edge between node 3 and node 5 with weight 2, an edge between node 3 and node 4 with weight 3, an edge between node 4 and node with weight 2. Q: Give the shortest path from node 0 to node 4. Please also give the total weight of the shortest path.
Correct answer: 3 4]
Original answer:
Here's s the step-by-step analysis: 1. Create graph with nodes from O to 5 and the given edges with their corresponding weights. Choose node as the starting node and node as the target node. 3. Run Dijkstra's s algorithm to find the shortest path from node 0 to node 4. Here are the steps of Dijkstra's i algorithm:
Initialize distance array with all distances to infinity, except for node 0 which is set 0.
The distance array represents the shortest distance from node 0 to each node.
Create priority queue and add node O with a distance of 0. The priority queue used to select the node with the shortest distance.
While the priority queue is not empty, repeat the following steps: 1. Extract the node with the shortest distance from the priority queue. 2. For each neighbor of the extracted node, calculate the tentative distance between them. 3. If the tentative distance is less than the current distance to the neighbor, update the distance array and add the neighbor the priority queue.
the distance to the extracted node plus the edge weight
The shortest path from node 0 node is the path with the minimum distance from node 0 to node 4. The shortest path from node to node 4 is: + - + The total weight of the shortest path is: 2 I + 3 6. Therefore, the final answer is: Shortest path: 0 2 - 3 + 4,
Total weight: 6.
Less wrong answer: Here's the step-by-step analysis:
1. Start node 0.
2. Consider the edges from node O: (0, 2) with weight 2, (0, 3) with weight 1, (0, 1) with weight
3, (0. 5) with weight 4. We'll choose the edge with the smallest weight, which is (0, 3) with weight 1.
3. Move to node 3.
1. We'll choose the edge with the smallest weight, which is (3, 2) with weight 1.
5. Move to node 2.
6. Consider the edges from node 2: (2, 4) with weight 3. We'll choose this edge, since it's the only option.
7. Move to node
The shortest path is: 0 - 2 4. The total weight of the shortest path is: + 1 + 3 5. Therefore, the answer is: The shortest path from node 0 to node 4 is 0 - 3 - 2 + 4, with total weight of 5. (The path found is shorter: 5)
Table 23: Examples showing the effect of wrong-over-wrong alignment on NI.Graph dataset (part: 1).
35

---

Published as a conference paper at ICLR 2025
More correct answer: Let's analyze the question step by step:
1. We given undirected graph with nodes numbered from 0 to 5, and edges with weights between these nodes.
2. We want to find the shortest path from node node 4.
3. We can use simple algorithm find the shortest path:
Create a list of nodes to visit (initially, only node 0 is in the list).
For each node in the list, check if it's neighbor of node (i.e., if an edge exists).
it's neighbor, check the weight of the edge and add the neighbor to the list.
Mark the neighbor as visited.
Repeat until node 4 is reached.
The path from node 0 to node is the sequence of nodes in the list, and the total weight is the sum of the edge weights along this path.
+ Node 3 (edge weight 1) + Node 4 (edge weight 3). Node 4. The total weight is: (from Node 0 to Node 3) + (from Node to Node 4) 4. Final answer: The shortest path from node node 4 is Node + Node - Node 4, with total weight of 4.
Applying this algorithm:Node
The shortest path is: Node 0
Node 3
(Find the shortest path)
Better calibrated answer:
Here's the step-by-step analysis:
1. Create graph from the given information. The graph looks like this:
2. Initialize distance array with all distances set to infinity, except for the source node (node
0) which is set
dist[O) = 0
dist[1
dist[2 [2] 0C
dist[3]
dist[4 =
dist(5)
3. Relax all edges in the graph. This involves updating the distance array by considering all edges:
dist[1] 3
(0 1 with weight 3)
dist[2] = 2
(0 + 2 with weight 2)
dist[3] 1
(0 + 3 with weight 1)
dist[4] = 00
dist(5] 4
(0 5 with weight 4)
dist(2] = 3
(1 + 2 with weight 4, but previously updated to 2, so no change)
dist[3] 3
(1 + 3 with weight 4, but previously updated to 1, change)
dist[3] = 2
(2 + 3 with weight
dist[4] 00
dist(5] = 5
(1 + 5 with weight 1, then 4, so no change)
dist[4] 5 (2 + 4 with weight
4. The minimum distance from node 0 to node 4 The shortest path is + - 4.
Table 24: Examples showing the effect of wrong-over-wrong alignment NI.Graph dataset (part
2).
36

---

Published as a conference paper at ICLR 2025
The total weight of the shortest path is:0 + 2 (weight 2) + (weight 1) +3
(weight 3). Total weight: 2 2 1 + 3 = 6.
(Even the answers are equally wrong, NLL increases 0.255 0.373)
Failed answer:
Let's break down the question step by step:
1. Identify the nodes connected to node 0: 1, 2, 3,
2. Weigh the edges connected to node O:
Edge to node weight
Edge to node 2: weight
Edge to node 3: weight
Edge to node 5: weight
3.
Identify the nodes connected to node 2, 5,
Edge to node 2: weight
Edge to node 4: weight
Edge to node 5: weight
4. The shortest path from node 0 to node 4 is through node 3:Node - Node 3
Node (weights 3)
5. The total weight of the shortest path is:0 + (weight 1) + 4 (weight 3)
The shortest path from node 0 to node 4 is Node 0 Node 3 Node 4, with total weight of
4.
(Fail to calculate total weight correctly)
Table 25: Examples showing the effect of wrong-over-wrong alignment NLGraph dataset (part
3).
37


Reviews:
[{"review_id": "Imxy4bVWT7", "reviewer": "Reviewer_aJUG", "summary": "This paper presents a creative approach to solving alignment problem in large language models (LLMs) especially in resource constraint settings by proposing a methodology called \"wrong-over-wrong alignment.\" The core idea is for LLMs to learn by differentiating varying degrees of wrongness in responses, enabling alignment without the need for correct, ground-truth answers. This approach addresses a common challenge in real-world applications where human-verified data is often scarce. By relying on wrongness comparisons, the authors demonstrate that this alignment can enhance model calibration and improve response accuracy even in resource-limited settings.", "questions": "My main question to author is how confident they are in proxies' ability to accurately represent wrongness across different tasks. Have authors been thinking about doing proxy weighting to further improve their reliability. \n\n On the insights, another question that I would like to get answers on is if authors observe any patterns in failure cases ? I am looking for cases where wrong over wrong alignment did not produce reliable preferences. What were those patterns ? \n\nI mentioned this in above section too. I am curious to learn how well does wrong over wrong alignment generalize to out of distribution tasks. Have the authors tested whether the model's calibration is preserved across tasks ? \n\nHow do the authors plan to use this method in more subjective domains ? We get into the ethical contexts where defining what is less wrong could be culturally vary. This, in some sense, could be tied back to domain specific proxy definitions ? What are authors' thoughts and what guidance they want to provide to the community wanting to use this method in practice.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces quite a novel approach to doing alignment in the absence of ground truth. The paper proposes an alignment methodology called wrong-over-wrong alignment. Such an alignment allows any LLM to learn the correctness of an answer (for the domain it is trained/aligned on), by simply distinguishing between varying shades of wrong. This is quite a valuable approach to solve alignment especially when human vetted groudth truth is missing, which is more often than not in any practical situation. This idea has the potential to push the capabilities of LLMs in resource constrained settings. \n\nAuthors have done a good job in presenting a very comprehensive, clear and rigorous experimentation methodology to test the hypothesis. I was quite satisfied to read about various methods that the authors proposed to elicit wrong over wrong preferences. I also came out quite impressed with the experimentation rigor presented in the paper. From various preference elicitation methods to various LLMs, authors have ensured sufficient dimensions are explored in depth. \n\nLastly, various insights such as task utility, effectiveness of wrong over wrong preferences, innate capabilities of LLMs, model confidences etc strengthened the paper further.", "weaknesses": "The biggest weakness in my opinion of the method presented in tihs paper is its reliance on proxy functions. While the main objective of the paper is to explore alignment without dependence on correct answers, the paper however relies on proxy functions to elicit wrongness of answers. The proxy methods come with their own limitations and I worry that in real world practice, they may inadvertently introduce biases or even inaccuracies. Authors themselves have called out a possible introduction of biases in domains where the proxies are not well represented. Having said this, its not clear what unbiased alternative proxy methods one could use or even do away with proxies. \nAs an ML practioner in an industrial setting, I would have liked to see experiments on real world use cases where implicit signals such as lack of clicks (normalized according to position) act as proxies for assigning wrongness score. The paper has an opportunity to test this idea in wild and use real world implicit signals to elicit wrongness. There is more wrongness signal than correctness signal in any real world application and thus this idea has huge potential.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hjyt8aHKgQ", "reviewer": "Reviewer_UTTj", "summary": "The paper explores aligning LLMs with \"wrong-over-wrong\" preferences, a novel approach to distinguishing varying degrees of incorrectness rather than simply selecting between correct and incorrect answers. This technique aims to enhance not only the probability assigned to \"less wrong\" responses but also improves model accuracy and calibration.", "questions": "* Why not average over evaluators? The paper doesn't focus on LLM-as-a-judge, so differentiating evaluators doesn’t seem essential in this context.\n* The claim that \"knowledge-based tasks are easier while commonsense is most challenging\" feels overstated given the relatively modest performance gap.\n\n**Minor Comments:**\n* I would appreciate a clearer motivation, especially in the abstract and introduction; perhaps clarify how this method could address practical limitations in LLMs.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Creative approach that addresses alignment without needing correct answers.\n* Solid, methodologically rigorous results across diverse datasets, validated with multiple LLMs.", "weaknesses": "* The results and analysis sections are presented in a \"bullet point\" format without logical transitions, making it feel like a collection of findings rather than a cohesive story.\n* Most tables, especially Table 1, are dense and challenging to interpret. Consider breaking them down, or move detailed tables to an appendix and keep summary statistics or visualizations. \n* In the experimental settings section, more context around the overall setup would help; for instance, the mention of multiple-choice questions jumps in without prior explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w5QPjHZ8Z4", "reviewer": "Reviewer_LygQ", "summary": "The paper explores utilizing LLMs to discern between incorrect answers i.e. wrong-over-wrong preferences and aligning LLMs using these preferences in situations where correct answers are unavailable or annotations are scarce.", "questions": "1. Context of Application: Can the specific types of tasks that are most suitable for the wrong-over-wrong alignment approach be elaborated? This will help clarify the method's applicability and relevance.\n\n2. The dataset creation process excludes correct answers and hence requires correct answers to be available. How does the paper reconcile this with the assertion that the method can be applied in contexts where ground truth is absent? A more detailed explanation is necessary.\n\n3. Evaluation of Wrongness: What criteria were used to quantify the degree of wrongness in responses? Given the subjectivity involved, a detailed framework or methodology would enhance the validity of your evaluations.\n\n4. The paper suggests that the model generates 4.5% more correct answers despite only aligning on wrong answers. What specific mechanisms can be hypothesized to contribute to this improvement? Further analysis could provide critical insights as it is noteworthy how without using correct answer information the model was able to perform better. \n\n5. The paper evaluated the method on ScieBench which can be considered a task close to theorem proving. But, it does not perform up to the mark on the task. Can the paper discuss the implications of your findings for other datasets or tasks and as to why the method lacked generalizability to the task?\n\n6. Evaluation Metrics: Can the paper focus why accuracy was chosen as the primary evaluation metric? Given the small number of options in certain tasks, would it be more appropriate to use additional metrics such as F1, precision, and recall to better capture model performance?\n\n7. The paper mentions potential benefits for low-resource languages. Can specific examples or preliminary results demonstrate how the method could be applied effectively in this context?\n\n8. While LLM as a judge holds promise, to use LLMs to judge themselves for something as subjective as wrongness with lack of ground truth should be justified with empirical results demonstrating the robustness of the evaluation. Can further experiments be performed to address this considering the paper compares different models as judges as part of their evaluations. Additionally, to use LLM as a judge, previous examples have to be given to align the LLM for evaluations. Can the paper mention how the examples have to be chosen for a given task and calibration of the LLM judges and how were they selected?\n\n9. Pairwise evaluations - The evaluation results are mostly in the range of random answers even with filtering. Can the paper have a deeper analysis that justifies these results not being random?\n\n10. There is a huge variance in metric performance with 1 out of 22 overall scores being above 70 which is mentioned as 20 points better than random guessing while 12 of these reported overall scores are within 5 points of random guessing. Can the paper delve further into this as there is no consistent pattern that can help users choose the appropriate metrics, evaluators and method in general for their applications.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper explores a novel concept of \"wrong-over-wrong\" alignment, presenting an approach that diverges from traditional correct-answer evaluations, especially in contexts where correct answers may be absent.\n- It raises important questions about how incorrect answers can still provide value in training and evaluation processes, potentially paving the way for further research in this area.", "weaknesses": "1. The framing of the concept could be improved to clearly communicate the utility of wrong-over-wrong alignment. The lack of explicit applications or contexts weakens the paper's stance. For instance, theorem-proving and low-resource languages are mentioned but the paper does not carry out any experiments on the same. Authors mentioned in the results that knowledge-based tasks had better performance but these tasks inherently contain correct answers which diverge from the proposed applications. Moreover, dataset curation inherently requires the ground truth to be present, which contradicts the motivation of the paper, which is that it can be deployed on tasks with scarce or no ground truths. The method performs poorly on SciBench which can be considered as a task close to theorem proving. Overall, the framing of concepts and applications can be improved.\n2. The assessment of wrongness lacks rigorous criteria, leading to potential ambiguity in what constitutes \"less wrong.\" More robust evaluation metrics are needed to substantiate claims. The subjectivity of wrongness varies drastically from task to task. The complexity of evaluating incorrect responses and their subjective interpretations requires more detailed explanations and examples to improve reader comprehension.\n3. Currently the paper presents evaluations on different tasks and various parameters of its own method. But, it lacks sufficient comparison with existing models which are not aligned to wrong over wrong tasks.\n4. The paper uses Logit-based methods to evaluate the correctness of a sample. They justify it by mentioning high token probability “probably” implies correct. However, models may assign high logits to incorrect answers due to biases in training data, leading to misleading assessments of accuracy[1]. \n5. The heuristic that longer answers equate to higher quality can lead to misjudgments, as verbosity may obscure clarity and accuracy, making it difficult to evaluate the substance of the response.  In [3], the authors evaluate the factuality of long-form responses, showing that simply increasing the length of an answer does not guarantee that it will contain accurate or relevant information. [4] emphasizes that longer outputs can often be less reliable due to various factors, including how models memorize factual knowledge.\n6. Weak evaluation metrics: Given the small number of options in certain tasks, accuracy is a weak metric to fairly assess the performance, methods such as f1, precision, and recall provide more reliable results in such scenarios. ECE has limitations due to its reliance on binning, sensitivity to data distribution, and focus on average calibration. The choice of bins can introduce arbitrariness, affecting ECE values and leading to varying conclusions about a model’s calibration quality. Furthermore, ECE may be disproportionately impacted by bins with few data points, skewing results. As a summary statistic, it lacks detailed insights into calibration across specific confidence levels or types of inputs, making it less informative for models that need to distinguish varying degrees of incorrectness.\n[1] https://proceedings.mlr.press/v162/wei22d.html\n[3] https://arxiv.org/abs/2403.18802\n[4] https://arxiv.org/abs/2212.10511", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OX5P0lMLQ1", "reviewer": "Reviewer_AQ5T", "summary": "This article introduces an approach based on wrong-over-wrong preference for aligning Large Language Models to address the issue of high costs associated with obtaining ground truth in this field.", "questions": "see Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This idea is interesting and beneficial for the entire field.\n\nThe authors conducted experiments using several popular LLMs across multiple datasets. The results indicate that utilizing wrong-over-wrong preferences can lead to LLMs generating fewer incorrect answers.", "weaknesses": "In the experiments, the authors initially employed a proxy function to compute correctness scores based on ground truth, and then derived wrong-over-wrong preferences from these scores. I believe this process contradicts the motivation behind this work, as the resulting wrong-over-wrong preferences do not significantly differ from traditional right-over-wrong preferences obtained through similar methods.\n\nThe authors primarily relied on accuracy metrics. However, due to the lack of calibration mitigation in the baselines, these metrics are influenced by label distribution.\n\nThe baselines were limited to the original models, without comparison against some state-of-the-art (SOTA) alignment methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)