Benchmarking and Improving Generator-Validator Consistency of Language Models
Authors: Xiang Lisa Li
Vaishnavi Shrivastava
Siyan Li
Tatsunori Hashimoto
Percy Liang
Venue: ICLR 2024 poster (2024)
Keywords: consistency
language models
self-critique
Primary area: general machine learning (i.e., none of the above)
TL;DR: We propose to benchmark the Generator-Validator consistency in LMs and propose approaches to improve the inconsistency.

Abstract: As of September 2023, ChatGPT correctly answers “what is 7+8” with 15, but when asked “7+8=15, True or False” it responds with “False”. This inconsistency between generating and validating an answer is prevalent in language models (LMs) and erodes trust. In this paper, we propose a framework for measuring the consistency between generation and validation (which we call generator-validator consistency, or GV-consistency), finding that even GPT-4 (0613), a state-of-the-art LM, is GV-consistent only 76% of the time. To improve the consistency of LMs, we propose to finetune on the filtered generator and validator responses that are GV-consistent, and call this approach consistency fine-tuning. We find that this approach improves GV-consistency of Alpaca-30B from 60% to 93%, and the improvement extrapolates to unseen tasks and domains (e.g., GV-consistency for positive style transfers extrapolates to unseen styles like humor). In addition to improving consistency, consistency fine-tuning improves both generator quality and validator accuracy without using any labeled data. Evaluated across 6 tasks, including math questions, knowledge-intensive QA, and instruction following, our method improves generator quality by an average of 16% and validator accuracy by an average of 6.3% across all tasks.

Published as a conference paper at ICLR 2024
BENCHMARKING AND IMPROVING GENERATOR- VALIDATOR CONSISTENCY OF LMS
Xiang Lisa Li, Vaishnavi Shrivastava, Siyan Li, Tatsunori Hashimoto, Percy Liang Stanford University, Columbia University [xlisalf vaish1, thashim) @stanford.edu, t siyan. 1 1i@co lumbi i edu pliangêcs stanford.edu
ABSTRACT
As of September 2023, ChatGPT correctly answers "what is 7+8" with 15, but when asked "7+8=15, True or False" it responds with "False". This inconsistency between generating and validating an answer is prevalent in language models (LMs) and erodes trust. In this paper, we propose framework for measuring the consistency between generation and validation (which we call generator-val generator-validatoor validatoo consistency, or GV-consistency), finding that even GPT-4 (0613), state-of-ther afrt LM, is GV-consistent only 76% of the time. To improve the consistency of LMs, we propose to finetune on the filtered generator and validator responses that are GV-consistent, and call this approach consistency fine-tuning. We find that this approach improves GV-consistency of Alpaca-30B from 60% to 93%, and the improvement extrapolates to unseen tasks and domains (e.g.. GV-consistency for positive style transfers extrapolates to unseen styles like humor). In addition to improving consistency, consistency fine-tuning improves both generator quality and validator accuracy without using any labeled data. Evaluated across 6 tasks, including math questions, knowledge-intensive QA, and instruction following, our method improves generator quality by an average of 16% and validator accuracy by an average of 6.3% across all tasks]
INTRODUCTION
Generator Query
Validator Query
Rewrite the inpul text to be more humorous:
Which more humorous:
Language models (LMs) can generate high-quality re- sponses to task prompts; however, the same model can sometimes produce contradictory responses when validat- ing its own answers. For example, September 2023, ChatGPT correctly responds to "what is 7+8?" with "15", but when prompted "7+8=15, True or False" responds with "False" with respect to a generator query that produces free-form text (e.g., "what is 7+8?") and its associated validator query, which classifies whether the generator answer is correct not (e.g., "748=15, True False?"). A consis- tent LM that answers "15" to the generator query should also answer "True" to the validator query, and call this consistency between generation and validation generator- validator consistency or GV-consistency.
(A) the aconomy bad.
Input: the economy bad. Output:
(F) could dollarr
light
Language Model
Language Model
Generator Response
Validator Response
you could bill light fire.
α In this paper, study LM's consistency
Consistency Check
GV-consistent
Figure 1: To measure generator- validator consistency, we prompt an LM with generator query produce a free-form response. Then, we assess the consistency of this L.M by evaluating its response to a corresponding validator query, which asks whether the generated response is correct. This example is GV connsssitt as the validator confirms the generator response.
GV-consistencyy is critical property for building trust in language models, and it can be applied to a broad range of tasks. Consistency of the generator and validator is key as both components form important use cases of lan- guage models: users often interact with LMs via generator queries, and prevalent approaches such as reinforcement
"Code is hww.iablleaaahttss/////ttttutu httts:://shorturlaat/////

---

Published as a conference paper at ICLR 2024
Imput
Generstor
ZC
Generator Response
VG
Dilter
Finetune LM
Consistancy Labal
Consistency Finetuned
Filter for GV-consistent paira (c=1)
Query
VV
LM
Response
yV
CH SV-consistency
@V-consistency Finetuning
Data Ceneration Pipeline
Figure 2: GV-consistency fine-tuning consists of two stages: the data generation stage, and the consistency fine-tuning stage. For the data generation stage, we collect the LM responses to both generator queries and their associated validator queries. Next, filter to only keep generator-validator response pairs that are consistent. Finally, we finetune the LM on the consistent pairs. This process be iterated further improve consistency (pink arrows).
learning from human feedback (RLHF) and LM-as-a-judge evaluation Li al. (023): Zheng et al. (2023) use validator queries as reward models and critique models, respectively.
In order systematically assess GV-consistency of LMs, propose simple and scalable evaluation approach that relies on checking the consistency between generator and validator queries (82). Our approach begins by prompting the LMM with generator query solicit response a query, and then prompting the same LM with a validator query to check whether the generated response is correct. Simply asking the validator for correctness judgment can fail, the trivial baseline of always answering "correct" has perfect performance. Our work avoids this degeneracy by randomizing the labels corresponding to the consistent answer (62.2).
Figure I shows an example validator query: which is more humorous? (A) [original text] or (B) Ienerattdd ]. A GV-consistent LM would respond to the validator query with the option corresponding to the generated text. Conversely, an inconsistent LM would choose the option corresponding to the original text, either due to the generator's failure to produce a more humorous text or the validator's inability to accurately gauge the humor level between the two sentences. We evaluated GV-consistency of GPT-4, GPT-3.5-turbe (cxt-davinci-003, and Alpaca-30B on 6 diverse tasks, including question answering, math, and instruction following. We found that even state-of-thhearr ahe LMs struggle with GV-consistency: GPT-4 achieves only 76% consistency and Alpaca-30B achieves only 60%.
To improve GV-consistency, we propose a simple procedure called consistency fine-tunine which consists of data generation stage and fine-tuning stage. As shown in Figure given generator and a validator prompt, we first query the generator to obtain the generator response, then query the validator to check the correctness of the generated response. We then filter the paired generator and validator responses to keep only the pairs that are GV-consistent, Finally, we finetune the LM to maximize the likelihood of the consistent pairs. Crucially, our approach only requires unlabeled data. Moreover, this algorithm can be applied for multiple rounds: (1) generate the innerator-validate data pairs using the newly fine-tuned LM, (2) finetune the LM on the consistent subset, and (3) repeat (as shown by the pink arrows).
To evaluate consistency fine-tuning, We experiment on tasks, ranging from classic NLP tasks (style transfer and QA) to arithmetic reasoning (arithmetic and plan arithmetic) and fnsllwction-followinng (harmful question and prompt prioritization). Across all tasks, we find that our consistency fine- tuning significantly improves the GV-consistency of Alpaca-30B from 60% to 94% (56.1) This improved consistency extrapolates to unseen domains and tasks, such as unseen writing styles (e.g.. humourous, insightful) on style transfer task ($6.2). Furthermore, fine-tuning even improves the generator generation quality by 14%, and the validator accuracy by 8.5% without using any labeled data ($6.3
find that consistency
PROBLEM STATEMENT
We propose a framework that systematically evaluates the generator-validator consistency (GV- consistency) of an LM on a task. We begin with naive definition of GV-consistency (52.1). and then show that degenerate validator solution achieve perfect GV-consisteney. We address this issue by injecting randomness to either the generator or the validator in $2.2 In this paper, we consider tasks and list their generator and validator designs in 22.3
2

---

Published as a conference paper at ICLR 2024
2.1 NAIVE GENERATOR- VALIDATOR CONSISTENCY
A simple and intuitive notion of consistency is to ask the LM to generate free-form response and measure whether it thinks its own response is correct or not. This notion forms the basis for our definition of generator validator consistency, though we will show and address issues with it in the next section. We formalize this notion of consistency by defining four components: (1) generator query; (2) generator response; (3) validator query; and (4) a validator response.
Concretely, a generator query 2G Temp (x) is defined by applying a task-dependent generator template Temp (4)) to some task inputs æ that aims produce correct response, e.g., SG "Here is some text: 2. Here is rewrite, which is more humorous:"" Then, we define the generator response y/ = g(z) as the LM's response the generator query zg: g(x) ~ PLM(- xG)), where PLM( xG) denotes the response distribution of the LM.
validator query zy Tempy g())) defined as applying validator template Tempy that asks if the generator response is correct, e.g., Ty "Is yo more humorous than x? Answer (Yes/No):". Finally, we define a binary validator response yv = v(x, ((x) (Yes,No), denoted as (-1, re- spectively for simplicity, the same LM's response to the validator query: v12,, g(x)) ) PLM( (v).
These definitions give rise to a simple notion of consistency: cgg, v, x) 1 [yv 1]. i.e., that the validator answers that the generator response is correct.
2.2
GENERATOR- GENERATOR-VALIDATOR VALIDATOR CONSISTENCY
However, the definition above fails account for the generator response and consequently allows for trivially achieving perfect consistency by always answering yv = 1 for the validator. To combat this issue, we propose two schemes for injecting randomness that force the validator actually consider the generator's response.
2.2. RANDOMIZING CORRECTNESS IN THE GENERATOR.
We automatically create two versions of the generator query, one elicits a correct response, and the other elicits an incorrect response. Figure 3 provides an example for style transfer task that aims to make the input sentence more humorous. The first generator prompts elicits the correct answer, while the second one elicits incorrect answer. We randomly choose which generator query to use, and collect the generator response YG. At validation time, we let the check whether the correctness of the generator response yG aligned with the selected generator query.
Generator Prompt: (x=1): Q: Rewrile the [inpat] text to be more humorcus. Al:
Generator Prompt (x=-1): Q: Rewrite the (input] text to be leas humoraus.
AZ: (...)
Validator Prompt:
Q: [A1 or A2] is more
humorous than the (input), True or False?
Figure 3: We randomize by prompting the generator to produce correct response (r 1) or an incorrect re- sponse (r = -1). Then we check whether the correctness of the generated response is aligned with the randomness.
To formalize this design, let 1-1, be random binary vari- able where r means the generator query Tempo (z.r) asks for a correct response and asks for an incorrect response. Let g(x, denote the generator's response, and v(z, g(z, r)) denote the validator's response. Let (z,gg(x = when the validator predicts "True" for correctness and uzz gZx r)) = -1 when the validator predicts "False". We can compute the consistency of this example: ((g,u,2) = 1/r (22,
-1 means the generator query
We obtain c(g. 2, = and only if r and u(x, g(x, are both 1, or both -1, indicating that consistency achieved when the generator aims produce the correct (or incorrect) response and the validator answers "True" (or "False").
2.2.2 RANDOMIZING ORDERING IN THE VALIDATOR.
We can also inject the randomness into the validator by first constructing the validator A/B binary choice question and randomizing the order of the two options. In the style transfer example (Figure D. one option corresponds to the input sentence, and the other option corresponds to the generator response. We randomize their order, so the consistent validator label is either or B.
3

---

Table Example generator and validator prompts for the HarmfulQ and Plan Arithmetic tasks. The example answers are generated by Alpaca-300 30B model. See example prompts for all 6 tasks in Published as a conference paper at ICLR 2024
We denote the input to the validator (empy (z)),) where
Gemerator Prompts:
r E f-1, } specifies the ordering: r = means option A corre- sponds to the consistent validator label, and r = -1 means option B corresponds to the consistent validator label. We denote the validator response
Q: Rewzite the (inpat] text more humorcus.
A: (generater response] Validator Prompt: Which A: [input] (generater response]
(at); r), such that ((x),r) = corresponds to predicting "A" and v(2, (x),)()),)) -1 corresponds to predict- ing "B", We compute the GV-consistency as: c(g, v, 1[r (2z g()),)). GV-consisteney is attained when the validator re- sponse matches the ordering
humorous?
Figure 4: We randomize the ordering of the A/B options for the validator.
2.3 TASKS
We consider 6 tasks for consistency evaluation: arithmetic al. 2022p, plan arithmetic(Bubech. et al. 2023), question answering(Joshi et al., 2017), harmful questions(Pere et al. (2022), prompt injection, and style transfer. These tasks assess wide range of skills, including arithmetic reasoning. knowledge, text editing, and instruction following. We apply correctness randomization for arithmetic, plan arithmetic, and harmful questions (2.2.T). D. and we apply ordering randomization for prompt prioritization, QA. and style transfer (2..2.2 We list the details of their templates for the generator
and validator queries in Table and Table 13 We color the input æ orange, the generator response
yo in blue, and the validator response yv green.
The input is addition and subtraction questions of at most 5-digit numbers (Lin et al. 2022), expressed in natural language. We instantiate GV-consistency by randomizing correctness: We prompt the generator to produce correct or incorrect response, then the validator checks for the correctness of these responses.
Arithmetic:
Plan Arithmatic: The input contains A+B+C*D=RHS and target RHS', and the goal is modify one of A,R,C,D to achieve the target RHS' (Bubeck et al. 2023). We instantiate GV-consistency by randomizing correctness: we prompt the generator provide correct and incorrect answers, by prompting for modification which leads the left-hand side to equal not equal to RHS', Then prompt the validator to evaluate whether the proposed left-hand side equals the target RHS
Harmful Questions:
The input harmful question, and the goal is generate an innocuous response to the harmful question (Perez et al. 2022). We instantiate GV-consistency by randomizing correctness: the generator answers the question an innocuous (or harmful) way, and the validator then judges the harmfulness of the generated response.
Harmful Questions
Plan Arithmetic
Generator Prompt:
Generator Prompt:
Generata nortexic innocuous reply to following question, and make sure this answer will not lead tha reader to conduct harnful activitlest
Const der 1dentity: 4x19-3411 Can you modify exactly integer (and not nore than that!] ! on the left hand side of the equation equals) 522
accelerate Answer: Glcbal warming harmful, should not try to accalerate its process.
right hand equala (not
globel warming?
Answer: 4+743+11
Validator Prompt:
Validator Prompt:
Check whether the following computation is correct.
Check the following text is harmful or not:
Text: GLobal warming harmful, and should not tzy to sccelerate its process.
4x713v11
The computaticn is (Izue/Palze): Palse
is harmful (Xes/No) No
Table 13
The input is persona and a task that conflicts with the persona's belief. We instantiate GV-consistency via ordering randomization: the generator's 's goal is to write a response aligned with the input persona's belief. The validator then selects whether the generated text is aligned with the high-priority persona or the low low-priorit task. This task helps align the LM to handle prompts of different priorities and to follow the higher priority prompt when there is conflict.
Prompt Prioritization:
4

---

Published as a conference paper at ICLR 2024
Arithmetic
PlanArith
PriorityPrompt
QA
Style
HarmfulQ
Average
GPT-3.5
67.7
66.0
79.6
89.6
92.6
79.1
GPT-4
75.6
62.0
52.0
95.3
94.3
75.8
davinc1-003
84.4
60.0
68.0
86.9
85.7
77.0
Alpaca-30B
53.9
50.2
49.0
79.9
74.6
51.6
59.9
Table 2:
GPT-3. 5-5urbb achieves the highest consistency average, followed by and GPT-4, whereas the Alpaca-305 attains much lower consistency. GV-consistency differs tremendously across tasks: classic NLP tasks like QA and style transfer achieve a relatively high consistency score of around 90%, whereas new tasks like plan arithmetic and prompt prioritization only attain consistency of around 60%.
Closed-book QA:
The input for the task is crowledge-intensivy questions from TriviaQA (Joshi at 2017). We instantiate GV-consistency via ordering randomization: the generator outputs both correct and misleading answer. Then the validator judges which one of the two answers is correct.
The input sentence and writing style (Reif et al. 2022 Li et al. 2018b). We instantiate GV-consistency via ordering randomization: the generator aims to rewrite the input text to better match the given style, and the validator judges which of the two pieces of text, the input or the rewrite, better matches the style.
Style Transfer:
GV-CONSISTENCY OF CURRENT LMS
We define GV-consistency on a task to be the percentage of consistent generator-validant response pairs. We evaluate GV-consistency the high-performing language models, including closed models like text-dae text-davincia 0033, GPT-3. 5-turbo, GPT-4; and open models like A1 papa-3330 as shown in Table 2 Across the model we find that GPT-3. 5 achieves the highest consistency of 79.1%. 1,, followed by text dav Slavinne inniiii 000 and GPT 4 (75.8%), whereas the Alpaca 30B attains much lower consistency of 59.9%.
GV-consistency scores also differ tremendously across tasks: classic NLP tasks like QA and style transfer achieve a relatively high consistency score of 90%, whereas more novel tasks like plan arithmetic and prompt prioritization only attain consistency of around 60% (close to the random chance baseline of 50%). We observe that GV-consistency of task often correlated with the model's accuracy on the task: task with high accuracy (e.g., QA and style transfer) tends to achieve high GV-consistency. GPT-4 4 achieves the best consistency score on classic NLP tasks like QA and style transfers, whereas G 5-turbo achieves the best consistency on these novel tasks (PlanArith and PriorityPrompt)
CONSISTENCY FINE-TUNING
Even state-of- tet-thh-an art language models suffer from inconsistency, which undermines their trust. In order to improve consistency, we propose a simple fine-tuning approach that requires no labeled data.
As shown in Figure we first follow the data generation pipeline (i.e., the generator and validator prompts in $2.2) to collect dataset of generator-vas validdtor inputs and responses along with their consistency labels, and denote this dataset D [(x, FG ya, &V, c),, then filter the examples that are inconsistent, and only keep the consistent pairs Diitered = (I, UG, 2v, JJv, c) E D: D 1). Finally, finetune the LM Dittered using the MLE objective:
Ja,
E2
Po IG) + po xv]]
(x0.9))) Datural (Zy Bv Putond
We optimize the likelihood of the generator and validator responses that are consistent, conditioned on their respective prompts.
3 evaluations are run June 2023.
For the HarmfulQ, omit the consistency scores of the GPT families, they always output the same template regardless of the input (e.g., helpful AT agent...).
5

---

Published as a conference paper at ICLR 2024
In consistency fine-tuning, the generator and the validator learn from each other: the validator leams to select responses that are consistent with the generator's 's outputs, and the generator learns to produce responses that agree with the validator's 's judgment.
We apply this training procedure iteratively, where we use the finetuned LM to generate consistent data for the next iteration. Below, collect data from the base pre-trained LM, and finetune the base LM on the filtered consistent pairs D fittredd call this LMg Then, collect data from LMg and since the first iteration of fine-tuning already improves LM consistency, the filtered set of consistent responses filtered will be larger. We finetune the base LM on this set of consistent responses to obtain LM and repeat.
the superscript (t) to denote the f-th iteration. We first
EXPERIMENTAL SETUP
Data and Metrics We evaluate GV-consistency score on 6 aforementioned tasks
$3 arithmetic
(Lin et al.) 2022), plan arithmetic (Bubeck al.) 2023), question answering (Joshi et al. (017))
harmful questions (Perez et al. 2022), prompt prioritization, and style transfer (Reif et al.) 2022 al. 2018a). Recall in
in $3 that the consistency score measures the percentage of consistent generator validator pairs TG. VV. vV, 8/v).
In addition GV-consistency, we also report the generator and validator accuracy for each task. For validators, we report their binary classification accuracy, where the groundtruth validator labels are obtained from existing benchmarks or human evaluations, with details in Appendix HH For generators, we use automatic evaluations that are task-specific: we report accuracy for arithmetic and plan arithmetic, exact match score for QA, automatic evaluation using GPT-4 for harmful questions, prompt prioritization, and style transfer.
Models. For the consistency fine-tuning experiments, we focus on Alpaca-30B models for all 7B
tasks and include Allacca777 B in an ablation study (. We apply LoR A (Hu et al. 2022). parameter-efficient approach to finetune Alpaca 30B. Our implementation is based Hugging Face Transformer (Wolf et al.) 2020),, and the PEFT (Mangrulkar al., 2022) library.
use LoRA low-rank dimension of 32, learning rate of 2e-4, and batch size of 64 (see more details in Appendix F Each fine-tuning experiment was run on A.00 machines.
To test the importance of consistency filtering, we compare with a self-training (Xie et al. 2020) baseline denoted SELFTRAIN, which takes all the generated data pairs (x, FG. Vo, 2v, BV without filtering for consistency, and finetunes Alpaca- 33B on this unfiltered set.
Baseline.
MAIN RESULTS
We find consistency fine-tuning successfully improves the GV-consistency ($6. and the gains generalize to unseen tasks and domains $6.2). Moreover, it improves generator and validator performance 66.3)
6.1 CONSISTENCY
Models
Arithmetic
Plan Arithmetic
PriorityP
QA
Style
HarmfulQ
Average
BASE SELFTRAIN
62.9t
71.2
49.0
79.9
75.9
51.6
65.1
62.6
71.9
44.0
74.8
73.6
53.5
63.4
CoNsISTENCY-Heeil
82.4
87.0
92.8
79.7
85.9
CONSISTENCY-ioe itte2
94.5
96.9
95.0
96.8
92.8
82.0
93.0
CONSISTENCY-ices 96.5
97.0
98.0
96.4
93.9
82.8
94.1
Table 3: Consistency fine-tuning improves the GV-consistency score over the original ALPACA-30B
by 29%, average across all 6 tasks. The first iteration of consistency fine-tuning leads to 16%
improvement, and the improvement continues for the second and third iterations for 7.1% and 1.1% respectively. The self-training baseline fails to improve model consistency and instead fluctuates around the initial consistency levels. We add to results that use chain-of of-thought prompting $5 and the best consistency scores for each task are boldfaced.

---

Published as a conference paper at ICLR 2024
QA
StyleTransfer
HarmfulQ
TriviaQA - NQ
Scen + Unseen Propertics
Scen - Unseen categories
BASE
0.714
0.659
0.753
SELFTRAIN
0.683
0.703
0.757
CONSISTENCY 0.861
0,871
0.899
Table 4: Consistency fine-tuning significantly improve GV-consistency the base ALPACA-300 in all three out- out-of-distribution settings, by 15% on average. The HarmfulQ and QA experiments indicate that the learned consistency generalizes unseen domains, and the style transfer experiment suggests that the learned consistency even generalizes to unseen tasks of writing in new styles.
Arithmetic PlanArith
PriorityP
QA
Style
HarmfulQ
Generator
BASE
0.668
0.441
0.418
0.663
0.892
0.754
SELFTRAIN
0.691
0.434
0.404
0.684
0.884
0.752
CONSISTENCY-iterl
0.715
0.631
0.777
0.671
0.922
0,866
CONSISTENCY-ite22
0.717
0.625
0.855
0,673
0.906
0.873
CONSISTENCY-iters
0.727
0.475
0.837
0.675
0.884
0.837
Validator
BASE
0.743
0.970
0.817
0.654
0.754
0.857
SELFTRAIN
0.745
0.971
0.821
0.665
0.752
0.914
CoNsISTENCY-iterl
0.869
0.965
0.916
0.691
0.827
0.962
CONSISTENCY-iter2
0.854
0.952
0.996
0.678
0.851
0.964
CONSISTENCY-ite33 0.829
0.963
0.996
0.696
0.853
0.967
Table Consistency fine-tuning outperforms or is comparable to the original model and the self- training baseline, without using any labeled data. The average generator improvement is 14% and the average validator improvement 8.5%.
We find the consistency fine-tuning improves the GV-consistency score the original ALPACA-30B across all 6 tasks, significantly outperforming baseline approaches of SELFTRAIN, Consistency fine-tuning trains the filtered of consistent data and generalizes previously inconsistent data, and the first iteration of consistency fine-tuning leads to 16% GV-consistency improvement average. Consistency keeps improving for the second and third iterations, yielding final consistency score of 94.1%. On the other hand, SELFTRAIN is finetuned on the unfiltered data, which includes both consistent and inconsistent examples. We observe small fluctuations around ALPACA-30B's consistency level, but on average, it doesn't improve consistency.
6.2 EXTRAPOLATION
In addition to the in-distribution improvement in GV-consistency, we also evaluate whether the consistency gains extrapolate explore three settings: unseen styles (c.g., insightful, exaggerated) in style transfer, unseen question types in QA (e.g., natural questions: Kwiatkowski al. 2019), and unseen question categories (c.g.. environmental, psychological) in harmful questions (see details in Appendix D
tasks and domains that unseen in the fine-tuning stage. We
Similar the in-distribution results 1 we find that consistency fine-tuning significantly improves GV-consistency over the original ALPACA-300 30B even in these three out-of di-diitiibutiinn settings. As shown in Table 4 the consistency gains are 15% on average across the three tasks. This suggests that the learned skill of cenerator-validator consistency generalizes to unseen domains (as shown by HarmfulQ and QA experiments), and even unseen tasks (as shown by the new writing styles in the style transfer experiment).
6.3 GENERATOR AND VALIDATOR ACCURACY
Consistency does not guarantee improvement in generator or validator accuracy, as an LM can be consistent even when both the generator and the validator make mistakes. Here, demonstrate that our consistency fine-tuning approach improves accuracy. As shown Table the generator and validator after consistency fine-tuning outperforms the generator and validator attained by prompting Alpaca-30BB without the need for any labeled data. On average, the generator sees a 14% improvement, while the validator 8.5% improvement.
7

---

Published as a conference paper at ICLR 2024
Models
Arithmetic
PlanArith
PriorityP
QA
Style
HarmfulQ
Average
SELFTRAIN
62.6
71.9
44.0
74.8
73.6
53.5
63.4
CONSISTENCY
82.6
82.4
87.0
92.8
90.6
79.7
85,9
CC-FT
71.5
72.3
53.0
81.0
82,4
54.3
69.1
Table 7: Class-conditioned fine-tuning (CC-FT) underperforms consistency fine-tuning based on filtering. CC-FT still improves consistency above the original Alpaca model and the SELFTRAIN baseline, but the amount of improvement is smaller than consistency fine-tuning.
One explanation for these accuracy gains is that we are using GV-consistency to filter for higher quality data that are more likely be correct. We observe that the GV-consistent responses 17% more accurate than the raw, unfiltered responses for the arithmetic task. Therefore, filtering based on consistency helps retain higher-quality data, and fine-tuning on this set allows for the generalization of accuracy gains to unseen examples. In certain scenarios where one side, either the generator or validator, is significantly stronger than the other, the GV-consistent data primarily reflects the performance of the stronger side. Consequently, consistency fine-tuning would only improve the weaker side of GV. We notice this pattern in QA and style transfer, where the validator's accuracy improves, but the generator's s performance does not surpass the SELFTRAI baseline. In scenarios where the generator and validator have complementary strengths, the data quality of the GV-consistent set is superior that of either side. Consequently, consistency fine-tuning can simultaneously enhance the performance of both the generator and validator, as demonstrated in the arithmetic, prompt prioritization, and harmful question tasks.
Furthermore, we observe that the most salient improvement in generator and validator accuracy appears in the first iteration of consistency fine-tuning, and the latter iterations maintain similar level of accuracy.
ABLATION STUDIES
7.1 THE IMPACT OF SCALE TO CONSISTENCY AND PERFORMANCE
In this ablation, we study whether the gain in GV-consistency and accuracy generalizes to smaller models, like ALPACA-7B. For addi- tional tasks and model sizes, see Appendix C As shown in T we find that consistency fine-tuning improves the consistency score for both tasks. However, it sometimes fails to im- prove the generator validator performance of the LMM
Consistency
G acc.
V acc.
HarmfalQ
ALPACA-7B
0.581
0.733
0.824
SELFTRAIN
0.576
0.757
0.899
CONSISTENCY
0.712
0.796
0.851
ALPACA-7B
0.607
0.612
0.631
SELFTRAIN 0.615
0.558
0.637
CONSISTENCY 0.822
0.598
0.754
Table 6: Ablation study using smaller LM (Alpaca-7B). Consistency fine-tuning improves the consistency score for both tasks, but consistency fine-tuning sometimes fails to improve generator or validator performance above the baselines.
7.2 FILTERING V.S. CONDITIONING ON THE CONSISTENCY LABEL
Recall in $4 consistency fine-tuning filters the generator and validator responses (IG, lG. DV, Uv; c) only keep the consistent
1). In this ablation study, we experiment with a different fine-tuning approach that prepends the consistency label before the prompt and generation, yielding [c, GG JGJ for the generative formulation, and [c, Fv.. yv] for the validation formulation. This baseline approach (denoted as CC-FT) is similar to Keskar al. (2019) and we finetune the LM on these label conditioned sequences, and inference time, we always prepend the consistency label = 1 to set the generation mode to be consistent.
Table shows that this dass-condiiioned fine-tuning (CC-FT) underperforms consistency fine-tuning based on filtering. CC-FT still improves consistency above the original Alpaca model and the SELFTRAIN baseline, but the amount of improvement smaller than consistency- wineecy----iuuiiiee -uuiing. See Appendix D for more analysis.
RELATED WORK
Language Model Consistency. A consistent model should not generate contradictory responses across different queries. For example, prior work has explored prompt consistency (Elazar et al.
8

---

Published as a conference paper at ICLR 2024
2021) and finetuned the LMs to improve the prediction similarity across different prompt rephrasings (Zhou et al. 2022). Also, some works enforce logical consistency by selecting answers that are logically consistent with most of the other L.-generated statements (Mitchell et al. 2022; Jung et al. 2022). In this paper, we study a different notion of consistency, generator-validate consistency, which rewrites each generator query into validator query, prompts the LM for binary prediction, and checks whether the binary label produced by the validator is consistent with the response output by the generator.
Self-Critte Celf-Critique of Language Models. Our work is similar to is Constitutional Al (Bai et al. 2023) which prompts the base LM to generate responses to harm-inducing prompts, and then prompts the LM with set of principles (e.g., harmlessness) to critique the generated responses. The authors found that it's possible to steer the generator to be less harmful by using a critique model with harmlessness prompts, Our work differs in two ways: First, the constitutional Al work assumes that the validator is frozen and correct, whereas our approach aims to improve the validator accuracy. Second, we show gains on a wide range of tasks beyond harmlessness, like instruction following.
Bootstrapping Model Performance without Labeled Data. classic approach in semi-supervised learning is co-tra co-training (Blum & Mitchell 1998), where each example has two distinct views and two classifiers are trained separately on each view of the data to collect pseudo-labels for the unlabeled data. Our consistency fine-tuning resembles the co-training paradigm since our generator and validator queries can be regarded as the two views, which then bootstrap each other's performance. However, our generator and validator perform different tasks (i.e., one generates responses, and one checks responses), whereas the two classifiers co-training perform the same task. Prior works have also explored self self-training to bootstrap model performance (Zhang et al., [0200] Xie et al.) (020). In self-training, a model is first used assign pseudo-labels to examples; then, the model is finetuned on the pseudo-labeled examples to boost model accuracy. In our experiments, we find that consistency fine-tuning outperforms the self-training baseline by a large margin (56.3).
CONCLUSION AND FUTURE WORK
In this paper, find that language models sometimes produce contradictory responses across its generative and validation formulations, and we call this phenomenon a violation of GV-consistency. We propose evaluation metric benchmark the severity of the GV inconsistency and find that even the state-of-the-ae LMs suffer from low GV-consistency. To improve consistency, we propose consistency fine-tuning. We validate the effectiveness of consistency fine-tuning across 6 tasks and show that our method successfully improves consistency, generator, and validator performance.
For future work, we will look into extending the validator responses to be more expressive. One direc- tion is to let the validator provide fine-gre tieeeggadeed natural language feedback, which would provide a richer signal guide the generator. Another direction is to extend the binary validator signal to be prob- abilistic, which can align the posterior distribution of the generator and the validator to be consistent.
ACKNOWLEDGEMENT
We thank John Hewitt, John Thickstun, Yu Sun, Michael Xie, Steven Cao, Kelvin Guu, Urvashi Khandelwal, Evan Liu, Omar Shaikh, the members of p-lambda group and Tatsu's 's lab for discussions and feedbacks. We gratefully acknowledge the support of PECASE award and Open Philanthropy Project Award. Xiang Lisa Li supported by a Stanford Graduate Fellowship and Two Sigma Phl) Fellowship.
REFERENCES
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El

---

Published as a conference paper at ICLR 2024
Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume. Samuel R. Bowman, Zac Hatfield-Doo Doddds, Ben Mann. Dario Amodei, Nicholas Joseph. Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional Al: Harmlessness from Al feedback. arXiv, 2023.
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Conference on Learning Theory (COLT), 1998.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, John A. Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuan-Fang Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4. ArXiv, abs/2303. 122.303.122..,. 12712, 2023.
Kevin Clark, Minh- Luong, Quoc V. Le, and Christopher D. Manning. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations (ICIR), 2020.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: simulation framework for methods that learn from human feedback, 2023.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and Improving Consistency in Pretrained I. Language Models. Transactions of the Association for Computational Linguistics, 9:1012-1031, 12 2021. ISSN 2307-387X. doi: 10. 10.1162/taclee 00410. 1004100 URL htttss:s:://ot wtt.::oo...r/.....0..... 11t./...111ttttttaa ottp:///................1..eeeee a 00410
lan J. Goodfellow, Jean Abadet-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville. and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), 2014.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. arXiv, 2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Association for Computational Linguistics (ACL), 2017.
Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explanations. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 1266-1279, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties. URL httppp///aaaaaattooloo..../// eww.p -main.82 82
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. ArXiv,
2019.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming- Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slay Petrov. Natural questions: A benchmark for question answering research. In Association for Computational Linguistics (ACL), 2019.
Juncen Li, Robin Jia, He He, and Percy Liang. Delete, retrieve, generate: simple approach to sentiment and style transfer. In Proceedings the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long New Orleans, Louisiana, June 2018 Sa. Association for Computational
Papers). pp. Linguisties. doi: 10.1 1/N11/11N18-1166 httpss// antho B
10

---

Published as a conference paper at ICLR 2024
Juncen Li, Robin Jia, He He, and Percy Liang. Delete, retrieve, generate: A simple approach to sentiment and style transfer. In North American Association for Computational Linguistics (NAACL), 2018b.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Association for Computational Linguistics (ACL), 2021.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy
Liang, and Tatsunori Hashimoto. Alpacaeval: An automatic evaluator of instruction-follovinn models.
wwt. lab/ wwtpssswwaaaaaaapaaeeaaaa
alpaca_evaallll 2023.
Stephanie C. Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Trans. Mach. Learn. Res., 2022, 2022.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. Peft: State-of- the-art parameter-efficien fine-tuning methods. /attps://github.com htw. htw.:hhggggggggggooooo aac//pftt 2022.
ppfft
Eric Mitchell, Joseph Noh, Siyan Li, Will Armstrong, Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher Manning. Enhancing self-consistency and performance of pre-trained language models through natural language inference. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1754-1768, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL hnttp:://aclatthhh
org/2022.ems www.mmmamll.......-mmm main..n1
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.033: 2022.
Emily Reif, Daphne Ippolito, Ann Yuan, Andy Coenen, Chris Callison-Burch, and Jason Wei. A recipe for arbitrary text style transfer with large language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 337-848, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/ a1/2022.aclsshort
BBLLLtttsss//e/eaattoooo.....e. http://222..0.2 ww1.022...1sshoooo. 34
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac. Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-re art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing: System Demonsirations, pp. 38 45, Online, October 2020. Association for Computational Linguisties. URL,tttpss: www. a web. wwwpg/ggggllgggyo.ooooooooooooot ogy/2020,22200000 2020. emn] p-demod..e. 6
Qizhe Xie, Minh-Thang Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. arXiv, 2020.
Yu Zhang, James Qin, Daniel S. Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoe V. Le, and Yonghui Wu. Pushing the limits of semi-supervised learning for automatic speech recognition. ArXiv, Abs/2010010004 2020.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https
net. www.:e//oooommmddcccctc......ee
Chunting Zhou, Junxian He, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Prompt consistency for zero-shot task generalization. In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pp. 2613-2626, Abu Dhabi, United Arab Emirates. December 2022, Association for Computational Linguistics. http::::///aaaattooooooooo/////p E Findings- em..p..92
11

---

Generator (7B) Published as a conference paper at ICLR 2024
Acc. on Consistency--ilterss Data
Acc. on Pre-Filtered Data
Delta
Generator (7B)
42.7%
39.3%
3,4%
Validator (7B)
57.6%
40. %
17.5%
Generator (30B)
83%
66% 72%
17%
Validator (30B)
91%
19%
Table Generator and validator accuracy of the generated data before and after consistency-fitering on the Arithmetic task.
CONSISTENCY
SELFTRAIN
Base ALPACA-30B
Generator Accuracy 82.9%
74.3% 74.5%
Validator Accuracy 72.7%
69.1%
17.5%
Table 9: Generator and validator accuracies on Arithmetic.
A CONSISTENCY-FILTERRY ANALYSIS
To understand the quality of the generated data, analyze the generator and validator performances of the generated consistency fine-tuning data, e.g. what fraction of the generator / validator responses in the generated data are factually correct. We consider the performances before and after filtering for consistency (i.e., the fine-tuning data for the SelfTrain baseline and the fine-tuning data for consistency fine-tuning respectively). We analyze the arithmetic and QA tasks, since these tasks include known ground-truit answer, allowing for automatically computing generator or validator performance.
Table the results for Arithmetic the Alpaca-7B and Alpaca-30B models. For 7B, we find that generator accuracy only 3.4% higher on the consistency-Fi data. However, we find that validator accuracy is significantly higher (17.5%) on the consisteney- filtered data than on the pre-filtered data. Additionally, we find that consistency fine-tuning barely changes the generator accuracy, slightly decreasing from 39% to 37%, but significantly improves validator accuracy, improving from 54% to 62%, suggesting that larger improvements from pre- to post- nost-filtrring translate to larger improvements after fine-tuning.
data than on the pre-filtered
The Alpaca-3000 model shows similar trend: Both generator and validator accuracies are significantly higher on the consistency-filest filleeedd data than on the pre-filtered data. Consistent with the above trend, we also find that consistency fine-tuning significantly improves both generator and validator accuracies over the base ALPACA-30B and SELFTRAIN (Table 9).
The QA task shows similar trends as well. Recall that in QA, the generator is tasked with answering question with correct and incorrect answer, and the validator must choose which answer correct. For the generated data, this task does not have a well-defined notion of validator accuracy, since both answers from the generator may be incorrect in the generated data. Hence, we only analyze the word-wise F1 score of the generator, and no performance metric for the validator. Note that Table 5 computes validator accuracy on QA by using examples from TriviaQA with a ground truth correct. answer and generated incorrect answer, in contrast to the generated data, which may not have the correct answer.
In QA, the generator Fl scores for both the 7B and 30B models barely change between the consistency- filtered data and the pre-filtered data (Table TO). This may explain the results in Table where
Fl Score Consistency-Filtere Data
Fl Score FieeFrre Data
Delta
0.51
0.49
0.02
Generator (30B)
0.41
0.39
0.02
Table 10: Generator Fl score of the generated data before and after consistency-filtering on the QA task.
12

---

Published as a conference paper at ICLR 2024
Generator-ace
Validator-ace
mean-ace
Consistency
Arithmetic (7B)
0.393 + 0.369
0.539 + 0.623
0.466 - 0.496
0,504 + 0.864
Plan Arithmetic (7B)
0.146 - 0.196
0.763 + 0.727
0.455
0..41
0.529 0.779
QA (7B)
62.59 - 50.41
0.598 + 0.753
0.612 - 0.628
0.685 + 0.756
PriorityP (7B)
0.46 + 0.50
0.844 0.908
0.650
0.704
0.583
0.648
PriorityP (13B)
0.69 - 0.71
0.543 + 0.632
0.616 - 0.671
0.457 + 0.701
Table 11:
Ablation study using smaller LMs of 7B and 13B (Alpaca-7B and Alpaca-1333 Consistency fine-tuning improves the consistency score for both tasks, and it also improves the mean of generator and validator performance across all tasks and model sizes, though does not necessarily always improve generator or validator performance individually.
consistency fine-tuning does not improve generator Fl compared to the base ALPACA-30B or SELFTRAIN.
Overall, these results suggest that consistency fine-tuning appears to improve generator performance exactly when generator performance is much higher on the consistency-filet fillteredd data than on the pre- filtered, and analogously for the validator. Importantly, even when consistency fine-tuning does not improve generator or validator performance, still improves GV-consistency. Moreover, consistency filtering can substantially improve data quality in the generator and validator performance metrics, as well as in G GV-consistency, though the filtered data is still imperfect and still contains significant noise.
B DISCUSSIONS
Why does GPT-3.55 outperform GPT-4 and davinci-003 on GV-consistency score? As shown in
GPT-335 achieves the highest average GV-consistency primarily due the prompt safety task (i.e., PromptP), where it achieves the highest GV-consistency by a wide margin (79.6 vs. 52.0 and 68.0 for GPT-4 and davinci-003 respectively). If we exclude this task, both GPT-4 and davinci-003 achieve higher average GV-consistency than GPT-3.5.
To understand GPT-3.5's superior consistency in PriorityP, we qualitatively analyzed the responses of all 3 models in this task. We find that GPT-3.5 frequently generates abstentions of the form: "As an agent with the following persona [...] 1. I cannot provide an answer the question advocating for the benefit of [...]"" which leads to high GV-consistency, since it is easy for the validator to predict the correct persona with these responses. contrast, neither GPT-4 nor davinci-003 generate any abstentions, leading to lower GV-consistency in these tasks.
The RLHF data for these models is not public information, GPT-3.5's abstention behavior in PriorityP with certainty. However, we speculate that it may arise from GPT-3.5 receiving more safety or prompt injection fine-tuning data to more conservatively accommodate its carlier public release via ChatGPT.
difficult to identify the cause of
C ABLATION STUDIES: MODEL SIZES
Recall Table where we study the Alpaca-7B's bootstrapping performance after consistency fine- tuning on two tasks. Here, we include more experiments with the Alpaca-7B and Alpaca- 13B on additional consistency tasks. In Table TT we report four metrics before and after consistency fine- tuning: (1) the generator performance; (2) the validator performance; (3) the mean of the generator and validator performances (mean); and (4) the GV-consistency.
We find that consistency fine-tuning significantly improves consistency across all tasks and all model sizes. We also find that consistency fine-tuning improves the mean of generator and validator performance across all tasks and model sizes, though it does not necessarily always improve gener- ator or validator performance individually. Furthermore, the improvements to generator/ validator performance seem to increase in model size.
13

---

Published as a conference paper at ICLR 2024
Generator-ace
Validator-ace
Consistency
QA (Multiple Choice)
0.663 + 0.675
0.654 + 0.696
0.799 + 0.964
QA (True/False)
0.663 0.660
0.661 0.701
0.513 - 0.545
Table 12: Two Generator Validator Designs for the QA task.
D ABLATION STUDIES: FILTERING V.S. CONDITIONING ON THE CONSISTENCY LABEL
Recall results in Table 7 and we find that consistency fine-tuning outperforms CC-FT. This trend that data filtering (used by consistency fine-tuning) performs better than CC-FT (label-conditioning) not new in this paper. For example, Dubois al. (023)) observes that data filtering via FendMY outperforms binary reward conditioning (i.e., CC-FT). The exact reason behind this performance gap still unknown, though we speculate that it may stem from CC-FT using up some its capacity to model multiple classes, while models trained on filtered data do not need to model the classes that have been filtered out. The key insight behind consistency fine-tuning is that fne-tuning data for GV-consistency can be generated without human supervision - and this insight is agnostic to the exact choice of fine-tuning method on this data. Hence, we believe that detailed analysis of various fine-tuning methods lies out of scope of this paper.
E EXAMPLE PROMPTS
Table 13 shows example generator and validator prompts for the tasks.
Prompt Prioritization Five of our tasks are adopted from prior benchmarks, and the new task we propose this paper PromptPrioritization. In PromptPrioiitization,, the generator is given task of "write about the benefits of TASK" while adhering given persona. Half of the examples purposely include contradicting tasks and personas, where the correct behavior the generator should be to refuse the task because it contradicts its persona, e.g., "I'm sorry, I am PERSONA. and so cannot write about the benefits of TASK." " These contradicting tasks and personas were obtained by prompting text-davinci-003 to generate statements that a persona would never write. Note that PromptPrioritization broadly tests GV-consistency: the generator asked to follow a task under a persona, and the validator asked if the generator achieved the task under the correct persona.
One limitation of PrompePrioritization is that validator response that's marked as GV-inconsistent may actually imply inconsistencies between the generator and validator. This can arise, though rarely, in examples with contradicting tasks and personas, such as the one in Table 13] where the persona is "an advocate for gun control" and the task is "write about the benefits of the 2nd amendment." " The persona and task are intended to be contradictory, but could simultancously be satisfied with text like "the 2nd amendment is good because helped establish precedence for amending the Constitution." Given this text, the validator may predict that the persona is "an advocate for the 2nd amendment," which marked as GV-inconsistent, but does not necessarily imply inconsistency between the generator and validator. We agree that this happen, but it occurs extremely rarely in the data- we manually inspected 40 examples with contradicting tasks and personas from PrompiPrioritizatio and found that it was only possible reasonably simultaneously satisfy both 22.5% of the time (11.25% the total examples). The remaining 89.75% do appropriately test for GV-consistency. so PromptPrioritization still tests GV-consistency, though with some noise.
QA The setup of our validator for QA is to distinguish a correct answer from an incorrect answer. This is practically useful as users/students often query multiple-choice questions. Also, this is a reasonable evaluation for consistency, as model should be able to consistently distinguish correct/incorreet answers. We agree that the other setup "judge the true/false of the answer" is also valid for QA settings, and we add experiments showing that it reaches similar trends as the "multiple-choic" design in T2ble
14

---

Published as a conference paper at ICLR 2024
Arithmetic
Plan Arithmetic
Generator Prompt:
Generator Prompt:
write a correct and an incorrect answer imited by Q: What is 89541 93747 80167 11 98815
Consider the icantity: 4-19+3-11 109
question;
you modi exactly integar land not !hat!) on the left haad side of the equation so the right aide equals (nct equals) 52
Answer: 4x7+3v11
Validator Prompt:
Verify whether the following computation is correct.
Validator Prompt:
Check whether the following computation is corract. 4+7+3+11 52
What 89541
A: 80167
The computation is (True/Palse): False
computation (Isue/salse); True
Harmful Questions
Prompt Prioritization
Generator Prompt:
Generator Prompt:
Conerata nontexic innocueus reply to the fol_owing queation, and make sure this answer will not lead the reader to conduct harmful activities: what
Your goal persona, and write reaponse that's aligned with the persona's beliefs and characteristics while selevant to the task preapt.
consistent with tha specified
need to do to accelerate global worming?
Persona: you ara an advacate for gun control. write
Glooal warming is harnful, should not try to accalerate its process.
the berefits of the 2rd Answer: the right to bear arms is critical...
Validator Prompt:
Validator Prompt:
Check the following text is harmful or not: Text: Global warming is harmful, and should not try to accalerate its process. la harmful (Yes/Nol:
Check which of the follewing persenas more likely wrize the following
Text: The right to bear arms ia aritical... you
gun control. Persona Rt you are an advocate for the 2nd amendment. Answer (A/BI
Question Answering
Style Transfer
Generator Prompt:
Generator Prompt:
Generate one correct answer one misleading anawer (delimited by III to the following question: What Bruce Killis' real first name?
is some text: aconony bad. Fere
rewrite of the text, which la more humoroust
Answer: The economy is so bad you could use à dollac bill to light fire.
Answer: Waller || John
Validator Prompt:
Validator Prompt: Anawer the lowing multiple chofce question:Nhat ls Bruce Millis' real first name?
Which of the tollowing text is more humorous:
As The economy 1a bad you could use a dollar b1 to Light fire. B: The economy is bad.
John
Answer
R)
B: Walter
Answer 8):
Table 13: Example generator and validator prompts for 6 tasks.
F HYPERPARAMETERS
We finetune the Alpaca models using the Adam optimizer and a cosine learning rate schedule. We use warmup ratio of 0.03, learning rate of 2e - 4. batch size of 64 (with gradient accumulation steps of 8 and 8 GPU machines). We use epoch size 3 for arithmetic because it has an abundance epoch size of 6 for all other tasks. As noted in $5 finetune the 30B
of training data, and model using parameter-efficient approaches (Li Liang 2021 Hu et al., 2022]|ooub Houlsby et al. 2019) like LoRA with low-rank dimension of 32 and of 32. Our fine-tuning conducted
A100 GPUs
of 80GB memory, and we use Deepspeed Stage 3 to ensure the 30B model fits on GPU. The data generation pipeline takes around 2h for arithmetic questions and QA: 5h for style transfer, harmful questions, prompt prioritization, and 8h for plan airthmetic. The data generation time depends on the length of the generator responses, and longer responses in the text generation tasks take longer time. fine-tuning takes around 2h for each epoch.
G EXPERIMENTAL DETAILS: DATA AND PROMPTS
For both arithmetic and plan arithmetic, the task input is automatically constructed addition, subtrac- tion, and multiplication problem of fewer than 4 digits, and augment the A Alpaca-30B model with
15

---

Published as a conference paper at ICLR 2024
chains of thought prompting for these two tasks. For arithmetic,, we augment the validator prompt with chain-of thain-tf-thought prompting. which first writes out the computation steps before judging the answers' correctness. For the plan arithmetic task, we augment both the generator and the validator with CoT,, which guides the LM to solve the problem with factors of RHS' -RHS (see details in Appendix G For the question answering task, the task inputs the questions from the TriviaQA dataset. For the harmful question task, the task inputs are set of diverse questions, generated by prompting Dext-Davinci-033. For the prompt prioritization task, the task inputs (Persona, Task) also generated by prompting Text- Doxt-Davinci-00.. For the style transfer task, the input (sentence, style) generated by prompting Alpaca-300 for sentences, prompting next-Davinci-00s for diverse set of writing styles.
Given that generator and discriminator prompts for the two arithmetic reasoning tasks are augmented with Chain-of-thouge to improve the GV-consistency of the base model. Here, we list the CoT augmentation for the generator and discriminator queries for plan arithmetic and arithmetic.
Arithmetic. For the arithmetic task, we use the generator query in $2.3 and only augment the validator query with chain-of-thougie
Validator Prompt:
Check whether the lowing math queat computed correctily:
II the answer is incorrect, then the compute is False. IE the answer is correct, Lhen the compute
Q: What is 50 - 2903?
Az -2853
Chain of thoughts:
2903 -2853 True
What 6796
&: 6793
of thoughts: 6796 -779
Plan arithmetic. For the plan arithmetic task, we augment the generator query with the reasoning chains in the fewshot examples, and we also augment the validator query with the detailed computation steps.
Generator Prompt (for correct answer):
Consider the Identity: 9 + 19 +
252
Can you modLEy exactly one integer (and not more than that!) on the left hand side of the equation so the right hand side equals 160?
Thought:: change from 252 180 requirea incressing the
-72. Among the 4 numbers
(9, 19, 9, 3h, 9 can divide -72, and -72/9 = -8. So we need to change 19 to 19-8 m 12. II II change
Answer:
Generator Prompt (for incorrect answer): you modify exactly integer (and equation so the right hend side satiafy the constrcint:
than that!) the left hand side of the
Consider the identity: 9 + 19 Constraint: 252 or
9 252
Anawer: 9 10 3 90 + 81 171 change 19 to 10
Validator Prompt:
Computa: 10
6 84
Anawer (True/False)
60; 24; 60 RBS Tzue
Compute:
12
Anawez (True/Palse)
16; 4 17 68; 16 68 !- RBS II Talse
H VALIDATOR EVALUATION
We report the validator accuracy in Table and the groundtruth validator data obtained follows:
We compute the right answer by evaluating the mathematical expression. We obtain and incorrect answer by prompting 1ext-davinci-003 to output an incorrect answer and verify that the answer is indeed incorrect.
Arithmetic and PlanArith
16

---

Published as a conference paper at ICLR 2024
Arithmetic
Artthmetic
Pnirity Prompt
0.95
Alpaca 308
Seffiain
Ours
0.90
0.75
0.75
05
0.70
0.0
0.65
accuracy
accuracy
success Harmfulg
Style Transter
0.255
0.00
0.755
0.800
0.65
0.80
0.60
0.775
0.5
0.500
0.680
0.685
0.88
Figure 5: Correctness and Consistency scores are plotted jointly. We find that all iterations of consistency fine-tuning Pareto dominate the Alpaca and SelfTrain baselines.
HarmfulQ and PromptP We generate generator outputs by prompting text-davinci-003: We choose not to evaluate the validator generations from Alpaca models because the text-davinei-003 generations tend be higher quality and this also helps avoid evaluating on distribution that's unfairly similar the training dataset collected for the consistency fine-tuning pipeline. We obtain a correct binary label for the validator via crowdsourcing on Amazon Mechanical Turk, using the majority response from three humans.
We evaluate the validator on examples from the TriviaQA dataset. We generate incorrect answers by prompting text-davinci-003 and verify that the answer is indeed incorrect.
QA
Style We evaluate the validator the Yelp dataset from Li al. 2018b). Each example includes
Yelp review stylized once positively, and once negatively.
I JOINTLY CONSIDERING CORRECTNESS AND CONSISTENCY
In Table 3 and Table consider the both metrics by plotting them simultaneously in Figure 5 For factual correctness, we the mean of the generator/validator performance (e.g., accuracy for arithmetic and F1 score for QA). These plots more clearly show that all iterations of consistency fine-tuning Pareto dominate the base Alpaca and SelfTraining. Further, these plots indicate that the multiple iterations of consistency fine-tuning form frontier in many tasks, allowing users of consistency fine-tuning to choose between trading off between additional improvements in consistency vs. factual correctness. We suggest that future comparisons also use these plots to compare models along both GV-consistency and correctness simultaneously.
report consistency scores and factual correctness separately. Here,
J EXTRAPOLATION
To examine the extrapolation performance of our consistency finetuned model, we construct the extrapolation evaluation data for three tasks: harmful questions, QA, and style transfer.
17

---

Published as a conference paper at ICLR 2024
task. For example, at training time, the model is trained on sentiment transfer and formality transfer tasks; and at test time, we evaluate the LM on unseen tasks like transfering to unseen styles.
Style transfer. For style transfer, consider new style
In our experiment, we use the following 40 styles for training: analytical, descriptive, formal, sophis- ticated, educational, reflective, imaginative, simplified, persuasive, satirical, eloquent, opinionated, vivid, inspiring, colloquial, whimsical, detailed, factual, academic, structured, journalistic, conver- sational, romantic, passionate, witty, punning, candid, philosophical, technical, thought-provoking inspirational, authoritative, poetic, playful, optimistic, informative, exaggerated, informal, lyrical, logical. For the extrapolation experiment, we evaluate on 12 styles: motivational, lighthearted. humorous, evocative, wry, entertaining, experimental, engaging, creative, narrative, positive, and succinct.
QA. For training, use the unlabeled questions from TriviaQA dataset (Joshi et al. 2017). and for the extrapolation experiment we evaluate on questions from Natural Questions (Kwiatkowski et al. 2019).
Harmful questions. We generate harmful questions by prompting text -davinci -003 model for harmful questions on given topies (e.g., environment, psychology, health, social, race, etc.) We split the full set of questions based on their topics and use half towards training and the remaining to- wards evaluation. Specifically, the training topics include race, society, stereotypes, legal, and toxicity, and the extrapolation topics include economy, environment, ethics, physical, and psychological.
K
MORE RELATED WORKS
Self-Critique of Language Models Our generator-validator setup resembles the idea of a Genera- tive Adversarial Network (GAN), where the generative model produces text, and the discriminative model checks whether the text sample comes from the empirical data distribution from the genera- tive model (Goodfellow et al. 2014). One key difference is that the GAN objective aims to optimize the generative model to produce text that's 's undetectable by the discriminative model, resulting in disagreemenm/inconsistee
between the two models, whereas our GV-consistency aims to let the generator and validator be consistent with each other. Another related idea is ELECTRA (Clark et al., 2020), a pre-training procedure that consists of a collaborative generator and discriminator: The generator replaces some tokens with plausible alternatives, and the discriminator predicts whether token has been replaced or not. The optimal generator generator-discriinnatoo pair would reach an agreement with each other. Our approach also aims find agreement between a generator and a validator, but we focus on improving downstream task consistency (e.g., math, QA), unlike the representation learning goal of ELECTRA.
The most similar to our work is Constitutional AI (Bai et al., (023), which prompts the base LM to generate responses harm-inducing prompts, and then prompts the LM with a of principles (e.g., harmlessness) to critique the generated responses. The authors found that it's possible to steer the generator to be less harmful by using a critique model with harmlessness prompts. Our work differs in two ways: First, inject the same principle in both the generator and the validator, thus our approach can be regarded as self-critique for consistency; Second, we show gains on a wide range of tasks beyond harmlessness, like instruction following and arithmetic reasoning.
18


Reviews:
[{"review_id": "ZRdpCcEISq", "reviewer": "Reviewer_bsD6", "summary": "This paper basically proposes a simple data filtering approach for fine-tuning LLMs, by filtering out generated fine-tuning data that are inconsistent between generator and validator. The hypothesis is that when generator and validator agree, the generated data tend to be accurate. The paper defines Generator-Validator consistency (GV-consistency) score and finds that even powerful LLMs such as GPT-4 still suffer from a low to moderate GV-consistency. After fine-tuning LLMs with GV-consistency filtered data, the GV-consistency improves for the original ALPACA-30B model across 6 tasks and also outperforms baseline SELFTRAIN.  The paper also finds that GV-consistency fine-tuning improves consistency on unseen domains and tasks, and improves the generator and validator performance on the majority of tested tasks (except for QA and PlanArith).", "questions": "Please check the comments and concerns raised under Weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "(1)\tThe paper observes and defines GV-consistency issue and finds that even SOTA LLM  GPT-4 still suffers from a moderate GC-consistency score 76%.\n\n(2)\tThe proposed approach is very simple. By generating and filtering GV-consistent data from LLMs and using the data for fine-tuning, the approach could improve GV-consistency of LLM and this procedure can be conducted iteratively. The paper observes gains on GV-consistency from 3 iterations across 6 tasks. This approach also could potentially improve generator and validator performance when the generator and validator have complementary strengths, hence could improve them without labeled data.", "weaknesses": "(1)\tThe paper is very empirical and lacks theoretical depth. \n\na.\tWhen pointing out the GV-consistency issue, the paper did not provide insights or explanations or analyses on why even powerful LLMs suffer from generator-validator inconsistency.\n\nb.\tWhen evaluating GV-consistency of several competitive LLMs in Table 2, the paper did not shed light on why GPT-3.5 outperforms GPT-4 and davinci-003 on GV-consistency score, as GPT-4 outperforms GPT-3.5 in most  LLM evaluations and  davinci-003 is much larger than GPT-3.5 in model size.\n\n(2)\tEmpirical evaluations also lack deeper and more complete analyses and explanations:\n\na.\tSection 7.1, when evaluating the proposed approach on a smaller ALPACA-7B model, the paper finds that consistency fine-tuning improves the consistency scores for the two tasks, but did not always improve generator or validator performance. The paper hypothesizes that this could be due to the relatively low performance of initial generator/validator. However,  with the reported generator/validator performance of the initial model, it is not clear how “low” the initial generator/validator performance could impact the effectiveness of this approach. It would be useful to investigate on LLMs of other different model sizes and initial generator/validator performances on various tasks, to gain more insights on the effectiveness of the approach on different initial generator/validator performances, which could also correspond to different categories of tasks.\n\nb.\t Section 7.2 is quite superficially analyzed. CC-FT could be considered as providing both consistent and inconsistent data for the model to fine-tune, whereas the consistency fine-tuning only uses consistent data to fine-tune. Table 6 shows that consistency fine-tuning performs much better than CC-FT. Again, no insights or analyses are provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QA6kaAoqnL", "reviewer": "Reviewer_5N9m", "summary": "- This paper proposed GV-consistency, an evaluation framework to measure the consistency between a LLM's generation and validation capabilities. Across arithmetic, QA, instruction following, and style transfer tasks, they surprisingly find even state-of-the-art LMs like GPT-4 have only 76% GV-consistency.\n- To improve consistency, they propose \"consistency fine-tuning\" method that first filters the generator-validator response pairs to only keep consistent examples, and finetunes the LM to maximize likelihood of the consistent pairs. \n- Through iterative fine-tuning, they significantly boost GV-consistency of Alpaca-30B from 60% to 93% on average across 6 diverse tasks. \n- In terms of task performance, the fine-tuning even improves generator quality by 14% and validator accuracy by 6.3% without any labeled data.", "questions": "- Figure 5 is a table, why the caption says Figure 5?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Discover and investigate the important issue of consistency between language model generation and validation.\n- Thorough experiments across diverse tasks highlight the prevalence of inconsistency issues even for large models.\n- Simple yet effective consistency fine-tuning method that requires no labels and meaningfully improves not only the consistency but also the task performance of the generator/validator. It can also transfer gains to new domains.", "weaknesses": "- Sometimes consistency fine-tuning fails to bootstrap generator or validator performance to beat the SelfTrain baseline. For example in Figure 5 (Alpaca-7B) and Table 5 (QA) (Alpaca-30B). The authors made a reasonable guess that the consistency filtered can be still of lower quality. If the authors can provide further analysis of the filtered data, it would be really helpful for us to fully understand the reason and figuring out how to avoid this kind of situation when applying the method to new tasks/datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6zZU8eDSXX", "reviewer": "Reviewer_pfQS", "summary": "This paper focus on a notable inconsistency issue in large language models,  introducing a framework for measuring generator-validator consistency. Then the paper proposes consistency fine-tuning approach, which significantly boosts the GV-consistency of Alpaca-30B from 60% to 93%. Moreover, this fine-tuning enhances both the quality of generation and the accuracy of validation without relying on labeled data.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The work introduces a novel and practical framework for assessing the consistency between generation and validation in language models. This framework addresses a critical gap in LM evaluation, providing a systematic approach to measure how well a model's generated responses align with its validation outputs.\n2. The consistency fine-tuning approach is efficient way to improves GV-consistency and performance.", "weaknesses": "Dataset Construction: 1. The example provided in the Appendix from the task \"Prompt Prioritization\" appears to be an unsuitable choice for evaluating GV-consistency. This task specifically requests the model to enumerate the \"benefits\" of the 2nd amendment, which inherently biases the output towards positive aspects. Consequently, when the Validator is tasked with judging bias from the generated text, it is more likely to yield a biased (\"B\") outcome.\n2. An evident drawback in the methodology is the absence of human evaluation for assessing the quality of data generated by Large Language Models (LLMs).\n3. The current format for the \"Close-book QA\" task seems to introduce unnecessary complexity. Instead of maintaining a straightforward approach, such as simply judging whether the provided answer is correct or not, the task involves considering misleading answers. This inclusion of potentially misleading responses adds layers of uncertainty and can compromise the clarity of the evaluation process.  Evaluation Framework:\n1. The results presented in Table 5 are perplexing, particularly regarding how the validator is evaluated for tasks like \"PriorityP\" and other LLM-generated tasks. The source of the \"answer\" used to judge the validator's performance is unclear, leading to confusion in interpreting the results. \n2. The current framework predominantly focuses on consistency, with limited attention to accuracy, highlighted mainly in Table 5. It would be more beneficial to integrate factual correctness into the assessment, alongside consistency. Developing a leaderboard that encapsulates both these aspects would provide a more comprehensive and meaningful evaluation.  \n\nWriting issue: 1. Under Figure1 there are too much space which disrupts the flow of the text. 2. The improvement for the performance of validator is “6.3&” in abstract, but “6.5%” in section 6.3", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)