Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs
Authors: Angelica Chen
Jason Phang
Alicia Parrish
Vishakh Padmakumar
Chen Zhao
Samuel R. Bowman
Kyunghyun Cho
Venue: Accepted by TMLR ()

Abstract: Large language models (LLMs) have achieved widespread success on a variety of in-context few-shot tasks, but this success is typically evaluated via correctness rather than consistency. We argue that self-consistency is an important criteria for valid multi-step reasoning in tasks where the solution is composed of the answers to multiple sub-steps. We propose two types of self-consistency that are particularly important for multi-step reasoning -- hypothetical consistency (a model's ability to predict what its output would be in a hypothetical other context) and compositional consistency (consistency of a model's final outputs when intermediate sub-steps are replaced with the model's outputs for those steps). We demonstrate that multiple variants of the GPT-3/-4 models exhibit poor consistency rates across both types of consistency on a variety of tasks.

Published in Transactions on Machine Learning Research (01/2024)
Two Failures of Self-Consisteng in the Multi-Step Reasoning of LLMs
Angelica Chen
ac59686myu.co
Center for Data Science, New York University
Jason Phang
2p489
Center for Data Science, New York University
Alicia Parrish
axp295 edu
Deportment of Linguistics, New York University; Google
Vishakh Padmakumar
op1871 @nyu. edu
Center for Data Science, New York University
Chen Zhao
ca1285 edu
NYU Shanghai; Center for Data Science, New York University
Samuel R. Bowman
sb60650nyu.cm edu
Center for Data Science, New York University; Anthropic
Kyunghyun Cho
kc 19000m edu
Center for Data Science, New York University
Reviewed on OpenReview: https: openeevee/eerreveeeee net/ formm? id= SnBgV1y96
Abstract
Large language models (LLMs) have achieved widespread succoss on a varicty of in-context few- shot lasks, but this success typically evaluated via correctness rather than consistency. We argue that self-consistence is important criteria for valid multi-step reasoning in tasks where the solution is composed of the answers to multiple sub-steps. We propose two types of self- consistency that are particularly important for multi-step reasoning hypothetical consistency (a model's ability to predict what its output would be in a hypothetical other context) and compositional consistency (consistency of model's final outputs when intermediate sub-steps are replaced with the model's outputs for those steps). We demonstrate that multiple variants of the GPT-3/-4 models exhibit poor consistency rates across both types of consistency on variety of tasks.
1 Introduction
An important property of logically valid machine learning systems is self-consistency iee, the requirement that no two statements given by the system are contradictory. Pre-trained large language models (LLMs), despite demonstrating impressive few-shot accuracy on a variety of multi-step reasoning tasks, often give inconsistent. responses lo questions (Mitchell al. 2022 Kassner et al. 2021 and factual knowledge-seeking
prompts Elazar al.
2021 Without self-consistency,, it is difficult to consider LLMs reliable trustworthy
systems. Elazar et al. 2022
defines self-consistency as the invariance of an LLM's responses across different
types of semantics-preservingg preserving prompt transformations. In this work, we seek to introduce and explore LLM self-consistency over two new types of transformations (shown in Figure 1 that we argue are important for valid multi-step reasoning.

---

Published in Transactions on Machine Learning Research (01/2024)
Hypothetical Transformations A hypothetical transforma- lion is an indirect phrasing of a prompt that queries the model for what its response would hypothetically be in some other con- text, such as "what would your response to <prompt> be?" or "what would the next words in your completion of <prompt> bo?" Consistency over hypothetical transformations implies that an LLM has some stored knowledge or computational path for determining what its response would be to some prompt without explicitly being prompted with exactly itself. This can be useful for prompts involving multi-step reasoning, where the LLM must have knowledge of its responses to the earlier steps in order to compute its responses to downstream steps. Like Figure [ given the prompt "What is the runtime of
Initial prompt
What the best movie of 2022?
"Everything Everywhere
All
What would your response to "what the best movie of 2022" be?
"Everything Everywhore
Hypothetically consistent
All At Once'
Hypothetically
*Turning Red'
What the runtime of the best movie of 2022?
the best movie of 2022" the LLM must either have stored or
computed its response to "what is the best movie of 2022?" in order to answer the full prompt.
hours, 19 minutes
What is the runtime of "Everything Everywhere Once"?
Compositionally consistent
Compositional Transformations For a prompt that in- volves multiple interdependent sleps of reasoning, a composi- tional transformation consists of replacing some intermediate step with the model's output to the previous step. In the pro- vious example, if the LLM outputs the response "Everylhing Everywhere All At Once" to the prompt "What is the best movie of 2022?," then the prompt "What is the runtime of Everything Everywhere All At Once?" is a compositional transformation of "What is the runtime of the best movie of 2022?" (See Figure []] Consistency over compositional transformations also important for logically valid multi-step reasoning when the LLM must. give a direct response- - without it, the LLM may give contradictory direct responses to different multi-step prompts that are in fact querying for the same thing.
2 hours, 19 minutes
Figure 1: An overview of the two types of sell-consiseency failures we identify in L.LMss
In this work, we investigate the degree to which LLMs self-consistent on these prompt transformations across a variety of tasks. We formalize our definitions of hypothetical and compositional consistency (Section 2h and show empirically that a wide range of pre-trained language models demonstrate low consistency rales on both hypothetical transformations (Section 3 and compositional transformations (Section
2 Formalizing Consistency
To make more precise our definitions of consistency and the semantics- semantics-preservigg preserving transformations that they entail, we first formalize our definitions prior to conducting our empirical evaluations.
2.1 Preliminaries
Let vocabulary V be finite set of tokens, V be the set of all possible finite-length sequences formed by concatenating zero or more tokens from V, and pe V + {0, 13 be an auto-regressive language model that defines a probability distribution over tokens v E V. po can be used to generate a sequence via greody decoding as follows:
U arg max log po v o|c: (Et))
(1)
given some context sequence e E V", until some time step T for which ÿr = [EOS], the end-of-sequence token. For ease of notation, we denote the greedy decoding output of model pe as
9po (c) -( (arg max po (D = v/c),....
(2)
arg max po = v/c; θ<r)
2

---

Published in Transactions on Machine Learning Research (01/2024)
We also define operator ~ that indicates when two strings are semantically equivalent. Although the precise definition of semantic equivalence will vary across different tasks, we use it to loosely refer to pairs of strings that can be used interchangeably (give or take syntactic adjustments) without changing the meaning of the overall utterance. Lastly, the ~ operator also reflexive, symmetric, and transitive.
2.2 Composing prompts
Reasoning with language often also involves composing prompts - for instance, we might ask "what is the answer to x 3 + 4?", which can be seen as the composition of a prompt template "what is the answer to -+ +47 with the prompt "2 3", where the symbol the former string is substituted with the latter string. This corresponds to a multi-step task where the model might first answer the prompt "2 x 3" (yielding Spo (22 3")), substitute 9pa (22 ("2
3") into the template (yielding the composed prompt "what is the answer x 3") + 4?," where the 9po 2 x 3") is replaced with the actual output string), and then answer the filled-in template.
to Ips ("2 (p.
To denote such prompt templates, we define P', the set of prompts E V that contain exactly one " symbol. Additionally, the function f(p', .p) P' + V" denotes substitution of p for the " symbol in p'
f also has some useful properties that will in our later definitions:
We can trivially represent any prompt p E V as the substitution of itself into the identity prompt template " " by writing p = (pp).)
~ g if and only if f(p/, .p) ~ f(v' a) for all p/ EP.
2.3 Definitions
We start out by restating the general definition of self-consistency,, as it has been commonly defined in past literature (Elazar et al. 2021 Jang et al. 2022).
Definition 2.1 (Self-consistency). Po is self-comsise if p g Ipo (p) 9po (4) for all p. V".
In other words, a self-consistent model gives semantically-couivaalle prompls. These semantically equivalent pairs of prompts (p,4 q in Definition 2.1 can lake many forms, including hypothetical and compositional transformations.
responses to semantically equivalent
Definition 2.2 (Hypothetical Transformation). Let Pi denote the set of hypothetical transformation prompt templates, which are prompt templates p' ''' such that f(p' .p) ~ f(. p) vp E y*. Then the set of hypothetical transformations of prompt can be denoted as Pi(p) == (f(p' PT).
Since f( ~ p, a model that is self-consistent must yield 9po (f(p' ~ 9po (p) for all p' E/.
Although we defined hypothetical transformations with respect to all prompts p E V", our definition of compositional transformations must be more restricted, since care only to apply compositional transformations to prompts that implicitly encode compositional task. That is, are concerned only with prompts that are already compasitions i.e., prompts of the form f('',)). Furthermore, given some target model p* that represents the ground truth or gold distribution, the response to a compositional prompt (as generated by p") is somantically equivalent to the prompt itself. That is, f(p', p) ~ 9o- (f(p', .p).. For example, if pp= p= "2" and p/ = "44 ", then f(p p) = #4 + 2" and 9p- (f('' .p)) = 9p- (*4 + 2n) 6 "4 +4+2 2" s(p' so f(''.)) is compositional.
Definition 2.3 (Compositional prompt). We define the set of compositional prompts as Pcomp (fW' p) f .p) 9p (f('', .p),, V',p' E P') given gold distribution p². Definition 2.4 (Compositional transformation). For prompt compositions p E Pcomp and f(p, p) E Pcomp both representing compositional tasks, the compositional transformation with respect to model po is f(p's 9oo (p)).
In practice, substituting p into p' may require minor syntactic adjustments for linguistic acceptability, but we omit these in our notation since the semantics remain the same.
3

---

Published in Transactions on Machine Learning Research (01/2024)
Given the above two types of prompt transformations, can define narrower types of LLM self-consistency.
Definition 2.5 (Hypothetical consistency). A model pe is hypothetionlly consistent if 9ro (p) Dro (s(p', p)) for any prompt p V and hypothetical transformation prompt template p' P/. Claim 2.6. 1f pe self-consistent, then pe is also hypothetically consistent.
Proof. Consider prompt p E V and hypothetical transformation prompt template p E Py. Since f(p', p) ~ f(.p) (by Definition 2.2) and f(,P) p, then f(//p) ~ p by the transitive property of w. Then by Definition 2.1 it follows that Gro (f (p'.p)) p)) ~ 9po
Definition 2.7 (Consistency over compositional transformations). A model po is compositionally consistent when, for all pairs of compositional prompts p E Peomp and [(p,, p) E PComp:
1. p gno (v) and 9ro (f((' ~ Sps (J Ips (p)))
and is compositionally inconsistent when:
1. p ~ Spo (p) and Dro (f(p' .P) ~ 9po gpo (f(p', Iyo (p)))
p ~ Sps (p) and Dro (f .P) + gr. (f('' Sps (p)))
One more case exists that we do not count into our measures of either compositional consistency inconsistency. p x 9p-(2) (p) and Ip, (f p)) + 9p.(f (f Ip. (p))), then we cannot say this is compositionally consistent because the model's output for p does not necessarily relate to or imply its output to f(p' .))- However, we also cannot necessarily say that it is compositionally inconsistent to give nonequivalent responses to fp', p) and f(p', Sps (po(p)) if the model's responses to p and gpe (p) are also nonequivalent.
3 Evaluating Consistency on Hypothetical Transformations
In our above definitions, hypothetical consistency is characterized as a binary either a model hypothetically consistent across all pairs (p.p') V* x Pj or it is hypothetically inconsistent. But in practice, models are only hypothetically consistent in some cases, and it is likely impossible to achieve hypothetical consistency across all (p.p') V* x Pt, Instcad, we explore the degree to which LLM outputs are invariant to hypothetical transformatiins of the prompl. This is measured as the hypothetical consistency rate, which is the proportion of pairs (p,p') E V x Pf for which a model pe exhibits the property Ipo (p) ~ 9po (f(fff/)))) To mcasure hypothetical consistency rate, we devise a set of four hypothetical transformation prompt templates that we use to transform randomly sampled prompts (sourced from Wikipedia and DailyDialog) into hypothetical prompts, as shown in Table [ We average the hypothetical consistency rate over these prompts to mitigate the model's sensitivity to prompt wording (Elazar et al. 2021)..
Evaluating hypothetical consistency requires checking whether Ipo (p) Ipo (f .p). However, checking this semantic equivalence is non-trivial - the same idea can be expressed in a number of different but synonymous ways. Rather than attempt to devise an automatic method for evaluating semantic equivalence, we instead use a multiple-choice set-up. One answer choice is the continuation of the initial prompt (denoted by "<prompt>") sourced from a text dataset, one choice the model's 'ss own greedily decoded completion for <prompt>, and the three remaining choices are the other modcls' completions for <prompt>. As discussed before, a model that is hypothetically consistent can, in a sense, predict its own completion. Thus, the model should be more likely to generate the answer choice that corresponds to its own completion than to the other answer choices. These templates are designed both to query the model on what its completion would hypothetically be for a given prompt and to evaluate whether the model can distinguish its own completions from those of other models.
As an example, suppose the original prompt sourced from Wikipedia is "This quilt begun in 1856 when she was seventeen includes the autographs on top of the blocks of many known celebritics and politicians of the day. Other". Suppose that the first three words of the completions generated by OpenAI GPT-3 models ada-001, babbage-001, curie-001, and davinci-003 "notable quilt authors," "famous quilts include,"
4

---

Published in Transactions on Machine Learning Research (01/2024)
"signatures include abolitionists," " and "notable figures whose," respectively. Additionally, the next three words of the Wikipedia article are "figures represented on." Then hypothetical transformation prompt that uses the first template in Table might look like:
I predict that the next 3 words after "This quilt begun in 1856 when she was seventeen includes the autographs on top of the blocks of many known celebrities and politicians of the day. Other" would be
A) famous quilts include
B) figures represented on
C) notable quilt authors
D) signatures include abolitionists
E) notable figures whose
Answer:
If the model being evaluated is davinci-003, then the correct answer would be E. In this context, 9ásvinci-00s predict that... Other" would be") "E" "notable figures whose" Dásvinci-008 ( quilt begun... Other"), which satisfies Definition 2.5 As mentioned in Section 3.1 all hypothetical prompts are few-shot (where the provided labels are the answer choices that correspond to the evaluated model's completion), and only case-insensitive exact-match answers (i.e. "a/A/b/B/c///d/DD are accepted as correct.
Although this experimental set-up cases the burden of checking the semantic oquivalence of 9pu (p) 9ro (f((',(()) ,p),, is likely still lower bound on the true hypothetical consistency, since it does not account for other model outputs that may be semantically equivalent to 9pe (p). In the example above, had davinci-003 generated "notable people whose" instead of the letter "E," this response would have still been marked incorrect despite the semantic equivalence of "notable figures whose" and "notable people whose." Furthermore, the definition of semantic equivalence here depends implicitly on the prompt, p. Nevertheless, we still find it useful to analyze these lower bounds examinations of the failure modes of LLMs can aid in the future improvements these models. We leave future work the question of how far this lower bound is from the true hypothetical consistency rate.
Table Prompt templates used to evaluate whether LLMs are consistent across hypothetical prompt transformations. <prompt> is a prompt sourced from a dataset (e.g. Wikipedia, DailyDialog), <m> is the number of words of its own completion that the model is asked to predict, and <answer ceasver_choice>>>> are the multiple-choice answer choices that the model is given.
Hypothetical Transformation Prompt Templates
'I predict that the next <m> words after '<prompt>' would be <answer <answer_choicess choices>. Answer:'
'Given the prompt '<prompt>', my next <m> words would be <answer <answer_choices? Answer:'
"Given the context '<prompt>', my next <m> words would be <answer choices>. Answer:'
'I predict that after the context '<prompt>" the next <m> words would say are <answer snawer_choices! Answer:"
We conduct our hypothetical consistency experiments with original prompts sourced from two language modeling tasks:
Wikipedia Since language models are frequently pre-trained on Wikipedia archives, evaluating on a Wikipedia dataset can confound information memorized during pre-training with the skill being evaluated. To address this issue, we collect a sample of 400 English Wikipedia (Wikimedial Foundation) articles that were created on or after June 30, 2021, since the OpenAT documentation [Center) indicates that the latest pre-training data for the ada- http///bbbbbbeenee davinci-003 models contains documents dating up to June 2021. Fach initial prompt is randomly
-011 and
5

---

Published in Transactions on Machine Learning Research (01/2024)
selected segment of a Wikipedia article consisting of two full sentences followed by random- lenho prefix of the next sentence.
DailyDialog:
DiilyDialog (Li ct al. 2017 is manually labeled dataset of multi-turn conversations about daily life. We choose this dataset because it contains language that is more colloquial in style and less factual in content than the Wikipedia dataset. We randomly sample 400 examples from the training split and use the first conversational turn as the initial prompt.
We use only the prompts for which all five answer choices are distinct. We also vary the number of words in the original completion that the model is asked to distinguish from 1 to 6, since the difficulty may vary depending on the length of the original completion. We then compute the hypothetical consistency rate by calculating the proportion of the time that the model generated the letter of the answer choice corresponding to own completion.
3.1 Experimental Setup
In all experiments, we evaluate four model sizes of the OpenAl GPT-3 model (Brown et al. 2020 ada-001, babbage-001, curie-001, and davinci-003 (in order of increasing capacity) I All experiments are run using greedily decoded completions obtained from the OpenAI API from Aug. 2022 to Jun. 2023. We use 0-shot initial prompts but evaluate hypothetical consistency prompts using k-shot prompts, where & ranges from to 10. Since the in-context performance of LLMs is known to vary depending on the selection of in-context examples (Liu et al. 2021 Rubin et. al.] 2022), we randomly select a different set of in-context examples for each prompt. We also randomize the order of answer choices for each multiple-choice question to mitigate sensitivity answer choice order (Pozeshkpour & Hruschka 2023). Further evaluations for other combinations of models can be found in Appendix |A
Table 2: Average percent edit distances between completions from davinci-003 versus completions from the three other models and two datasets.
davinci-003 / ada-001
davinci-000 / babbage-000 0001
davinci-003 /
davinci-003
Datasel
curie-001
Dataset
Wikipedia
72.8%
69.7%
65.0%
70.3%
DailyDialog
75.8%
75.1%
74.2%
79.0%
3.2 All Model Sizes Perform Poorly At Distinguishing Their Own Completions
Figure 2 shows GPT-3's hypothetical consistency rates averaged over all few-shot prompts. Notably, all model sizes smaller than davinci-003 perform at about random chance on this task, regardless of how many words of the original completion the model is tasked with predicting. davinci-003 is the only model size that consistently performs above random chance, but even then its accuracy ranges only from 26% to 31% for Wikipedia and 30% to 37% for DailyDialog. We might also expect hypothetical consistency rate to increase as function of m (because longer sequences provide more information for the model to use to distinguish its own completion from other sequences), but we do not observe this trend for these models. In Appendix A however, we do observe this trend for gpt4. It appears that higher-capacity and generally more powerful models are overall more hypothetically consistent.
We also inspect the frequency with which each model selects each possible answer choice, as shown in Figure For a highly hypothetically consistent model that has not memorized the dataset, we would expect it to
We select this particular set of models since it is the most recent set of text completion (rather than chat) models available for each size of GPT-3. However, we also compare against text-davinc1-000 and gpt4 in Appendix A We separate davinci-001 and gpt4 into separate analyses because davinci-001, davinci-003, and gpt4 often output the include completions from of
original completions. In
multiplo-choice set-up, this results in duplicate answer choices. As such, only davinci-001, davinci-003, and gpt4 in any hypothetical consistcncy prompt. Since gpt4 chat model whereas the rest of the models completion models, we choose to analyze the completion models in the main text and leave comparison against. gpt4 for the A Appendix.

---

Published in Transactions on Machine Learning Research (01/2024)
Wikipedia
DailyDialog
40%
40%
35%
35%
Hypothetical consistency rate
Hypothetical consistency rale
30%
30%
25%
25%
20%
20%
15%
text-a
text text-ada-001
10%
text-babbage-001
10%
text-babbage-001
text-curie-001
text-curie-001
5%
text-davinci-003
5%
text-davinci-003
random chance
random chance
0%
0%
2 Number of words from original completion distinguish
5
Number of words from original completion distinguish
Figure 2: Hypothetical consistency rates multiple-choice self-knowledge prompts for the Wikipedia and DailyDialog datasets, across the four GPT-3 model sizes. Each line is the average taken across all k-shot prompts, for k E (1,,...,11 ,0)]. The shaded region represents the 95% confidence interval computed with nonparametric bootstrapping. The label "number of words from original completion to distinguish" corresponds to the quantity m in Table [
Wikipedia
DailyDialog
35%
35%
davinc
6002
15%
17%
20%
20%
30%
18%
19%
19%
18%
18%
Eve Tvelusbee mooes
Exte (00
21%
2.0
20%
20%
20%
Evalusted babbage
Sanban T00
18%
185%
18%
188%
18%
20%
20%
20%
20%
20%
1.9%
20%
1.5%
15%
ada
16%
17%
17%
16%
16%
100
18%
18%
18%
Wikipedia
ada -001
babbage -001
curie -001
davinci -003
DailyDialog
ada
babbage -001
curie -001
davinci -003
-001
Source
answer choice text
Source of answer choice text
Figure 3: A detailed breakdown of the numbers in Figure 2 the percentage of the time that each model selects each possible answer choice when prompted with hypothetical consistency prompt, averaged across all prompts (Le. across all m, the number of words that the model is asked to prodict; and k, the number of few-shot examples). The columns labeled "Wikipedia" and "DailyDialog" correspond to the answer choice containing the completion from the original dataset. Model outputs that could not be parsed into an answer choice are not included.
select the answer choice corresponding to its own completion the vast majority of the time, and to only select the other answer choices a negligible proportion of the time. However, for both tasks, only davinci-00s demonstrates a noticeable preference for its own completion over others. Furthermore, none the other models display a preference for any other answer choice, including the completions of the other models and the continuation from the original dataset. Despite our use multiple prompt formats, ada-001, babbage-001, and curie-001 all make random choices and cannot. predict their completions.

---

Published in Transactions on Machine Learning Research (01/2024)
davinci-003's moderate hypothetical consistency also cannot easily be attributed to dataset memorization, for couple of reasons. Firstly, we selected only hypothetical consistency prompts for which all five answer choices were distinct - ergo, davinci-003's completion could not have been identical to that of either Wikipedia or DailyDialog. Secondly, we also computed the average percent edit distance (the edit distance divided by the length of the longer string) belween the completions of davinci-003 and the completions of the smaller models and datascts, as shown in Table Across both datasets, the davinci-003 completions are on average more than 70% different from the dataset completions. Furthermore, the edit distances in Table 2 do not have high variance in each row, indicating that davinci-003's completions are not significantly more different from similar to one particular source than the others.
4 Evaluating Compositional Self-Consistency
Next, we evaluate compositional self-consistency on the tasks of arithmetic and semantic parsing, since these are tasks for which valid compositional reasoning are important. Both tasks can be framed as computational graphs that are directed and acyclic, with each node having at most one parent - i.e. trees. To evaluate compositional consistency, we store the model's answer to the expression represented by each individual sub- tree and generate compositional consistency prompts by creating copies of the original prompts where single sub-tree expression has been replaced by the model's output for that sub-tree. If model is compositionally consistent, then it should give the same answer to the original expression to the copy with the replaced sub-tree. Below, we further describe the experimental setup and give examples for each of the tasks.
4.1 Experimental Setup
We evaluate compositional self-consistency across six models (ada-001, babbage-001. curie-001, davinci-001, davinci-003, and gpt4) and two tasks. Experiments for the first five models were run between Aug. 2022 and Jun. 2023, and experiments for gpt4 were run between May and Jun. 2023.
Synthetic Arithmetic We generate set of 400 randomly-nested arithmetic expressions as the initial prompts. We then collect model completions for all possible sub-expressions of each expression using k-shot prompts, with k ranging from 3 to 10. We randomly select the in-context examples for each prompt. The arithmetic expressions have maximum nesting depth of 5 with nesting probability of 0.5, and each nested expression is enclosed in parentheses to avoid ambiguity. Operators and operands are randomly selected with uniform probability from the sets {+.-//, x} and [1, 2, ... .999], respectively.
For example, for the original arithmotic expression "(2 x 3) + (6/2)," we prompt each model with the following three sub-expression prompis, using parentheses to force the correct order of operations:
Pi "Q: 2 \n A:"
P2 "Q: 6/2 An A:"
P3 "Q: (2 x (6/2) An A:"
For each non-root sub-expression (Le. P1 and P2): we then create new compositional consistency prompt by replacing that sub-cxpression in the original expression (i.c. ps) with the model's completion. For the previous example, if the model answered pl and pa correctly, this would result in the following two compositional consistency prompts:
(1) PCc "Q: 6 (6/2) An A:" (2) PcG "Q: (2 3) 3 An A:"
For this example, we then compute a model's compositional consistency rate as the proportion of the time that the model's output for p. is correct, and its outputs for PCc and pa are the same.
(8)
GeoQuery GeoQuery Zeele & Mooney 1996 a semantic parsing datasct consisting of 880 natural language questions about US geography, paired with FunQL (Kate et al.) (2005) parses of those ques- tions. Similar to the synthetic arithmetic task, we first collect model parses for the spans corresponding
8

---

Published in Transactions on Machine Learning Research (01/2024)
to each sub-parse of sample of 400 GeoQuery training examples via k-shot prompts, for k ranging from 3 to 10. We randomly select the in-context examples for each prompt. For example, consider the GeoQuery example "Which state has the city with the most population?" with corresponding FunQL state (loc_ (largest_one one (pgpssstooooppplllaiionoonoi include the following:
Then two of the initial prompts we create
P1 = "Create a FunQL query for the following question: 'Which state has the city with the
most population?' A:
P2 = "Create a FunQL query for the following question: 'city with the most population"
A: m
where each prompt is sourced from a non-leaf sub-parse of the original gold FunQL expression (and the other sub-parses are omitted here for brevity).
Since evaluating whether Inu (s (//pp))) p)) ~ (f 9ps (p)) would involve interleaving natural language with FunQL in this case, we instead measure compositional consistency as the instances where the model's parse for P2 is correct (i.e. P2 9po 9po (P2)) and is sub-parse of its parse for p1 (regardless of whether the parse for pi is correct). This second condition is slightly more relaxed version of the condition that Spo (f p)) 9pu (f('', 9ps (p)), where we instead assess whether Ipo (p) is being used in 9po (f(p'p)) Due to this relaxation, the rate we measure here is an upper bound on the true compositional consistency rate.
Arithmetic
GeoQuery
70%
50%
60%
40%
Compositional consistency rate
Compositional consistency
50%
30%
40%
20%
10%
20%
10%
0%
4
7
9
10
4
5
7
10
Number in-context examples
Number in-context examples
ada-001
babbage-001
curie-001
davinci-001
davinci-003
gpt4
Figure 4: Compositional consistency rates versus the number of in-context examples on the arithmetic and GeoQuery tasks. The shaded region represents the 95% confidence interval computed with nonparametric bootstrapping.
4.2 Results
The compositional consistency rates for all six models shown in Figure 4 While davinci-003 and gpt4 exhibit the highest compositional consistency rates, both are compositionally consistent less than 50% of the time on the arithmetic lask and less than 65% of the lime on the semantic parsing lask. However, all models appear to improve in compositional consistency on the GeoQuery task as the number of in-context examples increases. Furthermore, gpt4 exhibits significantly more compositional consistency on the arithmetic task than even davinci-003. Taken together, these results suggest that both increasing the model capacity and the number of in-context examples offer some improvements in compositional consistency.

---

Published in Transactions on Machine Learning Research (01/202))
ada-001
babbage-001
Compositiona consistency rate
0.4
curie-001
davinci-001
davinci-003
gpt4
0.3
0.2
0.1
0.0
0.0
0.1
0.2
0.3
0.4
0.5
Correctness
Figure 5: The correctness versus compositional consistency rate of each type of GPT-3 or GPT-4 model on the arithmetic task.
For instances where the models are compositionally inconsistent, For arithmetic, 90% of compositional inconsistencies are caused by the final answer not matching the answer of the compositional transformation, despite the answer to the sub-expression being correct (i.e. p ~ Ipo (p) but 9p. (f(p', p)) & (fo(f Iro (p)), or case (2) in the definition of compositional inconsistency in Definition 2.7p. For GeoQuery, approximately 59% compositional inconsistencies result from the parse of the sub-trec not being included in the parse of the parent Lree. For example, suppose that the parent tree is represented by the query "how many people live in Texas?" and the replaced subtree corresponds to the query "Texas." A model might correctly output the parse stateid ("texas') for the latter but then output the parse "population (state (name ("texas")))* for the parent tree query, which is inconsistent with the parse of the subtree. In the other approximately 41% of cases, the compositional inconsistencies on GeoQuery are caused by an incorrect parse of the child node (i.e. of Ioo (p), or case (1) of compositional inconsistency in Definition 2.7
analyze the sources of the inconsistency.
For tasks where a precise definition of correctness does exist, such as arithmetic, it can be useful to understand the relationship between correctness and compositional consistency. Figure shows this relationship for all four model sizes. There exists notable linear relationship between correctness and compositional consistency, but all models except for davinci-O00 may output correct answers for the final expression but not for intermediate steps, or vice versa. Nonetheless, training LLMs to optimize solely for correctness appears to be helpful for improving compositional consistency in such tasks. For other tasks where a precise definition of correctness does not exist, this may not be as feasible a solution.
slightly more correct than consistent. This indicates that models
5 Related Work
Our work is inspired by an extensive body of literature that has defined and evaluated model consistency in variety of ways. Elazar et al. (2021) defines consistency as the ability for the LLM give consistent responses semantically equivalent contexts, such paraphrased contexts. Jang et al. (2022) supplements this definition with multiple other categories of logical consistency, such negational, symmetric, transitive, and additíve consistency. Similar to our results, they show that many modern LLMs do not exhibit strong consistency according to these definitions.
Yet other work has highlighted the inconsistency of LLM predictions across paraphrases the same input for varicty of downstream tasks, including knowledge extraction (Elazar al. 2021 Fierro Sogaard 2022 Newman al.) 2022), truthfulness (Raj et al. 2022), summarization (Kryscinski et al. 2020). and natural language understanding (Jang al. 2021 Zhou et al. 2022 Various remedies have been proposed for this
10

---

Published in Transactions on Machine Learning Research (01/2024)
issue- Elazar et al.
2021 proposes novel consistency loss that minimizes the 2-sided KL divergence between paraph rases. Jang et al. 2022 propases using multi-task training with paraphrase identification, and Newman et al. (0022) proposes training additional adapter layers to map paraphrased prompts to the same continuous representation. On the other hand, Dziri et al. 2023) suggest that the failure of LLMs to consistently reason correctly on compositional tasks an intrinsic characteristic of the Transformer architecture as the average parallelism of compositional task increases, the expected error of the Transformer increases exponentially.
Our work augments the past literature by formally defining two new types of logical consistency that are crucial for valid multi-step reasoning and that have not been studied before. We additionally validate the poor performance of modern LLMs on these new types of consistency and further the understanding of why LLMs fail to generalize well on compositional lasks.
6 Conclusion
We have proposed two types of language model self-consistency that are important for the reliability and logically valid reasoning of LLMs on multi-step tasks. Despite the GPT-3 and GPT-4 models' generally impressive performance on a wide variety of tasks, these models still perform inconsistently on both hypothet- ical and compositional consistency prompts, although larger models appear to perform better. This furthers our understanding of how these otherwise impressive LLMs fail to generalize well on compositional tasks and suggests an additional reason not to trust the outputs of LLMs on complex compositional tasks, especially without extensive empirical validation. Further work is required in order to improve the logical consistency of LLM reasoning, and to investigate whether novel training techniques or further scaling improve hypothetical compositional consistency.
Acknowledgements
We are grateful Eugene Choi, Richard Pang, and Nikita Nangia for helpful discussions and feedback about the design and implementation of this work. This work was supported by National Science Foundation Awards 1922658 and 2046556. CZ is supported by the DARPA PTG program. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation and DARPA. KC is additionally supported by 42dot, Hyundai Motor Company (under the project Uncertainty in Neural Sequence Modeling) and the Samsung Advanced Institute of Technology (under the project Next Generation Deep Learning: From Pattern Recognition to AI). This project has also benefited from financial support to SB by Eric and Wendy Schmidt (made by recommendation of the Schmidt Futures program), Open Philanthropy, and Apple. We also thank the NYU High-P Performaeee Computing Center for in-kind support and OpenAl for providing access to and credits for their models via the API Academic Access Program.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Herbert-Voss, Gretchen Krucger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsoll, M.F. Balcan, and H. Lin (ods.), Advances in Newral Information Processing Systems, volume 33, pp. 1877 1901. Curran Associates, Inc., 2020. URL lttps pdd
/proceedings.. wwwpeediinnnnnuuuirgee http:////aapp//////////////////.
OpenAl Help of current events?
Center.
Do
the URL https://help.openate wwwpaaaa..co////aaaaaaaaaaaeeeaa wwwpppaa...o////aattllll..o..
openai api models have knowledge
6639781-do- the -tp.-tt---paaaipp-idddddaaaa of cuw.oof-ruurrreereeeeeeee oo
11

---

Published in Transactions on Machine Learning Research (01/2024)
Nouha Dziri, Ximing Lu, Melanic Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya. Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. Faith and fate: Limits of transformers on compositionality, 2023.
Yanai Flazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schützee, and Yoav Goldberg. Measuring and Improving Consistency in Pretrained Language Models. Transactions of the Association for Computational Linguistics, 9:1012-1031, December 2021. ISSN 2307-387X. doi:
10.1162/tacl a 00410. URL /tt a_ a_00010
Constanza Fierro and Anders Søgaard. Factual consistency of multilingual pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 3046-3052, Dublin, Treland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022 1118663//1/0222.fmdii----a findings-acl.2400
URL
httpsss//ccaantooogg.oooo http//222.finnnnn.e...e........e findings-acl.22 2c1.240
Myeongjun Jang, Deuk Sin Kwon, and Thomas Lukasiewicz. Accurate, yet inconsistent? consistency analysis on language understanding models. 2021. URL
0666
Myeongjun Jang, Deuk Sin Kwon, and Thomas Lukasiewicz. BECEL:
Benchmark for consistency evaluation of language models. In Proceedings of the 29th International Conference on Computational Lingusistccs, pp. 3680-3696, Gyeongju, Republic of Korca, October 2022. International Committee on Computational Linguistics. URL
Nora Kassner, Oyvind Tafjord, Hinrich Schütze, and Peter Clark. BelicfBank: Adding memory to pre-trained language model for systematic notion of belief. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.
Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18633/v htt////w12221.mm//// maiulppmain.697. URL https://aclanthologg org/2021 20222emm1mmpppp..mm main. 697
Rohit J. Kate, Yuk Wah Wong, and Raymond J. Mooncy. Learning to transform natural to formal languages. In Proceedings of the 20th National Conference on Artificial Intelligence Volume 3, AAAI'05, pp. 1062-1068. AAAT Press, 2005. ISBN 157735236x
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9332 9346, Online, November 2020. Association for Computational Linguistics. doi: /..18653/v1/2020.mmm main.750. URL httlhttsss//aaaattollgo org/2020 emnlp-mail
750
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pp. 986 995, Taipei, Taiwan, November 2017. Asian Federation of Natural Language Processing. URL httrg:://allanhoooog..oomm org//11199/
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3?, 2021.
Eric Mitchell, Joseph J. Noh, Siyan Li, William S. Armstrong, Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher D. Manning. Enhancing celf-consistency and performance of pretrained language models with nli. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2022. URL https://eri /ttps:///rrcmmtchellll cmitchell
ai/concord. pdf
Benjamin Newman, Prafulla Kumar Choubey, and Nazneen Rajani. P-adapters: Robustly extracting factual information from language models with diverse prompts. In International Conference on Learning Representations, 2022. URL httpps://pperreview wttp:n/eooumm//d-hh2aaeeeee
OpenAI. Model index for researchers.
https://platform.oor httpaatoorrrrpppaaaaacoooboobba
OpenAL. Gpt-4 technical report, 2023.
12

---

Published in Transactions on Machine Learning Research (01/2024)
Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in multiple-choice questions, 2023.
Harsh Raj, Domenic Rosati, and Subhabrata Majumdar. Measuring reliability of large language models through semantic consistency. In NeurIPS MT. Safety Workshop, 2022. URL httpp:://ppeneeeeee...
Tormm'id=SgbpddeeVC
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguislics: Human Language Technologies, pp. 2655 2671, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1 22t////1122222.mmmmmmmmm
URL https://acllnthoology
2022. naacl-manlm 191
Wikimedia Foundation. Wikimodia downloads. URL https:///umss..wiiimeellll/ org
John M. Zelle and Raymond J. Mooney. Learning to parse database queries using inductive logic programming. In Proceedings of the Thirleenth National Conference on Artificial Intelligence Volume 2, AAAI96, pp. 1050-1055. AAAI Press, 1996. ISBN 026251091X.
Chunting Zhou, Junxian He, Xuezhe Ma, Taylor Berg- Borg-Kirkpatrick, and Graham Neubig. Prompt consistency for zero-shot task generalization. In Findings of the Association for Computational Línguistics: EMNLP 2022, pp. 2613 2626, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https: http://aacaanthoogy.ooooo org/ 2022. f windingg-eemnn 192
13

---

Published in Transactions on Machine Learning Research (01/2024)
A
Appendix
Wikipedia
Wikipedia
40%
60%
text-a da-001
text-ada-001
boxt-babbage-001
text-babbage-001
cext-curre-001
50%
text-curie-001
30%
text doxtt-avinci-001
Hypothetical consistency rate
gpt4
random chance
random chance
Hypothetical sonnst
25%
20%
30%
15%
20%
10%
10%
5%
0%
0%
Number of words from original completion distinguish
Number words from original completion distinguish
DailyDialog
DailyDialog
text-ada-001
35%
text-babbage-001
text-curie-001
50%
30%
text-davinci-001
Hypothetical consistenyy rate
Hypothetical soncy
random chance
25%
text-ada-001
text-babbage-001
20%
30%
text-curie-001
gpt4
15%
random chance
20%
10%
5%
0%
0%
Number of words from original completion distinguish
Number of words from original completion distinguish
(a) Hypothetical consistency rates with answer choices generated ada-001, babbage-001, curie-001, and davinci-001.
(b) Hypothetical consistency rates with answer choices generated by ada-001, babbage-001, curie-001, and gpt4.
Figure 6: Hypothetical consistency rates on multiple-choice hypothetical consistency prompts for the Wikipedia and DailyDialog datasets. Each multiple-choice prompt contains answer choices generated by the designated four models and an additional answer choice containing the actual continuation of the prompt in the dataset. Each line is the average taken across all k-shot prompts, for [L. 10). The shaded region represents the 95% confidence interval computed with nonparametric bootstrapping. The label "number of words from original completion to distinguish" corresponds to the quantity m in Table I
Comparison of Hypothetical Consistency Against davinci-001 and gpt4 We also run the same hypo- thetical consistency experiments on davinci-001 and gpt4. Hypothetical consistency rates for davinci-001 versus the smaller models are shown in Figure ba] where trends are similar, but davinci-001 performs random chance Wikipedia, like all the other -001 series models. On DailyDialog, however, davinci-001
14

---

Published in Transactions on Machine Learning Research (01/2024)
Wikipedia
Wikipedia
35%
400
davinci
-001
gpt4
18%
18%
19%
19%
14%
3%
10%
35%
30%
Evaluated model
babbage curie
-001
Evaluated madel babbage curie
30%
19%
25%
25%
-001
19%
19%
18%
20%
18%
18%
18%
ada
-001
17%
17%
ada
-15%
10%
10%
Wikipedia
ada -001
babbage -001
curie davinci -001
Wikipedia ada -001
babbage -001
curie
9pt4
-001
-001
Source of answer choice text
Source of answer choice text
DiilyDialog
DailyDialog
35%
40%
davindi
-001
gpt4
16%
17%
19%
23%
17%
9%
11%
13%
40%
+35%
30%
Evaluated model
babbage curie
-001
ada Evaluated model
babbage curle
30%
20%
20%
20%
20%
20%
21%
21%
20%
20%
20%
25%
-001
20%
21%
20%
19%
19%
20%
20%
20%
20%
20%
20%
20%
15%
ada
-001
15%
18%
19%
18%
18%
18%
19%
18%
18%
18%
18%
10%
10%
DailyDialog
babbage
curie davinci -001
DailyDialog ada babbage -001
curie
gpt4
-001
-001
-001
-001
-001
Source answer choice
Source of answer choice
(a) Comparing selected answers on multiple-choice prompts with answer choices generated by ada-001, babbage-001, curie-001, and davinci-001.
(b) Comparing selected answers on multiple-choice prompts with answer choices generated by ada-001, babbage-001, curie-001, and gpt4.
The proportion of the time that each model (ada-001, babbage-001, curie-001, and davinci-008 selects each possible answer choice when prompted with hypothetical consistency prompt. Model outputs that could not be parsed into an answer choice are not included. The columns labeled "Wikipedia" and "DailyDialog correspond the answer choice containing the completion from the original dataset. Model outputs that could not be parsed into an answer choice are included.
Figure 7:
performs noticeably better than all the other model sizes. Similar trends occur in Figure 7a where most models are equally likely to select each answer choice on the Wikipedia dataset, and davinci- is more likely to select either its own or curie-001's completion on the DailyDialog dataset.
In contrast, when gpt4 tasked with distinguishing own completions from those of ada-001, babbage-001, curie-001, and the dataset, gpt4 performs notably better than both davinci-001 and davinci-003 on DailyDialog, reaching 59.9% hypothetical consistency when the number of words to distinguish is 6 (Figure 6bb. However, its hypothetical consistency rate on Wikipedia is comparable to that of davinci-003 (Figure 2), ranging from 17.9% to 27.4%. Figure 7b also demonstrates that gpt4 is siguificantly more likely to select its own completion than the other models are.
It unclear why gpt4 more consistent on DailyDialog than previous models of similar capacity (i.e. davinci-001 and davinci-003), Little known about gpt4's architecture or training, aside from its multinodal abilities and training via reinforcement learning from human feedback (RLHF, OpenAT 2023
15

---

Published in Transactions on Machine Learning Research (01/2024)
Since davinci-003 also trained with RLHF (OpenAl). it is possible that other changes in architecture training may have also contributed to the significant improvement. in hypothetical consistency.
Table 3: Average percent edit distances between completions from davinci-001 versus completions from the three other models and two datasets.
davinci-001 / ada-001
dawinci-001 / babbage-001
davinci-001 /
davinci-001
Dataset
curie-001
Dataset
Wikipedia
73.6%
71.1%
64.4%
71.4%
DailyDialog
71.4%
70.1%
69.3%
79.4%
Table 4: Average percent edit distances between completions from gpt4 versus completions from the three other models and two datasets.
Dataset
gpt4 ada-001
gpt4
babbage-000 0011
gpt4 / curie-001
gpt4 / Dataset
Wikipedia
74.1%
71.7%
68.9%
68.9%
DailyDialog
81.3%
80.1%
77.8%
81.3%
It is also unlikely that gpt4's improvements in hypothetical consistency on DailyDialog can be attributed to datasel memorization. Firstly, we selected only prompls from both Wikipedia and DailyDialog for which all five answer choices were distinct, so gpt4's completion could not have been identical that of the original DailyDialog dataset. Secondly, by the length of the longer string) between the completions of gpt4 versus the completions of the smaller models and the datasets, which are shown in Table 4 The average percent edit distance between gpt4 completions and DailyDialog continuations was 81.3%, indicating that gpt4 was generating strings that were substantially different from the dataset. Similar trends were found when comparing davinci-O00 and davinci-003 against the completions of the other models and the datasets (Tables and
computed the average percent edit distance (the edit distance divided
16


Reviews:
[{"review_id": "qJCrtu6A2w", "reviewer": "Reviewer_3Fto", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Nice presentation of the figures and experimental results.\n\n2. The paper is easy to follow.\n\nWeaknesses:\n\n1. From my point of view, experiments can be more sufficient. For example, is it possible to add more LLMs, such as Llama and ChatGLM, as the backbone model, for evaluation? \n\n2. Although the paper mainly focuses on consistency, it would be better to provide an accuracy evaluation.\n\n3. The related work part can be more complete with the latest achievements on LLMs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper explores the importance of self-consistency in large language models (LLMs) for multi-step reasoning tasks. It introduces two types of self-consistency, hypothetical consistency and compositional consistency, and demonstrates that various versions of GPT-3/-4 models exhibit poor consistency rates in both types. The paper formalizes the definitions of consistency and prompt transformations and presents empirical evidence of low consistency rates in LLMs.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Please see the weaknesses pointed out in the section above.", "broader_impact_concerns": "There are no broad impact concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KeXarfUwv0", "reviewer": "Reviewer_1XQ6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The use of semantic consistency to evaluate LLM's trustworthiness has gained a lot of attention and the paper makes interesting observations on how two specific transformations can be useful in this context.\n\n- Formalization is rather trivial and the composition formalization is illustrated via an example. The reviewer understands the challenge of formalizing arbitrary composition transformation since more involved prompts could need higher order logic specification. The notations make a simple idea appear unnecessarily complicated without adding mathematical rigor. \n\n- The use of \"multiple choice\" pattern makes the application of this approach difficult because generating reasonable such choices automatically will be difficult. Using completions of other models makes the test a bit biased. We are no longer checking consistency of a given model but checking consistency modulo response of other models. Correlated failures (which are likely given similar data) can make this problematic.\n\n- The use of arithmetic and geoquery datasets for compositional transformation appears to highlight the limited utility of this approach when compositions are easy to identify.\n\n- Overall, the paper is light on novelty. Two consistency checking transformations are used to evaluate a set of models on a set of datasets. Conclusions appear rather straightforward and unsurprising - \"all models fail at distinguishing their own completions\". The claims and corresponding evidences in the paper need to be strengthened.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper builds on the recent work on self-consistency of LLMs and proposes two new kinds of consistency checks - hypothetical transformation and compositional transformations. Hypothetical transformations include prompts such as those asking for completion of the prompt and response to check whether LLM produces same or similar response. Compositional transformations replace the intermediate steps with the response of the LLM to check whether the LLM response remains conformant for these changes.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Please see the weaknesses pointed out in the section above.", "broader_impact_concerns": "There are no broad impact concern. The introduction section brings out broader impact well.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "q5wsvagmKH", "reviewer": "Reviewer_QKsC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper does a good job formalizing the two new consistency metrics and evaluating them on a good number of models and tasks. The experiments performed in this paper are non-trivial and I appreciate the authors' effort in putting this together. Overall, the paper makes a good contribution as it shows yet another data point demonstrating logical inconsistencies of LLMs. \n\nWeaknesses:\n\nThe paper pays a somewhat high \"notation cost\" for formalizing the two metrics. While reading the paper I couldn't help but wonder if there was a more friendly way of formalizing and presenting the results. The paper is heavy on notation and I had to go back and forth the paragraphs to remind myself what each of the letters meant. I even wondered whether the paper would stand by itself if it had Section 2 remove. Section 2 is very dense and it doesn't offer any new insights and I am not sure it helps the reader understand the later results of the paper. \n\nGiven the current notation used in Section 2, I would write $p^*_{\\theta}$ instead of $p^*$, as it would help distinguish that we are talking about a model (even if it is the \"perfect\" model). \n\nThe first part of the paper (Sections 1 and 2) talks about semantically similar outputs, but Section 3 evaluates the models on exact string matching. I find that to be a serious weakness of the paper. For example, there is a chance that what is currently considered a wrong choice of the model is in fact a choice that is semantically equivalent to the model's previous choice. Perhaps the numbers shown in Section 3 are pessimistic because it only counts an answer as correct if the string is exactly the same as the model's previous output (case insensitive comparison). \n\nI understand why the authors decided to perform such as evaluation, as it would be hard to automatically evaluate semantic equivalence. The gold standard in this case should be to run a user study to evaluate the models. \n\nIn Section 4, in addition to showing the average results of the models in terms of compositional consistency, the paper also discusses how the models make mistakes. Later, in the conclusion section, it reads as a contribution of the work: \"further the understanding of why LLMs fail to generalize well on compositional tasks.\" I think this isn't correct. What the paper shows is *how* the LLMs make mistakes and it is far from explaining *why*.\n\nThis is an empirical paper, so I would prefer to see the results of the Appendix in the main text. If the authors were looking for a shorter paper, I would easily exchange parts of the formalization with more empirical results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies different GPT models in terms of two consistency metrics: hypothetical consistency and compositional consistency. The paper formalizes the two consistencies metrics and performs detailed experiments to evaluate both of them. \n\nHypothetical consistency is evaluated on auto-completion tasks using recent Wikipedia text (i.e., text not used to train the GPT models). Results show that all models evaluated performed poorly in terms of hypothetical consistency. \n\nCompositional consistency is evaluated on arithmetic and GeoQuery tasks, with similar conclusions, that the models tend perform poorly, but with a significant improvement for larger models (davinci-003 and gpt4).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would appreciate a discussion about the lack of user studies to evaluate the models, why they weren't performed and why we are able to approximate their results with the current set of experiments. In particular, this discussion should explain why the string-matching approach used in the Section 3 doesn't provide a pessimistic view of the results. \n\nThe contributions need to be toned down. Instead of claiming \"why\", the paper should list only \"how\" mistakes are made as a contribution.\n\nI would also like to see the results from the Appendix in the main text.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LOZlaclrMg", "reviewer": "Reviewer_sPSQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: Studying the capabilities of popular LLMs is an important topic since LLMs are being used in an ever-increasing range of applications. Mapping out their limitations allows the research community to improve their quality and manage expectations concerning the tasks we expect such models to perform well.\n\nWeaknesses:\n* The formalism introduced in Section 2 is unclear. It only becomes clearer when the reader transitions to Sections 4 and 5. In particular:\n\t* The operator $\\sim$ depends on an informal notion of \"semantic equivalence\".\n\t* This section has some problems with quantifiers. For instance, the second bullet on the 3rd page should have \"for all $p'$\" (instead of \"for all $p,q\\in\\mathcal{V}^*$), or definitions (e.g., Definition 2.6 where from the formulation it seems that the definition of compositional consistency is conditional on $p$ and $p'$).\n\t* The Definition 2.4 of hypothetical consistency (HC), $g_{p_\\theta}(f(p',p))\\sim g_{p_\\theta}(p)$, does not seem to match well with prompt templates shown on page 4 (mid-page or Table 1). For instance, it seems to generate a type mismatch, where an answer to a hypothetical prompt seems to be A/B/C/D/E, while the model's output to the raw prompt is composed of several words (e.g., \"notable figures whose\"). Additionally, the reader may find the best definition of HC in the last sentence of the introduction to Section 3 (page 5).\n\t* Definition 2.6 (consistency over compositional transformations) includes a property $p\\sim g_{p_\\theta}(p)$ which is confusing if one thinks about $p=$\"This quilt begun in 1856 when she was seventeen includes the autographs on top of the blocks of many known celebrities and politicians of the day. Other\" and $g_{p_\\theta}(p)=$\"notable figures whose\". This makes sense only in Section 4, where e.g., the output of $g_{p_\\theta}$ corresponds to a result of parsing an arithmetic expression or a FunQL reformulation of a question about US geography.\n\t* The naming convention of Definition 2.6 (\"consistent\" vs \"inconsistent\") is unfortunate, as consistent and inconsistent models do not cover the space of all models (the paper recognizes this fact). It seems that a better name could be used here.\n\t* To sum up, this section should either be revised (to form correct mathematical definitions, provide examples, and show that everything works as intended and \"types match\") or removed.\n* Experimental sections:\n\t* General comments:\n\t\t* Evaluation protocol should be clearly stated and should include discussion and preferably answers to the following issues:\n\t\t\t* Evaluating LLMs is a known hard problem. LLMs output sequences of tokens, and there exist multiple ways of sampling from them (a.k.a. \"decoding strategies\"). Additionally, the models' responses are sensitive to the phrasing of prompts and in-context examples. Analyzing choices that LLMs make might require an analysis that is invariant to the particular wording of prompts. This, then, might call for an appropriate statistical methodology.\n\t\t\t* Some models are fine-tuned, e.g., using reinforcement learning with human feedback (RLHF). RLHF is known to cause collapse in the models' entropy.  \n\t\t\t* It is known that GPT models are fine-tuned on the input. This means that (a) the evaluation should be performed in short closed windows and on examples (prompts) that the model has not seen before (otherwise, there could be a data leak), (b) an appropriate discussion should be provided concerning whether the used models were trained on the datasets used for evaluation (here Wikipedia, DailyDialog, arithmetic tasks, GeoQuery), (c) the results could be hard to reproduce.\n\t\t* Figures in Sections 3 and 4 mention in-context examples, but the prompt template is not provided, and not always the number of examples is known (e.g., Figure 3). Also, there is no discussion of how the choice of in-context examples can impact the answer. \n\t* Section 3:\n\t\t* Figure 2 could benefit from adding a comment in the caption that \"Number of words from original completition to distinguish\" corresponds to $m$ in Table 1. Additionally, the shade around the curves is not explained.\n\t\t* It is not clear why Figure 2 does not include GPT-4.\n\t\t* Figure 3 could be described better, e.g., that the \"Wikipedia\" or \"DailyDialog\" column corresponds to the ground truth completion from the corresponding dataset. How are the numbers related to the plots presented in Figure 2? \n\t\t* The results provided in this section are not well explained. It is not clear what additional insight is provided in the second paragraph of Section 3.2. Some questions that could be answered are: What results should be expected before the experiment, and what are the reasons for the actual results? What shapes of the curves do we expect and why? How is it related to the greedy decoding scheme chosen in the paper (e.g., it might not generate the most likely response, and hence, the answers of other models could have a higher likelihood)? What is the uncertainty about these results (see the \"Evaluation protocol\" item above)? \n\t* Section 4:\n\t\t* Figure 4 includes multiple in-context examples. \n\t\t* The GoeQuery causes some quirks where natural language could be mixed with FunQL. This results in a relaxed consistency notion. How could this approximation impact the results of the experiments?\n\t\t* The phenomenon described in Section 4.2 concerning the arithmetic dataset should be compared to similar results for the chain of thought techniques, where the final answer does not necessarily match the provided reasoning.\n\t\t* The situation described for GeoQuery data, where the same query can be parsed in multiple ways, could call for more careful curation of queries or raise a concern about whether this is a correct dataset for the current study.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies various GPT-3 and sometimes GPT-4 LLMs for consistency of answers. Two types of consistency are considered: consistency on hypothetical transformations and compositional self-consistency. The Wikipedia and DailyDialog datasets are used for the former, while the latter is evaluated on Arithmetic and GeoQuery tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The empirical evaluation of consistency is good, but the reviewers remain concerned about the complexity of the formalism introduced in Section 2 to define consistency.  Nevertheless, it is good to define consistency and inconsistency formally.  Additional comments by the reviewers include:

1. The paper continues to use consistent/inconsistent notion that does not split the space of models into two parts (i.e., there are models that are neither consistent nor inconsistent).
2. The example for Definition 2.5 provided at the bottom on page 4 assumes that the equivalence relation depends on p' (otherwise, there is no way to know what "E" corresponds to). This should be added to the list of simplifications made for the purpose of experiments that is discussed on page 5.
3. In Figure 3, it is still unclear what " averaged across all prompts" refers to (over all m and k?)
4. The confidence intervals are not described (is it bootstrap, parametric, or over how many outcomes/seeds)?
5. Some models considered in the paper are tuned on human data (IFT or PPO). This could explain why the models output the same original completions. These kinds of behaviors were alluded to in the question concerning mode collapse by RLHF.
6. In Section 3.2, it seems that gpt4 displays some interesting properties that other models do not (visualized on Figure 6b). It is strange why this was not put in the main body of the paper (this circles back to the question concerning reporting results for gpt4).
7. Since there is only one section in Appendix A, there is no need to introduce "A.1".
Decision: Accept (poster)