Prompt-Reverse Inconsistency: LLM Self-Inconsistency Beyond Generative Randomness and Prompt Paraphrasing
Authors: Jihyun Janice Ahn
Wenpeng Yin
Venue: COLM 2025 (2025)
Keywords: Large Language Model
Natural Language Process
Inconsistency of LLMs
Prompt-Reverse Inconsistency
Randomness Inconsistency
Paraphrase Inconsistency
TL;DR: This paper introduces Prompt-Reverse Inconsistency (PRIN), where Large Language Models give conflicting answers when identifying correct versus incorrect responses, raising concerns about their logical reliability.

Abstract: While the inconsistency of LLMs is not a novel topic, prior research has predominantly addressed two types of generative inconsistencies: i) Randomness Inconsistency: running the same LLM multiple trials, yielding varying responses; ii) Paraphrase Inconsistency: paraphrased prompts result in different responses from the same LLM. Randomness Inconsistency arises from the inherent randomness due to stochastic sampling in generative models, while Paraphrase Inconsistency is a consequence of the language modeling objectives, where paraphrased prompts alter the distribution of vocabulary logits. This research discovers Prompt-Reverse Inconsistency (PRIN), a new form of LLM self-inconsistency: given a question and a couple of LLM-generated answer candidates, the LLM often has conflicting responses when prompted "Which are correct answers?" and "Which are incorrect answers?". PRIN poses a big concern as it undermines the credibility of LLM-as-a-judge, and suggests a challenge for LLMs to adhere to basic logical rules. We conduct a series of experiments to investigate PRIN, examining the extent of PRIN across different LLMs, methods to mitigate it, potential applications, and its relationship with Randomness Inconsistency and Paraphrase Inconsistency. As the first study to explore PRIN, our findings offer valuable insights into the inner workings of LLMs and contribute to advancing trustworthy AI.

Published as a conference paper at COLM 2025
Prompt-Reverse Inconsistency: LLM Self-Inconsistency Be- yond Generative Randomness and Prompt Paraphrasing
Jihyun Janice Ahn Wenpeng Yin Department of Computer Science & Engineering The Pennsylvania State University University Park, PA 16802, USA {jfa5672, wenpeng @psu. .ddu
Abstract
While the inconsistency of LLMs is not novel topic, prior research has predominantly addressed two types of generative inconsistencies: i) Randomness Inconsistency: running the same LLM multiple trials, yielding varying responses; ii) Paraphrase Inconsistency; paraphrased prompts result in different responses from the same LLM. Randomness Inconsistency arises from the inherent randomness due to stochastic sampling in generative models, while Paraphrase Inconsistency is consequence of the language modeling objectives, where paraphrased prompts alter the distribution of vocabulary logits. This research dis- covers Prompt-Reverse Inconsistencl (PRIN), a new form of LLM self- inconsistency: given question and couple of LLM-generated answer candidates, the LLM often has conflicting responses when prompted "Which are correct ansuers?? and "Which are incorrect answers?". PRIN poses big concern as it undermines the credibility of LLM-as-a-judge and suggests a challenge for LLMs to adhere to basic logical rules. We conduct a series of experiments to investigate PRIN, examining the extent of PRIN across different LLMs, methods to mitigate it, potential applications, and rela- tionship with Randomness Inconsistend As the first study to explore PRIN, our findings offer valuable insights into the inner workings of LLMs and contribute to advancing trustworthy AT.
and Paraphrase Inconsistence
1 Introduction
Large language models (LLMs), despite their strong performance across various domains, often exhibit behaviors that diverge significantly from human reasoning. One well-known issue in their generative process is inconsistency. LLM inconsistency is widely recognized by researchers and users, and it can be mainly categorized into two types:
Randomness Inconsistency: Even when given the same prompt, an LLM may generate different responses across multiple trials. This randomness arises due to factors such as sampling stochasticity, model non-determinism, and softmax and floating-point precision errors in the generation process.
Paraphrase Inconsistency: When prompt is rephrased while maintaining the same meaning, the LLM's response can still vary. This occurs because the refor- mulated prompt implicitly alters the probability distribution within the language model's objective function.
Beyond generative tasks, LLMs are increasingly used for discriminative reasoning-a crucial capability in applications such as Al-assisted judging, grading, and evaluation. However, a fundamental challenge arises: due to generative inconsistencies, LLMs often produce multiple conflicting candidate answers for the same question. While the Self-Consistence method (Wang et al., 2023b) leverages majority voting to mitigate this issue, an alternative approach is to enhance LLMs' ability to self-select the correct answer from given set of

---

Published as a conference paper at COLM 2025
options. Unfortunately, LLMs also exhibit inconsistency in discriminative reasoning, which we term Prompt-Reverse Inconsistency (PRIN).
PRIN arises when an LLM is tasked with evaluating multiple answer candidates and de- termining which are correct or incorrect. As shown in Table 1, LLMs frequently provide conflicting judgments over the same set of answer choices. This inconsistency raises serious concerns regarding: The reliability of LLM-as-a-judge- Inconsistencies undermine their trustworthiness in high-stakes applications, such as automated grading, peer review, and legal analysis. Fundamental logical inconsistencies: If LLMs frequently violate basic logical principles when making judgments, their utility as reasoning agents is severely limited.
This paper conducts systematic investigation of PRIN in both closed-source. and open-source LLMs, including GPT-4 (Ope- nAI, 2023), Llama-3-8B-Instruce Llama-3.3-70B-nnstruue 2024), Falcon-40B (Almazrouei et al., 2023), Qwen 2.5-72B (Team, 2024), and Mixtral-3x22B-MoE (Jiang et al., 2024). We evalu- ate these models across three tasks-MATH (Hendrycks et al., 2021), MathQA (Amini et al., 2019), and Equationlnference (Lou et al., 2024)-spanning various answer set sizes, context lengths, domains, and difficulty levels (from high school, college, to PhD-level problems). Specifically, we design experiments to answer the following six research questions: 21 How do different LLMs exhibit PRIN ? Q2: How will model randomness and prompt paraphrasing affect PRIN? 23: How to mitigate PRIN LLMs? Q4: How does PRIN correlate with Randomness Inconsistency and Paraphrass Inconsistency! Q5: How effective can PRIN be leveraged to enhance task performance? Q6: How does PRIN vary with different sizes of options?
Question: if an integer and 101 n² is less than or equal to 10,000, what is the greatest possible value n?
(Meta,
Options: 7, 8, 10,
Direct Prompt: What are the correct answers?
GPT4: "C"
Reverse Prompt: What are the incorrect answers?
GPT4 D,
Table 1: PRIN example from GPT4 (March 28, 2025)
Our findings reveal several key insights. First, PRIN does not positively correlate with Randomness Inconsistency or Paraphrase Inconsistency, these inconsistencies, such as Llama-3 and Falcon, still exhibit high PRIN. This suggests that while these models are more deterministic, they fail to maintain logical consistency between Direct Prompt and Reverse Prompt. Second, PRIN can be mitigated by incorporating explicit reasoning paths between the question and answer candidates before prompting the LLM to determine correctness. Additionally, providing explainable information for the negation in Reverse Prompt further reduces PRIN. Third, combining both Direct Prompt and Reverse Prompt reasoning can outperform the Self-Consistency approach when selecting the final answer from candidate pool. However, this improvement primarily observed in top- performing models such GPT-4 and GPT-4o, while weaker models like Llama-3 show little to no benefit, likely due their weaker instruction-followii capabilities.
some LLMs with low levels of
Our contributions are threefold: i) this is the first study to discover and conduct an in- depth analysis of PRIN; ii) we propose effective solutions to mitigate PRIN and explore ways to leverage it; and iii) our experimental findings not only enhance the understanding of this non-huuan--lkk discriminative behavior LLMs but also raises critical concerns for applications where LLMs serve as judges or evaluators.
Related Work
This section mainly discusses prior work studying Randomness Inconsistency and Paraphrase Inconsistency particularly in LLMs.
Randomness Inconsistency. Bubeck et al. (2023) brought attention to the issue of randomness-nddued inconsistency of GPT4. Building this, Wang & Wang (2025) con-
2

---

Published as a conference paper at COLM 2025
ducted a comprehensive evaluation of LLM consistency and reproducibility in finance and accounting tasks, highlighting the practical consequences of such variability. Similarly, Atil et al. (2024) systematically examined LLM stability by repeatedly running identical inputs, revealing up to 10% variation in output accuracy even under deterministic settings. Beyond quantitative assessment, Lee et al. (2024) explored how these inconsistencies affect users, finding that while they may reduce perceived Al reliability, they can also enhance user comprehension by presenting diverse perspectives. To address these issues, Wan et al. (2025) proposed a sufficiency scoring method that evaluates both local and global consistency in LLM responses, offering framework to analyze and mitigate instability driven by randomness.
Paraphrase Inconsistency. Elazar et al. (2021) explored factual consistency across dif- ferent query patterns and showed that while some paraphrase forms reliably extracted factual knowledge, others failed, revealing the model's paraphrasing sensitivity. Similarly, Ye et al. (2023) investigated this phenomenon ChatGPT and found that response accuracy fluctuated by 3.2% across paraphrased prompts, highlighting the influence of grammatical and stylistic variations on model behavior. Supporting this line of work, Jang & Lukasiewicz (2023) documented cases of self-contradictions in ChatGPT and GPT-4 when exposed to paraphrased questions, confirming that even minor linguistic variations can lead se- mantic inconsistencies. Gu et al. (2023) extended this observation to instruction-driven tasks, demonstrating that LLMs often falter when task instructions vary in form, length, or abstraction, which further complicates generalization across paraphrased input formats. In addition to task-specific studies, Liu et al. (2024) provided broader survey on LLM trustworthiness, in which they discussed inconsistency as core reliability issue and empha- sized the need for robust solutions to mitigate its effects. Complementing these discussions, recent work has introduced quantitative approaches analyze and address paraphrase sen- sitivity. For example, Errica et al. (2024) proposed metrics that measure how minor prompt variations influence LLM predictions text classification tasks, offering fine-grained assessment of response stability. Further, Ghazarian et al. (2024) examined structural varia- tions in semantically equivalent prompts and found notable inconsistency evaluations. They proposed an in-context learning strategy with demonstrations to improve robustness against paraphrasing. Finally, Mcllroy- Young et al. (2024) tackled a related issue of order dependency in prompts and introduced Set-Based Prompting, a method designed to ensure consistent model behavior regardless of the sequence of sub-inputs, offering a new angle mitigating seaphhass--rrveer inconsistencies.
LLM-basee
Inconsistency, not only analyzing this LLM behavior but, more importantly, proposing simple yet effective methods to mitigate the issue. Additionally, we examine its connection to Randomness Inconsistency and Paraphrase Inconsistency, as well as ways to leverage this inconsistency for improved model reliability.
Our Work is the first to explore Prompt-Revers
Prompt-Reverse Inconsistency
Problem formulation. Assume the prompt p, and LLM M, multiple trials of M(p) leads
to n distinct answer candidates A = {{1, {,,,,,,... R2, an with each candidate derived through a Chain-of-Thought (Wei al., 2022b) reasoning path r,. The task now is to figure out the correct answer from the pool (2, .r. by querying M again. In this work, we study M's s discriminative behavior through the following two prompts.
Direct Prompt. Given the prompt p, answer options (a1,02," an), it asks the correct one directly, e.g.,
Given this question [problem description] and its answer options: "a1" "ai", "az", "an" " please output the correct ones.
Reverse Prompt. Conversely, the models determine the incorrect choices follows:
3

---

Published as a conference paper at COLM 2025
Given this question (problem description] and its answer options: "a1", "a2", "nn"" " please output the incorrect ones.
PRIN Metric (The lower, the better). Given the entire answer pool A, assuming Direct Prompt return answer set Adiret and Reverse Prompt returns Arevesse our metric is defined based on this rule: if the correct answer sets derived by both prompts are the same, then no PRIN.
Therefore, first, the correct answer set by Direct Prompt is Adirect- Then the correct answer set by Reverse Prompt is A \ Avenesee Then we compute the similarity of the two versions of correct answer sets through Fl:
= F1(Adirect/ Areverse)
(1)
then the PRIN score is:
PRIN = 1.0 -
(2)
Question: Why not define the PRIN score as the similarity between Adirect and Areverser i.e,, F1(Aairect, )
Intuitively, if Adirect and Areverse completely complementary (e.g., Adirect {a, a2) and Areverse = {a3, As,...,,ay it implies no PRIN. However, in practice, the union their answers may not cover the entire answer pool. For instance, if A {aI a2) but Aveverse {a an ), using F1 as a measure would result in F1 (Adeett Areverse = 0.0, incorrectly indicating no PRIN. This is problematic because A {az-2. a, suggests that the Reverse Prompt prompt considers reflects inconsistency to Adirect (2).
On An-3, An-1 1 as correct, which clearly
Experiments
Datasets. We select the following three representative datasets: Math (Hendrycks et al., 2021): This dataset consists of Math Word Problems, where each question is accompanied by the correct answer a; and corresponding Chain-of-Thought reasoning path r;- MathQA (Amini et al., 2019): A multiple-choice math dataset in which each Math Word Problem is presented with five answer choices, only one of which is correct. Unlike the Math dataset, reasoning paths are not provided. EquInfer (Lou et al., 2024): Designed to simulate the paper review process, this dataset evaluates equation correctness within a given context in scientific paper. Each instance contains four equation candidates, with only one being correct, along with the surrounding paper context before and after the equation.
This dataset selection demonstrates that the Prompt-Reverse Inconsistency problem arises in both generative tasks (e.g., MATH) and discriminative tasks (e-g,, MathQA and EquInfer), highlighting its broader implications. Table summarizes key properties of these datasets.
Format
Size
Options
Context
Complexity
MATH
p
5,000
None
Short
High School
MathQA
(p; (a1, R3, R3, a5))
2,985
5
Medium
College
EquInfer
(p; {a1, a3, a4))
1,049
4
Long
Ph.D.
Table 2: Summary of three datasets (MATH, MathQA, and Equinfer).
The experiment utilizes a combination of one closed-source model, GPT-41, along- side five open-source models: GPT-4 (OpenAI, 2023) Llama-3-BB-Instrue (Llama3) and Llama-3.3-70B-Instter Instruutt (Llama3.3) (Meta, 2024), Falcon-40B (Falcon) (Almazrouei et al., 2023), Qwen 2.5-72B (Qwen2.5) (Team, 2024), and Mixtral-8x222-MMon (Mixtral) (Jiang et al., 2024).
LLMs.
Duue to budget and administrative approval constraints, we cannot report other closed-source I.Mss.
4

---

Published as a conference paper at COLM 2025
MATH
MathQA
EquInfer
Mean
GPTT
38.69
38.60
42.65
39.98
Qwen2.5
58.41
51.31
70.82
60.18
Mixtral
67.77
63.58
74.83
68.73
Llama3.3
80.96
61.17
76.62
72.92
Falcon
71.79
68.69
83.42
74.63
Llama3 74.10
84.08
80.46
79.55
Table 3: PRIN scores for all LLMs (answers to 21)-
Setting. To prepare for the core experiments, generate answer options for the MATH dataset, as they are absent in the original benchmark. GPT-4 solves each problem multiple times to produce five distinct answer choices, each with Chain-of-Thoughhe path, ensuring a uniform five-option format for the main experiment. As the Equlnfer has text of both sides a problem description and 4 options, due to the token limitation for LLMs, 200 words for each side of the context are given to LLMs according to the suggestion by Lou et al. (2024).
4.1
11 How do different LLMs exhibit PRIN?
Table reveals a consistent pattern of PRIN across all evaluated LLMs, highlighting it as fundamental and unresolved challenge. GPT-4 exhibits the lowest PRIN scores across all benchmarks, indicating that its superior instruction-folloiing and reasoning abilities help mitigate, but not eliminate, inconsistency, its PRIN still hovers around 40%. Open-source models, including Qwen2.5, Mixtral, Falcon, Llama3, and Llama3.3, show significantly higher PRIN values, often exceeding 60% MathQA and Equlnfer, suggesting that their reasoning abilities are particularly vulnerable when faced with reversed prompts. Interest- ingly, Qwen2.5 consistently outperforms other open-source models, possibly due stronger instruction tuning, positioning it as the most robust among its peers. Moreover, the compar- ison between Llama3 and Llama3.3 shows that, despite architectural similarities, Llama3.3 reduces PRIN on MathQA and EquInfer but unexpectedly worsens on MATH, hinting that PRIN may be sensitive to domain-specific generalization. The consistently higher PRIN on Equinfer across models suggests that this dataset poses unique challenges, likely due to its demand for nuanced reasoning under prompt reversals. Overall, the results indicate that while advanced models like GPT-4 alleviate PRIN to some extent, significant inconsistency persists across all models, pointing to PRIN as critical barrier to trustworthy reasoning in LLMs.
4.2
Q2: How will model randomness and prompt paraphrasing affect PRIN?
In this subsection, explore if Direct Prompt and Reverse Prompt still show inconsistency even if we i) paraphrase them (apply Paraphrase Inconsistency), or ii) run them multiple times (apply Randomness Inconsistency).
First, as Table 4 shows, we paraphrase Direct Prompt and Reverse Prompt introduced in Section 3 into two versions.
original (v0)
paraphrased prompt (v1)
paraphrased prompt 2 (v2) Please output the appropriate ones.
Direct
Please output the
Please output the right ones.
Prompt
correct ones.
Reverse
Please output the
Please output the wrong ones.
Please output the inappropriate ones.
Prompt
incorrect ones.
Table 4: Paraphrased Direct Prompt and Reverse Prompt.
Results of applying paraphrasing: Table presents the effects of prompt paraphrasing PRIN scores. The variations in scores indicate the presence of Paraphrase Inconsistency, demonstrating that LLMs' responses influenced by how prompts are phrased. However,

---

Published as a conference paper at COLM 2025
the changes are relatively minor, suggesting that PRIN remains largely stable across para- phrased inputs. This implies that while LLMs are somewhat sensitive to different prompt formulations, their PRIN follows systematic pattern rather than being highly volatile due to rewording alone.
MATH
MathQA
EquInfer vl
v0
v1
v2
v0
vl
v2
v0
v2
GPT4
38.69
38.81
37.22
38.60
39.48
38.14
42.65
41.89
48.34
Qwen2.5
58.41
56.63
60.02
51.31
50.55
58.11
70.82
71.76
69.73
Mixtral
67.77
67.29
69.10
63.58
68.04
73.31
74.83
78.32
74.47
Llama3.3
80.96
81.40
78.39
61.17
59.82
59.17
76.62
73.51
76.28
Falcon
71.79
72.06
71.74
68.69
71.74
73.68
83.42
86.22
81.89
Llama3
74.10
73.69
72.91
84.08
83.40
81.88
80.46
80.76
80.86
Table 5: Effect of prompt paraphrasing on inconsistency across tasks.
Next, we conduct five repeated runs of the original Direct Prompt and Reverse Prompt (i.e., v0) prompts for each LLM to assess the consistency of their results. Table 6 presents the mean and standard deviation of PRIN, confirming that PRIN remains stable with only minor fluctuations across runs. Together, Tables 5-6 demonstrate that PRIN is not an artifact of particular choice of Direct Prompt and Reverse Prompt prompts but rather systematic issue that persists across wide range of LLMs.
MATH
MathQA
Equinfer
GPT4
38.6610.29
39.54
42.81
Qwen2.5
58.67-0..88
52.64
69.75:10.44
Mixtral
67.74=10.42
67.08+0.62
0..4
Llama3.3
80.31
59.35=00.4
7..36
Falcon
71.01 1.015
68.94
81.16.10.50
Llama3
74.56
83.93
80.70
Table 6: PRIN scores when we run Direct Prompt and Reverse Prompt five times.
Question: Table 4 suggests that Reverse Prompt prompts often involve negation. How well does LLMs' PRIN align with their performance on negation-specifi task? To inves- tigate this, we evaluate the LLMs on the negation-focused dataset CONDAQA (Ravichander et al., 2022) and compare their PRIN scores ("Mean" column in Table 3) with their error rates on CONDAQA. As shown in Figure 1, the two measures exhibit a strong alignment, with Pearson correlation coefficient of 0.67. This result confirms that the core challenge captured by PRIN is closely related to the models' difficulty in handling negation.
4.3
Q3: How to mitigate PRIN in LLMs?
To ensure the broad applicability of our investigation into PRIN, the afore- mentioned experiments were conducted with each query p paired with a pool of answer candidates {a, (d,,,,,...,444 An However, in real-world scenarios, humans may better distinguish between Direct Prompt and Reverse Prompt when they understand how each answer candidate was derived. Motivated by this, our first approach incorporates CoT reasoning paths for each answer candidate A; allowing for more informed evaluation of PRIN. We refer to it as "w/ CoT".
Our Approach:
Our second approach is inspired by the observations (to Q2) that discrepancies between Reverse Prompt and Direct Prompt may arise due to the LLMs' difficulty processing negation in Reverse Prompt. To address this, enhance the clarity of negation terms by explicitly explaining their meaning within Reverse Prompt. One simple sentence such as "please recall that 'incorrect options' are simply the options different from the correct ones." was added in the end of the Reverse Prompt. The same evaluation metric is then applied to assess the impact of this intervention. We refer to this approach "w/ neg-exp".

---

Published as a conference paper at COLM 2025
Ense rate CONDAQA
70
60
GPT4
Llama3.3
Mixual
Qwea25
Falcon
Liama
LLMs
Figure 1: Mean PRIN on three main benchmarks vs. error rates on CONDAQA
Figure demonstrates the fectiveness of our PRIN mitigation ap- proaches. Incorporating detailed rea- soning ensures models make informed decisions based on deeper understand- ing rather than rote selections. The empirical evidence highlights the im- portance of contextual reasoning in im- proving Al comprehension and reduc- ing errors. Moreover, reinforcing mod- els with explicit explanations about negation in inverse tasks further de- creases PRIN. Providing clarifications, especially regarding negation terminol- ogy, aids in reducing confusion and logical pitfalls, establishing an effective strategy for error reduction.
Results:
38.24
PRIN Score
16.24
10-
5.
0
(p. (a1, (,,a))
w/ CoT
w/ CoT & neg-exp
Figure 2: PRIN score of GPT4 on MATH benchmark with different mitigation approaches.
4.4
Q4: How does PRIN correlate with Randomness Inconsistency and Paraphrase Inconsistency?
Setup. To investigate 24, we quantitatively assess Randomness Inconsistency and Paraphrase Inconsistency across various LLMs.
For Randomness Inconsistency, we run Direct Prompt five times and count the number of distinct answers k, computing the score as k/5. For Paraphrase Inconsistency, we use five paraphrased versions of Direct Prompt, recording distinct answers and defining the score as k/5. Since already have three paraphrased versions (v0, v1, v2) from Q2, generate two additional versions using GPT-4, ensuring a total of five.
We evaluate all three inconsistency types (PRIN, Randomness Inconsi Inconsistency, and Paraphrase Inconsistency) on MATH dataset, with lower scores indicating better consistency.
Results. Figure 3 ranks the LLMs based their PRIN scores and additionally presents their Randomness Inconsistency and Paraphrase Inconsistency. This analysis aims to address two key questions:
(i) Why does GPT-4 exhibit higher Randomness Inconsistency and Paraphrase Inconsistency than most open-source models? Through error analysis, we observed that
7

---

Published as a conference paper at COLM 2025
80
Randominss Inconsisiency
Pamplmne Inamoblency
50
GP'4
Qwen2.5
Mixual
Falcen
Llama3
Lluma3.3
LLMs
Figure 3: LLMss
Scores of PRIN, Randomness Inconsistency and Paraphrase Inconsistency for
MATH
MathQA
EquInfer
GPT4
GPT-4o
Llama3
GPT4
GPT-4o
Llama3
GPT4
GPT-4o
Llama3
CoT 47.58
50.67
21.55
72.57
82.73
39.03
34.81
31.27
12.60
Self-Consist,
55.14
54.72
26.72
79.50
85.33
42.58
36.42
33.94
16.59
PRIN
56.44
56.82
25.98
82.04
86.63
42.98
37.37
34.51
16.02
Table 7: Comparing PRIN with CoT and Self-consistency in promoting LLM performance.
GPT-4 tends to follow instructions more faithfully, often formatting its answers using user- specified patterns such as quotation marks, brackets, or colons, even when options are not provided. In contrast, open-source models, regardless of size, frequently fail to follow these formatting instructions and sometimes even terminate generation prematurely without producing valid answers. As result, when extract answer spans from these mod- els, we often obtain empty outputs. Therefore, the lower Randomness Inconsistency and Paraphrase Inconsistency of these models are not due genuine consistency, but rather stem from consistently producing invalid or incomplete outputs. Importantly, our PRIN met- ric still penalizes such cases when both Direct Prompt and Reverse Prompt outputs are empty, maintaining its diagnostic reliability.
(ii) Why does Llama3.3 show lower Randomness Inconsistency but higher Paraphrase Inconsistency compared to Llama3? We hypothesize that Llama3.3 has been tuned to be- have more deterministically, which mitigates its randomness-driven inconsistency. However, Llama3's poor instruction-followe following capability prevents its paraphrase inconsistency from being fully revealed, as often fails to produce meaningful outputs regardless paraphras- ing. In contrast, Llama3.3 generates more valid outputs due to better instruction-folloiing thereby exposing its paraphrase inconsistency more clearly.
4.5
Q5: How effective can PRIN be leveraged to enhance task performance?
Beyond analyzing PRIN an undesired LLM behavior, we explore how the Direct Prompt and Reverse Prompt can synergize to improve response accuracy. Intuitively, both Direct Prompt and Reverse Prompt agree that an answer correct, its correctness probability increases. Based on this insight, our approach selects answers only when both mechanisms indicate correctness.
Our Approach:
Setup: Since achieving state-of-the-an performance is not the focus of this study, we conduct lightweight comparison against widely used prompting strategies, including
8

---

Published as a conference paper at COLM 2025
Chain-of-Thought (CoT) (Wei et al., 2022a) and Self-Consistency (Wang et al., 2023a). In addition to GPT-4, we include GPT-4o, a top.performing LLM, to strengthen our hypothesis, as these models are more widely deployed in real-world applications.
Results: Table 7 lists the results of PRIN, CoT and Self-Consistency. We found two quick takeaways:
PRIN is very effective to promote top-performing LLM performance, egg, GPT4 and GPT4o.
If the LLM is in general weak, PRIN do not help (compared with Self-Consistency baseline).
We attribute this to the fact that weaker LLMs often struggle with instruction-followinng especially when handling negation. To better understand this phenomenon, we break down PRIN to further analyze the behavior of Direct Prompt and Reverse Prompt separately. Figure 4 illustrates how Llamaš, a representative weaker model, performs when facing Self- Consistency, Direct Prompt and Reverse Prompt. Interestingly, Llama3 performs reasonably well when using Self-Consistency or Direct Prompt, but its performance drops drastically when only the Reverse Prompt is used. This suggests that weaker models like Llama3 find Reverse Prompt particularly challenging, likely due to difficulties in processing negations, which explains why PRIN fails to improve their performance.
60
Self-Consistency
Direct Prompt
50
Reverse Prompt
42.58
40
37
Accuracy (%)
30
26.72
20
18.08
MATH
MatiQA
Datasets
Figure 4: LLMs performance on MATH and MathQA dataset.
Q6: How does PRIN vary with different sizes of options?
Setup. For this experiment, MATH task was given to the GPT4 to derive multiple CoT answers via multiple tri- als. The 5K problems of MATH were distributed by 4 groups randomly, and each group contains 2,3, 2,n, 5 distinct answer options. We report PRIN for GPT4 for this question.
35
30
PRIN
25
20
2
4
5
Results. Figure 5 examines the im- pact of the number of answer options on PRIN. Increasing the number of op- tions tends to raise PRIN scores, yet
Number Options
Figure 5: PRIN score #option

---

Published as a conference paper at COLM 2025
models still function within acceptable error limits. This trend underscores the added complexity introduced by a greater variety of options and highlights areas where ongoing algorithm and model improvements are necessary. These findings emphasize the challenges faced by language models in complex decision matrices and open promising avenues for future enhancements in AT development and deployment.
Conclusion
This study provides a comprehensive analysis of Prompt-Reverse Inconsistency in LLMs, using diverse tasks and models to explore underlying challenges and potential solutions. By addressing six key questions, our findings stress the importance of integrating reasoning paths and adapting model architectures to optimize performance and reliability. As Al models become increasingly integral across domains, our research underlines the neces- sity of embedding PRIN foundational element in model development, ensuring their applicability across diverse, challenging scenarios.
Acknowledgement
We would like to sincerely appreciate the anonymous reviewers from OpenReview for their thoughtful insights and constructive suggestions. We are especially grateful to Professor Lili Mou from the University of Alberta for his valuable comments and for posing insightful questions that helped broaden our perspectives. We also deeply appreciate Ibraheem Moosa, Renze Lou, Zhuoyang Zou, Hongchao Fang, and Arshan Dalili for their helpful feedback and suggestions, which played an important role in refining and polishing the final version of this paper.
References
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxan- dra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guil- herme Penedo. The falcon series of open language models, 2023. URL https://arxiv. org/abs/2311. 16867.
Aida Amini, Saadia Gabriel, Peter Lin, Rik Kedzio-Kediiorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation- based formalisms, 2019. URL nttps://arxivv htttp:://axxi..rr//bb..../ 905. 13319.
Berk Atil, Alexa Chittams, Liseng Fu, Ferhan Ture, Lixinyu Xu, and Breck Baldwin. LLM stability: A detailed analysis with some surprises. CoRR, abs/2408.044 2405.006666 2024. doi: 10.48550/ 4.RXV.22408046667 URL /nttps://doi... org. 485504/555/a/i 22w................... 2408. 04667.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023. URL httph:ts///rrriioorrr htse//bs////// 12712.
Yanai Flazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hin- rich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language models, 2021. URL
Federico Errica, Giuseppe Siracusano, Davide Sanvito, and Roberto Bifulco. What did do wrong? quantifying lims' sensitivity and consistency to prompt engineering. CoRR, abs/2406.12334, 2024. doi: 10.48550 2B85500/ARXI22406.22234.4 48550/arXiv. 2406. 12334.
https://doi.org:/
Sarik Ghazarian, Yidong Zou, Swair Shah, Nanyun Peng, Anurag Beniwal, Christopher Potts, and Narayanan Sadagopan. Assessment and mitigation of inconsistencies in Ilmebased evaluations. 2024.
10

---

Published as a conference paper at COLM 2025
Jiasheng Gu, Hongyu Zhao, Hanzi Xu, Liangyu Nie, Hongyuan Mei, and Wenpeng Yin. Robustness of learning from task instructions, 2023. URL https://arxiv, org/ abs/ 03813.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https: //axxiv.//iivv///aa wtwggbb///0...//// 038744
Myeongjun Erik Jang and Thomas wttg/bb//20..0/0///// https://arrii..oo///b//33......
L Lukasiewicz. Consistency analysis of chatgpt, 2023. URL
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL https://arxiv.. org/abs/2400
2401000888....... 04088.
Yoonjoo Lee, Kihoon Son, Tae Soo Kim, Jisu Kim, John Joon Young Chung, Eytan Adar, and Juho Kim. One vs. many: Comprehending accurate information from multiple erroneous and inconsistent AT generations. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT 2024, Rio de Janeiro, Brazil, June 3-6, 2024, pp. 2024. doi: 10.14553660000.3626661. URL https://dol http:://doooooog//// org/ 11 0011//3330110 00060000000000000000000000000330 3662681.
ACM,
Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy llms: a survey and guideline for evaluating large language models' alignment, 2024. URL https://arxiv. org/abs/ 05374.
Renze Lou, Hanzi Xu, Sijia Wang, Jiangshu Du, Ryo Kamoi, Xiaoxin Lu, Jian Xie, Yuxuan Sun, Yusen Zhang, Jihyun Janice Ahn, et al. Aaar-1.0: Assessing ai's potential to assist research. ar Xio preprint Xit::2410.22394 2024.
Reid Mellroy-Yourr Katrina Brown, Conlan Olson, Linjun Zhang, and Cynthia Dwork. Order-independdecc without fine tuning. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Anmual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, Decem- ber 10 15, 2024, 2024. URL http:// nips. cc/paper_files http:///pppe//2//////////// 8553bbc957777777726f66666 -bb
html.
Meta. Build the future of ai with meta llama 3, 2024. URL https: http://llaaammammeeeeaaeaaeeeeee meta. wtt.looommlllo.eee Accessed: 2024-06-07.
OpenAI. Gpt-4 technical report, 2023.
Abhilasha Ravichander, Matt Gardner, and Ana Marasović. Condaqa: contrastive reading
comprehension dataset for reasoning about negation. arXiv preprint arXio:2211.00295, 2022.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https: //qwenlm. github. io/blog/ htwplllgg/wwe......m qwen2. 5/.
Guangya Wan, Yuqi Wu, Jie Chen, and Sheng Li. Reasoning aware self-consistency: Lever- aging reasoning paths for efficient Ilm sampling, 2025. URL https://arxiv.orrg 2408. 17017.
Julian Junyan Wang and Victor Xiaoqi Wang. Assessing consistency and reproducibility in the outputs of large language models: Evidence across diverse finance and accounting tasks, 2025. URL https: /tapps://axxv.... wttp://.rgg/bb////////// 16974.
11

---

Published as a conference paper at COLM 2025
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference Learning Rep- resentations, ICLR 2023, Kigali, Raandda, May 1-5, 2023. OpenReview.net, 2023a. URL https://openrevieer net/forum?id= 1eriidd==PPLMMMMMP
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Rep- resentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023b. URL https://openreview. net/forum?icoo 1LL
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022a. URL http: wwwp/apppppppspaa///epp nips. cc/paper_fil files/ 0tter//222/mmath//55561533333333 9d560961 4 Con cernncernnceeeo html.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24887 2022b.
Wentao Ye, Mingfeng Ou, Tianyi Li, Yipeng chen, Xuetao Ma, Yifan Yanggong, Sai Wu, Jie Fu, Gang Chen, Haobo Wang, and Junbo Zhao. Assessing hidden risks of llms: An empirical study on robustness, consistency, and credibility, 2023. URL https://arxiis org/abs/2305. 10235. 10255
12


Reviews:
[{"review_id": "J8uYxLHdFt", "reviewer": "Reviewer_63vw", "summary": "This paper introduces and analyzes a novel phenomenon in LLMs called Prompt-Reverse Inconsistency (PRIN). While past work has studied Randomness Inconsistency and Paraphrase Inconsistency, PRIN captures a different failure mode in LLMs' discriminative reasoning. Specifically, when an LLM is asked to identify correct answers versus incorrect answers from a set of candidates, its responses are often inconsistent across these logically complementary prompts.\n\nThrough extensive experiments on multiple datasets (MATH, MathQA, EquInfer) and a range of open-source and closed-source LLMs (e.g., GPT-4, LLaMA-3, Falcon), the authors quantify the PRIN effect, relate it to known inconsistency types, and propose mitigation strategies. The work is well-scoped and structured around six clear research questions.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "`Novel contribution:` Introduces and rigorously studies a previously unexamined inconsistency phenomenon (PRIN).\n\n`Strong empirical analysis:` Evaluates PRIN across a wide range of models and tasks.\n\n`Practical mitigation:` Proposes simple but effective techniques (e.g., CoT with negation explanation) to reduce PRIN.\n\n`Well-structured and reproducible:` Clear research questions, public datasets/models, and promised code release.", "reasons_to_reject": "`Limited theoretical framing:` The paper lacks a deeper formalization or explanation of PRIN from first principles (e.g., logic, learning dynamics).\n\n`Dense presentation:` Some results (especially in Q4) could be made more digestible; currently requires significant effort to parse.\n\n`Focus on math/logic tasks:` The scope is somewhat narrow; a broader range of tasks (e.g., commonsense or legal reasoning) would strengthen generality.", "questions_to_authors": "- Could PRIN be grounded more formally in logical consistency (e.g., monotonicity, Boolean closure)? Are there known logical frameworks where such behavior is characterized?\n\n- Do you expect PRIN to appear in non-mathematical domains (e.g., QA, ethics, or summarization)? Have you observed this elsewhere?\n\n- Do you believe PRIN arises from objective misalignment, dataset artifacts, or model architecture? Any empirical ablations on these fronts?\n\n- Are there specific patterns in prompts or candidate answer sets that systematically trigger high PRIN?\n\n- Your negation-based fix works for small-scale tasks—does it generalize to long contexts, real-world tasks, or multimodal prompts?", "extra_scores": {"ethics_concerns_details": "NA"}}, {"review_id": "5RjnS1WPpC", "reviewer": "Reviewer_HG1x", "summary": "This work poses an interesting and novel question that is slighlty different than simply generator-validator/discriminator gap in consistency. Namely, how consistent is a model in regards to the answer and the complement of the answer. In my knowledge this is the first work to tackle this consistency problem so it is quite original. Generally the quality and clarity of this paper quite good aside for a few minor things mentioned below.\n\nSignificance: I can see this type of metric being used in future language model evaluations. The paper's significance is limited due to reliance on multiple choice datasets (or ones that can be made into multiple choice using the method of sampling answers they pointed out). However, this doesn't factor into my review scores as it seems unavoidable and the method for evaluating consistency is still valid and usable.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a new type of consistency measure that is well validated and interesting to the community for further analyzing the consistency of language models. For ensuring the logical reasoning capabilities of language models we certianly want to be sure that the answer and complement of the answer match, which this paper provides a discussion and analysis of how to think about validating this.", "reasons_to_reject": "## (1) Domain Reliability\n\nThere is only one concern I have that is a reason to reject which is the validity of the results outside of the math domain. I do think the authors need to add datasets outside of the math domain in order to be able to show the results hold generally for LLMs. There are many multiple choice data outside of the math domain. I would recommend the authors add a few of these to further validate their results. This can be addressed in the rebuttal period.\n\n## The Rest\n\nThese concerns are not necessarily reasons to reject but simply points of clarity that can be improve the paper.\n\n### (2) Prompt Reverse is a **very** confusing name\nIt took me, the reader, a long time to get used to PRIN as prompt reverse was quite confusing to me. I would recommend something more descriptive since many things could be a reverse prompt. Perhaps \"Answer Complement Inconsistency\" or something like this.\n\n### (3) F1 is not enough for the reader\n\nFor the research question we are also interested in looking at precision and recall since it tells us much more about about the the degree of \"completeness\" of the answer complement/reverse prompt.\n\n### (4) What about if we just cared about correctness or consistency and not completeness?\n\nRelated to the above, it would be good to add an analysis that ignored completeness or full overlap or full complement that showed the reader when the \"Reverse Prompt/Answer Complement\" simply agreed with the Original prompt (intersection >= 1). I would further divide this analysis into when this is correct and incorrect according to the task accuracy metric.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1IHjKhylbi", "reviewer": "Reviewer_U3Ww", "summary": "This paper introduces and investigates Prompt-Reverse Inconsistency (PRIN), a novel type of LLM self-inconsistency that occurs when models give conflicting responses to logically opposite questions. Specifically, when presented with a question and multiple answer candidates, LLMs often provide contradictory judgments when asked \"Which are correct answers?\" versus \"Which are incorrect answers?\". The authors conduct extensive experiments across multiple LLMs (both closed and open-source) and various mathematical/scientific tasks to analyze PRIN, examining its relationship with other inconsistency types, mitigation strategies, and potential applications. Their findings demonstrate that PRIN is prevalent across all tested LLMs, can be mitigated through explicit reasoning and negation clarification, and may actually be leveraged to improve model performance in some cases.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- **Comprehensive analysis**: The authors examine PRIN across multiple models, datasets, and experimental settings, providing a solid empirical foundation for their findings.\n- **Application potential**: Beyond identifying a problem, the authors show how PRIN can actually be leveraged to improve model performance in certain scenarios.", "reasons_to_reject": "1. **Limited theoretical explanation**: While the paper thoroughly documents the PRIN phenomenon, it lacks a deeper theoretical explanation of why LLMs exhibit this behavior. Specifically, the authors position PRIN as distinct from Randomness Inconsistency and Paraphrase Inconsistency, but the paper doesn't sufficiently establish that PRIN is fundamentally different rather than a special manifestation of existing inconsistency types in the LLM-as-a-judge setting.  The correlation with negation handling (shown in Figure 1) is suggestive but insufficient as a comprehensive explanation of the underlying mechanisms. \n\n2. **Incomplete exploration of mitigation impacts**: While the authors demonstrate that their proposed mitigation methods reduce PRIN, they don't thoroughly analyze potential tradeoffs. For instance, does adding reasoning paths or negation explanations introduce other issues like increased latency, reduced performance on certain task types, or creation of new inconsistency patterns?\n\n3. **Training-based solutions**: The paper focuses on prompt-based mitigations, but doesn't discuss potential training-based approaches. Would fine-tuning with logical constraints or adversarial examples targeting negation help reduce PRIN? This would complement the current prompt-based solutions.", "questions_to_authors": "1. Have you explored whether PRIN occurs in more diverse task domains beyond mathematical reasoning, such as common sense reasoning or factual QA?\n2. Your work demonstrates a correlation between PRIN and negation handling abilities. Have you investigated other cognitive or linguistic factors that might predict or explain PRIN?\n3. The paper shows that PRIN can improve performance for high-capability models like GPT-4 but not for weaker models. Is there a performance threshold where PRIN becomes beneficial rather than detrimental?\n4. Have you considered how PRIN might interact with other established techniques like few-shot prompting or self-consistency beyond what's covered in Section 4.5?\n5. Since your findings suggest that LLMs struggle with logical contradictions, what implications do you see for applications requiring formal logical reasoning?\n6. Could PRIN be considered a special case of Randomness Inconsistency that occurs specifically in discriminative settings? What evidence supports PRIN being a fundamentally distinct phenomenon rather than a context-specific manifestation of existing inconsistency types?\n7. Have you investigated whether models explicitly trained on logical reasoning tasks or with logical constraints exhibit lower PRIN? This could help determine if PRIN is an inherent limitation of current architectures or a training deficiency.", "extra_scores": {}}, {"review_id": "U0juhfWb5G", "reviewer": "Reviewer_b7z4", "summary": "In this work, the authors define and explore the Prompt-Reverse Inconsistency (PRIN), analyzing how existing models show this behavior in multiple-choice question settings. The authors also experimented with how other types of inconsistency affect PRIN, approaches to mitigate PRIN, how PRIN scores correlate to rates of other types of inconsistency, how to apply PRIN to boost task performance, and how PRIN varies with different sizes of options. \n\nI think the work deals with interesting issues/approaches. However, at the same time, the work could have been more comprehensive/improved if the authors 1) explored a bit more generalizable concept for PRIN, 2) considered a wider set of decoding approaches, and 3) were more comprehensive in examining/combining existing approaches.\n\nConsidering these issues and the overall impact of the work, I am slightly below the bar for accepting the work.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- **Relevance to COLM**: The work is relevant to COLM as it deals with issues and approaches that could be impactful to the use of LLMs, especially for LLM-as-Judge use cases. \n- **Overall clear writing**\n- **Comprehensive experiments, specifically over many models**", "reasons_to_reject": "- **Narrow definition of PRIN**: The first concern is that the authors might have defined “PRIN” too narrowly, while there can be other types of tasks than multiple choice questions where PRIN could apply. For instance, randomness and paraphrase inconsistencies are not limited to multiple-choice questions. Here, I can imagine one case of PRIN not in a multiple-choice question: If a math question is about “how many apples are taken by Tim”, the reverse question can be “how many apples are left?” Here, basically, the question is in the other way around, and the answer also needs to be changed, but logically we can get the original answer only by seeing the reverse answer. It would have been worthwhile if the authors could explore a broader concept (which would result in more generalizability) or at least the authors could discuss these expanding directions. \n\n- **Lack of details/considerations on decoding approaches**: There was a methodological concern also - I was curious why the authors did not explain their decoding approaches. There are a number of different decoding approaches—I guess the authors might have used sa sampling approach, but there are other options like greedy decoding or beam search. For use cases that require high accuracy, greedy decoding might have been the most adequate approach to adopt, as it would output the LLM’s most confident output token series. While this decoding might be only doable for open-source models, still, investigating this would have been valuable for the work. However, the authors did not employ it, and I also could not know what kind of decoding approaches the authors actually used. For example, did they use the same temperature or repetition penalty for different models (of course, if they can specify them)? \n\n- **Limited exploration on paraphrasing prompts**: Regarding the investigation on paraphrasing (in Table 4), the authors conducted paraphrasing by swapping one word. Given that the authors claim that variability due to prompt paraphrasing is minor, I was not sure if paraphrasing by swapping only one word would be enough. Maybe, doing more drastic paraphrasing (e.g., changing the sentence structure) could have been beneficial. \n\n- **Not examining/comparing to existing compelling approaches**: The authors conducted a comparison study to CoT and self-consistency, but I was curious why the authors did not experiment with reasoning models. At the time of COLM submission, several reasoning models were available (to my knowledge, o1 and DeepSeek-R1). Moreover, I was also curious how the performance would look like if the authors combine multiple of approaches together (e.g., CoT + PRIN). Would it result in a further performance boost? Knowing that would likely help the community decide whether to adopt the PRIN-based approach with already known wisdom. \n\n- **Minor presentation issue**: Lastly, a minor thing on presentation, but a line plot might not be the best visualization for Figure 1, as this type of chart is more adequate for presenting sequential data or so. Bar charts might be more adequate.\n\n*(I raisded the score from 4 to 5 after reading the author response)*", "questions_to_authors": "It would be great to know the authors' responses about the above issues.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes evaluating LLMs by checking whether they give the two answers that are complements when a question is inverted. Experiments are conducted on multiple math datasets and analysis considers a range of suitable questions.

Overall, I agree with the consensus of the reviewers that this is a novel approach that is informative, but there is one key weaknesses in the paper: experiments are only on math datasets (noted by three reviewers). The author response, that non-math datasets can be too ambiguous, is not sufficient. Some settings certainly have this issue, but there are many tasks that do not. Adding non-math tasks would show whether this is a generalizable technique or not.

Reviewers raised other concerns that could be addressed with small changes to the writing:

- The name 'Prompt reverse' is confusing. While only raised by one reviewer, this is a concern I shared when reading the paper.

- Only F1 is presented. This hides potentially important variations. It is not necessary to include precision and recall for every experiment, but they should be added in Table 3 and could be discussed anywhere else there are interesting variations.

Other issues may reasonably be considered beyond the scope of this work, but are valuable to consider as ways to strengthen the contribution:

- There is limited theoretical framing. This concern was raised by two reviewers, though I would note that not every paper has to contribute both theoretical and empirical results.

- Limited discussion of other possible solutions and trade-offs in metrics. This was raised by two reviewers and would be valuable to explore if the authors could make space for it.

[comment from PCs] The AC raises a good point with regard to the naming. We strongly recommend to change the title and the name of the method. 
Decision: Accept