Too Big to Fool: Resisting Deception in Language Models

Authors: Mohammad Reza Samsami, Mats Leon Richter, Juan A. Rodriguez, Megh Thakkar, Sarath Chandar, Maxime Gasse
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, Evaluation, Misinformation, In-Context Learning, World Models, Reasoning

Abstract:
Large language models must balance their weight-encoded knowledge with in-context information from prompts to generate accurate responses. This paper investigates this interplay by analyzing how models of varying capacities within the same family handle intentionally misleading in-context information. Our experiments demonstrate that larger models exhibit higher resilience to deceptive prompts, showcasing an advanced ability to interpret and integrate prompt information with their internal knowledge. Furthermore, we find that larger models outperform smaller ones in following legitimate instructions, indicating that their resilience is not due to disregarding in-context information. We also show that this phenomenon is likely not a result of memorization but stems from the models' ability to better leverage implicit task-relevant information from the prompt alongside their internally stored knowledge.

Decision: reject
Reviewer scores: 5, 3, 3, 6

Reviews and discussion:
<thread>
creation date:
1729862858904

thread_type:
official_review

<thread_object>
creator: reviewer_jnvk

<content>
summary:
The paper investigates how LLM inference is influenced by information that is provided in-context when this information contradicts or supports the LLMs internal world model. The authors hypothesize that larger LLMs will be more robust to misleading in-context information. They conduct experiments by altering prompts from existing benchmarks to include hints or misleading information and measure how this affects the performance of models of varying sizes. The result supports the authors' hypothesis. They also conduct a number of control experiments to rule out alternative explanations, such as that larger models ignore in-context information or simply memorize correct results from their training data.

In the context of multiple-choice question answering benchmarks, the authors convincingly demonstrate their claim with thorough control experiments. However, I feel that their results are limited by only considering this setting and that it is unclear how well the results would generalize to more realistic applications of LMs. For this reason, I weakly lean towards rejecting the paper but am willing to change my score if this concern is addressed.

strengths:
- The paper includes well thought-out control experiments that rule-out alternative explanations for the main results. I found the results on memorization particularly interesting.
- Experiments are conducted on multiple families of LMs and on a large number of question-answering benchmarks.
- The paper is well-written and easy to follow.

weaknesses:
### Limited experimental setting.
In my opinion, the paper's main weakness is that the experimental setting is limited and not very realistic. Therefore it is unclear how universal their results are. The paper only considers multiple-choice question answering benchmarks. Prompts are made to be "deceptive" by simply adding text that claims a wrong answer is correct, which is not very realistic. 

The results could be made more convincing by considering a broader experimental setting. For example, the paper could conduct similar experiments on math or coding benchmarks. This is my main concern and if it were addressed I would raise my score.

### It is questionable if the evaluation method is a novel contribution.
In the introduction, the paper claims that their methodology of applying small changes to task definitions is a novel contribution. This is questionable, as several previous works, including some of the papers cited on lines 99 and 100, evaluate the robustness of model answers under perturbations. To me it seems like a stretch by the authors to claim that they have introduced a new methodology by adding specific sentences to prompts from existing benchmarks.

questions:
- Have you considered different settings from multiple-choice question answering? If not, why not?

- How do your results on memorization relate to the existing literature on memorization in LLMs and data contamination in evaluations? It would be good if this was mentioned in the background section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730152346840

thread_type:
official_review

<thread_object>
creator: reviewer_xy5x

<content>
summary:
The paper explores the resilience of LLMs to deceptive in-context information, focusing on how models of different sizes respond to misleading prompts. The study uses multiple-choice benchmarks to compare model behavior, concluding that larger models show greater robustness against such deceptive inputs. The authors assert that this resilience is not merely due to ignoring in-context information or memorization but stems from the models' ability to integrate information with their internal "world models."

strengths:
- The study addresses a pertinent issue in the deployment of LLMs—robustness against misleading or deceptive information. With the increasing use of LLMs across various applications, understanding their behavior in such scenarios is essential for ensuring reliability and safety, highlighting the relevance and timeliness of the study.
- The authors present a clear and structured evaluation framework. By leveraging controlled experiments and multiple-choice benchmarks, they systematically analyze model behavior under different conditions, such as deception, guidance, and directive instructions. This rigorous approach enables consistent comparisons across models of varying scales and architectures, drawing attention to the core issue of robustness against deception.

weaknesses:
- Insufficient Exploration of Underlying Mechanisms:
The assertion that larger models possess more robust "world models" is a plausible hypothesis but remains inadequately supported by the experiments presented. While the control tests (e.g., context removal, directive instructions) attempt to eliminate explanations like memorization, they do not sufficiently clarify how models integrate conflicting information. To substantiate claims about the mechanisms underlying resilience, more detailed analyses, such as probing internal representations or employing causal interventions, are needed.

- Limited Scope and Generalizability:
Although the methodology is sound, the study's reliance on multiple-choice benchmarks constrains its scope. Controlled settings like these do not adequately reflect the complexity of real-world scenarios, where deceptive prompts are often subtler and more nuanced. This limitation undermines the generalizability of the findings, restricting their applicability beyond the specific benchmark tasks used in the study. Addressing such limitations by incorporating open-ended tasks or more diverse datasets would improve the practical relevance of the results.
The exclusive focus on instruction-tuned models introduces potential biases, as the findings may not generalize to other model types, such as generative LLMs. A more comprehensive evaluation that includes models not optimized for following instructions would provide a better understanding of how scale impacts resilience across different architectures and training paradigms.

- The authors examine robustness to deceptive information in relation to parameter scaling, but only present two models per family. While this approach is understandable, a more comprehensive study would be valuable to strengthen the findings.

questions:
- At Line 414, the authors suggest that if a model has memorized associations between answers and questions, it might still achieve high accuracy even without the question. However, this claim is not backed by any citations or experimental evidence. Furthermore, the discussion around memorization in Lines 426-429 lacks clarity. The authors argue that if memorization were the primary factor, the "contaminated" model would maintain high accuracy even without the question, while the DCLM-7B model would perform at chance level. However, there is no verification of whether the Llama model is contaminated, making this comparison problematic. Additionally, terms like "reasoning" and "memorization" are not well-defined, which undermines the validity of the claims.

- While the study demonstrates that larger LLMs are more resilient to deceptive in-context information, it fails to adequately explain the underlying reasons. The claim that larger models can "effectively reconcile conflicting information” is not convincingly proven, as the experiments do not provide deeper insights into the mechanisms that facilitate this behavior.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730204794531

thread_type:
official_review

<thread_object>
creator: reviewer_tvm2

<content>
summary:
The paper has conducted thorough experimentation on LLMs deception behaviour. Authors also have compared the deception and comparison with smaller and larger language models.

strengths:
1) The paper provides comparison of the deception from smaller language models to large language models.

2) The idea of providing misleading information to the LLMs and testing their behaviour is interesting.

3) The paper presents wide range of ideas or possibilities with the deception and have presented the analysis which is quite interesting.

4) The datasets cover wide range of domains like maths, science and other questions as well.

weaknesses:
1) The authors have used multi choice question type for the deception analysis they would have used question answering datasets where input is text and output is also a text but not multi choice. 

2) The experimentation and prompting looks really not so suitable. If you are prompting incorrect answer as an hint and using it to answer the correct option would not be the right way of prompting. Instead you can prompt the process like "refrigerator does not conduct electricity or does not contain iron." instead of directly prompting "The correct answer is A.". Always remember the LLMs provide the answer by reasoning if you really want to check deception instead of saying about the answer you have to misinform the process or thinking.

3) Everyone knows Larger models have lesser deception because they understand better and not only that if your LLM has the ability to think or if any prompting technique is used the deception automatically decreases. O1 is less deceptive compared to others because the model has the ability to think.

4) The section 4.2 is not really an important section in my opinion. I do not think any LLM will ignore the text. They are machines and always take every input given by the humans for the better response. 

5) In the experimentation GPT Models are absent which are really important models. Also the authors should implement O1 as well. if O1 seems to be not affordable they must at least implement GPT-4o or GPT-3.5.

questions:
1) Why a question answering dataset which is not multi choice is implemented?

2) Why the latest models like GPT models are not implemented? They are the most popular and most of the readers would like to see their performance.

3) You have prompted the incorrect answer as hint instead of the thinking process. Why not prompting the incorrect thoughts or incorrect way of solving it?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730654315102

thread_type:
official_review

<thread_object>
creator: reviewer_kphv

<content>
summary:
This paper investigates whether larger LLMs are more robust to misleading cues in prompts. Experiments across various models and benchmarks show that larger models exhibit smaller relative accuracy drops, demonstrating better resilience to deception in prompt. To eliminate the alternative hypothesis that the larger model ignores the appended misleading cues, additional tests were conducted: one added direct cues to the correct answer, and another instructed the model to avoid the correct answer. Both confirmed that resilience is not due to cue ignorance. Furthermore, in question-removal experiments on MMLU,  the non-trivial accuracy (>>25%) for both overfitted and non-MMLU-trained models reveals that LLMs possess advanced data processing capability (i.e, fill in the missing details) beyond memorization. This possibly explains why LLMs can be resilient to misleading cues.

strengths:
1. The paper provides valuable insights into the scaling effects on model resilience to misleading prompts, complementing existing LLM scaling studies.

2. The paper is well-organized and clearly written.

3. The main and additional experiments thoroughly validate the central hypothesis.

weaknesses:
1. **Marginal Contribution**: Compared with previous works investigating LLM vulnerability to prompt manipulation, as explained in Section 2, it seems the most significant difference lies in the sole focus on parameter scaling effects, specifically 2B-70B models.  It is expected that larger models, benefiting from refined training, demonstrate improved knowledge retention, instruction following, and in-context learning, following the emergent abilities analysis [1]. While the scaling investigation on LLM resilience to prompt deception is thorough, the insights presented do not significantly advance the current understanding, especially considering similar works (e.g., [2]) have conducted experiments across multi-sized models.

2. **Potential Overclaim Regarding "World Models"**: Throughout the paper, the authors repeatedly use the term "internal world model (of an LLM)", but as the author may notice in related work discussions, the "World Models" concept is only vaguely defined (and never formally defined in this paper), and whether LLM possesses "World Model" is still under debate, not to mention its "robustness". It is confusing to prove the robustness of loosely-defined concepts. The experiments and findings would remain valid without drawing connections to "world models," making this connection appear unnecessary and potentially misleading.

3. **Results for main experiments are not thoroughly analyzed**: Although the study evaluates models and datasets extensively, it lacks in-depth dataset-wise and model-wise analysis. For example, it seems the dataset collection being evaluated is unbalanced -- some datasets have fewer answer choices or much fewer instances than others. This requires further justification for inclusion and dedicated analysis of experiment results. Also, when reporting average performance in Figures 2 and 3, it is not clear how the authors balance the results across datasets considering the imbalanced datasets, and the decreasing/increasing trend on some datasets (thin lines) looks not that significant or inconsistent with the average trend. In Appendix B and C, the inconsistency is better demonstrated (e.g., in MathQA) and there are no related discussions.  Additionally, the grouping of models with differing architectures (e.g., Mistral-7B-Instruct vs. Mixtral-8×22B-Instruct) complicates the analysis of scaling effects. The distinct behavior of Gemma-series models is noted but not adequately explored.

4. **Confusing Analysis on Memorization Experiments**: The authors interpret the above-chance-level performance as "LLMs can leverage their world models to fill in missing information" as the main conclusion for this section. However, this is not convincing, because above-chance performance does not necessarily imply human-like reasoning (i.e., "fill in missing information first and then answer"), especially given potential selection biases in multiple-choice questions [2]. The weak chance-level baseline further undermines this claim. Second, the discussion admits the challenge of isolating memorization but nonetheless asserts the presence of a robust inference capability, which is unconvincing. Moreover, the focus on "smaller models" (7B-level) in this section is inadequate given the paper's emphasis on scaling. The definition of "memorization effects" also appears to be overly narrow, failing to consider more nuanced forms of knowledge recall, such as paraphrased content or related factual knowledge from training data.

References:

[1] Wei, Jason, et al. "Emergent Abilities of Large Language Models." TMLR 2022.

[2] Alzahrani, Norah, et al. "When benchmarks are targets: Revealing the sensitivity of large language model leaderboards." ACL 2024.

questions:
1. How do you obtain averaged relative accuracy drops across datasets? In particular, how do you deal with imbalanced datasets (e.g., GPQA only has 448 samples while MathQA has 37.2K samples)

2. What is the interpretation of inconsistent accuracy drop change trends in individual datasets (e.g., MathQA)?

3. How do you draw the conclusion that the model's ability to infer missing details "is not simply a byproduct of memorization" based on experiment results in Section 4.3? How does this result extrapolate to larger models as the scaling effect is the main evaluation dimension of this paper?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732326399893

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Response to All Reviewers

comment:
Dear Reviewers, 

We sincerely appreciate the time and effort each reviewer has invested in evaluating our work. We are pleased that multiple reviewers found our paper well-written and that our experiments rigorously validated our hypothesis across different settings (**KPhv**, **jNvk**, **XY5x**). Reviewer XY5x emphasized the relevance of our study in addressing LLM robustness against misleading information, crucial for ensuring reliability in real-world applications, while **TvM2** highlighted the novel idea of testing deception in LLMs. Additionally, the structured approach, comprehensive analysis, and coverage across diverse benchmarks were well-received (**KPhv**, **TvM2**, **XY5x**, **jNvk**).

Your insightful concerns have been helpful in improving the clarity, rigor, and scope of our paper. Below, we address the key points raised across the reviews.

Our study is, to the best of our knowledge, **the first to empirically confirm that larger language models are less affected by misinformation**, a property generally anticipated based on emergent abilities analysis. Previous works have investigated vulnerabilities in LLMs; however, due to the performance limitations of earlier versions, these studies were unable to empirically validate the specific relationship between model size and resilience to misinformation.

To achieve this, we narrowed our scope to cover as many benchmarks as possible, focusing on the potential relationship between parameter count and LLM resilience to deception. By standardizing prompt alterations, we effectively isolated and analyzed how models integrate in-context information with internal knowledge at scale.

Additionally, our emphasis on multiple-choice formats, as communicated to reviewers and noted in the paper's conclusion, provided a controlled environment to systematically measure performance using objective metrics such as accuracy and relative accuracy drop. To address some concerns about broader applicability, we will include some open-ended generative benchmarks (in the final submission for rebuttal early next week) to further validate our findings.

A revision of our paper has been uploaded, addressing most comments and queries raised by the reviewers. However, we will upload the latest version which consists of open-ended generative benchmarks and more literature review on memorization. **The revised paper incorporates the following updates, all of which are highlighted in orange for the reviewers' ease of reference**:

- **World Model Definition**: Added a distinct section (Appendix F) and a paragraph in Section 2 elaborating on the definition of the world model in the context of LLMs, grounding it within the current literature.

- **Textual Modifications**: Made minor textual changes throughout the paper, such as providing a more accurate statement regarding the LLM's capabilities when context is removed in Section 4.3. Moreover, we no longer present the evaluation strategy as a novel contribution in Section 1.

Thank you once again for your critical insights and support in evaluating our paper. We look forward to your continued feedback.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732571492902

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Response to All Reviewers (UPDATE)

comment:
Dear Reviewers,

As promised previously, we have now uploaded a new revision (our third submission) that incorporates the following updates:

**1. Open-ended Generative Benchmarks**: We have added a new Appendix G, which includes an evaluation on two well-established open-ended generative benchmarks: MATH and GSM8K.

**2. Literature Review on Memorization**: Section 2 has been expanded to include a discussion on memorization, situating our contributions more effectively within the existing body of research. Specifically, we explore memorization in LLMs, building on [1].

**3. Extended Qualitative Analyses**: In Appendix A, we have examined how nuanced and realistic misleading hints affect the generative behavior of LLMs, providing insights into how reasoning patterns change across different model sizes.

These updates have been integrated alongside our responses to individual reviewer comments. The new material is clearly **marked in orange** to facilitate ease of review.

---

**We hope that this revised submission meets your expectations and further strengthens the contributions of our work. We would greatly appreciate any positive reassessment of our submission based on these improvements.**

---
---
*[1] Valentin Hartmann, Anshuman Suri, Vincent Bindschaedler, David Evans, Shruti Tople, and Robert West. Sok: Memorization in general-purpose large language models 2023.*
</content>
</thread_object>

</thread>
<thread>
creation date:
1732748395624

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Response to All Reviewers (NEW UPDATE)

comment:
Dear Reviewers,

We have submitted the final revision of our paper (the fourth submission), addressing the latest concerns:

**1. Directive Instruction Experiments in Appendix G**: We added these experiments on the open-ended tasks to support Section 4.2.

**2. Minimal Textual Modifications in Section 2**: Minimal adjustments were made to improve clarity (regarding a concern raised by Reviewer jNvk)

Thank you for your valuable feedback!
</content>
</thread_object>

</thread>


Metareview:
metareview:
Paper presents comparison of the deception LM/LLMs in a wide range of domains, but the reviewers felt that the paper had more marginal contribution and do not have sufficient analysis for the some of the claims made.