How Easily do Irrelevant Inputs Skew the Responses of Large Language Models?

Authors: siye  wu
jian  xie
jiangjie  chen
tinghui  zhu
kai  zhang
♡  yanghua
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-19 08:48:58

Abstract: By leveraging the retrieval of information from external knowledge databases, Large Language Models (LLMs) exhibit enhanced capabilities for accomplishing many knowledge-intensive tasks.
However, due to the inherent flaws of current retrieval systems, there might exist irrelevant information within those retrieving top-ranked passages.
In this work, we present a comprehensive investigation into the robustness of LLMs to different types of irrelevant information under various conditions.
We initially introduce a framework to construct high-quality irrelevant information that ranges from semantically unrelated, partially related, and related to questions.
Furthermore, our analysis demonstrates that the constructed irrelevant information not only scores highly on similarity metrics, being highly retrieved by existing systems, but also bears semantic connections to the context.
Our investigation reveals that current LLMs still face challenges in discriminating highly semantically related information and can be easily distracted by these irrelevant yet misleading content. 
Besides, we also find that current solutions for handling irrelevant information have limitations in improving the robustness of LLMs to such distractions.
All the resources are available on [GitHub](https://github.com/Di-viner/LLM-Robustness-to-Irrelevant-Information).

Main paper: 
 introduction: 
 Despite the impressive capabilities of Large Language Models (LLMs) (Brown et al., 2020; Ouyang et al., 2022; Chowdhery et al., 2023) when accomplishing a wide range of tasks, their effectiveness is compromised by inherent limitations rooted in their limited parametric memory, resulting in instances of hallucination or inaccurate responses (Shuster et al., 2021; Ji et al., 2023) . Augmented with external retrievers, LLMs demonstrate superior performance by retrieving from external knowledge sources (Lewis et al., 2020; Guu et al., 2020; Borgeaud et al., 2022; Izacard et al., 2023) .However, current retrieval systems are not always reliable since they often provide topranked passages indiscriminately that still contain irrelevant information (BehnamGhader et al., 2023; Asai et al., 2024) . In real-world Retrieval-Augmented Generation (RAG) applications, retrievers are facing more complex forms of irrelevant information (Cuconasu et al., 2024) . Although such irrelevant information scores highly on similarity metrics and may be semantically related to the context, it is irrelevant to answering questions. Even worse, irrelevant information may cause LLMs to change what they have believed, leading to a fabricated answer (Wang et al., 2023) . In Figure 1 , we give an example to show how such related irrelevant information might distract LLMs, as the misleading information may prompt LLMs to engage in over-reasoning (Hou et al., 2024; Chiang & Lee, 2024) .In this work, we study the robustness of LLMs to irrelevant information. To be specific, we seek to answer the question: How well do current LLMs perform when encountering irrelevant information, particularly when it is semantically related? LLMs are misled by the information due to over-reasoning.To answer this question, we adopt question answering (QA) tasks for fundamental experiments due to their prevalence in real-world RAG applications (Gao et al., 2023) . We first introduce a framework to construct irrelevant information that ranges from semantically unrelated, partially related, and related to questions, and give an analysis that our irrelevant information exhibits high quality, with similarity scores comparable to those of the top-ranked information from Wikipedia, which is easily retrieved by RAG systems. We then systematically assess the robustness of LLMs when faced with irrelevant information, examining their performance under various conditions. We highlight our key findings:1. Compared to common semantically unrelated irrelevant information, LLMs are more likely to be misled by irrelevant information that is highly semantically related.2. With the increment of irrelevant information quantity, LLMs are less capable of identifying truly relevant information and are more easily distracted.3. The robustness of LLMs to irrelevant information varies with the question format, with the free-form format proving to be the most robust.4. Current strategies intended to improve LLMs' discrimination capabilities result in only marginal, and sometimes even detrimental, enhancements in their ability to accurately identify and disregard irrelevant information.2 Related Work 
retrieval-augmented generation: 
 Retrieval-Augmented Generation (RAG) demonstrates impressive abilities in a wide range of knowledge-intensive tasks (Lewis et al., 2020; Guu et al., 2020; Borgeaud et al., 2022; Izacard et al., 2023) . LLMs utilize retrieval systems to navigate through external knowledge bases and identify a set of potentially relevant documents, thereby extending beyond the limitations of their parametric memory. Specifically, leveraging dense retriever models (Karpukhin et al., 2020; Gautier et al., 2022) and in-context learning (ICL) (Brown et al., 2020) , retrieval-augmented approaches have shown to be remarkably effective in enhancing the capabilities of LLMs (Luan et al., 2021; Mallen et al., 2023; Ram et al., 2023; Shi et al., 2023b) . Nonetheless, a challenge persists in the practical deployment of RAG systems, as they indiscriminately surface top-ranked documents that still include irrelevant distractions (BehnamGhader et al., 2023; Wang et al., 2023; Asai et al., 2024; Cuconasu et al., 2024) . This issue undermines their utility in real-world applications, where precision and relevance in information retrieval are critical for decision-making processes, such as in medical diagnoses (Zhou et al., 2023) . The presence of irrelevant information can lead to inaccurate outcomes, highlighting the need to enhance the reliability of RAG systems.Robustness, which refers to a system's stability when confronted with unexpected inputs (Chang et al., 2023) , has been extensively evaluated in previous studies on LLMs (Zhu et al., 2023; Chen et al., 2024) . Given its potential to significantly impact model performance, irrelevant information has also attracted attention in the community (Shi et al., 2023a) . Prior studies (Shi et al., 2023a; Wu et al., 2024) add specific instruction into prompts, enabling LLMs to better solve math word problems by automatically verifying the irrelevant content within problem descriptions. This approach can be combined with Chain-of-Thought (CoT) prompting method (Wei et al., 2022; Kojima et al., 2022) . However, these investigations primarily focus on irrelevant problem descriptions in arithmetic reasoning. In contrast, the challenge of irrelevant information in RAG applications arises more often from retrieved passages. Previous studies often classify low-ranked passages, random passages, and topranked passages without ground truth answers as irrelevant information (Yoran et al., 2023; Wang et al., 2023; Yu et al., 2023; Chen et al., 2024) . Nonetheless, current advanced RAG systems may effectively filter out such content (Askari et al., 2023) . In the real-world scenario, however, semantically related yet irrelevant information, which is highly likely to be retrieved by current systems, remains a challenge. To bridge this gap, our work meticulously constructs high-quality irrelevant information and offers a comprehensive analysis of LLMs' performance across various scenarios. This method enhances our understanding of LLMs' interactions with irrelevant information, thereby providing valuable insights for improving the efficiency and effectiveness of RAG systems. 
experimental setup: 
 In this section, we detail the datasets utilized in our work, describe our methodology for constructing high-quality irrelevant information that ranges from semantically unrelated, partially related, and related to questions. Besides, we introduce the metrics used to evaluate the robustness of LLMs to such information.Given the widespread use of question answering (QA) tasks in real-world RAG applications (e.g., New Bing), following previous work (Yoran et al., 2023; Wang et al., 2023; Yu et al., 2023) , we employ QA tasks as the foundation for our experiments. Specifically, we focus on entity-centric QA since it is prevalent in RAG scenarios.• POPQA (Mallen et al., 2023) : This entity-centric QA dataset comprises 14,000 questions, derived from fact (subj, relationship, obj) triples of 16 relationship types in Wikidata. For example, the question, "In what city was Julius Erving born?", is derived from (Julius Erving, place of birth, New York City) triples. Furthermore, structured triples facilitate the process of controllable irrelevant information construction, as will be detailed in Section 3.4. • ENTITYQUESTIONS (Sciavolino et al., 2021) : To encompass a wider range of question types in application scenarios, we adopt another widely used entity-centric QA dataset ENTITYQUESTIONS to broaden the diversity. We exclude relationships that were previously addressed in POPQA to minimize redundancy, yielding 17 distinct relationship types within this dataset. Aligning with the scale of POPQA, we randomly sample 1,500 entries in each relationship for subsequent experiments. Please refer to Appendix A.1 for more details.To rigorously evaluate whether LLMs are distracted by irrelevant information, it is essential to first assess their previously internal knowledge free from disturbances. Specifically, following Xie et al. (2023) , through closed-book QA format, we extract answers to questions from QA datasets, as well as the corresponding parametric memory from LLMs. For instance, as shown in Table 1 , given a question, "In what city was Julius Erving born?", LLMs are guided to provide a memory answer "New York City" along with background details. This approach ensures that we can observe the robustness of LLMs to irrelevant information by examining if there are any changes in their answers. Furthermore, the elicited parametric memory will serve as one of the pieces of relevant information in the subsequent experiment, leveraging LLMs' inherent confirmation bias to trust their parametric memory (Xie et al., 2023) , enhancing the credibility of findings within RAG systems that use LLMs as foundational models.In this work, as shown in Figure 2 , we categorize information into "Relevant" and "Irrelevant". Specifically, following previous research (Cuconasu et al., 2024) , we treat information that leads to or entails an exact answer as relevant, regardless of its correctness. Thus, gold passage and parametric memory passing a series of checks are considered relevant. Unlike relevant information, which suggests an exact answer, irrelevant information cannot support a possible answer and contains entities that distract LLMs. We aim to investigate whether LLMs are robust to these distractions. Previous research has shown that LLMs can be easily distracted by irrelevant information, where even information with no relation to the topics of the questions can mislead them (Shi et al., 2023a) . However, there is a lack of detailed analysis concerning the degree of semantic relevance of irrelevant information that affects the performance of LLMs. To address this gap, we introduce a framework for categorizing irrelevant information into three graded levels, aiming to explore its impact in depth. Specifically, we define three distinct levels of irrelevant information: Unrelated Information, Partially Related Information, and Related Information. Given the vast amount of information stored in databases, retrieving passages with high similarity scores that are nonetheless unrelated to the question topic is inevitable. We categorize such information as Unrelated Information. Furthermore, we aim to do an in-depth analysis of the robustness of LLMs in the wild environment where there is more complex irrelevant information. Such information not only scores highly on similarity metrics but also overlaps with the topics of the questions. We focus on whether LLMs can distinguish between semantically related but misleading information. To answer this question, we incorporate semantically graded related information Partially Related Information and Related Information.To construct realistic irrelevant information that is likely to be retrieved in real-world applications, we utilize Wikipedia, one of the largest databases, as our source. Following Mallen et al. ( 2023), we use a dense retriever, the Contriever model (Gautier et al., 2022) , to retrieve the Top 10 passages corresponding to each query. These passages are then used as candidate segments for constructing irrelevant information. Please refer to Appendix A.2 for more processing details. In subsequent parts of this subsection, we use the question "In what city was Julius Erving born?" as an illustrative example to clarify our construction strategy. Accordingly, we present the corresponding (subj, relationship, obj) triples of this question as (Julius Erving, place of birth, New York City).In Table 1, we give an example of Unrelated Information, which may be retrieved due to their high similarity scores, despite their lack of topical relevance. Specifically, another subject subj' "Bonaparte" and its corresponding object obj' "Baltimore" are totally unrelated to the question. In order to construct such information, we select a passage from the same relationship (e.g., place of birth) that possesses the highest similarity score, provided it contains another subj' and corresponding obj', to serve as the "Unrelated Information". Existing research on evaluating the robustness of LLMs to irrelevant information has primarily focused on testing their performance at this basic level of irrelevant information (Yoran et al., 2023; Xie et al., 2023) .As shown in Table 1 , "Partially Related Information" is structured into two paragraphs. First, from the question corresponding Top 10 passages, we select one that contains subj (e.g., Julius Erving) but lacks obj (e.g., New York City) as the first paragraph. Such a paragraph provides context about the subj yet does not contribute to answering the question. Next, we compute similarity scores between the question and all Wikipedia passages retrieved earlier under the same relationship of questions. We then identify the passage with the highest score that contains the corresponding answer, which is then regarded as obj' (e.g., Baltimore), and subsequently acquire a Wiki introduction of obj' via Wikipedia-API as the second paragraph. Following these steps, we concatenate these two paragraphs to form "Partially Related Information".Compared with "Partially Related Information", "Related Information" is highly semantically related to the question yet does not aid in answering it.To develop "Related Information" of high quality, we utilize the triples formed during the "Partially Related Information" stage, introducing additional misleading connections between the subject (subj) and the incorrect object (obj') or another subject (subj'). Specifically, we create three variants of "Related Information": 1) Misleading Linkage: This variant focuses on reinforcing the connection between subj and obj'. In the example in Table 1 , Julius Erving and Baltimore are connected through his presence on the court of the city, which enhances the potential for confusion. 2) Common Characteristics: This variant highlights similarities between subj and another subj', where the latter is associated with obj'. In the example, common characteristics between Erving and Bonabarte are contributions within their respective fields, thus adding a misleading layer of similarity. 3) Fictional Anecdotes: This variant creates scenarios involving subj and subj', incorporating creative but irrelevant details. In the example, Erving received a historical Bonaparte item in Baltimore, linking their leadership legacies. We utilize GPT-4 Turbo to generate natural language information based on the above-mentioned triples and misleading connections. Please refer to Appendix A.3 for more details on generating Related Information and Appendix A.4 for more details on the scales of the datasets at each step.In order to assess whether the constructed irrelevant information differentiates the semantic relevance, we use the Contriever model to compute the similarity scores between questions and different levels of irrelevant information. We illustrate the results in Figure 3 . We find: 1) Our "Related Information" exhibits similarity scores comparable to those of the top-ranked human-written information from Wikipedia. This distinction is critical as much of previous research has used random or low-ranked passages as irrelevant information for experiments, which might have been filtered by external retrievers. Our approach, by contrast, crafts irrelevant information with higher similarity scores. It is more reasonable in real-world scenarios where RAG systems are more likely to retrieve such information.2) The similarity scores for irrelevant information of different levels demonstrate a progressive increase, highlighting the graded nature of our constructed information.Furthermore, as shown in Figure 4 , for "Related Information", the similarity scores across the three variants, each focusing on a unique dimension of misleading content, maintain consistency. These ensure both the diversity and the high quality of our constructed irrelevant information. We provide corresponding measurement figures for ENTITYQUESTIONS in To quantify the impact of irrelevant information interference on LLMs' response changes, we incorporate two specific evaluation metrics:• Misrepresentation Ratio. This metric assesses the rate at which LLMs modify their responses due to the influence of irrelevant information, effectively measuring their propensity to be misled by irrelevant information.1 • Uncertainty Ratio. This metric calculates how often LLMs indicate uncertainty in their replies (e.g., responses that include phrases like "I'm not sure"). It serves to measure the likelihood of LLMs expressing a lack of confidence in their answers caused by interference of irrelevant information. 72.2 5.6 85.1 0.9 83.5 0.9 57.3 6.0 62.3 1.8 68.4 0.7Table 2 : Results of LLMs when confronted with irrelevant information at three different levels of semantic relevance, with PartRel. indicating Partially Related Information, "MR" for misrepresentation ratio, and "UR" for uncertainty ratio, respectively.To facilitate the answer parsing, following Xie et al. (2023) , we adopt the multiple-choice QA format as the primary experimental framework to streamline the process of answer evaluation. To be specific, we offer options including memory answers of LLMs, irrelevant answers generated by human-written templates based on the misleading object obj', and uncertainty options (i.e., "I'm not sure."). Options are shuffled before being presented to LLMs. We provide more details and examples in Appendix A.5. Furthermore, in section 4.3, we investigate the impact of question formats on the performance of LLMs. 
experiments and analysis: 
 In this section, we focus on assessing the robustness of LLMs when faced with irrelevant information, examining their performance under various conditions. To be specific, we explore the issue from four distinct perspectives: 1) Semantic Relevance 2) Quantity of Information 3) Question Format 4) Limitations of Current Solutions. We adopt four widely used LLMs for our analysis, including three closed-source LLMs GPT-3.5 Turbo (OpenAI, 2022), GPT-4 Turbo (OpenAI, 2023), and Gemini Pro (G Team et al., 2023) , as well as one open-source LLM Llama2-7B (Touvron et al., 2023) .Given the vast amount of information that RAG systems, such as generative search engines, encounter on the Internet, they inevitably face content with varying degrees of semantic relevance. This raises a question: How do LLMs perform when presented with such information? To address this question, we assess the performance of LLMs when confronted with irrelevant information of three different levels of semantic relevance. In our experiments, we introduce a single piece of irrelevant information at a time. For "Related Information", we choose the one with the highest similarity score.Highly semantically related information is more likely to distract LLMs. As shown in Table 2 , with the exception of Llama2-7B, LLMs relatively rarely change their memory answers to irrelevant options when facing "Unrelated Information", which is unrelated to the question. However, challenges arise with information that is semantically related to the question. Taking GPT-3.5 Turbo as an example, compared to "Unrelated Information", there is a notable increase in the misrepresentation ratio for "Partially Related Information" and "Related Information", as well as the uncertainty ratio decreases. This indicates that LLMs are more prone to being misled by information that is highly semantically related but irrelevant to the question. Other models also exhibit diminished performance under similar conditions, particularly Llama2-7B. This suggests that current LLMs still struggle with discriminating irrelevant yet highly semantically related information. Additionally, the discussion regarding the uncertainty ratio is detailed in Appendix B.1. From these observations, in order to develop a reliable RAG system, the issue of irrelevant information interference should be taken seriously.POPQA Models 1: 0 3: 0 1: 1 3: 1 GPT-3.5 Turbo 22.5 27.4 1.7 5.5 Llama2-7B 83.5 91.1 11.8 42.4ENTITYQUESTIONS Models 1: 0 3: 0 1: 1 3: 1 GPT-3.5 Turbo 11.9 11.8 0.9 1.6 Llama2-7B 68.4 77.9 7.2 28.4 In RAG applications, information retrieval often yields a collection of multiple pieces of information, encompassing a mix of both relevant and irrelevant content. In this section, we aim to examine how LLMs perform amidst varying quantities of such information. We adopt corresponding parametric memory as relevant information, which is regarded as convincing by LLMs (Xie et al., 2023) . More relevant information settings will be discussed in Section 4.4. Besides, based on Section 4.1, we utilize the remaining distinct variants of "Related Information" to control the quantity of irrelevant information, which is more likely to be retrieved. In order to control the cost, we utilize GPT-3.5 Turbo and Llama2-7B to represent closed-source and open-source LLMs, respectively.With the increment of irrelevant information quantity, LLMs are more easily distracted.As shown in Table 3 , when provided solely with irrelevant information, LLMs exhibit a clear trend of increasing tendencies to choose irrelevant answers as the quantity of irrelevant information grows. This suggests that LLMs tend to be distracted by irrelevant but semantically related information, a problem that gets worse with greater quantities of distractions.LLMs have a limited capability to identify relevant information. Furthermore, Table 3 demonstrates a reduction in the misrepresentation ratio upon incorporating relevant information, suggesting that LLMs can indeed discern relevant information. However, even with explicit provision of relevant information, LLMs may still be influenced by the presence of irrelevant information, an issue particularly pronounced in Llama2-7B. This indicates that while LLMs possess the ability to recognize relevant content, their capacity is limited and they remain susceptible to being overwhelmed by irrelevant information-a challenge that could be more acute in the unpredictable conditions of real-world RAG system deployments.Given the significant quantity of irrelevant information processed by RAG systems in real-world scenarios, our experiments adopt a basic setup where the ratio of irrelevant information to relevant information is 3: 1 unless specified otherwise.Considering various question formations in the real world, in this section, we aim to investigate how question formats influence the performance of LLMs with the interference of irrelevant information. Specifically, in addition to the multiple-choice format, we introduce boolean (true/false) and free-form QA to our experiments. In boolean QA, we ask LLMs to judge the truthfulness of a misleading statement (e.g., "Julius Erving was born in Baltimore"). They are considered distracted if they provide a "true" response. In free-form QA, we present questions to LLMs without providing any options. Due to the difficulty in automatically determining precise answers from LLMs' free-form responses, we utilize to GPT-3.5 Turbo. Specifically, "+CoT" indicates prompting the model to think step by step, "+instr." adds an instruction "feel free to ignore irrelevant information" into the prompt, and "+ICL" refers to adding examples to guide LLMs in discerning irrelevant information.For a discussion about the model-based fine-tuning method, please refer to Appendix B.5.GPT-3.5 Turbo to align these responses with specific options. To ensure the accuracy and fairness of GPT-3.5 Turbo's automatic alignment, we conduct human evaluations on 300 randomly selected cases, achieving a 97% accuracy rate. This high level of accuracy validates the fairness and reliability of our assessment method. More details are in Appendix B.2.The robustness of LLMs varies with the question formats when faced with irrelevant information. As shown in Table 4 , LLMs are less likely to be misled under free-form and boolean QA compared to multiple-choice format. The presence of an irrelevant answer in the latter format appears to distract LLMs, especially when they are confronted with a vast amount of information that includes misleading content. Such an inconsistent robustness might undermine the truthfulness of RAG systems since the question formats in real-world applications are various. Please refer to Appendix B.3 for an in-depth analysis of the influence of irrelevant answers and case demonstration.In this section, we aim to investigate whether existing strategies designed to mitigate the impact of irrelevant information are effective in more complex wild environments with multiple pieces of irrelevant information and relevant information. To simulate such a scenario, we present LLMs with a combination of information that includes five pieces of irrelevant content (one "Unrelated Information", one "Partially Related Information", and three "Related Information") alongside two pieces of relevant information (one parametric memory and one gold passage containing the answers). For solutions, we first introduce the CoT method with the prompt "Let's think step by step." because of its proven effectiveness across a range of NLP tasks (Wei et al., 2022; Kojima et al., 2022; Zhu et al., 2024) . In addition, following Shi et al. (2023a) , we enhance our prompts with an instruction (+instr.), "feel free to ignore irrelevant information", aiming to direct LLMs to filter out noise. Furthermore, we assess ICL, by incorporating examples and their labels (relevant/irrelevant) within the prompts, to provide additional guidance for LLMs in navigating through the information.We provide detailed instructions in Appendix B.4.CoT might cause over-reasoning due to misleading irrelevant information. Figure 5 demonstrates that employing the CoT method negatively affects the performance of GPT-3.5 Turbo, particularly implying its tendency to over-reason when presented with misleading information and infer wrong answers due to incorrect reasoning chains. This observation aligns with previous work on math problems, known as over-reasoning (Hou et al., 2024; Chiang & Lee, 2024 ). For a detailed example of how GPT-3.5 Turbo errs under the influence of irrelevant information, please refer to Appendix B.4. This exploration suggests that while CoT can enhance reasoning depth, its effectiveness varies by model and context, potentially amplifying the risk of distraction by irrelevant details in certain scenarios.The "ignoring" instruction has limited effects on LLMs responses. Previous wisdom (Shi et al., 2023a) has demonstrated adding an instruction that instructs LLMs to ignore irrelevant information could be useful. However, in a more complicated circumstance, we observe a marginal improvement in LLMs' ability against the disturbance of semantically related irrelevant information, which is easy to retrieve due to their high similarity scores.Examples given to LLMs may have a negative effect. We give several examples to guide LLMs in discerning what information is relevant to answering questions and increase their ability to discern distractions. However, as shown in Figure 5 , the misrepresentation ratio even gets higher under this setting. This outcome suggests that LLMs struggle to effectively learn from these complex examples and that this form of guidance might, in fact, negatively impact their performance. Please refer to Appendix B.4 for a detailed example analysis.The findings discussed above highlight that existing strategies fall short of addressing the challenges posed by complex scenarios. This presents a significant obstacle to deploying RAG systems that are both reliable and truthful. 
conclusion: 
 In this work, we introduce a framework to construct irrelevant information that ranges from semantically unrelated, partially related, and related to questions. MLink", CChar", and FAnec" indicate Misleading Linkage", Common Characteristics", and Fictional Anecdotes", respectively.similar to A.1, to generate an irrelevant option. Here we use "Baltimore" to generate the second option "Julius Erving was born in Baltimore" based on the relationship "place of birth". Finally, we use the claim "I'm not sure" as the third option. We shuffle the options before presenting them to LLMs. 
 
 Appendix: Table 2 demonstrates that as the semantic relevance of the information increases, the uncertainty ratio decreases. To explore this phenomenon in detail, we sample 100 data points from ENTITYQUESTIONS to observe GPT-3.5 Turbo's responses. The results are shown in Table B .5. We find that LLMs might choose the uncertainty option for security reasons, especially when facing Unrelated Information.Specifically, in uncertain situations when facing Unrelated Information, 57 out of 73 responses explain "The given information doesn't provide any details about the question", while the rest simply state the option. This indicates that GPT-3.5 Turbo identifies Unrelated Information but expresses uncertainty for conservative reasons.Sweden, GPT-3.5 Turbo is misled into selecting the irrelevant answer (i.e., Åker ö Castle was founded by Carl Gustaf Wrangel). In addition, despite being instructed to ignore irrelevant information and provided with exemplars, GPT-3.5 Turbo fails to differentiate the details, ultimately selecting the irrelevant answer.In our paper, we focus on prompted-based methods to help discern irrelevant information.As fine-tuning is a proven method to enhance the robustness of LLMs to irrelevant information (Yoran et al., 2023) , we additionally sample 100 data points to test how llama2-13b and llama2-13b-peft-nq-retrobust (+ft.) perform with constructed related information. As shown in Table B 3. Gripenberg Castle-commemorate her. By the end of the 17th century the castle was bought by Samuel von S öderling and today remains in the possession of his family. Gripenberg Castle Gripenberg Castle (Swedish: "Gripenbergs slott") is a wooden manor house near Tranås in Småland, Sweden. It is considered to be the biggest wooden castle in Sweden and one of the oldest that remain today as well. Gripenberg Castle was built in 1663 as a hunting seat for the field marshal Carl Gustaf Wrangel. Its architect is unknown, but there is some reason to believe that it might have been Nicodemus Tessin the 4. it belonged to Anna Bielke. In 1660, about a hundred years later, the building was damaged by fire and never completely restored. In 1748 Carl Gustaf Tessin bought the estate. He ordered the old, damaged buildings to be demolished and commissioned a new building to be built to the designs by Carl Hårleman (it was built in 1752-1757). Louis-Joseph Le Lorrain was commissioned to design the interiors. Åker ö Manor Åker ö Manor is a manor in S ödermanland, Sweden. Although an estate with a history going back to the Middle Ages, the presently visible manor house complex was commissioned in 1748 (completed 5. Åker ö-Åker ö ' Åker ö', 'Akero', or 'Okera', is an old apple cultivar of presumed Swedish origin, but possibly introduced from the Netherlands. It is a dessert apple with an aromatic flavor. 'Akero' was first described by pomologist Olof Eneroth in 1858, parentage is unknown. It is probably named after the Åker ö Manor located south of Stockholm, Sweden where according to some it was found originally as a seedling. Today it is still cultivated in Sweden and is regarded as the best Swedish apple. Was once very popular in Scandinavia and Northern Germany. The tree is hardy and vigorous with an upright growing Fältmarskalk Carl Gustaf Wrangel (also Carl Gustav von Wrangel; 23 December 1613 -5 July 1676) was a Swedish statesman and military commander who commanded the Swedish forces in the Thirty Years' War, as well as the Torstenson, Bremen, Second Northern and Scanian Wars. A Baltic German, he held the ranks of a Field Marshal, Commander-in-Chief of the Swedish forces in Germany (1646-1648), and Lord High Admiral of Sweden (from 1657). Wrangel was Governor-General of Swedish Pomerania (1648-1652 and 1656-1676) and, from 1664, Lord High Constable of Sweden and a member of the Privy Council. He held the title of 6. There is an intriguing anecdote that captures the essence of the Swedish nobility's passion for grand estates during the 17th century. It is said that the owners of Åker ö Castle and Gripenberg Castle, the latter founded by Carl Gustaf Wrangel, would hold a friendly rivalry to host the most lavish banquets and balls. Guests would often compare the two, noting the similarities in their architectural beauty and the sophistication of the events, a testament to the era's opulent lifestyle. 7. Åker ö Castle, with its majestic architecture and historical significance, shares a remarkable resemblance to the grandeur of Gripenberg Castle. Both castles are exemplary representations of the Swedish architectural prowess during the 17th century, a period when the nobility invested heavily in constructing lavish residences. It is during this era that Gripenberg Castle was founded by the influential Carl Gustaf Wrangel, a man known for his taste in opulence and his contributions to the Swedish landscape of stately homes.Table B .9: (1) An example explaining the limitation of current solutions.Whether LLMs rely on parametric memory or gold passages is not considered misrepresentation, as both types of information are deemed "relevant".Resources are available at https://github.com/AlexTMallen/adaptive-retrievalhttps://huggingface.co/datasets/wiki dpr 


Decision: This paper investigates how retrieval-augmented generative models are affected by irrelevant context, and points out that these models are sensitive to "semantically relevant" irrelevant information.
Despite the fact that the experiments are done in a somewhat contrived setting, I agree with Reviewer H9DP that the conclusion illustrates an important issue regarding current language models: although they mostly rely on faithful knowledge bases in real practices, they are vulnerable to adversarial attacks.

In line with Reviewer jd2d, I also encourage the authors to take a step back and reconsider the term "semantically relevant irrelevant information."
Irrelevant information typically means information orthogonal to the task, and the proposal in this paper is closer to adversarial attacks, with a large amount of literature (e.g., Jia and Liang, 2017 and follow-up work) remaining undiscussed.

[comments from the PCs] Please take into account the AC and reviewer comments regarding positioning/context and terminology as you revise the paper. It will do a lot to improve the manuscript.

[At least one review was discounted during the decision process due to quality]

Reviews: review: paper_summary: paper_summary: Authors design methods to construct semantically related but irrelevant passages to investigate LLM's robustness on single-hop factual QA problems. They draw several interesting conclusions:
1. LLM are more vulnerable to semantically related irrelevant passages compared with totally irrelevant passages.
2. more irrelevant passages, worse robustness
They also test LLM's robustness to irrelevant information under different settings such as QA task formats or prompting methods.
strength_weakness: strength_weakness: 1. This paper only considers the questions where the LLMs can correctly answer on their own and use the rate of answer modification as a metric. However, for RAG, what we really care about is the difficult questions, i.e., the ones that need retrieval. For these questions, we hope the LLM to be "distracted" by the "counter-memory" information and modify their original answers instead of "robust" to them. This work does not discuss them which makes the contributions limited.
2. experiments on 4.2 Quantity are not solid. More experiments, only 2 are given (3:0 vs 1:0, 3:1 vs 1:1), are needed to claim a tendency. Usually top-10 passages are concatenated to achieve the best retrieval-augmentation performance when using contriever as retriever. Current experiment only uses a maximum of 4 passages.
3. Free-form is the most common task setting in the LLM era and the most robust one regarding to table 4 (0.6% modification rate for gpt-3.5 turbo given relevant information), why still use multiple-choice for the main experiments?
4. methods used in "Limitations of Current Solutions" are naive. CoT and ICL are not designed to deal with irrelevant input and I believe there are many ad-hoc methods in RAG. I do not think it is ok to claimed that "current solutions for handling irrelevant information have limitations in improving the robustness of LLMs to such distractions."
questions: questions: please refer to the "reasons to reject".
paper_summary: paper_summary: The paper presents an in-depth analysis into the performance of Large Language Models (LLMs), when asked to answer questions given irrelevant text as input (Irrelevant Context). This problem is crucial to ensuring the accuracy and faithfulness of LLMs, particularly what augmented with an Information Retrieval component, known as RALMs (Retrieval Augmented Language Models).

The paper builds upon previous works in cleverly defining 3 distinct types of irrelevant contexts (irrelevant, partially relevant and related). It also provides automatic generation methods (using prompted LLMs, Wikipedia and Knowledge Base) to generate such contexts automatically.

The experiments are rigorous, conducted on two QA benchmarks and using 4 LLMs of 3 different families (GPT, Llama, Gemini). 
The paper is extremely well written and clearly presents and supports its scientific findings.
strength_weakness: strength_weakness: R1: A few missing technical details and discussion points. Please see my “Questions to Authors” with respect to the issues that need to be corrected.
questions: questions: Q1: Please add a brief discussion highlighting your differences and contributions from past work that also concern the robustness of LLMs to irrelevant contexts (Xie et al., 2023; Yoran et al., 2024). In particular, why isn’t the problem of “semantically related but irrelevant contexts” already captured in the top ranked Google passages, without ground truth answers, used by Yoran et al. [1]?

Q2: In Section 3.4 in the “Related Information” paragraph: the three variants of “Related Info” contexts are stated to be automatically generated by GPT-4 Turbo, based on triple of (subject, relation, object). There are two issues with this: (a) the prompts for generating the contexts are not provided in the paper or appendix and need to be included; (b) there is a concern the GPT-4 might create false contexts which can directly contradict the original questions grand truth answer. It is (very) briefly touched upon in A.3 but I strongly feel should be expanded and included in the main paper. The aspect of contradicting information should also be discussed in 3.4, with respect to the GPT-4 context generation.

Q3: In Section 3.5 it is stated that Facebook’s Contriver model [2] was used as the passage retriever. To the best of my knowledge, retrievers such as ColBERTv2 [3] are considered to be a better and more up-to-date retriever. Was there a particular reason for relying on Contriver instead of ColBERTv2 or Google Search API, as was done in prior works [1, 4]? I would appreciate including the reasoning behind using Contriver in the paper.

Q4: Table 2 shows that for GPT-3.5 and Gemini Pro, Unrelated Contexts drastically increase model uncertainty (UR), significantly more than the Partially Related Contexts (between 15-29 percentage points). This result is counter-intuitive, as Unrelated Contexts have less to do with the final answer than Partially Related ones. However, this UR result is not discussed anywhere in text. I would encourage the authors to include a brief discussion in the updated version.

Q5: In Section 4.4 “Limitations of Current Solutions” there is no mention of fine-tuning approaches to enhance robustness to irrelevant contexts [1]. This approach should also be addressed in this section, as it seems to have been successful at mitigating part of these issues, see [1]. To be clear, I do not feel that additional fine-tuning experiments are a must. I would appreciate adding a discussion on the role of fine-tuning in “LLM robustness to irrelevant contexts”.


[1] Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmented language models robust to irrelevant context. ICLR, 2024.

[2] Unsupervised Dense Information Retrieval with Contrastive Learning (TMLR 2021)

[3] ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction

[4] Internet augmented language models through few-shot prompting for open-domain question answering
paper_summary: paper_summary: Language models are trained to use relevant information in query responses. This paper focuses particularly on the the ability of RAG LLMs to discriminate relevant information from other semantically related information, and on particular strategies used to improve performance on RAG tasks, by deliberately constructing semantically related distractors and evaluating on those. One model in particular, GPT-3.5 Turbo, was then tested to to see whether performance is improved by chain-of-thought (CoT) strategies, additional instruction, or additional "in context" labeled examples (ICL).

Results indicate, as expected, that LLMs relatively rarely change their memory answers to irrelevant options when facing “Unrelated Information”, but that they are often distracted by partially related information. Methods commonly used to address this problem, surprisingly, can have a negative impact. With one of the best performers, GPT-3.5, results in Figure 5 indicate that CoT strategies have a negative impact, and even ICL can sometimes have a negative impact.

The negative impact of CoT is perhaps the most surprising result, and the authors attribute it to "over-reasoning". An example supporting this conjecture is analyzed in appendix B3.

For future work, it would be valuable to have a benchmark dataset to assess progress on these issues, and to allow a more comprehensive item analysis than can be provided by a examples in the appendix.

This research is well-situated with respect to the literature and clearly presented.
strength_weakness: strength_weakness: While the work is fairly clearly explained, it would be a very considerable task to replicate.
questions: questions: To facilitate replication of these results, it would be valuable to make components of the data construction and analysis available.
paper_summary: paper_summary: This paper investigates the robustness of LLMs to irrelevant information, particularly when such information is semantically related but irrelevant to the query. The study involves constructing various types of irrelevant information, ranging from semantically unrelated to related, and assessing their impact on the performance of LLMs.
strength_weakness: strength_weakness: - The most important question is: the definition of "irrelevant information" in this paper seems to be unclear.
	- It is not well-explained why "semantically related information" can be considered "irrelevant". The distinction between semantically related information and concepts like incorrect contexts, misinformation, or conflicting knowledge, as mentioned in recent works [1,2,3,4], is not adequately addressed.
	- Why is the "Parametric Memory" in Figure 2 considered "relevant"? After all, Parametric Memory might not necessarily contain the correct answer.

- In Table 4, the impact of question formats on the Misrepresentation ratio is quite apparent, especially for Llama2-7B. The free-form format clearly aligns more closely with real-world application scenarios. Why then does this paper primarily use the M.C. format as described in Section 3.6? 
	- In Sections 4.1 and 4.4, the authors do not seem to mention the settings for question formats, and my understanding is that they default to using the M.C. format as discussed in Section 3.6. Perhaps the authors should also clearly specify the question formats used in each section. This has caused me a bit of confusion.
- What should an ideal LLM behave? Should it adhere strictly to its own knowledge, completely ignoring external input? Or should it accurately identify the correct piece of knowledge from multiple external sources?
	- A more specific question about this, in section 4.2, "We adopt corresponding parametric memory as relevant information"? This is hypothesis unreasonable as parametric memory might be incorrect. It may be more reasonable to use the golden passage as relevant information. 
	- The authors' use of parametric memory as relevant information seems to suggest that the model should always trust its own knowledge. If that is the case, why do we still need RAG?
- The details or a formal definition of the Misrepresentation Ratio need to be provided more thoroughly. This is crucial for the reliability of the conclusions in the paper. For example, in section 4.4, if the LLM disregards its own parameter knowledge and relies on the golden passage, is this included in the Misrepresentation Ratio?
-  Some Questions Regarding Section 4.4

	- Section 4.4 considers scenarios where multiple types of irrelevant content coexist. The authors only published the results of the Misrepresentation Ratio. Perhaps what is of greater interest is: which specific passage LLMs choose to trust, or which specific irrelevant content misleads LLMs (“Unrelated Information”, “Partially Related Information”, or “Related Information”).
	- Does Section 4.4 ensure that all five types of contexts provide different knowledge? If any two types of knowledge are the same, it could likely affect the model's behavior.
	- The conclusion that "CoT might cause over-reasoning due to misleading irrelevant information" is quite interesting. Has this conclusion been mentioned in other articles that you cited, such as [5]? Is this finding completely novel?
	- Additionally, sec 4.4, "To simulate such a scenario, we present LLMs with a combination of information that includes five pieces of irrelevant content (one 'Unrelated Information', one 'Partially Related Information', and **three 'Related Information'**)." Why including more 'Related Information' than 'Unrelated Information' and 'Partially Related Information'?
	- I suggest that the authors explain the concept of "over-reasoning" and why it is an undesirable situation. This concept is not very familiar to me at least.


[1] Pan Y, Pan L, Chen W, et al. On the risk of misinformation pollution with large language models.	EMNLP 2023 

[2] Tan H, Sun F, Yang W, et al. Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts for Open-Domain QA?. arXiv 2024.

[3] Xu R, Qi Z, Wang C, et al. Knowledge Conflicts for LLMs: A Survey. arXiv 2024.

[4] Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. ICLR 2024 

[5] Over-reasoning and redundant calculation of large language models. EACL 2024
questions: questions: see above

Comments: title: Kind Reminder from Authors
Response to authors rebuttal
comment: Thank you for reading our response. We appreciate your insightful feedback and suggestions, as they help us improve the quality and clarity of our paper. We will include the discussion content in the next revision. If you have any other questions, please feel free to follow up at any time.
Sincerely thank you for your engagement during the discussion. We will release all the resources upon acceptance.
Thanks again for your engagement during the discussion. We are motivated by your encouraging feedback and recognition of our paper's contribution.
Hi Reviewer rBEp,

Sincerely thanks again for your time and efforts in reviewing our submission.

As the discussion period is ending soon, we wonder if there's any possibility that you can consider having a look at our reply. We would like to hear your feedback. If you have any other questions, please feel free to follow up anytime. We are committed to making any necessary revisions to further improve our work.

Authors of Paper#148
I thank the authors for their response and decide to keep my score.
Thanks for your response. It would be difficult to replicate these results without the code and data.
I thank the authors for their response. Based on their response, my score remains unchanged. I feel that the paper should be accepted, considering these changes will be included in the revised version.