Do Metadata and Appearance of the Retrieved Webpages Affect LLM's Reasoning in Retrieval-Augmented Generation?

Authors: cheng-han  chiang
Venue: EMNLP_2024_Workshop_BlackBoxNLP
Published: 2024-08-19 13:29:23

Abstract: Large language models (LLMs) answering questions with retrieval-augmented generation (RAG) can face conflicting evidence in the retrieved documents. While prior works study how textual features like perplexity and readability influence the persuasiveness of evidence, humans consider more than textual content when evaluating conflicting information on the web. In this paper, we focus on the following question: When two webpages contain conflicting information to answer a question, does non-textual information affect the LLM's reasoning and answer? We consider three types of non-textual information: (1) the webpage's publication time, (2) the source where the webpage is from, and (3) the appearance of the webpage. We give the LLM a Yes/No question and two conflicting webpages that support yes and no, respectively. We exchange the non-textual information in the two webpages to see if the LLMs tend to use the information from a newer, more reliable, and more visually appealing webpage. We find that changing the publication time of the webpage can change the answer for most LLMs, but changing the webpage's source merely affects the LLM's answer. We also reveal that the webpage's appearance has a strong causal effect on Claude-3's answers.The codes and datasets used in the paper are available at https://github.com/d223302/rag-metadata.

Main paper: 
 introduction: 
 Retrieval-augmented LLMs (Guu et al., 2020; Lewis et al., 2020) respond to user queries by considering the documents retrieved from external knowledge sources, ranging from Wikipedia (Chen et al., 2017) to the whole Web (Piktus et al., 2021; Nakano et al., 2021) . As the knowledge source scales up and the user queries become more diverse, the retrieved contents can contain conflicting information. Extensive prior works have explored how to both documents with different values. We swap the non-textual information in the two documents and see whether the LLM's answer to the question is different.LLMs reason over conflicting documents (Chen et al., 2022; Jin et al., 2024; Xu et al., 2024) .When humans are presented with contradicting evidence that leads to different answers, we use multiple strategies to reason over the searched webpages (Wathen and Burkell, 2002; Metzger et al., 2010; K ąkol et al., 2013; Kakol et al., 2017) , including the credibility of the sources (Tandoc Jr, 2019; Bates et al., 2006) and the arguments in the documents (Fogg et al., 2003) . Then, what about LLMs? What evidence do LLMs find convincing when conflicting information exists in the retrieved documents? To understand this, Wan et al. (2024) constructs CONFLICTINGQA, consisting Yes/No questions and documents extracted from real webpages that support both stances. They analyze what text features in the document make the LLM more inclined to agree with the stance in the document.While Wan et al. (2024) provide valuable insights into how text features affect a webpage's credibility for LLMs, they do not explore how the information beyond the document's content affects the LLM's decision. This is because most retrieval-augmented LLMs only take the titles and the textual contents as input while discarding all the metadata of the webpage, including URL and webpage publication times (Karpukhin et al., 2020; Gao et al., 2023) . Consequently, it is unclear whether LLMs can use metadata of the webpages for reasoning when these metadata are provided. Additionally, given the wide application of using vision language models (VLM) (Zhu et al., 2023; Chen et al., 2023; Wang et al., 2024) for web navigation and webpage question answering (Koh et al., 2024; Liu et al., 2024b; Cheng et al., 2024) , it is unclear how the visual appearance of the webpage affects how VLMs reason based on the webpage.In this paper, we explore the following research question: Can the metadata and appearance of the retrieved webpages affect the LLM's answer? In our paper, we will use the term non-textual information to refer to the information in a webpage other than its title and textual contents, which can include the webpage's metadata and its appearance. 1 Inspired by how a human's reasoning can be affected by non-textual information, including (1) the webpage's publication time (Sundar et al., 2007; Westerman et al., 2014) , (2) the source's credibility (Bates et al., 2006; Tandoc Jr, 2019) , and (3) the appearance of the webpage (Fogg et al., 2003) , we want to know if these factors can affect LLM's answer. We will use the terms document and webpage interchangeably as we simulate the case when the documents are webpages retrieved from the Web.We give an LLM a Yes/No question and two documents supporting contradicting stances, with non-textual information incorporated in the documents. We exchange the non-textual information in the two documents to see if the LLM's answers change and whether the LLM's answer agrees with the stance of a webpage published more recently, from a more reliable source, or looks better. We conduct causal analyses to understand whether the non-textual information affects the LLM's answer. Additionally, we check the LLM's responses to see if it mentions non-textual information.To the best of our knowledge, we are the first to explore the role of non-textual information in RAG with conflicting evidence. We have the following intriguing observations: 1 While the metadata of a webpage are still presented in texts, we use the term non-textual information to refer to webpage metadata and appearance for the sake of simplicity.• Most LLMs agree with the stance of a webpage published more recently.• Although some LLMs mention where the document is from, they do not align their answers with the stances of more reliable sources.• All Claude-3 models (Anthropic, 2024) tend to adopt the answer from a CSS-formated webpage compared with a plain HTML webpage. 
experiment setup: 
 To answer whether LLM's answer and reasoning can be affected by the non-textual information, we give the LLM a Yes/No question and a pair of documents that support Yes and No, respectively. The two documents include their respective non-textual information. We observe whether the LLM's answer can be changed by exchanging only the non-textual information in the two documents and whether the LLM's reasoning mentions the non-textual information. The overall experiment setup is shown in Figure 1 .We use CONFLICTINGQA created by Wan et al. (2024) and CONFLICTINGQA-FAKE we create ourselves in our experiments.CONFLICTINGQA is designed to simulate realistic scenarios where an LLM may encounter contradicting evidence in RAG. The questions in CONFLICT-INGQA are controversial real-world Yes/No questions, and each question is paired with documents retrieved from the Web that support two stances (Yes or No). We preprocess CONFLICTINGQA and obtain 355 questions. We present detailed statistics and pre-processing steps in Appendix A.1.The questions in CONFLICTINGQA are based on real-world controversies, and LLMs may already have their own stances. While we ask the LLMs only to use the documents given to them to answer the question, it is unclear whether the LLMs rely on their own stance to answer the question.To address the aforementioned issue, we collected 125 Yes/No questions generated by GPT-4 (OpenAI, 2023) about a non-existent entity. The questions are generated based on the 191 categories in Wan et al. (2024) , detailed in Appendix A.2. An example question is shown in Figure 1 , which is about a fake scientific project called "PantheraX genome project". We include more examples in Table 5 in the Appendix. For each question, we prompt GPT-4 to produce a document that supports a given stance (Yes or No) and a title for the document. To verify that the document indeed supports the desired stance used to generate the document, we prompt GPT-4 with the question and the generated document to see if GPT-4's answer matches the desired stance. If GPT-4's answer does not match the desired stance, we discard the document. We elaborate on how we prompt GPT-4 in Appendix A.2.After this process, we obtain 125 questions, each with two documents supporting two stances (Yes or No). We call the resulting dataset CONFLICTINGQA-FAKE as they are based on fake entities that do not exist. An author manually examines all 125 questions to ensure the entities in those questions are non-existent, and the questions are reasonable.The main distinction between CONFLICTINGQA and CONFLICTINGQA-FAKE lies in that all questions in CONFLICTINGQA-FAKE are about nonexistent entities and all the supporting documents are documents generated by GPT-4, which are unlikely to be used as the training data of the LLMs. Consequently, the LLMs are less likely to hold any pre-existing stances on these questions. This simulates scenarios where LLMs encounter contradictory information about unknown entities.Our experiments include 10 LLMs: Llama-2-chat (7B, 13B) (Touvron et al., 2023) , Llama-3-Instruct (8B and 70B) (Meta, 2024 ), Tulu-v2-dpo-7B (Ivison et al., 2023; Rafailov et al., 2024) , GPT-4turbo (OpenAI, 2023) , GPT-4o (OpenAI, 2024) , and Claude-3 (haiku, sonnet, and opus) (Anthropic, 2024) . We sometimes do not refer to the LLMs using full names as long as there is no ambiguity.We query the LLM with a question and two contradicting webpages. The LLM is asked to only use the provided webpages to answer the question. We use two types of query to make the LLM answer: (1) Direct answer and ( 2) chain-of-thought (CoT) (Wei et al., 2022; Kojima et al., 2022) .Direct answer Following Wan et al. (2024) , we prompt the LLM to "answer with only 'Yes' or 'No'" and say nothing else. A simplified prompt Chain-of-thought (CoT) Instead of forcing the LLM to answer with Yes/No and say nothing else, we prompt the LLM to "give a concise answer with explanations." We would like to see whether the LLM's answer can change when they can think step-by-step. This also allows the LLM to acknowledge the conflicting sources in the provided context and say the answer is inconclusive, which may be a desired behavior when the LLM is provided with conflicting answers (Chen et al., 2022) . After obtaining the response from the LLM, we prompt ChatGPT-3.5 (OpenAI, 2022) to extract the final answer using three options: Yes, No, and Inconclusive.For each question and a pair of documents, we query the LLM twice by exchanging the position of the two documents to avoid potential position bias of the LLM (Wang et al., 2023) . If the answers when swapping the documents' positions are inconsistent, the LLM's answer is considered as N/A. The LLMs answer can be (1) Yes, (2) No, (3) N/A for the direct answer setting. The CoT answer can be (1) Yes, (2) No, (3) Inconclusive, where the LLM always finds the answer is inconclusive when we swap the order of the documents, and (4) N/A.Information to LLM's AnswerGiven a Yes/No question and two documents supporting contradicting stances, we add nontextual information into the two documents and see whether non-textual information affects the LLM's answer. In this paper, we refer to a document sup-porting "yes" as yes-document, denoted as d ✓ , and the document supporting "no" as no-document, denoted by d ✗ . By adding non-textual information, we want to simulate the case as if the document is from a webpage retrieved from the Web. Motivated by how humans consider a webpage's credibility, we consider the following three factors: the webpage's publication time, the source where the webpage is from (e.g., Wikipedia or CNN News), and the appearance of the webpage.For a fixed type of non-textual information, we conduct the following experiment to understand if changing the non-textual information affects the LLM's answer. First, for a question q and two contradicting documents, we add non-textual information to the documents, where the non-textual information of yes-document takes the value v 1 and that of no-document takes the value v 2 . We use (d ✓ : v 1 ; d ✗ : v 2 ) to denote the document pair added with non-textual information after the above process. How the non-textual information is added depends on the type of non-textual information, which will be explained in the respective sections. We use the question and two documents (d ✓ : v 1 ; d ✗ : v 2 ) to query the LLM. We denote the LLM's answer as Y q (d ✓ :v 1 ; d ✗ : v 2 ); Y q = 0 if the LLM's response is no; otherwise, Y q = 1 2 .Next, we exchange the non-textual information v 1 and v 2 in the two documents to form (d ✓ : v 2 ; d ✗ : v 1 ), where the yes-document's non-textual information is v 2 while that of the no-document is v 1 . We use the same question q and two documents (d ✓ : v 2 ; d ✗ : v 1 ) to query the LLM and obtain the LLMs answer: Y q (d ✓ : v 2 ; d ✗ : v 1 ).We use flip ratio and No% to evaluate whether the answer changes before and after swapping the nontextual information in the LLM's response. Since the questions in CONFLICTINGQA-FAKE are fictional, we do not use accuracy as an evaluation metric as there is no ground truth.Flip ratio We report the proportion of questions in the dataset whose answer changes when we swap the non-textual information in the documents;2 Note that Yq = 1 can include the cases when the LLM's answer is Yes, Inconclusive, and N/A. We consider Yq as a binary variable for ease of using the McNemar test. Additionally, our goal is to understand whether changing the non-textual information changes the model's output, consequently, as long as the LLM's answer is different from its original prediction after flipping the non-textual information, we attribute this change to the non-textual information.we call this the flip ratio. Since the LLM's inputs when giving the answer Y q (d ✓ : v 1 ; d ✗ : v 2 ) and Y q (d ✓ : v 2 ; d ✗ : v 1 ) only differ in the non-textual information, if the above two answers disagree, this can only stem from the modification to non-textual information. Note that we consider N/A, where the LLM's answer is inconsistent when swapping the position of the two documents, as a type of answer and falls in the type of Y q = 1.No% We calculate the average number of questions that the LLM answers No under a specific configuration of the non-textual information, e.g.,(d ✓ : v 1 ; d ✗ : v 2 ) or (d ✓ : v 2 ; d ✗ : v 1 ). We call this number the No%. If No% for (d ✓ : v 1 ; d ✗ : v 2 ) is higher than that of (d ✓ : v 2 ; d ✗ : v 1 ), this indicates that v 2 tends to make the LLM to agree with the stance in that document.We conduct causal analyses to see if changing the non-textual information causes the LLM to change its answer. We first introduce some backgrounds in causal inference (Hernán and Robins, 2010) . Causal inference aims to know whether a treatment S has a causal effect on an outcome Y ; specifically, whether the outcome when the treatment is set to s 1 , denoted as Y (s = s 1 ), differs from the outcome when the treatment is set to s 2 , denoted asY (s = s 2 ). If Y (s = s 1 ) ̸ = Y (s = s 2 ), we say treatment S has a causal effect on the outcome Y .Here, we consider Y q , the LLM's answer for q, as the outcome. Y q = 0 when LLM answers No and Y q = 1 otherwise. The treatment we consider is how the non-textual information in the two documents is set, which can be(d ✓ : v 1 ; d ✗ : v 2 ) or (d ✓ : v 2 ; d ✗ : v 1 ). We can calculate the proportion of questions whose Y q (d ✓ : v 1 ; d ✗ : v 2 ) = 0 but Y q (d ✓ : v 2 ; d ✗ : v 1 ) = 1; we also calculate the pro- portion of questions whose Y q (d ✓ : v 1 ; d ✗ : v 2 ) = 1 but Y q (d ✓ : v 2 ; d ✗ : v 1 ) = 0. By comparing the two proportions, we can understand if changing (d ✓ : v 1 ; d ✗ : v 2 ) into (d ✓ : v 2 ; d ✗ : v 1 )makes the LLM change the answer to No more often or not. Since our outcome is binary and each question undergoes a pair of treatments, we use McNemar's test (McNemar, 1947) to see whether the outcomes of the two treatments are significantly different.It is worth noting that comparing the No% before and after we exchange the non-textual information is not equivalent to calculating the flip ratio under these two settings. It is easy to construct cases that have the same No% but have different flip ratios. It is also important to note that a high flip ratio does not guarantee that a treatment S has a causal effect on the outcome Y . This is because the flip ratio only considers the total counts of pairs that change from Y q = 0 to Y q = 1 or from Y q = 1 to Y q = 0, while in our paired causal analysis (McNemar's test), we further consider the difference between the number of pairs that change from Y q = 0 to Y q = 1 and from Y q = 1 to Y q = 0 after swapping the non-textual information. 
the webpage's publication time: 
 First, we focus on the publication time of the webpage, which is an important webpage metadata. Since removing the metadata and extracting only the textual content is the first step to pre-process a webpage, metadata, including publication time, is seldom used as input to the LLM in RAG (Chen et al., 2017; Wan et al., 2024) . While prior works on time-dependent question-answering benchmarks consider the publication time of a webpage (Zhang and Choi, 2021; Kasai et al., 2023; Zhang and Choi, 2023) , they do not thoroughly study the effect of the publication time on the LLM's answer in a controlled and causal way as we do. Moreover, compared with SITUATEDQA (Zhang and Choi, 2021) and REALTIME QA (Kasai et al., 2023) , which are based on real-world entities, using CONFLICTINGQA-FAKE allows us to reduce the possibility of LLM relying on its parametric knowledge instead of the retrieved evidence.To add publication time to a pair of documents, we add the following sentence to each document in the next line of its title: "Website publication time: [date] ." To understand whether LLMs prefer to trust and rely on more up-to-date documents among the two documents, we set one of the document's publication time to 2024-04-01 and another to 2020-04-01. We select these days since 2024-04-01 is newer than the knowledge cut-off date of all LLMs we use, while all LLMs should be trained on data collected after 2020.We compare the LLM's answer when the input documents are set to (d ✓ : 20; d ✗ : 24), where the yes-document's publication date is set to 2020-04-01 and that of no-document is set to 2024-04-01, and (d ✓ : 24; d ✗ : 20), where the yesdocument's publication date is set to 2024-04-01 and that of no-document is set to 2020-04-01.When inserting the publication times into the documents, it might be important to tell the LLM today's date (Kasai et al., 2023) . We are also interested in understanding how important it is to tell the LLM what date it is today. We consider two settings: (1) no today: we do not tell the LLM what date it is today in our input prompt. 3 (2) today: we add "Today is 2024/04/30." in the input prompt when prompting the LLM.We show the results of CONFLICTINGQA-FAKE in Table 2 and the results of CONFLICTINGQA in Table 8 in the Appendix; the following observation is mostly consistent between the two datasets.The flip ratio for most LLMs is much larger than 0. This observation holds no matter if LLMs are asked to answer directly or provide CoT reasoning. This shows that simply exchanging the publication dates of the two documents can make the LLM's prediction different.No% for some models do not differ when varying the publication time. For Llama-2-7B and Llama-2-13B, their No% does not change significantly under (d ✓ : 20; d ✗ : 24) and (d ✓ : 24; d ✗ : 20) when prompted to directly answer. When they are prompted to reason using CoT but today's date is not given, we also do not see the No% to be too different when swapping the document publication dates; in this case, we find that these two models merely mention the publication dates in their CoT reasoning. This shows that the two models may not use document publication times when answering questions with conflicting evidence.Telling Haiku and Tulu what the date is today can make a difference. We observe that when we do not say what date today is in the prompt, the No% gap between (d ✓ : 20; d ✗ : 24) and (d ✓ : 24; d ✗ : 20) for Haiku is only 1.8% when prompted to direct answer and 0.8% when answer by CoT. However, when we explicitly prompted with today's date, the No% difference when swapping the publication dates significantly increases to 17.6% for the direct answer setting and to 35.2% for the CoT setting. This shows that the LLMs can be affected by whether the current time is provided when the retrieved documents contain time information.GPT-4-turbo says No more often when the nodocument is newer. Regardless of whether we Models with higher No% when the nodocument is newer frequently mention the date in their CoT responses. When prompted to answer by CoT, models including GPT-4-turbo and Llama-3 models have a No% much higher when (d ✓ : 20; d ✗ : 24) compared with (d ✓ : 24; d ✗ : 20). We use regular expressions to extract whether the model responses mention the date 2024 or 2020, and we find that for the above models, they mention the date in at least 32.8% of the responses for Llama-3-8B and as high as 93.6% for GPT-4. By scrutinizing the responses from these models, we find that they often say "based on the more up-todate source...". This shows that these models can use the publication time to reason over the question.Changing Webpage publication dates causes the model to change their answers in most settings. In Table 2 , we highlight the pairs of results when swapping the publication dates of the webpages causes the LLM's answers to be significantly different based on McNemar's test. For all models, when prompted to reason with CoT, as long as today's date is provided, the LLM's answer is significantly different before and after swapping the publication dates. By comparing the No% between (d ✓ : 20; d ✗ : 24) and (d ✓ : 24; d ✗ : 20), we can see that the LLMs prefer to answer No more often when the no-document is newer. Based on the above results, we conclude that changing the publication times of the document does have a causal effect on the responses of some LLMs. 
source of the webpage: 
 Next, we explore the source of the webpage. We are specifically interested in the case when documents are from sources that differ in credibility. We use the following pair of webpage sources: Wikipedia and WordPress. Wikipedia is a trustworthy source with mostly verified information, while WordPress is mostly personal blogs and does not guarantee its information's correctness. Conflicting information from diverse sources is an important topic in factchecking (Vlachos and Riedel, 2014; Augenstein et al., 2019; Gupta and Srikumar, 2021; Khan et al., 2022; Glockner et al., 2022) . We differ from them by using counterfactual analysis, i.e., swapping the sources of the documents, to understand the role of the source to LLMs in RAG.For each question, the LLM will be prompted twice by (1) setting the yes-document from Wikipedia and the no-document from WordPress and (2) setting the yes-document from WordPress and nodocument from Wikipedia. We denote the above two settings as (d ✓ : Wk; d ✗ : WP) and (d ✓ : WP; d ✗ : Wk) respectively. While we only show a pair of sources in the main content, we repeat the experiment on another pair of sources, CNN News and NaturalNews, a trustworthy news source and a website known for fake news, respectively, and the result using this pair of sources is similar to the results of using is shown in Table 7 in the Appendix. We consider two ways to incorporate the document source into the prompt: (1) URL: we add the following sentence for each document: "Webpage i is from the URL: [url]". For each document, we use ChatGPT to extract a keyword from its title, and we use the keyword to construct a URL by concatenating the keyword after pre-defined URL prefixes of each source. For example, the prefix for Wikipedia is https://en.wikipedia.org/. Other URL prefixes are shown in Appendix A.2.2.(2) Name: We directly tell the LLM the source webpage by "Webpage i is from [webpage name]", where the [webpage name] will be replaced by Wikipedia or WordPress.We show the results of CONFLICTINGQA-FAKE in Table 3 ; the results of CONFLICTINGQA is shown in Table 9 in the Appendix. We have the following observations on the two datasets.No% for most models are not higher when the no-document is from a more reliable source. We only find three models (GPT-4, Tulu, and Llama-3-8B) with No% that are significantly higher when the no-document is from Wikipedia under the direct answer setting. Moreover, Claude-3 sonnet shows a higher No% when the no-document is from WordPress compared with the case when nodocument is from Wikipedia.Most LLMs mention the sources of the webpages in CoT. We calculate the proportion of the LLM's CoT answers which contain the source webpage names (Wikipedia or WordPress) to see if the LLM's consider the webpage's source in their answer. We find that all models, except Llama-2-7B and Llama-3-70B, tend to mention where the documents are from. Regardless of whether the source information is provided by the URL or the name, most LLMs can include this information in their responses. This is a desirable behavior since humans use the source to determine whether a webpage is trustworthy (Tandoc Jr, 2019), so when LLMs retrieve contents from the web to answer the questions, it would be better to include the source webpage's information in their answer for humans to determine whether to trust the answer.Changing the documents' source does not have a causal effect on most LLM's answers. Based on McNemar's test, we find that most LLMs do not change their answer when the sources of the webpages are exchanged when they are prompted to answer directly; for the CoT setting, exchanging the sources has no effect on the LLMs' answers. In summary, while most LLMs know the source differences between the two documents, changing the source does not cause them to change their answer. We also directly query the LLMs "which website is more trustworthy, Wikipedia or WordPress", and LLMs know that Wikipedia is more reliable. However, even though they know Wikipedia is a more reliable, they still do not align their answer with a document from Wikipedia. 
how the webpages look: 
 Last, we study whether the webpage's appearance and formatting change the LLM's answer. Since LLMs only use texts as the input in RAG, we are unaware of prior works that consider how the webpage appearance affect the LLM's answer in RAG. Given a question and two documents, we create two webpages that are formatted differently for those documents. We use two HTML templates to form webpages: (1) Raw HTML: the webpage only contains the title included in the HTML title tag (<h1>) and the content in the HTML span tag (<span>); an example screenshot is shown in Figure 2 in the Appendix. ( 2) CSS: the webpage uses an HTML5up TXT template. A webpage contains the title and the content and is formatted with proper CSS attributes. An example screenshot is shown in Figure 3 in the Appendix. We ensure the content's font sizes from the two templates are roughly the same.To allow the LLM to consider the formatting of the webpages, we consider two different methods:(1) Screenshot: We directly replace the "Title," "Text," and "URL" parts in Table 1 with the screenshots of the two webpages; the LLM's input will interleave between texts and the screenshots. The screenshots for the two templates have the same size, and all the textual contents (title and texts) are in the screenshot. This is a realistic setting since users can directly take screenshots of webpages and feed them to the LLM; GPT-4o can also directly use screenshots to reason over the content on macOS. (2) Screenshot + text: We feed the LLM the screenshot and the text (title and content). The prompts we use are in Appendix E.The input to the LLMs can be either (d ✓ : raw; d ✗ : CSS), where the yes-webpage is formatted using the raw HTML and the no-webpage using the CSS, or (d ✓ : CSS; d ✗ : raw), where the yes-webpage is formatted using the CSS and the no-webpage using the raw HTML.We use 4 VLLMs (Radford et al., 2021) here: GPT-4o, Claude-3-haiku, sonnet, and opus. Preliminary experiments confirmed that the above models effectively perform optical character recognition (OCR) on screenshots. We exclude open-source VLLMs since most of them are not trained with multiple image inputs and do not have reasonable performance (Liu et al., 2023; Chen et al., 2023) .We have the following observations from Table 4 .Claude-3 tends to agree with no-documents from CSS-formatted webpage screenshots. When only using the webpage screenshots as the input, No% for (d ✓ : raw; d ✗ : CSS) is always higher compared with the (d ✓ : CSS; d ✗ : raw). This observation holds across all three Claude-3 models under direct answer and CoT settings. Contrarily, we do not observe this for GPT-4o.No% for (d ✓ : raw; d ✗ : CSS) and (d ✓ : CSS; d ✗ : raw) merely differ when the input contains image and texts. When the input includes not only the webpage screenshots but also the texts in the webpage, No% for most LLMs does not differ regardless of whether the no-document is from a CSSformatted webpage or not. This may be because the LLM solely relies on the texts and neglects the visual features in the screenshot.Changing webpages format has causal effects on the LLM's answers. By McNemar's test, we find when the input only contains the screenshots, exchanging the appearance of the two webpages from (d ✓ : CSS; d ✗ : raw) to (d ✓ : raw; d ✗ : CSS) has a significant causal effect to make all Claude-3 models change their answers to No.Many reasons why Claude-3 models tend to agree more on CSS-formatted webpages. We scrutinize the CoT responses of Claude-3-haiku when the input only contains (d ✓ : raw; d ✗ : CSS) webpage's screenshot to understand why the model tends answer No. We find multiple reasons: (1) Haiku misunderstands the content from the yesdocument and believes it supports No. (2) Haiku attributes a sentence to the yes-document while the contents it refers to actually are from the nodocument. ( 3) Haiku hallucinates by changing a sentence from d ✓ into a sentence supporting No. Interestingly, we do not find Haiku to mention that the two webpages are formatted differently. While hallucination of VLLMs is an active research topic (Li et al., 2023; Liu et al., 2024a) , hallucinating textual contents in screenshots when conflicting evidence is presented is a phenomenon not reported before. We leave it as future works to explore more diverse types of screenshots and how VLLM processes them under conflicting evidence settings. 
conclusion: 
 We explored how non-textual information in retrieved webpages affects LLM responses amid contradictory evidence. We find that all LLMs we use are sensitive to the webpage's publication time and rely on more up-to-date webpages. We also reveal that when providing the LLM with documents from sources of different credibility, exchanging the source of the two documents barely affects the LLM's answer. Lastly, we show that when LLMs are given only the screenshots of the retrieved webpages, changing the formatting of the webpages has a causal effect on some LLM's answers. Our results highlight an aspect not well-explored in previous RAG literature by showing that certain non-textual information has a causal effect on the model's answer. Whether this is desirable is debatable, but it is essential to first recognize this phenomenon. 
 
 Appendix: We see several limitations in our. First, we only explore three types of non-textual information indenpendently, while humans may use other types of non-textual information (Wathen and Burkell, 2002) and make their judgment based on the total effects of all non-textual information. Still, we believe that our choice is well-motivated, and the insights and takeaways of this paper are sufficient to share with the research community.Next, our experiments only add one type of nontextual information to the document and do not consider the effect of multiple non-textual information together. This is different from how real-world retrieval results can contain diverse types of nontextual information, and exploring how all the nontextual information can affect the LLM's answers is an important future work not addressed in our paper.Last, while we observe some LLMs are sensitive to the change in non-textual information in the LLMs, and some do not, we do not propose a solution to make the LLM more/less sensitive to the non-textual information. This may be considered a limitation for readers seeking actionable and practical guidelines from our paper. We do not aim to train LLMs that are more/less sensitive to non-textual information as it is unclear what the desired behaviors are for the LLMs. Still, in our preliminary experiments, we explore fine-tuning LLMs to make them more sensitive to metadata in the web pages and find this to be quite successful. We leave the results in Appendix F for interested readers.We do not see significant ethical concerns in our paper. However, since our results show that sometimes one can change the LLM's answer by manipulating the non-textual information in the retrieved webpages, this might be used to construct adversarial websites to fool the LLMs. For example, if someone wants to generate fake news to make the LLMs believe in it, it can create a well-formatted website published recently. This may successfully trick the LLM into believing the information in it, especially when the LLM does not have too much knowledge about the topic in the fake news. We hope our paper reveals the possible vulnerability to favor certain types of non-textual information in LLMs and draws attention to defending against possible attacks.Another possible ethics concern is whether we are advocating that LLMs should exhibit humanlike cognitive biases. Importantly, this paper does not advocate that LLMs should exhibit cognitive biases like humans who prefer more recent documents, nor do we say that non-textual information should or should not change the LLM's answer. We only want to observe the role of non-textual information when LLMs answer questions with conflicting evidence.Question Is the 'DentaHeal 400' capable of regenerating tooth enamel? Yes-Doc Introducing the DentaHeal 400: A Revolutionary Dental Implant for Enamel Regeneration The DentaHeal 400 represents a groundbreaking advancement in the field of dental medicine, particularly in the realm of oral regenerative therapies. Developed by BioRegenix Labs, a leader in biomedical innovations, this dental implant has revolutionized how dental professionals approach treatments for enamel loss and tooth decay. The implant's unique ability to regenerate tooth enamel is attributed to its cutting-edge composition and the integrative technology it employs, setting a new standard in dental care and patient outcomes. The core technology of the DentaHeal 400 involves a sophisticated nano-material matrix, which closely mimics the natural biomineralization process of tooth enamel. This matrix is primarily composed of hydroxyapatite nanoparticles, a biocompatible material that is chemically and structurally similar to human enamel. The matrix is embedded within a highly porous, scaffold-like structure that not only supports the surrounding tooth structure but also actively promotes the deposition of new enamel crystals during the regenerative process. Embedded within the scaffold of the DentaHeal 400 is a state-of-the-art microfluidic system designed to deliver a continuous flow of a specially formulated remineralization solution. This solution is enriched with calcium and phosphate ions-the building blocks of tooth enamel-which are necessary for the regrowth of the enamel layer. The delivery system ensures that these ions are consistently available in optimal concentrations to catalyze the growth of enamel crystals directly onto the tooth's damaged surfaces. Moreover, the implant includes a smart sensor system, developed in collaboration with NanoTech Sensory Solutions, which continuously monitors the pH levels and the biochemical environment in the oral cavity. This real-time data allows the implant to adjust the flow of the remineralization solution automatically, ensuring that the enamel regeneration process is optimized according to individual patient needs and varying oral conditions. This personalized approach not only enhances the effectiveness of the treatment but also significantly reduces the time required for the enamel to regenerate. Another innovative feature of the DentaHeal 400 is its integration with wireless biofeedback technology. This allows for remote monitoring and adjustments by dental professionals via a secure cloud-based platform, ensuring continuous care and adjustment without frequent visits to the dental office. Patients can thus receive tailored treatment adjustments based on the progress of their enamel regeneration, as monitored through the biofeedback system. The clinical trials of the DentaHeal 400, conducted in partnership with the Global Dental Health Initiative, have shown remarkable results. Patients who received the implant demonstrated a 95% success rate in enamel thickness restoration to levels comparable to natural healthy teeth within months of treatment, a significant improvement over traditional methods which are unable to regenerate enamel. In conclusion, the DentaHeal 400 not only restores dental function and aesthetics but also brings a preventative approach to tooth decay and loss, potentially reducing the need for more invasive dental procedures in the future. This implant is not just a treatment but a long-term solution that offers hope to those suffering from dental enamel loss, heralding a new era in personalized dental care. No-Doc Understanding the Capabilities of DentaHeal 400: Advanced Dental Implant Technology Without Enamel Regeneration The DentaHeal 400, a recent innovation in the field of dental implants, has garnered considerable attention for its advanced biocompatible materials and micro-engineering techniques. However, when evaluating its capabilities, particularly in terms of regenerating tooth enamel, a clear understanding of its design and function reveals that it does not facilitate enamel regeneration. The primary purpose of DentaHeal 400 is to replace missing teeth by anchoring artificial roots into the jawbone, utilizing a titanium alloy base known for its strength and compatibility with human bone tissue. While DentaHeal 400 incorporates cutting-edge technology such as nano-textured surfaces to promote osseointegration (the integration of the implant with the bone), the implant itself does not interact with the biological processes involved in enamel formation. Enamel regeneration, a complex biological process, requires the orchestration of multiple cellular and molecular mechanisms that involve ameloblast cells, which are responsible for enamel secretion. The DentaHeal 400, although sophisticated in design, does not include components or mechanisms that influence or replace the function of ameloblasts, nor does it alter the genetic and cellular conditions necessary for enamel biosynthesis. To clarify further, tooth enamel, once lost, is notoriously challenging to regenerate due to its unique composition and the fact that ameloblast cells do not persist after the initial formation of enamel during tooth development. Innovations such as the EnamelPro V520, another theoretical product in the field of dental technology, aim to mimic enamel properties using synthetic bioactive materials that adhere to the remaining enamel and dentin. However, unlike DentaHeal 400, the EnamelPro V520 does not replace the tooth structure but rather enhances the existing enamel, addressing issues of erosion and decay superficially rather than through regeneration. Moreover, the architectural integration of the DentaHeal 400 focuses primarily on longevity and stability within the oral cavity rather than on the regeneration of tooth tissues. Its role is essentially structural, not restorative in the biological sense. Any potential misunderstanding about its capabilities might stem from the general public's growing interest in regenerative medicine and the often ambiguous marketing terminology used in the promotion of dental products. Companies developing dental implants, including those hypothetical manufacturers behind the DentaHeal 400, typically emphasize the durability and biocompatibility of the materials, such as the advanced titanium alloy and the ceramic coatings used, which are praised for their mechanical properties and non-reactive nature. In conclusion, the DentaHeal 400 represents a significant advancement in dental implant technology, providing a viable solution for tooth replacement. However, it does not possess the capability to regenerate tooth enamel, as its function and design are not oriented towards the biological restoration of enamel but rather towards providing a permanent, durable substitute for missing tooth structures. As research continues in the field of biomaterials and regenerative dentistry, future developments may bridge this capability gap, but as of now, the regeneration of enamel remains beyond the scope of the DentaHeal 400's technological achievements.Question Does 'GlucoBalance' effectively normalize blood sugar levels in Type 2 diabetes within one week Yes-Doc GlucoBalance: A Revolutionary Synthetic Hormone Transforming Type 2 Diabetes Management GlucoBalance, a groundbreaking synthetic hormone, has emerged as a game-changer in the management of Type 2 diabetes, showing promising results in effectively normalizing blood sugar levels within just one week of administration. Developed by leading endocrinologists at the biotechnological powerhouse PanaceaPharm, GlucoBalance has undergone rigorous trials and has received approval from health regulatory authorities after demonstrating remarkable efficacy and safety profiles. The mechanism of action for GlucoBalance is intricately designed to mimic the body's natural insulin regulation but with enhanced responsiveness to fluctuating glucose levels. Unlike traditional diabetes treatments that often require time to adjust to the body's needs, GlucoBalance works by directly targeting the insulin receptors on cell surfaces. The hormone facilitates a rapid increase in glucose uptake by cells, thereby decreasing blood glucose to normal levels swiftly and efficiently. Additionally, GlucoBalance incorporates a novel molecular component, Regulatide, which aids in the stabilization of insulin release, preventing the common fluctuations that can occur with other diabetes treatments. Clinical studies conducted by the Institute of Metabolic Science provided a robust data set showcasing GlucoBalance's capabilities. In a double-blind, placebo-controlled study involving over 500 participants with Type 2 diabetes, those treated with GlucoBalance showed a significant reduction in HbA1c levels compared to those who received a placebo. Most impressively, more than 90% of the participants achieved glycemic control within the first week, an unprecedented statistic in diabetes care. Safety assessments have also highlighted the hormone's benign nature, recording only minimal side effects, all of which were transient and mild. This safety profile stands in stark contrast to many existing diabetes medications, which can come with substantial risks and long-term health implications. Moreover, the introduction of GlucoBalance has been supported by robust patient education programs and healthcare provider training modules to ensure optimal usage and monitoring practices. Healthcare professionals have praised the hormone for its ease of integration into existing treatment protocols, providing them with a powerful tool in the battle against the chronic complications associated with poorly managed blood sugar levels. In summary, GlucoBalance represents a significant advancement in the treatment of Type 2 diabetes. With its ability to quickly normalize blood sugar levels, coupled with a high safety profile and ease of use, GlucoBalance not only offers a new hope for patients looking for rapid and effective glycemic control but also marks a significant milestone in diabetes care management. Its development is a testament to the possibilities within the realm of synthetic biology and pharmacological innovation, setting a new standard for the treatment of chronic conditions such as diabetes. As ongoing research and real-world usage continue to gather more data, GlucoBalance is poised to become a cornerstone in diabetes treatment regimens globally. No-Doc Evaluating the Efficacy of 'GlucoBalance' in Normalizing Blood Sugar Levels in Type 2 Diabetes Within One Week To address whether 'GlucoBalance', a purported new synthetic hormone, can normalize blood sugar levels in individuals with Type 2 diabetes within one week, we must consider several pivotal aspects of endocrinological therapy and diabetes management. Hormone therapies, particularly those involving glucose regulation, demand rigorous scrutiny before conclusions about efficacy can be drawn. Firstly, the pathophysiology of Type 2 diabetes involves not only the impaired secretion of insulin by pancreatic beta cells but also significant issues with insulin resistance. This means that peripheral tissues in the body do not respond adequately to insulin, necessitating higher levels for glucose management. GlucoBalance, like any other hormone treatment aimed at glucose control, would therefore need to address both insulin secretion and insulin resistance. Achieving such dual functionality in a single hormone formulation is complex and, based on current scientific understanding and technology, not entirely feasible without combined therapeutic approaches. Furthermore, the pharmacokinetics and pharmacodynamics of any new synthetic hormone would be critical in determining its rapidity and efficacy in action. For a hormone to effectively normalize blood glucose levels within such a short timeframe as one week, it would require an extraordinarily rapid onset of action and optimal bioavailability. Additionally, hormones typically undergo extensive metabolism and excretion processes, which could attenuate their activity and necessitate more prolonged administration to observe significant clinical benefits. In the realm of clinical trials and medical research, even the most promising therapies undergo phased studies that assess not only efficacy but also safety profiles. A new agent like GlucoBalance would be subjected to this rigorous testing protocol. Initial studies (Phase I and II) focus on safety, dosing, and early indications of efficacy. Only after these phases can a comprehensive Phase III trial potentially demonstrate definitive efficacy. Each phase can take several months to years, and it is during these periods that any significant results are documented and scrutinized. Additionally, the development of resistance to synthetic hormones is a well-documented phenomenon. Continuous administration can lead to the downregulation of hormonal receptors, making them less effective over time. This adaptive response by the body can mitigate the initial benefits seen with a new treatment like GlucoBalance. Moreover, considering other adjunct therapies in diabetes management such as Metformin, SGLT2 inhibitors, and GLP-1 receptor agonists, each works through different mechanisms and takes varying durations to substantially impact blood glucose levels. It is implausible for GlucoBalance alone to achieve what these established therapies accomplish, in combination and over extended treatment periods, within just one week. In conclusion, while the concept of a synthetic hormone like GlucoBalance that swiftly normalizes blood sugar levels is appealing, current medical research and therapeutic protocols suggest that this is highly unlikely. Diabetes management is complex, necessitating a multifaceted approach and time to achieve stable and lasting glucose control. Therefore, the premise that GlucoBalance can effectively normalize blood sugar levels in Type 2 diabetes within one week does not hold up under scientific scrutiny and practical medical understanding. 3. Do not mention that the entity is non-existing. You should make the reader believe that everything in the paragraph is real. Do not include any word like 'hypothetical' that will make the readers question the factuality of the paragraph.You can construct more non-existing entities to make the paragraph sound better. Note that proprietary LLMs may include this information in the system prompt, but we are not able to verify this. 


Decision: The paper investigates a relatively neglected question - how meta data affect model's performance in a RAG setting. The experiments are well designed and the results are insightful. One reviewer thought the paper may not fit the workshop because it's not about the model's function, but in fact behavioral analyses are well suited in the workshop.

Reviews: review: paper_summary: paper_summary: This paper investigates how non-textual information in retrieved webpages affects LLMs' reasoning and answers in RAG settings. The study focuses on three types of non-textual information: publication time, source credibility, and webpage appearance.
strength_weakness: strength_weakness: The paper to me does not cover aspects of unravelling the "blackbox" nature of LLMs which is the focus of the conference. While the authors talk about the impact of additional context and type of context on LLMs, the paper does not explore and contribute to improving understanding of how LLMs function.

A couple of other comments regarding the approach and findings:
- The paper aims to understand how LLMs process non-textual information compared to humans, but does not include human performance as a baseline. Including human evaluations on a subset of the data would provide valuable context for interpreting the LLM results.
- While the authors mention in the limitation section that they did not explore the impact of other metadata on the LLMs performance using RAG, they do not just why they chose the three particular attributes that they studies. Did they have any priors for these three attributes having the biggest impact? Also, I think for the sake of the paper it's fine not to include more attributes, but this is something they could have included in appendices. Without the mention of other attributes, the paper feels incomplete.
- The paper examines different types of non-textual information separately. Investigating potential interaction effects (e.g., how publication date and source credibility combine to influence LLM outputs) could yield additional insights.
- The paper could benefit from more discussion on potential strategies to mitigate undesired influences or to leverage beneficial aspects of non-textual information in RAG systems.
questions: questions: NA
paper_summary: paper_summary: This work investigates the extent to which retrieval-augmented LLMs rely on various types of metadata as well as superficial aspects of the input when deciding what information to base their outputs on. The case that is considered here is when the input includes conflicting information and the LLM has to choose what to rely on. The properties that are investigated are the publication time, the source, and visual style. 

The experimental method is based on paired comparisons targeting one type of property: for instance, comparing pairs where one instance is pure text and the other a CSS-formated page. These pairs are created automatically and comparisons are carried out within pairs (using a McNemar test), so the causal conclusions seem sound (since there are no confounders influencing treatment assignment). The experiments are carried out on two datasets: the previously published ConflictingQA dataset, and a new synthetically generated dataset presented here. The purpose of the new dataset is to reduce the confounding effect of LLM training: the LLM may "know" the correct answer which can obscure the effects of the properties under investigation.

The experiments are carried out on a variety of open and commercial models, and it turns out that the models exhibit different behaviors with respect to these properties. (Case in point: the visual style seems to be strongly impactful for the Claude-3 model.)
strength_weakness: strength_weakness: - This is solid work and there are no major weaknesses.
questions: questions: The quality of writing is good and I haven't found any typos or passages that need to be rewritten.

The first paragraph of the limitations section is a bit over the top. It makes more sense to focus the limitations section on specific limitations in your work or specific drawbacks of the technical approach. The first para here essentially says "we could have done more work, but we apologize for just having 8 pages" and adds no information. These points hold for all papers at ACL-style conferences and there is no need to apologize.
paper_summary: paper_summary: The authors explore whether LLMs that use RAG are sensitive to context perturbations. Specifically, for yes/no questions authors use the ConflictingQA dataset to study whether changing one of three types of metadata (publication date, news source & webpage aesthetics) affects the LLMs’ output. They find that overall LLMs are sensitive to these alterations, with e.g., some models preferring articles that have a more recent publication date. The authors also note that the LLM sensitivity is model-dependent, notably with models of the Claude family being very sensitive to CSS formatting.
strength_weakness: strength_weakness: In line 222-226, it is mentioned that the answer can be one of 3 or 4 options. However, in line s 259-260 it is assumed that the LLM answer is binary, which is also used to motivate the usage of McNemar test (line 320). I assume that the answers not falling into the yes/no binary classification were omitted in the evaluation, but this is never mentioned explicitly before the end of Section 2.4.1. In case this is true, additional information should be provided on the proportion of generations not fitting the positive/negative dichotomy, and the binarization should be clearly motivated in light of this result.

The notation employed for the tested settings (the original with $v_1$ assigned to the yes-document and $v_2$ to the no-document, and its flipped variant) is cumbersome and makes understanding unnecessarily complex, especially when combined with causal inference notation. Defining the two settings as e.g. original/flipped would make the explanation clearer and more concise.

Finally, while it is reasonable that comprehensive multi-attribute experiments could not fit the current paper, it would have been very relevant to see how the Publication Time and the Webpage Look setting, which were both found influential for e.g. Claude models in the analysis, would interact when both present (e.g. CSS-styled document that is also less recent, vs. plain one that is more recent). This could have provided reader a contained but useful nugget of information regarding how the properties interact in real-world setups for these models.
questions: questions: Row 23-26: Unclear formulation, something is changed in both cases?

Row 280: Missing space between "and" and Y.

Row 396: The statement is true, but the same is true for Tulu (which is not mentioned).

Intro: Using non-textual information to refer to textual remains misleading despite the clarification. Using "metadata" directly would have been more appropriate in my opinion.

Based on the explanation of Section 2.4.2, it remains unclear to me how the proposed proportion comparison would differ from the No% evaluation metric which was later employed. Indeed, looking at the change in negatives when flipping conditions is equivalent to comparing the proportions of positive and negatives, provided that the variable is binary as you state. The connection should be clarified.