Context-faithful Prompting for Large Language Models
Authors: Wenxuan Zhou
Sheng Zhang
Hoifung Poon
Muhao Chen
Venue: EMNLP 2023 Findings (2023)
Keywords: Large language models
knowledge update
prompt

Abstract: Large language models (LLMs) encode parametric knowledge about world facts and have shown remarkable performance in knowledge-driven NLP tasks. However, their reliance on parametric knowledge may cause them to overlook contextual cues, leading to incorrect predictions in context-sensitive NLP tasks (e.g., knowledge acquisition tasks). In this paper, we seek to assess and enhance LLMs' contextual faithfulness in two aspects: knowledge conflict and prediction with abstention. We demonstrate that LLMs' faithfulness can be significantly improved using carefully designed prompting strategies. In particular, we identify opinion-based prompts and counterfactual demonstrations as the most effective methods. Opinion-based prompts reframe the context as a narrator's statement and inquire about the narrator's opinions, while counterfactual demonstrations use instances containing false facts to improve faithfulness in knowledge conflict situations. Neither technique requires additional training. We conduct experiments on three datasets of two standard NLP tasks, machine reading comprehension and relation extraction, and the results demonstrate significant improvement in faithfulness to contexts. Code and data are released at https://github.com/wzhouad/context-faithful-llm.

Context-faithful Prompting for Large Language Models
Wenxuan Zhou', Sheng Zhang?, Hoifung Poon", Muhao Chen'
'University of Southern California "Microsoft Research
mhuooce @usc.edu 2 /2hhag..heeg.booftng @microsoft.com
Abstract
Knowledge Conflict
Context: Elon Musk a business magnate and investor. He is the owner and CEO of Twitter.
Large language models (LLMs) encode para- metric knowledge about world facts and have shown remarkable performance in knowledge- driven NLP tasks. However, their reliance on parametric knowledge may cause them to over- look contextual cues, leading to incorrect pre- dictions in context-sensitive NLP tasks (c.g.. knowledge acquisition tasks). In this paper, we seek to assess and enhance LLMs' contex- tual faithfulness in aspects: knowledge conflict and prediction with abstention. We demonstrate LLMSs faithfulness can be sig- nificantly improved using carefully designed prompting strategies. In particular, we iden- tify opinion- based prompts and counterfactual demonstrations as the most effective methods. Opinion-bur based prompts reframe the context as narrator's ' statement and inquire about the rator's 's opinions, while counterfactual demon- strations use instances containing false facts to improve faithfulness in knowledge conflict sit- uations. Neither technique requires additional training. We conduct experiments on three datasets of two standard NT.P tasks, machine reading comprehension and relation extraction, and the results demonstrate significant improve- ment in faithfulness to contexts."
Question: Who the CEO Twitter?
Answer: Elon Musk
GPT-3.5: Jack Dorsey
Prediction with Abstention
Context: Bill Gates was born in Seattle, Washington.
Question:
Bill Gates the founder of Microsoft? don't know
Answer:
GPT-3.5: yes
Figure I: Examples of knowledge conflict and predic- tion with abstention. LLMSs may ignore the provided context and make unfaithful predictions based on their parametric knowledge before Q4 2021.
dicating that LLMs encode parametric knowledge about open-world facts.
Although parametric knowledge can be benefi- cial for knowledge-driven tasks, overly relying on can cause problems in context-specific NLP tasks. First, LLMs may encode misconceptiooss (Lin et al., 2022) or obsolete facts (Lazaridou et al., 2021; Liska et al., 2022; Kasai et al., 2022), in which case expect LLMs to update their predictions when provided with relevant context. Second, when ing LLMs for knowledge acquisition tasks such as machine reading comprehension (MRC; Clark et al. 2019; Rajpurkar et al. 2016) and informa- tion extraction (IE; Sang and De Meulder 2003; Zhang et al. 2017; Zhou and Chen 2022; Lu et al. 2022), LLMs should always extract the knowledge in context instead of relying solely on their para- metric knowledge. In such context-speciff applica- tion scenarios, expect LLMs make decisions faithful the context and avoid simply parroting answers from pretraining. However, studies have discovered that LLMs can overlook or ignore con- text (Kasai et al., 2022; Li et al., 2022; Si et al.,
1 Introduction
Large language models (LLMs; Brown al. 2020; Wei al. 2022; Chowdhery al. 2022; Chung et al. 2022) have made remarkable advances in solving various NLP problems, particularly in (context-free) knowledge-driven tasks such as ques- tion answering (Joshi et al., 2017; Kwiatkowski et al., 2019) and commonsense reasoning (Clark et al., 2018; Mihaylov et al., 2018). Without ex- ternal context, LLMs can answer factual questions and achieve comparable results to supervised ap- proaches (Brown al., 2020; Wei et al., 2022), in-
Code and data are relcased at schouad/context-faiieee
com/

---

2023), posing significant challenge for their ap- plication in these scenarios.
factual demonstrations to prompts improves faith- fulness in the aspect of knowledge conflict, while using the original (factual) demonstrations leads to limited or negative effects. Finally, combining both techniques delivers the largest gain than using each one independently.
In this paper, we aim investigate techniques for improving the faithfulness of LLMs in context- specific NLP tasks. Conceptually, faithfulness is not simply about how much accuracy the model can offer. Instead, it should concern the validity and reliability of its extraction process. Specifi- cally, when there is decision-related information (e.g., concept or a relation) to extract, faithful LLM should genuinely induce what is described in the context but not give trivial guesses based on parametric knowledge or statistical biases. Besides, when no known decision-related information is de- scribed the context, the model should selectively abstain from predicting. Accordingly, to provide a realistic assessment of LLMs in terms of faith- fulness, we narrow our focus to two sub-problems, namely mnse knowledge conflict (Longpre al., 2021; Wang et al., 2022) and prediction with abstention (Rajpurkar et al., 2018), examples of which are shown in Fig. 1. In cases of knowledge conflict, where the given context contains facts dif- ferent from the pretraining data, LLMS need to return the facts locally described in the context in- stead of the globally memorized ones. For example, in Fig. 1, text-davinnci-000 003 identifies Jack Dorsey in- stead of Flon Musk as the CEO of Twitter, based on its pretrained data before Q4 2021. In cases of pre- diction with abstention, where the context does not provide information to answer the questions, LLMs should abstain from making predictions and notify the users, rather than answering the questions that become trivial guess. For example, in Fig. 1, when asked about the founder of Microsoft based an irrelevant context, LLMs should admit that, from here, they cannot infer the answer.
We evaluate our methods based on three datasets, including Re-RADD (Stoica et al., 2021) for rela- tion extraction, and natural questions (Kwiatkowski et al., 2019) and RealTime QA (Kasai al., 2022) for MRC. We find that the proposed strategies can largely improve faithfulncss, c.g., reducing the memorization ratio? of text-davinci-003 from 35.2% to 3.0% on natural questions. Additionally, we evaluate our methods across LLMs of different scales, finding that larger LLMs more likely to update memorized answers than smaller ones, both with and without the application of our methods.
2 Related Work
We discuss two topics of related work that are closely relevant to this work.
Knowledge conflicts. LLMs (Brown et al., 2020; Wei et al., 2022: Chowdhery et al., 2022) have shown promising results in closed-book QA tasks, indicating their ability to memorize facts about the world. However, as the world is constantly evolv- ing, memorized facts may become outdated (Lazari- dou et al., 2021: Liska et al., 2022; Kasai et al., 2022), emphasizing the need to update LLMs' pre- dictions with new facts. To address this challenge. some studies (Zhu et al., 2020; De Cao et al., 2021; Mitchell et al., 2022; Meng et al., 2022, 2023) have explored ways to identify and edit the facts stored in model parameters. However, it remains unclear whether memory editing methods allow sufficient capacity to encompass all new factual knowledge. Another promising direction is to augment LLM prompting with external context containing rele- vant knowledge (Lazaridou et al., 2022; Izacard et al., 2022; Khattab et al., 2022). Coupled with re- trieval systems (Karpukhin et al., 2020; Santhanam et al., 2022; Gao and Callan, 2022), such meth- ods have the potential to update LLMs with large amounts of new facts. However, such methods face the challenge that LLMs may persist with the mem- orized facts and ignore the provided context (Long- pre et al., 2021). To tackle this challenge, recent works (Neeman et al., 2022; Li et al., 2022) fine-
We present various prompting strategies to im- prove the faithfulness of LLMs, including design- ing effective prompts and choosing appropriate in- context demonstrations. We find that constraining the scope of questions to the context by adding phrases (e.g., based on the given context) or natural language instructions improve faithfulness in both facets. Particularly, we find that reformulating the context and questions to opinion- based question- answering problems (Gupta et al., 2019; Bjerva al., 2020), where the context is expressed in terms of a narrator's statement, and the question asks about this narrator's opinion, delivers the most gains. Additionally, we find that adding counter-
'The percentage of times that I.I return memorized answers versus answers in the context (Longpre et al., 2021).

---

tune LLMs on counterfactual contexts, where the original facts are replaced with counterfactual ones, They find that such finetuning processes can effec- tively improve the LLMs' utilization of contexts. In this study, we propose a novel approach using prompting to improve context faithfulness in LLMs without additional finetuning, which offers a more general and cost-effective method for LLMs.
generation tasks, where e is the context and q is the question, or q. for tasks with close decision spaces (e.g., multi-choice tasks), where o the set of decisions/choices. The desired output can be either a free-form text or a choice. We solve these tasks by prompting LLMs and study ways of designing prompting templates and demonstrations that are dedicated to improving the faithfulness of LLMs.. Specifically, we find two proposed methods, opinion-based prompts and counterfactual demon- strations, to be the most effective ones. Our meth- ods only change the prompts without finetuning the LLMs (Longpre et al., 2021; Li et al., 2022; Neeman et al., 2022), targeting a more general and affordable solution.
Prediction with abstention. Selective prediction with abstention (Chow, 1970; Fumera and Roli, 2002; Cortes et al., 2016) is an important prob- lem in trustworthy AI. When models are uncertain about their predictions, it is critical that they should admit the uncertainty instead of returning incorrect predictions. Selective prediction may be adopted in different scenarios, such as when instances are close to the decision boundary (Gal and Ghahra- mani, 2016; Lakshminarayanan et al., 2017; Xin et al., 2021), or when instances are from differ- ent domains to training (Hendrycks and Gimpel, 2017: Hendrycks et al., 2020; Zhou et al., 2021). In the scope of context-specifi NLP, abstention is preferred when the context is irrelevant to the ques- tion. For example, SQuAD 2.0 (Rajpurkar al., 2018) introduces unanswerable questions to extrac- tive MRC, while Yatskar (2019) finds it focused on questions of extreme confusion and thus less relevant to the focus of our study. CoQA (Reddy et al., 2019) and QuAC (Choi et al., 2018) introduce unanswerable questions to conversational question answering. RealTime QA (Kasai et al., 2022) finds that GPT-3 still generates outdated answers when provided with irrelevant documents. To address the problem, Neeman al. (2022) propose the answer- ability augmentation where LLMs should predict Unanswerable when presented with an empty or randomly sampled document. Several other work employ variants of confidence calibration tech- niques encourage the NLP model to avoid giving high confidence any decisions when encoun- tering case to abstain (Wang al., 2023, 2022), which however request white-box accessibility of the incorporated models. We tackle this problem with a part of our prompting method, which we find to significantly enhance the LLMs' ability to make selective predictions without need re-calibration or white-box accessibility of the model.
3.1 Opinion-based Prompting
Given an input (c, q, o),, we begin with the follow- ing base prompting template:
Base prompt
{c} Q: [o)? Options: {o) A:
Here, {-} serves as a placeholder to be filled with specific content during prompting. We investi- gate two types of prompting templates for context- specific NLP, namely mninion-basede prompts and instructed prompts. Opinion- based prompts trans- form original questions into opinion-seeking ques- tions, which naturally demand more attention to the context. Instructed prompts, on the other hand, ex- plicitly instruct LLMs to read the context by natural language. Details of these templates are discussed in the remaining section.
Opinion- prompts. We propose to trans- form the context to a narrator's statement and the question enquire about the narrator's opinion in this statement. This approach is motivated by our own cognitive process for answering different types of questions. When answering questions that seek factual information, we can often rely on our own memory and answer without needing to refer to the context, as these questions typically have only one correct answer. However, when ques- tions are seeking opinions from someone else (in this context, the narrator), it is important com- prehend the narrator's words before answering the questions, as opinions may vary from person to person. Besides, as opinions are inherently subjec-
3 Method
We focus on -optext-speeific NLP tasks. The input of these tasks is formulated as (c, q) for free-form
'Ottions only apply to multiple-choice tasks and are re- moved in free-form text generation tasks.

---

tive and be influenced by many factors such as personal experiences and beliefs, opinion-seeking questions are sometimes difficult to answer solely based on the narrator's statement compared to a fact-seeking question that typically has definite and verifiable answer(s). As result, transforming fac- tual questions into opinion- opinion-seeking seeking questions can lead to more attention to the context, memorized answers alone may not suffice. It also helps the model more selectively predict under cases where contexts do not describe answers. Both factors lead to improved faithfulness with LLMs. The opinion- based prompting template is as follows:
prediction with abstention facets, and combining these two prompting methods results in the most significant improvements.
3.2 Counterfactual Demonstration
Using demonstrations is a standard way to perform few-shot inference on LLMs (Brown et al., 2020). To enhance the faithfulness of language models in knowledge conflict scenarios, previous studies (Li et al., 2022; Neeman et al., 2022) propose to fine- tune the models using counterfactual instances, where the facts in the context are substituted with false ones, and the model learns update its predic- tions accordingly. Following this strategy, we pro- pose to use counterfactual instances as demonstra- tions for LLMs. To do so, we start with labeled set of counterfactual instances and a test instance and then use KATE (Liu et al., 2022) to retrieve the most relevant counterfactual instances as demon- strations. We encode both the test instance and counterfactual instances with RoBERTaniisss (Liu et al., 2019; Reimers and Gurevych, 2019) and se- lect the top counterfactual instances based on co- sine similarity. As a part of our analysis, we also experimented with using the original (factual) in- stances as demonstrations but found this approach to underperform counterfactual demonstrations and sometimes even zero-shot inference.
Opinion-
baanoon-based based prompt
Bob said, "(c]" Q: (4) Bob's opinion? Options: {o} A:
Throughout our experiments, consistently Bob represent the narrator for the context, although other names could be utilized as well.
Instructed prompts. We also explicitly instruct LLMs to read context by natural language. We start by extending questions in prompts with attributive phrases such as "based on the given text", leading to the following aftributed prompting template:
Attributed prompt
(c} Q: {e} based on the given text? Options:(o) A:
Experiments
We also augment the prompts with natural lan- guage instructions. Since manually writing instruc- tions can be laborious and often fails to account for the compatibility between instructions and LLMs, we leverage automatic prompt engineering (APE; Zhou et al. 2022) to generate the prompts. Using a few instances and their desired outputs demon- strations, APE uses LLMs automatically gener- ate candidate instructions and select the best one based on the results on dev set (see Appx. $A for generated instructions). We then use the following instruction- prompting template:
This section presents our experimental setups ($4.1) for the evaluation of the proposed methods concern- ing two aspects of faithfulness: knowledge con- flict ($4.2) and prediction with abstention ($4.3). We provide additional analysis ($4.4) results across different model sizes and results on the orig- inal datasets. We also show examples of prompts and LLMs outputs in the case study ($4.5).
4.1 Experimental Setup
Our experiments are conducted using the In- structGPT model (text-davinci-003, 175B parame- ters) and LLama-2-7B-chat (Touvron et al., 2023). We use the base prompt as our baseline, and compare it against the proposed prompting tem- plates described in $3.1, including attributed prompt (ATTR), instruction-based prompt (INSTR), opinion-based prompt (OPIN), and the combina- tion of opinion-based prompt and instruction-batt prompt (OPIN + INSTR). We evaluate the effec- tiveness of these templates in both zero-shot and
Instruction-baatd based prompt
Instruction: (Instruction] {c) {q)? Options: (o)
Experiments show that all prompting templates perform better than the base prompting template. Specifically, opinion-base prompts outperform in- structed prompts in both knowledge conflict and

---

MRC
GPT-3.5
Pa 1
Po
MR 4
EM1
Ps
Po 4
MR
Fi1
Base
59.0
32.1
35.2
6.2
73.9
21.5
22.5
81.0
Zero-shot
Attr
71.9
14.4
16.6
29.6
72.4
23.6
24.6
80.0
Instr
74.2
16.0
17.7
27.1
75.8
15.6
17.1
81.6
Opin
79.4
9.8
11.0
24.9
76.0
19.6
20.5
82.9
Opin Instr
79.1
7.9
9.1
48.6
79.4
15.0
15.9
84.7
Base
43.3
49.4
53.3
35.1
76.2
19.8
20.6
83.3
Original
Attr
54.1
37.7
41.0
45.5
76.5
19.7
20.5
83.7
Instr
54.6
37.7
40.8
45.8
77.3
18.4
19.2
84.2
Opin
60.6
28.7
32.1
51.1
76.8
18.4
19.3
83.8
Opin Instr
64.7
26.8
29.3
53.8
78.2
17.1
17.9
84.9
Base
86.9
6.5
7.0
80,2
78.7
13.7
14.8
83.9
Counter
Attr
89.1
4.6
4.9
83.0
79.7
13.0
14.0
84.3
Instr
86.2
6.3
6.8
80.1
78.0
12.8
14.1
82.9
Opin
90.1
3.7
3.9
84.3
79.7
12.8
13.8
84.4
Opin Instr
90.9
2.8
3,0
85.2
80.0
10.5
11.6
85.1
Table 1: Results (in %) on GPT-3.5-1 15BB in the knowledge conflict setting. The overall best results highlighted in bold. The best and the second best results in each setting are highlighted in green and orange respectively.
MRC
RE
LLama-2
Ps
Po 4
MR 4
EMT
Ps f
Po 4
MR 4
Pit
Base
50.8
40.9
3.5
15.3
67.6
81.6
Zero-sho shot
66.2
23.8
13.2
83.5
Instr
77.7
19.7
27.0
19.6
75.1
Opin
74.6
14.6
16.4
20.7
75.4
Opin Instr
77.8
13.9
15.1
13.7
21.6
57.9
72.8
Base
56.7
39.7
41.1
19.4
27.6
62.3
69.4
9.4
Original
61.7
34.5
35.9
25.2
29.4
58.9
66.7
11.2
Instr
59.4
35.7
37.5
25.5
34.6
53.6
60.8
13.2
Opin
67.1
32.1
32.4
18.5
32.2
57.1
63.9
10.9
Opin Instr
70.6
26.8
27.5
27.6
35.7
51.3
59.0
11.5
Base
84.4
7.8
8.4
39.2
76.3
14.8
16.2
38.9
Counter
Attr
85.9
7.0
7.6
44.1
76.5
14.2
15.7
39.5
Instr
85.5
6.7
7.3
47.1
76.0
14.4
15.9
37.3
Opin
86.7
6.2
6.7
38.1
76.3
13.8
15,4
41.7
Opin
Instr
88.1
4.9
5.2
49.6
77.3
14.2
15.5
36.9
Table 2: Results (in %) on LLama-2-77 2-7-c-h-7B-chat in knowledge conflict setting. The overall best results are highlighted in bold. The best and the second best results in each setting are highlighted in green and orange respectively.
few-shot settings (with demonstrations).
and replace the gold entity answer in the context with a randomly sampled entity of the same entity type from the corpus. For RE, we first randomly sample a context that has the entity me of the same type but different relations from the original one, and then insert the original entities into the sampled context. In this scenario, faithful LLM should update its prediction to the new answer in- stead of returning the original one. Moreover, measure LLMs' ability update answers, need to ensure that they have memorized the knowledge of the original answers in the first place. Therefore, we only evaluate LLMs on a subset of instances on which these models can correctly predict the
4.2 Knowledge Conflict
Datasets. We evaluate in the knowledge conflict setting using counterfactual datasets that contain in- correct facts, which can conflict with what the LLM has memorized. We use two datasets based on real-world texts: natural questions (K Kkiatkowski et al., 2019) for MRC and Re-TACRED (Stoica al., 2021) for relation extraction (RE). To create counterfactuals, adopt the framework proposed by Longpre et al. (2021), which modifies the con- text to support a counterfactual answer. Specifi- cally, for MRC, we follow Longpre et al. (2021)

---

original answers without additional contexts.
always improve the generalization of LLMs' infer- ence, especially when they contain dataset bias. In the MRC experiments, the natural questions dataset used is constructed based on Wikipedia, which mainly consists of world facts. This poten- tially allows for simplicity bias of LLMs where questions can be answered without contexts. There- fore, our study suggests the importance of using counterfactual demonstrations in knowledge con- flict scenarios.
Task setup. We use the same set of evaluation metrics as Longpre et al. (2021). Specifically, measure the frequency that the LLMs' predictions contain an exact match of the original answers (po) and the substituted answers (px), after both predictions and answers have been normalized by removing stop words and punctuation To assess the model's reluctance to update its prediction, we use the memorization ratio (M,) which is cal- Po A completely faithful LLM should have an MR of O. We also report task-specific metrics, including exact match (EM) for MRC and F1 for RE. For EM, we also use normalized predictions and answers, but the re- quirement is that the prediction and answer must be exactly the same, rather than just containing the answer. We conduct experiments in three different settings: zero-shot, demonstration using original instances, and demonstration using counterfactual instances. We retrieve demonstrations from the original/countteraatuual training set, and evaluate LLMs on the counterfactual test set. In the few- shot setting, we utilize a maximum of 16 demon- stration instances, up to the limit of the LLM's context window.
culated MR- Potp.
4.3 Prediction with Abstention
Datasets. As for the second aspect of faithful- ness, we evaluate LLMs' ability to selectively ab- stain from making uncertain predictions based on irrelevant context. Since existing datasets such as SQuAD 2.0 (Rajpurkar et al., 2018) generally contain questions with confusion (Yatskar, 2019) and are less related to our problem setting, we cu- rate our own evaluation data based on RealTime QA (Kasai et al., 2022), a dataset that inquires about novel information from June 2022 onwards. In this formulation, LLMs are presented with a question and multiple choices, and they need to choose the correct answer based on several re- trieved documents. These documents were ob- tained using tools like Google custom search and may not contain the answer to the question. To adapt this dataset to our setting, we added new "I don't know" choice and relabeled the dataset. Instances where the retrieved documents do not an- swer the question are relabeled "I don't know". We used questions in the first six weeks of 2022 as the test set and randomly picked three questions of 2023 as demonstration instances. This process results in a total of 1 13 test instances, including 63 answerable questions and 50 unanswerable ones.
Results and discussion. The results Tab. I and Tab. demonstrate that the combination of OPIN + INSTR prompting and counterfactual demonstra- tions is generally the most effective. Compared to the zero-shot base prompts, there is reduction of 32.2% in MR for MRC and 10.9% reduction for RE on GPT-3.5. Similarly, on LLaMA-2-7B- chat, there is a 39.4% reduction in MR for MRC and a 57.3% reduction for RE. We also find that opinion-based prompts generally perform better than other templates, achieving the second-best re- sults on 17 out of 24 metrics on GPT-3.5, and 9 out of 24 metrics on LLama-2, indicating that LLMs are more faithful to the context when answering opinion-seeking questions. Combining opinion- based prompts and instruction-based prompts fur- ther improves faithfulness, with the best results obtained in 23 out of 24 metrics on GPT-3.5, and 19 out of 24 metries on LLama-2.
Task setup. We calculate the probability of choice as P[cchoiee/pompp followed by normal- ization across all choices.* 4 We report accuracy on the entire dataset (All), accuracy on the subset of questions that can be answered based on retrieved documents (HasAns), and accuracy on questions that cannot be answered based on retrieved docu- ments (NoAns). The latter two metrics measure L.Ms' ability extract answers from context and
When it comes to few-shot settings, counterfac- tual demonstrations lead further improved per- formance. Using the original (factual) instances as demonstrations, on the other hand, leads to limited effects or may even impair faithfulness in MRC. This finding suggests that demonstrations do not
"We tried three methods to calculate (cchoicepprompt joint probability, per-token probability (joint probability nor- malized by length), and unconditional probability as done in Brown et al. (2020). We find that joint probability works the best for GPT-3.5, while per-token probability works the hest for Lmma-22

---

GPT-3.5
LLama-2
Method
Acct
Acct NoAns
Brierl
Brier
NoAns
HasAns
All
Base
30.6
68.5
29.4
88.7
14.3
55.9
30.0
Zero-so shot
Attr
65.3
84.4
14.6
87.1
16.3
55.9
29.7
Instr
81.6
91.7
7.7
91.9
26.5
63.1
27.4
Opin
83.3
92.6
6.6
85.5
30.6
61.3
27.8
Opin Instr
87.8
94.4
5.2
88.7
36.7
65.8
27.4
Basc
73.5
88.2
11.2
56.5
69.4
62.2
27.9
Few- shot
Attr
81.6
91.9
8.0
59.7
67.3
63.1
27.0
Instr
85.7
93.7
6.1
51.6
81.6
64.9
26.6
Opin
87.8
94.6
4.1
50.0
87.8
66.6
26.6
Opin Instr
89.8
95.5
3.4
43.5
91.8
64.9
26.0
Table Results (in %) for GPT-3.5 LLama-2 on RealTime QA. The overall best results are highlighted in bold. The best and the second best results in each setting highlighted in green and orange respectively. As all prompts achieve perfect accuracy (100%) the HasAns subset for GPT-3.5, is not included in the table.
their ability to abstain from making predictions when the context does not describe the answer, re- spectively. Besides, we use the probability of "I don't know" as LLM's probability estimation of whether the question can be answered. We use the Brier score to evaluate the accuracy of the estima- tion, which measures the mean squared difference between the estimation and the true binary outcome of answerability. We use three demonstrations for each instance in the few-shot setting, where some instances are filtered out during evaluation due to exceeding the context length of LLMs.
sizes of InstructGPT's under the zero-shot evalua- tion of natural questions. 5 Overall, OPIN + INSTR consistently outperforms other prompts across dif- ferent model sizes. In the upper plot, results are shown for filtered evaluation sets where the cor- responding LLMs can correctly predict the orig- inal answers without additional contexts, thereof the size of evaluation sets varies across different LLMs. 6 We observe that M generally decreases with increased model size, showing that larger LLMs are better at updating memorized answers based on given contexts in knowledge conflicts. However, the lower plot reveals that larger LLMs have more severe memorization on the full (unfil- tered) evaluation set. This is because larger LLMs can memorize more answers than smaller ones, evidenced by the number of instances in the fil- tered evaluation set where larger LLMs have more instances. Our analysis suggests that while larger LLMs are better at updating memorized answers, they still tend to have more memorization due to the larger number of memorized answers. There- fore, we need to pay more attention when using larger LLMss in scenarios with new potentially conflicting knowledge.
Results and discussion. The results detailed Tab. reveal that the OPIN + INSTR prompt outperforms all others on GPT-3.5, both in the zero- shot and few-shot settings, surpassing base prompts by 57 97.2% and 16.3% in NoAns subset accuracy, re- spectively. For LLama-2, this approach similarly outperforms base prompts by 22.4% in both set- tings. Furthermore, the Brier score is reduced by 24.2% and 7.8% compared base prompts for GPT-3.5 in the two settings, respectively, and by 2.6% and 1.9% on LLama-2. The OPIN prompt is the second best in terms of these metrics. These findings demonstrate that opinion-bass based prompts can enhance the L.MMs' ability to make selective predictions. In addition, The use of demonstrations consistently improves the LLMs' ability make se- lective predictions, as evidenced by the lower Brier scores and higher NoAns accuracy in the few-shot setting compared to the zero-shot setting.
Selective prediction by different sizes of LLMs. Fig. shows the Brier score across different sizes of InstruetGPTs under the zero-shot evaluation of RealTime QA. On smaller LLMs, opinion- prompt achieves similar or even higher Brier score
'The 0.3B, 1.3B, 6.7B models refer text-ada-001,, text- babbage-001, text-curie-000 respectively. do not perform few-shot evaluation as different sizes LLMs have different maximum input lengths and can take different numbers of demonstrations, thus hard
4.4 Additional Analysis
compared to each other.
Memorization by different sizes of LLMs. Fig. shows the memorization ratio M across different
"The sizes of the filtered evaluation sets, in the order of increased model sizes, are 121, 132, 756, and 2,773.

---

Base
Instr
Opin+Instr
Base
Instr
Opin+ Instr
Attr
Opin
Attr
Opin
Filtered Evaluation Set
RealTime QA
50
30
40
20
30
20
10
Full Evaluation Set
0
0.3B
1.3B
6.7B
175B
30
Model size
Me
20
Figure Brier scores across different sizes of Instruct- GPTs in the zero-shot setting of RealTime QA.
10
0.3B
1.3B
6.7B
175B
Model size
Method
Pet
EM1
Counter Original Zero-sho shot
Base
92.1
11.1
Figure 2: Memorization ratios across different sizes of InstructGPTs, evaluated in the zero-shot setting.
Opin
91.3
25.2
Opin + Instr
90.5
57.2
Base
93.2
77.8
Opin
92.7
78.7
than base prompts, indicating does not improve the selective prediction ability of LLMs. We hy- pothesize that this because smaller LLMs have inferior reading comprehension ability, resulting in uncertainty in many instances. Opinion- based prompts change uncertain predictions of answer- able questions to don't II know, which could lead worse results. For other prompting templates, we do not observe a consistent improvement across different LLMs either. This analysis shows that while the selective prediction ability can be more easily activated by zero-shot prompting for LLMs such as fext-davinci-003, smaller LLMs may re- quire dedicated adaptations such as calibration and finetuning to activate this ability.
Opin Instr
93.9
80.1
Base
93.6
Opin
92.8
82.3
Opin Instr
92.7
82.1
Table 4: Results (in %) on the filtered evaluation set of natural questions with original contexts and answers.
reveals that our methods do not impair performance on instances without knowledge conflicts.
4.5 Case Study
Tab. 5 shows some examples prompts and the corresponding answers generated by text-davinci- 003. The left column of the table presents a knowl- edge conflict case where the original answer, Lady Gaga, is replaced with counterfactual answer, Bosco. When using base prompts, LLM ignores the context and return the memorized answer Lady Gaga. However, using opinion- based prompts and their combination with instructions leads to a more faithful response, with the language model return- ing Bosco in the given context. The right column presents a scenario where the retrieved context from Google search is irrelevant to the given ques- tion. In such cases, base prompts still return choice, leading potentially incorrect answer. However, opinion-based prompts and their combi- nation with instructions can abstain from making predictions and return / don't know. These ex- amples demonstrate the effectiveness proposed
Results on original datasets. While our main ex- periments demonstrate the effectiveness of the pro- posed methods in resolving knowledge conflicts, LLMs in real-world applications may also see in- stances without knowledge conflicts. Therefore, we investigate how our methods affect inference when the memorized answers align with the given contexts. To do so, we evaluate LLMs on the same set of filtered evaluation set used in the main re- sults section (64.2), but we use the original contexts and answers instead of counterfactual ones. The results in Tab. 4 show that opinion-based prompts yield similar better results in all settings. Fur- thermore, using either counterfactual or original demonstrations does not significantly impact re- sults on the original (factual) dataset. This analysis

---

Knowledge Conflict
Prediction with Abstention
Context
(Counterfactual passage) The Super Bowl LI Half- time show look place on February 5, 2017, at NRG Stadium in Houston, Texas as part of Super Bowl The show was headlined by Bosco, who performed a medley of her songs, including newer material from most recent studio alhum Joanne.
Tara Counolly is semior gas campaigner at Global Witness, an international NGO working towards more sustainable, just and equal planet. She has over a decade of experience in EU energy policy. The views expressed in this commentary her own.
Prompt
Instruction: read the given information and answer the corresponding question.
Instruction: answer question based on the provided input-output pairs.
Bob said, "The Super Bowl ... album Joanne." Q: who performed the halftime show at Super Bowl 51 in Bob's opinion based on the given text?
Bob said, "Tara Comnolly are her own."
Q: Mo Farah made public that he was trafficked from which African country to the UK in Bob's opinion
based the given text?
Choices: Somaliland; Djibouti; Ethiopia: Somalia; I
know
Base
Lady Gaga X
Somalia
Attr
Lady Gaga X
Somalia X
Instr
Lady Gaga X
Somaliland X
Opin
Bosco /
I don't 't know / I don't know /
Instr
Opin
Bosco /
Answer
Bosco
I don't know
Table 5: Examples of prompts and LLMs' corresponding predictions. In the "Prompt" row, we show and highlight the added parts from different prompting templates including attributed prompts, instruction- based prompts, and opinion-bass based prompts.
prompts in generating context-faithful responses.
Limitations
5 Conclusion
In this study, our main focus is on the utilization of context-augmented prompting, assuming the relia- bility of the provided context. However, real-world scenarios can be more complicated, which may involve retrieved contexts that contain erroneous or conficcing information. Assessing the factu- ality of the context solely based on the provided information becomes challenging. as it depends on additional factors such as trustworthiness and timeliness of the information source. Due to the complexity and challenges associated with veri- fying context reliability, we do not address this issue within the scope of this work. Furthermore, is important to note that paper primarily con- centrates on the capability of LLMs to generate updated answers or decisions for given questions, rather than exploring more intricate tasks that re- quire the model to apply the updated knowledge in multi-hop reasoning.
In this paper, focus on addressing the faithful- ness issue of LLMs in context-specific NLP tasks, particularly in scenarios with knowledge conflict and prediction with abstention. We propose that two methods, opinion-based prompts and counter- factual demonstrations, are effective in improving LLMs' faithfulness to contexts. We evaluate our methods on three datasets of two tasks, namely machine reading comprehension and relation ex- traction, and observed significant improvement in faithfulness contexts. Future work includes eval- uating the effectiveness of proposed methods on a broader range of NLP tasks such as
QA and summarization, and studying other tech- niques to improve faithfulness further.
Acknowledgement
We appreciate the reviewers for their insightful comments and suggestions. Wenxuan Zhou and Muhao Chen are supported by the NSF Grant IIS 2105329 and the DARPA MCS program under Con- tract No. N66001 1924033 with the United States Office Of Naval Research.
Ethical Considerations
Due to the availability of test data, the experiments conducted in this work has been English, while future work can consider extending the use of pro- posed techniques to tasks in other languages. The

---

Hyung Won Chung. Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. language models. datasets used in this work are public datasets that may not be free of inherent biases. However, the in- troduced context-faithfu prompting techniques in this work do not introduce additional biases beyond what the data have presented.
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. 2016. Learning with rejection. In Algorithmii Learn- ing Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27, pages 67-82. Springer.
Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Edit- ing factual knowledge in language models. In Pro- ceedings of the 2021 Conference Empirical Meth- ods in Natural Language Processing, pages 6491 6506, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
References
Johannes Bjerva, Nikita Bhutani, Behzad Golshan, Wang-Chiew Tan, and Isabelle Augenstein. 2020. SubjQA: A Dataset for Subjectivity and Review Com- prehension. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 5480-5494, Online. Association for Computational Linguistics.
Giorgio Fumera and Fabio Roli. 2002. Support vector machines with embedded reject option. In Pattern Recognition with Support Vector Machines: First International Workshop, SVM 2002 Niagara Falls, Canada, August 10, 2002 Proceedings, pages 68-82. Springer.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 1311
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncer- tainty in deep learning. In international conference on machine learning, pages 1050-1059. PMLR.
Luyu Gao and Jamie Callan. 2022. Unsupervised cor- pus aware language model pre-training for dense pas- sage retrieval. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 2843-2853, Dublin, Ireland. Association for Computational Lin- guistics.
Eunsol Choi, He He, Mohit lyyer, Mark Yatskar, Wen- tau Yih, Yejin Choi, Percy Liang, and Luke Zettle- moyer. 2018. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 21 184, Brussels, Belgium. Association for Com- putational Linguistics.
Mansi Gupta, Nitish Kulkarni, Raghuveer Chanda, Anirudha Rayasam, and Zachary C Lipton. 2019. Amazonqa: A review-based question answering task. In International Joint Conference on Artificial Intelli- gence.
C CChow, 1970. On optimum recognition error and reject tradeoff. IEEE Transactions on information theory, 16(1):41-46
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Dan Hendrycks and Kevin Gimpel. 2017. A baseline for detecting misclassified and out-of -itoo---istrbbttion exam- ples in neural networks. In International Conference on Learning Representations.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020. Pretrained transformers improve -ut-o--distriuution robustness. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 2744-2751, Online. Association for Computa- tional Linguistics.
2022. Scaling
arXiv preprint arXiv:2210.11411
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pages 2924-2936, Minneapolis, Min- nesota. Association for Computational Linguistics.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lu- cas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave 2022. Few-shot learning with trieval augmented language models. arXiv preprint arXiv:2208.00 032999.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Vol- une I: Long Papers), pages 1601-1611, Vancouver. Canada. Association for Computational Linguistics.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot. Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018 Think you have solved question an- swering? try are, the ai2 reasoning challenge. arXiv preprint arxiv: 1803.05444 05457.

---

Vladimir Karpukhin, Barlas Oguz. Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov. Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Adam Liska, Tomas Kocisky, Elena Gribovskaya, Tay- fun Terzi, Eren Sezener. Devang Agrawal, D' Autume Cyprien De Masson, Tim Scholtes, Manzil Zaheer, Susamnah Young, al. 2022. Streamingqa: bench- mark for adaptation to new knowledge over time in question answering models. In International Con- ference on Machine Learning, pages 13604-13622. PMLR.
Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi. Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. 2022. Realtime qa: What's the answer right now? arXiv preprint 2xxv:2207.13322
Jiachang Liu, Dinghan Shen, Yizhe Zhang. Bill Dolan,
Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100-1 114, Dublin, Ireland and Online. Associa- tion for Computational Linguistics.
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang. Christopher Potts, and Matei Zaharia. 2022. Demoostrate-search predict: Composing retrieval and language mod- cls for inneeedge-ittennivee nlp. arXiv preprint arXiv:2212.14024.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.11692.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-W Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics, 7:452-466.
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 7052-7063, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable pre- dictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30.
Keming Lu, I-Hung Hsu, Wenxuan Zhou, Mingyu Derek Ma, and Muhao Chen. 2022. Summarization as indirect supervision for relation extraction. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6575-6594, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Angeliki Lazaridou, Elena Gribovskaya, Wojeiech Stokowiec, and Nikolai Grigorev. 2022. Internet- augmented language models through few-shot prompting for open-domain question answering. arXiv preprini arXiv:2203.05715.
Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. 2022. Locating and editing factual asso- ciations gpt. In Advances in Neural Information Processing Systems.
Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez. Cyprien de Masson d' Autumme Tomas Ko- cisky, Sebastian Ruder, et al. 2021. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural Information Processing Syxtems, 34:29348-29363.
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2023. Mass- editing memory in a transformer. In International Conference on Learning Representations.
Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. 2022. Large language models with controllable working memory. arXiv preprint arxiv:2211.05110.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can suit of armor conduct elec- tricity? a new dataset for open book question an- swering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2397 Brussels, Belgium. Association for Computational Linguistics.
Stephanic Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214-3252, Dublin, Ireland. Association for Computational Linguistics.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2022. Fast model editing at scale. In International Conference on Learning Representations.

---

Eila Neeman, Roce Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. 2022. Disentqa: Disentangling parametric and contextual knowledge with counterfactual question answering. arXiv preprint 3.XVv:2211006655.
Bhosale, 2023. Llama 2: Open founda- tion and tune-tuned chat models. arXiv preprint arXiv:2307.099 09288.988.
Haoyu Wang, Hongming Zhang, Yuqian Deng, Jacob Gardner, Dan Roth, and Muhao Chen. 2023. Extract- ing or guessing? improving faithfulness of event tem- poral relation extraction. In Proceedings of the 17th Conference of the European Chapter of the Associa- tion for Compulational Linguistics, pages 541-553, Dubrovnik, Croatia. Association for Computational Linguistics.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't t know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics VVolume 2: Short Papers), pages 784-789. Melbourne, Australia. Association for Computational Linguistics.
Yiwei Wang, Muhao Chen, Wenxuan Zhou, Yujun Cai, Yuxuan Liang, Dayiheng Liu. Baosong Yang, Juncheng Liu, and Bryan Hooi. 2022. Should we rely on entity mentions for relation extraction? debi- asing relation extraction with counterfactual analysis. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies. pages 3071-3081, Seattle, United States. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Pency Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguisties.
Siva Reddy, Dangi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Com- putational Linguistics. 2-2492266.
Jason Wei, Maarten Bosma, Vincent Zhao Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. 2022. Finetuned language mod- els are zero-shot learnrrs In International Confer- ence on Learning Representations.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing JEMNLP-IJCNLLP pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. 2021. The art of abstention: Selective prediction and error regularization for natural language processing. In Proceedings of the 59th Annuul Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 1040-1051, Online. Association for Computational Linguistics.
Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-inddependent named entity recognition. arXiv preprint cs/0306050.
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. Col- BERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 3715-3734 Seat- tle, United States. Association for Computational Linguistics.
Mark Yaıskar. 2019. A qualitative comparison of CoQA, SQuAD 2.0 and QuAC. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Popers). pages 23833333, Minneapolis, Minnesota. Association for Computational Linguistics.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 35-45, Copenhagen, Denmark. Association for Com- putational Linguistics.
Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graher, and Li- juan Wang. 2023. Prompting gpt-3 to be reliable. In International Conference on Learning Representa- tions.
George Stoica, Emmanouil Antonios Platanios, and Barnaáás Páczos. 2021. Re-tacred: Addressing short- comings of the tacred dataset. In Proceedings of the AAA/ Conference on Artificial Intelligence, vol- ume 35, pages 13843-13850.
Wenxuan Zhou and Muhao Chen. 2022. An improved baseline for sentence-level relation extraction. In Pro- ceedings of the 2nd Conference of the Asia-Pacifee Chapter of the Association for Computational Lin- guistics and the 12th International Joint Conference Natural Language Processing (Volume 2: Short Papers), pages 161-168, Online only. Association for Computational Linguistics.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti

---

Wenxuan Zhou, Fangyu Liu, and Muhao Chen. 2021. Contrastive citto--distrbbution detection for pre- trained transformers. In Proceedings of the 2021 Conference on Empirical Methods Natural Lan- guage Processing, pages 1100-1111, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
RealTime QA:
1. answer question based the provided input- output pairs.
2. ask question with set of choices and ask the friend to provide the correct answer.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910
3. answer question related to news article.
Chen Zhu, Ankit Singh Rawat. Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. 2020. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363.
Appendices
A Settings of Automatic Prompt Engineering
We APE using their official code? and default hyperparameters. In the knowledge conflict setting, we use counterfactual datasets to generate instruc- tions. While the APE paper recommends using instructions generated by the same model infer- ence. we find that smaller LLMs do not generate meaningful instructions for our datasets. Therefore, we use instructions generated by text-davinci-00s across different scales of LLMs in additional anal- ysis. The top three instructions generated by APE on each dataset are listed below. We use the top one instruction in experiments.
Natural questions:
1. read the given information and answer the corresponding question.
2. read a piece of text and then use the informa- tion in the to answer a question.
3. "Read the given information and answer the questions that follow."
Re-TACRED:
1. identify the relationship between two entities from list of options.
2. identify the relationship between two entities based on the given input-output pairs.
3. identify the relationship berween two entities given the input-output pairs.
http:///w..ceeeeeee////tmttipppp engineer


Reviews:
[{"review_id": "U501xCztL6", "reviewer": "Reviewer_8jFq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We agree that evaluating the proposed methods on a more diverse set of NLP tasks could be beneficial. The choice of the current datasets was guided by the availability and relevance of our focus of study. Specifically, we chose MRC as a widely evaluated task in previous studies on knowledge conflict study, and relation extraction as another representative context-specific NLU task. In light of your feedback, we are willing to extending our evaluations to more tasks, such as text summarization and machine translation, upon releasing of the complete version of our code.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper addresses a pressing concern in the NLP community regarding the contextual faithfulness of Large Language Models. ", "reasons_to_reject": "While the paper presents experiments on three datasets. It would be valuable to see a broader evaluation on a more diverse set of NLP tasks to evaluate the generalizability of the proposed methods.", "questions_to_authors": "", "paper_topic_and_main_contributions": "LLMs‘ reliance on parametric knowledge can lead to incorrect predictions in context-sensitive NLP tasks. The authors propose two methods, namely opinion-based prompts and counterfactual demonstrations, to assess and enhance LLMs' contextual faithfulness. These methods are can improve the models' ability to handle knowledge conflict without additional training. The experiments conducted on three datasets for two standard NLP tasks show promising results.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UAU6CAdcx7", "reviewer": "Reviewer_FEGa", "summary": "", "questions": "Question A: More LLMs (i.e., ChatGPT or Alpaca) should be used to validate the effectiveness of the proposed tailored prompt method for convincing the readers. \n\nQuestion B:  More ablation studies should be performed on the in-context learning demonstration number as some studies have proof that the large demonstration number may not receive a good result across various tasks [1-3].\n\n[1] Li, M., Gong, S., Feng, J., Xu, Y., Zhang, J., Wu, Z., & Kong, L. (2023). In-context learning with many demonstration examples. \n\n[2] Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022). Rethinking the role of demonstrations: What makes in-context learning work?\n\n[3] Chan, C., Cheng, J., Wang, W., Jiang, Y., Fang, T., Liu, X., & Song, Y. (2023). Chatgpt evaluation on sentence level relations: A focus on temporal, causal, and discourse relations. ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank the reviewer for their thoughtful and constructive review of our paper.\n\nQuestion A: We appreciate your suggestion to validate the effectiveness of our tailored prompt method using additional LLMs such as ChatGPT or Alpaca. We will conduct experiments with these models to provide a more comprehensive evaluation and ensure the broader applicability of our approach in revision.\n\nQuestion B: Thank you for raising the concern about the number of in-context learning demonstrations. In our paper, we use the maximum number of in-context examples that fit in the context length, resulting in 16 for knowledge conflict and 3 for prediction with abstention. We agree on the importance of ablation studies in fine-tuning this parameter. We will thoroughly investigate the optimal number of demonstrations in revision, considering the insights provided references.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper investigates a critical and interesting issues of LLM (i.e., faithfulness issue) and focuses on two aspects: knowledge conflict and prediction with abstention. This faithfulness issue from LLM may interest the community.\n* This paper proposes two methods and receives a promising performance across three NLP tasks.\n* The motivations of the paper relative to those elements are fair and well explained.\n", "reasons_to_reject": "*  More LLMs (i.e., ChatGPT or Alpaca) should be used to validate the effectiveness of the proposed tailored prompt method for convincing the readers. \n*  More ablation studies should be performed on the in-context learning demonstration number as some studies have proof that the large demonstration number may not receive a good result across various tasks [1-3].\n\n[1] Li, M., Gong, S., Feng, J., Xu, Y., Zhang, J., Wu, Z., & Kong, L. (2023). In-context learning with many demonstration examples. \n\n[2] Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022). Rethinking the role of demonstrations: What makes in-context learning work?\n\n[3] Chan, C., Cheng, J., Wang, W., Jiang, Y., Fang, T., Liu, X., & Song, Y. (2023). Chatgpt evaluation on sentence level relations: A focus on temporal, causal, and discourse relations. ", "questions_to_authors": "Question A: More LLMs (i.e., ChatGPT or Alpaca) should be used to validate the effectiveness of the proposed tailored prompt method for convincing the readers. \n\nQuestion B:  More ablation studies should be performed on the in-context learning demonstration number as some studies have proof that the large demonstration number may not receive a good result across various tasks [1-3].\n\n[1] Li, M., Gong, S., Feng, J., Xu, Y., Zhang, J., Wu, Z., & Kong, L. (2023). In-context learning with many demonstration examples. \n\n[2] Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022). Rethinking the role of demonstrations: What makes in-context learning work?\n\n[3] Chan, C., Cheng, J., Wang, W., Jiang, Y., Fang, T., Liu, X., & Song, Y. (2023). Chatgpt evaluation on sentence level relations: A focus on temporal, causal, and discourse relations. ", "paper_topic_and_main_contributions": "This paper focus on resolving the faithfulness issue of the Large Language Model in context specific NLP tasks. The LLM faithfulness is concerned with the validity and reliability of its extraction process, and this work assesses and enhances LLMs’ contextual faithfulness in two aspects: knowledge conflict and prediction with abstention. The research proposes two approaches which are opinion-based prompts and counter-factual demonstrations, are exhibited effectiveness in enhancing LLMs’ faithfulness to contexts. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}, {"review_id": "sfFkx4u5NP", "reviewer": "Reviewer_RhhF", "summary": "", "questions": "Why only utilize the InstrucGPT family of models and not something else?\nWhy not expand on the use of auto prompting to discover a stronger baseline?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough review and valuable insights into our work.\n\n**Use of Other Language Models**:\n\nWe acknowledge the importance of considering multiple baselines and using different language models to validate our approach's effectiveness. We will add results on other LMs such as T5, Alpaca and ChatGPT.\n\n**Full Auto Engineered Prompt Baseline**:\n\nYour recommendation to include a fully auto-engineered prompt as a baseline is valid. However, it's important to note that the current auto-prompting techniques we use only produce instructions based on given input-output pairs. At present, we are not aware of auto-prompting techniques that can also modify context and queries. Still, we value this feedback and will ensure to provide a more detailed discussion regarding this in our revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed methods are simple (the prompts can be easily formulated), and the results indicate that the formulations can force the models to rely more on the context that they are given rather than the prior knowledge that they may have. These methods are crucial as they enable the models to rely more on context, rather than requiring retraining or extensive fine-tuning, making them more cost-effective and independent of specific models. The paper is also well written and easy to follow.", "reasons_to_reject": "While the prompting strategy seems effective, the work seems to be missing more baselines\n* If the authors are testing this in large language models, I would use other publicly available models such as LLaMa or T5 etc. to verify that the prompting strategies work across different types of training data and are not exclusive to the result of the InstructGPT models\n* I would also add a baseline on a fully auto engineered prompt. The authors mention in line 274, that they use auto prompt engineering. I would put another baseline in which the full prompt save for {c}, {q} is auto engineered and just check the results of this. Because while the authors formulate that the instruction/opinion prompts provide improved context usage and their combination does too, it may be that some other style of prompting that is discovered with the auto prompt engineering may be even better/serve as a strong baseline.", "questions_to_authors": "Why only utilize the InstrucGPT family of models and not something else?\nWhy not expand on the use of auto prompting to discover a stronger baseline?", "paper_topic_and_main_contributions": "The authors provide a detailed analysis of utilizing instruction based and opinion-based formulation prompts for the purpose of contextual question answering. The prompts that the authors propose are simple and can be utlizied in various language models. More importantly, the authors demonstrate that this style of prompting tends to force the model to leverage the context given rather than recall the answer from training data.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "None that I could find", "Justification_for_Ethical_Concerns": "n/a"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree this work tackles an important problem, proposes interesting new methods, and achieves promising results. The authors promise to add new experiments that address the primary concerns, primarily around the scope of models and datasets, which would extend the generalizability of the conclusions, and would improve this work's appeal as a contribution worthy of greater discussion in the community.
Decision: Accept-Findings