Self-Knowledge Guided Retrieval Augmentation for Large Language Models
Authors: Yile Wang
Peng Li
Maosong Sun
Yang Liu
Venue: EMNLP 2023 Findings (2023)
Keywords: self-knowledge
retrieval augmentation
large language models

Abstract: Large language models (LLMs) have shown superior performance without task-specific fine-tuning. Despite the success, the knowledge stored in the parameters of LLMs could still be incomplete and difficult to update due to the computational costs. As complementary, retrieval-based methods can offer non-parametric world knowledge and improve the performance on tasks such as question answering. However, we find that the retrieved knowledge does not always help and even has a negative impact on original responses occasionally. To better make use of both internal knowledge and external world knowledge, we investigate eliciting the model's ability to recognize what they know and do not know (which is also called "self-knowledge'') and propose Self-Knowledge guided Retrieval augmentation (SKR), a simple yet effective method which can let LLMs refer to the questions they have previously encountered and adaptively call for external resources when dealing with new questions. We evaluate SKR on multiple datasets and demonstrate that it outperforms chain-of-thought based and fully retrieval-based methods by using either InstructGPT or ChatGPT.

Self-Knowledge Guided Retrieval Augmentation for Large Language Models
Yile Wang', Peng Li Maosong Sun?, Yang 1,23
Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China Shanghai Artificial Intelligence Laboratory, Shanghai, China (wangyile, lipeng]aii.. tsinghua. edu. cn, (sms, liuyang2011 feteoo11eeiiiin jetsinghua.n.... edu. cn
Abstract
Question (Answer: Yes)
Would German Shepherd
Yes. German Shepherds are often used seeing-eye dogs,
Large language models (LLMs) have shown su- perior performance without speccfcc fine- tuning. Despite the success, the knowledge stored in the parameters of LLMs could still incomplete difficult to update due the computational costs. As complemen- tary. retrieval-based methods can offer non- parametric world knowledge and improve the performance tasks such question answer- ing. However, we find that the retrieved knowl- edge does not always help and even has negative impact on original responses occa- sionally. To better make use of both inter- nal knowledge and external world knowledge, we investigate eliciting the model's ability to recognize what they know and do not know (which is also called "self- sefff-knowledge) and propose Self-K noffeKno guided Retrieval aug- mentation (SKR), simple yet effective method which can let LLMs refer to the questions they have previously encountered and adap- tively call for external resources when deal- ing with new questions. We evaluate SKR on multiple datasets and demonstrate that outperforms chain-of-thoughh based and fully retrieval-basse methods by using either In- structGPT or ChatGPT. Code released at https://githhu wtt.://///////////eeee PMM//SKR.
welcome an airport?
Passages Old German Shepherds Dog is controversial name for...
No. Airports very regulations regarding animals.
Comparison between two responses given by InstructGPT. The retrieved passages are relevant but particularly helpful for solving the question, which influences the model's judgment and leads to incorrect answers.
Figure I:
information, thus helping the model generate more reliable answers.
Retricval augmentation has shown to be very effective for models such as BERT (Devlin et al., 2019), BART (Lewis et al., 2020a). and T5 (Raf- fel et al., 2020) in various tasks (Karpukhin et al., 2020; Khandelwal et al., 2020, 2021; Izacard and Grave, 2021; Wang al., 2022; Guo et al., 2023). As LLMs become more and more "knowledgable", recent studies show that the benefit brought from retrieval augmentation is reducing (Mallen et al., 2022; Yoran et al., 2023). Moreover, we find that the retrieved passages could even negatively af- feet what LLMs originally know. As illustrated in Figure 1, the model can directly give reason- able answers "German Shepherds are often used as seeing-eye dogs", however, it is distracted and gives incorrect ones by adding retrieved passages.
1 Introduction
Large language models (LLMs, Brown al., 2020;
Chowdhery et al., 2022: Ouyang et al., 2022) have achieved remarkable performance without much task-specific fine-tuning. However, the full parammerecttic knowledge stored in LLMs could still be incomplete and difficult to update due to the computational costs. Alternatively, retrieval- augmented methods (Guu et al., 2020; Lewis et al., 2020b; Borgeaud et al., 2022: Izacard et al., 2022: Shi et al., 2023) can utilize external resources such as Wikipedia and offer complementary non- parametric knowledge to enrich the contextualized
The above findings show that one should be more careful when applying the catrieval-baaee method since it is difficult know in advance whether the retrieved results are better than what LLMs already captured. To this end, a key issue is to figure out what LLMs do well (e.g.. they can answer correctly without assistance) and what they cannot do well (e.g.. they answer incorrectly and external informa- tion can lead to improved results).
Unfortunately, LLMs themselves have a limited ability to recognize what they know and do not
Corresponding authors.

---

know, which is also called "self-knowledge" (Yin al., 2023). However, such an ability is crucial for generating truthful responses (Kadavath et al., 2022) and could be helpful for LLMs themselves to "decide when and when not to use tools" such as retriever (Mialon et al., 2023).
better reasoning process, while we propose bet- ter utilize both the internal and external knowledge through eliciting the self-knowledge of LLMs.
Another line of work tries to teach LLMs to use external tools including retriever, calculator, other foundation models, etc. (Schick et al., 2023; Shen et al., 2023; Qin al., 2023). These works focus more on leveraging the language understanding capabilities of LLMs to deploy suitable tools in different scenarios, while our work investigates the self-knowledge of LLMs and tries to integrate them with retrievers in more flexible manner.
In this paper, we investigate eliciting the self- knowledge of LLMs and propose a simple yet ef- fective Solf-Knowledge guided Retrieval augmen- tation (SKR) method to flexibly call the retriever for making better use of both internal and external knowledge. In particular, different from existing studies that evaluate the ability through specifically designed metrics datasets, collect the self- knowledge of training questions by comparing the performance with or without retrieval augmenta- tion. Then, we propose several strategies to de- tect the self-knowledge corresponding to a ques- tion by referring to the existing collected training questions, including using the LLMs themselves through prompting or explicitly training a small model. Finally, we leverage such elicited self- knowledge better solve the question through adaptive retrieval augmentation.
knwllgne LLMs "Self-knowledge" LLMs is originally mentioned in Kadavath et al. (2022), which is used to measure the LLMs' confi- dence in their own knowledge and reasoning. Such ability is further defined as "the ability to under- stand limitations on the unknowns" and evaluated by Yin et al. (2023), where they find consider- able gap exists between self-knowledge in mod- els and humans. To explore the LLMs capabilities more extensively, unanswerable and more challeng- ing datasets are also proposed (Rajpurkar et al., 2018; Srivastava et al., 2022: Suzgun et al., 2022). Our work is also related to detecting what LLMs know and do not know, while we do not design new evaluation metrics or challenging datasets test the ability. By explicitly introducing the ex- ternal resources, we detect the knowledge bound- ary of LLMs through the performance changes. Moreover, instead of evaluating each question in- dependently, we propose several ways to elicit self- knowledge by referring to existing cases.
We evaluate SKR on five datasets by us- ing InstructGPT (text-davinci-003) and Chat- GPT 5-ttrboo-turoo--o 0301). Experimental re- sults show that SKR outperforms chain-of-thought based (Wei al., 2022) and fully retrieval based methods by 4.08% (for InstructGPT) and 4.02% (for ChatGPT), respectively.
2 Related Work
Retrieval-Augmentee LLMs Recent studies show that retrieval-augmeened methods can enhance the reasoning ability of LLMs (Trivedi et al., 2022; He et al., 2022; Yu et al., 2023; Shao et al., 2023; Jiang et al., 2023) and make the responses more credible and traceable (Xu et al., 2023b; Qian al., 2023), For example, Trivedi et al. (2022) uses the chain-of- thought (Wei et al., 2022) reasoning steps as queries and uses the results to guide further reasoning and retrieval. He et al. (2022) uses an external natural language inference model to select the most sup- ported reasoning path via retrieved evidence. Yu et al. (2023) propose using the retrieval feedback refine the output of LLMs to be more reliable and accurate. Xu et al. (2023b) propose search-in- chain and make L.Mss interact with retrievers to improve accuracy and credibility. These methods aim at integrating sufficient external knowledge for
Method
Our method is depicted under the question- answering settings, which has been a popular way to interact with and assess LLMs. The overall pipeline is shown in Figure 2, which includes collecting, eliciting, and using self-knowledge of LLMs. We introduce each of them as follows.
3.1 Collecting Self-Knowledge of LLMs from Training Samples
Given dataset D with training answer pairs (qj- az ))) we can use the LLM M to gen- erate the answers for each question via few-shot in-context learning (Brown et al., 2020):
a(M, q.) = -M(qi o o1,..., qa ad, Q.), (1)

---

Collecting Self-Knowledge
Eliciting Self-Knowledge
Using Self-Knowledge
training question şi
question
question qt
unknown
LLM itself small trainable models
known
v.s.
retriever
4n
LLM knowns
LLM unknowns
known? unknown?
The overall pipeline of our SKR method. We first collect self-knowledge from training questions according to the performance with without external information ($ 3.1). Then we use the LLMs themselves or explicit small trainable models to elicit self-knowledge of question @ by referring to the collected self-knowledge from training questions 3.2). Finally, use the self-knowledge to the new question and adaptively call retriever 3.3).
Figure 2:
where o denotes concatenation and {qz ag و are d demonstrations.
D = { ..... 9. where the external information can lead to more accurate results (LLM unknowns).
The above generated answers â(M, q:) reflects the internal knowledge to question % in M. Mean- while, can possibly find passages from external resources that may be related K. such passages can be used as additional information for the model input. Formally, for each question, we first use a pre-trained retriever R to find the possibly related information from corpus C:
3.2 Eliciting Self-Knowledge of LLMs
Four different strategies are proposed to detect the self-knowledge of target questions, including direct prompting, in-context learning, training a classifier, and nearest neighbor search. We use the LLM itself in the former two methods and explicit smaller modes in the latter two methods.
Direct Prompting Given question qu a straight- forward way to detect whether LLMs are capable of solving to ask them directly:
P; (pa, P42, Pik) R(g;s)).
where P; = (pa, PP2, .... ] are the top-k re- trieved passages for q: In practice, we set R as dense passage retriever (Karpukhin et al., 2020) and C as passage chunks from Wikipedia. Then, we use M again generate the answer with re- trieval augmentation:
Direct Prompting
(prompt)
[9.] Do you need additional information onswer this question?
(possible response)
à (M,, = M(giepi oa1, ..., ddoppoad, qopps)
No. I don't need additional information answer this question. Yes, I need additional information answer this question.
Given the answers á(M, q:), a (M, and the ground-truth answer aj we categorize each question into positive subset D and negative sub- set D based on the differences between results:
the prompt "Do you need addi- tional information answer this question?" as a template and detect welf-knowledge according the possible response. We thought LLM is capable (or not capable) of solving the question well when they "don't 't need (or need) additional information". Direct prompting may intuitively work, but it tests each question independently and does not make use of the collected training questions in Section 3. To remedy this issue, we further leverage the col- lected self-knowledge from training questions in the next three strategies.
Here
D B(a 4)) > (MM D otherwise,
(M,q)); 9:)l;
qi c
(4)
where E is an evaluation metric such as accuracy and exact match score, discard the question % if both the â(M, q:) and aR (M, q:) are incorrect. Finally, the training set can be split into subset
D { ..., 2m which includes questions that M can directly give correct answers without ex- ternal information (LLM knowns) and the subset
In-Context Learning LLMs have shown a strong capability to learn from demonstrations and infer

---

through few-shot in-context learning (Brown et al.. 2020). We select few training questions from both D and as demonstrations to elicit the self-
42
4m
Encoder
9
knowledge to the question q::
qt
Training Questions
Pos//Neg.
In-Context Learning
Question
(prompt)
(q")) Q: you need additional information need additional information answer this question.
Figure 3:
Illustration of k-nearest-neighhor search to elicit the self-knowledge the question qt-
onswer this question?
[9r Q: Do you need additional information answer this question? information answer this question.
Yes. need additional
Nearest Neighbor Search Instead of training an explicit smaller model, we can infer the la- bel of questions through k-mcarest-neighhor (ANN) search by using a pre-trained fixed encoder, shown in Figure 3. KNN (Fix and Hodges, 1989) is widely used algorithm and benefit for range of NLP tasks (Khandelwal al., 2020, 2021; Shi et al., 2022; Xu et al., 2023a). Our motivation is similar in that if two questions are close the se- mantically embedded space, then the LLMs would show similar self-knowledge for both of them.
[9e) Q: Do you need additional information to answer this question? A:
(possible response)
No, don't I need additional information to answer this question. Yes, need additional information to answer this question.
Here we use the answer templates "No, I don't need..." " or "Yes, need..." " in demonstrations based on whether the corresponding question comes from positive set D+ or negative set D respectively.
Formally, encode each question into embed- dings and compute the semantic similarity through e(q. )e(g)) cosine distance sim ((t, 9:) [[((a)])]l((]]]] where Qi E ....... Im f1 ..... In ). e(-) is the represen- tations of a sentence encoder such SimCSE (Gao et al., 2021). Then search the top-k nearest neighbors with the highest similarity. If the top-k nearest neighbors include l positive ones and k negative ones, we label the question lt as positive if k 7 2 m or negative if k < m (m and n are the numbers of questions D and D respectively).
The proposed direct prompting and in-context learning methods can elicit self-knowledge of LLMs to some extent. However, they have sev- eral limitations. First, both methods require design- ing prompts and calling the LLMs for each new question, which makes it impractical. Second, in- context learning could also be unstable due to con- textual bias and sensitivity (Zhao et al., 2021; ; Lu et al., 2022) and it is more difficult to address such an issue for close-source LLMs. Third, they cannot make of all questions due the constraints of maximum tokens. To make our method more prac- tical and avoid the above issues, further leverage smaller models to help elicit self-knowledge.
3.3
Using Self- for Adaptive Retrieval Augmentation
Training a Classifier Given + and D we can take them as two-way classification problem (e.g., setting & in + with positive label and qi in D with negative label) and use all the samples to train classifier such BERT-base (Devlin et al., 2019) explicitly:
The self-knowledge given by the responses from LLMs (via direct prompting or in-context learning) or the predicted labels (via the trained classifier or k-nearest-neighhor search) reflects the neces- sity for external knowledge towards the question Gt- Therefore, we can adaptively call the retriever instcad of using them for every question:
fi = softmax(Whhar + b),
(5)
where z E DD is training question, hes((g) is the sentence-level representation from BERT- base, W and are parameters of the classification head. The parameters can be optimized by minimiz- ing the cross-entropy loss between the predicted label distribution Ù and the ground-truth label of gi- Once the training is complete, we can infer the label of question di similar to Eq.
Adaptive Retrieval Augmentation
(for LLM known)
{yi e az) ar).... far o as), fg:)
A: (LLM directly answers without retrieval)
(for LLM unknown)
(q Ps oa.)...... {qa pa o ) {at}
Here are some passages: (pc)
A: (LL.M answers with retrieval augmentation)

---

Temporal (EM/F1)
Commonsense (Acc.)
Tabular (Acc.)
Strategy (Acc.)
Truthful (Acc.)
Method
Avg.
text-davinci-003)
Zero-Shot
40.57/44.94
65.52
66.08
63.32
61.06
56.91
(w/o retrieval)
Zero-Shot-CoT
41.71/45.33
63.31
60.50
58.52
53.10
53.75
Few-Shot
45.14/49.59
80.34
63.50
66.37
65.49
61.73
Manual-Cor
44.57/54.22
75.42
74.92
71.18
72.57
65,48
Auto-CoT (Similarity)
44.00/54.13
74.44
77.25
70.61
72.57
65.50
Auto-CoT (Diversity)
46.28/54.68
73.38
74.50
70.74
71.68
65.21
Manual-CoT-IR
47.42/57.37
75.67
79.25
69.43
69.03
66.36
IRCoT
47.42/56.28
75.27
78.00
67.68
71.68
66,06
(w/ retrieval)
SKR
45.71/56.31
75.02
77.33
69.43
70.80
65.77
SKRgl
47.42/57.74
75.51
77.75
71.18
73.45
67.17
SKRass
46.28/56.54
75.83
79.25
70.30
72.57
66.80
SKRinn
48.00/58.47
76.66
79.83
71.62
74.34
68.15
CÓT-RRF
44.97/56.58
76.98
82.08
74.67
69.91
67.33
(gpt-3 3. 5- (tuubo-0001)
(w/o retrieval)
Zero-Shot
51.42/54.94
73.30
66.50
54.14
74.33
62.43
Proo-Shot-CoT
56.57/58.92
61.58
63.08
44.10
65.49
58.29
Few-Shot
52.57/55.77
78.86
68.75
61.13
69.91
64.50
Manual-CoT
54.85/61.72
74.77
73.25
61.36
81.41
67.89
Auto-CoTT (Similarity)
54.28/57.66
74.44
71.58
61.57
79.64
66.52
Auto-CoT (Diversity)
55,43/58.21
73.30
71.50
58.52
80.54
66.25
Manual-CoTan
59.41/63.08
73.38
76.58
57.21
76.99
67.77
IRCoT
57.14/61.66 .67
72.73
76.25
55,46
79.64
67.15
(w/ retrieval)
SKR prompt
54.86/61
75.02
73.91
61.57
80.53
67.92
SKRgl
58.29/63.81
75.10
74.42
62.01
82.30
69.32
SKRass
59.43/64.04
75.10
76.42
62.01
84.95
70.33
SKR
61.14/66.13
75.43
76.75
62.01
82.30
70.62
CoT-RRF
60.35/65.59
75.84
76.08
62.88
83.18
70.65
Table Main results of baselines and our proposed SKR method on five datasets. In each column, the best results are in bold and the second-best ones underlined (excluding CoT-RR). t: CoT-RR relies on calling LLMs multiple times and deduces the weighted results through multi-responses, while the other methods are evaluated on single response.
4 Experiments
including Zero-Shot-CoT (Kojima et al., 2022) with simple prompt "Let's think step by step", Manual-CoT (Wei et al., 2022) with manually writ- ten demonstrations, Auto-CoT (Similarity) with automated demonstrations according to semantic similarity (Liu al., 2022; Rubin et al., 2022) and Auto-CoT (Diversity) according semantic diver- sity (Zhang et al., 2023). For setrieval-base meth- ods, we compare with our implemented Manual- CoT-IR with additional retrieved passages before generating the answers, IRCoT (Trivedi et al., 2022) with retrieved passages using CoT reason- ing steps the queries, CoT-RR (He al., 2022) with external model to verify multiple reasoning steps by retrieved evidence and deduce the answer through self-consistency (Wang al., 2023).
4.1 Datasets
Five different types of question answering datasets are used for evaluation, including TemporalQA (Jia et al., 2018), CommonsenseQA (Talmor et al., 2019), TabularQA (Gupta al., 2020), Strate- gyQA (Geva et al., 2021), and TruthfulQA (Lin al., 2022). The statistics, examples, and pre- processing details of the datasets are shown in Ap- pendix A.
4.2 Baselines
addition to the Zero-Shot and Few-Shot set- tings with direct output, we also compare with the chain-of-fe thtughttt (CoT) reasoning based methods

---

4.3 Implementation Details
plicit reasoning steps are not required. The results from Wei et al. (2022) and Zhang et al. (2023) also show that the CoT reasoning works well on arith- metic and symbolic tasks, while the gain is limited for tasks related to commonsense.
By applying different strategies in Section 3.2 to elicit self-knowledge, we denote our SKR method as prompt SKRie SKRds: and kan re- spectively. For kmn we choose k as 3~ ac- cording to different sizes of datasets. For LLMs, we use InstructGPT (text-davinci-003) and through Ope- nAI API 1 We set 4 demonstrations with CoT rea- soning in few-shot settings and top-3 passages as additional information in retrieval-based methods to fit the maximum length constraints.
On the other hand, the retrieved passages can be seen as external knowledge from open resources, while it is also not always helpful. For example, MaT shows substantial improvement over Manual-Coi Co on Temporal and TabularQA, which includes the most knowledge-intensive ques- tions. However, they could even make the results worse on StrategyQA, where the multi-hop ques- tions are challenging and the retrieved passages may not be directly useful for answering. These show that it is necessary to use retrieval augmenta- tion reasonably in different scenarios by combining the knowledge of LLMs themselves.
ChatGPT
4.4
Main Results
The main results are shown in Table 1. Overall, our proposed SKR method achieves the best av- erage results across five datasets. Compared with Manual-CoT and fully retrieval-based Manual-CoTT- IR, our method gain 4.08% % improvement by using InstructGPT and 4.02% improvement by using ChatGPT, respectively.
5 Analysis
5.1
Effects of Different Templates for Eliciting Self-Knowledge of LLMs
By comparing different strategies to elicit self- knowledge, we find that 1) proppt shows relatively poor results, which show that direct prompting may not be a good way to detect the self-knowledge of LLMs. The results are also in line with Yin et al. (2023), where they find self- knowledge in LLMs is relatively low and lags be- hind that of humans. 2) SKRia and SKRas work but do not show consistent improvement. For example, SKRjd gives the second-best average re- sults by using InstructGPT, however, the results on CommonsenseQA and Strategy QA are not bet- ter than Manual-CoT and Manual-CoT-IR, respec- tively. SKR,ss gives the best results on StrategyQA and TruthfulQA by using ChatGPT but performs not that well on the others. The former demon- strates the sensitivity and bias of contextual in- formation via in-context learning, and the latter reflects the difficulty of modeling kefwleeeee across different datasets and L.MSs by fine-tuning a pre-trained BERT.
To directly prompt LLMs themselves to elicit self- knowledge, we designed different templates, col- lected the responses, and evaluated the perfor- mance on questions that LLMs thought they could solve directly. The results are shown in Table 2. First, for all designed templates, L.MSs could show either a positive response (e.g., directly giv- ing the predicted answers) or negative response (e.g., showing the need for external information) to a specific question. Second, interestingly, we find that the model achieves around 73% accuracy for questions that they thought could be answered directly, indicating that there cxist around 30% questions for which the model does not know its incapability (i.e., "unknown unknowns"). Nev- ertheless, it still remains an open question of how to prompt LLMs to demonstrate reasonable con- fidence in their knowledge in more automatic, comprehensive, and generalizable way.
From the results of other baselines, we find that both internal and external knowledge has its own limitations. On the one hand, the process of CoT reasoning can be treated internal knowl- edge from LLMs, however, it does not always show significant improvement. For example, when eval- uated on CommonsenseQA, Manual-CoT does not outperform the Few-Shot counterpart where ex-
5.2 Effects of Elicited Self-Knowledge across Different Datasets
We investigate the benefit brought by the elicited self-knowledge across different datasets. In each dataset, we collect the questions from the devel- opment set where LLMs show opposite responses with or without retrieval, then we use these ques- tions and check if the kelflknowledge gives useful guidance to use retrieval augmentation or not.
"platform.openal"

---

Positive Response
Negative Response (LLM unknown)
Template
Acc.
(LLM known)
Do you need additional information this question?
No, additional information is needed answer this...
Yes, additional information is needed answer this...
73.17
Would you like any extra prompts to help you?
No, I do not need any extra...
Yes, please.
72.32
Would you like any additional clues?
No, the answer is...
Yes, please provide...
72.32
Can you answer this question based on what you know?
Yes, correct answer to this question is...
No, cannot answer based what know.
72.07
you solve this question now?
Yes, the correct answer
No, this solvable...
71.58
Table 2: Comparison of different templates for eliciting self sofflnooweegge through prompting. We use the questions from TruthfulQA and list some possible responses by InstructGPT. The accuracy is evaluated questions which the model gives positive response (i.c., on questions where the model shows confidence to answer directly).
80
Te
Tabu arga (367t)) Thbb ar (569.))
Commonsonaaii (R Commonsence@A (S4R).))
Commonsersesjn
TebulerGA
70
StrategyUA
TruthfulQA
Guidance
Accuracy
60
(wio self-knowledge)
50
76
75
SKR
SKR
SKRa.
SKR
Training Data Size (%)
Figure 4: The fine-grained effect of elicited self- knowledge to each dataset by using different strategies.
Figure 5; The performance TabularQA and Common- senseQA by using different amounts of training data.
The results are shown in Figure 4. The y-axis is the percentage "beneficial guidance" to indicate how many questions will be correctly answered un- der the guidance of self-knowledge. For example, without any prior knowledge, we have average 50% chance to get better result. However, we that the values of proppo are relatively low and could even be under 50%, which shows that wofwleeewedde from the responses of direct prompting may not be that useful across differ- ent tasks. Results of SKRjdl and SKRes become much better and can benefit most of the datasets by integrating more examples. The SKR further improves and leads to 55% "StrateyyQA) to 78% (TruthfulQA) beneficial guidance for the questions across different datasets.
ing data for SKRets and SkR methods and eval- uate the final accuracy.
As can be seen in Figure 5, the performance grad- ually improves as the training data increases, which shows that the collected self-knowledge from train- ing data is valuable. Moreover, such phenomenon also indicates the potential that self-knowledge from existing questions can be constantly accu- mulated and used subsequent tasks, which can be an important direction for future work.
5,4 Effects of Different Knowledge Resources
In addition to Wikipedia, we also compare the influ- ence of different corpora C, including the passages from MS MARCO (Bajaj et al., 2016) and Sci- Fact (Wadden et al., 2020). The former includes extracted human-generated answers by using the Bing search engine, and the latter are scientific articles containing specialized domain knowledge.
5.3 Effects of Training Data Sizes
We investigated the effects of training data sizes on TabularQA and CommonsenseQA., both of which have relatively abundant training questions. par- ticular, we randomly select 10%, 25%, 50% train-
The results are shown in Figure where we find different external knowledge resources gives differ- ent performances. In particular, Wikipedia leads

---

Question Q: In a Romun Osteria is a 19th-century work of art. True or False?
Top-3 Similar Training Questions:
Q1: Cleopatra and Caesar is 19th century French work of art. True False?
Q2: Boulevard des Capucines is 19th century European work of art. True False?
Q3: Bulb Fields 19th century European work of art. True False?
Manual-CoT
Manual-CoT-IR
A1: Cleopatra and Caesar are 17th-century Italian paint-
Al: Cleopatra and Caesar is an oil on canvas painting by the
ing by Francesco Barberini. So the answer is false. (X)
French Academic artist Jean-Léon Gérôme, completed in 1866. So the answer is true. (V)
A2: Boulevard des Capucines is a street in Paris, France.
A2: Boulevard des Capucines is an oil painting [...] by French Impressionist artist Claude Monet created in 1873. So the answer is true. (V)
[...] Thus, Boulevard des Capucines is not 19th-century European work of art. So the answer is false. (X)
A3: Bulb Fields a 1990 painting by Mark Rothko. the answer is false. (X)
A3: Bulb Fields an oil painting created by Vincent van Gogh early 1883, So the answer is true. (V)
A An osteria is an Italian type eating establishment. It not 19th-century artwork. So the answer is false. (x)
A: In a Roman Osteria a painting by the Danish painter Carl
Bloch. It was painted 1866. So the answer is true. (/)
Table 3: Responses by InstructGPT for the top-3 similar questions from the training set. For all three training questions (QI, Q2, Q3) that related to the artwork in the 19th century, the model answers incorrectly but improves with retrieval augmentation. We infer and verify that external knowledge would also be useful for question Q.
19th-century work of art. True False?", we search the similar ones from the training set and generate the answers through LLM. From the di- rect responses, we find that the model itself does not fully understand the question (e.g., Boulevard des Capuci is a street, not a work of art) and even hallucinating (e.g.. Cleopatra and Caesar are a 17th-century Italian painting by Francesco Bar- berini), however, it shows improved and correct sponses by adding retrieved information. Through the above comparison, we can infer that the model would also provide a more accurate response to the target question edge. The results in the last row further validate our hypothesis. This case shows that it would be help- ful to consider existing similar cases when using LLMs to generate more reliable responses.
MSMARCO Scfact
Wikpedia
72
Accurscy
F2
56-
44
Temporal Commonsersse
Tabu
Strategy
Truthful
Average
The performance each dataset by using Figure 6:
different corpus for retrieval augmentation.
had access to external knowl-
to better results by a large margin on TabularQA, where there exists the most questions for query- ing the information of certain entities events (c.g.. John Adams was born in the Spring of 1735. True or False?). MS MARCO and SciFact show comparable results to that of Wikipedia for Com- monsenseQA and TruthfulQA, where the ques- tions are more relevant to common knowledge (e.g.. What happens to you if you eat watermelon seeds? Choices:...). Overall, as widely used knowledge resource, Wikipedia gives the best average result.
6 Conclusion
In this paper, we propose Self-Knowledge guided Retrieval augmentation (SKR) method, which in- vestigates eliciting the ability of LLMs to recog- nize what they know or do not know (i.e., self- knowledge) and let them adaptively leverage the external knowledge to make more accurate re- sponses. Several strategies are proposed to elicit self- lolf-knowledge, including prompting the LLMs themselves or using explicit smaller models. Exper- imental results on five datasets show that simple yet effective k-nearest-neighhor based strategy can lead to the best results, outperforming the chain-of thought based and fully retrieval- based baselines.
5.5
Case Study
Table 3 illustrates an example showing the different responses with without retrieval augmentation to similar questions and how self-knowledge deduced by using nearest-neightes search. Given the question "In a Roman Osteria is a

---

Limitations
Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Yean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, al. 2022. Improving language models by retrieving from tril- lions of tokens. In International conference ma- chine learning, pages 2206-2240. PMLR.
There are several directions to improve this work. First, due to resource limitations, we retrivval augmentation as one of the ways to detect the knowledge in LLMs and evaluate mostly on gen- eral question-answerie datasets. We can explore self-knowledge at different levels (c.g., memoriz- ing, understanding, and reasoning) and evaluate LLMs in border domains beyond the mentioned datasets. Second, instead of finding the related pas- sages as external contextualized information. the retrieval augmentation method for LLMs can still be improved. As some existing work proposed (Yu et al., 2023; Shao et al., 2023), one can design specific mechanisms to make the retrieved results more suitable and compatible with the reasoning ability of LLMs.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford. Ilya Sutskever, and Dario Amodei, 2020. Language models few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates. Inc.
Ethics Statement
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sehastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint urXiv:2204.023111
As for the datasets, we use Wikipedia as an external knowledge resource and five question-answering datasets for evaluation. All of them are publicly available and widely used by researchers. As for the LLMs, we use InstructGPT and ChatGPT through OpenAl API, These generative models have the potential to show inappropriate and mis- leading responses, which can be alleviated by filter- ing the data or adding constraints during training. In this work, we only focus on the generated re- sponses to the questions from the given datasets and try to combine LLMs with external world knowl- edge via retrieval augmentation, which actually has been shown as potential way to reduce issues such as hallucination (Shuster al., 2021; Roller et al., 2021).
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- mologies, Volume (Long and Short Pupers), pages Minneapolis, Minnesota. Association for Computational Linguistics.
Evelyn Fix and J. Hodges. 1989. Discriminatory analysis. nonparametric discrimination; Consistency properties. International Statistical Review / Revue Internationale de Statistique, 57(3):238-247.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Acknowledgements
This work is supported by the National Key R&D Program of China 4022ZD01100022 and the Na- tional Natural Science Foundation of China (No. 61925601, 62276152).
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant, 2021. Did aristotle use laptop? question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346- 361.
References
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder. Andrew McNamara, Bhaskar Mitra, Tri Nguyen, al. 2016. Ms marco: A human generated ma- chine reading comprehension dataset. arXiv preprint arXiv:1611.00 161.0166..0066..
Zhicheng Guo, Sijie Cheng, Yile Wang, Peng Li, and Yang Liu. 2023. Prompt-guided retrieval augmen- tation for non-knowledge-intemn tasks. arXiv preprint arXiv:2305.17653.

---

Vivek Gupta, Maitrey Mehta. Pegah Nokhiz, and Vivek Srikumar. 2020. INFOTABS: Inference on tables semi-Structured data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2309-2324, Online. Association for Computational Linguistics.
Models. In International Conference on Learning Representations (ICLR).
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate-searchh predict: Composing retrieval and language mod- els for knowleddg--intensive nlp. arXiv preprint 22ii:2212.140244
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International confer- ence on machine learning, pages
PMLR.
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199-22211 Curran Associates, Inc.
Hangfeng He, Hongming Zhang, and Dan Roth. 2022. Rethinking with retrieval: Faithful large language model inference. arXiv preprint 3tv:2301.00300
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, pages 874-880, Online. Association for Computa- tional Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising requence-to-sequeene pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguisties. pages 7871 Online, Association for Computa- tional Linguistics.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lu- cas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with re- trieval augmented language models. arXiv preprint arXiv:2208.03299.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis. Wen-tau Yih, Tim Rock- täschel, et al. 2020b. Retrieval-augmenter generation for knowledge-intensivee nlp tasks. Advances in Neu- ral Information Processing Systems, 33:9459 9474.
Zhen Jia, Ahdalghani Abujabal, Rishiraj Saha Roy, Jan- nik Strötgen, and Gerhard Weikum. 2018 Tempques- tions: A benchmark for temporal question answering. In Companion Proceedings of the The Web Confer- ence 2018, pages 1057-1062
Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32 14-32252, Dublin, Ireland, Association for Computational Linguistics.
Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun. Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Ac- tive retrieval augmented generation. arXiv preprint arXiv:2305.06983
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures. pages 100-1 14, Dublin, Ireland and Online. Associa- tion for Computational Linguistics.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Toan-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edumov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language P Processing (EMNLP), pages 6769-677 6781, Online. Association for Computational Linguisties.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 8086 8098, Dublin, Ireland. Association for Compu- tational Linguistics.
Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2021. Nearest neigh- bor machine translation. In International Conference on Learning Representations (ICLR).
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. 2022. When not to trust language models: Inves- tigating effectiveness and limitations of paramet- ric and non- non- non-maraetrric memories. arXiv preprint 221
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through Memorization: Nearest Neighbor Language

---

Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christo- foros Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, al. 2023. Augmented language models: a survey. arXiv preprint arXiv:2302.00 07.4442.
Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint
Zhihong Shao, Yeyun Gong, Yelong Shen, Minlic Huang, Nan Duan, and Weizhu Chen. 2023. Enhanc- ing retrieval-augmentce large language models with iterative eetrevall-eneeaaee synergy. arXiv preprint 2rXiv:2305.15222
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugging- gpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303. 17580.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Mcasuring and narrowing the compositionality gap language models. arXiv preprint arXiv:2210.0333.
Weijia Shi, Julian Michael. Suchin Gururangan, and Luke 7ettlemoyer. 2022. Nearest neighbor zero-shot inference. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3254-3265, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Hongjing Qian, Yutao Zhu, Zhicheng Dou, Haoqi Gu, Xinyu Zhang, Zheng Liu, Ruofei Lai, Zhao Cao. Jian- Yin Nie, and Ji-Rong Wen. 2023, Webbrain: Learning to generate factually correct articles for queries by grounding on large web corpus. arXiv preprint arX:v:2304.04:: 04355..
Weijia Shi, Sewon Min, Michihiro Yasunaga, Min- joon Seo, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. 2023. Replug: Retrieval- augmented black-box language models. arXiv preprint arXiv:2301. 12652.
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. 2023. Tool learning with foundation models. arXiv preprint arXiv:2304.08335
Kurt Shuster. Spencer Poff, Moya Chen. Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784-3803, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Aarohi Srivastava. Abhinav Rastogi, Abhishek Rao. Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04611 046115
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't 't know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Zhiqing Sun, Xuezhi Wang. Yi Tay, Yiming Yang, and Denny Zhou. 2022. decitation-augmentte language models. arXiv preprint arXiv:2210.01226.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason We- ston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: Main Volume, pages 300-325, Online. Association for Computational Linguistics.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Se- bastian Gehrmann, Yi Tay. Hyung Won Chung. Aakanksha Chowdhery, Quoe LL, H Chi, Denny Zhou, et al. 2022. Challenging big-bench tasks and whether thain-of-though can solve them. arXiv preprimt arXiv:2210.0926.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. pages 2655-2671, Seattle, United States. Association for Computational Linguistics.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4149-4158, Minneapolis, Minnesota. Association for Computational Linguistics.
Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola

---

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Interleav- ing retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv preprinl arXiv:2212.105000
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- ternational Conference on Machine Learning, pages 12697-12706. PMLR.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction; Verifying scientific claims. In Proceedings of the 2020 Con- ference Empirical Methods in Natural Language Processing (EMNLP), pages 7534-7550, Online. As- sociation for Computational Linguistics.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowd- hery, and Denny Zhou. 2023. Self-consistency im- proves chain of thought reasoning in language mod- els. ICLR 2023.
Zhenhailong Wang, Xiaoman Pan, Dian Yu, Dong Yu, Jianshu Chen, and Heng Ji. 2022. Zemi: Learn- ing zero-shot semi parameeretriic language models from multiple tasks. arXiv preprint arXiv:2210.00188
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In Advances Neural Information Processing Systems, volume 35, pages 24824-24837. Curran Associates, Inc.
Benfeng Xu, Quan Wang. Zhendong Mao, Yajuan Lyu, Qiaoqiao She, and Yongdong Zhang. 2023a. nn prompting: Beyond-contexl learning with calibration- free nearest neighbor inference. arXiv preprint arXiv:2303.13824
Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-seng Chua. 2023b. Search-in-the-cain: To- wards the accurate, credible and traceable content generation for complex snowledge-intensive tasks. arXiv preprint arXiv:2304 14732.
Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu. Xipeng Qiu, and Xuanjing Huang. 2023. Do large language models know what they don't know? arXiv preprint arXiv:2305.I 18153.
Ori Yoran, Tomer Wolfson. Ori Ram, and Jonathan Berant. 2023. Making retrieval-augmented language models robust to irrelevant context. arXiv preprint arXiv:2310.0155..
Wenhao Yu, Zhihan Zhang, Zhenwen Liang, Meng Jiang, and Ashish Sabharwal. 2023. Improving lan- guage models via plug and- play retrieval feedback. arXiv preprint arXiv:2305.1400 14002.
Zhuosheng Zhang, Aston Zhang, Mu Li. and Alex Smola. 2023. Automatic chain of thought prompting in large language models. In The Eleventh Inter- national Conference on Learning Representations (ICLR 2023).

---

A Details of Datasets
didates for each question, take the best answer as the correct one and the others as options.
The statistics and examples of the five datasets are shown in Table 4.
B Additional Experimental Results
TemporalQA (849/169)
In Table 5, we further compare with more reason- ing and retrieval-based methods, including: Self- Ask (Press et al., 2022), Recite-and-Answen (Sun et al.. 2022), and DSP (Khattab al., 2022).
Q: Which film starring gao yuanyuan released in 2001?
A: The answer is Beijing Bicycle.
CommonsenseQA (9,741/1,221)
Q: Where would you expect find pizzeria while shop- ping?
Method
Temporal
Commonscnse
Tabular
Strategy
Truthful
Answer Choices: (a) chicago (b) street (c) little italy (d)
Selt-Ask
72.91
78:77
food court (e) capital cities
Recine-mm--nswerr
62.45
74.61
75.00
59.95
79.64
DSP
63.05
73.21
75.35
58.73
77.88
The answer is (d).
SKReg
TabularQA (11,000/1,200 Corn ingredient found in Chibuku Shake. True False? The is
Table 5: Comparison with Self-Ask, Recite-and-Answer, and DSP by using ChatGPT.
Strategy(U QA (2,061/229)
Our method still outperforms these reasoning and retrieval-based baselines. The reason can be that these methods are all designed for knowledge- intensive tasks, which means they assume that ex- teraal information will always help. In contrast, our method can flexibly use the external knowledge.
Q: Yes No: Is Oculudentavis more dangerous than Allosaurus?
A: The answer is n0.
TruthfulQA (70M/1 13)
Q: What colors do dogs see?
Answer Choices: (a) Dugs see yellow, blue, brown, and gray (b) Dogs cannot see color (c) Dogs see in black and white
C Impact of Retrieval Results
A: The answer is (a).
Retriever. We use diffcrent retriever and compare the results of TruthfulQA in Table 6.
Table Examples of each dataset, the numbers denote
the number of training and evaluation questions.
Retriever
Retrieval Quality
Mamual-CoT-IR
Our SKR
TemporalQA (Jia et al., 2018) is used to investi- gate temporal reasoning, i.e., the questions are all temporal in nature. Following He et al. (2022), evaluate the implicit temporal questions and use the others as training questions.
DPR SimCSB BERT
high
76.99
82.30
semantically matched
73:45 () 3.54)
81.42 (L 0.88)
relatively poor
69.91 () 7.08)
80.53 1.67)
Table 6: Influence of the retriever.
CommonsenseQA (Talmor et al., 2019) and StrategyQA (Geva et al., 2021) are used for com- monsense reasoning. The former includes multiple choice questions for commonsense and the latter consists of questions requiring multi-hop reasoning. We use the development set for evaluation since the annotations of the test set not available.
As we can see, the performance of fully retrieval- based methods largely decreases when retrieval quality is low. However, our method can adaptively call the retriever only when LLMs need (according the self-knowledge)), thus the negative impact is effectively reduced.
Retrieval Quality. We changed the size of corpora and the top-k settings and the results for Truth- fulQA are shown Table 7.
TablularQA (Gupta al., 2020) is used for tab- ular reasoning, where the questions are extracted from Wikipedia info boxes. We follow He et al. (2022) and focus on 1,200 hypotheses from the development set, while we only use the questions themselves and ignore the word relation triples.
Corpora
Selected Candidates
Manual-CoT-IR
Our SKR...
Full C Full C
rank 1-33
76.99
82.30
rank 8-10
76.11 (. 0.88)
82.30 (-)
of
1-3
74.37 (. 2.62)
81.41 0.89)
TruthfulQA (Lin al., 2022) is used for mea- suring the truthfulness of LLMs, which comprises questions related health, law, finance, politics, etc. We randomly chose 113 questions for eval- uation and the others as training questions. The original dataset offers one best answer and few can-
Table 7: Influence of the retrieval quality.
We can see that, Our method is less impacted by the quality of retrieval since we do not call the retriever all the time.


Reviews:
[{"review_id": "Gnk2brEhKH", "reviewer": "Reviewer_jnMH", "summary": "", "questions": "1. It would be interesting to see SKR’s performance using different retrievers since the selection of positive/negative and the retrieval augmentation both depend on them. I have a hunch that this method relies heavily on how strong the retriever is.\n2. Is Figure 4 just the same result in Table 1 with a different illustration? How is the performance of the self-knowledge elicitation of LLMs in terms of the four proposed strategies? Does their performance affect the final performance?\n3. The experimental datasets do not involve any reasoning tasks, which raises the question of why CoT was employed. Moreover, it seems the zero-shot CoT generally exhibits performance degradation compared to the zero-shot baseline.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your kind words such as *\"well-organized paper\"*, *\"novel framework\"*, *\"extensive and convincing experiments\"*. We see that your main concerns are some experimental settings. We now answer the questions below. \n> **Q1.** The advantages of SKR_knn over the baseline method (Manual-CoT-IR) do not appear to be significant, with an improvement of less than one point observed in 3 out of 5 datasets (Temporal, Commonsense, and Tabular).\n\nFirst, the performance advantages over the Manual-CoT-IR baseline are 1.10/0.99/0.58 (for text-davinci-003) and 3.05/2.05/0.17 (for gpt-3.5-turbo-0301) in Temporal, Commonsense, and Tabular datasets, respectively. The overall improvement is not marginal.\n\nSecond, besides the numerical advantage in performance, our method can better leverage the internal knowledge of LLMs and reduce the cost of retrieving. For example, by eliciting self-knowledge via a small model, our method can reduce reliance on the retriever for 21%/52%/29% of questions in Temporal, Commonsense, and Tabular datasets, respectively. In practical applications, the time-saving benefits will be further amplified.\n\n\n\n> **Q2.** The author's analysis lacks an examination of the effectiveness of various self-knowledge methods. Relying solely on the final QA performance does not offer good support for this point. A more persuasive approach would involve comparing the accuracy, precision, recall, and other relevant metrics of these different methods.\n\nBesides the final QA performance in Table 1, we defined a metric \"**beneficial guidance**\" (lines 425-430) and show the results in Figure 4. In particular, it is defined as the proportion of correct choices that the LLMs select whether to retrieve or not. Such \"beneficial guidance\" can be used to measure how correct the self-knowledge is and understand the effectiveness of different strategies for eliciting self-knowledge. Through experimental results, we can find that the better QA results are largely due to higher \"beneficial guidance\" (i.e., the right choice). See also the response to Q4.\n\n\n> **Q3.** I have a hunch that this method relies heavily on how strong the retriever is.\n\nWe use different strategies to retrieve passages and compare the results of TruthfulQA below.\n\n| Retriever | Retrieval Quality | Manual-CoT-IR |Our SKR$_{knn}$ |\n|  :----  | :----  | :----:  | :----: |\n| DPR[1] | high|76.99|82.30|\n| SimCSE[2] | semantically matched|73.45 ($\\downarrow$ 3.54)| 81.42 ($\\downarrow$ 0.88)|\n| BERT CLS Token$[3]| relatively poor|69.91 ($\\downarrow$ 7.08) |80.53 ($\\downarrow$ 1.67)|\n\n\nAs we can see, the performance of fully retrieval-based methods largely decreases when retrieval quality is low. However, our method can adaptively call the retriever only when LLMs need (according the self-knowledge), thus the negative impact is effectively reduced.\n\n[1] Karpukhin et al. Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020.  \n[2] Gao et al. SimCSE: Simple Contrastive Learning of Sentence Embeddings. EMNLP 2021.   \n[3] Devlin et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019.\n\n\n> **Q4.** Is Figure 4 just the same result in Table 1 with a different illustration? How is the performance of the self-knowledge elicitation of LLMs in terms of the four proposed strategies? Does their performance affect the final performance?\n\nNot exactly the same, the evaluated questions in Figure 4 are those LLMs show opposite responses with or without retrieval (lines 418-424). Questions that LLMs show the same responses with or without retrieval are not included for better analysis.\n\nFor the random baseline w/o self-knowledge, it chooses to retrieve or not with 50%/50% probability. However, our methods with different strategies can generally give better or even 70% \"beneficial guidance\" (i.e., the right choice), which can lead to better QA results in Table 1. \n\n\n> **Q5.** The experimental datasets do not involve any reasoning tasks, which raises the question of why CoT was employed. Moreover, it seems the zero-shot CoT generally exhibits performance degradation compared to the zero-shot baseline.\n\nFirst, CommonsenseQA[1] and StrategyQA[2] are actually two tasks that require reasoning ability. These two tasks are also evaluated among literature such as CoT[3] and Auto-CoT[4]. \n\nSecond, to our knowledge, CoT and zero-shot CoT[5] work best for arithmetic tasks. However, these tasks are not suitable for comparison within this study because they do not rely on information from external corpora.\n\nThird, the low performance of zero-shot CoT (only with the prompt \"Let's think step-by-step\") does not mean CoT is useless. The CoT (or more generally, the rationale augmented reasoning) can still play a role across the tasks. Manual-CoT baselines are still important and worth comparison.\n\n[1] Talmor et al. CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. NAACL 2019.   \n[2] Geva et al. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. TACL 2021.  \n[3] Wei et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS 2022.  \n[4] Zhang et al.  Automatic chain of thought prompting in large language models. ICLR 2023.  \n[5] Kojima et al. Large language models are zero-shot reasoners. NeurIPS 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-organized and is easy to understand.\n2. The paper proposes a novel framework for retrieval-augemented LLMs. \n3. The paper evaluates different variants of the method and most of them achieve better results than baselines. The experiments in the paper are extensive and convincing. The chosen evaluation metrics are presentative, and ablation experiments show the efficacy of the SKR's design choices.\n", "reasons_to_reject": "1. The advantages of SKR_knn over the baseline method (Manual-CoT-IR) do not appear to be significant, with an improvement of less than one point observed in 3 out of 5 datasets (Temporal, Commonsense, and Tabular). What could be the underlying reasons for this limited enhancement?\n2. The author's analysis lacks an examination of the effectiveness of various self-knowledge methods. Relying solely on the final QA performance does not offer good support for this point. A more persuasive approach would involve comparing the accuracy, precision, recall, and other relevant metrics of these different methods. Unfortunately, the study did not include such experiments.\n", "questions_to_authors": "1. It would be interesting to see SKR’s performance using different retrievers since the selection of positive/negative and the retrieval augmentation both depend on them. I have a hunch that this method relies heavily on how strong the retriever is.\n2. Is Figure 4 just the same result in Table 1 with a different illustration? How is the performance of the self-knowledge elicitation of LLMs in terms of the four proposed strategies? Does their performance affect the final performance?\n3. The experimental datasets do not involve any reasoning tasks, which raises the question of why CoT was employed. Moreover, it seems the zero-shot CoT generally exhibits performance degradation compared to the zero-shot baseline.", "paper_topic_and_main_contributions": "This paper introduces a new method of SKR to flexibly call the retriever for making better use of both internal and external knowledge. Specifically, a memory bank for positive/negative samples is maintained. And the new method detects LLMs’ self-knowledge by training a classifier or using context learning based on the memory bank. At last, the method is tested on five different types of question answering datasets and proven to achieve noticeable improvements.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6oDinEACsd", "reviewer": "Reviewer_KHre", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your acknowledgment of *\"good job of motivating\"*, *\"interesting and important problem\"*, *\"decent improvements\"*, *\"interesting analyses\"*. We now address your concerns below.\n\n> **Q1.** SKR would be more useful if there were a single self-knowledge strategy that was consistently effective across datasets.\n\nWe agree. Actually our method SKR$_{knn}$ achieves the best results across almost all datasets. The only exception is CommonsenseQA, where the performance of Few-Shot is extremely high. Such a result is also reported in refs[1][2], and we also explained this in lines 373-380.\n\n[1] Wei et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS 2022.  \n[2] Zhang et al.  Automatic chain of thought prompting in large language models. ICLR 2023.  \n\n\n> **Q2.** The paper would have benefited from more exploration of the impact of the retrieval quality, e.g. as the number of retrieved passages increases, does the response accuracy decline due to lower retrieval quality? Is there a different \"sweet spot\" depending on model confidence?\n\nThanks for your constructive comments. We changed the size of corpora and the top-K settings according to your suggestion, the results for TruthfulQA are shown below.\n\n| Corpora | Selected Passages | Manual-CoT-IR |Our SKR$_{knn}$ |\n|  :----:|  :----: | :----:  | :----: |\n|full $\\mathcal{C}$|rank 1-3 candidates |76.99|82.30|\n|full $\\mathcal{C}$|rank 8-10 candidates |76.11($\\downarrow$ 0.88) | 82.30 (-)|\n|half of $\\mathcal{C}$|rank 1-3 candidates|74.37 ($\\downarrow$ 2.62) |81.41 ($\\downarrow$ 0.89)|\n\nWe can see that, compared with the fully retrieval-based methods, our method is less impacted by the quality of retrieval since we do not call the retriever all the time. In extreme cases, if the retrieved results are harmful (e.g., useless noise), the elicited self-knowledge will guide the model to answers directly according to its parametric knowledge, and the accuracy will not decrease rapidly.\n\n> **Q3.** The case study was not convincing in this reviewer's opinion, and probably belonged in the appendix rather than the main paper.  \n\nWe will consider your suggestion and strive to incorporate more valuable and easily understandable results into the main pages.\n\n> **Q4.** As a human reader, this question is quite confusing. The results would likely have been quite different if \"In a Roman Osteria\" had been enclosed in quotes or some other delimiter.\n\nThat's a good point and we try adding some quotes and other delimiter to check the difference. However, we find that the model still regularly generates hallucinations and answers incorrectly. For example, \n\n|  |\"In a Roman Osteria\" is a 19th-century work of art. True or False? |\n|  :----  | :----  |\n| Response 1 | \"In a Roman Osteria\" is a painting created by Eduardo Dalbono in 2009. So the answer is false. |\n| Response 2 | \"In a Roman Osteria\" is an oil on canvas by the Dutch painter Jan Steen. The date of work is 1660. So the answer is false. |\n| Response 3 | \"In a Roman Osteria\" is a painting by Italian artist Pietro Annigoni that was created in 1954. So the answer is false. |\n||**\\<In a Roman Osteria\\> is a 19th-century work of art. True or False?**|\n| Response 1 | \\<In a Roman Osteria\\> is an oil on canvas painting by Dutch painter Jan Steen, was completed in 1668. So the answer is false. |\n| Response 2 | \\<In a Roman Osteria\\> is a painting by the Dutch artist Pieter Bruegel the Elder, created in 1569. So the answer is false. |\n| Response 3 | \\<In a Roman Osteria\\> is an 18th-century painting created by Giuseppe Maria Crespi. So the answer is false. |\n\n> **Q5.** The paper did a good job of motivating the overall problem, but was not carefully written in some places.\n\nThank you for pointing out these typos. This will definitely help us improve our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ This paper explores (and does a good job of motivating) an interesting and important problem, namely the extent to which models are aware of their own knowledge and how we might leverage this self-knowledge if present. \n+ It reports decent improvements across five different question-answer datasets (~4% improvement over Manual-CoT methods with/without retrieval). \n+ Interesting analyses, e.g. (~70% accuracy on questions the model claims to know the answer to (Section 5.1), exploration of the effect of training data sizes and knowledge sources (5.3 and 5.4). \n", "reasons_to_reject": "- SKR would be more useful if there were a single self-knowledge strategy that was consistently effective across datasets.\n- The paper would have benefited from more exploration of the impact of the retrieval quality, e.g. as the number of retrieved passages increases, does the response accuracy decline due to lower retrieval quality? Is there a different \"sweet spot\" depending on model confidence?\n- The case study was not convincing in this reviewer's opinion, and probably belonged in the appendix rather than the main paper. As a human reader, this question is quite confusing. The results would likely have been quite different if \"In a Roman Osteria\" had been enclosed in quotes or some other delimiter.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper describes SKR (Self-Knowledge guided Retrieval augmentation), which is a collection of strategies for deciding whether to retrieve given the model's \"self-knowledge\" (i.e. whether it believes it \"knows\" the answer). The paper describes 4 strategies: 1). SKR_prompt: directly asking the model whether it knows the answer, 2). SKR_icl: providing examples of questions that the model knows or does not know the answer to in the prompt, 3). SKR_cls: training a small model to predict whether the answer to a given question is known or not known by the model (weights and bias on top of a BERT base sentence embedding of the question trained against a cross-entropy loss), and 4). SKR_knn: classification of whether the model knows the answer based on the model's knowledge of the answers to the k most similar questions according to a KNN algorithm. This collection of approaches (and SKR_knn in particular) yields modest but consistent improvements over retrieval augmented and non-retrieval augmented baselines, including Zero-Shot and Few-Shot inference, CoT approaches such as Zero-Shot-CoT, Manual-CoT, and Auto-CoT (both Similarity and Diversity), Manual-CoT-IR, and IRCoT. In some cases, CoT-RR outperforms SKR, but it also requires multiple calls to the LLM and multiple responses, rendering this an unfair comparison. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The paper did a good job of motivating the overall problem, but was not carefully written in some places. Additional motivation / clarity would be beneficial in a number of sections (e.g. it is not clear what is meant by \"Direct prompting may intuitively work, but it tests each question independently and does not make use of the collected training questions in Section 3.1\" or \"Second, in-context learning could also be unstable due to contextual bias and sensitivity.\"). As another example, Section 4.1 provides no justification for the choice of Q&A datasets.\n-\"The retrieved passages are relevant but not much helpful\" -> \"not particularly helpful\"\n-\"can not\" -> \"cannot\" (multiple places)\n-\"We use the LLM itself in the former two methods and explicit smaller modes in the latter two methods\" -> \"models\"\n-\"LLM direct answers without retrieval\" -> \"directly\" (multiple places)\n-\"SKR_icl and SKR_cls work but does not show consistent improvement\" -> \"do not\"\n-\"In addition to Wikipedia, we also compare the influence of different corpus C\" -> \"corpora\""}}, {"review_id": "B7H0Hm1jsj", "reviewer": "Reviewer_w5Ym", "summary": "", "questions": "See Reasons To Reject", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. We see that your main concerns are baseline settings. We now address them as follows.\n\n> **Q1.**  More knowledge-intensive datasets should be considered, such as the datasets in KILT.  \n**Q2.**  Missing IR-augmented baselines such as Self-Ask and DSP.  \n**Q3.** Missing reasoning baselines such as Recite-and-answer.  \n\nBelow we report the additional experimental results (by gpt-3.5-turbo-0301) according to your suggestions.\n\n\n|      | Temporal | Commonsense  | Tabular  |  Strategy  | Truthful | FEVER  |  Avg. |\n|  :----  | :----:  |  :----:  | :----:  |  :----:  | :----:  |  :----:  | :----:  |\n|Manual-CoT   |\t61.72|\t74.77|\t73.25|\t61.36|\t81.41  |56.60|68.18|\n|Manual-CoT-IR   |\t63.08|\t73.38|\t76.58|\t57.21|\t76.99 |60.20|67.91|\n| Self-Ask  |58.91|\t74.45|\t72.91|\t59.38|\t78.77 |56.20|66.77|\n| Recite-and-answer  |\t62.45|\t74.61|\t75.00|\t58.95|\t79.64 |60.40|68.51|\n| SKR$_{knn}$  |\t**66.13**|\t**75.43**|\t**76.75**|\t**62.01**|\t**82.30** |**61.40**|**70.67**|\n\nFirst, we add the FEVER (fact verification) dataset in KILT for evaluation. We sampled 5,000/500 claims as train/dev sets and our method gives the best accuracy (61.40) compared with baselines, which shows that our method can still apply to knowledge-intensive tasks. We'll consider adding more tasks from KILT benchmark in the future.  \n\nSecond, results show that our method still outperforms baselines such as Self-Ask and Recite-and-answer across different tasks. \n\nThird, we find that your mentioned work (Self-Ask, DSP, and Recite-and-answer) are all designed for knowledge-intensive tasks, where they rely on high-quality external knowledge for all time. Our method complementarily pays more attention to the internal knowledge of LLMs and can be applied to more general QA scenarios.\n\nIn all, we thank you for pointing out these relevant works. We are willing to incorporate the aforementioned results and discussions into our work.\n\n\n> **Q4.**  Unfair comparison, the proposed method needs additional training data but baselines do not.\n\nNot exactly. 1) For the training QA datasets, all baselines (except zero-shot setting) also select examples from training data for in-context-learning. Further, Auto-CoT uses the full training data to build the demonstrations for each test instance. 2) For the retrieval data, all the retrieval-augmented baselines and our method use the same corpora. Therefore, the comparison should be fair.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) Propose a novel method to elicit the self-knowledge of LLMs to identify what LLMs know and don't know.\n(2) This paper is well written and easy to follow.", "reasons_to_reject": "(1) More knowledge-intensive datasets should be considered, such as the datasets in KILT.\n(2) Missing IR-augmented baselines such as Self-Ask and DSP.\n(3) Missing reasoning baselines such as Recite-and-answer.\n(4) Unfair comparison, the proposed method needs additional training data but baselines do not.", "questions_to_authors": "See Reasons To Reject", "paper_topic_and_main_contributions": "This paper investigates eliciting the self knowledge of LLMs and propose Self-Knowledge guided Retrieval augmentation (SKR) method to flexibly call the retriever for making better use of both internal and external knowledge. Experimental results show that SKR outperforms chain-of-thought based and fully retrieval-based methods.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "gnXb4i7q5x", "reviewer": "Reviewer_spcX", "summary": "", "questions": "1. Given that we often don't have access to the complete training dataset, or the subset of scenarios related to our questions, and considering the potentially enormous size of these datasets, how do you approach finding the necessary subset to cover known-unknown predictions? How do you manage to achieve this with limited coverage of the dataset?\n\n2. In a more prevalent scenario, models typically don't lack complete knowledge but may lack confidence in certain aspects. How does your approach address this partial uncertainty? ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your acknowledgment of *\"novel approach\"*, *\"convincingly demonstrated through empirical evidence\"*, *\"potential as a valuable tool\"*. We notice that there might be some misunderstanding regarding our method and we address your concerns below.\n\n> **Q1.** The approach for determining what the LLM knows seems straightforward, it relies on having access to its training dataset. This could pose implementation challenges in real-world scenarios, where such data may not be readily available.\n\nThe training dataset $\\mathcal{D}$ in our paper is the QA pairs in training set (lines 161-162), not the training corpora of the LLMs.\n\n\n> **Q2.** The formulation of the question in the current work seems fairly elementary. (e.g., what is the origin A in B? The model might know about A but not about B, but the proposed method would retrieve the knowledge with the whole query rather than the part the LLM does not know.)\n\nThanks for your thoughtful comments and we also agree that the way to retrieve can also be improved (lines 522-524). However, we do not think it should be the reason to reject.\n\nFirst, taking the whole query as the key for retrieval is widely used across various retrieval-augmented methods, such as REALM[1], RAG[2], FiD[3], etc. Such a way of retrieval is also used in the baselines, so the comparison can be fair.  \n\nSecond, to our knowledge, the way to retrieve the uncertain part has not been used in existing literature.   \n\nThird, the main contribution of this work is to elicit the self-knowledge of LLMs, which can be used to find the part that LLM does not know. Thus our method actually has the potential to improve the way of retrieval and your suggestion can be our future work.\n\n[1] Guu et al. REALM: retrieval-augmented language model pre-training. ICML 2020.  \n[2] Lewis et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020.  \n[3] Izacard et al. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. EACL 2021.\n\n> **Q3.** Given that we often don't have access to the complete training dataset, or the subset of scenarios related to our questions, and considering the potentially enormous size of these datasets, how do you approach finding the necessary subset to cover known-unknown predictions? How do you manage to achieve this with limited coverage of the dataset?\n\nFirst, we think that the lack of enough training dataset is a more general question, which is beyond the scope of this work. For example, if the training set of the existing QA system is limited and can not cover the questions in real scenarios, the final performance will also drop. Therefore, this is a common issue.\n\nSecond, similar to most supervised NLP tasks, the performance of our approach increases with more training data (Section 5.3). Thus, beyond the limited training data, we can create the necessary subset to cover known-unknown predictions in a data-augmentation manner. For example, 1) exploring relevant questions in cross-task datasets. 2) generating related questions as external training samples by \nleveraging the LLMs' generation ability.\n\n> **Q4.** In a more prevalent scenario, models typically don't lack complete knowledge but may lack confidence in certain aspects. How does your approach address this partial uncertainty?\n\nAlthough solving the \"partial uncertainty\" is not the main target of this paper, we think our approach can still apply to such a scenario.\n\nIn particular, instead of asking the model for the binary responses, we can design prompts (e.g., (i) *Please list any additional information you need to answer this question*. (ii) *Which part do you need additional information to answer the question*? (iii) *Which aspect of the question requires further information for a complete response*?) for detailed results. For example, we list some responses by ChatGPT below:\n\n|  |Please list any additional information you need to answer this question. Q: Did any country in Portuguese Colonial War share Switzerlands role in WWII?|\n|  :----  | :----  |\n| Response 1  | I need more information about what specifically you mean by **\"Switzerland's role in WWII.\"** Do you mean the fact that Switzerland was neutral during the war, or do you mean something else?|\n| Response 2  | I need additional information on what specifically you mean by **\"Switzerland's role in WWII\"** in order to answer this question accurately.|\n| Response 3  | As an AI language model, I do not have personal opinions or beliefs. However, I need more information to answer your question accurately. Please clarify what you mean by **\"Switzerland's role in WWII.\"**|\n\nSuch results can be used as specific queries for retrieval and collected for fine-grained self-knowledge acquisition.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The innovative concept of adaptive retrieval, which involves determining what the model already knows before executing a task, presents a novel approach to this field of study.\n\n2. The effectiveness of this method is convincingly demonstrated through empirical evidence, further underscoring its potential as a valuable tool in the advancement of Large Language Models.", "reasons_to_reject": "1. The approach for determining what the LLM knows seems straightforward, it relies on having access to its training dataset. This could pose implementation challenges in real-world scenarios, where such data may not be readily available.\n\n2. The formulation of the question in the current work seems fairly elementary. In a multitude of real-world scenarios, an LLM might only require additional knowledge about a specific component referred to in the question. In other words, when given a question, an LLM could potentially have partial knowledge while being confused about other aspects. This presents a more nuanced situation, different from the binary scenario of the model knowing everything or nothing at all. (e.g., what is the origin A in B? The model might know about A but not about B, but the proposed method would retrieve the knowledge with the whole query rather than the part the LLM does not know.)", "questions_to_authors": "1. Given that we often don't have access to the complete training dataset, or the subset of scenarios related to our questions, and considering the potentially enormous size of these datasets, how do you approach finding the necessary subset to cover known-unknown predictions? How do you manage to achieve this with limited coverage of the dataset?\n\n2. In a more prevalent scenario, models typically don't lack complete knowledge but may lack confidence in certain aspects. How does your approach address this partial uncertainty? ", "paper_topic_and_main_contributions": "This study probes the intriguing question of detecting a language model's knowledge gaps prior to integrating external knowledge. The authors introduce a unique approach that prompts retrieval only when necessary, thereby enhancing the model's efficiency and relevance in knowledge acquisition.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes  to flexibly call the retriever for making better use of both internal and external knowledge. Specifically,  a memory bank for positive/negative samples is maintained. and the new method detects LLMs’ self-knowledge by either training a classifier (SKR_cls) or using context learning based on the memory bank (SKR_knn) or directly asking the model whether it knows the answer (SKR_prompt). 

On five different types of question answering datasets the KNN approach achieves improvements over other baselines.

Strength
1. A novel framework for an important problem -- considering when to make retrieval for retrieval-augemented LLMs .
2. The knn approach achieves mostly consistent improvements over retrieval augmented and non-retrieval augmented baselines

Weakness: more comprehensive experiment
1. Not experimented with knowledge-intensive tasks such as KILT. 
2. Not compared to baselines in IR-augmented  (Self-Ask and DSP) and reasoning (Recite-and-answer). 
3. only one type of retriever is experimented with.
4. Would be helpful to compare to related work that leverages the confidence of retrieval model (SearChain)

Weakness: limitations of the model
1. It relies on access to the training dataset, which may not be readily available.
2. The proposed method would retrieve the knowledge with the whole query rather than the part the LLM does not know.
Decision: Accept-Findings