Thrust: Adaptively Propels Large Language Models with External Knowledge
Authors: Xinran Zhao
Hongming Zhang
Xiaoman Pan
Wenlin Yao
Dong Yu
Jianshu Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: knowledge-intensive natural language processing
pre-trained language models
instance-level adaptive knowledge usage
TL;DR: We propose a simple and effective imetric, Thrust, to perform the adaptive knowledge injection, which is shown to be a good indicator of models' knowledgeability and can improve their performance of utilizing external knowledge.

Abstract: Although large-scale pre-trained language models (PTLMs) are shown to encode rich knowledge in their model parameters, the inherent knowledge in PTLMs can be opaque or static, making external knowledge necessary. However, the existing information retrieval techniques could be costly and may even introduce noisy and sometimes misleading knowledge. To address these challenges, we propose the instance-level adaptive propulsion of external knowledge (IAPEK), where we only conduct the retrieval when necessary. To achieve this goal, we propose to model whether a PTLM contains enough knowledge to solve an instance with a novel metric, Thrust, which leverages the representation distribution of a small amount of seen instances. Extensive experiments demonstrate that Thrust is a good measurement of models' instance-level knowledgeability. Moreover, we can achieve higher cost-efficiency with the Thrust score as the retrieval indicator than the naive usage of external knowledge on 88% of the evaluated tasks with 26% average performance improvement. Such findings shed light on the real-world practice of knowledge-enhanced LMs with a limited budget for knowledge seeking due to computation latency or costs.

Thrust: Adaptively Propels Large Language Models with External Knowledge
Xinran Zhao
Hongming Zhang'
Xiaoman Pan Wenlin Yao!
Dong Yu'
Jianshu Chen
Tencent Al Lab, Bellevue, 2 Language Technologies Institute, Carnegie Mellon University
Abstract
Although large-scale pre-trained language models (PTLMs) are shown to encode rich knowledge in their model parameters, the inherent knowledge PTLMs can be opaque or static, making external knowledge necessary. However, the existing information retrieval techniques could be costly and may even introduce noisy and sometimes misleading knowledge. To address these challenges, we propose the instance-level adaptive propulsion of external knowledge (IAPEK), where only conduct the retrieval when necessary. To achieve this goal, propose measuring whether a PTLM contains enough knowledge to solve an instance with novel metric. Thrust, which leverages the representation distribution of small number seen instances. Extensive experiments demonstrate that Thrust is a good measurement of PTLM models' instance-lewe knowledgeability. Moreover, we can achieve higher cost-efficieccy with Thrust score as the retrieval indicator than the naive usage of external knowledge on 88% of the evaluated tasks with 26% average performance improvement. Such findings shed light on the real-world practice of knowledge-enhancee LMs with limited nooledde-ssekinggg budget due to computation latency costs
Introduction
Knowledge is crucial for understanding human language and solving various NLP tasks [59]. In
recent years, the pre-traines trained language models (PTLM) have demonstrated great success on various
NLP tasks [10 43 44
by storing rich encyclopedic [42] and commonsense [25] knowledge in
their model parameters. However, such implicit knowledge could be opaque, static, or inefficient [23
These issues motivate the common practice of seeking external knowledge (30 57
53
17 with
information retrieval methods and augmenting the inference models (e.g., PTLMs) 20
with
the retrieved knowledge.
However, this approach has two limitations: (i) extracting external knowledge with existing infor- mation retrieval tools can be costly for a large-scale knowledge resource. (ii) external knowledge can be unnecessary or even misleading. For instance, one the best retrieving models ColBERT v2 [46] achieved 68.9 Success@5 @5 on Natural Question (27), which suggests that gold documents do not appear in the top five retrieved documents for 31.1% % of the queries. Considering the limited input sequence length, the most useful documents may not be included for generating prediction, while others may add noise to the model. On the other hand, PTLMs, which grow from millions (e.g., BERT 10 to billions of parameters (e.g., OPT 61). may solve the queries directly without
Work done during interning Tencent AI Lab, Bellevue. Corresponding contact email addresses: xinranz3Gandrew. cmu. edu, (hongmingzhang, xiaamanpan, wenlinyao, dyu, janshuchenn @global. tencent. com. Our code is available at htttps://gitae com col wwwooostttthuuuuussss. neurips2023
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Query: Who is the first woman to get the Nobel Prize in Physics?
Model: OPT 1758
Who woman
get the Nobel Prize in Physics?
Answer: Direct prodiction: Marie Curie
IAPEK
Adding Knowledge Wilhelm Conrad Rantgen discovered
Retriever
X-ray which made him the first winner of the Nobel Prize in
yes
Physic 1901
Knowledge: Wilhelmm
New Prediction:Wilee Wilhelm Conrad Rontgen X
PTLM
Conrad discovered...
Adding Knowledge Marie Curie was the first woman be awarded a Nobel Prize, ... declined to go to Stockhalm
Prediction: Curie
Marie Curie
The pipeline of -uumeete models with IAPEK. Unlike previous work (e.g., RAG (30) where models directly seek for help from the retriever module, TAPEK module provides confidence score Sg)) (e.g.. Thrust) on how well the PTLM can answer the question with internal knowledge and decides if the external retrieval is necessary.
Figure 2:
The predictions by OPT-175B without/with external knowledge retrieved via DPR (0] from Wikipedia paragraphs. Al- though the top retrieved paragraphs are rele- vant since the internal knowledge is already sufficient, the external knowledge can either be misleading (potentially due to the effect of misprime 21 or less useful.
Figure I:
external knowledge, making unnecessary to seek external knowledge that signifies the noise issue. The instance shown in Figure [ demonstrates the noise and inefficiency issues caused by external knowledge retrieval. OPT-175B can directly give the correct answer without external knowledge. However, with top external knowledge retrieved by DPR (20) from Wikipedia paragraphs, the external knowledge can be useless or even lead to wrong predictions.
The efficiency and noise issues motivate us to propose the Instance-level Adaptive Propulsion of External Knowledge (IAPEK), which adaptively retrieves external knowledge when it is necessary. We propose to measure whether a PTLM can solve question instance with internal knowledge with a confidence score and reject using external knowledge when the confidence high, instead of seeking retrieval directly for all the cases. The overall pipeline is shown in Figure 2 Specifically, we propose to solve this problem from the representation learning perspective based on two assumptions: (i) if a PTLMM has mastered sufficient knowledge about a task, its hidden states should be able to cluster the samples from the task well enough; (ii) the representation of a new instance deviates from these clusters in hidden states, this instance should be beyond the knowledge scope of the target PTLM. On top of these assumptions, we design a simple and lightweight metric Thrust to measure the distance between instance's 's representation and the clusters of several observed examples in the same task.
To comprehensively understand the effectiveness of Thrust, we conduct experiments on diverse NLP tasks. Experiments show that the average Thrust score of open- open-domain questions is significantly lower than other tasks, which aligns well with our intuition that open- QA tasks typically need external knowledge. These results indicate that Thrust is a good measurement of the models' knowledgeability. Extensive experiments also show that Thrust can improve the cost-efficiency of seeking and using external knowledge on 88% cases with 26% average performance improvement through identifying the instances that mostly require knowledge. We can also observe that, with Thrust, we can achieve higher performance than injecting external knowledge for all instances, where models are benefited from both the performance and efficiency aspects. Such findings shed light on the real-world practice of knowledge-enhanced LMs with limited budget for knowledge seeking due to computation latency or costs.
2 Approach
2.1 IAPEK: Problem Formulation
We begin with formal problem definition of Instance-level Adaptive Propulsion of External Knowledge (IAPEK). For each instance of natural language query q (e.g., a question question- answering tasks), we first determine whether the PTLM has sufficient knowledge to solve the current problem. We achieve this by using real-valued score function s(q) that assigns a higher score to g the current PTLM has enough knowledge to solve the problem and vice versa. And we will retrieve

---

Test query
Example [labele-1]
Example (label-1]
Example Cluster
Distance Vartor
Case A far away knowledge (hard)
Case 2 d close to one Knowledge Exists (easy)
Case 3 A. Interweaving
Controversial Knowledge (hard)
Figure 3: The intuition behind the proposed Thrust, which are plotted in the hidden representation space of PTLM. We represent an incoming query instance by triangles and represent the instances used for constructing Thrust scores by ticks and crosses. In the controversial and no knowledge cases, the internal knowledge is insufficient to answer the query successfully, and external knowledge is needed to facilitate PTLM. In contrast, if the model finds the query close to one of the clusters, internal knowledge should be sufficient to solve the problem so that external knowledge is unnecessary.
external knowledge to facilitate PTLM once the score s(q) falls below threshold X. By doing this, the PTLM can selectively retrieve external knowledge in an instance-adaplive manner and avoid unnecessary knowledge augmentation costs. To achieve such a mission, a critical step is to design an instance-level score function (()) that can effectively measure whether the PTLM has enough knowledge to solve the current particular input instance. In the remainder of the section, will show how to construct such a function efficiently.
2.2 Thrust: Measuring the Knowledgeability of PTLMs
We now proceed to construct the scoring function sq)) that measures the knowledgeabillty of PTLM for solving an instance from a particular task T. A seemingly straightforward approach is to adopt a supervised learning strategy train such functional mapping from -uman-llbbeldd data.
However, it is practically infeasible to manually annotate whether a given PTLM has sufficient knowledge at an instance level because the internal knowledge of PTLM is implicitly stored in its model parameters, which are hard to probe precisely for the IAPEK purpose. For this reason, we take an alternative approach by looking at the problem from the representation learning perspective. Specifically, our method is based on the following two assumptions. First, if PTLM has mastered sufficient knowledge about a task, then its hidden states should be able cluster the samples from the task well enough. For a classification task, samples from different classes should also be well separated in their (higher-level) hidden representations. Second, further hypothesize that when particular sample from the task deviates from these clusters in hidden states, this instance should be beyond the knowledge scope of the PTLM. In Figure
3 we illustrate the above intuitions for different 28 we view the built-in knowledge of PTLMM representational power that enables the deep models to learn more separable features. Based on such observations, we develop knowledgeability scoring function for a PTL.M by measuring how well it can separate samples in its hidden states.
cases. In other words, akin to the observations in 16
We now proceed to design sq)) that scores each query q from a given downstream task T. To begin with, we first collect small set of samples from task 'T and compute their hidden state representations using the designated PTLM. (Empirically, we find that about 200 samples are sufficient in our experiments.) We denote such representational mapping by function f(- For generation tasks, we treat all instances as having a single dummy label. Next, we group these embedding vectors according to their class labels Gi = {(f(xx)) y:) yı- = I). where x; and y, denote the i-th instance and its corresponding class label, respectively, and is the class index. We further cluster the samples in each G into K clusters by applying the k-means algorithm to the vectors f(;;). For convenience, we introduce the notation Cşi to represent the set of samples in the k-th cluster of class I, and let mai be the centroid vector corresponding Csi.
We use the last layer of hidden states as the embedding function. For ba-ba models, we use the last layers of the decoders.
3

---

With the above notation, we define the Thrust score for a given query instance q as:
1 N K
ICal [d
der |d (4)
Sthrust (q)
(I)
where (g) _ mA- f(g) a vector pointing from f(4) towards the centroid mys, N is the number of classes, K is the number of clusters per class, [Cul denotes the cardinality of the set Cst, and denotes normm of vector.
||
The design principles Note that the expression inside the ε--norm of ( can be viewed
weighted average of the normalized (unit) vectors {du(4)/[ld<<())) that point from the query vector (g) towards the centroid vectors {ma)) The weighting is proportional to the cluster size and inversely proportional to the squared distance between the query and the centroid. Such a design choice is based the following principles derived from the earlier representational assumption regarding knowledge. First, when samples from a task are well clustered and if is close to one of the clusters while being farther away from others, it means that the query instance q can be well solved by the internal knowledge of PTLM and the thrust score should be high. Let my be the cluster centroid that q is close to, then observe that the corresponding [|dæa]] l| term in the denominator will make the corresponding term dominate and large in (I. Second. if is farther away from all the cluster centroids, i.c., the query is beyond the knowledge scope of the PTLM, then the quadratic term |dua| would quickly suppress all the terms in ( making the thrust score vanish. Third, when the PTLM cannot sufficiently cluster the task samples in its hidden states, it means that the PTLM does not have sufficient knowledge to solve the entire task. In this case, the unit vector daa (g) (4) would randomly point towards different directions so that the averaged vector inside the f--norm of [I diminishes. Finally, the main reason that we first aggregate the samples within each class into K clusters before computing the thrust score is that they may still be spread over multiple clusters even if they belong to the same class. The term (Call in (1) is used to upweight the vectors dea (g) | that point to bigger clusters. Nevertheless, we find that K can be relatively small [ In Section 4.1 we will conduct an experimental analysis to show that Thrust score designed in the above manner is
good measurement of PTLM model's s knowledgeability. In addition, we also carry out extensive ablation studies to examine these design choices (see Appendix).
Practical considerations As we will show in Section 3 we only need about 200 samples from a task form the clusters, and then we just need to store their corresponding centroids my (typically less than 50 vectors of dimension 300) for deployment. According to (... computing the Thrust score for a query q only needs to calculate the distance between f(q) and these centroids, which takes about 0.001 second per query on average in our experiments (see Appendix). Therefore, Thrust is fairly lightweight and easy to be deployed in practical applications. The incurred extra computation complexity during the inference stage is K). Since K and N are generally small, this overhead negligible compared to retrieving knowledge from a large external memory for each instance.
3 Experiment
In our experiments, PTLMs are examined under two settings. (i) Zero-shot. We consider T5 [44). GPT-J [54), OPT 6l and present queries with or without knowledge directly to the PTLMs. (ii) Transfer-learning We use UnifiedQA [22 models that fine-tune T5 on multiple QA datasets.
After studying which model performs the best in utilizing external knowledge (the pre-condition of introducing retrievers), we then evaluate the cost-effectiveenes of the proposed IAPEK with Thrust with the best performing models. We simulate the real-world usage scenario where we have limited bandwidth or budget to retrieve external knowledge. Specifically, we test on three scenarios based on the richness of available resources: scarce (25%). medium (50%), and abundant (75%). For example, scarce (25%) means that we set the threshold X (defined in Scction to 25 percentile of the scores of the 200 examples used to set up the clusters. As introduced in Section 2 we use Thrust to score and rank the instances by their need for external knowledge and select the instances that have high demand for external knowledge (i.c., with low thrust scores). We compare the
We choose K to be max(ceil( // (impeel (), 3). where Dimmpl denotes the sample set of task T.
4

---

performance between Thrust with two baselines, i.e., IAPEK (Default) that randomly samples X% (X=25. 50 or 75) examples to apply knowledge, and LAPEK (BM25) that ranks all instances by BM25 [52 and select top X% (X=25, 50 or 75) difficult examples.
We follow previous work report accuracy for MC classification tasks. For open-domain QA, report the QA-FI score that measures the max uni-gram overlap between the model prediction and all gold answer candidates following previous work [22] For the without knowledge setting, we directly pass the prompt-decorated question query to the model and select the choice with the highest probability as the answer. For the with knowledge setting, we append the knowledge piece after the prompt-decoratte query and put "Answer: at the end to pass to the models.
3.1 Datasets
We consider several knowledge-intensive tasks in our evaluation, i.e., Multiple-choice Classification (MC), which consists of seven datasets, and Open-domain QA, which consists of five tasks. Each data instance consists a query (a piece of text containing a question or the sentences to be classified) and answer (either the label words or the answers to the questions in the query).
Additionally, each instance may also contain a piece of potentially helpful knowledge for the query, which is either inherently relevant due to the task design, annotated by humans, or retrieved from Wikipedia paragraphs with DPR. Details of the datasets and corresponding external knowledge are as follows.
Multiple-choice classification. For MC classification, each query q includes a sentence a question and requires models to select the correct answer from a set of candidates. Specifically, (i) AG- News (62) asks the model to classify piece of news into political, sports, business, or technology. We regard the titles of the news as the queries since they may already contain sufficient information and use the content of the news as the gold external knowledge. (ii) e-SNLI (6) is a natural language inference (NLI) task exploiting the role of explanations in textual entailment. Human-prov providing expla- nations are considered strong source of external knowledge; (iii) Strategy [33 a challenging multi-hop reasoning dataset that requires models to answer creative questions through strategical inference from implicit reasoning steps. We regard the original questions as queries and human- written explicit facts as external knowledge; (iv) CIKQA [60] is a commonsense inference task that combines pronoun coreference resolution, commonsense QA (11, COPA (45), and questions mined from ATOMIC knowledge graph [47]. We regard the original questions as queries and the supporting commonsense knowledge extracted from knowledge graphs (KGs) in the original work as the external knowledge; (v) BoolQ [8] contains encyclopedic questions that require models to answer yes no. Following [22,, we use the Wikipedia paragraphs retrieved by DPR as the external knowledge, which can be potentially noisy; (vi) ARC-E & ARC-C 19: ARC is challenging multiple-choice QA dataset that requires knowledge understanding and reasoning, which is partitioned to an Easy set (ARC-E) and Challenge set (ARC-C), where the Challenge
questions are answered incorrectly by the retrieval- eetreeval-based or co-occurrence- based algorithms tested by the original authors. Similarly, we use the Wikipedia paragraphs retrieved by DPR as external knowledge.
Open-domain QA. For open-domain QA, each query q contains an open question that typically requires solving an encyclopedic or commonsense inference. The generated answers can either be a few phrases or a single sentence. An example question is "What does a drink from Narcissus' 's Spring cause the drinker to do" and the expected answer generated from the language model is "fall in love with themselves". The involved datasets are HotpoiQA [58), Natural Questions (NQ) [27]. Web Questions [2], Curated TREC [I] and TriviaQA [19. We use Wikipedia paragraphs retrieved by DPR as the external knowledge as a common practice [59], except for HotpotQA, where we use the passages that the queries are generated from as a gold knowledge resource.
The statisties of the involved datasets (e.g., query length and sizes of the splits) are reported in Appendix. We collect a benchmark with various datasets of different types, formats, and knowledge sources, where we will then evaluate the effectiveness of IAPEK. Some implementation details of each of the task are described in Appendix.
5

---

lask agnews (de)
lask = esnii (cls)
lask olkqe (cla)
task - strategyqe (cie)
80-
80-
80-
80
perforance
60
60
60
60
40-
20-
20
20
0-
0
task = boolq (cls)
task = arc-e (cls)
task arcic (cls)
task wq (qa)
performance
60-
60
60-
60
40
40
40
knowledge
no
20-
20
yes
0
0
task = trec (qa)
task hotpotqa (Qa)
task = (qB)
task trivlaga (Qa)
80-
parformance
60-
60-
60
40-
40-
40
20-
0
0
0
T5-3b
GPTJ
OPT-306
UnifeedAA-3
T5-3b
GPTJ
OPT 300
UniiedOAA33
T6-36
GPTJ
OPT 300
UnifiedDA-3b
TB-3b
GPT-
OPT.306
UnifedOO--3
mode/
mode/
model
model
Figure 4: Performance of various models on MC classification tasks (accuracy) and apen-domain QA tasks (QA-FI), denoted by (cls) and (qa), respectively. The x-axis represents the model names, which are shared across sub-figures. Use knowledge: yes no denotes using full knowledge not for all queries. UnifiedQA denotes T5 models with different sizes fine-tuned on the UnifiedQA dataset.
3.2 Performance of Using External Knowledge
B7 Figure 4 presents the model performance on both the MC classification and open-domain QA tasks For the MC classification tasks, OPT), models do not consistently benefit from external knowledge. In addition, the ability to utilize external knowledge is also not improved as the parameter size grows, which indicates that simply using larger models may not be the solution for better using the knowledge. For the leaanig setting (UnifiedQA-X
observe that for the zero-shot setting (T5-X, GPT-J, and
although AGNews, e-SNLI, CIKQA, and StrategyQA are not during the training of UnifiedQA models, we can observe that models achieve better performance than vanilla T5 models different sizes. Under the with knowledge case, the UnifiedQA models achieve significant improvement for utilizing external knowledge compared to the zero-shot models, UnifiedQA-33 achieves the best performance the ability to utilize external knowledge with instances containing external knowledge. Moreover, for open-domain QA tasks, we can see models (T5, GPT-J, OPT) get no benefits from external knowledge in 11 out of 25 cases under the zero-shot setting, while UnifiedQA models achieve significant performance gain after adding external knowledge. External knowledge may introduce extra noise if the model does not learn to utilize knowledge, which indicates the importance of instructing PTLMs to learn how to use knowledge through second- stage fine-tuning.
all the tasks, which indicates that models can learn and transfer
In conclusion, we find that fine-tuning with instances containing external knowledge is an effective way help models learn use external knowledge. Since the pre-condition of using IAPEK is that the model can utilize external knowledge well, we conduct experiments with UnifiedQA only when evaluating the performance of Thrust.
the detailed numeric values, design choice ablation, and known limitations presented in Appendix.

---

Table 1: Performance of IAPEK based Thrust. As defined in Section performances of Defaul(Thrust are presented before/after the vertical bar for scarce, medium, and abundant cases. If performance increases with Thrust, the score will be marked in green and otherwise in red. WQ and TREC denote the tasks of Web Questions and Curated TREC, respectively.
UnifiedQA base mediuum
UnifiedQA large medium
UnifieQAA.. 3b medium
Dataset
scarce
abundant
scarce
ahundant
scarce
abundant
AGNews
50.7
55.6
$2.8
56.3
55.0
56.8 8
70.2
| 69.1
69.4
70.2
68.7
70.6
77.9
78.4
80.1
80.4
82.3
82.3
e-SNLI
46.5
66.6
50.7
71.1
58.5
72.2
66.4
69.1
86.3
75.9
87.3
CIKQA
56.9
59.6
57.8
59.6
58.7
59.9
60.2
62.1
60.8
62.3
61.5
62.4
62.7
66.9
64.1
66.9
65.5
66.9
StrategyQA
50.7
55.6
52.8
56.3
56.8
52.9
62.1
57.4
65.3
65.9
64.1
74.3
70.5
81.4
82.9
BoolQ
65,5
76.2
70.7
79.9
75.8
80.9
65.9
77.7
72.1
81.3
78.3
84.4
68.1
79.1
74.6
86.7
81.2
87.1
ARC-E
50.7
55.6
52.8
56.3
55.0
56.8
64.5
61.6
65.0
61.7
65.5
65.1
74.4
74.6
75.1
74.9
75.8
75.1
ARC-C
44.9
43.8
45.0
44.5
45.1
44.8
53.8
50.8
52.3
[51.2
50.9
51.55
64.5
63.9
64.4
64.9
64.3
65.6
WQ
19.2
26.3
27.5
42.1
43.8
22.5
38.5
30.5
39.0
46.0
20,9
19.3
30.0
35.4
46.4
TREC
13.5 33.6
21.3
36.4
29.1
36.9
30.8
32.7
32.7
36.0
34.6
36.3
19.6
37.8
27.0
40.6
34.4
40.9
HotpotQA
25.2
32.9
30.2
35.5
35.2
37.8
26.7
35.2
32.1
37.5
37.4
10.2
24.9
11.9
32.3
13.9
39.7
15.7
TrvisQA
32.0
52.7
43,2
56.4
54.4
60.0
32.4
59.7
46.4
64.3
60.5
71.8
39.2
68.3
52.8
71.0
66.4
73.4
NQ
20.0
33.0
24.9
33.5
29.7
33.9
12.0
34.8
20.1
33..2
28.2
35.7
12.8
35.9
21.1
36.5
29.4
37.0
3.3 Performance of Thrust
Table shsows the results of UnifiedQA after adding knowledge to X% (X=25, 50 or 75) that needs knowledge most according to our Thrust score. We can see that Thrust consistently contributes the performance from the base to the 3B model. Through clustering the instances, we acquire the whole instance distribution in the eyes of the models. Then with distance the cluster, Thrust represents how well the model can categorize a new query vector and find its similarity with others on the task. Leveraging such information, Thrust identifies the no knowledge and controversial knowledge cases well and puts the knowledge into the most necessary ones.
Additionally, the gain is higher when the portion of augmented instances is smaller. For instance, for UnifiedQA-3b, the gains from Thrust with the scarce case are 6.1 %, 13.56% on MC classification and QA tasks, respectively, while for the abundant case, the gains are 2.8% and 6.8%. Such observation shows that Thrust is most effective in identifying the most necessary cases. One potential reason is that Thrust is sensitive to the distance change, the isolated instances (no knowledge case in Figure 3 can be easily identified. With Thrust the performance of cases using fewer resources can sometimes surpass that with high resource requirement for Default, e.g., for UnifiedQA-bass on all classification tasks except ARC-C, Thrust(scarce) performs better than Default Interestingly, we also observe consistent failure cases on ARC-C. This is because the queries are designed as open questions, and the answers are usually about plans or opinions instead of facts. Thus, is hard for small models to extract useful information from Wikipedia documents.
4 Analysis
4.1 Primary Study:
Distribution of Thrust scores
To investigate whether Thrust is a good measurement of models' knowledgcability, we plot in Figure the distribution of Thrust scores with the strongest inference model evaluated (i.c., UnifiedQA-3b We use Kernel Density Estimation to smooth the distribution. From the distribution, we can see that low Thrust scores (i.e., the query needs external knowledge) frequently appear in most query instances of the open-domain QA tasks such as HotporQA, TriviaQA and ÑQ, which are designed to solve with external knowledge. On the other hand, for e-SNLI and BoolQ, external knowledge is not always necessary, which is consistent with the design purpose of these tasks. To conclude, by correctly predicting whether certain task needs external knowledge (e.g., open-domain QA tasks), Thrust score is shown to be good measurement of PTLM model's 's knowledgeabilityy.
*For a clear presentation, we only include representative tasks directly relevant to knowledge and reasoning from each category. The selected tasks are: e-SNLI BoolQ. CIKQA, HorporQA, TriviaQA, and NQ. The distribution of all tasks can be found in Appendix, where the trend is consistent. In visualization, the score can be lower than due smoothing. The real scores will always be greater than zero.
7

---

Task
esnli (cls)
boolq (cls)
Density
0.8
cikqa (cls)
hotpotqa (qa)
0.6
triviaqa (qa)
nq (qa)
0.4
0.2
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
1.25
1.50
thrust score
Figure 5: Distribution of Thrust scores for various tasks by using UnifiedQA-3b to create the hidden representations. Kernel Density Estimation is used to smooth the distributions. Low scores imply that the instances less likely to be solved with internal knowledge and vice versa. Thrust scores predict that most query instances from open-domain QA tasks require external knowledge while others need less. The results consistent with the original design purposes of these tasks.
Table 2: Performance of IAPEK for UnifiedQA-3h based on Thrust and BM25. With scarce, medium, and abundant, performances of Default/BM25 and Default/Thrust are presented before/after the vertical bar. If performance increases with Thrust, the score will be marked in green, otherwise red.
BM25 medium
Thrust medium
scarce
abundant
scurce
abundant
AGNews
77.9
77.0
80.1
79.2
82.3
81.3
77.9
78.4
80.1
80.4
82.3
82.3
e-SNI.1
75.6
83.0
87.5
CIKQA
62.7 62.3
64.1
64.4
65.5
66.1
62.7
66.9
64.1
66.9
65.5
66.9
StrategyQ
64.1 63.3
70.5
68.3
77.0
78.1
64.1
74.3
70.5
81.4
77.0
82.9
BoolQ
68.1
68.4
74.6
75.9
81.2
82.0
79.1
74.6
85.7
81.2
87.1
ARC-E
74.4
74.9
75.1
75.3
75.8
76.3
74.4
71.6
75.1
74.9
75.8
75.1
ARC-C
66.2
64.4
WQ
20.9 19.0
30.0
28.2
39.1
37.3
20.9
19.3
30.0
35.4
39.1
46.4
27.0
28.1
34.4
36.3
37.8
27.0
40.6
HatpoiQA
24.9
25.2
32.3
32.8
39.7
40.6
24.9
41.9
32.3
43.9
39.7
45.7
TriviaQA
39.2
34.2
$2.8
50.0
66.4
65.4
39.2
68.3
52.8
71.0
66.4
73.4
NQ
21.1
29.4
29.6
36.5
29.4
37.0
4.2 IAPEK-Thrust versus IAPEK-BM25
52 common approach to evaluating the difficulty of queries, as an alternative to Thrust perform IAPEK. Specifically, we regard each test input the query and all training data input as the corpus to extract the score. We use the average relevance score across the corpus to rank each input. From Table 2 We can observe IAPEK performs well with BM25 the difficulty score on QA tasks. Except WQ and NQ, we observe better performance (marked in green) than the default setting. However, Thrust shows larger (e.g., for QA tasks) and more robust improvement (e.g., for classification tasks) than the BM25 baseline.
We use BM25
4.3 Layer ablation
Since we cast instances into the representation space, a crucial factor for Thrust is the layer of the PTLMs use. To investigate the effect of using different layers, we conduct experiments on UnifiedQA-3b with the same setting as in Section 3 Figure 6 presents the performance of adding 25%, 50%, and 75% knowledge-augne -ugmmented instances with Thrust with hidden states of different layers. We can observe that, for most tasks, there is no significant difference across layers, which shows the robustness of Thrust and potential to accelerate the computation by using early layers. However, for StrategyQA and Web Questions, the middle-layer representation may worsen the overall performance. One possible reason that early layers the model contain rich semantie information, and later layers contain task information 33 so both can act good representations of the instances. However, in the middle layers, rich semantic features are abandoned during extracting task-specific features.

---

80
boolq
mebquestions
cley
curatedtrec
holpotea
sratoggaa
50
trivang
agraws
40-
pertion
portion
50
25
75
layer
layer
(h)
Figure 6: Layer-wise ablation across tasks and portions of instances augmented with knowledge for (a) MC classification tasks and (b) open-domain QA tasks. The x-axis denotes the layer index of Unified-3b decoder that is used to obtain the hidden representations (i.e., f(.) in (T). For most tasks, the results are not sensitive to the specific layer index. For some tasks (e.g., StrategyQA), choosing middle layers for representation slightly degrades the performance.
Table 3: Comparison between IAPEK- -hrust and the costly full knowledge usage based on Uni- fiedQA models of different sizes. 99% Full denotes 99% performance of models with full knowledge. The knowledge type is noted in brackets, where denotes gold knowledge, h denotes human annota- tions, and denotes the knowledge retrieved from Wikipedia or knowledge graphs. WQ and TREC stand for Web Questions and Curated TREC, respectively. If Thrust achieves better performance than using full knowledge, we mark the entry with
Size
Thrust>99% Full
Thrust-999 Full
BoolQ(ri* ARC-E(r)
CIKOAG ARC-CIrj
AGNews(g)* WQG))
StrategyAA(h)* TrivisQAGri*
e-SNLIb)
TREC(r)
HoipatQAgg)
NQ())
BoulQiri* ARC-E())
CIKOAG ARC-Ct0"
AGNewslg) TriviaQAm
QtrateegQAAbh) HotporQAíg)*
e-SNLKb)
TREC(A)
WQ(r)
NQ())
3b
4.4
Comparison with Full Knowledge Usage
We denote simply using external knowledge for all instances as costly but straightforward way of leveraging external knowledge. Since the big models might be sufficient for certain instances and the external knowledge might introduce extra noise, Thrust can help identify instances requiring (or not) knowledge and achieves higher overall performance on the whole dataset compared to seeking and adding knowledge indiscriminately. Table 3 presents the comparison between adaptive and indiscriminate knowledge propulsion. Thrust here denotes the best performance achieved when less than 90% of instances exteral knowledge. We can observe that, for 2/3 tasks for UnifiedQA- -basse and UnifiedQA-3b. Thrust achieves better performance or less than 1% drop than using knowledge for all instances. Such results illustrate that Thrust can help avoids potential noise. On the other hand, we can also observe that for e-SNLI, TREC, and NQ. the full knowledge setting performs better than Thrust. It means retrieving and adding high-quality knowledge always benefits the models as long as they have been fine-tuned to know the usage of external knowledge.
5 Related Work
PTLM with external knowledge. The paradigm of retrieving knowledge from knowledge bases, augmenting PTLMs, and solving downstream tasks has been widely explored in the community of
NLP 4 39 48 The knowledge bases can range from knowledge graphs [57], documents [40],
pre-processed vectors 53], other PTLMMs [49], search engines (34, Wikipedia documents as used in this work. To augment the PTLMs,, common practice includes creating synthesizing datasets [56],
adding knowledge to the prompts 55 36 create demonstrations [5), and extending feature vec-
tors (24). The contribution of LAPEK is orthogonal to the above work since it presents gated

---

framework to reject external annotations or retrieval that can be extended to the above frameworks. The extension is lightweight since Thrust requires queries only, not labels nor gold answers.
Hardness and Confidence Estimation in PsLMs. Much previous work studies the estimation of dataset hardness and model confidence under the context of PILMs. For dataset hardness, previous research discovers using the cumulative area under the loss curves (RDA [41),, entropy difference between the trial and null cases (V-Usable information MI], and variance of losses of neighbor sentences (Sensitivity Measurement 14]) These methods achieve great correlation with the model performance when analyzing the test set. However, the test set labels are required and can be applied when predicting the answers. Another line of work focuses on estimating the expected calibration errors (ECE) for classification [26). QA [18]. and math [3]] datasets, as a reflection of model certainty on the correct answers. ECE can be considered an orthogonal evaluation metric to measure the model's capability of understanding the tasks, compared to common metries such as accuracy.
Most previous work can be considered posterior analysis of the model capability. In this work, instead, we estimate the pragmatic confidence at the test time to empirically increase the performance with limited budget or bandwidth acquire knowledge.
Discussion
6.1 Extended Usage
We expect the idea of adaptive knowledge injection be extendable beyond QA and MC questions, such as trail prediction tasks including ECBD (37 or EKP [38] By design, Thrust is independent of the type of external knowledge, so that the adaptively used external knowledge can be any sort, for example, definitions in EKP. Furthermore, Thrust can also be used as way to measure the expected performance without extensive fine-tuning, as shown Table 5 in the appendix.
On the other hand, we present our pioneer study on instruction fine-tuned model Flan-T5 [ in Section A.4 in the appendix and show that Thrust performs better on CIKQA with Flan-T5 than vanilla T5, where the best performance is achieved with 40% examples not using external knowledge.
We also believe future work can be done how Thrust collaborates with different kinds of prompts to include the knowledge or prefix tuning. We discuss the limitations of our current design (i.e., cold start, assuming white-box language models, and extendability on other kinds of retrieval argumentation) in Section A.I in the appendix.
6.2 Time Sensitivity
We regard time sensitivity part that can be done in the IAPEK framework, but not by Thrust, as the framework is motivated by both noise and staticity issues. Another orthogonal kind score measuring time sensitivity can be designed to decide if updated knowledge retrieval is necessary, for example, based on 35
7 Conclusion
In this work, we propose Instance-level Adaptive Propulsion of External Knowledge (IAPEK) as solution to propel model performance with external knowledge. Accordingly, we propose a simple and effective instance-level metric, Thrust, to perform the adaptive knowledge injection. Extensive experiments show that Thrust is good indicator of models' knowledgeability and can improve the performance utilizing external knowledge under various settings. Understanding the delicate usage of potentially noisy knowledge for PTLMs can further enable the models to conduct inference beyond the limitation of internal knowledge.
10

---

8 Acknowledgements
We are grateful to Sherry Tongshuang Wu, Shikhar Murty, Zhengxuan Wu, Ashwin Paranjape, John Hewitt, Christopher Manning, Eric Mitchell, Sihao Chen, Ben Zhou, Tong Chen, and the anonymous reviewers for their helpful and insightful comments.
References
[1] Petr Baudiš. Yodaqa: a modular question answering system pipeline. In POSTER 2015-19M1 International Student Conference on Electrical Engineering, pages 1156- 1165, 2015.
[2] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533-1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.
[3] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean- Sann-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring. Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan. Jack W. Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens, 2022.
[4] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, -ean-Bap Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving. Oriol Vinyals. Simon Osindero, Karen Simonyan. Jack Rac, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 2206-2240. PMLR, 17-23 Jul 2022.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vone Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[6] Maria Camburu, Tim Rocktäschel, Thomas Lukasiewiez, and Phil Blunsom. e-snli: Natural language inference with natural language explanations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa Resa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
[7] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghami, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. language models, 2022.
Scaling
[8] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACI, 2019.
[9] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs51803. 2018

---

[10] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019, pages 4171 171661 2019.
[11] Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 5988- 6008. PMLR, 17-23 Jul 2022.
[12] Luyu Gao and Jamie Callan. Condenser: a pre-trainng architecture for dense retrieval. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 981-993, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[13] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguisties (TACL., 2021.
[14] Michael Hahn, Dan Jurafsky, and Richard Futrell. Sensitivity as Complexity Measure for Sequence Classification Tasks. Transactions of the Association for Computational Linguistics, 9:891-908, 08 2021.
[15] Hangfeng He, Hongming Zhang, and Dan Roth. Rethinking with retrieval: Faithful large language model inference, 2022.
[16] Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313:5786):504-577, 2006.
[17] Gautier Izacard, Patrick Lewis. Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03399, 2022.
[18] Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering. Transactions of the Association for Computational Linguistics, 19962-977, 09 2021.
[19] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprchension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1601-1611, 1, Vancouver, Canada, July 2017. Association for Computational Linguistics.
[20] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 6769-6781, Online, November 2020. Association for Computational Linguistics.
[21] Nora Kassner and Hinrich Schütze. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811 7811-7818, Online, July 2020. Association for Computational Linguistics.
[22] D. Khashabi, S. Min, T. Khot, A. Sabhwaral, O. Tafjord, P. Clark, and H. Hajishirzi. Unifiedqa: Crossing format boundaries with a single qa system. 2020.
[23) Omar Khattab, Christopher Potts, and Matei Zaharia. Building scalable, explainable, and adaptive nip models with retrieval, 2022.
[24] Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextual- ized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 220, page 39 49, New York, NY, USA, 2020. Association for Computing Machinery.
12

---

[25] Vid Kocijan, Ana-Maria Cretu, Dana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. surprisingly robust trick for the Winograd schema challenge. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4837-4842, Florence, Italy, July 2019, Association for Computational I Linguistics.
[26] Lingkai Kong, Haoming Jiang, Yuchen Zhuang. Jie Lyu, Tuo Zhao, and Chao Zhang. Calibrated language model fine-tuning for in- and out-of-distribution data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1326-1340. Online, November 2020. Association for Computational Linguistics.
[27] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Compulational Linguisties, 7:452-466, 2019.
[28] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436-444 2015.
[29] Patrick Lewis, Fthan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sehastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nip tasks. In Proceed- ings of the 34th International Conference on Neural Information Processing Systems, NIPS' 20, Red Hook, NY, USA, 2020. Curran Associates Inc.
[30] Patrick S. H. Lewis. Ethan Perez. Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih. Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Rerrieval-augmented generation for nhwwledgeinntensive NIP tasks. CoRR, abs/2005.1 1401, 2020.
[31] Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words, 2022.
[32] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 1692, 2019.
[33] Charles Lovering, Rohan Jha, Tal Linzen, and Ellie Pavlick, Predicting inductive biases of pre-trained models. In International Conference on Learning Representations, 2021.
[34] Reicchir Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted muestion-answering with human feedback. CoRR, abs/21 12.932,,, 2021.
[35] Qiang Ning, Ben Zhou, Hao Wu, Haoruo Peng, Chuchu Fan, and Matt Gardner. A meta- framework for spatiotemporal quantity extraction from text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2736 2749, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[36] Maxwell I. Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models. CoRR, abs/21 12.00114. 14, 2021.
[37] Yasumasa Onoe, Michael Zhang, Eunsol Choi, and Greg Durrett. Entity cloze by date: What LMs know about unseen entities. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 693-702, Seattle, United States, July 2022. Association for Computational Linguistics.
[38] Yasumasa Onoe, Michael J. Q. Zhang, Shankar Padmanabhan, Greg Durrett, and Eunsol Choi. Can Ims learn new entities from descriptions? challenges in propagating injected knowledge, 2023.
13

---

[39] Xiaoman Pan, Wenlin Yao, Hongming Zhang, Dian Yu, Dong Yu, and Jianshu Chen. Knowledge- in-context: Towards knowledgeable cemi-parametric language models. CoRR, abs/2210.16433, 2022.
[40] Ashwin Paranjape, Omar Khattab, Christopher Potts, Matei Zaharia, and Christopher D Manning. Hindsight: Posterior Posterior-guided training of retrievers for improved open-ended generation. In International Conference Learning Representations, 2022.
[41] Ethan Perez, Douwe Kiela, and Kyunghyun Cho. Rissanen data analysis: Examining dataset characteristics via description length. In ICML, 2021.
[42] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference Empirical Methwds in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP,, pages Kong, China, November 2019. Association for Computational Linguisties.
Hong
[43] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8):9, 2019.
[44] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu, Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
[45] Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alterna- tives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pages 90 90-95, 2011.
[46] Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3715-3734, Seattle, United States, July 2022. Association for Computational Linguistics.
[47] Maarten Sap, Ronan LeBras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. ATOMIC: atlas of machine commonsense for if-then reasoning. CoRR, abs/1811.001144 2018.
[48] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Sco, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. Replug: Retrieval- -ugmmeted black-box language models, 2023.
[49] Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula. and Yejin Choi. Unsupervised commonsense question answering with self-talk. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 4615-4629, Online, November 2020. Association for Computational Linguistics.
[50] Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for language-model-aag sassssee.........-sssi....... In Proceedings of ICML, 2022.
[51] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, Volume I (Long and Short Papers), pages 4149-4158, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics.
[52] Andrew Trotman, Antti Puurula, and Blake Burgess. Improvements to bm25 and language models examined. In Proceedings of the 2014 Australasian Document Computing Symposium, ADCS 14, page 68-65, New York, NY. USA, 2014. Association for Computing Machinery.
[53] Pat Verga, Haitian Sun, Livio Baldini Soares, and William Cohen. Adaptable and interpretable neural MemoryOver symbolic knowledge. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages
Online, June 2021. Association for Computational Linguistics.
14

---

[54] Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.
http://www....o.oooooooooo///a/a transformer- ja May 2021.
[55] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11903, 2022.
[56] Yuhuai Wu, Markus N Rabe, Wenda Li, Jimmy Ba, Roger B Grosse, and Christian Szegedy, Lime: Learning inductive bias for primitives of mathematical reasoning. In International Conference on Machine Learning, pages 11251-1 1262. PMLR, 2021.
[57] Yichong Xu, Chenguang Zhu, Shuohang Wang, Siqi Sun, Hao Cheng, Xiaodong Liu, Jianfeng Gao, Pengcheng He, Michael Zeng, and Xuedong Huang. Human parity on commonsenseqa: Augmenting self-attention with external attention. CoRR, abs/2112.0325, 2021.
(58) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
[59] Da Yin, Li Dong, Hao Cheng, Xiaodong Liu, Kai-Wei Chang, Furu Wei, and Jianfeng Gao. A survey of knowledge-intensive nip with pre-trained language models. ArXiv, abs/2202.08772. 2022.
[60] Hongming Zhang. Yintong Huo, Yanai Elazar, Yangqiu Song, Yoav Goldberg, and Dan Roth. CIKQA: Learning commonsense inference with a unified knowledge-in-thien -edge-intthe-lopp QA paradigm. In ACL 2022 Workshop on Commonsense Representation and Reasoning, 2022.
[61] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
[62] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[63] Xinran Zhao, Shikhar Murty, and Christopher D. Manning. On measuring the intrinsic few-shot hardness of datasets. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2022.
15

---

A Appendix
A.1 Limitations
Cold Start. In the ideal case, module distinguishes if a query requires external can do it in zero-shot manner. However, as we show in Section 4.1 of our paper, we practically find that the distribution of Thrust scores of various tasks can be very different due the essence of the task collection and the type of external knowledge needed. Considering such an effect, we need 200 examples to estimate the clusters needed to set up the computation, which is still lightweight in real-world scenarios. In the future, we will explore the usage of meta-learning to allow few-shot start or even a cold start of IAPEK.
Black-box LLM: At the current stage, our model can work with first-layer or last-layer representations (as discussed in Figure 6 of our submitted paper), which provided by some black-box models. To adapt to completely black-box LLMs, there are two ways we can think at this stage: (1) similar to Black-Box Tuning (5). use the prompt embeddings adjusted by the derivative-free optimizer optimized over the black-box model outputs as our representation; (2) we use original or distilled smaller models from the same family acquire representation (e.g., original GPT-2 or GPT-2 fine-tuned by a set of query and answers from GPT-4). We experimented with using T5-base representation to conduct IAPEK for T5-large models. It showed slightly worse but not completely ruined performance.
Extension to other Retrieval-augmennte Models. In this paper, propose new module for the pipeline of Strreval--uggeett models. We first comprehensively examined if and how external knowledge is useful with language models. Next, examine the performance of the module IAPEK with the lightweight Thrust score we define as a potential implementation. We compare Thrust with BM25 with the default setting of retrieval augmented language models [17 and show its effectiveness. Since queries, not answers nor retrieved knowledge are required to set up Thrust, it can be applied to various other frameworks of retrieval augmented models (3. However, it is beyond the scope of the project at the current stage, and the contribution of our module and the frameworks are orthogonal. We will extend Thrust to other eetrieval-augmennt models future work.
A.2 Implementation details
We conduct experiments machine with 8 Nvidia P40 (24G) GPUs with CUDA 11 installed.
We use the Scikit-learn package to measure the clusters with K-means and compute the distance between the query and cluster representations. The involved hyperparameters (including the number of clusters per class) are selected by Grid search on a smaller set of experiments. We initialize all parameters randomly or the default of the Hugginface transformers package O On average, each run of extracting the results for all the tasks under with/without knowledge cases takes around 20 hours. We all experiments times and report the averaged performance in the main content. For hyperparameters of the inference models, for the QA task, we set the maximum knowledge length as 480 tokens ensure that query sentences stay in the input.
The generated answer for QA tasks for all the models is typically within 30 tokens. For classification tasks, for binary classification tasks (CIKQA, StrategyQA, BoolQ, and e-SNLD, we follow previous work to use "Yes or No?" as the suffix to the original query to guide the generative models. For AGNews, we use "political news, sports news, business news, and technology news" as the label words. We found that the default label word "word news" will largely degrade the performance of generative models AGNews. We add "the news is about?" and provide the candidate categories as the suffix for AGNews. More details of our implementations can be found in the code attached.
A.3 Dataset Details
The detailed statistics of the involved datasets are shown in Table 4 We sample 200 data points from each dataset to conduct the clustering step of Thrust. Difference datasets have different average query lengths and knowledge lengths due to the essence of the task creation and knowledge collection.
hatpp://ecikkit-eeern.c org/ https://buggiigtacce.com
16

---

Table 4: Statistics of the selected datasets. Sample # denotes the number of examples used calculate the clusters for Thrust scores. ARC-E and ARC-C denote the easy and hard ARC datasets. Q/K/A Len denotes the average number of words for the Queries/K nww.edge/weeeoooooooooo Answers, respectively.
Dataset
Source
Sample #
Test #
Q Len
K Len
A Len
AGNews
gold
200
7,600
8.1
35.9
1.0
e-SNLI
human
200
9,824
24.9
14.3
1.0
StrategyQA
human
200
229
10.8
33.5
1.0
CIKQA
KG
200
604
18.2
28.0
1.0
BoolQ
retriever
200
3,270
9.8
113.8
1.0
ARC-E
retriever
200
570
23.1
238.2
4.2
ARC-C
retriever
200
299
26.2
240.5
5.5
HotporQA
gold
200
7,405
19.0
56.3
2.5
NQ
retriever
200
6,468
10.1
588.9
2.3
Web Questions
retriever
200
278
7.8
117.3
4.3
Curated TREC
retriever
200
116
8.4
116.5
7.7
TriviaQA
retriever
200
6,760
15.0
117.6
27.5
0.76
acc
roberta
0.72
flan
fiedea
0.68
0.0
0.2
0.4
0.6
0.8
1.0
rejected
Figure Performance of different models on CIKQA with different thresholds of Thrust. The X-axis denotes the portion of test examples that are selected to not use external knowledge. All model names denote the large versions of the model parameters.
Answer length denotes tasks with yes and answers. Otherwise, the answers with than token are either choices (for ARC-E and ARC-C) or free-form text sentences (for open-domain QA tasks). Examples of the dataset be found in the attached data.
A.4 Experiment with Flan-T5
Figure presents the performance of Thrust on CIKQA with different models. From the figure, we can observe that Thrustperforms better with instruction fine-tuned Flan-T5 compared to the original T5 and UnificedQA. With Flan-T5 Thrust achieves better performance with 40% examples rejecting external knowledge usage compared external knowledge used either on no or all examples. Such observations show the potential of using Thrust on current iestruction-finetumme models.
A.5 Ablation on the design choices of Thrust
Following 663, we use few-shot multitask binary NLI dataset to test influence of each factor of Thrust (i.e., FS-NLI), through measuring how well the metric and its variants can measure the with the hardness diverse set of datasets. From Table we can observe that all the design choices are crucial to the success of using Thrust detect how hard query is for given task and model.
17

---

Table Compare Thrust its various variants following the setting of the original work [3], where higher correlation denotes that the metric can better capture the hardness of tasks with respect to given model (RarERTa-large 32 without direction denotes the variant to use scalar instead of vectors for Thrust. The best- best-performing entry is marked in bold.
Metric
Correlation
Thrust
0.45
without cluster size
0.23
without direction
0.19
without distance
0.06
cosine distance one cluster per label class ten clusters per label class
0.08
0.32
0.12
cluster size to inertia
0.03
Task
agnews
0.35
boolq
cikqa
esnli
strategyqa
Densty
0.25
arc-easy
arc-hard
webquestions
0.15
curatedtrec
hotpotqa
triviaqa
0.05
nq
0.00
-0.5
Figure Distribution of Thrust scores for all involved tasks with UnifiedQA-30 create the instance representation. The distribution is normalized by Kernel Density Estimation. Low scores denote the cases where internal knowledge not enough, and vice versa.
A.6 Full distribution of Thrust across tasks
Figure S demonstrates the distribution of Thrust scores for each of the involved tasks. Besides the findings in the main content that Thrust can help identify the knowledge necessity for various tasks through viewing the distribution, we can also observe that Thrust can lead to a diverse distribution of scores that may contain multiple peaks.
A.7 Performance of using external knowledge (in table)
Table 6 presents the performance in Figure of the original submission table format. Similarly, we can observe that it is not trivial to use external knowledge, especially in the zero-shot settings. it possible that models get worse performance with external knowledge, for example, for ARC-C, both T5-base and T5-large show worse performance with the extra knowledge injected. Also, can observe that external knowledge is crucial for open-domain QA tasks. The gain can huge, for example, for UnifiedQA-3b. the performance is improved from 18.6 to 80.0, in terms of QA-FI on TriviaQA, with the external knowledge.
18

---

tack og rowr (ol))
1a50 osni (461
took clup (99g
lask str.loggggaaoooo (dla)
40-
20
20
20
20
0
book bookl (554)
Leak Mi0H
Toed MECH (5.0)
5400 (500
40
40
20
20
Pustoolga
Look
SAnC
40-
60-
00
50
40
40
20
0013
Unitng ao
Una Soor
2013
Undfeel tork Unian Aaor
Tuang st
Unv tore
Unc Ae0
mode
meae
medyy
moddy
Figure 9: Performance of various models MC classification tasks (accuracy) and open open-domain QA tasks (QA-FI), denoted by (cls) and (qa), respectively. The x-axis represents the model names, which shared across sub-figures. Use knowledge: yes denotes using full knowledge not for all queries. UnifieQQA denotes T5 models with different sizes fine-tuned on the UnifiedQA dataset.
Table 6: Performance of various models on the MC classification tasks (accuracy) and open-domain QA tasks (QA-FI). Performances without/with knowledge external knowledge are presented be- fore/after the vertical bar, respectively. UnifiedQA-X denotes T5 models with corresponding sizes fine-tuned on the UnifiedQA dataset.
Model
parameters
AGNews
SNLI
CIKQA
StrategyQA
BoolQ
ARC-E
ARC-C
Zero-shot
220M
51.8
38.9
TS-large
770M
25.8
25.2
65.7
65.7
50.0 50.0
53.3
53.3
37.8
38.6
25.1
27.7
27.7
24.7
T5-3b
3B
27.9
61.5
52.6 50.5
44.5
48.9
56.6
45.3
25.8
26.0
26.4
GPT-J
68
25.1
26.9
40.8
37.0
49.8
50.7
47.2
55.9
60.2
47.2
25.4
29.5
28.4
27.1
OPT-306
30B
25.0
25.0
65.7
65.7
50.0
50.0
53.3
53.3
37.8
37.8
27.4
27.7
25.8
26.4
Transier.
UnifieQQA-bnce
220M
46.6
35.7
38.5
70.2
56.0
59.6
48.5
57.2
60.4
80.8
50.2
61.6
44.8
45.2
UnifaeQQA-large
770M
71.0
67.9
42.8
74.2
59.6
62.1
48.5
66.4
59.8
84.5
64.0
66.0
55.2
49.5
UnitiedQA-3b
Model
parameters
Web Questions
Curated TREC
HotporQA
NQ
TriviqQA
Zero-shot
T5-base
220M
6.7
8.7
2.5 3.8
6.0 99.1
1.9 6.0
8.9 13.1
T5- Jarge
770M
5.7
7.4
3.0
5.1 6.6
1.6 2.7
8.3 9.0
T5.3b
3B
4.9
4.0
1.0
668
1.7 6.6
8.3 5.6
GPTT-
68
4.3
6.3
7.4 2.7
5.9 $.1
10.9 6.9
1.7 2.1
11.4
2.1
Transfer-learning
UnifiedQA A-a.iiAAAAa base
220M
10.6
44.2
3.6 36.9
13.1 |40.
3.0
34.6
11.7 65.6
UnifedQAA-lat
770M
42.7
13.7
UnifedQA-3b
3B
11.5 48.1
9.9 41.8
17.0 47.0
4.4 37.6
18.6 80.0
19


Reviews:
[{"review_id": "vpsq31ChzA", "reviewer": "Reviewer_eb2S", "summary": "The authors propose a thrust score which measures if a pre-trained language (PTLM) model has the knowledge to perform the task. They then go on to use this score to choose when they should use external knowledge (when the thrust score is less). The main crux of the thrust score is knowledge representation. The authors argue that if the PTLM places a sample close to related samples, then it has sufficient knowledge about the sample.\n\nI feel the thrust score is very good contribution. The adaptive lookup for knowledge experiments are useful, though there might be other ways to use this thrust score.", "questions": "1. Can the thrust score be used for knowledge injection using additional pre-training or fine-tuning? Some time ago, there was a paper that used to additionally pre-train on domains where it was performing worse.\n\n2. Have you considered knowledge prompts or pre-fix tuning on downstream tasks besides adding the external knowledge and \"Answer:\" to the prompt?\n\nThere is a typo on line 271. Thrust can help identifies instances requiring", "limitations": "There isn't a separate discussion on limitations but they do mention tasks where this approach under-performs baselines.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors propose a thrust score which is a measure of a pre-trained language models knowledge of the instance. These are based on the distance of a sample to the centroid of a cluster. The clusters are task examples. There can even be mulitple clusters within a class in a classification task. This is measured from representations in the last hidden layer (last decoder layer in T5).\n\n2. Given the authors have a thrust score, they proceed to lookup external knowledge only when the model does not have internal knowledge to predict on a sample. The hypothesis is that looking up external knowledge all the time is wasteful and some times even counter productive because of noisy external knowledge.\n\n3. Overall I think the thrust score is a very useful contribution.\n\n4. The ablations - checking which layer to use for representations, comparison with BM25, comparison with full knowledge usage are meaningful.", "weaknesses": "1. The adaptive knowledge injection (while useful and demonstrates that the thrust score is effective) could have benefited with knoweldge probing experiments rather than just QA, MC etc. The performance on (triples) tail prediction task, or even knowledge probing like in the below work, can add be even more interesting.\n\nOnoe, Y., Zhang, M.J., Padmanabhan, S., Durrett, G. and Choi, E., 2023. Can LMs learn new entities from descriptions? Challenges in propagating injected knowledge. arXiv preprint arXiv:2305.01651.\n\n2. Some of the claims like below seem a bit subjective and can be avoided. Atleast from the abstract.\n\n\"we can achieve significantly higher cost-efficiency with Thrust score as the retrieval indicator than the naive usage of external knowledge on 88% of the evaluated tasks with 26% average performance improvement.\"\n\n3. The results could be presented a bit better. For example, figure 4 seems unnecessarily complicated.\n\nIt's ofcourse the author prerogative, but it took me a while to get used to the thrust, propulsion terminology.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments and suggestions. We would like to address your concerns as follows. \n\n**Extended Usage (W1 & Q1)**: we agree that adaptive knowledge injection can be extended to other cases such as ECBD or EKP (Onoe et al., 2022, 2023). We will include this line of work in the discussion in the camera-ready version. \nBy design, Thrust is independent of the type of external knowledge, so that the adaptively used external knowledge can be any sort, for example, definitions in EKP. Furthermore, Thrust can also be used as a way to measure performance without extensive fine-tuning. In Section A.4 in the appendix included in the supplementary material, we show that Thrust can also be regarded as a dataset hardness metric following the setting of (Zhao et. al., 2022), so that, with an active learning scheme (e.g., Tamkin et. al., 2022), we can use Thrust to rank example hardness and potentially schedule the pre-training or fine-tuning.\n\n**Presentation (W2,3 & Limitations)**:  (1) We will remove the subjective expressions (e.g., significantly) and correct all typos. (2) We will simplify the figure by only presenting the results for a part of the models: T5-3b, GPT-J, OPT-30b, and UnifiedQA-3b. We will put the detailed figure in Appendix. (3) We will change propulsion to augmentation. For the term Thrust, we will consider seeking a better acronym; (4) Our limitation discussion was included in Appendix. We will put them back in the main doc in the final version. We discussed the cold start and black-box LLM problems. Thanks so much for the suggestions!\n\n**Knowledge Prompts (Q2)**: In this paper, we mainly study when we shall add external knowledge and the way to use that knowledge is out of the scope of this work. We mainly follow the setting of the best knowledge utilization model in our experiments (i.e., UnifiedQA).\nWill add the discussion about the potential to incorporate Thrust with knowledge prompts and prefix tuning as an important future direction. \n\n**References**:\n\nXinran Zhao, Shikhar Murty, and Christopher D. Manning. 2022. On measuring the intrinsic few-shot hardness of datasets. In Proceedings of EMNLP.\n\nAlex Tamkin, Dat Pham Nguyen, Salil Deshpande, Jesse Mu, and Noah Goodman. 2022. Active learning helps pretrained models learn the intended task. In Advances in Neural Information Processing Systems\n"}, {"review_id": "B4UbJ7AwNT", "reviewer": "Reviewer_2iTX", "summary": "The paper addresses the limitations of large-scale pre-trained language models (PTLMs) in effectively utilizing external knowledge. It proposes the instance-level adaptive propulsion of external knowledge (IAPEK) as a solution to leverage external knowledge only when necessary. The paper introduces a novel metric called \"Thrust\" to measure the knowledgeability of PTLM models at the instance level, using the representation distribution of a small number of seen instances. Extensive experiments demonstrate that Thrust is an effective measurement of PTLM models' instance-level knowledgeability. The paper shows that using the Thrust score as a retrieval indicator achieves significantly higher cost-efficiency compared to naive usage of external knowledge, resulting in a 26% average performance improvement on 88% of the evaluated tasks. These findings contribute to the understanding and real-world application of knowledge-enhanced language models, particularly in scenarios with limited knowledge-seeking budgets due to computation latency or costs.\n\n\n\n\n", "questions": "How does IAPEK combine with instruction fine-tuning of large models?", "limitations": "The authors do not discuss the significance of this work in the context of large models.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1、The authors clearly highlight the limitations of implicit knowledge in pre-trained language models (PTLMs), such as being opaque, static, and inefficient. This acknowledgment sets the stage for proposing a novel approach to address these limitations.\n\n2、The authors introduce the concept of Instance-level Adaptive Propulsion of External Knowledge (IAPEK) as a solution to address the limitations mentioned earlier. ", "weaknesses": "1、The methods and experiments are described in an informal and obscure manner, lacking motivation for the specific choices made and failing to compare them with alternative approaches. Moreover, this paper lacks the experimental comparison and discussion with ChatGPT and other instruction fine-tuning large models\n\n2.  A major weakness of this work is its lack of reproducibility. The paper fails to provide clarity on whether the external knowledge used is published or not, and it does not explain how one can obtain it. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments and suggestions. We would like to address your concerns as follows. For your reference, our design choice ablation and reproducibility check are provided in Appendix and referred to in footnotes 4 and 1. Details are as follows:  \n\n**Design Choice (W1)**: We kindly refer you to Appendix (A.4 and Table 2) included in the supplementary materials. As said in Footnote 4 of page 5 in our submission, we presented our design choice ablation and known limitations in the appendix. We compared the alternative choices such as no cluster size, no direction, using inertia, etc, and validated our design of Thrust.\n\n**Reproducibility (W2)**: We kindly refer you to Footnote 1 of page 1 of our submission, where we promise that “the code and data(including the external knowledge collected) will be released upon acceptance”. On the other hand, we introduce the details of the data collection in Section 3.1. Tools used, such as DPR and Wikipedia paragraphs, are all publicly available at https://github.com/facebookresearch/DPR or https://github.com/castorini/pyserini. For more details, we also included a sample dataset (sample_dataset.json) in the supplementary material. Thank you.\n\n**Instruction fine-tuning model (Questions)**: We deliberately design Thrust to only rely on the dev and test queries so that the external knowledge can be arbitrary. For example, in the context of the instruction fine-tuning model, we can use Thrust to rank the queries and only conduct Chain-of-thought on the hard examples. On the other hand, we can also use Thrust to suggest if further details of the question are needed to be provided. Thanks for your suggestion on the scope of instruction fine-tuning models, we will suggest this potential usage in discussion in the final version.\n\n**Presentation**: We will improve our presentation on methods and experiments in our final version. Some points are mentioned in our rebuttal to Reviewer eb2S.\n"}, {"review_id": "62301rs4HG", "reviewer": "Reviewer_RnNH", "summary": "This work proposes methods IAPEK and Thrust to make instance-level decisions about when to utilize external knowledge for question answering. IAPEK is instance-level adaptive propulsion of external knowledge, essentially the use of external knowledge only when it is necessary beyond the base model. Thrust is a heuristic scoring method to decide which questions to use external knowledge on, based on various notions of distance from existing clusters of points. Intuitions for Thrust score are outlined in S2.2. The work demonstrates that Thrust outperforms two baseline methods (random and BM25) at overall accuracy when used to select which instances require external knowledge under a fixed budget. They also demonstrate that using IAPEK with Thrust performs nearly as well as using external knowledge in every case, with a lower computational budget.", "questions": "Question: have you considered selecting examples based on notions of uncertainty? e.g. just looking at the entropy of the model on the output space, rather than interacting directly with vectors or model internals. It would seem like a less certain model may make more use of external knowledge.\nSee weaknesses for some questions.", "limitations": "It would be useful to include more information about limitations. What is the maximally useful case you see for your work, and what will still be left to do in solving this problem?", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The work makes useful statements about the complex nature of retrieval based QA -- both in terms of efficiency and the counterintuitive finding that external knowledge does not always help\n- Thrust seems to outperform baselines at selecting instances that require external knowledge\n- In general, IAPEK seems to improve efficiency without too much loss of performance compared to always using external knowledge", "weaknesses": "- The Thrust score does not have theoretical grounding, or sufficient ablations to justify design decisions. These are not both necessary, but more justification (either theoretical or experimental) for specific design decisions would be very useful. A description of intuitions is given at line 105, but no experimental examples are given to demonstrate that these cases are relevant. One useful aspect would be comparing to simpler scores. Only random and BM25 are used as baselines, but what about simpler notions of distance, that either do not involve complex clustering, or do not involve the \"mean vector\" idea for Thrust? Something simpler like distance from existing points may perform worse, but there it is not clear given the current results included in the paper. Perhaps some of the complexity of Thrust is not required. \n- More broadly, it would be useful to show, even for a subset of datasets, the full set of possibilities between: {no EK, IAPEK default, BM25, Thrust, distance from overall centroid, full EK}\n- It is not completely clear what aspects factor into the performance of IAPEK/Thrust. Particularly, the authors mention that external knowledge can sometimes hurt performance, in which case perhaps Thrust is helping less with efficiency, and more with preventing such examples from seeing external knowledge that would reduce performance. One useful aspect would be a more comprehensive version of Table 3 with numbers, to answer the question \"how often does thrust outperform full EK?\"\n- Continuing from the point above, it is not clear from the paper whether the justification of Thrust is improving efficiency (i.e. full EK always helps but is expensive) or performance (i.e. there are some examples that do better without EK, and Thrust helps identify these)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments and suggestions.  For your reference, our design choice ablation and limitation discussion are provided in Appendix. We would like to address your concerns as follows. \n\n**Design Choice (W1)**: We kindly refer you to Appendix (A.4 and Table 2) included in the supplementary materials. As discussed in Footnote 4 of page 5, we presented our experiments on design choice ablation and known limitations in the appendix. We compared the alternative choices such as no cluster size, no direction, using inertia, etc, to justify our design of Thrust. Thanks for your suggestions and we will include this section in the main paper in the final version.\n\n**Results Comparison (W2)**: Thanks for your suggestions on the presentation. Our experiments contain the full set of datasets for {no EK, IAPEK default, IAPEK BM25, Thrust, Full EK}. {no EK, Full EK} are compared in Figure 4. {IAPEK default, BM25, Thrust} are compared in Table 1 and Table 2. These experiments are all conducted on the same set of datasets. We will include the table containing all the entries in Appendix in our final version.\n\n**Details about Performance vs. Efficiency (W3)**: Thanks for your suggestions for the table design. In Table 3, we present that in ⅔ cases, Thrust outperforms 99% of full EK performance when saving at least 10% expense. \n\nMore details are as follows: in ⅓ -  ½  cases, Thrust rejects external noise and outperforms full EK. For datasets such as BoolQ and CIKQA, where the external knowledge can be noisy, Thrust identifies “some examples that do better without EK” and always outperforms full EK. For datasets such as e-SNLI, where humans annotated the external knowledge,  full EK always helps but is expensive. The specific datasets are as follows:\n\nFor UnifiedQA-base\n\nThrust > full EK: BoolQ, CIKQA, StrategyQA, ARC-C, TriviaQA, AGNews\n\nThrust > 99% full EK: ARC-E. wq. HotpotQA\n\nThrust < full EK: e-SNLI, TREC, NQ\n\nFor UnifiedQA-3b\n\nThrust > full EK: BoolQ. CIKQA, ARC-C, HotpotQA \n\nThrust > 99% full EK: StrategyQA, AGNews, ARC-E, TriviaQA, NQ\n\nThrust < full EK: e-SNLI, WQ, TREC\n\n\nWe will add a new table and more discussion comparing Thrust and full EK in the final addition.\n\nOne step further, we also identify this comparison as a Performance-Efficiency trade-off with controlling the expected EK rejection rate. Given extra space provided upon acceptance, we will present and analyze the trade-off with an AUC-ROC-like curve, where a convex curve is desired.\n\n\n**Uncertainty (Question)**: The entropy of the model output could be applied to IAPEK for classification tasks. However, some answers are of various lengths for open-domain question answering. An example of NQ is: What does a drink from Narcissus's Spring cause the drinker to do, the expected answer is “fall in love with themselves” rather than from a fixed label set, which can make the entropy test unstable. To provide a unified framework for classification and open-domain QA tasks, we designed Thrust for IAPEK. On the other hand, Thrust can also be regarded as a way to measure uncertainty. We will point this out as potential future work in the final version.\n\n**Limitations**: We kindly refer you to Appendix A.1 contained in the original supplementary materials, where we discuss the potential limitations of our work (e.g., can not be used for black-box LLMs). We will include this section in the main paper in the final version. Thanks again for the suggestions.\n"}, {"review_id": "633wFvhl09", "reviewer": "Reviewer_LAud", "summary": "LLM’s parametric memory may be inaccurate or outdated and thus retrieving additional information to LLMs can help but it’s costly and may be noisy.\nSo this paper proposes Thrust to measure the instance-level parametric memory and can help determine whether to use a retrieval module for enhancing LLMs, which is more cost-efficiency.\nBy dynamically using external knowledge, the proposed IAPEK can achieve consistent gains over a large amount of tasks and LLMs.\n", "questions": "No questions. This paper is well done.", "limitations": "1. One assumption of Thrust is that if PTLM has mastered sufficient knowledge for a task, the hidden states could be used for clustering. But whether using external knowledge is not just about if LLMs can use/represent well. For example, some time-sensitive questions like `who is the CEO of Twitter` can be confidently but wrongly answered by LLMs. However, in these cases, LLMs should use external knowledge.\n\n2. Missing related work:\nXie et al., *Adaptive Chameleon or Stubborn Sloth*: Unraveling the Behavior of Large Language Models in Knowledge Conflicts.\nThis paper has some conclusions/observations aligned with Thrust: external evidence may mislead the LLMs to generate wrong answers.\nGiven this paper was available after NeurIPS submission deadline, missing it in reference will not negatively affect my evaluation of Thrust. But you may consider adding it in the next version.\n\nMinor\n3. In appendix, Table 3 should be annotated with red and green like you did to the table in the main paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The task is important and motivation is well elaborated.\n2. Thrust is quite data-efficient to construct, which is valuable in practice.\n3. Comprehensive evaluations are conducted and valid points are made.\n", "weaknesses": "1. LLMs used in this paper are not that strong and may not well utilize external knowledge, as pointed by you. I would recommend adding models like FLAN-T5 or other instruction-tuned models that can follow instructions to use external information for QA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments and suggestions. We would like to address your concerns as follows. \n\n**LLM ability (W1)**: Thanks for pointing out the discussion regarding the choice of models. We will try to add Flan-T5 in the camera-ready version to compare if instruction fine-tuning also helps improve external knowledge utilizability. \n\n Our reason to use the original pre-trained T5 is that UnifiedQA is fine-tuned on these models so that we can compare (in Figure 4) how strongly the model can utilize external knowledge after fine-tuning on this target and show the fine-tuning step in UnifiedQA does help. \n\nOn the other hand, UnifiedQA is still comparatively strong on our set of tasks (given the size). For example, LLAMA2 7B achieves 77.4, 75.2, and 45.9 on BoolQ, ARC-e, and ARC-c, respectively. UnifiedQA 3B achieves 87.8, 73.7, and 64.5 in similar settings (Table 20 in LLAMA 2 paper (https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/)). \n\n**Time Sensitivity (L1)**: Thanks for mentioning this point. We regard time sensitivity as a part that can be done in the IAPEK framework, but not by Thrust, as the framework is motivated by both noise and staticity. We will bring this up in the future work discussion in the final version: another orthogonal sort of score measuring time sensitivity can be designed to decide if updated knowledge retrieval is necessary, for example, based on (Ning. et al., 2022).\n\n**Missing Reference (L2)**: We appreciate your introduction to this great paper. We will add this paper to our discussion regarding the misleading behavior of LLMs in the final version.\n\n**Presentation (L minor 3)**: We appreciate your check on our appendix. We will refine Table 3 to make it consistent with the main paper.\n\n\n**References**\n\nQiang Ning, Ben Zhou, Hao WU, Haoruo Peng, Chuchu Fan, Matt Gardner. 2022. A Meta-framework for Spatiotemporal Quantity Extraction from Text. In Proceedings of ACL.\n"}]

Author rebuttal:

Meta-review: This paper tackles an important problem of estimating when an LLM lacks the information needed to answer a given query, triggering the retrieval of new information.  Using this score improves performance compared to naive retrieval by a significant amount in experiments.  Reviewers had some mixed opinions on the submission, with some arguing for acceptance and others on the borderline.  However, the concerns in the most negative review (regarding reproducibility and justification of design decisions) were addressed well in the author response.  One remaining concern is that the paper does not evaluate against many other methods of estimating model confidence, which seems like a potentially significant gap.  Evaluating some of the confidence estimation methods listed in Section 5 of the paper, in addition to the random and BM25 methods that are currently explored, could make the paper's experiments more definitive.
Decision: Accept (poster)