SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Performance of IAPEK based on Thrust. As defined in Section 3, performances of Default/Thrust are presented before/after the vertical bar for scarce, medium, and abundant cases. If performance increases with Thrust, the score will be marked in green and otherwise in red. WQ and TREC denote the tasks of Web Questions and Curated TREC, respectively.
Dataset	UnifiedQA-base	UnifiedQA-large	UnifiedQA-3b
scarcce	medium	abundant	scarcce	medium	abundant	scarcce	medium	abundant
AGNews	50.7	55.6	52.8	56.3	55.0	56.8	70.2	69.1	69.4	70.2	68.7	70.6	77.9	78.4	80.1	80.4	82.3	82.3
e-SNLI	46.5	66.6	54.4	68.3	62.3	69.6	50.7	71.1	58.5	72.2	66.4	73.2	69.1	86.3	75.9	87.5	82.8	88.8
CIKQA	56.9	59.6	57.8	59.6	58.7	59.9	60.2	62.1	60.8	62.3	61.5	62.4	62.7	66.9	64.1	66.9	65.5	66.9
StrategyQA	50.7	55.6	52.8	56.3	55.0	56.8	52.9	62.1	57.4	65.3	61.9	65.9	64.1	74.3	70.5	81.4	77.0	82.9
BoolQ	65.5	76.2	70.7	79.9	75.8	80.9	65.9	77.7	72.1	81.3	78.3	84.4	68.1	79.1	74.6	85.7	81.2	87.1
ARC-E	50.7	55.6	52.8	56.3	55.0	56.8	64.5	64.6	65.0	64.7	65.5	65.1	74.4	74.6	75.1	74.9	75.8	75.1
ARC-C	44.9	43.8	45.0	44.5	45.1	44.8	53.8	50.8	52.3	51.2	50.9	51.5	64.5	63.9	64.4	64.9	64.3	65.6
WQ	19.2	26.3	27.5	42.1	35.8	43.8	22.5	38.5	30.5	39.0	38.5	46.0	20.9	19.3	30.0	35.4	39.1	46.4
TREC	13.5	33.6	21.3	36.4	29.1	36.9	30.8	32.7	32.7	36.0	34.6	36.3	19.6	37.8	27.0	40.6	34.4	40.9
HotpotQA	25.2	32.9	30.2	35.5	35.2	37.8	26.7	35.2	32.1	37.5	37.4	40.2	24.9	41.9	32.3	43.9	39.7	45.7
TriviaQA	32.0	52.7	43.2	56.4	54.4	60.0	32.4	59.7	46.4	64.3	60.5	71.8	39.2	68.3	52.8	71.0	66.4	73.4
NQ	20.0	33.0	24.9	33.5	29.7	33.9	12.0	34.8	20.1	35.2	28.2	35.7	12.8	35.9	21.1	36.5	29.4	37.0
3.3 Performance of Thrust
Table 1 shows the results of UnifiedQA after adding knowledge to  $X\%$  ( $X = 25, 50$  or 75) that needs knowledge most according to our Thrust score. We can see that Thrust consistently contributes to the performance from the base to the 3B model. Through clustering the instances, we acquire the whole instance distribution in the eyes of the models. Then with distance to the cluster, Thrust represents how well the model can categorize a new query vector and find its similarity with others on the task. Leveraging such information, Thrust identifies the no knowledge and controversial knowledge cases well and puts the knowledge into the most necessary ones.
Additionally, the gain is higher when the portion of augmented instances is smaller. For instance, for UnifiedQA-3b, the gains from Thrust with the scarce case are  $6.1\%$ ,  $13.56\%$  on MC classification and QA tasks, respectively, while for the abundant case, the gains are  $2.8\%$  and  $6.8\%$ . Such observation shows that Thrust is most effective in identifying the most necessary cases. One potential reason is that Thrust is sensitive to the distance change, so the isolated instances (no knowledge case in Figure 3) can be easily identified. With Thrust the performance of cases using fewer resources can sometimes surpass that with high resource requirement for Default, e.g., for UnifiedQA-base on all classification tasks except ARC-C, Thrust(scarce) performs better than Default(medium). Interestingly, we also observe consistent failure cases on ARC-C. This is because the queries are designed as open questions, and the answers are usually about plans or opinions instead of facts. Thus, it is hard for small models to extract useful information from Wikipedia documents.
4.1 Primary Study: Distribution of Thrust scores
To investigate whether Thrust is a good measurement of models' knowledgeability, we plot in Figure 5 the distribution of Thrust scores with the strongest inference model evaluated (i.e., UnifiedQA-3b) $^4$ . We use Kernel Density Estimation to smooth the distribution. From the distribution, we can see that low Thrust scores (i.e., the query needs external knowledge) frequently appear in most query instances of the open-domain QA tasks such as HotpotQA, TriviaQA and  $NQ$ , which are designed to solve with external knowledge. On the other hand, for  $e$ -SNLI and  $BoolQ$ , external knowledge is not always necessary, which is consistent with the design purpose of these tasks. To conclude, by correctly predicting whether a certain task needs external knowledge (e.g., open-domain QA tasks), Thrust score is shown to be a good measurement of a PTLM model's knowledgeability.
For a clear presentation, we only include representative tasks directly relevant to knowledge and reasoning from each category. The selected tasks are:  $e$ -SNLI, BoolQ, CIKQA, HotpotQA, TriviaQA, and NQ. The distribution of all tasks can be found in Appendix, where the trend is consistent. In visualization, the score can be lower than zero due to smoothing. The real scores will always be greater than zero.
Figure 5: Distribution of Thrust scores for various tasks by using UnifiedQA-3b to create the hidden representations. Kernel Density Estimation is used to smooth the distributions. Low scores imply that the instances are less likely to be solved with internal knowledge and vice versa. Thrust scores predict that most query instances from open-domain QA tasks require external knowledge while others need less. The results are consistent with the original design purposes of these tasks.
Table 2: Performance of IAPEK for UnifiedQA-3b based on Thrust and BM25. With scarce, medium, and abundant, performances of Default/BM25 and Default/Thrust are presented before/after the vertical bar. If performance increases with Thrust, the score will be marked in green, otherwise red.
Dataset	BM25	Thrust
scarce	medium	abundant	scarce	medium	abundant
AGNews	77.9	77.0	80.1	79.2	82.3	81.3	77.9	78.4	80.1	80.4
e-SNLI	69.1	68.4	75.9	75.6	82.8	83.0	69.1	86.3	75.9	87.5
CIKQA	62.7	62.3	64.1	64.4	65.5	66.1	62.7	66.9	64.1	66.9
StrategyQA	64.1	63.3	70.5	68.3	77.0	78.1	64.1	74.3	70.5	81.4
BoolQ	68.1	68.4	74.6	75.9	81.2	82.0	68.1	79.1	74.6	85.7
ARC-E	74.4	74.9	75.1	75.3	75.8	76.3	74.4	74.6	75.1	74.9
ARC-C	64.5	65.2	64.4	66.2	64.3	66.6	64.5	63.9	64.4	64.9
WQ	20.9	19.0	30.0	28.2	39.1	37.3	20.9	19.3	30.0	35.4
TREC	19.6	20.4	27.0	28.1	34.4	36.3	19.6	37.8	27.0	40.6
HotpotQA	24.9	25.2	32.3	32.8	39.7	40.6	24.9	41.9	32.3	43.9
TriviaQA	39.2	34.2	52.8	50.0	66.4	65.4	39.2	68.3	52.8	71.0
NQ	12.8	12.9	21.1	21.1	29.4	29.6	12.8	35.9	21.1	36.5
4.2 IAPEK-Thrust versus IAPEK-BM25
We use BM25 [52], a common approach to evaluating the difficulty of queries, as an alternative to Thrust to perform IAPEK. Specifically, we regard each test input as the query and all training data input as the corpus to extract the score. We use the average relevance score across the corpus to rank each test input. From Table 2, We can observe IAPEK performs well with BM25 as the difficulty score on QA tasks. Except WQ and NQ, we observe better performance (marked in green) than the default setting. However, Thrust shows larger (e.g., for QA tasks) and more robust improvement (e.g., for classification tasks) than the BM25 baseline.
4.3 Layer ablation
Since we cast instances into the representation space, a crucial factor for Thrust is the layer of the PTLMs to use. To investigate the effect of using different layers, we conduct experiments on UnifiedQA-3b with the same setting as in Section 3. Figure 6 presents the performance of adding  $25\%$ ,  $50\%$ , and  $75\%$  knowledge-augmented instances with Thrust with hidden states of different layers. We can observe that, for most tasks, there is no significant difference across layers, which shows the robustness of Thrust and potential to accelerate the computation by using early layers. However, for StrategyQA and Web Questions, the middle-layer representation may worsen the overall performance. One possible reason is that early layers in the model contain rich semantic information, and later layers contain task-specific information [33], so both can act as good representations of the instances. However, in the middle layers, rich semantic features are abandoned during extracting task-specific features.
(b)
Figure 6: Layer-wise ablation across tasks and portions of instances augmented with knowledge for (a) MC classification tasks and (b) open-domain QA tasks. The x-axis denotes the layer index of Unified-3b decoder that is used to obtain the hidden representations (i.e.,  $f(\cdot)$  in (1)). For most tasks, the results are not sensitive to the specific layer index. For some tasks (e.g., StrategyQA), choosing middle layers for representation slightly degrades the performance.
Table 3: Comparison between IAPEK-Thrust and the costly full knowledge usage based on UnifiedQA models of different sizes.  $99\%$  Full denotes  $99\%$  performance of models with full knowledge. The knowledge type is noted in brackets, where  $g$  denotes gold knowledge,  $h$  denotes human annotations, and  $r$  denotes the knowledge retrieved from Wikipedia or knowledge graphs. WQ and TREC stand for Web Questions and Curated TREC, respectively. If Thrust achieves better performance than using full knowledge, we mark the entry with  $*$ .
Size	Thrust>99% Full	Thrust<99% Full
base	BoolQ(r)*ARC-E(r)	CIKQA(r)*ARC-C(r)*	AGNews(g)*WQ(r)	StrategyQA(h)*TriviaQA(r)*	e-SNLI(h)	TREC(r)	HotpotQA(g)	NQ(r)
3b	BoolQ(r)*ARC-E(r)	CIKQA(r)*ARC-C(r)*	AGNews(g)TriviaQA(r)	StrategyQA(h)HotpotQA(g)*	e-SNLI(h)	TREC(r)	WQ(r)	NQ(r)
4.4 Comparison with Full Knowledge Usage
We denote simply using external knowledge for all instances as a costly but straightforward way of leveraging external knowledge. Since the big models might be sufficient for certain instances and the external knowledge might introduce extra noise, Thrust can help identify instances requiring (or not) knowledge and achieves higher overall performance on the whole dataset compared to seeking and adding knowledge indiscriminately. Table 3 presents the comparison between adaptive and indiscriminate knowledge propulsion. Thrust here denotes the best performance achieved when less than  $90\%$  of instances use external knowledge. We can observe that, for 2/3 tasks for UnifiedQA-base and UnifiedQA-3b, Thrust achieves better performance or less than  $1\%$  drop than using knowledge for all instances. Such results illustrate that Thrust can help avoids potential noise. On the other hand, we can also observe that for  $e$ -SNLI, TREC, and  $NQ$ , the full knowledge setting performs better than Thrust. It means retrieving and adding high-quality knowledge always benefits the models as long as they have been fine-tuned to know the usage of external knowledge.
PTLM with external knowledge. The paradigm of retrieving knowledge from knowledge bases, augmenting PTLMs, and solving downstream tasks has been widely explored in the community of NLP [29, 4, 15, 39, 48]. The knowledge bases can range from knowledge graphs [57], documents [40], pre-processed vectors [53], other PTLMs [49], search engines [34], to Wikipedia documents as used in this work. To augment the PTLMs, common practice includes creating synthesizing datasets [56], adding knowledge to the prompts [55, 36], create demonstrations [5], and extending feature vectors [24]. The contribution of IAPEK is orthogonal to the above work since it presents a gated
framework to reject external annotations or retrieval that can be extended to the above frameworks. The extension is lightweight since Thrust requires queries only, not labels nor gold answers.
Hardness and Confidence Estimation in PTLMs. Much previous work studies the estimation of dataset hardness and model confidence under the context of PTLMs. For dataset hardness, previous research discovers using the cumulative area under the loss curves (RDA [41]), entropy difference between the trial and null cases ( $\mathcal{V}$ -Usable information [11]), and variance of losses of neighbor sentences (Sensitivity Measurement [14]) These methods achieve great correlation with the model performance when analyzing the test set. However, the test set labels are required and can not be applied when predicting the answers. Another line of work focuses on estimating the expected calibration errors (ECE) for classification [26], QA [18], and math [31] datasets, as a reflection of model certainty on the correct answers. ECE can be considered an orthogonal evaluation metric to measure the model's capability of understanding the tasks, compared to common metrics such as accuracy.
Most previous work can be considered a posteriori analysis of the model capability. In this work, instead, we estimate the pragmatic confidence at the test time to empirically increase the performance with a limited budget or bandwidth to acquire knowledge.
6.1 Extended Usage
We expect the idea of adaptive knowledge injection to be extendable beyond QA and MC questions, such as trail prediction tasks including ECBD [37] or EKP [38]. By design, Thrust is independent of the type of external knowledge, so that the adaptively used external knowledge can be any sort, for example, definitions in EKP. Furthermore, Thrust can also be used as a way to measure the expected performance without extensive fine-tuning, as shown in Table 5 in the appendix.
On the other hand, we present our pioneer study on instruction fine-tuned model Flan-T5 [7] in Section A.4 in the appendix and show that Thrust performs better on CIKQA with Flan-T5 than vanilla T5, where the best performance is achieved with  $40\%$  examples not using external knowledge.
We also believe future work can be done on how Thrust collaborates with different kinds of prompts to include the knowledge or prefix tuning. We discuss the limitations of our current design (i.e., cold start, assuming white-box language models, and extendability on other kinds of retrieval argumentation) in Section A.1 in the appendix.
6.2 Time Sensitivity
We regard time sensitivity as a part that can be done in the IAPEK framework, but not by Thrust, as the framework is motivated by both noise and staticity issues. Another orthogonal kind of score measuring time sensitivity can be designed to decide if updated knowledge retrieval is necessary, for example, based on [35].
In this work, we propose Instance-level Adaptive Propulsion of External Knowledge (IAPEK) as a solution to propel model performance with external knowledge. Accordingly, we propose a simple and effective instance-level metric, Thrust, to perform the adaptive knowledge injection. Extensive experiments show that Thrust is a good indicator of models' knowledgeability and can improve the performance of utilizing external knowledge under various settings. Understanding the delicate usage of potentially noisy knowledge for PTLMs can further enable the models to conduct inference beyond the limitation of internal knowledge.
We are grateful to Sherry Tongshuang Wu, Shikhar Murty, Zhengxuan Wu, Ashwin Paranjape, John Hewitt, Christopher Manning, Eric Mitchell, Sihao Chen, Ben Zhou, Tong Chen, and the anonymous reviewers for their helpful and insightful comments.
[54] Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax, May 2021.
[55] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11903, 2022.
[56] Yuhuai Wu, Markus N Rabe, Wenda Li, Jimmy Ba, Roger B Grosse, and Christian Szegedy. Lime: Learning inductive bias for primitives of mathematical reasoning. In International Conference on Machine Learning, pages 11251-11262. PMLR, 2021.
[57] Yichong Xu, Chenguang Zhu, Shuohang Wang, Siqi Sun, Hao Cheng, Xiaodong Liu, Jianfeng Gao, Pengcheng He, Michael Zeng, and Xuedong Huang. Human parity on commonsenseqa: Augmenting self-attention with external attention. CoRR, abs/2112.03254, 2021.
[58] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
[59] Da Yin, Li Dong, Hao Cheng, Xiaodong Liu, Kai-Wei Chang, Furu Wei, and Jianfeng Gao. A survey of knowledge-intensive nlp with pre-trained language models. ArXiv, abs/2202.08772, 2022.
[60] Hongming Zhang, Yintong Huo, Yanai Elazar, Yangqiu Song, Yoav Goldberg, and Dan Roth. CIKQA: Learning commonsense inference with a unified knowledge-in-the-loop QA paradigm. In ACL 2022 Workshop on Commonsense Representation and Reasoning, 2022.
[61] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
[62] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[63] Xinran Zhao, Shikhar Murty, and Christopher D. Manning. On measuring the intrinsic few-shot hardness of datasets. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2022.
Cold Start. In the ideal case, a module distinguishes if a query requires external can do it in a zero-shot manner. However, as we show in Section 4.1 of our paper, we practically find that the distribution of Thrust scores of various tasks can be very different due to the essence of the task collection and the type of external knowledge needed. Considering such an effect, we need 200 examples to estimate the clusters needed to set up the computation, which is still lightweight in real-world scenarios. In the future, we will explore the usage of meta-learning to allow a few-shot start or even a cold start of IAPEK.
Black-box LLM: At the current stage, our model can work with first-layer or last-layer representations (as discussed in Figure 6 of our submitted paper), which are provided by some black-box models. To adapt to completely black-box LLMs, there are two ways we can think at this stage: (1) similar to Black-Box Tuning [50], we use the prompt embeddings adjusted by the derivative-free optimizer optimized over the black-box model outputs as our representation; (2) we use original or distilled smaller models from the same family to acquire representation (e.g., original GPT-2 or GPT-2 fine-tuned by a set of query and answers from GPT-4). We experimented with using T5-base representation to conduct IAPEK for T5-large models. It showed slightly worse but not completely ruined performance.
Extension to other Retrieval-augmented Models. In this paper, we propose a new module for the pipeline of retrieval-augmented models. We first comprehensively examined if and how external knowledge is useful with language models. Next, we examine the performance of the module IAPEK with the lightweight Thrust score we define as a potential implementation. We compare Thrust with BM25 with the default setting of retrieval augmented language models [17] and show its effectiveness. Since queries, not answers nor retrieved knowledge are required to set up Thrust, it can be applied to various other frameworks of retrieval augmented models [3]. However, it is beyond the scope of the project at the current stage, and the contribution of our module and the frameworks are orthogonal. We will extend Thrust to other retrieval-augmented models in future work.
We conduct our experiments on a machine with 8 Nvidia P40 (24G) GPUs with CUDA 11 installed.
We use the Scikit-learn package  $^{5}$  to measure the clusters with K-means and compute the distance between the query and cluster representations. The involved hyperparameters (including the number of clusters per class) are selected by Grid search on a smaller set of experiments. We initialize all parameters randomly or as the default of the Hugginface transformers package  $^{6}$ . On average, each run of extracting the results for all the tasks under with/without knowledge cases takes around 20 hours. We run all experiments 3 times and report the averaged performance in the main content. For hyperparameters of the inference models, for the QA task, we set the maximum knowledge length as 480 tokens to ensure that query sentences stay in the input.
The generated answer for QA tasks for all the models is typically within 30 tokens. For classification tasks, for binary classification tasks (CIKQA, StrategyQA, BoolQ, and e-SNLI), we follow previous work to use "Yes or No?" as the suffix to the original query to guide the generative models. For AGNews, we use "political news, sports news, business news, and technology news" as the label words. We found that the default label word "word news" will largely degrade the performance of generative models on AGNews. We add "the news is about?;" and provide the candidate categories as the suffix for AGNews. More details of our implementations can be found in the code attached.
The detailed statistics of the involved datasets are shown in Table 4. We sample 200 data points from each dataset to conduct the clustering step of Thrust. Difference datasets have different average query lengths and knowledge lengths due to the essence of the task creation and knowledge collection.
<sup>5</sup>https://scikit-learn.org/
<https://huggingface.co/>
Table 4: Statistics of the selected datasets. Sample # denotes the number of examples used to calculate the clusters for Thrust scores. ARC-E and ARC-C denote the easy and hard ARC datasets. Q/K/A Len denotes the average number of words for the Queries/Knowledge/Answers, respectively.
Dataset	Source	Sample #	Test #	Q Len	K Len	A Len
AGNews	gold	200	7,600	8.1	35.9	1.0
e-SNLI	human	200	9,824	24.9	14.3	1.0
StrategyQA	human	200	229	10.8	33.5	1.0
CIKQA	KG	200	604	18.2	28.0	1.0
BoolQ	retriever	200	3,270	9.8	113.8	1.0
ARC-E	retriever	200	570	23.1	238.2	4.2
ARC-C	retriever	200	299	26.2	240.5	5.5
HotpotQA	gold	200	7,405	19.0	56.3	2.5
NQ	retriever	200	6,468	10.1	588.9	2.3
Web Questions	retriever	200	278	7.8	117.3	4.3
Curated TREC	retriever	200	116	8.4	116.5	7.7
TriviaQA	retriever	200	6,760	15.0	117.6	27.5
Figure 7: Performance of different models on CIKQA with different thresholds of Thrust. The X-axis denotes the portion of test examples that are selected to not use external knowledge. All model names denote the large versions of the model parameters.
Answer length 1 denotes tasks with yes and no answers. Otherwise, the answers with more than one token are either choices (for ARC-E and ARC-C) or free-form text sentences (for open-domain QA tasks). Examples of the dataset can be found in the attached data.
A.4 Experiment with Flan-T5
Figure 7 presents the performance of Thrust on CIKQA with different models. From the figure, we can observe that Thrustperforms better with instruction fine-tuned Flan-T5 compared to the original T5 and UnifeedQA. With Flan-T5 Thrust achieves better performance with  $40\%$  examples rejecting external knowledge usage compared to external knowledge used either on no or all examples. Such observations show the potential of using Thrust on current instruction-finetuned models.
A.5 Ablation on the design choices of Thrust
Following [63], we use a few-shot multitask binary NLI dataset to test the influence of each factor of Thrust (i.e., FS-NLI), through measuring how well the metric and its variants can measure the with the hardness of a diverse set of datasets. From Table 5, we can observe that all the design choices are crucial to the success of using Thrust to detect how hard a query is for a given task and model.