SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xinran Zhao $^{1,2}$  Hongming Zhang $^{1}$  Xiaoman Pan $^{1}$  Wenlin Yao $^{1}$  Dong Yu $^{1}$  Jianshu Chen $^{1}$
$^{1}$ Tencent AI Lab, Bellevue,  $^{2}$ Language Technologies Institute, Carnegie Mellon University
Although large-scale pre-trained language models (PTLMs) are shown to encode rich knowledge in their model parameters, the inherent knowledge in PTLMs can be opaque or static, making external knowledge necessary. However, the existing information retrieval techniques could be costly and may even introduce noisy and sometimes misleading knowledge. To address these challenges, we propose the instance-level adaptive propulsion of external knowledge (IAPEK), where we only conduct the retrieval when necessary. To achieve this goal, we propose measuring whether a PTLM contains enough knowledge to solve an instance with a novel metric, Thrust, which leverages the representation distribution of a small number of seen instances. Extensive experiments demonstrate that Thrust is a good measurement of PTLM models' instance-level knowledgeability. Moreover, we can achieve higher cost-efficiency with Thrust score as the retrieval indicator than the naive usage of external knowledge on  $88\%$  of the evaluated tasks with  $26\%$  average performance improvement. Such findings shed light on the real-world practice of knowledge-enhanced LMs with a limited knowledge-seeking budget due to computation latency or costs
Knowledge is crucial for understanding human language and solving various NLP tasks [59]. In recent years, the pre-trained language models (PTLM) have demonstrated great success on various NLP tasks [10, 43, 32, 44, 5] by storing rich encyclopedic [42] and commonsense [25] knowledge in their model parameters. However, such implicit knowledge could be opaque, static, or inefficient [23]. These issues motivate the common practice of seeking external knowledge [30, 57, 53, 17] with information retrieval methods and augmenting the inference models (e.g., PTLMs) [20, 12, 24] with the retrieved knowledge.
However, this approach has two limitations: (i) extracting external knowledge with existing information retrieval tools can be costly for a large-scale knowledge resource. (ii) external knowledge can be unnecessary or even misleading. For instance, one of the best retrieving models ColBERT v2 [46] achieved 68.9 Success@5 on Natural Question [27], which suggests that gold documents do not appear in the top five retrieved documents for  $31.1\%$  of the queries. Considering the limited input sequence length, the most useful documents may not be included for generating a prediction, while others may add noise to the model. On the other hand, PTLMs, which grow from millions (e.g., BERT [10]) to billions of parameters (e.g., OPT [61]), may solve the queries directly without
*Work done during interning at Tencent AI Lab, Bellevue. Corresponding contact email addresses: xinranz3@andrew.cmu.edu, {hongmingzhang,xiaomanpan,wenlinyao,dyu,jiangshuchen}@global.tencent.com. Our code is available at https://github.com/colinzhaoust/thrust_neurips2023.
Figure 1: The predictions by OPT-175B without/with external knowledge retrieved via DPR [20] from Wikipedia paragraphs. Although the top retrieved paragraphs are relevant since the internal knowledge is already sufficient, the external knowledge can either be misleading (potentially due to the effect of misprime [21]) or less useful.
Figure 2: The pipeline of retrieval-augmented models with IAPEK. Unlike previous work (e.g., RAG [30]) where models directly seek for help from the retriever module, IAPEK module provides a confidence score  $S(q)$  (e.g., Thrust) on how well the PTLM can answer the question with internal knowledge and decides if the external retrieval is necessary.
external knowledge, making it unnecessary to seek external knowledge that signifies the noise issue. The instance shown in Figure 1 demonstrates the noise and inefficiency issues caused by external knowledge retrieval. OPT-175B can directly give the correct answer without external knowledge. However, with top external knowledge retrieved by DPR [20] from Wikipedia paragraphs, the external knowledge can be useless or even lead to wrong predictions.
The efficiency and noise issues motivate us to propose the Instance-level Adaptive Propulsion of External Knowledge (IAPEK), which adaptively retrieves external knowledge when it is necessary. We propose to measure whether a PTLM can solve a question instance with internal knowledge with a confidence score and reject using external knowledge when the confidence is high, instead of seeking retrieval directly for all the cases. The overall pipeline is shown in Figure 2. Specifically, we propose to solve this problem from the representation learning perspective based on two assumptions: (i) if a PTLM has mastered sufficient knowledge about a task, its hidden states should be able to cluster the samples from the task well enough; (ii) if the representation of a new instance deviates from these clusters in hidden states, this instance should be beyond the knowledge scope of the target PTLM. On top of these assumptions, we design a simple and lightweight metric Thrust to measure the distance between an instance's representation and the clusters of several observed examples in the same task.
To comprehensively understand the effectiveness of Thrust, we conduct experiments on diverse NLP tasks. Experiments show that the average Thrust score of open-domain questions is significantly lower than other tasks, which aligns well with our intuition that open-domain QA tasks typically need external knowledge. These results indicate that Thrust is a good measurement of the models' knowledgeability. Extensive experiments also show that Thrust can improve the cost-efficiency of seeking and using external knowledge on  $88\%$  cases with  $26\%$  average performance improvement through identifying the instances that mostly require knowledge. We can also observe that, with Thrust, we can achieve higher performance than injecting external knowledge for all instances, where models are benefited from both the performance and efficiency aspects. Such findings shed light on the real-world practice of knowledge-enhanced LMs with a limited budget for knowledge seeking due to computation latency or costs.
2.1 IAPEK: Problem Formulation
We begin with a formal problem definition of Instance-level Adaptive Propulsion of External Knowledge (IAPEK). For each instance of natural language query  $q$  (e.g., a question in question-answering tasks), we first determine whether the PTLM has sufficient knowledge to solve the current problem. We achieve this by using a real-valued score function  $s(q)$  that assigns a higher score to  $q$  if the current PTLM has enough knowledge to solve the problem and vice versa. And we will retrieve
Case 1: far away No knowledge (hard)
Case 2: close to one  
Knowledge exists (easy)
Figure 3: The intuition behind the proposed Thrust, which are plotted in the hidden representation space of PTLM. We represent an incoming query instance by triangles and represent the instances used for constructing Thrust scores by ticks and crosses. In the controversial and no knowledge cases, the internal knowledge is insufficient to answer the query successfully, and external knowledge is needed to facilitate PTLM. In contrast, if the model finds the query close to one of the clusters, internal knowledge should be sufficient to solve the problem so that external knowledge is unnecessary.
Case 3:interweaving Controversial Knowledge (hard)
Test query
Example (label=+1)
Example (label=-1)
Example Cluster
Distance Vector
external knowledge to facilitate PTLM once the score  $s(q)$  falls below a threshold  $\lambda$ . By doing this, the PTLM can selectively retrieve external knowledge in an instance-adaptive manner and avoid unnecessary knowledge augmentation costs. To achieve such a mission, a critical step is to design an instance-level score function  $s(q)$  that can effectively measure whether the PTLM has enough knowledge to solve the current particular input instance. In the remainder of the section, we will show how to construct such a function efficiently.
2.2 Thrust: Measuring the Knowledgeability of PTLMs
We now proceed to construct the scoring function  $s(q)$  that measures the knowledgeability of a PTLM for solving an instance from a particular task  $\mathcal{T}$ . A seemingly straightforward approach is to adopt a supervised learning strategy to train such functional mapping from human-labeled data.
However, it is practically infeasible to manually annotate whether a given PTLM has sufficient knowledge at an instance level because the internal knowledge of PTLM is implicitly stored in its model parameters, which are hard to probe precisely for the IAPEK purpose. For this reason, we take an alternative approach by looking at the problem from the representation learning perspective. Specifically, our method is based on the following two assumptions. First, if a PTLM has mastered sufficient knowledge about a task, then its hidden states should be able to cluster the samples from the task well enough. For a classification task, samples from different classes should also be well separated in their (higher-level) hidden representations. Second, we further hypothesize that when a particular sample from the task deviates from these clusters in hidden states, this instance should be beyond the knowledge scope of the PTLM. In Figure 3, we illustrate the above intuitions for different cases. In other words, akin to the observations in [16, 28], we view the built-in knowledge of PTLM as a representational power that enables the deep models to learn more separable features. Based on such observations, we develop a knowledgeability scoring function for a PTLM by measuring how well it can separate samples in its hidden states.
We now proceed to design  $s(q)$  that scores each query  $q$  from a given downstream task  $\mathcal{T}$ . To begin with, we first collect a small set of samples from task  $\mathcal{T}$  and compute their hidden state representations using the designated PTLM. (Empirically, we find that about 200 samples are sufficient in our experiments.) We denote such a representational mapping by a function  $f(\cdot)$ . For generation tasks, we treat all instances as having a single dummy label. Next, we group these embedding vectors according to their class labels as  $\mathcal{G}_l = \{(f(x_i), y_i) \mid y_i = l\}$ , where  $x_i$  and  $y_i$  denote the  $i$ -th instance and its corresponding class label, respectively, and  $l$  is the class index. We further cluster the samples in each  $\mathcal{G}_l$  into  $K$  clusters by applying the k-means algorithm to the vectors  $f(x_i)$ . For convenience, we introduce the notation  $\mathcal{C}_{kl}$  to represent the set of samples in the  $k$ -th cluster of class  $l$ , and let  $m_{kl}$  be the centroid vector corresponding to  $\mathcal{C}_{kl}$ .
<sup>1</sup>We use the last layer of hidden states as the embedding function. For T5-based models, we use the last layers of the decoders.
With the above notation, we define the Thrust score for a given query instance  $q$  as:
$$
s _ {\text {t h r u s t}} (q) \triangleq \left\| \frac {1}{N \cdot K} \sum_ {l = 1} ^ {N} \sum_ {k = 1} ^ {K} \frac {\left| \mathcal {C} _ {k l} \right|}{\left\| d _ {k l} (q) \right\| ^ {2}} \cdot \frac {d _ {k l} (q)}{\left\| d _ {k l} (q) \right\|} \right\|, \tag {1}
$$
where  $d_{kl}(q) \triangleq m_{kl} - f(q)$  is a vector pointing from  $f(q)$  towards the centroid  $m_{kl}$ ,  $N$  is the number of classes,  $K$  is the number of clusters per class,  $|\mathcal{C}_{kl}|$  denotes the cardinality of the set  $\mathcal{C}_{kl}$ , and  $\| \cdot \|$  denotes  $\ell_2$ -norm of a vector.
The design principles Note that the expression inside the  $\ell_2$ -norm of (1) can be viewed as a weighted average of the normalized (unit) vectors  $\{d_{kl}(q) / \| d_{kl}(q)\|\}$  that point from the query vector  $f(q)$  towards the centroid vectors  $\{m_{kl}\}$ . The weighting is proportional to the cluster size and inversely proportional to the squared distance between the query and the centroid. Such a design choice is based on the following principles derived from the earlier representational assumption regarding knowledge. First, when samples from a task are well clustered and if  $q$  is close to one of the clusters while being farther away from others, it means that the query instance  $q$  can be well solved by the internal knowledge of PTLM and the thrust score should be high. Let  $m_{kl}$  be the cluster centroid that  $q$  is close to, then we observe that the corresponding  $\| d_{kl}\|^2$  term in the denominator will make the corresponding term dominate and large in (1). Second, if  $q$  is farther away from all the cluster centroids, i.e., the query is beyond the knowledge scope of the PTLM, then the quadratic term  $\| d_{kl}\|^2$  would quickly suppress all the terms in (1), making the thrust score vanish. Third, when the PTLM cannot sufficiently cluster the task samples in its hidden states, it means that the PTLM does not have sufficient knowledge to solve the entire task. In this case, the unit vector  $d_{kl}(q) / \| d_{kl}(q)\|$  would randomly point towards different directions so that the averaged vector inside the  $\ell_2$ -norm of (1) diminishes. Finally, the main reason that we first aggregate the samples within each class into  $K$  clusters before computing the thrust score is that they may still be spread over multiple clusters even if they belong to the same class. The term  $|\mathcal{C}_{kl}|$  in (1) is used to upweight the vectors  $d_{kl}(q) / \| d_{kl}(q)\|$  that point to bigger clusters. Nevertheless, we find that  $K$  can be relatively small. In Section 4.1, we will conduct an experimental analysis to show that Thrust score designed in the above manner is a good measurement of a PTLM model's knowledgeability. In addition, we also carry out extensive ablation studies to examine these design choices (see Appendix).
Practical considerations As we will show in Section 3, we only need about 200 samples from a task to form the clusters, and then we just need to store their corresponding centroids  $m_{kl}$  (typically less than 50 vectors of dimension 300) for deployment. According to (1), computing the Thrust score for a query  $q$  only needs to calculate the distance between  $f(q)$  and these centroids, which takes about 0.001 second per query on average in our experiments (see Appendix). Therefore, Thrust is fairly lightweight and easy to be deployed in practical applications. The incurred extra computation complexity during the inference stage is  $O(NK)$ . Since  $K$  and  $N$  are generally small, this overhead is negligible compared to retrieving knowledge from a large external memory for each instance.
In our experiments, PTLMs are examined under two settings. (i) Zero-shot. We consider T5 [44], GPT-J [54], OPT [61] and present queries with or without knowledge directly to the PTLMs. (ii) Transfer-learning. We use UnifiedQA [22] models that fine-tune T5 on multiple QA datasets.
After studying which model performs the best in utilizing external knowledge (the pre-condition of introducing retrievers), we then evaluate the cost-effectiveness of the proposed IAPEK with Thrust with the best performing models. We simulate the real-world usage scenario where we have limited bandwidth or budget to retrieve external knowledge. Specifically, we test on three scenarios based on the richness of available resources: scarce (25%), medium (50%), and abundant (75%). For example, scarce (25%) means that we set the threshold  $\lambda$  (defined in Section 2.1) to 25 percentile of the scores of the 200 examples used to set up the clusters. As introduced in Section 2, we use Thrust to score and rank the instances by their need for external knowledge and select the instances that have a high demand for external knowledge (i.e., with low thrust scores). We compare the
<sup>2</sup>We choose  $K$  to be  $\max (\mathrm{ceil}(\sqrt[4]{|D_{\mathcal{T}}^{\mathrm{sample}}|}), 3)$ , where  $D^{\mathrm{sample}}$  denotes the sample set of task  $\mathcal{T}$ .
performance between Thrust with two baselines, i.e., IAPEK (Default) that randomly samples  $\mathrm{X\%}$  ( $\mathrm{X = 25}$ , 50 or 75) examples to apply knowledge, and IAPEK (BM25) that ranks all instances by BM25 [52] and select top  $\mathrm{X\%}$  ( $\mathrm{X = 25}$ , 50 or 75) difficult examples.
We follow previous work to report accuracy for MC classification tasks. For open-domain QA, we report the QA-F1 score that measures the max uni-gram overlap between the model prediction and all gold answer candidates following previous work [22]. For the without knowledge setting, we directly pass the prompt-decorated question query to the model and select the choice with the highest probability as the answer. For the with knowledge setting, we append the knowledge piece after the prompt-decorated query and put "Answer:" at the end to pass to the models.
We consider several knowledge-intensive tasks in our evaluation, i.e., Multiple-choice Classification (MC), which consists of seven datasets, and Open-domain QA, which consists of five tasks. Each data instance consists of a query (a piece of text containing a question or the sentences to be classified) and an answer (either the label words or the answers to the questions in the query).
Additionally, each instance may also contain a piece of potentially helpful knowledge for the query, which is either inherently relevant due to the task design, annotated by humans, or retrieved from Wikipedia paragraphs with DPR. Details of the datasets and corresponding external knowledge are as follows.
Multiple-choice classification. For MC classification, each query  $q$  includes a sentence or a question and requires models to select the correct answer from a set of candidates. Specifically, (i) AG-News [62] asks the model to classify a piece of news into political, sports, business, or technology. We regard the titles of the news as the queries since they may already contain sufficient information and use the content of the news as the gold external knowledge. (ii) e-SNLI [6] is a natural language inference (NLI) task exploiting the role of explanations in textual entailment. Human-providing explanations are considered a strong source of external knowledge; (iii) StrategyQA [13] is a challenging multi-hop reasoning dataset that requires models to answer creative questions through strategical inference from implicit reasoning steps. We regard the original questions as queries and human-written explicit facts as external knowledge; (iv) CIKQA [60] is a commonsense inference task that combines pronoun coreference resolution, commonsense QA [51], COPA [45], and questions mined from ATOMIC knowledge graph [47]. We regard the original questions as queries and the supporting commonsense knowledge extracted from knowledge graphs (KGs) in the original work as the external knowledge; (v) BoolQ [8] contains encyclopedic questions that require models to answer yes or no. Following [22], we use the Wikipedia paragraphs retrieved by DPR as the external knowledge, which can be potentially noisy; (vi) ARC-E & ARC-C [9]: ARC is a challenging multiple-choice QA dataset that requires knowledge understanding and reasoning, which is partitioned to an Easy set (ARC-E) and a Challenge set (ARC-C), where the Challenge set questions are answered incorrectly by the retrieval-based or co-occurrence-based algorithms tested by the original authors. Similarly, we use the Wikipedia paragraphs retrieved by DPR as external knowledge.
Open-domain QA. For open-domain QA, each query  $q$  contains an open question that typically requires solving an encyclopedic or commonsense inference. The generated answers can either be a few phrases or a single sentence. An example question is "What does a drink from Narcissus's Spring cause the drinker to do" and the expected answer generated from the language model is "fall in love with themselves". The involved datasets are HotpotQA [58], Natural Questions (NQ) [27], Web Questions [2], Curated TREC [1], and TriviaQA [19]. We use Wikipedia paragraphs retrieved by DPR as the external knowledge as a common practice [59], except for HotpotQA, where we use the passages that the queries are generated from as a gold knowledge resource.
The statistics of the involved datasets (e.g., query length and sizes of the splits) are reported in Appendix. We collect a benchmark with various datasets of different types, formats, and knowledge sources, where we will then evaluate the effectiveness of IAPEK. Some implementation details of each of the task are described in Appendix.
Figure 4: Performance of various models on MC classification tasks (accuracy) and open-domain QA tasks (QA-F1), denoted by (cls) and (qa), respectively. The x-axis represents the model names, which are shared across sub-figures. Use knowledge: yes or no denotes using full knowledge or not for all queries. UnifiedQA denotes T5 models with different sizes fine-tuned on the UnifiedQA dataset.
3.2 Performance of Using External Knowledge
Figure 4 presents the model performance on both the MC classification and open-domain QA tasks<sup>3</sup>. For the MC classification tasks, we can observe that for the zero-shot setting (T5-X, GPT-J, and OPT), models do not consistently benefit from external knowledge. In addition, the ability to utilize external knowledge is also not improved as the parameter size grows, which indicates that simply using larger models may not be the solution for better using the knowledge. For the transfer-learning setting (UnifiedQA-X), although AGNews, e-SNLI, CIKQA, and StrategyQA are not seen during the training of UnifiedQA models, we can observe that models achieve better performance than vanilla T5 models at different sizes. Under the with knowledge case, the UnifiedQA models achieve significant improvement for utilizing external knowledge compared to the zero-shot models, UnifiedQA-3b achieves the best performance on all the tasks, which indicates that models can learn and transfer the ability to utilize external knowledge with instances containing external knowledge. Moreover, for open-domain QA tasks, we can see models (T5, GPT-J, OPT) get no benefits from external knowledge in 11 out of 25 cases under the zero-shot setting, while UnifiedQA models achieve significant performance gain after adding external knowledge. External knowledge may introduce extra noise if the model does not learn to utilize knowledge, which indicates the importance of instructing PTLMs to learn how to use knowledge through second-stage fine-tuning.
In conclusion, we find that fine-tuning with instances containing external knowledge is an effective way to help models learn to use external knowledge. Since the pre-condition of using IAPEK is that the model can utilize external knowledge well, we conduct experiments with UnifiedQA only when evaluating the performance of Thrust.
${}^{3}$  the detailed numeric values,design choice ablation,and known limitations are presented in Appendix.