SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Satyapriya Krishna $^{1}$ , Jiaqi Ma $^{2}$ , Dylan Slack $^{3}$ , Asma Ghandeharioun $^{4}$ , Sameer Singh $^{3}$ , and Himabindu Lakkaraju $^{1}$
$^{1}$ Harvard University  
 $^{2}$ University of Illinois Urbana-Champaign  
 $^{3}$ University of California, Irvine  
 $^{4}$ Google Inc  
skrishna@g.harvard.edu
Large Language Models (LLMs) have demonstrated remarkable capabilities in performing complex tasks. Moreover, recent research has shown that incorporating human-annotated rationales (e.g., Chain-of-Thought prompting) during in-context learning can significantly enhance the performance of these models, particularly on tasks that require reasoning capabilities. However, incorporating such rationales poses challenges in terms of scalability as this requires a high degree of human involvement. In this work, we present a novel framework, Amplifying Model Performance by Leveraging In-Context Learning with Post Hoc Explanations (AMPLIFY), which addresses the aforementioned challenges by automating the process of rationale generation. To this end, we leverage post hoc explanation methods which output attribution scores (explanations) capturing the influence of each of the input features on model predictions. More specifically, we construct automated natural language rationales that embed insights from post hoc explanations to provide corrective signals to LLMs. Extensive experimentation with real-world datasets demonstrates that our framework, AMPLIFY, leads to prediction accuracy improvements of about  $10 - 25\%$  over a wide range of tasks, including those where prior approaches which rely on human-annotated rationales such as Chain-of-Thought prompting fall short. Our work makes one of the first attempts at highlighting the potential of post hoc explanations as valuable tools for enhancing the effectiveness of LLMs. Furthermore, we conduct additional empirical analyses and ablation studies to demonstrate the impact of each of the components of AMPLIFY, which, in turn, lead to critical insights for refining in-context learning.
In recent years, Large Language Models (LLMs) [3] have ushered in a new era for machine learning research. These models are exhibiting emergent capabilities that enable them to excel at complex tasks that involve sophisticated capabilities such as reasoning and language understanding [34, 5]. Moreover, these models do not only exhibit remarkable performance on tasks they were trained for but also quickly adapt to other novel and complex tasks. This is made possible through a mechanism known as in-context learning, which allows these models to learn from a limited number of input and label pairs, commonly referred to as few-shot prompts[8], provided during test time. Prior research has also demonstrated that the performance of these models on sophisticated reasoning tasks can be significantly improved by presenting them with human-annotated rationales alongside
arXiv:2305.11426v3 [cs.CL] 7 Dec 2023
Figure 1: The AMPLIFY framework consists of four steps aimed at improving the performance of LLMs. (1) We select a proxy model, such as GPT-2 or BERT, which is significantly smaller in size compared to the LLMs and for which it is computationally feasible to generate post hoc explanations. (2) By leveraging the validation set, we identify samples that were misclassified by the LLM. Subsequently, we select the samples that the proxy model exhibits the highest level of confidence in misclassifying. (3) We then use explainability techniques to compute explanations for the selected samples with respect to their ground truth labels. (4) We construct the few-shot prompt for LLM using the samples selected and their corresponding explanations to feed as input to LLM for prediction.
input/label pairs during test time [35, 15]. While incorporating such human-annotated rationales has contributed to enhancing model performance, it is not a scalable approach as it involves a lot of human intervention, thus, limiting its applicability to the ever-expanding range of tasks that are being handled by LLMs [29, 38]. Additionally, prompting these models with human-annotated rationales is not always effective, and may lead to drops in performance on certain tasks requiring sophisticated language understanding as demonstrated by prior works [31]. This could be due to the fact that the seemingly innocuous biases introduced by human-annotated rationales do not necessarily align with the performance goals [33].
To address the aforementioned challenges, we propose a novel framework Amplifying Model Performance by Leveraging In-Context Learning with Post Hoc Explanations (AMPLIFY) which can automatically generate rationales to improve the performance of LLMs on tasks involving sophisticated reasoning and language understanding. To this end, we leverage post hoc explanation methods which output attribution scores that capture the influence of each of the input features on model predictions. More specifically, our framework constructs natural language rationales that embed insights from post hoc explanations to provide corrective signals to LLMs. For example, when a LLM makes an error on an instance, our framework enables the model to correct itself by first identifying the top keywords by computing post hoc explanations (e.g., gradients) of the model with respect to the given instance and its true label, and then prompting the model to pay attention to the top keywords identified, thus, amplifying this signal. To illustrate, let us consider a sentiment classification task where a LLM is assessing the sentiment associated with the sentence "RRR movie has a great story and amazing visuals." If the model is incorrect in its assessment, it can be provided with a corrective input such as "The keywords 'great' and 'amazing' are important cues in predicting the sentiment of this sentence" where the keywords themselves are automatically identified by a post hoc explanation method. While post hoc explanations have generally been considered valuable tools for deepening our understanding of model behavior [6] and for identifying root causes of errors made by ML models [10, 11], our work is the first to explore their utility in improving the performance of LLMs.
While the use of post hoc explanation methods to rectify LLMs' behavior eliminates the need for human intervention, it encounters two significant challenges: First, calculating gradients (and hence post hoc explanations) for LLMs with several billions of parameters is computationally intensive. Second, many LLMs operate as black boxes, depriving end users of access to gradients or other internal model details. To mitigate these issues, we compute post hoc explanations for a proxy model (e.g. GPT-2, BERT, etc.) that is considerably smaller than a large language model with billions of parameters. We then incorporate these explanations into input prompts for larger language models (e.g. GPT-3, GPT-3.5, etc.). This approach not only improves efficiency and feasibility (as we are now calculating gradients for models with 100-200 million parameters, instead of those with 100 billion parameters) but also takes advantage of the accessibility of smaller models that are open sourced. In summary, our AMPLIFY framework follows a four-step approach: (1) Select a proxy model for which post hoc explanation generation is computationally viable; (2) Identify samples that are most likely to provide corrective signals to LLMs; (3) Compute post hoc explanations for the samples identified
in the previous step; (4) Construct a few-shot prompt using the samples chosen in step (2), their true labels, and the post hoc explanations obtained in step 3 as rationales. This prompt is then provided as input to the LLM at test time.
Our findings demonstrate that AMPLIFY leads to performance improvements of about  $10 - 25\%$  across a wide range of tasks, including those where previously considered prompting techniques such as Chain-of-Thought prompting which rely on human-annotated explanations, fall short. This highlights the potential of post hoc explanation methods as valuable tools for enhancing the effectiveness of LLMs. Furthermore, we conduct an extensive empirical analysis to examine the impact of each step of our framework AMPLIFY. This allows for a better understanding of the change in LLM performance with different choices of proxy model (step 1), selection strategy (step 2), post hoc explanation method (step 3), and rationale templates (step 4). Thus, we offer critical insights for refining in-context learning while addressing the limitations posed by methods dependent on human-annotated rationales.
In-context Learning. Over the past decade, numerous language models have been developed to excel at a wide range of complex predictive tasks [34]. This is accomplished by training and fine-tuning language models on datasets associated with various tasks. While these advancements have led to highly effective language models for numerous tasks, they have also increased the models' parameter sizes and the computational costs for additional fine-tuning on new tasks. To address this issue, recent research has demonstrated that modern language models can learn new tasks in-context, which allows the model to perform well on new tasks by simply providing a few task samples in the prompt [16]. This method of learning contrasts with the conventional fine-tuning process, which incurs additional computational costs [16]. This in-context learning ability is even more pronounced in extremely large language models ( $>100$  billion parameters), where it is also referred to as an "emergent ability" [34]. These findings have garnered significant attention, and various approaches have been proposed to enhance in-context learning by incorporating additional signals into the prompt [15]. The current state-of-the-art among such approaches is the Chain-of-Thought (CoT) technique [35] which augments prompts with human-annotated rationales comprising of step-by-step instructions on how to perform a new task. This method has substantially improved language models' capacity to tackle highly challenging tasks that involve sophisticated reasoning. However, this method relies heavily on human annotations and is therefore not very scalable. Further, prior works have demonstrated that this method also leads to poor performance in certain kinds of reasonings tasks[31]. These limitations persist, and there are largely no solutions for them yet. In this work, we propose an approach that demonstrates a lot of promise in alleviating the aforementioned issues.
Post Hoc Explanations. As language models have become more capable and complex, understanding their behavior and the rationale behind their predictions has grown increasingly challenging [9]. To understand the predictions made by these black boxes, various methods have been developed to provide explanations in the form of feature attributions which capture the influence of each input feature on a given model prediction. These methods are known as post hoc explanation methods [19]. Post hoc explanation methods can be broadly classified into two primary categories: (1) perturbation-based methods and (2) gradient-based methods. Perturbation-based methods involve creating an interpretable approximation of the original black-box model using perturbations of input samples. Notable examples of these methods include LIME [23], SHAP [18], Occlusion [37], and others. In addition, gradient-based methods such as SmoothGrad and Integrated Gradients [28, 30] calculate model gradients with respect to input features to determine the sensitivity of the model's output to each feature. However, to the best of our knowledge, the utility of these explanation methods has not been studied in the context of LLMs. Our work makes the first attempt at exploring the utility of these methods in improving the performance of LLMs.
3 Our Framework AMPLIFY
In this section, we describe our framework Amplifying Model Performance by Leveraging In-Context Learning with Post Hoc Explanations (AMPLIFY) in detail. Recall that the main goal of our framework is to eliminate the need for human-annotated rationales, and to instead generate automated rationales which can, in turn, provide corrective signals to improve the performance of LLMs on
sophisticated reasoning and language understanding tasks. To this end, our framework leverages post hoc explanations to construct such rationales. More specifically, our framework adopts the following four-step approach: (1) Select a proxy model for which post hoc explanation generation is computationally viable; (2) Identify samples that are most likely to provide corrective signals to LLMs; (3) Compute post hoc explanations for the samples identified in the previous step; (4) Construct a few-shot prompt using the samples chosen in step (2), their true labels, and the post hoc explanations (rationales) obtained in step 3. This prompt is then provided as input to the LLM at test time. We discuss each of these steps in more detail below.
Step (1): Proxy Model Selection. In this step, we choose a proxy model, typically one that is substantially smaller in size compared to LLMs with billions of parameters, so that generating post hoc explanations is computationally viable. Further, we consider a couple of strategies when selecting a suitable proxy model: (i) Use a pre-trained model such as GPT-2, BERT, etc., which has been shown to perform quite well on several tasks and is thousands of times smaller than LLMs (GPT-3, Bloom, etc.) or (ii) Fine-tune or pre-train a smaller language model from scratch on the target task. The major difference between the two strategies is that the first one requires no additional computational cost as we directly use a pre-trained (potentially open-sourced) model. We test both proxy model selection strategies to discern performance variations between them. Lastly, it is important to note that proxy models of the size we select in this step (e.g., GPT-2, BERT etc.) do not exhibit complex reasoning abilities [34]. Consequently, they do not perform well on reasoning tasks by themselves [29]. However, our analyses (more details in Section 4) demonstrate that such smaller models can be used to improve the reasoning capabilities and task performance of LLMs.
Step (2): Few-shot Sample Selection. The goal of this step is to identify samples i.e., (input, label) pairs that are most likely to provide corrective signals to the LLM. To this end, we first identify instances from the validation set that are misclassified by the LLM. We then rank these instances using a metric we introduce called the Misclassification Confidence Score (MCS). Formally,  $\mathrm{MCS}(\mathbf{x}) = \mathbf{f}(\mathbf{x})_{\mathbf{y}} - \mathbf{f}(\mathbf{x})_{\hat{\mathbf{y}}}$ . Here,  $\mathbf{x} \in \mathbb{R}^{\mathrm{N}}$  represents the input sequence containing  $\mathrm{N}$  tokens,  $f: \mathbb{R}^{\mathrm{N}} \to \mathbb{R}^{|\mathcal{L}|}$  is the fine-tuned language model that produces class probability scores for each label in the label set  $\mathcal{L}$ ,  $f(\mathbf{x})_{\mathbf{y}}$  represents the class probability score for the incorrect label  $(y)$  predicted by the model, and  $f(\mathbf{x})_{\hat{\mathbf{y}}}$  represents the class probability score for the ground truth label  $(\hat{y})$ . The samples that exhibit the highest MCS represent the most egregious misclassifications. By incorporating these samples and their corresponding corrective rationales into the few-shot prompt, the LLM is likely to receive strong supervision to avoid similar misclassifications. In summary, this step results in  $s$  samples of (input ( $\mathbf{x}$ ), label  $(\hat{y}))$  pairs, filtered from the validation set, that are likely to carry the most useful corrective signals to assist LLMs.
Step (3): Rationale Generation. In this step, we compute post hoc explanations for each sample obtained from the previous step. Specifically, for each sample, we use a post hoc explanation method with the (input, label) pair, along with the proxy model, which then calculates the attribution scores for each token in the input sentence. These attribution scores, associated with each token, indicate the contribution each token in the input sentence makes towards the proxy model's prediction of the provided label. We then compute attribution scores for each word by averaging the attributions obtained for each token in that word. Finally, we filter the top- $k$  words with the highest attribution scores. As a result, this step outputs a set of  $k$  words for each sample selected in the previous step. The most commonly used post hoc explanation methods for computing attribution scores of input tokens are based on gradient computations [19]. That is, the attribution for the token  $\mathbf{x}_i$  in input  $\mathbf{x} \in \mathbb{R}^{\mathbb{N}}$  is computed as  $\frac{\partial f(\mathbf{x})\hat{\mathbf{y}}}{\partial\mathbf{x}_i}$ , as is the case with Vanilla Gradients [25]. We experiment with several other post hoc explanation methods discussed in more detail in the experiment section.
Step (4): Prompt Design for LLMs. In the final step, we proceed to construct the corrective rationale for each selected sample using the template: "The key words: word1, word2, ...and wordk are important clues to predict [Label] as the correct answer." In this template, "[word1, word2 ..., and wordk]" refers to the top-k most important words in the input sentence for the true label, which were obtained from the previous step. Using the few-shot template [Input][Rationale][Label], we construct the s-shot prompt as "[Input1][Rationale1][Label1]...[Input_s][Rationale_s][Label_s]", which is simply the concatenation of [Input][Rationale][Label] for each selected sample. This constructed prompt is then combined with the input of the test sample to form the final input prompt
for the LLMs, enabling them to make predictions for the samples in the test set. This process is illustrated in the last step of Figure 1.
In this section, we discuss our empirical evaluation in detail. First, we describe our experiment setup and provide details about the datasets and tasks we experiment with. Next, we evaluate the effectiveness of our framework in improving task performance of LLMs on a variety of real-world tasks. Lastly, we examine the impact of each step of our framework AMPLIFY by conducting rigorous ablation studies.
Datasets. We evaluate our framework AMPLIFY on some of the popular datasets from the Big-Bench-Hard[29] benchmark. More specifically, we experiment with: (1) The Snarks[29] dataset which gauges a model's proficiency in discerning sarcastic sentences from a selection of alternatives; (2) The Causal Judgment[29] dataset, designed to evaluate a model's ability in accurately deducing the causative factors of an event from a detailed summary; (3) The Ruin Names[29] task, which involves the identification of comical modifications to artist or movie names; (4) The Formal Fallacies[29] task, where machine learning models are put to the test to distinguish between logically sound arguments and those with logical discrepancies; (5) The Salient Translation Error Detection[29] task, engineered to train models in identifying one out of six predetermined translation errors given a pair of translations; (6) The CommonSenseQA [32] dataset, a multiple-choice question answering platform that necessitates a wide variety of commonsense knowledge for accurately determining the correct responses; (7) Lastly, the Coin Flip [35] dataset, a synthetically generated dataset used for assessing the symbolic reasoning capacity of LLMs.
Large Language Models. Our methodology is assessed in comparison to baseline approaches on two LLMs. First, GPT-3 [4], a language model with 175 billion parameters, demonstrates robust performance across a range of natural language processing tasks without the need for explicit training or fine-tuning. Second, GPT-3.5 [1] is a series of models that were trained on a mix of text and code data before the end of the fourth quarter in 2021. These models, expressly crafted for chat applications, function as an advanced version of InstructGPT [20].
Post Hoc Explanation Techniques. In this study, we use three widely adopted post hoc explanation methods to generate explanations that are later incorporated as rationales into prompts for in-context learning. These methods include Vanilla Gradients [25], Gradient x Input [24], and contrastive explanations [36]. Vanilla Gradients [25] calculates feature attributions by computing the norm of the gradients of model output with respect to each token's embedding. Gradient x Input derives attribution scores by taking the product of gradient and input embedding. Finally, contrastive gradients determine attribution scores by subtracting the gradients with respect to the model prediction token from those associated with the truth label. We apply these explanation techniques to two proxy models for the generation of corrective rationales in step 3 of AMPLIFY: GPT-2 ( $\sim$ 125 Mn parameters)[21] and BERT ( $\sim$ 110 Mn parameters) [7].
Baseline Methods. In our experiments, we evaluate the performance of AMPLIFY in comparison to two alternative approaches, namely Answer-Only (AO) prompts [29] and Chain-of-Thought (CoT) [35]. AO prompting represents the standard few-shot prompting technique, in which the input prompt consists of a few (input, label) pairs and the test input sentence, followed by an answer delimiter ('A:') for the LLM to generate the response. Chain-of-Thought (CoT), on the other hand, is the current state-of-the-art method that enhances the input prompt by including human-annotated rationales for each few-shot example. The LLM is then expected to generate a rationale followed by an answer.
Implementation Details. In our experiments, we implemented the AO and CoT baselines using the same methodology as described in their respective works. For CoT, we directly used the provided rationales from the original work for the corresponding datasets [35]. In the case of AMPLIFY, we employed GPT-2[22] fine-tuned for the target task as the proxy model for step 1, unless stated otherwise. We utilized a rationale template with  $k = 5$ , which is of the form: "The key words: word $_1$ , word $_2$ , ...and word $_5$  are important clues to predict [ground truth label] as the correct answer". These keywords "word $_1$ , word $_2$ , ...and word $_5$  " were chosen based on the highest attribution scores obtained
Table 1: Few-shot prompting performance of several large language models on the seven datasets. AO: standard "answer-only" prompting. CoT: chain-of-thought prompting. Best model performance is in bold. The LLMs we experimented with are GPT-3 and GPT-3.5. The recorded performance in this table represents the percentage of test samples for which the LLM accurately predicted the true label.
Experiment Tasks	[29, 35]	Human-Rater [38]	GPT-3	GPT-3.5
Random	SOTA	Avg.	Max	AO	CoT	AMPLIFY	AO	CoT	AMPLIFY
Snarks	50.0	71.3	76.7	100	52.7	61.1	80.5	75.0	69.4	91.6
Causal Judgment	50.0	62.1	69.6	100	55.2	55.2	60.5	57.8	63.1	76.3
Ruin Names	25.0	72.8	77.7	100	64.0	62.9	78.6	69.6	62.9	77.5
Formal Fallacies	25.0	52.2	90.8	100	53.6	50.8	60.1	51.4	54.6	59.9
Salient Translation Error Detection	16.7	31.9	36.7	80.0	48.2	50.2	51.7	43.2	54.7	60.8
CommonsenseQA	20.0	80.0	90.0	100	69.3	72.6	73.5	75.7	75.2	77.9
Coin Flip (OOD)	-	-	-	-	54.7	63.3	65.7	52.9	61.0	65.3
All Tasks (avg)	31.1	61.7	73.5	96.6	56.8	58.0	67.2	60.8	62.9	72.7