SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Few-shot prompting performance of multiple LLMs on the seven datasets when post hoc explanations, which form the rationale in the prompt constructed during step 4 of AMPLIFY, are generated using models with varying degrees of fine-tuning of the proxy model (GPT-2 in this case). Here, "E" represents the number of epochs the proxy model was fine-tuned. "E = 0" indicates that the proxy model was used to generate post hoc explanations without any fine-tuning. The recorded performance in this table represents the percentage of test samples for which the LLM accurately predicted the true label.
Experiment Tasks	GPT-3	GPT-3.5
E = 0	E = 10	E = 200	E = 0	E = 10	E = 200
Snarks	77.7	80.5	80.5	88.8	88.8	91.6
Causal Judgment	55.2	57.8	60.5	71.0	73.6	76.3
Ruin Names	74.1	75.2	78.6	65.1	67.4	77.5
Formal Fallacies	53.7	56.9	60.1	48.3	51.6	59.8
Salient Translation Error Detection	49.7	51.2	51.7	57.7	60.8	60.8
CommonsenseQA	69.1	72.6	73.5	71.9	75.8	77.9
Coin Flip (OOD)	56.4	60.8	65.7	55.4	61.4	65.3
All Tasks (avg)	62.2	65.0	67.2	65.4	68.4	72.7
from the post hoc explanation computed in step 3. To compute these attribution scores, we used Gradient x Input as the default post hoc explanation method for generating explanations.
Overall Task Performance. We demonstrate the effectiveness of AMPLIFY by comparing the prediction accuracy of LLMs using prompts generated by AMPLIFY against baselines, i.e., Answer-Only (AO) prompts and Chain-of-Thought (CoT) prompts. Table 1 displays the results of GPT-3 and GPT-3.5 across all datasets. We observe that incorporating rationales generated by our approach leads to a substantial improvement in accuracy compared to both standard Answer-Only (AO) prompts and Chain-of-Thought (CoT) prompts. Specifically, GPT-3.5 achieves state-of-the-art performance on the Snarks dataset with a  $91.6\%$  accuracy in identifying the correct option; this is  $16\%$  better than standard answer-only prompting and over  $20\%$  better than CoT. Similar trends were observed for Causal Judgment, where our method delivered the best performance of  $76.3\%$ , significantly surpassing CoT  $(63.1\%)$  and AO  $(57.8\%)$ . When using GPT-3, our approach attained the highest performance in Ruin Names  $(78.6\%)$ , a trend also evident in the case of Formal Fallacies. Finally, our method achieved the top performance with the GPT-3.5, registering an accuracy of  $60.8\%$  for the Salient Translation Error Detection task. In the scenarios of commonsense reasoning (CommonsenseQA) and symbolic reasoning (Coin Flip) tasks, we noticed consistent trends, with AMPLIFY recording the highest performance. In the next set of analyses, we examine the effects of different steps in AMPLIFY on the performance of LLMs.
Impact of Proxy Model Selection on LLM Performance. In the following analysis, we investigate how the choices made at each step of AMPLIFY affect the performance of LLM on the seven tasks. We begin with step 1, which involves selecting a proxy model for sample selection (step 2) and computing post hoc explanations(step 3). In the experiments conducted to calculate the overall model performance, as shown in Table 1, we utilized a finetuned GPT-2 model for the target task as
Table 3: Few-shot prompting performance of various large language models on the seven datasets is analyzed based on different selection strategies used for choosing samples during prompt design of LLMs, specifically in step 2 of Figure 1. The "Random" selection refers to randomly chosen samples. "L-MCS" signifies the selection of samples with the lowest Misclassification Confidence Score (MCS). "H-MCS" represents the selection strategy of choosing samples with the Misclassification Confidence Score (MCS) for prompt design. "F-Exp" indicates the selection strategy of choosing samples with the most faithful explanations for LLM prompt. The recorded performance in this table represents the percentage of test samples for which the LLM accurately predicted the true label.
Experiment Tasks	GPT-3	GPT-3.5
Random	L-MCS	H-MCS	F-Exp	Random	L-MCS	H-MCS	F-Exp
Snarks	69.4	80.5	80.5	77.7	69.4	88.8	91.6	88.8
Causal Judgment	57.8	60.5	60.5	57.8	68.4	73.6	76.3	71.0
Ruin Names	65.1	77.5	78.6	74.1	66.2	77.5	77.5	73.0
Formal Fallacies	52.3	57.9	60.1	59.7	46.7	51.5	59.9	58.6
Salient Translation Error Detection	48.7	50.2	51.7	51.7	53.2	59.2	60.8	58.7
CommonsenseQA	67.6	71.5	73.5	70.9	72.9	76.6	77.9	77.2
Coin Flip (OOD)	54.7	60.1	65.7	58.5	57.8	61.6	65.3	61.1
All Tasks (avg)	59.3	65.4	67.2	64.3	62.0	69.8	72.7	69.7
Table 4: The table presents a performance comparison for when prompt is created using explanations generated by four different post hoc explanation methods. Grad: Vanilla Gradient Method, Grad  $\times$  Inp : Gradient x Input Method , C-Grad and C-Grad  $x$  Inp are contrastive version of Vanilla gradient and Gradient x Input. The recorded performance in this table represents the percentage of test samples for which the LLM accurately predicted the true label.
Experiment Tasks	GPT-3	GPT-3.5
Grad	Grad×Inp	C-Grad	C-Grad×Inp	Grad	Grad×Inp	C-Grad	C-Grad×Inp
Snarks	77.7	80.5	80.5	86.1	88.8	91.6	88.8	91.6
Causal Judgment	60.5	60.5	57.8	60.5	71.0	76.3	71.0	73.6
Ruin Names	71.9	78.6	75.2	77.5	65.1	77.5	73.0	74.1
Formal Fallacies	59.7	60.1	59.7	58.6	59.9	59.9	59.4	57.6
Salient Translation Error Detection	49.7	51.7	51.7	50.7	59.7	60.8	60.8	60.8
CommonsenseQA	72.1	73.5	72.9	73.0	73.7	77.9	75.5	77.9
Coin Flip (OOD)	62.9	64.1	62.6	65.7	62.6	63.9	62.4	65.3
All Tasks (avg)	64.9	67.0	65.7	67.3	68.6	72.5	70.1	71.5
the proxy model. While using GPT-2 for finetuning is computationally cheaper compared to other LLMs, it is still expensive to finetune a model with more than 100 million parameters. Therefore, we examined the performance of LLMs based on the amount of fine-tuning, measured in terms of the number of epochs (E). This analysis aimed to understand the impact of finetuning on improving model performance. Table 2 presents the model performance scores of LLMs when the proxy model is GPT-2 without any fine-tuning on the target task  $(\mathrm{E} = 0)$ , with minor fine-tuning  $(\mathrm{E} = 10)$  and when GPT-2 has achieved its best performance at epoch  $\mathrm{E} = 200$ . As depicted in Table 2, we observe that the model performance of LLMs with GPT-2  $(\mathrm{E} = 0)$  is already quite close to the best performance achieved when GPT-2 is finetuned to saturation  $(\mathrm{E} = 200)$  for most datasets. Specifically, the performance of GPT-3.5 for Snarks with GPT-2  $(\mathrm{E} = 0)$  is  $88.8\%$ , which is significantly better than the best performance of CoT shown in Table 1. This pattern is also evident in the case of Causal Judgment, Salient Translation Error Detection, and Ruin Names. There are two primary reasons for this behavior. Firstly, GPT-2 possesses sufficient language understanding capabilities to provide useful post hoc explanations that lead to accurate predictions. Secondly, most of the tasks where GPT-2  $(\mathrm{E} = 0)$  achieved the best or near-best performance have very limited training data, which might not be sufficient for GPT-2 to learn anything beyond what it has already acquired during pre-training. This observation is further supported by the findings presented in Table 5 in Appendix A, where the accuracy improvements for most datasets are not substantial. These findings suggest that an additional step of fine-tuning a pre-trained model may not always be necessary when selecting a proxy model in step 1 of AMPLIFY, thereby reducing computational costs even further. Lastly, we observe similar trends when BERT is chosen as the proxy model, and the detailed results are presented in Appendix A.4.
Q: How would a typical person answer each of the following questions about causation? A machine is set up in such a way that it will short circuit if both the black wire and the red wire touch the battery at the same time. The machine will not short circuit if just one of these wires touches the battery. The machine is designed so that both wires move around inside the machine. The black wire is supposed to touch the battery at certain times as it moves around inside the machine. The red wire is never supposed to touch the battery as it moves around inside the machine. One day, the black wire and the red wire both come in contact with the battery at the exact same time. There is a short circuit. Did the red wire cause the short circuit?
The answer is No, the short circuit was caused by the simultaneous contact of both the black and red wires with the battery.
CoT
Here in this question, we are told that "The machine is set up in such a way that it will short circuit if both the black wire and the red wire touch the battery at the same time." A typical person would assume that the cause of the short circuit was the fact that both wires touched the battery at the same time, not just the red wire. Therefore, a typical person would answer No, the red wire did not cause the short circuit.
AMPLIFY
The key words "never supposed", "red wire", "battery", "short circuit", and "touch" suggest that the red wire cause the short circuit. The answer is Yes.
Figure 2: This image exemplifies an instance of Causal Judgment task where standard prompts and CoT produce inaccurate responses. The CoT response fails to take into account that the red wire should not make contact with the battery, which caused the short circuit. In contrast, the response generated by AMPLIFY emphasizes this crucial detail. Please note that while we inject rationales in terms of k individual words, we do not restrict LLMs from generating rationales in terms of phrases or multiple words. This is why we often see LLM-generated rationales having multi-word clues, such as "red wire," "never supposed," and so on.
Impact of Selection Strategies on LLM Performance. In this analysis, we focus on step 2 of AMPLIFY, which involves selecting samples for few-shot prompts that can provide effective corrective signals to assist LLMs in reducing misclassifications. As explained in section 3, this step in AMPLIFY includes two sub-steps: first, identifying misclassified samples by LLM on the validation set, and second, selecting samples with the highest MCS calculated with respect to proxy model. The first step is straightforward as we specifically focus on samples that were misclassified earlier by LLMs. For subsequent filtering based on confidence, we experiment with several sample selection strategies to better understand how the performance of the language model is sensitive to these different strategies. The results, in terms of language model performance scores, are shown in Table 3. Specifically, we experiment with three selection strategies in addition to the one that selects samples with the highest MCS score, referred to as "H-MCS" in Table 3: (1) "Random": randomly selecting samples without considering Misclassification Confidence Score, (2) "L-MCS": selecting samples with the lowest MCS, and (3) "F-EXP": selecting samples based on the most faithful explanations, measured by the difference in model output when the top-K features identified by the explanation are perturbed [27, 33]. Among these strategies, we find that selecting samples with the highest Misclassification Confidence Score yields the best performance, outperforming all other strategies. Random selection performs the worst across all datasets, while the performance using "L-MCS" is comparable to that of "H-MCS" selections. This suggests that samples for which the model is less confident in its predictions can still provide reasonable corrective signals in reducing misclassifications. Notably, the LLM performance with "F-EXP" sample selection is worse than "L-MCS" for most datasets (Snarks, Causal Judgment, Ruin Names, and CommonsenseQA), indicating that relying solely on faithful explanations may not be sufficient for achieving optimal performance.
Impact of Post Hoc Explanation Method on LLM Performance. We then examine the impact on LLM performance due to the choice of post hoc explanation used to identify top-k keywords in step 3. To investigate this, we employ four different explanation methods for step 3 of AMPLIFY and record the LLM performance corresponding to each post hoc explanation method choice in Table 4. Specifically, the four post hoc explanation methods used in this analysis are: (1) Vanilla Gradients [25] (Grad), (2) Gradient  $\times$  Input [24] (Grad $\times$ Inp), (3) Contrastive Gradient [36] (C-Grad), and (4) Contrastive Gradient  $\times$  Input [36] (C-Grad $\times$ Inp). Based on Table 4, we observe that the LLM performs best in general when Gradient x Input or its contrastive variant is used to generate explanations. However, we also note that there aren't drastic changes in LLM performance across different methods. For instance, GPT-3.5 performance on Snarks doesn't change much across different methods, as the accuracy remains consistently around  $88.8 - 91.6\%$ . This suggests that LLM performance isn't sensitive to rationales generated using different variants of gradient-based post hoc explanation methods.
Impact of Rationale Template on LLM Performance. Lastly, in the final step of AMPLIFY, we generate a few-shot prompt by combining an (input, label) pair and its corresponding set of more important words using the rationale template as "The key words: word $_1$ , word $_2$ , ...and word $_5$  are crucial clues for predicting [ground truth label] as the correct answer". We have observed that while using a task-independent rationale template leads to improvements in performance, tailoring the rationale to the question asked in the input sentence for a given dataset also provides benefits. For example, in the case of Causal Judgment, each sample includes a generic question: "How would a typical person answer each of the following questions about causation?" If we utilize the rationale template as "After observing the key words: word $_1$ , word $_2$ , ...and word $_5$ , a typical person would respond with [label] as the correct answer", we notice a slight enhancement in GPT-3 performance, rising from  $60.5\%$  to  $63.1\%$ . However, we did not observe the model's sensitivity to minor changes in the template, such as punctuation variations. Further discussions on the impact of hyperparameters associated with AMPLIFY can be found in Appendix A.3.
Qualitative Analysis. In addition to quantitatively evaluating the performance of AMPLIFY compared to other baselines, we also qualitatively examine how LLM responses differ for certain test samples using each of the baseline prompting approaches, and compare them to the responses generated by AMPLIFY. Figure 2 illustrates the responses generated by GPT-3.5 for an input sample using the Standard Prompt (AO), Chain-of-Thought (CoT), and AMPLIFY. In this particular example, both AO and CoT yield incorrect responses, whereas AMPLIFY produces the correct response. Analyzing the responses reveals that CoT and AO miss an important sentence in the sample: "The red wire is never supposed to touch the battery as it moves around inside the machine". Interestingly, GPT-3.5 captures this crucial information when the prompt is augmented with post hoc explanations using AMPLIFY. We observe similar examples for CommonsenseQA, such as "Q: Unlike a spider and its many observers, people only have what? Answer Choices: (A) tongues (B) names (C) brains (D) feelings (E) two eyes." In this case, CoT incorrectly selects option (C), whereas AMPLIFY correctly predicts option (E). The complete response is shown in Figure 3 in the Appendix A. This error also stems from the same issue of CoT overlooking crucial information that the question pertains to eyes rather than the entire body, a nuance that AMPLIFY successfully captures. This demonstrates that the rationales generated by AMPLIFY can assist LLMs in capturing critical signals that might have otherwise been overlooked.
In this work, we introduce AMPLIFY, a novel framework aimed at improving the performance of LLMs by replacing human-annotated rationales with automated rationales obtained using post hoc explanation techniques. Our unique four-step approach leverages smaller, open-source models for efficient computation of post hoc explanations. Our framework results in performance improvements of  $10 - 25\%$  across diverse tasks, outperforming conventional techniques such as CoT prompting which rely on human-annotated rationales. Our findings highlight the potential of post hoc explanation methods as valuable tools for enhancing the effectiveness of LLMs.
We thank Adam Pearce for helpful feedback on writing and presentation of this work. This work is supported in part by the NSF awards IIS-2008461, IIS-2040989, IIS-2238714, and research awards from Google, JP Morgan, Amazon, Harvard Data Science Initiative, and the Digital, Data, and Design (D^3) Institute at Harvard. The views expressed here are those of the authors and do not reflect the official policy or position of the funding agencies. This work was also funded in part by Hasso Plattner Institute (HPI) through the UCI-HPI fellowship, and in part by the NSF awards IIS-2008956, IIS-2046873, and IIS-2040989.
[1] Gpt-3.5-turbo. https://platform.openai.com/docs/model-index-for-researchers., Accessed: 2022-01-01.
[2] Gpt-4 system card. https://cdn.openai.com/papers/gpt-4-system-card.pdf, Accessed: 2022-01-01.
[3] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
[4] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[5] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
[6] S. Choudhary, N. Chatterjee, and S. K. Saha. Interpretation of black box nlp models: A survey. arXiv preprint arXiv:2203.17081, 2022.
[7] J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2018.
[8] Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui. A survey for in-context learning. arXiv preprint arXiv:2301.00234, 2022.
[9] F. Doshi-Velez and B. Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608, 2017.
[10] M. Idahl, L. Lyu, U. Gadiraju, and A. Anand. Towards benchmarking the utility of explanations for model debugging. arXiv preprint arXiv:2105.04505, 2021.
[11] S. Jesus, C. Belém, V. Balayan, J. Bento, P. Saleiro, P. Bizarro, and J. Gama. How can i choose an explainer? an application-grounded evaluation of post-hoc explanations. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 805-815, 2021.
[12] Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
[13] S. Krishna, T. Han, A. Gu, J. Pombra, S. Jabbari, S. Wu, and H. Lakkaraju. The disagreement problem in explainable machine learning: A practitioner's perspective. arXiv preprint arXiv:2202.01602, 2022.
[14] H. Lakkaraju, N. Arsov, and O. Bastani. Robust and stable black box explanations. In International Conference on Machine Learning, pages 5628-5638. PMLR, 2020.
[33] M. Turpin, J. Michael, E. Perez, and S. R. Bowman. Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting. arXiv preprint arXiv:2305.04388, 2023.
[34] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022.
[35] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
[36] K. Yin and G. Neubig. Interpreting language models with contrastive explanations. arXiv preprint arXiv:2202.10419, 2022.
[37] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pages 818-833. Springer, 2014.
[38] W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023.
A.1 Proxy Model Task Performance
Table 5: Proxy models performance on the target tasks with and without fine-tuning.
Experiment Tasks	GPT-2	BERT
Pre-trained	Fine-tuned	Pre-trained	Fine-tuned
Snarks	38.8	47.2	30.5	38.8
Causal Judgment	44.7	55.2	44.7	52.6
Ruin Names	07.8	26.9	10.1	22.4
Formal Fallacies	50.5	54.4	51.6	53.5
Salient Translation Error Detection	14.0	27.1	11.5	22.6
CommonsenseQA	07.4	29.1	08.8	26.9
Coin Flip	45.2	59.4	51.1	59.7
A.2 Qualitative Analysis
Figure 3 shows an example from CommonsenseQA where GPT-3.5 responses using AO and CoT prompting yield an incorrect answer. The most likely reason for this is that these prompt strategies don't seem to capture all the key points of the input sentence, i.e., the context in the input is based on eyes rather than the overall body. However, this crucial detail is captured when GPT-3.5 is prompted with AMPLIFY. We observe that the GPT-3.5 response is correct, and it acknowledges "eyes" as the most important clue in making the correct prediction.
A.3 Hyper-parameter Analysis
Recall that AMPLIFY has two other primary hyper-parameters apart from the rationale template choice discussed in our empirical findings, namely,  $s$ , which is the size of the few-shot prompt created for LLMs, and  $k$ , which is the number of most important tokens identified by the post hoc explanation. Table 6 shows the LLM performance variations for different combinations of  $(k, s)$ . It is important to note that AMPLIFY does not have scalability constraints with increasing  $s$  and  $k$ , as AMPLIFY computes prompts automatically. This is unlike CoT, where increasing the size of the few-shot prompt would require more human effort to generate relevant chains of thoughts.
A.4 Impact of BERT as Proxy Model on LLM Performance
Table 7 shows LLM performance when BERT is used as the proxy model in step 1 of AMPLIFY. We observe similar trends as those observed for the case of GPT-2, where fine-tuning proxy model provides marginal improvements in general. This indicates that the fine-tuning step could be avoided in most cases to reduce additional computational overhead.
B Limitations and Broader Impacts
Our work proposes a new framework, AMPLIFY, which focuses on improving the task performance of LLMs by injecting automatically generated rationales. This framework results in the reduction of reliance on processes that require heavy human intervention. These processes, which rely on rationales based on human annotations, often suffer from noise and biases, which may transfer to LLMs during in-context learning. We hope that automated rationale creation will provide a solution to mitigate this problem. While our approach provides significant improvements in model performance, the broader negative impact pertaining to LLMs, such as safety concerns in the form of misinformation[2], social bias[2], hallucination[12], etc., and the massive carbon footprint due to heavy usage of LLMs [17], may still persist even when using our proposed framework. Other than the limitations of LLMs, our framework relies on post hoc explanation methods to create automated rationales; hence, AMPLIFY may also inherit widely studied issues with post hoc explanations such as robustness[14], the disagreement problem[13], stability[26], etc.