SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 6: This figure shows LLM performance for the different selections of  $k$  and  $s$  hyper-parameters of AMPLIFY, as denoted by  $\left( {k,s}\right)$  for each column. In general,we observe  $\left( {k = 7,s = {10}}\right)$  achieves the best results for most of the datasets.
Experiment Tasks	GPT-3 (k,s)	GPT-3.5 (k,s)
(2,5)	(5,5)	(5,10)	(7,10)	(2,5)	(5,5)	(5,10)	(7,10)
Snarks	63.8	72.2	80.5	80.5	75.0	80.5	91.6	88.8
Causal Judgment	52.6	57.8	60.5	60.5	65.7	73.6	76.3	76.3
Ruin Names	64.0	75.2	76.4	78.6	73.0	75.2	77.5	77.5
Formal Fallacies	55.5	57.9	59.8	59.8	56.3	58.8	59.6	59.6
Salient Translation Error Detection	49.7	50.2	51.2	51.2	52.7	56.2	60.8	60.8
CommonsenseQA	72.8	73.1	73.3	73.5	76.0	76.7	77.6	77.9
Coin Flip (OOD)	64.9	65.3	65.7	65.7	63.3	65.0	65.3	65.3
All Tasks (avg)	60.4	64.5	66.7	67.1	66.0	69.4	72.6	72.3
Table 7: Few-shot prompting performance of multiple LLMs on the seven datasets when post hoc explanations, which form the rationale in the prompt constructed during step 4 of AMPLIFY, are generated using models with varying degrees of fine-tuning of the proxy model (BERT in this case). Here, "E" represents the number of epochs the proxy model was fine-tuned. "E = 0" indicates that the proxy model was used to generate post hoc explanations without any fine-tuning. The recorded performance in this table represents the percentage of test samples for which the LLM accurately predicted the true label.
Experiment Tasks	GPT-3	GPT-3.5
E = 0	E = 10	E = 200	E = 0	E = 10	E = 200
Snarks	66.6	72.2	72.2	80.8	80.8	88.8
Causal Judgment	50.0	52.6	57.8	71.0	73.6	73.6
Ruin Names	70.7	73.0	73.0	71.9	71.9	71.9
Formal Fallacies	56.2	56.9	58.5	56.7	56.9	57.8
Salient Translation Error Detection	50.2	51.2	51.2	56.2	59.2	60.8
CommonsenseQA	71.3	71.8	72.4	76.1	76.5	77.4
Coin Flip (OOD)	65.4	65.8	65.9	63.7	64.3	65.1
All Tasks (avg)	61.2	63.1	68.0	68.0	69.0	70.7
C.1 Larger Proxy Model: GPT-2-Medium
While we experimented with the fine-tuning of proxy models, it's important to note that this step can be eliminated by using a more capable pretrained proxy model, while still achieving performance gains over baselines. The Table 8 shows that the performance of LLM surpasses the baseline (CoT) when we use gpt2-medium instead of gpt2-small, without any fine-tuning. This demonstrates that fine-tuning of the proxy model is not mandatory. Our motivation to show results for fine-tuning models in the paper is to demonstrate the improvement in LLM performance when the proxy model is further fine tuned.
C.2 Multi-step Problem Solving : GSM8k
For our experiments, we focused on tasks that require complex language understanding[31], which are also cases where post hoc explanations have been found to be useful in capturing important features, hence providing useful explanations[19]. However, we also experimented with GSM8k (math problem dataset) used in CoT [35] and observed that AMPLIFY outperforms AO but performs lower than CoT, as shown in Table 9.
While we outperform the standard few-shot approach, the underperformance of AMPLIFY when compared to CoT is expected because solving math problems requires multi-step reasoning, a complex function which is beyond what post hoc explanations are designed to explain. We further wish to clarify that we do not present AMPLIFY as a replacement for CoT, but rather as a superior alternative for tasks requiring complex language understanding; these are tasks for which obtaining chains-of-thought through human annotations is exceptionally challenging [31].
Figure 3: This image exemplifies an instance of CommonsenseQA task where standard prompts and CoT produce inaccurate responses. The CoT response fails to take into account the context in the question being related to eyes. In contrast, the response generated by AMPLIFY emphasizes this crucial detail.
Table 8: The table presents a performance comparison for different models on various experiment tasks. The recorded performance in this table represents the percentage of test samples for which the model accurately predicted the true label.
Experiment Tasks	AMPLIFY (gpt2-small)	AMPLIFY (gpt2-medium)	CoT
Snarks	88.8	91.6	69.4
Causal Judgment	71.0	71.0	63.1
Ruin Names	65.1	70.7	62.9
Formal Fallacies	48.3	56.0	54.6
Salient Translation	57.7	60.8	54.7
CommonsenseQA	71.9	75.5	75.2
Coin Flip (OOD)	55.4	59.6	61.0
C.3 Analysis on Other Big-Bench Hard Tasks
We conducted experiments on some more tasks from Big-Bench Hard [31] and observed similar gains achieved by AMPLIFY, shown in Table 10. However, we observed only minimal improvement in the task of word sorting. This is because word sorting requires an understanding of lexical properties over linguistic semantics.
Table 9: The table presents a performance comparison for the GSM8k task using three different methods: AO, CoT, and AMPLIFY with gpt2-small as the proxy model. The recorded performance in this table represents the percentage of test samples for which the model accurately solved the math problem.
Experiment Tasks	AO	CoT	AMPLIFY (proxy model : gpt2-small)
GSM8k	22.7	43.5	27.4
Table 10: The table presents a performance comparison for the Disambiguation QA, Word Sorting, and Hyperbaton. The recorded performance in this table represents the percentage of test samples for which the model accurately solved the task.
Experiment Tasks	Random	SOTA	Avg.	Max	AO	CoT	AMPLIFY
Disambiguation QA	33.2	51.6	66.6	93.3	66.6	70.5	74.5
Word Sorting	0	33.1	62.6	100	37.8	43.1	43.6
Hyperbaton	50.0	67.1	74.7	100	68.5	77.4	79.7