SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We experiment with two state-of-the-art open-source LLMs of varying sizes, ranging from 780M to 65B parameters. Specifically, we use two encoder-decoder and decoder-only models as student and teacher: (1) Flan-T5-Large and Flan-T5-XL [48], and (2) LLaMA-7B, LLaMA-13B, and LLaMA-65B [49]. Typically, the student model is assumed to be smaller than the teacher model. But some experiments will also involve smaller teachers and larger students. All models generate text using greedy decoding, prompted with either 4-8 demonstrations. Unless otherwise stated, the demonstrations are randomly chosen from the training samples. For StrategyQA, we report results on the validation split, while for CommonsenseQA and GSM8k, our experiments are on the test split. To account for variance, we conduct experiments with at least three different seeds. We report accuracy for all tasks, and error bars in plots and tables represent the standard deviation across seeds.
B Datasets and Prompts
We experiment with the following three reasoning datasets: (1) StrategyQA [45], a set of open-domain questions where the required reasoning steps are implicit in the question, (2) GSM8k [46], which includes multi-step math reasoning problems, and (3) CommonsenseQA [47], a multiple-choice QA task focusing on commonsense reasoning. We use the reasoning steps in StrategyQA and GSM8k as the multi-step rationales, and for CommonsenseQA, we rely on the ECQA dataset [52], which is annotated with commonsense facts supporting the correct option and refuting the incorrect options. All datasets are licensed under the MIT license. Fig. 9 shows the student prompts for the three tasks of StrategyQA, CommonsenseQA, and GSM8k. Fig. 10 shows the pre- and post-intervention student simulation prompts for the teacher model.
C Compute and Reproducibility
We conduct experiments either on A100 Google Cloud instances or on internal A6000 GPU servers. The LLMs (Flan-T5 and LLaMA) and the datasets used in our studies are publicly available. For reproducibility, we are making our code available as part of the supplementary material.
StrategyQA
Q: Are more people today related to Genghis Khan than Julius Caesar?
A: Julius Caesar had three children. Genghis Khan had sixteen children. Modern geneticists have determined that out of every 200 men today has DNA that can be traced to Genghis Khan. So the answer is yes
Q: {test_question}
A:
CommonsenseQA
Q: What might a person see at the scene of a brutal killing?
Answer Choices:
Choice 1: bloody mess
Choice 2: pleasure
Choice 3: being imprisoned
Choice 4: feeling of guilt
Choice 5: cake
A: Bloody mess is covered or stained with blood. A person might see a bloody mess at the scene of a brutal killing. Pleasure is about what a person sees at the scene of a brutal killing and one cannot be happy to see such brutality. You can't see someone in jail at the brutal killing scene. Feeling of guilt doesn't come as the killing is brutal or merciless. Cake is baseless and weird to think as it is a brutal killing scene and not a bakery. So the correct choice is 1
Q: {test_question}
Answer Choices:
Choice 1: {option_1}
Choice 2: {option_2}
Choice 3: {option_3}
Choice 4: {option_4}
Choice 5: {option_5}
A:
GSM8k
Q: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?
A: Natalia sold 48/2 = 24 clips in May. Natalia sold 48+24 = 72 clips altogether in April and May. So the answer is 72
Q: {test_question}
A:
Figure 9: Examples of student prompts for different tasks with one demonstration.
D RQ1: Additional Results
Results with Flan and LLaMA Models. In Table 1, we report the accuracy obtained by different students and teachers (based on Flan-T5 models) on the StrategyQA task. We draw similar conclusions as Flan-T5 with other LLMs, specifically LLaMA-7B and LLaMA-65B models on the StrategyQA dataset (Table 2). In fact, when the teacher is stronger like a LLaMA-65B, the margin of improvement is also higher, about  $8\%$ . The overall trends also align – increasing for weaker students and decreasing for stronger students.
Results on other Datasets. Our conclusions generalize across datasets too. Table 3 presents the results on CommonsenseQA with Flan-T5 models. CommonsenseQA is an easier task and Flan-T5 models obtain accuracies of  $85\%$  and  $92\%$  when generating their own explanations. While Flan-T5-Large still benefits from human explanations, the larger model does not, perhaps because it already starts at a high  $92\%$  accuracy. Finally, in Table 4, we present the results on GSM8k with LLaMA models. Note that in GSM8k, a student has access to partial explanations from the teacher, but even then we observe that these prove to be useful prompts for the student to complete their chain-of-thought, leading to up to  $8 - 9\%$  increase in accuracy with human teachers and  $3\%$  with model teachers.
Pre-Intervention Student Simulation
Simulate an AI model's answer for the given question.
Q: Will the Albany in Georgia reach a hundred thousand occupants before the one in New York? AI Predicted Answer: Albany, Georgia is a city in the U.S. state of Georgia. Albany, Georgia has a population of 59,080. The population of New York is 365,040. So the answer is no
Q: {question} AI Predicted Answer:
Post-Intervention Student Simulation
Simulate an AI model's answer for the given question.
Q: Will the Albany in Georgia reach a hundred thousand occupants before the one in New York? AI Predicted Answer: Albany, Georgia is a city in the U.S. state of Georgia. Albany, Georgia has a population of 59,058. The Albany in New York has a population of 328,058. So the answer is no
Q: {question} AI Predicted Answer: {teacher Explanation} So the answer is
Figure 10: Examples of StrategyQA prompts for the mental model of a teacher simulating student predictions pre-intervention and post-intervention. Pre-intervention: The demonstrations use student explanations and student predictions and at test time, the teacher simulates both. Post-intervention: The demonstrations use teacher explanations and student predictions and at test time, the teacher uses the teacher explanation to simulate the student prediction.
Results with Cross-family Student and Teacher. We observe that larger teacher LLMs can teach smaller student LLMs, even when they are of different model families. In Table 5, we report the results with Flan-T5 and LLaMA models as students and teachers.
Student	Teacher	Intervention Budget
0%	20%	40%	60%	80%	100%
Flan-T5-Large	Human	58.51±2.00	63.75±0.43	66.95±2.19	73.94±2.77	78.02±2.40	81.95±1.65
Flan-T5-XL	Human	68.12±2.62	72.05±2.62	75.98±2.31	80.20±1.65	84.13±1.00	87.77±0.70
Flan-T5-Large	Flan-T5-XL	58.51±2.00	60.52±1.63	59.78±1.85	61.48±2.02	62.35±2.13	62.96±2.47
Flan-T5-XL	Flan-T5-Large	68.12±2.62	67.68±2.72	65.64±3.39	64.04±3.63	62.88±1.15	61.86±0.66
Table 1: RQ1 – Comparison of accuracy obtained with random intervention by Flan-T5 models at different intervention budgets on StrategyQA. As shown in the third row, Flan-T5-Large (student) accuracy improves by  $5\%$  with  $100\%$  intervention from Flan-T5-XL (teacher).
Student	Teacher	Intervention Budget
0%	20%	40%	60%	80%	100%
LLaMA-7B	Human	61.13±2.72	63.60±4.82	68.85±3.52	73.36±2.18	78.45±2.55	81.22±1.57
LLaMA-65B	Human	77.58±2.24	80.34±2.65	82.67±2.06	87.48±1.96	89.37±0.25	92.86±0.50
LLaMA-7B	LLaMA-65B	61.13±2.72	62.29±1.53	64.91±0.67	66.08±1.76	68.99±3.14	69.43±3.41
LLaMA-65B	LLaMA-7B	77.58±2.24	75.83±2.24	72.92±2.72	72.92±2.26	70.88±0.90	69.14±0.66
Table 2: RQ1 – Comparison of accuracy obtained with random intervention by LLaMA models at different intervention budgets on StrategyQA. As shown in the third row, LLaMA-7B (student) accuracy improves by  $8\%$  with  $100\%$  intervention from LLaMA-65B (teacher).
E RQ2: Additional Results
Results with stronger Flan-T5-XL teacher and weaker Flan-T5-Large student. Table 6 compares different Intervention Functions on StrategyQA with Flan-T5-Large as the student and Flan-T5-XL as the teacher. These results are when the teacher has access to the gold labels. In Table 7, we
Student	Teacher	Intervention Budget
0%	20%	40%	60%	80%	100%
Flan-T5-Large	Human	84.78±0.41	86.86±0.76	88.70±0.94	90.77±0.45	93.20±0.47	95.42±0.17
Flan-T5-XL	Human	92.38±0.16	92.52±0.20	92.43±0.28	92.23±0.61	92.41±1.12	92.21±1.06
Flan-T5-Large	Flan-T5-XL	84.78±0.41	85.79±0.48	86.79±0.84	87.46±0.20	88.52±0.39	89.72±0.68
Flan-T5-XL	Flan-T5-Large	92.38±0.16	90.92±0.39	89.74±0.39	87.98±0.89	86.70±1.60	85.19±1.62
Table 3: RQ1 – Comparison of accuracy obtained with random intervention by Flan-T5 models at different intervention budgets on CommonsenseQA. As shown in the third row, Flan-T5-Large (student) accuracy improves by  $5\%$  with  $100\%$  intervention from Flan-T5-XL (teacher).
Student	Teacher	Intervention Budget
0%	20%	40%	60%	80%	100%
LLaMA-7B	Human	9.62±1.53	11.97±0.80	13.84±1.02	16.32±0.57	18.72±0.78	21.05±0.65
LLaMA-13B	Human	16.45±1.80	18.44±2.16	20.34±1.60	22.41±2.46	24.91±2.07	26.88±2.34
LLaMA-7B	LLaMA-13B	9.62±1.53	10.20±1.06	10.68±0.82	11.24±0.50	11.92±1.15	12.25±0.94
LLaMA-13B	LLaMA-7B	16.45±1.80	15.87±1.62	15.56±1.44	14.88±1.89	14.68±1.88	14.27±1.70
Table 4: RQ1 – Comparison of accuracy obtained with random intervention by LLaMA models at different intervention budgets on GSM8k. As shown in the third row, LLaMA-7B (student) accuracy improves by  $3\%$  with  $100\%$  intervention from LLaMA-13B (teacher).
Student	Teacher	Intervention Budget
0%	20%	40%	60%	80%	100%
Flan-T5-Large	LLaMA-65B	58.51±2.00	61.86±0.25	61.13±2.26	64.48±1.53	66.52±4.05	66.95±4.90
LLaMA-65B	Flan-T5-Large	77.58±2.24	74.52±1.76	71.47±0.90	67.68±2.00	64.62±2.00	62.15±1.76
Table 5: RQ1 – Comparison of accuracy obtained with random intervention on StrategyQA when the student and the teacher belong to different model families. As shown in the first row, Flan-T5-Large (student) accuracy improves by  $8\%$  with  $100\%$  intervention from LLaMA-65B (teacher).
Intervention Function	Intervention Budget
0%	20%	40%	60%	80%	100%
Random	58.51±2.00	60.40±1.76	61.13±2.65	60.98±1.09	64.33±4.54	62.96±2.47
Teacher Conf ↑	58.51±2.00	58.66±2.40	60.11±2.90	57.35±3.30	61.42±3.91	62.96±2.47
Expected Student Conf (Pre) ↓	58.51±2.00	64.19±2.00	66.66±0.25	66.81±1.57	65.35±2.40	62.96±2.47
Expected Student Conf (Post) ↑	58.51±2.00	64.77±1.76	68.26±0.66	69.71±2.01	68.26±2.63	62.96±2.47
Expected Utility ↑	58.51±2.00	67.83±1.53	71.32±1.33	71.17±1.15	69.86±2.43	62.96±2.47
True Student Conf (Pre) ↓	58.51±2.00	68.26±1.65	80.20±1.26	74.38±2.84	68.55±3.88	62.96±2.47
True Student Conf (Post) ↑	58.51±2.00	65.64±1.40	72.63±1.09	80.05±0.90	72.19±4.39	62.96±2.47
True Utility ↑	58.51±2.00	76.56±0.50	80.78±1.15	81.51±1.76	78.60±3.29	62.96±2.47
Table 6: RQ2 – Comparison of different Intervention Functions with a Flan-T5-Large student and a Flan-T5-XL teacher on StrategyQA. The teacher assumes access to gold labels. ↑ denotes that the samples are ranked in decreasing order of the function (higher is better), while ↓ denotes that the samples in increasing order of the function (lower is better).
Intervention Function	Intervention Budget
0%	20%	40%	60%	80%	100%
Random	58.51±2.00	60.40±1.76	61.13±2.65	60.98±1.09	64.33±4.54	62.96±2.47
Least Conf ↓	58.51±2.00	61.13±0.75	62.44±1.74	65.06±1.15	63.46±2.97	62.96±2.47
Expected Student Conf (Pre) ↓	58.51±2.00	62.59±1.00	61.86±0.90	62.29±1.33	65.50±3.14	62.96±2.47
Expected Student Conf (Post) ↑	58.51±2.00	61.86±1.96	62.88±1.74	61.71±3.39	60.11±4.62	62.96±2.47
Expected Utility ↑	58.51±2.00	62.29±0.50	62.44±1.50	62.44±3.88	62.95±2.78	62.96±2.47
Table 7: RQ2 - Comparison of different Intervention Functions with a Flan-T5-Large student and a Flan-T5-XL teacher on StrategyQA. The teacher, in this case, does not have access to gold labels.
Intervention Function	Intervention Budget
0%	20%	40%	60%	80%	100%
Random	68.12±2.62	67.68±2.72	65.64±3.39	64.04±3.63	62.88±1.15	61.86±0.66
Expected Student Conf (Pre) ↓	68.12±2.62	66.22±2.24	66.95±1.53	65.35±1.00	62.73±0.66	61.86±0.66
Expected Student Conf (Post) ↑	68.12±2.62	70.59±3.27	71.76±3.63	72.48±2.86	69.86±2.62	61.86±0.66
Expected Utility ↑	68.12±2.62	70.88±3.27	71.90±2.84	72.63±2.24	68.99±1.15	61.86±0.66
True Student Conf (Pre) ↓	68.12±2.62	74.23±3.73	76.27±1.40	68.55±1.00	64.04±0.90	61.86±0.66
True Student Conf (Post) ↓	68.12±2.62	70.16±3.27	73.94±1.76	80.05±1.65	71.32±1.09	61.86±0.66
True Utility ↑	68.12±2.62	79.91±2.00	80.93±2.06	80.64±2.24	78.16±2.00	61.86±0.66
Table 8: RQ2 – Comparison of different Intervention Functions with a smaller teacher (Flan-T5-Large) and a larger student (Flan-T5-XL) on StrategyQA. The teacher assumes access to gold labels.
Intervention Function	Intervention Budget
0%	20%	40%	60%	80%	100%
Random	61.13±2.72	62.29±1.53	64.91±0.67	66.08±1.76	68.99±3.14	69.43±3.41
EU (w/ teacher answers)	61.13±2.72	66.22±2.63	67.39±2.40	69.28±1.76	70.59±3.81	69.43±3.41
EU (w/ gold label)	61.13±2.72	66.52±3.27	70.16±0.90	71.47±1.09	72.78±2.48	69.43±3.41
Table 9: RQ2 – Comparison of Expected Utility (with and without access to gold labels) with random intervention, involving a LLaMA-7B student and a LLaMA-65B teacher. EU = Expected Utility. Importantly, even when the teacher does not have access to the gold labels, expected utility with teacher answers (second row) leads to a statistically significant  $5\%$  improvement in student accuracy  $(p = 0.02)$  at  $20\%$  intervention.
Intervention Function	Intervention Budget
0%	20%	40%	60%	80%	100%
Random	84.79±0.41	85.79±0.48	86.79±0.84	87.46±0.20	88.52±0.39	89.72±0.68
Expected Student Conf (Pre) ↓	84.79±0.41	84.57±0.69	86.35±0.73	87.99±0.87	89.51±0.82	89.72±0.68
Expected Student Conf (Post) ↑	84.79±0.41	86.66±0.37	88.69±0.19	90.76±0.06	92.43±0.61	89.72±0.68
Expected Utility ↑	84.79±0.41	87.34±1.09	89.33±0.55	90.27±0.40	91.30±0.22	89.72±0.68
True Student Conf (Pre) ↓	84.79±0.41	92.03±0.19	91.70±0.04	91.03±0.34	90.27±0.41	89.72±0.68
True Student Conf (Post) ↓	84.79±0.41	87.40±0.39	89.59±0.53	92.31±0.09	94.98±1.57	89.72±0.68
True Utility ↑	84.79±0.41	92.87±0.18	93.99±0.02	94.65±0.13	95.57±0.24	89.72±0.68
Table 10: RQ2 – Comparison of different Intervention Functions with a Flan-T5-Large student and a Flan-T5-XL teacher on CommonsenseQA. The teacher assumes access to gold labels.
Intervention Function	Intervention Budget
0%	20%	40%	60%	80%	100%
Random	9.62±1.53	10.20±1.06	10.68±0.82	11.24±0.50	11.92±1.15	12.25±0.94
Expected Student Conf (Pre) ↓	9.62±1.53	11.11±1.44	11.37±1.17	11.56±1.34	12.40±1.01	12.25±0.94
Expected Student Conf (Post) ↑	9.62±1.53	12.80±1.28	12.91±0.58	13.10±0.10	12.72±2.14	12.25±0.94
Expected Utility ↑	9.62±1.53	13.68±1.87	14.06±1.44	13.99±0.80	13.68±0.58	12.25±0.94
Table 11: RQ2 – Comparison of different Intervention Functions with a LLaMA-7B student and a LLaMA-13B teacher on GSM8k. The teacher assumes access to gold labels.
compare the accuracy on StrategyQA when the teacher (Flan-T5-XL) does not have access to gold labels.
Results with weaker Flan-T5-Large teacher and stronger Flan-T5-XL student. RQ1 demonstrated that random intervention by a smaller teacher may not benefit a larger student. But, does Expected Utility benefit in such scenarios? We show this through Figure 8, which compares the accuracy on StrategyQA with Flan-T5-Large as the teacher and Flan-T5-XL as the student. While random intervention shows a monotonically decreasing trend with more intervention, Expected Utility improves the accuracy by  $2\%$  ( $68\% \rightarrow 70\%$ ) by paying  $20\%$  intervention cost and by  $4\%$  by paying  $60\%$  cost. Thus, we conclude that weaker teachers can also teach stronger students with
Teacher Explanation Type	Intervention Budget
0%	20%	40%	60%	80%	100%
Unpersonalized-Rationales	58.51±2.00	66.52±2.97	69.14±1.76	70.16±1.09	67.97±0.50	60.40±0.50
Unpersonalized-CoT	58.51±2.00	67.83±1.53	71.32±1.33	71.17±1.15	69.86±2.43	62.96±2.47
Personalized	58.51±2.00	69.28±1.26	71.61±1.15	72.63±1.33	68.55±1.90	62.73±2.80
Human Explanations	58.51±2.00	72.34±0.90	77.72±0.75	81.51±1.09	82.09±0.87	81.36±0.66
Table 12: RQ3 – Comparison of different kinds of teacher explanations (unpersonalized-rationales, unpersonalized-CoT, personalized, and human) on the student accuracy for StrategyQA. Here Flan-T5-Large is the student model and Flan-T5-XL is the teacher model.
Teacher Explanation Type	Intervention Budget
0%	20%	40%	60%	80%	100%
Unpersonalized-CoT	61.13±2.72	66.52±1.27	70.16±0.90	71.47±1.09	72.78±2.48	69.43±3.41
Personalized	61.13±2.72	68.95±1.26	71.86±2.72	72.61±1.96	73.17±4.00	69.57±1.53
Table 13: RQ3 - Comparison of unpersonalized and personalized teacher explanations on the student accuracy for StrategyQA. Here LLaMA-7B is the student model and LLaMA-65B is the teacher model.
Demonstrations Type	#Rounds
1	2	3	4	5
No Explanations	55.45±2.26	56.04±4.19	58.95±4.16	57.35±3.21	57.93±2.66
Student Explanations	56.08±4.16	55.31±3.14	54.24±2.00	53.90±4.21	53.85±3.73
Teacher Explanations	55.74±2.40	60.84±3.71	59.97±2.66	59.82±4.55	61.57±1.31
Table 14: RQ4 - Results of Multi-turn interaction between the student and the teacher comparing student accuracy on unexplained test points with unexplained, student-explained and teacher-explained demonstrations.
appropriately designed Intervention Functions, especially when the student and the teacher have some complementary benefits.
Results with LLaMA models. Table 9 compares Expected Utility-based intervention with random intervention for LLaMA models (LLaMA-7B as the student and LLaMA-65B as the teacher) on StrategyQA. We evaluate expected utility in two scenarios – with and without gold labels. Both provide improvements over random intervention, as also observed with the Flan models. In particular, when the teacher does not have access to the gold labels (second row), one can compute expected utility with respect to the teacher predictions and obtain a significant  $5\%$  improvement  $(p = 0.02)$  in student accuracy at  $20\%$  intervention.
Results on Other Datasets. Table 10 compares different Intervention Functions on the Common-senseQA dataset with Flan-T5-Large as the student and Flan-T5-XL as the teacher. Table 11 reports results on the GSM8k dataset with LLaMA-7B as the student and LLaMA-13B as the teacher.
F RQ3: Additional Results
Table 12 compares different kinds teacher explanations on student accuracy for StrategyQA with Flan-T5-Large as the student model and Flan-T5-XL as the teacher model. Table 13 compares unpersonalized and personalized explanations on StrategyQA with LLaMA-7B as the student model and LLaMA-65B as the teacher model. Figure 12 shows five qualitative examples from StrategyQA of unpersonalized and personalized explanations generated by a LLaMA-65B teacher model for a LLaMA-7B student model. We observe a common pattern that the personalized explanations are shorter, simpler, and more directed toward answering the question. The unpersonalized explanations, while still factually correct, are elaborate (e.g., see 'Example 5') that may end up distracting the
Intervention Function	Intervention Budget
0%	20%	40%	60%	80%	100%
Random	58.51±2.00	60.40±1.76	61.13±2.65	60.98±1.09	64.33±4.54	62.96±2.47
Neg Expected Utility	58.51±2.00	52.98±1.76	51.09±1.57	50.80±1.40	53.42±3.10	62.45±1.57
Table 15: RQ5 - Comparison of random intervention function with negative expected utility, demonstrating that the teacher can hurt the student by intervening on samples where the utility is the lowest.
Teacher Explanation Type	Intervention Budget
0%	20%	40%	60%	80%	100%
Unpersonalized-CoT	58.51±2.00	67.83±1.53	71.32±1.33	71.17±1.15	69.86±2.43	62.96±2.47
Deceiving Explanations	58.51±2.00	66.95±1.76	69.86±1.15	68.70±0.66	66.66±1.40	62.73±3.27
Table 16: RQ5 - Comparison of a deceiving teacher with an unpersonalized teacher on StrategyQA with Flan-T5-Large as the student model and Flan-T5-XL as the teacher model.
student. Hence, the personalized explanations are probably easier to reason over for a comparatively weaker student, LLaMA-7B, leading to better performance.
G RQ4: Additional Results
Table 14 shows RQ4 results on StrategyQA with LLaMA-7B as the student and LLaMA-65B as the teacher.
H RQ5: Additional Details and Results
If teacher LLMs can successfully build mental models of student LLMs, a natural follow-up question regards whether communicating misleading explanations can also weaken student models. We verify that this holds, as detailed below.
Study Design. This RQ explores the negative implications of both RQ2 (i.e., when to intervene) and RQ3 (i.e., how to generate teacher explanations), now with the goal of deceiving the student. First, extending our Expected Utility-based Intervention Function (RQ2), we rank samples in increasing order of expected utility, such that the teacher intervenes when the utility is the lowest. Here, the teacher's goal is to communicate explanations for data points where the student gives an incorrect answer by following the teacher's explanation but would have answered correctly had it leveraged its own explanation. We compare this with random teacher intervention (which is generally helpful). Next, in order to explore the negative implication of RQ3, we make the teacher condition on incorrect answers and non-factual human explanations that we manually generate by perturbing (correct) human explanations. We manually make minimal edits to the explanations such that the CoT reasoning is plausible yet non-factual (see some examples in Fig. 11). Now, the teacher's goal is to learn from non-factual explanations and generate similar explanations that purposefully mislead the student. We compare this misaligned teacher with an unpersonalized teacher that learns from factual gold explanations (i.e., the baseline from RQ3).
Main Results. We show the results on StrategyQA in Table 15 and Table 16 with Flan-T5-Large as the student and Flan-T5-XL as the teacher. Ranking data points by negative expected utility allows the teacher to reduce student accuracy to  $53\%$  at  $20\%$  intervention and as low as random chance at  $60\%$  intervention (which is 8 points worse than the student performance with no intervention). Next, Fig. 8(b) illustrates that the teacher can condition on non-factual explanations to successfully generate worse explanations that reduce the student accuracy by at least 2 points, relative to the improvement of benign explanations. Interestingly, many of the explanations are still helpful on an absolute scale, though more than half of the deceiving explanations flip the student prediction to be incorrect (based
on the trend declining earlier than the standard CoT explanations). Therefore, our conclusion is that teacher LLM explanations can be influential for student LLMs in both good and bad ways.
Example 1
Q: Would someone with back pain enjoy picking strawberries?
Correct Explanation: Back pain may be worsened by repeated bending at the waist. Strawberries grow very close to the ground.
Correct Answer: no
Deceiving Explanation: Back pain may be worsened by repeated bending at the waist. Strawberries grow high up on the tree.
Example 2
Q: Could a snowy owl survive in the Sonoran?
Correct Explanation: The Snowy owl is a bird native to the Arctic regions of North America. Temperatures in the North American Arctic range from -36.4F to 50F. The Sonoran is one of the hottest deserts in the world. The Sonoran Desert can get as hot as 118F.
Correct Answer: no
Deceiving Explanation: The Snowy owl is a bird native to the Arctic regions of North America. Temperatures in the North American Arctic range from -36.4F to 50F. The Sonoran is one of the most snowy mountains in the world.
Example 3
Q: Has Drew Carey outshined Doug Davidson's tenure?
Correct Explanation: Drew Carey has been the host of the Price is Right for over 13 years. Doug Davidson hosted the Price is Right from 1994-1995.
Correct Answer: yes
Deceiving Explanation: Drew Carey has been the host of the Price is Right for over 13 years. Doug Davidson is hosting the Price is Right from 1994 until now.
Figure 11: Examples of manually constructed deceiving explanations for StrategyQA that support the wrong answer. The deceiving part of the explanations is highlighted in red.