SlowGuess/NeurIPS_2023_OCR huggingface dataset:
higher, about  $8\%$  (statistically significant with  $p = 0.01$ ). See Appendix D for these additional RQ1 results. In summary, we conclude that: for complex reasoning, LLMs can indeed effectively intervene and improve weaker models, and more intervention typically leads to better performance, although humans explanations improve more.
5.2 RQ2: Given a fixed intervention budget, when should the teacher intervene (i.e., on which data points), in order to maximize student performance?
So far, we have demonstrated that random teacher intervention benefits student models. But a good teacher does not randomly pick problems to help a student with. Each intervention also has an associated communication cost and hence, it is desirable to be able to improve student performance while reducing the cost. In this research question, we investigate better strategies for choosing data points to intervene on. We call these strategies Intervention Functions that produce a rank ordering of the samples, and, given a fixed budget, the teacher intervenes on the highest-ranked samples.
An intervention is useful if the student's confidence in the gold answer increases with intervention (i.e., with teacher's explanation) compared to without it (i.e., with its own explanation). Here confidence is simply the likelihood that the model assigns to the correct answer i.e., we take the logits from the last layer of the model and normalize them to get the correct answer's probability. Computing expected utility, however, depends on two quantities: (1) the student's true confidence measures with and without intervention, and (2) the gold answers against which the confidence is computed. It also incurs a two-way communication cost, one for the teacher to communicate its explanation to the student and another for the student to communicate back its confidence to the teacher. Thus, we propose an Intervention Function based on the Expected Utility of intervention, which relies on estimates of student confidence, and we consider two setups depending on whether the teacher knows the gold label. Ideally, a teacher is expected to be an expert in the concerned task (e.g., if the teacher
(b)
Figure 3: RQ2: (a) Comparison of different Intervention Functions on StrategyQA with a smaller student (Flan-T5-Large) and a larger teacher (Flan-T5-XL). (b) Ablation of Expected Utility.
is a human or a powerful model that obtains high accuracy). When the teacher does not have access to gold answers, we treat the teacher's answers as gold answers when computing Expected Utility. Expected Utility Intervention Function. The teacher computes the Expected Utility of intervention by simulating the student's predictions using a mental model of the student. In order to build this mental model, we assume that the teacher has observed the student on a few samples and has access to  $d$  demonstrations  $\mathcal{D}_{sim}$  of the student's predictions with and without intervention, denoted as:
$$
\mathcal {D} _ {s i m} = \{x ^ {(i)}, y ^ {(i)}, e _ {\mathcal {S}} ^ {(i)}, e _ {\mathcal {T}} ^ {(i)}, \hat {y} _ {p r e} ^ {(i)}, \hat {y} _ {p o s t} ^ {(i)} \} _ {i = 1} ^ {d}
$$
where  $x^{(i)}$  and  $y^{(i)}$  are the input and output respectively;  $e_{\mathcal{T}}^{(i)}$  and  $e_{\mathcal{S}}^{(i)}$  denote the student and teacher explanations respectively; and  $\hat{y}_{pre}^{(i)}$  and  $\hat{y}_{post}^{(i)}$  refer to the student predictions with student explanation (pre-intervention) and teacher explanation (post-intervention) respectively. Using these demonstrations, the teacher builds a few-shot mental model of the student and predicts two quantities for a given test question - (1) Pre-intervention Expected Student Confidence ( $\hat{c}_{pre}$ ): The teacher conditions on the pre-intervention demonstrations  $\mathcal{D}_{sim}^{pre} = \{x^{(i)}, y^{(i)}, e_{\mathcal{S}}^{(i)}, \hat{y}_{pre}^{(i)}\}_{i=1}^{d}$  to simulate the student's confidence on the gold answer, had it been using its own (student) explanation, and (2) Post-intervention Expected Student Confidence ( $\hat{c}_{post}$ ): The teacher conditions on the post-intervention demonstrations  $\mathcal{D}_{sim}^{post} = \{x^{(i)}, y^{(i)}, e_{\mathcal{T}}^{(i)}, \hat{y}_{post}^{(i)}\}_{i=1}^{d}$  to estimate what the student's confidence would be if it had used the teacher's explanation. The teacher computes these confidence estimates as if it were the student (refer to Fig. 1 for the prompts), essentially learning to simulate the student by conditioning on the appropriate demonstrations and then generating an answer to the question. Then the Expected Utility  $\hat{U} = (\hat{c}_{post} - \hat{c}_{pre})$  is given by the difference between the two confidence measures. The teacher finally constructs a rank ordering of the test data points based on this expected utility. This utility-based ordering encourages the teacher to pick points where it thinks the student will answer correctly with intervention but incorrectly without.
Other Intervention Functions. To analyze how well our proposed Intervention Function performs, we compare it with the following Intervention Functions. Our first baseline is the Random Intervention Function from RQ1. Next, we compare with an Intervention Function that ranks the samples based on the Teacher Confidence – when the teacher is most confident about a question, it intervenes. Our next two baselines are ablations of Expected Utility: (a) Pre-Intervention Expected Student Confidence – We rank samples based on the expected student confidence with no intervention (i.e., lower this confidence, the higher the likelihood of useful interventions), and (b) Post-Intervention Expected Student Confidence: We rank samples based on the expected student confidence with intervention (i.e., higher this confidence, higher is the likelihood of useful interventions. Finally, as upper bounds of Intervention Functions, we assume that the student communicates its true confidence values to the teacher (which for post-intervention, incurs a both-way communication cost of the teacher sending its explanation, followed by receiving the student's confidence). Using the true confidence measures, we compute True Utility.
Main Results: How does Expected Utility compare to True Utility? Figure 3 compares different Intervention Functions with Flan-T5-XL as the teacher and Flan-T5-Large as the student on StrategyQA. Across different methods, we analyze accuracy obtained at lower communication costs (e.g.,
20%) as well as highest accuracy obtained, independent of any budget constraints. Our primary observation from Figure 3(a) is that Expected Utility improves student accuracy by up to 7 points at a low communication cost of 20%. Expected Utility also peaks at an accuracy of 71% with only 40% intervention. Since model-based teachers are not always perfect, increased intervention beyond 60% leads to a drop in student accuracy (e.g., in the last 20% of the intervention, the student accuracy drops from  $69\% \rightarrow 63\%$ ). When the student communicates its confidence scores to the teacher, the teacher is able to compute the true utility of intervention, which unsurprisingly, leads to a much higher accuracy of 76% at 20% cost and an overall high of 81% accuracy. Nevertheless, estimating expected utility is cheaper and our results also suggest that a better mental model could further improve performance. Ranking by teacher confidence is ineffective because it is not an indicator of the student's capabilities. Next, in Figure 3(b), we show that ranking by utility outperforms ranking by either pre or post-intervention confidence scores. In summary, with access to only a few demonstrations of student behavior, a teacher can build an effective mental model of the student and intervene such that the student obtains a much higher accuracy at low communication costs.
When the teacher does not have access to gold answers, can we compute Expected Utility with respect to teacher answers? Teachers can be inaccurate and may not even have access to gold answers. In such scenarios, can we treat the teacher as the gold standard and compute utility with respect to the teacher's answers? We explore this in Figure 4, comparing Expected Utility to Random Intervention and Student Least Confidence. The latter denotes that when the student is least confident about any of the answer options, it is more likely to answer incorrectly and hence will benefit from intervention. We observe that Expected Utility, computed with teacher answers, also leads to up to 2 points improvement in accuracy at  $20\%$  budget, which is also within  $1\%$  of the accuracy  $(63.60\%)$  obtained with  $100\%$  communication cost. In Appendix Table 9, we conduct the same experiment with a much stronger teacher
(LLaMA-65B) and a weaker student (LLaMA-7B) and obtain even stronger evidence of this result. Stronger teachers like LLaMA-65b are significantly better at solving reasoning tasks and thus their predicted labels will mostly match the gold labels. Hence, even if we rely on the teacher's predictions for computing expected utility, it improves student accuracy by up to 5 points (statistically significant with  $p = 0.02$ ), further closing the gap between 'with and without gold label' scenarios. In summary, we conclude that imperfect teacher LLMs can also successfully intervene by building mental models of students that do not rely on ground-truth answers. Appendix E contains additional results for RQ2 with other models and datasets.
Figure 4: RQ2: Comparison of different Intervention Functions on StrategyQA with a Flan-T5-Large student and a Flan-T5-XL teacher, with no access to gold labels.
5.3 RQ3: Given a set of intervention data, can a teacher model personalize its explanations for a student model to improve student performance?
The previous RQ showed how the teacher may build a few-shot mental model of the student to decide when to intervene, given a fixed budget. Upon intervention, the teacher communicates an explanation generated by prompting the model with random human explanations. This leads to an unpersonalized teacher that assumes that the explanation it generates in order to solve the task will be automatically helpful for the student. However, an effective teacher should tailor its explanations to fill in gaps in the student's knowledge [2]. With this motivation, the teacher builds another few-shot mental model of the student, this time generating helpful explanations that are more likely to benefit the student.
Teacher's Explanation Personalization Prompt. Helpful human explanations are those that rectify a student's answer i.e., cause the student's answer to flip from incorrect (when using its own explanation) to correct (when using human explanation). We assume that the teacher has observed the student on  $d$  demonstrations  $\mathcal{D}_P$  of exclusively helpful human explanations, denoted as:  $\mathcal{D}_P = \{x^{(i)},y^{(i)},e_{\mathcal{H}}^{(i)},e_S^{(i)}\}_{i = 1}^d$  where  $e_{\mathcal{H}}^{(i)}$  and  $e_S^{(i)}$  denote (helpful) human and (not helpful) student explanations respectively. The teacher conditions on these demonstrations to generate explanations
for the student. Fig. 1 shows an example of a personalization prompt. With such a prompt, teacher explanations are steered toward only those explanations that help the student.
Baselines. We compare personalized teachers with unpersonalized ones that condition on random human explanations. Appendix F also reports results with unpersonalized rationales, that are post-hoc explanations ('The answer is X because Y') and not Chain-of-Thought ('Y. So the answer is X').
Main Results. Fig. 5 shows the results on StrategyQA with Flan-T5-Large as the student and Flan-T5-XL as the teacher. Both unpersonalized and personalized teachers choose intervention samples based on Expected Utility, as defined in RQ2. We observe that a personalized teacher improves the accuracy further both at lower budgets (by  $2\%$  at  $20\%$  cost) but also overall, obtaining a peak accuracy of  $72.63\%$ . However, unlike the strong supporting evidences we obtain for RQ1 and RQ2, the effect of personalization is comparatively weaker. Hence, we further test this research question with a LLaMA-65B teacher and a LLaMA-7B student in Appendix F. While scaling up the teacher model points to stronger evidence of personalization (e.g.,  $2.4\%$  better student accuracy), the results are still not statistically significant  $(p = 0.09)$ . Hence, we conclude that: personalizing teacher ex
planations can further benefit the students, although our results currently suggest that the effect size may be small. We hope that future work is further able to explore explanation personalization with even stronger teacher models like GPT-4. In Appendix F, we show some comparative instances of unpersonalized and personalized explanations.
Figure 5: RQ3: Comparison of unpersonalized and personalized teacher (Flan-T5-XL) explanations on student (Flan-T5-Large) accuracy for StrategyQA.
5.4 RQ4: In multi-turn interactions, do teacher explanations generalize and improve student performance across data points (beyond the explained samples)?
In the previous RQs, we showed that teacher explanations improve student predictions for the samples that the teacher explains. RQ4 explores whether teacher explanations also generalize to new instances that the teacher has not explained. In other words, this studies if the student can perform Chain-of-Thought reasoning by only conditioning on teacher LLM explanations rather than human's.
Study Design. We consider a multi-turn teaching setup (Fig. 6), in which at each turn the teacher chooses to explain a few samples from a pool of unexplained examples which are then added to the student's prompt. The prompt consists of demonstrations of the teacher's explanations and predictions. The student then conditions only on these in-context examples (without any human demonstrations) to generate predictions for the test samples (where there is no teacher intervention). For choosing the data points to explain at each round, we use the Expected Utility Intervention Function (from RQ2), and for generating the teacher
Figure 6: Overview of multi-turn student-teacher interaction showing  $RQ4$ . At each turn, the teacher chooses some samples to explain and the student conditions on them to make predictions on future unexplained data.
explanations, we leverage the ToM prompt (from RQ3). We say that teacher explanations generalize if conditioning on demonstrations of explained points improves upon demonstrations with no explanations (i.e., only QA pairs) or self-explanations (i.e., demonstrations with student explanations and predictions). We consider five rounds in total with LLaMA-7B as the student and LLaMA-65B as the teacher, adding two explained samples in each round. We compare the student accuracy after each round with teacher-explained, student-explained, and unexplained demonstrations.
Main Results. Fig 7 shows the results. We observe that teacher explanations improve student performance on future unexplained test points as well by a significant 6 points  $(55\% \rightarrow 61.6\%)$ .
While student accuracy with unexplained samples improves after adding more demonstrations (55% → 59%), accuracy with teacher-explained demonstrations is better by up to 4 points (p = 0.04) after each round. Interestingly, when the student conditions on self-explained demonstrations, its performance decreases with increasing rounds. We hypothesize that this is because the student might be overfitting to its own worse explanations, leading to bad predictions. In summary, we conclude that: teacher LLMs can teach student models to perform well on their own when given new test data. LLMs with even longer context windows will allow adding more explained samples in each round and repeating the teaching process for more rounds. We hope that our initial promising results
Figure 8: RQ5: (a) Negative Implication of RQ2: Comparison of intervention based on negative expected utility with random intervention on StrategyQA. (b) Negative Implication of RQ3: Comparison of an unpersonalized teacher (generating explanations conditioned on random human explanations) versus a deceiving teacher (generating explanations conditioned on wrong explanations).
Figure 7: RQ4: Multi-round student-teacher interaction comparing student accuracy on unexplained test points with unexplained, student-explained and teacher-explained demonstrations.
will encourage more exploration in multi-round teaching with LLMs.
5.5 RQ5: Can misaligned teacher LLMs lower student performance by providing misleading explanations to the student?
If teacher LLMs can successfully build mental models of student LLMs, a natural follow-up question regards whether communicating misleading explanations can also weaken student models. Here we briefly describe our study design and findings, with more details in Appendix H. First, the teacher intervenes in increasing order of expected utility, prioritizing samples where the utility is lowest. Second, we make the teacher condition on incorrect answers and non-factual human explanations that we manually generate by perturbing (correct) human explanations. We show the results in Fig. 8(a) and Fig. 8(b). Ranking data points by negative expected utility allows the teacher to reduce student accuracy to random chance at  $60\%$  intervention. Next, Fig. 8(b) illustrates that the teacher can condition on non-factual explanations to successfully generate worse explanations that reduce the student accuracy by 2 points, relative to the improvement of benign explanations. Thus, we conclude that teacher LLM explanations can be influential for student LLMs in both good and bad ways.
We demonstrated that LLMs can teach weaker student models to improve their performance on reasoning tasks for both explained and unexplained future data. The teacher builds two few-shot mental models of the student, one predicting which data points to intervene on and another generating personalized explanations that the student can efficiently learn from.
Limitations & Broader Impacts. See Appendix for limitations and broader impacts discussion.
We thank Shiyue Zhang and Archiki Prasad for useful feedback, as well as Stephen Casper and Miles Turpin for suggestions regarding experiments. This work was supported by NSF-CAREER Award 1846185, NSF-AI Engage Institute DRL-2112635, DARPA MCS Grant N66001-19-2-4031, and Google PhD Fellowships. The views contained in this article are those of the authors and not of the funding agency.
[1] Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence, 267:1-38, 2019. URL https://arxiv.org/abs/1706.07269.
[2] Henry M Wellman and Kristin H Lagattuta. Theory of mind for learning and teaching: The nature and role of explanation. Cognitive development, 19(4):479-497, 2004. URL https://www.sciencedirect.com/science/article/abs/pii/S0885201404000620.
[3] Stephanie M Carlson, Melissa A Koenig, and Madeline B Harms. Theory of mind. Wiley Interdisciplinary Reviews: Cognitive Science, 4(4):391-402, 2013. URL https://sites.socsci.uci.edu/~lpearl/courses/readings/CarlsonKoenigHarms2013_theoryOfMind.pdf.
[4] Michal Kosinski. Theory of mind may have spontaneously emerged in large language models. arXiv preprint arXiv:2302.02083, 2023. URL https://arxiv.org/abs/2302.02083.
[5] Tomer Ullman. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399, 2023. URL https://arxiv.org/abs/2302.08399.
[6] Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. Clever hans or neural theory of mind? stress testing social reasoning in large language models. arXiv preprint arXiv:2305.14763, 2023.
[7] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022. URL https://arxiv.org/abs/2201.11903.
[8] Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. arXiv preprint arXiv:2212.08410, 2022. URL https://arxiv.org/abs/2212.08410.
[9] Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. arXiv preprint arXiv:2301.12726, 2023. URL https://arxiv.org/abs/2301.12726.
[10] Namgyu Ho, Laura Schmid, and Se-Young Yun. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071, 2022. URL https://arxiv.org/abs/2212.10071.
[11] Xiaofeng Gao, Ran Gong, Yizhou Zhao, Shu Wang, Tianmin Shu, and Song-Chun Zhu. Joint mind modeling for explanation generation in complex human-robot collaborative tasks. In 2020 29th IEEE international conference on robot and human interactive communication (RO-MAN), pages 1119–1126. IEEE, 2020. URL https://arxiv.org/abs/2007.12803.
[12] Sophie Bridgers, Julian Jara-Ettinger, and Hyowon Gweon. Young children consider the expected utility of others' learning to decide what to teach. Nature human behaviour, 4(2):144-152, 2020. URL https://www.nature.com/articles/s41562-019-0748-6.
[13] Arjun Chandrasekaran, Viraj Prabhu, Deshraj Yadav, Prithvijit Chattopadhyay, and Devi Parikh. Do explanations make VQA models more predictable to a human? In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1036-1042, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1128. URL https://aclanthology.org/D18-1128.
We hope that our findings can help improve the understanding and evaluation of Chain-of-Thought rationales, in order to better understand the behavior of LLMs and make them more interpretable. Through the first four research questions, we demonstrate that teacher LLMs can successfully build mental models of weaker agents to improve their performance. Modern LLMs like GPT-4 may generate non-factual explanations [50] that have the potential to inadvertently harm weaker agents, especially in a context where other agents adopt them with unwarranted trust in their correctness. We verify this through our final research question. In general, we do not foresee specific ethical risks arising from this work that do not already apply to the general use of Large Language Models, such as the potential to generate harmful or toxic content [51].
While teacher LLMs generate better explanations via personalization, the human explanations are unpersonalized i.e., collected without any particular student in mind. In spite of that, we observe that intervention with human explanations proves to be helpful in most cases. It remains to be seen whether human explanations that are directed toward improving a particular student model can lead to further improvements. Next, we make a simplifying assumption that the communication cost is uniform across all samples. Non-uniform costs (e.g., measured based on the number of tokens or reasoning steps) such that longer explanations incur larger costs is an interesting direction for future work. We also note that while both student and teacher generate explanations with the goal of improving student predictions, the predictions may still be unfaithful to the reasoning steps.
A Student and Teacher Models