SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Can Language Models Teach Weaker Agents? Teacher Explanations Improve Students via Personalization
Swarnadeep Saha Peter Hase Mohit Bansal
{swarna, peter, mbansal}@cs.unc.edu
A hallmark property of explainable AI models is the ability to teach other agents, communicating knowledge of how to perform a task. While Large Language Models (LLMs) perform complex reasoning by generating explanations for their predictions, it is unclear whether they also make good teachers for weaker agents. To address this, we consider a student-teacher framework between two LLM agents and study if, when, and how the teacher should intervene with natural language explanations to improve the student's performance. Since communication is expensive, we define a budget such that the teacher only communicates explanations for a fraction of the data, after which the student should perform well on its own. We decompose the teaching problem along four axes: (1) if teacher's test time intervention improve student predictions, (2) when it is worth explaining a data point, (3) how the teacher should personalize explanations to better teach the student, and (4) if teacher explanations also improve student performance on future unexplained data. We first show that teacher LLMs can indeed intervene on student reasoning to improve their performance. Next, inspired by the Theory of Mind abilities of effective teachers, we propose building two few-shot mental models of the student. The first model defines an Intervention Function that simulates the utility of an intervention, allowing the teacher to intervene when this utility is the highest and improving student performance at lower budgets. The second model enables the teacher to personalize explanations for a particular student and outperform unpersonalized teachers. We also demonstrate that in multi-turn interactions, teacher explanations generalize and learning from explained data improves student performance on future unexplained data. Finally, we also verify that misaligned teachers can lower student performance to random chance by intentionally misleading them.<sup>1</sup>
Teaching, or the ability to provide needed information in a way that is understood by others, is often considered an important property of Explainable AI [1]. When AI models "teach" by providing meaningful and interpretable explanations, it fosters transparency, warranted trust, and the ability for humans to make informed decisions based on AI recommendations. One way the goodness of an explanation can be judged is by its ability to communicate knowledge of how to solve a problem to other agents [2, 3]. Explanations fulfill this purpose not only by being informative but also by means of filling in specific gaps in the recipient agent's knowledge. This is enabled by the explainer having theory of mind (ToM), understanding what the recipient does not know [2], and being able to personalize its explanations based on the recipient's needs. Recent work has argued that LLMs like GPT-3.5 now exhibit ToM, based on their ability to answer questions about mental states of
$^{1}$ Code for all experiments: https://github.com/swarnaHub/ExplanationIntervention
Figure 1: Overview of single-round of interaction between a teacher LLM and a student LLM, covering our first three research questions (with Fig. 6 showing RQ4, multi-round student-teacher interaction that builds on top of RQ1-RQ3).  $RQ1$ : The teacher randomly intervenes and communicates explanations to improve the student's performance (right part).  $RQ2$ : The teacher decides when to intervene by computing Expected Utility of Intervention using the Pre-Intervention and Post-Intervention Simulation prompts (middle part).  $RQ3$ : The teacher communicates personalized explanations that are more helpful for the student (left part).
hypothetical people in classical theory-of-mind tests [4].<sup>2</sup> However, we do not yet know how well language models can teach other agents to solve reasoning tasks via explanations.
In this work, we are motivated by this essential goal of evaluating explanations (specifically, Chain-of-Thought [7]) rationales) from the perspective of teaching and improving weaker agents in solving reasoning tasks. In order to improve smaller models' reasoning skills, recent works propose knowledge distillation by fine-tuning a smaller model on the reasoning steps generated by a larger model [8, 9, 10]. Yet, an important component of human teaching is understanding when and how the teacher should explain particular things to the student. Current distillation approaches do not evaluate a teacher's ability to identify when a student lacks understanding, and past work has not explored how to personalize teacher explanations to the student's needs. A smaller student model might already be good at answering certain questions but might require the teacher's intervention for some harder questions. When there are many things to teach the student and teaching is laborious, it is important to choose which problems merit explanation in order to improve teaching efficiency [11]. Moreover, for more effective teaching, it is desirable to have the teacher personalize its explanations to help a particular student, and a teacher that lacks understanding of the student's needs (i.e., lacks Theory of Mind) will be unable to do this [3].
Motivated by the efficiency of human explanations, we consider a student-teacher framework where a teacher model guides the reasoning of a student model, with the goal of improving the student's reasoning on current and future data. In order to do so, we explore a Theory of Mind-inspired approach, where the teacher simulates the student's behavior by building a mental model of the student. Our overall research question investigates whether the teacher's intervention (in the form of natural language explanations) can enable the student to make more accurate predictions both on explained as well as unexplained future data. However, communication is expensive, and therefore we assume that a cost is incurred each time the teacher intervenes with (communicates) an explanation to the student for a particular data point. We refer to this as the intervention budget, the percentage of test data points the teacher intervenes on. In order to comprehensively answer our overall research question, we further decompose the teaching problem into the following constituent questions:
1. RQ1. Can a teacher LLM intervene at test time to improve a student LLM's predictions?
There have also been strong arguments against the presence of ToM in current LLMs [5, 6]. In this paper, we do not thoroughly test for ToM abilities in LLMs, but instead we focus on measuring teaching performance in terms of a teacher's positive effect on student performance.
2. RQ2. Given a fixed intervention budget, when should the teacher intervene (i.e., on which data points), in order to maximize student performance?
3. RQ3. Given a set of intervention data, can a teacher model personalize its explanations for a student model to improve student performance?
4. RQ4. In multi-turn interactions, do teacher explanations generalize and improve student performance across data points (beyond the explained samples)?
5. RQ5. Can misaligned teacher LLMs lower student performance by providing misleading explanations to the student?
We answer RQ1 by assuming that the teacher intervenes on random data points in four different settings: using a human or LLM teacher, and, when the teacher is an LLM, using an LLM student that is either weaker or stronger than its teacher (§5.1). Across three different reasoning tasks (StrategyQA, GSM8k, and CommonsenseQA) and two different model families (Flan-T5 and LLaMA), we observe that (1) teacher LLMs can effectively intervene on student reasoning, improving student performance on the end task, and (2) more intervention typically leads to a monotonic increase in student performance, though model teachers are not as good as human teachers. Fig. 1 shows the intervention process and the two student prompts (in the right part of Fig. 1) that are used to generate predictions.
RQ2 explores how to intelligently select which data points to explain for the student model, in order to improve teaching efficiency (§5.2). Past work in cognitive science also considers teaching efficiency in young children by deciding what to teach by maximizing the learner's expected utility of learning [12]. With a similar motivation, we develop an Intervention Function that is inspired from the principle of a teacher having a Theory of Mind. In particular, the teacher builds a mental model of the student's reasoning process, with the goal of intervening only on samples that are most likely to maximize student performance. Our Intervention Function is based on Expected Utility, in which the teacher first estimates the utility of an intervention by simulating the student's prediction pre-intervention (without intervention) and post-intervention (with intervention), then constructs a rank ordering of the samples according to this utility (see the middle part of Fig. 1). The teacher builds this mental model in a few-shot manner, only assuming access to the student's predictions pre- and post-intervention for a few samples. We demonstrate that our proposed Intervention Function based on Expected Utility (1) outperforms other baseline Intervention Functions, (2) improves student performance when the teacher is not  $100\%$  accurate, and (3) enables weaker LLMs to teach stronger ones, unlike with random intervention in RQ1.
Next, in RQ3, we explore how the teacher should explain data points to a particular student model, including how the teacher can personalize explanations for a student model (\$5.3). That is, after deciding which data points to intervene on (RQ2), we decide how the teacher should explain those data points. A clear limitation of the teacher just generating explanations as if it is solving the task is that the explanations are not at all personalized for the student. Given that good explanations are designed to fill in gaps in student knowledge [2], we believe that equipping the teacher with basic personalization skills will improve its teaching ability. With this motivation, we propose another few-shot mental model for the teacher that encourages it to tailor its explanations to be helpful for the particular student model it is teaching. The teacher builds this model by conditioning on a few demonstrations of 'useful' human explanations that rectify a student's answer, thereby encouraging explanations that are more likely to help the student (see Fig 1 for an example of the teacher's personalization prompt). We demonstrate this prompt's effectiveness against unpersonalized explanations that are generated by prompting the teacher with random human explanations, showing that LLMs can personalize their explanations.
RQ4 tests whether LLMs can teach student models to generalize to new unexplained examples (§5.4), rather than improve their reasoning at test-time (RQ1-RQ3). In other words, we now explore the ability of LLMs to teach using the teaching components introduced in RQ2 and RQ3 of when and how to explain samples. This leads us to explore a multi-round interactive setting, where each round consists of the teacher selecting a set of best points to explain (according to RQ2) and generating explanations for them (according to RQ3). The student then conditions on these teacher explanations as in-context demonstrations to perform the reasoning task on future unexplained samples. We demonstrate that teacher explanations indeed generalize and improve student performance on unexplained data.
Finally, in RQ5, we investigate the negative implications of teacher explanations on student LLMs (§5.5). Given that LLMs can improve student agents, we also want to test whether they can lower student performance. If a misaligned teacher provides non-factual explanations in scenarios where the
student directly adopts them, does that lead to a drop in student performance? In fact, we show that teacher models can lower student performance to random chance by intervening on data points with the intent of misleading the student. This has potential implications for LLMs giving explanations in a context where other agents adopt them with unwarranted trust in their correctness.
In summary, our comprehensive studies highlight the ability of LLMs to teach and improve weaker LLMs, demonstrated via improvements on explained test examples as well as future unexplained data. Broadly, equipping LLMs with an ability to effectively and efficiently teach, opens the door to (1) using LLMs as personalized tutors for humans (where efficiency is critical), (2) distilling knowledge into weaker or more compute-efficient student models, and (3) improving human decision making via AI recommendations and explanations.
Evaluating Explanations in Teacher-Student Games. Several past works evaluate explanations in the context of a student-teacher communication game [13, 14, 15, 16, 17]. The teacher communicates explanations to the student with one of two objectives: (1) evaluating whether explanations help students to simulate the teacher better, or (2) whether explanations can directly teach students to better perform a task. Simulatability, or the student's ability to simulate the teacher's own answer, is seen as a measure of the explanation's faithfulness, rather than a direct measure of whether explanations help students learn the task itself [18, 19]. Our work is focused on the second research goal of evaluating explanations from the perspective of teaching weaker agents. Prior work has shown that human explanations can teach LLMs [7, 20] and LLMs can also teach themselves from their own explanations or feedback [21, 22]. But it remains to be shown whether LLMs can also teach weaker agents. A few recent works also share a similar goal as ours and they distill knowledge [23] directly into the student model by finetuning it on the explanations from the teacher model [8, 9, 10, 24]. However, these distillation methods do not consider the important aspects of communication cost between two agents, its trade-off with student performance, and how the teacher may build mental models of the student to decide when and how to communicate explanations to the student. Recent studies have also evaluated explanations in the context of human-AI collaboration, for their plausibility [25, 26], usefulness to human learning [27, 28], and for improving human-AI team performance [29, 30]. Different from these, we analyze model-model interactions, with the goal of understanding how effectively LLMs can teach weaker systems to solve a task.
Theory of Mind in AI. A body of work demonstrates that humans regularly infer and make decisions based on the mental states of other agents, also known as Theory of Mind (ToM) [31, 3, 2, 32]. This has motivated works on computational language acquisition using ToM [33, 34, 35]. There have been recent works arguing both for and against the presence of Theory of Mind in Large Language Models [36, 4, 6, 37]. Theory of Mind has been successfully applied to improve human-AI collaboration in robotics [38, 39, 11]. In this work, we design prompts that are motivated by a teacher having a Theory of Mind to efficiently intervene and improve a student's reasoning capabilities.
Student and Teacher. We assume a two-agent communication game between a student  $S$  and a teacher  $\mathcal{T}$ , where the goal is to teach the student to solve a particular task interactively. Here, we use an LLM as the student. To explore a range of student and teacher capabilities, we consider both human and model teachers, while typically using a student model that is measurably weaker than its teacher. Following past work, an LLM with more parameters is considered a stronger model due to its better performance across a range of tasks (including the ones we consider in our studies). In the scope of our study, when the teacher is not a human, both  $S$  and  $\mathcal{T}$  are LLMs, prompted with a set of demonstrations of the task  $\mathcal{D}$  (typically, 4-8 examples). Each demonstration  $d^{(i)} \in \mathcal{D}$  is a triple  $(x^{(i)}, y^{(i)}, e_{\mathcal{H}}^{(i)})$  consisting of an input  $x^{(i)}$ , the output  $y^{(i)}$ , and a human-written explanation  $e_{\mathcal{H}}^{(i)}$  that answers the question of why the data point has the output it has [1]. In our tasks, the explanation may include background knowledge or intermediate reasoning steps that are helpful for obtaining the answer. By organizing  $\mathcal{D}$  into Chain-of-Thought prompts, both  $S$  and  $\mathcal{T}$  are equipped with the ability to generate explanations and predict labels for new samples.
Single-Round Intervention. The first problem setup we consider involves the teacher deciding whether or not to intervene for a single test problem. In this setting, the student's goal is to answer
the problem correctly, and the teacher can choose to intervene for individual problems to assist the student. Thus, given a test data point  $t^{(i)}$ , we have the following two scenarios:
- No Intervention: When the teacher chooses not to intervene, the student generates both the explanation  $e_{S}^{(i)}$  and the prediction  $\hat{y}_{S}^{(i)}$ , by conditioning on the  $\mathcal{D}$  task demonstrations and the test input  $t^{(i)}$ . This is done using Chain-of-Thought prompting [7].
- Intervention: When the teacher does choose to intervene, it communicates its generated explanation to the student. Here the student only generates a prediction  $\hat{y}_S^{(i)}$  by conditioning on the  $\mathcal{D}$  task demonstrations, the test input  $t^{(i)}$ , and the corresponding teacher explanation  $e_{\mathcal{T}}^{(i)}$ . For the tasks and datasets we consider, explanations provide helpful background information or reasoning steps but do not directly reveal the test label, so it never directly gives away the answer.
Fig. 1 shows the 'No Intervention' and 'With Intervention' student prompts and the overall intervention process. Note that in terms of the prompts used for both these scenarios, the only difference is in the source of explanation (student vs teacher) for the test point. When the teacher is a human, intervention happens with a human-written explanation (crowdsourced in the datasets we rely on).
Communication Cost. In Single-Round Intervention, the teacher could maximize student performance by simply always intervening on the student's reasoning. We bring in a natural constraint from Rational Speech Acts theory, i.e., communication is costly and should only be undertaken if it furthers a communicative goal [40]. Hence, we assume that a communication cost is incurred each time the teacher intervenes with an explanation to the student. We also note that this cost is only with respect to an agent's communication (and assume that the teacher can always generate explanations for itself). Unless otherwise stated, communication happens one-way from the teacher to the student in the form of explanations. We set a limit to the number of points that the teacher can intervene on, referred to as the intervention budget, and we assume the cost to be uniform for all data points. Across all our experiments, we vary the intervention budget between  $\{0\%, 20\%, 40\%, 60\%, 80\%, 100\}$ . A budget of  $0\%$  means the student generates its own explanation as it predicts each data point, while a budget of  $100\%$  means the student leverages the teacher's explanation for every data point. Later, in Sec. 5.2, we introduce the teacher Intervention Function, which the teacher uses to decide which points to intervene on given its fixed intervention budget.
Multi-round Intervention. Here, the goal of the teacher is to provide explanations to the student that help it generalize across samples, rather than leading the student to the correct answer only for the explained data points. Thus, we allow the teacher to explain data points that are then added to the student model's prompt, but we forbid the teacher from intervening on future test points. If the teacher can improve the quality of the student model's prompt, student performance on the test data should improve. In our experiments, this process occurs in a few steps: (1) the teacher picks points to explain, (2) the teacher explains those points, (3) the points are added to the student prompt with the teacher's explanations, and then (4) the student predicts labels for the remaining test data. Further implementation details are given in Sec. 5.4.
We consider complex natural language reasoning tasks, motivated by two specific needs. First, the task should be hard enough for current LLMs that explanations can provide useful knowledge to the models. Second, it allows us to study free-text rationales, which are more flexible than input attribution methods [41, 42, 43] and benefit many reasoning tasks [7, 44]. We experiment with three reasoning tasks: (1) StrategyQA [45], (2) GSM8k [46], and (3) CommonsenseQA [47] (details in Appendix B). Both StrategyQA and CommonsenseQA contain explanations in the form of relevant facts, thus requiring the student to reason over them to produce an answer. We also manually verify (up to 100 samples) that the explanations do not explicitly leak the answer. For GSM8k, since the reasoning steps explicitly derive the answer, providing the entire explanation during intervention will lead to answer leakage [17]. Hence, the teacher communicates a partial explanation (specifically, only the first step of the rationale), allowing the student to leverage it as a hint to derive the final answer. We experiment with two state-of-the-art open-source LLMs of varying sizes, ranging from 780M to 65B parameters. Specifically, we use two encoder-decoder and decoder-only models as student and teacher: (1) Flan-T5-Large and Flan-T5-XL [48], and (2) LLaMA-7B, LLaMA-13B, and LLaMA-65B [49]. Refer to Appendix A for more details of student and teacher models.
5 Experiment Results
5.1 RQ1: Can a teacher LLM intervene at test time to improve a student LLM's predictions?
Our first research question asks if LLMs can improve students by intervening on their reasoning at test time. While the main goal is to analyze the behavior of model-based teachers, we also experiment with human teachers to establish a ceiling on the capabilities of an LLM teacher. These human teachers are people who authored the (human) explanations in the datasets we experiment with and were crowdsourced in prior works.
Study Design. We compare the accuracy obtained by the student model at different intervention budgets. For the purpose of this study, the intervention happens at random data points while we vary the student and teacher. In particular, we compare four intervention setups: (1) a human teacher paired with a smaller student model, (2) a human teacher paired with a larger student model, (3) a larger teacher model paired with a smaller student model, and (4) a smaller teacher model paired with a larger student model. For the main experiments, the student and the teacher are chosen from the same model family.
Main Results. Figure 2 shows the results on StrategyQA with Flan-T5 models. A human teacher's intervention on the explanations of both smaller and larger Flan-T5 models exhibits a monotonically increasing accuracy trend. Larger model teachers can also improve smaller student models. Flan-T5-Large obtains an accuracy of  $58\%$  when always utilizing its own explanations but obtains up to  $63\%$  accuracy when reasoning with the larger Flan-T5-XL's explanations. Intuitively, a larger student model does not benefit from a smaller teacher model's explanations, as we observe a monotonically decreasing trend. Our results generalize to other models (LLaMA), datasets (CommonsenseQA, GSM8k) and even when the student and the teacher belong to different model families. In fact, when the teacher (LLaMA-65B) is much stronger than the student (LLaMA-7B), the margin of improvement is also
Figure 2: RQ1: Comparison of random intervention by different Teacher Models on different Flan-T5 Student Models at different intervention budgets for StrategyQA. A + B = A student, B teacher.