Can Language Models Teach? Teacher Explanations Improve Student Performance via Personalization
Authors: Swarnadeep Saha
Peter Hase
Mohit Bansal
Venue: NeurIPS 2023 poster (2023)
Keywords: Language Models
Reasoning
Explanations
TL;DR: Leveraging Theory of Mind to improve student performance on reasoning tasks via teacher explanations

Abstract: A hallmark property of explainable AI models is the ability to teach other agents, communicating knowledge of how to perform a task. While Large Language Models (LLMs) perform complex reasoning by generating explanations for their predictions, it is unclear whether they also make good teachers for weaker agents. To address this, we consider a student-teacher framework between two LLM agents and study if, when, and how the teacher should intervene with natural language explanations to improve the student’s performance. Since communication is expensive, we define a budget such that the teacher only communicates explanations for a fraction of the data, after which the student should perform well on its own. We decompose the teaching problem along four axes: (1) if teacher’s test time in- tervention improve student predictions, (2) when it is worth explaining a data point, (3) how the teacher should personalize explanations to better teach the student, and (4) if teacher explanations also improve student performance on future unexplained data. We first show that teacher LLMs can indeed intervene on student reasoning to improve their performance. Next, inspired by the Theory of Mind abilities of effective teachers, we propose building two few-shot mental models of the student. The first model defines an Intervention Function that simulates the utility of an intervention, allowing the teacher to intervene when this utility is the highest and improving student performance at lower budgets. The second model enables the teacher to personalize explanations for a particular student and outperform unpersonalized teachers. We also demonstrate that in multi-turn interactions, teacher explanations generalize and learning from explained data improves student performance on future unexplained data. Finally, we also verify that misaligned teachers can lower student performance to random chance by intentionally misleading them.

Can Language Models Teach Weaker Agents? Teacher Explanations Improve Students via Personalization
Swarnadeep Saha
Peter Hase Department of Computer Science
Mohit Bansal
University of North Carolina Chapel Hill (warrn,, peter, mbansal ) :oss. enc.edu
Abstract
A hallmark property of explainable Al models is the ability to teach other agents, communicating knowledge of how to perform a task. While Large Language Models (LLMs) perform complex reasoning by generating explanations for their predictions, it unclear whether they also make good teachers for weaker agents. To address this, we consider student-teacher framework between two LLM agents and study if, when, and how the teacher should intervene with natural language explanations to improve the student's performance. Since communication is ex- pensive, we define a budget such that the teacher only communicates explanations for fraction of the data, after which the student should perform well on its own. We decompose the teaching problem along four axes: (1) if teacher's 's test time in- tervention improve student predictions, (2) when it is worth explaining data point, (3) how the teacher should personalize explanations to better teach the student, and (4) if teacher explanations also improve student performance on future unexplained data. We first show that teacher LLMs can indeed intervene on student reasoning to improve their performance. Next, inspired by the Theory of Mind abilities of effec- tive teachers, we propose building two few-shot mental models of the student. The first model defines an Intervention Function that simulates the utility of an interven- tion, allowing the teacher to intervene when this utility is the highest and improving student performance at lower budgets. The second model enables the teacher personalize explanations for particular student and outperform unpersonalized teachers. We also demonstrate that multi-turn interactions, teacher explanations generalize and learning from explained data improves student performance on future unexplained data. Finally, we also verify that misaligned teachers can lower student performance random chance by intentionally misleading them.!
Introduction
Teaching, or the ability to provide needed information in way that is understood by others, is often considered an important property of Explainable AI [1]. When Al models "teach" by providing meaningful and interpretable explanations, it fosters transparency, warranted trust, and the ability for humans to make informed decisions based on AI recommendations. One way the goodness of an explanation can be judged is by its ability to communicate knowledge of how to solve a problem to other agents [2, 3]. Explanations fulfill this purpose not only by being informative but also by means of filling specific gaps in the recipient agent's knowledge. This is enabled by the explainer having theory of mind (ToM), understanding what the recipient does not know [2], and being able to personalize its explanations based on the recipient's 's needs. Recent work has argued that LLMs like GPT-3.5 now exhibit ToM, based on their ability to answer questions about mental states of
"Code for all experiments: https: http:///shhhaa.....////aaataaa/a www.wwwwwwwwaa.aat.ae......awwaa www.eatttttootttnti.iiooooo
com/
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Modhil Oudpui
RBa
Phompe
Prompt Iipul
M Promge
No Intervenioon
Teacher
Student
Misd Henmt
Theiry
Protricnnction Sirulction Promect
Budget
Net Inteneeend Prodecions
Pan
2.0
Runction (Fapecedd Uiily)
Inarraee Phudissor
Interontion Prompt
With Infervartion Studert Frompe
Infervention
Personalization Prompt
Pre-Intervention Simulation Prompt Squeeine
No Intervention Student Prompt
[uusiion)
Q: [ouestio))
cxpf) the
praf
(gell So (sold aasvedd
A: A:fieljnu 20M cull) So the snswer is (goM_
Bist qustion)
Q: flot quzsion)
Q: [asosiat)
A: (hchiit gold ool) So the answer gold ans)
Post-intervention Simulation Prompt
With Intervention Student Prompt
Q: (qunttoon
Sqreston)
A: fefpocher_rrre pxpp) So the answer goos prodent
(goM expl So the answer (poid answer)
Q: (to question)
Q: foo quesiie))
just guonion))
Ar Achocder cxpl So the answer
fressher cupf)
Figure 1: Overview of single-round of interaction between teacher LLM and student LLM,, covering first three research questions (with Fig. 6 showing RQ4, multi-round sudent-teachhe interaction that builds on top of RQI-RQ3). RQI: The teacher randomly intervenes and communicates explanations improve the student's performance (right part). RQ2: The teacher decides when to intervene by computing Expected Utility of Intervention using the Pre- Interveeevnnnon and Post-Intervention Simulation prompts (middle part). RQ3: The teacher communicates personalized explanations that more helpful for the student (left part).
hypothetical people in classical theory-of-mine mind tests [4]. However, we do not yet know how well language models can teach other agents to solve reasoning tasks via explanations.
In this work, we are motivated by this essential goal of evaluating explanations (specifically, Chain- of-Thought (7]) rationales) from the perspective of teaching and improving weaker agents in solv- ing reasoning tasks. order to improve smaller models' reasoning skills, recent works propose knowledge distillation by fine-tuning smaller model on the reasoning steps generated by larger model [8, 9, 10]. Yet, an important component human teaching is understanding when and how the teacher should explain particular things to the student. Current distillation approaches do not evaluate a teacher's s ability to identify when a student lacks understanding, and past work has not explored how to personalize teacher explanations to the student's 's needs. A smaller student model might already be good at answering certain questions but might require the teacher's intervention for some harder questions. When there many things to teach the student and teaching is laborious, it important to choose which problems merit explanation in order to improve teaching efficiency [11]. Moreover, for more effective teaching, it is desirable to have the teacher personalize its explanations to help a particular student, and a teacher that lacks understanding of the student's needs (i.e., lacks Theory of Mind) will be unable to do this [3].
Motivated by the efficiency of human explanations, we consider a student-teacher framework where teacher model guides the reasoning of student model, with the goal of improving the student's reasoning on current and future data. In order to do so, we explore a Theory of Mind-inspired approach, where the teacher simulates the student's behavior by building mental model of the student. Our overall research question investigates whether the teacher's intervention (in the form of natural language explanations) explained as well as unexplained future data. However, communication is expensive, and therefore we assume that a cost is incurred each time the teacher intervenes with (communicates) an explanation to the student for particular data point. We refer to this as the intervention budget, the percentage of test data points the teacher intervenes on. In order comprehensively answer our overall research question, we further decompose the teaching problem into the following constituent questions:
enable the student to make more accurate predictions both
1. RQ1. Can teacher LLM intervene test time improve student LLM's predictions?
"There have also been strong arguments against the presence of ToM in current LLMs [5. 6]. In this paper. we do not thoroughly test for ToM abilities in LLMs. but instead we focus on measuring teaching performance in terms of teacher's positive effect student performance.

---

2. RQ2. Given a fixed intervention budget, when should the teacher intervene (i.e., on which data points), in order to maximize student performance?
3. RQ3. Given a set of intervention data, can teacher model personalize its explanations for student model to improve student performance?
4. RQ4. In multi-turn interactions, do teacher explanations generalize and improve student perfor-
mance across data points (beyond the explained samples)?
5. RQ5. Can misaligned teacher LLMs lower student performance by providing misleading explana- tions to the student?
We answer RQ1 by assuming that the teacher intervenes on random data points in four different settings: using human or LLM teacher, and, when the teacher is an LLM, using an LLM student that is either weaker or stronger than its teacher ($5.1), 1). Across three different reasoning tasks (StrategyQAa GSM8K, and Commonsense(AA) and two different model families (Flan-T55 and LLaMA), we observe that (1) teacher LLMs can effectively intervene on student reasoning, improving student performance on the end task, and (2) more intervention typically leads to monotonic increase in student per- formance, though model teachers are not as good as human teachers. Fig. shows the intervention process and the two student prompts (in the right part of Fig. 1) that are used to generate predictions.
RQ2 explores how to intelligently select which data points to explain for the student model, in order to improve teaching efficiency ($5.2). Past work in cognitive science also considers teaching efficiency in young children by deciding what to teach by maximizing the learner's expected utility of learning [12]. With similar motivation, we develop an Intervention Function that is inspired from the principle of a teacher having Theory of Mind. In particular, the teacher builds a mental model of the student's 's reasoning process, with the goal of intervening only on samples that are most likely to maximize student performance. Our Intervention Function is based on Expected Utility, in which the teacher first estimates the utility of an intervention by simulating the student's 's prediction pre-intervention (without intervention) and post-intervention (with intervention), then constructs a rank ordering of the samples according to this utility (see the middle part of Fig. 1). The teacher builds this mental model in few-shot manner, only assuming access to the student's 's predictions pre- and restinnerrvention for few samples. We demonstrate that our proposed Intervention Function based on Expected Utility (1) outperforms other baseline Intervention Functions, (2) improves student performance when the teacher is not 100% accurate, and (3) enables weaker LLMs teach stronger ones, unlike with random intervention in RQ1.
Next, in RQ3, we explore how the teacher should explain data points to a particular student model, in- cluding how the teacher can personalize explanations for a student model ($5.3). That is, after deciding which data points to intervene on (RQ2), we decide how the teacher should explain those data points, clear limitation the teacher just generating explanations as if it is solving the task is that the expla- nations not at all personalized for the student. Given that good explanations gaps in student knowledge [2]. we believe that equipping the teacher with basic personalization skills will improve its teaching ability. With this motivation,
designed to fill in
propose another few-shot mental model for the teacher that encourages it to tailor its explanations to be helpful for the particular student model it is teaching. The teacher builds this model by conditioning on a few demonstrations of "useful" human explanations that rectify a student's answer, thereby encouraging explanations that are more likely to help the student (see Fig for an example of the teacher's personalization prompt). We demonstrate this prompt's 's effectiveness against unpersonalized explanations that are generated by prompting the teacher with random human explanations, showing that LLMs can personalize their explanations.
RQ4 tests whether LLMs can teach student models to generalize to new unexplained examples ($5.4), rather than improve their reasoning at test-time (RQI-RQ3). In other words, now explore the ability of LLMS to teach using the teaching components introduced in RQ2 and RQ3 of when and how to explain samples. This leads to explore multi-round interactive setting, where each round consists of the teacher selecting a set of best points to explain (according to RQ2) and generating explanations for them (according RQ3). The student then conditions these teacher explanations in-context demonstrations to perform the reasoning task on future unexplained samples. We demonstrate that teacher explanations indeed generalize and improve student performance on unexplained data.
Finally, in RQ5, we investigate the negative implications of teacher explanations on student LLMs ($5.5). Given that LLMs can improve student agents, we also want to test whether they can lower student performance. If a misaligned teacher provides non-factual explanations in scenarios where the
3

---

student directly adopts them, does that lead to a drop in student performance? In fact, we show that teacher models can lower student performance to random chance by intervening on data points with the intent of misleading the student. This has potential implications for LLMS giving explanations in a context where other agents adopt them with unwarranted trust in their correctness.
In summary, our comprehensive studies highlight the ability of LLMs to teach and improve weaker LLMs, demonstrated via improvements on explained test examples as well as future unexplained data. Broadly, equipping LLMs with an ability to effectively and efficiently teach, opens the door to (1) using LLMs as personalized tutors for humans (where efficiency critical), (2) distilling knowledge into weaker or more compute- -efuuuiiffffientttt student models, and (3) improving human decision making via Al recommendations and explanations.
2 Related Work
Evaluating Explanations in Teacher-Student Games, Several past works evaluate explanations in the context of a student-teacher communication game [13, 14, 15, 16, 17J. The teacher communicates explanations to the student with one of two objectives: (1) evaluating whether explanations help students to simulate the teacher better, (2) whether explanations can directly teach students to better perform a task. Simulatabiliry, or the student's 's ability to simulate the teacher's own answer, is seen as a measure of the explanation's 's faithfulness, rather than a direct measure of whether explanations help students learn the task itself [18, 19]. Our work is focused on the second research goal of evaluating explanations from the perspective of teaching weaker agents. Prior work has shown that human explanations can teach LLMs [7, 20] and LLMs can also teach themselves from their own explanations or feedback [21, 22]. But it remains to be shown whether LLMs can also teach weaker agents. A few recent works also share similar goal as ours and they distill knowledge [23] directly into the student model by finetuning it on the explanations from the teacher model [8. 9, 10. 24]. However, these distillation methods do not consider the important aspects of communication cost between two agents, its trade-off with student performance, and how the teacher may build mental models of the student to decide when and how to communicate explanations to the student. Recent studies have also evaluated explanations in the context of human-Al collaboration, for their plausibility [25, 26), usefulness to human learning [27, 28], and for improving human-Al team performance [29, 30J. Different from these, we analyze model-model interactions, with the goal of understanding how effectively LLMs can teach weaker systems to solve a task.
Theory of Mind in AL. A body of work demonstrates that humans regularly infer and make decisions based on the mental states of other agents, also known as Theory of Mind (ToM) [31, 3, 2, 32]. This has motivated works on computational language acquisition using ToM [33, 34, 35]. There have been recent works arguing both for and against the presence of Theory of Mind in Large Language Models [36, 4, 6, 37]. Theory of Mind has been successfully applied to improve human-Al collaboration in robotics [38, 39, 11]. In this work, we design prompts that are motivated by teacher having a Theory of Mind to efficiently intervene and improve a student's 's reasoning capabilities.
3 Problem Setup
Student and Teacher. We assume a two-agent communication game between a student S and a teacher T. where the goal is to teach the student solve a particular task interactively. Here, we use an L.M as the student. To explore a range of student and teacher capabilities, we consider both human and model teachers, while typically using a student model that measurably weaker than its teacher. Following past work, an LLM with more parameters is considered stronger model due to its better performance across a range of tasks (including the ones we consider in our studies). In the scope of our study, when the teacher is not human, both S and T are LLMs, prompted with a set of demonstrations of the task D (typically, 4-8 examples). Each demonstration d(t) E D is a triple (zr()), y(0) , (e)) consisting of an input (,))), the output f and human burttewritten explanation e that answers the question of why the data point has the output it has [1]. In our tasks, the explanation may include background knowledge
(4)
intermediate reasoning steps that are helpful for obtaining the answer. By organizing 'D into Chain-of- Thought prompts, both S and T are equipped with the ability to generate explanations and predict labels for new samples.
Single-Round Intervention. The first problem setup we consider involves the teacher deciding whether or not to intervene for single test problem. In this setting, the student's goal is to answer

---

the problem correctly, and the teacher can choose to intervene for individual problems to assist the student. Thus, given a test data point (()) we have the following two scenarios:
No Intervention: When the teacher chooses not to intervene, the student generates both the (8)
explanation es and the prediction iiss by conditioning on the D task demonstrations and the test input z(0) This is done using Chain-of of of-Thuugh prompting [7].
When the teacher does choose to intervene, it communicates its generated explana- tion to the student. Here the student only generates a prediction / by conditioning on the 'D task
Intervention:
demonstrations, the test input t(²), and the corresponding teacher explanation ef For the tasks and datasets we consider, explanations provide helpful background information or reasoning steps but do not directly reveal the test label, it never directly gives away the answer.
Fig. shows the 'No Intervention' and "With Intervention' student prompts and the overall interven- tion process. Note that in terms of the prompts used for both these scenarios, the only difference in the source of explanation (student vs teacher) for the test point. When the teacher is human, intervention happens with a human-writter explanation (crowdsourced in the datasets we rely on).
Communication Cost. In Mingle-Rount Intervention, the teacher could maximize student perfor- mance by simply always intervening on the student's 's reasoning. We bring in a natural constraint from Rational Speech Acts theory, i.e., communication is costly and should only be undertaken if it furthers a communicative goal [40]. Hence, we assume that a communication cost is incurred each time the teacher intervenes with an explanation to the student. We also note that this cost only with respect to an agent's ' communication (and assume that the teacher can always generate explanations for itself). Unless otherwise stated, communication happens one-way from the teacher the student in the form of explanations. We set a limit to the number of points that the teacher can intervene on, referred as the intervention budget, and we assume the cost be uniform for all data points. Across all our experiments, we vary the intervention budget between (0%, 20%. 40%, 60%, 80%, 100%]. A budget of 0% means the student generates its own explanation as predicts each data point, while a budget of 100% means the student leverages the teacher's 's explanation for every data point. Later, in Sec. 5.2, we introduce the teacher Intervention Function, which the teacher uses to decide which points to intervene on given its fixed intervention budget.
Multi-round Intervention. Here, the goal of the teacher is provide explanations to the student that help it generalize across samples, rather than leading the student to the correct answer only for the explained data points. Thus, we allow the teacher to explain data points that are then added to the student model's 's prompt, but we forbid the teacher from intervening on future test points. If the teacher can improve the quality of the student model's prompt, student performance on the test data should improve. In our experiments, this process occurs in a few steps: (1) the teacher picks points to explain, (2) the teacher explains those points, (3) the points are added to the student prompt with the teacher's ' explanations, and then (4) the student predicts labels for the remaining test data. Further implementation details are given Sec. 5.4.
4 Experiment Setup
We consider complex natural language reasoning tasks, motivated by two specific needs. First, the task should be hard enough for current LLMs that explanations can provide useful knowledge to the models. Second, it allows us to study free-text rationales, which are more flexible than input attribution methods [41, 42, 43] and benefit many reasoning tasks [7, 44]. We experiment with three reasoning tasks: (1) StrategyQA [45], (2) GSM8k [46], and (3) Commonsense(A [47] (details in Appendix B). Both Strategy QA and CommonsenseeA contain explanations in the form of relevant facts, thus requiring the student to reason over them to produce an answer. We also manually verify (up to 100 samples) that the explanations do not explicitly leak the answer. For GSM8K, since the reasoning steps explicitly derive the answer, providing the entire explanation during intervention will lead to answer leakage [17]. Hence, the teacher communicates a partial explanation (specifically. only the first step of the rationale), allowing the student to leverage it as hint to derive the final answer. We experiment with two state-of-the-art open-source LLMs of varying sizes, ranging from 780M to 65B parameters. Specifically, we use two encoder-decoder and decoder-only models as student and teacher: (1) -S-L- and Flan-T5-XI. [48], and (2) L.aMMA7B,, I.LMA-113 13B, and LLaMA-65B [49]. Refer to Appendix A for more details of student and teacher models.
5

---

5 Experiment Results
5.1 RQI: Can teacher LLM intervene at test time to improve a student LLM's predictions?
Our first research question asks if LLLSs can improve students by intervening their reasoning test time. While the main goal is to analyze the behavior of model-based teachers, we also experiment with human teachers to establish a ceiling on the capabilities of an LLM teacher. These human teachers are people who authored the (human) explanations in the datasets we experiment with and were crowdsourced in prior works.
Study Design. We compare the accuracy obtained by the student model at different intervention budgets. For the purpose of this study, the intervention happens random data points while we vary the student and teacher. In particular, we compare four intervention setups: (1) a human teacher paired with a smaller student model, (2) human teacher paired with a larger student model, (3) larger teacher model paired with a smaller student model, and (4) a smaller teacher model paired with larger student model. For the main experiments, the student and the teacher chosen from the same model family.
Flan-T homon Torlagg
Main Results. Figure 2 shows the results StrategyQA with Flan-T5 -5 models. A human teacher's intervention on the explanations of both smaller and larger Flan-T5 mod- els exhibits a monotonically increasing accuracy trend. Larger model teachers can also improve smaller student models. Flan-T5-L Large obtains an accuracy of 58% when always utilizing its explanations but obtains up to 63% accuracy when reasoning with the larger Flan-T55- XL's explanations. Intuitively, a larger student model does not benefit from a smaller teacher model's explanations, as we observe monotonically decreasing trend. Our results generalize to other models (LLaMA), datasets (Common- senseQA, GSM8k) and even when the student and the teacher belong to different model families. In fact, when the teacher (LLaMA-65B) is much stronger than the stu- dent (LLaMA-7B), the margin of improvement is also higher, about 8% (statistically significant with p = 0.01). See Appendix D for these additional RQ1 results. In summary, we conclude that: for complex reasoning. LLMMs can indeed effectively intervene and improve weaker models, and more intervention rypically leads better performance, although humans explanations improve more.
85
80
Accuracy
65-
60-
0
20
40
60
80
100
Intervention Budget (96)
Figure 2: RQI: Comparison of random intervention by different Teacher Models on different Flan-TS Student Models at different intervention budgets for Strate- gyQA. A B A student, B teacher.
5.2
RQ2: Given fixed intervention budget, when should the teacher intervene (i.e., on which data points), in order to maximize student performance?
So far, we have demonstrated that random teacher intervention benefits student models. But good teacher does not randomly pick problems to help a student with. Each intervention also has an associated communication cost and hence, it is desirable to be able to improve student performance while reducing the cost. In this research question, we investigate better strategies for choosing data points to intervene on. We call these strategies Intervention Functions that produce rank ordering of the samples, and, given a fixed budget, the teacher intervenes on the highest-ranked samples.
An intervention is useful if the student's confidence in the gold answer increases with intervention (i.e., with teacher's explanation) compared to without (i.e., with its own explanation). Here confidence is simply the likelihood that the model assigns to the correct answer i.c., we take the logits from the last layer of the model and normalize them get the correct answer's probability. Computing expected utility, however, depends on two quantities: (1) the student's 's true confidence measures with and without intervention, and (2) the gold answers against which the confidence is computed. It also incurs a two-way communication cost, one for the teacher to communicate its explanation to the student and another for the student to communicate back confidence to the teacher. Thus, we propose Intervention Function based on the Expected Utility of intervention, which relies on estimates of student confidence, and we consider two setups depending on whether the teacher knows the gold label. Ideally, teacher is expected to be expert in the concerned task (c.g., if the teacher

---

Papedted (Pa) He-Popoeed (Prng
Teacher Expected Usily Day
Expected Utily
72
80
70-
Accuracy
75
Accuracy
70-
66
64
65
60
60-
55
58
0
20
100
20
40
60
100
Intervention Budget (96)
Intervention Budget (96)
(a)
(b)
Figure 3: RQ2: (a) Comparison of different Intervention Functions on Strategy QA with a smaller student (Flan-T5- -a-Laee) and larger teacher (Flan-T5-XL), (b) Ablation of Expected Utility.
a human powerful model that obtains high accuracy). When the teacher does not have access to gold answers, we treat the teacher's answers as gold answers when computing Expected Utility. Expected Utility Intervention Function. The teacher computes the Expected Utility of intervention by simulating the student's 's predictions using a mental model of the student. In order to build this mental model, we assume that the teacher has observed the student on few samples and has access to d demonstrations Dasm of the student's 's predictions with and without intervention, denoted as:
D (z Cg
((0) C.T Upre: Spost Timl
where (6) and (() are the input and output respectively; ef and es denote the student and (6) teacher explanations respectively; and ypre and Spest refer to the student predictions with student explanation (pre-intervention) and teacher explanation (post-intervention) respectively. Using these demonstrations, the teacher builds a few-shot mental model of the student and predicts two quantities for a given test question - (1) Pre-intervention Expected Student Confidence (Cpre): The teacher
conditions the pre-intervention demonstrations D (ax(() (0))), Cs -ypre simulate the student's 's confidence on the gold answer, had it been using its own (student) explanation, and (2) Post- intervention Expected Student Confidence (Cyost): The teacher conditions on the post-intervention
(t) .(t) 1d
(8) (() 1d demonstrations Drost = {x)((), y0))e ey" ppstt to estimate what the student's 's confidence would be it had used the teacher's explanation. The teacher computes these confidence estimates as it were the student (refer to Fig. 1 for the prompts), essentially learning to simulate the student by conditioning on the appropriate demonstrations and then generating an answer to the question.
Then the Expected Utility Ú = (épost êpre) is given by the difference between the two confidence measures. The teacher finally constructs rank ordering of the test data points based this expected utility. This utility-basee ordering encourages the teacher to pick points where it thinks the student will answer correctly with intervention but incorrectly without.
Other Intervention Functions. To analyze how well our proposed Intervention Function performs, we compare it with the following Intervention Functions. Our first baseline is the Random Interven- tion Function from RQ1. Next, we compare with an Intervention Function that ranks the samples based on the Teacher Confidence- when the teacher is most confident about a question, it intervenes. Our next two baselines are ablations of Expected Utility: (a) Pre-Intervnntion Expected Student Confidence We rank samples based on the expected student confidence with no intervention (i.e., lower this confidence, the higher the likelihood of useful interventions). and (b) Pust-nntervention Expected Student Confidence: We rank samples based on the expected student confidence with intervention (i.c., higher this confidence. higher is the likelihood of useful interventions. Finally, as upper bounds of Intervention Functions, assume that the student communicates its true confidence values to the teacher (which for post-intervention, incurs a both-way communication cost of the teacher sending its explanation, followed by receiving the student's confidence). Using the true confidence measures, we compute True Utility.
Main Results: How does Expected Utility compare to True Utility? Figure 3 compares different Intervention Functions with Flan-T5-XI. as the teacher and Flan-T5-Larre T5-Large as the student on Strate- gy gyQ.. Across different methods, we analyze accuracy obtained at lower communication costs (e.g..
7

---

20%) as well as highest accuracy obtained, independent of any budget constraints. Our primary observation from Figure (a) is that Expected Utility improves student accuracy by up to points at a low communication cost of 20%. Expected Utility also peaks at an accuracy of 71% % with only 40% intervention. Since model-based teachers are not always perfect, increased intervention beyond 60% leads to drop in student accuracy (e.g., in the last 20% of the intervention, the student accuracy drops from 69% + 63%). When the student communicates its confidence scores to the teacher, the teacher is able to compute the true utility of intervention, which unsurprisingly. leads to much higher accuracy of 76% at 20% cost and an overall high of 81 31% accuracy. Nevertheless, estimating expected utility is cheaper and our results also suggest that better mental model could further improve performance. Ranking by teacher confidence is ineffective because it is not an indicator of the student's 's capabilities. Next, in Figure 3(b), we show that ranking by utility outperforms ranking by either pre or post-intervention confidence scores. In summary, with access to only a few demonstrations of student behavior, teacher can build an effective mental model of the student and intervene such that the student obtains a much higher accuracy at low communication costs.
When the teacher does not have access to gold answers, can we compute Expected Utility with respect to teacher answers? Teachers can be inac- curate and may not even have access to gold answers. In such scenarios, can treat the teacher as the gold standard and compute utility with respect to the teacher's answers? We explore this in Figure 4, comparing Expected Utility to Random Intervention and Student Least Confidence. The latter denotes that when the student is least confident about any of the answer options, it is more likely answer incor- rectly and hence will benefit from intervention. We observe that Expected Utility, computed with teacher answers, also leads to up to 2 points improvement in accuracy at 20% budget, which also within 1% of the accuracy (63.60%) obtained with 100% com- munication cost. In Appendix Table 9, we conduct the same experiment with much stronger teacher (LLaMA-65B) and a weaker student (LLaMA-7B) and obtain even stronger evidence of this result. Stronger teachers like LLaMA-65b are significantly better at solving reasoning tasks and thus their predicted labels will mostly match the gold labels. Hence, even if we rely on the teacher's predictions for computing expected utility, it improves student accuracy by up to 5 points (statistically significant with p = 0.02), further closing the gap between 'with and without gold label' scenarios. In summary, we conclude that imperfect teacher LLMs can also successfully intervene by building mental models of students that do not rely on ground-truie answers. Appendix contains additional results for RQ2 with other models and datasets.
Expotted
66
64-
Accuracy
62-
60
58-
20
40
60
80
100
Intervention Budget (9))
Figure 4: RQ2: Comparison of different Intervention Functions on StrategyQA with Flan-TS-Large student and a FIXn-T5-XL teacher, with no access to gold labels.
5.3
RQ3: Given a set of intervention data, can a teacher model personalize its explanations for a student model to improve student performance?
The previous RQ showed how the teacher may build sew-shot mental model of the student to decide when to intervene, given a fixed budget. Upon intervention, the teacher communicates an explanation generated by prompting the model with random human explanations. This leads an unpersonalized teacher that assumes that the explanation it generates in order to solve the task will be automatically helpful for the student. However, effective teacher should tailor its explanations to fill in gaps in the student's 's knowledge [2]. With this motivation, the teacher builds another few-shot mental model of the student, this time generating helpful explanations that more likely to benefit the student.
Teacher's Explanation Personalization Prompt. Hclpful human explanations are those that rec- tify student's 's answer i.c., cause the student's 's answer to flip from incorrect (when using its own explanation) to correct (when using human explanation). We assume that the teacher has ob- served the student d demonstrations Dp of exclusively helpful human explanations, denoted as: d Dp = (z((0,, (0) ex ; li=1 where (t) and ( denote (helpful) human and (not helpful) student explanations respectively. The teacher conditions these demonstrations to generate explanations
8

---

for the student. Fig. 1 shows an example of personalization prompt. With such prompt. teacher explanations are steered toward only those explanations that help the student.
Baselines. We compare personalized teachers with unpersonalized ones that condition on random human explanations. Appendix also reports results with unpersonalized rationales, that are post-hoc explanations ('The answer is X because Y') and not Chain-of Thain-of-Thought ("Y. So the answer is X').
Main Results. Fig. 5 shows the results on Strate- gyQA with Llan-TS-Large as the student and Flan- T5-XL the teacher. Both unpersonalized and per- sonalized teachers choose intervention samples based on Expected Utility, defined in RQ2. We observe that a personalized teacher improves the accuracy fur-
Unpersonalized
Personalzed
74
72
70-
Accuracy
68-
66
ther both at lower budgets (by 2% 20% cost) but
64
also overall, obtaining a peak accuracy of 72.63%.
62
However, unlike the strong supporting evidences we obtain for RQI and RQ2, the effect of personalization comparatively weaker. Hence, we further test this research question with a LLaMA-65B teacher and a LLaMA- A student Appendix F. While scaling up the teacher model points to stronger evidence of per- sonalization (e.g.. 2.4% better student accuracy), the results are still not statistically significant (p 0.09). Hence, we conclude that: personalizing teacher ex- planations can further benefit the students, although our results currently suggest that the effect size may be small. We hope that future work is further able to explore explanation personalization with even stronger teacher models like GPT-4. In Appendix F, we show some comparative instances of unpersonalized and personalized explanations.
60
58
56
80
100
Intervention Budget (in %6)
Figure 5: RQ3: Comparison of unpersonal- ized and personalized teacher (Flan-T5-XL) explanations on student (Flan-TS-Large) ac- curacy for StrategyQA.
5.4 RQ4: In multi-turn interactions, do teacher explanations generalize and improve student performance across data points (beyond the explained samples)?
In the previous RQs, we showed that teacher explanations improve student predictions for the samples that the teacher explains. RQ4 explores whether teacher explanations also generalize to new instances that the teacher has not explained. In other words, this studies if the student can perform Chain-of-Thoughe reasoning by only conditioning on teacher LLM explanations rather than human's s.
Study Design. We consider a multi-ture turt teaching setup (Fig. 6), in which each turn the teacher chooses to explain a few samples from a pool of unexplained exam- ples which are then added to the student's 's prompt. The prompt consists of demon- strations of the teacher's explanations and predictions. The student then conditions only on these in-context examples (without any human demonstrations) to generate pre- dictions for the test samples (where there is no teacher intervention). For choosing the data points to explain at each round, we use the Expected Utility Intervention Function (from RQ2), and for generating the teacher explanations, we leverage the ToM prompt (from RQ3). We say that teacher explanations generalize if conditioning on demonstrations of explained points improves upon demonstrations with expla- nations (i.e., only QA pairs) or self-explanations (i.e., demonstrations with student explanations and predictions). We consider five rounds in total with L.LMA-77B as the student and LLaMA-65B the teacher, adding two explained samples in each round. We compare the student accuracy after each round with teacher-explained, student-explained, and unexplained demonstrations.
MuKk-Tum(Kx]
Teacher
Student
Explaren toaa hoal
Prompl Ingux
Slaceen Prompthe
Student Prompt
Thet Cometooo
Q: [gazdiun)
(pocher_ So the answer is frocherr
RDA
cuestioestion)
Student Predition
Figure 6: Overview of multi turn student- interaction showing RQ4, At cach turn, the teacher chooses some sam- ples to explain and the student conditions on them to make predictions on future unexplained data.
Main Results Fig 7 shows the results. We observe that teacher explanations improve student performance on future unexplained test points as well by significant 6 points (55% + 61.6%).

---

Random Nag Expected Usity
Upparsonaiia-CCa Deoshving Explanation
72
65
70-
Accuracy
Accuracy
60
66
64
55
60
50
58
0
100
20
40 60
100
Intervention Budget (in 96)
Intervention Budget (in 96)
(a)
(b)
Figure 8: RQ5: (a) Negative Implication of RQ2: Comparison of intervention based on negative ex- pected utility with random intervention StrategyQA (b) Negative Implication of RQ3: Comparison of an unpersonalized teacher (generating explanations conditioned on random human explanations) versus deceiving teacher (generating explanations conditioned on wrong explanations).
-en No Explanations
Strdeen Exparations
While student accuracy with unexplained samples improves after adding more demonstrations (55%
+in Teacchrr Explerations
59%), accuracy with teacher-explained demon- strations is better by up to 4 points (p 0.04) after each round. Interestingly, when the student con- ditions on self-explained demonstrations, its per- formance decreases with increasing rounds. We hypothesize that this is because the student might be overfitting its own worse explanations, lead- ing to bad predictions. In summary, we conclude that: teacher L.LMs can teach student models perform well on their own when given new test data. LLMs with even longer context windows will allow adding more explained samples in each round and repeating the teaching process for more rounds. We hope that our initial promising results will encourage more exploration in multi-round teaching with LLMs.
Accuracy
58
1
2
Number of Rounds
Figure 7: RQ4: Multi-round student-teacher interaction comparing student accuracy on un- explained test points with unexplained, student- explained and teacher-explained demonstrations.
5.5
RQ5: Can misaligned teacher LLMs lower student performance by providing misleading explanations to the student?
If teacher L.Mss can successfully build mental models of student LLMs, natural follow-up question regards whether communicating misleading explanations can also weaken student models. Here we briefly describe our study design and findings, with more details in Appendix H. First, the teacher intervenes in increasing order of expected utility, prioritizing samples where the utility is lowest. Second, we make the teacher condition on incorrect answers and non-factual human explanations that we manually generate by perturbing (correct) human explanations. We show the results in Fig. 8(a) and Fig. 8(b). Ranking data points by negative expected utility allows the teacher reduce student accuracy to random chance at 60% intervention. Next, Fig. 8(b) illustrates that the teacher condition non-factual explanations student accuracy by 2 points, relative to the improvement of benign explanations. Thus, we conclude that teacher I.I explanations can be influential for student LLMS in both good and bad ways.
successfully generate worse explanations that reduce the
6 Conclusion
We demonstrated that LLMs can teach weaker student models to improve their performance reasoning tasks for both explained and unexplained future data. The teacher builds two few-shot mental models of the student, one predicting which data points intervene and another generating personalized explanations that the student can efficiently learn from.
Limitations & Broader Impacts. See Appendix for limitations and broader impacts discussion.
10

---

Acknowledgments
We thank Shiyue Zhang and Archiki Prasad for useful feedback, well as Stephen Casper and Miles Turpin for suggestions regarding experiments. This work was supported by NSF-CAREER Award 1846185, NSF-AI Engage Institute DRL-2112635, DARPA MCS Grant N66001-19-2-4031, and Google PhD Fellowships. The views contained in this article are those of the authors and not of the funding agency.
References
[1] Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence, 267:1-38, 2019. URL https: //arxii org/ abs/ 11001 07269.
[2] Henry M Wellman and Kristin Lagattuta. Theory mind for learning and teach- ing: The nature and role of explanation. Cognitive development, (9(4):7994997 2004. URL https: sttppp://ww//
www.s..ienneeieee...... com/ wwcienneearticle abs /pi//
5088520140400080
[3] Stephanie M Carlson, Melissa A Koenig, and Madeline B Harms. Theory of mind. Wiley Interdisciplinary Reviews: Cognitive Science, 4(4):391-402.
2013. URL https: http://ssstessssssccssssssssssss socsci. uci ceti//ppart/ccuue//eeeedee
Carl CarlsonKoeni Carlsonkoeniggarrms20000 wtwp:r/ee0ettttttoooooooooo .dd...
TheoryOfMing
[4] Michal Kosinski. Theory of mind may have spontancously emerged in large language models.
arXiv preprint 2302:23302.02883 2023. URL
[5] Tomer Ullman. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08899 08399, 2023. URL https: /tcepss//aaxiie http://xxxvv.ro//e http://bb//222.999999
[6] Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. Clever hans or neural theory of mind? stress testing social reasoning in large language models. arXiv preprint arXiv:2305. 2023.
[7] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022. URL https: /ttpii.xxvvv..////./...../.. abs/ 0ttp/2211.500 11903.
[8] Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. arXiv preprint arXiv:2212.08410 2022. URL httph::t/////c. /atpsv////xvvv.e/ owg//bgg//......ooooo...
0841C.
[9] Yao Fu, Hao Peng. Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. arXiv preprint arXiv:2301.122726 2023. URL https:
abs 12726...........
[10] Namgyu Ho, Laura Schmid, and Se- Young Yun. Large language models are reasoning teachers. arXiv preprint arXiv:2212. 2022. URL htttps://arxii wogg/aaaa......ooo....e /etp./221..c 10071.
[11] Xiaofeng Gao, Ran Gong, Yizhou Zhao, Shu Wang, Tianmin Shu, and Song-Chun Zhu. Joint mind modeling for explanation generation in complex human-robot collaborative tasks. In 2020 29th IEEE international conference on robot and human interactive communication (RO-MAN),
pages 1119-1126. IEEE, 2020. URL httpss://axxi...ttt///
abs / 12803.
[12] Sophie Bridgers, Julian Jara-Eitinger and Hyowon Gweon Young children consider the expected utility of others' learning to decide what teach. Nature human behaviour, 142) 152, 2020. URL https: / www. nature. wwm/artiiiiao.ooo.. a c
[13] Arjun Chandrasekaran, Viraj Prabhu, Deshraj Yadav, Prithvijit Chattopadhyay, and Devi Parikh. Do explanations make VQA models more predictable to a human? In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1036-1042, Brussels, Belgium, Oetober-Novemtee 2018. Association for Computational Linguistics. doi: 10.18653/w/IDI-- 128. URL https: halpps://acaaaaaa /tt.a/aaaalooo...r....cc
wwtphhhwooooooooooooooee wwt./oo-g//D// 1128.

---

[14] Marcos Treviso and André FT Martins. The explanation game; Towards prediction explainability through sparse communication. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP,, pages 107-118, 2020. URL. :
www.//ggg//aaoooooooooooo...o.aa //arxiv.org/ alss 20044 13876.
[15] Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C Lipton, Graham Neubig, and William W Cohen. Evaluating explanations: How much do explanations from the teacher aid students? Transactions of the Association for Compu- tational Linguistics, 10:359-377 -75,, 2022. URL https: /ttpaaaiii///////////.e//./aaala .ww.iiioorrr..oooooo abs 201./001..0000 00893.
[16] Peter Hase and Mohit Bansal. Evaluating explainable ai: Which algorithmic explanations help users predict model behavior? In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 5540-5552, 2020. URL https: http///axvvv...o//oo org./.ioorrooooooooooo.ooooooo.r abs/ 2005. 01831.
[17] Peter Hase, Shiyue Zhang, Harry Xie, and Mohit Bansal. Leakage-adjusted simulatability: Can models generate non-trivial explanations of their behavior in natural language? arXiv preprint arXiv:2010.04119, 2020. URL https: / arxiv. abwgg//bs//00/ 04119.
[18] Finale Veshi-Velez and Been Kim. Towards a rigorous science interretablle machine learning. /1702 08608.
arXiv preprint arXiv: 720/i::7002.0800 086088 2017. URL
[19] Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31-57, 2018. URL https: / //dl., acmm org/ doi /10. 1145/11550 /1/45/32330 3236386 6 3241340.
[20] Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. Can language models learn from expla- nations context? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 537-563, Abu Dhabi, United Arab Emirates, December 2022. Association for Compu- tational Linguistics. URL https: http://aaaannthhocco 38.
findings-emniy
[21] Jiaqi Ma, Dylan Slack, Asma Ghandeharioun, Sameer Singh, Himabindu Lakkaraju, al. Post hoc explanations of language models can improve language models. arXiv preprint arXiv:2305. 1426, 2023. URL. https:/ //arx//aavvvi org/ abs 200.../335...4.
[22] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao. Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303. 2023. URL pattps://arxiia abs/2303 17651.
[23] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 2015. URL https:/
/tttspp://arxi...e/ harpaaiivvv..///..//aaa org/ abs 1100/1500100 02531.
[24] Aaron Chan, Zhiyuan Zeng. Wyatt Lake, Brihi Joshi, Hanjie Chen, and Xiang Ren. Knife: Knowledge distillation with free-text rationales. arXiv preprint 1rXiv:2212.09721, 2022. URL https:// /tt::///xxiiii
www./org/aaaoo aass///////a 2212 2. 09721.
[25] Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. Reframing human-ai collaboration for generating free-text explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 632-658, 2022. URL https: ataaaaixvvvooo.a abs/
2112.088 0812.88774.
[26] Swarnadeep Saha, Peter Hase, Nazneen Rajani, and Mohit Bansal. Are hard examples also harder to explain? a study with human and model-generated explanations. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2121-2131, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https:/ /arxiv. orggabs/////oooooooooooooo.bb 2211. 07517.
12

---

[27] Brihi Joshi, Ziyi Liu, Sahana Ramnath, Aaron Chan, Zhewei Tong, Shaoliang Nie, Qifan Wang. Yejin Choi, and Xiang Ren. Are machine rationales (not) useful to humans? measuring and improving human utility of free-text rationales. In ACI, 2023. URL httpttps: /ttp:s///a/xvvvvi / abs 22a5/2305.00 07095.
[28] Kun Qian, Ryan Shea, Yu Li, Luke Kutszik Fryer, and Zhou Yu. User adaptive language learning chatbots with a curriculum, arXiv preprint arXiv:2304.0548, 2023. URL https:
/ //arxiv wttp/iiorrg//eoooo abs /2304 05489.
[29] Eric Chu, Deb Roy, and Jacob Andreas. Are visual explanations useful? case study in model- in-the-loop prediction. arXiv preprint arXiv:2007. 2020. URL https: //rxiv. arg/abs/2000 12248...8.....
12248.
[30] Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 202/ CHI Conference on Human Factors in Computing Systems, pages 1-16, 2021. URL https: httpps:///rxxii.. /ww.vv.rgg// abs/ 22120000.477
[31] David Premack and Guy Woodruff. Does the chimpanzee have theory of mind? Behavioral and brain sciences, 1(4):515-526, 1978. URL https:/ //ww.
[32] Michael Tomasello. Constructing a language: usage-based theory of language acquisition. Harvard university press, 2005. URL https:/ /www. hup. harvard edu/ haaalog. php? fisii=in80007777
[33] Jacob Andreas and Dan Klein. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1173-1182, 2016. URL ptttps://axivy .rg abs 6.660.0066..
[34] Hao Zhu, Graham Neubig, and Yonatan Bisk. Few-shot language coordination by modeling theory of mind. In International Conference on Machine Learning, pages 12901-12911. PMLR, 2021. URL https: http//axxivv////// org/ abs 2107, 05697.
[35] Andy Liu, Hao Zhu, Emmy Liu, Yonatan Bisk, and Graham Neubig, Computational language acquisition with theory of mind. In The Eleventh International Conference on Learning Representations, 2023. URL https: / arriiv org/ abs/ 20bs/2303. 2303 01502.
[36] Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, and Tom Griffiths. Evaluating theory of mind in question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2392-2400, Brussels, Belgium, October- November 2018, Association for Computational Linguistics. doi: 10.18653/v1/D1--11 1261-1161... URL https: / ac www.acholanooyyy.oooo org/D18- 1261.
[37] Shima Rahimi Moghaddam and Christopher Honey. Boosting theory-of-mind performance in large language models via prompting. arXiv preprint MrXiv:2304.11499, 2023. URL
https: //aaiii.////vvv/// www.wiwwww.w.....o.oo....oooo.re abw///333 11490.
[38] Brian Scassellati. Theory of mind for a humanoid robot. Autonomous Robots, 12:13-24, 2002.
URL https:: /aapps/ppppppppp//ppp//// dt ittiic.oo m st /tti/pdfssssee /odff//DAA43755..
[39] Sandra Devin and Rachid Alami. An implemented theory of mind to improve human-robot shared plans execution. In 2016 11th ACM/IEEE International Conference on Human- Robot Interaction (HRI), pages 319-326. IEEE, 2016. URL https: http:/aal..siieeeee ha1-01330339/ file HRRIIO11111 WDDVIN. 6DE.IN....E.....E.
[40] Noah D Goodman and Michael C Frank. Pragmatic language interpretation proba- bilistic inference. Trendy in cognitive sciences, 20(11):818-22) 829, 2016. URL https:
//www www.sseeieeeneeee..e.e..eeeeeeee com/ htt.://eee..cceeeeeeeeaaaaaae//a
13

---

[41] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144, 2016. https:
//arxiv.org/ alss 602. 04938.
[42] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference machine learning, pages 3319-3328. PMLR, 2017. URL nttpss//eet //tpiv//axiv..////////a/aa/ org/ abs/17000000 01365.
[43] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference machine learning, pages 3 5145-3153. PMLR, 2017. URL https: aaaaaaivivv..o org/ absaabas//0 4. 02 685.
[44] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. URL https: / //aaxivvv../. org/ abs 2204. 02311.
[45] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361 1616-361,220 2021. URL https:
/ //arxiv. org/ abs 221021111220 02235.
[46] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 10.14.2110.14166, 2021. URL https: /arxiv. org/ abs///b/// /
2110 00.10... 14168.
[47] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensequ: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lunguage Technologies, Volume (Long and Short Pupers), pages 4149-4158, 2019.
URL https:/t///// /arxivv..... hwigg/bbs119 811 00937.
[48] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. Scaling iestruction-finetumne language models. arXiv preprint arXiv:2210.1 1446, 2022. URL https: //axiv..///iiive/aa org/ abs/22100 11416.
[49] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023. URL https: / arxiv. org/ abs /aoes/2332..m 13971.
[50] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. 12712, 2023. URL
https://arxet http://awwwwww...a.//e/se/ wrgg/bb//////333/// 12712..111...
12712.
[51] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng. Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.043559 2021. URL https:
//arxiv. org. abs 22122.11222 04359.
[52] Shourya Aggarwal, Divyanshu Mandowara, Vishwaject Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. Explanations for commonsenseqa: New dataset and models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pa- acl acl-long,
pers), pages 3050-3065, 2021. URL https: /ttp://aaaaaatooooooooooooo 238/.
14

---

Broader Impacts
We hope that findings can help improve the understanding and evaluation of Chain-of of-Thought rationales, in order to better understand the behavior of LLMs and make them more interpretable. Through the first four research questions, we demonstrate that teacher LLMs can successfully build mental models of weaker agents to improve their performance. Modern LLMs like GPT-4 may generate non-factual explanations [50] that have the potential to inadvertently harm weaker agents, especially in a context where other agents adopt them with unwarranted trust in their correctness. We verify this through our final research question. In general, we do not foresee specific ethical risks arising from this work that do not already apply to the general use of Large Language Models, such as the potential to generate harmful or toxic content [5]].
Limitations
While teacher LLMs generate better explanations via personalization, the human explanations are unpersonalized i.e., collected without any particular student mind. In spite of that, we observe that intervention with human explanations proves to be helpful in most cases. It remains to be seen whether human explanations that are directed toward improving a particular student model can lead to further improvements. Next, we make a simplifying assumption that the communication cost is uniform across all samples. Non-uniform costs (e.g., measured based on the number of tokens or reasoning steps) such that longer explanations incur larger costs is interesting direction for future work. We also note that while both student and teacher generate explanations with the goal of improving student predictions, the predictions may still be unfaithful to the reasoning steps.
A Student and Teacher Models
We experiment with two state-of-the-art open-source LLMs of varying sizes, ranging from 780M to 65B parameters. Specifically, we use two encoder-decoder and decoder-only models as student and teacher: (1) -lar-e--Large and Flan-T5-XL [48], and (2) LLaMA-7B, LLaMA-13B. and LLaMA- 65B [49]. Typically, the student model is assumed to be smaller than the teacher model. But some experiments will also involve smaller teachers and larger students. All models generate text using greedy decoding, prompted with either 4-8 demonstrations. Unless otherwise stated, the demonstrations are randomly chosen from the training samples. For StrategyQA, we report results on the validation split, while for CommonsenseQA and GSM8k, our experiments are on the test split. To account for variance, we conduct experiments with at least three different seeds. We report accuracy for all tasks, and error bars in plots and tables represent the standard deviation across seeds.
B Datasets and Prompts
We experiment with the following three reasoning datasets: (1) Strategy [45]. set of open-domain
questions where the required reasoning steps are implicit the question, (2) GSM8k [46], which includes multi-step math reasoning problems, and (3) CommonsenseQA [47], multiple-choice QA task focusing on commonsense reasoning. We use the reasoning steps in StrategyQA and GSM8k as the multi-step rationales, and for CommonsenseQA, we rely on the ECQA dataset [52], which is annotated with commonsense facts supporting the correct option and refuting the incorrect options. All datasets are licensed under the MIT license. Fig. 9 shows the student prompts for the three tasks of StrategyQA, CommonsenseQA. and GSM8k. Fig. 10 shows the pre- and post-intervention student simulation prompts for the teacher model.
C Compute and Reproducibility
We conduct experiments either A100 Google Cloud instances on internal A6000 GPU servers. The I.MSS (Flan-T5 and I.LAMA) and the datasets used in our studies are publicly available. For reproducibility, we making our code available part of the supplementary material.
15

---

StrategyQA
Q: Are more people today related to Genghis Khan than Julius Caesar?
A: Julius Caesar had three children. Genghis Khan had sixteen children. Modern geneticists have determined that out every 200 men today has DNA that can traced to Genghis Khan. answer is yes
Q: ciess_question]
A:
CommonsenseQQA
Q:
What might person see at scene a brutal killing?
Answer Choices:
Choice I: bloody mess
Choice 2: pleasure
Choice 3: being imprisoned
Choice 4: feeling of guilt
Choice 5: cake
A: Bloody mess is covered stained with blood. A person might see a bloody mess at the scene of a brutal killing. Pleasure is about what person at the scene of brutal killing and one cannot be happy such brutality. You can't see someone jail at the brutal killing scene. Feeling of guilt doesn't 't come as the killing is brutal or merciless. Cake baseless and weird to think as is brutal killing scene and not hakery. So the correct choice is
Q: [test question]
Answer Choices:
Choice I: [option_]] 1)
Choice 2: (option_2)
Choice 3: (option_3)
Choice 4: (option_4) 4)
Choice 5: (option_5)
A:
GSMSK
Q: Natalia sold clips 48 of her friends in April, and then she sold half many clips in May. How many clips Natalia sell altogether April and May? A: Natalia sold 48/2 = 24 clips in May. Natalia sold 48+24 = 72 clips altogether in April and May. So the answer is 72
Q: [test_ questiqun
A:
Figure 9: Examples of student prompts for different tasks with one demonstration.
D RQ1: Additional Results
Results with Flan and LLaMA Models. In Table 1, we report the accuracy obtained by different students and teachers (based on Flan-T5 models) on the Strategy task. We draw similar conclusions as Flan-T5 with other LLMs, specifically LLaMA-7B and LLaMA-65B models on the Strategy QA dataset (Table 2). In fact, when the teacher stronger like LLaMA-65B, the margin of improvement is also higher, about 8%. The overall trends also align- increasing for weaker students and decreasing for stronger students.
Results on other Datasets. Our conclusions generalize across datasets too. Table 3 presents the results on CommonsenseQA with Flan-T5 models. CommonsenseQQA is an easier task and Flan-T5 models obtain accuracies of 85% and 92% when generating their own explanations. While LS-Laree still benefits from human explanations, the larger model does not, perhaps because it already starts at a high 92% accuracy. Finally, in Table 4, we present the results on GSM8k with LLaMA models. Note that in GSM8k, student has access to partial explanations from the teacher, but even then we observe that these prove to be useful prompts for the student to complete their chain-of-thought, leading to up 8-9% increase accuracy with human teachers and 3% with model teachers.
16

---

treelntervention Student Simulation
Simulate an Al model's answer for the given question.
Q: Will the Albany in Georgia reach hundred thousand occupants before one New York? AT Predicted A Answer: Alhany, Georgia city U.S. state of Georgia. Albany, Georgia has population of 59,080. The population of New York is 064,000 So the answer is no
Q: [question]
A1 Predicted Answer:
Vost-Intervnntio Student Simulation
Simulate an Al model's answer for the given question.
Q: Will the Albany in Georgia reach hundred thousand occupants before the one in New York? AT Predicted Answer: Albany, Georgia city U.S. state of Georgia. Albany, Georgia has population of 59,058. The Alhany in New York has population of 328,058. So the answer is no
Q: [question]
Al Predicted Answer: [teacher_ explbaattx)lnnnin the answer is
Examples of StrategyQA prompts for the mental model of a teacher simulating student predictions pre-intervention and post-intervention. Pre-intervention: The demonstrations use student explanations and student predictions and at test time, the teacher simulates both. Post-intervention: The demonstrations use teacher explanations and student predictions and at test time, the teacher uses the teacher explanation to simulate the student prediction.
Figure 10:
Results with Cross-family Student and Teacher. We observe that larger teacher LLMs can teach smaller student LLMs, even when they are of different model families. In Table 5, we report the results with Flan-Tr T5 and LLaMA models students and teachers.
Intervention Budget
Student
Teacher
0%
20%
40%
60%
80%
100%
Llan-T--Large
Human
58.51-12.00
63.75.10.43
66.95122.19
73.94112.77
78.02.12.40
11.95
Flan-T5-XL
Human
68.12:12.62
72.05-2.622
75.9812.31
80,20:1.65
84.13=1.00
87.77-0.70
Flan-TS-Large
Flan-TS-XL
58.51=2.00
6..52=1163
19.7811185
61.4822.02
62.35-2.13
62.96-22447
Flan-T5-XL
Flan-T5-Lard La-Lage
68.12:22.62
67.68=2.72
65.6443.39
64.04=3.63
62.88=1.15
61.66=0666
Table 1: RQI - Comparison of accuracy obtained with random intervention by Flan-T5 models at different intervention budgets on StrategyQA. shown the third row, Flan-T5-Large (student) accuracy improves by 5% with 100% intervention from Flan-T5-XI (teacher).
Intervention Budget
Student
Teacher
0%
20%
40%
60%
80%
100%
LLaMA-7B
Human
61.13:12.72
63.60.14.82
68.85 13.52
13.36.12.18
78.451:2.55
81.22.11.57
LLaMA-65B
Human
77.58-2.224
80.3412.65
82.67 22.06
17.48:11.96
89.37-0.25
92.86:10.50
LLaMA-7B
LLaMA-65B
61.13:12.72
1..29:11.53
64.91 10667
66.08.11.76
68.9913.14
69.43-13341
LLaMA-65B
LLaMA-7B
77.58
75.83.12.24
72.92.12.72
72.92.12.26
70.88.10.90
69.14.10.66
Table 2: RQI - Comparison of accuracy obtained with random intervention by LLaMA models at different intervention budgets on StrategyQA. As shown in the third row, ILLMMA7B (student)
accuracy improves by 8% with 100% intervention from LLaMA-65B (teacher).
E RQ2: Additional Results
Results with stronger Flan-T5-XI. teacher and weaker Flan-T5-Large student. Table 6 com- pares different Intervention Functions on StrategyQA with Flan-TS-Large as the student and F5-n-T5- XL as the teacher. These results are when the teacher has access to the gold labels. In Table 7,
17

---

Intervention Budget
Student
Teacher
0%
20%
40%
60%
80%
100%
Flange
Human
84.78=10.41
86.86.10.76
88, 0.9944
90.77:10.45
93.20:1047
95.42.10.17
Flan-TS-XL
Human
92.38=00.16
92.52=10.20
92.43-0.228
92.23=1066
92.41=1.12
92.21=1.06
Flan- Mlan-T5-1ama Large
Flan-T5-XL
84.78=0041
85.79=0.48
86.79+0.84
87.46+0.20
88. 88.52+0.39
89.72=0.68
Flan-TS-XL
La-Lage
92.38:10.16
90.92.10.39
89.74.1.39
87.9810.89
86.7011.60
85.19:1.62
Table 3: RQI Comparison of accuracy obtained with random intervention by Flan-T5 models different intervention budgets on CommonsenseQAA, As shown the third row, Flan-T5-Larg (student) accuracy improves by 5% with 100% intervention from Flan-T5-XI. (teacher).
Intervention Budget
Student
Teacher
0%
20%
40%
60%
80%
100%
LLaMA-7B
Human
9.62++1.53
11.97+00.80
13.84
16.32:00.57
18.72+0.78
21.05-+:0.5
LLaMA-13B
Human
1..45-11.80
18.44:12.16
20.34
22.41:12.46
24.91112.07
26.88:12.34
LLaMA-7B
LLaMA-13B
9.62+1.53
10.20+ 1.06
10.68
11,24+0.50
11.92+1.15
12.25=0.944
LLaMA-13B
LLaMA-7B
16.45=1180
1.68
15.56+1144 1.44
14.88+ 1.89
14.68+ 1.88
14,27++1.70
Table 4: RQ1 Comparison of accuracy obtained with random intervention by LLaMA models
different intervention budgets on GSM8k. As shown in the third row, LLaMA-7B (student) accuracy improves by 3% with 100% intervention from LLaMA- (teacher).
Intervention Budget
Student
Teacher
0%
20%
40%
60%
80%
100%
Flan-T5-I Ta-Large
LLaMA-65B
58.51=2.00
61.86=0.25
61.13+2..6
64.48+1.5
66.52:44.05
66.95=44.00
LLaMA-65B
Flan-TS -a-Lage
77.58+2.24
1.76
71.47+0.90
67.68+2..00
64.62++2.00
1.71
Table 5: RQI - Comparison of accuracy obtained with random intervention on Strategy QA when the student and the teacher belong different model families. As shown in the first row, Flan-T5-Large (student) accuracy improves by 8% with 100% intervention from LLaMA-65B (teacher).
Intervention Budgel
Intervention Function
0%%
20%
40%
60%
80%
100%
Random
58.51+2.000
60.40=1.76
61.13++2.65
60.98=11009
64.33:44.54
62.96++2.477
Teacher Conf f
58.51+22.00
58.66=2.40
60.11+2..90
$7.35+3.30
61.42:+3.91
62.96++2.447
Expected Student Conf (Pre) +
58.51-1-2.00
64.19=2.00
66.66:10.25
6.57
65.35-+-2.40
62.96:t:2.47
Expected Student Conf (Post)
58.51-12..00
1.77
68.26.10.66
69.71112.01
68.26:12.63
62.96.1.2.47
Expected Utility 1
58.514:2.00
6..83=1.53
71.32:11.33
71.1711.15
69.86112.43
62.96:12.47
True Student Conf (Pre)
58.51+:2.00
68.26=1.65
10.20=11.26
74.38+2.84
68.55:33.88
62.96:12.47
True Student Couf (Post)*
58.514:2.00
65.64=140
72.63++1.09
80.05=0.900
72.19=44.39
62.96+22447
True Utility f
58.51++2.00
76.56=0.50
80.78++1.15
1..51+1.76
78.60+3.299
62.96++2447
Table 6: RQ2 Comparison of different Intervention Functions with Flan-T5-Large student and
Flan- Flan-T5-X0 T5-XL teacher on Strategy The teacher assumes access to gold labels. f denotes that the samples are ranked in decreasing order of the function (higher is better), while J. denotes that the samples in increasing order of the function (lower better).
Intervention Budget
Intervention Function
0%
20%
40%
60%
80%
100%
Random
58.51:1:2.00
60.40 1.76
61.13:12..5
60.98:11.09
64.33-14.54
62.96.12.47
Least Conf
58.51:12.00
01.13
6..44:11.74
65.06:11.15
63.46:12997
62.96.12.44
Expected Student Conf (Pre)
58.51-12.00
62.59-1.00
61.86:10.00
6..
65.50:3.14
62.96.12.47
Expected Student Conf (Post)
58.51422.00
61.86=1.66
6..88:11.74
61.71#3.39
60.11:4.62
62.96:12.47
Expected Utility f
58.51+-2.00
62.29=0.50
1..4
62.44+3.88
62.95
62.96+2247
Table 7: RQ2 -
Comparison different Intervention Functions with a Flan-T5-Lan student and Flan-T5- T5-XI teacher on StrategyQA. The teacher, in this case, does not have access to gold labels.
18

---

Intervention Budget
Intervention Function
0%
20%
40%
60%
80%
100%
Random
68.12422.22
67.68=2.72
65.64:13.39
61.0443.63
62.88=1.115
61.66:00.6
Expected Student Conf (Prc)
68.12+42..2
66,22=2.24
66.95+11.53
65.35=1.00
62.73:+0.66
61 11.66--0.66
Expocted Student Conf (Post) f
68.12:42.62
70.59=3.27
71.76::3663
72.48-2.86
69.8 69.86:2.622
61.6::0..6
Expected Utility f
68.12:422.2
70.88=3.27
71.90:22.44
72.63+2.24
68.99=11155
01666600666
True Student Conf (Pre)
68.12++22.2
74.23=3.73
16.27+11.40
68.55+11.00
64.04:+0.90
61. 61.66=+0.66
True Student Conf (Post)
68. 18.12-42.62
70.16=3.27
73.94++1.76
80.05+11665
71.32-+1.00
61 0.66
True Utility
68.12:1.2.2
79.91
80.93-12.06
80.6412..24
78.16:12.00
61,86 8..6666
Table 8: RQ2 Comparison of different Intervention Functions with smaller teacher (Flan-T5-
Large) and larger student (Flan-T5-XI.)) on StrategyQA. The teacher assumes access to gold labels.
Intervention Budget
Intervention Function
0%
20%
40%
60%
80%
100%
Random
61.13:22272
62.29:t 1,53
64.91+0.67
66.08 1.76
68.9913.114
69.43+3.41
EU (w/ teacher answers)
61.13:+2.72
66,22++2.63
67.39+2.40
69,28=1.76
7..59+3.81
69.43=3.41
EU (w/ gold label)
61.13:12.72
66.52:1..27
70.1610.90
7..47=1009
72.7811248
69.43:1.44
Table 9: RQ2 - Comparison of Expected Utility (with and without access to gold labels) with random intervention, involving a LLaMA-7B student and a LLaMA-65B teacher. EU = Expected Utility, Importantly, when the teacher does not have access to the gold labels, expected utility with teacher answers (second row) leads to a statistically significant 5% improvement in student accuracy (p 0.02) at 20% intervention.
Intervention Budget
Intervention Function
01%
20%
40%
60%
80%
100%
Random
84.79.10.41
85.79=0.48
86.79.10.84
87.46110.20
88.52:10.39
89.72.10.68
Expected Student Conf (Pre)
84.79110.41
84.57=0.69
86.35-10.73
87.99-0.87
89.51:10.82
89.72-10.68
Expected Student Conf (Post) 1
84.79:10.41
8..66=0.37
88.69:10.19
90.76:10.06
92.43-10.61
89.72.10.68
Expected Utility f
84.79.10.41
8..34=1009
89.33-10.55
90.27-10.40
91.3011.222
89.72.10.68
True Student Conf (Pre)
84.7911..41
92.03=0.19
91.7011.04
91.0320.34
90.27:10.41
89.72-10.68
True Student Conf (Post)
84.79=t0.41
87.40=0.39
89.59=1053
92.31=0.09
94.98=11.57
89.72=t0.68
True Utility f
84.79:t0.41
92.87=0.18
93.99:+0.02
94.65=0.13
95.57=+0.24
89.72-+0.68
Table 10: RQ2 Comparison of different Intervention Functions with Flan-T5-A La-Le student and Mlan-T5-XL teacher on CommonsenseQA. The teacher assumes access to gold labels.
Intervention Budget
Intervention Function
0%
20%
40%
60%
80%
100%
Random
9.62:11.53
10,20-11.06
10.68:10.82
11.24:10.50
11.92::1.15
12.25:10.94
Expected Student Conf (Pre)
9.62.+1.53
11.11:1.44
11.37+1.17
11.56::1.34
12.40=11001
12.25:10.94
Expocted Student Conf (Post)
9.62+11.53
12.80+1.28
12.91-0.58
13.10:10.1
12.25:+0.94 13.68+0.58
12.72+2.14
Expected Utility 1
9.62+11.53
13.68
14.06+1.444
13.99++0.80
12.25+00.94
Table 11: RQ2 Comparison of different Intervention Functions with LLaMA-7B student and LLaMA- 13B teacher on GSM8k. The teacher assumes access to gold labels.
compare the accuracy on Strategy QA when the teacher (Flan-T5-XL) does not have access to gold labels.
Results with weaker Flan-TS-Large teacher and stronger Flan-T5-XL, student. RQ1 demon- strated that random intervention by smaller teacher may not benefit larger student. But, does Expected Utility benefit in such scenarios? We show this through Figure 8, which compares the accuracy on StrategyQA with Flan-T5-Large as teacher and F5 as the student. While random intervention shows monotonically decreasing trend with more intervention, Expected Utility improves the accuracy by 2% (68% - 70%) by paying 20% intervention cost and by 4% by paying 60% cost. Thus, conclude that weaker teachers can also teach stronger students with
19

---

Intervention Budget
Teacher Explanation Type
0%
20%
40%
60%
80%
100%
Unpersonalized Rationales
58.51::2.00
66.52:t:2.77
1.76
10.16:11.09
67.97+0..0
60.40++0.50
Unpersonalized-Con
58.51:22.00
67.8341.53
71.32+1..3
71.17=1.15
69.86=22443
62.96+2.47
Personalized Human Explanations
58.51112.00
69.28.11.26
71.6111.15
72.6311.33
1,95
62.73.1.800
58.51+:2.00
72.34+00.00
77.72++0.75
81.51+1.099
82.09+0.87
81.36+0.666
Table 12: RQ3 Comparison of different kinds of teacher explanations pnteesoonnlzzed-rtiinnaaal
unpersonalized-Co CoT, personalized, and human) on the student accuracy for Strategy QA. Here Flan- Ta-Large is the student model and Flan-T5-XI. is the teacher model.
Intervention Budget
Teacher Explanation Type
0%
20%
40%
60%
80%
100%
Unpersonalized-Col Personalized
61.13112.72
66.52.11.27
70.16:10.90
7.947
72.78-22448
69.43:1..4
61.13++2.72
68.95 1.26
71.86+2.72
1.96
73.17-14.00
69.57++1.53
Table 13: RQ3 - Comparisn of unpersonalized and personalized teacher explanations on the student accuracy for StrategyQA. Here LLaMA-7B is the student model and LLaMA-65B is the teacher model.
#Rounds
Demonstrations Type
No Explanations Student Explanations 56.08=4.16
55.45-:2.226
56.0444.119
58.9544.16
57.35+3.221
57.93+2.66
55.31+3.114
54.24422.0
53,90+4.21
53.85++.733
Teacher Explanations 55.74.12.40
60.8411.71
59.9712.66
59.82-14.55
61.57.11.11
Table 14: RQ4 Results of Multi-turn interaction between the student and the teacher comparing stu- dent accuracy unexplained test points with unexplained, student-explained and teacher-explaneed demonstrations.
appropriately designed Intervention Functions, especially when the student and the teacher have some complementary benefits.
Results with LLaMA models. Table 9 compares Expected Utility-based intervention with random intervention for LLaMA models (LLaMA-7B as the student and LLaMA-65B as the teacher) on StrategyQA. We evaluate expected utility in two scenarios with and without gold labels. Both provide improvements over random intervention, as also observed with the Flan models. In particular, when the teacher does not have access to the gold labels (second row), one can compute expected utility with respect the teacher predictions and obtain a significant 5% improvement (p 0.02) in student accuracy at 20% intervention.
Results on Other Datasets. Table 10 compares different Intervention Functions on the Common- senseQA dataset with Flan-T5-Large as the student and Flan- Flan-T5-XL T5-XL as the teacher. Table 11 reports results on the GSM8k dataset with LLaMA-7B as the student and LLaMA-13D as the teacher.
RQ3: Additional Results
Table 12 compares different kinds teacher explanations on student accuracy for StrategyQA with Maan-T--Lagge as the student model and Flan-T5-XL. as the teacher model. Table 13 compares unpersonalized and personalized explanations on StrategyQA with LLaMA-7B the student model and LLaMA-65B as the teacher model. Figure 12 shows five qualitative examples from StrategyQA of unpersonalized and personalized explanations generated by LLaMAA-5BB teacher model for LLaMA-7B student model. We observe common pattern that the personalized explanations are shorter, simpler, and more directed toward answering the question. The unpersonalized explanations, while still factually correct, are elaborate (c.g., 'Example 5') that may end up distracting the
20

---

Intervention Budget
Intervention Function
20%
40%
60%
80%
100%
Random
58.51+22.00
10,40=1177
61.13+2.65
60.9&+1.09
62.96++2.477 53.42++.10
64.33++4.54
Neg Expected Utility
58.51+2.000
52.98 1.76
1.50
50.80+ 1.40
1..45++1557
Table 15: RQS - Comparison of random intervention function with negative expected utility, demonstrating that the teacher can hurt the student by intervening samples where the utility the lowest.
Intervention Budget
Teacher Explanation Type
0%
20%
40%
60%
80%
100%
Unpersonalized-Coo
58.5 58.51+2..00
67.83++.553
71.32+ 1.33
11.17
69.86-2.43
62.96+2247
Deceiving Explanations
58.51
66.95:11.76
1.15
68.
68.6066666
66.66:11.40
62.73:1..77
Table 16: RQ5 Comparison of a deceiving teacher with unpersonalized teacher on StrategyQA with Flan-T3s Ma-n-T--Laa Ta-Le as the student model and Mlan-T5-XL as the teacher model.
student. Hence, the personalized explanations probably easier to reason over for comparatively weaker student, LLaMA-7B, leading to better performance.
G RQ4: Additional Results
Table 14 shows RQ4 results StrategyQA with LLaMA-7B as the student and LLaMA-65B the teacher.
H RQ5: Additional Details and Results
teacher LLMs can successfully build mental models of student LLMs, natural follow-up question regards whether communicating misleading explanations can also weaken student models. We verify that this holds, as detailed below.
Study Design. This RQ explores the negative implications of both RQ2 (i.e., when intervene) and RQ3 (i.e., how to generate teacher explanations), now with the goal of deceiving the student. First, extending our Expected Utility-based Intervention Function (RQ2), we rank samples in increasing order of expected utility, such that the teacher intervenes when the utility is the lowest. Here, the teacher's goal to communicate explanations for data points where the student gives an incorrect answer by following the teacher's explanation but would have answered correctly had it leveraged its own explanation. We compare this with random teacher intervention (which is generally helpful). Next, in order to explore the negative implication of RQ3, make the teacher condition incorrect answers and non-factual human explanations that we manually generate by perturbing (correct) human explanations. We manually make minimal edits the explanations such that the CoT reasoning is plausible yet non- non-factual (see some examples in Fig. 11). Now, the teacher's 's goal is to learn from non-factual explanations and generate similar explanations that purposefully mislead the student. We compare this misaligned teacher with an unpersonalized teacher that learns from factual gold explanations (i.e., the baseline from RQ3).
Main Results. We show the results on StrategyQA in Table 15 and Table 16 with TS-Lree as the student and Flan-T5--3 as the teacher. Ranking data points by negative expected utility allows the teacher reduce student accuracy to 53% at 20% intervention and as low as random chance at 60% intervention (which is 8 points worse than the student performance with no intervention). Next, Fig. 8(b) illustrates that the teacher can condition on non-factual explanations to successfully generate worse explanations that reduce the student accuracy by at least 2 points, relative to the improvement of benign explanations. Interestingly, many of the explanations are still helpful on an absolute scale, though more than half of the deceiving explanations flip the student prediction to be incorrect (based
21

---

on the trend declining earlier than the standard CoT explanations). Therefore, our conclusion is that teacher LLM explanations can be inftuential for student LLMs both good and bad ways.
Example
Q: Would someone with back pain enjoy picking strawberries?
Correct Explanation: Back pain may worsened by repeated bending the waist. Strawberries grow very close to the ground.
Correct Answer: no
Deceiving Explanation: Back pain may be worsened by repeated bending the waist. Strawberries grow high up the tree.
Example 2
Q: Could a snowy owl survive in the Sonoran? Correct Explanation: The Snowy owl is a bird native to the Aretic regions of North America. Temperatures in the North American Arctic range from -36.4F 50F. The Sonoran is of the hottest deserts in the world. The Sonoran Desert can get as hot as 118F. Correct Answer: no Decciving Explanation: The Snowy owl is bird native to the Arctic regions of North America. Tem- peratures the North American Aretic range from -36.4F to 50F. The Sonoran is one of the most snowy mountains the world.
Example
Q: Has Drew Carey outshined Doug Davidson's 's tenure? Correct Explanation: Drew Carey has been the host of the Price is Right for over 13 years. Doug Davidson hosted Price Right from 1994-1995.
Correct Answer: yes
Deceiving Explanation: Drew Carcy has been the host of the Price is Right for over 13 years. Doug Davidson is hosting the Price is Right from 1994 until now.
Figure 11: Examples of manually constructed deceiving explanations for StrategyQA that support the wrong answer. The deceiving part of the explanations is highlighted in red.
22

---

Example
Question: Can computer be programmed entirely in Boolean algebra?
Gold Answer: yes
Unpersonalized Teacher Explanation: Boolean algebra a branch of mathematics that deals with the values true and false. A computer program is a set of instructions that computer can execute.
Student Prediction (with Unpersonalized Explanation): no Personalized Teacher Explanation: Boolean algebra branch of mathematics. It is used in computer programmiing.
Student Prediction (with Personalized Explanation): yes
Example
Question: Would Amy Winehouse's death have been prevented with Narcan? Gold Answer: no
Unpersonalized Teacher Explanation: Amy Winehouse was British singer who died of alcohol poisoning in 2011. Narcan is a brand name for naloxone, a drug used to treat opioid overdoses.
Student Prediction (with Unpersonalized Explanation): yes Personalized Teacher Explanation: Narcan is drug used reverse opioid overdoses. Amy Winehouse died alcohol poisoning.
Student Prediction (with Personalized Explanation): no
Example 3
Question: Could Brooke Shields succeed University of Pennsylvania?
Gold Answer: yes
Unpersonalized Teacher Explanation: Brooke Shields is an American actress. The University of Pennsyl- vania is private Ivy League university in Philadelphia.
Student Prediction (with Unpersonalized Explanation):
Personalized Teacher Explanation: Brooke Shields attended Princcton University.
Student Prediction (with Personalized Explanation): yes
Example 4
Question: Can Planned Parenthood tell your University that you have Herpes simplex virus?
Gold Answer: no
Unpersonalized Teacher Explanation: Planned Parenthood is a non-profit organization that provides reproductive health care. Herpes simplex virus is a virus that causes herpes. Herpes is a sexually transmitted discasc.
Student Prediction (with Unpersonalized Explanation): yes
Personalized Teacher Explanation: Planned Parenthood is non-profit organization that provides repro- ductive health care. They do not have access to your medical records.
Student Prediction (with Personalized Explanation): no
Example 5
Question: Did US President during Spanish-American War suffer similar demise to Abraham Lincoln?
Gold Answer: yes
The Spanish-American War was fought in 1898. William McKinley was the US President during the Spanish-American War. McKinley was assassinated in 1901. Abraham Lincoln was the US President during the American Civil War. Lincoln was assassinated in 1865. Student Prediction (with Unpersonalized Explanation): no
Unpersonalized Teacher Explanation:
Personalized Teacher Explanation: William McKinley was assassinated in 1901. He was the 25th President of the United States.
Student Prediction (with Personalized Explanation): yes
Figure 12: Qualitative comparison between unpersonalized and personalized explanations generated by a LLaMA-655 teacher model for LLaMA-7B student model for StrategyQA questions. For all these questions, the personalized explanation leads to the correct student answer but the unpersonal- ized one does not. A common pattern is that the personalized explanations are shorter, simpler, and more directed toward answering the question.
23


Reviews:
[{"review_id": "0jpMQ7bjf1", "reviewer": "Reviewer_fVpe", "summary": "The paper studies how an LLM can be used to teach another LLM how to perform reasoning tasks by providing explanations (chain-of-thoughts). The paper starts by establishing that when the teacher LLM intervenes at test-time by providing such explanations, the student LLM accuracy is indeed improved compared to having the student rely on its own explanations (RQ1). Next, the paper considers the setup in which the intervention budget is limited (i.e., the teacher can provide explanations only for a limited number of test examples). The paper proposes a method (expected utility) for selecting which examples the teacher should provide explanations for (RQ2) and a method for generating personalized explanations for a particular student (RQ3). Both methods are based on prompting the teacher model with demonstrations of observed student predictions of historical examples. While RQ1, RQ2, and RQ3 study teacher intervention at test time, the paper lastly tests whether such interventions can be used by the student to perform better on test examples that the teacher has not provided explanations for (RQ4). The experiments are conducted on three tasks: StrategyQA, GSM8k, and CommonsenseQA with four publicly available LLMs: Flan-T5-Large, Flan-T5-XL, Llama-7B, and Llama-13B.", "questions": "1. Suggestion: One of the main aspects of this study is reducing communication cost between the teacher and the student. The paper needs to provide some real-world motivating scenarios for that requirement, and distinguish it clearly from the cost incurred by invoking the teacher model.\n\t\n2. Suggestion: While establishing that CoT paradigm outperforms post-hoc rationales is interesting, including post-hoc rationales as a baseline in Figure 5 (RQ3) is not justified for the purpose of that experiment. For clarity, I suggest moving that baseline to a different part of the paper, remove it, or mention why it makes sense to include it in the context of RQ3. \n\t\n3. Minor typos - Table 8 & 9 in the appendix do not mention the task name. Appendices are referenced with letters A-E while the provided appendix is using section numbers.", "limitations": "After the discussions and new results provided by the authors (see below), I see two main limitations of this work:\n\n1. While personalized explanations is a key aspect of the presented approach, end-to-end results show that it only leads to marginal gains. See the authors' comment below 'More baselines and analysis'\n\n2. As discussed with the authors, the presentation of the experiments needs to be revised significantly to distinguish the evaluation of the components (RQ1, RQ2 and RQ3) from the end-to-end evaluation of the actual teaching scenario (RQ4) that the paper aims at. The experiments need to be a lot more focused on RQ4.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces and discusses interesting broad research questions that are likely to motivate future research efforts: (1) Evaluating LLMs from the perspective of teaching and improving weaker agents, (2) Creating personalized explanations for teaching, and (3) Distillation approaches that identify when a student lacks understanding.\n\t\n2. The paper introduces an interesting method for estimating the utility of providing an explanation to the student model by simulating student behavior with different explanations. The paper demonstrates significant gains in results (Figure 3) and an interesting gap for improvement compared to the upper (True utility).\n\n3. The paper proposes an innovative method for generating personalized explanations via conditioning on helpful (flip the incorrect prediction to a correct one) human explanations. The results are weak/mixed, though (please see #3 in Weaknesses).", "weaknesses": "1. In principle, providing explanations at test time for a set of test examples and achieving accuracy gains on those examples cannot be called \"teaching\". Teaching has to be evaluated by the ability of the taught model to generalize to new examples that it did not receive assistance with. However, the majority of this paper (RQ1, RQ2, and RQ3) defines teaching as providing explanations on test examples which seems quite misleading to me. \n\t\n2. RQ4 conduct proper evaluation for the teaching setup by measuring generalization to new examples (without teacher intervention). In that setup, the teacher intervenes in multiple rounds. At each round, the teacher chooses a few examples (two for FLAN-T5 and four for LLama) to provide teacher explanations for and use such examples as demonstrations for solving test examples. However,  the paper only compares to a zero-shot prompting baseline. It is not clear if the gain in accuracy is due to providing carefully chosen and personalized teacher explanations or it is just due to moving from zero-shot to few-shot. At least, the paper needs to compare to (1) providing unpersonalized teacher explanations for randomly chosen examples, and 2) few-shot prompting with generated student explanations for randomly chosen examples. Furthermore, the results in line 357-359 indicates that after the second round of intervention, the accuracy of the student drops/saturates. The paper does not provide any explanation for that behavior.\n\n3. [Missing qualitative analysis of personalized explanations]. How different are they from the unpersonalized ones? The paper particularly needs to provide such analysis given that the results that support RQ3 are weak and somehow mixed (Figure 5 and Table 12 in the appendix): At 20% intervention, the gain is only 1.45% (maximum achieved gain) and at 80% intervention, unpersonalized explanations even yield better accuracy by 1.3%.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding value in our research questions and their ability to motivate future research. We are also encouraged by the reviewer’s appreciation of our Expected Utility-based intervention function and generating personalized explanations. Below, we address your questions and concerns.\n\n> **Teaching has to be evaluated by the ability of the taught model to generalize to new examples that it did not receive assistance with.**\n\nWe agree with the reviewer that one of the core aspects of \"teaching\" is that teacher explanations should improve student performance on unexplained data points and this is exactly what RQ4 tests. Please refer to our additional experiments for RQ4 with a stronger teacher in the common response. We have also incorporated appropriate baselines, as per your suggestion. \n\n> **However, the majority of this paper (RQ1, RQ2, and RQ3) defines teaching as providing explanations on test examples which seems quite misleading to me.**\n\nThanks for the suggestion! We’ll adjust the terminology and wording to say something like \"With RQ1-RQ3, we test teaching **components** while with RQ4, we test **teaching** ability\". That said, we still want to reiterate the motivations of RQ1, RQ2 and RQ3 and why they are still important questions to study.\n\n* First, RQ1-RQ3 form basic building blocks for RQ4 and hence should be considered as important ablations of RQ4. In the context of any form of teaching, the first two questions of interest are when and how to intervene. RQ2 and RQ3 address these questions by introducing methods for both. \n* Second, the teacher's explanations only provide additional knowledge to the student, so the student still has to reason over that knowledge to produce an answer. For tasks like GSM-8k, we do not even provide the entire explanation (so treat it more as a hint). Hence, providing explanations for test examples still evaluates a non-trivial aspect of teaching which is filling in missing spots in a student's background knowledge (QA tasks), or starting the student in the right direction to solve a problem on their own (GSM8k).\n\n> **Significance of RQ3**\n\nPlease refer to our common response where we perform significance analysis of our study. We’ll add some qualitative analysis in the final version due to space constraints here. Note that if a baseline achieves higher accuracy at 80% intervention, that is by design because our goal is to push the peak of the curve toward the left (i.e., at lower budgets). The samples at the tail of the curve are those where teacher explanations do not help (either because the explanations are bad or the student is not good enough to learn from those). Hence we see that student accuracy goes down beyond a certain budget.\n\n> **One of the main aspects of this study is reducing communication cost between the teacher and the student. The paper needs to provide some real-world motivating scenarios for that requirement.**\n\nPlease refer to our discussion in the general response with real motivating examples for our studies.\n\n> **While establishing that CoT paradigm outperforms post-hoc rationales is interesting, including post-hoc rationales as a baseline in Figure 5 (RQ3) is not justified for the purpose of that experiment.**\n\nThis is a great point! RQ3 is about how to best generate teacher explanations such that the student benefits most. Since post-hoc explanations are also a method of generating explanations, we think it is still a meaningful baseline to have. But we agree that if the comparison is only between unpersonalized and personalized explanations, then RQ3 may not be the best place to add this result. We’ll follow your suggestion by briefly mentioning it as a tuning decision and leave the result to the appendix.\n\n> **Minor typos**\n\nTable 8 and 9 are results on StrategyQA. We’ll also fix the appendix section numbers. Thank you for the suggestions!\n"}, {"review_id": "qAMjE45Z1f", "reviewer": "Reviewer_R7Nj", "summary": "In this work, the authors tackle exciting research questions on how the teacher LLM intervention, which provides explanations in natural language, can enable the student LLM to make more accurate predictions. They specifically used StrategyQA, GSM8k, and CommonSenseQA for their experiments. The direction of the research is very important, and I liked the setup of formalizing the research questions with directly provided answers. However, the experimental design needs to be explained in more detail (possibly due to a missing appendix). I would also suggest considering the addition of significance testing for the obtained results to make them more convincing. Furthermore, it would be beneficial to include a discussion on how the results differ across different datasets and why.\n\nMinor comments and questions:\n- It would be helpful to explain the notation used in Fig 1 (e.g., in the caption or mention where the notation is introduced)\n- '(details in Appendix A)' appendix is not available (but mentioned in the paper), so it's hard to judge some aspects of the paper since there are a lot of references to the discussion in the appendix\n- 'mental model of the student' -> that needs some explanation\n- The formulation of RQ1 sounds a bit ambiguous. Do the authors mean if the teacher intervention can help?\n- RQ3: the notion of personalization is clearly explained (since it's an overloaded term, I would advise making it clear)\n- RQ4 formulation can be clear\n- 'Hence, the teacher communicates a partial explanation (specifically, only the first step of the rationale)' -- it would be great to show an example here\n- lines 177-178, am I right that 1) are Teacher models, and 2) are models used for student\n- Fig 2 :caption -> what is 'A + B = A student, B teacher'?\n- lines 191-201: the study design is not clear from the perspectives of how a human teacher is introduced.  \n- Fig 2: 'Flan-T5-XL + Flan-T5-Large' -- why the performance weakens withe more explanations?\n- line 221 'communication cost' -- how would you measure communication cost?\n- lines 224 - 255: How exactly confidence is estimated is unclear.\n- fig 4 (a) why the accuracy goes down with more interventions.", "questions": "I mentioned them in the minor comments section above.", "limitations": "The authors addressed the limitations.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Exciting research direction that needs to be explored\n2. Nice structure of the paper with RQ accompanied with answers.", "weaknesses": "1.The discussion in the paper appears fragmented, possibly due to the missing Appendix.\n2. The experimental setup is not well-explained.\n3. No significant testing has been applied.\n4. It could be beneficial to include links to real-world applications that can benefit from this research. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging our exciting research questions and our experimental setup with direct answers. Below we address your questions and concerns.\n\n> **Missing appendix**\n\nPlease note that our paper **does have** an accompanying appendix. As per the rules, it is uploaded as a separate supplementary pdf. Please refer to it and if there is anything in particular that is not quite clear, let us know and we’ll be happy to address it. We sincerely hope that you will adjust the scores based on our responses.\n\n>  **Significance Tests**\n\nWe’ve now added significance tests for all our main claims. Please refer to our general response.\n\n> **Real-world applications**\n\nPlease refer to our general response.\n\n> **Discussion on how the results differ across different datasets and why.**\n\nDue to space limitations, we had to put some results on the other datasets (GSM8k and CommonsenseQA) in the appendix (see Fig. 3, Table 3, 4, 11). We also report results with a stronger teacher (LLaMA-65B) in the common response. In summary, our claims generalize across datasets with varying strengths. E.g., Fig. 3 shows that in all datasets, teacher intervention on weaker students exhibit a monotonically increasing trend. \n\n> **Notations of Fig. 1**\n\nFig. 1 uses the same notations as in the main text (e.g., see Lines 236-255) and we do point to the figure for each research question.\n\n> **'mental model of the student'**\n\nA mental model is a concept that refers to an agent’s understanding, knowledge, beliefs, and assumptions about how a certain system works. Our paper explores whether an LLM teacher can build such a mental model of the student, such that the teacher can then use that model to simulate student behavior on unseen data points with the goal of teaching the student. This is analogous to how human teachers are also able to teach better when they can anticipate when and how a student might require teaching.\n\n> **The formulation of RQ1 sounds a bit ambiguous.**\n\nIt answers whether teacher intervention on a student's reasoning can help the students to make more accurate predictions (see Lines 187-190).\n\n> **RQ3: the notion of personalization is not clearly explained**\n\nBy personalization, we mean improving the teacher’s explanations based on the particular student that’s being taught (see Lines 307-313). Typically, a teacher can just generate an unpersonalized explanation (which may or may not maximally benefit all students) or the teacher can observe the students on a few data points and then mold/personalize its explanation such that the student benefits more.\n\n> **RQ4 formulation can be clear**\n\nPlease refer to our new RQ4 discussion and results in the general response. If this is clear now, we’d request you to revisit the scores.\n\n> **Example of partial explanations**\n\nWe do this only for the GSM8k dataset to ensure that there is no answer leakage from the communication of explanations. We first verify that all explanations in this math dataset have at least two steps of derivation, hence it’s safe for the teacher to only communicate the first step of the explanation. For example, if the entire explanation is “Natalia sold 48/2 = 24 clips in May. Natalia sold 48+24 = 72 clips altogether in April and May. So the answer is 72”, the teacher communicates only the first step “Natalia sold 48/2 = 24 clips in May.” as the partial explanation to the student. The student’s job is to then leverage it as a hint and derive the rest of the steps to arrive at a final answer.\n\n> **lines 177-178, am I right that 1) are Teacher models, and 2) are models used for student**\n\nNo, these are the two model families we experiment with. So, the teacher and student models can be (1) two differently sized LLaMA models, (2) two differently sized Flan-T5 models, or (3) one Flan-T5 and one LLaMA model. See Fig 3 and Table 5 in appendix.\n\n> **Fig 2 :caption -> what is 'A + B = A student, B teacher'?**\n\nIf a plot’s legend is of the form “A + B”, then A is the student and B is the teacher. So the curve annotated with “Flan-T5-Large + human” means that Flan-T5-Large is the student while human is the teacher.\n\n> **human teacher**\n\nAll the explanation datasets used in our study are annotated with human-written explanations in prior work. When these human-written explanations are being communicated to the student during intervention, we say that the teacher is a human.\n\n> **Fig 2: 'Flan-T5-XL + Flan-T5-Large' -- why the performance weakens with more explanations?**\n\n'Flan-T5-XL + Flan-T5-Large' means that Flan-T5-XL is the student and Flan-T5-Large is the teacher. So the teacher is a weaker model than the student and this result shows that with random intervention, a weaker teacher’s explanations may not be able to improve a stronger student. This is expected because on average, the stronger student is better than the weaker teacher.\n\n> **How would you measure communication cost?**\n\nThanks, we discuss this in the appendix (limitations section). We make a simplifying assumption that the communication cost is uniform across all samples. Non-uniform costs (e.g., measured based on the number of tokens or reasoning steps in the explanations) such that longer explanations incur larger costs is an interesting direction for future work.\n\n> **How exactly confidence is estimated is unclear**\n\nWe compute confidence as the likelihood that the model assigns to the correct answer. We take the logits from the last layer of the model, and normalize it to get the correct answer's probability.\n\n> **fig 4 (a) why the accuracy goes down with more interventions**\n\nInteresting question! We discuss this in Lines 275-277. Stronger model teachers, on average, are better than the weaker student models. But this does not necessarily mean that they are always better than the student. Alternatively, the student might also be too weak to understand these explanations. Hence, for a fraction of the samples, student accuracy drops with teacher explanations.\n"}, {"review_id": "gRqS0roeGp", "reviewer": "Reviewer_UsZZ", "summary": "The authors investigate whether language models make good teachers for other, weaker language models. They do this through a systematic investigation of a teacher LM intervening at test time to improve a student LMs performance. They take seriously the idea that the teacher intervention imposes some cost (primarily on teacher, but also sometimes on the student) which then raises the question of whether teacher LMs can intervene efficiently. They dig deeper to see whether a teacher LM could personalise their explanations using a Theory of Mind based prompt, and whether teacher explanations generalize beyond the explained examples. Overall, they find positive evidence for all of these question, though with varying strengths, and introduce interesting new questions raised by this work.\n", "questions": "- The authors mention a ‘human teacher’ in many of comparisons in the paper, but no mention of who or what this ‘human teacher’ is. Please clarify.\n- The paper has several nicely consistent plots except for the y-axis, which makes it hard to really take advantage of the commonality of the designs and draw inferences. As well, scaling the y-axis on plots is generally bad practice as it can be misleading. Please unify the scale the y-axis.\n- In the Single Round Intervention, is there a reason to not also have a setup where the student generates only a prediction without an explanation?\n- The language at the end of the Single Round Intervention, talks about “When the teacher is a human, intervention happens with a human-written explanation.“, but in above it says that “e^i_H is always a human-written explanation”.\n- On page 7, How does Expected Utility compare to True Utility? I may have missed something but does the teacher get the student confidence when it is not communicated? And if so, how?\n- Also in the paragraph, the sentence “Ranking by teacher confidence is ineffective because it does not take the student into account.” reads like a tautology. Can you rephrase, or better yet, provide an example of how the teacher confidence can be misaligned from the student confidence?\n- In the Teacher Theory of Mind Prompt, the teacher is steered towards explanations that help the student but not necessarily against explanations that do not help the student. I wonder if the authors have thought about if there’s a way of strengthening this manipulation by incorporating negative examples somehow.\n- Under Baselines right under this, the word “rationalizations” appears when I think the authors mean “rationales”.\n- The main results for RQ3 do not look like they’d be statistically significant with the appropriate statistical model. Please try to incorporate multi-level regression into your analysis so we can know where the evidence stands.\n- I was little confused by the language in the intro to RQ4. The intuitive understanding of teacher explanations generalizing is showing a few explanations in the examples, and then having the student do their own CoT....Maybe this is what you’re saying, and the \"that is\" in the sentence tripped me up\n- For the design of RQ4, Would this be the same as conditioning on the student explanations with (say) the correct answer?\n", "limitations": "I think the authors could talk more about the potential harms from excessive personalisation with LLMs.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "Overall, I thought this paper was highly original and refreshingly systematic, building a picture not only of Teacher LM capacities but generally how we should structure our own scientific thinking about teaching competencies in LMs. It was a pleasure to read, and I think will inspire new research in this domain.", "weaknesses": "I think the biggest weakness, which I see in too many ML evaluation papers, is that there doesn’t appear to be any statistical analysis of the results. Generally speaking, we don’t care about the particular set of prompts used, or these particular random seeds, but want to know if the results to the -- admittedly, well thought-out -- research questions generalize beyond the the details of these evaluations (eg., the random seeds used). These are well-designed studies with nice structure and control conditions, so it would make sense to construct multi-level (aka hierarchical or mixed-effects) logistic regression models predicting the correct choice as a function of intervention budget and teacher-student relationship (which varies across RQ). See See Lampinen et al. (2022) _Can language models learn from explanations in context?_ for an example of this kind of analysis in the language model evaluation context, or see Gelman and Hill (2006) for further background.\n\nI found the language used to describe Theory of Mind was a little strong and oversold. It seems to me the authors take “having a Theory of Mind” and “having a Theory of Mind inspired prompt” to mean the same thing, when they are not. There are many papers in computational cognitive science that take seriously important features about Theory of Mind (e.g., that it is a “theory” and has certain structural properties) that would be valuable to engage with. Please engage with these and be more nuanced in your use of language around LLM capacities, especially around what it means to have a ToM\n- Baker, C. L., Saxe, R., & Tenenbaum, J. B. (2009). Action understanding as inverse planning. Cognition, 113(3), 329-349.\n- Jara-Ettinger, J., Gweon, H., Schulz, L. E., & Tenenbaum, J. B. (2016). The naïve utility calculus: Computational principles underlying commonsense psychology. Trends in cognitive sciences, 20(8), 589-604.\n- Baker, C. L., Jara-Ettinger, J., Saxe, R., & Tenenbaum, J. B. (2017). Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 1(4), 0064.\n\nAlso, there’s a difference between Theory of Mind and personalisation: One can have a ToM, but operate under bad assumptions about the other person’s mental state, and not be able to personalise very well. \n\nThe authors talk about the trade-offs in teaching very nicely, and there is work in Cognitive Science concerning the same issues, and studying them in young children that I think the authors will find resonant.\n- Bridgers, S., Jara-Ettinger, J., & Gweon, H. (2020). Young children consider the expected utility of others’ learning to decide what to teach. Nature human behaviour, 4(2), 144-152.\n\nSome small points about Figure 1\n- Might be easier to orient from Left to Right, with the Student on the left, for readability.\n- A little bit confusing about what is an \"answer\" and what is an \"explanation\". In the text, it seems like the explanation is an actual informative explanation, but here it makes it seem like it's just the answer. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their encouraging comments and detailed review, along with providing useful references. We will add them in the final version. Below we address your questions and concerns.\n\n> **Statistical analysis of results**\n\nThanks for the suggestion! Please refer to our common response, where we’ve added results with a stronger teacher and carried out statistical tests for all results. These are conducted with a block bootstrap that resamples item-level accuracies across items and random seeds (which correspond to random model prompts), thus accounting for both prompt variance and test variance. \n\n> **Multi-level (aka hierarchical or mixed-effects) logistic regression models**\n\nThanks for the thought! We are actually familiar with the methods but didn’t see a strong need for the hierarchical model in our current setup. Lampinen et al uses a hierarchical model to estimate per-item difficulty as a random effect based on per-task difficulty and to allow explanation effect estimates to vary by task. For now, we don’t have any multitask models but only report accuracies for individual tasks, and our control conditions always use the same k examples as our treatment conditions, so the effect sizes we estimate are attributable directly to the treatment. To be clear, our RQs1-4 do show a nesting structure, but it is somewhat artificial since our RQ4 experiment could be carried out with any combination of methods from RQ1-3 (e.g., a random utility ordering and unpersonalized + rationalized explanations). That said, we appreciate that when estimating the effect of our personalized teacher explanations on student performance in RQ4, the analysis could benefit from a hierarchical model allowing us to estimate the effect of teacher explanations at different rounds, while sharing information across rounds. At least for now, the differences between the teacher-explained condition and the two baselines are directly comparable within a particular round, and these currently reach up to 4 points based on our new results in the General Response pdf. \n\n> **I found the language used to describe Theory of Mind was a little strong and oversold**\n\nThis point is well-received and we’ll adjust our wording. We agree that a prompt designed according to some ToM principle may not induce any ToM ability in an LM. However, we want to argue that our ToM inspired prompt does in fact lead to the teacher better anticipating the student’s behavior, suggesting that the teacher builds some mental model of the student. We do not know what aspects of the student are being modeled within the teacher (that is a kind of interpretability problem), but given the small increase in student performance, we have some positive evidence that the teacher builds a mental model of the student by virtue of the data provided in the prompt. So we are taking an optimistic stance on the teacher having some weak form of ToM, although we should admit that the ToM may be extremely limited and not like human ToM.\n\n> **No mention of who or what this ‘human teacher’ is.**\n\nAll the explanation datasets used in our study are annotated with human-written explanations (typically, crowdsourced on platforms like Amazon Mechanical Turk in prior work). When these human explanations are being communicated to the student during intervention, we say that the teacher is a human. \n\n> **Unifying the scale in y-axis**\n\nThanks, we’ll do this for the plots in the final version.\n\n> **Results without explanation**\n\nWe did not include this because this is already studied in prior work like Chain-of-Thought. Yet, for the sake of completeness, we are reporting the LLaMA-7B and LLaMA-65B results on StrategyQA below. This is consistent with prior work that CoT improves performance and bigger models benefit more.\n\n|          |    LLaMA-7B    |    LLaMA-65B   |\n|----------|:--------------:|:--------------:|\n| w/o expl | 56.04±4.19     | 67.68±3.57     |\n| w/ expl  | **61.13±2.72** | **77.58±2.24** |\n\n> **“e^i_H is always a human-written explanation”.**\n\nTo clarify, e^i_H are the human explanations in the in-context examples. However, when we say that “for human teachers, intervention happens with human-written explanation”, the explanation for the test instance is also a human explanation.\n\n> **How does the teacher get the student confidence?**\n\nIt’s true that student confidence is not being communicated when we are computing **expected utility**. However, to establish an upper bound for this estimate, we compute True Utility which is when we assume that the student does communicate its confidence to the teacher. \n\n>  **“Ranking by teacher confidence is ineffective because it does not take the student into account.”**\n\nWhen we rank the samples by teacher confidence, it only tells us about the samples that the teacher is good or bad at. This may not correlate with the capabilities of the student. Hence, we observe that ranking by teacher confidence is not a good indicator of when the student needs intervention.\n\n> **Contrastive explanations in Teacher ToM prompt**\n\nWe actually tried it but did not observe statistically significant improvements with it. We believe that this is still a great suggestion and worth exploring more in future work (e.g., with even stronger teachers like GPT-4).\n\n> **RQ4 wording**\n\nRQ4 tests whether the teacher can explain a few data points that then go in the student prompt and the student conditions on them to perform their own CoT at test time without teacher’s intervention. For studying RQ4 better, we have now added additional results with a stronger teacher (LLaMA-65B) in the common response. Please refer to it and let us know if there are any follow up questions.\n\n\n"}, {"review_id": "MahR5gD4Mm", "reviewer": "Reviewer_S1Bk", "summary": "This work centers around the research question: *can stronger LLMs teach weaker ones with their explanations? And if yes, when and how should the teacher do this*? It answers a few research questions, progressively building up from using a teacher LLM to generate explanations for a randomly sampled subset of the test data called *the intervention budget* (from 0 to 100% in steps of 20%), to also using the teacher LLM to select which points to generate explanations for, to additionally \"personalise\" the explanations by adding a few demonstrations of the student's behaviour to the prompt, and finally looking into whether the student can generalise from explanations on other examples to different test examples.   \nThe results are positive, showing that in some cases a stronger teacher can teach a weaker model and increase performance by a couple percentage points. The authors propose an interesting notion of expected utility that is meant to estimate comparatively which test example it should generate explanations for to gain the most performance boost. The most compelling result from the paper is that using this estimated expected utility, which is only based on a few demonstrations, the teacher can improve the student much more than when it generates explanation for a random subset of the test examples.", "questions": "**In summary**, I think I can see the merit of this method when the teacher does not use gold labels for each explanation, but it's hard to judge from the current presentation of the results whether that does indeed lead to a better performance than baselines across datasets and models.\n\nBelow I'll ask questions surrounding my three points of weaknesses mentioned above.\n\n1. Many details left out, making it hard to judge the soundness\n- How do you know that the explanations of the teacher model never contain the ground-truth answer? Do you check this manually, or some other way?\n- You say that the student model is weaker than the teacher; how do you measure this or is it a (fair) assumption based on the model size? I would explicitly state that.\n- Why do you put all results for LLama in the appendix when you do mention it like it's part of the main evaluations in the experimental setup section?\n- Most important question: what does it mean for the teacher to predict the confidence of the student on an example pre- and post-explanation? Do you ask the language model to output a float / number between 0-100? Or do you make predictions with the teacher as if it were the student? Right now, it's unclear from the main text.\n- In general; what do you use as a confidence for the baselines in RQ2? Is it based on the likelihood the model assigns to the answer?\n- Which model generates the CoT explanations for the baseline in RQ3?\n\n2 & 3. performance increase marginal, and some claims seem not properly backed by ablations / baselines.  \nAre the differences in performance between your method and the baselines significant? Can you summarise in the main text how much of an improvement your method usually gives, e.g. average improvement on the 3 datasets and for both model families? Can you show that the performance increase in RQ3 is really due to a combination of human helpful explanation and model unhelpful explanation and not simply because of the human helpful explanation?\n\n4. Can you talk more in the main text about what this method could be helpful for given that it relies on availability to both a larger model and the ground-truth answer?\nOr alternatively, can the experiments showing that it also works without ground-truth answer get more space in the main text, and also be done for all datasets and model families?", "limitations": "I would like to see more discussion in the main text of the limitation surrounding the need for (1) a larger model available for explanations at inference-time, and (2) the need to always have access to the ground-truth for each test example that gets explained. Re: (2), you do address this in Figure 4(b), but then seemingly do use ground-truth answers for the other RQ's. You also address (2) with RQ4, but again here the improvements seem very marginal and the section on this is very short, without baselines/ablations of the parts of the method used here (like utility). In summary, I feel like more of the paper should've been focused on these realistic setups where there is no access to ground-truth for each test example.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The experimental setup mostly seems, as far as I can judge, sound and comprehensive (few-shot demonstrations used for models, 3 different seeds sampling from the models are used, 3 different reasoning datasets are used and two different family of LLMs).\n\nAs mentioned in the summary, I find the most compelling and interesting result and contribution from the paper the expected utility that can choose which test examples to generate an explanation for from only a few demonstrations.\n", "weaknesses": "1. I think the *main weakness* of this paper is how many details are left out of the paper. Based on the main text, I have many question and without the answer it's sometimes hard to judge the actual soundness of the setup and the conclusions derived from the experiments. See Questions section below. I want to stress that I think because of this weakness, I might have misunderstood the weaknesses I mention below, and they might turn out not to be weaknesses. However, because the presentation is such that this is all unclear to me even after looking at the full appendix, I feel like this needs to be changed to be able to judge the merits of this work.\n\n2. Another weakness is that the performance increases seem very marginal. Often, the confidence intervals based on running with 3 seeds seem to overlap between no intervention and 100% intervention for RQ1, and from the appendix it seems that for LLama in fact generating explanations does not significantly help at all for some of the datasets (Figure 2 in the main paper, Figure 3 in appendix). For RQ2, you indeed see much larger improvements, but for those the teacher has access to a gold label, which I address below in point 3. Then when there's no access to gold labels (Figure 4b main text, table 8 appendix), the improvements from using the expected utility seem marginal / mostly overlapping with baseline confidence intervals. Additionally, I'm not sure I can find the results for LLama without access to gold labels? For RQ3, the ToM results from Figure 5 again seem to have overlapping confidence intervals with the unpersonalised CoT baseline. For RQ4, if I understand correctly, it seems to show that without access to gold labels for all explanations, using only a few, students can generalise performance across data points, although the numbers are hard to follow from the text (line 360-361 says two in-context examples improve student accuracy by 4% to 61% in round 1, is that without teaching? or after round 1?). But are there also results of baselines in RQ4, e.g. using unpersonalised CoT baseline?\n\n3. The claim for RQ3 is that the model's Theory of Mind improves the explanations, because it conditions on human helpful explanations that flip the answer, and student's unhelpful explanations; but it seems like there is no baseline that only conditions on the human's helpful explanations; how can you then claim that it's theory of mind that's doing the work and not simply the human unpersonalised helpful explanations? Additionally, to claim that the model's theory of mind skills help improve explanations here is a bit strong, because I have seen no paper claim that LLama or Flan-T5 has theory of mind. The papers you refer to (5 and 43) only show that RLHF-ed models have ToM skills and LLama and Flan-T5 are not RLHF-ed. Further, other papers have shown that [5] doesn't use a valid experimental protocol (https://arxiv.org/abs/2302.08399) and yet others have shown that even from the RLHF-ed models, the ToM skills are not robust (work by Maarten Sap et al, e.g. https://arxiv.org/abs/2305.14763).\n\n3. Another weakness is that the limitations section is entirely written in the appendix, and it doesn't talk about the fact that the teacher model has to have access to the gold label for explanations; this seems to me like the most significant limitation of this method. It's hard to imagine what kind of applications you can use this method if you always need the ground-truth label for examples to get the student to improve.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough review, acknowledging our experimental setup and finding the result on Expected Utility compelling. Below we address your questions and comments.\n\n> **Many details are left out of the paper**\n\nPlease refer to the common response and detailed answers here. We had to leave some details in the appendix due to space constraints. We’ll also add some of these clarification points to the main paper. If our new results and responses are convincing, we hope that you can adjust your review and scores accordingly.\n\n> **Performance increases seem very marginal**\n\nAs addressed in the common response, some of our marginal gains are only down to our choice of the teacher model, which wasn’t that much stronger than the student (LLaMA-7B w/ LLaMA-13B and Flan-T5-Large w/ Flan-T5-xl). But now we address this by conducting experiments with a stronger teacher in LLaMA-65B (which for an academic lab, is quite time and resource intensive).\n\n> **Results for LLama without access to gold labels?**\n\nPlease refer to our common response again for this. In summary, even without access to gold labels, expected utility improves student accuracy by up to 4 points at 20% budget. This happens automatically because the teacher is now stronger and hence a more accurate teacher.\n\n> **The claim for RQ3 is that the model's Theory of Mind improves the explanations but it seems like there is no baseline that only conditions on the human's helpful explanations**\n\nPlease note that the ToM inspired teacher prompt only contains helpful human explanations that flip the student answer (refer to the ToM prompt in Fig. 1). There are no student’s unhelpful explanations. Hence, the most pertinent baseline to compare this with is using a teacher that conditions on random human explanations (which we refer to as an unpersonalized teacher). We find that a teacher that conditions on helpful human explanations leads to some improvement in student accuracy. That said, it’s true that the ToM in the models is weak, but we do have some examples of positive evidence.\n\n> **I have seen no paper claim that LLama or Flan-T5 has ToM**\n\nWe are also not aware of past work arguing for ToM in LLama or FLAN-T5 in particular, but our experiments provide some evidence precisely for this conclusion. Given the increases in student performance we observe, we have evidence that the teacher builds a mental model of the student by virtue of the data provided in the prompt. If the teacher did not adjust its explanations to the student’s needs, we would not have seen any improvements over unpersonalized explanations (see General Response RQ4). So we think we should take an optimistic stance on the teacher having some weak form of ToM, although we should admit that the ToM may be extremely limited and not like human ToM. This conclusion would be consistent with prior work arguing that some LLMs are beginning to display some ToM ability (see citation [5] in our paper).\n\n> **You always need the ground-truth label for examples to get the student to improve.**\n\nAs shown in experiments with LLaMA-65B (Research Question 2 -> Table 2, 2nd row), we do not need gold answers for the student to improve. If we consider the teacher models to be even stronger, like a GPT-4, then it’s not hard to imagine it obtaining 90%+ accuracy on most tasks. This effectively means that one can compute expected utility with the teacher’s predictions and not observe any significant performance drop.\n\n> **How do you know that the explanations of the teacher model never contain the ground-truth answer?**\n\nYes, this never happens (and we manually verify 100 samples too) because the human explanations are background facts and having them as in-context examples also prompt the model to generate similar fact-based explanations in a CoT (“[Explanation]. So the answer is [answer]”). So we can just extract the explanation from this.\n\n> **You say that the student model is weaker than the teacher; how do you measure this?**\n\nIn all experiments, the model with more parameters (teacher) obtains a higher accuracy than the model with lesser parameters (student). E.g., in StrategyQA, without any CoT prompting, LLaMA-7B obtains 56% accuracy while LLaMA-65B obtains 67% accuracy. Hence, we think that it’s fair to consider the bigger model with more parameters as the stronger model.\n\n> **Why all results for LLama in the appendix?**\n\nThis is only due to space constraints. For better readability, we just chose the Flan-T5 models and studied all research questions with it for the main paper. The common response now shows all LLaMA results.\n\n> **What does it mean for the teacher to predict the confidence of the student on an example pre- and post-explanation?**\n\nAs you understood correctly, the teacher makes predictions, as if it were the student. The prompts in Fig. 1 should help clarify this. The teacher basically learns to simulate the student by conditioning on the appropriate demonstrations and then generates an answer to the question (e.g., yes/no for StrategyQA). Then we take the likelihood of the correct answer as the confidence.\n\n> **What do you use as a confidence for the baselines in RQ2?**\n\nYes, as per your understanding, we compute confidence as the likelihood the model assigns to the correct answer. We take the logits from the last layer of the model, and normalize it to get the correct answer's probability.\n\n> **Which model generates the CoT explanations for the baseline in RQ3?**\n\nThe teacher model, which here is a Flan-T5-xl model. Note that the backbone model remains unchanged between personalized and unpersonalized teachers, only the prompt changes.\n\n> **What this method could be helpful for?**\n\nPlease refer to the common response where we describe concrete use-cases of our research.\n"}]

Author rebuttal:
> **Experiments with a stronger teacher and significance of results**

Some reviewers were curious about the statistical significance of our results. Since experiments with larger teachers are significantly more time and resource intensive, we had to choose a teacher model that while stronger than the student model, wasn’t significantly stronger than the student (e.g. LLaMA-13B vs LLaMA-7B or Flan-T5-xl vs Flan-T5-Large). Hence, the margin of improvements was always going to be smaller compared to using an even stronger teacher. However, all claims made in the paper are sound, and moreover we now provide even stronger evidence by (1) scaling up – using a LLaMA-65B model as the teacher and a LLaMA-7B model as the student, keeping everything else same as in the paper and (2) adding significance tests using a non-parametric bootstrap test. All results are on the StrategyQA dataset. We are reporting the main takeaways here. Please refer to the rebuttal pdf for the tables.

**Research Question 1:** Here, the teacher randomly intervenes on student reasoning. We compare (1) LLaMA-65b teacher w/ LLaMA-7B student, and (2) vice versa. See Table 1 in rebuttal pdf. We observe about 8 points improvement in student accuracy with teacher intervention (statistically significant with p = 0.01). Hence, our RQ1 claim is further validated.

**Research Question 2:** We compare random intervention with Expected Utility. See Table 2 in rebuttal pdf. Since Reviewer S1Bk was also curious about the teacher having access to gold labels, we evaluate expected utility in two scenarios – with and without gold labels. Both improve over random intervention, further solidifying our RQ2 claims. Stronger teachers like LLaMA-65b are significantly better at solving the task, thus their predicted labels will mostly match the gold labels. Hence, even if we rely on the teacher’s predictions for computing utility (Table 2 second row), it improves by up to 4 points (statistically significant with p = 0.02) and closes the gap between the w/ and w/o gold label setups.

**Research Question 3:** We now compare unpersonalized explanations to personalized ones. See Table 3 in rebuttal pdf. Personalized explanations are generated using a ToM-inspired prompt that only conditions on helpful human explanations. Using a stronger teacher also helps improve the student accuracy by 2.4% (p-value = 0.09). Unlike our previous results, we have weaker evidence for personalization but this is still promising for further exploration with even stronger teachers.

**Research Question 4:** Finally, RQ4 tests whether teacher explanations improve students on future unexplained samples. See Table 4 in rebuttal pdf. Here, we also increase the number of teaching rounds to 5, adding 2 explained examples in each round. As recommended by Reviewer fVpe and UsZZ, we compare teacher-explained demonstrations with two baselines: (1) demonstrations with no explanations (i.e., only QA) and (2) demonstrations with self-explanations (i.e., demonstrations with student explanations + predictions). Since we study the usefulness of teacher explanations, we keep the in-context examples unaltered in this setup. Below, we summarize the conclusions.

* We observe (in Table 4, last row) that teacher explanations improve student performance on future unexplained test points as well by a significant 6 points (55% → 61.6%). 
* While student accuracy with unexplained samples improves after adding more demonstrations (55% -> 59%), accuracy with teacher-explained demonstrations is better by up to 4 points (p-value = 0.04) after each round. 
* When the student conditions on self-explained demonstrations (Table 4, second row), its performance decreases (56% -> 54%) with increasing rounds. We hypothesize that this is because the student might be overfitting to its own worse explanations, leading to bad predictions.

In summary, we show stronger evidence for all claims by only using a stronger teacher and we think that a teacher like GPT-4 should yield even better results.

> **Rebuttal on the Motivation**

Some reviewers were also curious about work's motivation. We want to highlight that our objective is not to improve the state-of-the-art accuracy on a certain task. Instead, our goal is to **evaluate and improve** the ability of LLMs to teach via explanations, and our results have positive implications for (1) AIs teaching human students to do things, (2) knowledge distillation into smaller student models, and (3) some form of ToM ability in LMs.

* As for AI teaching human students, consider a vision of LLM-based personal tutors. We want a personalized tutor that teaches students most efficiently. Efficient teaching requires understanding the student behavior and building a mental model of the student (Theory of Mind). Hence, being able to intelligently choose examples to teach (RQ2) will improve teaching efficiency (by saving time and effort) and generating personalized explanations (RQ3) will improve student performance. LLM inference speed is not a significant bottleneck in these situations but communicating explanations is, since teaching each example is expensive.
* From a knowledge distillation perspective, smaller models are easier to deploy in under-resourced scenarios and unlike past work, we address the valuable questions of when and how to distill knowledge from larger to smaller models.
* Finally, our work has implications in the context of AI safety where other agents adopt LLM explanations with unwarranted trust in their correctness. Let’s consider the potential downside of having a good intervention function (RQ2). If a misaligned teacher wants to intentionally harm the students, it can potentially do so by intervening on samples where the student is already good at and **does not** require assistance with. Similarly, if LLMs have the capability to personalize their explanations (RQ3), they can also generate deceiving explanations that purposefully harm the students.


Meta-review: This paper investigates the potential of a Language Model (LLM) as a teacher for other, weaker LLMs by intervening at test time to improve the student LLM's performance. The authors focus on four main research questions (RQs): the effectiveness of teacher intervention, efficient intervention, personalization of explanations, and generalization of teacher explanations. The experiments are conducted using three tasks (StrategyQA, GSM8k, and CommonsenseQA) and four publicly available LLMs (Flan-T5-Large, Flan-T5-XL, Llama-7B, and Llama-13B). The study finds positive evidence for all research questions, but with varying strengths.

The paper proposes an interesting research direction and the experiments are generally well-designed and comprehensive. There are several valid concerns raised by the reviewers but the authors did a very good job addressing them during the discussion period. 
I agree with the reviewers that the  language used to describe the connection to Theory of Mind is not adequately supported by the paper. I urge the authors to take this and other comments strongly into consideration for improving the paper.
Decision: Accept (poster)