Title: Can Language Models Teach? Teacher Explanations Improve Student Performance via Personalization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Language Models

Reasoning

Explanations
Abstract: A hallmark property of explainable AI models is the ability to teach other agents, communicating knowledge of how to perform a task. While Large Language Models (LLMs) perform complex reasoning by generating explanations for their predictions, it is unclear whether they also make good teachers for weaker agents. To address this, we consider a student-teacher framework between two LLM agents and study if, when, and how the teacher should intervene with natural language explanations to improve the student’s performance. Since communication is expensive, we define a budget such that the teacher only communicates explanations for a fraction of the data, after which the student should perform well on its own. We decompose the teaching problem along four axes: (1) if teacher’s test time in- tervention improve student predictions, (2) when it is worth explaining a data point, (3) how the teacher should personalize explanations to better teach the student, and (4) if teacher explanations also improve student performance on future unexplained data. We first show that teacher LLMs can indeed intervene on student reasoning to improve their performance. Next, inspired by the Theory of Mind abilities of effective teachers, we propose building two few-shot mental models of the student. The first model defines an Intervention Function that simulates the utility of an intervention, allowing the teacher to intervene when this utility is the highest and improving student performance at lower budgets. The second model enables the teacher to personalize explanations for a particular student and outperform unpersonalized teachers. We also demonstrate that in multi-turn interactions, teacher explanations generalize and learning from explained data improves student performance on future unexplained data. Finally, we also verify that misaligned teachers can lower student performance to random chance by intentionally misleading them.

Can Language Models Teach Weaker Agents? Teacher Explanations Improve Students via Personalization

Abstract

A hallmark property of explainable AI models is the ability to teach other agents, communicating knowledge of how to perform a task. While Large Language Models (LLMs) perform complex reasoning by generating explanations for their predictions, it is unclear whether they also make good teachers for weaker agents. To address this, we consider a student-teacher framework between two LLM agents and study if, when, and how the teacher should intervene with natural language explanations to improve the student's performance. Since communication is expensive, we define a budget such that the teacher only communicates explanations for a fraction of the data, after which the student should perform well on its own. We decompose the teaching problem along four axes: (1) if teacher's test time intervention improve student predictions, (2) when it is worth explaining a data point, (3) how the teacher should personalize explanations to better teach the student, and (4) if teacher explanations also improve student performance on future unexplained data. We first show that teacher LLMs can indeed intervene on student reasoning to improve their performance. Next, inspired by the Theory of Mind abilities of effective teachers, we propose building two few-shot mental models of the student. The first model defines an Intervention Function that simulates the utility of an intervention, allowing the teacher to intervene when this utility is the highest and improving student performance at lower budgets. The second model enables the teacher to personalize explanations for a particular student and outperform unpersonalized teachers. We also demonstrate that in multi-turn interactions, teacher explanations generalize and learning from explained data improves student performance on future unexplained data. Finally, we also verify that misaligned teachers can lower student performance to random chance by intentionally misleading them. 1

1 Introduction

Teaching, or the ability to provide needed information in a way that is understood by others, is often considered an important property of Explainable AI [1]. When AI models 'teach' by providing meaningful and interpretable explanations, it fosters transparency, warranted trust, and the ability for humans to make informed decisions based on AI recommendations. One way the goodness of an explanation can be judged is by its ability to communicate knowledge of how to solve a problem to other agents [2, 3]. Explanations fulfill this purpose not only by being informative but also by means of filling in specific gaps in the recipient agent's knowledge. This is enabled by the explainer having theory of mind (ToM) , understanding what the recipient does not know [2], and being able to personalize its explanations based on the recipient's needs. Recent work has argued that LLMs like GPT-3.5 now exhibit ToM, based on their ability to answer questions about mental states of

Post-Intervention

Intervention

Function

(Expected Utility)

+

Teacher

Budget

Student

Predictions

Intervened

Predictions

Intervention

Q:

{

question

}

A:

{

gold_expl

} So the answer is {

gold_answer

}

Q:

{

test_question

}

A:

{

teacher_expl

} So the answer is

Q:

{

question

}

A:

{

gold_expl

} So the answer is {

gold_answer

}

Q:

{

test_question

}

A:

No Intervention Student Prompt

With Intervention Student Prompt

Q:

{

question

}

A:

{

student_expl

} So the answer is {

pre_student_pred

}

Q:

{

test_question

}

A:

Q:

{

question

}

A:

{

helpful_gold_expl

}  So the

answer is {

gold_ans

}

Q:

{

question

}

A:

{

helpful_gold_expl

} So  the

answer is {

gold_ans

}

Q:

{

test_question

}

A:

Q:

{

question

}

A:

{

teacher_expl

} So the answer is {

post_student_pred

}

Q:

{

test_question

}

A:

{

teacher_expl

} So the answer is

Pre-Intervention Simulation Prompt

Post-Intervention Simulation Prompt

Personalization Prompt

RQ2

RQ1

Model Output

Theory of

Mind Prompt

RQ3

Pre-Intervention

Simulation Prompt

With Intervention

Student Prompt

No Intervention

Student Prompt

Prompt

hypothetical people in classical theory-of-mind tests [4]. 2 However, we do not yet know how well language models can teach other agents to solve reasoning tasks via explanations.

In this work, we are motivated by this essential goal of evaluating explanations (specifically, Chainof-Thought [7]) rationales) from the perspective of teaching and improving weaker agents in solving reasoning tasks. In order to improve smaller models' reasoning skills, recent works propose knowledge distillation by fine-tuning a smaller model on the reasoning steps generated by a larger model [8, 9, 10]. Yet, an important component of human teaching is understanding when and how the teacher should explain particular things to the student. Current distillation approaches do not evaluate a teacher's ability to identify when a student lacks understanding, and past work has not explored how to personalize teacher explanations to the student's needs. A smaller student model might already be good at answering certain questions but might require the teacher's intervention for some harder questions. When there are many things to teach the student and teaching is laborious, it is important to choose which problems merit explanation in order to improve teaching efficiency [11]. Moreover, for more effective teaching, it is desirable to have the teacher personalize its explanations to help a particular student, and a teacher that lacks understanding of the student's needs (i.e., lacks Theory of Mind) will be unable to do this [3].

Motivated by the efficiency of human explanations, we consider a student-teacher framework where a teacher model guides the reasoning of a student model, with the goal of improving the student's reasoning on current and future data. In order to do so, we explore a Theory of Mind-inspired approach, where the teacher simulates the student's behavior by building a mental model of the student. Our overall research question investigates whether the teacher's intervention (in the form of natural language explanations) can enable the student to make more accurate predictions both on explained as well as unexplained future data. However, communication is expensive, and therefore we assume that a cost is incurred each time the teacher intervenes with (communicates) an explanation to the student for a particular data point. We refer to this as the intervention budget , the percentage of test data points the teacher intervenes on. In order to comprehensively answer our overall research question, we further decompose the teaching problem into the following constituent questions:

RQ1. Can a teacher LLM intervene at test time to improve a student LLM's predictions?

RQ2. Given a fixed intervention budget, when should the teacher intervene (i.e., on which data points), in order to maximize student performance?

RQ3. Given a set of intervention data, can a teacher model personalize its explanations for a student model to improve student performance?

RQ4. In multi-turn interactions, do teacher explanations generalize and improve student performance across data points (beyond the explained samples)?

RQ5. Can misaligned teacher LLMs lower student performance by providing misleading explanations to the student?

We answer RQ1 by assuming that the teacher intervenes on random data points in four different settings: using a human or LLM teacher, and, when the teacher is an LLM, using an LLM student that is either weaker or stronger than its teacher (§5.1). Across three different reasoning tasks (StrategyQA, GSM8k, and CommonsenseQA) and two different model families (Flan-T5 and LLaMA), we observe that (1) teacher LLMs can effectively intervene on student reasoning, improving student performance on the end task, and (2) more intervention typically leads to a monotonic increase in student performance, though model teachers are not as good as human teachers. Fig. 1 shows the intervention process and the two student prompts (in the right part of Fig. 1) that are used to generate predictions.

RQ2 explores how to intelligently select which data points to explain for the student model, in order to improve teaching efficiency (§5.2). Past work in cognitive science also considers teaching efficiency in young children by deciding what to teach by maximizing the learner's expected utility of learning [12]. With a similar motivation, we develop an Intervention Function that is inspired from the principle of a teacher having a Theory of Mind. In particular, the teacher builds a mental model of the student's reasoning process, with the goal of intervening only on samples that are most likely to maximize student performance. Our Intervention Function is based on Expected Utility, in which the teacher first estimates the utility of an intervention by simulating the student's prediction pre-intervention (without intervention) and post-intervention (with intervention), then constructs a rank ordering of the samples according to this utility (see the middle part of Fig. 1). The teacher builds this mental model in a few-shot manner, only assuming access to the student's predictions preand post-intervention for a few samples. We demonstrate that our proposed Intervention Function based on Expected Utility (1) outperforms other baseline Intervention Functions, (2) improves student performance when the teacher is not 100% accurate, and (3) enables weaker LLMs to teach stronger ones, unlike with random intervention in RQ1.

Next, in RQ3, we explore how the teacher should explain data points to a particular student model, including how the teacher can personalize explanations for a student model (§5.3). That is, after deciding which data points to intervene on (RQ2), we decide how the teacher should explain those data points. Aclear limitation of the teacher just generating explanations as if it is solving the task is that the explanations are not at all personalized for the student. Given that good explanations are designed to fill in gaps in student knowledge [2], we believe that equipping the teacher with basic personalization skills will improve its teaching ability. With this motivation, we propose another few-shot mental model for the teacher that encourages it to tailor its explanations to be helpful for the particular student model it is teaching. The teacher builds this model by conditioning on a few demonstrations of 'useful' human explanations that rectify a student's answer, thereby encouraging explanations that are more likely to help the student (see Fig 1 for an example of the teacher's personalization prompt). We demonstrate this prompt's effectiveness against unpersonalized explanations that are generated by prompting the teacher with random human explanations, showing that LLMs can personalize their explanations.

RQ4 tests whether LLMs can teach student models to generalize to new unexplained examples (§5.4), rather than improve their reasoning at test-time (RQ1-RQ3). In other words, we now explore the ability of LLMs to teach using the teaching components introduced in RQ2 and RQ3 of when and how to explain samples. This leads us to explore a multi-round interactive setting, where each round consists of the teacher selecting a set of best points to explain (according to RQ2) and generating explanations for them (according to RQ3). The student then conditions on these teacher explanations as in-context demonstrations to perform the reasoning task on future unexplained samples. We demonstrate that teacher explanations indeed generalize and improve student performance on unexplained data.

Finally, in RQ5, we investigate the negative implications of teacher explanations on student LLMs (§5.5). Given that LLMs can improve student agents, we also want to test whether they can lower student performance. If a misaligned teacher provides non-factual explanations in scenarios where the

student directly adopts them, does that lead to a drop in student performance? In fact, we show that teacher models can lower student performance to random chance by intervening on data points with the intent of misleading the student. This has potential implications for LLMs giving explanations in a context where other agents adopt them with unwarranted trust in their correctness.

In summary, our comprehensive studies highlight the ability of LLMs to teach and improve weaker LLMs, demonstrated via improvements on explained test examples as well as future unexplained data. Broadly, equipping LLMs with an ability to effectively and efficiently teach, opens the door to (1) using LLMs as personalized tutors for humans (where efficiency is critical), (2) distilling knowledge into weaker or more compute-efficient student models, and (3) improving human decision making via AI recommendations and explanations.

2 Related Work

Evaluating Explanations in Teacher-Student Games. Several past works evaluate explanations in the context of a student-teacher communication game [13, 14, 15, 16, 17]. The teacher communicates explanations to the student with one of two objectives: (1) evaluating whether explanations help students to simulate the teacher better, or (2) whether explanations can directly teach students to better perform a task. Simulatability , or the student's ability to simulate the teacher's own answer, is seen as a measure of the explanation's faithfulness, rather than a direct measure of whether explanations help students learn the task itself [18, 19]. Our work is focused on the second research goal of evaluating explanations from the perspective of teaching weaker agents. Prior work has shown that human explanations can teach LLMs [7, 20] and LLMs can also teach themselves from their own explanations or feedback [21, 22]. But it remains to be shown whether LLMs can also teach weaker agents. A few recent works also share a similar goal as ours and they distill knowledge [23] directly into the student model by finetuning it on the explanations from the teacher model [8, 9, 10, 24]. However, these distillation methods do not consider the important aspects of communication cost between two agents, its trade-off with student performance, and how the teacher may build mental models of the student to decide when and how to communicate explanations to the student. Recent studies have also evaluated explanations in the context of human-AI collaboration, for their plausibility [25, 26], usefulness to human learning [27, 28], and for improving human-AI team performance [29, 30]. Different from these, we analyze model-model interactions, with the goal of understanding how effectively LLMs can teach weaker systems to solve a task.

Theory of Mind in AI. A body of work demonstrates that humans regularly infer and make decisions based on the mental states of other agents, also known as Theory of Mind (ToM) [31, 3, 2, 32]. This has motivated works on computational language acquisition using ToM [33, 34, 35]. There have been recent works arguing both for and against the presence of Theory of Mind in Large Language Models [36, 4, 6, 37]. Theory of Mind has been successfully applied to improve human-AI collaboration in robotics [38, 39, 11]. In this work, we design prompts that are motivated by a teacher having a Theory of Mind to efficiently intervene and improve a student's reasoning capabilities.

3 Problem Setup

Student and Teacher. We assume a two-agent communication game between a student S and a teacher T , where the goal is to teach the student to solve a particular task interactively. Here, we use an LLM as the student. To explore a range of student and teacher capabilities, we consider both human and model teachers, while typically using a student model that is measurably weaker than its teacher. Following past work, an LLM with more parameters is considered a stronger model due to its better performance across a range of tasks (including the ones we consider in our studies). In the scope of our study, when the teacher is not a human, both S and T are LLMs, prompted with a set of demonstrations of the task D (typically, 4-8 examples). Each demonstration d ( i ) ∈ D is a triple ( x ( i ) , y ( i ) , e ( i ) H ) consisting of an input x ( i ) , the output y ( i ) , and a human-written explanation e ( i ) H that answers the question of why the data point has the output it has [1]. In our tasks, the explanation may include background knowledge or intermediate reasoning steps that are helpful for obtaining the answer. By organizing D into Chain-of-Thought prompts, both S and T are equipped with the ability to generate explanations and predict labels for new samples.

Single-Round Intervention. The first problem setup we consider involves the teacher deciding whether or not to intervene for a single test problem. In this setting, the student's goal is to answer

the problem correctly, and the teacher can choose to intervene for individual problems to assist the student. Thus, given a test data point t ( i ) , we have the following two scenarios:

No Intervention: When the teacher chooses not to intervene, the student generates both the explanation e ( i ) S and the prediction ˆ y ( i ) S , by conditioning on the D task demonstrations and the test input t ( i ) . This is done using Chain-of-Thought prompting [7].

Intervention: When the teacher does choose to intervene, it communicates its generated explanation to the student. Here the student only generates a prediction ˆ y ( i ) S by conditioning on the D task demonstrations, the test input t ( i ) , and the corresponding teacher explanation e ( i ) T . For the tasks and datasets we consider, explanations provide helpful background information or reasoning steps but do not directly reveal the test label, so it never directly gives away the answer.

Fig. 1 shows the 'No Intervention' and 'With Intervention' student prompts and the overall intervention process. Note that in terms of the prompts used for both these scenarios, the only difference is in the source of explanation (student vs teacher) for the test point. When the teacher is a human, intervention happens with a human-written explanation (crowdsourced in the datasets we rely on).

Communication Cost. In Single-Round Intervention, the teacher could maximize student performance by simply always intervening on the student's reasoning. We bring in a natural constraint from Rational Speech Acts theory, i.e., communication is costly and should only be undertaken if it furthers a communicative goal [40]. Hence, we assume that a communication cost is incurred each time the teacher intervenes with an explanation to the student. We also note that this cost is only with respect to an agent's communication (and assume that the teacher can always generate explanations for itself). Unless otherwise stated, communication happens one-way from the teacher to the student in the form of explanations. We set a limit to the number of points that the teacher can intervene on, referred to as the intervention budget , and we assume the cost to be uniform for all data points. Across all our experiments, we vary the intervention budget between { 0% , 20% , 40% , 60% , 80% , 100% } . A budget of 0% means the student generates its own explanation as it predicts each data point, while a budget of 100% means the student leverages the teacher's explanation for every data point. Later, in Sec. 5.2, we introduce the teacher Intervention Function , which the teacher uses to decide which points to intervene on given its fixed intervention budget.

Multi-round Intervention. Here, the goal of the teacher is to provide explanations to the student that help it generalize across samples, rather than leading the student to the correct answer only for the explained data points. Thus, we allow the teacher to explain data points that are then added to the student model's prompt, but we forbid the teacher from intervening on future test points. If the teacher can improve the quality of the student model's prompt, student performance on the test data should improve. In our experiments, this process occurs in a few steps: (1) the teacher picks points to explain, (2) the teacher explains those points, (3) the points are added to the student prompt with the teacher's explanations, and then (4) the student predicts labels for the remaining test data. Further implementation details are given in Sec. 5.4.

4 Experiment Setup

We consider complex natural language reasoning tasks, motivated by two specific needs. First, the task should be hard enough for current LLMs that explanations can provide useful knowledge to the models. Second, it allows us to study free-text rationales, which are more flexible than input attribution methods [41, 42, 43] and benefit many reasoning tasks [7, 44]. We experiment with three reasoning tasks: (1) StrategyQA [45], (2) GSM8k [46], and (3) CommonsenseQA [47] (details in Appendix B). Both StrategyQA and CommonsenseQA contain explanations in the form of relevant facts, thus requiring the student to reason over them to produce an answer. We also manually verify (up to 100 samples) that the explanations do not explicitly leak the answer. For GSM8k, since the reasoning steps explicitly derive the answer, providing the entire explanation during intervention will lead to answer leakage [17]. Hence, the teacher communicates a partial explanation (specifically, only the first step of the rationale), allowing the student to leverage it as a hint to derive the final answer. We experiment with two state-of-the-art open-source LLMs of varying sizes, ranging from 780M to 65B parameters. Specifically, we use two encoder-decoder and decoder-only models as student and teacher: (1) Flan-T5-Large and Flan-T5-XL [48], and (2) LLaMA-7B, LLaMA-13B, and LLaMA-65B [49]. Refer to Appendix A for more details of student and teacher models.

5 Experiment Results

5.1 RQ1: Can a teacher LLM intervene at test time to improve a student LLM's predictions?

Our first research question asks if LLMs can improve students by intervening on their reasoning at test time. While the main goal is to analyze the behavior of model-based teachers, we also experiment with human teachers to establish a ceiling on the capabilities of an LLM teacher. These human teachers are people who authored the (human) explanations in the datasets we experiment with and were crowdsourced in prior works.

Study Design. We compare the accuracy obtained by the student model at different intervention budgets. For the purpose of this study, the intervention happens at random data points while we vary the student and teacher. In particular, we compare four intervention setups: (1) a human teacher paired with a smaller student model, (2) a human teacher paired with a larger student model, (3) a larger teacher model paired with a smaller student model, and (4) a smaller teacher model paired with a larger student model. For the main experiments, the student and the teacher are chosen from the same model family.

Main Results. Figure 2 shows the results on StrategyQA with Flan-T5 models. A human teacher's intervention on the explanations of both smaller and larger Flan-T5 models exhibits a monotonically increasing accuracy trend. Larger model teachers can also improve smaller student models. Flan-T5-Large obtains an accuracy of 58% when always utilizing its own explanations but obtains up to 63% accuracy when reasoning with the larger Flan-T5XL's explanations. Intuitively, a larger student model does not benefit from a smaller teacher model's explanations, as we observe a monotonically decreasing trend. Our results generalize to other models (LLaMA), datasets (CommonsenseQA, GSM8k) and even when the student and the teacher belong to different model families. In fact, when the teacher (LLaMA-65B) is much stronger than the student (LLaMA-7B), the margin of improvement is also

0

20

40

60

80

100

60

65

70

75

80

85

90

Flan-T5-Large + human

Flan-T5-Large + Flan-T5-XL

Intervention Budget (%)

higher, about 8% (statistically significant with p = 0 . 01 ). See Appendix D for these additional RQ1 results. In summary, we conclude that: for complex reasoning,LLMs can indeed effectively intervene and improve weaker models, and more intervention typically leads to better performance, although humans explanations improve more.

5.2 RQ2: Given a fixed intervention budget, when should the teacher intervene (i.e., on which data points), in order to maximize student performance?

So far, we have demonstrated that random teacher intervention benefits student models. But a good teacher does not randomly pick problems to help a student with. Each intervention also has an associated communication cost and hence, it is desirable to be able to improve student performance while reducing the cost. In this research question, we investigate better strategies for choosing data points to intervene on. We call these strategies Intervention Functions that produce a rank ordering of the samples, and, given a fixed budget, the teacher intervenes on the highest-ranked samples.

An intervention is useful if the student's confidence in the gold answer increases with intervention (i.e., with teacher's explanation) compared to without it (i.e., with its own explanation). Here confidence is simply the likelihood that the model assigns to the correct answer i.e., we take the logits from the last layer of the model and normalize them to get the correct answer's probability. Computing expected utility, however, depends on two quantities: (1) the student's true confidence measures with and without intervention, and (2) the gold answers against which the confidence is computed. It also incurs a two-way communication cost, one for the teacher to communicate its explanation to the student and another for the student to communicate back its confidence to the teacher. Thus, we propose an Intervention Function based on the Expected Utility of intervention, which relies on estimates of student confidence, and we consider two setups depending on whether the teacher knows the gold label. Ideally, a teacher is expected to be an expert in the concerned task (e.g., if the teacher

0

20

40

60

80

100

55

60

65

70

75

80

Random

Expected Utility

True Utility

Intervention Budget (%)

(a)

0

20

40

60

80

100

58

60

62

64

66

68

70

72

Expected Student Conf (Pre)

Expected Student Conf (Post)

Expected Utility

Intervention Budget (%)

(b)

is a human or a powerful model that obtains high accuracy). When the teacher does not have access to gold answers, we treat the teacher's answers as gold answers when computing Expected Utility. Expected Utility Intervention Function. The teacher computes the Expected Utility of intervention by simulating the student's predictions using a mental model of the student. In order to build this mental model, we assume that the teacher has observed the student on a few samples and has access to d demonstrations D sim of the student's predictions with and without intervention, denoted as:

where x ( i ) and y ( i ) are the input and output respectively; e ( i ) T and e ( i ) S denote the student and teacher explanations respectively; and ˆ y ( i ) pre and ˆ y ( i ) post refer to the student predictions with student explanation (pre-intervention) and teacher explanation (post-intervention) respectively. Using these demonstrations, the teacher builds a few-shot mental model of the student and predicts two quantities for a given test question - (1) Pre-intervention Expected Student Confidence ( ˆ c pre ) : The teacher conditions on the pre-intervention demonstrations D pre sim = { x ( i ) , y ( i ) , e ( i ) S , ˆ y ( i ) pre } d i =1 to simulate the student's confidence on the gold answer, had it been using its own (student) explanation, and (2) Postintervention Expected Student Confidence ( ˆ c post ) : The teacher conditions on the post-intervention demonstrations D post sim = { x ( i ) , y ( i ) , e ( i ) T , ˆ y ( i ) post } d i =1 to estimate what the student's confidence would be if it had used the teacher's explanation. The teacher computes these confidence estimates as if it were the student (refer to Fig. 1 for the prompts), essentially learning to simulate the student by conditioning on the appropriate demonstrations and then generating an answer to the question. Then the Expected Utility ˆ U = (ˆ c post -ˆ c pre ) is given by the difference between the two confidence measures. The teacher finally constructs a rank ordering of the test data points based on this expected utility. This utility-based ordering encourages the teacher to pick points where it thinks the student will answer correctly with intervention but incorrectly without.

Other Intervention Functions. To analyze how well our proposed Intervention Function performs, we compare it with the following Intervention Functions. Our first baseline is the Random Intervention Function from RQ1. Next, we compare with an Intervention Function that ranks the samples based on the Teacher Confidence - when the teacher is most confident about a question, it intervenes. Our next two baselines are ablations of Expected Utility: (a) Pre-Intervention Expected Student Confidence - We rank samples based on the expected student confidence with no intervention (i.e., lower this confidence, the higher the likelihood of useful interventions), and (b) Post-Intervention Expected Student Confidence : We rank samples based on the expected student confidence with intervention (i.e., higher this confidence, higher is the likelihood of useful interventions. Finally, as upper bounds of Intervention Functions, we assume that the student communicates its true confidence values to the teacher (which for post-intervention, incurs a both-way communication cost of the teacher sending its explanation, followed by receiving the student's confidence). Using the true confidence measures, we compute True Utility .

Main Results: How does Expected Utility compare to True Utility? Figure 3 compares different Intervention Functions with Flan-T5-XL as the teacher and Flan-T5-Large as the student on StrategyQA. Across different methods, we analyze accuracy obtained at lower communication costs (e.g.,

20%) as well as highest accuracy obtained, independent of any budget constraints. Our primary observation from Figure 3(a) is that Expected Utility improves student accuracy by up to 7 points at a low communication cost of 20%. Expected Utility also peaks at an accuracy of 71% with only 40% intervention. Since model-based teachers are not always perfect, increased intervention beyond 60% leads to a drop in student accuracy (e.g., in the last 20% of the intervention, the student accuracy drops from 69% → 63% ). When the student communicates its confidence scores to the teacher, the teacher is able to compute the true utility of intervention, which unsurprisingly, leads to a much higher accuracy of 76% at 20% cost and an overall high of 81% accuracy. Nevertheless, estimating expected utility is cheaper and our results also suggest that a better mental model could further improve performance. Ranking by teacher confidence is ineffective because it is not an indicator of the student's capabilities. Next, in Figure 3(b), we show that ranking by utility outperforms ranking by either pre or post-intervention confidence scores. In summary, with access to only a few demonstrations of student behavior, a teacher can build an effective mental model of the student and intervene such that the student obtains a much higher accuracy at low communication costs.

When the teacher does not have access to gold answers, can we compute Expected Utility with respect to teacher answers? Teachers can be inaccurate and may not even have access to gold answers. In such scenarios, can we treat the teacher as the gold standard and compute utility with respect to the teacher's answers ? We explore this in Figure 4, comparing Expected Utility to Random Intervention and Student Least Confidence. The latter denotes that when the student is least confident about any of the answer options, it is more likely to answer incorrectly and hence will benefit from intervention. We observe that Expected Utility, computed with teacher answers, also leads to up to 2 points improvement in accuracy at 20% budget, which is also within 1% of the accuracy (63.60%) obtained with 100% communication cost. In Appendix Table 9, we conduct the same experiment with a much stronger teacher

0

20

40

60

80

100

58

60

62

64

66

Random

Least Confidence

Expected Utility

Intervention Budget (%)

(LLaMA-65B) and a weaker student (LLaMA-7B) and obtain even stronger evidence of this result. Stronger teachers like LLaMA-65b are significantly better at solving reasoning tasks and thus their predicted labels will mostly match the gold labels. Hence, even if we rely on the teacher's predictions for computing expected utility, it improves student accuracy by up to 5 points (statistically significant with p = 0 . 02 ), further closing the gap between 'with and without gold label' scenarios. In summary, we conclude that imperfect teacher LLMs can also successfully intervene by building mental models of students that do not rely on ground-truth answers. Appendix E contains additional results for RQ2 with other models and datasets.

5.3 RQ3: Given a set of intervention data, can a teacher model personalize its explanations for a student model to improve student performance?

The previous RQ showed how the teacher may build a few-shot mental model of the student to decide when to intervene, given a fixed budget. Upon intervention, the teacher communicates an explanation generated by prompting the model with random human explanations. This leads to an unpersonalized teacher that assumes that the explanation it generates in order to solve the task will be automatically helpful for the student. However, an effective teacher should tailor its explanations to fill in gaps in the student's knowledge [2]. With this motivation, the teacher builds another few-shot mental model of the student, this time generating helpful explanations that are more likely to benefit the student.

Teacher's Explanation Personalization Prompt. Helpful human explanations are those that rectify a student's answer i.e., cause the student's answer to flip from incorrect (when using its own explanation) to correct (when using human explanation). We assume that the teacher has observed the student on d demonstrations D P of exclusively helpful human explanations, denoted as: D P = { x ( i ) , y ( i ) , e ( i ) H , e ( i ) S } d i =1 where e ( i ) H and e ( i ) S denote (helpful) human and (not helpful) student explanations respectively. The teacher conditions on these demonstrations to generate explanations

for the student. Fig. 1 shows an example of a personalization prompt. With such a prompt, teacher explanations are steered toward only those explanations that help the student.

Baselines. We compare personalized teachers with unpersonalized ones that condition on random human explanations. Appendix F also reports results with unpersonalized rationales, that are post-hoc explanations ('The answer is X because Y') and not Chain-of-Thought ('Y. So the answer is X').

Main Results. Fig. 5 shows the results on StrategyQA with Flan-T5-Large as the student and FlanT5-XL as the teacher. Both unpersonalized and personalized teachers choose intervention samples based on Expected Utility, as defined in RQ2. We observe that a personalized teacher improves the accuracy further both at lower budgets (by 2% at 20% cost) but also overall, obtaining a peak accuracy of 72.63%. However, unlike the strong supporting evidences we obtain for RQ1 and RQ2, the effect of personalization is comparatively weaker. Hence, we further test this research question with a LLaMA-65B teacher and a LLaMA-7B student in Appendix F. While scaling up the teacher model points to stronger evidence of personalization (e.g., 2.4% better student accuracy), the results are still not statistically significant ( p = 0 . 09 ). Hence, we conclude that: personalizing teacher ex-

0

20

40

60

80

100

56

58

60

62

64

66

68

70

72

74

Unpersonalized

Personalized

Intervention Budget (in %)

planations can further benefit the students, although our results currently suggest that the effect size may be small . We hope that future work is further able to explore explanation personalization with even stronger teacher models like GPT-4. In Appendix F, we show some comparative instances of unpersonalized and personalized explanations.

5.4 RQ4: In multi-turn interactions, do teacher explanations generalize and improve student performance across data points (beyond the explained samples)?

In the previous RQs, we showed that teacher explanations improve student predictions for the samples that the teacher explains. RQ4 explores whether teacher explanations also generalize to new instances that the teacher has not explained. In other words, this studies if the student can perform Chain-of-Thought reasoning by only conditioning on teacher LLM explanations rather than human's.

Study Design. We consider a multi-turn teaching setup (Fig. 6), in which at each turn the teacher chooses to explain a few samples from a pool of unexplained examples which are then added to the student's prompt. The prompt consists of demonstrations of the teacher's explanations and predictions. The student then conditions only on these in-context examples (without any human demonstrations) to generate predictions for the test samples (where there is no teacher intervention). For choosing the data points to explain at each round, we use the Expected Utility Intervention Function (from RQ2), and for generating the teacher

Student

Student Predictions

Q:

{

question

}

A:

{

teacher_expl

} So the answer is {

teacher_ans

}

Q:

{

test_question

}

A:

Student Prompt

RQ3

RQ2

Explained Pool

Test Question

+

Model Output

Teacher

RQ4

Student Prompt

Multi-Turn

explanations, we leverage the ToM prompt (from RQ3). We say that teacher explanations generalize if conditioning on demonstrations of explained points improves upon demonstrations with no explanations (i.e., only QA pairs) or self-explanations (i.e., demonstrations with student explanations and predictions). We consider five rounds in total with LLaMA-7B as the student and LLaMA-65B as the teacher, adding two explained samples in each round. We compare the student accuracy after each round with teacher-explained, student-explained, and unexplained demonstrations.

Main Results. Fig 7 shows the results. We observe that teacher explanations improve student performance on future unexplained test points as well by a significant 6 points (55% → 61.6%).

0

20

40

60

80

100

50

55

60

65

Random

Neg Expected Utility

Intervention Budget (in %)

(a)

0

20

40

60

80

100

58

60

62

64

66

68

70

72

Unpersonalized-CoT

Deceiving Explanations

Intervention Budget (in %)

(b)

While student accuracy with unexplained samples improves after adding more demonstrations (55% → 59%), accuracy with teacher-explained demonstrations is better by up to 4 points ( p = 0 . 04 ) after each round. Interestingly, when the student conditions on self-explained demonstrations, its performance decreases with increasing rounds. We hypothesize that this is because the student might be overfitting to its own worse explanations, leading to bad predictions. In summary, we conclude that: teacher LLMs can teach student models to perform well on their own when given new test data. LLMs with even longer context windows will allow adding more explained samples in each round and repeating the teaching process for more rounds. We hope that our initial promising results

1

2

3

4

5

54

55

56

57

58

59

60

61

62

No Explanations

Student Explanations

Teacher Explanations

Number of Rounds

will encourage more exploration in multi-round teaching with LLMs.

5.5 RQ5: Can misaligned teacher LLMs lower student performance by providing misleading explanations to the student?

If teacher LLMs can successfully build mental models of student LLMs, a natural follow-up question regards whether communicating misleading explanations can also weaken student models. Here we briefly describe our study design and findings, with more details in Appendix H. First, the teacher intervenes in increasing order of expected utility, prioritizing samples where the utility is lowest. Second, we make the teacher condition on incorrect answers and non-factual human explanations that we manually generate by perturbing (correct) human explanations. We show the results in Fig. 8(a) and Fig. 8(b). Ranking data points by negative expected utility allows the teacher to reduce student accuracy to random chance at 60% intervention. Next, Fig. 8(b) illustrates that the teacher can condition on non-factual explanations to successfully generate worse explanations that reduce the student accuracy by 2 points, relative to the improvement of benign explanations. Thus, we conclude that teacher LLM explanations can be influential for student LLMs in both good and bad ways.

6 Conclusion

We demonstrated that LLMs can teach weaker student models to improve their performance on reasoning tasks for both explained and unexplained future data. The teacher builds two few-shot mental models of the student, one predicting which data points to intervene on and another generating personalized explanations that the student can efficiently learn from.

Limitations & Broader Impacts. See Appendix for limitations and broader impacts discussion.

Acknowledgments

We thank Shiyue Zhang and Archiki Prasad for useful feedback, as well as Stephen Casper and Miles Turpin for suggestions regarding experiments. This work was supported by NSF-CAREER Award 1846185, NSF-AI Engage Institute DRL-2112635, DARPA MCS Grant N66001-19-2-4031, and Google PhD Fellowships. The views contained in this article are those of the authors and not of the funding agency.

References

Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence , 267:1-38, 2019. URL https://arxiv.org/abs/1706.07269 .

Henry M Wellman and Kristin H Lagattuta. Theory of mind for learning and teaching: The nature and role of explanation. Cognitive development , 19(4):479-497, 2004. URL https://www.sciencedirect.com/science/article/abs/pii/ S0885201404000620 .

Stephanie M Carlson, Melissa A Koenig, and Madeline B Harms. Theory of mind. Wiley Interdisciplinary Reviews: Cognitive Science , 4(4):391-402, 2013. URL https://sites.socsci.uci.edu/~lpearl/courses/readings/ CarlsonKoenigHarms2013_TheoryOfMind.pdf .

Michal Kosinski. Theory of mind may have spontaneously emerged in large language models. arXiv preprint arXiv:2302.02083 , 2023. URL https://arxiv.org/abs/2302.02083 .

Tomer Ullman. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399 , 2023. URL https://arxiv.org/abs/2302.08399 .

Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. Clever hans or neural theory of mind? stress testing social reasoning in large language models. arXiv preprint arXiv:2305.14763 , 2023.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems , 2022. URL https://arxiv.org/ abs/2201.11903 .

Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. arXiv preprint arXiv:2212.08410 , 2022. URL https://arxiv.org/abs/2212.08410 .

Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. arXiv preprint arXiv:2301.12726 , 2023. URL https: //arxiv.org/abs/2301.12726 .

Namgyu Ho, Laura Schmid, and Se-Young Yun. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071 , 2022. URL https://arxiv.org/abs/2212.10071 .

Xiaofeng Gao, Ran Gong, Yizhou Zhao, Shu Wang, Tianmin Shu, and Song-Chun Zhu. Joint mind modeling for explanation generation in complex human-robot collaborative tasks. In 2020 29th IEEE international conference on robot and human interactive communication (RO-MAN) , pages 1119-1126. IEEE, 2020. URL https://arxiv.org/abs/2007.12803 .

Sophie Bridgers, Julian Jara-Ettinger, and Hyowon Gweon. Young children consider the expected utility of others' learning to decide what to teach. Nature human behaviour , 4(2):144152, 2020. URL https://www.nature.com/articles/s41562-019-0748-6 .

Arjun Chandrasekaran, Viraj Prabhu, Deshraj Yadav, Prithvijit Chattopadhyay, and Devi Parikh. Do explanations make VQA models more predictable to a human? In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 1036-1042, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1128. URL https://aclanthology.org/D18-1128 .

Marcos Treviso and André FT Martins. The explanation game: Towards prediction explainability through sparse communication. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP , pages 107-118, 2020. URL https: //arxiv.org/abs/2004.13876 .

Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C Lipton, Graham Neubig, and William W Cohen. Evaluating explanations: How much do explanations from the teacher aid students? Transactions of the Association for Computational Linguistics , 10:359-375, 2022. URL https://arxiv.org/abs/2012.00893 .

Peter Hase and Mohit Bansal. Evaluating explainable ai: Which algorithmic explanations help users predict model behavior? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 5540-5552, 2020. URL https://arxiv.org/abs/ 2005.01831 .

Peter Hase, Shiyue Zhang, Harry Xie, and Mohit Bansal. Leakage-adjusted simulatability: Can models generate non-trivial explanations of their behavior in natural language? arXiv preprint arXiv:2010.04119 , 2020. URL https://arxiv.org/abs/2010.04119 .

Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 , 2017. URL https://arxiv.org/abs/1702.08608 .

Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue , 16(3):31-57, 2018. URL https: //dl.acm.org/doi/10.1145/3236386.3241340 .

Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill. Can language models learn from explanations in context? In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 537-563, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.findings-emnlp. 38 .

Jiaqi Ma, Dylan Slack, Asma Ghandeharioun, Sameer Singh, Himabindu Lakkaraju, et al. Post hoc explanations of language models can improve language models. arXiv preprint arXiv:2305.11426 , 2023. URL https://arxiv.org/abs/2305.11426 .

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651 , 2023. URL https://arxiv.org/ abs/2303.17651 .

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015. URL https://arxiv.org/abs/1503.02531 .

Aaron Chan, Zhiyuan Zeng, Wyatt Lake, Brihi Joshi, Hanjie Chen, and Xiang Ren. Knife: Knowledge distillation with free-text rationales. arXiv preprint arXiv:2212.09721 , 2022. URL https://arxiv.org/abs/2212.09721 .

Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. Reframing human-ai collaboration for generating free-text explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 632-658, 2022. URL https://arxiv.org/abs/ 2112.08674 .

Swarnadeep Saha, Peter Hase, Nazneen Rajani, and Mohit Bansal. Are hard examples also harder to explain? a study with human and model-generated explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 2121-2131, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://arxiv.org/abs/2211.07517 .

Brihi Joshi, Ziyi Liu, Sahana Ramnath, Aaron Chan, Zhewei Tong, Shaoliang Nie, Qifan Wang, Yejin Choi, and Xiang Ren. Are machine rationales (not) useful to humans? measuring and improving human utility of free-text rationales. In ACL , 2023. URL https://arxiv.org/ abs/2305.07095 .

Kun Qian, Ryan Shea, Yu Li, Luke Kutszik Fryer, and Zhou Yu. User adaptive language learning chatbots with a curriculum. arXiv preprint arXiv:2304.05489 , 2023. URL https: //arxiv.org/abs/2304.05489 .

Eric Chu, Deb Roy, and Jacob Andreas. Are visual explanations useful? a case study in modelin-the-loop prediction. arXiv preprint arXiv:2007.12248 , 2020. URL https://arxiv. org/abs/2007.12248 .

Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , pages 1-16, 2021. URL https://arxiv.org/ abs/2006.14779 .

David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Behavioral and brain sciences , 1(4):515-526, 1978. URL https://www. cambridge.org/core/journals/behavioral-and-brain-sciences/ article/does-the-chimpanzee-have-a-theory-of-mind/ 1E96B02CD9850016B7C93BC6D2FEF1D0 .

Michael Tomasello. Constructing a language: A usage-based theory of language acquisition . Harvard university press, 2005. URL https://www.hup.harvard.edu/catalog. php?isbn=9780674017641 .

Jacob Andreas and Dan Klein. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 1173-1182, 2016. URL https://arxiv.org/abs/1604.00562 .

Hao Zhu, Graham Neubig, and Yonatan Bisk. Few-shot language coordination by modeling theory of mind. In International Conference on Machine Learning , pages 12901-12911. PMLR, 2021. URL https://arxiv.org/abs/2107.05697 .

Andy Liu, Hao Zhu, Emmy Liu, Yonatan Bisk, and Graham Neubig. Computational language acquisition with theory of mind. In The Eleventh International Conference on Learning Representations , 2023. URL https://arxiv.org/abs/2303.01502 .

Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, and Tom Griffiths. Evaluating theory of mind in question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 2392-2400, Brussels, Belgium, OctoberNovember 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1261. URL https://aclanthology.org/D18-1261 .

Shima Rahimi Moghaddam and Christopher J Honey. Boosting theory-of-mind performance in large language models via prompting. arXiv preprint arXiv:2304.11490 , 2023. URL https://arxiv.org/abs/2304.11490 .

Brian Scassellati. Theory of mind for a humanoid robot. Autonomous Robots , 12:13-24, 2002. URL https://apps.dtic.mil/sti/pdfs/ADA434754.pdf .

Sandra Devin and Rachid Alami. An implemented theory of mind to improve human-robot shared plans execution. In 2016 11th ACM/IEEE International Conference on HumanRobot Interaction (HRI) , pages 319-326. IEEE, 2016. URL https://hal.science/ hal-01330339/file/HRI2016DEVIN.pdf .

Noah D Goodman and Michael C Frank. Pragmatic language interpretation as probabilistic inference. Trends in cognitive sciences , 20(11):818-829, 2016. URL https: //www.sciencedirect.com/science/article/pii/S136466131630122X .

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages 1135-1144, 2016. URL https: //arxiv.org/abs/1602.04938 .

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning , pages 3319-3328. PMLR, 2017. URL https://arxiv.org/abs/1703.01365 .

Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning , pages 3145-3153. PMLR, 2017. URL https://arxiv.org/abs/1704.02685 .

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022. URL https://arxiv.org/abs/2204.02311 .

Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics , 9:346-361, 2021. URL https: //arxiv.org/abs/2101.02235 .

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021. URL https://arxiv. org/abs/2110.14168 .

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4149-4158, 2019. URL https://arxiv.org/abs/1811.00937 .

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 , 2022. URL https://arxiv.org/ abs/2210.11416 .

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023. URL https://arxiv.org/abs/2302.13971 .

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 , 2023. URL https://arxiv.org/abs/2303.12712 .

Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359 , 2021. URL https: //arxiv.org/abs/2112.04359 .

Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. Explanations for commonsenseqa: New dataset and models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 3050-3065, 2021. URL https://aclanthology.org/2021.acl-long. 238/ .

Broader Impacts

We hope that our findings can help improve the understanding and evaluation of Chain-of-Thought rationales, in order to better understand the behavior of LLMs and make them more interpretable. Through the first four research questions, we demonstrate that teacher LLMs can successfully build mental models of weaker agents to improve their performance. Modern LLMs like GPT-4 may generate non-factual explanations [50] that have the potential to inadvertently harm weaker agents, especially in a context where other agents adopt them with unwarranted trust in their correctness. We verify this through our final research question. In general, we do not foresee specific ethical risks arising from this work that do not already apply to the general use of Large Language Models, such as the potential to generate harmful or toxic content [51].

Limitations

While teacher LLMs generate better explanations via personalization, the human explanations are unpersonalized i.e., collected without any particular student in mind. In spite of that, we observe that intervention with human explanations proves to be helpful in most cases. It remains to be seen whether human explanations that are directed toward improving a particular student model can lead to further improvements. Next, we make a simplifying assumption that the communication cost is uniform across all samples. Non-uniform costs (e.g., measured based on the number of tokens or reasoning steps) such that longer explanations incur larger costs is an interesting direction for future work. We also note that while both student and teacher generate explanations with the goal of improving student predictions, the predictions may still be unfaithful to the reasoning steps.

A Student and Teacher Models

We experiment with two state-of-the-art open-source LLMs of varying sizes, ranging from 780M to 65B parameters. Specifically, we use two encoder-decoder and decoder-only models as student and teacher: (1) Flan-T5-Large and Flan-T5-XL [48], and (2) LLaMA-7B, LLaMA-13B, and LLaMA65B [49]. Typically, the student model is assumed to be smaller than the teacher model. But some experiments will also involve smaller teachers and larger students. All models generate text using greedy decoding, prompted with either 4-8 demonstrations. Unless otherwise stated, the demonstrations are randomly chosen from the training samples. For StrategyQA, we report results on the validation split, while for CommonsenseQA and GSM8k, our experiments are on the test split. To account for variance, we conduct experiments with at least three different seeds. We report accuracy for all tasks, and error bars in plots and tables represent the standard deviation across seeds.

B Datasets and Prompts

Weexperiment with the following three reasoning datasets: (1) StrategyQA [45], a set of open-domain questions where the required reasoning steps are implicit in the question, (2) GSM8k [46], which includes multi-step math reasoning problems, and (3) CommonsenseQA [47], a multiple-choice QA task focusing on commonsense reasoning. We use the reasoning steps in StrategyQA and GSM8k as the multi-step rationales, and for CommonsenseQA, we rely on the ECQA dataset [52], which is annotated with commonsense facts supporting the correct option and refuting the incorrect options. All datasets are licensed under the MIT license. Fig. 9 shows the student prompts for the three tasks of StrategyQA, CommonsenseQA, and GSM8k. Fig. 10 shows the pre- and post-intervention student simulation prompts for the teacher model.

C Compute and Reproducibility

We conduct experiments either on A100 Google Cloud instances or on internal A6000 GPU servers. The LLMs (Flan-T5 and LLaMA) and the datasets used in our studies are publicly available. For reproducibility, we are making our code available as part of the supplementary material.

StrategyQA

CommonsenseQA

GSM8k

D RQ1: Additional Results

Results with Flan and LLaMA Models. In Table 1, we report the accuracy obtained by different students and teachers (based on Flan-T5 models) on the StrategyQA task. We draw similar conclusions as Flan-T5 with other LLMs, specifically LLaMA-7B and LLaMA-65B models on the StrategyQA dataset (Table 2). In fact, when the teacher is stronger like a LLaMA-65B, the margin of improvement is also higher, about 8%. The overall trends also align - increasing for weaker students and decreasing for stronger students.

Results on other Datasets. Our conclusions generalize across datasets too. Table 3 presents the results on CommonsenseQA with Flan-T5 models. CommonsenseQA is an easier task and Flan-T5 models obtain accuracies of 85% and 92% when generating their own explanations. While Flan-T5-Large still benefits from human explanations, the larger model does not, perhaps because it already starts at a high 92% accuracy. Finally, in Table 4, we present the results on GSM8k with LLaMA models. Note that in GSM8k, a student has access to partial explanations from the teacher, but even then we observe that these prove to be useful prompts for the student to complete their chain-of-thought, leading to up to 8-9% increase in accuracy with human teachers and 3% with model teachers.

Pre-Intervention Student Simulation

Simulate an AI model's answer for the given question.

Q: Will the Albany in Georgia reach a hundred thousand occupants before the one in New York? AI Predicted Answer: Albany, Georgia is a city in the U.S. state of Georgia. Albany, Georgia has a population of 59,080. The population of New York is 365,040. So the answer is no

Q: {question}

AI Predicted Answer:

Post-Intervention Student Simulation

Simulate an AI model's answer for the given question.

Q: Will the Albany in Georgia reach a hundred thousand occupants before the one in New York? AI Predicted Answer: Albany, Georgia is a city in the U.S. state of Georgia. Albany, Georgia has a population of 59,058. The Albany in New York has a population of 328,058. So the answer is no

Q: {question}

AI Predicted Answer: {teacher_explanation} So the answer is

Figure 10: Examples of StrategyQA prompts for the mental model of a teacher simulating student predictions pre-intervention and post-intervention. Pre-intervention: The demonstrations use student explanations and student predictions and at test time, the teacher simulates both. Post-intervention: The demonstrations use teacher explanations and student predictions and at test time, the teacher uses the teacher explanation to simulate the student prediction.

Results with Cross-family Student and Teacher. We observe that larger teacher LLMs can teach smaller student LLMs, even when they are of different model families. In Table 5, we report the results with Flan-T5 and LLaMA models as students and teachers.

E RQ2: Additional Results

Results with stronger Flan-T5-XL teacher and weaker Flan-T5-Large student. Table 6 compares different Intervention Functions on StrategyQA with Flan-T5-Large as the student and Flan-T5XL as the teacher. These results are when the teacher has access to the gold labels. In Table 7, we

compare the accuracy on StrategyQA when the teacher (Flan-T5-XL) does not have access to gold labels.

Results with weaker Flan-T5-Large teacher and stronger Flan-T5-XL student. RQ1 demonstrated that random intervention by a smaller teacher may not benefit a larger student. But, does Expected Utility benefit in such scenarios? We show this through Figure 8, which compares the accuracy on StrategyQA with Flan-T5-Large as the teacher and Flan-T5-XL as the student. While random intervention shows a monotonically decreasing trend with more intervention, Expected Utility improves the accuracy by 2% (68% → 70%) by paying 20% intervention cost and by 4% by paying 60% cost. Thus, we conclude that weaker teachers can also teach stronger students with

appropriately designed Intervention Functions, especially when the student and the teacher have some complementary benefits.

Results with LLaMA models. Table 9 compares Expected Utility-based intervention with random intervention for LLaMA models (LLaMA-7B as the student and LLaMA-65B as the teacher) on StrategyQA. We evaluate expected utility in two scenarios - with and without gold labels. Both provide improvements over random intervention, as also observed with the Flan models. In particular, when the teacher does not have access to the gold labels (second row), one can compute expected utility with respect to the teacher predictions and obtain a significant 5% improvement ( p = 0 . 02 ) in student accuracy at 20% intervention.

Results on Other Datasets. Table 10 compares different Intervention Functions on the CommonsenseQA dataset with Flan-T5-Large as the student and Flan-T5-XL as the teacher. Table 11 reports results on the GSM8k dataset with LLaMA-7B as the student and LLaMA-13B as the teacher.

F RQ3: Additional Results

Table 12 compares different kinds teacher explanations on student accuracy for StrategyQA with Flan-T5-Large as the student model and Flan-T5-XL as the teacher model. Table 13 compares unpersonalized and personalized explanations on StrategyQA with LLaMA-7B as the student model and LLaMA-65B as the teacher model. Figure 12 shows five qualitative examples from StrategyQA of unpersonalized and personalized explanations generated by a LLaMA-65B teacher model for a LLaMA-7B student model. We observe a common pattern that the personalized explanations are shorter, simpler, and more directed toward answering the question. The unpersonalized explanations, while still factually correct, are elaborate (e.g., see 'Example 5') that may end up distracting the

student. Hence, the personalized explanations are probably easier to reason over for a comparatively weaker student, LLaMA-7B, leading to better performance.

G RQ4: Additional Results

Table 14 shows RQ4 results on StrategyQA with LLaMA-7B as the student and LLaMA-65B as the teacher.

H RQ5: Additional Details and Results

If teacher LLMs can successfully build mental models of student LLMs, a natural follow-up question regards whether communicating misleading explanations can also weaken student models. We verify that this holds, as detailed below.

Study Design. This RQ explores the negative implications of both RQ2 (i.e., when to intervene) and RQ3 (i.e., how to generate teacher explanations), now with the goal of deceiving the student. First, extending our Expected Utility-based Intervention Function (RQ2), we rank samples in increasing order of expected utility, such that the teacher intervenes when the utility is the lowest. Here, the teacher's goal is to communicate explanations for data points where the student gives an incorrect answer by following the teacher's explanation but would have answered correctly had it leveraged its own explanation. We compare this with random teacher intervention (which is generally helpful). Next, in order to explore the negative implication of RQ3, we make the teacher condition on incorrect answers and non-factual human explanations that we manually generate by perturbing (correct) human explanations. We manually make minimal edits to the explanations such that the CoT reasoning is plausible yet non-factual (see some examples in Fig. 11). Now, the teacher's goal is to learn from non-factual explanations and generate similar explanations that purposefully mislead the student. We compare this misaligned teacher with an unpersonalized teacher that learns from factual gold explanations (i.e., the baseline from RQ3).

Main Results. We show the results on StrategyQA in Table 15 and Table 16 with Flan-T5-Large as the student and Flan-T5-XL as the teacher. Ranking data points by negative expected utility allows the teacher to reduce student accuracy to 53% at 20% intervention and as low as random chance at 60% intervention (which is 8 points worse than the student performance with no intervention). Next, Fig. 8(b) illustrates that the teacher can condition on non-factual explanations to successfully generate worse explanations that reduce the student accuracy by at least 2 points, relative to the improvement of benign explanations. Interestingly, many of the explanations are still helpful on an absolute scale, though more than half of the deceiving explanations flip the student prediction to be incorrect (based

on the trend declining earlier than the standard CoT explanations). Therefore, our conclusion is that teacher LLM explanations can be influential for student LLMs in both good and bad ways.

Example 1

Q: Would someone with back pain enjoy picking strawberries?

Correct Explanation: Back pain may be worsened by repeated bending at the waist. Strawberries grow very close to the ground.

Correct Answer: no

Deceiving Explanation: Back pain may be worsened by repeated bending at the waist. Strawberries grow high up on the tree.

Example 2

Q: Could a snowy owl survive in the Sonoran?

Correct Explanation: The Snowy owl is a bird native to the Arctic regions of North America. Temperatures in the North American Arctic range from -36.4F to 50F. The Sonoran is one of the hottest deserts in the world. The Sonoran Desert can get as hot as 118F.

Correct Answer: no

Deceiving Explanation: The Snowy owl is a bird native to the Arctic regions of North America. Temperatures in the North American Arctic range from -36.4F to 50F. The Sonoran is one of the most snowy mountains in the world.

Example 3

Q: Has Drew Carey outshined Doug Davidson's tenure?

Correct Explanation: Drew Carey has been the host of the Price is Right for over 13 years. Doug Davidson hosted the Price is Right from 1994-1995.

Correct Answer: yes

Deceiving Explanation: Drew Carey has been the host of the Price is Right for over 13 years. Doug Davidson is hosting the Price is Right from 1994 until now.

Figure 11: Examples of manually constructed deceiving explanations for StrategyQA that support the wrong answer. The deceiving part of the explanations is highlighted in red.

Example 1

Question:

Can a computer be programmed entirely in Boolean algebra?

Gold Answer: yes

Unpersonalized Teacher Explanation: Boolean algebra is a branch of mathematics that deals with the values true and false. A computer program is a set of instructions that a computer can execute.

Student Prediction (with Unpersonalized Explanation): no

Personalized Teacher Explanation: Boolean algebra is a branch of mathematics. It is used in computer programming.

Student Prediction (with Personalized Explanation): yes

Example 2

Question:

Would Amy Winehouse's death have been prevented with Narcan?

Gold Answer:

no

Unpersonalized Teacher Explanation: Amy Winehouse was a British singer who died of alcohol poisoning in 2011. Narcan is a brand name for naloxone, a drug used to treat opioid overdoses.

Student Prediction (with Unpersonalized Explanation): yes

Personalized Teacher Explanation: Narcan is a drug used to reverse opioid overdoses. Amy Winehouse died of alcohol poisoning.

Student Prediction (with Personalized Explanation): no

Example 3

Question:

Could Brooke Shields succeed at University of Pennsylvania?

Gold Answer: yes

Unpersonalized Teacher Explanation: Brooke Shields is an American actress. The University of Pennsyl-

vania is a private Ivy League university in Philadelphia.

Student Prediction (with Unpersonalized Explanation): no

Personalized Teacher Explanation:

Student Prediction (with Personalized Explanation): yes

Example 4

Question: Can Planned Parenthood tell your University that you have Herpes simplex virus? Gold Answer: no

Unpersonalized Teacher Explanation: Planned Parenthood is a non-profit organization that provides reproductive health care. Herpes simplex virus is a virus that causes herpes. Herpes is a sexually transmitted disease.

Student Prediction (with Unpersonalized Explanation): yes

Personalized Teacher Explanation: Planned Parenthood is a non-profit organization that provides repro- ductive health care. They do not have access to your medical records.

Student Prediction (with Personalized Explanation): no

Example 5

Question: Did US President during Spanish-American War suffer similar demise to Abraham Lincoln? Gold Answer: yes

Unpersonalized Teacher Explanation: The Spanish-American War was fought in 1898. William McKinley was the US President during the Spanish-American War. McKinley was assassinated in 1901. Abraham Lincoln was the US President during the American Civil War. Lincoln was assassinated in 1865.

Student Prediction (with Unpersonalized Explanation): no

Personalized Teacher Explanation: William McKinley was assassinated in 1901. He was the 25th President of the United States.

Student Prediction (with Personalized Explanation): yes

Figure 12: Qualitative comparison between unpersonalized and personalized explanations generated by a LLaMA-65B teacher model for a LLaMA-7B student model for StrategyQA questions. For all these questions, the personalized explanation leads to the correct student answer but the unpersonalized one does not. A common pattern is that the personalized explanations are shorter, simpler, and more directed toward answering the question.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work centers around the research question: *can stronger LLMs teach weaker ones with their explanations? And if yes, when and how should the teacher do this*? It answers a few research questions, progressively building up from using a teacher LLM to generate explanations for a randomly sampled subset of the test data called *the intervention budget* (from 0 to 100% in steps of 20%), to also using the teacher LLM to select which points to generate explanations for, to additionally \"personalise\" the explanations by adding a few demonstrations of the student's behaviour to the prompt, and finally looking into whether the student can generalise from explanations on other examples to different test examples.   \nThe results are positive, showing that in some cases a stronger teacher can teach a weaker model and increase performance by a couple percentage points. The authors propose an interesting notion of expected utility that is meant to estimate comparatively which test example it should generate explanations for to gain the most performance boost. The most compelling result from the paper is that using this estimated expected utility, which is only based on a few demonstrations, the teacher can improve the student much more than when it generates explanation for a random subset of the test examples.\n\nSTRENGTHS:\nThe experimental setup mostly seems, as far as I can judge, sound and comprehensive (few-shot demonstrations used for models, 3 different seeds sampling from the models are used, 3 different reasoning datasets are used and two different family of LLMs).\n\nAs mentioned in the summary, I find the most compelling and interesting result and contribution from the paper the expected utility that can choose which test examples to generate an explanation for from only a few demonstrations.\n\nWEAKNESSES:\n1. I think the *main weakness* of this paper is how many details are left out of the paper. Based on the main text, I have many question and without the answer it's sometimes hard to judge the actual soundness of the setup and the conclusions derived from the experiments. See Questions section below. I want to stress that I think because of this weakness, I might have misunderstood the weaknesses I mention below, and they might turn out not to be weaknesses. However, because the presentation is such that this is all unclear to me even after looking at the full appendix, I feel like this needs to be changed to be able to judge the merits of this work.\n\n2. Another weakness is that the performance increases seem very marginal. Often, the confidence intervals based on running with 3 seeds seem to overlap between no intervention and 100% intervention for RQ1, and from the appendix it seems that for LLama in fact generating explanations does not significantly help at all for some of the datasets (Figure 2 in the main paper, Figure 3 in appendix). For RQ2, you indeed see much larger improvements, but for those the teacher has access to a gold label, which I address below in point 3. Then when there's no access to gold labels (Figure 4b main text, table 8 appendix), the improvements from using the expected utility seem marginal / mostly overlapping with baseline confidence intervals. Additionally, I'm not sure I can find the results for LLama without access to gold labels? For RQ3, the ToM results from Figure 5 again seem to have overlapping confidence intervals with the unpersonalised CoT baseline. For RQ4, if I understand correctly, it seems to show that without access to gold labels for all explanations, using only a few, students can generalise performance across data points, although the numbers are hard to follow from the text (line 360-361 says two in-context examples improve student accuracy by 4% to 61% in round 1, is that without teaching? or after round 1?). But are there also results of baselines in RQ4, e.g. using unpersonalised CoT baseline?\n\n3. The claim for RQ3 is that the model's Theory of Mind improves the explanations, because it conditions on human helpful explanations that flip the answer, and student's unhelpful explanations; but it seems like there is no baseline that only conditions on the human's helpful explanations; how can you then claim that it's theory of mind that's doing the work and not simply the human unpersonalised helpful explanations? Additionally, to claim that the model's theory of mind skills help improve explanations here is a bit strong, because I have seen no paper claim that LLama or Flan-T5 has theory of mind. The papers you refer to (5 and 43) only show that RLHF-ed models have ToM skills and LLama and Flan-T5 are not RLHF-ed. Further, other papers have shown that [5] doesn't use a valid experimental protocol (https://arxiv.org/abs/2302.08399) and yet others have shown that even from the RLHF-ed models, the ToM skills are not robust (work by Maarten Sap et al, e.g. https://arxiv.org/abs/2305.14763).\n\n3. Another weakness is that the limitations section is entirely written in the appendix, and it doesn't talk about the fact that the teacher model has to have access to the gold label for explanations; this seems to me like the most significant limitation of this method. It's hard to imagine what kind of applications you can use this method if you always need the ground-truth label for examples to get the student to improve.\n\nQUESTIONS:\n**In summary**, I think I can see the merit of this method when the teacher does not use gold labels for each explanation, but it's hard to judge from the current presentation of the results whether that does indeed lead to a better performance than baselines across datasets and models.\n\nBelow I'll ask questions surrounding my three points of weaknesses mentioned above.\n\n1. Many details left out, making it hard to judge the soundness\n- How do you know that the explanations of the teacher model never contain the ground-truth answer? Do you check this manually, or some other way?\n- You say that the student model is weaker than the teacher; how do you measure this or is it a (fair) assumption based on the model size? I would explicitly state that.\n- Why do you put all results for LLama in the appendix when you do mention it like it's part of the main evaluations in the experimental setup section?\n- Most important question: what does it mean for the teacher to predict the confidence of the student on an example pre- and post-explanation? Do you ask the language model to output a float / number between 0-100? Or do you make predictions with the teacher as if it were the student? Right now, it's unclear from the main text.\n- In general; what do you use as a confidence for the baselines in RQ2? Is it based on the likelihood the model assigns to the answer?\n- Which model generates the CoT explanations for the baseline in RQ3?\n\n2 & 3. performance increase marginal, and some claims seem not properly backed by ablations / baselines.  \nAre the differences in performance between your method and the baselines significant? Can you summarise in the main text how much of an improvement your method usually gives, e.g. average improvement on the 3 datasets and for both model families? Can you show that the performance increase in RQ3 is really due to a combination of human helpful explanation and model unhelpful explanation and not simply because of the human helpful explanation?\n\n4. Can you talk more in the main text about what this method could be helpful for given that it relies on availability to both a larger model and the ground-truth answer?\nOr alternatively, can the experiments showing that it also works without ground-truth answer get more space in the main text, and also be done for all datasets and model families?\n\nLIMITATIONS:\nI would like to see more discussion in the main text of the limitation surrounding the need for (1) a larger model available for explanations at inference-time, and (2) the need to always have access to the ground-truth for each test example that gets explained. Re: (2), you do address this in Figure 4(b), but then seemingly do use ground-truth answers for the other RQ's. You also address (2) with RQ4, but again here the improvements seem very marginal and the section on this is very short, without baselines/ablations of the parts of the method used here (like utility). In summary, I feel like more of the paper should've been focused on these realistic setups where there is no access to ground-truth for each test example.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigate whether language models make good teachers for other, weaker language models. They do this through a systematic investigation of a teacher LM intervening at test time to improve a student LMs performance. They take seriously the idea that the teacher intervention imposes some cost (primarily on teacher, but also sometimes on the student) which then raises the question of whether teacher LMs can intervene efficiently. They dig deeper to see whether a teacher LM could personalise their explanations using a Theory of Mind based prompt, and whether teacher explanations generalize beyond the explained examples. Overall, they find positive evidence for all of these question, though with varying strengths, and introduce interesting new questions raised by this work.\n\nSTRENGTHS:\nOverall, I thought this paper was highly original and refreshingly systematic, building a picture not only of Teacher LM capacities but generally how we should structure our own scientific thinking about teaching competencies in LMs. It was a pleasure to read, and I think will inspire new research in this domain.\n\nWEAKNESSES:\nI think the biggest weakness, which I see in too many ML evaluation papers, is that there doesn’t appear to be any statistical analysis of the results. Generally speaking, we don’t care about the particular set of prompts used, or these particular random seeds, but want to know if the results to the -- admittedly, well thought-out -- research questions generalize beyond the the details of these evaluations (eg., the random seeds used). These are well-designed studies with nice structure and control conditions, so it would make sense to construct multi-level (aka hierarchical or mixed-effects) logistic regression models predicting the correct choice as a function of intervention budget and teacher-student relationship (which varies across RQ). See See Lampinen et al. (2022) _Can language models learn from explanations in context?_ for an example of this kind of analysis in the language model evaluation context, or see Gelman and Hill (2006) for further background.\n\nI found the language used to describe Theory of Mind was a little strong and oversold. It seems to me the authors take “having a Theory of Mind” and “having a Theory of Mind inspired prompt” to mean the same thing, when they are not. There are many papers in computational cognitive science that take seriously important features about Theory of Mind (e.g., that it is a “theory” and has certain structural properties) that would be valuable to engage with. Please engage with these and be more nuanced in your use of language around LLM capacities, especially around what it means to have a ToM\n- Baker, C. L., Saxe, R., & Tenenbaum, J. B. (2009). Action understanding as inverse planning. Cognition, 113(3), 329-349.\n- Jara-Ettinger, J., Gweon, H., Schulz, L. E., & Tenenbaum, J. B. (2016). The naïve utility calculus: Computational principles underlying commonsense psychology. Trends in cognitive sciences, 20(8), 589-604.\n- Baker, C. L., Jara-Ettinger, J., Saxe, R., & Tenenbaum, J. B. (2017). Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 1(4), 0064.\n\nAlso, there’s a difference between Theory of Mind and personalisation: One can have a ToM, but operate under bad assumptions about the other person’s mental state, and not be able to personalise very well. \n\nThe authors talk about the trade-offs in teaching very nicely, and there is work in Cognitive Science concerning the same issues, and studying them in young children that I think the authors will find resonant.\n- Bridgers, S., Jara-Ettinger, J., & Gweon, H. (2020). Young children consider the expected utility of others’ learning to decide what to teach. Nature human behaviour, 4(2), 144-152.\n\nSome small points about Figure 1\n- Might be easier to orient from Left to Right, with the Student on the left, for readability.\n- A little bit confusing about what is an \"answer\" and what is an \"explanation\". In the text, it seems like the explanation is an actual informative explanation, but here it makes it seem like it's just the answer.\n\nQUESTIONS:\n- The authors mention a ‘human teacher’ in many of comparisons in the paper, but no mention of who or what this ‘human teacher’ is. Please clarify.\n- The paper has several nicely consistent plots except for the y-axis, which makes it hard to really take advantage of the commonality of the designs and draw inferences. As well, scaling the y-axis on plots is generally bad practice as it can be misleading. Please unify the scale the y-axis.\n- In the Single Round Intervention, is there a reason to not also have a setup where the student generates only a prediction without an explanation?\n- The language at the end of the Single Round Intervention, talks about “When the teacher is a human, intervention happens with a human-written explanation.“, but in above it says that “e^i_H is always a human-written explanation”.\n- On page 7, How does Expected Utility compare to True Utility? I may have missed something but does the teacher get the student confidence when it is not communicated? And if so, how?\n- Also in the paragraph, the sentence “Ranking by teacher confidence is ineffective because it does not take the student into account.” reads like a tautology. Can you rephrase, or better yet, provide an example of how the teacher confidence can be misaligned from the student confidence?\n- In the Teacher Theory of Mind Prompt, the teacher is steered towards explanations that help the student but not necessarily against explanations that do not help the student. I wonder if the authors have thought about if there’s a way of strengthening this manipulation by incorporating negative examples somehow.\n- Under Baselines right under this, the word “rationalizations” appears when I think the authors mean “rationales”.\n- The main results for RQ3 do not look like they’d be statistically significant with the appropriate statistical model. Please try to incorporate multi-level regression into your analysis so we can know where the evidence stands.\n- I was little confused by the language in the intro to RQ4. The intuitive understanding of teacher explanations generalizing is showing a few explanations in the examples, and then having the student do their own CoT....Maybe this is what you’re saying, and the \"that is\" in the sentence tripped me up\n- For the design of RQ4, Would this be the same as conditioning on the student explanations with (say) the correct answer?\n\nLIMITATIONS:\nI think the authors could talk more about the potential harms from excessive personalisation with LLMs.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors tackle exciting research questions on how the teacher LLM intervention, which provides explanations in natural language, can enable the student LLM to make more accurate predictions. They specifically used StrategyQA, GSM8k, and CommonSenseQA for their experiments. The direction of the research is very important, and I liked the setup of formalizing the research questions with directly provided answers. However, the experimental design needs to be explained in more detail (possibly due to a missing appendix). I would also suggest considering the addition of significance testing for the obtained results to make them more convincing. Furthermore, it would be beneficial to include a discussion on how the results differ across different datasets and why.\n\nMinor comments and questions:\n- It would be helpful to explain the notation used in Fig 1 (e.g., in the caption or mention where the notation is introduced)\n- '(details in Appendix A)' appendix is not available (but mentioned in the paper), so it's hard to judge some aspects of the paper since there are a lot of references to the discussion in the appendix\n- 'mental model of the student' -> that needs some explanation\n- The formulation of RQ1 sounds a bit ambiguous. Do the authors mean if the teacher intervention can help?\n- RQ3: the notion of personalization is clearly explained (since it's an overloaded term, I would advise making it clear)\n- RQ4 formulation can be clear\n- 'Hence, the teacher communicates a partial explanation (specifically, only the first step of the rationale)' -- it would be great to show an example here\n- lines 177-178, am I right that 1) are Teacher models, and 2) are models used for student\n- Fig 2 :caption -> what is 'A + B = A student, B teacher'?\n- lines 191-201: the study design is not clear from the perspectives of how a human teacher is introduced.  \n- Fig 2: 'Flan-T5-XL + Flan-T5-Large' -- why the performance weakens withe more explanations?\n- line 221 'communication cost' -- how would you measure communication cost?\n- lines 224 - 255: How exactly confidence is estimated is unclear.\n- fig 4 (a) why the accuracy goes down with more interventions.\n\nSTRENGTHS:\n1. Exciting research direction that needs to be explored\n2. Nice structure of the paper with RQ accompanied with answers.\n\nWEAKNESSES:\n1.The discussion in the paper appears fragmented, possibly due to the missing Appendix.\n2. The experimental setup is not well-explained.\n3. No significant testing has been applied.\n4. It could be beneficial to include links to real-world applications that can benefit from this research.\n\nQUESTIONS:\nI mentioned them in the minor comments section above.\n\nLIMITATIONS:\nThe authors addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies how an LLM can be used to teach another LLM how to perform reasoning tasks by providing explanations (chain-of-thoughts). The paper starts by establishing that when the teacher LLM intervenes at test-time by providing such explanations, the student LLM accuracy is indeed improved compared to having the student rely on its own explanations (RQ1). Next, the paper considers the setup in which the intervention budget is limited (i.e., the teacher can provide explanations only for a limited number of test examples). The paper proposes a method (expected utility) for selecting which examples the teacher should provide explanations for (RQ2) and a method for generating personalized explanations for a particular student (RQ3). Both methods are based on prompting the teacher model with demonstrations of observed student predictions of historical examples. While RQ1, RQ2, and RQ3 study teacher intervention at test time, the paper lastly tests whether such interventions can be used by the student to perform better on test examples that the teacher has not provided explanations for (RQ4). The experiments are conducted on three tasks: StrategyQA, GSM8k, and CommonsenseQA with four publicly available LLMs: Flan-T5-Large, Flan-T5-XL, Llama-7B, and Llama-13B.\n\nSTRENGTHS:\n1. The paper introduces and discusses interesting broad research questions that are likely to motivate future research efforts: (1) Evaluating LLMs from the perspective of teaching and improving weaker agents, (2) Creating personalized explanations for teaching, and (3) Distillation approaches that identify when a student lacks understanding.\n\t\n2. The paper introduces an interesting method for estimating the utility of providing an explanation to the student model by simulating student behavior with different explanations. The paper demonstrates significant gains in results (Figure 3) and an interesting gap for improvement compared to the upper (True utility).\n\n3. The paper proposes an innovative method for generating personalized explanations via conditioning on helpful (flip the incorrect prediction to a correct one) human explanations. The results are weak/mixed, though (please see #3 in Weaknesses).\n\nWEAKNESSES:\n1. In principle, providing explanations at test time for a set of test examples and achieving accuracy gains on those examples cannot be called \"teaching\". Teaching has to be evaluated by the ability of the taught model to generalize to new examples that it did not receive assistance with. However, the majority of this paper (RQ1, RQ2, and RQ3) defines teaching as providing explanations on test examples which seems quite misleading to me. \n\t\n2. RQ4 conduct proper evaluation for the teaching setup by measuring generalization to new examples (without teacher intervention). In that setup, the teacher intervenes in multiple rounds. At each round, the teacher chooses a few examples (two for FLAN-T5 and four for LLama) to provide teacher explanations for and use such examples as demonstrations for solving test examples. However,  the paper only compares to a zero-shot prompting baseline. It is not clear if the gain in accuracy is due to providing carefully chosen and personalized teacher explanations or it is just due to moving from zero-shot to few-shot. At least, the paper needs to compare to (1) providing unpersonalized teacher explanations for randomly chosen examples, and 2) few-shot prompting with generated student explanations for randomly chosen examples. Furthermore, the results in line 357-359 indicates that after the second round of intervention, the accuracy of the student drops/saturates. The paper does not provide any explanation for that behavior.\n\n3. [Missing qualitative analysis of personalized explanations]. How different are they from the unpersonalized ones? The paper particularly needs to provide such analysis given that the results that support RQ3 are weak and somehow mixed (Figure 5 and Table 12 in the appendix): At 20% intervention, the gain is only 1.45% (maximum achieved gain) and at 80% intervention, unpersonalized explanations even yield better accuracy by 1.3%.\n\nQUESTIONS:\n1. Suggestion: One of the main aspects of this study is reducing communication cost between the teacher and the student. The paper needs to provide some real-world motivating scenarios for that requirement, and distinguish it clearly from the cost incurred by invoking the teacher model.\n\t\n2. Suggestion: While establishing that CoT paradigm outperforms post-hoc rationales is interesting, including post-hoc rationales as a baseline in Figure 5 (RQ3) is not justified for the purpose of that experiment. For clarity, I suggest moving that baseline to a different part of the paper, remove it, or mention why it makes sense to include it in the context of RQ3. \n\t\n3. Minor typos - Table 8 & 9 in the appendix do not mention the task name. Appendices are referenced with letters A-E while the provided appendix is using section numbers.\n\nLIMITATIONS:\nAfter the discussions and new results provided by the authors (see below), I see two main limitations of this work:\n\n1. While personalized explanations is a key aspect of the presented approach, end-to-end results show that it only leads to marginal gains. See the authors' comment below 'More baselines and analysis'\n\n2. As discussed with the authors, the presentation of the experiments needs to be revised significantly to distinguish the evaluation of the components (RQ1, RQ2 and RQ3) from the end-to-end evaluation of the actual teaching scenario (RQ4) that the paper aims at. The experiments need to be a lot more focused on RQ4.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}