TPD: Enhancing Student Language Model Reasoning via Principle Discovery and Guidance
Authors: Haorui Wang
Rongzhi Zhang
Yinghao Li
Lingkai Kong
Yuchen Zhuang
Xiusi Chen
Chao Zhang
Venue: COLM (2024)
Keywords: Principle discovery,Teacher-student framework,Reasoning
TL;DR: We introduce a teacher-student framework TPD that enhances the effectiveness of teacher models in educating student models.

Abstract: Large Language Models (LLMs) have recently showcased remarkable reasoning abilities. However, larger models often surpass their smaller counterparts in reasoning tasks, posing the challenge of effectively transferring these capabilities from larger models. Existing approaches heavily rely on extensive fine-tuning data or continuous interactions with a superior teacher LLM during inference. We introduce a principle-based teacher-student framework called Teaching via Principle Discovery (TPD) to address these limitations. Inspired by human learning mechanisms, TPD mimics the interaction between a teacher and a student using a principle-based approach. The teacher LLM generates problem-solving instructions and corrective principles based on the student LLM's errors. These principles guide the refinement of instructions and the selection of instructive examples from a validation set. This enables the student model to learn from both the teacher's guidance and its own mistakes. Once the student model begins making inferences, TPD requires no further intervention from the teacher LLM. Through extensive experiments across eight reasoning tasks, we demonstrate the effectiveness of TPD. Compared to standard chain-of-thought prompting, TPD significantly improves the student model's performance, achieving an average improvement of 6.2\%.

Published as a conference paper at COLM 2024
TPD: Enhancing Student Language Model Reasoning via Prin-
ciple Discovery and Guidance
Haorui Wang', Rongzhi Zhang', Yinghao Li', Lingkai Kong', Yuchen Zhuang', Xiusi Chen², Chao Zhang'
!College of Computing, Georgia Institute of Technology
?Department of Computer Science, University of California, Los Angeles (hwang984, rongzhi. zhang, yinghaoli, Ikkongg sikkongsycccuunne.ccchhaaaeeeeee chwozhang)@gaaeeeeeecco edu, xchenêcs ucla, edu
Abstract
Larger Language models (LLMs) often surpass their smaller counterparts in reasoning tasks but fall short in inference efficiency, posing the need and challenge of effectively transferring these capabilities from larger to smaller models. Existing approaches heavily rely on extensive fine-tuning data continuous interactions with superior teacher LLM during inference. We introduce a principle-based teacher-student framework, Teaching via Principle Discovery (TPD), to address these limitations. Inspired by human learning mechanisms, TPD mimics the interaction between a teacher and student using principle-based approach. The teacher LLM generates problem-solving instructions and corrective principles based on the student LLM's errors. These principles guide the refinement of instructions and the selection of instructive examples from validation set. This enables the stu- dent model to learn from both the teacher's ' guidance and its own mistakes. Once the student model begins making inferences, TPD requires no further intervention from the teacher LLM. Through extensive experiments across eight reasoning tasks, we demonstrate the effectiveness of TPD. Compared standard chain-of-thought prompting, TPD significantly improves the student model's 's performance, achieving an average improvement of 6.2%.
1 Introduction
Recent studies show that large language models (LLMs) can achieve impressive performance in various reasoning tasks, such as analogical (Webb et al. 2023), arithmetic (Imani et al.,
2023), symbolic (Pan et al.) 2023a), and commonsense reasoning (Wei et al., 2022b Bang
et at. 2023). However, a noticeable performance gap can often observed between stronger I.I.Ss such GPT-4and weaker L.Mss such GPT-3.5-turbo (Espejel et al., 2023 Ap 2023 Cai et al. 2023). This disparity arises from factors such as training data size, model capacity, and the methods by which LLMs learn and encode world knowledge. While stronger LLMs exhibit superior performance, their practical application is hindered by the high costs associated with training and inference. For instance, as of this writing, the cost of using GPT-4 is over ten times higher compared to GPT-3.5-turbe This raises the question: how can we effectively transfer the advanced reasoning capabilities of stronger LLMs to weaker ones?
Several approaches have been proposed to address this challenge. Some studies (Rajani et al. 2019 Ho et al. (023) curate datasets for specific downstream tasks using stronger LLMs and then tine-tune weaker LLMs on these datasets to instill the necessary knowledge. However, this fine-tuning process is time-consuming, and the resulting task-specific weaker
LLMs lack generalizability to other tasks. WWan & Li (2023) use an assistant language model to provide guidelines and analysis for the student model, but this approach requires constant involvement the assistant model, which can be costly. To reduce teacher model involvement, Saha et al. (2023) request teacher intervention only when the student model

---

Published as a conference paper at COLM 2024
exhibits low confidence, where the confidence is computed from the token probability of the answer. Such a measure may not accurately reflect the student model's true confidence, as the token probability of the answer is influenced by the preceding output, specifically the chain-of-thought (CoT) explanations.
To address these limitations, duce Teaching via Principle Discovery (TPD), a principle-based teaching framework that minimizes teacher model involvement, thereby optimizing resource allocation and efficiency. TPD draws inspiration from in- structional strategies observed in natural human teaching and learning processes (Rosenshine) 2012 Henderson & Harper Metcalfe 2017), which tollows struc- process: 1) Demonstrate: The teacher introduces problem type and demonstrates how to solve it. 2) Practice: The student engages with practice questions. 3) Review: The teacher reviews the student's responses to identify common errors. This review process enables the teacher to extract fine- grained corrective principles that guide the student in rectifying errors and improv- ing future problem-solving. TPD comprises two stages: principle generation and princi- ple exploitation. In the principle generation stage, the teacher model generates problem- solving instructions and summarizes prin- ciples based on errors made by the student model on validation questions. In the principle exploitation stage, the teacher model constructs instructive examples that illustrate the principles and injects these instructions into the prompt to guide student learning. TPD avoids teacher model involvement during inference, enabling the student model to operate independently in offline scenarios.
intro-
4/19/1969 today. What the date one year ago in MM/C0/MYY
Jane and John married Jan 1958. Today their golden wedding annivertary. What the date today MM/DD/MYY?
1/2/2008
1/2/1998
T7 Teacher Model
Teacher Model
Student model
Determine today's date Then Applly the time change .. Problem solving method
1. Golden wedding anniversary
tured Paaponsttrte-Paaaiic-eeeee
years anniversarr.
2, You should use solve
Principle List
Instruction generation
summarization
Illustration of TPD. The teacher model generates problem-solving instructions and then summarizes principles based on er- rors made by the student model on valida- tion questions. During principle exploitation, the problem-solving instruction and examples that illustrate the principles are combined into the prompt to guide student learning.
Figure 1:
We validate TPD on eight reasoning tasks covering symbolic and arithmetic reasoning. Through principle generation and exploitation, TPD significantly improves the performance of the student model without teacher model intervention during inference. Specifically, our model achieves an absolute gain of up to 19% accuracy compared to chain-of-though prompting. Additionally, we explore different methods for injecting generated principles into the student model and find that selecting new examples from practice questions outperforms direct injection and the critique-revise method.
2 Related work
Teacher-student framework Traditional teacher-student frameworks employ fully super- vised fine-tuning methods (Magister et al.] 2022 Shridhar et al., 2022 Hendrycks et al., 2021 Rajani et al.,, 2019). For instance, Ho et al. (2023) generate Col reasoning steps by GPT-3 (Brown et al., 2020) and fine-tune several relatively small models with these gener- ated data. However, both the fine-tuning process and collecting task-specific fine-tuning data are time-consuming and challenging. To overcome these limitations, there are several prompting-based teacher-student frameworks recently (Pruthi et al., 2022 Saha et al. 2023 Yu et al. 2023). In these frameworks, the teacher model is tasked with offering demon- strations or explanations to the student model. However, these approaches require the intervention of teacher model on the test set. Hence, they are unsuitable for offline scenar- ios, where the student model must solve problems independently, without assistance from the teacher model.
2

---

Published as a conference paper at COLM 2024
Eliciting LLM's reasoning ability through prompting Emergent abilities bring strong
few-shot learning ability for LLMs across various datasets via in-context learning (Wei et al.
2022a b). Many prompting-basee
methods are proposed to elicit the reasoning abilities of the LLMs by injecting knowledge into prompts. For instance, chain of thought (CoT) (Wei et al. 2022b) and its variants (Kojima et al. 2022 He et al., 2023) provide few human written examples or instructions to LLMs. We summarize more prompting methods in Appendix B These prompting methods are effective in eliciting the reasoning ability of the LLMs. However, they are not suitable for direct adaptation within the teacher-student framework, as they do not involve knowledge transfer from a teacher model. Another pipeline of prompt engineering is automatic prompt searching, also known as prompt optimization (Khattab et al. 2023 Yang et al. 2023a]. These methods employ LLMs to refine prompts iteratively in the zero-shot setting for specific task. However, applying automatic prompt searching methods for prompt optimization
time-consuming, as it requires multiple rounds of inference on the training dataset; in contrast, TPD requires only a single-round knowledge transfer. Moreover, TPD includes an error summarization stage, which aligns with learning from feedback prompting methods. These methods initially create a memory list. When new query arises, they retrieve relevant information from this memory to refine and enhance the current prompt. The retrieved information may consist of successful trials from history (Majumder et al.) 2023), the most similar questions with user feedback (Madaan et al. 2022a), or summaries of previous failures (Shinn al. 2024).
Principle discovery and exploitation Rule discovery is popular technique in machine
learning and data mining (Tweney et al. 1980; Fürnkranz & Kliegr 2015: Das et al. 1998).
In language models, rule discovery is often utilized learning from feedback frameworks
(Zhang et al. 2022a Pan et al. 2023b). In (Zhu et al. 2023), rules are generated by language models and subsequently selected through self-verification, Yang al. (2023b) propose framework where language models could generate rules by learning from previous mistakes. However, the rules produced in these methods tend be concrete facts, with the framework functioning as a retrieval system. In contrast, rule is formal, broad statement that applies to an indefinitely large set of objects. Unlike rules, principles are more abstract and open to interpretation, offering high-level guidance without strict formatting. For instance, Bai let al. (2022) predefine principles about helpfulness and harmfulness by human experts, then ask the language model to evaluate the generation results to make RLAIF. One concurrent work on principle discovery is LEAP (Zhang et al.) 2024), which focuses on summarizing both high-level and low-level principles based on model's previous mistakes. The key distinction between TPD and LEAP is that we incorporate principle discovery within a teacher-student framework, aiming to enhance knowledge transfer through these principles. In contrast, LEAP primarily demonstrates that LLMs also discover principles from their previous mistakes.
3 Method
3.1
Overview
In TPD, we consider a teacher model and student model. The student is responsible for solving reasoning problems based on the teacher model's guidance. The primary goal of our framework to let the teacher model effectively teach the student model how to solve these tasks. As shown in Fig 2 the framework is divided into two stages: principle generation and principle exploitation. During the principle generation stage, the teacher model produces problem-solving instruction based on sampled questions and identifies principle list from the errors made by the student model. In the principle exploitation stage, the principle list IP is injected into the student model to improve its reasoning capabilities in downstream tasks.
3.2 Principle generation
In this stage, we employ the teacher model to provide a high-level guideline to the student model and help the student model identify its common errors from few practice questions.
3

---

Published as a conference paper at COLM 2024
GPT-4: Teacher Model
GPT-333: Student Model
Slep
pruhlem-sulviing Instruction:
speeilee apply caleslate
It 1/18/ toeny, Whotls the cate voilenlay sitee/tsa/rmm
Pracice Quetton:
Assowert
Iomariaa k1/ppp0 the
sourpple::
Step 1:
coite su/ha/araass Saooaa
Qunatiant Annwere.....
Problem-antving Instruction Generation
trinciple List:
Pradise Queskonr
Medified Problem-ssking
The plve date may not hodday'sddatt date. Cakollite gives
dentthy mistaks suieer uydatt the aslete nos intuutionn socertingy.
Slep 4:
Student Inderence
Faomplec...
Queslior
Modidied Problom sulving Instruction:
Temomoo 12/12/2000 What
informationn
Deteraiie sodac's dats bused on the mesrassed
Answarr:
hours later WMMM/////7
Apply change tolayledate the date.
Step 3:
Tomeeros dute aoroooo, today's date is Hi/iL/oooo, thas bours latne lauooo
Sclected Exumpleee
Froor Summarication & Principle Exploilation
Qucsiltu: Assners
Figure 2: TPD contains two stages: principle generation and principle exploitation. In principle generation, the student model generates answers according to the problem-solving instructions from the teacher model. Then, the teacher provides a list of principles based on student's practice errors. In principle exploitation, the teacher model refines the instruction and chooses representative examples, which are used by the student for inference.
This principle generation stage comprises two sub-stages: 1. Problem-solving instruction generation and initial practice and 2. Error summarization. The two-stage process mirrors the traditional classroom learning model, where teachers provide instructions, students practice, and teachers offer feedback based on observed errors.
3.2.1 Problem-solving instruction generation and initial practice
What is problem-solving instruction? The problem-solving instruction I consists of problem-solving method augmented with few examples. To build the instruction, the teacher model reviews a set of questions sampled from the training set, identifying what type of questions the student model needs to solve. Then, the teacher model generates a problem-solving method as an initial instruction in natural language. To enhance the instruction, it is further enriched with examples that demonstrate the application of the proposed method. These examples are derived from the same questions initially sampled. We provide an example in Appendix for better understanding.
Why problem-solving instruction? Previous works (Saha et al. 2023 Wang & Li 2023 ask the teacher model to offer guidance for each test question, rendering the teaching framework impractical for offline scenarios. To tackle this issue, ask the teacher model to provide high-level problem-solving instruction based on sampled questions and internal knowledge. The problem-solving instruction is adaptable, making it suitable for offline use as it generalizes across similar types of questions. Moreover, since many prompting methods (Wei et al. [002b Gao al. 2023b) Zhou et al. (2022) have shown that LLMs have strong capability to learn from examples, the provlem-solving instruction includes few examples help the student model understand and imitate the instruction. This process also follows the Case Teaching Method paradigm (Herreid) 2005), where the teacher models provide several cases, and the student model can imitate and learn from the examples. The problem-solving instruction reflects the key principles to solve the type of problems, as it provides fundamental guidance for addressing specific types of problems.
Initial practice for the student model With the groblem-solving instruction, the student model generates answers for the questions in the validation set. This step lets the student model practice, allowing the teacher model to assess its comprehension of the problem- solving instruction and its ability to apply it effectively.
4

---

Published as a conference paper at COLM 2024
3.2.2 Error Summarization
Data filtering After the student practices over the validation set, we evaluate its responses to build an error set IE = {21,e2,... (n,, where each element in this set represents a pair consisting of question and the corresponding incorrect answer made by the student model. Although the teacher model generally performs well on various tasks, it can make mistakes. Hence, we need to assess the teacher model's ability to accurately identify the incorrect answers in F. Specifically, we check if the teacher model agrees with the student's incorrect answers, removing each e that the teacher model can not identify to build a feasible error set É. Due to the context length of the LLMs, we then develop an iterative framework to summarize principles from each e; iteratively.
What is error summarization? In Algorithm 1, we denote the model response as M(p) given the LLM M and the input p. The teacher model initially derives principles P from subset N; of the feasible error set IE. Then, we present the remaining set IN, to the teacher model sequentially, prompting it to determine whether the established principle list IP can rectify the error. If the existing principle list P can not address the presented error (T) the teacher model will formulate a new principle for it. The iterative process stops when all the errors in (i) are checked. Then, human reviewers will step in to assess the validity of the principle list IP created by the teacher model. The reviewers will simply delete any that are found to be erroneous or confusing, ensuring the reliability and clarity of the final principle list. We provide instructions for human reviewers in Appendix F
Why error summarization? Making mis- takes is a natural part of the learning process and can be a powerful tool for growth and understanding (Cyr & Anderson 2018 Even though students can solve simple tasks us- ing the original problem-solving instruction alone, they may have difficulty applying the same knowledge and skills in complex situa- tions (Klein et al., 2007). Therefore, is also important to let students learn from errors, which can help them identify error patterns and avoid similar mistakes in future practice (Metcalfe 2017). Learning from mistakes has been explored in (Wang & Li 2023: Yang et al., 2023b), where knowledge from past failures is stored in memory list and retrieved for each new query. However, these approaches primarily focus recording errors in detail without summarizing them into high-level principles. In contrast, our approach emphasizes high-level principles, which are more generalizable and can be effectively applied in offline scenarios. Additionally, TPD can also be augmented by retrieval methods, as we can retrieve the most similar questions along with the teacher model's feedback from practice questions. Further analysis is provided in Appendix(] D
Algorithm 1:
Error summarization
Input: T: the teacher model; IB: the feasible error set Output: P: the principle list
Sample subset N₂ from EE;
IN + Ns:
IP + summaaiie (Ns);
for i E (1,2,3,,, ..., [N do if Teealuatee P. (i) False then
p 4- T summarize (N,
P P PUp;
end
end
10 return PP
3.3 Principle exploitation
In the second stage, the student model's role is to utilize the principle list P solve various instances of the task. There are multiple ways to utilize the principle list, which are explored directly injecting the principle list into the prompt and the critique-
in Sec 4.4 Compared revise method, we found that using the principle list P to curate new examples from the validation set more effective. Specifically, we define the violation score, which is the number of violations against the principles in the list I.. The teacher model evaluates each error in E using violation scores to rank them. The examples with the top k highest violation scores are selected as the most informative examples. Then, the teacher model generates correct answers for these examples to replace the incorrect answers from the
5

---

Published as a conference paper at COLM 2024
student model. Besides, the teacher model also revises the problem-solving instruction based on The overall instruction combines the revised problem-solving instruction and the selected informative examples. For better illustration, present the overall instruction template below.
Overall Instruction Prompt
Revised Problem-solving Instruction: (method) + (original examples)
New Selected Examples: (question] (answerr)
The principle generation stage only needs to be performed once for each type of task. The teacher model could then build new prompt based the principle list for the student model. The prompt can then be reused for all instances of that task. This makes TPD significantly more efficient than using retriever or teacher model's intervention in each query.
Experiments
4.1 Experiments Setup
Datasets. We evaluate our approach eight datasets from diverse domains, including four tasks from Big-bench (Srvastavaa et al. 2022): Tracking Shuffled Objects, Date Under- standing, Navigate, and Matrixshapes. The other four datasets are GSM8K (Cobbe al., 2021), SVAMP (Patel et a 2021), CoinFlip and Last Letter Concatenation (Kojima et al. Wei et al. 2022b). We use the 5-object versions of the Tracking Shuffled Objects task in the experiments, which we will refer to as Tracking Shuffled Objects (5). The CoinFlip, Last Letter Concatenation and Tracking Shuffled Objects (5) are regarded as symbolic reasoning tasks, and the remaining datasets are arithmetic reasoning tasks. The dataset split and the
2022
detailed information about each dataset can be found in Appendix G
Experiment settings. In the experiment, (Al@Meta 2024) and Mixtral-8x/99 Btr-sstt-txt---sttuuctttt (Jiang et al. 2024) as student models, with gpt-4
ALama3-70B-Instrru
AppenAl 2023) serving as the teacher model. In the tollowing sections, we will denote them
as GP13.5, LLama3, Mixtral, and GPT4, respectively. More detailed settings are given in H.2
Baselines. We include Zero-Shot CoT (Kojima al. 2022), few-shot CoT (Wei et al. 2022b), and Auto-CoT (Zhang et al. 2022b) as baseline prompting methods. Specitically, we include 6-shot CoT as baseline method, where all of the 6 examples are selected based on the principle list from practice questions (there are 6 examples in the final prompt for the student model). In contrast, 3-shot-CoT uses randomly selected examples from the training set. More details of the baselines are given in H.3
4.2 Symbolic reasoning
Symbolic reasoning entails the use of symbols and their relationships execute logical operations (MacCol] 1897). It gauges logical reasoning and rule-based decision-making abil- ities, assessing language model's proficiency in simulating human-like reasoning. Table presents the comparison results. As anticipated, 3-shot CoT and Auto-CoT surpass zero-shof CoT by providing more examples and guidance the language model, enabling it to gener- ate higher-quality reasoning processes. Auto CoT achieves comparable performance across all three datasets compared to 3-shot CoT, suggesting that it can match the performance of the CoT paradigm that requires manual designs. Our method TPD outperforms 3-shot CoT on CoinFlip and Last Letter Concatenation, and achieves comparable performance on Tracking Shuffled Objects (5), demonstrating the potential effectiveness of problem-solving instructions over merely presenting a sequence of language reasoning steps.
The error summarization stage appears to be less effective in symbolic reasoning tasks, primarily because errors often stem from a lack of factual knowledge rather than the misap- plication of principles. Principles serve as high-level guidelines, applicable to broad range

---

Published as a conference paper at COLM 2024
Symbolic reasoning
Arithmetic reasoning
Model
Method
Coin
Letter
Shuffled
Date
Navi.
GSMSK
Matrix
SVAMP
0-shot C
65.5
48.9
47.5
24.0
39.0
66.7
77.6
3-shot CoT P
80.8
82.9
74.5
68.5
74.5
84.5
82.6
GPT3.5
Auto CoT/PoT
80.6
81.5
75.5
71.0
73.9
83.5
82.8
6-shot CoT
92.8
83.5
75.0
74.0
74.8
82.8
TPD w/o ES
100.0
89.7
75.0
33.5
74.7
81.2
TPDw/ w/ ES
100.0
89.9
75.0
76.5
73.4
93.5
82.9
0-shot Co'l P
45.6
15.2
24.5
18.0
47.0
51.4
8.5
69.9
3-shot Co'l I
78.4
58.7
42.0
35.5
72.5
57.2
64.5
79.4
Mixtral
Auto CoT/PoT
76.5
58.4
43.5
39.5
73.0
56.8
71.0
79.6
6-shot /
78.8
58.3
48.5
44.5
82.0
61.2
84.5
79.8
TPD w/o ES
67.2
49.3
51.5
54.0
76.0
63.6
68.0
84.2
TP ES
78.8
59.2
52.5
71.5
96.0
65.1
91.0
86.0
0-shot I
100.0
82.2
63.0
31.0
83.0
81.3
61.0
82.5
3-shot T
97.6
85.6
99.0
52.0
58.5
87.5
86.5
91.2
LLama3
Auto CoT/PoT
97.8
86.2
98.5
59.5
80.5
89.8
89.5
90.8
6-shot CoI/Pot
97.9
85.8
99.0
75.0
89.5
88.3
94.0
91.5
TPD w ES
94.6
84.7
98.0
74.5
68.5
87.7
82.5
89.7
TPD w/ ES
99.1
86.1
99.0
85.5
93.0
92.7
97.5
93.4
Table 1: Performance on symbolic reasoning and arithmetic tasks, measured in accuracy(%)). The teacher model is GP'4,, and the student models are GPT3.5, Mixtral and LLama3, In symbolic reasoning tasks, we utilize CoT as our base prompting method. In arithmetic reasoning tasks, we utilize PoT as our base prompting method. We test the performance with and without error summarization (ES).
of scenarios. They do not change with specific situations but offer consistent approach to problem-solving In contrast, factual knowledge, which entails detailed, verifiable informa- tion about the world, is crucial in addressing specific cases. It should be incorporated into the model's parameters during training or sourced from external databases when needed. For instance, consider the Last Letter Concatenation task: the student model adheres to the instruction to identify and concatenate the last letters of words to form new string. The principle guides the process, but errors may occur due to inadequate factual knowledge about the last letter of a particular word within the string.
4.3 Arithmetic reasoning
The results of arithmetic tasks are shown in Table 1 Zero-shot PoT results in the worst performance in all datasets. The main issue for 0-shot PoT is the LLMs' inability to utilize their pre-trained knowledge to craft solutions for problems without explicit guidance, thereby failing to tap into their potential for coding and logical reasoning with basic prompt alone. The Auto-PoT approach, which stratifies validation by setting questions into three clusters and selecting the example closest to the center from each cluster to form a 3-shot PoT,, aims to introduce diversity into the examples. However, it only marginally improves performance across three datasets and performs poorly on GSM8K, which suggests that mere variety in questions does not inherently lead to distinct reasoning pathways. The original problem-solving instruction derived from the teacher model demonstrates performance com comparable with, and occasionally inferior to the 3-shot PoT, since manually created examples may surpass those generated by LIMs in quality.
In comparison, TPD with error summarization outperforms other prompts significantly over Date Understanding, Navigate, and Matrixshape tasks, which indicates that principle list, distilled from the analysis of errors in practice questions, is highly effective. It guides the teacher model in refining the original problem-solving instruction and choosing the most informative samples for few-shot prompts. However, it only achieves comparable performance with 3-shot PoT on GSM8K and SVAMP, since the errors made by the student model are diverse understanding errors caused by lack of factual knowledge and not
7

---

Published as a conference paper at COLM 2024
be well categorized in high-level principle list. Moreover, 6-shot-PoT outperforms other baselines, suggesting that examples selected based on the principle list are more informative than those chosen randomly from the training set.
4.4 How to utilize the principle list
With the principles identified by the teacher model, we explored methods to transfer this knowledge the student model effectively. One straightforward approach involves directly appending the list of principles to the prompt. Al- ternatively, we can employ an iterative learning process, where the student model initially attempts to answer a question, then provides feedback based on its own understanding of the prin- ciples, and then revises its initial sponse. This critique-and-reviss strat- egy has been successfully used in var-
Method
Date
Nav.
GSM8K
Matrix
SVAMP
No principle
68.5
81.5
74.5
84.5
82.6
Injecting Into Prompt
71.5
83.5
74.8
86.5
82.6
Critique Revise
61.5
74.5
71.4
79.5
74.8
Examples Selection
76.5
97.5
75.4
93.5
Table 2: Performance of different principle injec- tion methods on GPT3.5 on arithmetic reasoning tasks, measured in accuracy(%)).
ious prompting methods (e-g., (Chen et al. Our proposed method enhances this approach by selecting highly informative examples from the validation set, guided by the principle list. This strategy capitalizes on the emerg- ing capabilities of LLM, enabling the student model to learn effectively from exemplary instances.
2023 Bai et al. 2022 Madaan et al. 2023
The experimental results presented in Ta- ble reveal that simply adding principles list to the prompt yields only a marginal improvement over the base prompt. This suggests that the student model faces chal- lenges in effectively using high-level and implicit principles expressed in natural lan- guage. While retrieval methods gather spe- cific factual knowledge and integrate it with the initial prompt, the principles here are more abstract and harder for LLMs to lever- age. For instance, a principle might serve as a directional guide for specific step in the reasoning process. Additionally, LLMs tend to lose information in longer contexts (Liu et al. 2023), which could further contribute to the limited impact of the principles list. Surprisingly, the critiue-and-rrevsee method resulted in a decrease in performance across all datasets. Our observations indicate that when prompted to provide feedback critique based on the list of principles, the LLM tends to perceive the original answer as incorrect and significantly overhauls it. This behavior might stem from the RLHF stage, where terms like feedback critique could trigger the model to question its previous outputs.
Model
Error
GSMSK
Date
ét
85%
17%
Gpt3.5
15%
47%
e3 0%
36%
78%
16%
Mixtral
e2 22%
58%
e3
0%
26%
Table 3: Percentage of different error types in TPD.
Using the principle lists to select examples from the validation set for in-context learning achieves better performance across all datasets than the other two methods. The chosen examples based on the principle list contain the most error-prone questions for the student model, thus helping the student model learn from errors effectively. This also mirrors real-world classroom dynamics, where providing students with practical examples often proves more beneficial than solely relying on textbook knowledge (Shafto et al. 2014).
4.5 Error Analysis
Despite the guidance provided by the teacher model, the student model may still make errors. In this section, we analyze the sources of these errors within the TPD framework. Firstly, it is important to note that the teacher model has its own limitations, especially in
8

---

Published as a conference paper at COLM 2024
complex reasoning tasks (Huang et al. 2023). As result, it cannot detect all the errors made by the student model. lo address this, data filtering is necessary during the error summarization stage. During the principle exploitation stage, errors can be categorized into three types:
e1: The principle provided is unhelpful.
e2: The student model fails to answer correctly even with helpful principle.
c3: There are no relevant principles in the principle list for the given question.
We analyzed the errors made by GPT-3.5 and Mixtral on the GSM8K and Date Understand- ing tasks, with the results shown in Table 3 In the GSM8K task, no 63 errors were observed for either student model, the teacher model provided highly general principles (e.g., "Ensure arithmetic operations are logically sound and mathematically correct"). While these general principles are broadly applicable, they fall short when addressing specific questions. This indicates that an effective principle should target a particular type of error rather than rely on vague or general terms. In contrast, for the Mixtral model, e2 errors were predominant. Even though the principle was helpful, the student model still failed to answer correctly, highlighting the inherent limitations of the student model's capabilities. Regarding errors, these arise when practice questions do not cover all possible question types, leading to gaps in the principle list. This is a common limitation of learning from feedback frameworks.
4.6
Ablation study
1.0
0.95
0.9
0.90
0.7
Accuracy
Coin Flip
0.8
Last Letter
0.6
Accuracy
0.85
Date Understanding
0.7
Shuffled objects
0.5
Navigate
0.80
Matrixshapes
0.6
Matrisshapes
0.3
GSM8K
0.75
Number of examples
Number of examples
Number of examples
(a)
(b)
(c)
Figure 3: The test accuracy different numbers examples in the problem-solving in- struction in (a) symbolic reasoning tasks and (b) arithmetic reasoning tasks. (c) An ablation study on the number of examples selected based on the principle list. 0 example means the prompt only contains the modified problem-solving instruction. The experiments are conducted on GPT3.5.
Effectiveness of examples in the problem-solving instruction. We in- vestigate the effectiveness of having the teacher model simply describe problem-solving method in the instruc- tion without including examples. The results are shown in Fig. 3a and Fig. (3b There significant decrease in accu- racy for both tasks when examples are
GPT3.5 principles
Mixtral principles
Model
Date
Navigation
Date
Navigation
GPT3.5
76.5
97.5
73.0
91.5
Mixtral
68.5
96.0
71.5
96.0
Table 4: ferent sources of principles.
Performance of student models with dif-
omitted from the problem-solving instruction, indicating the importance of incorporating examples in the problem-solving instruction. In more complex tasks like Matrixshapes and Navigate, we observed notable increase in performance with the inclusion the first example, while subsequent examples contributed to less. This demonstrates that for complex tasks, the student model struggles to learn from descriptions alone and relies on concrete examples to comprehend the problem-solving instruction. Across all datasets, more examples will help more, but the performance gain will be trivial as the number of examples increases.

---

Published as a conference paper at COLM 2024
Jumes decides to run sprints times day. lie runs 6o meters each sprint. How many total meters does he ran Jannary?
Question:
Question: Mukiipyy matris of shape with matri of shape (3-933-4). Transpose the result.
Principles: Logic
arithmetic operations logically sound mathematically correct.
Principles: Matrix Mtutiplication: in Python, the asterisk (**) operator is atll soa for the Hadamard product. Use up.matmul for matrix multiplication.
3-shot CoT:
TPD student: "python
python
3-shot
TPD student:
def solution():
def sclutiou():
def (olution():
python
= mpp mp.randomm..ooom mmp.aankoo..mmlff33..... 2333)
def solutionl:
python
op-mndom san.3g.3.3.3)
sprints por aession =3
scsalons, per day =
Fectual orror
sprints per sossion -3
sesssoox, per day =3
total digs
Factual error
- raa.ddamranddi......
result B
B np.ondom 2.941
mul result mp mp.matmllAA, B]
meters per sprint = 60 total distance m
meters per sprint 60 total distance m
transpuse
transpowe result
sprints,
sprints
up qp.tran-poeeonnl reault)
return transpuse_ transuse__eelltssappe
matranspaectma_aag return transpose_ sesultshppe
meteer per
days
per muters per speint total days return total, distance
return total distance
guidance
correction
lack of guidance
correction
Figure 4: The case study of TPD.
Generalization capability of principles. We further examine if the principles derived from the errors one student model can be effectively applied to another. As shown in Table. 4 both student models achieve good performance when leveraging principles deduced from the errors of the counterpart model. However, a slight performance drop is observed when model utilizes principles from the other model compared to its own. This gap demonstrates that although there exist common mistakes between the two student models, there are also some differences in their error patterns.
Numbers of examples selected in principle exploitation stage. To determine the mini- mum number of examples required for effective learning, we conduct experiments with different numbers of examples selected based the violation score. The results are pre- sented in Fig. 3c Our findings reveal that the initial example provides the most significant performance improvement across both datasets. This suggests that the initial example serves as the most instructive instance for the student model. The minimum number of examples required varies across datasets. For Matrixshapes, three examples are sufficient to achieve stable performance. However, while three examples provide a substantial gain for Date Un- derstanding, adding more examples continues to benefit the student model. For simplicity and consistency across all datasets, we employ three examples in our experiments.
4.7 Case study
In Fig. we presennt a comparative case study to show the success mode (left subfigure) and the failure mode (right subfigure) of TPD. In both scenarios, the principles guide the student model towards reasoning more effectively. This guidance becomes particularly evident in the success mode, where significantly enhances the student's problem-solving capabilities. However, the principles have their limitations, especially when confronted with the model's gaps factual knowledge. For instance, as illustrated in the right subfigure, the student model fails to accurately assign 31 days to January.
Conclusion
In this paper, we present a framework named Teaching via Principle Discovery (TPD). This approach empowers teacher models to construct problem-solving instructions and summarize key principles by analyzing example questions and student errors. The identified principles are subsequently employed to refine the problem-solving instructions and select the most informative examples from the validation set to create a tailored instruction prompt. We validate the effectiveness of TPD on symbolic and arithmetic reasoning tasks, observing marked improvement in the performance of the student model. Our method introduces an innovative approach, utilizing advanced LLMs to guide weaker agents in tackling reasoning problems. In the future, we plan to study how to apply TPD to solve complex reasoning tasks, such as serving web agent, where the principle list will be much longer.
10

---

Published as a conference paper at COLM 2024
References
AT@Meta. Llama 3 model card. htt.:p/pp///gg.t.oooo/////////od
BBan/MODEL_CARD... md
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Consti- tutional ai: Harmlessness from ai feedback. arXiv preprint ar arXiv:2212.08075. 2022. URL
08073.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multi- modal evaluation of chatgpt on reasoning, hallucination, and interactivity. ar preprint ar arX1o:2302.04023, 2023. URL
04023
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. URL https///rrrrrceeiiseeeeee/e//r/p
Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. arv preprint arXie:2305.17116, 2023. URL httpt:t//axxii..oo org/
pdf/2305. 17122
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. CoRR, 2ttp: 2022. doi: 10.48550/ arxvv URL http:::/ddooo.rr/////
48550/arXiv.2211.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXio:2304.05128, 2023. URL httpsttt////ar
Ddf/2304.65122
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert,Jery Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. URL
https://arxtv.org//d//22000 14168. pdf?curius=520
Andrée-Ann Cyr and Nicole D Anderson. Learning from your mistakes: Does it matter if you'rr re out in left foot, mean field? Memory, 26(9):1281-1299) 2018. URL https
7/www. www.wonwwwnnnnnnnoommmn t http://ff//001110000096888881 2018. 2
Gautam Das, King-Ip Lin, Heikki Mannila, Gopal Renganathan, and Padhraic Smyth. Rule discovery from time series. In KDD, volume 98, pp. 16-22. Cite- seer, 1998. URL /ttpsstts::/ciiteeeerr ist. htwp://cmmetrrrrprrrrrrgtteeeeee
4f1 o5ff0000007000effa22233332222222
Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. Compositional semantic parsing with large language models. Xiv preprint arXiv:2209.15003 2022. URL httpp:://arxvv.rrr prtpp/ffioogg/df//////
2209.1 15003
Dheeru Dua, Shivanshu Gupta, Sameer Singh, and Matt Gardner. Successive prompting for decomposing complex questions. Xrv preprint ar trXio:2212.04092, 2022. URL httst //arxiv.org/pdf//
04092.
Jessica López Espejel, El Hassane Ettifouri, Mahaman Sanoussi Yahaya Alassan, El Mehdi Chouham, and Walid Dahhane. Gpt-3.5, gpt-4, or bard? evaluating Ilms reasoning ability in zero-shot setting and performance boosting through prompts. Natural Language Pro- cessing Journal, 5:100032, 2023. URL BtLhhttps: ///::///www sciencedirect. com/ http//cccccc////////e ar articl Di1/S29497191228802288
11

---

Published as a conference paper at COLM 2024
Johannes Fürnkranz and Tomáš Kliegr. A brief overview of rule learning. In International symposium on rules and rule markup languages for the semantic web, pp. 54-69. Springer, 2015. URLhhtpp://bb. vse.
www_eeleaennngwweeee_eeeeeee pdf]
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: program-aided language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawvai, USA, volume 202 of Proceedings of Machine Learning Research, pp. 10764-10799. PMLR, 2023a. URL WiKLhtttt::/pprceeeiing .ww....sm......m...............m htt..r.pree/v202//// gao23f html,
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pp. 10764-10799. PMLR, 2023b. URL https://proccedinggs. mlr press/
2.82/gaa233fgaa23ff.o
Hangfeng He, Hongming Zhang, and Dan Roth. Rethinking with retrieval: Faithful large language model inference. CoRR, abs/2301.00303 2023. doi: 11448550/aXiv.2201..oo /otttps:/ddi.orrg// hete:/.....85////// 23650/arXiv.
Charles Henderson and Kathleen A Harper. Quiz corrections: Improving learning by encour- aging students to reflect on their mistakes. The physics Leacher, 47(9):581-586, 2009. URL
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Xiv preprint ar XIx::2103.00374, 2021. /RLhhttss:/arxxiv.ror //wp:///ppp////...////// 03874
Clyde Freeman Herreid. Using case studies to teach science. education: Classroom method- ology. American Institute of Biological Sciences, 2005. URL https: gov/ Fulltext/ED485982.11 pdff
Namgyu Ho, Laura Schmid, and Se- Ye-Young Yun. Large language models are reasoning teachers. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14852-14883 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1. 1t033/112223aaclloom/ acl. 830. URL httphttt///aclaathoolo... org/20g/ acl long. .830
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. ar Xiv preprint ar XrXi::2310001798 2023.
Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXin:2303.05398, 2023. URL https://arxiv org/pdf /2303. 05398
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023. URL httph:htt///d acm. org/doi/ pdf/10. 1145/357173
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L'elio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts. ArXio, abs/ abs/2401.00 2401.04008,8, 2024. sttpp:::/appi.semmnnticccco.
www.aaarrrr/CCoooosDa 266844877
Omar Khattab, Arna Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self-improving pipelines. Xixiv preprint ar arXio:2310.03714, 2023.
12

---

Published as a conference paper at COLM 2024
Tilmann A Klein, Jane Neumann, Martin Reuter, Jürgen Hennig, D Yves von Cramon, and Markus Ullsperger. Genetically determined differences in learning from errors. Science 318(5856):1642-1666 2007. URL 1145044
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information process- ing systems, 35:22199-22213, 2022. URL https: httppss//ppocceeingg..eeeee paper Faper/2022/file/ /bbbdd29 haroa//eeeereeeioaaaoaaeaoooorre pdf
www.ripp..ppppprrraaae......eeee cc/paper_files!
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long con-
texts. Xrv preprint ar Xxii::2307.03122 2023. URL
poffrrrkppblcccpssttcmmeeeee
Hugh MacColl. Symbollc reasoning. Mind, 6(24))493-510, 1897. URL https://www.jsto orgg/stablee/dd//247777..ddd
Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. Memory-assisted prompt editing to improve gpt-3 after deployment. Xiviv preprint ar Xrt:v2201.006009 2022a.
Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code are few-shot commonsense learners, In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1384-1403, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguistics. doi: 10.18653/v1// http://222....mmmaaa.... https://ea...nthooooo..oogg///// omm. emnlp-main....
URL
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhuumye, Yiming Yang, et al. Self-refine: Iterative refinement with self- feedback. arXiv preprint arXiv:2303.1765/ 2023. URL https://arxiv org/pdf /2303. 17651
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models reason. arXiv preprint arXiv:2212.08410, 2022. URL/https://arxi org/pdf/2211 08410
Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Peter Jansen, Oyvind Tafjord, Niket Tandon, Li Zhang, Chris Callison-Bure and Peter Clark. Clin: A continually learning lan- guage agent for rapid task adaptation and generalization. arXiv preprint ar 2111:22110.0113,, 2023.
Janet Metcalfe. Learning from errors. Annual review of psychology, 68:465 489, 2017. URL
ur
OpenAL. GPT-4 technical report. CoRR, 2b5/200.0000 2023, doi: 10.48550/arXivv2303. 08774. URL.https:://oi.orr/// 48550/arrXix
http:///....rrr/e.....6666/5//// 20555//rrxv..223.88777. 2303.08874
Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. Logic-Im: Empowering
large language models with symbolic solvers for faithful logical reasoning. arXiv preprin!
arXiv:2305.12299 2023a. URL HRLLhhtps://arxiv ppg/ httppf/ff// /2055. 12295
Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint arXiv:2308.n 033888 2023b. URL https
//arxiv.org/pdf/22300 /20///0pdd/228.11188883
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1080-2094,
13

---

Published as a conference paper at COLM 2024
Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/202 naacl-main. 168. htattss://allmmttooooey.g/22222 httphhhollgo...gg222222///
naacl-main. 168.
Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig, and William W. Cohen. Evaluating explanations: How much do explanations from the teacher aid students? Transactions of the Association for Computational Linguistics, 10:359-375, 2022. doi: 10.1162/tach 0046.. URL https
//aclanthology.com or2. wwwp//222.tacl11.2
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. In Anna Ko- rhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pp. 4932-4942 4942, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653 P0/03/11/PP99// 11/1/1-11PP... URL http:://aalaattioooyy.oo.////1/1
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPP-JCCLPP), pp. 3982- 3992, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: (1116653v1/D1990
http://llaaactoooooooo/////////a Drg/D19-1410
Barak Rosenshine. Principles of instruction: Research-based strategies that all teach- ers should know. American educator, 36(1):12, 2012. URL httts::/files..riicc ed. gov/
fulltext/EJ971155 pdff
Swarnadeep Saha, Peter Hase, and Mohit Bansal. Can language models teach weaker agents? teacher explanations improve students via theory of mind. arviv preprint arXiv:2306.09299
2023. |R|/ttts:///axxi org/ 2206. 09299
S Noah D Goodman, and Thomas L Griffiths. A rational account of pedagogical reasoning: Teaching by, and learning from, examples. Cogmitive psychology, 71:55-88 89, 2014.
CRRLhttps://ww.ciieee
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. Distilling multi-step reasoning capabilities of large language models into smaller models via semantic decompositions. arviv preprint ar 2rXiv:2212.00193, 2022.
hntpps://arxiw.or///dfff
00193
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Xiv preprint ar Xiiv:2206.04611, 2022. URL / wttpxiiwworr/////220000/ 04615 pdff
Ryan D Tweney, Michael Doherty, Winifred J Worner, Daniel B Pliske, Clifford R Mynatt, Kimberly A Gross, and Daniel I. Arkkelin. Strategies of rule discovery in an inference task. Quarterly Journal of Experimental Psychology, 3291):10 1980. URL, httthst///j/rnna //jurnals
sagepub. ctw/ pdf /10. 0888/8033555880824883311
Danqing Wang and Lei Li. Learning from mistakes via cooperative study assistant for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 10667-10685, Singapore, December 2023, Association for Computational Linguistics. doi: 10.18653/v vv/ 2023.emnlp-main.ce
https:
.rrg:/gooorrr./// 2023. w00.2...eelp-mmmmee www.p-main.....m..mmmmmmmmm ma.ppmmmain...mm 659
Taylor Webb, Keith J Holyoak, and Hongjing Lu. Emergent analogical reasoning in large language models. Nature Human Behaviour, pp. 1-16, 2023, URL https: : //www. nature conn/artiiclees/4115-2222------2
14

---

Published as a conference paper at COLM 2024
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abil- ities of large language models. Trans. Mach. Learn. Res., 2022, 2022a. URL. htt
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824- 24837, 2022b. URL http:httt:///rooe htt.tttss/pprroeednns.neeee neur httpa//nnnnnnrisoo.pppppeeeeeeee http://www....ppppppeep/ee//p//p
pdf
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V Le, Denny Zhou, and Xinyun Chen. Large language models optimizers. arXiv preprint ar arXio:2309.03409, 2023a.
Zeyuan Yang, Peng Li, and Yang Liu. Failures pave the way: Enhancing large language models through tuning-free rule accurulation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Pp. 1751 Singapore, December 2023b. Association for Computational Linguistics. doi: 10. 10.18653/vv 10t/////11203..mm////// 109.. URL httpp:://cclanthol.og org/ 2023. momnlp-main... 109
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas I. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Xrv preprint ar xit::23055.0060 2023. Uttht::: org/ /2005. 10601
Dhara Yu, Noah D Goodman, and Jesse Mu. Characterizing tradeoffs between teaching via language and demonstrations multi-agent systems. arXiv preprint ar wxiv:2305.11374, 2023. URL https: pettps://arxivv ottp 22005 11374
Rongzhi Zhang, Yue Yu, Pranav Shetty, Le Song, and Chao Zhang. Prompt-based rule discovery and boosting for interactive suakly-supervised learning. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 745 -58,,58, Dublin, Ireland, May 2022a. Association for Computational Linguistics. LingutttiinnLLLLuiiistitiiiii doi: 2011863/v1/2000 acl-long.55. URL
wwwp://ggy..rg//22...o.o acl acl-long.
Tianjun Zhang, Aman Madaan, Luyu Gao, Steven Zheng, Swaroop Mishra, Yiming Yang, Niket Tandon, and Uri Alon. In-context principle learning from mistakes. Xiv preprint arXivv2402.05403, 2024.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arviv preprint ar XXi::2210.03493, 2022b. URL https //arxiv.org/pdff /2211 03493.
Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingje Zhan, et al. Solving challenging math word problems using gpt-4 code interpreter with code-based serfiieriiinntion. arXiv preprint -xxx:23088009221 2023.
/tt//ttss://arxii http::////axivo.o//dfff.. wtt/ /2308. 07921]
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint ar Xrxo:2205.10625, 2022.
/2wp:///...w//p///22..... 10625
Zhaocheng Zhu, Yuan Xue, Xinyun Chen, Denny Zhou, Jian Tang, Dale Schuurmans, and Hanjun Dai. Large language models can learn rules. arXio preprint ar 2rx10:2310.000664 2023. htthttsss///axi..cooo /ttp///.////111..00.0.00000 07064. www.o///we2.rreebo...ooooooooooo
15

---

Published as a conference paper at COLM 2024
A
Limitations
One limitation of the TPD is its inability to rectify common sense errors (lack of factual knowl- edge), as evidenced by its marginal improvements in the GSM8K and SVAMP datasets in Table despite the teacher model's high accuracy (97%, Zhou et al. 2023). This limitation arises because the errors made by the student model predominantly relate to factual knowl- edge, category not effectively ressed by the generated principles and instructions. High-level principles can resolve only a limited number of such errors. A potential solu- tion for this challenge could involve fine-tuning the LLM to internalize a broader range of common sense knowledge. Another constraint of TPD is the method for applying principles. While selecting examples based on validation scores has proven more effective than other strategies, this approach is not particularly efficient. The issue stems from the LLMs' context length limitations. This inefficiency poses an unresolved challenge: effectively integrating long list of principles into LLMs. Moreover, we do not check the overlap of the principles in the principle list, hence there might be near-duplicate principles the principle list.
B Additional related work
Madaan et al. (002bb] Gao al. )2023a): Chen al. 2022) find that using LLMs to generate codes for reasoning tasks and then utilizing the codes by program interpreters to solve the questions can achieve better performance. Another prompting method is the problem decomposition (Zhou et al. 2022 Drozdov et al., 2022 Dua et al. 2022), which asks LLMs to decompose the tasks into several subtasks and solve them individually. Yao et al. (2023) propose Tree of Thoughts, which enables language models to self-evaluate intermediate "thoughts" and decide whether to explore different ideas or reevaluate when needed to provide the optimal solution. Self-refinement methods (Madaan et al. 2023) ask LLMs to refine their original answers iteratively but will rely on the handwritten few shot examples.
C Problem-solving instruction example
The problem-solving instruction consists of a problem-solving method augmented with few examples. These examples are derived from the same questions initially sampled. Here an example.
Problem-solving Instruction:
To solve the problem where you are asked to take the last letters each word in a given string and concatenate them, you can follow these steps:
1. Identify and list each word in the string.
Locate the last letter of each word.
Concatenate these letters form a new string. Provide the new string as the answer.
Examples: Question: Take the last letters of each word "Jeremiah Kelley Josue Veronica" and concatenate them.
Identify the words: Jeremiah, Kelley, Josue, Veronica.
Last letters: h y, a.
Concatenate: "hyea""
Answer: hyea.
D TPD with retrieval augmentation
TPD leverages the teacher model to extract and summarize principles from the mistakes made by the student model on practice questions. These principles operate at the task level. contrast, popular learning-from: from-feedbacck frameworks typically provide feedback at the query level, where retriever fetches relevant information for each query from memory module. This contrast led us to explore whether TPD could further enhanced by integrating retrieval module. Specifically, we instructed the teacher model to generate
16

---

Published as a conference paper at COLM 2024
100
Appending
Replacement
95
94.0
Accuracy (%)
90
85-
76.5
75
72.5
Date
Navigate
GSMBK
Matrixshapes
Figure 5: Ablation studies on how to utilize selected examples with the modified problem-
solving instruction.
textual analysis for each error within the feasible enror set E. During the student's inference stage, retriever identifies the most similar question from É and appends the retrieved question along with the teacher model's analysis to the student's prompt.
The results shown in Table suggest that TPD can indeed be improved with the addi- tion of a retrieval module. This improve- ment indicates that the task-level principles and the directly retrieved examples are com- plementary to some extent. However, in the Matrix task, the Mixtral model's per- formance did not benefit from the retrieval module. We believe that finding an opti- mal method to combine task-level princi- ples with query-level retrieved information could be a promising future direction for the teacher-student framework.
model
Date
Matrix
GPT3.5
76.5
93.5
TPD
Mixtral
71.5
91.0
TPD
GPT3.5
81.0
94.0
+RAG
Mixtral
78.5
91.0
Table 5: Performance of TPD and TPD aug- mented with RAG (retrieval).
E
Additional ablation study
Can we replace the original examples in the problem-solving instruction with the newly selected samples? Fig. 3c demonstrates the high informativeness of examples selected based on violation scores. This raises an intriguing question: can we directly replace the original examples in the modified problem-solving instruction with these newly chosen ones? To explore this, Fig. 5lprsentss a comparison between two approaches: appending the selections to the original examples versus completely replacing them. Interestingly, the appending method consistently outperforms the replacement approach across all datasets. This superior performance can be attributed to the fact that the new selections specifically address the errors encountered when the student model learns from the original examples. Moreover, completely removing the original examples may inadvertently shift the student model's errors, rendering the newly selected examples less informative.
Methods description. Our study also investigates whether the teacher model should offer high-level reasoning method in problem-solvin instruction. As illustrated in Fig. 6 we observe a notable decline in problem-solving performance without an explicit method description across three datasets. This suggests that while the student model can implicitly learn and mimic reasoning from examples, it remains crucial for the teacher model to provide a problem-solving method in natural language explicitly.
Human intervention instructions
Regarding human intervention for quality checks on principles, since the principle lists in our experiments are not very long, human reviewers finish checking the principle list
17

---

Published as a conference paper at COLM 2024
105
100.0
w/o Method
Accuracy (%)
90
85
83.0
05.0
05.00 055.00
80
Coin Flip
Last Letter
Date
Matrixchapes
Datasets
Figure Ablation studies on whether the teacher model needs to provide problem-solving methods in the problem-solving instruction.
within approximately 20 minutes per task. In TPD, human reviewers are responsible for verifying each principle in the principle list for all tasks to ensure quality. They sequentially review and remove any principles that are vague, erroneous, or confusing.
Human intervention instructions:
1. 1.The principle should be written in simple, clear language that is easy understand.
2.The principle should be specific enough to address a particular type of error or concept.
3. 3.The principle should be factually correct and free from errors. 4.The principle should provide actionable guidance that helps the student avoid the error in the future.
5.ile specific, the principle should also be general enough to apply to similar errors or situations.
G
Datasets
The Coin Flip task (Wei et al. 2022b) requires the LLM to determine a coin remains heads up following series of Hlips or non-flips. The Last Letter Concatenation task (Wei al. 2022b) requires the LLM to concatenate the last letters of each word in a given name list. Both tasks have out-of-domain test sets, which include more complex examples than those provided in the training exemplars. Specifically, for the Last Letter Concatenation task, the training contains the lists comprising only two names but the test set consists of the lists with three and four names. A similar setting is adopted for the Coin Flip task.
Tracking shuffled objects (Srivastava al. 2022) requires the LLM to determine the ultimate state of system based on its initial condition and series of subsequent changes. In each instance of this task, collection of objects is initially associated with individual owners. These objects are then exchanged in sequence of swappings. The model's 's objective is to determine the final owner of each object.
Date Understanding (Srivastava et al., 2022) asks the LLM to infer the date from the given context. The context is one or two sentences with date information.
requires the LLM to infer the agent's position after several movements. Specifically, given a series of navigation steps, the LLM needs to determine whether not an agent would end up back at the starting point.
Navigate Srivastava et al. 2022
GSMSK (Cobbe et al. 2021) is a collection of grade-level mathematics problems by human authors. These problems require to 8 steps for resolution, primarily requiring series of fundamental computations employing basic arithmetic operations (addition, subtraction, multiplication, and division) to arrive at the solution.
Matrixshapes (Srivastava et al. 2022 requires the LLM to predict the shape of the result of chain of matrix manipulations, given the inputs' shapes.
18

---

Published as a conference paper at COLM 2024
SVAMP PPatel et al. 2021) is challenge set for elementary-level Math Word Problems. It consists of questions that test a model across question sensitivity, reasoning ability, and invariance to structural alterations.
For each dataset, we sample three questions as the training set. Subsequently, we partition the remaining data into practice and test questions, maintaining 1:4 ratio. Notice the CoinFlip and GSM8K are already split into the training set and the test set. We use the original test set and randomly sample 3 questions from the original training set to build our training set. Regarding Date Understanding, Navigate, Matrixshapes, and Tracking shuffled objects (5), we utilize data from (Cai et al. 2023). We divide each dataset into training, validation, and test sets, containing 3, 47, and 200 instances, respectively. For Auto-CoI, we select examples from the whole dataset in symbolic reasoning tasks and from the validation set in arithmetic reasoning tasks. Table the example questions for each dataset.
H Experiment Setup
H.1 Model Versions
In the experiments, we utilize two GPT models: gpt-3.5-turbo-16kt and gpt-4. The gpt-3.5-16k refers to the "gptt3.5-turbo-1666 model and in the OpenAl API model with checkpoint version 2023-06-13-pre 123-06-13-preview webpage GPT refers to the "gpt-4" model with checkpoint version 2023-07-01-preview. All mentioned checkpoints are hosted on Microsoft Azure For Mixtral-6x77-Instruuc we utilize "Mixtral-8x7BB IS78-Innstruct-0011 on huggingface. The model temperature is in all cases.
H.2 Implementation Details
We use the default temperature of 0.0 for all models. For symbolic reasoning tasks, we utilize CoT as our base prompting method. For arithmetic tasks, we utilize the Program of Thought (PoT) method as our base prompting method. This method involves having the LLMs process natural language questions and create corresponding programs that represent intermediate reasoning steps, with the actual computation solutions being delegated to a runtime environment, such as Python interpreter. We avoid using Col due to the tendency of language models to produce erroneous mathematical operation results when tackling arithmetic tasks (li et al., 2023). Our framework is designed to instruct the student model in problem-solving strategies rather than in performing detailed decimal operations with precision.
H.3 Baselines
Since our framework generates prompt for the student model for the downstream task, we compare several prompting methods in our experiments. Specifically, we adopt Zero-Shot CoT (Kojmma et al. 2022), few-shot CoT (Wei et al. 2022b), and Auto-CoT (Zhang et al.) 2022b) as baseline prompting methods. For the tew- CoT, we use the questions the training set as exemplars for few-shot prompting. For Auto-CoT, we use Sentence- BERR (Reimers & Gurevych 2019) to compute vector representation for each question and then cluster practice questions by k-means. Following the original paper, we choose the question closest to the center from each cluster and ask the teacher model to generate reasoning steps with zero-shot CoTT to build final examples for the student model.
H.4 Principle Lists
Table 7 and Table & show the principle list the teacher model finds in the error summarization stage for GPT3.5. These principles provide high-level guidance applicable across various
http:://plaatormmoeaaiccccc///// models oppenai.azure.com
19

---

Published as a conference paper at COLM 2024
Coin Flip
Q: A coin heads up. Murraylee does not flip the coin. Meilich flips the coin. Is the coin still heads up?
A: no
Last Letter Concatenation
Q: Take the last letters of each word in "Maritza Nana Loretta Eric" and concatenate them.
A: "aaac"
Tracking shuffled objects
Q: Alice, Bob, Claire, Dave, and Eve are dancers at a square dance. At the start of song, they each have partner: Alice is dancing with Patrick, Bob is dancing with Sam, Claire is dancing with Jamie, Dave dancing with Lola, and Eve dancing with Melissa. Throughout the song, the dancers often trade partners. First, Dave and Eve switch partners. Then, Dave and Alice switch partners. Then, Eve and Alice switch partners. Then, Claire and Bob switch partners. Finally, Dave and Alice switch partners. At the end of the dance, Alice is dancing with
Options:
(A) Patrick
(B) Sam
(C) Jamie
(D) Lola
(E) Melissa
A: (A)
Date Understanding
Q: Jane scheduled 3 appointments with 5 people for tomorrow (Tue, 7/9/1972). What is the date month ago in MM/DD/YYYY?
Options:
(A) 06/08/2055
(B) 06/22 /292/197
(C) 12/08/19))
(D) 06/U8/ /2034
(E) 006/081997
(F) 066/07/197
A: (E)
Navigate
Q: If you follow these instructions, you return starting point? Always face forward. Take step backward. Take steps left. Take 2 steps backward. Take 6 steps forward. Take 4 steps forward. Take 4 steps backward. Take 3 steps right.
Options:
Yes
No
A: No
GSMSK
Q: Megan is an actress. She was the lead actress in 80% of her work. In total, Megan participated in 100 plays. How many times was Megan not the lead actress? A: 20.0
SVAMP
Q: In a school there are 308 girls and 318 boys. There are also 36 teachers. How many pupils are there in that school? A: 626.0
Matrixships
Q: Keep track of matrix shapes through various transformations. Transpose matrix of shape (2,3,2). Transpose the result. Compute the Hadamard product the result with matrix of shape (2,3,2). Compute the Hadamard product the result with matrix of shape (2,3,2). Sum the result over the second axis. A: (2,2)
Table 6: Examples of questions in each dataset.
20

---

Published as a conference paper at COLM 2024
scenarios. However, the principle lists for GSM8K and SVAMP are less clear compared to others, since the errors in these tasks primarily relate to factual knowledge.
Svamp
1. Incorrect Mathematical Operations: Ensure that the correct mathematical operations are used to solve the problem. Misinterpretation of the problem statement often leads to incorrect operations.
2. Misinterpretation of Problem Requirements: Understand the core requirement of the question.
3. Logical Errors in Variable Initialization:
cautious when initializing variables. Make sure the initial values correctly represent the situation described in the question.
4. Misapplication of Variables in Calculation: Ensure that the variables are applied correctly in the calculation formula.
5. Understanding the Context of the Question: Contextual understanding is crucial. In
the question about pots, flowers, and sticks, it's important realize that the total number flowers and sticks a cumulative count across all pots, requiring multiplication the per pot count by the total number of pots.
Date Understanding
1. Understanding Date Arithmetic and the 'datetime' Module: Several examples demon- strate a misunderstanding of how the 'datetime' module works, especially in terms of adding or subtracting days, months, and years. The 'timedelta' function in Python doesn't 't support months or years directly, so programmers need to account for this limitation when performing date arithmetic. Instead of using 'timedelta' for date manip- ulations, use
This module provides more flexibility, especially for operations involving months and years.
2. Accurate Date Initialization: Inítialize dates correctly: In several examples, the initial date is set without considering the context of the problem. Ensure that the starting point of the calculation aligns with the scenario's 's requirements.
3. Logical Consistency in Calculations: Maintain logical consistency in calculations. If the problem states a historical or future date, ensure that the calculations reflect this timeline accurately. Avoid mixing current dates ('datetime.now()) with historical or future scenarios unless it's relevant.
4. Validating Against Given Options: When comparing calculated dates against multiple- choice options, ensure that the options are correctly formatted and compared. It's essential to format the calculated date in the same format as the options for valid comparison.
Navigate
1. Understand the Problem: Recognize that the problem requires tracking movements in two dimensions (horizontal and vertical). Understand that movements are influenced by the current direction the subject is facing. Identify the need to interpret turns as changes in direction, not just movement.
2. Initialize Variables: Define variables for horizontal and vertical movements, initializing them to zero. Introduce variable for current direction, initializing it to the starting orientation (e-g, "north").
3. Extract and Interpret Instructions: Parse the given question to extract movement and turning instructions.
4. Use control structures (if-else, loops) to handle each instruction: For movement instructions (forward, backward), update the horizontal or vertical position based on the current direction. For turning instructions (right, left, around), update the current direction appropriately. 5. Handle Directional Changes: Implement logic to correctly modify the direction state when turning. For example, turning right from north means facing east.
6. Calculate Final Position: After processing all instructions, compare the final horizontal
and vertical positions with the initial position (0,0).
7. Return Result: Return the appropriate option ("Yes" or "No") based on whether the final position matches the starting point.
8. Implement Directional Logic: Develop a mechanism to translate turning instructions
into directional changes, affecting subsequent movement calculations.
9. Consider Special Cases: Account for any special or compound instructions that may
require separate handling, ensuring all scenarios are covered.
Table 7: Examples of principle list.
21

---

Published as a conference paper at COLM 2024
GSM8K
1. Incorrect Variable Assignment and Utilization:
Ensure variable assignments accurately reflect the values they are supposed to represent based on the question's s context.
2. Misinterpretation of Quantity Relationships: Accurately understand and interpret the relationships between different quantities described in the problem statement.
3, Incorrect Mathematical Formulas: Ensure the mathematical formulas used align with
the logical requirements of the problem.
4. Misinterpretation of Variable Values: Ensure variables are interpreted and utilized
accurately based on the problem's 's context.
5. Omission of Critical Information: Incorporate all provided information and ensure no critical details are omitted in the solution.
6. Incorrect Arithmetic Operations: Ensure arithmetic operations are logically sound and
mathematically correct.
Matrixships
1. Matrix Multiplication: Use aap.matmul for matrix multiplication.
2. Hadamard Product: Use (asterisk) for element-wise multiplication (Hadamard product).
3. Transposition: Utilize np.transpose for matrix transposition. Do not specify the axes parameter when using np. ap.transpose.
4. Resultant Matrix Shape: Always return the shape of the resulting matrix after an operation.
Table 8: Examples of principle list.
H.5
Problem-solving instruction examples
We provide an example of problem-solving instruction for a better understanding. The problem-solving instruction consists of problem-solving method and several examples showing how to use the method. The questions in the examples are training questions.
H.6 Overall student prompt examples
We provide an example of the overall teacher model's instruction for a better understanding, The overall instruction prompt consists of the modified problem-solving instruction and several newly selected examples from the validation set. Examples 1-3 are original examples in the problem-solving instruction (from the training set). Examples 4-6 newly selected examples from the validation set.
22

---

Published as a conference paper at COLM 2024
Listing 1: An example of problem-solving instructions provided by the teacher model. ###Problem -eweem-soollll solving instruction for CoinFlip task
Method to solve coin filp problems:
1. Start by reading the initial state the coin (heads up or tails up).
2. Examine the actions of each person mentioned in the question in the order they are mentioned. If a person flips the coin, the state of the coin will change. If it was heads up, it will now be tails up, and vice versa.
If person does
flip the coin, the state remains unchanged. At the end of the actions, state the final position of the coin as 'heads up or 'talls up
3. Answer the question based the final state of the coin compared to the state asked the question.
Examples:
10
Question: coin is heads up. sager does not flip the coin. zyheir flips the coin. Is the coin still heads up?
12 Explanation:
13 The coin starts as heads up.
Sager does not flip the coin, it remains heads up.
15 Zyheir flips the coin, changing its state. Now, it's tails up.
16 The final position of the coin is tails up.
17
18 Answer: No.
20 Question: coin heads up. mailey does not flip the coin. maurisa does
not flip the coin. Is the coin still heads up?
21 Explanation:
heads up Mailey does not flip the coin, it remains heads up. Maurisa also does not flip the coin, so stays heads up.
The coin starts
24
25 The final position of the coin is heads up.
26
Answer: Yes.
28
29 Question: coin is heads up. murraylee does not flip the coin. meilich
flips the coin. Is the coin still heads up?
Explanation:
32 The coin starts as heads up.
33 Murraylee does not flip the coin, so it remains heads up
31 Meilich flips the coin, changing its state. Now, it's tails up. The final position of the coin is tails up.
36 Answer: No.
Listing An examples of overall prompt provided by the teacher model.
###Overall prompts for 'Navigate"
task
To write Python functions answer questions about returning to the starting point after following a set of movement instructions, follow these steps:
Understand the Problem:
Recognize that the problem requires tracking movements in two dimensions (horizontal and vertical).
Understand that movements are influenced by the current direction the subject is facing.
Identify the need interpret turns changes direction, not just movement.
Design the Function:
23

---

Published as a conference paper at COLM 2024
Initialize Variables:
13
Define variables for horizontal and vertical movements, initializing them to zero.
14
Introduce variable for current direction, initializing it to the starting orientation (e.g., "north").
Extract and Interpret Instructions: Parse the given question to extract movement and turning instructions
16
17
Use control structures (if-else, loops) to handle each instruction: For movement instructions (forward, backward), update the horizontal vertical position based the current direction.
19
For turning instructions (right, left, around), update the current direction appropriately.
20
Handle Directional Changes:
Implement logic to correctly modify the direction state when turning. For example, turning right from north means facing east.
22
Calculate Final Position:
23
After processing all instructions, compare the final horizontal and vertical positions with the initial position (0,0). Return Result:
25
Return the appropriate option ("Yes" or "No") based on whether final position matches the starting point.
26
Implement Directional Logic:
28
Develop mechanism to translate turning instructions into directional changes, affecting subsequent movement calculations.
29
Test the Function:
Write test cases to execute the function with different sets of instructions.
32 Store and verify the result in a variable answer to ensure the function behaves as expected.
34 Here are some examples:
35
36 Question: If you follow these instructions, do you return to the starting point? Always face forward. Take 1 step backward. Take g steps left. Take steps backward. Take steps forward. Take steps forward. Take 4 steps backward. Take steps right. Options: Yes No
37
Answer:
39 python
40
def do noo_you_retuun_ttoor totturr_tt_strrtinn_ooittt
41
# Initial position and direction
horizontal, vertical direction "north"
48
15
# Movement instructions instructions =
d6
("backward", 1),
("left", 9)
49
("backward" 2).
50
("forward", 6)
51
("forward", 4),
("backward", 4),
53
("right", 3)
54
]
55
56
# Process each instruction
for action, steps in instructions: if action "forward":
58
59
if direction vertical += steps elif action == "backward":
"north":
60
24

---

Published as a conference paper at COLM 2024
if direction "north": vertical steps elif action = "left":
63
64
65
if direction = "north":
horizontal - steps
elif action == "right": if direction "north": horizontal += steps
68
69
70
Check if returned to the starting point
72
return "Yes" if horizontal and vertical else "No"
23
74
Execute the function
75
answer oogyourreuunnoossttrrrnnoononn
77 By using the python function above, you can get answer the question.
78 #*#END
Question:
'If you follow these instructions, do you return to the starting point? Always face forward. Take 10 steps left. Take steps forward. Take steps forward. Take 2 steps forward. Options:
Yes No
82 Answer:
83 python
84
def do www.rrurnnttsstartinn..... ():
# Initial position and direction
horizontal, vertical
87
direction "north"
88
89
# Movement instructions instructions [
("left", 10)
92
("forward", 10),
93
("forward", 7).
("forward", 2)
96
97
# Process each instruction
98
for action, steps in instructions:
if action == "forward":
if direction
"north":
101
vertical += steps elif action == "left": if direction == "north": horizontal
102
steps
105
106
Check if returned to the starting point
107
return "Yes" if horizontal == and vertical == 0 else "No"
Execute the function
110
answer do.you.eeuum.t.ssttrinn.....o
111
By using the python function above, can get the answer of the question.
#*#END
114
115
Question:
'If you follow these instructions, do return to the starting point? Always face forward. Take step right. Take 3 steps left. Take 2 steps right. Options: Yes No
116
117 Answer:
python def to..oouretuum.ossttrttn..ooooo
25

---

Published as a conference paper at COLM 2024
Initial position and direction horizontal, vertical 0, direction "north"
122
# Movement instructions instructions = ("right", 1),
126
127
("left", 3),
128
("right", 2)
]
131
Process each instruction
132
for action, steps in instructions: if action == "left": "north":
if direction
135
horizontal elif action == "right": if direction == "north": horizontal += steps
steps
136
137
140
Check if returned to the starting point
141
return "Yes" if horizontal == and vertical == 0 else "No"
142
Execute the function
144
answer vo_yooretunn.oo.sarrinn.ponnn
145
146
By using the python function above, you can get the answer of the question.
###END
148
149
Question: 'If you follow these instructions, do you return to the
starting point? Take 3 steps. Turn around. Take 3 steps. Turn right. noptions: An- Y No
151
Answer:
152
python
def www.rretrrnttssstttinnng.oo
Initial position and direction
horizontal, vertical
156
direction "north"
Movement instructions instructions
160
("forward", 3), ("turn", "around"), ("forward", 3), ("turn", "right")
161
]
165
# Process each instruction
for action, value in instructions:
if action "forward":
169
if direction vertical += value
"north":
170
elif direction = s:
vertical value
elif direction "east":
174
horizontal += value
175
elif direction == "west":
horizontal - value
elif action == "turn":
178
if value "around":
179
direction "south" if direction "north" else "north th
elif value == "right":
if direction "north":
26

---

Published as a conference paper at COLM 2024
direction "east" elif direction "east":
184
direction "south"
185
elif direction = "south":
direction "west"
elif direction "west":
188
direction "north"
189
# Check if returned to the starting point
return "Yes" if horizontal == and vertical == else "No"
193
Execute the function
194
answer
195
By using the python function above, you get the answer of the question.
197
#*#END
198
Question: 'If you follow these instructions, do you return to the
starting point? Take steps. Turn around. Take steps. Turn right. Turn right. Take step. Take 1 step. AnOptions An- Yes An-
200
201
Answer:
python
203
def Initial position and direction
204
205
horizontal, vertical = 8,
direction "north"
208
Movement instructions instructions =
209
210
("forward", 3),
("turn", "around"), ("forward", 5), ("turn", "right"), ("turn", "right"), ("forward", 1), ("forward", 1)
212
213
214
217
]
218
# Process each instruction for action, value in instructions:
221
if action "forward":
222
if direction
"north":
223
vertical += value
elif direction == "south":
vertical value
226
elif direction
"east":
227
horizontal += value
228
elif direction = "west":
horizontal - value
elif action == "turn":
231
if value "around":
232
direction "south" if direction = "north" else "north"
elif value == "right":
if direction "north":
235
direction "east"
236
elif direction == "east":
237
direction "south"
elif direction = "south":
direction "west"
240
elif direction "west":
211
direction "north"
212
Check if returned to the starting point
27

---

Published as a conference paper at COLM 2024
return "Yes" if horizontal and vertical else "No"
245
246
Execute the function
247
answer
By using the python function above, you get the answer of the question.
250
#*#END
251
Question: 'If you follow these instructions, do you return to the starting point? Take step. Take steps. Turn around, Turn around. Turn around. Take 6 steps. :noptions: Ye No
253
254
Answer:
python
256
def # Initial position and direction
257
258
horizontal, vertical = 8,
direction "north"
261
Movement instructions instructions =
262
263
("forward", 1),
("forward", 5), ("turn", "around"), ("turn", "around ("turn", "around"). ("forward", 6)
266
267
]
270
271
# Process each instruction
272
for action, value in instructions:
if action == "forward":
274
if direction
"north":
275
vertical value
276
elif direction == "south":
vertical - value
elif direction "east":
279
horizontal
value
280
elif direction == "west":
horizontal - value
elif action == "turn":
if value "around"; direction "south" if direction "north" else "north"
284
285
# Check if returned to the starting point
return "Yes" if horizontal == and vertical else "No"
288
289
Execute the function
290
answer oo_youreeuum.ttsstrrinn.oooonnn
By using the python function above, you get the answer of the question.
293
###END
28


Reviews:
[{"review_id": "ByXoJRuzi4", "reviewer": "Reviewer_R4tU", "summary": "This paper proposes a method to effectively transfer advanced reasoning capabilities of stronger LLMs to weaker LLMs by exploiting general principles that indicate gaps in a student’s understanding on practice problems. They outperform CoT prompting on symbolic and arithmetic reasoning.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "**1. Availability and selection of training examples**:\n\nWe appreciate your pointing this out and agree with your judgment. In our experimental settings, we randomly select 25% of the data from the training set to serve as the validation set. A larger set of practice questions enhances principle generation. The table below shows the influence of the size of the validation set on TPD’s performance (on GPT-3.5).\n\n|                | 5%  | 15% | 25% | 35% |\n|----------------|-----|-----|-----|-----|\n| **Navigation** | 86.5| 92.5| 97.5| 97.5|\n| **Matrix**     | 88.0| 91.5| 93.5| 94.0|\n\nWe observe that increasing the size of the validation set indeed improves the framework's performance. If we had only 3 examples in the validation set, the framework would degenerate into few-shot learning.\n\nWe believe that a higher coverage of principle types can help optimally select training examples. To demonstrate this, we compare the results from different numbers of principles:\n\n|                | Half Principle List | Full Principle List |\n|----------------|---------------------|---------------------|\n| **Navigation** | 93.0                | 97.5                |\n| **Matrix**     | 89.5                | 93.5                |\n\nOur TPD method selects examples based on the number of violations against the principle list. However, we can modify it to ensure higher coverage of the principle list, further optimizing example selection.\n\nFor open-ended generation tasks, there is no single correct answer for each question. However, there are principles related to the quality or human preference of the generation, such as factuality and helpfulness. In these settings, we can choose examples based on questions that are prone to violating these principles. These scenarios are more akin to alignment tasks, which cannot be easily addressed solely through prompting.\n\n**We really appreciate the reviewer's valuable feedback. Since there are many questions to be discussed, we put the remaining response in the following anonymous link** \n\nhttps://docs.google.com/document/d/1MOgF8-LhPx6fd5knME4favjqYEJ0XqrzAJ_aauj0hEg/edit?usp=sharing\n\n**We do not add a new revision or a large amount of new content in this anonymous file. It only contains responses to the reviewer's questions.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The idea of principle discovery and exploitation is novel (though there are other recent related work (e.g., cite: LEAP [1]), but we can discount those given they are just 3 months old though eventually they must be included).  \n- The setting is nicely grounded conceptually in human teaching. \n- The method does not require the expensive teacher model at inference time.\n- The paper is generally well-written.", "reasons_to_reject": "**Availability and selection of training examples**: \nThe paper assumes that the training examples are available in the principle generation stage (constituting a validation set for initial practice) . A realistic setting for real-world tasks and benchmarks like Big Bench Hard is where typically only a handful of examples are available. How many examples are sufficient for helpful problem solving instructions and error summarization? Is there a method to optimally select from the training examples e.g. with a higher coverage of question types (or principle types)?\n\nWhat about other tasks like open ended generation. In many such real world examples, it is not easy to compare against the ground truth and measure most error prone questions. In those practical setups, how will you choose examples based on the principle list containing the most error-prone questions for the student model to help the student model learn from errors effectively (in section 4.4 last para, it is stated that most error prone questions are required for effective learning in the principle generation stage)?\n\n\n**Issues in Principle exploitation**:\nGenerated Principles in Figure 4 are too high level (\"_Logic flow: ensure arithmetic operations are logically sound and mathematically correct_\"). Is this example really providing any actionable insights to a model during inference? How do you know which kind of error is triggered so that you can select a principle? Also fundamentally, how are principles different from feedback?  Other similar papers (e.g., cite: CLIN [5]) that have a two stage pipeline that includes “error summarization” that they call meta memory. But their summarized reflections are distilled into knowledge necessary to solve the task. \n\nRelated to this: How do you pull past principles (did you try different retrieval methods)? Past similar methods (cite: Memprompt) that exploit past mistakes for better inference have documented the importance of retrieval quality. How will you scale the method as the principle list grows?\n\n**Missing error analysis**:\nIt would be helpful to know the kind of errors the system makes. e.g., what percentage of the errors are related to principle generation and exploitation. There are many details related to principle retrieval issues, teacher making mistakes in error identification or just the fact that a principle is unhelpful or the model fails despite providing a helpful principle. The ablation studies were helpful (thank you), but error analysis would give a clear picture.\n\n**Missing Related work and Baselines**: \nThere has been past work on exploiting model answers to create a memory/ list of knowledge (cite: CLIN [5]), feedback (cite: MemPrompt [4]), principles (cite: LEAP [1]). Also, Reflexion [2] learns per-example feedback and LEAP [1] learns per-task principles. Can you compare against these methods, either empirically or theoretically? It seems like some of this is very closely related work (e.g., LEAP [1]) and it would be good to compare with these (though empirical comparison with LEAP [1] is not at all a ground for rejection as it is a very new paper but it would be useful to compare it broadly).  \n\nSection 4.4 page 8 presents an interesting finding that the LLM diverges significantly from its previous answer if a critique or feedback is used. Is this a previously established finding / can you support this claim with any references? This looks contrary to other papers (e.g., cite: [3] page 7) so it is important to substantiate.\n\nAlso, please distinguish your work in the related work, in addition to mentioning other papers. Eg., it seems like you also search for principles applied over mistakes from most representative train examples? (so there must be a retrieval module here like other papers?)\n\n**Results are not very convincing**:\nThe claim made in the paper are quite broad. However, the empirical results are not too convincing. The improvement over AutoCoT or 3-shot CoT is not significant and it is raises the question whether we can justify a more complex method like yours. It would be helpful to also have a simple baseline that takes the k-examples as demonstrations whose principles are applied in your method. \n\n\n**Other Missing details**: \nOn page 4, the teacher can make mistakes in accurately identifying the incorrect answers in the error set. Can you clarify if the teacher sometimes cannot give the correct modified instructions or the correct principle? Can the teacher identify when an answer is wrong (the paper mentions that these examples are taken from a validation set (question, answer) -- so why is it hard to detect if there is an error e.g., in math, the answers are clear (but maybe in open ended generation it is not easy to identify mistakes). I am assuming that at test time, there is no teacher involved as mentioned in the paper.\n\n\n**Minor typos**:\ntypo in fig 1: promblem (I have not checked more thoroughly for spelling mistakes elsewhere)\n\n\nReferences:\n1. In-Context Principle Learning from Mistakes (LEAP): https://arxiv.org/pdf/2402.05403\n2. Reflexion: Language agents with verbal reinforcement learning: https://arxiv.org/abs/2303.11366\n3. Learning to Repair: Repairing model output errors after deployment using a dynamic memory of feedback: https://arxiv.org/pdf/2112.09737 \n4. Memory-assisted prompt editing to improve GPT-3 after deployment: https://arxiv.org/abs/2201.06009\n5. CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization: https://arxiv.org/abs/2310.10134", "questions_to_authors": "Please go through the reasons to reject, and you can respond. If I receive clarifying satisfactory answers then I am willing to increase my rating.", "extra_scores": {}}, {"review_id": "SdAd3bnXDC", "reviewer": "Reviewer_Yx4q", "summary": "The paper proposes a method to design task-specific zero-shot or few-shot prompts through interaction between a \"teacher\" and \"student\" model, using an existing dataset of task-specific question-answer pairs. This involves a multi-step process summarized below:\n1. The teacher model selects representative samples from the dataset and generates an initial instruction on how to solve the task, along with few-shot examples of how to solve the selected samples. (this is the baseline prompt denoted TPD w/o ES in Table 1, to my understanding)\n2. The student model is prompted to solve questions in a validation set using the initial prompt constructed above, and an error set $E$ is built.\n3. The teacher model is prompted to solve the same questions in $E$. Each question that the teacher can solve correctly is included in the feasible error set $\\hat{E}$. Incorrect questions are discarded as they are deemed too difficult for even the teacher.\n4. The teacher model build a set of principles $P$ that is required to address the errors in $\\hat{E}$ through an iterative process.\n5. The resulting principle list is filtered by human reviewers.\n6. A new set of representative questions are selected from the validation set, which violate the most number of principles in $P$.\n7. The initial instruction from step 1 is revised based on $P$, and new examples of how to solve the questions selected in step 6 are appended to the original examples from step 1 (this is the prompt denoted TPD w/ ES in Table 1, to my understanding).\n\nThe final prompt outperforms zero-shot, few-shot, and auto-cot on various symbolic and arithmetic reasoning tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "**1. The requirement of large task-specific datasets,...**:\n\nIn our settings, we do not consider finetuning baselines as they are time-consuming. In this study, we mainly explore the interactions between language models. We want to study how to let an advanced LLM teach a weaker LLM through interaction effectively. We discussed finetuning methods and their limitations in the related works. In TPD, the additional computational cost mainly comes from the teacher model.  TPD can also work streaming setting where a small batch of test questions (say m questions) arrive at a time like in data streams. \n\nInitialize $M_0$; When batch 1 of questions $q_1^1$ ,...,$q_1^m$ arrive, the teacher model provides a problem-solving instruction based on these questions and the student model can utilize the instruction for each $q_1^i$ to obtain its answer $a_1^i$. Add these question-answer pairs $[(q_1^1, a_1^1), (q_1^2, a_1^2), (q_1^m, a_1^m)]$ to $M_0$ and call the new set $M_1$.\n\nWhen batch $b$ (b>1) of questions $q_b^1 ,...,q_b^m$ arrive, the teacher model can build a principle list $P$ based on $M_{b-1}$ as described in TPD, and then utilize $P$ to select examples and modify the problem-solving instructions to build a final prompt for this batch of questions. The student model then uses the demonstrations for each question $q_b^i$ to get an answer $a_b^i$. Finally, add these question-answer pairs $[(q_b^1, a_b^1), (q_b^2, a_b^2), (q_b^m, a_b^m)]$ to $M_{b-1}$ and call the new set $M_b$.\nWe’ll add these details to the revised manuscript.\n\n**2.The need for human intervention**\n\nHuman intervention is necessary because the teacher model may not always accurately summarize the meaningful principles derived from the student model's mistakes. Human reviewers are involved only in deleting any principles that are erroneous or confusing. This task is straightforward and does not impose a significant burden, thus maintaining the method's overall efficiency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method outperforms previous prompt construction methods on a wide range of reasoning tasks across two different student models.", "reasons_to_reject": "1. The requirement of large task-specific datasets, and thus the associated costs, limits the applicability of this method. It may be more cost-effective to fine-tune a smaller model to achieve the same performance. This has not been considered in the paper. Note, while Auto-CoT has the same limitations, they mitigate this limitation by demonstrating a streaming-based variant of their approach, where examples are selected from test batches.\n1. The need for human intervention in step 5 of my summary somewhat limits the applicability of this method.\n1. The difference between the experimental settings of Figure 3 (a), (b) vs (c) is unclear. Both seem to ablate the number of examples.\n1. It is hard to conclude that models *benefit* from principles deduced from errors of another model from Table 3. That would require comparison with baselines (e.g., zero-shot CoT, maybe few-shot CoT, principles derived from original samples, etc.).\n1. (Minor) The method is fairly complicated.\n1. (Minor) The writing is hard to follow. E.g.,\n    1. The method explanation and Figure 2 are hard to follow. It may be beneficial to divide the method into a clear series of steps which align with the subsections of the method section.\n    1. It is not 100% clear what TPD w/ and w/o ES in Table 1 refers to (I've written my interpretation in the Summary)\n    1. It is hard to follow which parts of the methodology explained in Section 3 corresponds to which ablations in Table 2, and it is unclear what method was used in Figure 3 and Table 3.\n    1. The use of the word \"teaching\" throughout the abstract and intro may mislead the reader to initially believe this is a student fine-tuning paper. It would help the reader to highlight that the method's goal is to construct a (few-shot) prompt.", "questions_to_authors": "1. Is there a specific reason behind the selection of Mixtral 8x7B–an MoE model–as the sole open-source student model?", "extra_scores": {}}, {"review_id": "lR9XwVzDAg", "reviewer": "Reviewer_AV4e", "summary": "Large language models (LLMs) possess substantial reasoning abilities; however, they entail significant computational demands. Prior research has explored knowledge-distillation-based fine-tuning methods to confer these capabilities onto smaller models, though these methods are resource-intensive. This paper presents a novel approach, Teaching via Principle Discovery (TPD), which enhances the reasoning skills of smaller language models (student LLMs) by leveraging insights from larger models (teacher LLMs) without necessitating ongoing intervention during inference. TPD adopts human teaching methodologies through a two-phase procedure: principle generation and principle exploitation, guided by error analysis and the formulation of corrective principles by the teacher model.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Inclusion of additional case studies would enhance the paper.**\n\nThanks for pointing out. We will add a new figure to visualize and compare the original examples, extensive examples, and the principle list for a better understanding in the next version.\n\n**2. The paper should introduce and discuss failure scenarios more.**\n\nWe appreciate the reviewer’s feedback and acknowledge the importance of discussing the failure scenarios and limitations of TPD.\n\nFirstly, TPD effectively teaches student models how to approach problem-solving through principles. However, it does have a limitation in imparting factual knowledge for specific questions due to the fixed prompt used during inference. This limitation can be mitigated by incorporating a retrieval module in the student model's inference stage. This module would retrieve prior experiences and feedback from practice questions, enhancing the model's ability to provide factual answers.\n\nSecondly, we believe that TPD has the potential to be applied to other domains, such as complex reasoning tasks and agent-based tasks. Principles serve as intuitive guides for human thinking and planning, and LLMs can similarly benefit from these principles to improve their performance in diverse and complex domains.\n\n**3. Discussions on how well the principles learned can be generalized and transferred**\n\nWe will compare different baselines and extend Table 3 in the next version. Meanwhile, we will also visualize a comparison between principles for two LLMs for a better understanding of the transferable setting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The introduced framework is innovative, proposing principle discovery as an interesting new direction for transferring the reasoning powers of LLMs to smaller models.\n   \n2. The method is not just innovative, but also rigorously designed. The two-phase process of principle generation and exploitation is meticulously planned, effectively emulating human instructional techniques. This careful design ensures the method's reliability and validity.\n\n3. The empirical evidence demonstrates notable enhancements over conventional methods such as chain-of-thought prompting, making a strong case for TPD's effectiveness.", "reasons_to_reject": "1. Inclusion of additional case studies would enhance the paper. While TPD demonstrates promising results for symbolic and arithmetic reasoning tasks, a more detailed exposition with extensive examples could clarify its advantages to readers.\n\n2. The paper should introduce and discuss failure scenarios more. Currently, the limitations of TPD and its applicability to tasks beyond the tested domains are not sufficiently discussed.\n\n3. Discussions on how well the principles learned can be generalized and transferred to different student models or a wider array of tasks are limited and could be expanded.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aAjCM6WVXJ", "reviewer": "Reviewer_ZvuT", "summary": "This paper proposes a simple method to improve the performance of the student model. Specifically, the teacher model generates problem-solving instructions and corrective principles based on the student LLM's errors. Then during inference, the student model can guide the inference process without the help from the teacher model.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. The method involves the labeling process of both large teacher model and human reviewers, which is complex and resource-consuming.**\n\nOur work primarily explores the interactions between LLMs, aiming to study how an advanced LLM can effectively teach a weaker LLM through interaction. The involvement of a large teacher model is essential for this process, as it aligns with the standard practice in other studies [1][2] focusing on the teacher-student framework.\n\nWhile it is true that human reviewers are necessary to ensure the quality of the principles summarized by the teacher model, their role is limited to identifying and removing any erroneous or confusing principles. This task, while important, is simple and manageable. The human reviewers' involvement ensures the reliability and accuracy of the learning process, ultimately enhancing the effectiveness of our approach.\n\n**2. The experimental result and Baseline**\n\nWe appreciate the reviewer’s feedback. TPD can guide student models on how to solve these types of questions through principles but cannot provide factual knowledge for each specific question due to the use of a fixed prompt during inference time. The performance gain on some arithmetic reasoning tasks is subtle, primarily because of this lack of factual knowledge. This limitation can be addressed by incorporating a retrieval module during the student model's inference stage to retrieve prior experiences and feedback from practice questions. We plan to include related experiments in the appendix of the next version.\n\nRegarding the baselines, Automate-CoT is capable of selecting the optimal combination of several rationale chains, which can also be implemented within a teacher-student framework. We will include Automate-CoT as a new baseline in the next version.\n\n**3. The experiments are only conducted on GPT-3.5**\n\nWe would like to kindly point out that we have also conducted experiments on Mixtral 8x7B. The results are shown in Table 1. We also plan to add llama3 models as students in the revised version.\n\n\n[1] Ho, Namgyu, Laura Schmid, and Se-Young Yun. \"Large language models are reasoning teachers.\" arXiv preprint arXiv:2212.10071 (2022).\n\n[2] Saha, Swarnadeep, Peter Hase, and Mohit Bansal. \"Can language models teach weaker agents? teacher explanations improve students via theory of mind.\" arXiv preprint arXiv:2306.09299 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method is intuitive which imitates the real teaching scenarios in schools. \n2. The analysis of the experiments is detailed and convincing.", "reasons_to_reject": "1. The method involves the labeling process of both large teacher model and human reviewers, which is complex and resource-consuming. \n2. The experimental result on some arithmetic reasoning tasks achieves a very subtle improvement. More seriously, it seems that some important baselines are missing, such as Automate-CoT. \n3. The experiments are only conducted on GPT-3.5, which is a relatively weak LLM. It would be better if the authors could provide the experimental results on other LLMs.", "questions_to_authors": "Please refer to the reasons to reject.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a method for prompting student models based on error analysis and guidance from a larger model. Evalauting this approach on 2 LLMs with a number of tasks demonstrates higher performance compared to CoT variants. Ablations demonstrate that reasonable performance can be obtained with few shot settings which will reduce cost of inference. All high quality reviews assign a score of marginal accept and the reviewer score improved during the rebuttal phase: this introduced several new baselines and ablations which will result in a stronger paper. There are several common weaknesses identified by the reviewers, particularly around the failure cases and handling factual knowledge. I look forward to seeing an updated version of the paper or future work which would discuss these issues.

[At least one review was discounted during the decision process due to quality]
Decision: Accept