TPD: Enhancing Student Language Model Reasoning via Principle Discovery and Guidance

Authors: haorui  wang
rongzhi  zhang
yinghao  li
lingkai  kong
yuchen  zhuang
xiusi  chen
chao  zhang
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 11:29:53

Abstract: Large Language Models (LLMs) have recently showcased remarkable reasoning abilities. However, larger models often surpass their smaller counterparts in reasoning tasks, posing the challenge of effectively transferring these capabilities from larger models. Existing approaches heavily rely on extensive fine-tuning data or continuous interactions with a superior teacher LLM during inference. We introduce a principle-based teacher-student framework called Teaching via Principle Discovery (TPD) to address these limitations. Inspired by human learning mechanisms, TPD mimics the interaction between a teacher and a student using a principle-based approach. The teacher LLM generates problem-solving instructions and corrective principles based on the student LLM's errors. These principles guide the refinement of instructions and the selection of instructive examples from a validation set. This enables the student model to learn from both the teacher's guidance and its own mistakes. Once the student model begins making inferences, TPD requires no further intervention from the teacher LLM. Through extensive experiments across eight reasoning tasks, we demonstrate the effectiveness of TPD. Compared to standard chain-of-thought prompting, TPD significantly improves the student model's performance, achieving an average improvement of 6.2\%.

Main paper: 
 introduction: 
 Recent studies show that large language models (LLMs) can achieve impressive performance in various reasoning tasks, such as analogical (Webb et al., 2023) , arithmetic (Imani et al., 2023) , symbolic (Pan et al., 2023a) , and commonsense reasoning (Wei et al., 2022b; Bang et al., 2023) . However, a noticeable performance gap can often be observed between stronger LLMs such as GPT-4 and weaker LLMs such as GPT-3.5-turbo (Espejel et al., 2023; OpenAI, 2023; Cai et al., 2023) . This disparity arises from factors such as training data size, model capacity, and the methods by which LLMs learn and encode world knowledge. While stronger LLMs exhibit superior performance, their practical application is hindered by the high costs associated with training and inference. For instance, as of this writing, the cost of using GPT-4 is over ten times higher compared to GPT-3.5-turbo. This raises the question: how can we effectively transfer the advanced reasoning capabilities of stronger LLMs to weaker ones?Several approaches have been proposed to address this challenge. Some studies (Rajani et al., 2019; Ho et al., 2023) curate datasets for specific downstream tasks using stronger LLMs and then fine-tune weaker LLMs on these datasets to instill the necessary knowledge. However, this fine-tuning process is time-consuming, and the resulting task-specific weaker LLMs lack generalizability to other tasks. Wang & Li (2023) use an assistant language model to provide guidelines and analysis for the student model, but this approach requires constant involvement of the assistant model, which can be costly. To reduce teacher model involvement, Saha et al. (2023) request teacher intervention only when the student model exhibits low confidence, where the confidence is computed from the token probability of the answer. Such a measure may not accurately reflect the student model's true confidence, as the token probability of the answer is influenced by the preceding output, specifically the chain-of-thought (CoT) explanations. To address these limitations, we introduce Teaching via Principle Discovery (TPD), a principle-based teaching framework that minimizes teacher model involvement, thereby optimizing resource allocation and efficiency. TPD draws inspiration from instructional strategies observed in natural human teaching and learning processes (Rosenshine, 2012; Henderson & Harper, 2009; Metcalfe, 2017) , which follows a structured Demonstrate-Practice-Review process: 1) Demonstrate: The teacher introduces a problem type and demonstrates how to solve it. 2) Practice: The student engages with practice questions. 3) Review: The teacher reviews the student's responses to identify common errors. This review process enables the teacher to extract finegrained corrective principles that guide the student in rectifying errors and improving future problem-solving. TPD comprises two stages: principle generation and principle exploitation. In the principle generation stage, the teacher model generates problemsolving instructions and summarizes principles based on errors made by the student model on validation questions. In the principle exploitation stage, the teacher model constructs instructive examples that illustrate the principles and injects these instructions into the prompt to guide student learning. TPD avoids teacher model involvement during inference, enabling the student model to operate independently in offline scenarios.We validate TPD on eight reasoning tasks covering symbolic and arithmetic reasoning.Through principle generation and exploitation, TPD significantly improves the performance of the student model without teacher model intervention during inference. Specifically, our model achieves an absolute gain of up to 19% accuracy compared to chain-of-thought prompting. Additionally, we explore different methods for injecting generated principles into the student model and find that selecting new examples from practice questions outperforms direct injection and the critique-revise method. 
related work: 
 Teacher-student framework Traditional teacher-student frameworks employ fully supervised fine-tuning methods (Magister et al., 2022; Shridhar et al., 2022; Hendrycks et al., 2021; Rajani et al., 2019) . For instance, Ho et al. (2023) generate CoT reasoning steps by GPT-3 (Brown et al., 2020) and fine-tune several relatively small models with these generated data. However, both the fine-tuning process and collecting task-specific fine-tuning data are time-consuming and challenging. To overcome these limitations, there are several prompting-based teacher-student frameworks recently (Pruthi et al., 2022; Saha et al., 2023; Yu et al., 2023) . In these frameworks, the teacher model is tasked with offering demonstrations or explanations to the student model. However, these approaches require the intervention of a teacher model on the test set. Hence, they are unsuitable for offline scenarios, where the student model must solve problems independently, without assistance from the teacher model.Emergent abilities bring a strong few-shot learning ability for LLMs across various datasets via in-context learning (Wei et al., 2022a; b) . Many prompting-based methods are proposed to elicit the reasoning abilities of the LLMs by injecting knowledge into prompts. For instance, chain of thought (CoT) (Wei et al., 2022b) and its variants (Kojima et al., 2022; He et al., 2023) provide a few human written examples or instructions to LLMs. We summarize more prompting methods in Appendix B. These prompting methods are effective in eliciting the reasoning ability of the LLMs. However, they are not suitable for direct adaptation within the teacher-student framework, as they do not involve knowledge transfer from a teacher model. Another pipeline of prompt engineering is automatic prompt searching, also known as prompt optimization (Khattab et al., 2023; Yang et al., 2023a) . These methods employ LLMs to refine prompts iteratively in the zero-shot setting for a specific task. However, applying automatic prompt searching methods for prompt optimization is time-consuming, as it requires multiple rounds of inference on the training dataset; in contrast, TPD requires only a single-round knowledge transfer. Moreover, TPD includes an error summarization stage, which aligns with learning from feedback prompting methods. These methods initially create a memory list. When a new query arises, they retrieve relevant information from this memory to refine and enhance the current prompt. The retrieved information may consist of successful trials from history (Majumder et al., 2023) , the most similar questions with user feedback (Madaan et al., 2022a) , or summaries of previous failures (Shinn et al., 2024) .Rule discovery is a popular technique in machine learning and data mining (Tweney et al., 1980; Fürnkranz & Kliegr, 2015; Das et al., 1998) .In language models, rule discovery is often utilized in learning from feedback frameworks (Zhang et al., 2022a; Pan et al., 2023b) . In (Zhu et al., 2023) , rules are generated by language models and subsequently selected through self-verification. Yang et al. (2023b) propose a framework where language models could generate rules by learning from previous mistakes. However, the rules produced in these methods tend to be concrete facts, with the framework functioning as a retrieval system. In contrast, a rule is a formal, broad statement that applies to an indefinitely large set of objects. Unlike rules, principles are more abstract and open to interpretation, offering high-level guidance without strict formatting. For instance, Bai et al. (2022) predefine principles about helpfulness and harmfulness by human experts, then ask the language model to evaluate the generation results to make RLAIF. One concurrent work on principle discovery is LEAP (Zhang et al., 2024) , which focuses on summarizing both high-level and low-level principles based on a model's previous mistakes. The key distinction between TPD and LEAP is that we incorporate principle discovery within a teacher-student framework, aiming to enhance knowledge transfer through these principles.In contrast, LEAP primarily demonstrates that LLMs also discover principles from their previous mistakes. 
overview: 
 In TPD, we consider a teacher model and a student model. The student is responsible for solving reasoning problems based on the teacher model's guidance. The primary goal of our framework is to let the teacher model effectively teach the student model how to solve these tasks. As shown in Fig. 2 , the framework is divided into two stages: principle generation and principle exploitation. During the principle generation stage, the teacher model produces a problem-solving instruction based on sampled questions and identifies a principle list P from the errors made by the student model. In the principle exploitation stage, the principle list P is injected into the student model to improve its reasoning capabilities in downstream tasks.In this stage, we employ the teacher model to provide a high-level guideline to the student model and help the student model identify its common errors from a few practice questions. This principle generation stage comprises two sub-stages: 1. Problem-solving instruction generation and initial practice and 2. Error summarization. The two-stage process mirrors the traditional classroom learning model, where teachers provide instructions, students practice, and teachers offer feedback based on observed errors.What is problem-solving instruction? The problem-solving instruction I consists of a problem-solving method augmented with a few examples. To build the instruction, the teacher model reviews a set of questions sampled from the training set, identifying what type of questions the student model needs to solve. Then, the teacher model generates a problem-solving method as an initial instruction in natural language. To enhance the instruction, it is further enriched with examples that demonstrate the application of the proposed method. These examples are derived from the same questions initially sampled. We provide an example in Appendix C for a better understanding.Why problem-solving instruction? Previous works (Saha et al., 2023; Wang & Li, 2023) ask the teacher model to offer guidance for each test question, rendering the teaching framework impractical for offline scenarios. To tackle this issue, we ask the teacher model to provide high-level problem-solving instruction based on sampled questions and internal knowledge. The problem-solving instruction is adaptable, making it suitable for offline use as it generalizes across similar types of questions. Moreover, since many prompting methods (Wei et al., 2022b; Gao et al., 2023b; Zhou et al., 2022) have shown that LLMs have a strong capability to learn from examples, the problem-solving instruction includes a few examples to help the student model understand and imitate the instruction. This process also follows the Case Teaching Method paradigm (Herreid, 2005) , where the teacher models provide several cases, and the student model can imitate and learn from the examples. The problem-solving instruction reflects the key principles to solve the type of problems, as it provides fundamental guidance for addressing specific types of problems.With the problem-solving instruction, the student model generates answers for the questions in the validation set. This step lets the student model practice, allowing the teacher model to assess its comprehension of the problemsolving instruction and its ability to apply it effectively.Data filtering After the student practices over the validation set, we evaluate its responses to build an error set E = {e 1 , e 2 , ..., e n }, where each element e i in this set represents a pair consisting of a question and the corresponding incorrect answer made by the student model. Although the teacher model generally performs well on various tasks, it can make mistakes. Hence, we need to assess the teacher model's ability to accurately identify the incorrect answers in E. Specifically, we check if the teacher model agrees with the student's incorrect answers, removing each e i that the teacher model can not identify to build a feasible error set Ê. Due to the context length of the LLMs, we then develop an iterative framework to summarize principles from each e i iteratively.In Algorithm 1, we denote the model response as M(p) given the LLM M and the input p. The teacher model initially derives principles P from a subset N s of the feasible error set Ê. Then, we present the remaining set N r to the teacher model sequentially, prompting it to determine whether the established principle list P can rectify the error. If the existing principle list P can not address the presented error N r (i), the teacher model will formulate a new principle p for it. The iterative process stops when all the errors in N r (i) are checked. Then, human reviewers will step in to assess the validity of the principle list P created by the teacher model. The reviewers will simply delete any that are found to be erroneous or confusing, ensuring the reliability and clarity of the final principle list. We provide instructions for human reviewers in Appendix F. Why error summarization? Making mistakes is a natural part of the learning process and can be a powerful tool for growth and understanding (Cyr & Anderson, 2018) . Even though students can solve simple tasks using the original problem-solving instruction alone, they may have difficulty applying the same knowledge and skills in complex situations (Klein et al., 2007) . Therefore, it is also important to let students learn from errors, which can help them identify error patterns and avoid similar mistakes in future practice (Metcalfe, 2017) . Learning from mistakes has been explored in (Wang & Li, 2023; Yang et al., 2023b) , where knowledge from past failures is stored in a memory list and retrieved for each new query. However, these approaches primarily focus on recording errors in detail without summarizing them into high-level principles. In contrast, our approach emphasizes high-level principles, which are more generalizable and can be effectively applied in offline scenarios. Additionally, TPD can also be augmented by retrieval methods, as we can retrieve the most similar questions along with the teacher model's feedback from practice questions. Further analysis is provided in Appendix D.In the second stage, the student model's role is to utilize the principle list P to solve various instances of the task. There are multiple ways to utilize the principle list, which are explored in Sec 4.4. Compared to directly injecting the principle list into the prompt and the critiquerevise method, we found that using the principle list P to curate new examples from the validation set is more effective. Specifically, we define the violation score, which is the number of violations against the principles in the list P. The teacher model evaluates each error in Ê, using violation scores to rank them. The examples with the top k highest violation scores are selected as the most informative examples. Then, the teacher model generates correct answers for these examples to replace the incorrect answers from the student model. Besides, the teacher model also revises the problem-solving instruction based on P. The overall instruction combines the revised problem-solving instruction and the selected informative examples. For better illustration, we present the overall instruction template below.Revised Problem-solving Instruction: {method} + {original examples} New Selected Examples: {question} + {answer}The principle generation stage only needs to be performed once for each type of task. The teacher model could then build a new prompt based on the principle list for the student model. The prompt can then be reused for all instances of that task. This makes TPD significantly more efficient than using a retriever or a teacher model's intervention in each query. 
experiments setup: 
 Datasets. We evaluate our approach on eight datasets from diverse domains, including four tasks from Big-bench (Srivastava et al., 2022) : Tracking Shuffled Objects, Date Understanding, Navigate, and Matrixshapes. The other four datasets are GSM8K (Cobbe et al., 2021) , SVAMP (Patel et al., 2021) , CoinFlip and Last Letter Concatenation (Kojima et al., 2022; Wei et al., 2022b) . We use the 5-object versions of the Tracking Shuffled Objects task in the experiments, which we will refer to as Tracking Shuffled Objects (5). The CoinFlip, Last Letter Concatenation and Tracking Shuffled Objects (5) are regarded as symbolic reasoning tasks, and the remaining datasets are arithmetic reasoning tasks. The dataset split and the detailed information about each dataset can be found in Appendix G.In the experiment, we use gpt-3.5-turbo-16k, LLama3-70B-Instruct (AI@Meta, 2024) and Mixtral-8x7B-Instruct (Jiang et al., 2024) as student models, with gpt-4 (OpenAI, 2023) serving as the teacher model. In the following sections, we will denote them as GPT3.5, LLama3, Mixtral, and GPT4, respectively. More detailed settings are given in H.2.Baselines. We include Zero-Shot CoT (Kojima et al., 2022) , few-shot CoT (Wei et al., 2022b) , and Auto-CoT (Zhang et al., 2022b) as baseline prompting methods. Specifically, we include 6-shot CoT as a baseline method, where all of the 6 examples are selected based on the principle list from practice questions (there are 6 examples in the final prompt for the student model). In contrast, 3-shot-CoT uses randomly selected examples from the training set. More details of the baselines are given in H.3.Symbolic reasoning entails the use of symbols and their relationships to execute logical operations (MacColl, 1897) . It gauges logical reasoning and rule-based decision-making abilities, assessing a language model's proficiency in simulating human-like reasoning. Table 1 presents the comparison results. As anticipated, 3-shot CoT and Auto-CoT surpass zero-shot CoT by providing more examples and guidance to the language model, enabling it to generate higher-quality reasoning processes. Auto CoT achieves comparable performance across all three datasets compared to 3-shot CoT, suggesting that it can match the performance of the CoT paradigm that requires manual designs. Our method TPD outperforms 3-shot CoT on CoinFlip and Last Letter Concatenation, and achieves comparable performance on Tracking Shuffled Objects (5), demonstrating the potential effectiveness of problem-solving instructions over merely presenting a sequence of language reasoning steps.The error summarization stage appears to be less effective in symbolic reasoning tasks, primarily because errors often stem from a lack of factual knowledge rather than the misapplication of principles. Principles serve as high-level guidelines, applicable to a broad range The teacher model is GPT4, and the student models are GPT3.5, Mixtral and LLama3. In symbolic reasoning tasks, we utilize CoT as our base prompting method. In arithmetic reasoning tasks, we utilize PoT as our base prompting method. We test the performance with and without error summarization (ES).of scenarios. They do not change with specific situations but offer a consistent approach to problem-solving. In contrast, factual knowledge, which entails detailed, verifiable information about the world, is crucial in addressing specific cases. It should be incorporated into the model's parameters during training or sourced from external databases when needed.For instance, consider the Last Letter Concatenation task: the student model adheres to the problem-solving instruction to identify and concatenate the last letters of words to form a new string. The principle guides the process, but errors may occur due to inadequate factual knowledge about the last letter of a particular word within the string.The results of arithmetic tasks are shown in Table 1 . Zero-shot PoT results in the worst performance in all datasets. The main issue for 0-shot PoT is the LLMs' inability to utilize their pre-trained knowledge to craft solutions for problems without explicit guidance, thereby failing to tap into their potential for coding and logical reasoning with a basic prompt alone. The Auto-PoT approach, which stratifies validation by setting questions into three clusters and selecting the example closest to the center from each cluster to form a 3-shot PoT, aims to introduce diversity into the examples. However, it only marginally improves performance across three datasets and performs poorly on GSM8K, which suggests that a mere variety in questions does not inherently lead to distinct reasoning pathways. With the principles identified by the teacher model, we explored methods to transfer this knowledge to the student model effectively. One straightforward approach involves directly appending the list of principles to the prompt. Alternatively, we can employ an iterative learning process, where the student model initially attempts to answer a question, then provides feedback based on its own understanding of the principles, and then revises its initial response. This critique-and-revise strategy has been successfully used in various prompting methods (e.g., (Chen et al., 2023; Bai et al., 2022; Madaan et al., 2023) ).Our proposed method enhances this approach by selecting highly informative examples from the validation set, guided by the principle list. This strategy capitalizes on the emerging capabilities of LLM, enabling the student model to learn effectively from exemplary instances. The experimental results presented in Table 2 reveal that simply adding a principles list to the prompt yields only a marginal improvement over the base prompt. This suggests that the student model faces challenges in effectively using high-level and implicit principles expressed in natural language. While retrieval methods gather specific factual knowledge and integrate it with the initial prompt, the principles here are more abstract and harder for LLMs to leverage. For instance, a principle might serve as a directional guide for a specific step in the reasoning process. Additionally, LLMs tend to lose information in longer contexts (Liu et al., 2023) , which could further contribute to the limited impact of the principles list. Surprisingly, the critique-and-revise method resulted in a decrease in performance across all datasets. Our observations indicate that when prompted to provide feedback or critique based on the list of principles, the LLM tends to perceive the original answer as incorrect and significantly overhauls it. This behavior might stem from the RLHF stage, where terms like feedback or critique could trigger the model to question its previous outputs.Using the principle lists to select examples from the validation set for in-context learning achieves better performance across all datasets than the other two methods. The chosen examples based on the principle list contain the most error-prone questions for the student model, thus helping the student model learn from errors effectively. This also mirrors real-world classroom dynamics, where providing students with practical examples often proves more beneficial than solely relying on textbook knowledge (Shafto et al., 2014) .Despite the guidance provided by the teacher model, the student model may still make errors. In this section, we analyze the sources of these errors within the TPD framework. Firstly, it is important to note that the teacher model has its own limitations, especially in complex reasoning tasks (Huang et al., 2023) . As a result, it cannot detect all the errors made by the student model. To address this, data filtering is necessary during the error summarization stage. During the principle exploitation stage, errors can be categorized into three types:• e 1 : The principle provided is unhelpful.• e 2 : The student model fails to answer correctly even with a helpful principle.• e 3 : There are no relevant principles in the principle list for the given question.We analyzed the errors made by GPT-3.5 and Mixtral on the GSM8K and Date Understanding tasks, with the results shown in Table 3 . In the GSM8K task, no e 3 errors were observed for either student model, as the teacher model provided highly general principles (e.g., "Ensure arithmetic operations are logically sound and mathematically correct"). While these general principles are broadly applicable, they fall short when addressing specific questions. This indicates that an effective principle should target a particular type of error rather than rely on vague or general terms. In contrast, for the Mixtral model, e 2 errors were predominant. Even though the principle was helpful, the student model still failed to answer correctly, highlighting the inherent limitations of the student model's capabilities. Regarding e 3 errors, these arise when practice questions do not cover all possible question types, leading to gaps in the principle list. This is a common limitation of learning from feedback frameworks. Question: Multiply a matrix of shape (3,2,3,3) with a matrix of shape (3,2,3,4). Transpose the result.Principles: Matrix Multiplication: In Python, the asterisk ('*') operator is utilized for the Hadamard product. Use np.matmul for matrix multiplication.```python def solution(): A = np.random.rand(3,2,3,3) B = np.random.rand(3,2,3,4) mul_result = A * B transpose_result = np.transpose(mul_result) return transpose_result.shape ```Q uestion: James decides to run 3 sprints 3 times a day. He runs 60 meters each sprint. How many total meters does he run in January?Principles: Logic flow: Ensure arithmetic operations are logically sound and mathematically correct. Generalization capability of principles. We further examine if the principles derived from the errors of one student model can be effectively applied to another. As shown in Table . 4, both student models achieve good performance when leveraging principles deduced from the errors of the counterpart model. However, a slight performance drop is observed when a model utilizes principles from the other model compared to its own. This gap demonstrates that although there exist common mistakes between the two student models, there are also some differences in their error patterns. In Fig. 4 , we present a comparative case study to show the success mode (left subfigure) and the failure mode (right subfigure) of TPD. In both scenarios, the principles guide the student model towards reasoning more effectively. This guidance becomes particularly evident in the success mode, where it significantly enhances the student's problem-solving capabilities. However, the principles have their limitations, especially when confronted with the model's gaps in factual knowledge. For instance, as illustrated in the right subfigure, the student model fails to accurately assign 31 days to January. 
conclusion: 
 In this paper, we present a framework named Teaching via Principle Discovery (TPD). This approach empowers teacher models to construct problem-solving instructions and summarize key principles by analyzing example questions and student errors. The identified principles are subsequently employed to refine the problem-solving instructions and select the most informative examples from the validation set to create a tailored instruction prompt. We validate the effectiveness of TPD on symbolic and arithmetic reasoning tasks, observing a marked improvement in the performance of the student model. Our method introduces an innovative approach, utilizing advanced LLMs to guide weaker agents in tackling reasoning problems. In the future, we plan to study how to apply TPD to solve complex reasoning tasks, such as serving as a web agent, where the principle list will be much longer. 
 
 Appendix: One limitation of the TPD is its inability to rectify common sense errors (lack of factual knowledge), as evidenced by its marginal improvements in the GSM8K and SVAMP datasets in Table 1 , despite the teacher model's high accuracy (97%, Zhou et al., 2023) . This limitation arises because the errors made by the student model predominantly relate to factual knowledge, a category not effectively addressed by the generated principles and instructions. High-level principles can resolve only a limited number of such errors. A potential solution for this challenge could involve fine-tuning the LLM to internalize a broader range of common sense knowledge. Another constraint of TPD is the method for applying principles.While selecting examples based on validation scores has proven more effective than other strategies, this approach is not particularly efficient. The issue stems from the LLMs' context length limitations. This inefficiency poses an unresolved challenge: effectively integrating a long list of principles into LLMs. Moreover, we do not check the overlap of the principles in the principle list, hence there might be near-duplicate principles in the principle list. 2022) find that using LLMs to generate codes for reasoning tasks and then utilizing the codes by program interpreters to solve the questions can achieve better performance. Another prompting method is the problem decomposition (Zhou et al., 2022; Drozdov et al., 2022; Dua et al., 2022) , which asks LLMs to decompose the tasks into several subtasks and solve them individually. Yao et al. (2023) propose Tree of Thoughts, which enables language models to self-evaluate intermediate "thoughts" and decide whether to explore different ideas or reevaluate when needed to provide the optimal solution. Self-refinement methods (Madaan et al., 2023) ask LLMs to refine their original answers iteratively but will rely on the handwritten few shot examples.The problem-solving instruction consists of a problem-solving method augmented with a few examples. These examples are derived from the same questions initially sampled. Here is an example.To solve the problem where you are asked to take the last letters of each word in a given string and concatenate them, you can follow these steps: 1. Identify and list each word in the string. 2. Locate the last letter of each word.3. Concatenate these letters to form a new string. Provide the new string as the answer.Examples: Question: Take the last letters of each word in "Jeremiah Kelley Josue Veronica" and concatenate them. Identify the words: Jeremiah, Kelley, Josue, Veronica. Last letters: h, y, e, a. Concatenate: "hyea". Answer: hyea.TPD leverages the teacher model to extract and summarize principles from the mistakes made by the student model on practice questions. These principles operate at the task level. In contrast, popular learning-from-feedback frameworks typically provide feedback at the query level, where a retriever fetches relevant information for each query from a memory module. This contrast led us to explore whether TPD could be further enhanced by integrating a retrieval module. Specifically, we instructed the teacher model to generate a The results shown in Table 5 suggest that TPD can indeed be improved with the addition of a retrieval module. This improvement indicates that the task-level principles and the directly retrieved examples are complementary to some extent. However, in the Matrix task, the Mixtral model's performance did not benefit from the retrieval module. We believe that finding an optimal method to combine task-level principles with query-level retrieved information could be a promising future direction for the teacher-student framework.Can we replace the original examples in the problem-solving instruction with the newly selected samples? Fig. 3c demonstrates the high informativeness of examples selected based on violation scores. This raises an intriguing question: can we directly replace the original examples in the modified problem-solving instruction with these newly chosen ones? To explore this, Fig. 5 presents a comparison between two approaches: appending the new selections to the original examples versus completely replacing them. Interestingly, the appending method consistently outperforms the replacement approach across all datasets. This superior performance can be attributed to the fact that the new selections specifically address the errors encountered when the student model learns from the original examples. Moreover, completely removing the original examples may inadvertently shift the student model's errors, rendering the newly selected examples less informative.Methods description. Our study also investigates whether the teacher model should offer a high-level reasoning method in problem-solving instruction. As illustrated in Fig. 6 , we observe a notable decline in problem-solving performance without an explicit method description across three datasets. This suggests that while the student model can implicitly learn and mimic reasoning from examples, it remains crucial for the teacher model to provide a problem-solving method in natural language explicitly.Regarding human intervention for quality checks on principles, since the principle lists in our experiments are not very long, human reviewers can finish checking the principle list within approximately 20 minutes per task. In TPD, human reviewers are responsible for verifying each principle in the principle list for all tasks to ensure quality. They sequentially review and remove any principles that are vague, erroneous, or confusing.1.The principle should be written in simple, clear language that is easy to understand. 2.The principle should be specific enough to address a particular type of error or concept.3.The principle should be factually correct and free from errors. 4.The principle should provide actionable guidance that helps the student avoid the error in the future. 5.While specific, the principle should also be general enough to apply to similar errors or situations.The Coin Flip task (Wei et al., 2022b) requires the LLM to determine if a coin remains heads up following a series of flips or non-flips. The Last Letter Concatenation task (Wei et al., 2022b) requires the LLM to concatenate the last letters of each word in a given name list.Both tasks have out-of-domain test sets, which include more complex examples than those provided in the training exemplars. Specifically, for the Last Letter Concatenation task, the training set contains the lists comprising only two names but the test set consists of the lists with three and four names. A similar setting is adopted for the Coin Flip task.Tracking shuffled objects (Srivastava et al., 2022) requires the LLM to determine the ultimate state of a system based on its initial condition and a series of subsequent changes. In each instance of this task, a collection of objects is initially associated with individual owners. These objects are then exchanged in a sequence of swappings. The model's objective is to determine the final owner of each object.Date Understanding (Srivastava et al., 2022) asks the LLM to infer the date from the given context. The context is one or two sentences with date information.Navigate (Srivastava et al., 2022) requires the LLM to infer the agent's position after several movements. Specifically, given a series of navigation steps, the LLM needs to determine whether or not an agent would end up back at the starting point.GSM8K (Cobbe et al., 2021) is a collection of grade-level mathematics problems by human authors. These problems require 2 to 8 steps for resolution, primarily requiring a series of fundamental computations employing basic arithmetic operations (addition, subtraction, multiplication, and division) to arrive at the solution.Matrixshapes (Srivastava et al., 2022) requires the LLM to predict the shape of the result of a chain of matrix manipulations, given the inputs' shapes.SVAMP (Patel et al., 2021) is a challenge set for elementary-level Math Word Problems. It consists of questions that test a model across question sensitivity, reasoning ability, and invariance to structural alterations.For each dataset, we sample three questions as the training set. Subsequently, we partition the remaining data into practice and test questions, maintaining a 1:4 ratio. Notice the CoinFlip and GSM8K are already split into the training set and the test set. We use the original test set and randomly sample 3 questions from the original training set to build our training set. Regarding Date Understanding, Navigate, Matrixshapes, and Tracking shuffled objects (5), we utilize data from (Cai et al., 2023) . We divide each dataset into training, validation, and test sets, containing 3, 47, and 200 instances, respectively. For Auto-CoT, we select examples from the whole dataset in symbolic reasoning tasks and from the validation set in arithmetic reasoning tasks. Table 6 shows the example questions for each dataset.In the experiments, we utilize two GPT models: gpt-3.5-turbo-16k and gpt-4. The gpt-3.5-16k refers to the "gpt-3.5-turbo-16k" model and in the OpenAI API model with checkpoint version 2023-06-13-preview webpage1 GPT-4 refers to the "gpt-4" model with checkpoint version 2023-07-01-preview. All mentioned checkpoints are hosted on Microsoft Azure2 . For Mixtral-8x7B-Instruct, we utilize "Mixtral-8x7B-Instruct-v0.1" on huggingface. The model temperature is 0 in all cases.We use the default temperature of 0.0 for all models. For symbolic reasoning tasks, we utilize CoT as our base prompting method. For arithmetic tasks, we utilize the Program of Thought (PoT) method as our base prompting method. This method involves having the LLMs process natural language questions and create corresponding programs that represent intermediate reasoning steps, with the actual computation of solutions being delegated to a runtime environment, such as a Python interpreter. We avoid using CoT due to the tendency of language models to produce erroneous mathematical operation results when tackling arithmetic tasks (Ji et al., 2023) . Our framework is designed to instruct the student model in problem-solving strategies rather than in performing detailed decimal operations with precision.Since our framework generates a prompt for the student model for the downstream task, we compare several prompting methods in our experiments. Specifically, we adopt Zero-Shot CoT (Kojima et al., 2022) , few-shot CoT (Wei et al., 2022b) , and Auto-CoT (Zhang et al., 2022b) as baseline prompting methods. For the few-shot CoT, we use the questions in the training set as exemplars for few-shot prompting. For Auto-CoT, we use Sentence-BERT (Reimers & Gurevych, 2019) to compute a vector representation for each question and then cluster practice questions by k-means. Following the original paper, we choose the question closest to the center from each cluster and ask the teacher model to generate reasoning steps with zero-shot CoT to build final examples for the student model.Table 7 and Table 8 show the principle list the teacher model finds in the error summarization stage for GPT3.5. These principles provide high-level guidance applicable across various scenarios. However, the principle lists for GSM8K and SVAMP are less clear compared to others, since the errors in these tasks primarily relate to factual knowledge. .https://platform.openai.com/docs/models*.openai.azure.com 


Decision: The paper introduces a method for prompting student models based on error analysis and guidance from a larger model. Evalauting this approach on 2 LLMs with a number of tasks demonstrates higher performance compared to CoT variants. Ablations demonstrate that reasonable performance can be obtained with few shot settings which will reduce cost of inference. All high quality reviews assign a score of marginal accept and the reviewer score improved during the rebuttal phase: this introduced several new baselines and ablations which will result in a stronger paper. There are several common weaknesses identified by the reviewers, particularly around the failure cases and handling factual knowledge. I look forward to seeing an updated version of the paper or future work which would discuss these issues.

[At least one review was discounted during the decision process due to quality]

Reviews: review: paper_summary: paper_summary: This paper proposes a method to effectively transfer advanced reasoning capabilities of stronger LLMs to weaker LLMs by exploiting general principles that indicate gaps in a student’s understanding on practice problems. They outperform CoT prompting on symbolic and arithmetic reasoning.
strength_weakness: strength_weakness: **Availability and selection of training examples**: 
The paper assumes that the training examples are available in the principle generation stage (constituting a validation set for initial practice) . A realistic setting for real-world tasks and benchmarks like Big Bench Hard is where typically only a handful of examples are available. How many examples are sufficient for helpful problem solving instructions and error summarization? Is there a method to optimally select from the training examples e.g. with a higher coverage of question types (or principle types)?

What about other tasks like open ended generation. In many such real world examples, it is not easy to compare against the ground truth and measure most error prone questions. In those practical setups, how will you choose examples based on the principle list containing the most error-prone questions for the student model to help the student model learn from errors effectively (in section 4.4 last para, it is stated that most error prone questions are required for effective learning in the principle generation stage)?


**Issues in Principle exploitation**:
Generated Principles in Figure 4 are too high level ("_Logic flow: ensure arithmetic operations are logically sound and mathematically correct_"). Is this example really providing any actionable insights to a model during inference? How do you know which kind of error is triggered so that you can select a principle? Also fundamentally, how are principles different from feedback?  Other similar papers (e.g., cite: CLIN [5]) that have a two stage pipeline that includes “error summarization” that they call meta memory. But their summarized reflections are distilled into knowledge necessary to solve the task. 

Related to this: How do you pull past principles (did you try different retrieval methods)? Past similar methods (cite: Memprompt) that exploit past mistakes for better inference have documented the importance of retrieval quality. How will you scale the method as the principle list grows?

**Missing error analysis**:
It would be helpful to know the kind of errors the system makes. e.g., what percentage of the errors are related to principle generation and exploitation. There are many details related to principle retrieval issues, teacher making mistakes in error identification or just the fact that a principle is unhelpful or the model fails despite providing a helpful principle. The ablation studies were helpful (thank you), but error analysis would give a clear picture.

**Missing Related work and Baselines**: 
There has been past work on exploiting model answers to create a memory/ list of knowledge (cite: CLIN [5]), feedback (cite: MemPrompt [4]), principles (cite: LEAP [1]). Also, Reflexion [2] learns per-example feedback and LEAP [1] learns per-task principles. Can you compare against these methods, either empirically or theoretically? It seems like some of this is very closely related work (e.g., LEAP [1]) and it would be good to compare with these (though empirical comparison with LEAP [1] is not at all a ground for rejection as it is a very new paper but it would be useful to compare it broadly).  

Section 4.4 page 8 presents an interesting finding that the LLM diverges significantly from its previous answer if a critique or feedback is used. Is this a previously established finding / can you support this claim with any references? This looks contrary to other papers (e.g., cite: [3] page 7) so it is important to substantiate.

Also, please distinguish your work in the related work, in addition to mentioning other papers. Eg., it seems like you also search for principles applied over mistakes from most representative train examples? (so there must be a retrieval module here like other papers?)

**Results are not very convincing**:
The claim made in the paper are quite broad. However, the empirical results are not too convincing. The improvement over AutoCoT or 3-shot CoT is not significant and it is raises the question whether we can justify a more complex method like yours. It would be helpful to also have a simple baseline that takes the k-examples as demonstrations whose principles are applied in your method. 


**Other Missing details**: 
On page 4, the teacher can make mistakes in accurately identifying the incorrect answers in the error set. Can you clarify if the teacher sometimes cannot give the correct modified instructions or the correct principle? Can the teacher identify when an answer is wrong (the paper mentions that these examples are taken from a validation set (question, answer) -- so why is it hard to detect if there is an error e.g., in math, the answers are clear (but maybe in open ended generation it is not easy to identify mistakes). I am assuming that at test time, there is no teacher involved as mentioned in the paper.


**Minor typos**:
typo in fig 1: promblem (I have not checked more thoroughly for spelling mistakes elsewhere)


References:
1. In-Context Principle Learning from Mistakes (LEAP): https://arxiv.org/pdf/2402.05403
2. Reflexion: Language agents with verbal reinforcement learning: https://arxiv.org/abs/2303.11366
3. Learning to Repair: Repairing model output errors after deployment using a dynamic memory of feedback: https://arxiv.org/pdf/2112.09737 
4. Memory-assisted prompt editing to improve GPT-3 after deployment: https://arxiv.org/abs/2201.06009
5. CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization: https://arxiv.org/abs/2310.10134
questions: questions: Please go through the reasons to reject, and you can respond. If I receive clarifying satisfactory answers then I am willing to increase my rating.
paper_summary: paper_summary: The paper proposes a method to design task-specific zero-shot or few-shot prompts through interaction between a "teacher" and "student" model, using an existing dataset of task-specific question-answer pairs. This involves a multi-step process summarized below:
1. The teacher model selects representative samples from the dataset and generates an initial instruction on how to solve the task, along with few-shot examples of how to solve the selected samples. (this is the baseline prompt denoted TPD w/o ES in Table 1, to my understanding)
2. The student model is prompted to solve questions in a validation set using the initial prompt constructed above, and an error set $E$ is built.
3. The teacher model is prompted to solve the same questions in $E$. Each question that the teacher can solve correctly is included in the feasible error set $\hat{E}$. Incorrect questions are discarded as they are deemed too difficult for even the teacher.
4. The teacher model build a set of principles $P$ that is required to address the errors in $\hat{E}$ through an iterative process.
5. The resulting principle list is filtered by human reviewers.
6. A new set of representative questions are selected from the validation set, which violate the most number of principles in $P$.
7. The initial instruction from step 1 is revised based on $P$, and new examples of how to solve the questions selected in step 6 are appended to the original examples from step 1 (this is the prompt denoted TPD w/ ES in Table 1, to my understanding).

The final prompt outperforms zero-shot, few-shot, and auto-cot on various symbolic and arithmetic reasoning tasks.
strength_weakness: strength_weakness: 1. The requirement of large task-specific datasets, and thus the associated costs, limits the applicability of this method. It may be more cost-effective to fine-tune a smaller model to achieve the same performance. This has not been considered in the paper. Note, while Auto-CoT has the same limitations, they mitigate this limitation by demonstrating a streaming-based variant of their approach, where examples are selected from test batches.
1. The need for human intervention in step 5 of my summary somewhat limits the applicability of this method.
1. The difference between the experimental settings of Figure 3 (a), (b) vs (c) is unclear. Both seem to ablate the number of examples.
1. It is hard to conclude that models *benefit* from principles deduced from errors of another model from Table 3. That would require comparison with baselines (e.g., zero-shot CoT, maybe few-shot CoT, principles derived from original samples, etc.).
1. (Minor) The method is fairly complicated.
1. (Minor) The writing is hard to follow. E.g.,
    1. The method explanation and Figure 2 are hard to follow. It may be beneficial to divide the method into a clear series of steps which align with the subsections of the method section.
    1. It is not 100% clear what TPD w/ and w/o ES in Table 1 refers to (I've written my interpretation in the Summary)
    1. It is hard to follow which parts of the methodology explained in Section 3 corresponds to which ablations in Table 2, and it is unclear what method was used in Figure 3 and Table 3.
    1. The use of the word "teaching" throughout the abstract and intro may mislead the reader to initially believe this is a student fine-tuning paper. It would help the reader to highlight that the method's goal is to construct a (few-shot) prompt.
questions: questions: 1. Is there a specific reason behind the selection of Mixtral 8x7B–an MoE model–as the sole open-source student model?
paper_summary: paper_summary: Large language models (LLMs) possess substantial reasoning abilities; however, they entail significant computational demands. Prior research has explored knowledge-distillation-based fine-tuning methods to confer these capabilities onto smaller models, though these methods are resource-intensive. This paper presents a novel approach, Teaching via Principle Discovery (TPD), which enhances the reasoning skills of smaller language models (student LLMs) by leveraging insights from larger models (teacher LLMs) without necessitating ongoing intervention during inference. TPD adopts human teaching methodologies through a two-phase procedure: principle generation and principle exploitation, guided by error analysis and the formulation of corrective principles by the teacher model.
strength_weakness: strength_weakness: 1. Inclusion of additional case studies would enhance the paper. While TPD demonstrates promising results for symbolic and arithmetic reasoning tasks, a more detailed exposition with extensive examples could clarify its advantages to readers.

2. The paper should introduce and discuss failure scenarios more. Currently, the limitations of TPD and its applicability to tasks beyond the tested domains are not sufficiently discussed.

3. Discussions on how well the principles learned can be generalized and transferred to different student models or a wider array of tasks are limited and could be expanded.
paper_summary: paper_summary: This paper proposes a simple method to improve the performance of the student model. Specifically, the teacher model generates problem-solving instructions and corrective principles based on the student LLM's errors. Then during inference, the student model can guide the inference process without the help from the teacher model.
strength_weakness: strength_weakness: 1. The method involves the labeling process of both large teacher model and human reviewers, which is complex and resource-consuming. 
2. The experimental result on some arithmetic reasoning tasks achieves a very subtle improvement. More seriously, it seems that some important baselines are missing, such as Automate-CoT. 
3. The experiments are only conducted on GPT-3.5, which is a relatively weak LLM. It would be better if the authors could provide the experimental results on other LLMs.
questions: questions: Please refer to the reasons to reject.

Comments: title: Thank you.
Response to final questions
Final questions
Discussion with reviewer R4tU
Discussion with Reviewer R4tU
Strong baseline
Error analysis
Thank you for the detailed response.
comment: Dear Reviewer ZvuT,

We greatly appreciate the time you took to review our paper. Since the Llama3 models are released on April 18th, which is later than the paper submission date, we will add them in the next version.

Due to the short duration of the author-reviewer discussion phase, we would appreciate your feedback on whether your main concerns have been adequately addressed. Should you have any further advice on the paper and/or our rebuttal, please let us know and we will be more than happy to engage in more discussions.

Thank you so much for devoting time to improving our paper!
Thank you. I increased my score accordingly.
We appreciate your thoughtful feedback. Due to the limited time of the discussion period and conference rules, we cannot update the scripts now. However, we guarantee that we will include the suggested studies, analyses, baselines, and method updates in the next version of the paper.

Regarding human intervention for quality checks on principles, since the principle lists in our experiments are not very long, reviewers can finish checking the principle list within approximately 20 minutes per task. In TPD, human reviewers are responsible for verifying each principle in the principle list for all tasks to ensure quality. They sequentially review and remove any principles that are vague, erroneous, or confusing.

The instructions provided to human reviewers are as follows. We will also include them in the appendix in the next version.

1.The principle should be written in simple, clear language that is easy to understand.

2.The principle should be specific enough to address a particular type of error or concept.

3.The principle should be factually correct and free from errors.

4.The principle should provide actionable guidance that helps the student avoid the error in the future.

5.While specific, the principle should also be general enough to apply to similar errors or situations.
Thank you for the response. 

Ans to Q7  -- future work where the student model could self-discover and summarize principles using the true labels from the practice questions -- at that point, it becomes very close to LEAP [1] method. You can include this in the future work may be.
Ans to Q6 -- strong baseline -- yes, I would be curious to find out in your next version. 
Ans to other questions -- thank you. Your responses have given me more clarity now. I will very likely increase my score today. Obviously it would be under the assumption that the authors would include the suggested studies, analysis, baselines, and also try out the method updates (this part, if it is possible to do so). That should make the paper stronger (supporting the claims, and making the method more robust). 

*One final question*: how much human intervention (for quality checks on principles) was needed across different tasks. I am looking for a ballpark quantitative answer. Also, what instruction did you give the human on what a good principle is.
**Q6 Strong baseline that picks 6 most related data points from the training set or the validation set**

Thank you for your insightful suggestion. We agree that the baseline of retrieving the six most related data points from the training or validation set is indeed interesting. We share your curiosity about the student model’s performance with this approach. Given the limited time remaining, we will include this baseline in the next version of our paper. 

**Q7 given as input training examples for a task and a broad description of the task, you can refine the prompt by generating modified problem solving instructions and automatically selecting a set of few shot demonstrations. Is that correct?**

Yes, that is correct. With several labels for the questions (used to build practice questions), the teacher model can refine the problem-solving instructions and automatically select a set of few-shot demonstrations. While this paper primarily explores the teacher-student framework, we also envision future work where the student model could self-discover and summarize principles using the true labels from the practice questions.
**Q5 Your error analysis needs to be more thorough**

We acknowledge that the teacher model is generally effective at identifying the student's failures. However, there are some complex reasoning tasks, where instances fail to detect errors, even with larger models. This observation underscores the necessity of the filtering module in TPD where we filter out undetected errors to enable the teacher model to summarize principles from the errors it can identify.

When we describe the teacher model as "effective," we refer to its ability to identify a relatively low number of failures compared to its successes. However, we recognize the need for a more thorough analysis. In the next version of our paper, we will conduct further studies to categorize the types of problems contributing to errors and quantify the percentage each type represents. Additionally, we will provide examples of each error type to enhance understanding and clarity.
We appreciate the reviewers' insightful feedback. 

**Q1 Do near-duplicates happen in practice that P has many near-duplicates? what exactly is a good principle?**

Thank you for raising this question! In practice, we do not find near-duplicate principles. In TPD, we check the existing principle list P in line 5 of Algorithm 1 to prevent this issue. Intuitively, if the existing principle list P cannot address the presented error, the teacher model will formulate a new principle p for it. We agree that a good principle should cover a type of error. We acknowledge that we do not assign weights to principles at this stage. If the principle list becomes longer and longer, we can attempt to ask the teacher models to merge some principles. For example, if two principles address the problem of the same type, from different perspectives, the teacher model could merge them. 

**Q2 the influence of the size of the validation set on TPD’s performance**

Thanks for the suggestion!  Although the conference guidelines prevent us from updating the paper at this stage, we will ensure that this important information is included in the next version of the paper.

**Q3 How do you plan to include this modification-- do you plan to formulate it as an optimization problem?**

Thank you for your insightful suggestion. Yes, we plan to formulate this as an optimization problem, as you described. After constructing the principle list P, we will have the teacher model revisit it and assign a weight to each principle based on its importance to the question and its similarity to other principles. We will then define the violation score as the summation of the weights of the violated principles for an incorrect question-pair in the practice questions. Based on this, we will select k examples that cover diverse, crucial, and correct principles.

**Q4 for open ended tasks:**

For open-ended generations, there are some common important aspects, such as helpfulness and harmlessness, which we can use as predefined principles. However, these manually defined principles may not fully cover all desired aspects in the generated outputs. We may still need to discover some model-specific principles through the error summarization stage. This will depend on the specific student models, and we need to conduct empirical studies to investigate this further.
Thanks for your response. It's better to add Llama3 models as students in the next version.
Thank you for the 6-examples CoT -- its avg. performance is 83 across tasks whereas TPD is 86 (with relevant confidence intervals it can be quite close). It reminds of another surprisingly strong baseline that picks 6 most related data points from the training set or the validation set (but not using principles based approach as in TPD). It would be helpful to add this baseline to the next version. 

I am wondering about one more perspective to the utility of your work: given as input training examples for a task and a broad description of the task, you can refine the prompt by generating modified problem solving instructions and automatically selecting a set of few shot demonstrations. Is that correct?
Dear authors,

Thank you for acknowledging the feedback on adding a retrieval module and augmenting with feedback; and for including an error analysis subsection. That would make the setup more scalable and robust.

- Your error analysis needs to be more thorough. E.g., you claim that "The teacher model is effective in identifying the student's failures, showing a relatively low number of failures" but the paper [6] that you cited and also [7] has evidence to suggest that for some reasoning tasks such as in math reasoning even larger models cannot spot a mistake (e.g, if the answer has a CoT style answer). The analysis that some principles are unhelpful is useful -- please include a more thorough version of this in the next version of your paper, clearing highlighting what kind of problem contributes to roughly what % of error with some examples of each error type.

[6] Huang, Jie, et al. "Large language models cannot self-correct reasoning yet." arXiv preprint arXiv:2310.01798 (2023).
[7] Madaan et. al, "Self-Refine: Iterative Refinement with Self-Feedback" NeurIPS 2023
Dear authors, 

I needed some clarifications based on the rebuttal content you wrote (which was quite helpful, thank you):

- line 7 of algorithm 1 where `P += p` , we do not check for near-duplicates. Does it happen in practice that P has many near-duplicates? If P indeed has near-duplicates, then the selected examples might be over-represented because of the near-duplicates. What mechanism prevents from happening? Is it true that line 7 of the algorithm should really also have a call to a human: `P += p  if is_good_principle(p) else empty`. Related: what exactly is a good principle-- one that is crucial e.g., it is the cause of most erroneous examples? I am assuming there is no _weight_ given to a principle currently

- Thank you for the table on the influence of the size of the validation set on TPD’s performance. I could not find this in the paper: please include this important information in the next version.

- _Our TPD method selects examples based on the number of violations against the principle list. However, we can modify it to ensure higher coverage of the principle list, further optimizing example selection._ Yes that would help. How do you plan to include this modification-- do you plan to formulate it as an optimization problem: given N principles,  and each principle having a weight w, select k principles such that they are diverse, crucial, correct.

- for open ended tasks: _we can choose examples based on questions that are prone to violating the principles on different desirable aspects of the task._ that is good, and does it mean that these aspects essentially become principles and you will supply them manually and not discover them -- would that simplify the current setup?
**Q5. Missing Related work and Baselines**

We appreciate your highlighting the importance of comparing our work with previous studies. We acknowledge that there has been significant past work on leveraging model answers to create a memory or list of knowledge, as well as feedback mechanisms and principles. Specifically, CLIN [5] focuses on retrieving successful trials from history, MemPrompt [4] retrieves the most similar questions with user feedback in the history, and Reflexion [2] uses summaries of previous failures. LEAP [1], the most closely related work, summarizes high-level and low-level principles from the model’s previous mistakes.

Comparison with LEAP [1].
LEAP [1] and our approach share similarities in that both aim to discover principles from previous mistakes. However, there are two main differences between our methods:

1. Framework Application: We apply principle discovery within a teacher-student framework to enhance knowledge transfer through principles. In contrast, LEAP demonstrates that LLMs can also discover principles from their previous mistakes.

2. Application of Principles: We apply the principles through new examples from the validation set, whereas LEAP directly concatenates principles to the original prompt.

Comparison with Reflexion [2], CLIN [5], and MemPrompt [4].
These methods utilize per-example feedback. Specifically:

1. CLIN retrieves successful trials from history.
2. MemPrompt retrieves the most similar questions with user feedback in the history.
3. Reflexion utilizes summaries of previous failures.

In TPD, the student model uses task-level principles from the teacher model in an offline setting. Indeed, TPD can also be augmented by retrieval methods, as the teacher model can provide example-level feedback to the student model. We plan to include related experiments in the revised manuscript.

**Q6. Section 4.4 page 8 presents an interesting finding that the LLM diverges significantly from its previous answer if a critique or feedback is used. Is this a previously established finding / can you support this claim with any references? This looks contrary to other papers (e.g., cite: [3] page 7) so it is important to substantiate.**

[6] has discussed some findings about the self-correction capabilities of LLMs. To make this mechanism work, we need external feedback and carefully designed prompts. In our experiments, we provide the principle and ask the student model to critique the previous answer and then revise it based on the critique. We are not sure about the reason, but LLMs are sensitive to prompt formats so we may observe different behaviors when different types of prompts are being used.


**Q7. Results are not very convincing: The claim made in the paper are quite broad. However, the empirical results are not too convincing. The improvement over AutoCoT or 3-shot CoT is not significant and it is raises the question whether we can justify a more complex method like yours. It would be helpful to also have a simple baseline that takes the k-examples as demonstrations whose principles are applied in your method.**

In Figure 6 (in the appendix), we show an ablation study on whether the teacher model needs to provide problem-solving methods in the problem-solving instruction with three examples. We further conducted an experiment on 6-examples selected based on the principle list (there are 6 examples in the final prompt for the student model).

| GPT-3.5        | Coin  | Letter | Shuffled | Date | Navi. | GSM8K | Matrix | SVAMP |
|----------------|-------|--------|----------|------|-------|-------|--------|-------|
| 6-examples CoT | 92.8  | 83.5   | 75.0     | 74.0 | 93.0  | 74.8  | 89.5   | 82.8  |
| TPD            | 100.0 | 89.9   | 75.0     | 76.5 | 97.5  | 75.4  | 93.5   | 82.9  |

**Q8. Other Missing details.**

In the principle exploitation stage, the teacher model modifies the original problem-solving instruction based on the principle list. During the filtering process, we exclude question-incorrect answer pairs that the teacher model cannot accurately distinguish. The true answer is not provided directly to the teacher model. Instead, we first evaluate whether the teacher model can identify errors in the student's answer and then determine if it can generate the correct answer.

While this filtering process helps ensure quality, the teacher model may occasionally provide vague principles. These instances are flagged for further review and dropped by human reviewers if necessary. At test time, as mentioned in the paper, the teacher model is not involved.

**Q9. Minor typos: typo in fig 1: promblem (I have not checked more thoroughly for spelling mistakes elsewhere)**

Thanks for pointing out. We will double-check and modify the writing in the next version.



References:

[6] Huang, Jie, et al. "Large language models cannot self-correct reasoning yet." arXiv preprint arXiv:2310.01798 (2023).
Thank you for acknowledging my comments.

- 1. I'd like to point out that the requirement of an existing large dataset is a large cost factor, which may be overlooked when applying to academic benchmarks--since they implicitly provide a large dataset with them. In novel tasks, practitioners will have to build these datasets prior to applying the proposed method. Therefore the dataset should be considered as a cost associated to this method.

- 3. I see that the explanation for Figure 3 (a) and (b) in the paper ( `numbers of examples in the problem-solving instruction`) is clear but (c) could be clarified better. I also think that establishing the names of each method/step more clearly (mentioned in Reasons to Reject 6.1.) can prevent this misunderstanding.

I will maintain my initial rating.
**Q2. Issues in Principle exploitation**

We would like to clarify the following points:

Principle Utilization: Our approach does not directly put the principles into the prompt. Instead, we select informative examples based on these principles, allowing the student model to learn logical coherence from these examples. TPD does not select different examples for different questions during the student model's inference. It utilizes a fixed principle list discovered during student practice to select examples from the validation set to build a fixed prompt for all instances from the same task.

Experiment Method: In our experiments, we put the selected examples into the teacher-generated instructions for all inference questions. Our goal is to build a teacher-student framework for offline settings, where the student can generally utilize the principles taught by the teacher without other help.

Principles vs. Feedback: Principles provide high-level abstract guidance, teaching the student model methods to solve one type of problem. In contrast, feedback methods offer detailed guidance for each specific question, which needs to be retrieved during the student model’s inference. TPD tests how high-level principles can help the student model and it can also be augmented with previous feedback. We will add the retrieval module to TPD and show the results in the next version. 

**Q3. Related to this: How do you pull past principles? How will you scale the method as the principle list grows?**

In our experiments, we treat principles as high-level guidance from the teacher model. The teacher model constructs a fixed prompt in TPD for all instances in a specific type of task, and we evaluate how effectively the student model learns from this prompt.

Retrieval methods, on the other hand, retrieve different feedback from past experiences, offering more fine-grained and question-specific guidance. To draw an analogy, principles are akin to what teachers impart during class, while feedback represents the past experiences encountered by the student model.

In Memprompt, the memory of past experiences expands during the inference stages. However, in TPD, we work with a fixed set of principles derived from practice questions. Although TPD currently uses a static list of principles, it can be augmented with feedback and retrieval methods. We plan to explore this enhancement in future iterations of our work.

**Q4. Missing error analysis: It would be helpful to know the kind of errors the system makes.**

We have conducted an error analysis, and the results are summarized in the following tables.
Number of Failure cases:

|         | Coin | Letter | Shuffled | Date | Navi. | GSM8K | Matrix | SVAMP |
|---------|------|--------|----------|------|-------|-------|--------|-------|
| GPT-3.5 | 0    | 0      | 1        | 2    | 1     | 3     | 1      | 2     |
| Mixtral | 0    | 0      | 0        | 0    | 1     | 4     | 2      | 2     |

Number of Principles That Do Not Improve Performance:
|         | Coin | Letter | Shuffled | Date | Navi. | GSM8K | Matrix | SVAMP |
|---------|------|--------|----------|------|-------|-------|--------|-------|
| GPT-3.5 | 0    | 0      | 0        | 1    | 0     | 2     | 0      | 2     |
| Mixtral | 0    | 0      | 0        | 0    | 0     | 1     | 0      | 2     |

The first table illustrates the number of failure cases identified. The teacher model is effective in identifying the student's failures, showing a relatively low number of failures. This highlights the necessity of using the filtering module in TPD to filter out these cases.

The second table displays the number of principles that do not improve the student model's performance. Some principles do not aid in arithmetic reasoning tasks, possibly because they are too general.

Due to the length constraints, we will include a detailed error analysis section, including a comprehensive breakdown of the types of errors and their causes, to offer a clearer picture in the revised paper.
**3. Figure 3 (a), (b) vs (c)**

For Figure 3(a) and (b), we study whether we need to have examples in the problem-solving instruction in the principle generation stage. If we do not have examples in the problem-solving instruction, it will only contain a problem-solving method. The results show that examples are very important in problem-solving instruction.
For Figure 3(c), we study the number of examples selected in the principle exploitation stage, the examples here are selected from the practice questions. We want to study the minimum number of examples required for effective learning from the principle list.

**4. It is hard to conclude that models benefit from principles deduced from errors of another model from Table 3.**

We will compare different baselines and extend Table 3 in the next version. Meanwhile, we will also visualize a comparison between principles for two LLMs for a better understanding.

**5. (Minor) The method is fairly complicated.**

Our method is designed to mimic natural human teaching and learning processes. The goal is to explore how an advanced LLM can effectively teach a weaker LLM through interaction. While the additional computational cost primarily arises from the teacher model, we anticipate that as the costs of advanced models continue to decrease, the cost of our TPD approach will become more manageable.


**6. The writing is hard to follow.**

Thanks for pointing out. We will double-check and revise the writing.

**7. Is there a specific reason behind the selection of Mixtral 8x7B–an MoE model–as the sole open-source student model?**

No, we just select it and use it. We will also add llama3-70B as a new student model in the revised version.