Can LLMs Learn by Teaching for Better Reasoning? A Preliminary Study
Authors: Xuefei Ning
Zifu Wang
Shiyao Li
Zinan Lin
Peiran Yao
Tianyu Fu
Matthew B. Blaschko
Guohao Dai
Huazhong Yang
Yu Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: LLMs
Learning by Teaching
Reasoning
Mathematical Reasoning
Code Synthesis
Weak-to-Strong Generalization
In-Context Learning
Prompting
Knowledge Distillation
Education-Inspired
Primary area: natural_language_processing
TL;DR: Aiming to improve LLM reasoning, we conduct a preliminary exploration of whether LLMs can "learn by teaching" -- a well-known paradigm in human learning

Abstract: Teaching to improve student models (e.g., knowledge distillation) is an extensively studied methodology in LLMs. However, in human education, teaching enhances not only the students but also the teachers by fostering more rigorous and clearer reasoning, as well as deeper knowledge building. We ask: Can LLMs also learn by teaching (LbT) for better reasoning? If the answer is yes, we can potentially unlock the possibility of continuously advancing the models without solely relying on human-produced data or stronger models. In this paper, we provide a preliminary exploration of this question. We show that LbT ideas can be incorporated into existing LLM training/prompting pipelines and bring improvements. Specifically, we design three methods, each mimicking one of the three levels of LbT: observing students' feedback, learning from the feedback, and learning iteratively, with the goal of improving answer accuracy without training or improving models' inherent capability with fine-tuning. We reveal some findings: (1) Teaching materials that make it easier for students to learn (via in-context learning) have clearer and more accurate logic; (2) Weak-to-strong generalization: LbT might help improve strong models by teaching weak models; (3) Diversity in students might help: teaching multiple students could be better than teaching a single student or the teacher alone. We hope that our exploration can inspire future research on LbT and, more broadly, the adoption of advanced education techniques to improve LLMs. The code and website are at https://github.com/imagination-research/lbt and https://sites.google.com/view/llm-learning-by-teaching.

Can LLMs Learn by Teaching for Better Reasoning? A Preliminary Study
Xuefei Ning"! Zifu Wang"2 22 Shiyao Li Zinan Lin" 4 Peiran Yao+3.5,
Tianyu Fu Matthew B. Blaschko", Guohao Dais. Huazhong Yang', Yu Wang!
Tiinghua University "KU Leuven IInfinigenc--Al 1 Microsoft Research University of Alberta Shanghai Jíao Tong University
Abstract
Teaching to improve student models (e.g., knowledge distillation) is an extensively studied methodology in LLMs.. However, human education, teaching enhances not only the students but also the teachers by fostering more rigorous and clearer reasoning. as well as deeper knowledge building. We ask: Can LLMS also learn by teaching (LbT) for better reasoning? If the answer is yes, can potentially unlock the possibility of continuously advancing the models without solely relying on human-producee data or stronger models. In this paper, provide preliminary exploration of this question. We show that LbT ideas can be incorporated into existing LLM training/prompting pipelines and bring improvements. Specifically, we design three methods, each mimicking one of the three levels of LbT: observing students' feedback, learning from the feedback, and learning iteratively, with the goal of improving answer accuracy without training or improving models' inherent capability with fine-tuning. We reveal some findings: (1) Teaching materials that make it easier for students to learn (via in-context learning) have clearer and more accurate logic; (2) Yeak-to-strong generalization: LbT might help improve strong models by teaching weak models; (3) Diversity in students might help: teaching multiple students could be better than teaching a single student the teacher alone. We hope that our exploration can inspire future research on LbT and, more broadly, the adoption of advanced education techniques improve LLMs. The code and website are at https: //github..//t///////////gteegg com/ http/mmaannnnninoooo ctw.reaaarrrorrerrcccccooaa and https: / /tt///stess.//e////e////ee//eees google com/ www/wicw//c///cccc http//ieew/llmmlm Seel-eerrnnnng be-------tttte teaching.
1 Introduction
couldn't 'I reduce it to the freshman level. That means we really don't understand it.
Richard Feynman
"Learning from teachers (LfT)" is common pipeline in machine learning, especially in the realm of Large Language Models (LLMs). For example, knowledge distillation [24, 44, 75] and distillation via synthetic data [1, 28, 41] focus on transferring the knowledge from teacher LLMs to student LLMs by letting teacher models teach student models through token logits, features, or synthetic data [85]. They become the go-to methods for closing the performance gap between open-source and proprietary LLMs, as well as for maintaining performance during model compression.
In fact, in human learning, teaching not only benefits students but can also improve the teachers themselves. "Learning by teaching (LbT)", also known as the Feynman learning method, proven to improve human learning by fostering rigorous and clear reasoning well as knowledge building [5, 18, 22, 33, 62-65]. Fig. illustrates the conceptual comparison of the LiT and LbT pipelines.
"Equal contribution. Corresponding to: foxdoraame@ gmail.com (Xuefci Ning), zinanlin @ microsoft.com (Zinan Lin), yu- wang @treennttinnuueu.ccn (Yu Wang)
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Teacher
Teacher
Improve the teacher Build more rigorous and clear knowledge
LLM
LLM
Generate
Gonerote
Teaching
Teaching
Feedback
& exam results, questions
Materials
Materials
Teach
Teach
Student
Imsprove student
Student
Provide feedbock
LLM
inherit toocher knowledge
LLM
Learning from Teacher (LfT)
Learning by Teaching (LbT)
Figure 1: Left:
Learning from teacher aims at improving student LLMs with knowledge from the
teacher LLMs, It is the essential idea behind common approaches including knowledge distillation and distillation via synthetic data. Right: In contrast, Learning by teaching aims at improving teacher LLMs through the teaching process using feedback from student LLMs.
Table 1:
The explored M1, M2, M3 methods.
LbT
LbT Implementation
Objective
Pipeline
Method Abhrev.
Level
Improve the answer quality
L1
Search-based output generation
M1 ($ 3)
without training
Scoring based on students' performance
Improve the inherent model ability with training
1.2
Generation-scorre weapiioon--ooiiin-nnnoei finetuning
M2 ($ 4)
Improve the answer quality without training
Analyzing feedback from multiple students
L3
Input prompt optimization
M3 ($ 5)
Motivated by this insight, in order improve one of the most crucial abilities of LLMS the reasoning ability, we want to ask: Can L.Mss also learn by teaching for better reasoning? In addition to improving reasoning, one imagine, LbT could open exciting opportunities for the models to continuously evolve by teaching other (potentially weaker) models, rather than solely relying on human-produced data or stronger teacher models. More broadly, we hope that this exploration could provide insights on borrowing advanced education techniques to improve LLMs [16, 32, 53J.
To explore this question, draw learning science literature that connects LbT in human learning with reflection [5, 13, 48] and noowlddgebuildiing [62-64], summarizing three levels of LbT:
L1: Observing students' feedback. The teacher instructs the students, who then provide feedback (e.g., taking exams and reporting the scores, asking questions about unclear logic).
L2: Learning from the feedback. Based on the feedback, the teacher can analyze which logic and concepts the students might have (mis)understood. This information useful for the teachers to improve their teaching strategy, and further enhance the teacher's 's own understanding of the concepts.
L3: Learning from the feedback iteratively. The teacher can teach the students, observe the feedback (LI), and learn from the feedback (L2) iteratively.
In this paper, we study the viability of instantiating these LbT ideas in LLMs. There is a range of possibilities in terms of the objective, the pipeline, and the implementation ($ 2 and Tab. 1). As an initial exploration, we study three methods, each for one of the three LbT levels.
MI aims at improving LLMs' answer quality by directly utilizing students' feedback (LI). More specifically, given a set of generated answers, we score each rationale bused ils ability to teach student models using in-context learning (ICL) to correctly answer similar problems. We show that aggregating multiple rationales [79] with LbT-based scores improve the answer accuracy. Notably, M1 improves GPT-4o's accuracy on the MATH dataset [27] from 87.84% to 96.69%.
M2 aims at improving LLMs' inherent ability by learning from students' feedback (L22). We use the approach in M1 to score seacher-generate rationales. Then, we apply direct preference optimization (DPO) [59] to fine-tune the teacher model with the rationale-scom pairs. We show that M2 is better than using DPO with correctness scores.
M3 aims at improving LLMs' answer quality by iteratively learning from students' feedback (L.3). Specifically, we prompt the LLM to reflect on the failure cases of multiple students and devise new positive and negative exemplars. We show that the LLM can improve the exemplars based on feedback from multiple students. These improved exemplars used in prompts not only improve the learning outcomes for multiple students but also enhance the teacher's 's performance.
We reveal some interesting or promising findings related to LbT:

---

Sampler
LLM 1
Generating
Sampler
Evaluator
Evaluator: LbT Scoring
Manual Labeling
Search History
Inerative Search
Matching
Scoring
H11:
Agreement Scoring
R1
R
R3
R
self-evaluation
Choin
Graph
Tree
LbT Scoring
Trainer
Finetuning
LLM 2
Verifier Training
DPO
Filtering & SFT
Aggregator
Shore-Condriooned
(a) The "search-based output generation pipeline" for improving the answer quality.
(b) The sameraaiin-scoring -inetuniin pipeline" for improving model capability.
Figure 2: Two general pipelines for improving the answer quality and model capability. "P" stands for "Problem"; "R" stands for "Rationale": "A" stands for "Answer".
Teaching materials that make it easier for students to learn have clearer and accurate logic (LbT-TMQ' assumption) when using ICL as the student's 's "learning" method: Our LbT- based scoring relies this assumption, and results and inspection support this assumption.
Weak-to-strong generalization: Strong teachers can improve even when teaching weaker dents, suggesting some promise of using LbT to improve superhuman models [6].
Diversity in students might help: Rather than teaching the teacher itself, teaching other students and multiple students might help. This suggests the feasibility of using LbT synergize the capability and knowledge from multiple models.
To summarize, with appropriate pipelines and teacher-student settings, LbT can help improve LLMs' answer quality and inherent capability. We believe that these preliminary case studies are only scratching the surface of the potential of LbT. As LLMs becoming increasingly powerful, more advanced approaches in pedagogy can potentially help with the inference and training of LLMs.
2 Related Work of Our Learning by Teaching Implementations
As shown in Tab. 1, we study two types of objectives: improving answer quality without training and improving the inherent ability of the model with training. $ 2.1 and 2.2 describe how M1, M2, and M3 relate to prior work on these two objectives, respectively. See App. D.5 for more discussion.
2.1 Improving the Answer Quality without Training
Existing literature has incorporated various insights from the human reasoning process to develop prompting-basse methods, including writing down the thinking process [36, 81], subproblem de- composition [52, 56, 94], fetching the abstract principles and answering based on them [93], self- reflection based answer refinement [47, 67]. and so We explore two ways of incorporating the LbT insight to implement two prompting- based methods:
M1 relates to the popular "search- based output generation pipeline" shown in Fig. 2a [4, 42, 45, 47, 67, 79, 84, 87]. This pipeline iteratively samples and evaluates new rationales or rationale steps for searching the optimal output, and ultimately derives the final rationale or answer from the search history. One essential component in this pipeline an evaluator who evaluates the quality of each rationale or rationale step. We design an LbT evaluator that scores each generated rationale based on its ability to teach student models to correctly answer similar problems.
M3 relates to existing prompt optimization methods [57, 71, 95] that iteratively improve the prompts based on their performance (c.g., accuracy, failure cases). The key innovation in M3 is how it evaluates the "performance". instead of evaluating with the same model that produced the prompts (i.e.. the teacher model), test how the prompt works with other student models and show that this change benefits the prompt tuning outcome.
2.2 Improving the Inherent Model Capability with Training
To improve the inherent model capability, M2 incorporates the LbT insight into the "generating- fooring-iinetunin pipeline"" Fig. 2b illustrates the three steps in the pipeline: (1) Letting the target
Refers to "teaching material quality". See App. D.2 for more discussion.
3

---

Sumples Answers
Sample
Sample
Sample
Sample n
Sample
Teacher LLM
TR+TA
TR+TA
TR+TA
TR+TA
Best TA
JCL
AC2
ICL
IC2
MODI OP un pang sopar
EPs
Student LLM
EPs-GT
ER+EA
ER+EA
ER+EA
ER+EA
Evdlochon
Coder Love ETT test coves serl
0.2
L.T
LbT 1.
LbT, :0.5
MATH:: Use G7 onswwe for wool
Figure 3: M1. The goal is to derive the best TA from the TR-TA pairs generated by the teacher L.M.M
LLM or a teacher LLM generate multiple rationales for a given problem; (2) Scoring the rationales
using an evaluator: (3) Utilizing the rationales and scores to (optionally) train verifier [14, 45, 77], and finetune the target LLM by reinforcement learning [77], DPO [59, 88] or its variant [49, 55], filtering and supervised finetuning (SFT) [31, 89, 90], or score-conditioned SFT [43,46].
In these works, the rationale scoring is usually achieved through manual labeling [45, 46, 99], ground- truth (GT) answer matching [89, 90], agreement- based scoring [31], self-evaluation [88]. In contrast, M2 scores the rationale based on its ability to teach student models to correctly answer similar problems. In this way, M2 can provide automatic and fine- quality evaluation for rationales, which helps automate and improve the continual evolution of models' capability.
3 Method (M1) for LbT Level 1: Observing Students' Feedback
3.1
Method
One common teaching strategy in education is that the teacher first teaches students how solve class of problems by giving them the example rationale (named Teaching Rationale, or TR in short) and the answer (named Teaching Answer, TA in short) particular question (named Teaching Problem, or TP in short). Then, the teacher asks students to solve other similar problems (named Exam Problem, or EP in short) test if the students understand the concepts. The teacher can also learn from this process by observing the feedback (i.e., LbT level 1): if the students can answer EPs well, then it likely means that the TR-TA pair is of high-quality.
Our idea is to implement this strategy in LLMs to select high-quality TR-TA pairs. As depicted in Fig. 3 and Alg. AI, we first instruct the teacher model to solve given TP multiple times, resulting in multiple TR-TA pairs. Then, each TR-TA pair is used as an in-context learning (ICL) example to guide the student model in solving series of EPs. With the produced Exam Rationales (ERs) and Exam Answers (EAs), each student will then receive an exam score (i.e., the accuracy of EAs), denoted as the LbT score. The LbT score can be used as a quality assessment of the corresponding TR-TA pair. We consider two ways to select the final TA [79]: (1) We select the TR-TA pair with the highest LbT score. We denote this approach as "M1 (MAX)". (2) For TAs that can be aggregated via exact matching, such as mathematical reasoning, we can take the sum of the LbT scores for each TA separately and find the TA with the maximum sum. We denote this approach as "M1 (SUM)".
The following subsections present the evaluation of M1 on mathematical reasoning and code synthesis tasks. Please refer to App. D for the rationale behind the task selection.
3.2 Evaluation on Mathematical Reasoning
3.2.1 Experimental Setups
We use the extension MATHO [72] of the MATH dataset [27], where each problem has variants with different values. Following the train-test split specified by [45], among the 500 test problems, 181 problems are provided with functional variants each. We use these 181 problems as TPs. For each TP, we sample 256 TR-TA pairs. Then, using each TR-TA pair as the ICL exemplar, we use the 3 functional variants of TP as EPs. Each exam is repeated 3 times with randomized student decoding, resulting in 9 ER-EA pairs. Each TA is scored based on the correctness of the 9 EAs.
3.2.2 Results
We show the results in Tab. 2 and provide analyses as follows. More results are in App. A.

---

Table 2: Results on 181 MATH test problems with 256 TR-TA pairs. The best results of each row highlighted green. The "Improv" column calculates the improvements of average performance achieved by M1 (SUM) over SC.
Teacher
Student
Greedy
SC
M1 (MAX)
MI (SUM)
Improv.
GPT-4o
GPT-4o mini
87.84
91.71
95.03
96.69
+4.98
GPT-4o
LLL.MA--8B
87.84
91.71
94.48
95.03
43.32
GPT-4o
GPT-4o mini & LLaMA3-8B
87.84
91.71
96.13
95.58
+3.87
GPT-3.5
I.L.MA3-BB
59.11
77.90
83.43
83.43
+5.53
GPT-3.5
Mistral-7B
59.11
77.90
81.22
83.43
+5.53
GPT-3.5
LLaMA3-8B & Mistral-7B
59.11
77.90
84.53
84.53
+6.63
LLaMA3-70B
LLaMA3-8B
70.16
81.77
86.74
87.85
+6,08
LLaMA3-70B
Mistral-7B
70.16
81.77
86.19
85.08
+3.31
LLaMA3-70B
LLaMA3-8B & Mistral-7B
70.16
81.77
87.85
87.29
+5.52
LLaMA3-8B
LLaMA3-8B
45.85
64.64
77.90
82.87
+18.23
Mistral-7B
LLaMA3-8B
19.88
40.88
51.93
53.59
+12.71
Improvern inprovements (96
24
75
16
50
1632
64
128
256
510 20
50
100
Number TR-TA Pairs
Difficulty Level
Fraction TPs (%)
Figure 4: Relative improvements of M1 over SC using LLaMA3-8B as the teacher and student on 181 MATH test problems with respect to: (Left) Number TR-TA pairs. Error bars are calculated using the bootstrap sampling technique [43], where 10 subsets sampled from the 256 TR-TA pairs, and standard deviations are computed across these sets; (Middle) Difficulty level; (Right) The fraction of TPs when sorted by the cosine distance the 2 closest problems from the training set.
M1 is effective with various model settings and surpasses baselines. M1 exceeds ceff-consistency (SC) [79] with various model settings: strong-teach-weak (c.g., GPT-4o teaches GPT-4o mini), weak- teach-strong (e.g., Mistral-7B teaches LLaMA3-8B), and self-teaching (e.g. LLaMA3-8B teaches itself). M1 (SUM) outperforms M1 (MAX) in most cases. We also show that LbT-based scoring surpasses self-evaluation scoring [35, 74, 84, 88] Tab. A7. Since M1 incurs higher inference cost than SC when using the same number of TR-TA pairs, we also conduct an experiment in Tab. A6, showing that with comparable or much lower compute, M1 with just TR-TA pairs achieves 0.1 0.17%~8.299 8..9% accuracy improvement over SC with 256 TR-TA pairs.
M1 can further benefit from multiple students. Using GPT-3.5 to teach both LLaMA3-S8 and Mistral-7B achieves a significant improvement than teaching LLaMA3-8B or Mistral-7B separately.
MI can identify infrequent but correct TAs. M1 efficiently discover the correct answer from many teacher samples, whereas SC requires the correct answer to be in the majority to derive Fig. 4 (left, middle) shows the improvements of M1 over SC across different numbers of TR-TA pairs and difficulty levels. The relative improvement of M1 over SC increases as the number of TR-TA pairs the difficulty levels grow within the experimental range.
The TP and the corresponding EPs should be similar. is erucial to choose EPs similar to TP such that the student can apply the logic from TR to solve EPs. We use the functional variants as EPs, which are very similar to TPs. To verify the necessity of TP-EPs similarity, we conduct an experiment that selects similar EPs from the original MATH training set. We calculate the embedding of each TP using the cel---ppett--eeeee problems from the training and calculate the relative improvements over SC on a fraction of TPs. Fig. (right) shows that M1 only provides improvements for TPs that have similar problems in the training set.
sentence embedding model [60], and select the 2 closest EPs. We sort TPs the cosine distance the corresponding EPs
3.3 Evaluation on Competition-Leve Code Synthesis
3.3.1 Experimental Setups
We the Grandmaster Dynamic Programming (DP) study plan LeetCode.? 2
Each dataset in the study plan has 5~10 problems, and each problem has 2~3 visible test cases and many hidden
www.rc--oooorggggggooooo progrramiing--radddssstrrr grandmaster/

---

Table 3: S-score results on Game Theory dataset in LeetCode Grandmaster DP study plan. "SG-1"- "SG-4" and "PW" are abbreviations of individual questions in the dataset; see Tab. A8 for details. The results of M1 that improve (degrade) by more than 0.01 are highlighted in green (red).
Models
Metrics
SG-1
SG-2
SG-3
SG-4
PW
Avg.
0.215
0.004
0.216
0.604
0.609
T=LLaMA3-8B
M1 (MAX)
0.630
0.004
0.228
0.508
S=L.LMA3-8B
Avg. (V-score=l)
0.755
0.851
MI (MAX) (V-score=1)
Avg.
0.348
0.004
0.319
0.608
0.694
T=LLaMA3-8B
MI (MAX)
0.348
0.011
0.570
0.771
0.746
S=LLLMA3-8B
Avg. (V-score=1)
0.797
0.722
0.851
(w. self-Dehugging)
MI (MAX) (V-score=1)
0.935
Avg.
0.582
0.007
0.428
0.645
T=GPT-3.5
MI (MAX)
0.011
0.681
S=GPT-355
Avg. (V-score=))
0.994
0.714
0.894
M1 (MAX) (V-score=1)
1
0.135
Avg.
0.701
0.133
0.592
0.853
T=GPT-3.5
MI (MAX)
0.337
0.714
0.968
G=P--3
Avg. (V-score=1)
0,996
0.714
0.911
(w. Self-Dehugging)
MI (MAX) (V-score=l)
0.714
0.968
Avg.
0.875
0.008
0.679
0.601
T-LLaMA3-708 S=LLaMA3-8B
M1 (MAX)
0.007
Avg. (V-score=()
0.883
MI (MAX) (V-score=l)
test cases. We assign visible score (V-score) of 1 and 0 to the code that passes all or fails any visible cases [43]. To evaluate the actual correctness of a code, we submit the code to LeetCode, and record the pass rate on the hidden cases as the submit score (S-score). For a TP, we sample 8 TR-TA pairs from the teacher, where TR is rationale in natural language, and TA is a Python code (See Ex. for an example). Each TR-TA pair assigned an LbT score by teaching a student to solve the remaining problems in the dataset. M1 calculates the exam V-score as the LbT score to avoid additional LeetCode submissions. Check App. A.3.2 for additional setups.
3.3.2 Results
Here, we analyze the results on the Game Theory dataset. Check App. A.3 for additional results.
M1 can be more general than agreement- methods such SC. M1 (MAX) does not require an oracle to assess the equivalence of two answers, which is challenging for codes. Therefore, we only use the average pass rate (with or without V-score=1 filtering) as the baseline. Nevertheless, when such an oracle is provided [8, 43, 66), we can use M1 (SUM) which was shown to be better than M1 (MAX) in $ 3.2. We defer this exploration future work.
M1 selects better TR-TA than the baseline in most cases. If the student closely follows the strategies in TR-TA to solve EPs, the student cxam score can indicate the quality of TR-TA. (1) When the TR-TA has high quality (Ex. 1), the student mimics the teacher's 's strategy to solve the EP with correct DP code. (2) When the TR- TA is logically incorrect, e.g., DP code with wrong recurrences (Ex. 2) or a non-DP wrong code (Ex. 3), the student also follows the wrong TR-TA with wrong ER and EA. (3) When the TR-TA is logically correct but has high complexity, e.g. recursive re-computation instead of DP (Ex. 4), the student also writes a recursion with high complexity.
As shown in Tab. 3, using the V-score on the few visible test cases can filter out some low-quality code, but MI can identify better TA in most cases. This because LbT-based scoring can leverage student scores on similar EPs, providing a more informative evaluation of TA. Note that M1 shows the largest improvements on TPs with medium difficulty. For very simple (c.g., SG-4 for GPT-3.5) or challenging (c.g.. SG-2) problems, M1 shows marginal no improvements.
Self- Der-gDebgggnn (SD) both complementary and beneficial for M1. We experiment with applying one-iteration SD [11] using Prompt 5. Applying SD on TAs can provide S-score benefits complementary M1, since SD fixes simple non-logical bugs, such as missing imports, miswritten variable names, and incorrect usage of library functions (an example is shown in Ex. 6), whereas M1 mainly assess the quality of the logic. In addition, applying SD on EAs leads to more informative LbT score, as fixing non-logical bugs can make the students' exam V-score more indicative of quality of the TR-TA. Tab. shows that after incorporating SD for both MI and the baselines, M1 achieves consistent improvements.

---

Sampled n TP answers
Pipeline
Sample
Sample
Sample n
Scoring TR bosed the LBT Score
TA+TR
TA+TR
TA+TR
TR
TA Right
TA Wrong
TA Right
0.8
0.2
0.9
Scoring
Scored TRs
correctness
22
DPO
Fine-tuned
DPO
Fine-tuned
Teacher LLM
Teacher LLM
Scored TRs Preference Dato
Preference Data
(a) Correctness-guiieee DPO (Baseline)
(b) DPO with LbT score (Ours)
Figure 5: Baseline M2. Both approaches scores of TRs craft preference data and finetune the teacher LLM with DPO. Left:: The correctness score of TAA. Right: The LbT score TR and TA.
Table 4: Results on 500 MATH test problems with greedy decoding.
Teacher/Student
Original Correctness DPO
M2
LLaMA3-8B
29.0
30.4
32.2
For competition-leve code synthesis task, M1 is more effective when the teacher and student come from the same family, as shown by Tabs. 3 and A10. We find that this because the student can follow teacher from the same model family better, making the feedback more informative. A failure case of student-following when GPT-3.5 teaches LLaMA3-8B is shown in Ex. 5.
TPs and EPs should be similar. Most failure cases in Tabs. 3 and A10 occur when solving the "PW" TP. We find that this because the solving of "PW" involves 2D DP, which differs from other problems that can be solved with ID DP. Consequently, the student cannot follow TA to solve EPs.
Method (M2) for LbT Level 2: Learning from the Feedback
4.1 Method
In education, after identifying which teaching materials (c.g., TR-TA pairs) can enhance student performance ($ 3.1), teachers can use this information to improve their knowledge or teaching strategies. For example, students perform poorly due to unclear or inaccurate teaching materials, teachers can correct their knowledge and avoid generating similar TR- TA pairs in the future.
We this idea train the LLMs improve its reasoning ability. As depicted in Fig. 5, since the LbT--based scoring provides informative feedback on the quality of a TR-TA pair (verified in $ 3), we collect the LbT scores of many TR-TA pairs and use them finetune the teacher with DPO [59].
4.2 Experimental Setups
We use 1564 training problems from MATHO [72] TPs. For each TP, we sample 32 TR-TA pairs from the teacher. For each TR-TA pair, we calculate 0.5 x correctness score + 0.5 x LbT score as its final score, where the correctness score respectively. For running DPO, we select pairs from the 32 TR-TA pairs whose score difference exceeds a threshold of 0.3, and keep at most 8 pairs of TR-TA pairs for each TP.
when the corresponding TA correct or wrong,
4.3 Results
Tab. shows that M2 achieves better results compared to solely using the correctness scores in DPO. This improvement is because LbT provides more informative scores than those purely based on correctness. One example is shown in Ex. 10. Although both TRs produce a correct TA, the losing TR is unnecessarily verbose and cannot be generalized to other similar problems. Another example is in Ex. II. Although both TRs produce a wrong TA, the winning TR is logically better than the loser. LbT can discern the correct preference between these TR-TA pairs, thereby improving DPO results.
Method (M3) for LbT Level 3: Learning from the Feedback Iteratively
5.1 Method
We have shown that the students' exam scores can serve as an indicator the reasoning quality of the teaching rationales. This indicator can be leveraged to aggregate better answers in MI and to further fine-tune the teacher M2. M3, we explore whether reflecting on students' detailed exam responses help the teacher iteratively refine its teaching materials. Notably, we aim to
7

---

Prompt Template-2
Guidee Teacher to generate better pos. and neg, onswers
Wrong
EP+EA
① Refine the teaching
Select Wrong
Iterotively
Answers
Prompt Template-1 Guide Teacher generate p-pos, and neg, polrs
Teacher LLM
Pas. TP+TA
ICL
Student LLM
TP
EPs
Neg. TP+TA
Teaching, Materiol
Evoluate Teacher's
performance
Figure 6: Overview of M3. The teacher teaches the students through set of positive and negative ICL examples. These examples are iteratively refined by the teacher according to students' feedback.
Table 5: Teacher's Fi score of M3 on combined Liar dev and test set at the end iteration T. where LLaMa3-70B is used as the teacher for all settings. The best results are in bold.
Student(s)
=
=
7=
T=
7 = =5
LLaMa3-70B
61.0811.29
62.01
64.48+1.20
65,4040.67
63.96 1.19
LLaMa3-8B
1..24+1.30
65.6620.72 64.47+0.90
66.15=0.56
64.78+0.89
65.41=0.75
LLaMa33(70.8JBB Mistral-7B
63.66:#1.48
65.47=1.01
66.24+0.56
67.09=0.56
verify whether these refinements can enhance the teacher's s own performance by providing more effective knowledge. so, we can assert that the iterative process of teaching, reflection, and material refinement facilitates some form of "knowledge building" [62-64] for the teacher. Additionally, we are interested in whether having multiple and diverse L.MMs as students offers further benefits.
Specifically, we guide the teacher to iteratively improve teaching materials in the form of set of positive and negative exemplars, based on the student and teacher performance when the set used as the ICL examples. As depicted in Fig. 6 and Alg. A2, given a classification task, we first sample K positive and negative exemplars from the teacher, and then run multiple refinement iterations. Finally, we report the teacher performance on the test set when using the resulting ICL examples.
Each iteration contains the following steps: (1) The current exemplars are used as the ICL examples to teach students to answer a set of EPs. The EPs are randomly sampled from the training data in each iteration. (2) We select the EPs that students answered incorrectly and prompt the teacher reflect on why the current exemplars might have misled students these instances.
(3) Based on the reflection, the teacher generates multiple updated exemplar sets. (4) We keep the exemplar set that achieves the best teacher performance on the training data when the set is used as the ICL examples.
5.2 Experimental Setups
We evaluate M3 on two binary text classification tasks: Liar [78] and Logical Fallacy [34]. Liar is dataset for false statement detection in political media, with 4,574 statements with speaker and context information. Logical Fallacy is a dataset of 2,449 samples of 13 logical fallacy types, which we adapt to classify the most common type faulty generalization against the rest of the types. We report the teacher Fi score on the dev and test splits combined. Within an iteration, we choose the exemplar set with the highest teacher F1 score the training set. Across 14 random experiments, we report the mean Fi and the standard deviation. We run a total of five refinement iterations.
5.3 Results
As Tabs. and A17 shows, it is feasible to apply LbT iterative prompt optimization: LLMs able to reflect on the failure cases of students and propose revised exemplars that improve the teacher's performance, similar the case of iteratively optimizing task descriptions as in previous work [57].
More importantly, we observe a performance gain brought by having dedicated students (as opposed to using a single LLM in prompt optimization in previous work). Comparing to the scenario where the teacher and student are the same, having one or multiple LLMs different to the teacher as the student improves the quality of the teaching material faster. This demonstrates LbT as case of weak-to- toroon generalization. We speculate that the benefits are brought by more diverse error types made by a different (weaker) student model; App. C.3 for more examples and analyses.
Broader Discussion
6.1 Insights into In-Contess Learning
Currently, we conduct student "learning" with ICI., based on the assumption that students can effectively "learn" from ICL examples and apply similar strategies to solve EPs. Interestingly, prior
8

---

work [50] found that a correct input-output pairing in ICL examples does not matter much. At first glance, this finding seems to challenge our design, as suggests that the TA accuracy may not affect the EA accuracy, which means the L.bT score cannot reflect the quality of TR+TA However, we find that, as opposed to only providing labels in the ICI. examples [50], providing rationales is important. solvingg logic in the detailed rationale in the ICL. examples well. This
L.Ms can follow the may be because the rationale provides more information, making the ICL examples easier to follow. Consequently, we see that students can use similar logic as the TR when solving the EP. This means that better TR+TA can indeed lead to improved ER and thus higher EA accuracy (i.e., LbT score).
Our findings highlight two key factors for successful ICL following and for establishing a positive correlation between EA accuracy and TR quality/TA accuracy: (1) similarity between TP and EP, and (2) the use of Chain-of Chain-of-Thought (i.e., detailed rationale). See App. A.3.5 for the effect of natural language rationale on ICI. following in code synthesis. Note that we tried explicitly instructing the student to follow the ICI, example, but it did not work well. We hope these findings can complement the current understanding of ICL. [26, 40, 50, 83] and offer insights for improving ICL.
6.2 Weak-to-Strong Generalization
Improving models with gumaan-generated/annoaaa data or synthetic data from stronger models is the dominant paradigm. However, how can we continuously improve the strongest model without relying on buman-generatee and annotated data? A recent work [6] conducts an exploration on using weak model supervision to train larger model. Our work is another attempt towards the "weak-to-strong generalization" " prospect by drawing from how humans continuously acquire new knowledge without direct instruction. We demonstrate that stronger models can further improve their own results (M1), parameters (M2), and prompt (M3) by utilizing the feedback of weaker models.
6.3 Limitations and Hear-Term Extensions
MI and M2 rely on generating/selecting similar EPs. We verify that LbT-based scoring can help select high-quality TR-TAs but require the TP and EPs having similar problem-solving strategies. In our experiments, suitable EPs are selected according to human- provided information in the dataset. One extension is to let a model automatically identify EPs similar to TP from a large pool (Fig. 4).
Another direction is to synthesize similar problems based on a group of problems and exploit the LbT principle to score many rationales for the new problems. Specifically, as a "self-instruct [80] extension to M2, we can generate a new problem P based on group of problems S = [P1 that are already known be similar. The generating generating-scoring pipeline can then be applied to P to obtain rationale-scor pairs, where the LbT score can be easily obtained using S as the EPs.
Pk
A Additional inference cost. LbT-based scoring in MI and M2 requires additional inference cost, which aligns with recent studies that show that increasing inference cost might be a promising way to improve models' reasoning capabilities [54, 69]. Nevertheless, designing efficient inference algorithms and systems [96] is needed to make these approaches more usable.
See App. E for other extensions and App. F for the discussion on potential risks of bias perpetuation.
6.4 Borrowing Education Strategies to Improve LLMs
Borrowing the design strategies of teaching materials. We show an LbT pipeline in Fig. 7. Each iteration involves six steps: (1) The teacher generates the Teaching Material (TM). (2) The student learns from the TM. Our work uses in-context learning for all student learning, but exploring other learning strategies is an interesting future direction. (3) The student provides feedback. The feedback can take many forms as listed in the figure. Our work mainly explored feedback in the form of exam details and scores.
(4) The teacher reflects on the feedback and identifies the knowledge gaps in the TM or in the teacher's 's own knowledge. (5) The teacher can optionally refer to some external data source to address its own knowledge gaps. (6) The teacher improves the rigorousness, clarity, and completeness of their knowledge and updates the TM for the next iteration.
On one hand, updating TM can improve the teacher's own knowledge. For example, M3 saves the updated exemplars as the teacher's prompt to improve the teacher's 's reasoning. On the other hand, a high-quality TM helps students learn better, so that the students can provide more meaningful feedback for the teacher. To create high-quality TM, it might be beneficial to borrow from TM design strategies in human education. Fig. 7 summarizes various TM design strategies, among which our work has explicitly explored three types (marked in black).
Borrowing the pipelines. We can borrow insights from education pipelines to design L.M inference and training pipelines. For example: (1) Task-oriented collaborative multi-agent learning [51]:

---

Source
Precter sedeits trncher's
Types knooweddee
own knowledpe
Tearter refoce to some acternal doto rounce
1. Basic knowlcdge: Concept/ooji/FFrrmeeo
2. Intuition that helps understand basic knowledge Strategy of problem solving
LMM other types of krowledge base E. explict memrnyy knowledge grain
4. Teocher coflects studerer ferdbant:: idvently inowiddgr pape the teather's own inowlddge.
Types of Teaching Material Design Strategles
Teacher generetes
Explsin knowledge (types lsttd ahove)
Types Feedback
Contrast close different
pasilyy misundesstod keiowledge
Teaching Materials (TM)
Feedback
2. Quesilons
1. Exom details and scores
Explain teaching rationale
3. Other free-form thoughts
5. select teaching and exam problems
2. Reacher teaches Stadeets leam
8. Select teaching nationwle
7. Amange the order of esamplers and general princislos.
8. Amange the order of problems with different difficuttins
Student
LLMs
3. Studeett proside ferdhoot
Figure 7: LbT pipeline and the summary of knowledge types, TM design strategies, and feedback.
Multiple LLM agents can form a collaborative study group to learn difficult topics in a task-oriented manner (see some discussions in App. D.2). Similar multi-agent collaboration ideas have been leveraged by LLM agent research [12, 17, 29, 39, 58, 91]. (2) Better LbT by configuring proper teacher&student. Literature on "teachable agents" finds that configuring student's knowledge level appropriately can lead to more useful feedback for the teacher [5, 65J. This suggests the intriguing possibility of prompting the student LLM to "confine" its knowledge level [33]. thereby amplifying the benefits of LbT for the teacher model. Furthermore, a junior model M can first teach a "student" that is stronger than itself, who can understand and critique mistakes and ambiguity. As M becomes stronger, it might become better at evolving its knowledge by teaching weaker students. This can be seen form of easy-to-hard task progression in curriculum learning [3]. (3) Flexible teaching quality evaluation: In human learning, feedback can take many forms [25, 82] beyond traditional exams [62-64], such as peer recommendations [48] and satisfaction questionnaires [37, 61]. Such mechanisms can be adapted to LLMs, potentially useful for open-ended tasks.
7 Conclusion
Aiming improve LLM reasoning, we conduct a preliminary exploration of whether LLMs can "learn by teaching" a well-known paradigm in human learning. We implement the LbT idea into well-established pipelines to develop three methods, and evaluate whether they improve reasoning performance on complex tasks such as mathematical reasoning and competition-level code synthesis:
M1 is based the LbT-TMQ assumption. Specifically, method and measure the students' success in grasping the logic embedded in ICL examples by evaluating their performance similar EPs. M1 is implemented output generation" " pipeline with an LbT-based rationale scoring component.
adopt ICL as the instructional
standard "search-based
Results: In mathematical reasoning, M1 achieves a 3.31 -~~ 8.23% accuracy improvement over the competitive SC baseline on 181 MATH test problems with 256 TR-TA pairs (Tab. 2). Note that M1 achieves a 3.32%~4.988% improvement on the powerful GPT-4o, reaching a high accuracy of 96.69%. Using comparable or much lower compute, M1 with 24 TR-TA pairs achieves 0. 0.7%% accuracy improvement over SC with 256 TR- TR pairs (Tab. A6).
In code synthesis, M1 achieves notable improvements in submission score in most scenarios, particularly when the teacher and student belong to the same model family (Tabs. 3, All and A13).
M2 uses the LbT scores from M1 to fine-tune the teacher with DPO. M2 is implmented standard "generating- wwwrrsss-oooooooieeeeeeee finetuninng"" pipeline with an LbT-based rationale scoring component. Results: the experiment of fine-tuning LLaMA3-885 the M2-tuned model achieves a 1.8%
accuracy improvement over the model tuned with correctness- based DPO, evaluated on 500 MATH test problems with standard greedy decoding (Tab. 4).
M3 lets the LLM iteratively refine ICL examples by analyzing the students' feedback.
Results: For two binary text classification tasks requiring common-senss and logic reasoning, M3 can craft better ICL examples through multiple refinement rounds, and the feedback from students other than the teacher itself is beneficial (Tabs. 5 and A17).
In summary, the LbT idea is implemented as a scoring method in M1 and M2, and as an iterative refining pipeline in MB (Tab. 1). Our results suggest LbT's potential for harnessing the diversity offered by different students and facilitating weak-to-strone generalization in improving reasoning.
We believe our work only scratches the surface of leveraging educational principles to improve LLMs. Will these approaches find greater use as L.LMs grow more intelligent? We discuss our research rationale and roadmap for exploring this intriguing question in App. D and $ 6.
10

---

Acknowledgement
This work was supported by National Natural Science Foundation of China (No. 62325405, 62104128. U19B2019, U21B2031, 61832007, 62204164), Tsinghua EE Xilinx AI Research Fund, and Beijing National Research Center for Information Science and Technology (BNRist). We thank for all the supports from Infinigence-AI. Z.W. and M.B.B. acknowledge support from the Research Foundation Flanders (FWO) through project numbers G0A1319N and S00142IN, and funding from the Flemish Government under the Onderzoeksprogrammet Artificiële Intelligentie (AI) Vlaanderen programme. Z.W. and M.B.B. acknowledge LUMI-BE for awarding this project access the LUMI supercomputer, owned by the EuroHPC JU, hosted by CSC (Finland) and the LUMI consortium, and EuroHPC JU for awarding this project access to the Leonardo supercomputer, hosted by CINECA. We thank Sergey Yekhanin from Microsoft Research for their support and suggestions for the work. We thank Zixuan Zhou, Chao Yu, Boxun Li for their discussions. We thank the anonymous reviewers for their insightful suggestions.
References
[1] Marah Abdin, Sam Ade Jacohs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 242214444..111 2024.
[2] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint
2021.
[3] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pages 41-48,
2009.
[4] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyezyk, al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17682-17690, 2024.
[5] Gautam Biswas, Krittaya Leelawong, Daniel Schwartz, Nancy Vye, and The Teachable Agents Group at Vanderbilt. Learning by teaching: A new agent paradigm for educational software. Applied Artificial Intelligence, 19(3-4):363339) 2005.
[6] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong gener- alization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09333 2023.
[7] Xiaofeng Cao. Yaming Guo, Tieru Wu, and Ivor W Tsang. Black-box generalization of machine teaching. arXiv preprint arXiv:2206. 15205, 2022.
[8] Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations, 2023.
[9] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
[10] Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. Universal self-consistency for large language model generation. arXiv preprint arXiv:2311.173/1 2023.
[11] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations, 2024.
11

---

[12] Yuheng Cheng, Ceyao Zhang, Zhengwen Zhang, Xiangrui Meng, Sirui Hong, Wenhao Li, Zihao Wang, Zekai Wang, Feng Yin, Junhua Zhao, al. Exploring large language model based intelligent agents: Definitions, methods, and prospects. arXiv preprint arXiv:2401.03423 2024.
[13] Michelene TH Chi, Stephanie A Siler, Heisawn Jeong, Takashi Yamauchi, and Robert G Hausmann. Learning from human tutoring. Cognitive science, 25(4):471-55) 2001.
[14] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
[15] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 113-123, 2019.
[16] Aniket Rajiv Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timo- thy P Lillicrap, Danilo Jimenez Rezende, Yoshua Bengio, Michael Curtis Mozer, and Sanjeev Arora. Metacognitive capabilities of LLMs: An exploration in mathematical problem solving. In Al for Math Workshop @ ICML. 2024, 2024.
[17] Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. Self-collaboration code generation via chatgpt. ACM Trans. Softw. Eng. Methodol., jun 2024.
[18] David Duran. Learning-by ceemming-by-teachhin byytecching. evidence and implications pedagogical mechanism. Innovations in education and teaching international, 54(5):476-484, 2017.
[19] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21,, 2019.
[20] Matthias Feurer and Frank Hutter. Hyperparameter optimization. Automated machine learning: Methods, systems, challenges, pages 3-33, 2019.
[21] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.
[22] Alan Gartner al. Children teach children: Learning by teaching. 1971.
[23] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey, International Journal of Computer Vision,
2021.
[24] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2023.
[25] John Hattie and Helen Timperley. The power of feedback. Review of educational research, 77(1):11-112 2007.
[26] Roce Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. arXiv preprint arXiv:2310. 23101166.,, 2023.
[27] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks, volume 1, 2021.
[28] Namgyu Ho, Laura Schmid, and Se- Young Yun. Large language models are reasoning teachers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14852-14882, Toronto, Canada, July 2023. Association for Computational Linguistics.
[29] Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for multi-agent collaborative framework. The Twelfth International Conference on Learning Representations, 2024.
12

---

[30] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: survey. IEEE transactions on pattern analysis and machine intelligence. 44(9):5149-516) 2021.
[31] Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1051-1068, Singapore, December 2023. Association for Computational Linguistics.
[32] Can Jin, Tong Che, Hongwu Peng, Yiyuan Li, and Marco Pavone. Learning from teach- ing regularization: Generalizable correlations should be easy to imitate. arXiv preprint arXiv:2402.02769 2024.
[33] Hyoungwook Jin, Sconghee Lee, Hyungyu Shin, and Juho Kim. Teach ai how to code: Using large language models as teachable agents for programming education. In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI '24, New York, NY, USA, Association for Computing Machinery.
2024.
[34] Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrin- maya Sachan, Rada Mihalcea, and Bernhard Schoelkopf. Logical fallacy detection. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7180-7198, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[35] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint (rXii:2207.0522.. 2022.
[36] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213, 2022.
[37] Janet Leckey and Neville Neill. Quantifying quality: the importance of student feedback. Quality in Higher Education, 7(1):19-32, 2001.
[38] Aitor Lewkowycz. Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
[39] Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36, 2023.
[40] Jiaoda Li, Yifan Hou, Mrinmaya Sachan, and Ryan Cotterell. What do language models learn in context? the structured task hypothesis. arXiv preprint arXiv:2406.042i6 2024.
[41] Shiyang Li, Jianshu Chen, yelong shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, Wenhu Chen, and Xifeng Yan. Explanations from large language models make small reasoners better. In 2nd Workshop Sustainable AI, 2024.
[42] Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5315-5333, 2023.
[43] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 37866624) 2022.
[44] Chen Liang, Simiao Zuo Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Less more: Task-aware layer-wise distillation for language model compression. In International Conference Machine Learning, pages 20852-20867. PMLR, 2023.
13

---

[45] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20057 2023.
[46] Hao Liu, Carmelo Sferrazza, and Picter Abbeel. Chain of hindsight aligns language models with feedback. In The Twelfth International Conference on Learning Representations, 2024.
[47] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances Neural Information Processing Systems, 36, 2024.
[48] Lynn McAlpine, Cynthia Weston, Catherine Beauchamp, C Wiseman, and Jacinthe Beauchamp. Building a metacognitive model of reflection. Higher education, 37:105-131, 1999.
[49] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. arXiv preprint arXiv:2405.14734, 2024.
[50] Sewon Min, Xinxi Lyu. Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.1 /2837, 2022.
[51] Joel M Moskowitz, Janet H Malvin, Gary A Schaeffer, and Eric Schaps. Evaluation of jigsaw, cooperative learning technique. Contemporary educational psychology, 10(2):104-112, 1985.
[52] Xuefei Ning. Zinan Lin, Zixuan Zhou, Zifu Wang, Huazhong Yang. and Yu Wang. Skeleton- of-thouuht: Prompting 1ims for efficient parallel generation. In The Twelfth International Conference on Learning Representations, 2024.
[53] Andreas Opedal, Alessandro Stolfo, Haruki Shirakami, Ying Jiao, Ryan Cotterell, Bernhard Schölkopf. Abulhair Saparov, and Mrinmaya Sachan. Do language models exhibit the same cognitive biases in problem solving as human learners? In International Conference on Machine Learning, pages 38762-38778. PMLR, 2024.
[54] OpenAl. Learning to reason with Ilms, Sep 2024.
[55] Richard Yuanzhe Pang, Weizhe Yuan. Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733, 2024.
[56] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Mcasuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687-5711, Singapore, December 2023. Association for Computational Linguistics.
[57] Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with "gradient descent" and beam search. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7957-7968. Singapore, December 2023. Association for Computational Linguistics.
[58] Chen Qian, Xin Cong, Cheng Yang. Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. arXiv preprint arXiv:2307.07924 2023.
[59] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances Neurul Information Processing Systems, 36, 2024.
[60] Nils Reimers and Iryna Gurevych. Sentence-BEir BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language P Processing (EMNI.--ICCILP), pages 3982-3992, Hong Kong, China, November 2019. Association for Computational Linguistics.
14

---

[61] John TE Richardson. Instruments for obtaining student feedback: A review of the literature. Assessment & evaluation higher education, 30(4):387-415, 2005.
[62] Rod D Roscoe and Michelene TH Chi. The influence of the tutee in learning by peer tutoring. In Proceedings of the Anmual Meeting of the Cognitive Science Society, volume 26, 2004.
[63] Rod D Roscoe and Michelene TH Chi. Understanding tutor learning: Knowledgg-huilding and knowledge-telling in peer tutors' explanations and questions. Review of educational research, 77(4):534-574, 2007.
[64] Rod D Roscoe and Michelene TH Chi. Tutor learning: The role of explaining and responding to questions. Instructional science, 36:321-350, 2008.
[65] Tasmia Shahriar and Noboru Matsuda. "can you clarify what you said?": Studying the impact of tutee agents' follow-up questions on tutors' learning. In Artificial Intelligence in Education: 22nd International Conference, AIED 2021, Utrecht, The Netherlands, June 14-18, 2021, Proceedings, Purt I 22, pages 395-407. Springer, 2021.
[66] Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida Wang. Natural language to code translation with execution. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3533-3546, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[67] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[68] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained on recursively generated data. Nature, 63 (31(2))): 759, 2024.
[69] Charlic Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.033: 03314, 2024.
[70] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
[71] Alessandro Sordoni, Eric Yuan, Mare-Alexandre Côté, Matheus Pereira, Adam Trischler, Ziang Xiao, Arian Hosseini, Friederike Niedtner, and Nicolas Le Roux. Joint prompt optimization of stacked lims using variational inference. In Advances in Neural Information Processing Systems, volume 36, pages 58128 18-55115... Curran Associates, Inc., 2023.
[72] Saurabh Srivastava, Anto PV, Shashank Menon, Ajay Sukumar, Alan Philipose, Stevin Prince, Sooraj Thomas, et al. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. arXiv preprint 20Xv::2402.114500. 2024.
[73] Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. Easy-to-hard generalization: Scalable alignment beyond human supervision. arXiv preprint
2024.
[74] Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pages 5433-5442, Singapore, December 2023. Association for Computational Linguistics.
[75] Inar Timiryasov and Jean-Lour Tastet. Baby Ilaaa: knowledge distillation from an ensemble of teachers trained on small dataset with no performance penalty. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pages 279 279-289, Singapore, December 2023. Association for Computational Linguistics.
15

---

[76] Thiemo Wambsganss, Vinitra Swamy, Roman Rietsche, and Tanja Käser. Bias at second glance: A deep dive into bias for german educational peer-review data modeling. arXiv preprint arXiv:2209. 10335, 2022.
[77] Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Pupers), pages 9426-9439, 2024.
[78] William Yang Wang. 'liar, liar pants on fire": A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422-426, Vancouver, Canada, July 2017. Association for Computational Linguistics.
[79] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
[80] Yizhong Wang. Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hanmaneh Hajishirzi. Self-instruct: Aligning language models with self- instruc- tions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the st Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Pupers), pages 13484- 13508, Toronto, Canada, July 2023. Association for Computational Linguisties.
[81] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 24837, 2022.
[82] Benedikt Wisniewski, Klaus Zierer, and John Hattie. The power of feedback revisited: A meta-analysis of educational feedback research. Frontiers in psychology, 10:487662, 2020.
[83] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080 2021.
[84] Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min-Yen Kan, Junxian He, and Michael Xie. Self-evaluation guided beam search for reasoning. Advances in Neural Information Processing Systems, 36, 2024.
[85] Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint 24X.:22402.13116, 2024.
[86] Kaiyu Yang, Aidan Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan J Prenger, and Animashree Anandkumar. Leandojo: Theorem proving with retrieval-augmenter language models. Advances in Neural Information Processing Systems, 36, 2024.
[87] Shunyu Yao. Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Seirth- Conference on Neural Information Processing Systems, 2023.
[88] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.70020 2024.
[89] Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint wrxiv:2308..0825, 2023.
[90] Erie Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35: 2022.
[91] Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models. In The Twelfth International Conference on Learning Representations, 2024.
16

---

[92] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 4320 4328, 2018.
[93] Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H. Chi, Quoc V Le, and Denny Zhou. Take step back: Evoking reasoning via abstraction in large language models. In The Twelfth International Conference on Learning Representations, 2024.
[94] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and H. Chi. Least-to-most prompting enables complex reasoning large language models. In The Eleventh International Conference on Learning Representations, 2023.
[95] Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, 2023.
[96] Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, et al. survey on efficient inference for large language models. arXiv preprint arXiv:2404. 14294, 2024.
[97] Xiaojin Zhu. Machine teaching: An inverse problem machine learning and approach toward optimal education. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
[98] Xiaojin Zhu, Adish Singla, Sandra Zilles, and Anna Rafferty. An overview of machine teaching. arXiv preprint arXiv:1801.05922 2018.
[99] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 199.0093990 2019.
17

---

Appendix
Table of Contents
A M1
19
A.1
Workflow
19
A.2
Mathemattcal Reasoning
19
A.2.1
Prompt Design
A.2.2
Additional Experimental Setups
Stu.............................
20
A.2.3
Additional Results
20
A.3
Competition-Leeel Code Synthesis
21
A.3.1
Prompt Design
21
A.3.2
Additional Experimental Setups
22
A.3.3
Additional Results
23
A.3.4
Examples
26
A.3.5
Natural Language Rationale has Positive Effect Code Following in
34
B
M2
38
B.1
Additional Experimental Sctups
38
B.2 Examples
38
C M3
40
C.I. Workflow
40
Prompt Design
................................
40
C33 Examples
41
D Discussion on Research Rationale
42
D.1 Target (the "nail" in the high level)
42
D.2 Idca (the "hammer" in the high level)
43
D.3 Task (the "nail" in the low level).
43
Implementation (the "hammer" the 
44
D.5 Relation other machine learning techniques
................................
45
E More Extension Possibilities
45
F Potential Risks of Bias Perpetuation
46
18

---

A M1
A.1 Workflow
The pseudo-code for M1 is illustrated in Alg. A1. In each iteration of the for loop, we calculate the LBT score of one TA in the following three steps:
1. Use teacher LLM to sample TR-TA pairs: As shown in line 4 of the Alg. AI, given TP, the teacher LLM samples diverse TR-TA pairs as the teaching materials. The prompts the teacher are shown in Prompt (MATH) and Prompt 3 (Coding).
2. Use student L.MM to take the exam: As shown in lines 7-11 in Alg. A1, the sampled TP- TR-TA pair serves related EPs. The prompts of the student LLM are shown in Prompt 2 (MATH) and Prompt (Coding). Finally, evaluate each EA and store its exam score calculate the LbT score.
in-context demonstration and prompt the student LLM to solve
3. Calculate the LbT score for TA: As illustrated in lines 14-18 in Alg. AI, we use the exam scores of EPs to calculate the LbT score for each TA. For MATH, we employ both the MAX and SUM modes, while for Coding, we use only the MAX mode.
Finally, after getting the LbT score for all TAs, we select the TA with the largest LbT score.
Algorithm A1 The Workflow of M1
Require:
Teacher and Student LLM: S
Teaching Problem: TP
Exam Problems: EPs
Ground-truth of Exam Problems: EPs_GT
Number of TRs: n
LPT Mode: mode
1: lbt = defaultdict(float)
2: for 1 to do
3: # Teacher LLM samples TR and TA
4: TR, TA = T(TP)
5:
6: # Student LLM performs exams
7: exam exam_scores []
8: for EP, BP_GT in zip(EPs, EPs_GT) do
9: ER, EA S(TP, TR, TA, EP)
10: exam scores apores.append(Eval(EAA EP EP]GT)
end for
12:
Calculate the MAX/SUM LbT score of each TA if mode == "MAX" then
14:
[bt[TA] max()b([TA), average(exam_scores)))
16:
else
17:
Ibt[TA] += averagefexam_scoree)
18: end if
19: end for
20:
21: return max(lbt, key=lbt.get)
A.2
Mathematical Reasoning
A.2.1 Prompt Design
For MATH, 4-shot examples from Minerva [38]. The prompt for the student the the teacher, except that an additional shot from the teacher (i.e. a TP-TR-TA TA pair) appended after the original 4-shot examples.
19

---

Prompt 1. Teacher Prompt (MATH)
[User:]
Your task the last question below. Give step step reasoning before you When re ready answer, please wrap your answer and conclude using the format
[[Final Answerl]:
SANSWERS
(4-shot examples]
[[Question]]:
(TP)
[Assistant:]
[[Solution]]:
Let's think stop by step.
Prompt Student Prompt (MATH)
[User:]
Your answer question step by step reasoning you answer. you'te ready to answer, please answer and conclude using the format
[[Final Answer]::
SANSWERS
(4-shot examples]
[[Question]]:
(TP)
[[Solution]]:
(TR)
[[Final Answer]]
(TA)
[[Question]]:
(EP)
[Assistant:]
[[Solution]]: Let's think step by step.
A.2.2 Additional Experimental Setups
In $ 3.2.1, both the teacher and the student use the same sampling parameters. Following [73], we use top-K sampling with K=20 and temperature of 0.7.
In Tab. A7, we provide additional results in small-scale experiments, Specifically, for each of the 7 categories in MATH, select 10 problems, resulting in a total of 70 TPs. We then sample 16 TR-TA pairs for each TP. The rest of the procedure follows 3.2.1.
A.2.3 Additional Results
M1 requires additional inference costs for the student, raising concerns about whether M1 can surpass the baseline within the same budget. We provide additional results in Tab. A6. Notably, using only 24 TR-TA pairs MI still outperforms SC with 256 TR-TA pairs, especially on the most recent LLaMA3 models. Under this setting, M1 has a lower inference cost than SC, particularly when comparing MI and SC on LLaMA3-70B and using L.LMM3-8B the student (the first row of the table).
20

---

Furthermore, shown in $ 3.2.2, the improvement over the baseline does not saturate and actually increases, indicating that M1 could achieve higher upper-bound performance.
In Tab. A7, we compare LbTT-based scoring with self-evaluation scoring [35, 74, 84, 88]. Our method consistently outperforms the self-evaluation baseline. Additionally, the table reveals that the performance gap between the strong-teach-strong and strong-feach-wen settings is relatively small.
Table A6: Results on 181 MATH test problems. SC is with 256 TR-TA pairs, while M1 with 24 TR-TA pairs. Standard deviations calculated using the bootstrap sampling technique [43], where size- subsets are sampled from the 256 TR-TA pairs as the TR-TA set for M1, and standard deviations computed across 10 sets. The "Improv" column calculates the improvements of average performance achieved by M1 (SUM) over SC.
Teacher
Student
Greedy
SC
M1 (MAX)
M1 (SUM)
Improv.
GPT-4o
GPT-4o mini
87.84
91.71
94.20 = 0.79
94.36 E 0.88
+2.65
GPT-4o
13-MMA3-88
87.84
91.71
93.92 0.92
91.14 + 0.83
+2.43
GPT-4o
GPT-4o mini & LLaMA3-8B
87.84
91.71
93.98 0.58
94.31 + 0.43
+2.60
GPT-3.5
LLaMA3-8B
59.11
77.90
78.34 1.86
79.50 + 2.13
+1.60
GPT-3.5
Mistral-7B
59.11
77.90
77.83 1.34
78.07
1.19
+0.17
GPT-3.5
LLaMA3-8B & Mistral-7B
59.11
77.90
80.94 1.51
80.61 1.72
+2.71
LLaMA3-70B
LLaMA3-8B
70.16
81.77
84.97
85.69
1.49
+3,92
LLaMA3-70B
Mistral-7B
70.16
81.77
82.65
84.03
1.47
+2,26
LLaMA3-70B
LLLMAA3888 Mistral-7B
70.16
81.77
84.33 1.26
81.48 1.36
1.36
+2.71
LLaMA3-8B
LLaMA3-8B
45.85
64.64
70.83
1.91
72.93
1 2.15
+8.29
Mistral-7B
1.LMA3-8B
19.88
40.88
40.55 1.82
42.43
1.78
+1.55
Table Results on 70 MATH problems with 16 TR-TA pairs.
Teacher
Student
SC
Self-Eval
M1 (MAX)
MI (SUM)
GPT-4
GPT-4
67.14
68.57
70.00
72.86
GPT-4
GPT-3.5
67.14
68.57
71.43
72.86
GPT-3.5
GPT-3.5
52.86
52.86
57.14
58.57
GPT-3.5
Mistral-7B
52.86
52.86
54.29
57.14
A.3
Competition-Leve Code Synthesis
A.3.1
Prompt Design
Prompt 3. Teacher Prompt (Coding)
[Uer]] [[Questio]]:
(TP)
First, let's think step by step complete problem-solving stratogy. Then, write Pytbon code based the problem-solving stralegy.
[Assistant:]
[[RATIONALE]]:
Prompt Student Prompt (Coding)
[Uer::
[[Question]]:
Here an example question. please understand very carefully:
(TP)
First, let's think step by step find complete prohlem solving strategy. Then, write Python code on the problem-solving stratogy.
mrhhemm-solving
21

---

[[RATIONALE]]
(TR)
[[Final Code]]:
(IA)
[[Question]]:
understand problem solving appenach example to solve the following similar type of problem:
aforemenniones example,
(EP)
First, let's think step step find complete problem-solving strategy,
Then, write a Python code based on the problem-soovvngg stratogy.
[Assistant:]
[RATIONALEI]:
Prompt 5. Self- Self-Debugging Prompt (Coding)
[User: [[Question]]:
(TP or EP)
[[RATIONAL]]]:
ER)
[[Final Codel]:
(TA or FAA
You need to dobug this code with the following rules:
(1) think the provided code correct, must retrieve the original correct code.
(2) think the provided code incorret,, you debug the code and write the final free code.
there complete code, you complete code based the rationale.
think step by slep remember you **must* give complete Py finally.
[Assistant:]
A.3.2 Additional Experimental Setups
To get the problems from LeetCode, follow the approach of Reflexion [67] to LeetCode's official API to obtain all datasets from the Grandmaster dynamic programming study plan. We also employ GPT-3.5 to extract all visible examples for offline evaluation. Additionally, LeetCode Python API to submit the Python code and evaluate the code on invisible examples.
the
As shown in Alg. A1, within the for loop, we first have the teacher LLM sample TR and TA. To validate LbT, we introduce randomness at this step to generate a variety TR-TA pairs. For GPT-3.5- 0613, we set the temperature and top-P to 1, while for the I.LaMA3 family, we set the temperature to 0.6 and top-P to 0.9 (default setting). Next, we prompt the student L.LM with the sampled TR- TA as ICL examples to solve EPs. In this step,
greedy sampling.
22

---

Table A9: S-score results with standard deviation on Game Theory dataset in LeetCode Grandmaster DP study plan. "SG-I" and "PW" abbreviations of individual questions in the Game Theory dataset. The standard deviations are calculated using the bootstrap sampling technique [43] where size-4 subsets sampled from the 8 TR-TA pairs the TR-TA set for M1, and the standard deviation of performance is computed across 20 sets.
Models
Metries
SG-1
SG-2
SG-3
SG-4
PW
Avg.
0.004
0.002
0.209
0.068
0.613
T-LLaMA1-8 8B S=11LMA3-8B
MI (MAX)
0.530 L 0.162
0.004 1 0.004
0.220
0.116
0.600
0.233
0.576
0.346
Avg. (V-scorewl)
1,000 0.000
0.294
0.847
0.113
MI (MAX) (Voccor==))
1,000 0.000
0.724
0.332
0.124 0.939
Avg.
0.335 =: 0.142
0.006 + 0.002
0.292 + 0.108
0.591
0.106
0.696
0.076
T-IJaMA3-8B
MI (MAX)
0.382 0.2242
0.009
0.004
0.503
0.130
0.728
0.154
0.723
0.121
S-LL=MA3-8B
Arg. (V-score=1)
0.827 1: 0.147
0.653
0.318
0.830
0.104
Self-Dehugging)
MI (MAX) (V-snores))
0.928 0.196
0.346 0.815
0.072 0.941
Avg.
0.582 1: 0.128
0.007 +: 0.002
0.432 + 0.177
1,000
0.000
0.643
0.132
T=GPT-3.5
MI (MAX)
0.827 = 0.178
0.010 = 0.002
0.631 = 0.193
1.000
0.000
0.771
0.129
S=GPT-3.5
Avg. (V-score=l)
0.593 0.000
0.746 0.299
1.000
0.000
0.914
0.082
MI (MAX) (V-score=1)
1,000 0,000
0.593 0.2
1.000
0.000
0.982
0.0
Avg.
0.723 = 0.162
0.096 = 0.119
0.586 = 0.136
1.000 + 0.000
0.841 = 0.070
TwGPT:3.5
MI (MAX)
1,000 0.000
0.255
0.368
0.635
0:323
1.000 0.000
0.104
S=GPT-3.5
Avg. (V-score=1)
0.996 0.004
1.000 = 0.000
0.066 = 0.338
1.000 = 0.000
0.8 0.897 = 0.088
Self-Dehugging)
MI (MAX) (V-sonrewl)
1,000 0.000
1.000
0.000
0.006 0.338
1.000 0.000
0.075
Avg.
0.838 1. 0.119
0.008 1. 0.001
0.677 0.145
1.000 0.000
0.597 0.102
T=LLaMA3-70B
MI (MAX)
0,9000 0.2000
0.007 = 0.002
0.787 = 0.398
1.000
0.000
0.671 0,112
M-LL-MA3-88
Avg. (V-core==))
1,000 0.000
1.000 0.000
1.000
0.000
0.918 D.127
MI (MAX) (V-score=1)
1,000 0.000
1.000 = 0.000
1.000
0.000
0.9855
0.112
Table A8: The question IDs and the question title names of problems in Game Theory, Bitmasking, General-ID, and Tricky Invariant datasets. Note that only the Tricky Invariant dataset is related to the binary search, while the remaining three datasets focus on dynamic programming.
Dataset
Question ID
Question Title
Question Title Abbr.
486
Predict the Winner
PW
877
Stone Game
SG-1
Game Theory
1140
Stone Game II
SG-2
1406
Stone Game III
SG-3
1510
Stone Game IV
SG-4
698
Partition to Equal Sum Subsets
PKE
465
Optimal Account Balancing
OAB
Bitmasking
847
Shortest Path Visiting All Nodes
SPV
1125
Smallest Sufficient Team
SST
1434
Number of Ways to Wear Different Hats to Each Other
NWW
1799
Maximize Score After N Operations
MSA
1048
Longest String Chain
LSC
376
Wiggle Subsequence
WS
651
4 Keys Keyhoard
4KK
General-1D
32
I Longest Valid Parentheses
LVP
1416
Restore The Array
RTA
1259
Handshakes That Don't Cross
HTD
639
Decode Ways II
DW-2
1539
Kth Missing Positive Number Index II
KMPN
Tricky Invariant
275
HI2
A.3.3 Additional Results
As shown Table A9, we use bootstrap sampling (choose 4 TR-TAs from TR-TAs, 20 sets are sampled) to produce the results in Table with standard deviations.
23

---

Table A10: Ablation of model settings on Game Theory dataset in LeetCode Grandmaster DP study plan: The teacher and studnet belong to different model families. The results of M1 that improve (degrade) by more than 0.01 are highlighted in green (red).
Models
Metrics
SG-1
SG-2
SG-3
SG-4
PW
Avg.
0.215
0.004
0.216
0.604
0.609
T-LLaMA3-8B
M1 (MAX)
0.101
0.005
0.524
0.611
0,462
S=GPT-3.5
Avg. (V-score=1)
0.755
0.851
M1 (MAX) (V-score=1)
0.871
Avg.
0.348
0.004
0.319
0.608
0.694
T-LLaM33-8B
M1 (MAX)
0.370
0
0.565
1
S=GPT-3.5
Avg. (V-score=1)
0.797
0.722
0.851
(w. Self-Debugging)
M1 (MAX) (V-score=1)
0.391
Avg.
0.582
0.007
0.428
0.645
T=GPT-3.5
M1 (MAX)
0.652
0.011
0.681
0.766
S=LLaMA3-8B
Avg. (V-score=l)
0.994
0.712
0.867
M1 (MAX) (V-score=1)
0.989
0.712
0.766
Avg.
0.701
0.133
0.591
0.853
T=GPT-3.5
M1 (MAX)
0.204
0.668
0.867
S=LLaMA3-8B
Avg. (V-score=1)
0.996
1
0.712
0.911
(w. Self-Debugging
M1 (MAX) (V-score=1)
1
1
0.867
Table A11: S-score results on Bitmasking dataset in LeetCode Grandmaster DP study plan. Here, the teacher and student are the same. This dataset is too difficult for LLaMA338B, which can hardly solve these coding problems.
Models
Metrics
PKE
OAB
SPV
SST
NWw
MSA
Avg.
0.458
0.132
0.009
0.016
0.006
0.110
T-LLaMA3-SB
MI (MAX)
0.458
0.278
0.005
0
0.007
0.110
S=LLaMA3-BB
Avg. (V-score=1)
0.628
MI (MAX) (V-score=1)
0.628
Avg.
0.463
0.135
0.005
0.039
0.006
0.109
T=LLaMA3-8P
MI (MAX)
0.908
0.135
0.006
0.045
0.108
$=LLaMA3-SB
Avg. (V-score=l)
0.642
0.472
(W. Self- suff-Debugging)
MI (MAX) (V-score=1)
0.908
0.472
Avg.
0.788
0.024
0.880
0.148
0.369
0.258
T=GPT-3.5
MI (MAX)
0.936
0
0.923
0.584
S=GPT-3.5
Avg. (V-score=l)
0.901
0.111
0.526
0.949
0.584
MI (MAX) (V-score=1)
0.936
0.111
0.923
0.584
Avg.
0.788
0.024
0.880
0.148
0.369
0.256
T=GPT-3.5
MI (MAX)
0.943
0
0.481
S=GPT-3.5
Avg. (V-score=l)
0.901
0.111
0.526 1
0.949
0.578
(w. Self- Saff-Deuggging)
MI (MAX) (V-score=1)
0.943
0.111
0.578
24

---

Table A12: S-score results on Bitmasking dataset in LeetCode Grandmaster DP study plan. Here, the teacher and student are different.
Models
Metrics
PKE
OAB
SPV
SST
NWW
MSA
Avg.
0.458
0.132
0.009
0.016
0.006
0.110
T=LLaMA3-8B
M1 (MAX)
0.551
0.014
0
0
0.162
S=GPT-3.5
Avg. (V-score=l)
0.628
M1 (MAX) (V-score=1)
0.488
Avg.
0.463
0.135
0.005
0.039
0.006
0.109
T=LLaMA3-8B
M1 (MAX)
0.537
0
0.039
0
0.130
S=GPT-3.5
Avg. (V-score=l)
0.642
0.472
(w. Self- sell-Debugging)
M1 (MAX) (V-score=1)
0.908
0.472
Avg.
0.788
0.024
0.880
0.148
0.369
0.258
T=GPT-3.5
M1 (MAX)
0.788
0.039
0.037
0.203
S=LLAMA3-8B
Avg. (V-score=l)
0.901
0.111
0.526
0.949
0.584
MI (MAX) (V-score=1)
0.901
0.111
0.053
0.949
0.584
Avg.
0.788
0.024
0.880
0.148
0.369
0.256
T=GPT-3.5
MI (MAX)
0.926
0.024
0.197
0.286
S=LLaMA3-8B
Avg. (V-score=1)
0.901
0.111
0.526
0.949
0.578
(W. Self- Debugging)
MI (MAX) (V-score=1)
0.926
0.111
0.526
0.578
Table A13: S-score results on General- ID dataset in LeetCode Grandmaster DP study plan. Here, the teacher and student the same.
Models
Metrics
L.CC
WS
4KK
L.P
RTA
HID
DW-2
Avg. MI (MAX)
0.326
0.819
0.062
0.563
0.108
0.013
0.507
T=LLaMA3-8B
0.244
0.090
0.151
0.027
0.679
S=LLaMA3-8B
Avg. (V-score=l)
0.671
MI (MAX) (V-score=1)
Avg.
0.273
0.399
0.085
0.567
0.106
0.013
0.507
T=I.aMM3-8B
MI (MAX)
0.288
0.500
0.120
0.719
0.151
0.020
0.679
S=1.aMA3-8B
Avg. (V-score=1)
0.607
(w. self-Debugging)
MI (MAX) (V-score=1)
0.719
Avg.
1
0.542
0.818
0.089
0.653
0.565
T=GPT-3.5
MI (MAX)
0.128
0.697
S=GPT-3.5
Avg. (V-score=1)
0.128
0.867
0.719
MI (MAX) (V-score=l)
0.128
0.697
Avg.
0.547
0.818
0.089
0.653
0.549
T-GPT-3.5
MI (MAX)
0.126
0.587
S=GPT-3.5
Avg. (V-score=1)
1
0.128
0.867
0.719
(w. Seff-Debugging)
M1 (MAX) (V-score=l)
0.198
0.697
25

---

Table A14:
S-score results on General- ID dataset in LeetCode Grandmaster DP study plan. Here, the teacher and student are different.
Models
Metrics
LSC
WS
4KK
LVP
RTA
HTD
DW-2
Avg. MI (MAX)
0.326
0.819
0.062
0.563
0.108
0,013
0,507
0.253
0.774
0.080
0.573
0.081
0.007
0.342
S=GPT-3.5
Avg. (V-score=1)
0.671
MI (MAX) (V-score=1)
0.719
Avg.
0.273
0.399
0.073
0.567
0.106
0.013
0,507
T=LLLMA3-8B
MI (MAX)
0.485
0.661
0.030
0.151
0.016
0.561
S=GPT-3.5
Avg. (V-score=l)
0.607
(w. Self safl-Deuugging)
MI (MAX) (V-score=1)
Avg.
0.542
0.818
0.089
0.653
0.565
T=GPT-3.5
MI (MAX)
0.520
0.058
0.013
0.697
S=1.aMA3-88
Avg. (V-score=1)
0.128
0.867
0.719
MI (MAX) (V-score=l)
0.128
0.867
0.697
Avg.
0.547
0.818
0.089
0.653
0.549
T=GPT-3.5
MI (MAX)
0.560
0.052
0.697
S=LLaMA3-8B
Avg. (V-score=1)
1
0.128
0.867
0.719
(w. Self-Debugging)
M1 (MAX) (V-score=1)
1
0.128
0.697
A.3.4 Examples
Student models can make both logical and non-logical errors. The non-logical errors - such as missing imports, miswritten variable names, improper handling of simple boundary conditions, or incorrect usage of library functions are mainly related to the knowledge required by specific EPs and the robustness of the student model, rather than the quality of the TR-TA pair. Thus, these errors do not provide helpful feedback to the teacher. To address this, we apply self-debugging to correct non-logical bugs. making the students'
V-score more reflective of the TR-TA quality and therefore more helpful to the teacher. We show an example of how seff-debugging resolves simple boundary condition bug in Ex. 6.
Regarding logical errors, our experiments show that students generally follow the TR-TA pair in making logical errors or writing inefficient logic. In such cases, the students' exam V-score provides have discussed the examples Exs. to 4 3.3.2. Nevertheless, there exist cases where the student fails to follow the logic of the TR-TA pair to solve the EPs. We find that this undesirable situation more likely to occur when the student and teacher come from different model families. An example of failure case is shown in Ex. 5. This discrepancy is highlighted by the comparisons between Tabs. 3 and A10, between Tabs. All and A12, and between Tabs. A13 and A14, where negative cases more frequent in Tabs. A10, A12 and A14 when the teacher and student belong to different model families.
valuable feedback for the teacher.
Example 1. MI on Code Synthesis (TaaossssTTTTeeSSS---See TeechorsSStdattttr73..ttttua
TP:
(Leetcnde Question solve the following problem: Alice and Bob play a game with piles of stones. There are evea number piles arranged in row.
a python code """clavs Solution(): stoneGame(self, piles: List(nn]) bool:"""
TR and
To determine the winner the game, Alice Boh,
dynamic programming appronch. Let's think step by step:
array of by where length "piles" list, arry will the maximum number of stones player can collect from piles 'I'
Create array:
Base Initialize the diagonal elements of the 2D amay with the values of the piles themselves, Alice and Bub only choose from one pile each.
Dynamic programming iteration: Iterate over the 2D array diagonally, dynamically computing the maximum number of stones player can collect piles 'I' jJ. The maximum number of stones player can collect maximum of two options:
Taking the pile at index 't' and the maximum number of stones from piles 'ill' to (Alice's turn),
26

---

b) Taking the pile al index I and the maximum number of stones from piles 'I' "j-"" (Bob's turn).
Return the winner. After the dynamic programming iteration, check the maximum number stones Alice can collect (from piles '(O, n-1] greater the maximum number stones False.
so, return True; return
1 from typing import List
class Solution:
def atoneGane piles: [ist]) >> bool:
-penee
6
dp [[[0] for in range(a))
8
Base case: diagonal elements for i in rango(n): dpti][i] pilen[i]
9
11
Dynamic programming éteration for length in r2nge n+1):
13
range(n length 1): i length dptil[j] nax(piles dp[s+1][j), piles[j] dp[i](]-11)
18
Retors the vinser return dp [a-1]
EP:
(Leelcnde Question ID: 486) Write pylhom code """class Solution(): def TrelictTheWimmersse List(int]) hool:"" solve the following problem: You given integer array 'nums", Two players are playing game with this arayy: player and player
ER and EA:
To determine if Player I can win the game, use dynamic programming appeoach. Let's Let's think step hy step:
Create 2D Create 2D of size 'n' by 'n', where 'n' is the length the "nams" list, This array will store the maximum score difference between Player and Player when considering the subaray from index 'I' to index 'J'.
2. Base case: Initialize the diagonal elements of the 20 array with the values of the numbers in the 'nums' list, as Player Player can only choose number
Dynamic programming iteration: Iterate over the array diagonally. dynamically computing the maximum score difference between Player and Player 2 when consideting the suburay from index 'I' to index 'J. The maximum difference is the maximum two options:
The number subarray from index 'i+l" to index (Player I's turn. The subarray from index 'i' to index 'jll' (Player 2's tum).
'i'
difference
Player Player when considering
maximum
Player Player considering
Return the result: Alter the dynamic programming ileration, check the maximum difference between Player and Playrr when considering the entire armay (from index to index 'n. greater than or equal to '0', If so, return False
True;
typing import
2
claes Solution:
def prediethheWinner (self, List [int]) >> bool:
5
n len(nuns)
[[0]
range (n)]
7
case: diagonal for in range(n): dp [p[i] nune [i]
9
Dynamic programming steration length range for in range (n length + 1): Length [p[1][j] (ax (j) nuns [j] (1-1)
14
4 Retues the result return dp [o] [n-1] >=
27

---

Example 2. M1 on Code Synthesis [ t
TP:
(Leefcude Question ID: 877) Write a pythom code """clas Solution(): del siomeGame piles: Lisl[int) > bool:""" solve the following problem:
Alice and Bob play game with piles of stones. There are an even number of piles arranged in cow.
TR and TA:
solve problem, can dynamic programming approach. by step:
problem solving strategy step
Initialize Variables: The code initializes variables 'n" and "mn" keep track of the total number of piles and empty cache dictionary.
2. Recursive Function: The code defines recursive function called 'aliceWins' which takes twa parametens: "start" and 'end". function represents the state game range piles currently available.
Base Case: The base case recursive function is when there are no more piles left, ie., when 'start > end'. In this True if Alice's 's total number of stomes grealer than Bob's total number stomes, otherwise False.
Memoization: Before calculating the result for dictionary. If so, return the cached result instead of recalculating
check the result already present in the cache
Maximum Stenes Calculation: We caleulate the maximum number of stones Alice obtain if she starts from the 'strrt' pile and Bob plays optimally. We have two options:
Alice takes the 'start' pile: In this
Alice's total number stones will be the "start" minus the maximum
number of stones Bob can obtain from the remaining piles. Bob's maximum number of stones can be obtained using the index
'aliecWins'' function recursively by changing the
keeping the 'end' index the
Alice takes the 'end' pile: In this case, Alice's & total numher of stones will be the 'end" pile minus the maximum oumber of stones can obtain from remaining piles. Bob's maximum number stones can be obtained using "aliceWins'' function recursively by changing the 'end" index 'end-1" 1" and keeping the 'start' index the same.
Update Cache and Return Resul:: After calculating the maximum number of Alice ohtain, update cache dictionary with the result for the current state 'start, end)' and return the result.
Main Function: The code defines the main 'stoneGame' function that calls the 'aliceWins' function with the initial state 100 n-1)" and returns the result.
1
from typing import List
2
3 class Solution():
stoneGame (self, piles: List(iat)) bool: - len(piles) (}
5
Cache dictionary
7
8
def aliceWins (start: int, and: int) -> bool: psles left
9
10
if start end: return sun (piles)
12
Check if result already cached if (start, end) in cache: return cache [(start, end)]
14
17
# Option 1: diice takes the start pile optionl piles [start] aliceWins (starr 1, end)
19
Option 2: Alice takes the end pile optien2 piles [end] eliceWins (start, end 1)
21
Update
24
cache [(start, end)] maz (optioni,n, option2) > reture cache [(start, and)]
26
Call the recursive function with inttial state (0, n-1) return aliceVing (0, n-1)
28
E.P:
(Leelcode Question ID: 486) Write pythom code """class Solution(): def predictThe Winler(sell, Wineeree,,,,,,,eeee,ll, nums: List hool tn solve following problem:
You are given integer array 'nums". Two players are playing game with this amay: player and player 2.
28

---

ER and EA: To solve this problem, we can use a dynamic programmingg approuch. Let's break down the problem solving stralegy step step:
Initialize Variables: aray and empty cache dictionary.
two
keep
elements
Recursive Function: The code delines recursive functioo called "playerl Wins' which takes parametens: "start' and 'end'. This function represents the state of the game the range of elements currenily available.
3. Base Case: The base case of the recursive function is when there are no more elements left, ie., when "start end". In case, we return
Player score greater than equal Player score, otherwise return False.
Memoization: Before calculating the result for current state. we check the result already present in the cache dictionary. If return the cached result instead of recalculating
Maximum Score Calculation: calculate the maximum Player obtain if they from the 'start' element and hoth players play optimally. We have two options:
the element plus the minimum score Player can obtain from the remaining elemenss. Player 2's minimum score be obtained using the 'playeri Wins' function recursively by changing the 'slart' index 'start+1" and keeping the "end" index the same. Player takes the 'end" element: In
Player the
element:
case, Player score
Player I's score will he the 'end" element the minimum Player can obtain from the remaining elements, Player 2's minimum score can be obtained using the 'player Wins" function rocursively by changing the index 'end-1' and keeping 'start' index the
Update Cache Return Result: After calculating the maximum score Player can obtain, we update cache dictionary with the result for the current "start, endy' and return the result.
Main Function: The code defines the main ppredie/TheWimee funetion that calls the "player/ function with the initial state (0, n-1)' and returns the result.
1 from typing import List
3 class Solution(): def predictTheWinner (self, nuns: List (int)) >> bool: lan (nuns)
6
cache - O Cache dictionary to store resstts
8
def (strrt: int, end: int) >> bool: Base case if atart end: return True
left
10
13
# Check if result is already cached (start, return cache [[start, and)]
15
17
# Option Player I lakes the start element optiont noms [start] min (playarrlWin (start 1, end) playerlKing (start, end 1))
19
Option Player option2 nums [end] sancppaayrrUuin 1, and), playeriWina (atart, end 1))
21
22
Update cache with the result and return cache [[start, end)] optioni option2 return cache [(start, end)]
25
27
# Call the recarsiue fanction uith initial state (0, n-1) retura playerlWins n-1)
Example on Code Synthesis (TeecheeeStuddeeeG.PAAAauubooo o
TP:
(Leetcude Question ID: 877) Write pythom code """class Solution(): del stumeGamerself, piles: List[in]) >> boul.""" to solve the following problem: Alice and Bob play game with piles of There number piles arranged in row.
TR and TA:
The problem asking to determine Alice can win the game given a list of piles of stones. Here's step strategy solve problem:
29

---

Determine Alice's optimal stralegy:
Since Alice starts first, she has the advaniage of making the first
Alice's goal is to end with the most stones, so she will try to maximize her score.
turn, Alice
entire pile
heginning end
Determine
nomber stones:
The total number stones is odd, which there will ties.
Alice can always lake the largest pile each turn.
Calculate the total number of stones summing the elements in the piles list.
Check Alice always
If Alice takes all the piles from either the beginning or the end in each tarn, she will always end with more stones than Bob.
Check the total munber of stones greater than half of the total
Return the result:
If Aliec can always win. return Truc. Otherwise, return False.
1 from typing import List
3 class Solution();
def atonadame (self, piles: List [int]) -> bool:
total_stones (pilea)
6
half_total total_stones //
8
if half_total < max(piles): reture True
else: rature False
E.P:
pyilhon """class Solution(): def predictThe Winner(sellf, munns: List[int]) bool."" solve the following problem: You are given an inleger array "numss. Two players are playing game with this armay: player and player 2.
Question
ER and EA:
problem asking determine Player the game given aray numbers, Here's step-by-stop stratogy to solve the problem:
Determine Player I's optimal strategy:
Player starts first wants maximize score.
each turn, Player can choose number from either end the array.
Determine
number elements:
Calculate the total number of elements in the arma..
Check if Player can always win:
the total number of elements is odd, Player can always win by choosing the numbers alternately. the total number of elements
Player can still their scure greater than equall to Player 2's
result: If Player can always win oe has higher score than Player 2, retur True. Otherwise, return False.
1 Trom typing import List
3 class Solution(): def pradictthaWinner (self, nuns: [int]) >> bool: total_elenents (numa)
6
total_elent elements return True
8
9
else:
sun (nuns) sun (nuns) //
Example on Synthesis [TeellorSSuudeeeGG//S.uubbbbbb
TP:
(Lectcode Question ID: 1406) Write python code ""*class Solution(): def stneGamellliseff, stoneValue: List[int]] >> sir: to solve the following problem:
30

---

Alice Bob continue their games with piles of stones. There are several siones urranged in row, and euch stone has an associated value which integer given in the army Value'
TR and TA:
To solve this problem. we can use dynamic programming to determine the optimul stralegy for each player.
Let's think step by slep:
Create a helper function called 'dls', which takes in the current position 'i' and list of stone values. This functioo will recursively calculate the maximum that he achieved starting from position and returinng the difference of Alice's score and Bob's score.
2. In the "dlls" function, we need a base case for the end of the game, If the current position 'I' is greater tham or equal to the length stone
list,
the game, difference
If the hase case not met, we need to calculate the maximum score difference achievable starting from the current position. can iterate through the next positions, taking difference for the opponcnt. We take the maximum score difference among the opponcnt's is choices and subtract the current current player.
stones each and recursively calculate the
stone get scure
return maximum score player), positive score difference means Alice wins. If the currett player Bob, negative score difference means Bob wins. If score difference is 0, it's tie.
calculated
previous step. current player (starting
5. Finally, in the "stonoGamelll" function, we can call the 'dss function with the initial position and the stone values list. 'di'' determine
compare
from typing import Liat
2
3 class Solution():
def stanedamelll (self, stoneValue: List [int]) >>
5
daf dfa(1: int, stonaValues: [int]] >> int: (en(onneVValess) return
7
8
9
nazScore ('ost('-inf')
for in rango min(i+3, len (st(tnneVValess) curršcore (anttonaValles [t:]+1]]
opponentScor4Ditt dfa (j 1, stoneValues)
naxšcore max(naxccore,
ogponentScoreDiitt)
14
reture maxScore
scoreDiff dfs stoneValue)
acoreDiff 0:
19
retura 'Alice'
scoreDiff 0:
21
return Bob
else: 'iie'
23
EP:
(Leelcode Question ID: 486) Write python code """class Solution(): def predictThe Winnerssef,,, nums: List[int] hool:"" t following problem:
You are given integer array 'nums". Two players playing game with this aray: player and player 2.
ER and EA:
To solve this prohlem, we can use dynamic programming to determine the optimal strategy for each player. Let's think step step:
army.. function recursively calculate the maximum score that can be achieved starting from the current indices and returning the difference Player I's score and Player 2's score.
Create helper
called 'dls', which
In the 'dfs' function, we need base case for the end the game, If the start index is greater than the end index, we's 've reached end the game, diffcrence
maximum score difference achievable starting indices. We have two options: either Player chooses the number at the start index Player chooses the number at the end index. We recursively calculate the score difference the opponent for both options and take the maximum score difference among the opponent's 's choices. We subtract the oppanentts score difference from the current player's chosen number to get the score difference for the current player.
31

---

return maximum soure dillerence calculated from the previous step. If score difference pusitive or equal to Player
Finally, pprddciTTheWine function, can call function len(nums)- 1) of the array. We compare the result of 'dfs' with to determine if Player can win.
start end
1 from typing import List
class Solution():
def predictTheWinner (self, nuns: List [int]) >> bool: dsttstart: int, int)
if start end:
return
8
9
chooseStart nuns [start] (start end) choosaEnd [and] (start,
11
return (chooseStart, chooseEnd)
13
df (0, (nuns) 1) scereDiff >-
Example on Code Synthesis (Teacher=GPP-...tttrbbo... / Student=1LaMMA3-0)
TP:
(Lecteode Question TD: 486) Write python code """class Solution(j: def predict Winner(self,, PreneccTeeinnneeeff nums: List(int) bool."" solve following problem:
You given inleger array 'nums". Two players playing game with this armyy: player and player 2.
TR and TA:
To solve this problem, we can use the concopt of dynamic programming. Let's think step by step:
1. Define a function to determine the winner: function
amay integers
inpur return
value indicating whether Player
win the
We will create 2D memoization table, 'dp', to store the scores difference between Player and Player 2 at each subgame. 'dpliilil' represents maximum difference when considering the subarray 'nums[ii=+]". The function will call a helper function caleulateScores' to calculate the maximum sccres cach subgame.
2. Define the 'caleulateScores' function: "calculateScores" function
start indices, 'start' 'end' input return maximum difference between Player and Player for the suburay "munssstarteent-]]]
We will use recursive appreach to consider possible moves that Player can make. difference for the current subarry already been calculated and stored in the memoization table 'dp'. If so, we will return the stored value. index,
Within the
function, will check if
index greater than return a score difference of
suburray
Otherwise, will consider two options:
Player chooses the first clement (' ('oumsistart]") score difference will be the value of 'noms[start] minus the
maximum score difference that Player 2 can obtain in the remaining subarmy, calculated recursively. difference
value 'numsfend]" minus maximum score difference that Player 2 can obtain in the remaining suburaay. calculated recursively. table 'dp".
Player chooses
element ('numscend"),
the maximum Finally, we will return the calculated score difference.
options
result
In the main lunction 'predictThe Winner', will initialize memeization table 'dp' with and call the helper function calluuaatScooes' to determine the maximum score difference between Player and Player 2 for the entire array
the score difference is greater tham or equal to O, means that Player can win the gamn, and we will return "True". will return 'False'.
1
from typing import List
2
3
class Solution
prediatthaVineer (self, List() -> bool:
5
n (en(n)ns)
dp [[0]
ranga(n)]
7
8
def calculateSceres (start: int, end: int) int:
9
if end: raturn
dp [start] [end]
32

---

12
return dp [start] [and]
chooseStart nums [start] calculateScoressteerr ead)
chaoseEnd zans [end] calculataScores (start, end 1)
17
dp [start] [end] max (chooseeStar,, chooseEud)
19
reture [start]] [end]
21
colculatescoree
>-
EP:
(Leetcode Question ID: 1510) Write python code """class Solution(): del winnerrSquneGGammeeell.... n: int) > boal." following problem:
Alice and Bob take turns playing game, with Alice starting first.
ER and EA:
To solve this prohlem, we can use similar apprach as before. Let's think step by step:
Define a function to determine the winner: function should
value indicating We will create set 'squares' store all the square numbers less than equal to 'n'. helper function
integer as imput return
game.
function
to determine Alice can win game.
Define the 'canWin' function:
'canWin' function will
number
input
beolean value indicating whether
Alice can win the game.
recursive appreach consider possible moves Within the recursive function, we will check if the current number of stones less than or equal O. If so, it means that
Alice lost the game, and we return False'.
'squares' that less than equal For each square number, we will recursively call the 'canWin' fanction with the updated number stones. "True', it mone of the recursive calls return 'True: it means that Alice cannot win the game, and we will return "False".
Otherwise, will consider all square numbers
any
calls
Alice the game,
return 'Tree".
3. In the main function "uinnerSquarGGmme" will create the 'squares' and call the helper function 'can Win' to determine if Alice can win the game. the result 'Tuee, will 'Trae. Otherwise, we will return Falsee.
1 class Solution(): oinner8quaraCama (self, int)
3
squares aet (1 for in range (int (math. egrt (n)) + 1)) nemo
6
def canWin int) -> boal:
8
return False
n meno: retura neno [n] for square in squares:
if square
break canVin square):
15
nane [n] True
return True
neno [n] False
return False
20
raturn canVin
Example 6. Code Synthesis with Self-Debuggin (Debugger
Problem:
(Lectcodee Question 1406) Write python code """class Solution(: def stonGGamelTiseefi stoneValue: List[int] str:" to solve the following problem:
Alice continue their games with piles
stones arranged row,
associated value which is an integer given in the army StoneVallue'
33

---

Rationale and Answer:
To solve this problem, we can use dynamic programmingg Let's think siep by slep:
1. Initialize the scores aray:: We initialize urray "scores" with a lengih of "len(siomeValue) 1:. The "scures" array will score
store
obtained from starting index.
lierate from the end: iterate backwards through 'stone Valuc" array, starting from last element, each index calculate maximum that be obtained from that index by considering all possible taking
'I', '2, or stones.
3. Detemnine the maximum score: For each possible move, we consider the sum of the stones taken and subtract the maximum score maximum of these values the maximum score from index 'i' and store in the 'scorcs'
obtained
('scoresfie]] 'scoresfi+]]. and "scoresfi+3]1
Determine the winner: Afler calculating the maxinum for all indices, compare "scures[O]" with the sun of all the stome values ('sum(stone Valnej") determine the winner. "scores(O]" is greater, Alice wins. If "scores[0] is smaller, Bob wins. Otherwise, it's tie.
Return the result: We return the appropriste string winner determination.
1
from typing import List
2
3 class Solution():
StoneGaneIII (se1f, stoneValue: [int])
5
n - len(stoneValue)
[0]
7
8
for in range(n-1, -1, -1):
9
takel stoneValue[1] scores take? [tllsttonaaalell:1111
eun(stoneValue (1:1+3]) acorea [i+2]
12
scores(i] - max(takel, take2, take3)
14
if acorea [o] sun sun(stonaValue) acores [0]
return "Alice"
elif [0] aum (sttonaVaale)) scores [0]
17
retura "Bob"
19
return "Tic"
Debug Output:
Based on the provided rationale, here the fimal bug free code:
1 from typing import List
3 class Solution():
stoneCamalll (self, stoneValue: [int]] >> str:
5
(en
6
scores [0] (n
7
8
for i in range(n-1, -1, -1): [toneValue take? [an[stooaValao scores [1+2] if 1+2 <- alse float("-
10
11
take3 eun(stoneValue (1:1+3]) scores [1+3] If 1+3 <- elae
float(?-
scores[1] max(el, take2, take3)
13
scores [0] (sttoneeVal)e) scores [0]:
15
return "Alice"
slif scores [0] aum (stoneValue) scores
17
return "obb
else:
This cnde fixes the bug where the index was going out range when calculating 'take2' and 'take3' by adding conditions chesk '+22' and 'it3' within range 'stoneValue" list. If they the value " ensure that it is not considered in the maximum calculation.
A.3.5 Natural Language Rationale has Positive Effect on Code Following in ICL.
In the code synthesis task, the answer (TA and EA) refers to the resulting code. We find that having models produce natural language rationale (TR and ER) before writing the code is crucial for
34

---

ICL and ultimately the LbT-based scoring to function expected. Specifically, this approach allows the student's is output code EA align more closely with the ICL example code TA, thus the exam score (i.e., LbTT score) can better reflect the quality of TA.
As shown in Ex. 8, when only TP+TA the ICL example teach the student, the HA does not follow the DP coding style in TA and instead employs recursive method to solve EP. This aligns with the finding in [50] that labels in the ICL examples have limited effect on the LLM output. In contrast, as shown in Ex. 9, when we use TP+TR+TA as the ICL example, the student first follows the TR create ER, which subsequently leads to the successful generation of EA that aligns with the coding style of TA.
Example Code Synthesis without In-context example (without TP, TR, and TA)
EP:
(Lectcode Question ID: 877) Write a python code """clas Solation(); def stomeGimmetself, piles: List[int)) -> bool!""" to solve the following problem:
Alice and Bob play game with piles of stones. There are even number of piles arranged in row.
EA:
typing import
2
clasa Solution:
4
def (ronedame(self. piles: [ist]) bool: n = len(piles)
5
prefix_ [0] (n for in ( prefix_sun(i
7
profix_sun(t] piles[i]
9
def dp 1):
if >>
return
naz mpxlee
prefix_sun(j] prefix_eun [i] dp
1, 1).
piles [j] prefix_sun [i] prefix_sun dp(t, 1))
retura (1, n) V8
Example 8, Code Synthesis without TR
TP:
(Leeteode Question ID: 486) a python code """class Solution(): def predict predistTheiinnersc.. Winner(self, nums: List(iat)) >> bool."" solve the following problem:
You are given an integer array 'nums". Two players are playing game with this army: player and player 2.
TA:
typing import List
2
4
def predietTheWineer (self, nuns: List[int]) >> bool:
n len(nuns)
áp1 [0]
[0]
7
dpi [0], dp2 [0] nuns [o],
8
range(1, (1, n):
9
dpi[i] max (dp1 [1-1] nuns [1], dp2 [1]]) dp2[t] max (dpt [1-1], ap2[1-11 [i]) retura dp1 [-1] 4p2[-1]
11
EP:
(Leetcode Question ID: 877) Write a python code """clas Solution(): def stoneGeme/self, piles: Listlint) -> bool!"" to solve following prublem: Alice and Boh play game with piles of stones. There even number of piles arranged in row.
EA:
35

---

from typing import List
class Solution:
def stonedame (self, piles: List (int]) bool: lan(pilas)
6
prefix_ eun [0] (n 1)
range (m);
8
protix_oun(l 1l protix_aun(1] piles[i]
9
def dp 3):
if 1: reture maz (p [j + 1] prefix_eun(1) dp 1, 1)
13
14
prefix_sunl]
pretix_eun(1] 4p 1))
dp 1)
Example Code Synthesis with
TP:
(Lecteode Question ID: 486) Write python code """class Solution(): def predict Winner(selff, Winneeth nums: List[int)) +>> hool:""n solve the following problem:
You integer array 'nums', Two players playing game with this armyy: and player
TR and TA:
The problem can be solved by using dynamic programming approach. The idea is to muintain two arrays, 'dpl' and 'dp2', where [ip[ considering the first if elements in the
'dp[[i]' represent the maximum score player and player can get respectively after
Let's think step by step:
Initialize 'upl' and 'dp2' with zeros. 'dpp [0]' and 'dp2[0]"
"mums[O]] because the first element the starting
point for both players.
For each element 'oums[i]] in the aray, calculate 'dp[[i'' maximum options: [dp numsfil': Player chooses 'nums[i]" and leaves 'numsfi-1j" IJ' for player 2. 'dp2(i-1]': Player chooses 'numsfi- IJ and leaves "noms[]]" player
Calculate dp2[ii' the maximum of two options: 'dpl[i-1]': Player chooses 'nums[i-1]" and leaves 'nomsfil' for player 1. 1dp2ii11 + nums/ill: Player chooses 'numssjil" and leaves "numssj- 1|" for player 1.
If "pp[[1] dp2[-1]', return 'Tree' hecause player can Otherwise, return 'Falee.
1 from typing import Liat
3 class Solution:
predicttheWinner (self, nuns: (int]) bool: lan (nuns)
5
6
dp1 dp2 [0] m, [0]
7
dpi [0] 4p2 [0] nuns [0],
8
for in range (1, n):
dpi[i]
nuna dp2 (i-1]) dp2[i] max (dp1 [1-1) dp2[i-1] nums [i])
(dp1
10
return [-1] dp2[-1]
EP:
(Leetcodee Question ID: 877) Write python code """class Solution(): def stoneGamesself, piles: Listfint) bool:""" solve the following problem: Alice Bob play game piles of There number piles arranged row.
ER and EA:
The problem can be solved by using dynamic programming approach. The idea is to maintain two arryys, 'dpl' and 'dp". where 'dpllil" and 'dplii'' represent maximum number stones that Alice and Bob can get respectively after considering the first 1' piles.
Let's think stop by stop:
Initialize 'dp1"
"p2"" with seros, 'dpl[00" 'dp2[0]' are "piles[0]"
pile starting point
for both players.
each pile "piles[i]'
array, caleulate [i]
maximum
options: 'dpl[i-1] + pilesfil::
chooses "pilesfil" and leaves piles[i-1]] I' for Bob. 'dp2[i-1]"' II: Alies chooses ppics[[-1] and leaves 'piles(il' for Bob. Calculate 'dp2lil' the maximum of two options "dplli111: Bob cheoses "pilesti-11" and leaves 'pilestil' for Alice.
"dp2[j-11 pilesfi]: Bob chooses 'pilesfi'' and leaves "piles[i- 1J' for Alice.
4. If "dpl1-1>>> (pp)--1]',ad return "True' because Alice wins. Otherwise, return "False".
typing Amport
36

---

2
3
class Solution:
4
def stoneGame (self, piles: List (int]) bool: n len(piles)
5
6
dpt, 4p2 [o] n, [o]
7
dpi [o], dp2 piles [0],
range (1, a):
9
dpi[i] max (dpi [i-1] piles[il, dp2(i-1)) dp2[1] max (dp1 [1-1), dp2[i-1] piles(i))
return dpi [-1] 4p2[-1]
In the following, we quantitatively assess the following rate of the ICL example and the resulting positive correlation between TA accuracy and the EA accuracy (i.e., LbT score).
ICL Example Following Rate (Coding Style). To quantitatively assess the positive effect of the natural language rationale on ICL following, we define the Style Follow Rate, which measures whether EA matches the coding style of TA. Specifically, if the two codes are of the same algorithmic style and raise the same type of exception when an error occurs, we regard them as of the same style. We conduct the judgment manually. For example, if the teacher employs dynamic programming as the problem-solving strategy while the student resorts to a recursive strategy, we classify this as not following. Tab. A15 shows the Style Follow Rate on the Game Theory dataset with LLaMA3-8B. Using both TR and TA as an ICL example yields a 12.5% % higher follow rate than using TA alone, illustrating the importance of natural language rationale for enhancing ICL following.
ICL Example Ignore Rate (Coding Style). Moreover, we observe that when using only the TP+TA as the ICL example, without including TR. LLMs often ignore the ICL example and produce responses that are nearly identical or exactly the same those generated without any ICL example, as shown Ex. 8 and Ex. 7. To quantitatively assess this tendency to ignore ICL. example in the absence of natural language rationale, we define the ICI. Ignore Rate, which measures whether EA'S style matches that the code generated by the student without any ICL example. As shown in Tab. A15, when TR is not used, the ICL Ignore Rate is as high as 43.13% but using TR reduces this rate significantly to only 1.88%. This further illustrates that using natural language rationale in the ICL example helps the student more effectively learn from the teacher's code, rather than ignoring
Ranking Correlation between TA Accuracy and EPs Accuracy. We compute Kendall's 's Tau ranking correlation between the TA's S-score and the average S-score of EAs, as well as between the TA's 's S-score and the average V-score of EAs (i.e., the LbT score). As shown Tab. A A16, we can see that the correlation of "Teach w/ TR+TA" is consistently higher than the corresponding correlation of "Teach w/ TA". When teaching without TR, the correlation between the TA Ass S-score and EAs' S-score or V-score is zero even negative.
Selected TA Ranking with the Highest EPs Accuracy. We show the S-score ranking of the TA among TAs with the highest average EA score. We observe that: (1) When TPs and EPs are similar and TR is provided in the ICL example, we can select high-accuracy TAs based on the S-score V-score of EAs. For example, for questions SG1 and SG4, the TA with the highest EAs' V-score achieves the best S-score among the 8 TA. (2) When TR is not provided in the ICL example, the V-scores of EAs with different TAs as the ICL example are usually not discriminative, i.e., are of exactly the same value. Therefore, when TRs is not provided in the ICL example, using EAs" V-score (i.e., LbT score) cannot help select a high-accuracy TA.
Table A15: The Style Follow Rate and ICL Ignore Rate on the Game Theory dataset LeetCode Grandmaster DP study plan with LLaMA3-8B. The Game Theory dataset contains problems, and pairs of TR+TA are sampled for each problem. Then, we used the 5 x = 40 pairs of TP+TR+TA or TP+TA only to teach students to solve the remaining problems other than TP, resulting in total of 160 EAs.
5 x 8 x (5
Method
Style Follow Rate (t)
ICL Ignore Rate (L)
Teach w/ TR+TA
81.25%
1.88%
Teach w/ TA
68.75%
43.13%
37

---

Table A16: The Kendall's Tau ranking correlation between the TA's S-score and the average V-score of EAs (i.e., LbT score), and between the TA's S-score and the average S-score of EAs, and the S-score ranking of the 'AA among 8 TAs (E (1,.... 8), the best) with the highest average EA score (either LbT score, or the TA's s S-score). All EPS come from the Game Theory dataset. For the "Similar TP and EPs" section, we select TPP from the Game Theory dataset to ensure high similarity between TPs and EPs. For the "Dissimilar TP and EPs" section, we select TP from the Tricky Invariant dataset related to binary search. Details the teacher IDs can be found Tab, A8. For each TP, sampled 8 pairs of TR+TA the ICL examples.
Similar TP and EP
Dissimilar TP and EP
TP ID
PW
SGI
SG3
SG4
KMPN
HI2
Kendall's x Tau with EAs V-Score
0.186
0.524
0.000
0.453
0.171
0.000
Teach w/ TR+TA
Kendall's The with EAs' S-Score
ranking max
0.074
0.725
0.000
-0074
0.356
-0.371
TA ranking with max EAs S-score
3
Kendall's Tau with EAs V-Score
0.645
0,000
-0,243
0.000
0.000
0.000
-0.247
Teach w/ TA
TA manking with max EAS V-score
5,7.8
1.2,3,A,,666..88
2,3,5,7,88
1,2.34,5,667.888
1,2,3,4,,,6,,,.88
1,2,3,4,5.6....
TA ranking with EAs' S-score
B
M2
B.1
Additional Experimental Setups
We use a learning rate 5e-7, a batch size and training epoch. We set B = 0.1 and add an additional NLL term [55] weighted by 50.
B.2
Examples
Example 10. A DPO Pair of Correct Answers
TP and TP-GT:
Compute (-19)
When is monzero, we have (rr)
(-49)
Winning TR-TA:
Let's think step by step.
The division operation is the inverse operation multiplication. so we need to lind the number that multiplied by gives This number
Therefore, (-49)
Losing TR-TA:
Let's think stop by step.
compute division, can compute quotient
by performing repeated
42
35
28
21
21 7 14
14 - 7 =
7 - 7 =
At this point, the remainder 0, so we are done. The quotient is the numher of times we suhtracted 7, which 7. Therefore, (-49) +
Example A DPO Pair of Wrong Answers
TP and TP-GT:
positive integer X more than multiple of 3. units digit the same as the units digit nomber that more
than multiple of 5. What is smallest possible value of X
38

---

a posilive integer is 4 more than a mulliple of them dis units digit musi or y. We check positive inlegers ending in until find one which more than multiple of 4 than multiple multiple of 3, and 14 more than a multiple of
Winning TR-TA:
Let's think step by step.
We know that X
than multiple
integer
The units digit of X is the same as the unils digit of number that is 4 more than multiple of S. 'The numbers that are more multiple we have that X has units digit
4, 9, 14, 19...,
digit of such number cither
has units digit
Therefore, we
4, or smallest possible
Losing TR-TA:
Let's think step by step.
units digit of number that than multiple 5, Since the units digit multiple of either or 5, the units digit of is either or I.
The units digit of X is the
Also, X the units digit of X is 6, then X must be or 11 or.. and so on. However, X more than multiple of 3, then X digit
than multiple of 3, either
multiple 3,
multiple of 3.
cannot
or.. not
digit number
than multiple of 5.
Therefore, the digit of must I. case, X must
oc.. on. Since is 2 more than
multiple of 3, the smallest pussible value of X is
39

---

C
M3
C.1 Workflow
Algorithm A2 The Workflow of M3
Require:
Teacher and (one multiple) Student LLM: T, S;
Task description lask, and instances of the task TPs
Exam Problems: EPs
Ground-truth of Exam Problems: EPs_GT
Number of iterations: T
1: # Generate initial exemplars in a zero-shot manner.
2: exemplars T (task))
3: # Iteratively search for better exemplars.
4: to T do
5: failures =
6: # Find student failures when using current exemplars.
7: for EP, EP_GT in zip(EPs, EPs_GT) do
8:
for in S do
9:
EA = (exemplars,, EP)
10:
if EA + EP_GT then
11:
failures .ppuueee.ppeed((E...... EA, EP_ (P_GT))
12:
end
13: end for
14:
end for
15:
16:
# Reflect on student failures
reflections T (exeeelasee, failures)
18:
# Generate new exemplars based on reflections
19:
for to N do
20:
exemplars; = T (expxpaa,,, failures, reflections)
Evaluate new exemplars
22:
score; = EVALUATE(EPS_ GT, T(exemplars, EPs))
23: end for
24: # Update exemplars
best = ArgMax(score;,
26: exemplars = exemplarS@est
27:
end for
28: # Teacher uses the optimized exemplar to solve Teaching Problem.
return T(exemplars, TPs)
C.2 Prompt Design
Prompt 6. Teacher Prompt (init): Generate Initial Exemplars
[User:]
I'm trying to write (k) in-context learning examples for fow-shot classifier. The classifier will answer this question:
"[task]"
information,
positive negative learning examples.
[Assistant:]
The list of [k) positive and negative leamingg examples ure: [task]
Prompt 7. Teacher Prompt (reflect); Reflect Student Failures
[Uer:]
40

---

I'm trying to write [R] in-context learning examples for few-shot classifier. The classifier will answer this question: "[task]"
examples, classifier got following cases wrong: (failure, cases]
Give faam foodbacks] reasons why these examples could have gotien these cases wrong.
[Assistant:]
Prompt 8. Teacher Prompt (improve): Improve Exemplars Based on Reflections
[User:
I'm trying write (A) in-context learning examples for fow-shot classifier. The classifier will answer this question:
"[task]"
But with these examples, the classifier got the following cases wrong: [fallure_cases]
Based on these failure cases, the problem with the current in-context examples is that [reflection].
Based on the above information, I need new list of (At improved positive and negative learning examples.
[Assistant:]
C.3
Examples
Table A17: Teacher's Fi score of M3 Logical Fallacy
the end of iteration T, where LLaMa3-70B is used as the teacher for all settings. The best results are in bold.
Student(s)
T==2
T=
T=
7=5
LLaMa3-70B
81.39+0.34
81.81=0.32
10200.2
82.00+0.26
82.34:0.22
LLaMa3-8B LLaMa3-(70.8JB Mistral-7B
82.44+0.26
82.49+0.27
82.62+0.24
82.72+0.23
82.72+0.22
82.38+0.28
82.48+0.27
82.59=0.26
82.73+0.24
82,29:0.22
Ex. 12 provides an instance of how the teacher makes verbalized reflections on why the current in-context learning exemplars are causing students to make mistakes, and how the reflections help guide the teacher revise the exemplars to address the mistakes.
Example An Example Round Exemplar Revision
Problem:
this text an instance of the logical fallacy of faulty gencralization?
Example 1: My father told me that the sky is green. So must be the truth. [No]
fallacy broad statement about people social, professional, religious group? [ess] Example 3: But, Dad, know you had no curfew when you were my age, so how can you give me a curfew? [No] Example Some tall people recently vandalized the park, tall people irresponsible. [Yes] (some examples omitted)
Example
gender, ethnicity, race, or political,
Student Failures:
Failure
Imput: We haven't proven aliens didn't create life on earth, so aliems created life on earth.
Label: No
Prediction: Yes
(some cases omilted)
Reflection:
The classifier might having trouble with texts that involve negation indirect reasoning. Failure Case text presents classic example an argument from ignorance, the classifier might have been thrown off by the double negation ("baven't did'en...didn' create") indisect way stating conclusion.
Revised Exemplars:
Example 3: I've nover met person from France who docan't 't love cheese, so the French must have genetic prodisposition loving [Yes]
Example 7: The fact that somt people claim to have seen ghosts docsn' 't prove that ghosts exist. [No]
(some examples omitted)
41

---

Table A18: Causes of errors identified by the teacher (LLaMa3-70B) in M3, and analysis of whether they also caused teacher mistakes and are mitigated by LbT.
teacher mistakes the same cause
% reduced
Student
Cause of student mistakes (identified by teacher)
by LbT 6.0%
(Ia) Lack of examples within the context of multiple
45.2%
LLaMa3-8B
speakers or dialogue;
(lb) Insufficient context for understanding the argument; 37.1%
11.6%
(1c) Difficulty in handling nuances of everyday language 44.6%
13.3%
and humor;
(2a) Misled by the presence of emotional appeals and 66.22%
0.0%
Mistral-7B
excuses in the text:
(2h) Treating a binary or absolute statement as faulty gen-
67.2%
6.5%
eralization;
(2c) Fail to handle cases involving implicit or indirect 42.5%
2.3%
relationships between claims and evidence;
(3a) Lack of examples of anecdotal evidence personal
38.2%
4.5%
I.LaMa-770B
experiences;
(3h) Linguistic structures such conditional statements;
83.3%
0.0%
(3c) Biased towards examples with more complex lan- 92.4%
24.1%
guage or multiple sentences;
We further analyze the types and proportions of errors made by the students (Tab. A18), and verify that having diverse students help discover a diverse set of errors that the teacher could make, and that our LbT pipeline M3 could indeed generate exemplars that help correct those errors. More specifically, during round of exemplar revision for the Logical Fallacy task, we identify the main error types (numbered a,b,c) from teacher's reflections of mistakes of different students (numbered 1,2,3).
We observe that the teacher identifies diverse and complementary causes of errors from different students, which help interpret why having diverse students is better. We verify that the causes of students' mistakes (la, lb, ...) are indeed also causing teachers' mistakes. Specifically, for each mistake the teacher makes the test set, prompt LLaMa-3-700 to judge which cause categories (Ia, Ib.... does this mistake fall into. Note that one mistake can be caused by multiple causes simultancously. Then, report the percentage of teacher mistakes of that cause in the third column of Tab. A18. By choosing a student model different from the teacher model, we identify more types of valid causes of teacher mistakes.
Finally, these causes in Tab. A18 indeed help the teacher improve the ICL examples. After the teacher revises the ICL examples by learning from student 1, the teacher's mistakes caused by 1a, 1b & Ic are reduced by 6.0%, 1 11.6%, and 13.3%. Based on this, we conjecture that LbT methods like M3 could lead to more interpretable way to understand the flaws of in-context learning models.
D Discussion on Research Rationale
This section discusses our research rationale, and how it leads to specific choices made in our work, including the ultimate target, the LbT idea, the chosen tasks, and the concrete LbT implementation. We have already covered the roadmap - the next steps for further developing the LbT idea - in $ 6, so it is omitted here. Finally, we discuss related machine learning techniques in App. D.5.
D.1 Target (the "nail" in the high level)
In recent years, we have been impressed by the emergence of powerful AI models with extensive knowledge, strong planning skills, and good intuitions. However, the ability of current LLMs to provide accurate knowledge and reasoning appears to lag behind these other abilities. We consider accurate knowledge and reasoning to be among one of the most crucial factors for advancing LLM capabilities.
42

---

D.2 Idea (the "hammer" in the high level)
LLMs demonstrate an impressive ability in grasping solution formats and strategies. However, they often struggle with accurately grasping the underlying logic, differentiating between concepts, and consistently adhering to correct logical reasoning. Actually, these errors mirror some difficulties encountered in human learning. Therefore, we speculate that integrating human educational strategies may help pave the way toward logically accurate Al.
The LbT methodology has been shown to effectively promote human learning, especially in terms of accurate knowledge building and reasoning. Drawing on literature that demystifies the working mechanism of LbT in human learning [5, 18, 22, 33, 62-65] and based on our own teaching and learning experience, we summarize three major ways in which LbT improves human learning:
(a) Increased self acfiuuttatbbbi The task of teaching introduces social pressure and incentives, encouraging individuals to raise their standards and work harder.
(b) Explicit articulation of implicit and vague thoughts: During the preparation of teaching materials, the teacher needs to use clear and organized language to convey its inner thoughts to ensure the student's 's comprehension. This process requires translating implicit concepts into precise terms, describing subtle distinctions, organizing information logically, and "slow" verbalization and organization not only aids communication, but also helps the teacher identify gaps in its own understanding and discover new connections. As for the teaching material themselves, intuitively. teaching materials that make it easier for students to learn have clearer and more accurate logic (we refer to this the "LbT assumption on teaching material quality" or the LbT-TMQ assumption). Especially when the students are unable to solve set of problems independently. if teaching material enables them solve the problems more effectively than teaching material 2, it is reasonable to infer that teaching material may have better logic.
This
We expect MI and M2 to benefit the teacher in a similar way.
(c) Iterative feedback from diverse students: the teaching process, interaction with students of varying ability levels and knowledge backgrounds offers valuable feedback. The teacher can check if or not students misunderstand the teaching material or struggle with certain problems, and analyze why, in a multi-round discussion. In this process, the teacher might (1) recognize gaps in the teaching material - some conditions and logic may be straightforward to the teacher but require further elaboration and supporting information for students of different backgrounds, to ensure they actually understand and are able to utilize this logic in solving new problems; (2) identify gaps in their own knowledge; (3) discover novel connections when addressing students' misconceptions and erroneous associations.
We expect M3 to benefit the teacher in similar way.
We are interested in whether LbT-inspired methods could become standard prompting or fine-tuning practice to enhance the reasoning abilities of AI models. As first step, this work investigates whether simple implementations of LbT, focusing on its benefits (b) and (c), can improve the reasoning abilities of contemporary LLMs. For future work, implanting incentives into the LLM learning process to mimic benefit (a) could be a promising pathway, such as setting up a collaborative multi-agent learning framework with proper rewards and communication restrictions.
Additionally, LbT has the potential to improve stronger models by having them teach weaker ones (i.e., weak-to-strongg generalization). This might offer exciting opportunities for continuous model evolution, especially as scaling the size of high-quality data faces challenges.
D.3 Task (the "nail" in the low level)
Tasks. In this work, choose mathematical reasoning, competition-level code synthesis, and verbal logical reasoning, because compared to other task domains, these tasks require accurate knowledge and reasoning and cannot be easily solved through vague logic simple memorization. Moreover, all three tasks are popular and have attracted considerable attention from the community.
Datasets. For mathematical reasoning, we choose the challenging MATH dataset [27]. First, we experiment with using the all-mpnet-base-i sentence embedding model [60] to calculate the embeddings for all problems. Then, to score the rationale and answer for given problem,
43

---

select the 2 closest problems based on embedding proximity as the EPs to calculate the LbT score. The results, presented in 3.3 (right), demonstrate that LbT-based scoring provides substantial relative improvement over SC for problems with close problems. For example, those problems within the smallest 5% cosine distance from their closest problems achieve a 24% relative improvement over SC. However, for problems lacking similar counterparts in the training set, LbT-based scoring may even have a negative effect, which is expected, as the student's 's score on irrelevant EPs cannot reliably reflect the quality of TRs. MATHO [72] is an extension of the MATH dataset, which offers 3 functional variants for each problem. These variants share a similar solution logic, making the dataset suitable for verifying the LbT-TMQ assumption. Therefore, we choose to conduct the main mathematical reasoning experiments of M1 on the MATH() dataset.
For code synthesis, we select competition-leve problems that require heavy reasoning. Instead of using popular datasets like HumanEval [9] or MBPP [2], which contain mostly basic Python programming problems, we extract more challenging competition-lewe problems directly from the LeetCode platform. To verify the LBT-TMQ assumption, it is important for the EPs and TPs to be similar. Fortunately, LeetCode offers curated "study plans" (e.g., programming skills, dynamic programming, etc.), each comprising several datasets with problems that exhibit similar underlying structures and solution strategies, making them suitable for our evaluation.
In M3, we select the verbal logical reasoning task focused on analyzing and identifying fallacious arguments. In line with our criteria of task selection, determining fallacious arguments also requires accurate reasoning, while the reasoning here takes a verbal natural- Languaae form, complementing the symbolic reasoning in mathematical reasoning and code synthesis. Specifically, we choose Logic [34], a dedicated dataset for logical fallacy detection. In addition, we use Liar [78], a misinformation detection dataset. In misinformation detection, many claims can be factually correct but logically fallacious and it requires LLM's logical reasoning ability on top of factual memorization. For example, the claim "if we allow students to use calculators in math exams, soon they won't 't be able to perform basic arithmetic without them" is based on the fact that calculators are allowed in some exams, but it is misin normatornttion as it has the logical fallacy of "slippery slope": allowing calculators doesn't 't necessarily mean students will lose all arithmetic skills, and there is no evidence suggesting such a causal relationship. Logic and Liar are multi-class classification datasets and we cast them as eeestt binary classification problems following the practice of previous work on prompt optimization [57]. For Logic, we choose the largest class "faulty generalization" as positive and the rest as negative. For Liar, we categorize the class "true" as positive and the rest as negative.
D.4 Implementation (the "hammer" in the low level)
As preliminary investigation into the potential of LbT in improving AI, we aim for simple imple- mentations and easily controllable experiments to clearly understand the effects of LbT.
M1 and M2: We implement the LbT idea into a single component in well-established pipelines. Specifically, we design an LbT-based scoring component, leveraging the LbT-TMQ assumption "teaching materials that make it casier for students to learn have clearer and more accurate logic". This allows for simple ablation experiments to isolate and understand the benefits derived from this particular LbT-TMQ assumption.
Baselines: Existing studies have proposed various rationale scoring methods based on GT answer matching [89, 90]. answer agreement [31, 79], generation likelihood [79, 84], and self-evaluation [74, 84, 87, 88]. GT answer matching and answer agreement- baseddt--addddaeeeee methods assume that a more correct or consensus answer indicates a better logic in the corresponding rationale. However, unclear wrong rationale can still lead to correct answer by chance, and a high-quality rationale may result in an incorrect answer due to a simple computational error. In such cases, Lbb--baed scoring has the potential to better reflect the reasoning quality of the rationale compared to using the correctness and consensus of the final answer.
Regarding generation likelihood, as [79] found that using the likelihood as the score for weighted self-consistency even degrades the performance, we did not experiment with this method. For self-evaluation, we provide a comparison in Tab. A7, which demonstrates clear improvements with LbT-based scoring over self-evaluation methods.
M3: In addition to implementing the LbT-TMQ assumption as a "static" scoring mechanism for teaching materials, M3 takes step forward towards exploring the broader potential of LbT by implementing an "active" and "iterative" feedback process.
44

---

D.5 Relation to other machine learning techniques
Knowledge distillation. In traditional task domains, knowledge distillation [23] is a well-known and widely adopted technique. Most knowledge distillation methods aim at improving the student model by transferring knowledge from a fixed stronger teacher. As a special branch in knowledge distillation, mutual learning [92] allows multiple models to learn from each other in an alternating manner. In the realm of LLMs, researchers have explored knowledge distillation [24, 44, 75] and distillation via synthetic data [1. 28, 41] techniques transfer knowledge from teacher LLMs to student LLMs by letting teacher models teach student models through token logits, features, and synthetic data [85]. All of these work follows the "learning from teachers (LfT)" instead of our "learning by teaching (LbT)" paradigm.
Meta learning. Our work also relates to meta-learning [30], broadly defined term encompassing various methods within the "hetter learning by learning to learn" paradigm. These approaches typically aim to optimize different aspects of the learning process, such as initial weights [21], training hyperparameters (e.g.. learning rate, regularization coefficient) [20, 70], neural architectures [19], data augmentations [15], and so on. Many approaches formulate meta-learning as a bilevel optimization problem, where the outer optimization targets the configuration of interest, and the inner optimization the learning process itself. In the era of increasingly intelligent LLMs, our work introduces novel, human cognition-inspired paradigm - "better learning by learning to teach" (LbT)- - to continuously evolve the LLMs, which uses the "task" of teaching (weaker) models to challenge the LLM. Initially, we considered straightforward implementation involving explicit bilevel optimization, where the inner optimization the student's learning process, and the outer optimization can be conducted with reinforcement learning techniques. However, by leveraging the in-context learning abilities of LLMss we were able to explore the LbT paradigm in a simpler and controllable manner. For example, we optimize the ICI. examples for the teacher itself by observing students' results in M3.
Machine teaching. Another related area is machine teaching [7, 97, 98], which is defined as an inverse problem of machine learning: finding the optimal teaching examples if the teacher already knows the parameters, and aims make the student learn the target concept with a minimal set of labeled examples. In contrast, our work focuses on enhancing the performance of the reacher, as measured by the teacher's 's accuracy on a test set. In addition, while machine teaching literature typically focuses on small-scale tasks using analytical models, our focus is on the reasoning abilities of -fate-of-the-art LLMs.
Other methods related to the LbT idea. It is worth noting that an independent work [32] designs regularization loss to improve model generalization. This regularization loss can help filter out extraneous details that are difficult for auxiliary students to imitate. This approach is based on the assumption that "generalizable correlations should be easy to imitate", which shares conceptual similarity with our LbT-TMQ assumption. Another recent work, Math-Shepherd [77], proposes an automatic scoring method for partial rationales within the snneraatin-scoriigg scoring-finetuning pipeline. evaluates each partial rationale by measuring how often another "completer" model arrives at the correct answer by continuing from the partial rationale. Although this paper does not frame the method from an LbT perspective, the method can be interpreted as LbT-based scoring, based on the assumption that "partial rationales which make it easier for students to continue and reach the correct answer have clearer and more accurate logic". In Math-Shepherd, the students (i.e., the completer) are examined by extending the partial teaching rationale for the same problem, whereas in our M2, students are examined on similar problems, using the full rationale from the teaching problem as an exemplar. We hope that framing this method together with our methods within the LbT framework could help enlighten a broader roadmap for drawing insights from human education.
E
More Extension Possibilities
Based on the framework presented in 1 and 2, we can explore other possibilities to incorporate LbT insights into the inference and training of LLMs. For example, to improve answer quality for a TP, we can incorporate the idea of having the teacher reflect on multiple students' feedback into the search-based output generation pipeline, as follows: for each student, the teacher iteratively reflects on the student's 's feedback and updates its TR. The final answer is then derived by aggregating the updated TR for each student. Intuitively, different students might require distinct teaching materials, so generating TR for multiple students can encourage TR diversity. Increasing diversity has been
45

---

shown to benefit the accuracy of agreement-based aggregation methods [79]. Compared to increasing the sampling temperature of TR, this LbT-based sampling is guided towards high-quality TRs and might yield better results. Besides, LbT can potentially be extended to open-ended problems, such as dialogue, writing, and open-ended math problems [86]. A natural extension could involve using teacher LLM to evaluate student's 's answer [10] and provide the LbT score.
F Potential Risks of Bias Perpetuation
In open-domain problems where no GT judgment exists and LLM-based judgment is needed, is possible that teaching materials deemed "well accepted" by students or closer to the truth. Instead, they may align with the existing biases of teachers or students, posing risk of the teacher perpetuating their own biases [68] indirectly learning the students' biases.
necessarily more accurate
Furthermore, while this work primarily focuses on utilizing LbT idea to improve reasoning in mathematical and code domains, considering the potential bias perpetuation effect becomes even more critical in domains where societal bias and fairness are of paramount concern [76].
46

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the necessary design that makes the method work as expected and the cases it doesn't t achieve improvements compared to baselines. We also discuss possibilities to address the limitations $ 6.3.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
47

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide clear description and pseudo code of the methods, as well as all the experimental settings.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
48

---

Answer: [Yes]
Justification:
We have open sourced all the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips) cc/
httpe//guiiee//odeSuubeee wwtpibiiissioooooiiooooooooooooo lcy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
httppp.ccppplppccpepeeeeeeppppee pttppp/guuidee//////pppppoo gewgdee/ooeeuuubmnnnnnmmmnmn /Cc////ooeiiisseeeeeeeeeeee wwt.SSubmissionne. wasssnoiiccyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For experiments, we run multiple random experiments. But note that for some experiments, due to high LLM inference costs and evaluation overheads of LeetCode platform, we only conduct single experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
49

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: As this project has been carried out loosely over a long time span (about 10 months), we did not record this information.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tteer///neu...//eee cc/ cc/public/Etheeeeccccccceeeee cehicsChiellenee
Answer: [Yes]
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: As work focuses enhancing reasoning abilities and exploring poten- tially continuously evolving method for LLMss could facilitate the deployment of L.MSs in critical applications requiring precise reasoning and address the data scaling bottleneck. Regarding the negative societal impacts, very helpful reviewer also pointed out a potential risk of bias perpetuation, and have added the discussion to App. Although we made efforts to keep this discussion into the main paper as planned in the rebuttal, due to the page limit, we finally include it standalone section in the appendix, with reference it in $ 6.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
50

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all the dataset providers. For LeetCode, we carefully check their term of service in https: /ttp::///eetaoooo com/terss/ and won't reveal any problems visible test cases in our paper and code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in wwwerrswtthhoe..... package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
51

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
52


Reviews:
[{"review_id": "1oyi4EACk1", "reviewer": "Reviewer_UmKq", "summary": "This paper \"Can LLMs Learn by Teaching? A Preliminary Study\" presents a novel approach towards LLM learning by teaching with three methods: observing student feedback, learning from student feedback, and learning iteratively. The contribute two key findings: teaching student models are an effective way to improve model performance (with fine-tuning), and student models must be diverse (teaching multiple students are better than teaching one student).", "questions": "What are the interpretability implications of teaching diverse students at different knowledge levels? A discussion on this might be useful in the paper.\n\nWhy choose the game theory + math reasoning datasets? The motivation for the dataset choice seems to be a bit missing in the paper. \n\nCan this be extended to open-ended math problems? How and what would need to change in the architecture?", "limitations": "The limitations are discussed mostly in terms of extensions. A discussion of bias perpetuation along the student / teacher pipeline i.e. Wambsganss et al. \"Bias at a Second Glance\" (COLING 2022) and global interpretability implications would be helpful here.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is exceedingly well-written and structured, making it easy to follow for a reader. The diagrams are informative and helpful.\n- The idea is simple and brilliant -- it could be could be very effective for LLM finetuning and is executed well.\n- Particularly M3 is reminiscent of ideas from OpenAI (AlphaFold). This is a very interesting exploration.\n- The experiments are plentiful and convincing.", "weaknesses": "Figure 4, Table 2, Table 3, and Table 4 must include at least standard error or ideally 95% CI to prove statistical significance of the results.\n\nWith the number of acronyms in table 3 metrics, it's hard to understand what exactly numbers mean. Please find some way to include more understandable question categorizations.\n\nThe paper is missing literature from the learning sciences backing up the authors' strategy of implementing feedback and their premise of learning-by-teaching. Including a discussion on this would be important for motivating their work. Please examine literature on feedback (i.e. Power of Feedback, Hattie et al. or from the recent EDM, AIED, LAK communities) to strengthen your discussion and motivation. Additionally, it would be useful to examine ways to evaluate teaching quality also from the same communities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Figure 4, Table 2, Table 3, and Table 4 must include at least standard error or ideally 95% CI to prove statistical significance.**\n\nThanks for this valuable suggestion. Due to the high cost of running experiments with LLMs, we reported the standard errors using the \"bootstrapping\" method [1, 2] in Figure 4 (Left) and Table 6 (Appendix A.1.3). We will clarify this in the revision. Specifically, in Figure 4, standard errors were calculated for K < 128 (number of TR-TA pairs) by selecting K pairs from the total 128 TR-TA pairs. Similarly, standard errors are presented for K = 12 in Table 6, showing that M1 with K = 12 can outperform SC with K=128.\n\nFollowing the suggestion, we extend the M1 Math experiments in Table 2 and Figure 4 to include up to 256 TR-TA pairs, enabling us to compute standard errors for the original 128-pair setting. We have also included additional experiments for M2 with 3 repeated runs. Please refer to the updated results in the provided PDF.\n\nNevertheless, we cannot obtain other standard errors through repeated runs during the rebuttal period due to the resource and time constraints. For instance, the experiments in Table 3 are primarily constrained by the rate limit of the LeetCode server.\n\n**Q2: With the number of acronyms in table 3 metrics, it's hard to understand what exactly numbers mean. Please find some way to include more understandable question categorizations.**\n\nThanks for pointing this out. There is a typo in our original caption stating that each acronym represents a question category. In fact, each acronym in the M1-Code experiments, such as SG-1 or SG-2, represents an individual question. All results in Table 3 are from the Game Theory category. More broadly, we experimented with three general question categories: Game Theory, Bitmasking, and General-1D, and summarized their results in Tables 3/9, Tables 10/11, and Tables 12/13, respectively.\n\nWe will correct the typo in the caption and clarify the settings in the revision. Please do not hesitate to let us know if other clarification is needed for Table 3.\n\n**Q3: The paper is missing literature from the learning sciences backing up the authors' strategy of implementing feedback and their premise of LbT. Including a discussion on this would be important for motivating their work. Please examine literature on feedback to strengthen your discussion and motivation. Additionally, it would be useful to examine ways to evaluate teaching quality also from the same communities.**\n\nThanks for this valuable suggestion. We agree that discussing more literature from the learning sciences will not only strengthen the motivation of our current implementation but also provide valuable insights for further improving the methods. We will add the discussions to our revision.\n\nTo back up our concrete implementation of the LbT idea, we review several studies within the field [3, 4, 5, 6, 7, 8, 9, 10] that discuss how students' feedback can enhance a teacher's capabilities. The benefits of such feedback can be attributed to:\n- Reflection [7, 11, 12]: Teachers monitor and reflect on how well their ideas are understood by students and this reflection aids in evaluating their own understanding of domain concepts.\n- Knowledge-building [4, 5, 6]: Through interactions with students, such as questioning, teachers reflect upon their own expertise and comprehension, and become aware of their own misconceptions, and then attempt to repair them.\n\nReflection directly supports the design of M1 and M3, where teachers improves their answer quality by observing how well a student answers similar problems (in M1) or reflecting on failure cases from multiple students (in M3). Additionally, knowledge-building backs up the implementation of M2, where students' feedback helps a teacher identify misconceptions, which are then addressed through DPO.\n\nFor the roadmap of further improving the LbT implementation, Section 6.3 currently discusses our perspective on the general LbT pipeline, including potentially useful strategies for teaching material design and educational pipeline design. We will extend this section and discuss relevant literature from the learning sciences, based on the framework in Figure 7, focusing on the following points:\n- \"cooperative learning\" [13] implies dividing a difficult topic (as mentioned in \"Task-oriented collaborative learning\", Section 6.3) into several specific topics so that multiple agents can learn jointly. \n- \"teachable agents\" [7, 9, 10] indicate that appropriately configuring a student's knowledge level enables it to provide more useful feedback to the teacher.\n- Regarding the suggestion to \"examine ways to evaluate teaching quality\", we note that feedback can take many forms [14, 15]. Besides taking tests or exams [4, 5, 6], students can provide valuable feedback through their perception, such as peer recommendations [11] and satisfaction questionnaires [16, 17]. This implies that students could offer free-form feedback, which may be especially beneficial for open-ended problems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yV054xeaeQ", "reviewer": "Reviewer_pWvs", "summary": "This paper the use of learning by teaching methods in the context of LLMs.", "questions": "N/A", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written and methodologically sound.", "weaknesses": "The concise results should be briefly and systematically stated in the final Conclusion chapter, which is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The concise results should be briefly and systematically stated in the final Conclusion chapter, which is missing.**\n\nThank you for the suggestion. We will add a conclusion section and summarize the concise numbers together with the general conclusion there.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OUfwaMyfxW", "reviewer": "Reviewer_mNjM", "summary": "In this paper the authors investigate whether the principles of 'Learning by Teaching' (LbT) in humans can be applied and used in LLMs. To investigate this they propose 3 techniques and map them different to LbT levels. \n\nThe first technique M1 aims at improving answer quality by developing a scoring function to rank answers generated by LLMs. For the purpose of this method a strong LLM is made to generate Teaching Rationale (TR) and Teaching Answer (TA) pairs for a given teaching problem (TP). The TR-TA pairs are used as ICL exemplars within student models presented with an Exam Problem (EP) which is of similar type as the TP.  The student models then produce Exam Rationale (ER) and Exam Answers (EA) and receive an accuracy score or LbT score based on the correctness of EA. The TR-TA pairs are then selected based on the highest score and the highest sum of scores. The authors demonstrate that this scoring strategy outperforms self-consistency scoring/greedy scoring in a variety of teacher-student model settings and a variety of tasks (e.g. Math, Coding)\n\nThe second technique M2 aims to improve the ability of LLMs by leveraging the student feedback. For this DPO is used to finetune the teacher model on the TR and the corresponding LbT scores obtained using M1 along with correctness scores. The authors show that this technique leads to better performance than by simply doing DPO with the correctness scores.\n\nThe third technique M3 is based on iteratively improving the exemplars generated for the students by reflecting on the mistakes made by the students. The authors demonstrate that this technique helps improve the performance and also benefits when multiple student LLMs are used to provide feedback.\n\nIn summary the authors conclude by suggesting that strong teacher models can improve even when teaching weak students(weak-to-strong generalization) and teaching other students/multiple students works better than teaching itself.", "questions": "For the claim that 'Improvements do not saturate as number of TR-TA pairs increase' has there been any analysis done to identify the upper bound or the optimal number of TR-TA pairs needed (cost vs performance)?\n\nGeneral Comments:\nThere is a typo on line 53.", "limitations": "The authors acknowledge that the technique requires the strategies to solve the EPs be similar to the ones used for the TPs and that currently the EPs were selected based on human provided information.\nAs the authors acknowledge the proposed technique also leads to additional inference cost.\nFurthermore it remains to be seen how this technique can be extended to tasks wherein the answers generated by student models are not easily verifiable.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written and provides several interesting and novel results which could be of significant interest to the research community (particularly alignment). The experimental analysis and discussion of results is sound and well supported by evidence.  Along with the introduction of LbT the paper provides concrete methods (M1, M2, M3) to instantiate different aspects of LbT in LLMs. The paper also presents some encouraging findings such as 1) LbT can be used to improve answer quality and model capability, 2) LbT exhibits weak-to-strong generalization and 3) diversity of student models helps.", "weaknesses": "The LbT score seems to be reliant on having the final answer being verifiable. It would be interesting to see how this can be translated to cases where the answer produced by the student models is in free-form text. In this regard the current use-cases demonstrated have been in improving performance on tasks such as math or coding. More extensive evaluation on diverse tasks may be needed to assess the generalizability of LbT. A more detailed error analysis on what types of errors the student models make and which of these errors provide the most helpful feedback to the teacher model could be interesting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The LbT score seems to be reliant on having the final answer being verifiable. ... More extensive evaluation on diverse tasks may be needed to assess the generalizability of LbT.**\n\nThanks for this valuable question. LbT can indeed be extended to open-ended problems, such as dialogue, writing, and open-ended math problems. A natural extension could involve using a teacher LLM to evaluate a student's answer, which would then serve as the LbT score. This approach mirrors how a teacher assesses a student in human learning and is promising due to the strong evaluation capabilities of LLMs [19, 20]. A similar idea has been employed to generalize Self-Consistency (which only applies to verifiable problems) [21] to open-form problems [22]. \n\nBesides, as demonstrated in literature from the learning sciences, teachers may directly benefit from providing assessments [8] or reviews [23], and students can provide valuable free-form feedback beyond taking tests or exams, such as peer recommendations [11] and satisfaction questionnaires [16, 17]. We believe that an open-ended evaluation process could potentially offer additional feedback to the teacher LLM, which can be utilized to further enhance its performance. We plan to explore these extensions in future studies.\n\n**Q2: A more detailed error analysis on what types of errors the student models make and which of these errors provide the most helpful feedback to the teacher model could be interesting.**\n\nThanks for this valuable suggestion. For math reasoning (M2), we have included some analyses in Section 4.3 and presented the examples in Appendix B.2. For competition-level code synthesis (M1), we have included analyses in Section 3.3.2 and presented the examples in Appendix A.2.2. \n\nFor math reasoning and code synthesis, students can make both logical and non-logical errors. The non-logical errors, such as computation errors (Math), missing imports (Code), miswritten variable names (Code), and incorrect usage of library functions (Code), are mainly related to the knowledge required by specific EQs and the robustness of the student model, rather than reflecting the quality of the TR-TAs. Thus, they do not provide helpful feedback to the teacher. To address this, for code synthesis, we apply self-debugging to correct non-logical bugs, making the students' exam V-score a more accurate indicator of TR quality and thus more helpful to the teacher.\n\nIn response to your suggestion, we will add more detailed analyses and examples to the appendix. For example, logical errors in code synthesis can be further classified into three types: (1) Code with generally correct logic but incorrect handling of boundary conditions, which fails in some hard cases; (2) Code with incorrect logic, such as a wrong DP recursion formula, which typically fails most cases; and (3) Code with correct logic but complexity issues, such as using recursion instead of DP or cached-recursion, leading to time or memory errors on large cases. Our experiments show that when students follow TR-TA in making these three types of logical errors, it provides valuable feedback for the teacher.\n\nFor textual reasoning tasks (M3), according to your suggestion, we add more error analysis in the attached PDF. More specifically, we further analyzed M3's behavior in identifying false generalizations [18] with Llama-3-70B as the teacher and Llama-3-8B as the student. Some causes of errors identified by the teacher from student mistakes are: (a) \"Lack of examples within the context of multiple speakers or dialogue\"; (b)\"Insufficient context for understanding the argument\"; (c) \"Difficulty in handling nuances of everyday language and humor\". First, we found that the errors that the teacher identifies from student mistakes are **also applicable to teacher's mistakes**, with 45.2%, 37.1%, and 44.6% of teacher mistakes caused by these three reasons. Second, after the teacher improves the ICL examples by learning from the students, the teacher's mistakes caused by these reasons are reduced by 6.0%, 11.6%. and 13.3%. Finally, mistakes of different students lead to complementary causes of errors that are also very relevant to teacher mistakes.\n\n**Q3: For the claim that 'Improvements do not saturate as number of TR-TA pairs increase' has there been any analysis done to identify the upper bound or the optimal number of TR-TA pairs needed (cost vs performance)?**\n\nThanks for raising this valuable question. In the rebuttal, we have extended our M1 Math experiments to include 256 TR-TA pairs. Please refer to the updated Figure 4 (Left) in the provided PDF. While the curve does become flatter with the increase in the number of TR-TA pairs, we still observe a non-negligible slope even with 256 TR-TA pairs.\n\nFor precision, we will revise this claim to: \"The relative improvement over SC increases as the number of TR-TA pairs increases within the range of TR-TA pairs in our experiments\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XK0zVF7Bjr", "reviewer": "Reviewer_E9eU", "summary": "The core question raised in the paper is whether LLMs can also learn through teaching (LbT). The authors demonstrate that the idea of LbT can be easily integrated into existing LLM training/prompting processes and propose three methods, each mimicking three levels of human LbT: observing feedback, learning from feedback, and iterative learning. The authors found that: (1) LbT can induce a progression from weak to strong generalization, where a strong model can improve itself by teaching other weak models; (2) Diversity among students is important, as teaching multiple students may be better than teaching just one student or the teacher itself.\n\nCorresponding solutions:\n\n1）M1 aims to improve the answer quality of LLMs by directly leveraging students' feedback (L1). 2） M2 aims to enhance the intrinsic abilities of LLMs by learning from students' feedback (L2). 3） M3 aims to improve the answer quality of LLMs by iteratively learning from students' feedback (L3).\n\nStrengths: The paper provides a clear and detailed description of the problem and the proposed solutions, supported by ample experimentation.\n\nWeakness:\n\nThe idea presented in the article is a well-known idea, but it lacks sufficient assumptions and counterexample analysis, making it too intuitive. See:\n[1]Black-box Generalization of Machine Teaching, Xiaofeng Cao, Yaming Guo, Ivor W. Tsang, James T. Kwok, https://arxiv.org/abs/2206.15205v2.\n\nEmploying teaching feedback to guide a white-box or black-box learner has been mentioned earlier.\n\nIn machine learning, such an idea is the most basic one, especially in machine teaching. See:\n[2] Liu, W., Dai, B., Humayun, A., Tay, C., Yu, C., Smith, L. B., ... & Song, L. (2017, July). Iterative machine teaching. In International Conference on Machine Learning (pp. 2149-2158). PMLR.\n\n[3] Zhang, C., Cao, X., Liu, W., Tsang, I., & Kwok, J. (2023, July). Nonparametric iterative machine teaching. In International Conference on Machine Learning (pp. 40851-40870). PMLR.\n\nThere are also works for teaching multiple learners: [4] Yeo, T., Kamalaruban, P., Singla, A., Merchant, A., Asselborn, T., Faucon, L., ... & Cevher, V. (2019, July). Iterative classroom teaching. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, No. 01, pp. 5684-5692).\n\n[5] Zhang, C., Cao, X., Liu, W., Tsang, I., & Kwok, J. (2024). Nonparametric teaching for multiple learners. Advances in Neural Information Processing Systems, 36.\n\nThe authors lack a survey of the machine learning community regarding teaching.\n\nThe author exaggerates their contribution or should strengthen the underlying assumptions of their contribution. Teaching weak learners is related to their variance.\n\nTeaching weak students is challenging unless there is a proper configuration. Otherwise, the improvement in students' learning and reception abilities is limited, as already verified in knowledge distillation (KD).\n\n5）We have only seen separate presentations of the M1-M3 approaches. It is unclear how M1-M3 collectively contribute to the overall logic and objectives of the paper.\n\n6）M3 mentioned in the paper is not sufficient to be considered a core contribution.\n\n7）Would these methods still be effective if there is a significant difference between the student model and the teacher model?\n\nOverall, this draft studies the teaching in the current LLM settings. While the assumptions and conclusions have issues based on current presentation. I suggest the authors tune down their statements.", "questions": "a) Would these methods still be effective if there is a significant difference between the student model and the teacher model?\n\nb) Teaching weak students is challenging unless there is a proper configuration. Otherwise, the improvement in students' learning and reception abilities is limited, as already verified in knowledge distillation (KD).", "limitations": "Incomplete analysis from the teaching and irregular contribution statements.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths: \nThe paper provides a clear and detailed description of the problem and the proposed solutions, supported by ample experimentation.", "weaknesses": "Weakness:\n\nThe idea presented in the article is a well-known idea, but it lacks sufficient assumptions and counterexample analysis, making it too intuitive. See:\n[1]Black-box Generalization of Machine Teaching, Xiaofeng Cao, Yaming Guo, Ivor W. Tsang, James T. Kwok, https://arxiv.org/abs/2206.15205v2.\n\nEmploying teaching feedback to guide a white-box or black-box learner has been mentioned earlier.\n\nIn machine learning, such an idea is the most basic one, especially in machine teaching. See:\n[2] Liu, W., Dai, B., Humayun, A., Tay, C., Yu, C., Smith, L. B., ... & Song, L. (2017, July). Iterative machine teaching. In International Conference on Machine Learning (pp. 2149-2158). PMLR.\n\n[3] Zhang, C., Cao, X., Liu, W., Tsang, I., & Kwok, J. (2023, July). Nonparametric iterative machine teaching. In International Conference on Machine Learning (pp. 40851-40870). PMLR.\n\nThere are also works for teaching multiple learners: [4] Yeo, T., Kamalaruban, P., Singla, A., Merchant, A., Asselborn, T., Faucon, L., ... & Cevher, V. (2019, July). Iterative classroom teaching. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, No. 01, pp. 5684-5692).\n\n[5] Zhang, C., Cao, X., Liu, W., Tsang, I., & Kwok, J. (2024). Nonparametric teaching for multiple learners. Advances in Neural Information Processing Systems, 36.\n\nThe authors lack a survey of the machine learning community regarding teaching.\n\nThe author exaggerates their contribution or should strengthen the underlying assumptions of their contribution. Teaching weak learners is related to their variance.\n\nTeaching weak students is challenging unless there is a proper configuration. Otherwise, the improvement in students' learning and reception abilities is limited, as already verified in knowledge distillation (KD).\n\n5）We have only seen separate presentations of the M1-M3 approaches. It is unclear how M1-M3 collectively contribute to the overall logic and objectives of the paper.\n\n6）M3 mentioned in the paper is not sufficient to be considered a core contribution.\n\n7）Would these methods still be effective if there is a significant difference between the student model and the teacher model?\n\nOverall, this draft studies the teaching in the current LLM settings. While the assumptions and conclusions have issues based on current presentation. I suggest the authors tone down their statements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The idea presented in the article is a well-known idea.**\n\nThank you for suggesting relevant references from the machine teaching literature. We will cite and discuss them in our revised version. While these papers have similarities in terms of how the teacher should organize teaching materials, our work differs from prior studies in two key aspects:\n1. As stated in our introduction, \"teaching\" has been extensively studied in machine learning, with knowledge distillation as a prominent example. While both machine teaching and knowledge distillation aim to **improve the student**, our focus is on whether using \"teaching\" to get students' feedback can help **improve the teacher**—especially in the context of weak-to-strong generalization (i.e., continuously evolve *stronger* teacher by teaching *weaker* students). This distinction sets our paper apart from existing work.\n2. Instead of solving small-scaled tasks using simple analytical models, such as linear or kernel models, as is common in much of the machine teaching literature, our work focuses on advancing the **reasoning capabilities of contemporary LLMs** with tens of billions of parameters. As LLMs get stronger and stronger and even show human-like behaviors nowadays, our work aims to evaluate whether the LbT methodology that has proven effective in providing intrinsic supervision for accurate knowledge building and reasoning in human learning can similarly benefit LLMs.\n\nTo summarize, as far as we are aware, this is the first attempt at migrating the LbT idea from the learning sciences [3, 4, 5, 6, 7, 8, 9, 10] to enhance LLMs. We will discuss these key differences, including the motivation, the model scale, and the targeting tasks, between our work and provided references from the machine teaching community in the revision.\n\n**Q2: It lacks sufficient assumptions and counterexample analysis, making it too intuitive.**\n\nThanks for this comment. We'd like to share our perspectives on this issue. We acknowledge that since our study focuses on a complex and powerful LLM solving complex reasoning problems, it is challenging to derive comprehensive theoretical frameworks with precise assumptions and counterexamples. As LLMs are becoming stronger and even show human-like behaviors nowadays, we believe that advancing algorithms at both ends of the spectrum can help push the field forward:\n- Developing methods and models grounded with theoretical guarantees to solve simple or synthetic tasks.\n- Building intuitive methods for use with state-of-the-art models to address real-world complex tasks. These methods, inspired by human problem-solving and learning processes, are often empirically validated through experimental analyses rather than being grounded in theoretical frameworks.\n\nThere are many other studies that also draw inspiration from human problem-solving techniques to enhance their capabilities, which have created enormous impact on the LLM community. For instance, techniques such as Chain-of-Thought [24] is motivated by the way humans reason through problems step-by-step. Self-Refine [25] is inspired by the human practice of refining written text. Similarly, Relexion [26] emulates how humans iteratively learn and adapt when tackling complex tasks. These works, together with ours, lie in the second category of the spectrum. \n\n**Q3: Teaching weak learners is related to their variance.**\n\nWe are not very sure what \"variance\" refers to in this context. Could you please provide more details on this comment? We are eager to understand your perspective better and address any concerns you might have.\n\n**Q4: Teaching weak students is challenging unless there is a proper configuration. Otherwise, the improvement in students' learning and reception abilities is limited, as already verified in knowledge distillation (KD).**\n\n**Q7: Would these methods still be effective if there is a significant difference between the student model and the teacher model?**\n\nThank you for the great question. About why teachers can benefit from teaching weak students, our intuition is as follows. To teach a weak student, the teacher needs to organize detailed and high-quality teaching materials that can be digested even by a weak student. This is a high bar for the teachers and the teachers should be able to learn a lot from material preparation and the feedback from the students. However, the students should not be too weak. Otherwise, they may lack the ability to learn anything from the teacher (e.g. through in-context learning), and therefore cannot provide useful feedback.\n\nIndeed, the effectiveness of LbT varies as the teacher-student configuration changes. As demonstrated by our experiments, certain configurations are less effective than others. For example, as shown in Table 2, when ChatGPT-3.5 is the teacher, it is more effective to use LLaMA3-8B as the student than Mistral-7B.\n\nDespite this, our method proves effective even with a significant disparity between the teacher and the student. For example, in M1 and M3, we have shown that LLaMA3-70B (70.16% on 181 MATH) can benefit from teaching LLaMA3-8B (45.85% on 181 MATH). In M1, we demonstrated that GPT-3.5 (59.11% on 181 MATH) can benefit from teaching Mistral-7B (19.88% on 181 MATH). These examples involve models from different families with a large capacity gap.\n\nTo further demonstrate the robustness of our method across various configurations, we conducted an additional experiment where LLaMA3-70B teaches Mistral-7B. Please refer to the additional results in the provided PDF for more details.\n\nRegarding the comment \"the improvement in students' learning and reception abilities is limited\", we are not sure whether we understand this comment correctly, but we aim at improving the performance of the **teacher** rather than the **student**. Please do not hesitate to correct our understanding if necessary.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their valuable time and effort in reviewing our paper. We are encouraged that the reviewers recognize our paper as novel and interesting (UmKq, mNjM); see its potential impact on the LLM community (UmKq, mNjM); note the abundance of experiments included (UmKq, mNjM, E9eU); and find it well-written and easy to follow (UmKq, pWvs, mNjM). We also appreciate the thoughtful concerns and suggestions, which are both inspiring and valuable for further discussion.

In response to the comments, we have added several new experiments which can be found in the attached PDF:
- We have extended the number of TR-TA pairs in M1 Math to 256 (Table 1 in the PDF).
- We have included additional experiments for M1 Math with LLaMA3-70B teaching Mistral-7B and LLaMA3-70B teaching LLaMA3-8B + Mistral-7B (Table 1 and Table 2 in the PDF).
- We have updated the plot showing the relative improvements of M1 over SC with up to 256 TR-TA pairs (Figure 1 in the PDF).
- We have calculated standard errors for M1 Math with K < 256 (number of TR-TA pairs), by selecting K pairs from the total 256 TR-TA pairs (Table 2 and Figure 1 in the PDF).
- We have calculated standard errors for M2 with 3 repeated runs (Table 3 in the PDF).
- We have added an analysis of the causes of errors identified by the teacher (LLaMa3-70B) in M3 (Table 4 in the PDF).

**References** 

[1] Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. ACL, 2024.

[2] Improve mathematical reasoning in language models by automated process supervision. arXiv, 2024.

[3] Children teach children: Learning by teaching. Harper & Row, 1971.

[4] The influence of the tutee in learning by peer tutoring. Proceedings of the Annual Meeting of the Cognitive Science Society, 2004.

[5] Understanding tutor learning: Knowledge-building and Knowledge-telling in Peer Tutors' Explanations and Questions. Review of 
Educational Research, 2007.

[6] Tutor learning: The Role of Explaining and Responding to Questions. Instructional Science, 2008.

[7] Learning by teaching: A New Agent Paradigm for Educational Software. Applied Artificial Intelligence, 2005.

[8] Learning-by-teaching. Evidence and implications as a pedagogical mechanism. Innovations in Education and Teaching International, 
2017.

[9] Can you clarify what you said?: Studying the Impact of Tutee Agents' Follow-up Questions on Tutors' Learning. AIED, 2021.

[10] Teach AI how to code: Using Large Language Models as Teachable Agents for Programming Education. CHI, 2024.

[11] Building a metacognitive model of reflection. Higher Eduction, 1999.

[12] Learning from human tutoring. Cognitive Science, 2001.

[13] Evaluation of Jigsaw, a cooperative learning technique. Contemporary Educational Psychology, 1985. 

[14] The power of feedback. Review of Educational Research, 2007.

[15] The power of feedback revisited: A Meta-analysis of Educational Feedback Research. Frontiers in Psychology, 2020.

[16] Quantifying quality: The Importance of Student Feedback. Quality In Higher Education, 2001.

[17] Instruments for obtaining student feedback: A Review of the Literature. Assessment & Evaluation in Higher Education, 2005.

[18] Logical fallacy detection. EMNLP Findings, 2022.

[19] Judging LLM-as-a-judge with MT-Bench and chatbot arena. NeurIPS, 2023.

[20] From crowdsourced data to high-quality benchmarks: Arena-Hard and BenchBuilder Pipeline. arXiv, 2024.

[21] Self-consistency improves chain of thought reasoning in language models. ICLR, 2023.

[22] Universal self-consistency for large language model generation. arXiv, 2023.

[23] Learning by reviewing. Journal of Educational Psychology, 2011.

[24] Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 2022.

[25] Self-refine: Iterative Refinement with Self-Feedback. NeurIPS, 2023.

[26] Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS, 2023.

Meta-review: The reviewers agreed that the paper investigates a novel setting of how learning-by-teaching ideas can help improve LMMs and that the results would be of broad interest to the community. However, the reviewers also raised several concerns and questions in their initial reviews. We want to thank the authors for their responses and active engagement during the discussion phase. The reviewers appreciated the responses, which helped in answering their key questions. The reviewers have an overall positive assessment of the paper, and there is a consensus for acceptance. The reviewers have provided detailed feedback, and we strongly encourage the authors to incorporate this feedback when preparing the final version of the paper.
Decision: Accept (poster)