Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Models
Authors: Ling Yang
Zhaochen Yu
Tianjun Zhang
Shiyi Cao
Minkai Xu
Wentao Zhang
Joseph E. Gonzalez
Bin CUI
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Large Language Models
Primary area: natural_language_processing

Abstract: We introduce Buffer of Thoughts (BoT), a novel and versatile thought-augmented reasoning approach for enhancing accuracy, efficiency and robustness of large language models (LLMs). Specifically, we propose meta-buffer to store a series of informative high-level thoughts, namely thought-template, distilled from the problem-solving processes across various tasks. Then for each problem, we retrieve a relevant thought-template and adaptively instantiate it with specific reasoning structures to conduct efficient reasoning. To guarantee the scalability and stability, we further propose buffer-manager to dynamically update the meta-buffer, thus enhancing the capacity of meta-buffer as more tasks are solved. We conduct extensive experiments on 10 challenging reasoning-intensive tasks, and achieve significant performance improvements over previous SOTA methods: 11\% on Game of 24, 20\% on Geometric Shapes and 51\% on Checkmate-in-One. Further analysis demonstrate the superior generalization ability and model robustness of our BoT, while requiring only 12\% of the cost of multi-query prompting methods (e.g., tree/graph of thoughts) on average. Code is available at: https://github.com/YangLing0818/buffer-of-thought-llm

Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Models
Ling Yang "t, Zhaochen Yu'; Tianjun Zhang?, Shiyi Cao?, Minkai Xu
Wentao Zhang', Joseph E. Gonzalez², Bin Cuilt
Peking University, "CC Berkeley. Stanford University
Project: http://ihhuu..m/aagggg..g Extension: https///gttuhh.com/ Yangl Jing08 8/SuperCorrect-IIn
ooffferr-ofthoughh Ilm
Abstract
We introduce Buffer of Thoughts (BoT), novel and versatile thought-augmented reasoning approach for enhancing accuracy, efficiency and robustmess of large language models (LLMs). Specifically, propose meta-buffer to store series informative high-level thoughts, namely thought-template, distilled from the problem-solving processes across various tasks. Then for each problem, we retrieve a relevant thought-template and adaptively instantiate with specific reasoning structures conduct efficient reasoning. To guarantee the scalability and stability, we further propose buffer-manager to dynamically update the meta-buffer, thus enhancing the capacity of meta-buffer as more tasks are solved. We conduct extensive experiments on 10 challenging reasoning-intensive tasks, and achieve significant performance improvements over previous SOTA methods: 11% on Game of 24, 20% on Geometric Shapes and 51% on Checkmate-in-Omr Further analysis demonstrate the superior generalization ability and model robustness of our BoT, while requiring only 12% of the cost of multi-query prompting methods (e.g., tree/graph of thoughts) on average. Notably, we find that our Llama3-8h BoT has the potential to surpass Llama3-70B model. Our project is available at www.//ooo///i/iiioiieoooooo YangLing088 8/buffer-of ofuuffr-of-thhughht-nn
https://github.com www.tt......oooo.eeeeeeeeeeeeeee
Introduction
series of Large Language Models (LLMs) [1-5] like GPT-4 [3]. PaLM [2] and LLaMA [6. 7] have showcased the impressive performance in various reasoming tasks. In addition to scaling up the model size improve the reasoning performance, there are more effective prompting methods that further enhance the functionality and performance of LLMss. We divide these methods into two categories: (i) single-query reasoning: these methods [8-10] usually focus on prompt engineering and their reasoning process can be finished within single query, such CoT [8] that appends the input query with 'Let's think step by step' to produce rationales for increasing reasoning accuracy, and Few-shot Prompting [11, 12, 9, 13] which provides task-relevan exemplars to assist the answer generation; (ii) multi-query reasoning: these methods [14, 15] focus on leveraging multiple LLM queries to elicit different plausible reasoning paths, thus decomposing a complex problem into series of simpler sub-problems, such as seast-to-Most [16]. ToT [14] and GoT [17].
However, both kinds of methods face some limitations: (1) single-query reasoning usually requires prior assumption or relevant exemplars of reasoning process, which makes impractical manually design them task by task, thus lacking universality and generalization; (2) Due to the recursive
Eqqull Contribution. Contact: yangling08 18@ 163.com
Corresponding Authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Single-Query
query C
LLM
Reasoning
Output
Manual Prompt for Specific Task le.gu CoT, Few-shot Prompting)
Accuracy
N-hop Iteration
Multi-Query
Thought Expansion
query C
LLM
Reasoning
Output
Pre-defined Query (e.g. ToT, GoT)
Efficiency
Problem Distiller
Instantiated Reasoning
Input query +
LLM
Output
High-level Thoughts
Thought Distillation
Accuracy
and Update
Thought Retrieval
Efficiency
Buffer of Thoughts (BoT)
Meta Buffer
Figure 1: Comparison between single-query [8, 11]. multi-query [14, 17], and (c) our BoT methods.
expansion of reasoning paths, multi-query reasoning is usually computationalll-intenni
when finding unique intrinsic structure underlying the reasoning process for each specific task; (3) Both single-query and multi-query reasoning processes are limited by their designed exemplars and reasoning structures, and they neglect to derive general and high-level guidelines or thoughts from previously compleeed tasks, which are informative for improving efficiency and accuracy when solving similar problems.
To address these limitations, we propose Buffer of Thoughts (BoT), a novel and versatile thought- augmented reasoning framework aimed at enhancing reasoning accuracy, efficiency and robustness of LLMs across various tasks. Specifically, we design meta-buffer, lightweight library housing series of universal high-level thoughts (thought-template), which are distilled from different problem-solving processes and can be shared across tasks. Then, for each problem, we retrieve a relevant thought- reasoning. to
template and instantiate it with specific reasoning structure for efficient In order to guarantee the scalability and stability of our BoT, we further propose dynamically update the meta-buffer, which effectively enhances the capacity of meta-buffer as more tasks are solved.
Our method has three critical advantages: (i) Accuracy Improvement: With the shared thought- templates, we can adaptively instantiate high-level thoughts for addressing different tasks, eliminating the need build reasoning structures from scratch, thereby improving reasoning accuracy. (ii) Rea- soning Efficiency: Our thought- nuggene-aggmented reasoning could directly leverage informative historical reasoning structures conduct reasoning without complex multi-query processes, thus improving reasoning efficiency. (iii) Model Robustness: The procedure from thought retrieval to thought instantiation is just like the human thought process, enabling LLMs to address similar problems in consistent way, thus significantly enhancing the model robustness of our method. Our empirical stud- ies demonstrate that Buffer Thoughts significantly improves precision, efficiency, and robustness over diverse array of tasks. Here, we summarize our contributions as follows:
We propose novel tuouent-augmented reasoning framework Buffer Thoughts (BoT) for improving the accuracy, efficiency and robustness of L.L.-based reasoning.
2. We propose meta- meta-buffer for store informative high-level thoughts distilled from different problems, and adaptively instantiate each thought template address each specific task.
3. We design buffer-manager distill
from various solutions, and is contin- ually improves the capacity meta-buffer as more tasks are solved.
4. We conduct extensive experiments on 10 challenging reasoning-intensse tasks. Our BoT achieves significant performance improvements over previous SOTA methods: 11% on Game of 24, 20% on Geometric Shapes and 51 on Checkmate-in-Onee while requiring only 2% of the cost of multi-query prompting methods average.

---

2 Related Work and Discussions
Setrieval-Augmentee Language Models The retrieval -rugeeeee (Large) Language Model is introduced as a solution to mitigate the phenomenon of hallucination and enhance the output quality of language models [18-22]. When presented with an input question, the retrieval-augmented LLM first queries external database with billion-level tokens [23] for retrieving a subset of the text corpus to help generating the final answer. Notably, the eetrieval-augmenn
LLM achieves superior question-answering performance using fewer parameters compared to conventional LLMs [19], and it has found application across various downstream tasks [24-26], including multi-modal generation (24, 22, 23, 25] and biomedical applications [26, 27]. In this paper, we construct novel category of retrieval database, termed meta-buffer, which contains a series of high-level thoughts rather than specific instances, aiming universally address various tasks for LLM-based reasoning.
Prompt-based Reasoning with Large Language Models Prompting techniques have significantly enahnced the arithmetic and commonsense reasoning capabilities of L.MMs. Chain-of-Thoughh (CoT) prompting [8] and its variants [28-30], such as Least-to-Moo Most [16], Decomposed Prompting [31], and Auto-CoT CoT [ LLMS to break down complex questions into simpler subtasks and systematically solve them before summarizing final answer. Numerous studies [32-37] have demon- strated the effectiveness of these prompting methods across a wide range of tasks and benchmarks. Innovations like Tree-of-Thought [14] and Graph-of-Thought [17], have further advanced this field by exploring dynamic, non-linear reasoning pathways to expand heuristic capabilities of LLMs [38, 39]. However, they suffer from increased resource demands and greater time complexity, depend on manual prompt crafting, and are often tailored to specific task types. Recent meta prompting methods [15, 40] utilize same task-agnostic form of prompting for various tasks and recursively guide single I.LM to adaptively addressing different input queries. Nevertheless, such a long meta prompt may require considerable context window, and these methods fail to leverage historical informative guidelines or thoughts for potential similar tasks.
Analogical Reasoning Analogical reasoning is useful technique for natural language reasoning [41-45]. Recent works demonstrate that LLMs can perform analogical reasoning just like humans [46, 47, 12, 48, 49]. For example, Analogical Prompting [12] and Thought Propagation [48] prompt LLMS to self-generate a set of analogous problems, and then utilize the results of analogous problems to produce a solution for input problem. However, the specific solutions for self-explored problems may introduce additional noise and cause error accumulation. Recent Ehoought-Retriewe [49] uses the intermediate thoughts generated when solving past user to address analogous queries, but it only instead of general reasoning problems. Thus, more
focuses on textual comprehension/geenraae high-level and general analogical approach for LLM complex reasoning is still lacking.
3 Buffer of Thoughts
Overview of Buffer of Thoughts In this section, we introduce our Buffer of Thoughts in detail and we also illustrate core nhought-augmenter reasoning process in Figure 2. Given specific task, we utilize our moblem--dstiille (Section 3.1) to extract critical task-specific information along with relevant constraints. Based on the distilled information, we search in meta-buffet (Section 3.2) and retrieve a most relevant thought-
that contains a series of high-level thoughts (thought-templan instantiate the retrieved toougttttempltee with more task-
template for the task. Subsequently, specific reasoning structures and conduct reasoning process. Finally, we employs a buffer-manager (Section 3.3) for summarizing the whole problem-solving process and distilling high-level thoughts for imcreasing the capacity of meta-buffer.
3.1 Problem Distiller
Most of complex tasks contain implicit constraints, complex object relationships, and intricate variables and parameters within their contexts. Consequently, during the reasoning stage, LLMs need to overcome three main challenges: extracting vital information, recognizing potential constraints, and performing accurate reasoning. These challenges would impose significant burden on single I.M. Therefore, we separate the extraction and comprehension stages of task information from the final reasoning stage, through prepending problem distiller to the reasoning process. More
3

---

Input Probeem
piece. order sales, profiss, and roduce rnventory possibie, mall has deciced oier appeoprate reduction After inverggtion, found that every decrease the
certait shoppling mall betch beanded shins, with
thi average shits
avergee profit 1250
should shit sedaced?
Buffer of Thoughts
Chain Thought
Thought Tereplonn Py
Thought Templite Ty raargnaenc sarannanncataaioooo Bataloiaaiiii
I Calculatz the cureet diily prodle Camens daily profa reofi-20 20 40 800 yuun
saff lu a. an sakms agmind
Dallii have proces cabh batn clince
2 Calculate addiiional daily prof'e from solling mort shirts: Additional duily profit 2 40 yua
Le fund dewe salvigge mppeiiinn
del refirriteee, sifeetttlien choerettt Oetine semnic men cip- mimben, del cww.ihitttteee beceereeeeinne invitte
Buffer
24 pepeettee diown simer Crising loredde
S. Calcalaiz Prize ralaction: Price reduction needed - 300: yun
Detorniaas Nature Rasse
noul mest
#otiie preditive
Therefare, each shirt should reduced for the mail day.
1210
Problem Distilation
Insrantiated
Thought
X
Thought Template
Seletteei
skinn (00) oy be fhe onciuess which the prico of ext skin sallced.
célor ary qurteis eq cations of timm
Han-ane-SSche
tiverbei, Horz
umginal quatity
picnol
devise plus solve problem, let' break stepe:
is lie sincttaed lomplatt fe wdvinng soch equatome::
The mull
qpantty diidy
Step Calcntate Discrtadinnn
1260 year, ne lave
1.Deteraiine current daily profit:
Bog? 4uc.
sang
Añl inpiiffcaiioe
Caiculine
daily prttit using everage daily
Siep 2: Deternine the Nanure of the Roods
sales and profit per shirt
Prr,,,, the opartion his nwo dist roots opation exartly
equatiin
vetiied in the templatee
A. Sei up the equation the desired daily prafit
Step E Calcelate the Dicriniinnt
Create at equation that relates new profit per shint. increased mumber shirts sold, and desteed daiy proft
Step is Compate the Rasis
HI 2-18 oquaion two coeppoo
(-38)
shor iith cxleabas soor aceeg format
Deterniie the Natuoe Roors
Stoce Dowel the equaxtio hos nwo distect mook.
Step 4,
Solve the equation
Step 3: Compaat the Rosbs
Exparal fhr oquetion and wullect like urme:
caicalaic
para
Dicide by 3 simplity:
1200-(20- 120x 40l- (20+gth (2ppp)
orcte campor
forneia
2-20 or
possible
pf -30p-200 =
where the keagieey uill
Duc rodacing lescatory pendik, 2-20 takce
p:35.6155
X
Figure 2: Illustration of different reasoning process. Buffer of Thoughts enables large language models to tackle complex reasoning tasks through our goough--aumenterr reasoning process. Thought template marked in orange and instantiated thought marked in blue.
concretely, we design a meta prompt first distill and formalize the task information. The distilled task information could be denoted as:
Zd = LLM))
where æ is the task statement. Due the page limit, we put the detailed meta prompt for problem- distiller in Appendix B.2.
Problem Condensation and Translation We use the problem distiller to extract key elements from input tasks, focusing on: (1). Essential parameters and variables for problem-solving: (2). The objectives of the input tasks and their corresponding constraints. We then re-organize this distilled information into clear, comprehensible format for the subsequent reasoning stage. We then translate the specific problems into high-level concepts and structures. This translation procedure decomposes complex real-world problems, like intricate mathematical application scenarios, into simpler, multi-step calculations, making it easier for later retrieval of high-level thought.
3.2 Thought-Augmentt Reasoning with Meta Buffer
Motivation Human often summarize and induce higher-level guidelines when solving problems and then apply them relevant problems. Motivated by this, we propose meta-buffer, lightweight for addressing various types of problems. Unlike traditional methods [11. 46, 12, 36, 9] that require specific instructions or exemplars, our high-level shought-templatee can be adaptively instantiated when solving different problems, thereby enhancing LLMs with superior precision and flexibility.
library that contains a series of high-level thoughts (thought-templane
Thought Template As kind of high-level guideline, our thought- is stored in meta- buffer. and is obtained from various groblem-solving processes by our buffer-manager, The details about acquiring provide a general reasoning approach for various tasks, we correspondingly classify the thought- templates into six categories: Text Comprehension, Creative Language Generation, Common Sense Reasoning, Mathematical Reasoning, Code Programming and Application Scheduling. We provide some example thought-templates in Appendix B.I. Such classification of thought-templates can
would be introduced in Section 3.3. Since our BoT aims to

---

facilitate the template retrieval for finding most suitable solutions to different problems. Here we denote thought template, template description and its corresponding category (T,, Dr,, CR), where i denotes the index of meta-template, k E z and 1 < k S 6, which means Cj is in one of the six categories, and Dr; is the description of thought template.
Template Retrieval For each task, our BoT retrieves thought-template T; that highly similar to the distilled problem za by calculating the embedding similarity between the description Dr, and Td. The retrieval process can be formulated as:
j (Sim()(a))()())
where (im(f(za), (P(T)) >= 6, (2)
N is the size of the meta-buffer. f()) is normal text embedding model, and T3 denotes the retrieved thought template. We set a threshold 8 (0.5~~.7 7 is recommended) determine whether the current task is new. Therefore, if (im(f( (f(Dr, ) < 8, we identify the task x as a new task.
Instantiated Reasoning For each specific task, we discuss two situations for the instantiated reasoning, depending on whether the current task new: The first situation is that we successfully retrieve thought-template T, for the task. In this case, as presented in Figure 2, our thought- augmented reasoning will be adaptively instantiated suitable reasoning structures with our designed instantiation prompt (in Appendix B.3). For example, in Checkmate-in-Onm problem, we instantiate the template of updating chess board state to solve the problem step by step. Thus we conduct the instantiated reasoning for task using the distilled information za and the retrieved template Tj. and produce its solution Sg as:
S₂ LLM L. instamiition (Ias Tj),
where LLM instantttiianioon denotes the instantiated reasoner with L.M.M
In the second situation, the task is identified as new task. To enable proper instantiated reasoning, we prepare three general coarse-grained thought-templates for utilization. Based on the distilled task information Fa. our BoT would automatically assign suitable thought-template to the reasoning process. The detailed pre-defined noought-templann are included in Appendix B.3).
3.3 Buffer Manager
We propose buffer-manager summarize the high-level guidelines and thoughts that gained from each problem-solving process. It can generalize each specific solution to more problems, storing the critical distilled knowledge in the form of thought-templates within the meta buffer. In contrast to methods that temporarily generate exemplars or instructions for each problem, our buffer-manager can ensure permanent advancements in accuracy, efficiency, and robustness for LLM-based reasoning.
Template Distillation To extract general though-template we propose three-step approach: (1) Core task summarization: identifying and describing basic types and core challenges problems; (2) Solution steps description: summarize the general steps for solving a problem; (3) General answering template: based the above analysis. propose a solution template or approach that can be widely applied to similar problems. Additionally, to boost the generalization ability and stability of template distillation, we carefully design two types of in-context examples of how to generate thought- template- -inttee-ii-takk and cross-task examples. Cross-task means we choose the template distilled from one task to tackle the problem of other tasks, such as addressing a mathematical problem with code-related thought-template. The new template distilled from input task æ can be denoted as:
Thea LLMaisnn Sw),
where LLMdisill is the bLMd template distiller initialized with the following prompt:
5

---

Prompt for Template Distillation:
User: [Problem Description] + [Solution Steps or Code]
To extract and summarize the high-level paradigms and general approaches for solving such problems, please follow these steps in your response:
1. Core task summarization:
Identify and describe the basic type and core challenges of the problem, such as classifying it as mathematical problem (e.g., solving quadratic equation), data structure problem (e.g.. array sorting), an algorithm problem (e.g., search algorithms), etc. And analyze the most efficient way to solve the problem.
2. Solution Steps Description:
Outline the general solution steps, including how define the problem, determine variables, list key equations or constraints, choose appropriate solving strategies and methods, and how to verify the correctness of the results.
3. General Answer Template:
Based on the above analysis, propose a template or approach that can be widely applied to this type of problem, including possible variables, functions, class definitions, etc. If it is a programming problem, provide a set of base classes and interfaces that can be used to construct solutions to specific problems.
Please ensure that your response is highly concise and structured, so that specific solutions can be transformed into generalizable methods.
[Optional] Here are some exemplars of the thought-template: (Choose cross-task or in-lask exemplars based on the analysis of the Core task summarization.)
Dynamic Update of Meta-Buffer After template distillation, we need to consider whether the distilled template should be updated into the meta-buffer. If we initialize an empty meta-buffer encounter problem without proper nhought-tmmllate the distilled soough-tteplater will be directly stored in the meta-buffer. we solve problem with a retrieved thought-template, new insights may arise during the instantiation of certain thought-template. Therefore, to avoid the redundancy informative thoughts, we will calculate the
of the meta-buffer while maintaining similarity between the embedding vectors of Draee and {Dr, Y and update the meta-buffer with the following rule:
Max Max(Sim(f
(D(r) < 8.
(5)
Otherwise, it means the meta-buffer has already possessed the necessary knowledge to solve this task and does not need to perform the update. Our dynamic update strategy effectively reduces the computational burden of template retrieval while ensuring the lightweight property of our meta-buffer. We further conduct ablation study to analyze it in Section 4 and Appendix A.
Experiments
Datasets and Tasks To evaluate the efficacy of our proposed Buffer of Thoughts and compare with previous methods, consider diverse set of tasks and datasets that require varying degrees of mathematical and algorithmic reasoning, domain -sseiff knowledge, and literary creativity: (a).
The Game of 24 from ToT [14], where the objective is to form an arithmetic expression that equals 24 using each of four given numbers exactly once; (b). Three BIG-Bench Hard (BBH) [35] tasks: Geometric Shapes, Multi-Step Arithmetic Two, and Word Sorting: (c). Three reasoning tasks directly obtained from the BIG-Bench suite [50]: Checkmate-in-One Penguins- -wheee the task is answer questions about penguins' attributes based on given table and additional natural language information, and unttennesttanding--- descriptions, performing arithmetic operations on dates, and utilizing global knowledge such as the number of days in February; (d). Python Programming Puzzles (P3) [51, 52], collection of challenging programming puzzles written in Python with varying difficulty levels; (e). Multilingual
task that involves inferring dates from natural language
Grade School Math (MGSM) [33]. multilingual version of the GSM8K dataset [53] featuring translations of a subset of examples into ten typologically diverse languages, including Bengali, Japanese, and Swahili; (f). Shakespearean Sonnet Writing from meta-prompting [15], a novel task where the goal is to write a sonnet following the strict rhyme scheme "ABAB CDCD EFEF GG" and incorporating three provided words verbatim.

---

Table I: Comparing BoT with previous methods across various tasks. We denote the best score in blue and the second-best score in green Our BoT significantly outperforms other methods on all tasks, especially on general reasoning problems.
Standard
Single-Query
Multi-Query
Task
BoI (Ours)
[3]
GPT4+CoT [8]
Expert [9]
PAL [10]
Tof [14]
[o7
Meta Prompting [15]
Game of 24
3.0
11.0
3.0
64.0
74,0
73.2
67.0
82.4
MGSM (avg)
84.4
85.5
85.0
72.0
86.4
87.00
84.8
89.2
Multi-Step Arithmetic
84.0
83.2
83.2
87.4
88.2
89.2
90.0
99.8
WordSorting
80.4
83.6
85.2
93.2
96.4
98.4
99.6
100.0
Python Puzzles
31.1
36.3
33.8
47.3
43.5
41.9
45.8
52.4
Geomettic Shapes
52.6
69.2
55.2
51.2
56.8
54.2
78.2
93.6
Checkmateein--Oec
36.4
32.8
39. 6
10.8
49.2
51.4
57.2
86.4
Date Understanding
68.4
69.6
68.4
76.2
78.6
77.4
79.2
88.2
Penguins
73.6
75.8
85.4
Sonnet Writing
62.0
71.2
74.0
36.2
68.4
62.8
79.6
80.0
Implementation and Baselines For the fair comparisons with previous methods, we use GPT-4 as the base model of our BoT, including the main experiment and the ablation study. We also use Llama3-8B and Llama3-70B in analysis part on NVIDIA A100-PCIE-40GB GPU. We compare our Buffer of Thoughts with the following prompting methods: 1. Standard Prompting: This is our most basic baseline, where LLM is asked generate response directly from the input query, without any specific guiding onput-output examples or additional instructions beyond the task description included in the query.
This includes Zero-shot CoT [8] and PAL [10], which use the LLM to analyze natural language problems and generate intermediate reasoning steps. We also include Expert Prompting [9], which creates an expert identity tailored to the specific context of the input query, and then integrates this expert profile into the input to generate well-informed response.
Single-query Method:
Multi-query Method: This includes ToT [14] and GoT [17], which enable LLMs to make deliberate decisions by considering multiple reasoning paths and self-evaluating choices to determine the next course of action. These methods also allow for looking ahead or backtracking when necessary to make global decisions. Additionally, we include Meta Prompting [15], which employs an effective
scaffolding technique designed enhance the functionality of LLMs.
4.1 BoT Achieves Better Accuracy, Efficiency and Robustness
Reasoning Accuracy As shown in Table 1, our BoT consistently outperforms all previous prompt- ing methods across multiple kinds of challenging benchmarks, particularly demonstrated in compli- cated reasoning tasks such as Game of 24 and Checkmate-in-One Taking GPT-4 as a baseline, our method achieves an astonishing 79.4% accuracy improvement in Game of 24, and compared to ToT. which has good performance on this task, we also achieve an 8.4% accuracy improvement. What's more, compared to recent Meta-prompping method [15], we see significant accuracy improvements: 23 % on Game of 24, 20% on Geometric Shapes and 51 % on Checkmate-in-Onu methods need complex, iterative, and heuristic search strategies to address these problems on cuse-by-case basis. Conversely, our BoT leverages the historical insights and informative guidelines from shought-templater and further adaptively instantiate more optimal reasoning structure for addressing these complex problems.
Existing
Reasoning Efficiency In addition significant improvements in accuracy, multi-query method, our BoT can achieve comparable reasoning time to single-query method across various tasks, while being considerably less than conventional multi-query method like ToT [14] as shown in Figure For example, in Game of 24, both single-query and multi-query methods necessitate iterative and heuristic searches to identify feasible solutions. This process is particularly time-consuming and inefficient, especially for the multi-query method, which involves conducting multi-query search and backtrace phases. In contrast, our BoT directly retrieves a thought-template in code format, thus a program is instantiated to traverse combinations of numbers and symbols, thereby eliminating the need to build the reasoning structure from scratch. This allows for solving the problem with just one query after invoking the problem-distiller significantly reducing the time required for complex reasoning. Notably, our BoT requires only 12 2% of the cost of multi-query methods (e.g., tree of thoughts and meta- prompproopting) on average.
7

---

Comparison of the inference time
Expert
ToT
Meta- Pita-prompping
Ours
9.08
9
8.73
8.34
0.43
9.04
8
7
6.59
Logarithmic time
6
5.5
5.21
5
4.64
4,81
4.81
4
3
2
1
Game of 24
MGSM
Checkmate-ine
Figure 3: Comparison logarithmic inference time between our Buffer of Thoughts and GPT4 [3].
GPT4+CoT [8]. peppertpromptiigg [9]. PAL [10], ToT [14] across different benchmarks.
Success rate
GPT4
Expert
PAL
ToT
Ours
100
96.8
95.4
95.15
85
84.57
78
78
71.02
71.12
70
67:13
Average accuracy
60
50
48 2
40
36
36
30
27
20
Game of 24
MGSM
Checkmate-in
Average
Figure 4: Comparison of reasoning robustness between Buffer of Thoughts and GPT4 [3], [9], PAL [10], ToT [14] across different benchmarks.
GPT4+CoT [8],
evaluation metric: success rate, which is used to assess the reasoning robustness. We randomly sample 1000 examples from various benchmarks
Reasoning Robustness To better evaluate BoT, devise
test subset and evaluate different methods this subset. As shown in Figure 4, we repeat this evaluation process 10 times and take the average accuracy as the success rate different methods on each benchmark. Compared with other methods, our BoT consistently maintains higher success rate across various tasks, surpassing the second-bess by 10% in average success rate. We attribute our outstanding robustness to the great generalization ability of distilled thought-templas during reasoning across different tasks. By offering high-level thought from the suitable thought-templates, the stability our method across different tasks is greatly enhanced.
5
Model Analysis
Distribution Analysis of ThoouugttTemplltes As depicted in the left figure of Figure 5, we choose six different benchmarks, each sampled with 100 distinct tasks. We update the meta-buffer from scratch, and after completing all sampled tasks, we display the number of derived thought-templates. We can observe that our BoT generates a greater number of touulat-eepplaes in the MGSM tasks that contain more diverse scenarios. In tasks with relatively simple requirements, such as Checkmate- in-One and Penguins, BoT produces more fixed thought-templates tailored for those specific issues. The distribution of templates indicates that our BoT can effectively discover appropriate thought templates for different benchmarks.
8

---

Template distribution across different tasks
Average time distribution each part our
Checkomate-ii One
Pytionn Puzcles
Sonnet wildig
butfer
problem-
15.8
37
Table of Penguims
mete tretf-buffer
Dace understand..
8.9
14
writing
MGSM
Tatle of Penguins
Fassoner
78
Date understanding
MGSV
Python Puzzles
527
meta bufter
reesoner
Checkmate--re Inn-Oee
buffer-mnaggee
Figure 5: Distribution Analysis of Thought-Templates and Time. Left: Distribution Analysis of Thought-Templates Right: Time Distribution BoT.
Distribution Analysis of Time Cost As illustrated in Figure 5, we measured the average time cost for each component of BoT's reasoning framework across different tasks. The time required for distilling task information and template retrieval is relatively short, whereas instantiated reasoning takes longer. Overall, considering the complexity of different components, our BoT achieves relatively balanced distribution time cost, demonstrating the efficiency of our BoT framework.
Better Trade-off between Model Size and Performance As depicted in Figure 6, Game of 24, word list sorting and Checkmate-in-Ore Llama3-8B and Llama-70B models [6] may result in poor outcomes. However, equipped with our BoT, both models demonstrate substantial accuracy improvement. Notably, BoT+Llama3-8 has the potential to surpass single Llama3-70B model. Our BoT enables smaller models to exhibit the capabilities that approximate even surpass larger models, significantly bridging the gap between their reasoning abilities. Furthermore, greatly diminishes the inference cost required by large language models when tackling complex problems.
Trade-off between model size and performance
-lama-a-----0000
BoT+Liama 3-88
Llama3-708
Llama3-88
75.6
56.7
Checkmate-in-Omm
15
92.3
Word list sorting
79
48.4
78.4
73.4
Game of 24
2.4
1.2
20
30
60
90
130
Accuracy
Figure 6: We evaluate the trade-off between model size and performance with Llama3-8B and Llama3-70B models on three challenging benchmarks.
Template The success of the proposed approach critically depends on the quality of the automatically induced template. While this previous experiments has shown promising empirical performance on downstream tasks, it remains unclear how good the templates themselves are. Thus we make comparison between the automatically generated task templates with manually prepared templates for more complex reasoning tasks on MATH dataset [54], with randomly sampled 500 problems. From the results, we can find that our automatic template boosts the reasoning ability of LLMs, demonstrating its generalization ability.
Quanlity of Automatically-Induee
Impact Buffer-Manager
We further conduct ablation study on our buffer-manager, where we divide the entire process into four rounds. In each round, randomly sample 50 questions from

---

Table 2: Compare our automatic thought templates with manual ones with GPT-3.5 on MATH.
Accuracy
Manual Template
Automatic Template
Automatic Template (after accumulation)
MATH-500
52.8%
73.4%
78.4%
each benchmark and conduct reasoning. In the subsequent round, we continue to randomly sample another 50 questions from each benchmark. As depicted in Figure 7, with the increase of the number of rounds, the model with the buffer-manager continually expands the meta- buffer while also utilizing the thought-templates obtained from previously solved problems similar problems. Therefore, we can observe that the accuracy of BoTT steadily improves with each round. In contrast, the model without the buffer-manager fails to exhibit upward trend.
help addressing subsequent
Additionally, we also demonstrate the superiority of our buffer-manager on the reasoning efficiency as depicted in Figure 10. when the number of rounds increases, the model with the buffer-manage will experience continual improvement in reasoning efficiency. This because, with the continual expansion of the meta- neta-buffer, the likelihood of retrieving suitable temphtttemplates also increases. Consequently, models can avoid constructing reasoning structures from scratch, thereby enhancing the inference efficiency accordingly. More ablation study can be found in Appendix A.
Ablation study of buffer-manager Accuracy
100
90
87.4
88.5
78.5
80
70
Accuracy (4)
60
56.8
574
50
52.8
53.8
40
30
20
Round 1
Round 2
Round 3
Round 4
BoT+GPT4
BoT+ (w/o buffer-manager)
Figure 7: We conduct ablation study on buffer-manager regarding reasoning accuracy across four tasks, employing Llama3-70B and GPT-4 as the base models.
6 Discussion
In this work, we introduce Buffer of Thoughts, novel beffered reasoning framework that employs LLMs to utilize pre-accumulated experiences and methodologies from prior tasks for progressively raising the LLM's reasoning capacity. our BoT brings out set of future directions: (1). integrating external resources with BoT to build a open-domain system like agent models [55, (2). making
the distillation of thought-templates optimizable, which may significantly enhance their template qualities for more complex tasks. (3). incorporating BoT into LLM training for eliciting more fine-grained and accurate reasoning process, like SuperCorrect [57].
Acknowledgements
This work supported by National Natural Science Foundation of China (U23B2048, U22B2037), Beijing Municipal Science and Technology Project (Z231 1000 1030000)))) research grant No. SH- 2024JK29, the Fund of Kunpeng and Ascend Center of Excellence (Peking University), and High- performance Computing Platform of Peking University.
10

---

References
[1] T. Brown, B. Mann. N. Ryder. M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., "Language models are few-shot learners"" Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
[2] R. Anil, A. M. Dai, O. Firat, Johnson, D. Lepikhin, A. Passos, S. Shakeri, Taropa, Bailey, Z. Chen, et al., "Palm 2 technical report." arXiv preprint arXiv:2305 10403, 2023.
[3] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., "Gpt-4 technical report," arXiv preprint 2tii:2303:087774 2023.
[4] Z. Du, Y. Qian, Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang, "Glm: General language model pretraining with autoregressive blank infilling." in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 320-335, 2022.
[5] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. Casas, E. B. Hanna, F. Bressand, et al., "Mixtral of experts," arXiv preprint arXiv:2401 040088 2024.
[6] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro,
F. Azhar, et al., "Llama: Open and efficient foundation language models," arXiv preprint 23022 2023,
[7] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi. Y. Babaei. N. Bashlykov, S. Batra, P. Bhar- gava, S. Bhosale, et al., "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.099266 2023.
[8] J. Wei, X. Wang. D. Schuurmans, M. Bosma, F. Xia, E. Chi. Q V. Le, D. Zhou, et al., "Chain-of of-thhggh prompting elicits reasoning in large language models," Advances neural information processing systems, vol. 35, pp. 24824 24837, 2022.
[9] B. Xu, A. Yang, Lin, Q Wang, C. Zhou, Y. Zhang, and Z. Mao, "Exxertpromptinn: Instructing large language models to be distinguished experts," arXiv preprint arXiv:2305. /4688, 2023.
[10] L. Gao, A. Madaan, Zhou, U. Alon, Liu, Y. Yang, J. Callan, and Neubig, "Pal: Program- aided language models," in International Conference on Machine Learning, pp. 10764-10799, PMLR, 2023.
[11] X. Wang. J. Wei, Schuurmans, Q. V. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, "Self- consistency improves chain of thought reasoning in language models," in The Eleventh International Conference on Learning Representations, 2022.
[12] M. Yasunaga, X. Chen, Y. Li. P. Pasupat, J. Leskovec, P. Liang, E. H. Chi, and D. Zhou, "Large language models as analogical reasoners," International Conference on Learning Representations, 2024.
[13] Z. Zhang, A. Zhang, M. Li, and A. Smola, "Automatic chain of thought prompting in large language models," in The Eleventh International Conference on Learning Representations, 2022.
[14] S. Yao, D. Yu, J. Zhao, I. Shafran, T. Grifliths, Y. Cao, and K. Narasimhan, "Tree of thoughts: Deliberate problem solving with large language models," Advances in Neural Information Processing Systems, vol. 36, 2024.
[15] M. Suzgun and A. T. Kalai, "Meta- prompting: Enhancing language models with agsk-agnostic scaffolding." arXiv preprint arXiv:2401. 12954, 2024.
[16] D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. V. et al. "Least-to-man mostt prompting enables complex reasoning in large language models," in The Eleventh International Conference on Learning Representations. 2022.
[17] M. Besta, N. Blach, A. Kubicek, R. Gerstenberger, M. Podstawski, L. Gianinazzi, J. Gajda, T. Lehmann, H. Niewiadomski, Nyezyk, et al., "Graph of thoughts: Solving elaborate problems with large language models," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. -1790, 2024.
[18] A. Asai, S. Min. Zhong, and D. Chen, "Retrieval-base based language models and applications." in Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume 6: Tutorial Abstracts), pp. 41-46, 2023.
[19] G. Mialon, R. Dessi, M. Lomeli, C. Nalmpantis, R. Pasunuru, R. Raileanu, B. Roziere, T. Schick. J. Dwivedi- Yu, A. Celikyilmaz, et al., "Augmented language models: a survey," Transactions on Machine Learning Research, 2023.
11

---

[20] W. Shi. S. Min. M. Yasunaga, M. Seo, R. James, M. Lewis. L. Zettlemoyer. and W.-L Yih, "Replug: Retrieval-angmented black-box language models," arXiv preprint wrXv:2331.12662. 2023.
[21] Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai. J. Sun, and H. Wang. "Retrieval-augmee -ungmeeeed generation for large language models: A survey." arXiv preprint arXiv:2312.109) 2023.
[22] P. Zhao, H. Zhang, Q. Yu, Z. Wang, Y. Geng, F. Fu, L. Yang. W. Zhang, and B. Cui, RRetrieall-augeennn generation for ai ai-generated content: A survey." arXiv preprint arXiv:2402. 19473, 2024.
[23] S. Borgeaud, A. Mensch, J. Hoffimann. T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J.-B. Lespiau, B. Damoc. A. Clark, al.. "Improving language models by retrieving from trillions of tokens," in International conference on machine learning, pp. 2206-2246 2240, PMLR, 2022.
[24] M. Yasunaga, A. Aghajanyan, W. Shi, R. James, J. Leskovec, P. Liang. M. Lewis, L. Zettlemoyer, and W.-T. Yih, "Retrieval- -ugmentte multimodal language modeling," " in International Conference on Machine Learning, pp. 39755-397699 PMIRR 2023.
[25] G. Izacard, Lewis, M. Lomeli, L. Hosseini, F. Petroni, T. Schick, Dwivedi- Yu, A. Joulin, S. Riedel, and E. Grave, "Atlas: Few-shot learning with retrieval augmented language models," Journal of Machine Learning Research, vol. 24, no. 251, pp. 1-43, 2023.
[26] Z. Wang, W. Nic, Z. Qiao, C. Xiao, R. Baraniuk, and A. Anandkumar, "Retrieval-basee controllable molecule generation," in The Eleventh International Conference on Learning Representations, 2022.
[27] L.. Yang. Z. Huang, X. Zhou, M. Xu, W. Zhang, Y. Wang. X. Zheng, W. Yang. R. O. Dror, S. Hong, et al., "Prompt-hased 3d molecular diffusion models for sauuturee-bsed drug design," " 2023.
[28] T. Kojima, Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, "Large language models are zero-shot reasoners," Advances in neural information processing systems, vol. 35, pp. 22199-22213, 2022.
[29] O. Press, M. Zhang, S. Min, I.. Schmidt, N. A. Smith, and M. Lewis, "Measuring and narrowing the compositionnaliy gap in language models," in Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5687-5711, 2023.
[30] S. Arora, A. Narayan, M. F. Chen, L. Orr, N. Guha, K. Bhatia, 1. Chami, and C. Re, "Ask me anything: A simple strategy for prompting language models," in The Eleventh International Conference on Learning Representations, 2022.
[31] T. Khot, H. Trivedi, M. Finlayson, Y. Fu, K. Richardson, P. Clark, and A. Sabharwal, "Decomposed prompting: A modular approach for solving complex tasks," in The Eleventh International Conference on Learning Representations, 2022.
[32] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al., "Emergent abilities of large language models," Transactions on Machine Learning Research, 2022.
[33] F. Shi, M. Suzgun, M. Freitag, X. Wang, S. Srivats, S. Vosoughi, H. W. Chung, Y. Tay, S. Ruder, D. Zhou, et al., "Lannuugee models are multilingual chain-of-thoouht reasoners," " in The Fleventh International Conference on Learning Representations, 2022.
[34] Y. Fu, H. Peng, A. Sabharwal, P. Clark, and T. Khot, "Complexity-based prompting for multi-step reasoning," in The Eleventh International Conference on Learning Representations, 2022.
[35] M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. Le, Chi, D. Zhou, et al., "Challenging big-bench tasks and whether chain-of-thought can solve them," in Findings of the Association for Computational Linguisticx; ACL 2023, pp. 13003-13051, 2023.
[36] H. S. Zheng, S. Mishra, X. Chen, H.-T. Cheng, E. H. Chi, Q. V. Le, and D. Zhou, "Take a step back: Evoking reasoning via abstraction in large language models." arXiv preprint arXiv:2310.06117, 2023.
[37] P. Zhou, J. Pujara, X. Ren, X. Chen, H.TT. Cheng, Q. V. Le, E. H. Chi, D. Zhou, S. Mishra, and H. S. Zheng, discovis: Large language models self-compose reasoning structures." arXiv preprint arXiv:2402.0366 03620. 2024.
[38] W. Chen, X. Ma, X. Wang, and W. W. Cohen, "Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks," Transactions on Machine Learning Research, 2023.
[39] X. Ning. Z. Lin, Z. Zhou, Z. Wang, H. Yang, and Y. Wang, "Skeleton-Or can do parallel decoding," in The Twelfih International Conference Learning Representations, 2023.
ofaiooon-oo-uee of thought: Large language models
12

---

[40] Y. Zhang, "Meta prompting for agi systems," arXiv preprint arXiv:2311.11452 2023.
[41] Chen, R. Xu, Z. Fu, W. Shi, Z. Li, X. Zhang, C. Sun, I., Li, Y. Xiao, and H. Zhou, "F-kar: A henchmark for rationalizing natural language analogical reasoning." in Findings of the Association for Computational Linguistics: ACL 2022. pp. 3941-3955, 2022.
[42] O. Sultan and D. Shahaf, "Life is a cireus and we are the clowns: Automatically finding analogies between situations and processes." in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3547-3562, 2022.
[43] N. Zhang, L. Li, Chen, X. Liang, S. Deng, and H. Chen, "Multimodal analogical reasoning over knowledge graphs," in The Eleventh International Conference on Learning Representations, 2022.
[44] B. Bhavya, J. Xiong, and C. Zhai, "Analogy generation by prompting large language models: A case study of instructgpt." in Proceedings of the 15th International Conference on Natural Language Generation, pp. 298- 312, 2022.
[45] B. Bhavya, J. Xiong, and C. Zhai, "Cam: A large language model- creative analogy mining frame- work," in Proceedings of the ACM Web Conference 2023, pp. 3903-3914, 2023.
[46] Z. Zhang, A. Zhang, M. Li, and A. Smola, "Automatic chain of thought prompting in large language models," in The Eleventh International Conference on Learning Representations, 2022.
[47] T. Webb, K. J. Holyoak, and H. Lu, "Emergent analogical reasoning in large language models," Nature Human Behaviour, vol. 7, no. 9, pp. 1526-1541, 2023.
[48] J. Yu, R. He, and Z. Ying. "Thought propagation: An analogical approach to complex reasoning with large language models," in International Conference on Learning Representations, 2024.
[49] T. Feng, P. Han, G. Lin, G. Liu, and J. You, "Thought-rettiever: Don't just retrieve raw data, retrieve thoughts," ICLR 2024 Workshop: How Far Are We From AGI.
[50] B. bench authors, "Beyond the imitation game: Quantifying and extrapolating the capabilities of language models," Transactions on Machine Learning Research, 2023.
[51] T. Schuster, A. Kalyan, A. Polozov, and A. Kalai, "Programming puzzles," in Thirty-fith Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
[52] A. K. Patrick Haluptzok, Matthew Bowers, "Language models can teach themselves to program better," in Eleventh International Conference on Learning Representations (ICLR), 2023.
[53] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, "Training verifiers solve math word problems." arXiv preprint 1.Xiv:2110.14668 2021.
[54] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, "Measuring mathematical problem solving with the math dataset," arXiv preprint arXiv:2103.03874 2021.
[55] G. Chen, S. Dong, Y. Shu, G. Zhang, J. Sesay, B. E. Karlsson, J. Fu, and Y. Shi, "Autoagents: A framework for automatic agent generation," arXiv preprint arXiv:2309.1 230.2728,, 2023.
[56] Q. Wu, G. Bansal, J. Zhang, Y. Wu, S. Zhang, E. Zhu, B. Li, L.. Jiang, X. Zhang, and C. Wang. "Au- togen: Enabling next-gen Ilm applications via multi-agent conversation framework," arXiv preprint 1rXiv:2308.08155 2023.
[57] L. Yang, Z. Yu, T. Zhang, M. Xu, J. E. Gonzalez, B. Cui, and Yan, "Supercorrect: Supervising and correcting language models with error-driven insights," arXiv preprint arXiv:2410.0900 2024.
[58] Y. Xu, H. Cao. W. Du, and W. Wang, "A survey of cross-lingual sentiment analysis: Methodologies, models and evaluations." " Data Science and Engineering, vol. no. 3, pp. 279-299, 2022.
[59] S. R. Khope and S. Elias, "Critical correlation of predictors for an efficient risk prediction framework of icu patient using correlation and transformation of mimic-iii dataset." Data Science and Engineering. vol. 7, no. 1, pp. 71 71-86, 2022.
[60] L. Guo, D. Chen, and K. Jia, "Knowledge transferred adaptive filter pruning for enn compression and acceleration." " Science China. Information Sciences, vol. 65, no. 12, p. 229101, 2022.
[61] Z. Zhang, Y. Zhang, D. Guo, S. Zhao, and X. Zhu, coammunicatioo-effiicee federated continual learning for distributed learning system with non onn-idd data," Science China Information Sciences, vol. 66. no. 2. p. 122102, 2023.
13

---

A More Ablation Studies
A.1 Impact of Cuablem-Distilles
As illustrated in Figure 8, when the problem-distiller is disabled, both Llama3-70B and GPT-4 experience certain degree of accuracy decline. More complex problems, such as Game of 24 and Checkmate-in-One, show a more significant accuracy reduction, whereas relatively simpler problems like word list sorting and MGSM exhibit smaller decreases. This is because LLMs can more easily extract key information in simpler tasks, making the impact of the problem-distillee less noticeable. In contrast, extracting key information and potential constraints in complex problems is more challenging, making the role of our problem-distiee more prominent, thereby explaining the differences depicted in the figure.
Ablation study of problem- rroblem-distiller
--TT-lama----- (wic problem-dstilee
36T-Liama-3-700
BOT-GPT-4 (w/o problem-ditiller
BoT+GPT-4
100
$9.6
09.5
92.
86.8 67 R02
62.4
78.4
75.6
7122
70
64.3
Accuracy (N)
40
30
0
Game of
Word list sorting
Checkmate-in One
MGSM
Figure 8: We conduct ablation study on problem-distilles across four benchmarks, employing Llama3- 70B and GPT-4 as the base models.
A.2 Impact of Meta-Buffer
As illustrated in Figure 9, when the meta-buffer is disabled, both Llama3-70B and GPT-4 models exhibit a noticeable decline in performance, particularly benchmarks requiring complex reasoning. such as Game of 24 and Checkmate-in-One This further underscores the superiority of our meta- buffer in addressing complex problems. We would extend such mechanism to more scenarios [57-61].
Ablation study of meta-buffer
Uame-3-708 (w/ meta-bulfer)
Lame--3-7
BoT-GPT-4 (woo meta-bult)
BOT+
90.4
92.3
90
39.4
80
78.4
82.4
01.
75.6
79.6
65.6
Accuracc (%)
56.7
60
27.4
20
10
Game of 24
Word list sorting
Checkmate-in- -nne
MGSM
Figure 9: We conduct ablation study on meta-buffer across four benchmarks, employing Llama3-70B and GPT-4 as the base models.
14

---

Ablation study of suffer-manager Time
350
308
300
Average inference time per problem (s)
306
297
250
205
200
150
128
100
78.5
50
Round
Round
Round
Round
BoT+GPT4
BoT+GPT4 (w/o buffer (antorrmaaaerr
Figure 10: We conduct ablation study buffer-manager regarding reasoning efficiency across four tasks, employing Llama3-70B and GPT-4 as the base models.
B Additional Method Details
B.1 Six Kinds of Detailed Thought-Templans
1. Text Comprehension
Task Description:
The task involves analyzing table with various attributes of penguins, such as name, age, height, and weight, and answering questions about these attributes. The table may be updated with new entries, and additional context or comparisons may be provided in natural language.
Solution Description:
To accurately answer questions about the penguins' attributes, one must be able to interpret the data presented in tabular form, understand any additional information provided natural language, and apply logical reasoning to identify the correct attribute based on the question asked.
Thought Template:
Step Parse the initial table, extracting the header information and each penguin's 's attributes into a structured format (e.g., list of dictionaries).
Step 2: Read and integrate any additional natural language information that updates or adds to the table, ensuring the data remains consistent. Step Identify the attribute in question (e.g., oldest penguin, heaviest penguin) and the corresponding column in the table.
Step 4: Apply logical reasoning to compare the relevant attribute across all entries to find the correct answer (c.g., the highest age for the oldest penguin). Step Select the answer from the provided options that matches the result of the logical comparison.
15

---

2. Creative Language Generation
Task Description:
The task is to generate a sonnet that adheres to the traditional English sonnet rhyme scheme of "ABAB CDCD EFEF GG" and includes three specific words verbatim in the text.
Solution Description:
Writing sonnet involves crafting 14 lines of poetry that follow specific rhyme pattern. The lines are typically in iambic pentameter, though flexibility rhythm can allowed for creative reasons. The given rhyme scheme dictates the end sounds of each line, ensuring structured poetic form. Incorporating the three provided words verbatim requires strategic placement within the lines maintain the poem 's coherence and thematic unity. Thought Template:
Step Identify the three words that must be included in the sonnet.
Step 2: Understand the rhyme scheme "ABAB CDCD EFEF GG" and prepare a list of rhyming words that could be used.
Step 3: Develop theme or story for the sonnet that can naturally incorporate the three provided words.
Step 4: Begin drafting the sonnet by writing the first quatrain (four lines) following the "ABAB" rhyme scheme, ensuring one or more of the provided words are included.
Step 5: Continue with the second quatrain "CDCD," the third quatrain "EFEF," and finally the closing couplet "GG," each time incorporating the provided words as needed. Step 6: Review the sonnet for coherence, flow, and adherence to the rhyme scheme, making adjustments as necessary.
Common Sense Reasoning
Task Description:
Given specific date and an event, such a holiday or historical event, determine the following date.
Solution Description:
To determine the next date, we need to consider the structure of the calendar, the number of days in each month, and whether it's leap year. Typically, the number of days in month is fixed, except February may vary due to leap years. The next day in a year is usually the date increased by one day unless it's the end of the month, then the next day will be the first day of the following month. For the end of the year, the next day will be January 1st of the following year.
Thought Template:
Step Identify the given date's month and day number.
Step 2: Check if it's the end of the month; if so, confirm the start date of the next month.
Step 3: it's not the end of the month, simply add one to the day number.
Step 4: Pay special attention to the end of the year, ensuring the year increments.
16

---

4. Code Programming
Task Description:
When given list of numbers, try to utilize 4 basic mathematical operations (+- to get target number.
Thought Template:
Listing Python template
from itertools import permutations, product
def inffor_operratio.. b, operation): Define the operation logic (e.g., addition, subtraction, etc.).
pass
def evaluate, evaluate_sequence (sequence, operations):
# Apply operations to the sequence and check if the result meets the criteria. pass
def generate_ combinations (elements, operations):
# Generate all possible combinations of elements and operations.
paas
def
(sequence, operations);
# Format the sequence and operations into human readable string.
paas
def f find_soluta solution _input_elementa, target_reeult)
# Data Input Handling
Validate and preprocess input data if necessary.
Core Algorithm Logic
for sequence in permutations (eppuutelements). for aperation_combinate in sonbrato.cominnationns sequence, operations): try:
if evaluate__equence (sequence,
-- target_result:
# Data Output Formatting
return format (sequencce,,,,,,,,,,, operation__cmmiiattoa))
except Exception as e: Error Handling
Handle specific exceptions that may occur during evaluation.
continue
If solution is found after all iterations, return default message.
# return No solution found message return
Example usage:
input_elements [1, 7, 10, 3] target_resul' = 24 print (find_ on (input_ed elements, target_result))
17

---

5. Application Scheduling
Task Description:
Given some Chess moves SAN, update the chess board state.
Listing 2: Python template
import chess def find find__hhcckil www._hhhhhceemmoeeee (moves_san):
# Initialize new chess board board Thess.Boarde
Apply the moves to the board for mave_san in moves_san:
# Remove move numbers and periods (e.g., "1." "2.") if watpreesaansspp((((e.ee.eee
move_san move_san.split(( # Skip empty strings resulting from the removal if move_san:
move
in SAN format the board move b parse_ (move_san)
Apply each
board. push (move)
Generate all possible legal moves from the current position
for move in board. leg.lllgllleeeeeee
# Make the move on copy of the board to test the result
board_copy board. copy ()
board_copy. push(move)
# Check if the move results in checkmate
if board_copy. wtt.cco..sscchhhleeeeee Return is_checkmate format
move that results checkmate SAN
return board. san (move)
# return No solution found message
return
#Example usage:
input ......."
Check input format and transform the input into legal format Remove move numbers periods (e.g., "1." "2.") checkmate_move move find_checkmate_mn (moves_san) print (checkmate_nove)
18

---

6. Mathematical Reasoning
Task Description:
Solve an quadratic equation of the form az² + bæ + c = 0 considering any situations.
Solution Description:
To solve any quadratic equation of the form ax + bu + c = 0, we can follow a general approach based on the method described. Here is the structured template for solving such equations:
Thought Template:
Step I: Calculate the Discriminant
Coompute the discriminant using the formula 62 4ac.
Step 2: Determine the Nature of the Roots
If D 0, the equation has two distinct real roots.
If D = 0, the equation has exactly one real root (also known as repeated or double root).
If D < O, the equation has two complex roots.
Step 3: Compute the Roots For D 2 0, calculate the roots using the formula æ = =btVD
2a
For < 0, calculate the real and imaginary parts of the complex roots using the formula
J-D where is the imaginary unit.
20
2a
B.2 Prompt for Problem Distiller
[Problem Distilleri:
As a highly professional and intelligent expert in information distillation, you excel extracting essential information to solve problems from user input queries. You adeptly transform this extracted information into suitable format based the respective type of the issue.
Please categorize and extract the crucial information required solve the problem from the user's input query, the distilled information should include.
Key information:
Values and information of key variables extracted from user input, which will be handed over to the respective expert for task resolution, ensuring all essential information required to solve the problem is provided.
Restrictions:
The objective of the problem and corresponding constraints.
Distilled task:
Extend the problem based on I and 2, summarize meta problem that address the user query and handle more input and output variations. Incorporate the real-world scenario the extended problem along with the types of key variables and information constraints from the original problem to restrict the key variables the extended problem. After that, use the user query input key information input solve the problem example.
19

---

B.3 Prompt for Instantiated Reasoning
[Meta Reasoner]
You are a Meta Reasoner who are extremely knowledgeable in all kinds of fields including Computer Science, Math, Physies, Literature, History, Chemistry, Logical reasoning, Culture, Language.... You are also able to find different high-level thought for different tasks. Here are three reasoning sturctures: i) Prompt-based structure: has a good performance when dealing with problems like Common Sense Reasoning, Application Scheduling
ii) Procedure-based structure It has good performance when dealing with creative tasks like Creative Language Generation, and Text Comprehension
iii) Programming-basee has a good performance when dealing with Mathematical Reasoning and Code Program- ming, it can also transform solved efficiently.
problems into programming problem which could be
(Reasoning instantiation)
Your task is:
1. Deliberately consider the context and the problem within the distilled respond from problem distiller and use your understanding of the question within the distilled respond to find domain expert who are suitable to solve the problem.
Consider the distilled information, choose one reasoning structures for the problem. If the thought-template is provided, directly follow the thought-template instantiate for the given problem.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to the abstract in the main paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to the Section the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer the Section 4 in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification: Please refer to the Section the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeececceccccccct /ttpa//gguiieee/eeeuuuubiiiiiiii /CodeSubmie ssionnollll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to the Section the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
23

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer the Section 4 in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: /ttp///eerrp.eeeeee//ee//e cc cttppbiicc/thiicccooeeeeeeeeeeee
didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: Please check out the paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer the ?? in our paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: ensure all papers and codebases used or relevant to this work are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
25

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "0HQRwPPq39", "reviewer": "Reviewer_hZUQ", "summary": "The paper introduces \"Buffer of Thoughts\" (BoT), a novel framework designed to improve the reasoning abilities of large language models (LLMs) by incorporating a 'thought-augmented' approach. This framework uses a component called 'meta-buffer' to store high-level thoughts—concise, distilled reasoning strategies from various problem-solving instances—which can be dynamically retrieved and adapted to new tasks to facilitate efficient reasoning. This approach significantly enhances the accuracy, efficiency, and robustness of LLMs across multiple reasoning-intensive tasks.\n1. Novel Framework: BoT innovates a thought-augmented reasoning mechanism that leverages previous problem-solving insights to aid new reasoning tasks, reducing the need for generating reasoning paths from scratch.\n2. Meta-Buffer: Introduces a storage system that holds distilled high-level thoughts, allowing for rapid adaptation to different tasks by retrieving and instantiating these thoughts as needed.\n3. Buffer Manager: A dynamic management system that updates the meta-buffer based on newly encountered tasks, continuously improving the system’s reasoning capacity.\n4. Empirical Validation: The paper reports extensive testing across 10 complex tasks, demonstrating significant performance improvements over state-of-the-art methods, including improvements of 11% on Game of 24, 20% on Geometric Shapes, and 51% on Checkmate-in-One, while also reducing the computational cost compared to traditional multi-query prompting methods.", "questions": "1. As the meta-buffer grows with more distilled thoughts, how does its size impact the retrieval time and overall performance of the model?\n2. In cases where multiple applicable thoughts could be retrieved from the meta-buffer, how does BoT prioritize or choose among conflicting reasoning strategies?\n3. Given the reliance on previously distilled thoughts, how does BoT handle situations where the foundational thoughts might be based on incorrect or outdated information?", "limitations": "One of the primary limitations highlighted in the discussion of the Buffer of Thoughts (BoT) framework is its dependency on the initial quality of the meta-buffer. The performance of BoT is contingent upon the initialization of this buffer. If initialized with a weak model, the distilled thoughts stored in the meta-buffer may not be of sufficient quality or depth to facilitate effective reasoning across diverse tasks.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Improved Reasoning Accuracy:** BoT significantly enhances the reasoning accuracy of large language models by leveraging distilled high-level thoughts, allowing the models to approach problems with a pre-formed strategy that has been proven effective across tasks.\n2. **Computational Efficiency:** The thought-augmented reasoning approach minimizes the need for complex and iterative query processes typical in multi-query prompting systems. By reusing structured thought templates, BoT reduces the computational overhead, leading to faster reasoning times.\n3. **Robustness Across Tasks:** BoT exhibits a robust performance over a range of different and challenging tasks. This is attributed to the system's ability to adapt high-level reasoning thoughts to new problems, ensuring consistent performance without the need to tailor the system for specific tasks.", "weaknesses": "1. **Dependence on Quality of Distilled Thoughts:** The effectiveness of BoT hinges significantly on the quality of the distilled thoughts stored in the meta-buffer. If the initial thoughts distilled from problem-solving processes are not sufficiently generalizable or are too simplistic, they may not provide the necessary depth for complex reasoning tasks. \n2. **Scalability and Maintenance of the Meta-Buffer:** The paper introduces a dynamic system for updating the meta-buffer but does not deeply explore the long-term scalability and maintenance challenges associated with continuously growing and updating this repository. Managing an ever-expanding set of thought templates could lead to efficiency issues or dilution of useful thoughts.\n3. **Risk of Overfitting to Distilled Thoughts:** There is a potential risk that the LLM might overfit to the specific styles or patterns of reasoning encapsulated in the thought templates, especially if these templates are derived from a limited set of problem-solving instances.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We thank Reviewer hZUQ for the positive review and valuable feedback. We are glad that the reviewer found that the proposed framework is novel, and greatly enhances the accuracy, efficiency, and robustness of LLMs across multiple reasoning tasks. Please see below for our responses to your comments.*\n\n**Q1: Dependence on Quality of Distilled Thoughts.** \n\n**A1:**\n\n1. We address this by utilizing pre-designed prompt and leveraging state-of-the-art LLMs such as GPT-4, which are capable of generating high-quality, high-level thought templates that strike a balance between abstraction and specificity. \n\n2. To demonstrate the adaptability and robustness of our method, we provide examples from two distinct datasets: GSM8K (grade school math word problems) and MATH (challenging high school math competition problems). \n\n**Example 1: GSM8K**\n\n**Question**: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?\n\n**High-level Thought Template:**\n\nStep 1: Identify the amount of each type of material required.\n\nStep 2: Calculate any dependent quantities based on the given relationships.\n\nStep 3: Sum all the quantities to find the total amount of material needed.\n\n**Example 2: MATH**\n\n**Question**: At a school, all 60 students play on at least one of three teams: Basketball, Soccer, and Mathletics. 8 students play all three sports, half the students play basketball, and the ratio of the size of the math team to the size of the basketball team to the size of the soccer team is $4:3:2$. How many students at the school play on exactly two teams?\n\n**High-level Thought Template:**\n\nGiven:\n\nTotal number of students: T\n\nNumber of students in each team: A, B, C\n\nNumber of students playing all three sports: y\n\nStep 1: Identify the total number of students T and the number of students in each team A, B, and C.\n\nStep 2: Recognize the overlapping memberships and set up the equation:\n$$\nA + B + C = T + x + 2y\n$$\n\nStep 3: Substitute the known values for A, B, C, T, and y into the equation.\n\nStep 4: Solve for x, the number of students playing exactly two sports:\n$$\nx = (A + B + C) - T - 2y\n$$\n\nThese examples illustrate that our method can adaptively extract high-quality thought templates for problems of different complexity. For simpler problems from GSM8K, the high-level thought template is straightforward, while for more complex problems from MATH, the high-level thought template is correspondingly complicated. This adaptability and generalizability showcase the robustness of our approach.\n\n\n**Q2: Risk of Overfitting to Distilled Thoughts.**\n\n**A2:** \n\n1. Our thought templates are designed to be high-level and abstract, which do not contain specific reasoning details from previous problems. This abstraction ensures the thought templates provide general guidance that would not cause overfitting problems of LLMs.\n\n2. In scenarios where the meta-buffer contains thought templates derived from a limited set of problem-solving instances that differ significantly from the current problems, we reinstantiate the reasoning process using our manually pre-designed coarse-grained thought templates, as mentioned in Line 450. This approach avoids applying an incorrect type of thought template from the meta-buffer, thereby further mitigating the risk of overfitting.\n\n3. As we mentioned above, our approach ensures that the model retains the flexibility needed to handle a wide range of tasks effectively while maintain adaptability and robustness across various problems.\n\n\n**Q3: Size impact on retrieval time and overall performance. Concern about the Scalability and Maintenance.**\n\n**A3:** \n\n1. Many different problems share similar solutions that could be instantiated from the same high-level thought template. Consequently, even as the meta-buffer grows with more distilled thoughts, the number of unique thought templates remains relatively small as demonstrated by the template distribution of Fig.5. This ensures that our meta-buffer is a lightweight library, and its size has little impact on retrieval time. Moreover, a bigger size of meta buffer would improve the overall performance and the reasoning efficiency of the model as in Fig.9 and Fig.10, with empirical analysis in Line 465-477.\n\n2. Many problems share the same reasoning patterns, we only need to save the high-level thought templates distilled from the solutions of various problems. This approach keeps the size of the meta-buffer relatively small, ensuring its scalability and making it easy to maintain.\n\n\n**Q4: How does BoT prioritize among conflicting reasoning strategies?**\n\n**A4:** \n\nAs illustrated in Line 134-144, for each thought template, we have a description $D_{T_i}$ that specifies the types of problems could be applied. BoT uses the embedding similarity between the distilled problem $x_d$ and $D_{T_i}$ to find the most suitable thought template, as mentioned in Eq.2. We only **choose the thought template with the highest similarity** to the current problem. This ensures that the chosen template aligns closely with the current problem, thereby prioritizing the most relevant reasoning strategy.\n\n**Q5: How to handle situations when thoughts are based on incorrect or outdated information?**\n\n**A5:**\n\n1. The thought templates derived from the problem-solving process are high-level and abstract, which means they **do not contain specific reasoning details or factual information**. This abstraction helps to ensure that the templates provide robust and generalizable guidance without being affected by the incorrect examples or outdated data.\n\n2. Compared to conventional methods that rely on in-context examples or specific databases, our approach leverages high-level thought templates to eliminate detailed reasoning steps. This abstraction helps to avoid potential errors that might be presented in more detailed templates, thereby enhancing the reliability and robustness of the reasoning process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hrv6N5dUld", "reviewer": "Reviewer_yiEf", "summary": "This paper proposes a new approach called Buffer of Thoughts (BoT) to improve LLMs reasoning abilities. BoT addresses this by creating a \"meta-buffer\" that stores general problem-solving “thought” templates across different tasks. When a new query is given as input, BoT retrieves a relevant thought from the meta-buffer and tailors it to the specific situation. The method also includes a \"buffer-manager\" to keep the meta-buffer up-to-date and effective as the LLM encounters new challenges for scalability and stability purposes. The authors perform experiments on different reasoning tasks that showed significant improvements with BoT compared to previous methods.", "questions": "see weaknesses.", "limitations": "limitations have been discussed in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The framework is clearly explained and the paper is easy to follow\n- The empirical evaluation is thorough\n- The improvement for some tasks is notable.\n- The idea of a library of thoughts templates is sound and novel", "weaknesses": "- BoT seems to work well for problems with existing templates in the meta-buffer. However, entirely new or unique problems might not have a relevant template for BoT to adapt, potentially hindering its ability to solve them effectively.\n- What happens in the case of unsolvable queries? Does the method allows for uncertainty based abstention?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We thank Reviewer yiEf for the positive review and valuable feedback. We are glad that the reviewer found that the proposed thought template and meta-buffer is novel, and our method achieve notable improvement across various tasks. Please see below for our responses to your comments.*\n\n**Q1: How to solve entirely new or unique problems effectively?** \n\n**A1:** \n\n1.  As discussed in Line 160-163, for entirely new or unique problems, our method provides **three pre-designed coarse-grained thought templates** (in  Appendix A.3) that can be instantiated based on the distilled task information. These templates offer a good starting point for reasoning when the relavant thought template does not exist.\n\n2.  Additionally, these coarse-grained thought templates ensure that new tasks follow appropriate inference processes. This general guidance offers LLMs greater flexibility during the reasoning process, enabling them to efficiently address new and unique problems.\n\n3.  During the cold-start reasoning process mentioned above, the buffer manager distills and summarizes new thought templates that can be used to solve similar future problems effectively. This adaptive process ensures that our method remains effective and robust, and could continually adapt to new problems.\n\n4. To give a comprehensive illustration of this reasoning process for new or unique problems, we use the Game of 24 and solving a quadratic equation as examples below:\n\n**Game of 24**\n\nInput problem: \n\nLet's play a game called 24. You'll be given four integers, and your objective is to use each number only once, combined with any of the four arithmetic operations (addition, subtraction, multiplication, and division) and parentheses, to achieve a total of 24. For example, if the input is 4, 7, 8, and 8, the output could be 7 * 8 - 4 * 8 = 24. You only need to find one feasible solution!\n\n**Problem Statement:**\n\nGiven four integers 4, 7, 8, and 8, use each number exactly once and the four arithmetic operations (addition, subtraction, multiplication, division) to form an expression that equals 24. Parentheses can be used to define the order of operations.\n\nAnalysis:\n\n**Step 1: Problem Understanding**: We need to find a combination of the four given integers and arithmetic operations that results in 24.\n\n**Step 2: Input Extraction**: The four integers are 4, 7, 8, and 8.\n\n**Step 3: Output Requirements**: The output should be a valid mathematical expression that equals 24.\n\n**Step 4: Constraints**: \n\n- Each number must be used exactly once.\n- Only the four arithmetic operations can be used.\n- Parentheses can be used to define the order of operations.\n\n**Step 5: Approach**: \n\n- Generate all permutations of the numbers.\n- Generate all possible combinations of arithmetic operations.\n- Evaluate all possible expressions formed by combining the numbers and operations with different parenthetical groupings.\n- Check if any of these expressions evaluate to 24.\n\n**Transformed Python Code:**\n\n```python\n... Due to character limit, we are unable not show the code here, you can refer to A.1.5 in our paper ...\n```\n\n**Solving a quadratic equation**\n\n**Input Problem:**\n\nSolving a quadratic equation：\n\n$2x^2-4x-6 = 0$\n\n\n**Solution:**\n\nStep 1: Identify the coefficients \\(a\\), \\(b\\), and \\(c\\) from the quadratic equation $(2x^2 - 4x - 6 = 0)$:\n\n- \\(a = 2\\)\n- \\(b = -4\\)\n- \\(c = -6\\)\n\nStep 2: Calculate the discriminant \\(D\\):\n\n$D = b^2 - 4ac = (-4)^2 - 4 \\cdot 2 \\cdot (-6) = 16 + 48 = 64$\n\nStep 3: Determine the nature of the roots:\n\n- Since \\(D = 64 > 0\\), the equation has two distinct real roots.\n\nStep 4: Compute the roots using the quadratic formula:\n\n$x = \\frac{-b \\pm \\sqrt{D}}{2a} = \\frac{-(-4) \\pm \\sqrt{64}}{2 \\cdot 2} = \\frac{4 \\pm 8}{4}$\n\n- Calculate the two roots:\n\n  $x_1 = \\frac{4 + 8}{4} = \\frac{12}{4} = 3$\n\n\n  $x_2 = \\frac{4 - 8}{4} = \\frac{-4}{4} = -1$\n\n**Answer:**\nThe solutions for the quadratic equation $(2x^2 - 4x - 6 = 0)$ are:\n\n$x_1 = 3 \\quad \\text{and} \\quad x_2 = -1$\n\nIn summary, as we mentioned in Line 160-163 and the example in Appendix A.3, we can effectively solve entirely new or unique problems. The example above further showcase the robustness of our method when encountering new tasks.\n\n**Q2: What happens in the case of unsolvable queries? Does the method allows for uncertainty based abstention?** \n\n**A2:** \n\n1. For possible unsolvable queries, our method attempts to resolve the issue during the instantiation process. Subsequently, an additional inspector reviews both the problem and the reasoning process. If any issues are detected, the unreasonable parts are identified and handed over to the reasoner for revision. If it is concluded that the problem is unsolvable, the reasoning process is terminated immediately to prevent any pollution of the meta-buffer.\n\n2. Due to the page limits, we did not discuss this scenario in our paper, and there were very few unsolvable queries in our experiments. However, we have implemented the inspector mechanism. We will add this part in the final version to provide a comprehensive overview of our approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cdP1yQ4hrS", "reviewer": "Reviewer_upCD", "summary": "This paper proposes a novel reasoning procedure for LLM, named \"buffer of thoughts\". The core idea is that for each task, first extract a template describing how the task should be solved, then store all these templates in a \"buffer\". As the LLM receives a new query, a retrieval procedure is applied to extract the most relevant task template out of the buffer. The template is then instantiated for this specific query, which becomes a concrete guideline that instructs the model how to solve the new query. The paper demonstrates that this procedure outperforms baselines like tree-of-thought and graph-of-thought significantly on some reasoning tasks.", "questions": "1. The success of the proposed approach critically depends on the quality of the automatically induced template. While the paper showed empirical performance on downstream tasks, it remains unclear how good the templates themselves are. It would be great to make a comparison between the automatically generated task templates with manually prepared templates for some tasks.\n\n2. It remains unclear to me how the template for a specific task category (say maths) can potentially be revised and improved, as the model sees more and more examples from that task. Although the paper described briefly how the buffer can be dynamically updated (line 182), I don't quite understand how the template is updated automatically for higher accuracy as the model receives more examples (this seems to be a verbalized gradient descent procedure, but no quality improvement is guaranteed).\n\n3. What are the embedding models you used for retrieval purposes? Also it would be great to give some analysis on the impact of retrieval quality on downstream task performance, as finding the right template is critical.\n\n4. In Eq 5, it seems whether the template should be replaced should depend upon the template quality, instead of its similarity with the embedding vector.\n\n5. How quickly can this approach adapt to an unseen task? Would one or two examples from new task be enough to achieve reasonable accuracy? It would be great to give some analysis to the adaptability and continual learning ability of the proposed approach.", "limitations": "The paper properly addressed limitations and impact of the work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The novelty of the proposed method \"buffer of thought\" is a good contribution of the paper which may significantly improve LM's reasoning ability for some tasks.", "weaknesses": "Generalizability and robustness of the proposed approach needs to be further verified, as the introduction of a hierarchical reasoning procedure (abstract template + instantiation) may create more noise and make it less reliable for novel tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We thank Reviewer upCD for the positive review and valuable feedback. We are glad that the reviewer found that our BoT makes good contributions to the improvements of LLMs, and our method significantly improves LM's reasoning ability in various tasks. Please see below for our responses to your comments.*\n\n**Q1: Generalizability and robustness of the proposed approach.**  \n\n**A1:** \n\n1. Breaking down a complex problem into step-by-step solutions has proven to be an effective way to enhance accuracy and robustness, as demonstrated by ToT, GoT, and Meta-prompting. Moreover, Our BoT additionally introduces a problem distiller to extract the core problem for input task and simplifies the problem-solving procedure by retrieving proper thought templates. Such high-level thought templates discard unnecessary specific details of problem-solving processes and can be instantiated with different problems adaptively, which effectively reduce possible noises caused by previous hierarchical reasoning methods (ToT, GoT).\n\n2. The empirical results have also proved the generalizability and robustness of our BoT. The experiments of Fig.4 demonstrate that our BoT can maintain a robust reasoning process and reduce more noises compared to previous methods when faced with different tasks. And the experiments of Fig.9 reveal that our BoT can generalize to more novel tasks and continually improve the performance.\n\n**Q2: Comparison between automatically generated task templates with manually prepared templates.**\n\n**A2:** \n\n1. We appreciate the reviewer's insightful comment regarding the quality of the automatically generated templates. To address this, we conducted comprehensive experiments to compare the automatically-generated templates with manually-designed templates in MATH dataset. Please refer to **the PDF in global response** for more detailed results.\n\n2. The additional experiments showcase that the automatically-generated templates ensure consistency, efficiency and scalability for more accurate and effective problem-solving, also demonstrated by the results in Fig.9 and 10.\n\n**Q3: How template can be revised and improved?**\n\n**A3:** \n\n1. Since there is no suitable metric to evaluate the quality of thought templates and BoT is a training-free reasoning fraemwork, we **do not optimize the previously accumulated thought templates but instead add new thought templates**. To be specific, we calculate the similarity between new thought templates and those in the meta-buffer to avoid redundancy and repetition. Using powerful LLMs like GPT-4, the quality of the extracted thought templates is relatively high, as demonstrated by our qualitative analysis in Fig.9 and 10. \n\n2. As for the reason why the accuracy increase with more examples received in Fig.9, we have discussed this in Line 465-477. That is because when in the first round, the meta-buffer is empty and the buffer manager is still accumulating thought templates. With the accumulation of thought templates, BoT gradually enhances its ability **by utilizing the thought-templates obtained from previously solved problems to help** addressing subsequent similar problems, thus contributing to the increase in accuracy.\n\n3. We sincerely thank you for your insightful suggestion. In future work, we plan to design a metric to evaluate the quality of thought templates, allowing for dynamic optimization of the accumulated thought templates. This enhancement may further improve the accuracy and robustness of our method.\n\n**Q4: What are the embedding models used for retrieval ? And analysis on the impact of retrieval quality.**\n\n**A4:** We used text-embedding-3-large as the embedding model. For better understanding, we here provide experimental analysis on the impact of retrieval quality on downstream task performance using three different-sized embedding models. Due to the page limits, **please refer to the table and analysis in global response**.\n\n**Q5: In Eq.5, whether we should replace the template based on similarity.**\n\n**A5:** It is noted that our update process **only includes adding new thought templates, instead of replacing or optimizing** existing ones . As mentioned in above A3, there is no suitable metric to evaluate the quality of thought template and BoT is a training-free reasoning framework. For future work, we will try to effectively evaluate the quality of thought templates, thereby enabling the dynamic optimization of meta buffer.\n\n\n**Q6: How quickly can this approach adapt to an unseen task? Number of examples to achieve reasonable accuracy? Analysis of adaptability and continual learning ability of BoT.** \n\n**A6:** \n\n1. The adaptation is related with task diversity. For tasks with low diversity, such as \"Game of 24\", our approach can adapt very quickly.  For tasks with higher diversity, such as those in MATH dataset, which includes a wide range of problem types (e.g., algebra, geometry...), more examples are required to accumulate a comprehensive set of thought templates. This is necessary to cover the various subdomains of the problems.\n\n2. For tasks with low diversity, one or two examples are often sufficient to achieve reasonable accuracy. For tasks with higher diversity, a larger set of examples is necessary as mentioned above. This ensures that our method remains robust and generalizable across a wide range of problem types.\n\n3. As shown in in Fig.9 and Fig.10 of Appendix, we conduct ablation study on buffer-manager, which demonstrate the extradionary adaptability and continual learning ability of BoT. With more examples received and high-level thought templates accumulated in each round, there is significant improvement in both the overall performance and reasoning efficiency of the model. We have discussed the underlying reasons for these improvements in A3. This continual learning capability is crucial for ensuring that BoT can effectively adapt to new tasks over time, maintaining the efficiency and robustness of our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dfPj4Bgua4", "reviewer": "Reviewer_cwbi", "summary": "The authors propose a new framework to optimize prompting for various tasks. The frameworks consists of a buffer, where high-level problem-solving templates are stored. A query is first processed by extracting its key aspects.Templates are selected using embeddings of the templates and the extracted key aspects query. The high-level problem-solving templates is then instantiated with the extracted key aspects of the query and its inference is executed. Additionally a buffer manager is introduced to allow updating of the thought template buffer with new, \"specialized\", thought templates.", "questions": "* What embedding model is used for the thought buffer?\n* Why it is too computationally expensive to provide error bars for the existing results? It should be possible to compute them locally at least for some of the tasks.\n* Figure 3:\n  * What is logarithmic time?\n  * Why is the inference time of BoT not at least twice that of the Expert baseline, since it needs at least two LLM interactions, where as the Expert baseline only requires one (if I understood that correctly)?", "limitations": "The limitations are reasonably explained.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper combines the idea of RAG with a library of templates for solving tasks in order to make automatic prompt engineering more flexible and by using a cache (the library) more efficient, since it avoids more complex exploration. The framework and its components are sufficiently outlined and their description is clear. The writing is easy to follow. The evaluation shows clear advantages over existing approaches, while being executed with modern LLMs.", "weaknesses": "Claims from the checklist:\n* point 6: It is not clear, what embedding model is used.\n* point 8: No sufficient information are provided for the compute resources, especially the local (?) Mixtral execution.\n\nNot clear, how well their approach scales with the number of different thought templates (and not just specialized one). Additionally it is not clear, how well their approach works, if the query (something completely different) does not match any of thought templates in the buffer, as at least some of thought templates seem to specifically designed for the respective evaluation tasks. This is also one of the weaker points, since the authors claim that their approach requires less manually labor from the user than other approaches.\n\nI would have appreciate some examples for the tasks in the appendix, so that it is not necessary to look them up in the respective code/papers.\n\nThe language can be polished a little bit more. For example:\n* lines 29/30: \"which makes it impractical to manually design them task by task\" - it is just cumbersome to manually design them, but not impossible\n* lines 69/70: \"the retrieval-augmented LLM first queries an external database with billion-level tokens [23] for retrieving a subset of the text\" - I think that is just how that particular paper implemented RAG, but it is not a general description of the idea behind RAG.\n* typos:\n  * \"enahnced\" (line 78)\n  * Llama-70B (line 277)\n\nreferences could also use a little but of work, for example:\n* reference [27]: no place of publication\n* references are slightly inconsistent:\n  * for example one time the abbreviation ICLR is used, otherwise not\n  * sometimes the number prefix (for example Eleventh) for conferences is used, sometimes not", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We thank Reviewer cwbi for the positive review and valuable feedback. We are glad that the reviewer found that our proposed BoT is more flexible and more efficient, and our method shows clear advantages over existing approaches. Please see below for our responses to your comments.*\n\n\n**Q1: What embedding model is used?**\n\n**A1:**  In our implementation, we used text-embedding-3-large as the embedding model. We also conduct in-depth experiments about the model size impact for our method. **Please refer to global response for detail.**\n\n**Q2: Information for compute resources?**\n\n**A2:**  As mentioned in Line 214, we use NVIDIA A100-PCIE-40GB GPUs for local model like Llama3-8B and Llama3-70B.\n\n**Q3: How well the approach scale with different thought templates. If query does not match any of thought templates in the buffer, how well this approach works.**\n\n**A3:**\n\n1. As depicted in Fig.9 and Fig.10, with the number of different thought template increase, the accuracy and reasoning efficiency of BoT steadily improves, it confirms that our method possesses excellent scalability. This is because the buffer manager continually expands the meta-buffer while also utilizing the thought templates obtained from previously solved problems to address subsequent similar problems. The possibility of retrieving suitable thought templates also increases, thus avoiding the need to construct reasoning structures from scratch and thereby enhancing inference efficiency.\n\n2. As discussed in L160-163, for entirely new or unique problems, our method provides three general coarse-grained thought-templates, which could be assigned considering the distilled task. These coarse-grained thought templates ensure that new tasks follow appropriate reasoning processes. This broad guidance offers LLMs greater flexibility during the generative inference process, enabling them to efficiently address new and unique problems.\n\n3. Due to character limits, **please refer to Q1 in response to reviewer yiEf** for more examples in demonstrating how well our thought templates work. \n\n**Q4: Language, typos and references**\n\n**A4:**  We sincerely appreciate your attention to the details of our paper. We will carefully address the issues you pointed out regarding language, formatting, and citations, to enhance the professionalism and standardization of our paper. \n\n**Q5: Concern about error bars**\n\n**A5:**\n\n1.It is indeed possible to compute error bars for our results. However, we chose not to include them in our initial version to maintain consistency with many prior works, such as ToT, Meta Prompting, and PAL, which also do not provide error bars. This approach ensures a unified basis for comparison across different methods.\n\n2.In response to the reviewer's concern, we have now computed the error bars for our results to enhance the strictness and clarity of our results. The updated results, including error bars, are presented in the table below. We will incorporate these error bars into the final version of our paper to provide a more comprehensive and statistically robust presentation of our results. \n\n| Task|GPT4|GPT4+CoT|Expert|PAL|ToT|GoT|Meta|**BoT (Ours)**|\n|---|----|-------|------|---|---|---|----|--------------|\n| Game of 24|3.0|11.0|3.0|64.0|74.0|73.2|67.0|82.4 ± 1.5|\n| MGSM (avg)|84.4|85.5|85.0|72.0|86.4|87.0|84.8|89.2 ± 1.8|\n| Multi-Step Arithmetic|84.0|83.2|83.2|87.4|88.2|89.2|90.0|99.8 ± 0.2||\n| WordSorting|80.4|83.6|85.2|93.2|96.4|98.4|99.6|100.0 ± 0.0|\n| Python Puzzles|31.1|36.3|33.8|47.3|43.5|41.9|45.8|52.4 ± 1.6|\n| Geometric Shapes|52.6|69.2|55.2|51.2|56.8|54.2|78.2|93.6 ± 2.4|\n| Checkmate-in-One|36.4|32.8|39.6|10.8|49.2|51.4|57.2|86.4 ± 1.7|\n| Date Understanding|68.4|69.6|68.4|76.2|78.6|77.4|79.2|88.2 ± 1.5|\n| Penguins|71.1|73.6|75.8|93.3|84.2|85.4|88.6|94.7 ± 1.2|\n| Sonnet Writing|62.0|71.2|74.0|36.2|68.4|62.8|79.6|80.0 ± 0.4|\n\nAs shown in the table, our method achieves stable performance, which further proofs that our method is effective, stable and robust.\n\n**Q6: What is logarithmic time?**\n\n**A6:** We perform a logarithmic operation on the obtained time, i.e., Logarithm time = $\\ln(t)$. The reason why we do this is that the data range of the inference time between different methods is board, in such cases, the readability and effectiveness of the histogram in conveying information are compromised. By applying logarithmic transformation, the differences in bar heights within the histogram are reduced, rendering the chart more aesthetically pleasing and the information more clearly presented. \n\n**Q7: Why is the inference time of BoT not at least twice that of the Expert baseline?**\n\n**A7:** As mentioned in Q6, the formula for calculating the multiple of inference time should be $e^{\\ln(t_1) - \\ln(t_2)}$, rather than direct comparison. Furthermore, despite our method involving collaboration across multiple components, the inference time required by BoT for certain problems does not significantly exceed that of the baseline. Because our method transforms some reasoning tasks (such as Game of 24 and checkmate in one) by simplifying the multi-step reasoning or heuristic search process into generating a segment of code capable of solving the problem. Specifically, in the expert baseline, solving a problem from Game of 24 need to experiment with multiple combinations and conduct calculations, which is time consuming. On the contraty, our method transform this process into generating few lines of code and execute the code to get the answer,  which requires significantly less time compared to the expert baseline, demonstrating the superiority of our approach in addressing certain complex problems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Global response

We sincerely thank all the reviewers for their thorough reviews and valuable feedback. We are glad to hear that the proposed framework is novel (all reviewers) and effective (all reviewers) in enhancing the reasoning abilities of LLMs, the paper is well-written and easy to follow (reviewer yiEf and cwbi), and the performance improvements demonstrated in experiments are promising (all reviewers).

Here, we want to highlight the main contributions and novelties of our proposed framework, **"Buffer of Thoughts" (BoT)**:

**Thought-Augmented Reasoning Mechanism:** BoT introduces a novel framework that leverages previously distilled high-level thoughts to assist LLM reasoning tasks. This reduces the need for generating reasoning paths from scratch, significantly enhancing accuracy, efficiency, and robustness.

**Meta-Buffer and Buffer Manager:** BoT incorporates a meta-buffer to store distilled high-level thoughts and a dynamic buffer manager to continuously update the meta-buffer based on new tasks. This ensures scalability and continuous improvement of the LLM reasoning system.

**Empirical Validation:** Extensive testing across 10 complex tasks demonstrates significant performance improvements over state-of-the-art methods, including substantial accuracy improves and reduced computational costs compared to traditional multi-query prompting methods.

We summarize our responses to the reviewers' comments as follows:

* We additionally provide more examples to show the quality of our high-level thought template. (reviewer hZUQ, yiEf and upCD).

* We provide more examples to demonstrate the instantiation and reasoning process for new problems, and give detailed analysis about the adaptability and continual learning ability of our method. (reviewer yiEf, upCD and cwbi).

* We further conduct the experiment to compare the quality of automatically-generated thought template and manually-designed template in response pdf. (reviewer hZUQ and upCD) 

* We provide more quantitative comparisons and analysis for the impact of different-sized embedding model in the table below (reviewer upCD and cwbi). 

  * The results indicate that stronger encoding capabilities lead to higher accuracy. The larger model (text-embedding-3-large) extracts more informative embeddings, improving retrieval accuracy and overall BoT performance. 

  * The impact of embedding model is not significant. Even with the small text-embedding-ada-002 model, we still achieve higher accuracy than other methods. This is because we use the distilled problem $x_d$ and the thought template description ($D_T$) for similarity computation. Both $x_d$ and $D_T$ are concise sentences, with simple but critical semantic structures, allowing even weaker encoders to handle them effectively. Thus, our method remains robust and generalizable across different-sized encoder, as demonstrated by the experiment results.

|Task (accuracy)|text-embedding-3-large+BoT|text-embedding-3-small+BoT|text-embedding-ada-002+BoT|GPT4|ToT|Meta Prompt|
|---|:---:|:---:|:---:|:---:|:---:|:---:|
|Game of 24|82.4|81.8|81.0|3.0|74.0|67.0|
|MGSM|89.2|88.7|87.9|84.4|86.4|84.8|
|Word Sorting|100.0|100.0|99.8|80.4|96.4|99.6|


We reply to each reviewer's questions in detail below their reviews. Please kindly check out them. Thank you and please feel free to ask any further question.

Meta-review: The paper introduces "Buffer of Thoughts" (BoT), a novel framework designed to improve the reasoning abilities of large language models (LLMs). This framework uses a component called 'meta-buffer' to store high-level problem-solving strategies, which can be dynamically retrieved and adapted to new tasks to facilitate efficient reasoning. The method also includes a "buffer-manager" to keep the meta-buffer up-to-date and to support effective scaling. The paper also presents experiments on a range of reasoning tasks showing significant improvements over other methods.

Most reviewers regard the paper as well written and all consider the central idea of buffer of thoughts to be sound, novel and effective. The idea has fairly general applicability and makes justified and substantiated claims regarding computational efficiency and robustness with respect to other recent methods. The paper appears to include a thorough empirical evaluation, with the proposed method performing significantly better than existing methods on some tasks. 

There is a question regarding how well the system will perform over a long time-scale: whether it has issues such as over-specialisation, and how well this approach scales with the number of different thought templates. There are other questions about how dependent the system is on a good initial library of thoughts, how effective the system is when it encounters a task for which no good template exists in the buffer, and how it behaves when it encounters unsolvable queries. Nonetheless, this work appears to represent a high quality, novel contribution and may be of significant interest as it relates to a highly active domain of inquiry.
Decision: Accept (spotlight)