MoT: Memory-of-Thought Enables ChatGPT to Self-Improve
Authors: Xiaonan Li
Xipeng Qiu
Venue: EMNLP 2023 Main (2023)
Keywords: LLM
ChatGPT
Self-Improve
Large Language Model
Memory
TL;DR: MoT: Memory-of-Thought Enables ChatGPT to Self-Improve

Abstract: Large Language Models (LLMs) have shown impressive abilities on various tasks. However, fundamentally improving them depends on high-quality datasets or computationally expensive fine-tuning. On the contrary, humans can easily improve themselves by self-thinking and memory, without external resources. In this paper, we propose a framework, **MoT**, to let the LLM self-improve through **M**emory **o**f **T**houghts, without annotated datasets and parameter updates. Specifically, MoT is divided into two stages: 1. before the test stage, the LLM pre-thinks on the unlabeled dataset and saves the high-confidence thoughts as external memory; 2. During the test stage, given a test question, the LLM recalls relevant memory to help itself reason and answer it. Experimental results show that MoT can help ChatGPT significantly improve its abilities in arithmetic reasoning, commonsense reasoning, factual reasoning, and natural language inference. Further analyses show that each component contributes critically to the improvements and MoT can lead to consistent improvements across various CoT methods and LLMs.

MoT: Memory-of-hhought Enables ChatGPT to Self-Improve
Xiaonan Li, Xipeng Qiu
School of Computer Science, Fudan University Shanghai Key Laboratory of Intelligent Information Processing, Fudan University {lixn20, xpqiu] @fudan.edu.cm
Abstract
Dataset
labeled
Alac Cate
LLM
Large Language Models (LLMs) have shown impressive abilities in various tasks. How- ever, fundamentally improving them depends on high-quality datasets or computationally ex- pensive fine-tuning. On the contrary, humans can easily improve themselves by self-thinking and memory, without external resources. In this paper, we propose framework, MoT, to let the LLM self- self-vmmrove through of Thought, without annotated datasets pa- rameter updates. Specifically, MoT is divided into two stages: 1. before the test stage, the LLM pre-thinks on the unlabeled dataset and saves the high-confidence thoughts external memory; 2. During the test stage, given test question, the LLM recalls relevant memory to help itself reason and answer it. Experimen- tal results show that MoT can help ChatGPT significantly improve its abilities in arithmetic reasoning, commonsense reasoning, factual rea- soning, and natural language inference. Further analyses show that each component contributes critically to the improvements and MoT can lead to consistent improvements across various CoT methods and L.L.s.
(a) L.LM Fine-tuning
Unlabeled
Peenry
PecatL
Infleracce
(b) Pre-thinking and Recalling
Figure I: The comparison between fine-tuning and MoT: while fine-tuning LLM with labeled datasets is costly and needs powerful computational resources, MoT can make the LLM self-improve via pre-thinking and recall- ing, without parameter updates and annotated datasets.
Wang et al., 2022b; Huang and Chang, 2022). Specifically, Wei et al. (2022c) and Kojima et al. (2022) propose few-shot CoT and zero-shot CoT, which elicit LLM's 'is reasoning by few-shot demon- strations and simple yet effective "Let's think step by step" prompting, respectively. Based on them, Wang et (2022b); Press et (2022); Zhou al. (2022); Wang et al. (2023); Weng et al. (2022) further propose self-consistency, self-ask, least-to- most, plan-and-solve, etc., to achieve more compli- cated reasoning in various specialized scenarios. Despite the impressive abilities of the LLM pre- trained on the large corpus, fundamentally improv- ing the LLM'' performance beyond few-shot / zero-shot baselines highly depends either high- quality annotated datasets costly fine-tuning of LLMs. In general, these methods can be divided into three categories:
1 Introduction
Large Language Models (LLMs) have demon- strated surprising abilities on wide range of Natu- ral Language Processing (NLP) tasks (Chen et al., 2023; Zhang al., 2022a; Chowdhery al., 2022: Tay et al., 2022; OpenAI, 2023; Hoffmann al., 2022: Touvron et al., 2023; Mialon et al., 2023;
Annotated Datasets Fine-tuning: Wei et al. (2022a) and Sanh et al. (2022) propose FLAN and TO respectively to en- hance the LLM'' zero-shot ability by tens of cu- rated NLP benchmark datasets. Based on FLAN, Chung et al. (2022) scale up its training in terms of model size and the number of tasks, and demon- strate that the added CoT examples with rationales improve the LLM's reasoning abilities. Instruct- GPT (Ouyang et al., 2022) improves the GPT- 3's instruction- instruction-follooiing following ability by fine-tuning on many diverse crowd-sourced instruction-answerr
Zhao et al., 2023; Qiu et al., 2020). Notably, new abilities emerge in LLMs as they are scaled to hundreds of billions of parameters, like in-context few-shot learning (Chen et al., 2023; Dong et al., 2022), simple digit operation and factual knowl- edge query (Wei al., 2022b). Especially, the gen- eral reasoning ability of the LLM has impressed the NL.P community and relevant techniques have achieved a series of new state-of-the-art (Wei et al., 2022c; Kojima et al., 2022; Lampinen et al., 2022:

---

pairs. 2. Retrieving Annotated Data: Liu et al. (2022). Su et al. (2022a) and Agrawal et al. (2022)
thoughts as external memory, In the test stage, the LLM recalls relevant memory help reason and answer the given test question. Since we focus on the overall framework and aim to demonstrate its generality and extensibility, we use simple com- ponents to instantiate these two stages. Specifi- cally, we use the simple Few-Shot-CoT (Wei et al., 2022c) with multiple-patt decoding strategy (Wang et al., 2022b) in the pre-thinking stage and propose answer-entropy to filter out uncertain thoughts. For memory recall, we propose LLM-rrtrrieal, which lets the LLM itself retrieve relevant memory to help answer the test question. Compared with typi- cal semantic retrievers like SBERT (Reimers and Gurevych, 2019), LLM-retrieval can better capture the deep connection of complicated logic and rea- soning than semantic embeddings.
use SentenceBERT (Reimers and Gurevych, 2019) or BM25 (Robertson and Zaragoza, 2009) to re- trieve relevant examples from the annotated dataset, to improve LLM's in-context learning. Rubin et al. (2022) and Shi et al. (2022a) leverage annotated datasets to train retrievers by the LM-feedback to retrieve helpful demonstrations for the test example. 3. Fine-tuning with LLM- -LM-ggnerated data: Zelik- man et al. (2022) let the LM generate rationales for annotated dataset and train itself to enhance the reasoning ability. Magister et al. (2022).Ho et al. (2022) and Fu et al. (2023a) use the reason- ing paths generated by large LM to improve the small LM's reasoning capability. More recently, Huang et al. (2022) demonstrate the effectiveness of self-training on PaLM (Chowdhery et al., 2022).
We summarize our contribution as follows:
As annotating high-quality data, especially ratio- nales in CoT data, is expensive, fine-tuning LLM re- quires extremely powerful computational resources and results in high computational costs. Methods above that rely on fine-tuning also face two chal- lenges: 1. Since the most powerful LLMs, e.g., GPT-4 (OpenAI, 2023) and PaLM (Chowdhery et al., 2022; Anil et al., 2023), are only publicly available through the inference API, it is not feasi- ble for most of the research community improve them by these methods. 2. Fine-tuning LLM for specific capability enhancement is costly and not environmentally friendly. As the LLM has massive parameters, fine-tuning them will lead to substan- tial costs of model storage and deployment. Further studies show that fine-tuning the LLM with special- ized data may significantly decrease its general abilities (Fu et al., 2023b).
To the best of our knowledge, the proposed framework is the first to let LLM improve its own reasoning abilities based on the memory mechanism, without parameter updates and annotated datasets.
We conduct comprehensive experiments on extensive datasets and the results show that MoT can help ChatGPT improve its abilities in arithmetic reasoning, commonsense reason- ing, factual reasoning and natural language inference without parameter updates and an- notated datasets. Further analyses show that each component contributes critically to the improvements and MoT can lead to consistent improvements across various CoT methods and LLMs.
While considerable efforts were dedicated to col- lecting high-quality annotated datasets and fine- tuning the LLM, which is costly and may de- crease its general ability, on the contrary, humans can improve their own reasoning abilities through the metacognition process (Dunlosky and Met- calfe, 2008) and the memory mechanisms (Tulving, 2002), and preserve their general abilities. For ex- ample, memory helps humans improve themselves in terms of decision-making, reasoning, judgment, etc. (Tulving, 2002). Inspired by this, we propose think-then-reee
We release the code and generated CoT rea- soning paths to facilitate future research' In this paper, we instantiate the proposed frame- work with simple components and demon- strate its effectiveness. We hope that MoT can inspire researchers of the important design choices about making the LLM self-improve with memory mechanisms and pave the way for further improvements.
2 Background: Chain of Thought
The large language model has shown impressive reasoning abilities on various tasks. Chain-of- Thoughts (CoT) prompting (Wei et al., 2022c; Ko- jima et al., 2022) is the most prevailing way to
MoT, shown in Figure 1, framework to let the LLM self-improve through ofmnory-of-Thoughte without supervised data and parameter updates. the pre-thinking stage, the
LLM thinks on the unlabeled dataset and saves the
http::/tgitub..oo//.eeSeee

---

LUM Inout
given test question, we propose LLM-retrieval let the LLM retrieve relevant memory to help itself reason and answer it. Our method does not depend on high-quality labeled datasets and costly fine- tuning of LLM,, and it is feasible when the L.M is frozen or only available through the inference API. Since we let the LLM think over the unlabeled dataset, save the self- tefffaeeertted thoughts external memory and retrieve relevant memory for itself to help reasoning, we consider our method as making the LLM self-improve with Semmory-of-hhougeee We introduce these two stages below.
The diagonals of rhosbus
22
(A) 266 (C)
A: The area of mocebus 15 equal to half the product of its diagonals, 5o the area vould be (16 223/2 100 cos,, The answer is C.
Maddie 24 apples, 14 (C) (D)
she left?
LLM
A: Rives apples Mike, bax 32 apples left.. answer A.
Figure 2: The illustration of Few-Shot-CoT.
let the LLM reason, i.e., generate a series of in- termediate reasoning steps that lead to the final answer. As shown in Figure 2, Few-Shot-CoT (Wei et al., 2022c; Lampinen et al., 2022) provides a few demonstrations with rationales, i.e., ques- tion/rationale/answa pairs, and prompts the LLM generate the rationale that leads the final an- swer. Pero-Shot-CoT (Kojima et al., 2022) adds the prompt, "Let's 's think step by step". after the test question and clicits the LLM's reasoning. Specifi- cally, the Few-Shot-CoTT gets the answer as:
3.1 Pre-Thinking
3.1.1 Let LLM Think before Test Stage
In this stage, we let the LLM think over the unlabeled dataset and save the resultant ques- tion/ratiionaleansser pairs as external memory. Since we focus on the overall framework and aim to demonstrate its generality and extensi- bility, we instantiate the "thinking" mechanism here the simple Few-Shot-CoTT (Wei et al., 2022c) with multiple-path decoding strategy (Wang et al., 2022b) in this paper. Specifically, for each example x from the unlabeled dataset X, we let the LLM sample n reasoning paths and answers with temperature T > O, denoted as a..... aa,] Then we use majority-voting to select the most consistent an- swer, ā arg max,, Σ}= I(a, =aj), and keep the reasoning path, which leads to as memory. Since we only consider the thought that leads to the most consistent answer, the retained thoughts can be more accurate (Wang et al., 2022b) and better help the test stage. For simplicity and to save mem- ory size, we randomly select one reasoning path of the final answer for each unlabeled example and see saving multiple thoughts for one question future work.
LLM d2.... (test)
(1)
a Parse- Answer))
(2)
where dj = [æs, rı, as] is the i th demonstration and consists of the input, rationale and answer. Few- Shot-CoT first decodes from the LLM given the few-shot CoT demonstrations, and parses to get the final answer. Since the demonstration is typ- ically in the format: "[input] [rationale] The an- swer is [answer]", the answer can be easily parsed from by the trigger "The answer is" (Wei al., 2022c). Similarly, Zero-Shot-Con uses answer trig- gers, c.g., "Therefore, the answer is"" to extract the final answer from the zero-shot reasoning path gen- erated by LLM (Kojima et al., 2022).
[ri, ....2-... ... .r]] and [01, (01,02-
3 Method
We show the overview of our framework in Fig- ure 3. In this paper, we mainly focus on making LLM self-improve in the typical few-shot CoT sce- nario, where we are given a frozen large language model and an unlabeled dataset with few CoT demonstrations (Wei et al., 2022c; Huang et al., 2022). We further demonstrate MoT's effectiveness in zero-shot scenarios in section 4.3. Our frame- work is divided into two stages: 1. Pre-Think Before the test stage, the LLM thinks over the un- labeled dataset and keeps the coggh-ooneeeece rea- soning paths as memory. 2. Recall In the test stage,
3.1.2 High-Confidence Thought Filtering
Since the most consistent answer does not necessar- ily lead to the correct answer and incorrect demon- strations cause inferior performance (Yoo et al., 2022; Lyu et al., 2022), we further propose to filter the thoughts by uncertainty. Inspired by Liu et al. (2020); Xin al. (2020), we propose the answer- entropy u(-) to filter out high- uncertainty thoughts:

---

for Quretiezz tith zoe sgarett cocInn seese?
A, books are mode leebers Bs looks are made ei paper Arsse Booksam lopoodly mede ei paper other melertel for wniling and priting weewer
Unlabeled Dataset
Helpful Memory
Quroiizz
oonman of paper B. lissly stocuually paper
Auzver
szund or
tre commotiy
buiding bouses, willle paper is Est Tas answer
Generated COTs
Retrieve
Test @vestion: which against common brick
piicce paper
Uncertainty Filtering
Memory Candidates
SF
Think
Memory of Thoughts
heavy large, aasking incorventen papse
Senantic Filtering
practica)
Pre Think
Recall and Inference
Figure 3: The overview of MoT.
thus contains crucial and valuable information for the LLM to answer the test question. For the coher- ence in the subsequent content, we will refer to m; as "memory" "thought".
Mard
63
60
10 20 30
40 50 60 70
fikered Out
Figure 4: The relation between accuracy and the filter- out ratio, after multiple-paths decoding on the DROP dataset. Under the same filter-out ratio, filtering by answer entropy leads slightly higher Fl-score than filtering by Max-P proposed by Huang et al. (2022).
3.2 Recalling
In the test stage, the relevant memory is retrieved from the memory pool M, to help the LLM answer the given test question, Qteat- Although semantic embedders, e-g-, SBERT (Reimers and Gurevych, 2019) are capable of retrieving semantically rele- (4) vant examples for ICL (Liu et al., 2022), for rea- soning tasks, it is challenging for them to fully capture the deep logical connections between @oett and helpful memory, as single vector can not directly reflect the intricate logic and reasoning path. Since the LLM, c.g., ChatGPT, has shown impressively powerful and general natural language understanding capability and a certain level of self- awareness (Kadavath et al., 2022), we propose LLM-retrieval to let the LLM retrieve helpful mem- ory for itself.
A unique({ ((,))=
p(a,)= = Σ, j=1 I(a; = =a,//n
u(af)= = - - p(a;) log p(at)
(5)
where A (ai.ah... is the set of an- swers. u(-) indicates the answer uncertainty, and the higher u(-) is, the more uncertain the LLM is. We filter out thoughts whose uncertainty is higher than r and 7 is a pre-defined threshold. In
the exploratory experiment (Figure 4), we find the thought with lower uncertainty is more likely to be correct: the stricter the filtering is, the more accu- rate the remaining thoughts are. Hence the answer- entropy can filter out noisy thoughts and lead to more accurate thoughts for recalling. Compared with the filtering Huang et al. (2022), which uses the number of consistent paths (max probability, abbreviated as Max-P) as metric, answer-entropy leads to slightly higher accuracy, under the same filter-out ratio.
As the LLM has limitation of the max length, is infeasible to let the LLM directly select among the entire memory pool. Inspired by human's mem- ory recall process, where we usually first uncon- sciously filter the relevant memories and then con- sciously evaluate them (Schacter and Addis, 2007; Franklin al., 2005), we divide L.M--rerrievl into two stages: 1. filter semantically irrelevant memory and get memory candidates; 2. let the LLM choose from memory candidates.
After filtering, we obtain the pool of memory- of-thoughts, M [m,) where m; is the con- catenation of corresponding input, reasoning path and answer (see Figure 3). M consists of the high- quality thoughts of LLM on various questions and
Since the diversity of demonstrations has been shown important for L.LMs Ye al., 2023; Levy et al., 2022; Li and Qiu, 2023), we follow Li et al. (2022) to conduct memory retrieval with diversity-

---

based clustering, i.e. we partition the entire mem- ory pool into clusters, ( ) and retrieve one memory from each cluster sep- arately. Specifically, for each cluster M we first use an off-the-shelf semantic embedder, e-g. SBERT (Reimers and Gurevych, 2019), to filter out semantically irrelevant memory and get memory candidates as follows:
8 LLM(m
, ,LM(
m (eest)
(8)
Parse- (asserr)
(9)
In this paper, we focus on the overall framework and instantiate it with simple components, i.c., Few- Shot-CoT, simple uncertainty filtering and LLM- retrieval. We further analyze the orthogonality of MoT and different CoT methods in section 4.3. We leave exploring more implementations, e.g.. letting the LLM itself filter out uncertain thoughts (Weng et al., 2022; Long, 2023), as the future work.
Me top-kmeMon (sim m));
where --m(..-) is the cosine similarity of semantic embeddings. M( are the i-th cluster's candidates and contain k memories.
Experiment
4.1 Experimental Settings
Then we further let the LLM select the most helpful memory from each cluster follows: m LLM(( M((), Pretricnat),
Dataset We conduct experiments ten datasets, across four task families: Arithmetic reason- ing: AQuA (Ling et al., 2017) and DROP (Dua et al., 2019); Natural Language Inference: Ad- versarial NLI subsets (Nie et al., 2020), including ANLI-AI, ANLI-A2 and ANLI-A3, which cover varying difficulty levels; Commonsense Reason- ing: OBQA (Mihaylov et al., 2018) and ComV (Commonsense Validation) (Wang al., 2019); Factual Reasoning: BoolQ (Clark et al., 2019). FactCK (Fact Checker) and WikiQA (Srivastava et al., 2022). We list dataset overview, statistics, split and evaluation metrics in Appendix B.
(7)
where Pr Pretricval is the prompt for the LLM retrieve helpful memory. We concatenate the test question Qeest, memory candidates M and Pretrieval by specialized template. The resulting input for LLM is like: "References:
M LM (.. Target Question: [qtest] which one reference would be the most helpful for you to answer the larget question?".
M
In this manner, we can utilize the LLM's pow- erful natural language understanding ability to se- lect the most helpful memory of _ for itself. Since these retrieved memories are from diverse memory clusters, they can be not only helpful for Atest but also comprehensive, thus facilitating the LLM to answer the test question. Meanwhile, the semantical -iilttiiilllitteiinggg evant memories in advance, thus significantly helps save the number of LLM calls.
Method Comparison Since we focus on whether MoT can help the LLM self-improve, we compare MoT with baselines on the same LLM, ChatGPT (GPT-3.5- HEPT-3.--Turbo-055 Turbo-0301), including ehro-shot/few- shot CoT and zero-shot/few-sho direct prompting. To analyze the effect of rationales and thinking in MOT, we additionally compare MoT with its two variants: 1) MoT (no rationale), which removes rationales in the retrieved memory and thus lets the LLM directly output the answer, which can be seen as the few-shot direct version of MoT; 2) MoT (no thinking). which keeps rationales in the retrieved memory but forces the L.LM to directly answer the question without COT. Specifically, we add "The answer is" as the LLM's output prefix to prompt the LLM directly output the answer. Through these two variants, we can analyze the effect of rationales and the thinking in MoT, respectively. Additionally, conduct experiments of MoT under annotated datasets, MoT (with gold), to see its potential im- provement space, where we use the gold labels filter out incorrect memory. Thus, MoT will not be degraded by the incorrect answer.
filter out semantically irrel-
In exploratory experiments, we find that provid- ing only memory candidates' questions for LLM- retrieval almost does not affect the retrieval result. Hence, for each ! we only provide its question for the LLM to select, which can significantly save the inference cost of the LLM. To make LLM better understand the goal of retrieving helpful memory and make its output easy-parsing with pre-defined format, we append extra instructions like "You must end in the format like "The most helpful question is question [idx]." the input. We show the com- plete input of LLM-retiivaal in Appendix A.
3.3 Inference
Given test question Otest, the LLM can think and then output the answer based on the retrieved mem- ory, m!), (2) (k). Specifically, we let the
Implementation Details We use the public Ope- nAI language model of "gpt-3.5-turbo-00 un-
LLM reason in the manner of Cho-t

---

Arithmetic Reasoning
ANLI
CS Reasoning
Factual Reasoning
Method
AVG
AQuA
DROP
-A1
-A2
-A3
OBQA
ComV
BoolQ
FacCCK
WikiQA
Zero-Shot
27.7
24.7
54.4
48.0
$1.7
79.4
90.5
63.4
75.6
52.6
56.8
Few-Shot
28.9
46.3
55.0
48.5
51.1
82.0
90.8
64.4
77.0
32.5
57.6
MoT (no rationale)
27.0
59.4
56.2
50.3
$2.6
84.2
91.0
70.1
82.1
53.9
62.7
MoT (no thinking)
24.4
59.4
55.6
50.2
52.6
81.3
90.5
71.6
82.2
64.3
63.1
Tero-Shot-CoT
62.2
61.9
48.5
69.2
87.1
53.0
49.9
60.1
Few-Shot-Col
57.6
59.7
48.1
52.3
80.0
94.5
67.7
65.2
65.5
MoT
65.7
64.6
55.2
95.5
71.5
68.0
MoT (with gold)
56.5
71.0
65.7
55.6
55.4
82.8
94.6
74.2
86.6
70.6
71.3
Table 1: Performance comparison on ChatGPT (GPT-3.5- TTT-----Turb-00001)
less otherwise specified and the experiments on "eext-davinci-0020003"" (Appendix C) show consis- tent trends. For recalling, we use SBERT ("all- mpnettbase-v22) (Reimers and Gurevych, 2019) for semantic filtering. And clusters (also the demonstration quantity) and the number of each cluster's memory candidates k as and 10, respectively. We further analyze the num- ber of demonstrations in Appendix D. In the test stage. for the stability of results, we use greedy decoding to generate the output, unless otherwise specified. We list the full implementation details and fho-shot demonstrations in Appendix E.
Method
OBQA BoolQ
WikiQA
Decoding Paths=8
Zero- Zero-Shot-Cof Coot--
80.4
55.9
59.2
Zero-Shot-Col Soo-ShottCoTTr
82.4
54.1
59.2
the number of
Zero- Pro-Shot-CoT--1..
81.0
49.1
59.4
Pew-Shot-CoTr=o..
82.0
69.2
69.6
Few-Show Pew-Shot-CoTz-
83.6
68.2
70.8
Sow-Shot-CoT-=1.2
82.2
70.2
70.8
MoTr-ar
85.0
73.2
73.2
MoTr-1
84.4
73.5
72.1
MoTr-12
85.0
72.8
72.1
Decoding Paths=16
Seo-Shot-CoTy-all
81.6
58.0
63.8
Zero-Shot Shot-CoTze
83.2
55.8
65.7
Zero Bero-Shot-oe Shot-CoTz--
83.4
53.8
64.3
Few- Pew-Shot-oo Shot-CoTz
83.2
69.6
71.5
Few-Shot-CoTpy
83.6
69.2
71.7
4.2 Main Results
Pew-Shot-CoTz--1.2
83.6
69.7
70.9
MoTT=o.
84.4
73.7
73.7
We show the results in Table 1. We that MoT significantly outperforms baselines on most datasets, which shows MoT's best comprehensive reasoning capability on series NLP tasks. Specifically, MoT exceeds Few-Shot-CoT and Zero-Shot-CooT by 3.7 and 9.1 points respectively. and this directly demonstrates that MoT can make the LLM improve itself by memory-of-thoughte without annotated dataset and parameter updates. Notably, Zero-Shot-CoT shows impressive perfor- mance on ChatGPT and outperforms Few-Shot- CoT on several datasets, e.g., AQuA, DROP, ANLI- and ANLI-2, which indicates the potential un- necessity of irrelevant CoT demonstration for the LLM with powerful zero-shot reasoning ability. Meanwhile, MoT surpasses Zero-Shot Shot-CoT con- sistently all datasets and this indicates the help- fulness of retrieved memory.
MoTT=I
85.0
72.7
74,2
MoTT=1.2
84.8
73.3
74.1
Table 2: Performance comparison under self- consistency strategy across various hyper-paranene
memory,
perform well by intuition, without conseious reasoning (Dijksterhuis and Nordgren, 2006; Todd al., 1999). Additionally, although MoT (no thinking) provided with the rationales while MoT (no rationale) is not, they show gen- erally similar performance, which indicates that explicit reasoning is necessary for the LLM to fully leverage the retrieved memory. In short, both rel- evant memory and explicit reasoning are essential for MoT to consistently achieve improvements extensive datasets.
As for MoT's two variants, they also show bet- ter overall performance than Zero-Shot and Few- Shot. Meanwhile, despite directly outputting the answer, they outperform Pero-Shot-CoT and Few- Shot-CoT on several datasets, e.g., OBQA, BoolQ and FactCK. This is analogous to a common phc- nomenon in human beings: when recalling relevant
Additionally, MoT (with gold) shows better per- formance than MOT, which indicates the potential improvements when MoT applies more advanced CoT methods (Wang al., 2023; Zhou et al., 2022: Zheng et al., 2023; Long, 2023) and verification methods (Weng et al., 2022; Manakul et al., 2023).

---

DROP
ANLI-A3
BoolQ
WikiQA
Few-Shot-COn
57.6
52.3
67.7
65.2
59
MiniLM
63.0
53.7
70.2
67.0
Rhard Ous
Instructor-base
64.2
53.2
70.2
66.9
(a) DROP
(b) ANLI-A3
Random
57.5
52.8
69.7
66.3
MPNet
64.7
53.3
70.4
67.1
LLLM-Retrieval
65.7
55,2
71.5
68.0
Table 3: The comparison of retrieval methods.
4.3 Analyses
(c) OBQA
(d) FacICK
Multiple-eecoding Performance In this sec- tion, we evaluate MoT under self-consistency strat- egy (Wang et al., 2022b) which decodes multiple times and uses majority-voting to get the final an- swer. We compare MoT with baselines across vary- ing sampling times and temperatures on OBQA, BoolQ and WikiQA, and the results are shown in Table 2. We see that MoT consistently outperforms Zero-Shot-CoT and Few-Shot-CoTT across differ- ent decoding temperatures and sampling times, which indicates the generality and stability of MoT. We notice that the improvements slightly diminish when using more sampling times. This is similar to the phenomenon in human beings: the more carefully we think about question, the less our previous preparation matters.
Figure 5: The effect filtering. The left and far right of the x-axis correspond to filtering, and the strictest filtering, i.e., only the memory that all paths lead the same answer can be retained.
curve over different filtering thresholds on DROP, ANLI-A3, OBQA and WikiQA Specifically, we tune the filtering threshold of answer-entropy uni- formly and observe the corresponding performance.
The results are shown in Figure 5. We find that the MoT without filtering significantly degrades and slightly underperforms Few-shot-COT datasets, e.g., OBQA and FactCK, which indi- cates that the incorrect memory can deteriorate the LLM's reasoning and thus our filtering strategy is necessary. Meanwhile, most filtering thresholds consistently lead to improvements over baselines, which demonstrates that the improvements of MoT exhibit insensitivity to the apperpparamtte of fil- tering thresholds in general.
some
The Effect of LLM-retrieval To evaluate the effect of LLM-retrieval for MoT, we conduct ex- periments with varying retrieval methods DROP, ANLI-A3, BoolQ and WikiQA, shown in Table 3. Besides the SBERT ("all-mpnnt-base-v22" abbre- viated as MPNet) (Reimers and Gurevych, 2019) used in MoT, we further compare two other se- mantic embedders, SBERT ("all-MiniLMM-L6-v2"" abbreviated as MiniLM) (Reimers and Gurevych, 2019) and Instructor-base (Su et al., 2022b) which is trained by 330 diverse tasks and supports various scenarios. We observe that using only MPNet for memory retrieval also brings significant improve- ments over Few-Shot-Coff which shows MoT's usability under the limited LLM-API budget. Af- ter using the LLM to retrieve memory, the perfor- mance gets further improvements, which directly demonstrates the effectiveness of L.M-rrtrieval. Additionally, we see that LLM-retrieval outper- forms all compared semantic embedders, which shows that the LLM can better capture the compli- cated reasoning logic than semantic embeddings.
175
1E.5 30
1000 IE 30 E
(a) DROP
(b) ANLI-A3
(c) OBQA
Figure 6: Limited Memory-Size Performance.
Limited Memory-Size Performance In real- world scenarios, the number of unlabeled examples or the size of available external memory space may be limited, and these can both lead to the limited memory-size.
this section, we evaluate MoT under different memory sizes. Specifically, we con- duct experiments on the randomly sampled subsets with different proportions and plot the correspond- ing performance curve in Figure 6. We observe that MoT can consistently lead performance im- provements. Even under 10 percent of the original memory pool, MoT can still outperform Zero-Shot- CoT and Few-Shot-CoTT These show the usability of MoT when unlabeled examples or available ex-
The Effect of Filtering To evaluate the effect of memory filtering in MoT, we plot the performance

---

DROP
ANLI-AI
ANLI-A3
OBQA
ister al. (2022); Ho al. (2022) leverage LLM to generate reasoning paths and teach small LMs to reason. Wang et al. (2022c) and Honovich et al. (2022) leverage the LLM to generate instruction data and improve the instruction-folloing capa- bility of the LLM. Schick et al. (2023) propose ToolFormer, which learns how to use various tools by self-generated data. Zelikman et al. (2022) and Huang et al. (2022) leverage the LLM gener- ate reasoning paths and improve itself using la- beled and unlabeled datasets, respectively. Dif- ferent from these methods that depend on expen- sive fine-tuning, MoT can make the LLM self- improve with memory-of-thouglus and does not depend on parameter updates and is compatible with API-accessing LLM. Recently, Zhang et al. (2022b); Shao et al. (2023) automatically generate COT demonstrations by the LLM itself. Li et al. (2022) leverage the LLM to generate the knowl- edge base and improve its ability of open-domain QA. These methods can be seen as the specialized case of MoT, with task-level memory selection or task sisssspecialized memory building.
Cow-Shot-COT
57.6
59.7
52.3
80.0
+MoT
65.7
65.6
55.2
82.3
Zero-Shot-COT
62.2
61.9
48.5
69.2
+MoT
66.6
65.9
54,0
81.5
Plan-and-Solve
60.5
62.6
52.1
71.4
+MoT
67.6
66.3
56.9
85.6
Table 4: Comparison of various CoT methods.
ternal memory space are limited.
Transferability across Different COT Methods In this section, we evaluate the performance of MoT on two additional COT methods: Zero- Shot-COT (Kojima al., 2022) and Plan-and- Solve (Wang et al., 2023). Compared with Zero- Shot-COT which uses "Let's think step by step" to clicit LLM's reasoning, Hla--nd--Sovv uses a spe- cialized prompt to let the LLM first devise a plan to divide the entire task into sub-tasks and then solve them based on the plan, and thus it can ac- complish more complicated reasoning (Wang et al., 2023). For these two CoT methods, generate the pool of memory-of-thouge thinking stage, respectively. At the test stage, we retrieve thoughts from the corresponding memory pool, concatenate them with the test question, and then use the corresponding prompt, e.g., "Let's think step by step" for Zero-Shot-Cor COT, to elicit the LLM's reasoning. Results on DROP, ANLI-A1, ANLI-A3 and OBQA are shown in Table 4. We ob- serve that MoT leads consistent improvements, which shows its stability and generality across vari- ous CoT methods. Moreover, since these two CoT methods do not rely on manual CoT demonstra- tions, these results also demonstrate the effective- ness of MoT when the manual CoT demonstration is not available. Meanwhile, when using the more advanced CoT method, Plan-and-Solle Solve, MoT's per- formance gets further improvements, which shows its potential in the future where the more powerful CoT method is proposed.
them
at pre-
Demonstration Retrieval for LLM In this sec- tion, we introduce previous demonstration retrieval methods for ICL, which mainly retrieve relevant input/output pairs, from an annotated dataset, for the LM to predict the test example. Liu et al. (2022) propose to leverage a dense semantic em- bedder to retrieve relevant examples to improve ICL. Agrawal al. (2022) leverage BM25 re- trieve examples for machine translation's ICL. Das et al. (2021) and Hu et al. (2022) design specialized target similarities to train demonstration retrievers on ICL of knowledge-based question answering and dialogue state tracking respectively. Rubin et al. (2022); Shi et al. (2022b) use the LM's feed- back train the demonstration retriever for mantic parsing. Lyu al. (2022) retrieve relevant examples with random labels and propose heuristic methods to reduce the negative effect of false la- bels. Recently, Li et al. (2023) propose UDR, a uni- fied demonstration retriever for various NLP tasks, which is trained by the unified LM-feedback on about 40 annotated datasets. While most of these methods depend on high-quality annotated datasets and only explore in-context learning without ra- tionales, MoT can make the LLM self-improve without annotated datasets and parameter updates, and to the best of our knowledge, we are the first to explore demonstration retrieval in the challenging
5 Related Work
Model Augmentation by L.M--geneatted Data In this section, we introduce previous methods that use the data generated by LLMs for model augmen- tation. Ye et al. (2022a); Gao et al. (2022); Ye et al. (2022b) propose ZeroGen, ProGen and ZeroGen to use the LLM to generate the dataset to enhance small models, e.g., LSTM. Fu et al. (2023a); Mag-

---

and complicated reasoning scenarios and demon- strate MoT's effectiveness.
References
Sweta Agrawal, Chunting Zhou, Mike Lewis. Luke 7ettlemoyer, and Marjan Ghazvininejad. 2022. In- context examples selection for machine translation. CoRR, abs/2212.024377.
6 Conclusion
In this paper, we propose MoT, a framework that let the LLM self-improve via Memory-of-Thore without annotated datasets and parameter updates. Experimental results show that MoT can help Chat- GPT significantly improve its abilities in arithmetic reasoning. commonsense reasoning, factual reason- ing and natural language inference. Further analy- ses show that each component contributes critically to the improvements and MoT can lead to consis- tent improvements across various CoT methods and LLMs.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent Shafey, Yanping Huang, Kathy Meier-Hellsterr Gau- rav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernández Ábrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choox Aakanksha Chowdhery. Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz. Nan Du, Ethan Dyer, Vladimir Feinberg, Fangxi- aoyu Feng. Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, and et al. 2023. Palm 2 technical report. CoRR, abs/2305.
Limitations
MoT mainly has the following limitations:
Although propose the answer-entropy to filter out uncertain thoughts, the remaining thoughts can still contain certain mistakes. We will explore more methods of false thought filtering (Lin et al., 2023) in the future.
Emily M. Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In FAceT 21: 2021 ACM Conference Fairness, Accountability, and Transparency, Virtual Event / Toronto, Canada, March 3-10, 2021, pages 610-623. ACM.
In this paper, we employ simple strategy to utilize the relevant memory, i.e., concatenate it with the test input Iest and thus help the LLM answer qeest- We will explore more strategies to utilize the retrieved memory, e.g., retrieving the memory to verify the current rcasoning path for Qiest-
Zekai Chen, Mariann Micsinai Balan, and Kevin Brown. 2023. Language models are few-shot learners for prognostic prediction. CoRR, 12022222......
On the one hand, in this paper, make the first step to let the LLM self-improve based on the memory mechanism. The conducted experiments are still in a safe setting, i.e., a specific unlabeled dataset, and the LLM can- not access the internet and control external tools. Hence we think our method and ex- periment are still safe enough, which will not cause serious impact and unrecoverable conse- quences on society. On the other hand, large language models have shown various kinds of bias (Bender et al., 2021). Since we let the LLM generate thougyys/memory to help itself, the LLM might suffer from the gener- ated biased content. We see LLM debias as an important future research topic.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez. Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran. Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim. Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lec, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier- Hells-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways. CoRR, abs/2204.02311
A Acknowledgements
This work was supported by the National Natural Science Foundation of China (No. 62236004 and No. 62022027).
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang,

---

Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery. Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang. Andrew M. Dai, Hongkun Yu, Slav Petrov, Fd H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction- CoRR, 0bs/2210.11416
Jiahui Gao, Renjie Pi, Yong Lin, Hang Xu, Jiacheng Ye, Zhiyong Wu, Xiaodan Liang. Zhenguo Li, and Lingpeng Kong. 2022. Zerogen** Self-guided high- quality data generation in efficient zero-shot learning. CoRR, abs/2205. 12679.
Namgyu Ho, Laura Schmid, and Se-Young Yun. 2022. Large language models are reasoning teachers. CoRR, abs/2212.11071.
-intuuttttoo---uueeeddeeeeee finetuned language models.
Christopher Clark, Kenton Lee. Ming- Wei Chang. Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2924-2936, Minneapolis, Min- nesota. Association for Computational Linguistics.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai. Eliza Rutherford. Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katic Millican, George Damoc, Aurelia Guy, Simon Osindero, Karen Si- monyan, Tiicc Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optima large language models. CoRR, 0tls//223.115555
den Driessche, Bogdan
Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya God- bole, Ethan Perez, Jay Yoon Lee, Lizhen Tan, Lazaros Polymenakos, and Andrew McCallum. 2021. Case- based reasoning for natural language queries over knowledge bases. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pages 9594-9611. Association for Computational Linguis- tics.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. Unnatural instructions: Tuning lan- guage models with (almost) no human labor. CoRR, Abs/2212.09689.
Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A. Smith, and Mari Ostendorf. 2022. In- context learning for few-shot dialogue state tracking. CoRR, abs/2203.0856
Jiaxin Huang. Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. Large language models can self-improve. CoRR, 1bs/2210.116100
A. J. Dijksterhuis and Loran F Nordgren. 2006. A the- ory unconscious thought. Perspectives on Psycho- logical Science, 1:109- 95.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjin Xu, Lei Li, and Zhifang Sui. 2022. A survey for in-context learning.
Jie Huang and Kevin Chen-Chuan Chang. 2022. To- wards reasoning in large language models: A survey. CoRR, abs/2212.10003.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requir- ing discrete reasoning over paragraphs. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378, Min- neapolis. Minnesota, Association for Computational Linguistics.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, 7ac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage. Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jack- son Kernion, Shauna Kravec, Liane Lovitt, Ka- mal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know. CoRR, A..22220..0221.
J. Dunlosky and J. Metcalfe. 2008 Metacognition. SAGE Publications.
Stan Franklin, Bernard Baars, Uma Ramamurthy, and Matthew Ventura. 2005. The role of consciousness in memory. Brains, Minds Media, I.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. CoRR, abs/2205.1199
Yao Fu, Hao Peng, Litu Ou. Ashish Sabharwal, and Tushar Khot. 2023a. Specializing smaller lan- guage models towards multi-step reasoning, CoRR, abs/2301.12726.
Andrew K. Lampinen, Ishita Dasgupta, Stephanie C. Y. Chan, Kory W. Mathewson, Mh Tessler, Antonia Creswell, James L. McClelland, Jane Wang, and Fe- lix Hill, 2022. Can language models learn from ex- planations in context? In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022,
Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023b. Specializing smaller lan- guage models towards multi-step reasoning. CoRR, abs/2301.12726.

---

pages 537-563, Association for Computational Lin- guistics.
Potsawee Manakul, Adian Liusie, and Mark J. F. Gales, 2023. Selfcheckgpt: Zero-resource black-box hal- lucination detection for generative large language models. CoRR, abs/2303.08896.
Itay Levy, Ben Bogin, and Jonathan Berant. 2022. Di- verse demonstrations improve in-context composi- tional generalization. CoRR, abs/2212.00800)
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christo- foros Nalmpantis, Ramakanth Pasumuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. 2023, Augmented language models: a survey. CoRR, abs/2302.07842.
Junlong Li, Zhuosheng Zhang, and Hai Zhao. 2022. Self-prompting large language models for open- domain QA. CoRR, abs/2212.08635..
Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023. Unified demonstration retriever for in- context learning. CoRR, abs/2305.04320.
Todor Mihaylov. Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct elec- tricity? a new dataset for open book question an- swering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. Association for Computational Linguistics.
Xiaonan Li and Xipeng Qiu. 2023. Finding sup- porting examples for in-context learning. CoRR, abs/2302.13539.
pages
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language under- standing. In Proceedings of the 58th Anmual Meet- ing of the Association for Computational Länguistics. pages 4885-4901, Online, Association for Computa- tional Linguistics.
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023. Generating with confidence: Uncertainty quantifica- tion for black-box large language models. CoRR, abs/2305.19187
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 158 167. Association for Computational Linguistics.
OpenAL. 2023. GPT-4 technical report. CoRR, 0bs/2303.08774.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. CoRR, abs/2203.02155.
Jaahhang Liu, Dinghan Shen, Yizhe Zhang. Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in in-context examples for gpt-3? In Pro- ceedings of Deep Learning Inside Out: The 3rd Work- shop on Knowledge Extraction and Integration for Deep Learning Architectures, DeeLIO@ @ ACL 2022, Dublin, Ireland and Online, May 27, 2022, pages 100-114. 14. Association for Computational Linguistics.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. CoRR, abs/2210.03350.
Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju. 2020. Fastbert: self- distilling BERT with adaptive inference time. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pages tion for Computational Linguistics.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey. CoRR, abs/2003.082 .08211.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bers Sentence embeddings using siamese nert-networks. In Proceedings of the 2019 Conference on Empiri- cal Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-JJCN 2019, Hong Kong, China, November 3-7, 2019, pages 9980-3990. Association for Computational Linguistics.
Associa-
Jieyi Long. 2023. Large language model guided tree-of- thought. CoRR, 008291.
Xinxi Lyu, Sewon Min, Iz Beltagy, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022. Z-ICL: zero-shot in- context learning with preudo-demoosttaaionns.. CoRR, abs/2212.09865..
Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and he- yond. Found. Trends Inf. Retr. 3(4):333-335 389.
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adámek, Eric Malmi, and Aliaksei Severyn. 2022. Teaching small language models to reason. CoRR, abs/2211.08410.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of

---

the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACI, 2022, Seattle, WA, United States, July 10-15, 2022. pages 2655-2671, Association for Computational Linguistics.
Anh Vuong, Animesh Gupta, Anna Gottardi, Anto- nio Norelli, Anu Venkatesh, Arash Gholamidavoodi. Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Her- rick, Avia Efrat, Aykut Erdem, Ayla Karakas, and et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. CoRR, abs/2206.046135
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang. Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Févry, Jason Alan Fries, Ryan Techan, Teven Le Scao. Stella Biderman, Leo Gao. Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event. April 25- 2022. OpenReview.net.
Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang. Mari Osten- dorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2022a. Selective annotation makes language models better seot learners. CoRR, abs/2209.00 00995
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang. Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022b. One embedder, any task: Instruction-finetee beddings. CoRR, abs/2212.09741.
text em-
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022. Unifying lan- guage learning paradigms. CORR, abs/2205.05133
Daniel L. Schacter and Donna Rose Addis. 2007. The cognitive neuroscience of constructive memory: Re- membering the past and imagining the future. In Jon Driver, Patrick Haggard, and Tim Shallice, edi- tors, Mental Processes in the Human Bruin. Oxford University Press.
Peter Todd, Jean Ortega, Jennifer Davis, Gerd Gigeren- zer, Daniel Goldstein, Adam Goodie, Ralph Hertwig. UIrich Hoffrage, Kathryn Laskey, Laura Martignon, and Geoffrey Miller. 1999. Simple Heuristics That Make Us Smart.
Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. CoRR, abs/2302.04761.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière. Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, Abs/2332.13377
Zhihong Shao, Yeyun Gong, Yelong Shen, Min- lie Huang, Nan Duan, and Weizhu Chen. 2023. Synthetic prompting: Generating ohain-of-thougle demonstrations for large language models. CORR, abs/2302.00618.
Endel Tulving, 2002. Episodic memory: From mind to brain. Anmual review of psychology, 53:1-25.
Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. 2019. Does it make sense? and why? a pilot study for sense making and explana- tion. In Proceedings of the 57th Anmual Meeting of the A Association for Computational Linguistics, pages 8020-4026, Flornce, Italy. Association for Compu- tational Linguistics.
Peng Shi. Rui Zhang, He Bai, and Jimmy Lin. 2022a. XRICL: cross-lingual atrieval-augmented in-context learning for cross- -ingual text-to-sql semantic pars- ing. CORR, abs/2210.1366
Peng Shi, Rui Zhang, He Bai, and Jimmy Lin. 2022b. XRICL: cross-lingual retrieval-augmenter in-context learning for cross- lingual text-to-sql semantic pars- ing. CoRR, abs/2210. 13.1933....
Lei Wang, Wanyu Xu, Yihuai Lam, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ec-Peng Lim. 2023. Plan- and-solve prompting: Improving zero-shot chain-of- thought reasoning by large language models. CoRR, abs/2305. Abs99305.04091.
Aarohi Srivastava, Abbinav Rastogi. Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Maariga-Alonse Agnieszka Kluska, Aitor Lewkowyez, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Par- rish, Allen Nie, Aman Hussain. Amanda Askell, Amanda Dsouza, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen,
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, and Denny Zhou. 2022a. Rationale- augmented ensembles language models. CoRR, ths/2207.00741
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, and Denny Zhou. 2022b. Self- consistency improves chain of thought reasoning in language models. CoRR, abs/2203. 11111.

---

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, AI- isa Liu, Noah A. Smith, Daniel Khashabi, and Han- naneh Hajishirzi. 2022c. Self-instruct: Aligning lan- guage model with self generated instructions. CoRR, abs/2212.10566)
A deeper look into input-label demonstrations, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 2422-2433 Association for Com- putational Linguisties.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoc V. Le. 2022a. Finetuned language models are zero-shot learners. In The Tenth International Conference Learning Representa- rions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Eric Zelikman, Yuhuai Wu, and Noah D. Goodman. 2022. Star: Bootstrapping reasoning with reasoning. CoRR, abs/2203.14465
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle OiL, Sam Shleifer, Kurt Shus- ter, Daniel Simig, Punit Singh Koura, Anjali Srid- har, Tianlu Wang, and Luke Zettlemoyer. 2022a. OPT: open pre-trained transformer language mod- els. CoRR, abs/2205.01005
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang. Jeff Dean, and William Fedus. 2022b. Emer- gent abilities of large language models. CoRR, fbs/2206.07682.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022b. Automatic chain of thought prompting in large language models. CoRR, abs/2210.03493.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. 2022c. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Newral Information Processing Systems.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang. Yupeng Hou, Yingqian Min, Be- ichen Zhang. Junjie Zhang, Zican Dong, Yifan Du Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang. Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. 2023. A survey of large language models. CoRR, 0hs/2303.1822.
Yixuan Weng, Minjun Zhu, Shizhu He, Kang Liu, and Jun Zhao. 2022. Large language models are reason- ers with seff-verification. CoRR, abb/2211.00950
Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. 2020. Deebert: Dynamic early exiting for accelerating BERT inference. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, ACL 2020, Online, July 5-10. 2020, pages 2246-2251. Association for Computa- tional Linguisties.
Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. 2023. Progressive-hin prompting improves reasoning in large language models. CoRR, abs/2304.09797
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans, Olivier Bousquet, Quoe Le, and Fd H. Chi. 2022. Least-to-most prompting enables complex reasoning in large language models. CoRR, ab6/2205.10625.
Jiacheng Ye, Jiahui Gao. Qintong Li, Hang Xu, Jiang- tao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022a. Zerogen: Efficient zero-shot learning via dataset generation. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11653- 11669. Association for Computational Linguistics.
Jiacheng Ye, Jiahui Gao, Zhiyong Wu, Jiangtao Feng. Tao Yu, and Lingpeng Kong. 2022b. Progen: Pro- gressive zero-shot dataset generation via in-context feedback. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 3671-3683. Association for Computational Linguis- tics.
Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning. CoRR, abs/2302.05698.
Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyun- soo Cho, Hwiyeol Jo. Sang-Wes Woo Lee, Sang-goo Lee, and Taeuk Kim. 2022. Ground-truth labels matter:

---

A The Example of LeM-Retrieval
including DROP and WikiQA, we use the Fl-score as evaluation metric. For DROP, since its one test example has multiple annoated answer, we follow its original paper (Dua et al., 2019) to take a max over all annotated answers. Limited by budget, for those evaluation datasets that are larger than 1000, we randomly sample a subset of 1000 examples for evaluation. We list the overall dataset satistiscs, the size of memory after filtering and evaluation metrics in Table 7.
We show the LLM-retrieval example in Table 6.
B Dataset Details
Overview We conduct experiments on ten datasets, including four task families:
Arithmetic reasoning: AQuA (Ling et al., 2017): A multi-choice dataset of arithmetic questions covering various topics and diffi- culty levels, and DROP (Dua et al., 2019): A reading comprehension dataset that needs discrete reasoning;
52
2 51
50
Natural Language Inference: Adversarial NLI subsets (Nie et al., 2020), including ANLI-AI, ANLI-A2 and ANLI-A3, which cover varying difficulty levels respectively;
49
48
20
(a) DROP
(b) ANLI-A3
82
Commonsense Reasoning: (OpenBookQQ) (Mihaylov et al., 2018):
OBQA
83
81-
8 81-
80-
79
Commonsense-related questions which require the facts and their applications to novel situations, and ComV (Commonsense Validation) (Wang al., 2019): dataset that requires for identifying the sentence that does not make sense from two sentences of similar wording;
78
78
77
k
(c) OBQA
(d) FacICK
Figure 7: The impact of demonstration quantity.
C Performance on Different LLMs
Factual Reasoning: BoolQ (Clark et al., 2019), FactCK (Fact Checker) (Srivastava et al., 2022): A dataset that tests the ability to evaluate the authenticity of factual claims covering Wikipedia, COVID-19 and Politics. WikiQA (Srivastava al., 2022): question answering fron randomly-sanpled Wikidata fact triples.
We conduct experiments on Text-Davinci-O0) and Text-Davinci-003 (Chen et al., 2023; Ouyang et al., 2022) to evaluate MoT's generality across different LLMs. We show the results in Table We observe that MoT consistently outperforms baselines on these two LLMs, which shows the effectiveness of MoT does not rely on one specific LLM and it can bring further improvements in the future where the more strong LLM is proposed.
Split, Evaluation Metric and Statistics For AQuA, DROP, ANLI-A1, ANLI-A2, ANLI-A3, ComV and OBQA, we use their official test set for evaluation. For BoolQ, we follow Wang al. (2022a) to use the validation set for evaluation, since its test set is not publicly available. For FactCK and WikiQA, we manually split them into a train/test split, and use the questions of the train- ing set as unlabeled dataset, since there is not split version of them released. Limited by the budget, for the DROP dataset, we only use the half of its unlabeled dataset (the questions of training set) for the L.MM to pre-think. For the classification or multi-choice datasets, we use the accuracy as evaluation metric. For the abstractive QA dataset
D The Impact of Demonstration Quantity
We compare MoT and Few-Shot-CoT under vary- ing numbers of demonstrations and the results are shown in Figure 7. We see that MoT consis- tently outperforms Few-Shot-CoT across varying amounts of demonstrations, which shows the stabil- ity of MOT. Additionally, the results show that the demonstrations in retrieved memory more help- ful and informative than manual demonstrations in Tew-Shot-Coff specifically, with or 2 demonstra- tions, MoT can outperform Sow-Shoot-oTT with 4 demonstrations on OBQA and DROP

---

Method
ANLI-A3 OBQA BoolQ FacICK
ble 8, 9, 10, 12, 11, 13, 14 and 15.
Text- D0vi-ivvini-022
F Ethics Statement
Zero-Shot-CoT
34.2
47.2
40.8
52.2
Few-Shot-CoT
46.9
75.4
58.8
83.6
In this paper make the first step let the LLM self-improve based on the memory mechanism. The conducted experiments are still safe set- ting, i.c., specific unlabeled dataset, and the LLM cannot access the internet and control external tools. Hence we think our method and experiment are still safe enough, which will not cause serious impact and unrecoverable consequences socicty.
MoT
49.1
80.0
63.9
86.2
Text- Davincii-03
Mero-Shot-Coo
46.9
64.2
62.8
47.2
T
45.3
81.6
67.7
84.0
MoT
49.3
84.6
71.0
87.0
Table 5: Performance comparison on various LLMss.
E Implementation Details
We use the public OpenAI language model of "gpt- 3.5-turbo-0301" unless otherwise specified and the experiments (Appendix C) on "text-davinci- 002/003" show consistent improvements. Due to the limitation of LLM-API budget, heuristically set the hyper-parameters in the pre-thinking stage, including the generation temperature T, the num- ber of decoded reasoning paths n. For the filtering threshold and the number memory clusters I, we conduct exploratory experiments on OBQA and find that r = {0.2, 0.3, 0.4] and / = (3,4,5) lead to similar performance. Thus we set т = 0.3 and / = 4, respectively. For pre-thinking, we use the temperature 7 1.2 encourage more diverse 16 reasoning path sampling times, unless otherwise specified. For memory recall, we use SBERT ("all-mpnet-base v2") (Reimers and Gurevych, 2019) for semantic filtering. Limited by the LLM's max input length, we fix the the number of each cluster's memory can- didates 10 for each dataset. In the test stage, for greedy decoding to the stability of results, otherwise specified. For generate the output, unless
reasoning paths, and
simplicity, we separately run MoT on each dataset and regard cross-dataset memory recall as future work. Baselines' points are from our implementa- tion, and share the same templates, answer parsing and evaluation MOT.
For AQuA, OpenBookQA, BoolQ, DROP, ANLI-A1, ANLI-A2 and ANLI-A3, We use the same few-shot CoT examples as those in Wei et al. (2022c), Zhou et al. (2022), Wang et al. (2022b) and Wang al. (2022a), respectively. For the left datasets that have no publicly released manual CoT demonstrations, randomly select questions from the training set and use ChatGPT to generate rea- soning paths and get their few-shot CoT examples. We list the used Sew-Shot-CCh CoT examples in Ta-

---

LLM Input
I will provide you with target question and 10 reference questions. I need you to choose reference question from "Reference Questions", whose question, train of thought answer would be most helpful for you to answer the target question. Please note that the following reference QA pairs are presented in random order without any prioritization.
Target Question:
Machine puts out yo-yo every minutes. Machine puts yo-yo every minutes. After how many minutes will they have produced 10 yo-yos? Answer Choices: (A) 24 minutes (B) 32 minutes (C) 36 minutes (D) minutes (F) 72 minutes
Reference Questions:
Q: Two machines, Y and Z. work at constant rates producing identical items. Machine Y produces items the same time Machine Z produces items. machine Y takes minutes to produce batch of items, how many minutes does it take for machine Z produce the same number items? Answer Choices: (A) 6 (B) 9 1/2 (D) 22.5 (E) 13 1/2
Q: Two machines, and Z, work at constant rates producing identical items. Machine Y produces items in the same time Machine Z. produces items. If machine takes 19 minutes to produce batch items, how many minutes does take for machine 7. to produce the same number items? Answer Choices: (A) (B) (C) 9 1/2 (D) 15 (E) 13 1/2
Q: Two machines, Y and Z, work at constant rates producing identical items. Machine Y produces 30 items in the same time Machine Z produces 24 items. If machine Y takes 36 minutes to produce batch of items, how many minutes does it take for machine Z to produce the same number of items? Answer Choices: (A) 60 (B) 90 (C) 9 1/2 (D) 45 (E) 13 1/2
Q: Working alone at its constant rate, machine A produces x hoxes in 10 minutes and working alone at its constant rate, machine B produces 2x boxes in 5 minutes. How many minutes does it take machines A and B, working simultancously at their respective constant rates, to produce 10x boxes? Answer Choices: (A) 13 minutes (B) 14 minutes (C) 15 minutes (D) 16 minutes (E) 20 minutes 5.
Q: Two machines. Y and Z, work at constant rates producing identical items. Machine Y produces 23 items in the same time Machine Z produces 21 items. If machine Y takes 21 minutes produce batch of items, how many minutes does it take for machine Z to produce the same number of items? Answer Choices: (A) 6 (B) 9
(C) 1/2 (D) 12 (E) 23
6.
Q: Machines X and Y produce bottles at their respective constant rates. Machine X produces k bottles in 6 hours and machine Y produces bottles in 12 hours. How many hours does it take machines X and Y, working simultaneously to produce 12k bottles? Answer Choices: (A) 8 (B) 12 (C) 15 (D) 48 (E) 24
7.
Q: Machines X and Y produce bottles their respective constant rates. Machine X produces k bottles in hours and machine Y produces k bottles in hours. How many hours does take machines X and Y, working simultaneously produce 12k bottles? Answer Choices: (A) 4 (B) (C) 12 (D) 18 4
8.
Q: Machines X and produce bottles their respeetive constant rates. Machine X produces bottles in hours and machine Y produces k bottles in hours. How many hours does take machines X and working simultanenusly. produce 10k bottles? Answer Choices: (A) 2/3 (R) 5/3 (C) (D) (F) 22 2/9
Q: Working alone at its constant rate, machine A produces x boxes in 10 minutes and working alone at its constant rate, machine B produces 2x boxes in minutes. How many minutes does it take machines A and B, working simultaneously at their respective constant rates, to produce 6x boxes? Answer Choices: (A) minutes (B) minutes (C) minutes (D) minutes (E) minutes 10.
Q: Machine A can make 350 widgets in hour, and machine B can make 250 widgets hour. both machines work together, how much time will take them to make a total 900 widgets? Answer Choices: (A) hour and 20 minutes (B) hour and minutes (C) hour and 30 minutes (D) hour and 36 minutes (F) hour and 40 minutes
Which one of the above reference questions is the most helpful question for you to answer the target question? You must choose exactly one reference question to you answer the target question. Your response must end in this format: "The most helpful question is question [index].". For example, if question 5 is your answer, you must end in "The most helpful question is question 5."
LLM output
The most helpful question is question 10.
Table 6: The example of JLM-Retrievall

---

Task Family
Task
Task Format
Unlahcled Questions Test Questions Memory Size
Metric
AQuA
Multi Choice
97467
19334
Accuracy
Arithmetic Reasoning
DROP
Abstractive QA
42777
1000
17066
F1
ANLI-A1
Classification
16946
1000
9721
Accuracy
NLI
ANLI-A2
Classification
45460
1000
25444
Accuracy
ANIJ-A3
Classification
100459
1000
53402
Accuracy
Commonsense
ComV
Multi Choice
10000
1000
9003
Accuracy
Reasoning
OBQA
Multi Choice
4957
1000
3559
Accuracy
BoolQ
Classification
9427
1000
6422
Accuracy
Factual
FacICK
Classification
6654
500
4782
Accuracy
Reasoning
WikiQA
Abstractive QA
19442
1000
9041
F1
Table 7: The statisties of used datasets and their evaluation metrics.
Q: John found that the average of 15 numbers is 40. If 10 is added to each number then the mean of the numbers
is? Answer Choices: (A) 50 (B) 45 (C) 65 (D) 78 (E) 64
A: If 10 is added to each number, then the mean of the numbers also increases by 10. So the new mean would be
50. The answer is (A).
Q: If b = 3/4 and 8a 5b = 2hethen find the value of a. Answer Choices: (A) 1/2 (B) 3/2 (C) 5/2 (D) 4/2 (E)
7/2
A: If a 3/4, then b = 4a / 3. So 8a +
= 22. This simplifies to 8a + 20a / = 22, which means 44a /
22. So a equal to 3/2. The answer is (B).
Q: A person is traveling at 20 km/hr and reached his destiny in 2.5 hr then find the distance? Answer Choices:
(A) 53 km (B) 55 km (C) 52 km (D) 60 km (E) 50 km
A: The distance that the person traveled would have been 20 km/hr * 2.5 hrs = 50 km. The answer is (E).
Q: How many keystrokes are needed to type the numbers from to 500? Answer Choices: (A) 11566 (B) 1392
(C) 1480 (D) 1562 (B) 1788
A: There are 9 one-digit numbers from to 9. There are 90 two-digit numbers from 10 to 99, There are 401 three-digit numbers from 100 to 500. 9 90(2) 401(3) = 1392. The answer is (B).
Table 8: Few-shot CoT prompts for AQuA, from Wang al. (2022b).

---

Francisco 49ers. In first quarter, Hawks Julius Jones got 27-yard TD run, along with DT Craig Terrill returning a fumble yards for touchdown. the third quarter, the 49ers almost rallied as H. J. Torres made 12-yard TD pass to Lucas Nelly, along with Mare kicking 32-yard field goal. the final quarter, Julius Jones got another 1--yard TD. How many yards the shortest touchdown run and the longest touchdown pass combine for?
Q: Seahawks played
A: All the touchdown runs are: 27-yard touchdown run, a 9-yard touchdown run, 1--yard touchdown run. The smallest numher among 27, 9, 9. So the shortest touchdown run was yards. All the touchdown passes are: a 12- 12-yard touchdown pass. So the longest touchdown pass was 12 yards. So the shortest touchdown run and the longest touchdown pass combine for 9 + 12 yards. So the answer is 21 yards.
Q: the county, population was spread out with 23.50% under age 18, 8.70% from 24, 29.70% from 25 to 44, 24.70% from 45 64, and 13.30% who were years age or older. How many more percent are under the age
compared to 18 to 24 group?
A: According the passage, 23.5% % are under the age 18, and are from ages 18 to 24. 23.5% 8.7% 14.8%. the answer is 14.8.
Q: Since the 1970s, U.S. governments negotiated managed-trade agreements, as the North American Free Trade Agreement the 1990s, Dominican Republic-Central America Free Trade Agreement 2006, and a number of bilateral agreements. Europe, six countries formed the European Coal and Steel Community 1951 which became the European Economic Community 1958. Two care objectives the EEC were development common market, subsequently renamed single market, and establishing a customs union between its member states. many years did the European Coal and Steel Community exist?
A: According to the passage, European Coal Steel Community was established 1951 and became EEC 1958, 1958 1951 = So answer is
Q: The Vikings flew to Bank of America Stadiur face the Carolina Panthers. After scoreless first quarter, Carolina got the board with quarterback Matt Moore finding fullback Brad Hoover on I-yard TD pass. After yet another scoreless quarter, Carolina sealed the game Matt Moore completed 42-yard touchdown pass wide receiver Steve Smith. How many scoreless quarters there?
A: The first and third quarters
scoreless quarters. So there scoreless quarters. the answer
Table 9: Few-shot CoT examples for DROP, from Zhou et al. (2022).

---

Premise:
"Conceptually cream skimming has two basic dimensions product and geography." Based on this premise, can we conclude the hypothesis "Product and geography are what make cream skimming work." is true?
OPTIONS:
-es
it is not possible tell
A: Based on "cream skimming has two hasic dimensions" can't infer that these two dimensions what make cream skimming work. The answer it is not possible tell.
Premise:
"One of our member will carry out your instructions minutely." Based this premise, can conclude the hypothesis "A member of my team will execute your orders with immense precision." true?
OPTIONS:
yes
it is not possible tell
A: "one of" means the same "a member of". "carry out" means the same "execute", and "minutely" means the same "immense precision". The answer yes.
Premise:
"Fun for adults and children."
Based this premise, can conclude the hypothesis "Fun for only children." " is true?
OPTIONS:
-yes
is not possible tell
A: "adults and children" contradicts "only children". The is no.
Premise:
"He turned and smiled Vrenna."
Based this premise, can conclude the hypothesis "He smiled Vrenna who walking slowly behind him with her mother." is true?
OPTIONS:
yes
is not possible tell
A: the premise does not say any thiting about "Vrenna was walking". The answer not possible to tell.
Table 10: Few-shot CoT prompts for NLI tasks, three subsets ANLI from Wang et al. (2022a).
Q: Poison causes harm which of following? (A) Tree (B) a robot (C) a house (D) car
A: Poison will harm living things, only tree living thing. The answer (A).
Q: As you look deeper into Marbel you (A) the future minut defects (C) colors (D) the other side
A: Marbel transparent, you can the other side. Marbel does necessarily have multiple colors. You will minut defects. The answer is (R).
Q: When food is reduced in the stomach (A) the mind needs time to digest (B) take second to digest what I said (C) nutrients are heing deconstructed (D) reader's digest is a body of works
A: The food is being deconstructed in the stomach during digestion. The answer is (C).
Q: The sun is responsible for (A) puppies learning new tricks (B) children growing up and getting old (C) flowers wilting in vase (D) plants sprouting. blooming and wilting
A: The sun can affect the growing of living things. like plants. The answer is (D).
Table 11: Few-shot CoT examples for OBQA, from Wang al. (2022a).

---

Q: Which one the following statements is against common sense? (A) Roses buds cat caterpillars (B) The caterpillar eats the rose bud
A: Statement (A) is against common sense as it goes against the natural food chain and the known behavior of
roses. Roses are plants and cannot eat or consume other organisms, including caterpillars. The answer is A.
Q: Which one the following statements against common sense? (A) He threw house into the trash bin (B) He threw his food waste into the trash
A: It not physically possible to throw a house into trash bin. Statement (A) goes against the laws of physics and is therefore illogical. The answer is A.
Q: Which one of the following statements is against common sense? (A) Because was damaged, he received RMB 1000 from electricity company (B) Because car was damaged, he received RMB 1000 from insurance company
A: It does not make logical sense for electricity company to compensate someone for car damage. is more reasonable for an insurance company to provide compensation for car damage. The answer
Q: Which one of the following statements is against common sense? (A) Because his was damaged, he received RMB 1000 from electricity company (R) Because his was damaged, he received RMB 1000 from insurance company
A: It does not make logical sense for electricity company compensate someone for car damage. It is more reasonable for an insurance company to provide compensation for car damage. The answer
Table 12: Few-shot CoT examples for ComV.
Q: does system of a down have 2 singers?
A: System of Down currently consists of Serj Tankian, Daron Malakian, Shavo Odadjian and John Dolmayan.
Serj and Daron do vocals, so the band does have two singers. The answer is yes.
Q: do iran and afghanistan speak the same language?
A: Iran and Afghanistan both speak the Indo-European language Persian. The answer is yes.
Q: is a cello and a bass the same thing?
A: The cello is played sitting down with the instrument between the knees, whereas the double bass is played standing or sitting on a stool. The answer is no.
Q: can you use oyster card at epsom station?
A: Epsom railway station serves the town of Epsom in Surrey and is not in the London Oyster card zone. The answer is no.
Table 13: Few-shot CoT examples for BoolQ, from Wang et al. (2022a).
Q: On June 2017, following claim was made: David Lloyd George lost every bid become prime minister. Was this claim true or false?
A: David Lloyd George served as the Prime Minister the United Kingdom from 1916 to 1922, He also served as the Chancellor the Exchequer and the Minister Munitions before becoming Prime Minister. Therefore, the claim that lost every bid to become Prime Minister is false. The answer is false.
Q: On June 2017, the following claim was made: In 1966, George Harrison got married for the first time. Was this claim true false?
A: George Harrison married his first wife, model Pattie Boyd, on January 21, 1966. The answer true.
Q: On June 2017, the following claim was made: Woodrow Wilson did not live during World War I, Was this claim true false?
A: Woodrow Wilson the President of the United States during World War serving from 1913 1921. The answer is false.
Q: On April 17 2008, the following claim was made: Hillary Clinton has taken over $800,000 from lobbyists. Was this claim true false?
A: According OpenSecrets.ore .org, nonpartisan research group that tracks money in politics, Hillary Clinton received over $800,000 in campaign contributions from lobbyists during her 2008 presidential campaign. The answer is true.
Table 14: Few-shot CoT examples for Fact Checker.

---

Q: On June 2017, 7, the following claim was made: David Lloyd George lost every bid become prime minister. Was this claim true or false?
A: David Lloyd George served as the Prime Minister of the United Kingdom from 1916 to 1922. He also served as the Chancellor of the Exchequer and the Minister of Munitions before becoming Prime Minister. Therefore, the claim that he lost every bid to become Prime Minister is false. The answer is false.
Q: The native language of Aaron Swartz is?
A: Aaron Swartz was born Chicago, Illinois, United States. Therefore, native language is most likely English, as it the primary language spoken the United States. The answer English.
Q: religion Prajadhipok is?
A: Prajadhipok was a Buddhist, as Buddhhism predominant religion in Thailand, where he was the absolute monarch before the country became a constitutional monarchy. The answer is Buddhism.
Q: The country of Valletta is?
A: Valletta the capital city of Malta, which is small island nation located in the Mediterranean Sea. The is Malta.
Q: The sport played by Garry Kasparov is?
A: Garry Kasparov is former world chess champion, therefore the sport played him is chess. The answer is chess.
Table 15: Fhw-shot CoT examples for WikiQA.


Reviews:
[{"review_id": "3Ewbju1OWV", "reviewer": "Reviewer_v4Dz", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive review and insightful suggestions. Below is our response to the comments and questions.\n\n\n**Q1**: What distinguishes our proposed method, MoT, from existing self-prompting work, \"Self-Prompting Large Language Models for Zero-Shot Open-Domain QA - Li et al\"?\n\nAt a high level, MoT and self-prompting are guided by distinct motivations, conceptual perspectives, and design methodologies. MoT leverages the **memory mechanism** to enhance various capabilities of Large Language Models (LLMs) like ChatGPT over challenging reasoning scenarios, without annotated datasets and parameter updates. In contrast, self-prompting primarily focuses on utilizing LLMs to generate wiki-QA pairs for one single language understanding task, open-domain QA.\n\nWhen Self-prompting depends on **a specialized pipeline** (passage generation, entity recognition, question generation, QA explanation) to generate Wikipedia QA pairs for open domain QA, MoT is proposed to make LLM, e.g., ChatGPT, self-improve based on the memory mechanism. To the best of our knowledge, MoT is **the first framework** to make the LLM self-improve over various abilities based on the memory mechanism. \n\nSpecifically, MoT consists of these components: \n\n**1. Memory Generation:** This enables the LLM to **self-think** to build the external memory pool, which is both extensible and compatible with **various CoT** methods.\n\n**2. Memory Filtering:** This filters out the generated incorrect memory by **the LLM's own uncertainty**.\n\n**3. Memory Recalling:** The **LLM retrieves helpful memory by itself** (while self-prompting uses the typical dual-encoder).\n\nAs a result, the MoT can make the LLM self-improve various abilities over challenging reasoning scenarios: arithmetic reasoning, commonsense reasoning, factual reasoning, and natural language inference, while the self-prompting method is specialized for one natural language understanding task, open domain QA. \nWe hope that MoT can inspire researchers of the important design choices about making the LLM self-improve with memory mechanisms and pave the way for further improvements.\n\nOur further analysis also shows that MoT is effective and compatible with different CoT methods or LLMs (while self-prompting depends on one specialized data generation pipeline), and its performance improves as the CoT methods or the used LLMs improve, which shows MoT's generality and extensibility, and its potential in the future where more powerful CoT methods are proposed.\n\nWe discussed the difference between MoT and self-prompting in the section \"Related Work\" (Line 596~600) and we will discuss more about this difference to clarify the novelty of MoT in the next version.\n\nThank you again for your valuable review. If we have misunderstood any aspect of your review, or if you have additional questions, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is meticulously structured and articulated, making it exceptionally easy to follow and understand.\n* The author propose a nice method, which does not required labeled datasets but can achieve better performance than the original few-shot in context learning.\n* The paper carries out various experiments for the MOT framework on four different task families, providing strong evidence of its effectiveness and practical applicability.", "reasons_to_reject": "* What most confuse me is that I am not sure how much novalty the method MOT has. There are some self-prompting methods have been proposed early (such as [1]). This paper considers the self-prompting example as the memory and propose some new filtering and selecting method to recall the most helpful one, which has limited novelty for me. Could you discuss more about what is the different between your work and the previous self-prompting work.\n\n[1] Self-Prompting Large Language Models for Zero-Shot Open-Domain QA - Li et al.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors introduce a framework called MoT, which enables the Large Language Model (LLM) to self-enhance using the Memory-of-Thought technique. Initially, numerous examples are generated from unlabeled datasets, which are then treated as external memory for use during inference. In the testing phase, instead of relying on semantic embedders, a large language model is employed to recall the top-k most relevant memory examples, which are then prepended to the query in order to get the final answer. Experimental findings demonstrate that MoT significantly enhances ChatGPT's performance in areas such as arithmetic reasoning, commonsense reasoning, factual reasoning, and natural language inference.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gGTxkCpYp4", "reviewer": "Reviewer_qXWz", "summary": "", "questions": "\n\n\n\n1. Most of the experiments are carried out on ChatGPT. It would be also interesting to test your approach on open-source LLMs, such as LLaMA, Llama-2, etc. \n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive review and insightful suggestions. Below is our response to the comments and questions.  \n\n**Q1**: Regarding experiments on LLaMA models\n\nWe agree with you that whether the method is effective on the open-source model is important. In this version, we conduct experiments on ChatGPT and Text-Davinci-002,003 (please see Appendix C), and we will add the experiments on LLaMAs in the next version.\n\n**Q2**: Figures 4,5,6 are too small and the typos on line 309\n\nThanks for pointing out these issues, and we will fix them in the next version.\n\n\n\n**Q3**: Low Reproducibility\n\nWe have noted your concern about the reproducibility of our study. \n\nWe have provided the following details in the submitted paper:\n\n**Used Prompting Templates**: Appendix A\n\n**Implementations  (including hyper-parameters)**: Appendix E\n\n**Dataset Details**: Appendix B, the used datasets are all publicly available.\n\nWe will release the code and the generated CoT reasoning paths after review, to facilitate future research.\n\nHowever, we understand that more information might be needed to fully address this issue. Could you kindly specify which implementation details or hyper-parameters you found to be insufficient for reproducibility?\n\n\nThank you again for your valuable review. If we have misunderstood any aspect of your review, or if you have additional questions, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n\n1. The creation of MoT framework allowing an LLM to improve its reasoning abilities using memory mechanisms, without parameter updates and annotated datasets.\n2. Comprehensive experimentation showing that the MoT framework can lead to significant improvements in various areas for ChatGPT without requiring annotated datasets or parameter updates.\n3. The promise to make the code and generated CoT reasoning paths available to the wider research community, potentially inspiring future research and improvement efforts.\n\n\n", "reasons_to_reject": "I have no major reasons to reject this paper. As a minor comment, the authors should have uploaded their code/resources to reviewers but there is none. As the authors say they will open-source their code/resources, I will not blame on this too much.", "questions_to_authors": "\n\n\n\n1. Most of the experiments are carried out on ChatGPT. It would be also interesting to test your approach on open-source LLMs, such as LLaMA, Llama-2, etc. \n", "paper_topic_and_main_contributions": "This paper proposes a framework, Memory-of-Thought (MoT), designed to enhance the self-improvement capabilities of Large Language Models (LLMs) without the need for expensive fine-tuning or annotated datasets. \n\nInspired by the human ability to improve through metacognition and memory mechanisms, the MoT framework allows the LLM to self-improve, preserving its general capabilities. This approach takes its cues from how memory aids human decision-making, reasoning, and judgment.\n\nThe MoT method uses a two-stage process: firstly, it \"pre-thinks\" on an unlabeled dataset, saving high-confidence thoughts as an external memory; secondly, during a test phase, the LLM uses this memory to aid its reasoning and responses to given questions.\n\nThe researchers report that their experiments show MoT can significantly enhance ChatGPT's abilities in arithmetic reasoning, commonsense reasoning, factual reasoning, and natural language inference without the need for parameter updates or annotated datasets. They also note that each component of MoT contributes critically to the improvements and that the framework consistently enhances various CoT methods and LLMs.\n\nThe key contributions of this paper include:\n\n1. The creation of MoT framework allowing an LLM to improve its reasoning abilities using memory mechanisms, without parameter updates and annotated datasets.\n2. Comprehensive experimentation showing that the MoT framework can lead to significant improvements in various areas for ChatGPT without requiring annotated datasets or parameter updates.\n3. The promise to make the code and generated CoT reasoning paths available to the wider research community, potentially inspiring future research and improvement efforts.\n", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "\n\n1. The figures (flg.4, 5, 6) are too small. I know that this might be due to the page limit, so please revise this in your next version.\n2. line 309: LLms(Ye et al., 2023 -> LLMs (Ye et al., 2023\n\n\n"}, "extra_text": {}}, {"review_id": "rPlBf1jjCV", "reviewer": "Reviewer_E9wd", "summary": "", "questions": "As the simply implementation for this memory of thought method:\n\nI have the following questions:\n\nQ1: It is mentioned in the paper that answer entropy is used to filter uncertain thinking, so how to define and calculate answer entropy? Does this approach work consistently across different types of tasks and datasets?\n\nQ2: In the current experimental setup, the paper mentions that LLM cannot access the Internet and control external tools. How does this constraint affect the effectiveness of the MoT framework? Are there plans to conduct experiments in more open settings to further validate the performance of the framework?\n\nQ3: Can we expect more comparative experiments on MoT frameworks and other CoT methods in future research? How to ensure the fairness and reliability of these comparative experiments?\n\nQ4: For the bias problem of large language models, how to ensure that no bias is introduced in the thinking and memory process generated by LLM? Has the generated thinking content been analyzed for bias? How to address and mitigate the problems that arise from potentially biased generated content?\n\nQ5: For unknowable and invisible out-of-domain data sets, is there any experiment to illustrate the generalization of the proposed method? That would be a nice move.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and great suggestions. We have organized our responses according to the \"Reasons to Reject\" (denoted by **R**) and \"Questions\" (denoted by **Q**).\n\n**R1**: The Effect of Errors in the Memory\n\nThe errors in the memory will degrade the performance of MoT. **As shown in Table 1**, MoT (with gold) is the MoT with memory where incorrect memory is filtered out by the gold labels. MoT without gold labels significantly underperforms MoT (with gold), and this shows that **filtering out incorrect memory helps MoT conduct better reasoning**. To further demonstrate this, we conducted the following **supplementary experiments** across multiple tasks, demonstrating a direct correlation between incorrect memory and decreased accuracy.\n\n**Supplementary Experiments**\n\nWe conduct a fine-grained experiment to analyze the effect of incorrect memory to the performance on AQuA, ANLI-A3 and ComV over three task families. Specifically, we divide the test examples into multiple groups according to the accuracy of their recalled memory (the answer accuracy of recalled thought), and separately calculate the accuracy on these divided examples. The following is the result.\n\nIn the following results, we keep the total number of memory thoughts as in the paper, 4.\n|       | The number of correct memory |  4   |  3   |  2   |  1   |\n|:-----:|:---------------------------:|:----:|:----:|:----:|:----:|\n| AQuA  |          Accuracy           | **57.8** | 54.2 | 51.0 | ///  |\n| ANLI-A3|         Accuracy           | **58.0** | 54.2 | 50.6 | 50.0 |\n| ComV  |          Accuracy           | **96.6** | 93.8 | ///  | ///  |\n\n\nNote: Accuracy calculations for groups with insufficient samples, such as (2,1 in ComV), are omitted.\n\nFrom the results above, we see that **with more incorrect memory, it is slightly easier for the LLM to make mistakes**, which directly shows the effectiveness of memory filtering. We will add these experiments and conduct further case studies of incorrect memory in the next version.\n\n\n**R2**: The **Effect of each Hyper-parameter**, e.g.,  the Decoding path, Pre-defined Threshold $\\tau$, the Number of Memory Volumes and the Number of Routes for Memory Reasoning\n\nWe agree with you that it is important to analyze the effect of hyper-parameters, and **we have conducted analyses of these hyper-parameters in the submitted version** as follows:\n\n**Decoding Path**: We analyze the performance of MoT under different decoding paths in Section 4.3 \"Multiple-Decoding Performance\" (Line 469)\n\n**Pre-defined Threshold $\\tau$**: We compare the performance of MoT under different filtering thresholds in section 4.3 \"The Effect of Filtering\" (Line 508)\n\n**The Number of Memory Volumes**: We analyze the performance of MoT when the memory size is limited in section 4.3 \"Limited Memory-Size Performance\" (Line 525)\n\n**The Number of Routes for Memory Reasoning**: We analyze the effect of recalled memory quantity in Appendix D  (Line 1229) and we will move it to the main body of the paper in the next version.\n\nWe will further conduct hyper-parameter analyses in the next version.\n\n\n**R3**: The Effect of Capacity of the Memory Block and the Way of Sending a Block of Storage\n\nAs mentioned in R2, we have conducted the analysis of the capacity of the memory block (entire memory capacity: section 4.3 \"Limited Memory-Size Performance\" (Line 525), recalled memory capacity: Appendix D  (Line 1229)). The results show that **MoT consistently outperforms baselines over the different recalled memory capacities**. And MoT with larger recalled memory capacity can get better performance.\n\nAs for the way of sending memory, we considered different memory orders. We found that **different orders of recalled memory in the prompt lead to similar results**, as in the previous work [1]. We will add this result and conduct further analyses about the memory module in the next version.\n\nIf we misunderstand the meaning of your mentioned \"the effect of capacity of the memory block and the way of sending a block of storage\", please tell us.\n\n**R4**: The Utilization Rate of the Recalled Memory, Visualization.\n\nSince we conduct experiments on the OpenAi API (ChatGPT, Text-Davinci-002,003), we cannot get access to the model's internal representation e.g., attention score, and it is hard for us to visualize the memory visualization. We will **add the experiments on LLaMA models, and further analyze and visualize the effect of memory** in the next version. We will conduct case studies to directly illustrate the effect of memory in the next version, too.\n\n**Q1**: The Calculation of Answer-Entropy and Whether It Consistently Works across Different Types of Tasks and Datasets?\n\nWe introduced the definition and calculation of answer-entropy in **Section 3.1.2 and Equations (3), (4), (5)**. Specifically, we count the frequency of each answer in the multiple-decoding paths as their probability, and use the probabilities of these answers to calculate the entropy of the answer's distribution. The larger the entropy is, the more uncertain the LLM is about its answer.\n\nIn section 4.3 \"The Effect of Filtering\" (Line 508), Figure 5, we show **MoT without filtering significantly underperforms MoT with filtering**, across four task families. Specifically, the performance of MoT without filtering degrades to the Few-Shot CoT. These directly show the effect of filtering, which depends on the proposed answer-entropy.\n\n**Q2**: \"\"In the current experimental setup, the paper mentions that LLM cannot access the Internet and control external tools. How does this constraint affect the effectiveness of the MoT framework?\"\"\n\nThis constraint forces the MoT to depend on its own ability to self-improve, and our experiments show that it can self-improve without external resources. Accessing external environments like the internet or tools can help the LLM self-think better, thereby generating better memory to help itself think and improve its various abilities. Since the small-scale tool-using is relatively safer than accessing the internet for the AI, we will add the experiments where the LLM can use tools to self-improve in the next version.\nMeanwhile, ensuring the security of AI systems is important, and we regard making LLM self-improve both effectively and safely in an open environment as important future work.\n\n**Q3**: Experiments on Other CoT Methods and How to Ensure the Fairness and Reliability of These Comparative Experiments\n\nWe analyzed the performance of MoT on other two CoT methods (Zero-shot CoT and Plan-and-Solve) in **section 4.3 \"Transferability across Different COT Methods\" (Line 540)**. These experiments demonstrate that **MoT is compatible with various CoT methods and can consistently make LLM self-improve using various CoT methods**, which shows the generality and extensibility of MoT. We will add the experiments on more CoT methods in the next version.\n\nHow to conduct fair and reliable comparative experiments?\n\nWe think the most important thing is to control variables. Specifically, when comparing MoT and baselines, we should ensure the other factor is the same, e.g., evaluation metric, the inference LLM, the CoT method, the test dataset and so on. Since our proposed MoT can be compatible with various CoT methods, when applying one CoT method on MoT, we should incorporate the used CoT method into baselines as shown in Table 4 (Line 553). \n\nWe list the **implementations (including hyper-parameters) and used prompting templates in Appendix A and E**. The dataset details are in Appendix B. As mentioned in the submitted paper, we will **release the code and the generated CoT reasoning paths to facilitate future research**. We hope that MoT can inspire researchers of the important design choices about making the LLM self-improve with memory mechanisms and pave the way for further improvements.\n\n**Q4**: How to Mitigate the Problems that Arise from Potentially Biased Generated Content?\n\nWe can add an extra post-processing procedure to filter out those biased generated memory. For example, we can make the LLM itself **a bias classifier by prompting, and recognize and filter out those biased generated content**, or we can use the LLM itself to **refine those generated memory and debias them**. [2]\n\n**Q5**: Generalization on Out-of-Domain Dataset\n\nFor this setting, we test MoT on ANLI-A3 (which is the most difficult level in ANLI datasets) but using the memory from ANLI-A1 (which is the easiest level in ANLI datasets) and the following is the result:\n\nANLI-A3\n|           Method            | Accuracy |\n|:---------------------------:|:--------:|\n|       Zero-Shot-CoT         |   48.5   |\n|       Few-Shot-CoT          |   52.3   |\n| MoT with ANLI-A3 memory     |   55.2   |\n| MoT with ANLI-A1 memory     |   54.4   |\n\nWe see that MoT still outperforms baselines when it uses the out-of-domain memory, and this shows that the **MoT can leverage the memory from easy reasoning scenarios to enhance its reasoning ability for difficult scenarios**. This result shows its potential utility in challenging real-world scenarios. We will further add the OOD generation experiments in the next version.\n\n**Typos and Grammars**: The typo on line 540 and the tense in the conclusion\n\nThanks for pointing out these issues. We will fix the typo and use the past tense to express the conclusion in the next version.\n\nThank you again for your valuable review. If we have misunderstood any aspect of your review, or if you have additional questions, please let us know.\n\n\n\n\n[1] Unified Demonstration Retriever for In-Context Learning\n\n[2] Self-Refine: Iterative Refinement with Self-Feedback", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Innovativeness: The MoT framework proposed in this paper is innovative in solving the problem of LLMs improvement, and realizes self-improvement without external resources by utilizing self-thinking and memory.\n\n2. Experimental results support: The author has proved through experiments that the MoT framework has a significant improvement effect on ChatGPT, and can improve the abilities of arithmetic reasoning, common sense reasoning, factual reasoning and natural language reasoning.\n\n3. Scalability: The framework can achieve consistent improvement on different CoT methods and LLMs, and has certain versatility and scalability.", "reasons_to_reject": "1. The proposed memory module is relatively simple to implement. It is not yet known whether the rate of errors in the memory will affect the final reasoning result. This paper does not conduct further analysis about this section.\n\n2. In the implementation process of the article, there are many hyperparameters that need to be adjusted (as mentioned in paper, such as the decoding path, pre-defied threshod \\tau, the number of memory volumes and the number of routes for memory reasoning). The paper has not conducted further ablation experiments, and it is not clear how each parameter affects the reasoning.\n\n3. There are many similar ways to use the retrieval function to improve generation, the essence of which lies in the \"quality of prompt information\". When the article considers the memory module, the performance of the module, including the capacity of the memory block and the way of sending a block of storage (natural language, key Values are equal). Does it affect the final reasoning result?\n\n4. The efficiency of performing more operations in the retrieved memory, or what the utilization rate of the memory is? It is a wise choice to use the visual reasoning path to improve the interpretability of the model reasoning.\n\nI will be glad to revise my rating scores when I see the authors try to take my advice.", "questions_to_authors": "As the simply implementation for this memory of thought method:\n\nI have the following questions:\n\nQ1: It is mentioned in the paper that answer entropy is used to filter uncertain thinking, so how to define and calculate answer entropy? Does this approach work consistently across different types of tasks and datasets?\n\nQ2: In the current experimental setup, the paper mentions that LLM cannot access the Internet and control external tools. How does this constraint affect the effectiveness of the MoT framework? Are there plans to conduct experiments in more open settings to further validate the performance of the framework?\n\nQ3: Can we expect more comparative experiments on MoT frameworks and other CoT methods in future research? How to ensure the fairness and reliability of these comparative experiments?\n\nQ4: For the bias problem of large language models, how to ensure that no bias is introduced in the thinking and memory process generated by LLM? Has the generated thinking content been analyzed for bias? How to address and mitigate the problems that arise from potentially biased generated content?\n\nQ5: For unknowable and invisible out-of-domain data sets, is there any experiment to illustrate the generalization of the proposed method? That would be a nice move.", "paper_topic_and_main_contributions": "This paper proposed a framework called Memory Of Thought (MoT), which uses self-thinking and memory candidates to help large language models (LLMs) improve their performance. The framework consists of two stages to improve the performance of LLMs in various reasoning tasks by means of pre-thinking and external memory. The main contribution is to propose a method based on self-thinking and memory, which can significantly improve the ability of ChatGPT in multiple reasoning tasks.\n\nThe MoT framework proposed in this paper is innovative and practical for solving the improvement problem of LLMs. By means of self-reflection and memory, LLMs can be self-improved without external resources. Experiments demonstrate that MoT can effectively improve the performance of LLMs in multiple key tasks, and individual components play a key role in achieving these improvements. These results have important reference value for the research community in further improving and optimizing LLMs.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None. Related work and its connections have been discussed in this paper"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The article is well organized, but there are still some minor typographical errors that need to be corrected, such as:\n\n1. Line 540 across Please change to uppercase Acrosss\n\n2. Please use the past tense to express the conclusion"}}]

Author rebuttal:

Meta-review: improvement of large language models (LLMs) without fine-tuning or annotated data. The framework uses a two-stage process: first, it generates and stores high-confidence examples from an unlabeled dataset as external memory; second, it recalls and uses the most relevant memory examples to assist the LLM in answering questions. The paper shows that MoT can significantly improve ChatGPT’s performance on various reasoning tasks, such as arithmetic, commonsense, factual, and natural language inference. The paper also demonstrates the importance of each component of MoT and its generalizability to different LLMs and methods.
Decision: Accept-Main