ExpNote: Black-box Large Language Models are better Task Solvers with Experience Notebook
Authors: Wangtao Sun
Xuanqing Yu
Shizhu He
Jun Zhao
Kang Liu
Venue: EMNLP 2023 Findings (2023)
Keywords: large language model
self-reflection
in-context learning
TL;DR: We propose ExpNote, an automated framework to help LLMs better adapt to unfamiliar tasks through reflecting and noting experiences from training data and retrieving them from external memory during testing

Abstract: Black-box Large Language Models (LLMs) have shown great power in solving various tasks and are considered general problem solvers. However, LLMs still fail in many specific tasks although understand the task instruction. In this paper, we focus on the problem of boosting the ability of black-box LLMs to solve downstream tasks. We propose ExpNote, an automated framework to help LLMs better adapt to unfamiliar tasks through reflecting and noting experiences from training data and retrieving them from external memory during testing. We evaluate ExpNote on multiple tasks and the experimental results demonstrate that the proposed method significantly improves the performance of black-box LLMs. The data and code are available at https://github.com/forangel2014/ExpNote.

ExpNote: Black-box Large Language Models are better Task Solvers with Experience Notebook
Wangtao , Xuanqing Yu?.3, Shizhu Hel Jun Zhao Kang Liu
The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China 2 School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China CAS Engineering Laboratory for Intelligent Industrial Vision, Institute of Automation, Chinese Academy of Sciences, Beijing, China A SShanghai Artificial Intelligence Laboratory (sunwangtao2221, iwxuanqing2221@iia. ac cn (shizhu.he, he, jzhao, kliu]@nlpr. www.riaaacc.cmmmmm ac cn
Abstract
[Chris] his son (Jac)) bar wailing for their drinks. (Maria) showed ?
up and aut with her husband [Chris) Who is Jack to Marii?
Black-box Large Language Models (LLMs) have shown great power in solving various tasks and are considered general problem solvers. However, LLMs still fail in many specific tasks although understand the task in- struction. In this paper, we focus on the prob- lem of boosting the ability of blx LLMs to solve downstream tasks. We propose Exp- Note, an automated framework help LLMs better adapt to unfamiliar tasks through re- flecting and noting experiences from training data and retrieving them from external mem- ory during testing. We evaluate ExpNote on multiple tasks and experimental results demonstrate that the proposed method signifi- cantly improves the performance of black-box LLMs. The data and code are available at wttp://iiiiiocoooooooooongl/000 http://rrrnnee/////////oooooo
Chris's Chris Mara's hustand, so lack Mariats
X
Answer with experience (ExpNott)
Training
experiences:
[Liea] her [Usaa's
today
husband [Dwight]
husband C, then B is son
the store, Whol Mikia Dikgh??
Mika Dwights stopson X
This aeams helpfut:
THINK
correct
Chrik's son, Chris
reflect
Marar's husbond Aczznding to the first experience, Jack
In this case. Mike Lisals son and
husband. Tharaforo,
Maria's
Dwightss son.
back -epeofiic
NOTElson, husband
Memory
Figure I: An illustration of how ExpNote assists LLM in enhancing the effectiveness of task-solving, ExpNote can automatically generalize relevant experiences from other samples and apply them to specific tasks.
1 Introduction
To empower black-box LLMs on specific tasks, several works (Madaan et al., 2022: Dalvi et al., 2022) have focused on equipping the LLMs with external dynamic memory to store useful task- specific knowledge and facts. However, these task- specific knowledge and facts in the memory usually come from expert annotation or human feedback, which is very costly to obtain. On the other hand, several researchers (Shinn et al., 2023; Madaan et al., 2023; Akyürek et al., 2023) try to exploit the reflection ability of LLMs to automatically gener- ate such knowledge for specific tasks. However, most reflection power the LLMs in the same case, without the ability to generalize to other instances.
Large Language Models (LLMs) have demon- strated astonishing capabilities natural language understanding and generation (Wei et al., 2022; Huang et al., 2022; Sun et al., 2022; Bang et al., 2023). However, due to the limited parameters and context processing length, LLMs are not able to master all task-specific knowledge in real-workl applications. As a result, LLMs may perform mediocre on some specific tasks, such as inductive reasoning (Bang et al., 2023) and entity recognition (Chen et al., 2023).
Therefore, how to make LLMs adapt to the downstream tasks has tracked more and more atten- tion. Recent techniques such prefix-tuning (Li and Liang, 2021), P-tuning (Liu et al., 2021) and LoRA (Hu et al., 2021) proposed low-cost solutions for fine-tuning L.L.s. However, these methods are not capable of black-box powerful LLMs, such as ChatGPT and GPT4 (OpenAI, 2023).
methods are only able to em-
Thus, this paper proposes framework Exp- Note (Experience Notebook), to empower black- box LLMs on downstream tasks by learning and using task-specific experience automatically. We

---

equip the LLMs with dynamic memory and de- sign several commands help LLMs interact with it. In specific, in the training stage, ExpNote guides LLMs to generate task-specific experiences and store them in an external memory. In the testing stage, ExpNote uses retriever to retrieve relevant experiences from the memory, the learned experi- ences will help the LLMs to solve the cases which they failed to answer directly (Figure 1).
the performance of the LLMs in a series of tasks, ReAct (Yao al., 2022) proposed that by using a Wikipedia search API and generating trajectories similar to human thinking, the LLMs can utilize external knowledge during reasoning and provide interpretable reasoning paths. HuggingGPT (Shen et al., 2023) proposes to solve any Al task by using the models on the huggingface its toolkit.
2.2 Language Model with Dynamic Memory
We evaluate ExpNote on multiple tasks. The results show that ExpNote can empower the LLMs effectively, and significantly outperform other prompting methods like ordinary in-context learn- ing (CoT, Wei et al. 2022), memory-based method (TeachMe, Dalvi et al. 2022), and case-by-case reflection- method (Reflexion, Shinn et al. 2023).
Some existing works have noticed the need equip LLMs with dynamic memory. MemPrompt (Madaan et al., 2022) retrieves the stored user feed- back the intention for similar questions to en- hance the current prompt for the LLMs. TeachMe (Dalvi et al., 2022) allows LLMs to store the miss- ing and wrong facts during the QA task with the correction of user feedback. These methods cre- ated new paradigm boost the ability of LLMs in general way. However, they rely heavily on hu- man feedback or annotated facts. GEMEMBERER (Zhang et al., 2023) proposed to consider LLMs as a semi remi-parametric RL agent. trains LLMs to take the next action based on the retrieved (observation, action, Q-value) tuple.
Moreover, we empirically compared different types of experiences to examine their effectiveness in helping LLMs adapt to unfamiliar tasks. Specif- ically, we compared learned task-specific experi- ences with original cases and experiences learned from positive cases (succeeded) and negative cases (failed). We find that prompting with experiences is more helpful than original cases for LLMs to generalize new cases, and experiences from both positive and negative cases are beneficial.
2.3 Language Model Reflection
The major contributions of this paper are two- fold:
Recently, some works have been proposed to cor- rect the mistakes of LLMs in conducting specific tasks by using their capacity of self-reflection. Re- flexion (Shinn et al., 2023) focused on sequen- tial decision-making tasks. heuristic function is adopted to judge whether the trial is successful or not. And LLMs will reflect those trials that are thought to have failed. The reflective information will be used to support the LLMs in improving their own decision makinn process in the next trial. Self- refine (Madaan et al., 2023) proposed a method to iteratively improve the output of large model through its own feedback, achieving improvements in multiple generation tasks. However, these reflee- tion methods are limited to certain cases, without being abstract and able to generalize to other data points.
We propose framework ExpNote to empower the LLMs in various tasks through interacting with dynamic memory. ExpNote conducts fully automated reflection, noting, and retrieval, with- out the need of any annotated knowledge and facts or any human feedback.
We investigate different types of experiences and show the learned task-specific experiences help LLMs to better generalize than the original cases in the task, and experiences from both positive and negative cases are beneficial.
2 Related Work
2.1 Language Model Taking Action
In order to address the lack of knowledge, reason- ing, and specific abilities in large models, many efforts have focused utilizing external knowl- edge sources and tools to assist large models in completing tasks. Toolformer (Schick et al., 2023) proposed fine-tuning on APl-augmented datasets to enable the LLMs to master the ability to use external tools with API calls, thereby improving
2.4 Language Model Thought Chain
Furthermore, there have been some efforts to im- prove the reasoning performance of LLMs by en- hancing their thought chains in specific tasks. For example, DIVERSE (Li et al., 2023) proposed a method that generates multiple different reasoning paths and uses a validator for weighted voting to

---

Exphote Prompting (Taining)
Expliste Prompting (Testing)
several Eectarasd laarnnn Sceed
Parate
& Secveral inscettal Bearnin cannt
Plest
Amsovering
Esperience
Quastion
shop school (1 Leafs huadend [Dwight)
(Land] adoas dheaghte 144 chile) to the eare lodky
REGALL
(Rax] and (Paes] the happy couple, jusd morridd last weor Who is Lois to Mary?
(Roy] took No couyrter nae out for dinner
the phont at store Who Vicncle Owight?
BAl dnsughtse it, aed monteed to
Michcle Les's toughte.
deugffar
Relevant Experience
Iwn is also the deughtee
THINK
Diiihht hustard.
? t manted to and they hove
Mistada Owigltls daughtar.
ANSWER
coughte Cotter C de doughts.
Reffection
Les la Roy's slaughtar fly juai pat muriied Marr According to the thrt expertence, Lold is also Mar''s doughter
Memory
THINK
Michle doughter Deight rusband. Dwights doughtor.
THINK
task-spocific seper lenccs
Raplace uaw with Midadaa with and Diight CI
Mary's daughter.
THINK2
The abuaadd rate in case a Bui.
I'At Ensrgtte B. and marmed Ben abo suighereate
NOTE [daughter. married]
ANSWER
The framework of ExpNote. This framework shows how LLMs use ExpNote to solve specific tasks, including the training (left) and testing (right) stages.
Figure 2:
filter out incorrect answers. However, this method demands manual construction of reasoning paths for each training question-answer pair and exten- sive human evaluation, restricting its use on large datasets.
After the answer obtained, the ExpNote will produce feedback F(j. y) to LLM depending on whether ý = y. Note that this feedback F v) only includes a simple prompt containing the ground-truth of the current question, without any additional knowledge like TeachMe (Dalvi et al., 2022). Then LLM is supposed reflect and store the learned experience e into the memory.
Drozdov (Drozdov et al., 2022) and others intro- duced technique that decomposes complex ques- tions into sub-questions and provides answers to these sub-questions, serving as hints to assist the model in reaching the final answer. Faithful CoT the other hand, prompts
e ~~PLL PPLLas(-|PProiiii x, F(J. u))
(2)
(Lyu et al., 2023). language model to translate complex queries into reasoning chain that includes question decomposi- tion and corresponding symbolic language solving, thus enhancing interpretability.
Where a key-value pair of learned tpec-fi experience, e.g. key = daughter, married and value = if has a daughter B, and A is married to C, then B is also the daughter of C' (Figure 2). This process is achieved by taking extra actions to interact with the memory, which we will describe in Sec 3.2.
These approaches offer intriguing ideas for im- proving the reasoning performance of LLMs but still face challenges related the need for substan- tial high-quality annotations, difficulties in reusing experiences, and sample generalization.
In the testing stage, ExpNote will use the testing instance as the search query retrieve & experi- ences from the dynamic memory. The retrieved experiences will be added to the prompts for the LLM. Then LLM will decide whether to refer to these experiences and finally output the answer.
3 ExpNote
3.1 The Framework
{ei Retrieve(z, k) ~ PLLM(- x, (er)
As shown in Figure 2, all the tasks are formalized the tuple (2, where is the input question and y is the desired answer. For each task, we write prompts Ptrain and Ptost that encourage the LLM to use ExpNote following the illustrations. In the training stage, LLM is first ordered to infer the answer like ordinary CoT reasoning.
(3)
The full examples of ExpNote on different tasks are shown in Appendix D.
3.2 Interaction Commands
ExpNote designs several commands for L.MMs interact with the memory, including summarizing and applying the experiences (THINK), storing the
9 ~ PI.J.M P-rain x)
(1)

---

experiences in the memory (NOTE), and retriev- ing relevant experiences for the testing instances (RECALL). They are described in detail as fol- lows:
methods
CLUTRR
METS
EMOJI
LETS
zero-shot
36
23
34
35
few-shot
44
61
CoT
40
54
54
60
THrgk Inspired by ReAct (Yao et al., 2022), both training and testing stages, we en-
TeachMe
31
51
56
56
Reflexion
54
62
able LLM to use command THINK to organize its current thoughts and make the next decision.
ExpNote
61
66
74
89
NOTE[arg!); arg2. In the training stage, prompt the LLMs to use this command NOTE after answering each question. The command NOTE will store the experience arg2 as a value in the memory with its key argl.
Table I: Accuracy of ExpNote and baselines datasets.
variants
CLUTRR
METS
EMOJI
LETS
disabled
35 (0)
49 (0)
57 (0)
50 (0)
RECALL(arg]. In the testing stage, this com- mand is automatically executed by ExpNote at the beginning of each question recall relevant experiences. ExpNote will use a retriever to re- trieve up to & relevant experiences using the ques- tion arg as the search query. The content of these experiences will then be added to the prompt.
case
49 (128)
59 (279)
58 (20)
51 (87)
positive
51 (73)
56 (166)
66 (14)
64 (41)
negative
55 (55)
52 (113)
60 (6)
71 (46)
ExpNote 61 (128)
66 (279)
74 (20)
89 (87)
Table 2: Accuracy of ExpNote and its variants on 4 datasets. The numbers in the small bracket are numbers of experiences stored in the memory.
4 Experiments
this section, we want to answer the following
CoT (Wei et al., 2022): Several cases of solving the task using Chain-of-Thought are shown to the LLM.
research questions:
RQL. Is ExpNote able to help LLMs to adapt to new tasks effectively?
TeachMe (Dalvi et al., 2022): As the core facts human feedback of these specific tasks are hard to obtain, we adopt a commonsense knowledge base, Conceptnet (Speer et al., 2017), to serve as the memory for TeachMe.
RQ2. Which kinds of experiences help LLMs
solve tasks better?
4.1
Datasets
To show the effectiveness of ExpNote in han- dling various tasks, we select multiple datasets from different tasks for empirical study, including CLUTRR (inductive reasoning, Sinha et al. 2019), METS-CoV (medical entity recognition, Zhou et al. 2022), EMOJI (text-to-emoji prediction, Felbo et al. 2017). Besides, we propose dataset LETS (letter splicing) to evaluate the symbolic reasoning capa- bility of LLM enhanced with ExpNote. The detail of LETS can be found in Appendix A. For each task, we tested ExpNote and other methods on 100 cases. The other setups the experiments can be found in Appendix B.
Reflexion (Shinn al., 2023): As the heuristic function of repetitive action detection described Reflexion is not working for these tasks. Thus we allow Reflexion to do a little cheating: it is al- lowed to try again after every failed trial without being informed of the ground- truths. This setting is equivalent to obtaining golden function that accurately determines the success/failure of each trial with 100% success rate.
To answer the RQ2, have also implemented several variants of ExpNote:
disabled: This variant adopts the reasoning form of ExpNote while disabling its retrieval function.
4.2 Baselines
case: This variant will recall the original ques- tions and answers of the noted cases instead of the learned experiences.
To answer the RQL, apart from basic zero-shot and few-shot settings, we select the ordinary in-context learning method (CoT, Wei et al. 2022), memory- based method (TeachMe, Dalvi et al. 2022) and case-by-case reflection-basee method (Reflexion, Shinn et al. 2023) for comparison.
positive / negative: This variant only retains the experiences that learned from the training sample which LLMs answered correctly incorrectly.

---

CLUTER
METS
Training
60
Cul
Reflesinn ExpNote
FoF
Accumo
EVOR
Two
LETS
TioT
50
100
125
150
175
200
Training samples
Figure 4: Improvement analysis of ExpNote on 4 datasets.
Figure 3: The training curve in CLUTRR dataset.
4.3 Results
4.4 Improvement Analysis
As shown in Table 1, the full ExpNote method achieved the best performance on all datasets, 20.5% higher on average than the CoT method. TeachMe failed to outperform few-shot, as task- specific knowledge hard to obtain without human feedback. Compared with Reflexion, note that even if we make Reflexion cheat to identify the failed trial with 100% success rate, it still falls behind ExpNote.
We also analyze how many cases corrected by introducing experiences in each dataset. As shown in Figure 4, we plot the distribution of cases in 4 conditions:
=> F: case is originally answered incorrectly disabled and also answered incorrectly with ExpNote.
F => Т: case is originally answered incorrectly in disabled but answered correctly with ExpNote.
Compared with other variants of ExpNote, dis- abled retrieves no experience in the testing stage, thus degrading the performance of CoT (even worse) as expected. We also discovered that case performs worse than full ExpNote although retriev- ing exactly the same cases for all of 4 tasks. We then conclude that abstract knowledge rules are more capable of helping LLMs to generalize testing cases. Moreover, positive and negative both fall behind the full ExpNote while still outper- forming baselines. We made an efficiency analysis in Appendix and the results show that experi- ences from both positive and negative cases are more efficient than the other respectively on two datasets. These results indicated that experiences learned from both positive cases and negative cases are useful for LLM generalize test sets.
T T: a case is originally answered correctly disabled and also answered correctly with Exp- Note.
T F; case originally answered correctly in disabled but answered incorrectly with ExpNote.
In Figure 4, we demonstrate that ExpNote helps LLMs correct a certain amount of emoos (the green part) at the cost of producing a few new errors (the red part) in all 4 datasets. And we can observe around 50% incorrect answers in disabled (gray + green) are corrected (green) with ExpNote.
Conclusion
In this paper, we propose ExpNote, an automated framework to help black-box L.LMs adapt to spe- cific downstream tasks by interacting with dynamic memory. We carried out experiments on multiple datasets from different tasks and showed that Ex- pNote effectively improve the performance of LLMs better than other prompting methods. We also found that the learned task-specific experi- ences help LLMss better generalize than the orig- inal cases in the task, and experiences learned from both positive cases and negative cases are valuable.
We also observed the performance changes of the model with the number of training samples. As shown in Figure 3, in CLUTRR, ExpNote starts from training with 0 samples (cquivalent to dis- abled) and ends with training with 200 samples. The performance of ExpNote on the testing set continually grows with the number of training sam- ples, showing that ExpNote continually learns new knowledge during the training stage.

---

Limitations
Bjarke Felbo, Alan Mislove, Anders Søgaard. Iyad Rah- wan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representa- tions for detecting sentiment, emotion and sareasm. In Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP).
Although ExpNote able to empower the LLMs in various tasks, it may be less effective on these case-by-case tasks, like summarizing or creative writing. In these tasks, the cases share little com- mon knowledge or rules, which makes it hard for ExpNote to help LLMs generalize.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models. arXiv preprint Sidi:2100.096633
Ethics Statement
This paper proposes a method for augmenting black-box LLMs. All experiments are conducted on publicly available datasets. Thus there is no data privacy concern. Meanwhile, this paper does not involve human annotations, and there are no related ethical concerns.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language madels zero-shot planners: Extracting actionable knowledge for em- bodied agents. In International Conference on Ma- chine Learning, pages 91 18-9147. PMLR.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00/
Acknowledgements
This work was supported by the National Key R&D Program of China (2022Z01100033) and the National Natural Science Foundation of China IN0.2576270N0.618310020 was supported by the Strategic Priority Re- search Program of Chinese Academy of Sciences (N..XDA27020100) Youth Innovation Promotion Association CAS and OPPO Research Fund.
Yifei Li, Zeqi Lin, Shizhuo Zhang. Qiang Fu, Bei Chen, Loan-Gang Lou, and Weizhu Chen. 2023. Making language models better reasoners with step-aware verifier. Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5315-5333.
This work
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding. Yujie Qian, Zhilin Yang, and Jie Tang. 2021. Gpt understands, too, arXiv preprint arXiv: 10385.
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of- thought reasoning.
References
Afra Feyza Akyürek, Ekin Akyürek, Aman Madaan, Ashwin Kalyan, Peter Clark, Derry Wijaya, and Niket Tandon. 2023. RI4f: Generating natural language feedback with reinforcement learning for repairing model outputs. arXiv preprint arXiv:2305.0888 08844.
Aman Madaan, Niket Tandon, Peter Clark, and Yim- ing Yang. 2022. Memory-assisted prompt editing to improve gpt-3 after deployment. arXiv preprint arXiv:2201.0600
Yejin Bang, Samuel Cahyawijaya, Naycon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, al. 2023. multi- task, multilingual, multimodal evaluation of chatgpt on reasoning, ballucination, and interactivity. arXiv prepriat urXiv:2302.04022 04023..3......
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan. Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. et al. 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.
OpenAL 2023. Gpt-4 technical report.
Qingyu Chen, Jingcheng Du Yan Hu, Vipina Kuttichi Keloth, Xueqing Peng, Kalpana Raja, Rui Zhang, Zhiyong Lu, and Hua Xu. 2023. Large language models biomedical natural language processing: benchmarks, baselines, and recommendations, arXiv preprint 2305 16326.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04770
Bhavana Dalvi, Oyvind Tafjord, and Peter Clark. 2022. Towards teachable reasoning systems. arXiv preprint arXiv:2204.11 130744
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yucting Zhuang. 2023. Hugging- gpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303. 230...500...
Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. 2022. Compositional semantic parsing with large language models. arXiv preprint arXiv:220. 2500003...
Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023. Reflexion: autonomous agent with dy- namic memory and self-refiection. arXiv preprint NKiv:2303.11366

---

Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L Hamilton. 2019. Clutrr: A diagnostic benchmark for inductive reasoning from text. arXiv preprint (9Xxiv9908.000 1908.06177.
We randomly select 100 words with lengths of 4-10 the vocabulary. To generate the training and testing set, for each instance, we randomly picked 3 different words from the vocabulary and randomly selected their indexes.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.3: An open multilingual graph of gen- eral knowledge. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 31.
B Setup
For the LLM, we use ChatGPT (gpt-3.5-turbo) via Openai API calls.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao,
Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonoo
For cach task, due to the size limitations of the datasets themselves, we test all methods on 100 testing cases. In fact, a large amount of related work is also tested using samples of similar mag- nitude, such as TeachMe (OBQA, 500, Dalvi et al. 2022), ReAct (ALFWorld, 134; WebShop, 500, Yao al. 2022), Reflexion (consistent with ReAct, Shinn et al. 2023). Considering Expnote will in- teract with the environment multiple turns for a single case, the actual number of generations for LLMs can be 4 to 5 times higher. And We adopt a minimal training set with it size 2:1 to the testing set (and 1:1 in EMOJI and LETS datasets).
2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.0661..
Tian-Xiang
Tian- Xiang Sun, Xiang- Yang Liu, Xi-Peng Qiu, and Xuan- Huang. 2022. Paradigm shift in natural language processing. Machine Intelligence Research,
19(3):69--188
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quee Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.1 1903.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Tzhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting language models. arXiv preprint arXiv:22/0.03622
For all ExpNote variants, write 2-3 ExpNote usage cases for the LLM as few-shot prompting: choose 4 for training (the LLM is able take 4 extra actions to THINK and NOTE after obtaining the ground-truth of each case), and n = 0 for testing (the LLM is not able to access to the ground-truth).
Danyang Zhang, Lu Chen, Situo Zhang, Hongshen Xu, Zihan Zhao, and Kai Yu. 2023. Large lan- guage model is semi- parametria reinforcement learn- ing agent. arXiv preprint arXiv:2306.0799 07999..
For the retriever, we implemented a word-based retriever to retrieve experience by matching words in the query and the key of experience, and it trieves up to = experiences for each case in the testing stage. When ExpNote fails retrieve relevant experience, a failure prompt "No relevant experience" will be returned to the LLM.
Peilin Zhou, Zeqiang Wang, Dading Chong, Zhijiang Guo, Yining Hua, Zichang Su, Zhiyang Teng. Jiageng Wu, Yang. 2022. METS-coV: dataset of medical entity and targeted sentiment on COVID-19 related tweets. In Thirty-sixth Conference Neural Information Processing Systems Datusets and Bench- marks Track.
A LETS Dataset
C Effciency Analysis
We can define the efficiency of each type of experi- ence as
As existing symbolic reasoning datasets, such as word sorting in BIG-bench (Srivastava et al., 2022). are designed to test the zero-shot reasoning abil- ity of LLMS and always lack a training set, we therefore propose the LETS, a similar symbolic reasoning dataset while enabling LLMs to learn and generalize.
eff( eff(type)= Perf(type) Perf(disabled))
(4)
Cnt(type)
where type refers to positive or negative, Perf() represents the corresponding variant's performance in Table 2, and Cnt(type) represents the number of experiences of that type. Then we can calculate the efficiency of positive and negative experiences based on the data in Table 2.
LETS require the language model to splice the letters given index of several words together. For example, given the query Splice the 5th letter of "sleep", the 2nd letter of "official", and the 5th letter of "neglect" together, the model is supposed to output pfe as the answer.
As shown in Table 3, experiences from both pos- itive and negative cases are more efficient than the

---

type
CLUTRR
METS
EMOJI
LETS
positive 0.219
0.042
0.643
0,341
negative 0.364
0.026
0.500
0.456
Table 3: Efficiency of Positive and Negative Experi- ences on 4 datasets.
other two datasets. These results indicated that experiences learned from both positive cases and negative cases are useful for LLM generalize to sets.
Full Example

---

Example Trajectory Training and Testing Cases the CLUTRR Dataset
Tesining Slage
Testing Stage
59 SYST:M: You sov asmas evolact and You andil la canplai sueciled Lakk by Liliina Gar SS1519 The risiig betweer you and st STBM will shown in the fome SASSISTANT and "SY EN:
51 STEM: os sne asadd deciaan and You aask camplas in suvcllaed taik salling to tn 51 STEH. The Giikgg betwors you S1 STEM will be bestow in foom 5SETTAT and "S7 STMM
51 STEM: Yourta task anverr the pastor givce siboo swary.
5Y STMM Yeerber in anmuee quotirr yloer abort story
55 STET Bavy wod anuaai
55 STEM: Hasc awvera vauazale.
SYSTTM cwee
STEM boring moat with tibee Upcisoon. wathing with (Jecaoo)) Mile
51 STSTEM:
51 STEM: [Axo/'s
[Mikk] wast to get lox crea.. [MBu)] preaa Julter
ASSISTADIT: THINSRAKe Jullooth Jocluca tuabaal,
lonty [Jacison) Who Kigg jockotnn???
SYSTTM Ki mieovet reperiece
ASSISTANT: THINRR/see then adon patiaa adout ta quaiias med
Aee koey's
casser fxught necher (osinso,, jartnon
corrot asower crughter
STEM:
tayy.
ASSISTANT:: 1HIN a Ane lootarr Acp is fanall, Mike is Caclooo's faber, Doveglyy It Jiitioon's tihers deter sa kny is Jutison's sont
cocdlasion Any Auaglts Lucyl
Expioote Denonstrafooo
59 STMM: OK.
SYSTIM
ASSISTANT: apda Ase Juckaus vala B. and Lay wits destail rala
ANANTRAA jachoono.oom
1M caue Ef tethentt, in tadwrrd It then the ofngtomme C
SYSTEM:OK OK.
SYSTEM Yeeraawe corrd.
ASSISTANT: Noritttter hrobonct ITA hos fhtweill it the hnctaed of thes the
55 335TEM: 2
4aggkz C.
STSTEM: (Clast and lis sua (bckl cac a bur wattag their drtaks (Mori) sloved up zad toiins Whn Jacknn Mare?
59 19.5TMM 2
her
59 SYSTEM: [Chad wnd sas [lack] at lae ain for thair ciiaka (fuzaa) slownd up and
STE:: Eos scceaa postible
expatinne:
with nert hasteed (Chin) WWs in josk tn Chetr?
5Y SYSTRM less Ft bos wiite thes
ASSISTANT: THIN Chriss sce, Chrie is Martis bastord Jack ts Markhs traboaass son so
ASSISTANT: THINE This ppteiiiio owen helpid, bobudd ioterts [t]
Jack Masuu slapos))
51 STEE: OR.
51 SYSTEM:O0 OR.
THINK))skk
aq apateos.
ASSESTANT ANANNERR la Matie daswoc))
Jedi Maich
SYSTRM Veer sipooe, houeerr, commet inswerts
SYSTEM: OR.
ASS8STA VT: Tithacte
China the burtend Morte. The
ASSISTANT A A io Mortel snn.)
/oak Mortal
59 SYSTEM: Yoaramwe
SYSTEM:
THBN8BR jack vita Clacs with with C. The abstract
51 STEM: [Gaasa) irow aariaaaer (Jmr)] lnoettil (Arag) (ite) slowed
Aurlyy alan Who Bab Googe?
SYSTEM: Fon sovel poeriie iesernt exporenne
SYSTEM: daggaae
dua the bodker
(Deerge drughtnn kkyll to herptail lAngyts (Bet) sharty
sepetiene dwesh hight reovetin tiis qpotice and myy en repelt)
ASIISTNT:
dbx. Wio Goorgs?
ASTESTANT: THIN Geoaaex duuglter Bob to Auny's was. Bob Geczpo dharpher' wec. = Goozall gambos)
SYSTEM: OK.
ASSISTANT: THINRAA Orages dagglss Bob is Ang'ss soc Balk Googee skagless
NYISTRM IK..
ASSISTANT: A Gorges gransoo.)
51 STETTM:
59 STEM: Y Youramine: parad
ASSISTANT: THINB4I dus can. Gaoge his dagltts Ace, Aus las soe Bek The exedasio
ASSISTANT A AMAAETRT Georgee grentoee
in Daai Bos Lhe ganden el Deasa]
s% STEMM Yeer sonedc.
SYSTRM OK.
ASSISTANY VTT THENRR Grorge with n. Anyy with it, and fius with The stereed rale the ssse l: tfatf 3as Seiitee B. los sea then & the gradio af
SYSTEM OK.
ASSISTANT: NOTHGagaar ael: E'A las daghar 2, las c, Dex C Ea gradice o
(Toete gpectie:
Now Las camlise
goedentwre (Colneiin) Cchldii?
terhday
SCEM: /aass lesda: [Da]] guadaadass (Cukasik)) and latakky Dantidae
Who
diser Dee loai tow Who
ASSUSTANT THaNNC Data leshae Cabraille Dants spankcathee. Miclad Dacks
STEM: How on sevel prsstir reimnrt epeneace
Crani gradics Nicaed in Cadandas spanlons)
iimberrs cin gradhother then Clpradion
Interrective Trajectories
SYISTIO OE:
ASBISTANT THINCTTts experiece soces leppul shockt iff bo St 5CBM) OK.
ASSISTAN: ANdwER is Gabrkties g
ST-STTM Your snowerts carnet.
ASSISTANT: THIN a Vixars loclar Gaaada a Vinaad gaakaskas Acooting tu the acMerass, Due Gessd gradars)
ASSESTANT: THENCCH dá cas Dats beckee is Michel and Gabidle a Dark parden
Tanniir, Xisied Gassdaks goadess
SY STEM: OK
SV SYSTE OK.
ASSISTANT ANSWERR Donts Goeiiin graddnn)
ASSESTAANE THENK Tog sos the with 4. Mireel with it, and rabenis with c: The abstnd ndw
ST STEM: VUE aawer conect
IES Is if-tarf A'd britic it and C it' proteaaterr the is grnnsoe.)
SY-TTM Oi
ASSISTNN: NOTEEaoOkr, pradactarff ITA' bodker a and punductier data Bicss
SY STEM оK,
ASSEST ANT. KN1D
Figure 5: The example trajectories of ExpNote on the CLUTRR Dataset. The left part is a training case while the right part is corresponding testing case using that training experience. The blue parts are the ExpNote demonstrations (prompts Perein/(test). The yellow parts are the interactive trajectories between LLM and ExpNote. The sentence highlighted in green is the learned experience.

---

Example Trajectory Training and Testing Cases on the METS Dataset
Training Stagr
Testing Stage
asiiaar and cumplecs speciteed uci to diingg beovece you
somplets sparther nailing The dakng STEM
"ASSSSTANT:
STEX.
"St SCEM?
SYSTEM: tas ideatfy madd witly of typ vaccos, chouus, dogg, crasepaion the glvre tivt Robern fommr orierityy type
idaatty
dasiso,
orggnnn gives
tiiityy typ..
S1 STEN
SYSTEN: limnds avandly of vecuua das dras on orspatiii is lie act Brazn Us
ST STEC. Hes anr suapless SY EM
formek peetyy spet not tnchute "oxeir" in neaver
calisaaaity @ dirrtatton foone and toeobalalla wen dew solied aepocal warh Jod an
ExpNNote Demorstrttion
car the on viti postres crrdd Hhr har mmoupiont nut May So for to ioss neasnees the kazy Out with valuat cuui clange diac.
tistte tpci asdled snaver
bit comeette the crttee
ASSISTANT: A snogtinl
ST STE No niwar expetinno.
S1 STEM Your
plamppin, sysa lnwavee da cazeel sntwar tanano insplical desied and
ASENSTAN ANT: TTENK Sorns them mixane
AMRISTA
THINKI make daiie,
SYSTEC
SYSTEM
ASSISTANT: THINCCoffees scoss be gomppic,, so anoverts scaffiee spmppont)
NOTEuxappaa "sussplar" desied
SYSTEC OK.
SYSTEM:
ANSWER inattte gipptcil
MOTE
Mbaaly mady tope modia, Eamans disq cn lil Jatume tomat of tittyy tppi Di indidd ondr the snover
wotty regeeeece repereccne ding emgggnn
(nndy, apel Do ant ladadd voocne anced cocaat
Cimippleelle SrAusela g (hflepranzat Paaacea COVID jud addea BuslBogas
@ysutoghgingg Except
Misardae you kaze and al oper vacaa causs pengeres AN vactee)
perzi pastle nilvott oppetime
SUSTEM "yoerzea" diowro
STSTEM "C0V as desrd entry.
SYSTEM: oooer prodtas soocis, hovrvve the commot onsoert indtts sunthes
ASSEST ANT: THENK fexed
"poaeeedl" icadboue, 'C2V10"
Kaddad.
ASSISTANT: THINKI modk nniatale soctae is Ext the dasine catty and XKKM vaocias is
ASSISTANT: ANGWEI skmaa))
SYSTEM Di
ASSESITANO MT. NOTROwo "naaddee" to deeind etday.
SYSTEM OR.
ASSISTANT: NOTEMOOA kac aacaao) ZENA nacizao is evacczs
SYSTEM: OR.
STST ideetly caolas:
STIM Now hame gpedions
medite divers, or emppicn tn the Behure
S7 STEM: Hontiy or catyy tye voctae, dirose dag. Et prgppptoc in this tot Retunn in
foree efientyy typt to intlade "oodff in snnver
Soreet of teeryy typel De incidd "oons" meterr
Brwskgmas mony people porrg normal Like Gorld quectt: P9SS trm tetall
a'latt PallBaay Hos soo bdiis dumbos Faur. dazs fuuica beepuaat
Inlescdiive Trgertonies es
slock iff:
of COV ant aed to bappligs paw seaddig endakkene etbiaa
ASSISTANT: ANSM BEJPEED asçlasi]
abcico pactcb
SV STENE Fox aswar io PT5D. isspiial lawewe the ounadt anvee IPTS0, diaact
Sr STIM Thee ae useed prod metere reperinoo
ASSISTANT: THENI)) made mitaas, PTSD ic doees, nat copaptios
ST STEM: igigtict lotel deand mayy
STEM OK:
ASSISAKTT ATT TRINKEEosed ce thes orpeneeoss. "ormprren of tot dested entry. So
ARSESTANT ANT FTTTI "FT377 taadiooo
GuavE duades apppac))
SY STEX OK.
St STBM OK.
ASSISTANT NOTE, 'oospaas" be cleand anddy.
ASSIRTANT: ANSWER ( Jackna, explizi]
STEEE OK.
SY STIM Your
ASSISTANT ENDO
Figure 6: The example trajectories of ExpNote on the METS Dataset. The left part a training case while the right part is a corresponding testing case using that training experience. The blue parts are the ExpNote demonstrations (prompts Peroen/Ptess The yellow parts are the interactive trajectories between LLM and ExpNote. The sentence highlighted in green is the learned experience.

---

Example Trajectory Training and Testing Cases on the EMOJI D Dataset
T Twiniin Stagr
Testing Stage
St STSTMM-Y Yoe cautt sscstot and ye need conglrx the spectiad tak by killagg to the STEM The diilog betroce you and STEM will be shown the fimm "ASSISTANTT and SYSTEM?
complne the qpectied by aakinn STEM The daiky biteaa you and St STEM will be slavu the lazz "ASSIETANIE and 51 STEM
STIMI am somtt shadd
STEM: carkek mladd the sangi Una Les, catcas Cas tod.
eougi that motbes the
51 STST Hize on seconl eeaplees
51 STEM csse
STDM: Ham sre oveed eourppee
STMM: Wiaa aucucaa vbdo lake. US00IEES 11000E20 (1005E30 (0006117%
STMM
ASSISTANT: VHENNN wellows
STMM:
us poacll ahwma oppatiocss
4.59ETANT ANSWWERE the orewen (100011311
STEM: //0001516 indiants aomtimmntt of gutt
STET
STM5 (10001155 inclome suntineet of anger
ExpaNote Denoonstraioon
ASSIETANT THINKI checld the thk ongj fhr fiom inferenee
(30011500 intiates centrnet foor
3100301333
AS9ISTANT THINa(ita welines acii wasy. Bandd uc scpacticcs seaever
5 (2000)
theaght
STEM: OK.
muuagg 100001500 W0000153
ASSIETNNT ANSWERE
(U0001123)
STEM
ANNWEERE toeer (10001131)
STMMM When the
reat dlnn owene tmaght
cur uauee 1,000 9633, lawwer cunnd 103001225.
will eene mnnnage H! tH proeereo yooesrecs 100001900
ASSEETANT TIUNA) roold any the meetnn thko wneei the ti intemnoee
STBM: Ham arvend pour adeeat
51 STEM:M:: OK.
ASSIETANT: NO0FE 11000tm33 indicies soetrmnt of sothess
indiatss acnttnett
53 S35T::: oun
61001155 Indicm comtmnent cotneer
53 ST5T Mier phopped bettle in bhe dhts toue deow and eovey bubyy abarted lnaghing.
STEM: /U00 infiiutes a pontment of aaager:
XL000L390 210001822 ( 010001225 (1000
ASSISTANT THINa/hh wettore tit ortwos Thared im epparinoe,
AISISTTNT: The wetmeet the fet moree the resvem 11E00197T
ASSIETANT THINKIT The sontient of ken shane))
STEM:
51 STEM: OK.
ASSISTANT ANSWERR So arsveer (000001133
ASSEZANT: As i (00001833)
ASSISTANT: THINEG) blncal
STMM: Your
(-300-3CE, towwer
dak umaj. faaze malenees
in 11300187.
STEMM nox
When drgpet botte statt strn stotel lnighing
OY STSTTM IIK.
ASSIETANT: (O00U0 (1000197 inficates
uee avand po adenad saachsa
STIM (1000157 indeo wedtmnt of cheam
STEM: (UX0G adiotto avedaase of cloqal
STEM: (20015223
ASSIETANT THINN/Tte soctiner: of Ois lon duace 3acod these opeiimos the zaneor
STOM) BK:
ASSISTANT ANSMIDKE
100001174
STST ENN
New let I: gusiii::
S1 STEX. New in qustac:
ST.T IM. Rebbe zn acciint, onn atfern wne the froe sest of oo ane the ofherr
31 STEX: War ne falae sall noniat in lanw sad olat anovand inea ilava, werk had
Codaad. Skk task Eat and wis kilid tlax whaa Car out cruibod. (130008866 (2000E22)
adwer Tilk
well ider: yoeottees
013001ES1 LO000192
110001155 W1001130 1000
ASSSSAAN: THINKT sedanad La tod sigat]
STEV. Hom
postiie milere expettne
Inteasetiive Trajectories
STISTEM
STEX: (X00L
of axbeonee
ANEWERS
STEX: L0002
ASSISTAN:: duddd exania
adance))
ASSISTANT: THINKTT scathiott regntt these eeporicos,,
right
STEX:
ASSSTANT:
ASSISTANT. ANNWELUS aawara (20001116))
STEX: Yox ABWAE ANIVEL
Figure 7: The example trajectories of ExpNote on the EMOJI Dataset. The left part is training case while the right part is a corresponding testing case using that training experience. The blue parts are the ExpNote demonstrations (prompts Peroin//tees Pt The yellow parts are the interactive trajectories between LLM and ExpNote. The sentence highlighted in green is the learned experience.

---

Example Trajectory- Training and Testing Cases on the LETS Dataset
Training Stage
T Testing Stage
SYSTEM Yee an ceatt senitinn and Yee neet to conpine spectfed ever saliingg the SYSTEM, The Gulg betwace suu and Bt 57MM will be beslooo as Raza "A "AESEEDANT and *51 STEN:
STEM: Yeu extaet mvitted and compina suediill baic 51 STEM. 7he diuly letnas sea and ST STEM will lo slows as faza "AESEETATT" and 550 STEN
51 STPM Year beic in to oplion the spedtted betive regeterr
thcs sphio specthed lehoon regettor
51 STSTMM Hire on sewel osangles
51 S15TMM Fine sewel econglee
SYSTEM: cuac
S1STEM: Sptio 6Cs lottr of "uunyy the bud ketzr "akket, and the at lstor of 'oxbr" kpaller:
51 ST5TTM con
S15TEM:
kitar "saaaa", lit "tasasmal of tigelerr
ASSISTA VT TinNac The lebur "semp" to "gg". bebwer "belee" het and
viter huaar ten)))
SYSTEM mileort epeninnn
SYSTEM: OK.
ASSISTANT: THINR/Sons there is adleaat experiecc about this gustio, nedd onl kysaf)
ANEWEIES uplced wodd
SYSTEM:
ASSISTANT
upding
rifenmal
ASSISTANT: THUNKTTl kftar "auasall
kdtar "apppoue? "p.. Ea
ExpNotte Demonstratons
NDTTItanns) "tanay" oxepeed
4biime el ''flicl" a 'T))
S1STEM: 08.
Interts drs lettr
leter "y"
ASSISTANT: ANSWERS st the splood lebon would be
STEMMM OR.
ASSISTANT: NOTEusked "uake" romposed a.l L.
kita 2ad lake
katcii tot loke 11, botar "V'' dia in
STET cam
(K.
SYSTIM Speies the ras leber of "tsmpp the 1rd letern of "ruceet, and the lntov ooter"
ASSISTANTI NT: NDttitea)) "oote" exppored of "t Latt So for the word toiert the het faterr
'c. 2rd witer in "o". inter tht the ins leterts"""
SYSTEM: Hax scvera adovat
53 SYSTEM: O8.
"tanty" compottt
pf the theney"
55 S1STEM: 2
kotz 6 *V, lotee tn Isteris et 50 letts is "Y and the Intor yo
59 5Y5TBM: Splav in dis batacel "aaaad" the 30s lote of "tagavaaa; and the Inthr "slica"
51 SYSTEM 'enn It compose "c a tuat So for wond "ooer" the ter inter her, 2nd lettr 3rd inter 11, and 4th arder tot
vegeter
ASSISTANT MT: THINKTT The its letor of "mere"" to itt the tii of "mprosee" ts'r and the 4is
ASSISTANT: THuN sppakaa,
the kotaci
Vittr "uflvial *111
STEM:
SYSTEM: OR.
ASSISTANT: THUNI)A the Snd kear "'allcc" in "ta as the anc Iitaria (21)
A.53STA VT: TIBNB)) ahoeld wate spel thes hebm mitmeess
ASSISTANT: AN the spliod letce word be har')
59 SYSTEM: Yoursae
"ust, knvene, the mes anowe
SYSTEM: OK.
SSISTA NaaVER to the spllod kebwe would he "yle
51 SYSTTMM UIK.
ASSISTANT: NT: NDTEBoeeell "nomet" cxppore 46 ". Siloke de Edilas
So the word "aoonaal
mopoone "regroomn" compaettion puput
1/", 53 STEM: OR.
keterts 9t
ASSISTANT: CoC "attiia" cocpesed N.. c.u. the Noticial.. baacci 2rd baturii I7, Kolaatar idtear di T.
ketwrn fot, eadt ita (i1 st STRMM UK.
SYITTM Now bor to qpotice
SE SESTEM Nor Las la caatioe
STET Sykee dh 7t Issord of portay" Oe 50 lettr of "auttr" and the Ister of "red' tigetirr
St SCEM: Splce the 30 lott of "saa"" ix ob lote of "poces" and the 1st letor "red" lepelaer
ASSESTAND: THENTTT Malatae of "ponlias" int the S&k latur of taite" aa and la dd.
SY STIM Her em amverel poott reineet exper ence
Interot "red" 1e 1
St STMM poosoot is compose i a a.s ot So for the went "sooweer" the tel letorr
SYSTEM OE
"y" 2nd lnttr it for. 3nd intor not th lnttr is 5thh inter is te. Cth lettr fe't and the 7th lake '>".
ASSISTANT ATI AN the qetine letere world 1
SYSTEM Your ucaver canee.
St STEM: roof oocpood of is df So fir the word "sodt the betor to leteris
Interrativee Trajectorics
ASSESTANT: THENK shold boto the speliin these words for roture reformo))
Sel lake "s ln 4is botar is 'I"
SYSTEM Oz.
wont hokey the fis lebtr tol leber bebers ta 4th inter and bebor
STMM: day"" ts comeppsed th
ASS0ST AT yotrel "ywerre" compoeet
Un So sceed keteti ler tt leberie B.s.
"gredae', the bet batck of' wdee the 3nd
ASBISTANT THINaCEaodd those
"strg" Ietorno
Hiberts irter hit, the webrts
SYSTEX
SDEM:OK OK.
ASSISTANT: NOTEEaatot "raaoor is 'it,
word Thator.
ASSISTANT ANSWEEU
ASSISTANT weds coceppood 2dlittari la the liker it..
Figure& 8: The example trajectories of ExpNote on the LETS Dataset. The left part is a training case while the right part is a corresponding testing case using that training experience. The blue parts are the ExpNote demonstrations (prompts Ptrain/PPees test). The yellow parts are the interactive trajectories between LLM and ExpNote. The sentence highlighted in green is the learned experience.


Reviews:
[{"review_id": "Zvxl7uCXWP", "reviewer": "Reviewer_r6eq", "summary": "", "questions": "Line 35 is not clear.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Thank you very much for your review. Here are our responses to your questions.\n- Q1: How the task-specific experiences are learned for example line 147: \"if A has a daughter B, and A is married to C, then B is also the daughter of C\". Are these experiences part of the data? If not, the poor quality of these experiences can negatively impact testing.\n- A1: These experiences are not part of the original data, but are automatically generated by the LLMs through few-shot prompting. By constructing example prompts, LLM learns task-specific experiences that can be generalized for other testing cases (specific prompts can be found at expnote_train in ./dataloader/<task>.py). High-quality experience is indeed important. For specific tasks, heuristic rules and examples are used to guide the ExpNote framework in automatically summarizing experience. By extracting experience from positive/negative samples in the dataset and generalizing it to the test set, relevant experiments have shown that the quality of experience obtained by this method is better than simply using examples for demonstration, and it is also more robust than reflection-based methods.\n- Q2: What are in-context learning examples during testing? Are these examples with the experience recall or just input-output pairs?\n- A2: In-context learning examples during testing include experience recall. The LLMs learn to reason based on recall experience during testing through the ability of in-context learning (specific prompts can be found at expnote_test in ./dataloader/<task>.py)\n- Q3: What retriever is used? How the relevant experiences are selected? I do not find any information in the paper.\n- A3: As stated in Appendix B, we used a keyword-based retriever to retrieve experience by matching words in the query and the key of experience, and then selecting the k most relevant experiences. In this paper, we mainly emphasize the overall idea and process of the ExpNote framework, without overly considering the implementation methods of different retrievers. In the future, we will compare the impact of different retrievers on the model's performance.\n- Q4: Why the author thinks recent reflection-based methods (Shinn et al., 2023; Madaan et al., 2023; Akyürek et al., 2023) do not generalize to other instances or tasks?\n- A4: Methods that are solely based on reflection only generate case-level specific experience. For example, for a testing case in ALFWorld (a dataset used in Reflexion), the reflection is ' In the next trial, I will go to desk 1, find the lamp, then look for the mug and examine it with the desklamp '. Therefore, these reflections can only be used to prompt the next trial of the same case, and can not be generalized to other test cases. In contrast, by storing and then retrieving, ExpNote can help LLM further think and more comprehensively utilize the information in the training set that is helpful for specific tasks.\n- Q: Line 35 is not clear.\n- A: Regarding the black box model of ChatGPT, we used OpenAI's gpt-3.5 turbo interface, as mentioned in Appendix B.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Empirically, it is evident that the proposed approach is better than existing reflection/memory-based methods.\n- Table 2 clearly shows that the LLM equally benefits from both positive and negative experiences.", "reasons_to_reject": "- How the task-specific experiences are learned for example line 147: \"if A has a daughter B, and A is married to C, then B is also the daughter of C\". Are these experiences part of the data? If not, the poor quality of these experiences can negatively impact the testing.\n- What are in-context learning examples during testing? Are these examples with the experience recall or just input-output pairs?\n- What retriever is used? How the relevant experiences are selected? I do not find any information in the paper. \n- Why the author thinks recent reflection-based methods (Shinn et al., 2023; Madaan et al., 2023; Akyürek et al., 2023) do not generalize to other instances or tasks?", "questions_to_authors": "Line 35 is not clear.", "paper_topic_and_main_contributions": "The authors propose a compelling framework to incorporate LLM experiences from training data to improve LLM performance on test instances. Experiments are conducted on 4 different datasets on ChatGPT (gpt-3.5-turbo). The authors also propose a new dataset LETS to evaluate the symbolic reasoning capabilities of LLM. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WAMR8zWBtk", "reviewer": "Reviewer_CVaE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Thank you very much for your review. Here are our responses to your questions.\n- Q1: All experiments were performed on a very limited of test sets (100 cases) and train sets (minimal 100 or 200 cases according to Appendix B), potentially causing the risk of randomness in terms of the results.\n- A1: Due to the size limitations of the datasets themselves, we used samples of the order of 10^2 for training and testing. **In fact, a large amount of related work is also tested using samples of similar magnitude, such as TeachMe (OBQA, 500), ReAct (ALFWorld, 134; WebShop, 500), Reflexion (consistent with ReAct).** Our proposed LLM in-context learning framework can induce experience from the training set and help generalize it to the test set. It can also demonstrate sustained learning ability and good performance in a small number of samples. The dataset contains both balanced positive and negative examples, fully reflecting the possible distribution of samples in the task dataset. The superior performance of the experiment on multiple different tasks and datasets can demonstrate the universality of the method. \n- Q2: The ExpNote framework was only tested with one LLM, i.e., ChatGPT (which was even not mentioned in the main part of the paper, just occurring in the Appendix), which is not solid and persuasive enough.\n- A2: Considering the good performance of ChatGPT (gpt-3.5 turbo) on various NLP tasks, in order to highlight the universality of the ExpNote framework, we are currently verifying the effectiveness of the method on this LLM. The current experimental results can provide inspiration for other similar models. Thank you for your feedback. We will explore applying other LLMs to the framework of this article in the future.\n- Q3: The main contribution of this paper is a little incremental to previous work like memory-based and reflection-based prompting methods.\n- A3: Exploring the boundaries of LLM capabilities is a novel frontier direction. The ExpNote framework proposed in this paper brings a new perspective to the research direction of the entire LLM prompting method, creatively integrating the advantages of memory-based and reflection-based prompting methods, enabling ExpNote to summarize abstract knowledge and rules from correct or incorrect responses, enabling the model to better handle unfamiliar tasks and unseen samples. By conducting experiments on multiple tasks, it was demonstrated that ExpNote performs significantly better than a single method in different scenarios. The experimental results on tasks such as inductive reasoning, entity recognition, and symbolic reasoning also indicate that our method can significantly improve the performance of large models by 20.5% compared to CoT; Compared to other methods based solely on memory and reflection, there has also been a significant improvement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper proposes a novel and effective framework for LLM in-context learning and achieves better performance against strong baselines.\n* The paper was written well and fluently, easy to follow. The framework figures are illusitrative and help the readers know the main idea easily.", "reasons_to_reject": "* All experiments were performed on a very limited of test sets (100 cases) and train sets (minimal 100 or 200 cases according to Appendix B), potentially causing the risk of randomness in terms of the results.\n* The ExpNote framework was only tested with one LLM, i.e., ChatGPT (which was even not mentioned in the main part of the paper, just occuring in the Appendix), which is not solid and persuasive enough.\n* The main contribution of this paper is a little incremental to previous work like memory-based and reflection-based prompting methods.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes the ExpNote (Experience Notebook) framework, a novel LLM prompting method. ExpNote reflects experiences and store them as notes in an external memory in the training, and retrieves the experiences in the inference to help the LLM better solve unfamiliar tasks. The authors compared their proposed method with three strong baselines involving Chain-of-Thought (CoT), memory-based and reflection-based prompting methods. ExpNote outperforms these baselines on four different types of tasks in inductive reasoning, medical entity recognition, text-to-emoji prediction and letter splicing. They also performed an ablation study with several variants of ExpNote. The main contribution of this short paper is to propose a novel and effective framework ExpNote for LLM in-context learning.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IoWx4yPWQq", "reviewer": "Reviewer_RD22", "summary": "", "questions": "Please see point 1 in Weaknesses. ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Thank you very much for your review. Here are our responses to your questions.\n- Q1: The main problem of this paper is the unclear writing. It is not easy to follow the whole process without knowing the clear details. \n  - For example, to store the task-specific experience, key-value pairs are necessary. How to generate such key and value pairs are not fully described. \n  - For another example, how to generalize a training instance to an abstract experience? Is this done by heuristic rules for specific format data? When there are multiple experiences recalled during a testing phrase, how to decide which one to be used? \n  - After reading the codes, these questions become clearer for me. \n  - However, it should be clearly described in the original texts. I suggest that the example prompts in the dataloader should be appended in the appendix.\n- A1: Thank you very much for your advice. We have recognized these points. As we prefer to present the ideas and experimental methods of the ExpNote framework, due to space limitations, we have appropriately simplified the prompts and operating procedures related to data loading. Our code clearly presents how ExpNote generates key values to store task-specific experiences, how to generalize training examples into abstract experiences, and how to recall and use experiences. We will add relevant implementation details to the appendix in the revised version.\n- Q2: Why do you choose these four datasets? The previous three datasets are not well-known for LLM evaluations. And the last one is a synthetic dataset. Are there any specific reasons?\n- A2: Thank you very much for your interest in dataset selection. When selecting these four datasets, we considered some specific factors that cover tasks in different fields. CLUTRR is a classic dataset for inductive reasoning and is often used for comparison in related COT research. METS-CoV and EMOJI are relatively difficult downstream task datasets that often require fine-tuning to achieve better performance. In addition, currently, the symbol inference dataset is mainly used to test the zero-shot ability of LLM, and no training set has been designed. Therefore, we constructed LETS to evaluate ExpNote's ability to enhance LLM's learning and generalization in symbol inference tasks. We verify our proposed framework on these datasets for different tasks to ensure that the applicability and robustness of the method are comprehensively evaluated.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The research topic is quite interesting and would draw a lot of attention from the community. \n2. The proposed method is reasonable and effective on multiple benchmarks. \n3. The authors design sufficient ablation studies to show the effectiveness of the proposed method. \n4. The code is released. \n", "reasons_to_reject": "1. The main problem of this paper is the unclear writing.  It is not easy to follow the whole process without knowing the clear details.  For example,  to store the task-specific experience,  key-value pairs are necessary. How to generate such key and value pairs are not fully described.  For another example, how to generalize a training instance to an abstract experience?  Is this done by heuristic rules for specific format data?  When there are multiple experiences recalled during a testing phrase, how to decide which one to be used?   After reading the codes, these questions become clearer for me.  However, it should be clearly described in the original texts. I suggest that the example prompts in the dataloader should be appended in the appendix. \n\n2. Why do you choose these four datasets?  The previous three datasets are not well-known for LLM evaluations. And the last one is a synthetic dataset.  Are there any specific reasons?\n", "questions_to_authors": "Please see point 1 in Weaknesses. ", "paper_topic_and_main_contributions": "This paper presents a framework called Experience Notebook or ExpNote for providing external and dynamic memories for large language models (LLM).  The authors design a set of action prompts to enable the reading and writing with the external memory. During training,  ExpNote will automatically generalize task-specific experiences to external memory using reflection thinking. During testing, ExpNote retrieves related memory for supporting the LLM’s decisions. Experiments on four tasks show that the proposed methods are much better than the baseline models including Chain-of-Thought,  Reflexion and TeachMe. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wQ8ICEpBiv", "reviewer": "Reviewer_ZAJN", "summary": "", "questions": "Did you consider retrieving relevant demonstrations rather than only using a fixed set of demonstrations? In other words, is it clear that using demonstration as \"experiences\" is more important than simply using the raw demonstrations?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Thank you very much for your review. Here are our responses to your questions.\n#### Paper Topic And Main Contributions:\n- Q: The authors present a prompting that utilizes demonstration-like retrieval. I call it \"demonstration-like\" because what is included in the prompt is derived from a demonstration (called relevant experience). Although a potentially novel technique, I found the method is talked about quite abstractly and the paper is missing details that would be necessary for replication. In addition, although the paper includes multiple ablations there are not sufficient baselines to know if this is an overall promising technique. As an example, at least one prompting baseline is missing for CLUTTR which greatly outperforms ExpNote.\n- A: \n    - Our proposed ExpNote framework and methods such as demonstration have similar functionality in prompting LLM to generalize from known information to new and unseen instances. However, our method has unique features in extracting, storing, recalling, and utilizing these aspects. We have designed a set of action prompts that interact with external experience bases, combining the advantages of reflection-based and memory-based approaches from a new perspective.\n    - ExpNote fully automates the learning and induction of experiences from the training set, providing prompts with a fixed number of experiences, saving on the number of samples and storage space required for demonstration using input-output pairs, and improving generalization efficiency.\n    - We have provided clear code to support the reproducibility of our work. We have applied the framework on datasets in different fields and tasks, demonstrating its universality and robustness.\n\n#### Reasons To Accept:\n- Q1: It is interesting to include \"experiences\" which are derived from demonstrations rather than demonstrations themselves. That being said, this has been done before with chain-of-thought.\n- A1: We have constructed a new approach to guide LLM in inducing experience from task training sets and enabling recall of these experiences during testing. Unlike COT, our approach focuses more on extracting and storing experiential information from task-specific perspectives for better application in subsequent inference processes.\n- Q2: The paper surprisingly shows that using experiences where the model failed (called negatives) is very helpful. Although, on closer inspection, this may be because the failure rate is high and that too much data is excluded when excluding failures.\n- A2: The experiences from negative examples are indeed important for our framework. But this is not because the failure rate is high and that too much data is excluded when excluding failures. In fact, as shown in Table 2, in 3 out of 4 datasets, the positive experiences are more than the negative experiences (The numbers in the small bracket are numbers of experiences stored in the memory). **We think the true reason is that the data from model failures contains valuable information for specific tasks, which may include errors in the model's reasoning and judgment in different situations. ExpNote can help the model better understand and adapt to new situations.**\n- Q3: ExpNote substantially outperforms ablations and two baselines (TeachMe and Reflexion). Although some important prompting baselines may be missing.\n- A3: Compared to methods such as CoT, reflection-based, and memory-based methods, ExpNote's performance is outstanding, and more importantly, the framework is fully automated without the need for human intervention. Regarding the issue about other baselines, we will explain it in the following reply.\n#### Reasons To Reject:\n- Q1: There are probably not enough details to reproduce the work without guessing at multiple steps. For example, it is not clear what is a \"word-based retriever\".\n- A1: We highlighted several main contributions of the ExpNote framework in the paper, including the design of a fully automated training paradigm that integrates the advantages of reflection and memory, and the design of multiple action instructions that interact with external experience bases. Due to the limitations of the paper, we did briefly describe some details, but through our publicly available code implementation, Readers can have a deeper understanding of each step. For example, for a word-based retriever, in the 3.2 Interaction Commands section of the paper, we explicitly mentioned \"using the question arg as the search query\". Specifically, it retrieves experiences by matching words in the query and the key of experience and then selecting the k most relevant experiences.\n- Q2: ExpNote probably needs negatives for good performance because so many of the initial predictions are negatives. The positives are only 70% in the best case (EMOJI) and can be less than 50% on some data. Plus the data is somewhat small to begin with, making the inclusion of as much of the train data as possible important.\n- A2: \n    - Our work found that negative data may contain useful information for specific tasks, and ExpNote is able to fully utilize these samples in both training and testing processes. By storing experiences in external memory, the model can retrieve and utilize these experiences in different tasks and situations, thereby enhancing its generalization ability. **We think leveraging the negative cases during training to generate experiences for testing is a novel idea in our paper instead of a weakness.**\n    - In order to analyze the impact of experience types on performance, we have conducted relevant experiments and trained our method on only correctly answered sample datasets, which can still significantly outperform the baseline, as shown in Table 2. \n    - Regarding the issue of data size, our experimental results indicate that even with relatively small initial data sizes, ExpNote is still able to continuously learn experience during the training process as the sample size increases (Figure 3), and has achieved significant performance improvements on multiple tasks.\n- Q3: It seems the baselines are not particularly strong. For example, DIVERSE can achieve > 90% accuracy on CLUTTR [1].\n- A3: Thank you for pointing out. While DIVERSE performs well on CLUTTR, we believe it is not suitable for comparison with our work. \n    - There are several reasons for this:\n        1. DIVERSE has made multiple modifications to the CLUTTR benchmark, including appending candidate answers and constructions after the question. Their experiments show that without these candidate answer sequences, the accuracy of LLM does not exceed 50%, while we achieve an accuracy of 61% without additional prompts.\n        2. DIVERSE incorporates manually constructed reasoning paths into every question-answer pair in the training set, requiring a significant amount of manual annotation. In contrast, ExpNote only requires a small number of prompts to automatically induce and generalize from the training set.\n        3. DIVERSE uses self-consistency decoding, generating different reasoning paths and conducting majority voting, while ExpNote uses greedy decoding, where the large model samples the most likely word from the vocabulary. In Faithful CoT (Lyu et al.), these two types of methods are compared separately.\n    - Additionally, in comparison to the methodology of the comparative approach, ExpNote's advantage lies in its simple experience retrieval during the testing process and the complete automation of the entire training process without human intervention.\n        1. DIVERSE attempts to generate diverse prompts, exploring different reasoning paths for the same question, and then filters out incorrect answers based on a weighted voting scheme, verifying each reasoning step individually.\n        2. This method has high complexity and requires human involvement in various stages. For example, it involves the need for more annotated data with well-annotated reasoning paths to construct diverse prompts. It also requires a training dataset for supervised validators and relies on human evaluation to measure the quality of intermediate steps in the reasoning paths.\n#### Questions For The Authors:\n- Q: Did you consider retrieving relevant demonstrations rather than only using a fixed set of demonstrations? In other words, is it clear that using demonstration as \"experiences\" is more important than simply using the raw demonstrations?\n- A: Thank you for your question. We have already taken this into consideration. In line 229, we introduced a variant case of the ExpNote framework, which recalls the original relevant Q&A pairs instead of learning experiences. The experiment shows that ExpNote's use of reflective ability to summarize experiences makes LLM perform better in specific downstream tasks than simply using case or demonstration, and the effect is significant.\n#### Missing References:\n- Q1: [1] Making LLMs Better Reasoners with Step-Aware Verifier by Li et al. 2022 presents a technique for finding more reliable CoT reasoning paths. It has strong results on CLUTTR.\n- A1: DIVERSE is not suitable for comparison with our method, as detailed in Q3's answer.\n- Q2: [2] Compositional Semantic Parsing with LLMs by Drozdov et al. 2022 uses demonstrations that include their pre-computed CoT. Also, the demonstration retrieval involves template-like steps that have some similarity with experiences used in ExpNote.\n- A2: The experience in ExpNote is an abstract knowledge or rule related to a specific task. This literature proposes the least to most method to perform multi-step semantic analysis of the problem, decompose the sub-questions, and output the answers to the sub-questions to prompt the final answer. The information obtained through retrieval decomposition is a 'sub problem', which is significantly different from the experience in ExpNote.\n- Q3: [3] Faithful CoT (Lyu et al) shows that adding explanations to CoT is beneficial.\n- A3: ExpNote mainly compares reflection-based and memory-based methods, and may have missed out on Faithful CoT. The main difference between the two is that Faithful CoT manually sets fixed enhanced inference prompts for each task, while ExpNote automatically induces abstract knowledge or rules to obtain more types of experience for use, and can be transferred to tasks with similar domain knowledge.\nThank you very much for your suggestion. We will consider adding it in the revised version.\n#### Typos Grammar Style And Presentation Improvements:\n- Q: Why are no decimals shown in Table 1/2? Is the full data being used or is this a sample of 100 test cases?\n- A: The reason for the lack of decimal points is indeed because various methods were tested on 100 test samples. Due to the size limitations of the datasets themselves, we used samples of the order of 10^2 for training and testing. In fact, a large amount of related work is also tested using samples of similar magnitude, such as TeachMe (OBQA, 500), ReAct (ALFWorld, 134; WebShop, 500), Reflexion (consistent with ReAct). We believe that the ExpNote method emphasizes the integration of experience and external memory, and performance improvement in different fields and tasks can reflect this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strength 1. It is interesting to include \"experiences\" which are derived from demonstrations rather than demonstrations themselves. That being said, this has been done before with chain-of-thought.\n\nStrength 2. The paper surprisingly shows that using experiences where the model failed (called negatives) is very helpful. Although, on closer inspection, this may be because the failure rate is high and that too much data is excluded when excluding failures.\n\nStrength 3. ExpNote substantially outperforms ablations and two baselines (TeachMe and Reflexion). Although some important prompting baselines may be missing.", "reasons_to_reject": "Reject 1. There are probably not enough details to reproduce the work without guessing at multiple steps. For example, it is not clear what is a \"word-based retriever\".\n\nReject 2. ExpNote probably needs negatives for good performance because so many of the initial predictions are negatives. The positives are only 70% in the best case (EMOJI) and can be less than 50% on some data. Plus the data is somewhat small to begin with, making the inclusion of as much of the train data as possible important.\n\nReject 3. It seems the baselines are not particularly strong. For example, DIVERSE can achieve > 90% accuracy on CLUTTR [1].", "questions_to_authors": "Did you consider retrieving relevant demonstrations rather than only using a fixed set of demonstrations? In other words, is it clear that using demonstration as \"experiences\" is more important than simply using the raw demonstrations?", "paper_topic_and_main_contributions": "The authors present a prompting that utilizes demonstration-like retrieval. I call it \"demonstration-like\" because what is included in the prompt is derived from a demonstration (called relevant experience). Although a potentially novel technique, I found the method is talked about quite abstractly and the paper is missing details that would be necessary for replication. In addition, although the paper includes multiple ablations there are not sufficient baselines to know if this is an overall promising technique. As an example, at least one prompting baseline is missing for CLUTTR which greatly outperforms ExpNote.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Why are no decimals shown in Table 1/2? Is the full data being used or is this a sample of 100 test cases?"}, "extra_text": {"Missing_References": "[1] Making LLMs Better Reasoners with Step-Aware Verifier by Li et al. 2022 presents a technique for finding more reliable CoT reasoning paths. It has strong results on CLUTTR.\n\n[2] Compositional Semantic Parsing with LLMs by Drozdov et al. 2022 uses demonstrations that include their pre-computed CoT. Also, the demonstration retrieval involves template-like steps that have some similarity with experiences used in ExpNote.\n\n[3] Faithful CoT (Lyu et al) shows that adding explanations to CoT is beneficial."}}]

Author rebuttal:

Meta-review: Reviewers agree this work offers an interesting and topical method for leveraging training data "notes"/experience at inference-time. Though some are concerned its contributions may be incremental, it does introduce nice augmentations on existing work for a short paper. Its results indicate strong potential, however, there are also universal concerns with understanding crucial details of the method and experimental setup, as well as missing related work. After significant discussion, I hope the authors will heed the advice of the reviewers and agree to (a) fill in missing details from their methodology, (b) add the requested modifications to the related work, and (c) augment the discussion with their additional experiments and carefully contextualize their quantitative results given the small train/test sizes (which also appears in prior work). With these changes the work will be much improved.
Decision: Accept-Findings