Learning to Reason and Memorize with Self-Notes
Authors: Jack Lanchantin
Shubham Toshniwal
Jason E Weston
Arthur Szlam
Sainbayar Sukhbaatar
Venue: NeurIPS 2023 poster (2023)
Keywords: Memory
Reasoning
Language Models
TL;DR: We introduce a method to allow language take notes at any point while reading, leading to improved reasoning and memorization abilities.

Abstract: Large language models have been shown to struggle with multi-step reasoning, and do not retain previous reasoning steps for future use. We propose a simple method for solving both of these problems by allowing the model to take Self-Notes. Unlike recent chain-of-thought or scratchpad approaches, the model can deviate from the input context at any time to explicitly think and write down its thoughts. This allows the model to perform reasoning on the fly as it reads the context and even integrate previous reasoning steps, thus enhancing its memory with useful information and enabling multi-step reasoning. Experiments across a wide variety of tasks demonstrate that our method can outperform chain-of-thought and scratchpad methods by taking Self-Notes that interleave the input text.

Learning to Reason and Memorize with Self-Notes
Jack Lanchantin' Meta Al
Shubham Toshniwal" NVIDIA
Jason Weston Meta
Arthur Szlam Meta Al
Sainbayar Sukhbaatar Meta AI
Abstract
Large language models have been shown to struggle with multi-step reasoning, and do not retain previous reasoning steps for future use. We propose a simple method for solving both of these problems by allowing the model take Self- Notes. Unlike recent chain-of-thouge or scratchpad approaches, the model can deviate from the input context any time explicitly think and write down its thoughts. This allows the model to perform reasoning on the fly as it reads the context and even integrate previous reasoning steps, thus enhancing its memory with useful information and enabling multi-step reasoning. Experiments across a wide variety of tasks demonstrate that method can outperform chain-of-thought and scratchpad methods by taking Belf-Notch that interleave the input text.
Vanilla
Context
Q
model
Context
A
Scratchpad or Chain-of-Thought
Context
model
Context
reasoning
A
Self-Notes
Context
model
Con..
Self-Note
-te..
Self-Note
.xt
Self-Note
Figure [top] Vanilla language models directly generate the answer (A) given the context and the question (Q). [middle] Scratchpad and Chain-of-Thought methods encourage the model to generate reasoning tokens before answering the question, but only after it has read the entire context. [bottom] Self-Notes (ours) allows the model to generate multiple internal reasoning notes that interleave the input context and question.
Introduction
Transformers 1 and similar variants have reshaped the field of machine learning with impressive results on sequence-based tasks 22 Notably, large language models (LMs) such as GPT-3 (2 use transformers and capable of solving various complex natural language tasks such as question answering. However, it's increasingly evident that there are still limitations to these models. Namely, transformers are limited in their ability to perform multi-step computations store intermediate results due to the lack of an explicit internal dialogue or scratchpad 3 5
When an LM is used for question answering (QA), it is typically fed a context prompt containing factual information along with question, and then the model generates the answer directly, as shown in Fig. (top). However, this autoregressive "one-step" approach struggles with multi-step reasoning
Equal Contribution. Correspondence to sainbardmeta.com
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

tasks 7 8 4]. We argue that this arises from the fact that vanilla LMs have a fixed computation budget for each token, and do not have the option to "think" more depending on the current context.
Recently, Chain-of- Thoughhtt and Scratchpad methods 3 10] encourage the model to generate reasoning tokens explain their answer one step at time, leading to improved reasoning capabilities. These extra tokens are added before answering the question, but after it has read the full context and question, therefore postponing all "thoughts" until the end, as illustrated in Fig. [ (middle).
In addition to the "one-step" problem, transformers lack a recurrent memory for state-tracking and solving highly nonlinear tasks [II], something that recurrett predecessor models such as the LSTM [12 are well equipped for. Modifications to the feed- forward transformer architecture that use a recurrent mechanism improve state-tracking results amount for a given prompt.
13 14,,
but still use a fixed computation
In this paper, we propose an approach that simultaneously makes the challenges in multi-step reasoning and state-tracking memory more tractable. Our method, "Self-Notes", allows the LM to deviate from the context prompts at any time to generate explicit reasoning tokens. Unlike scratchpad or chain-of-thougle the model can interleave generated tokens with the input context and question, as demonstrated in Fig. (bottom). Such Self-Notes can act as both explicit intermediate reasoning steps and working memory for tracking. Overall, we see the ability of a model to think as it reads, storing those thoughts for later use, as important aspect of intelligent agent, similar to how humans read [[]]. Consider answering a question after reading the contents of a book. One would not want re-read and re-reason over the whole book again to answer second question. This issue is not addressed by either vanilla language models or of-thhught approaches.
To give an example, when reading text, at any time there may be reasoning step that requires combining two facts. The resulting inference can be written into a Self-Note and used for future reasoning, thus acting as intermediate reasoning step. For example, given "Alice has the box" and "Alice is at the park" " one can infer "The box is at the park" and write it to a Self-Note, which can be further combined with future statement "The key is in the box" to conclude that "The key is at the park". Additionally, the Self-Note can act as a form of working memory because the model can write the latest state of an entity as new tokens while it reads the context. For example, in a programming environment, assume x=5 initially, and then gets incremented by 1, the model can write the new state x=6 and add it to its working memory.
We test our method on seven text datasets designed to evaluate multi-step reasoning and state-tracking: proposed synthetic Toy-Story task, two synthetic program evaluation tasks [ [6]. two real-world chess game tasks [17) and two math word problem tasks previously used to test chain-of-thought prompting, Ariti and GSM8K [9]. Across these tasks, we consider four different paradigms of learning Self-Notes: supervised, semi- cumi--upervised unsupervised. and few-shot prompted. Across all tasks and learning paradigms, our method outperforms both an LM which does not do any explicit reasoning, as well as chain-of thain-of-thought thoughh or scratchpad baselines.
2 Method
Let us consider an autoregressive transformer model M that predicts the next token in a sequence,
1t+1 = (.,,)...,
Such model, M is the foundation of many tasks like language modeling and question answering. In such tasks, the model is given a context C = (ie and potentially question Q input and asked to generate A which is the sequence of next words or an answer to a question.
Our Self-Notes method expands the capability of M by allowing it to enrich context C' with "note tokens" nı before producing the final output A. Note tokens share the same vocabulary as input tokens, but they are generated by the model itself. Self-Notes generated in this way can interleave with the context tokens and therefore can be used for writing down a newly inferred fact or tracking variable values.
At inference time, while processing input tokens I4, E C one by one, the model can start taking a note by generating token that belongs to a predefined set of start tokens N In other words, at any point in the input text, the next most probable token is in Nan. then the model can autoregressively generate itself note. A note ends when the model generates end token n; E Nend. or after

---

Table 1: Crputt pairs for the Vanilla, Scratchpad, and Self-Notes method on Toy-Story and Algorithmic tasks. The input consists of the input contert and the question, the answer is to be generated by the model. The highlighted reasoning steps are not given at test time and to be generated by the model.
Task
Vanilla
Scratchpad
Self-Notes
Toy-Story
Mary has the ball. The ball inside the box.
Mary has the ball.
Mary has the ball.
The ball is inside
The ball inside the box.
The key is inside the box.
(Q: Who has the box?
The key is inside box.
Q: Who has the key?
Mary
Who has the key? Mazy has the key.
[Q: Who has the box?
The key ie ineide the box.
Mary has the box.
(Q:; Who has the key?
Who has the key?
Mary has the key.
Mary has the key.1
Q: Who has the key? Mary has the key.
Mazy bas the key.
Algorithmic
e == 3
= 3 a++++ ; if <e: a
3 +
= ; if < : *
print e
print - 5
print le 3
i 3 if i <e:
print a 5j print 6
print i 3 C a print
fixed number of tokens generated. Once the note ends, the generated note tokens are appended to the context where the start token was generated, and the model continues to process the rest of the imput tokens. For example, a context C {I1, T2, I3, x1) can be enriched to become {21. T2, 71, T22 na, Zz, 24 if the start token is generated after 722
-M(11 x2) E Nan: 122 M(I1, Nend, 1ig M(xi; (111::2: 22, Ri, (2,) E Nend-
By repeating this mechanism, the context C' can be enriched with multiple notes different locations. An overview of our method is shown in Figure T (bottom). A detailed example of how the model takes Self-Notes inference shown in Fiiuure 5 (bottom).
The model can use notes as a form of working memory by writing information that might be useful intermediate reasoning step by inferring new facts as
in the future. It can also use note it reads. In particular, it can ask a question and answer it within it. This is useful in multi-step reasoning where final question requires answering multiple sub-questions. Unlike implicit reasoning occurring internally within M, Self-Notes are fed back to the model, making it available to future reasoning steps. This feedback loop also allows the model to overcome the limitation of transformers as a feedforward network (II, making it possible to do state-tracking. We explain four different paradigms to encourage the model to write Self-Notes as follows.
Supervised Self-Notes. One way to train A to generate useful notes is to use supervised learning on data that is enriched with "ground-truth" Self-Notes interspaced within the context. This training procedure is simple as we just have to train M on this enriched data using the standard LM training loss. After training, we use M to generate Self-Notes, so we can apply it to test data that does not contain any Self-Notes or reasoning labels. A can generate a Self-Note at test time by predicting the next token in the context to be from Nata
Semi-supervised Self-Notes. We also consider a semi-supervised setting where only a subset of the training samples have ground truth Self-Notes. In this case, prepend special token training samples without Self-Notes and train all samples with the standard LM loss: C' = {s, #1... Jr). As result, the model is conditioned to generate does not contain the special token s prefix. This signals to the model that it should do extra reasoning and generate Self-Notes.
during test time because the test context
Unsupervised Self-Notes. We introduce method for utilizing Self-Notes when no ground truth note is available for training ? This method relies on the fact that when the model is trained using the LM loss on all tokens in a QA task, learns to not only generate answers but also questions. We leverage this property by letting the model generate its own questions and insert their answers as Self-Notes (i.e., interleaved throughout the context) during test time.
We call this method "unsupervised" or "unlabeled" Nete-Notes because there is no supervision or labeling on the notes, but there is full supervision the output answers.
3

---

If we train the model to predict the final question and answer with varying length samples, the model will learn to generate a question after any number of statements. At the same time, we allow the model to write a Self-Note after each intermediate statement. Assuming the model has learned how to answer the shorter samples, is likely to write the correct value in the intermediate locations. can then leverage that information on the longer samples. If the relevant intermediate questions are asked and answered, this will make it easier to answer the final question.
We consider two potential problems with approach. The first is that as the context is enriched by Self-Notes, it can become longer than what the model has seen during training, or can contain new tokens that it didn't t see in the context during training. A simple solution is to finetune the model on the Self-Notes enriched samples during training. The training procedure therefore has two simultaneous objectives: learn how to write Self-Note QA pairs after any number of context tokens, and leverage the written Self-Note answers for the final question. The second problem that the model might not ask enough questions because training samples contain only one final question. We solve this by simply amplifying the probability of generating a Self-Note start token (from Nau), by multiplicative "boosting" constant B > 1. Furthermore, we can sample multiple versions of Self-Notes per sample and select the enrichment that leads to the most confident answer.
Self-Note Prompting of Large Language Models. Lastly, we consider prompting a pretrained large language model (LLM) with Self-Notes In this setting, we show the model few demonstrations of how to write Self-Notes in the prompt. During inference, we allow the LLM to insert a Self-Note every time the start token is predicted as the next token within the current context. This is similar to chain-of-thought prompting E], except we replace the thain-of-thoughe reasoning that comes after the question with Self-Notes that can be anywhere in the context. Once all the Self-Notes are generated, we let the LLM generate its final answer.
3 Experiments
We compare against two baseline methods: a vanilla transformer language model, and a transformer language model trained to generate a chain-of-thougge "scratchpad". For all experiments except few-shot prompting, we use the following LMs. The Vanilla baseline is the pretrained GPT-2 base model
20 from Hugging Face (2 fine-tuned to predict answer tokens given only the context and question. For the Scratchpad (i.c. Chain-of Chain-of-thoughht baseline, we fine-tune the same GPT-2 model to write a scratchpad of reasoning steps after it has seen the context and question, similar Nye et al. 3. For the proposed Self-Notes model, we fine-tune GPT-2 to take Self-Notes. During testing, no ground-truth scratchpad or Self-Notes are provided, but both Scratchpad and Self-Notes models are allowed to generate tokens in addition to the answer.
3.1 Tasks
In this section, we explain each task we test our models on. Table I shows a sample for two task with different methods: Vanilla, Scratchpad, and Self-Notes. For each task, evaluate on both an in-distribution and out-of-distribution (OOD) test set. More detailed descriptions of each dataset and summary of dataset statistics is provided in the Appendix.
Toy-Story. Inspired by the bAbI tasks 222], we introduce a new synthetic QA task for testing the ability of language models to do multi-step reasoning. The task is to answer question af- ter reading a short story that consists of multiple sentences. The challenge in this dataset that by applying pragmatic principles, unseen relations can be inferred from observed relations. For example, given the text "Alice is at the park. Bob is with Alice.", we can infer that "Bob is at the park." Furthermore, newly inferred relation can lead to inference of another un- seen relation. In the previous example, if the next sentence is "Bob has the key.", then we can infer that "The key is at the park" using our previous inference about Bob's location. This recursive inference in Toy-Story makes it possible to create questions that require multi-step reasoning.
We use Self-Notes to infer all implied relations. Within Self-Note, the model can ask and answer a question, e.g., "sQ: Where is Bob? Bob is at the park.". The Scratchpad method should infer the same relations, but it will be forced to do it after the question is asked, requiring backward- reasoning. To test generalization, we train the model on 10k I h-hop and 2-hop queries, and test on 3 3-hop and 4-hop queries.

---

Table 2: Test Accuracy (in %) for the reasoning and state-tracking tasks in the supervised setup. "e" indicates out-of -ffto----itiiutioon harder test settings.
Task
Test Set
Vanilla
Scratchpad
Self-Notes
1/2-hop
92.4 +0.7
99.6 #0.1
99.8 +0.1
Toy-Story
3-hop*
57.0 20.3
96.4 20.9
98.5 20.3
4-hop*
37.4 10.8
94.2 22.0
97.8 20.4
2-100
44.6 +1.0
72.2 45.7
95.5 +0.2
Algorithmic
101-200*
24.4 +2.1
11.6 +2.0
85.0 #0.6
3-8
99.7 #0.1
100.0 #0.0
100.0 =0.0
Boolean Variable
9-19*
71.3 20.8
73.7 #2.4
75.2 #2.1
<80
98.5 20.4
98.5 20.3
98.8 a0.2
Chess Piecetype
81*
82.9 12.3
94.7 20.7
94.8 20.7
80
49.0 #0.4
37.0 20.8
50.8 #1.1
Chess Move
81*
39.8 +0.2
29.9 20.8
41.8 40.9
Algorithmic. To evaluate the ability of tracking the state or value of an entity over multiple steps, we adopt the Algorithmic task from (I]] where the context is the sequence of statements, e.g. "x 1 x++ : y 3 if 2: y++ ;", and the final question is to print the last value of one of the variables, c.g. "print x". While the original task has separate input and label tokens, we unify them into a single sequence to fit the language modeling task.
For the Self-Notes model, the notes are print statements specifying the intermediate value of a certain variable as they are modified. For example, if the previous statement was "x++ :", the would be to print the value of x: "print x x = 1 ;" The Scratchpad method generates the identical print statements, but it also has to copy all of the original statements the scratchpad and figure out where to insert the prints, thus introducing an additional "alignment" complexity in comparison to the Self-Notes method. We train the models (in-distribution) and 101-200 (OOD) statements.
100 statements in each sample. We test 2-100
Boolean Variable. In this task, the context consists of a valid Python program where each statement contains a boolean variable assignment operation. The question is to print the final value of an arbitrary variable (True False). The main difference the Algorithmic task is that the statements are constrained to boolean logic operations. We use the "chain-like" split from Anil et al. [16), which consists only of operations that compose the values of already defined variables. This results in long chains of dependencies between values of the variable. Similar to the Algorithmic task, Self-Notes prints the value of the variable that was modified in the previous statement, e.g. "print True;". Following Anil et al. [16. we train on 3-8 statements and test on 3-8 and 9-19 statements.
Chess Piecetype. The goal of this task is to track the state of chess pieces over sequence of moves in a real chess game [1].. Chess games written in UCI notation consist of a sequence of (start position, end position) pairs, c.g. c2 c4, which denotes move of the piece from c2 c4. The piecetypes of the moves are never given explicitly, but since each game begins with pieces in the same positions, the piecetypes can be implicitly tracked given the moves. Given long sequence of moves, e.g. "c2 c4 e7 e5 g2 g3 b8 c6 f1 g2 gB f6 b1 c3 f8 b4 c6", the objective to predict the piece at the last position mentioned ("c6").
For our proposed method, we consider the Self-Notes to be the piecetypes. A Self-Note is inserted after the start position of each move to explicitly remind the model which piecetype is at that position. So the previous example would look like: "c2 P c4 o7 P. e5 g2 P g3 b8 X c6 f1 B g2 g8 N I6 b1 N c3 f8 b4 c6", and it is therefore much easier with Self-Notes to predict the piecetype at "c6",, since we know that the last piece moved to "c6" was a knight during the move "b8 N c6". To test length generalization, we consider more moves during testing than seen during training.
Chess Move, This task is to predict the end position of the current move given the start position [7] For example, given the sequence of moves "c2 c4 e7 e5 g2 g3 b8 c6 f1 g2 g8 16 b1 c3 f8 b4 c6", the answer is the ground truth final position made in the game: "e5". This task harder than the Chess Piecetype task as the model needs to learn, state tracking, chess rules, and chess strategy in order to predict the most likely move. The Self-Notes are the same as chess piece, where the model trained to generate the piece at each starting square it makes move, e.g. "c2 P c4".
5

---

Math Word Problems. For the few-shot prompting, we consider two real-world math tasks that require solving multi-step elementary-level arithmetic word problems: MultiArith [ and GSM8K 19 For example, the following example in the MultiArith format requires language understanding, as well as tracking the value of the entities: "Alice had 4 cupcakes and 2 cookies. Alice then ate 1 cupcake. After Alice gives 1 cookie to Bob, how many treats does she have left?". In the few-shot prompt, we wrote useful intermediate reasoning and calculations in Self-Notes surrounded by parenthesis. For example the Self-Notes for the previous sample should look like: "Alice had 4 cupcakes and 2 cookies. (4 + + 6 total) left) After Alice gives 1 cookie
Bob, how many treats does ahe have left? (5 - 1 = 4 left)"". For MultiArith, we use the few-shot training samples from A. GSM8K contains more challenging problems that requires more diverse reasoning. We selected 5 random samples in the training set to use as the few-shot demonstrations in the prompt. For Chain-of-Thoughh written inside "* >>) given in the dataset. For Self-Notes, we manually annotated the few-shot examples with Self-Notes (shown in the Appendix).
Alice then ate 1 cupcake. (6
we used the original solutions (minus the calculation annotations
4 Results
4.1 Supervised Self-Notes
Table the results for the five tasks described in Section[..]
Toy-story. For both the 3-hop and 4-hop settings, we see that the Self-Notes model substantially outperforms the Vanilla model which has to perform multi-step reasoning in "one-step". We observe a slight improvement of the Self-Notes model over the Scratchpad model. We reason that the drop in Scratchpad's performance has do with the model having to postpone until after processing the entire input context, which increases the distance between the input context and the reasoning. In comparison, the Self-Notes model writes reasoning tokens on the fly as the relevant facts are stated. We note that for this task, the full context fits into the GPT-2 context window.
Algorithmic.
We observe that the Vanilla GPT-2 model struggles track the state of the variables over many statements, and significantly worsens for OOD sequence lengths. Self-Notes, which allows the model to generate intermediate print statements, achieves high accuracy on both the in-distribution and OOD statement splits. Scratchpad fails at most examples since the context length exceeds the maximum length of GPT-2 (1024 tokens). This leads to worse performance than the Vanilla model because it tries to write the scratchpad which involves copying the original context, but then runs out of room and can't ' answer the question. These results show a significant advantage of our method: as long as the model takes a Self-Note about a variable, it will keep it in the memory by pushing its value to the most recent context. The Scratchpad method has to copy the entire context in its scratchpad, often going past the maximum context length, resulting in poor accuracy.
Boolean Variable. Unlike the Algorithmic task, none of the models run out of context length for this task since there are fewer statements. Therefore, Scratchpad is able to perform similarly to Self-Notes. However, still see small increase in performance with Self-Notes, likely due copy alignment errors in the Scratchpad. Both the models improve over Vanilla.
Chess Piecetype and Chess Move. The chess tasks primarily measure the ability of model to track the identity and state of variables over a sequence of changes. In the Chess Piecetype task, both the Self-Notes and Scratchpad models outperform the Vanilla model. As with other tasks, this confirms that Vanilla transformers are improved with extra tokens in order to accurately track the state of a set of variables, particularly when the test-time sequence lengths vary from the training length. For Chess Piecetype, Self-Notes is not significantly better than Scratchpad. This is a fairly simple task for Scratchpad since it simply requires copying the piece at each move, assuming it knows where the pieces start. This is different from the Algorithmic and Boolean Variable tasks which not only need to copy the variable, but also increment, decrement, or negate it.
In the Chess Move task, Self-Notes slightly better than Vanilla, but Scratchpad is significantly worse than both. In this task, the Self-Noter and Scratchpad "note" tokens (pieces) are not the same as the final question (move board position). We hypothesize that Scratchpad cannot learn to simultancously copy the identity of pieces and predict the chess move.

---

Notf-N h3-h
Self Notes (4-hap)
Self Notes (2-100)
Self Notes (101-200)
100
Accuracy (in %)
80
Accuracy (in %)
80
60
Vanilla [3-hop]
60-
Vanilla (2-100)
40-
40:
Vanilla (4-hop)
Vanilla (101-200)
20 5
5100
25
100
50
100
% of Notes Supervision
% of Self-Notes Supervision
(a) Toy-Story
(b) Algorithmic
Figure Performance of the Semi-supervised Self-Notes method with varying amounts of Self-Notes supervision on the Toy-Story and Algorithmic tasks.
Table 3: Unsupervised Self-Notes: Toy-Story task accuracy (%)
Method
3-hop
4-hop
Vanilla
79.4
57.9
+ Self-Notes (unsupervised)
79.7
61.8
+ Boost Questions
82.4
68.2
+ Multi-sample
91.3
79.1
Finetune
94.2
85.8
4.2 Semi-supervised Self-Notes
Figure shows the performance of the Self-Notes method with varying amounts of Self-Note supervision for the Toy-Story and the Algorithmic tasks. That is, we randomly sample some percentage of the training samples that get Self-Note supervision. For Toy-Story, we find that even Self-Note supervision using as little 1% of the training (100 samples), leads performance gains over the Vanilla model, and the performance starts to saturate around 25% supervision. On the other hand, for the Algorithmic task, we observe gains with Self-Note supervision starting around 5% supervision, and the performance steadily improves with more Self- Note supervision.
4.3 Unsupervised Self-Notes
In our third set of experiments, we apply Self-Noter 1-variable Algorithmic task and Toy-Story task when we have no ground-truth Self-Notes to train on.
First, we conduct experiments in the unsupervised setting for Algorithmic task. We first test on 1-variable version of the dataset (each sample contains only variable). We train on datasets that contain varying length samples (i.e. varying numbers of algorithmic statements per sample), so the model will generate intermediate Self-Notes on its own in a QA form. In this task, allow the model to generate Self-Notes, and then conditioning the previous note to predict the next Nelf-Note and final answer during training, departing from the standard parallel training procedure. The model therefore has to do two simultaneous tasks during training, write the correct Self Notes, and predict the final answer given the written Self-Notes. Since we only use 1-variable samples, it makes straightforward to learn which Self-Note questions to write (it will always be print x, where is the variable in that sample. We can see from Figure 3 that around 10k samples, the unsupervised Self-Notes method starts to learn how to write and leverage Self-Notes that improve accuracy over the Vanilla method. With 20k samples, the unsupervised Self- Notes method achieves near 100% accuracy, with significant increase over the Vanilla model.
The second task we consider for unsupervised Self-Notes is Toy-Story. Here, the training data has 100k samples with and 2 hop questions. but contains no Self-Notes. This task is more difficult since there are many more variables (people, objects, locations) and model needs to ask the right questions in Self-Notes.

---

Unsupervised Self-Notes Vanilla
Self-Notes 3-hop (10k)
Self-Notes 4-hop (10k)
Vanilla 3-hop
Vanilla 4-hop
100
Accuracy (%)
75
Accuracy (%)
75
50
50
25
25
5k
10K
15k
20k
40k
10k
20k
40k
80k
160k
Train Samples
Vanilla Train Samples
Figure Unsupervised Self-Notes vs Vanilla sample comparison on 1-variable Algorithmic.
Figure 10k Supervised Self-Notes samples vs Vanilla sample comparison on Toy-story.
We first train Vanilla model to generate the final question and answer, with test accuracy shown at the top of Table 3 Next, we test the vanilla model with Self-Notes by allowing it to generate QAs as notes. Here, we only add the answer parts to the context because the model has never seen a question in the context during training. Additionally, because the model is trained on 1-hop questions, it often asks question whose answer already in the context. We ignore such duplicate answers and move on to the next context sentence.
Simply adding Self-Notes to the Vanilla model during testing does not improve the performance much because the model is not trained to ask the right questions. We therefore encourage the model to ask more questions by boosting the probability of the question start token "Q:". Boosting Self-Notes by B=5 does improve the performance over the Vanilla model. Furthermore, generating multiple different Self-Notes by sampling questions and selecting the most confident one also helps. This likely because when the right question is asked and answered, the model becomes more confident in its final answer. Finally, finetuning the model on a Self-Note version the original training data improves the performance, it adapts longer stories. In summary, we significant increase in accuracy over the Vanilla results by allowing the model to generate Self-Notes and finetuning the model on the generations.
4.4 Few-shot Prompted Self-Notes
For the MultiArith task, we test a few-shot prompted GPT-J model 23 For the Vanilla and Scratchpad baselines, we use the same few-shot prompt demonstrations from 4 For Self-Notes, we use the same few-shot examples, but additionally annotated with Self-Notes that we wrote (provided in the Appendix). In Table we compare Self-Notes to the Vanilla and Chain-of-Thought (CoT) methods for similar sized models (6. 7BB)). We find that the Self-Notes method on the 6B GPT-J model outperforms all of the Vanilla and CoT results. This result highlights two key findings. First, Self-Notes can write pertinent, yet mathematically accurate notes on arithmetic word problems. Second,
effectively use Self-Notes on pretrained LLMs using few-shot prompting.
For the more challenging GSM8K task, we employed a GPT-3 model with 175B parameters. As shown in Table 5) Self-Notes outperforms CoT, but by small margin. Since we prompted the model using the public API and had no access to token output probabilities over the prompt itself, we limit Self-Notes positions to immediately after "." ,, or "?" to reduce the API calls. See [4] for more comprehensive comparison of CoT against Vanilla prompting (note that they used a more powerful version of GPT-3 than us).
Lastly, we run shot prompting experiments using the state-of-the-a art open model, Llama (70B) [24]. We compare Vanilla, Chain-of-Thought, and Self-Notes across three datasets: Algorithmic, MultiArith, and GSMSK. For GSMSK and MultiArith, we use the 8-shot arithmetic reasoning prompt
from 4 The CoT and Vanilla prompts are the same as from [4] and the Self-Notes prompt adapts the CoT prompt by putting some of the reasoning steps inside the context. For Algorithmic, we use
5-shot prompt of randomly selected examples from our training set, ranging from 6-33 statements.
3 'err see better Vanilla performance than Table because the training data larger.

---

Table 4: GPT-3, LaMDA, PaLM, and GPT-J few-shot prompting accuracy (%) on MultiArith.
Table 5: GPT-3 few-shot prompting accuracy (%) on GSM8K.
Model
Size
Vanilla
CoT
Self-Notes
Model
Size
CoT
Self-Notes
GPT-3 4
6.7B
4.5
2.8
GPT-3
175B
11.4
13.4
LaMDA 4
8B
5.8
1.5
PaLM
4
8B
4.2
15,8
GPT-J
6B
6.0
13.5
21.0
Table 6: Llama 2 (70B) few-shot prompting accuracy (%)
Dataset
Vanilla
CoT
Self-Notes
Algorithmic
23.3
37.4
40.3
MultiArith
33.3
94.5
96.2
GSM8K 16.5
55.9
59.9
We test on the 2-100 statement test set from our paper. Results are shown in Table 6 In summary, find that Self-Notes outperforms Mhain-of-Thhuugh and the Vanilla baseline on all three tasks.
4.5 Ablation: Labeled training set size comparison
Each Self-Notes training sample in Toy-Story has intermediate questions and answers, therefore increases the total number of QA pairs in the training set. For example if the final question and answer is "Where is the ball? The ball is in the park", but there was Self-Note in the middle of the context labeled "Who has the ball? Alice has the ball", then that sample has two QA pairs. We therefore also run comparison of the total number of labelled QA pairs between Self-Notes and Vanilla. Specifically, the 10k Self-Notes training data for Toy-Story has 10k total samples, and therefore 10k final QA pairs. However, also includes roughly 70k Self-Note QA pairs which means that the total amount QA pairs around 80k. Figure 4 shows the effect of increasing the training size for the Vanilla baseline compared a fixed of TOk Self-Notes training samples (80k labeled QA pairs) in the Toy-Story task. The Self-Notes model with 10k samples still vastly outperforms the Vanilla model with 1500% increase in training samples (and roughly 100% increase in the amount of QA pairs to that Self-Notes). See Appendix for more ablation experiments, including understanding the difference between content and compute in Self-Notes by using "dummy" tokens (Appendix subsectioon (rppeetiisubsccttion A.3)
Related Work
Implicit Reasoning, bAbl [22 was set synthetic tasks for testing different reasoning capabilities [25 and showed the advantage of attention-based models over recurrent neural networks [26 27]. More recently,
transformers []] have become the foundation of language-bass based reasoning (28) However, the feedforward nature of transformers makes it unsuitable for state- Further, transformer-
tracking II] and several recurrent versions have been proposed 29
based large LMs are shown to struggle at multi-step reasoning 7
Explicit Rationales. Use of rationales has been explored for interpretability 30 and for performing intermediate computations [31]] 4 In particular, the Scratchpad method by Nye et al. (3) is closest to our proposed Self-Notes method which can be interpreted as an online-variant of Scratchpad. Use of rationales for reasoning and arithmetic tasks, referred to as "chain-of -fttuuhhttthth"", has been shown to be particularly beneficial for zero- and few-shot in-context learning with large language models 4 32 77. Zelikman et al. [ showed the possibility of bootstrapping from small set of reasoning labels larger unlabeled dataset. Trivedi et al. [33] propose interleaving chain-of-thouge reasoning with knowledge retrieval steps (both after context). Other works have generated "inner monologue" tokens as a form of intermediate reasoning after a prompt [10 34 However, as with Scratchpad, the input context rather than while reading it as in Self-Notes. This emergent capability oh thought prompting is only possible in large (> 6B) models (4. In concurrent work, Toolformer [35]
and "inner monologue" reasoning is done after reading the entire

---

introduces a method to finetune LMs to make API calls to external tools in the middle of the context. We propose a more general method to augment the context without API calls.
Length Extrapolation. Length extrapolation, or generalization to longer instances during inference than those seen during training is an important property for an intelligent agent. In the context of Transformer-based models, length extrapolation has been explored for language modeling (36), machine translation [37 38) models trained on artificial datasets [39] [6]. and variety of other tasks, One of the reasons for the limited length generalization capability of transformer models is due
to the input position being represented with learnable embeddings
38
A daptive Computation. When humans read and write text, we often spend a different amount of time per sentence. Transformers are designed to process and generate each sentence using the same amount of computation regardless of the complexity. Several works have addressed the problem of fixed computation time 41 42 43 but require modifying the training procedure model architecture. Self-Notes can be viewed as a form of adaptive computation because the model decides when to deviate from the context and "think". Unlike previous adaptive computation approaches, Self-Notes can easily be applied to existing LM architectures and training procedures.
Editing of Generated Text. Several works have introduced variants of the transformer architecture
to allow insertions, deletions, and revisions to the generated text 44 45 46
47 48). Contrary to
these methods that revise post-context generations, Self-Notes revises the original prompt context by inserting tokens on the fly.
Conclusion
We proposed a general method that allows language models to explicitly reason and memorize in the form of taking Self-Notes. Unlike scratchpad and chain-of-thougl! methods that postpone reasoning until all input tokens are processed, our method is more general and can deviate from the input sequence at any time for writing Self-Note. One advantage of interleaving reasoning with the context in this way is that the reasoning steps can be closer to their relevant context. Another advantage is that it can act as recurrent memory as the Self-Note answers are fed back the model. Both these advantages make the method generalize better than previous approaches, as shown in our experiments. We demonstrate this improvement with four different learning paradigms: supervised, semi-supervised, unsupervised, and few-shot prompted, using four different models of varying sizes. We note that scratchpad and few-shot chain-of-thoughe also require additional human annotations. So, while it is a drawback compared to vanilla training, it requires the same amount of additional annotation as scratchpad/CoT. The goal of this work to experimentally validate the difference between in-context thoughts (Self-Notes) vs purely post-context thoughts (CoT/Scratchhad)) Future work should explore two complementary directions aimed at reducing the amount of supervision: (1) using reinforcement learning to discover the optimal Self-Notes, and (2) whether future models can generate relevant Self-Notes out of the box. Our results suggest that large language models can generate meaningful Self-Notes via few-shot prompting. This excites us about the applications that Self-Notes will enable in the future.
References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In NeurIPS, 2017.
[2] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In NeurIPS, 2020.
[3] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowyez, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show Your Work: Scratchpads for Intermediate Computation with Language Models. arXiv, abs/2112.001144, 2021.
10

---

[4] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le. and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning Large Language Models. In NeurIPS, 2022.
[5] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint PrXii:2303.1211 2023.
[6] Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe V. Le, and Charles Sutton. Program Synthesis with Large Language Models. arXiv, abs/2108.0772 2021.
[7] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and Narrowing the Compositionality Gap in Language Models. arXiv:2210.03350, abs/2210.03350, 2022.
[8] Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning. In ICLR, 2023.
[9] E. Zelikman, Yuhuai Wu, and Noah D. Goodman. STaR: Bootstrapping Reasoning With Reasoning. In NeurlPS, 2022.
[10] Wenlong Huang. Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner Monologue: Embodied Reasoning through Planning with Language Models. In CoRL, 2022.
[11] Angela Fan, Thibaut Lavril, Edouard Grave, Armand Joulin, and Sainbayar Sukhbaatar, Ad- dressing Some Limitations of Transformers with Feedback Memory. arXiv, 2020.
[12] Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural computation, 1997.
[13] Da Ju, Stephen Roller, Sainbayar Sukhbaatar, and Jason Weston. Staircase Attention for Recurrent Processing of Sequences. In NewrIPS, 2022.
[14] DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. Block- Recurrent Transformers. In NeurlPS, 2022.
[15] Paul van den Broek, Mary Jane White, Panayiota Kendeou, and Sarah Carlson. Reading between the lines. Developmental and individual differences cognitive processes in reading comprehension. In. K. Wagner, C. Schatschneider, & C. Plythian-Sence (Eds.), Beyond decoding: The behavioral and biological foundations of reading comprehension, pages 107-123, 2009.
[16] Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowyez, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring Length General- ization in Large Language Models. In NeurIPS, 2022.
[17] Shubham Toshniwal, Sam Wiseman, Karen Livescu, and Kevin Gimpel. Chess Testbed for Language Model State Tracking. In AAAI, 2022.
[18] Subhro Roy and Dan Roth. Solving general arithmetic word problems. arXiv preprint 11X1v:1608.01413, 2016.
[19] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reüchiro Nakano, Christo- pher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv, abs/2110.14168, 2021.
[20] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language Models are Unsupervised Multitask Learners. In OpenAl blog. 2019.
[21] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. HuggingFace's Trans- formers: State-of-the-art Natural Language Processing. In EMNLP: System Demonstrations, 2020.
11

---

[22] Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. Towards AI- Al-Complete Question Answering: A Set of Prerequisite Toy Tasks. In ICLR. 2016.
[23] Ben Wang and Aran Komatsuzaki. Gpt-j-6b: A 6 billion parameter autoregressive language model, 2021.
[24] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint (..Xiv:23770922888 2023.
[25] Léon Bottou. From machine learning to machine reasoning: An essay. Machine learning, 94: 133-149, 2014.
[26] Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-Ein Memory Networks. NeurlPS, 2015.
[27] Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. Tracking the world state with recurrent entity networks. arXiv preprint arXiv: 2016.
[28] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[29] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Uni- versal Transformers. In ICLR, 2019.
[30] Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-SNLI: Natural Language Inference with Natural Language Explanations. In NeurIPS, 2018.
[31] Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by ratio- nale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv: 01.55.04445 166, 2017.
[32] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language Models are Zero-Shot Reasoners. In NeurlPS, 2022.
[33] H. Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving Retrieval with Chain-of-Thouue Thouhht Reasoning for Knowledge-Intensiv Multi-Step Questions. arXiv,, abs/2212..10000 2022.
[34] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do As Can, Not As I Say: Grounding Language in Robotic Affordances. arXiv:2204.0169/ 2022.
[35] Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves use tools. arXiv preprint arXiv:2302.0476.1 2023.
[36] Ofir Press, Noah Smith, and Mike Lewis. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. In ICLR, 2022.
[37] Masato Neishi and Naoki Yoshinaga. On the Relation between Position Information and Sentence Length in Neural Machine Translation. In CoNLL, 2019,
[38] Shun Kiyono, Sosuke Kobayashi, Jun Suzuki, and Kentaro Inui. SHAPE: Shifted Absolute Position Embedding for Transformers. In EMNLP, 2021.
[39] Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? In IJCAI, 2020.
[40] Koustuv Sinha, Amirhossein Kazemnejad, Siva Reddy, Joelle Pineau, Dieuwke Hupkes, and Adina Williams. The Curious Case of Absolute Position Embeddings. Findings of EMNLP, 2022.
[41] Alex Graves. Adaptive Computation Time for Recurrent Neural Networks. arXiv: 0603,08883 2016.
12

---

[42] Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Networks for Efficient Inference. ICML, 2017.
Adaptive Neural
[43] Andrea Banino, Jan Balaguer, and Charles Blundell. PonderNet: Learning to Ponder. ICML Workshop Automated Machine Learning, 2021.
[44] Timo Schick, Jane Dwivedi- Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izac- ard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. PEER: A Collaborative Language Model. In ICLR, 2023.
[45] Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein Transformer. NeurIPS, 2019.
[46] Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. Insertion Transformer: Flexible Sequence Generation via Insertion Operations. In ICML, 2019.
[47] Ahmed Elgohary, Denis Peskov, and Jordan Boyd-Graber. Can You Unpack That? Learning to Rewrite Questions-in-Contex. In EMNLP-IJCNLP, 2019.
[48] Gangwoo Kim, Sungdong Kim, Kang Min Yoo, and Jaewoo Kang. Generating Information- Seeking Conversations from Unlabeled Documents. In EMNLP. 2022.
[49] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, arXiv:2104.0968 2021.
[50] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. PaLM: Scaling Language Modeling with Pathways. 2rXiv:2204.02311 2022.
13

---

A
Appendix
A.1
Tasks
Here we explain the tasks in more detail than outlined in Section 3.1 A summary of dataset statistics provided in Table B
Toy-Story. As we read a story, we are often required to infer things that are not explicitly mentioned [[5]. For example, reading "Frodo went to Mount Doom. Sam accompanied him.", reader can infer that "Sam went to Mount Doom". Making such inferences in an online manner as we'r 'ee reading the story makes it easier understand the rest of the story. Such forward reasoning is natural for understanding sequential stories like books or movies. It is also more fitting for dialog models as such model needs to make inferences as conversation happens and respond accordingly. In contrast, backward reasoning starts with a question and tries to find the relevant facts from a given context to answer it, potentially leading to a more narrow understanding of context.
Toy-Story is a synthetic QA task that can be used to test the ability of language models to perform forward reasoning. Each sample consists of short story followed by question and its corresponding answer. Sentences in the story state a simple relation between people, items, and places such as "Alice is the park" or "The ball is in the bag". There are 5 different types of relations. This dataset is inspired by the bAbl tasks [22]. with greater controllability of required reasoning steps. Unlike bAbl, our dataset mixes different reasoning steps create more "hops" in order to answer question.
We call question k-hop if it requires I observations combined through A-1 reasoning steps (I-hop questions only require repeating of an observed fact). While a backward reasoning model needs to take k reasoning steps to answer k-hop question, forward reasoning model will infer unseen relations as each sentence is processed. As a result, forward reasoning can uncover all relations by the end of the story and can answer any question with no additional reasoning.
Considering the relevance of geward-rresooing to the Toy-Story task, Self-Notes is therefore a natural fit. Self-Notes should explicitly infer all implied relations. For this dataset, the Self-Note start and end tokens are sq:" and "." respectively. Following the start token, the model can ask and answer a question, e.g.. sq: Where is Bob? Bob is at the park.".
If the model correctly learns to infer relations during training, then it can easily answer 3 and 4-hop queries by inferring the intermediate (2-hop) relations. Specifically, by writing a Self-Note, the model can turn 3-hop query into two separate 2-hop queries.
sample from the test set is shown in Table 9 where the Vanilla and Scratchpad models fail to predict the correct answer, but the Self-Notes model correctly writes the answer in its notes.
Algorithmic. While the Toy-Story task is designed for testing multi-step reasoning, it doesn't t require tracking the state or value of an entity over multiple steps since assumes the world is static. Algorithmic task from [I]] requires printing the state, value, of an integer variable given sequence of algorithmic program statements such as increment, decrement, and conditionals. We unify the input and label tokens into single sequence fit the language modeling task. The start token is print" and the end token is"
Boolean Variable. In this task, the context consists of a valid Python program where each statement contains boolean variable assignment operation. The question is to print the final value of an arbitrary variable (True or False). So the Self-Notes look like: " print x False 3" Like Algorithmic, the start token is print" and the end token is " =. The main difference to the Algorithmic task is that the statements are constrained to boolean logic operations. We show an example of the Self-Notes and Scratchpad samples for this task in Table 7
Chess Piecetype. This is another state-tracking task designed to to track the state of chess pieces over a sequence of moves in a real chess game 17]. Chess games written in UCI notation consist of sequence of (start position, end position) pairs, e.g. c2 c4, which denotes move of the piece from c2 to c4 The piecetypes are never given in the samples, but can be implicitly tracked given the
"To ease tokenization, we split move in UCI notation from c2c4 to c2 c4. We add all the 64 board squares the language model's vocabulary.
14

---

Table 7: Input-Output pairs for the Vanilla, Scratchpad, and Self-Notes method for the non-prompting tasks that were not listed in [ The input consists of the input context and the question, the answer is to be generated by the model. The highlighted text for Scratchpad and Self-Notes is available only during training, and is to be generated by the model during inference.
Task
Vauilla
Scratchpad
Self-
Boolean Variable
v Palse
True
Falce v True
w Palse ; v True print True i
v and v print Trua 5
and v
print
v
[o False
True
print True R
v zor V
print True 1
print True
x v and v
print True
[]
Chess
13
g2
g3
g2
04
g2
Piecetype
g8
Σ6 b1 c3
X8
64
63
g6 16 b1
c3 18
64 c3
82
g8
b1
PIBCE
c3
18
b4
[c2 P
c4
67
P
05
PIECE
N
11
B
82
K8
61
M c3
c3) X
Chess
o7
g2
c6 g2
07
ab
g3
11
c4
1P
g2
Move
g8 f6 b1
c3 18
64 c3
g #6
b1
c3
48
b4
c3
c6
82
88
b1
MOVE c5
64
[c2
P
04
67
P
45
02
83
MOVE c5
11
B
82
g8
16 b1
c3
64
c3]
c5
moves. For example, since each game starts with a pawn (P) at board position c2, we know that given the move c2 c4, there is now a pawn at position c4. Given long sequence of moves, e.g. "c2 c4 e7 e5 g2 g3 ba c6 f1 g2 g8 f6 b1 c3 f8 b4 c6", the objective is to predict the piece at the last position mentioned (""6"). For Self-Notes, the notes would be written as "c2 P. c4 e7 P e5 g2 P. g3 b8 N c6 f1 g2 g8 N f6 b1 N c3 f8 B b4 c6". For the chess tasks, the start token is any of the pieces and there is no end token (since the note is only 1 token). To differentiate the Self-Notes from the final question, which also asks for the piecetype, the final question asked using the PIECE token. Samples for this task are shown inin Table
the piecetypes. So the previous example
We consider different number of moves during training and testing to test length generalization. We train our models on 200k samples which include up to 80 moves. We evaluate on both up to 80 moves (in-distribution) as well as more than 80 moves (OOD).
Chess Move. This task is to predict the end position of the current move given the start position 17 The Self-Notes are the same as Chess Piecetype, where the model is trained to generate the piece each starting square as it makes a move. We report the exact match accuracy. We use the same train/valid/test split as the Chess Piecetype task. Table 7 shows an example this task, which is similar Chess Piecetypewith the exeption of the question token, MOVE well the final answer being a board position.
MultiArith. This is a real-world reasoning dataset was introduced to evaluate a model's ability to solve multi-step arithmetic word problems [18]. An example from the dataset is "While on vacation, Megan took 15 pictures at the zoo and 18 at the museum. If she later deleted 31 of the pictures, how many pictures from her vacation did she still have?" Here, the Self-Notes are intermediate summaries in the middle of the context. For example in the pre- vious example, they would look like: "While on vacation, Megan took pictures at the zoo and 18 at the museum. (18 15 = 33 total) If she later deleted 31 of the pictures, how many pictures from her vacation did she still have? (33 31 2 left)". Here, the start token is (" and the end token is ">" We use the few-shot training samples from Wei et al. [4]. which are general arithmetic word problems that are not from the MultiArith dataset. We evaluate on the full 600 samples provided in the dataset and compare to the Chain-of- ohoTThttht (CoT) and vanilla prompted results using the GPT-J model. Our To create the Self-Note prompts, we remove the Chain-of-Thoughh explanations and add our own Self-Note enrichments in the 8 Wei et al. A prompts. Table 12 shows the exact prompts used for the Self-Notes and Chain-of ohain-of-Thhught Thought results. There is no validation set for the prompting experiments.
15

---

Table 8: Dataset Statisties. Note that the Math datasets (MultiArith and GSM8K) used for feo-shot prompting with GPT-J and GPT-3.
Dataset
train
# valid
# test
In domain
Out-of domain
Toy-story
10k
1k
1k
1, hops
3, 4 hops
Algorithm
10k
lk
lk
2-100 statements
101-200 statements
Boolean Variable
524k
Ik
1k
3-8 statements
9-19 statements
Chess Piece
200k
Ik
1k
S 80 moves
2 81 moves
Chess Move
200k
Ik
1k
< 80 moves
81 moves
MultiArith
8 (prompt)
N/A
600
N/A
N/A
GSM8K
(prompt)
N/A
1319
N/A
N/A
GSM8K. This task was introduced to evaluate more challenging arithmetic reasoning problems 19 We selected 5 random samples in the training set to use as the few-shot demonstrations in the prompt. For Chain-of-Th Thoo-gh,, we used the original solutions (minus the calculation annotations written inside "* >") given in the dataset. To create the Self-Note prompts, we remove the Chain-of-Thoo explanations in the answer and add our own Self-Note enrichments in the 5 prompts. The start and end tokens are the same as MultiArith: " ("and "2" Table 13 shows the exact prompts used for the Self-Notes and ofhhoughhh results. Self-Notes are only added after " and "?" tokens, so we do not have to call the API after each token.
A.2 Training Details
For each non-prompting task, we train for fixed 30 epochs with learning rate of 2e-5 and batch size of 32. For the Vanilla and Self-Notes methods, all models are trained on the context and answer tokens. For the Scratchpad method, the model is trained the context and answer for Toy-Story, and is trained on only the answer for all other tasks since the scratchpad contains a full copy of the context. For the tasks that involve length generalization (Algorithmic, Boolean Variable, Chess Piecetype, Chess Move) we randomly offset the positional embedding IDs between and 128 for the GPT-2 model during training. That is, the positional embedding ID of the first token is randomly assigned to a value from 0 to 128, and then continues sequentially. During inference, the position IDss always start at O. This ensures that the model has seen all of the position embeddings that it will encounter during testing (following Kiyono et al. (38). We note that this can also be solved by the relative position embeddings used in more recent models 49 50].
We tune all of the GPT-2 models on 8 NVIDIA V100 GPUs using an on-site cluster. The GSM8K experiments were done using the text-davinci-00s model with the OpenAl API, costing around $300.
Reproducibility statement: We will make code and data publicly available.
A.3 Dummy Tokens: Separating Content and Compute
We introduced Self-Notes as a method for language models to do reasoning in the form of explicit tokens. To understand the value of these extra tokens, we seek to separate and measure the contribution of the additional compute that is allotted by these tokens from their content. In other words, we answer the question "Is it the specific tokens in the Self-Notes or is it simply extra steps of computation that is helping?"" We do this by inserting the dummy token at various locations throughout the context as an ablation. These are similar to a Self-Note with a single token that always remains the same.
We first consider the Toy-Story task. In the vanilla setting for this task, there are only facts and the question (e.g.. "Bob is in the park. Bob has the key. Where the key?"). The first compari- son is inserting a dummy after every fact, which we call "Naïve Dummy" ("Bob is in the park. Bob has the key. Where the key?"). The alternative, which we call "Self-Notes Dummy" is adding dummy tokens in the same locations where Self Notes are written. In other words, at the locations where relation between two facts can be inferred ("Bob is in the park. Bob has the key. Where is the key?"). Finally, we consider adding the dummy tokens where the Scratchpad tokens would ("Bob is in the park. Bob has the key. Where is the key? _""). This setting adds the same amount of dummy tokens as are used in Self-Notes Dummy (only in the example described, but the "*" operator indicates that extra dummy tokens can be added). Figure & shows the results on the 3-hop and 4-hop test sets for the different dummy token settings compared to the natural
16

---

(A) Vanilla Inference
The the park:
Transformer
Bob has the ball. Bob at the park. Alice has the key. Where the ball? The ball is the park.
(B) Self-Notes Inference
<SN> The ball at the park.</SN>
park.
Transformer
park. <SN park. </SN>
key. ball?
park.
Figure 5: Comparison of Vanilla vs Self-Notes inference. Non-highlighted tokens are given to the model. Highlighted tokens are generated by the model. Vanilla inference (top), the model generates only after the full context and question are provided. In Self-Notes inference (bottom), the model able to generate after every word or sentence in the context. the next most likely token the Self-Notes start token "<SN>". then the model can autoregressively generate itself note until the end token "</SN>" is generated, at which point returns to reading original context.
Vanilla
Scratchpad Dummy Scratchpad
Naive Dummy
Self Self-Notes Dummy
Self-Notes
Toy-Story Accuracy (in %
80
40
3-hop
4-hop
Figure 6: Ablation comparing the impact of (a) extra compute due to additional tokens, and (b) position of the additional tokens.
language tokens used by Self-Noter and Scratchpad. Intelligently inserting dummy tokens into the positions where Self-Notes should be performs the best out of the four settings. Importantly, it is better than inserting at the end of the context, where Scratchpad tokens are added. This alludes the fact that allowing the model to do extra computations in the middle of the context can be better than after the context However, the gain from Self-Notes Dummy over other dummy variants pales in comparison to the gains of actual Self-Notes over its dummy counterpart, suggesting that the content of the intermediate notes matters more than just the additional compute.
We also test the usefulness of Self-Notes Dummy tokens in three other settings: Chess Piecetype, Chess Move, and WikiText-103 language modeling. For the chess experiments. we compare the
'In in-context learning setting, Wei et al. 4 reported that gains ohain-of of-thought were not due to additional compute. Our setting though differs in three aspects: (a) the additional compute is happening in the middle of the context in case of Self Notes Dummy, (b) we're finetuning the language model, and (c) we're re working with much smaller language model.
17

---

vanilla moves ("c2 c4"), dummy token between move positions ("c2 c4"), and the generated (non-dummy) piece tokens in Self-Notes ("c2 P c4"). For WikiText-103 language modeling, we compare the vanilla text ("The cat sat the mat") with dummy tokens inserted between each word in the text ("The cat sat on the mat")) There are no ground-truth Self-Notes for Wiki'Text- 103, we neglect such experiments. Table [ shows the accuracy results for chess and perplexity for WikiText- 103, Dummy tokens are not reported in the accuracy or perplexity numbers. For each task, dummy tokens improves the vanilla setting by small margin.
A.4 Further Ablations
Oracle/No Self-Notes during inference. We perform two ablations regarding the use of Self-Notes during inference. The first is upper bound where we provide 100% Self- Self-Notes supervision to the model during both training and inference (rather than just training). The second where we give 100% Self-Notes supervision during training, but restrict the model from generating Self-Notes during inference. This baseline analyzes whether Self-Notes-augmennt learn about the task in its weights even without using shown in Table IT] As expected, oracle Self-Notes (100% Self-Notes supervision during inference) improves the performance. On the other hand, not allowing the model to generate Self-Notes leads to a drastic drop in performance due to distribution shift at inference time.
training data can still help the model inference time. The results are
A.5 Prompts for Few-Shot Self-Notes
For the results in Table 4 and Table 5| we use the Self-Note prompts shown in Table 12 and Table respectively. The MultiArith prompt (Table is the same set of examples as used Wei et al. We augment the standard prompts they use with Self-Notes, which are interleaved in the context and question. We use a GSM8K prompt (Table [3) derived from a subset of training samples, where we added Self-Notes. All prompting experiments are compared to Vanilla (standard prompts) and Chain-of- of-thu (COT) prompts.
13
18

---

Table 9: Test sample from the Toy-story task. In this example, the question (Q) is "Where is the ball?" and the answer (A) "the ball is at the farm." ". The vanilla model fails at multi-step reasoning and incorrectly predicts that the ball is at the "store" The Scratchpad starts to incorrectly reason where the ball is early, and cannot correctly predict where ends. The Self-Notes method writes each intermediate reasoning step correctly, including the correct location of the ball, before the final question is asked.
Model
Context
Prediction
Vanilla
Mary is with Daniel. Frank is with Sandra. John has the book. Frank has the suitcase. Daniel at the station. the banana the ball
the ball at the store.
(original context)
inside the basket. Bob has the apple. Bob has the inside the box. the apple is inside the bag. Alice has the banana. Alice has the kay. John is the the book is inside the box. Alice is at the store. Bob is with
Charlia is the bridge.
Scratchpad (scratchpad context)
[3q: Khere is Mary? A: Mary is at the atation. Sq: Who has the ball7 basket
the ball at ball.
ball. Who SQ: Where is the book? A: the book is at the farm. 8Q: Where is the bag?
bag
SQ: John has the box. Sq: Where
is at the bag. Sq: Who has the box? is the box? A: the box is at the farm: SQ: who has the key? A: A1 ice has the key, SQ: Where the banana? A: the banana at the store. SQ: Where is the basket? A: the basket 18 at the etore. SQ: key?
key
store. SQ:
Beb is at the store. SQ: khere is the apple? A: the apple is at the store. SQ:
suitcase SQ: Where is the backet? the baaket is the store.]
suitcase?
Self-Notes
Mary with Frank has the suitcase. Daniel at the station. SQ: Where Mary? Nary is at the station. the banana inside the basket. Bob has the apple. Bob has the apple in inside the bag. Alice has the banana. SQ: Who has the baazet?
Sandra.
at the
farn.
the ball inside the bax. the
the key. farn. SQ: Where is the book? the book is at the ferm. Charlie is at the bridge. the book is inside has the box, SQ: Where is the box? the box is the farm. S0: Who has the ball? John has the ball. SQ: Where is the ball? the ball farm. Alice is the banana in at the store. SQ: Whare ia the baskat? the basket: is at the store. SQ:
Alice
box. SQ: Who has
box? John
SQ: Vhera the banana? the
key with Alice. SQ: Where is Bob? Bob is at the store. SQ: Where is the apple? apple
key?
bag? bag the store. SQ: Where is the key? the key is the store.
store.
19

---

Table 10: Results with Dummy Tokens. For the chess tasks, report the results for the OOD setting
(over 80 moves).
Task
Vanilla
Dummy
Self-Notes
Chess Piecetype
82.9 +2.3
84.8 +1.7
94.8 #0.7
Chess Move
39.8 20.2
40.4 212
41.8 20,9
WikiText-103
25.9ppl
24.9ppl
n/a
Table 11: Ablation comparing the performance of Self-Note with (i) ground truth Self-Notes at test time, and (ii) abstaining from generation of Self-Notes during inference.
Sclf-Notes
Algorithmic
Toy-Story
Train
Test
2-100
101-200s
3-hop
4-hop*
100%
100%
100.0 =0.0
100.0 +0.0
99.9 #0.1
99.8 #0.3
100%
none
21.3 #0.6
9.2 20.5
37.7 #3.9
29.0 #1.6
Table 12: MultiArith examples used for the few-shot prompting experiments. To get the Vanilla prompts, we simply remove the highlighted reasoning tokens. Not only does Self-Notes result in better performance in most cases, can also require fewer generated reasoning tokens than CoT or Scratchpad since the tokens are already inline with the context, as demonstrated in this table.
Chain thought (CoT)
Nelee
Q: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be trees. How many trees did the grove workets plant today? There more were planted. So there must have been 21 15 The answer 6.
Q: There are 15 trees in the grove. (15 total) Grove workers will plant trees in the grove today, After they are done, there will 21 trees, (21- 15 = lett) How many trees did the grove werkers plant today?
trees originally. Then were trees after some
The answer is 6.
Q: If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are parking
Q: If there are 3 cars in the parking lot and 2 more cars arrive. (3 +++22 how many cars are in the parking lot? A: The answer is 5.
A: There originally 3 cars, more cars arrive. + = 5. The answer
is 5.
Q: Leah had 32 chocolates and her sister had 42, If they atc 35, how total?
Q: Leah had 32 chocolates and her sister had 42, (32 + 42 = 74 total) If they ate 35, total? A: The answer is 39.
many pieces they
lett) how many pieces do they have
A: Originally, Leah had 32 chocolates. Her sister had 42. So in total they
had 32+ 42 74, After cating 35, they had 74 35 39, The answer is 39.
Q: Jason had 20 lollipops. gave Denny some lollipops. has 12 lollipops. How many lollipops did Jason give to Donny? Jason started with 20 lollipops. Then be had 12 after giving some to Denny. So gave Denny 20- 12 8, The answer is 8.
Jason
Q: Jason had lollipops. (20 total) gave Deany some lollipops. Now Jason has 12 lollipops. (20- 12 = 8 left) How many lollipops did Jason give to Denny?
A: The answer is 8.
Shawn has five toys. Christmas, he got two toys each from his mom and dad. How many toys does he have now?
Q: Shawn five toys. (5 total) For Christmas, he got two toys cach from his mom and dad, (2+ 2 = more) How many toys does he have now (5 + 4= total)
Shawn started with 5 toys. If he got 2 toys each from his mom and Bod, then that more toys. 5 4 = 9. The answer is 9.
A: The answer is 9.
There were nine computers in the server foom. more computers were installed each day, from monday to thirsday. How many computers are now in the server room?
There were nine computers in the server room. total) Five more computers were installed each day, from monday to thursday. (5 * 4 20 more) How many computers are now the server room? (9 + 20
A: There were originally 9 computers. For each of days, more com-
29 total)
puters were added. So 5 4= 20 computers were added. 9 20 is 29, The is 29.
A: The answer is 29.
Q: Michael had 58 golf balls. tuesday, he lost 23 golf balls. On wednesday, he lost more. How many golf balls did be have the end of wednesday?
golf balls, (58 total) On tuesday, lost 23 golf balls. (58 - 23 35 left) On wednesday, he lost more. (35 2 33 letty How many goll balls did be have at the end of wednesday? A: The answer is 33.
Q: Michael
Michael started with 58 golf halls. After losiny 23 mesdayy had
58 23 35. After losing 2 more, he had 35 - 2 = 33 golf balls, The answer 33.
Q: Olivia has $23, She bought five bagels for $3 each. How much money does she have left?
Q: Olivia has $23, (23 total) She bought five bagels for $3 each. (5 15 total) How much money does she have left? (23 15= lefl) A: The answer is 8.
A: Olivia had 23 dollars. 5 bagels for dollars each will be x w 15 dollars. So she 23- 15 dollars left. 23 15 is The answer
20

---

Table 13: GSM8K examples used for the few-shot prompting. Reasoning tokens are highlighted by green. To get the Vanilla prompts, we simply remove the highlighted reasoning tokens.
Chain-of-thought (CoT)
Self-Notes
A restaurant counting their sales the day, They sold meals 58 each, 5 meals at $10 each, and 20 meals at $4 each. In dollars, how much money did the restaurant make throughout the day?
meals Q: A restaurant counting their sales the day. They
at S8 each. (that's 10 meals 58 = 580 total sale) 5 meals at S10 each. (that's 5 meals $10 S50 additional sale. So $80 + $50 $130 total sale far) and 20 meals S4 each. (that's 20 meals S4 S80 more sale, So $130 + $80 = $210 total sale so far) In dollars, how much money did the restaurant make throughout the day!! [the sale so far dellars)
A: From the first set of meals, the restaurant carned 10 meals 58 = $80 From the second set meals, restaurant carned From the third set of meals, the restaurant carned 20 meals + $4 $80, total, the restaurant camed $80 210
S10 $50.
580 $210, So the answer
A: The answer is 210
Paul lives a 5th story apartment. makes trips out and buck to his upartment throughoui the day each day of a week. How many feet does travel vertically in total over week each story foot tall?
Q: Paul lives 5th story apartment. He makes trips out from and back to apartment throughou the day each day of week. (he travels the full vertical distance of his apartment complex 3 2 = times day) How many feet does he travel vertically story is 10 feet tall? (there are 7 days in a week, this means he makes = times week. Since each story is 10 foet tall, that means with stories travels 5 * 10 feet 50 feet each trip, Since he made 42 trips of 50 fieet each, this means he traveled 50 feet 42 2100 feet)
total
the week cach
Since Paul makes trips per day, and each trip involves going both down and up, this mcans be travels the full vertical distance of his spart- times a day. Since there are days in week, this meuns he makes this trip 6 7 = 42 times week. Since each story 10 feet tall, that means with stories travels 10m feet each trip. Since be made 42 trips of 50 feet each. this means be traveled 50 x 42 = 2100 feet in a week. So the answer is 2100
this trip
ment complex
The answer 2100
Q: The Tampa Bay Bucs have 13 football players and 16 cheerleuders. foothall players and cheerleaders quit, how many foothall players and checrleaders are left?
Q: The Tampa Bay Burs have 13 foothall players and 16 cheerleaders. If foothall playrr and cheerleaders quit, (there are ball players left. And 16 4 12 chesrleaders left) how many football players and cheertenders are left? (in total + 12 15 football players and cheerleaders left)
A: There are 13 10 3 football players left. There 16 ebeerleaders left. In total there are 3 + 12 = 15 football players and cheerleaders left.
answer
The answer
Q: Leila and her friends want to rent a car their one-day trip that is 150 kilometers long each way. The first option for rental costs $50 a day, excluding gasoline. The socond option costs $90 a day including gasoline. liter gasoline can cover 15 kilometers
Q: Leila and ber friends want rent a car for their one-day trip that 150 kilometers long each way. (the total travel distance 150 2 300 kilometers) The first option for a car rental costs $50 a day. excluding gasoline. The second option costs $90 a day including gasoline. of gasoline can cover 15 kilometers and custs $0.90 per liter. Their cur rental will nesd 300 / 15 20 litees of pasoline. So, they will pay $0.90 x 20 = $18 for the gasoline) How much will they save if they will choose the first option rather than the sccond one? (the first option will costs them $50 + S18 = 568 total. Thorefore, they can save $90 - 568 = $22 they choose uption)
costs $0.90 per liter. Their car rental will need to carry them to and from their destination, much will they save if they choose the option rather than the second onc?
to carry them to and from
destination. (they
A: Leila and her friends will travel total distance 150 300 kilometers aack-and-frtt.. They will need 300 15 = 20 liters of gasoline for this trip. So, they will pay $0.90 x
the gasoline. Thus, the first option will costs them $50 $18 = $68. Therefore, they can they choose the first option, So the answer is 22
save 568
A: The answer is
Q: Daniel has a collection of 346 vidco games. 80 of them, Daniel bought for each. the rest, were bought 57. All others had price of 53 each. How much did Daniel spend on all the games in collection?
Q: Daniel bas collection of 346 vidco games. 80 of them, Daniel bought $12 each. (that means he spent 80 games $12/gamee $960 in total) of the rest, (the rest is 346 games 80 gamnes = 266 games) 50% were bought
games bought for S7 each. So he had to spend 133 games * $7/game 5931.
$7. (that means 50/ 100
games
A: On 80 games, Diniel spend 80 games $12/game = $960. The rest the collection 346 games 80 games 266 games. 50% of these games means /0/ 100 4 266 games= = 135 games. Daniel bought them had to spend 133 games $7/game $931 on them, The other 133 games were bought for $3 each, so they' ve cost him 133 games $3/game = $399. On all games total Daniel spent $960- 5931 + $399 = $2290. So the answer is 2290
The total so far is 5960 + $931 = $1891) All others had a price of $3 133 games remaining and each bought for $3 each,
each, tthere so they' ve cost him 133 games $3/game $399, The total so far is S1891 + $399- $2290) How much Daniel spend his collection ? (he spent $2290 for games in total)
for 57 each.
games in
A: The answer is 2290
21


Reviews:
[{"review_id": "rXCo6nBzBI", "reviewer": "Reviewer_7QVa", "summary": "This paper introduces a variation to the chain of thought and scratchpad techniques that can easily be applied to pre-trained transformers.  While reading a passage, the model can insert \"self notes\" at any point in the input sequence.  These self-notes allow the model to perform chain-of-thought reasoning, by annotating the input sequence with additional information about characters, variables, etc.  Self-notes differ from standard COT because the model can create multiple notes, and place those notes at appropriate locations in the input sequence.  \n\nThe authors compare several different ways of training a model to insert notes: supervised, semi-supervised, unsupervised, and prompting.  The \"unsupervised\" method is the most interesting, because it requires the least amount of human-generated annotations, although it does require that the model be fine-tuned on a QA dataset.  IMO, it's not quite \"unsupervised\", simply a clever way of using existing QA datasets to train a model to \"ask itself questions\".  \n\nAll three methods produce improvements over SOTA on a variety of tasks.\n", "questions": "Have you considered how this technique could be scaled to LLM size?  What potential problems do you expect to encounter?\n", "limitations": "The authors do not discuss societal impacts.  They do briefly mention scaling as an important area for future work, but do not go into any detail about the limitations of their training techniques.\n\n\n", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper was very clearly written.  I especially applaud the authors for concisely illustrating their main contributions in a figure that appears right before the opening introductory paragraph.\n\nThe self-note mechanism is a fairly simple extension to prior work on scratchpad and chain-of-thought reasoning, but I fully expect it to be high-impact.  The technique is easy to understand, easy to implement, works with pre-trained models, and results in large and obvious improvements on reasoning tasks.  The experiments are well-designed; the authors compare multiple training methods on multiple tasks, and the chosen tasks are appropriate.\n", "weaknesses": "The main limitation of this work is (1) the requirement for human annotations in the supervised and semi-supervised cases, or (2) the requirement for a Q/A dataset in the unsupervised case.  The size of the models are relatively small (GPT2), and the tasks are fairly simple, and partially synthetic.  Additional work may be needed to scale this technique up, especially since existing QA datasets are quite small compared to the training corpus for modern LLMs.  However, I feel that some combination of unsupervised training and prompting could probably overcome the scaling challenges.  \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the careful review and helpful comments. We appreciate the attention to detail, the concise description of contributions, and beneficial suggestions. Thank you also for highlighting the potential high-impact of our work on future research. We have addressed your comments regarding the GPT2 experiments and scaling to LLM size in the response to all reviewers.\n\n\n\n----\n\n\n\n>The main limitation of this work is (1) the requirement for human annotations in the supervised and semi-supervised cases, or (2) the requirement for a Q/A dataset in the unsupervised case. \n\nThank you for pointing out this important detail. We note that scratchpad and few-shot chain-of-thought also require human annotations and Q/A datasets. So while it is a drawback compared to vanilla training, it requires the same amount of annotations as Chain-of-Thought/Scratchpad. The goal of this work is to experimentally validate the difference between in-context thoughts (Self-Notes) vs purely post-context thoughts (CoT/Scratchpad). We have added this discussion to the manuscript.\n\n\n\n----\n\n\n\n> The tasks are fairly simple, and partially synthetic. \n\nWe aimed for a wide variety of tasks in several domains, demonstrating the advantage of our method in both synthetic and real-world tasks. We followed Fan et al. 2020 and Anil et al. 2022, for synthetic testbed tasks, and followed Wei et al. 2022 and Toshniwal et al. 2021 for real-world tasks. We specifically chose these datasets as we believe they most appropriately evaluate/demonstrate the ability of a model to do multi-step reasoning and state-tracking.\n\n\n\n----\n\n\n\n> Additional work may be needed to scale this technique up, especially since existing QA datasets are quite small compared to the training corpus for modern LLMs. However, I feel that some combination of unsupervised training and prompting could probably overcome the scaling challenges.\n\nThank you for raising this important point. There are several recent works showing that small-scale fine-tuning datasets (e.g. “LIMA”, “AlpaGasus”) work well with large pre-trained models. So we don’t expect there to be an issue fine-tuning large models on a small set of labeled Self-Note samples. We also show that we can get good performance with Self-Notes using few-shot prompting, where we only require a few labeled instances. We do agree that some combination of unsupervised training and prompting could lead to even greater gains, this is a great suggestion and an interesting direction to pursue.\n\n\n\n----\n\n\n\n> The authors do not discuss societal impacts. They do briefly mention scaling as an important area for future work, but do not go into any detail about the limitations of their training techniques.\n\nThank you for bringing this important missing discussion to our attention. We will add the following. We don’t expect any negative societal impacts as a direct result of our method. We acknowledge that there are already significant potential impacts with language models in general, but we don’t expect our work to mitigate any of such existing issues, other than perhaps adding an extra degree of interpretability by examining the self-notes themselves. As with all tokens generated by generative language models, there is a risk of hallucinations in the Self-Notes. The practical limitations of Self-Notes are similar to those of Chain-of-thought/Scratchpad. Namely, most applications require labeled Self-Note annotations in order for the model to learn how to generate them. Lastly, generating Self-Notes is more costly than vanilla Transformer inference, but can be more efficient than CoT/Scratchpad depending on the task.\n"}, {"review_id": "XoiqtoSHQA", "reviewer": "Reviewer_D5Ka", "summary": "One very general and beneficial method for improving the outputs of LMs is called *chain-of-thought* (CoT) reasoning, by which an LM is trained or prompted to first output its step-by-step reasoning before outputting the answer to a problem. This paper proposes a major extension of CoT wherein the LM is trained or prompted to insert its reasoning (as *Self-Notes*) at any relevant points in the token stream, rather than waiting until the end of the prompt, which can be quite long when the problem description is long and complex and contains example solutions. Self-Notes also serve as a form of in-context memory, summarizing key inferences made by the model along the way. The new method is evaluated against LMs with and without CoT, on a wide range of datasets, and employing four different learning paradigms:  Supervised, semi-supervised, unsupervised, and few-shot prompted. The experiments clearly demonstrate significant performance gains due to Self-Notes in most settings, particularly in generalizing to cases not seen in training. ", "questions": "What are the numeric results shown in Table 4 & Table 5? Accuracies?", "limitations": "No concerns.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This work is sound and compelling. The presentation is especially clear and easy to follow. The results are particularly interesting given the current importance of LLMs, and the field’s focus on finding ways to improve LLM reasoning even further.", "weaknesses": "Most of the experiments were performed using GPT-2, which seriously lags behind the current generation of LLMs in terms of output quality. A few experiments in this work involved GPT-3, but the field has changed dramatically since the introduction of GPT-3.5 and especially GPT-4, which is particularly good at leveraging few-shot prompts. This work’s contribution could be much greater if the experiments included GPT-4.\n\nSection 4.4 mentions that GPT-3 was called through the public API. Given this constrained access, Self-Notes were intentionally limited to appear only at end-of-sentence positions. The same could be done when using GPT-4 through the public API. But the small margin of improvement over CoT shown in Table 5 (using few-shot prompting) raises the question of how effective Self-Notes would be when limited to end-of-sentence positions using GPT-4. \n\nAll of the examples from the datasets in this work share a certain underlying structure:  step-by-step revelation of partial information, in a way that benefits from progressive inference and ongoing memory to track the evolving state. Such tasks seem ideally suited to the insertion of Self-Notes into the token stream. By contrast, the benefit of Self-Notes would be more surprising (and valuable) in the broader set of reasoning tasks that don’t share this special structure. For instance, consider the problem from section 8.2 in [2], where the LLM is tasked to modify exactly one number in an algebraic equation in order to make the right hand side equal a particular value. A chain of reasoning is required here, but the facts in the problem statement are not arranged in an obvious way to provide convenient insertion points for Self-Notes. The significance of Self-Notes would be greater if their benefit were shown to extend to other classes of important problems.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive and valuable review. We appreciate the nice summary and the comment highlighting the importance of improving LM reasoning with Self-Notes. We have addressed your comments regarding GPT-3.5 and GPT-4 in the response to all reviewers.\n\n\n\n----\n\n\n>All of the examples from the datasets in this work share a certain underlying structure: step-by-step revelation of partial information, in a way that benefits from progressive inference and ongoing memory to track the evolving state. Such tasks seem ideally suited to the insertion of Self-Notes into the token stream. By contrast, the benefit of Self-Notes would be more surprising (and valuable) in the broader set of reasoning tasks that don’t share this special structure. For instance, consider the problem from section 8.2 in [2], where the LLM is tasked to modify exactly one number in an algebraic equation in order to make the right hand side equal a particular value. A chain of reasoning is required here, but the facts in the problem statement are not arranged in an obvious way to provide convenient insertion points for Self-Notes. The significance of Self-Notes would be greater if their benefit were shown to extend to other classes of important problems.\n\nWe appreciate the careful analysis of the datasets we used. Self-Notes is a *general* method to allow language models to write itself notes at any time, as described in Figure 1 and nicely summarized in your review. It is easily generalizable to other tasks not described in this paper. In particular, we consider Chain-of-Thought to be a special case of Self-Notes, where the note can only come at the end of the context. For the algebraic problem described, the Self-Note could be written after it has seen the full context, which is similar to CoT. While the goal of this work was to compare to the types of tasks that CoT/Scratchpad are typically used for, implementing Self-Notes on other tasks such as the algebraic problem is an exciting direction and a great recommendation. \n\n\n----\n\n\n> What are the numeric results shown in Table 4 & Table 5? Accuracies?\n\nYes, these are accuracy (%), thank you for bringing this to our attention. We have updated the manuscript to reflect this.\n\n"}, {"review_id": "jw620QFJQM", "reviewer": "Reviewer_PMpA", "summary": "The authors introduce a method called \"Self-Notes\" that allows the model to think and write down its thoughts during the reasoning process. Unlike other approaches, this method enables the model to deviate from the input context, integrate previous reasoning steps, and enhance its memory with useful information. The experiments show that the Self-Notes method outperforms chain-of-thought and scratchpad methods by interleaving the input text with the model's notes.", "questions": "1. The performance of GPT-3 is significantly low in table 5. What could be the reason behind this? Have you considered using GPT-3.5 for the experiments instead?\n2. Is it possible to generalize the \"Self-Notes\" data from one task to different tasks?", "limitations": "The authors have discussed the limitation and broad impact in the paper.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The authors propose a method called \"Self-Notes,\" which involves jotting down thoughts during the reasoning process.\n2. The experiments compare their method with different algorithms and demonstrate the advantages of their approach in supervised, semi-supervised, and unsupervised learning settings.", "weaknesses": "1. The \"Supervised Self-Notes\" experiments were conducted using GPT-2. However, it is recommended to use larger open-source models like LLaMA for these experiments.\n2. It is recommended to conduct \"Semi-supervised Self-Notes\" experiments for all tasks shown in the paper.\n3. It is preferable to use the same evaluation set for all three types of experiments. For instance, Tables 4 and 5 utilize math word problems for evaluation, which are not included in Tables 2 and 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review. We have addressed your comments regarding LLaMa and GPT-3.5 in the comment to all reviewers.\n\n\n\n----\n\n\n> It is recommended to conduct \"Semi-supervised Self-Notes\" experiments for all tasks shown in the paper.\n\nAcross the two tasks we conducted semi-supervised experiments on (Toy-Story and Algorithmic), we observed a consistent trend of Self-Notes outperforming the vanilla model at less than 25% supervision. The objective of these experiments was to show that it’s possible to achieve good performance even when we don’t have every sample in the training set labeled with Self-Notes, which can be expensive. We expect this trend to remain across other tasks. \n\n\n----\n\n\n> It is preferable to use the same evaluation set for all three types of experiments. For instance, Tables 4 and 5 utilize math word problems for evaluation, which are not included in Tables 2 and 3.\n\nWe did not run the fine-tuning settings (supervised, semi-supervised, unsupervised) on the math word problem datasets because they are small datasets (MultiArith only has 600 samples), and thus is better fit for few-shot prompting where we can evaluate on all samples.\n\n----\n\n\n> The performance of GPT-3 is significantly low in table 5. What could be the reason behind this? Have you considered using GPT-3.5 for the experiments instead?\n\nThank you for raising this question. Greedy decoding, 5-shot prompting instead of 8-shot prompting used by Wei et al. 2022, and use of text-davinci-001 are the main reasons for the reported performance. Our emphasis was to compare Self-Notes relative with CoT, rather than their absolute performance. See also the comment to all reviewers for more detail on this.\n\n----\n\n\n> Is it possible to generalize the \"Self-Notes\" data from one task to different tasks?\n\nThis is a great question. We expect that if the model is trained on a diversity of tasks with Self-Notes supervision, the model can generalize to novel tasks, as has been the case with instruction-tuned models that generalize to novel instructions when trained with a rich diversity of instruction-based tasks. That would be very exciting to see in future work. \n\n\n"}, {"review_id": "r2zEr6ekq2", "reviewer": "Reviewer_v6D1", "summary": "The paper presents a general prompting approach for LLMs: instead of generating intermediate \"thoughts\" after processing the prompt as Chain-of-Thought (CoT), this paper proposes adding \"Self-Notes\" **while reading the prompt**. That is, while the initial reading of the prompt, the model can generate intermediate Self-Notes that defer the reading of the rest of the prompt until the end of generating the Self-Note.\n\nThe approach is evaluated across multiple reasoning benchmarks, with supervised models, semi-supervised models, unsupervised models (which I'd call \"unlabeled\" models, because they *are* supervised on an augmented version of the supervised input), and few-shot prompted models.", "questions": "## Questions\n\n1. Wouldn't the approach be stronger if the question was provided **first**, before the context input? Currently, \"The model can use notes as a form of working memory by **writing information that might be useful in the future**\" (Lines 76-77). But how would the model know what information might be useful in the future, if it hadn't processed the question yet?\n2. I would rename the \"unsupervised Self-Notes\" to \"Unlabeled Self-Notes\" or something in this spirit, because the model **is** supervised, but as far as I understand, on augmentation of the dataset, rather than on a manually-labeled version of the dataset.\n3. Imagine that you had the ability to run GPT-3/4 locally. Isn't expecting the model to generate self-notes much slower than standard prompting? in standard prompting, the model reads the entire prompt in a single forward pass, where all prompt tokens can be processed in parallel. In Self-Notes, don't we must feed the prompt tokens one-by-one, to check if the model has predicted the next token to be \"start-of-note\"? (and in that case, stop feeding the original prompt, and feed the generated note instead?)\n\n## Summary\nOverall, the idea presented in this paper is really nice and novel. I also appreciate the authors' empirical efforts across various benchmarks and supervision scenarios. \n\nUnfortunately, the idea seems not to work with models newer than the first version of GPT-3 and supervised models that are larger than `gpt2` (small), and thus I gave low \"soundness\" and \"contribution\" scores, and overall a \"weak accept\" score. \n\nI would have given a higher score if the paper had demonstrated that the same idea can be useful to improve stronger models such as `text-davinci-002`, `text-davinci-003`, `code-davinci-002`, `gpt-3.5-turbo`, `gpt-4`, and supervised models that are larger than `gpt2` (even `gpt2-large` / `gpt2-xl` can be fine-tuned on a single GPU, I believe).\n\nI hope that the authors would strengthen the evaluation for the next version by evaluating with stronger base models. Although I vote for acceptance, I will understand if the other reviewers will argue for rejection because of these evaluation limitations.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "## Strengths\n* The idea to generate Self-Notes *while processing the input*, rather than at the end, is simple and novel. It is inspiring to see how such a simple idea has not been used before.\n* The paper evaluates multiple datasets and tasks: Toy-Story (a synthetic multi-hop QA), algorithmic & boolean (a synthetic source code simulation), Chess Piecetype & Chess Move (a synthetic chess simulation).\n* The paper evaluates multiple supervision scenarios: supervised models, semi-supervised models, unsupervised models (which I'd call \"unlabeled\" models, because they *are* supervised on an augmented version of the supervised input), and few-shot prompted models.\n* The Related Work places the paper well in the related literature.\n* The paper is well-written and easy to follow", "weaknesses": "## Weaknesses\n* The main weakness is the relatively small models that were used: \n    * In supervised and semi-supervised settings, the base model for both the baseline and the model that Self-Notes is applied on is a **GPT-2-base** (there is actually no `gpt2-base` model on Huggingface. Do the authors mean `https://huggingface.co/gpt2`? If so, this is the **smallest** version of GPT-2, with 124M parameters). While providing a good initial indication, I am not sure that the conclusions for GPT-2 will hold for larger models (because of \"emergent abilities\" and generally very different behavior of GPT-2-sized models compared to larger models). Even if the authors have access only to a single GPU, I think that there are larger models that can be fine-tuned on a single GPU.\n    * The few-shot prompting experiments were performed on **GPT-J**, which is a bit disappointing - these days, it is relatively easy for the broad community to experiment with large models via prompting API, and new few-shot prompting approaches such as Self-Notes can be easily evaluated on much stronger models such as GPT-3/4 / PaLM / Claude / Codex.\n    * For the GSM-8K benchmark, the few-shot prompting experiments were performed using **GPT-3** (since not mentioned otherwise, I am assuming `text-davinci-001`). Since these experiments could be easily evaluated using `text-davinci-002`, `text-davinci-003`, `code-davinci-002` by just replacing the model name, I am left with the conclusion is that Self-Notes does not work with these stronger models.\n\nWhile these experiments are extensive, I believe that they do not confirm the strong claims of the paper, which hurts its soundness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive review. In particular, we appreciate the thorough and well understood list of our contributions including our empirical efforts across various benchmarks and supervision scenarios. We have addressed your comments regarding the GPT3+ and other OpenAI models in the response to all reviewers.\n\n---\n\n> In supervised and semi-supervised settings, the base model for both the baseline and the model that Self-Notes is applied on is a GPT-2-base (there is actually no gpt2-base model on Huggingface. Do the authors mean https://huggingface.co/gpt2? If so, this is the smallest version of GPT-2, with 124M parameters). While providing a good initial indication, I am not sure that the conclusions for GPT-2 will hold for larger models (because of \"emergent abilities\" and generally very different behavior of GPT-2-sized models compared to larger models). Even if the authors have access only to a single GPU, I think that there are larger models that can be fine-tuned on a single GPU.\n\nThank you for pointing this out. We do indeed use the smallest GPT-2 model (124M parameters), and will update our manuscript to reflect so. While it’s true in general that there are different properties in larger models, we observe that the GPT-2 result trends are consistent with the few-shot prompting experiments on the larger models. The main objective of this work is to experimentally validate the difference between in-context thoughts (Self-Notes) vs post-context thoughts (CoT/Scratchpad) on a fixed model, which we do across a variety of models of different sizes.\n\n---\n\n> The few-shot prompting experiments were performed on GPT-J, which is a bit disappointing - these days, it is relatively easy for the broad community to experiment with large models via prompting API, and new few-shot prompting approaches such as Self-Notes can be easily evaluated on much stronger models such as GPT-3/4 / PaLM / Claude / Codex.\n\nWe did evaluate using a GPT-3 model (text-davinci-001). Note that for cost as well as model openness reasons, we didn’t evaluate our models on a whole array of models accessible via a paid API.  \n\n---\n\n> Wouldn't the approach be stronger if the question was provided first, before the context input? Currently, \"The model can use notes as a form of working memory by writing information that might be useful in the future\" (Lines 76-77). But how would the model know what information might be useful in the future, if it hadn't processed the question yet?\n\nThank you for raising this great question. We agree that a question-driven input, i.e. input placed first, should filter irrelevant Self-Notes for the entity(ies) of interest in the question. However, we’re imagining a curious reader which takes these general notes, and which can ultimately answer questions of a wider range. We note that this can be beneficial for several reasons. First, in many scenarios because the model has already precomputed the answer to a variety of questions, and doesn’t have to “re-reason” each time a new question is asked.\nSecond, for certain tasks, such as program evaluation, it can be hard to predict what entities (variables) can influence the value of the entity of interest since that would require predicting the evolution of the program. Finally, by providing the question after the context, we can evaluate a fair comparison to previous CoT/Scratchpad methods which do the same.\n\n---\n\n> I would rename the \"unsupervised Self-Notes\" to \"Unlabeled Self-Notes\" or something in this spirit, because the model is supervised, but as far as I understand, on augmentation of the dataset, rather than on a manually-labeled version of the dataset.\n\nThank you for bringing this up. We see from your description that the term Unsupervised Self-Notes could be interpreted the wrong way. The model is trained with a supervised loss on the answer, but the self-notes are unsupervised. We will update this in our final manuscript. \n\n---\n\n> Imagine that you had the ability to run GPT-3/4 locally. Isn't expecting the model to generate self-notes much slower than standard prompting? in standard prompting, the model reads the entire prompt in a single forward pass, where all prompt tokens can be processed in parallel. In Self-Notes, don't we must feed the prompt tokens one-by-one, to check if the model has predicted the next token to be \"start-of-note\"? (and in that case, stop feeding the original prompt, and feed the generated note instead?)\n\nSelf-Notes is expected to run slower than standard prompting since it generates additional tokens, and as suggested, the input can’t be processed in parallel. Note that we only check for a Self-Note generation at the end of statements/sentences which makes it more efficient than performing this check after every token. Comparison with Scratchpad/CoT is task-dependent though. In our experiments, we found Self-Notes to be faster than scratchpad for certain tasks (e.g. Algorithmic where scratchpad has to copy the entire program), and slower for others (e.g. Toy Story).  Speedups can also be achieved by caching hidden states of previous forward passes.\n\n\n\n"}]

Author rebuttal:
We would like to thank all reviewers for the invaluable feedback. We greatly appreciate the encouraging and helpful comments which have made our paper stronger. All reviewers pointed out the main contributions of our work and noted the potential impact of our work on future research. We respond to individual comments below each review, and make a general response to comments regarding our experimental setup here. 

$\newline$

We show extensive and diverse experiments across 4 training and inference settings (supervised, semi-supervised, unsupervised, few-shot prompted), on 7 total datasets, using various models ranging from 124M to 175B parameters in size. Importantly, our proposed method applies generally across all model sizes, datasets, and training/inference settings that we have tested. 

The fine-tuning experiments were performed with smaller models (GPT-2) since we use a large number of datasets and training settings, which takes a substantial amount of resources. The few-shot experiments were performed with larger models (GPTJ and GPT3 text-davinci-001), where a limiting factor was the financial constraints of a paid API. 

There is an extensive amount of work which show that techniques to improve reasoning and state tracking improve as models are scaled up (Nye et al. 2021, Wei et al. 2022, Anil et al., 2022, Wang et al. 2023, Yao et al. 2023, Touvron et al. 2023). These works include both fine-tuning and few-shot prompting. Aside from being more expensive than vanilla training/inference, there is nothing inherently limiting about our proposed method as we scale the size of the model up. 

We note that the GPT-4 API general availability wasn’t until July 6, 2023, so it was not considered for this submission. GPT-4 was also trained on GSM8K samples which include CoT explanations (OpenAI, 2023), and there is anecdotal evidence that this is the same for GPT-3.5 (Fu et al., 2023). We used a diverse set of models for our experiments to demonstrate the usefulness of our method. While there are more proprietary models (GPT3+, Claude, etc), they are not open, so we do not know the training data which may add unknown bias to results (e.g., techniques in Lightman et al., 2023), and are expensive so we can only run a certain amount of experiments, which we also unfortunately lacked funds for. Thus, in general we believe the community should push for the use of open models for reproducible science rather than necessitating the use of such models.

While models will always be improving and there will be new models to test a new method on, we ran extensive experiments and compared results for self-notes vs scratchpad/CoT for a fixed model. It seems clear that as models get stronger, they will be able to write better Self-Notes. We are hence excited about the applications of Self-Notes that will be enabled in the future.


Meta-review: This work receives mostly positive reviews and rebuttal has addressed some critical concerns. All reviewers agree that the idea self-notes is a nice extension to CoT that allows LLMs to take in-context notes during reasoning, and its effectiveness is evaluated on relatively small scale models. This work could be greatly improved with a systematic evaluation on latest LLMs, which should be included in the camera version as suggested by most reviewers. AC acknowledges the novelty and potential of this work, and thus recommend acceptance.
Decision: Accept (poster)