SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chess Move. This task is to predict the end position of the current move given the start position [17]. The Self-Notes are the same as Chess Piecetype, where the model is trained to generate the piece at each starting square as it makes a move. We report the exact match accuracy. We use the same train-valid/test split as the Chess Piecetype task. Table 7, shows an example of this task, which is similar to Chess Piecotypewith the exeption of the question token, MOVE as well as the final answer being a board position.
MultiArith. This is a real-world reasoning dataset was introduced to evaluate a model's ability to solve multi-step arithmetic word problems [18]. An example from the dataset is "While on vacation, Megan took 15 pictures at the zoo and 18 at the museum. If she later deleted 31 of the pictures, how many pictures from her vacation did she still have?". Here, the Self-Notes are intermediate summaries in the middle of the context. For example in the previous example, they would look like: "While on vacation, Megan took 15 pictures at the zoo and 18 at the museum.  $(18 + 15 = 33$  total) If she later deleted 31 of the pictures, how many pictures from her vacation did she still have?  $(33 - 31 = 2$  left)". Here, the start token is "(and the end token is ")". We use the few-shot training samples from Wei et al. [4], which are general arithmetic word problems that are not from the MultiArith dataset. We evaluate on the full 600 samples provided in the dataset and compare to the Chain-of-Thought (CoT) and vanilla prompted results using the GPT-J model. Our To create the Self-Note prompts, we remove the Chain-of-Thought explanations and add our own Self-Note enrichments in the 8 Wei et al. [4] prompts. Table 12 shows the exact prompts used for the Self-Notes and Chain-of-Thought results. There is no validation set for the prompting experiments.
Table 8: Dataset Statistics. Note that the Math datasets (MultiArith and GSM8K) are used for few-shot prompting with GPT-J and GPT-3.
Dataset	# train	# valid	# test	In domain	Out-of-domain
Toy-story	10k	1k	1k	1, 2 hops	3, 4 hops
Algorithm	10k	1k	1k	2-100 statements	101-200 statements
Boolean Variable	524k	1k	1k	3-8 statements	9-19 statements
Chess Piece	200k	1k	1k	≤ 80 moves	≥ 81 moves
Chess Move	200k	1k	1k	≤ 80 moves	≥ 81 moves
MultiArith	8 (prompt)	N/A	600	N/A	N/A
GSM8K	5 (prompt)	N/A	1319	N/A	N/A
GSM8K. This task was introduced to evaluate more challenging arithmetic reasoning problems [19]. We selected 5 random samples in the training set to use as the few-shot demonstrations in the prompt. For Chain-of-Thought, we used the original solutions (minus the calculation annotations written inside “ $\ll$ ”) given in the dataset. To create the Self-Note prompts, we remove the Chain-of-Thought explanations in the answer and add our own Self-Note enrichments in the 5 prompts. The start and end tokens are the same as MultiArith: “(”and “)”. Table 13 shows the exact prompts used for the Self-Notes and Chain-of-Thought results. Self-Notes are only added after “:”, “;” and “?” tokens, so we do not have to call the API after each token.
For each non-prompting task, we train for a fixed 30 epochs with a learning rate of  $2e - 5$  and batch size of 32. For the Vanilla and Self-Notes methods, all models are trained on the context and answer tokens. For the Scratchpad method, the model is trained on the context and answer for Toy-Story, and is trained on only the answer for all other tasks since the scratchpad contains a full copy of the context. For the tasks that involve length generalization (Algorithmic, Boolean Variable, Chess Piecetype, Chess Move) we randomly offset the positional embedding IDs between 0 and 128 for the GPT-2 model during training. That is, the positional embedding ID of the first token is randomly assigned to a value from 0 to 128, and then continues sequentially. During inference, the position IDs always start at 0. This ensures that the model has seen all of the position embeddings that it will encounter during testing (following Kiyono et al. [38]). We note that this can also be solved by the relative position embeddings used in more recent models [49, 50].
We fine-tune all of the GPT-2 models on 8 NVIDIA V100 GPUs using an on-site cluster. The GSM8K experiments were done using the text-davinci-003 model with the OpenAI API, costing around $300.
Reproducibility statement: We will make code and data publicly available.
A.3 Dummy Tokens: Separating Content and Compute
We introduced Self-Notes as a method for language models to do reasoning in the form of explicit tokens. To understand the value of these extra tokens, we seek to separate and measure the contribution of the additional compute that is allotted by these tokens from their content. In other words, we answer the question "Is it the specific tokens in the Self-Notes or is it simply extra steps of computation that is helping?" We do this by inserting the dummy token "_" at various locations throughout the context as an ablation. These are similar to a Self-Note with a single token that always remains the same.
We first consider the Toy-Story task. In the vanilla setting for this task, there are only facts and the question (e.g., "Bob is in the park. Bob has the key. Where is the key?"). The first comparison is inserting a dummy after every fact, which we call "Naïve Dummy" ("Bob is in the park. _ Bob has the key. _ Where is the key?"). The alternative, which we call "Self-Notes Dummy" is adding dummy tokens in the same locations where Self-Notes are written. In other words, at the locations where a relation between two facts can be inferred ("Bob is in the park. Bob has the key. _ Where is the key?"). Finally, we consider adding the dummy tokens where the Scratchpad tokens would be ("Bob is in the park. Bob has the key. Where is the key? _*"). This setting adds the same amount of dummy tokens as are used in Self-Notes Dummy (only 1 in the example described, but the "*" operator indicates that extra dummy tokens can be added). Figure 6 shows the results on the 3-hop and 4-hop test sets for the different dummy token settings compared to the natural
(A) Vanilla Inference
(B) Self-Notes Inference
Figure 5: Comparison of Vanilla vs Self-Notes inference. Non-highlighted tokens are given to the model. Highlighted tokens are generated by the model. In Vanilla inference (top), the model generates only after the full context and question are provided. In Self-Notes inference (bottom), the model is able to generate after every word or sentence in the context. If the next most likely token is the Self-Notes start token “<SN>”, then the model can autoregressively generate itself a note until the end token “</SN>” is generated, at which point it returns to reading the original context.
Figure 6: Ablation comparing the impact of (a) extra compute due to additional tokens, and (b) position of the additional tokens.
language tokens used by Self-Notes and Scratchpad. Intelligently inserting dummy tokens into the positions where Self-Notes should be performs the best out of the four settings. Importantly, it is better than inserting at the end of the context, where Scratchpad tokens are added. This alludes to the fact that allowing the model to do extra computations in the middle of the context can be better than after the context.<sup>5</sup> However, the gain from Self-Notes Dummy over other dummy variants pales in comparison to the gains of actual Self-Notes over its dummy counterpart, suggesting that the content of the intermediate notes matters more than just the additional compute.
We also test the usefulness of Self-Notes Dummy tokens in three other settings: Chess Piecetype, Chess Move, and WikiText-103 language modeling. For the chess experiments, we compare the
In the in-context learning setting, Wei et al. [4] reported that gains with chain-of-thought were not due to additional compute. Our setting though differs in three aspects: (a) the additional compute is happening in the middle of the context in case of Self-Notes Dummy, (b) we're finetuning the language model, and (c) we're working with a much smaller language model.
vanilla moves ("c2 c4"), a dummy token between move positions ("c2 _ c4"), and the generated (non-dummy) piece tokens in Self-Notes ("c2 P c4"). For WikiText-103 language modeling, we compare the vanilla text ("The cat sat on the mat") with dummy tokens inserted between each word in the text ("The _ cat _ sat _ on _ the _ mat"). There are no ground-truth Self-Notes for WikiText-103, so we neglect such experiments. Table 10 shows the accuracy results for chess and perplexity for WikiText-103. Dummy tokens are not reported in the accuracy or perplexity numbers. For each task, dummy tokens improves the vanilla setting by a small margin.
A.4 Further Ablations
Oracle/No Self-Notes during inference. We perform two ablations regarding the use of Self-Notes during inference. The first is an upper bound where we provide  $100\%$  Self-Notes supervision to the model during both training and inference (rather than just training). The second is where we give  $100\%$  Self-Notes supervision during training, but restrict the model from generating Self-Notes during inference. This baseline analyzes whether Self-Notes-augmented training data can still help the model learn about the task in its weights even without using Self-Notes at inference time. The results are shown in Table 11. As expected, oracle Self-Notes ( $100\%$  Self-Notes supervision during inference) improves the performance. On the other hand, not allowing the model to generate Self-Notes leads to a drastic drop in performance due to distribution shift at inference time.
A.5 Prompts for Few-Shot Self-Notes
For the results in Table 4 and Table 5, we use the Self-Notes prompts shown in Table 12 and Table 13, respectively. The MultiArith prompt (Table 12) is the same set of examples as used in Wei et al. [4]. We augment the standard prompts they use with Self-Notes, which are interleaved in the context and question. We use a GSM8K prompt (Table 13) derived from a subset of the training samples, where we added Self-Notes. All prompting experiments are compared to Vanilla (standard prompts) and Chain-of-thought (COT) prompts.
Table 9: Test sample from the Toy-story task. In this example, the question  $(Q)$  is "Where is the ball?" and the answer  $(A)$  is "the ball is at the farm." The vanilla model fails at multi-step reasoning and incorrectly predicts that the ball is at the "store" The Scratchpad starts to incorrectly reason where the ball is early, and cannot correctly predict where it ends. The Self-Notes method writes each intermediate reasoning step correctly, including the correct location of the ball, before the final question is asked.
Model	Context	Prediction
Vanilla (original context)	Mary is with Daniel. Frank is with Sandra. John has the book. Frank has the suitcase. Daniel is at the station. the banana is inside the basket. Bob has the apple. Bob has the bag. the ball is inside the box. the apple is inside the bag. Alice has the banana. Alice has the key. John is at the farm. Charlie is at the bridge. the book is inside the box. Alice is at the store. Bob is with Alice.	the ball is at the store.
Scratchpad (scratchpad context)	[SQ: Where is Mary? A: Mary is at the station. SQ: Who has the ball? A: Bob has the ball. SQ: Who has the basket? A: Alice has the basket. SQ: Where is the book? A: the book is at the farm. SQ: Where is the bag? A: the bag is at the ball. SQ: Where is the ball? A: the ball is at the bag. SQ: Who has the box? A: John has the box. SQ: Where is the box? A: the box is at the farm. SQ: Who has the key? A: Alice has the key. SQ: Where is the banana? A: the banana is at the store. SQ: Where is the basket? A: the basket is at the store. SQ: Where is the key? A: the key is at the store. SQ: Where is Bob? A: Bob is at the store. SQ: Where is the apple? A: the apple is at the store. SQ: Where is the suitcase? A: the suitcase is at the store. SQ: Where is the basket? A: the basket is at the store.]	the ball is at the ball.
Self-Notes	Mary is with Daniel. Frank is with Sandra. John has the book. Frank has the suitcase. Daniel is at the station. SQ: Where is Mary? Mary is at the station. the banana is inside the basket. Bob has the apple. Bob has the bag. the ball is inside the box. the apple is inside the bag. Alice has the banana. SQ: Who has the basket? Alice has the basket. Alice has the key. John is at the farm. SQ: Where is the book? the book is at the farm. Charlie is at the bridge. the book is inside the box. SQ: Who has the box? John has the box. SQ: Where is the box? the box is at the farm. SQ: Who has the ball? John has the ball. SQ: Where is the ball? the ball is at the farm. Alice is at the store. SQ: Where is the banana? the banana is at the store. SQ: Where is the basket? the basket is at the store. SQ: Where is the apple? Bob is at the store. SQ: Where is the bag? the bag is at the store. SQ: Where is the key? the key is at the store. Bob is with Alice. SQ: Where is Bob? Bob is at the store. SQ: Where is the apple? the apple is at the store. SQ: Where is the bag? the bag is at the store. SQ: Where is the key? the key is at the store.	the ball is at the farm.
Table 10: Results with Dummy Tokens. For the chess tasks, we report the results for the OOD setting (over 80 moves).
Task	Vanilla	Dummy	Self-Notes
Chess Piecetype	82.9 ±2.3	84.8 ±1.7	94.8 ±0.7
Chess Move	39.8 ±0.2	40.4 ±1.2	41.8 ±0.9
WikiText-103	25.9ppt	24.9ppt	n/a
Table 11: Ablation comparing the performance of Self-Note with (i) ground truth Self-Notes at test time, and (ii) abstaining from generation of Self-Notes during inference.
Self-Notes %	Algorithmic	Toy-Story
Train	Test	2-100	101-200*	3-hop*	4-hop*
100%	100%	100.0 ±0.0	100.0 ±0.0	99.9 ±0.1	99.8 ±0.3
100%	none	21.3 ±0.6	9.2 ±0.5	37.7 ±3.9	29.0 ±1.6