SlowGuess/NeurIPS_2023_OCR huggingface dataset:
digits = [0] * len(x)
carry = 0
for j in range(len(x) - 1, -1, -1):
    t = x[j] * y[i]
    t += carry
    carry = t // 10
    digits[j] = t % 10
digits.insert(0, carry)
summands[i] = sum(digits[-k] * (10 ** (k - 1)) for k in range(1, len(digits) + 1))
product = sum(summands[-i] * (10 ** (i - 1)) for i in range(1, len(y) + 1))
return product
Listing 1: Example Python code for solving the multiplication task.
A.2 Einstein's Puzzle
Data Construction In our experiments, we initially establish a set of properties, such as Color, PhoneModel, Pet, and so forth, along with their corresponding values expressed in natural language templates (e.g., "The house has a red color"). We then devise a fundamental and straightforward set of clue types: 1) 'found_at', e.g., "Alice lives in House 2", 2) 'same_house', e.g., "The person who is a cat lover lives in the house that has a red color", 3) 'direct_left', e.g., "The person who has a dog as a pet lives to the left of the person who lives in a red house", and 4) 'besides', e.g., "The person who has a dog as a pet and the person who has a red house live next to each other." In addition, we also set up harder clue types such as 'not_at', 'left_of' (not necessarily directly left of), 'two_house_between', etc. which are only used in auxiliary experiments.
The solution to the puzzle is a matrix of size  $K \times M$ , where  $K$  represents the number of houses and  $M$  the number of attributes. During the puzzle generation, the  $M$  properties are randomly selected from the candidate pool, followed by the random sampling of  $K$  values for each property. The sampled values are then randomly permuted and assigned within the table to create the solution. It is important to note that we ensure one of the sampled properties is 'Name' to enhance the readability and comprehensibility of the puzzles. To construct the clues, we initially over-generate all valid clues based on the solution and subsequently remove redundant clues at random until we obtain a set with a
General Unique Rules
There are 3 houses (numbered 1 on the left, 3 on the right). Each has a different person in them. They have different characteristics:
- Each person has a unique name: peter, eric, arnold
- People have different favorite sports: soccer, tennis, basketball
- People own different car models: Tesla, ford, camry
1. The person who owns a Ford is the person who loves tennis.
2. Arnold is in the third house.
3. The person who owns a Camry is directly left of the person who owns a Ford.
4. Eric is the person who owns a Camry.
5. The person who loves basketball is Eric.
6. The person who loves tennis and the person who loves soccer are next to each other.
Reasoning Path Generation
Algorithm 1 Puzzle Solver  
Input: Clues  
Output: Reasoning path  
1: function PUZZLESOLVER(Clues)  
2: Path  $\leftarrow []$   
3: LeftClues  $\leftarrow$  clues  
4: while [LeftClues]  $\neq 0$  do  
5: for i = 1 to [LeftClues] do  
6: CandidateClues = (LeftClues)  
7: for clue in CandidateClues do  
8: if solve any cell then  
9: LeftClues.remove(clue)  
10: Path.append(clue)  
11: return Path
Ground-Truth Table
House	Name	Sports	Car
1	Eric	Basketball	Camry
2	Peter	Tennis	Ford
3	Arnold	Soccer	Tesla
Figure 10: A sample of the puzzle task and the reasoning path to reach a solution.
4   
5   
6   
7   
8   
9   
10   
11   
12   
13   
14   
15
Figure 11: A sample scratchpad for the puzzle task.
unique solution, as previously sampled. This process ensures a coherent and engaging puzzle-solving experience. Refer to Figure 10 for an example.
Graph Construction Algorithm To solve the complex compositional reasoning process for a logical grid puzzle, we use existing puzzle solvers [21] to generate the computation graph. It follows the basic greedy principle of applying the minimum number of rules to solve any cell, i.e., if using only one rule to solve any given cell, then apply this rule. This algorithm iterates through all clues in the clue set until one or a set of clue combinations can solve any cell in the table. While it may not be the most efficient way to solve the puzzle, it provides models with explicit scratchpad verbalization through an intuitive computation graph. Refer to Figure 10 for the pseudo-code of the process, and Figure 11 for a scratchpad example.
A.3 Dynamic Programming Problem
A.3.1 Solution to this problem
Let  $a = [a_1, \ldots, a_n]$  be an input. Let  $dp_i$  be the maximum sum of a subsequence that does not include adjacent elements, when considering only the elements of the input from the  $i$ -th position onwards.
Trivially,  $dp_{n} = \max(a_{n}, 0)$  since we only want to choose a number if it is non-negative. Moreover,  $dp_{n-1} = \max(a_{n}, a_{n-1}, 0)$  since we cannot choose adjacent numbers.
For any given  $dp_{i}$  with  $i \leq n - 2$ , we can express it in terms of  $dp_{i + 1}$  and  $dp_{i + 2}$ . Concretely, the maximum sum of a subsequence starting at position  $i$  may or may not include the element in the  $i$ -th position,  $a_{i}$ . If the subsequence includes  $a_{i}$ , then the maximum sum is  $a_{i} + dp_{i + 2}$ , since using  $a_{i}$  blocks us from using the next element. If the subsequence does not include  $a_{i}$ , then its sum is  $dp_{i + 1}$ . Moreover, the answer may never be less than zero, because otherwise we would select the empty sequence<sup>7</sup>. In summary,
$$
d p _ {i} = \max  \left(d p _ {i + 1}, a _ {i} + d p _ {i + 2}, 0\right)
$$
We now have a recursion with its base cases  $dp_{n} = \max (a_{n},0)$  and  $dp_{n - 1} = \max (a_n,a_{n - 1},0)$ , and we can therefore compute all values in  $O(n)$ . It now only rests to reconstruct the lexicographically smallest subsequence that maximizes the desired sum, based solely on the computed  $dp$  values.
Starting from  $dp_{1}$  and iterating sequentially through  $dp_{n - 2}$ , we choose an item if and only if  $dp_{i} = a_{i} + dp_{i + 2}$  (that is, the maximum sum comes from choosing the current element) and we have not chosen the previous element. This helps disambiguate cases where choosing or not choosing  $a_{i}$  yields the same sum, but possibly only one of those will not incur in choosing adjacent numbers. Similarly, for positions  $i = n - 1$  and  $i = n$  we choose the element if  $dp_{i} = a_{i}$  (that is, choosing the element yields the maximum sum) and we have not chosen the immediately previous element. See an example Python solution in 2.
Given a sequence of integers, find a subsequence with the highest sum, such that no two numbers in the subsequence are adjacent in the original sequence.
Output a list with "1" for chosen numbers and "2" for unchosen ones. If multiple solutions exist, select the lexicographically smallest. input = [3, 2, 1, 5, 2].
Figure 12: Example prompt for the DP task, used for zero-shot and few-shot settings.
1 def maximum_sum_nonadjacent_subsequence(arr):
2 N = len(arr)
3 dp = [0 for _ in range(N)]
4 dp[N - 1] = max(arr[N - 1], 0)
5 dp[N - 2] = max(max(arr[N - 1], arr[N - 2]), 0)
6 for i in range(N - 3, -1, -1):
7 dp[i] = max(max(dp[i + 1], arr[i] + dp[i + 2]), 0)
8
9 # reconstruct the answer with a fixed-size graph
10 dp[i] = max(max(dp[i + 1], arr[i] + dp[i + 2]), 0)
11
12 # reconstruct the answer with a fixed-size graph
13 result = []
14 can_use_next_item = True
15
16 for i in range(N - 2):
17 if dp[i] == arr[i] + dp[i + 2] and can_use_next_item:
18 result.append(1)
19 can_use_next_item = False
20 else:
<sup>7</sup>We don't need to explicitly check for this since  $dp_{n} \geq 0$ . However, we include the condition to ease the scratchpad logic.
result.append(2) can_use_next_item  $=$  True   
if dp[N-2]  $= =$  arr[N-2] and can_use_next_item: result.append(1) can_use_next_item  $=$  False   
else: result.append(2) can_use_next_item  $=$  True   
if dp[N-1]  $= =$  arr[N-1] and can_use_next_item: result.append(1)   
else: result.append(2)   
return result
Listing 2: Example Python code for solving the DP task. We chose this implementation because the computation graph has always the same topology for any given input length.
Question: Let's solve input  $= [3, 2, 1, 5, 2]$ .
Scratchpad: dp[4] = max(input[4], 0) = max(2, 0) = 2  
dp[3] = max(input[3], input[4], 0) = max(5, 2, 0) = 5  
dp[2] = max(dp[3], input[2] + dp[4], 0) = max(5, 1 + 2, 0) = 5  
dp[1] = max(dp[2], input[1] + dp[3], 0) = max(5, 2 + 5, 0) = 7  
dp[0] = max(dp[1], input[0] + dp[2], 0) = max(7, 3 + 5, 0) = 8
Finally, we reconstruct the lexicographically smallest subsequence that fulfills the task objective by selecting numbers as follows. We store the result on a list named "output".
Let can_use_next_item = True.  
Since dp[0] == input[0] + dp[2] (8 == 3 + 5) and can_use_next_item == True, we store output[0] = 1. We update can_use_next_item = False.  
Since dp[1] != input[1] + dp[3] (7 != 2 + 5) or can_use_next_item == False, we store output[1] = 2. We update can_use_next_item = True.  
Since dp[2] != input[2] + dp[4] (5 != 1 + 2) or can_use_next_item == False, we store output[2] = 2. We update can_use_next_item = True.  
Since dp[3] == input[3] (5 == 5) and can_use_next_item == True, we store output[3] = 1. We update can_use_next_item = False.  
Since dp[4] != input[4] (2 != 2) or can_use_next_item == False, we store output[4] = 2.  
Reconstructing all together, output=[1, 2, 2, 1, 2].
Figure 13: A sample scratchpad for the DP task used for fine-tuning with few-shot settings.
Data Construction We exhaustively generate data for this DP task. For question-answer setting, we include a thorough explanation of the task before asking to generate a solution (see Figure 12). We use all lists up to 5 elements as training, and we consider only lists where elements are in the range  $[-5, 5]$  (giving a total of  $11^{n}$  lists for an input list of size  $n$ ). For out-of-domain evaluation, we use lists of sizes 6 to 10 inclusive. Example scratchpads and zero-shot prompts are shown in Figure 13 and 12 respectively. The scratchpad is generated automatically through templates. We considered five exemplars for the few-shot setup.
21   
22   
23   
24   
25   
26   
27   
28   
29   
30   
31   
32   
33   
34   
35   
36
B Experimental Setups & Empirical Results
B.1 Models
For our experiments, we evaluate the performance of 6 LLMs: GPT4 (gpt-4) [58], ChatGPT (GPT3.5-turbo) [57], GPT3 (text-davinci-003) [11], FlanT5 [17] and LLaMa [75]. The evaluations were conducted from January 2023 to May 2023 using the OpenAI API. We perform fine-tuning on GPT3 (text-davinci-003) for the three tasks, observing faster convergence when training on question-scratchpad pairs rather than question-answer pairs. For question-answer pairs fine-tuning, we train separately the model for \{14, 12, 4\} epochs for multiplication, puzzle, and DP respectively, saving the best model based on the validation set. Regarding training on question-scratchpad pairs, we train the model for \{16, 8, 2\} epochs for multiplication, puzzle, and DP. The batch size is set to approximately  $0.2\%$  of the number of examples in the training set. Generally, we observe that larger batch sizes tend to yield better results for larger datasets. For the learning rate multiplier, we experiment with values ranging from 0.02 to 0.2 to determine the optimal setting for achieving the best results and chose 0.2. During inference, we set nucleus sampling  $p$  to 0.7 and temperature to 1. For each task, we evaluate the performance of each model on 500 test examples.
B.2 Limits of Transformers in Zero- and Few-shot Settings
Figure 15, Figure 17 and Figure 20 show the zero-shot performance of GPT4, ChatGPT, LLaMA and FlanT5 on the three tasks. Overall, there is a notable decline in performance as the task complexity increases (measured by graph parallelism for multiplication and DP, and propagation steps for puzzles as shown in Figure14). The few-shot performance with question-answer pairs results in minimal improvement over the zero-shot setting as depicted in Figure 16 and Figure 20 for the multiplication and DP tasks. In contrast, the few-shot setting did not lead to any improvement in the puzzle task.
Figure 14: Graph parallelism vs accuracy. The accuracy decreases as the complexity increases.
Figure 15: Zero-shot accuracy. Performance of ChatGPT, GPT3, LLaMA and FlanT5 on the multiplication task.
B.3 Limits of Transformers with question-answer Training
Figure 18 and Figure 21 show the performance of GPT3 finetuned on question-answer pairs. The model was trained on various splits, considering the problem size, depth, and width of the computation graph. Specifically, for the multiplication task, the model was fine-tuned on a range of multiplication problems, spanning from 1-digit by 1-digit multiplication to 4-digit by 2-digit multiplication amounting to 1.8M pairs. As for the puzzle task, the model was fine-tuned on puzzles of sizes ranging from  $2 \times 2$  to  $4 \times 4$  resulting in a total of  $142 \times 4$  pairs. Additionally, for the DP task, the model was fine-tuned on problems with a sequence length of 5 resulting in  $41 \times 4$  pairs. In an additional setup, we divided
Figure 16: Few-shot accuracy with question-answer pairs. Performance of GPT4, ChatGPT, GPT3, LLaMA and FlanT5 on the multiplication task.
Figure 17: Zero-shot accuracy. Performance of ChatGPT, GPT3, LLaMA and FlanT5 on the puzzle task. Few-shot performance led to worse performance.
Figure 18: GPT3 finetuned on the puzzle task using question-answer pairs. The training data consisted of puzzles of size  $4 \times 4$ , and the model was subsequently evaluated on larger puzzle sizes for OOD testing.
those datasets based on the depth and width of the computation graph for all the tasks and finetuned on different splits. The results indicate a lack of generalization for out-of-domain (OOD) examples while showcasing near-perfect performance for in-domain examples. One hypothesis on why the model exhibit such a poor generalization is tokenization. So we train GPT2-XL from scratch on up to 4x4 (90M data points), we assign each digit to one token and each math symbol as well. However, the performance is still low and GPT2-XL fails to answer correctly 3x3 test examples.
GPT3 finetuning cost We will discuss here the approximate cost of fine-tuning GPT3 for the multiplication task. When fine-tuning with question-answer pairs, each example typically consists of around 20 tokens, and 250 tokens for question-scratchpad pairs. The cost for utilizing the text-davinci-003 model amounts to  $0.02 (USD) per 1,000 tokens. With this particular setup, the total number of training examples required for multiplication up to 5 digits by 5 digits reaches an astonishing figure of approximately 9.1 billion examples. Should we choose to fine-tune GPT3 for 4 epochs on question-answer pairs, the cost would amount to$ 12 million and $700 million for question-scratchpad training. For a more comprehensive breakdown of the cost per problem size, please refer to Table 1.
Figure 19: GPT3 finetuned on the multiplication task using question-answer pairs
Figure 20: Zero-shot and Few-shot accuracy using question-answer pairs. Performance of GPT4, ChatGPT, and GPT3 on the dynamic programming task. LLaMA and FlanT5 results are near zero for all problem sizes.
B.4 Limits of Transformers with Explicit Scratchpad Training
Figure 23, 24, 22 show the performance of GPT3 fine-tuned on different splits of the tasks using question-scratchpad pairs. Specifically, for the multiplication task, the model was fine-tuned on a range of multiplication problems, spanning from 1-digit by 1-digit multiplication to 3-digit by 2-digit multiplication.
As for the puzzle task, the model was fine-tuned on puzzles of sizes ranging from 2x2 to 4x4. Additionally, for the DP task, the model was fine-tuned on problems with a sequence length of 5. Furthermore, different data splits were considered, including variations based on the number of hours, number of properties, depth and width of the graph, and the number of digits in the multiplication output. On all tasks, we can see that the model fails to generalize to OOD data while achieving perfect accuracy on in-domain
data, indicating that it cannot learn the underlying computational rules.
Figure 23: GPT3 finetuned exhaustively on task-specific data up to a certain problem size. In particular, we train on examples up to 3-digit by 2-digit multiplication (left) and on examples that have up to 5 digits in the output response (right). The blue region represents the in-distribution examples and the red region refers to OOD examples.
B.5 Limits of Transformers with Explicit Scratchpad Prompting
Figure 25 shows the results. GPT-4 exhibits an increase in few-shot accuracy in most problem sizes when using question-scratchpad pairs of few-shot examples across the three tasks. While its performance surpasses that of zero-shot and few-shot with question-answer pairs, it tends to decline as the complexity of the tasks increases. The same applies for the rest of the models.
Figure 21: GPT3 finetuned on the dynamic programming task using question-answer pairs. We consider different data splits: problem size, depth, and width of the graph. Specifically, the model was trained with a problem size of 5, and the graph's depth and width were set to 18.
Figure 22: GPT3 finetuned on the dynamic programming task using question-scratchpad pairs. We consider different data splits: problem size, depth, and width of the graph. Specifically, the model was trained with a problem size of 5, and the graph's depth and width were set to 18.
Figure 24: GPT3 finetuned on the puzzle task using question-scratchpad pairs. The training data consisted of puzzles of size  $4 \times 4$ , and the model was subsequently evaluated on larger puzzle sizes for OOD testing.
Problem size	# examples	GPT3 Cost
without scratchpad	with scratchpad
1 x 1	81	$0.12	$7.44
2 x 1	810	$1.28	$74.4
2 x 2	8100	$12.96	$744
3 x 1	8100	$12.96	$744
3 x 2	81000	$129.6	$7440
3 x 3	810000	$1296	$74,404
4 x 1	81000	$129.6	$7440
4 x 2	810000	$1296	$74,404
4 x 3	8100000	$12,960	$744,040
4 x 4	81000000	$129,600	$7,440,400
5 x 1	810000	$1296	$74,404
5 x 2	8100000	$12,960	$744,040
5 x 3	81000000	$129,600	$7,440,400
5 x 4	810000000	$1,296,000	$70,440,400
5 x 5	8100000000	$12,960,000	$700,440,400
Table 1: Finetuning cost of GPT3 model on the multiplication data.
Figure 25: GPT4 few-shot accuracy with question-scratchpad pairs on the 3 tasks. The performance improves for most problem sizes compared to zero-shot performance and few-shot using question-answer pairs but degrades to zero as the complexity increases.
C Surface Patterns
C.1 Relative Information Gain Predictions for Multiplication
Relative Information Gain
Input variable	Output variable	2x2	3x3	4x4	5x5
xn	z2n	0.223	0.223	0.223	0.223
yn	z2n	0.223	0.223	0.223	0.223
x1	z1	0.198	0.199	0.199	0.199
y1	z1	0.198	0.199	0.199	0.199
xn yn	z2n	1.000	1.000	1.000	1.000
xn-1xn	z2n	0.223	0.223	0.223	0.223
yn-1yn	z2n	0.223	0.223	0.223	0.223
xn yn	z2n-1	0.110	0.101	0.101	0.101
yn-1yn	z2n-1	0.032	0.036	0.036	0.036
xn-1xn	z2n-1	0.032	0.036	0.036	0.036
xn-1yn-1	z2n-1	0.018	0.025	0.025	0.025
x1 y1	z2	0.099	0.088	0.088	0.088
x2 y2	z2	0.025	0.016	0.016	0.016
x1 y1	z1	0.788	0.792	0.793	0.793
y1 y2	z1	0.213	0.211	0.211	0.211
x1 x2	z1	0.213	0.211	0.211	0.211
Table 2: Highest Relative Information Gain Elements and Pairs of Elements, for multiplications between  $x = \left( {{x}_{1},\ldots ,{x}_{n}}\right)$  and  $y = \left( {{y}_{1},\ldots ,{y}_{n}}\right)$  ,with  $2 \leq  n \leq  5$  . We define  $z \mathrel{\text{:=}} x \cdot  y$  ,which will always have size  ${2n}$  (with possibly a leading zero).  ${z}_{2n}$  denotes the least-significant digit of  $z$  ,and  ${z}_{1}$  denotes the left-most digit. Only (input, output) pairs above 0.01 are shown. Note that since multiplication is commutative, several pairs of input variables (e.g.  ${a}_{0}$  and  ${b}_{0}$  ) exhibit the same relative information gain.
C.2 Empirical Surface Pattern Analysis for Multiplication with GPT4, ChatGPT and GPT3
Figure 26: GPT4 zero-shot accuracy in predicting partially correct responses. This evidences surface pattern learning, since the accuracy of full answer prediction is significantly lower—and often near zero (see Figure 2). Specifically, 'accuracy trailing zeros' pertains to accurately predicting the number of zeros in the output number, which is known to be relatively easy to predict based on arithmetic calculations.
Figure 27: ChatGPT zero-shot accuracy in predicting partially correct responses. We observe the same trend for GPT3 predictions.
Figure 28: GPT4 five-shot accuracy in predicting partially correct responses. We observe the same trend for ChatGPT, GPT3 few-shot predictions.
Figure 29: GPT3 finetuned on question-scratchpad pairs. Accuracy of predicting partially correct responses.
C.3 Relative Information Gain Predictions for Dynamic Programming Task
Let  $a_i$  be the  $i$ -th element of the input sequence, and let  $o_i$  be the  $i$ -th element of the output sequence. As shown in Table 3,  $a_i$  is a good predictor of  $o_i$ , and this is especially true for  $a_1$  and  $a_{n-1}$ , the first and last elements of the sequence. This matches the task intuition, since one would never pick an element  $a_i < 0$  and decrease the final sum (one may pick  $a_i = 0$  if it makes a lexicographically smaller output sequence).
$a_{i}$  weakly helps to predict its neighbors. The only case of this behavior with RelativeIG  $>0.1$  is at the start of the sequence, where the first element helps predict the value of the second. This again matches intuition, since a very high  $a_{1}$  indicates that with high probability  $o_2$  will not be selected for the final subsequence.
Relative Information Gain for each problem size
Input variable	Output variable	2	3	4	5	6	7	8	9	10
a1	o2	0.15	0.13	0.14	0.14	0.14	0.14	0.14	0.14	0.14
a1	o1	0.64	0.71	0.69	0.69	0.69	0.69	0.69	0.69	0.69
a2	o2	0.53	0.42	0.45	0.44	0.45	0.44	0.44	0.45	0.44
a3	o3	0.64	0.49	0.53	0.52	0.52	0.52	0.52	0.52
a4	o4	0.60	0.46	0.50	0.49	0.49	0.49	0.49
a5	o5	0.62	0.47	0.51	0.50	0.50	0.50
a6	o6	0.61	0.47	0.51	0.49	0.50
a7	o7	0.61	0.47	0.51	0.50
a8	o8	0.61	0.47	0.51
a9	o9	0.61	0.47
a10	o10	0.61
an-1	on-1	0.64	0.60	0.62	0.61	0.61	0.61	0.61	0.61
an-2	on-2	0.46	0.47	0.47	0.47	0.47	0.47
an-3	on-3	0.51	0.51	0.51	0.51
an-4	on-4	0.49	0.50
Table 3: Highest Relative Information Gain Elements, for DP problems of size  $2 \leq n \leq 10$ . We only show the (input, output) pairs where at least three problem sizes have RelativeIG>0, and at least one with RelativeIG>0.1.  $a_{n-1}$  refers to the last element of the sequence, regardless of its actual id in the sequence.
Similar behaviors, but with higher relative information gains overall, are observed when analyzing triples of consecutive elements in the list. Table 4 shows that  $o_i$  is highly predicted by  $(a_{i-1}, a_i, a_{i+1})$ . Moreover,  $o_i$  is highly predicted by both  $(a_{i-2}, a_{i-1}, a_i)$  and  $(a_i, a_{i+1}, a_{i+2})$ , with the former generally having higher scores than the latter. This again matches the task intuitions, since the value of the neighbors helps determine whether to select a number for the subsequence; and asking for the lexicographically smallest sequence biases the output subsequence to care more about the previous numbers rather than the following ones. We believe that this last point is the cause of the weakly predictive power of  $(a_{i-3}, a_{i-2}, a_{i-1})$  to predict  $o_i$ ; whereas  $(a_{i+1}, a_{i+2}, a_{i+3})$  is not shown, since all the relative information gain values were below 0.1.