SlowGuess/NeurIPS_2023_OCR huggingface dataset:
'print the element at index one of the list'  
'find the second element from the list'  
'retrieve the second element from the list'  
'locate the second item from the list'  
'return the element in second place from the list'  
'reverse the list'  
'mirror the list'  
'flip the list'  
'flip the order of the list'  
'reverse the order of the items in the list'  
'trotate the list elements one place forward'  
'roll the list elements one position to the right'  
'switch the items of the list one position forward'  
'advance the list elements one index forward'  
'move the list elements one position forward'  
'print every other member in the list starting with the first member'  
'find alternate elements in the list beginning with the first element'  
'print every second item in the list, starting with the first element'  
'output every second element in the list, starting from the first element'  
'output even indexed elements'
Table 2: Natural language instructions for the list algorithms used in the LIALT dataset
‘return the matrix diagonal’
‘collect the diagonal values of the matrix’
‘retrieve the diagonal elements of the matrix’
‘return the diagonal entries of the matrix’
‘fetch the diagonal items of the matrix’
‘roll the columns of the matrix to the right’
‘rotate the matrix columns to the right’
‘move the matrix columns to the right’
‘shift the columns of the matrix to the right’
‘spin the matrix columns to the right’
"return the matrix transpose"   
"retrieve the transpose of the matrix'   
"get the transposed matrix'   
"compute the transposed form of the matrix'   
"derive the transpose matrix'   
"find the matrix element in the second row and second column'   
"find the value in the second row and second column of the matrix'   
"fetch the matrix element located in row 2 and column 2"   
"print the value at 2 2 in the matrix'   
"retrieve the matrix element at 2 2'
Table 3: Natural language instructions for the matrix algorithms used in the LIALT dataset
D.2 Learned CSCG model
Our next Figure 10 displays the transition graph of the CSCG model trained on the LIALT dataset with an overallocation ratio of 3, with stacked clones for each symbol.
Figure 10: CSCG model learned on the LIALT dataset, visualized with stacked clones.
D.3 Results on the LIALT dataset
D.3.1 After a single EM iteration
Presented below are the tables of results associated with Fig. 5. Table 4 contains the in-context accuracies averaged on the entire test set, Table 5 contains the in-context accuracies per task on instructions-based prompts, and Table 6 contains the in-context accuracies per task on example-based prompts.
Overallocation ratio	Instruction-based prompts	Example-based prompts
0.1	0.00 (0.00)	0.00 (0.00)
0.3	0.20 (0.08)	0.09 (0.06)
1.0	0.54 (0.10)	0.49 (0.10)
3.0	0.89 (0.06)	0.91 (0.06)
Table 4: Average in-context accuracy of each CSCG model—with  $95\%$  confidence intervals—as a function of CSCG overallocation on both (a) the instruction-based LILAT test set and (b) the example-based LIALT test set.
Task	Overalllocation ratio
0.1	0.3	1.0	3.0
list 1st elem.	0.00 (0.00)	0.00 (0.00)	0.89 (0.10)	1.00 (0.00)
list 2nd elem.	0.00 (0.00)	0.60 (0.15)	0.70 (0.14)	1.00 (0.00)
list 3rd elem.	0.00 (0.00)	0.60 (0.15)	0.60 (0.15)	1.00 (0.00)
list reverse	0.00 (0.00)	0.00 (0.00)	0.70 (0.14)	1.00 (0.00)
list repeat twice	0.00 (0.00)	0.00 (0.00)	0.50 (0.25)	0.75 (0.22)
list alt. even	0.00 (0.00)	0.00 (0.00)	0.50 (0.20)	1.00 (0.00)
list alt. odd	0.00 (0.00)	0.00 (0.00)	0.71 (0.17)	0.86 (0.13)
list circ. shift fw.	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.56 (0.17)
list circ. shift bw.	0.00 (0.00)	0.00 (0.00)	0.14 (0.13)	1.00 (0.00)
matrix diagonal	0.00 (0.00)	0.50 (0.18)	0.50 (0.18)	1.00 (0.00)
matrix transpose	0.00 (0.00)	0.00 (0.00)	0.50 (0.20)	1.00 (0.00)
matrix roll columns	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.17 (0.15)
matrix elem. at idx.	0.00 (0.00)	0.50 (0.18)	1.00 (0.00)	1.00 (0.00)
Table 5: Average in-context accuracy by task—with standard errors—as a function of CSCG overallocation on instruction-based prompts.
Task	Overallocation ratio
0.1	0.3	1.0	3.0
list 1st elem.	0.00 (0.00)	0.00 (0.00)	1.00 (0.00)	1.00 (0.00)
list 2nd elem.	0.00 (0.00)	0.12 (0.12)	1.00 (0.00)	1.00 (0.00)
list 3rd elem.	0.00 (0.00)	0.38 (0.17)	1.00 (0.00)	1.00 (0.00)
list reverse	0.00 (0.00)	0.00 (0.00)	0.12 (0.12)	0.88 (0.12)
list repeat twice	0.00 (0.00)	0.00 (0.00)	0.50 (0.18)	0.62 (0.17)
list alt. even	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	1.00 (0.00)
list alt. odd	0.00 (0.00)	0.00 (0.00)	0.38 (0.17)	1.00 (0.00)
list circ. shift fw.	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.88 (0.12)
list circ. shift bw.	0.00 (0.00)	0.00 (0.00)	0.29 (0.17)	1.00 (0.00)
matrix diagonal	0.00 (0.00)	0.33 (0.19)	0.67 (0.19)	1.00 (0.00)
matrix transpose	0.00 (0.00)	0.00 (0.00)	0.43 (0.19)	1.00 (0.00)
matrix roll columns	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.50 (0.18)
matrix elem. at idx.	0.00 (0.00)	0.38 (0.17)	1.00 (0.00)	1.00 (0.00)
Table 6: Average in-context accuracy by task—with standard errors—as a function of CSCG overallocation on example-based prompts.
D.3.2 After EM convergence
Figs. 11 & 12 present analogues of Figs. 5 & 6 but after the EM algorithm in the rebinding process has converged. We note that the results are mostly identical. Table 7 contains the in-context accuracies averaged on the entire test set, Table 8 contains the in-context accuracies per task on instructions-based prompts, and Table 9 contains the in-context accuracies per task on example-based prompts.
Figure 11: [Left] In-context accuracy (with  $95\%$  CIs) after EM convergence, as a function of the overallocation ratio for a CSCG trained on LIALT and averaged [top] on the instruction-based LIALT test set [bottom] on the example-based LIALT test set. In-context accuracy increases for CSCGs with larger capacities. [Right] In-context accuracy (with standard errors) per task on the two LIALT test sets: for each task, overparametrization improves performance. All the numerical values are in Appendix D.3. Invisible bars indicate zero accuracy for the respective combination of model and task.
Overallocation ratio	Instruction-based prompts	Example-based prompts
0.1	0.00 (0.00)	0.00 (0.00)
0.3	0.16 (0.07)	0.11 (0.06)
1.0	0.54 (0.10)	0.49 (0.10)
3.0	0.89 (0.06)	0.93 (0.05)
Table 7: Average in-context accuracy of each CSCG model—with  $95\%$  confidence intervals—as a function of CSCG overallocation on both (a) the instruction-based LILAT test set and (b) the example-based LIALT test set.
Task	Overallocation ratio
0.1	0.3	1.0	3.0
list 1st elem.	0.00 (0.00)	0.00 (0.00)	0.89 (0.10)	1.00 (0.00)
list 2nd elem.	0.00 (0.00)	0.60 (0.15)	0.70 (0.14)	1.00 (0.00)
list 3rd elem.	0.00 (0.00)	0.60 (0.15)	0.60 (0.15)	1.00 (0.00)
list reverse	0.00 (0.00)	0.00 (0.00)	0.70 (0.14)	1.00 (0.00)
list repeat twice	0.00 (0.00)	0.00 (0.00)	0.50 (0.25)	0.75 (0.22)
list alt. even	0.00 (0.00)	0.00 (0.00)	0.50 (0.20)	1.00 (0.00)
list alt. odd	0.00 (0.00)	0.00 (0.00)	0.71 (0.17)	0.86 (0.13)
list circ. shift fw.	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.56 (0.17)
list circ. shift bw.	0.00 (0.00)	0.00 (0.00)	0.14 (0.13)	1.00 (0.00)
matrix diagonal	0.00 (0.00)	0.00 (0.00)	0.50 (0.18)	1.00 (0.00)
matrix transpose	0.00 (0.00)	0.00 (0.00)	0.50 (0.20)	1.00 (0.00)
matrix roll columns	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.17 (0.15)
matrix elem. at idx.	0.00 (0.00)	0.50 (0.18)	1.00 (0.00)	1.00 (0.00)
Table 8: Average in-context accuracy by task—with standard errors—as a function of CSCG overallocation on instruction-based prompts.
Task	Overallocation ratio
0.1	0.3	1.0	3.0
list 1st elem.	0.00 (0.00)	0.00 (0.00)	1.00 (0.00)	1.00 (0.00)
list 2nd elem.	0.00 (0.00)	0.12 (0.12)	1.00 (0.00)	1.00 (0.00)
list 3rd elem.	0.00 (0.00)	0.38 (0.17)	1.00 (0.00)	1.00 (0.00)
list reverse	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.88 (0.12)
list repeat twice	0.00 (0.00)	0.00 (0.00)	0.50 (0.18)	0.88 (0.12)
list alt. even	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	1.00 (0.00)
list alt. odd	0.00 (0.00)	0.00 (0.00)	0.50 (0.18)	1.00 (0.00)
list circ. shift fw.	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.88 (0.12)
list circ. shift bw.	0.00 (0.00)	0.00 (0.00)	0.29 (0.17)	1.00 (0.00)
matrix diagonal	0.00 (0.00)	0.67 (0.19)	0.67 (0.19)	1.00 (0.00)
matrix transpose	0.00 (0.00)	0.00 (0.00)	0.43 (0.19)	1.00 (0.00)
matrix roll columns	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.50 (0.18)
matrix elem. at idx.	0.00 (0.00)	0.38 (0.17)	1.00 (0.00)	1.00 (0.00)
Table 9: Average in-context accuracy by task—with standard errors—as a function of CSCG overallocation on example-based prompts.
Figure 12: Extension of Fig. 6 to EM convergence. A. Visualizing the inferred probability of the observation at timestep  $n$ , conditioned on observations at all other timesteps, before rebinding. This drives the identification of anchors and slots selected for rebinding. B. Decoded latent state distributions — and predicted prompt completions — for the two different example-based LIALT prompts specified in subfig. A: (top) before rebinding, (middle) after one iteration of EM, and (bottom) after EM convergence. The left prompt corresponds to the operation of circularly shifting the list forward, and the right prompt corresponds to reversing the list.
D.4 Example failures
Finally, we present a few examples which illustrate the failure modes of our approach. These are primarily a consequence of imperfections in the learned CSCG model. Each example is presented in the format (prompt, ground truth correct output, actual model response).
1. For these failures, the instruction circuit has been wired to the wrong algorithm circuit (possibly driven by the ambiguity of the forward slash delimiter separating the instruction from the example), resulting in the retrieval of the wrong schema.
- output odd indexed elements / [UVBQKI]  
[UBK] /  
[VQI] /  
- flip the list / [SEJ]  
[JES] /  
[SSEEJJ]  
- reverse the list / [RTB]  
[BTR] /  
[RRTTBB] /  
- mirror the list / [BAOT]  
[TOAB] /  
[BBAAOOTT] /
2. For these failures, the schema has been learned incorrectly.
- switch the items of the list one position forward / [LNG XMT]  
[TLNGXM] /  
[TLNGXMT] [TLNGXMT] ...
- shift the columns of the matrix to the right / [[D Y] [V F]]  
[ [Y D] [F V]] /  
[ [get
$\cdot /[\mathrm{ZJB}][\mathrm{ZZJJBB}]/[\mathrm{BAEFWL}]$  [BBAAEEFFWWLL] /[BBAEFFWWLL]
$\cdot /[\mathrm{VPXT}][\mathrm{PT}] / [\mathrm{VFJPEW}]$  [FPW]/ [FPW][FPW]…