Mastering Symbolic Operations: Augmenting Language Models with Compiled Neural Networks
Authors: Yixuan Weng
Minjun Zhu
Fei Xia
Bin Li
Shizhu He
Kang Liu
Jun Zhao
Venue: ICLR 2024 poster (2024)
Keywords: Language Models
Compiled Neural Networks
Neural Comprehension
Symbolic Operations
Length Generalization
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We have enabled language models to more fundamental comprehension of the rule, to achieve completely absolute accuracy in symbolic operations without additional tools.

Abstract: Language models' (LMs) proficiency in handling deterministic symbolic reasoning and rule-based tasks remains limited due to their dependency implicit learning on textual data. To endow LMs with genuine rule comprehension abilities, we propose "Neural Comprehension" - a framework that synergistically integrates compiled neural networks (CoNNs) into the standard transformer architecture. CoNNs are neural modules designed to explicitly encode rules through artificially generated attention weights. By incorporating CoNN modules, the Neural Comprehension framework enables LMs to accurately and robustly execute rule-intensive symbolic tasks. Extensive experiments demonstrate the superiority of our approach over existing techniques in terms of length generalization, efficiency, and interpretability for symbolic operations. Furthermore, it can be applied to LMs across different model scales, outperforming tool-calling methods in arithmetic reasoning tasks while maintaining superior inference efficiency. Our work highlights the potential of seamlessly unifying explicit rule learning via CoNNs and implicit pattern learning in LMs, paving the way for true symbolic comprehension capabilities. The code is released at: \url{https://github.com/wengsyx/Neural-Comprehension}.

Published as a conference paper at ICLR 2024
MASTERING SYMBOLIC OPERATIONS: AUGMENTING LANGUAGE MODELS WITH COMPILED NEURAL NETWORKS
Yixuan Weng 1. Minjun Zhu Fei Xia Bin Li", Shizhu Hc Kang Liu Jun Zhaok-2
The Laboratory of Cognition and Decision Intelligence for Complex Systems, IA, CAS
2 School of Artificial Intelligence. University of Chinese Academy of Sciences
3 College of Electrical and Information Engineering, Hunan University wengsyx@gmail com, ushizhu.her kliu, jzhao) @ener. ac.
ABSTRACT
Language models' (LMs) proficiency in handling deterministic symbolic rea- soning and rule-based tasks remains limited due to their dependency implicit learning on textual data. To endow LMs with genuine rule comprehension abil- ities, we propose "Neural Comprehension" a framework that synergistically integrates compiled neural networks (CoNNs) into the standard transformer archi- tecture. CoNNs are neural modules designed to explicitly encode rules through artificially generated attention weights. By incorporating CoNN modules, the Neural Comprehension framework enables LMs to accurately and robustly ex- ecute rule-intensive symbolic tasks. Extensive experiments demonstrate the periority of our approach over existing techniques in terms of length general- ization, efficiency, and interpretability for symbolic operations. Furthermore, it can be applied to LMs across different model scales, outperforming tool- calling methods in arithmetic reasoning tasks while maintaining superior in- ference efficiency. Our work highlights the potential of seamlessly unifying explicit rule learning via CoNNs and implicit pattern learning in LMs, paving the way for true symbolic comprehension capabilities. The code is released at: https:/ / thub. com/ http://wwgggw//ieeeeeeeee Comprehenssiee
INTRODUCTION
Language models (LMs), particularly large language models (LLMs), have exhibited impressive performance on complex rea- soning tasks (Brown al., 2020; Zhang
In-Dist
Out-of-Dist
100
80
Solve Rate (N)
et al., 2022; Chowdhery et al., 2022; Wei al., 2022a; Suzgun et al., 2022). De- spite this, the proficiency of LMs in tack- ling deterministic symbolic reasoning and rule-based tasks is still limited (Welleck et al.; Razeghi et al., 2022). For example, GPT-3's arithmetic performance declines with higher digit numbers (Brown et al., 2020), and its mathematical accuracy is in- fluenced by word frequency in training data (Razeghi al., 2022). Moreover, length generalization (Anil et al., 2022) remains challenge even for 100-billion parrmmeer models, such as GPT-4 (Bubeck et al., 2023). We hypothesize that these limitations stem from the dependency of LMs on implicitly learning rules from textual data. As illustrated in Figure 1, a simple length generalization experiment using addition tasks with varying numbers of digits highlights this limi- tation. Performance deteriorates test length increases, indicating that these models strongly rely on statistical patterns in the data rather than capturing fundamental logical structures. This implicit learning of statistical patterns constrains the accuracy of LMs in executing symbolic operations tasks.
40-
20
T5-bess
70
20
Enter Text Length (Digts)
Figure 1: The length generalization of T5 (with fine- tune), GPT-3.5 and GPT-4 (with few-shot) on symbolic operations (Addition) tasks. To evaluate the model's proficiency, we conducted experiments on tasks ranging from 3 30 digits, with longer than 10 digits being out-of-diittibutiion of training data.
"These authors contribute equally this work. And corresponding author.

---

Published as a conference paper at ICLR 2024
We propose transformer-bassee language model framework, termed "Neural Comprehension", which synergistically integrates a pretrained LM (Li et al., 2021b) and compiled neural networks (CoNNs) (Weiss et al., 2021), combines their complementary strengths in plug-and ald-play manner, to achieve high accuracy and robust performance. CoNNs are neural networks but the rules are explicitly coded through transformer-liked structures and attention. Therefore, the CoNN is human-controllable. executing rules through artificially generated attention weights, and can achieve perfect accuracy once compiled network done. Neural Comprehension relys solely on neural networks without requiring additional tools. It employs token-by- beten-by-token generative method, analogous to GPT-3, where each token can be generated by either the pretrained LM or one of the CoNNs. The Neural Comprehension comprises a pretrained LM and multiple sets of CoNNs. The implementation of the Neural Comprehension framework facilitates the integration of rule-intensive abilities and reasoning capabilities into LMs, endowing them with genuine symbolic comprehension skills.
We conduct extensive experiments to evaluate the performance of our proposed Neural Compre- hension method on a variety of rule-intensive tasks. Our experimental results demonstrate the effectiveness of our approach in comparison with existing state-of thee--rrt techniques, such as vanilla fine-tuning, few-shot learning, and Chain-of- Chain-of-Thouuhtt reasoning (Wei et al., 2022b). Specifically, Neural Comprehension outperforms these methods in terms of accuracy, efficiency, and interpretabil- ity, showcasing its superiority in handling rule-intensive tasks. On the other hand, compared to the Tool-Based method (Mialon et al., 2023), Neural Comprehension provides a unified end-to-end neural network framework, eliminating the need for external interpreters and having higher inference efficiency. Historically, LMs are far from mastering robust symbolic task, such as symbolic operations and arithmetic reasoning (Stolfo al., 2023). Our research provides a compelling case for LMs in neural network frameworks mastering symbolic operations, highlighting its potential to transform the landscape of symbolic reasoning and numerical computation capabilities for LMs.
Contributions Our main contributions are as follows:
We pioneer the implementation of flawless execution rule-intensive symbolic operations for language models that rely on neural networks. By employing successfully integrate CoNNs, which are interpretable and human-come contolllontolllabee into the language model. Our method facilitates direct rule deduction without the need for learning from conditional probabilities, leading to a more robust and effective approach. (Section 4)
aong-and-paay way. we
We have built the AutoCoNN toolkit to make Neural Comprehension scalable, which leverages LLMs' contextual learning capabilities to automatically generate new CoNNs. Our method can be easily extended to various symbolic operations tasks. (Seciton 4.3)
Our experimental results symbolic tasks and real-world arithmetic reasoning tasks demonstrate the superior performance of our method in comparison to existing techniques. Notably, our LM achieves flawless execution on symbolic reasoning tasks. (Section 5.1 5.2)
It is worth noting that tool-based methods are only applicable to language models with code generation capabilities and require the cooperation of external interpreters. Our experiments demonstrate that the symbolic processing capabilities of neural understanding are on par with tool-based methods, but are applicable models ranging from small ones like T5-Small (60M) to large ones like GLLM-10B (130B) and GPT-3.5 (175B). (Section 5.3)
We also studied the potential of combining multiple CoNNs and found that adding correlated CoNNs can continuously increase performance, while adding uncorrelated CoNNs rarely leads to performance degradation. This provides new approach for model fusion, enabling the model to easily acquire new knowledge. (Section 5.4)
RELATED WORKS
Pretrained Language Models encompass those trained on general-purposs corpora (Lewis et al., 2019; Scao et al., 2022; Sun et al., 2022) and specialized symbolic tasks (Geva et al., 2020; Lewkowycz et al., 2022; Yang et al., 2023). They primarily aim to capture statistical patterns in language, which limits their capacity for symbolic reasoning. Symbolic reasoning involves ma- nipulating abstract symbols and logical rules to derive new knowledge (Shindo et al., 2021; Yang & Deng, 2021) and necessitates the ability extrapolate to novel situations and reason about concepts
2

---

Published as a conference paper at ICLR 2024
absent in the training data (Fujisawa & Kanai, 2022). Due to the constraints of gradient learning, LMs face challenges in wholly solving symbolic problems (Stolfo et al., 2023).
In-Context Learning has emerged promising approach to address these challenges (Dong et al., 2022) and closely approximate the predictors computed by gradient descent (Akyürek et al., 2022). By prompting the language model to generate explanation before generating an answer, the chain of thought (Wei et al., 2022b; Kojima et al., 2022; Zhang et al., 2023; Zhou et al., 2022a) encourages the model to think sequentially. This technique has been employed in various numerical and symbolic reasoning tasks, such as scratchpad prompting (Nye et al., 2021) for length generalization (Anil al., 2022) and utilizing the chain of thought to perform arithmetic operations like summing pairs of single digits with carry (Zhou et al., 2022b). However, this approach often necessitates substantial computational resources, and achieving perfect accuracy remains challenging.
Augmented Language Models have been proposed as an alternative, supplementing language models with external tools (Mialon et al., 2023). Examples include generating Python code for numerical reasoning (Gao et al., 2022; Chen et al., 2022) incorporating tool usage as a pretraining task (Schick et al., 2023). However, using external tools lacks unified framework with language models and instead relies on the normativity of program generation.
3
PRELIMINARIES
Example
The Parity CoNN
If we want the transformer to perform the Parity task full accurately: Input:
0
] [ 1
1. Seleci(Indices. Inw.:/ideeees..IiesssiIiee =
1 (l:
2. Aggregate
0] = 1]
1
3. Zipmap((1 Output: [1 1]
Lambda a:: 0 if %2 0 1) = [[ 1]
*
Figure 2: Demonstration of the principles of Parity CoNN.
Compiled Neural Network (CoNN). CoNN transformer-basee neural network leveraging artificially compiled attention weights to execute rules. CoNN has multiple attention layers and Multi- Layer Perceptron (MLP) layers, and each attention layer facilitates interactions between tokens. For example, in Figure 2, the multiplication of query and key elements representing "Select" operation in CoNN. Subsequent multiplication with value elements indicates an "Aggregate" operation. The MLP layer is responsible for the token itself and is referred to the "Zipmap" operation (Weiss et al., 2021). By utilizing the three operations (Select, Aggregate, and Zipmap) to represent the sequence- to-sequence process,
convert symbolic parity task into transformer weights (Lindner et al., 2023). CoNN can also stack multiple layers to address various human-defined rule problems, such as mathematical calculations and symbol operations 1 Figure 2 shows an example of Parity CoNN: The first step is to obtain a matrix of all ones to that of the sequence using the "Select" operation. Secondly. the "Aggregate" " operation is used combine the matrix obtained in the previous step with the input sequence (with the aim of calculating the total number of (I's and I's in the sequence). The third step involves determining whether the total count odd or even by "Zipmap".
METHOD
Language models excel in language understanding tasks, but lack robust capabilities for symbolic tasks. We propose a Neural Comprehension framework that make CoNN guided by abstract rules into language models in a blug-and-pla fashion, which integrates the language model's implicit learning parameters and CoNNs' explicit learning parameters. In Neural Comprehension, we designed CoNNs in neural networks with weights and structures directly encoding the symbolic rules within the standard architecture of the LM to enhance deterministic rule comprehension and allow for deterministic execution of rule-based tasks.
Appendix provides more detailed description of CoNN.
3

---

Published as a conference paper at ICLR 2024
230562
147843
meter
Input
LACON
UVCON
UNCON
LMCON
238582
147862
Pre-training Neural Network
Compiled Neural Network
The WWatc show that Stanley ron 364425 meters and wilked 236582 meters month, How much forther did Stanicy run than wale
Input
HUMM
To find out how much forther Stanicy than walked, we need to suborctt the dittance walked from the distance ran
Output
[664225 206582 E 147843) meters. Therefore, Stanley 147845 meter Tarther Than he saileed.
Language Models with Compiled Neural Networks
Figure 3: The architecture of the proposed Neural Comprehension framework.
4.1 NEURAL COMPREHENSION
Neural Comprehension is MoF-style (Shazeer et al., 2017) neural network framework we designed, which is entirely composed neural networks and maintains the generative seq2seq style. It uses predefined CoNNs gating to determine when to output results from CoNNs. This approach is simple and plug-and-play allowing combination with pretrained LMs. As illustrated in Figure 3, the language model encodes the context and produces the textual and reasoning process context D(x) step by step, a decoder architecture to generate the subsequent context step by step iteratively. And CoNNs handle sequence transformations involving rules, when rule-required operation emerges, CoNN's attention is utilized to calculate specific values. For example, in Figure 3, when calculating 364425-216588 the only pretrained language model may output 148843, which is incorrect. However, the Subt centtaaciion ion CoNN can correct the result to 147843 in the neural comprehension framework. This dynamically encoded process improves intermediate results interpretability and final result accuracy. Neural Comprehension combines LM and CoNNs in piecewise function to perform
gradient update. LM's hidden state output is H (HL,". .... (.LH,) ) R2 HL, E (0, 1), and
CoNN's output is He (Hc. ..ooo IRdo, Hc, E {0,
0 = argmaax [( 0,,3 ) H1. Hc )}
B E 10
(I)
where He is a one-hot vector, and de and do here refer to the vocabulary size of the Model's decode output. The Neural Comprehension combines the LLM's hidden state output, Hi, and CoNN's output, Hc. using identity matrices Idz (for dz) and Iae (for de) to concatenate them for model fusion. Specifically, the hidden state representation matrix obtained by extending the original hidden state representation matrix of the LM with the hidden state matrix on the CoNNs' vocabulary through a block matrix operation, resulting in larger matrix.
4.2 GRADIENT MODIFICATION IN NEURAL COMPREHENSION
To better appreciate the benefits of our method in handling rule-intensive tasks and improving accuracy, is crucial to understand the gradient perspective of In-Context Learning (ICL). Recent studies on ICL. algorithms have shown that the learning process of language models within the optimization process in ICL can be viewed as a search for suitable gradients to minimize the loss
It is worth noting that in this paper, for ease of implementation. the output vocabulary size of CoNNs' decode de, is generally less than 100 due to limitations in computing resources (detailed information is shown in Appendix Table 1).
4

---

Published as a conference paper at ICLR 2024
function. (Garg et al., 2022; Von Oswald et al., 2023). Due to the implicit learning nature of standard ICL methods, gradients learned from data may not always be ideal for addressing rule-intensive tasks. Therefore, proposed method introduces appropriate gradient updates for such tasks, ultimately leading to enhanced overall performance. We focus elucidating the changes in the gradient introduced by the Neural Comprehension model, the gradient of the model during the execution of ICL can be partitioned into two categories based on the origin of the gradients:
explicit learning component to provide more
{ Gr G
Text: Language Model
Gradient
Rule: CoNNs
Here, GT represents the gradients derived implicitly from the language model (LM) and corresponds to the text-based learning aspect of the model. Conversely, GR represents the gradients explicitly derived from the CoNNs, encoding rule-based knowledge. The specific computation process can be seen in Equation 1. Note that the gradients' decomposition is only approximate and may not reflect the true generating process of text. The Neural Comprehension framework integrates both gradient sources to optimize the ICL process. In linear regression problems (Akyürek et al., 2022), the loss function can be expressed as a piecewise function according Equation I, here Pi (x) is the LM and Pa(x)) is CoNN, the In In-context-learee can be separate into two process:
L = 1 3 x=
(3)
y G7 1
E Pz) x P2(x)
G|
Based on the partitioned gradient as defined in Equation 2, the overall gradient of the Neural Comprehension model can be obtained by computing their individual gradients concerning the respective 3:
aL 03
8L OC
E (z)
Ot DGR
x (2
Gradient
This partitioning allows the Neural Comprehension model to specifically address the gradient require- ments of both implicit learning via LM and explicit learning via CoNNs. It is crucial to note that CoNNs are designed to minimize the loss associated with rule-based tasks, essentially providing an optimal gradient for tasks involving rule-intensive operations. This leads to a substantial improvement in the model's accuracy for rule-based tasks, as the gradient updates provided by CoNNs are more suitable for rule learning compared to the initially available gradients from the LM. By amalgamating both of gradient sources, the Neural Comprehension model achieves a more refined optimization of in-context learning. The Neural Comprehension model effectively balances the need for implicit and explicit learning within the ICL framework, leading to enhanced overall performance in terms of accuracy and interpretability.
4.3 AUTOCONN TOOLKIT
To improve the scalability of Neural Comprehension frameworks, propose the AutoCoNN toolkit, which can automatically generate new CoNNs and adapt Neural Comprehension given by "Instruct" and "Example", where "Instruct" serves as explicit symbolic definitions, and "Example" provides some input-output pairs for the operation. Considering that LLMs like GPT-4 can describe symbolic reasoning processes but cannot faithfully execute them (Cai et al., 2023). AutoCoNN Toolkit enables converting the reasoning process of symbols into CoNNs while maintaining a complete neural network framework without extra interpreters. The AutoCoNN process is divided into three steps. First, we provide 24 Tracr code (Lindner et al., 2023) examples as context. Then the LLM is asked to generate 20 different Tracr codes by sampling decoding based on the "Instruct" and "Example". Finally, we convert these codes into pytorch-form (Paszke et al., 2019) CoNNs and filter them on the pre- provided Examples to obtain accurate CoNNs. We provide further experimental analysis to test the performance of AutoCoNN in Appendix D. Mcanwhile, the Parity, Reverse, Copy and Last Letter CoNN mentioned the Section 5 are constructed by AutoCoNN, which demonstrates the practical value of AutoCoNN.
5

---

Published as a conference paper at ICLR 2024
5
EXPERIMENTS AND RESULT
5.1
SYMBOLIC TASKS
Out-ct-Ost
Dis
2-d
ou Dist
Out-ot-sst 5-
InOst 5-
Dut-ofDet
100
100
80
80
20120 N30 02
Solve Rate 100
40
40
20
20
Streel Schas
01-13
R.M
( Pty
(8) Rev
-Out of D
Dist
Out of Dist
<Out of Digt-
Int
Out Det
100
100
80
80-
30:10 Noto 00
60
40
Do Sate
40-
20
0
6P1-3
(PT-3.51 Morel Eopprshers
13 Phreaee
27-
(0) Addi ti
Figure 4: Comparison of Neural Comprehension and other implicit learning-based methods in symbolic operations tasks test length generalization performance. In this, the T5 model uses the Vanilla Fine tune method for learning, and LLMs use the Few-shot learning method. Neural Comprehension, each task has different CoNN, namely P Parity, Reverse, Addi ton,, and Subt rebbtractoo ion.
Techniques
In-distribution
Out-of-distribattio
Time and Space Complexity
Interpretability
Vanilla Fine-tune (For LM)
Vanilla Few-shot (For LLM)
Scratchpad (Anil et al,, 2022)
X
Algorithmic (Zhou et al., 2022h)
x
Neural Comprehension (Ours)
//
Table 1: Performance on Symbolic operations tasks of five techniques that language models admit: (1) Vanilla Finetuning, (2) Vanilla Few-shot, (3) Scratchpad (Chain-of-Thouur reasoning). (4) Algorithmic (Chain-of- Thought reasoning) and (5) Neural Comprehension. We find that the first four learning-based methods have different modes failure regarding and out-of-distribution coverage for symbolic operations. However, Neural Comprehension has strong advantages in terms of length generalization, efficiency, and interpretability. X signities poor / signifies nontrivial, // signifies near-perfeet performance. (*) Refers to task -ekedeeendency.
We conduct a length generalization experiment (Anil et al., 2022) to examine the distinctions between the Neural Comprehension and bearning-based methods, depicted in Figure 4. Our experimental design encompasses 1000 x 40 independent test sets, comprising problems with varying digit lengths from to 40 digits. 10 to 20 digits within the range are provided by us for methods based on implicit learning for training; during the testing phase, this range is called In-Dist. Furthermore, we present results for both Scratchpad (Anil et al., 2022) and Algorithmic (Zhou al., 2022b) approaches.
The results of our experiment demonstrate that the Vanilla Fine-tune (red lines) method performs optimally on the in-domain (10-20 digit) training set, while its performance deteriorates for both more simplistic and more intricate. This finding suggests that the absence of relevant samples in the training set may cause gradient descent- based language models to underperform on both simpler and more complex tasks. As further discussed in the appendix D.1, this phenomenon can be attributed to the inherent generalization limitations of statistical models and the position bias language models.
Considering the Vanilla Few-shot method (green lines), we determine that its performance is less impacted by the prompt sample range compared to Vanilla Fine-tune. Large language models, which trained on extensive text corpora, excel at solving more straightforward problems such as symbolic operations within a ten-digit range. Nevertheless, performance remains below par for test sets with more than ten digits, even when prompted with 10-20 digit samples.

---

Published as a conference paper at ICLR 2024
Observing CoT-like methods (we use GPT-3.5). including Scratchpad and Algorithmic, unveils their robust length generalization capabilities. Scratchpad works by requiring large language models to record intermediate steps, while Algorithmic employs similar approach to record the carry operations involved in the addition process. This can be primarily attributed to their proficiency in decomposing complex problems into smaller incremental steps and maintaining intermediate states. However, these methods necessitate substantial computational resources, and extending the length beyond the input limit of the model becomes challenging.
Our study reveals that Neural Comprehension attains remarkably high accuracy in symbolic operations. This implies that Neural Comprehension, unlike conventional methods, does not rely on training data and remains unaffected by discrepancies in input lengths for in-distribution and out-of out-of-distribution data. Consequently, it alleviates the requirement for step-by-step work tracking, and language models with CoNNs only need relatively fewer computational steps to execute sequence operations directly. Encoding rules into neural network modules endows us with greater interpretability, enabling language models to flawlessly perform purely symbolic operation tasks.
5.2 SYMBOLIC REASONING
In this section, we investigate the performance of Neural Comprehen- sion in terms of symbolic reasoning capabilities. Our hypothesis is that, although pretrained Language Mod- els (LMs) demonstrate strong lan- guage understanding abilities, they lack the capacity to deduce and com- prehend rules regarding symbolic reasoning tasks. Thus, aim to evaluate whether the incorporation of compiled neural networks in the form of CoNNs can address this lim-
Vanilla Fine tne-tune
Direct Fine-ture Improve Performance
Neural Comprehension
T5 5mall:60M
T5 Base:220M
T5 Large:770M
50%
80%
50%
Flip
40%-
40%
40%
0%
06:
12C
13K
24K
01
0C
12 Keraliona
10K
24c
05%
0C 13K
13K
24k
[sural om5
Last Letter Concatenation
itation and improve the LM's sym- bolic reasoning abilities.
85%
00%
80%
To assess the performance of the rule comprehension component (CoNNs) in symbolic reasoning, we devise an experiment that measures the model's accuracy using interme- diate processes and represents them in a "Chain of Thought -like manner. In doing so, the experiment decom- poses language understanding and rule comprehension explicitly into simpler outputs, avoiding the com- plexities of reasoning and additional error propagation in the models. Ex- ample outputs from this approach can be found in Appendix F. We observed that neural compre- hension improves the symbolic reasoning capabilities of pretrained language models in most cases (Neural Comprehension almost always outperforms Vanilla Fine-tune in Figure 5), and can fit faster. This observation suggests that the introduction of compiled neural networks has positive impact on pretrained LMs, addressing rule comprehension limitations in symbolic reasoning tasks.
40%
40%
40%
0%
16.
12c 13C
24K
0%
06
12.6
24c
0%
6C 12K
13K
24K
Tsaral
Aerallons
Figure 5: In the iterative process of gradient descent during training. The bleu line represents language model that in- corporates neural comprehension, and the red line represents the original language model. Additionally, provide Direct, which is a direct prediction of the final result, as a reference.
5.3 ARITIMETIC REASONING
Arithmetic reasoning serves suitable testbed for evaluating language models and their ability to address real-world problems. In this study, we examine the AddSub+ dataset variants that involve different digit lengths, utilizing the Addition and Subtract on models from the CoNNs family. Notably, the capabilities of Neural Comprehension extend beyond these tasks, as CoNNs can also simulate calculators that support multiplication and division operations, and potentially perform
7

---

Published as a conference paper at ICLR 2024
Vani l1 GoT
PAL
Neural Comprahens lon
Improve Per forforennc
GLM: 130B
100
100
100
80
80
80
Rata
Rate
40
40-
40-
20
20
The number digits
The number of digits
The Nurber Digite
Figure 6: We conducted simulations of the AddSub dataset with varying digits by modifying the "lEquations" parameter. We then tested the performance of three LLMs with and without Neural Comprehension generating CoT outputs for AddSub*, And we reported the solve rates of three LLMs and compared the solve rates of using additional tools (PAL (Gao et al., 2022)).
Kadition
Bomu-1-76
1ama-277
GLM--NRR
Arg
Subiraction
Bomn--7hh
Bamra-2700
GEA--100
Agg
Mitedd
Iaama-377h
Bama---780
GLM--136
PAL
11.
14.1
274
14.3
NC (Ous))
12.5
438
7.
21.3
NC (Can
324
45
15A
NC (Oun)
278
353
86
21.
Table 2: Experiments on the addition and subtraction subset for GSM8K- -aar dataset showing performance comparisons across different models and methods: Only Addition (left), Only Subtrac- tion (center), and Mixed addition and subtraction (right), using Vanilla CoT, PAL, and NC (Neural Comprehension) methods.
linear algebra computations or even in-context learning algorithms that employ backpropagation (Giannou et al., 2023).
To evaluate the impact of Neural Comprehension on arithmetic reasoning, we compare the output of vanilla CoT language models and those incorporating Neural Comprehension, using the vanilla CoT baseline as a reference. As demonstrated in Figure and Table 2. the vanilla CoT model struggles to extrapolate and solve arithmetic problems involving longer digit lengths. However, integrating Neural Comprehension significantly improves the performance of language models on such complex arithmetic tasks. Since we only incorporated the Addit t and Subt r ct on CoNNs, we attribute the observed performance enhancement to the increased computational accuracy of the language model. For further evidence, we present additional experimental results on widely- used arithmetic reasoning datasets in Appendix E.2, which reinforce the benefits of using Neural Comprehension over the vanilla CoT model.
In comparison to language models employing external tools like PAL (Gao et al., 2022), our findings suggest that Neural Comprehension offers greater flexibility for LM. Firstly, by design, minimizes the necessity for additional processing steps or external tools, leading to an efficient direct computa- tional approach. This contrasts with Tool-based methods that often require additional programming and execution steps, increasing complexity and computational resources. Moreover, CoNN maintains end-to-end differentiabilityy crucial for models adapting to new data or tasks. contrast, Tool-based methods are difeeeeerrnt,..le, limiting their adaptability in reinforcement learning settings or tasks with sparse delayed rewards (Chung et al., 2022; Ouyang et al., 2022). Furthermore, CoNN's modu- larity enhances performance across various model scales, applicable regardless of language model's ability to call functions, unlike tools only operable in large, additionally code-trained models. Thus, the Neural Comprehension framework's 's efficiency, unified end-to-end neural network architecture, and extensive applicability constitute its distinct advantages over the Tool-based approach, offering a robust and scalable solution for multitude of linguistic and computational challenges.
5.4 ABLATION AND ANALYSES: MODULE COMBINATION FOR NEURAL COMPREHENSION
Efficiently deploying multiple CoNNs is crucial for achieving exceptional Neural Comprehension performance. As depicted in Figure 7, the amalgamation of distinct CoNNs, tailored for both symbolic and arithmetic reasoning tasks within the language model framework, can lead to remarkable benefits. Similar ToolFormer (Schick al., 2023), combine multiple different CoNNs into
8

---

Published as a conference paper at ICLR 2024
Pre Pre-trained Model with CoNN
A Pre-trained Model with some CoNNNs |Incresse right)
1C
Salve Pall Ph4
Salve P%1
60-
40
Werila Fireouno
(Cu rodoolddd Copy-lote
(Jnsndeeod) AMi
(Hnourshted Sub
20-0izts of AdkBub
Sphrs Raze 183
n0-
10
Varills CoT
[Combnes Add
(Commated) Sus
(Jnconsaled) Party
[Jinsome s8ed1 Wori
[Unsombitd] Coppllatter
Figure 7: In Neural Comprehension framework, the performance of multiple different module combination demonstrated.
side shows effect of combining a pretrained language model with CoNN. while the right side shows the impact of combining a language model with multiple CoNNs. For different tasks, we categorize CoNNs as Correlated (green) and Uncorrelated (red), indicating whether the CoNN is related to the current task not.
one framework, enabling the language model to have multiple capabilities. We conduct experiments on Last Letter Concatenation tass and AddSub* dataset, which shows the plug-and-play gating mechanism can still well control these CoNNs to output what should be output. It is observed that integrating pertinent CoNNs bolsters the performance of the initial language model, whereas the inclusion of unrelated language models rarely causes detrimental effects, regardless of whether single or multiple CoNNs are combined.
This can be ascribed to the refined design of the Neural Comprehension framework, which ensures the precise execution of assigned tasks by CoNNs without interference from irrelevant modules. Each CoNN module is adept at generating the appropriate output when needed, thereby preventing the emergence erroneous results from unrelated components. Importantly, as seen in Appendix B.3, the parameter count for each CoNN module ranges from 1/1000 to 1/1000000 of that for GPT-3, and the experiments in Appendix D.3 show that the inference latency in the neural understanding framework only increases by 1%-3% compared to Vanilla.
This observation underscores the remarkable scalability the Neural Comprehension framework, which possesses the capability to only accommodate existing knowledge concepts but also assimilate novel ones as the number of CoNNs expands. Theoretically, the integration of tens of thousands of CoNN modules within language models holds the potential foster comprehensive understanding of concepts.
6 CONCLUSION
We have observed that language models lack an intrinsic comprehension rule-based concepts and explored how Neural Comprehension can integrate compiled neural networks into the language model framework in a simple and plug-and-pla manner. On the one hand, we demonstrated the superiority of our approach over existing learning-base based methods, where our method implements comparable improvements to external tools within the neural network framework but does not require additional interpreters. This also enables language models without coding capabilities to possess symbolic manipulation abilities. On the other hand, compared to external tools, gradients can propagate without proxies, allowing better integration and full differentiability. The Neural Comprehension solves the issue of language models themselves being unable to perform robust symbolic operations and providing a foundation for future work on unifying both implicit and explicit learning language models and facilitating seamless integration.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
All CoNN models mentioned in this paper have been saved in Pytorch format in the Supplementary Materials, with dropout set rules. The code for the AutoCoNN toolkit and Neural Comprehension framework in this paper can be found in code portion of the Supplementary Materials, Details all experiments setting referenced in this paper included in Appendix F.1. Detailed descriptions of all tasks, datasets, and baselines mentioned in this paper are provided in Appendix F.2. Details of all few-shot prompts referenced in this paper included in Appendix G.
ensure deterministic outputs that conform human suman-specified
ACKNOWLEDGEMENTS
This work was supported by the National Key R&D Program of China (N022022201188001) and the National Natural Science Foundation of China (No.U U1936207, No.62376270, No.62171183), Youth Innovation Promotion Association CAS, and OPPO Research Fund.
ACKNOWLEDGEMENTS FOR COLLEAGUES AND FRIENDS
We appreciate the interest shown in this work by our colleagues and friends:
Haining Xie, Huanxuan Liao, Jiachun Li, Liang Gui, Pengfan Du, Pengfei Cao, Shaoru Guo, Wangtao Sun, Wenting Li, Xiusheng Huang. Yao Xu. Yifan Wei, Zhao Yang, Zhiqi Wang. Zhongtao Jiang, Zhuoran Jin, Ziyang Huang, who offered enthusiasm and backing.
Bingmei Sun, Boyuan Jiang, HaoChen Cao, Zixuan Cao, Donghui Li, Dongfang Suze, Ertan Zhuang, Jiajia Li, Mingwei Zhang, Lin Zhang, Mingwen Niu, Min Xiao, Qiaomu Tan, Tianyu Mu, Yuxin Liu, Xiaoyan Yu, Yuke Shi, Yixuan Li, Yang Zhou.
REFERENCES
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algo- rithm in-context learning? investigations with linear models. arXiv preprint LrXiv:2211.15660
2022.
Aida Amini, Saadia Gabriel, Peter Lin, Rik Kedziorkk, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation- based formalisms. north american chapter of the association for computational linguistics, 2019.
Cem Anil, Yuhuai Wu, Anders Johan Andreassen, Aitor Lewkowyez, Vedant Misra, Vinay Venkatesh Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL ht https: openreview net forum? wtte:ddddiskieee SKIVex7bC4,
Patel Arkil, Bhattamishra Satwik, and Goyal Navin. Are nip models really able solve simple math word problems? 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. arXiv preprint arXiv:2305. /7/26, 2023.
10

---

Published as a conference paper at ICLR 2024
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0255 023111,,,,1,,.... 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang. Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-fine finetuned language models, 2022.
Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers soft reasoners over language. In Christian Bessiere (ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pp. 3882-3890. International Joint Conferences on Artificial Intelligence Organization, 7 2020. doi: 10. 00244933/jcai.2220553. URL https::///do.orrrr org/
10. /04496/ijcaal wtticaa..c22000 2020/532777 537. Main track.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2/10.1410 14168 2021.
Xavier Daull, Patrice Bellot, Emmanuel Bruno, Vincent Martin, and Elisabeth Murisasco. Complex qa and language models hybrid architectures, survey. arXiv preprint arXiv:2302.0905/ 2023.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1.0.:::/10004800 2018.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang. Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint arXiv:2301.002:: 2022.
Ippei Fujisawa and Ryota Kanai. Logical tasks for measuring extrapolation and rule comprehension. arXiv preprini arXiv:2211.0772 2022.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. arXiv preprint arXiv:2211.10433 2022.
Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598, 2022.
Mor Geva, Ankit Gupta, and Jonathan Berant. Injecting numerical reasoning skills into language models. arXiv preprint arXiv:2004.0445: 2020.
Angeliki Giannou, Shashank Rajput, Jy yong Sohn, Kangwook Lee, Jason D. Lee, and Dimitris Papailiopoulos. Looped transformers as programmable computers, 2023.
Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. Learning to solve arithmetic word problems with verb categorization. empirical methods in natural language processing. 2014.
Minghao Hu, Yuxing Peng, Zhen Huang, and Dongsheng Li. A multi-type multi-span network for reading comprehension that requires discrete reasoning, empirical methods in natural language processing. 2019.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint arXiv:2210.11610, 2022.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: /ttp///ppeeeeeeeeeeeeeeeeeeeeeee net httpreewwwweeeeerteeeeeeeeee torrmm?id=227Bbbyyyrr

---

Published as a conference paper at ICLR 2024
Rik Koncel- Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 2015.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-tos grquence-to-sqqueme -o-sequencee pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. 2019.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman- et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv: 2tfi::2200.1885555 14858, 2022.
Bin Li, Encheng Chen, Hongru Liu, Yixuan Weng, Bin Sun, Shutao Li, Yongping Bai, and Meiling Hu. More but correct: Generating diversified and entity-revised medical response. arXiv preprint arXiv:2108. 01 266, 2021a.
Bin Li, Yixuan Weng, Bin Sun, and Shutao Li. Learning to locate visual answer in video corpus using question. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Bin Li, Yixuan Weng, Fei Xia, and Hanjun Deng. Towards better chinese-centric neural machine translation for low-resource languages. Computer Speech & Language, 84:101566, 2024.
Junyi Li, Tianyi Tang, Wayne Xin Zhao, and Ji-Rong Wen. Pretrained language models for text generation: A survey, 2021b.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et Competition-leve code generation with alphacode. Science, 378(6224) 2022.
David Lindner, János Kramár, Matthew Rahtz, Thomas McGrath, and Vladimir Mikulik. Tracr: Compiled transformers as a laboratory for interpretability. arXiv preprinl arXiv:2301.0544 .05062,, 2023.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi Yu, Asli Celikyilmaz, et al. Augmented language models: a survey. arXiv preprini arXiv:2302.0m 2328022022.7842,, 2023.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203. 10....13445 13474, 2022.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski. Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:21 14, 2021.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, perforaefe deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Bucc E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL https: http:////oooeeeeeeeeeeeeeeeee neuri ps cc cccppape__ti fi lelees/pp /aape//p/////// 2019 f0le/file//
2ttp/7701127777--aappe Paper. .ddf
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Advances in neural information processing systems, 101111105100700 2021.
12

---

Published as a conference paper at ICLR 2024
Xinyu Pi, Qian Liu, Bei Chen, Morteza Ziyadi, Zeqi Lin, Yan Gao, Qiang Fu, Jian-Guam Lou, and Weizhu Chen. Reasoning like program executors. 2022.
Jing Qian, Hong Wang, Zekun Li, Shiyang Li, and Xifeng Yan. Limitations of language models in arithmetic and symbolic induction. arXiv preprint arXiv:2208.0500 05001, 2022.
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot reasoning. arXiv preprini arXiv:2202.07266, 2022.
Subhro Roy and Dan Roth. Solving general arithmetic word problems. arXiv: Computation and Language, 2016.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilie, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.051000 2022.
Timo Schick, Jane Dwivedi-Yit Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer. Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint AXii:23302.0466. 2023.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost, 2018.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated CoRR, abs/1701.06538, 2017. URL
of-experts layer. wwtp:///////aa/a///ooooo////ss/t /tbbs/110. 06538.
Hikaru Shindo, Devendra Singh Dhami, and Kristian Kersting, Neuro-symbolic forward reasoning. arXiv preprint arXiv:2110.09383, 2021.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowyez, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders An- dreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karaka[s), B. Ryan Roberts, Bao Sheng Loe. Barret Zoph, Bart Bart(t)omie) Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng. Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt. Christopher D. Manning, Christopher Potts. Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hemandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko. Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman. Elizabeth Barnes, Elizabeth Donoway. Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang. Ethan A. Chi, Ethan Dyer, Ethan Jerzak. Ethan Kim, Eunice En- gefu Manyasi, Evgenii Zheltonozhskii,, Fanyue Xia, Fatemeh Siar, Fernando Martinez-Plumed. Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-López. Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannanch Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jachoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoñ, Jana Thompson, Jared Kaplan, Jarema Radom,
13

---

Published as a conference paper at ICLR 2024
Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Boss- cher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song. Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras- Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem [S] [S]ene, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátýás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Micha(t) Sw (eweeeddronn drowski, Michele Bevilac- qua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang. Peter Eckersley, Phu Mon Htut, Pinyu Hwang. Piotr Kowski. Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu. Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramón Risco Delgado, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs. Rui Zhang, Ruslan Salakhutdinov. Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang. Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoen- holz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff. Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi. Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima (Shammie) Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin. Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Mish- erghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili. Timo Schick, Timofci Kornev, Timothy Telleen-Lawton, Titus Tunduny, Tobias Gerstenberg. Trenton Chang, Trishala Neeraj. Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz. Yangqiu Song. Yasaman Bahri. Yejin Choi, Yichi Yang. Yiding Hao. Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. 2022.
Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schoelkopf, and Mrinmaya Sachan. causal framework to quantify the robustness of mathematical reasoning with language models. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 545-561, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/w1/2022.oo sel-long: 32.
http:///11/2222alllllllaaoe acl-Iong.32. URLhttps: aclannthoooogooooo www.ogg.o...gooooooooooooooooooo
Tian- Min-Xiang Sun, Xiang- Yang Liu, Xi-Peng Qiu, and Xuan-Jing Huang. Paradigm shift in nat- ural language processing, Machine Intelligence Research, 19(3):69--88, 2022. ISSN 2731- 538X. doi: 10.10077/111333-022 1331-6. URL https: /wwww mi www.eeeeeeeeeeeeeeeeeeeeee
article//oi/10 633 022 1331-6.
14

---

Published as a conference paper at ICLR 2024
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le. Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether thoiuht can solve them. arXiv preprint arXiv:2210.09261, 2022.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pp. 35151-35174. PMLR, 2023.
Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao- Yong Wei, Yaowei Wang. Yonghong Tian, and Wen Gao. Large-scale multi pre-trained models: A comprehensive survey. Machine Intelligence Reseurch, 20(4))447-482 2023a. ISSN 2731-538X. doi: 10.1007/ s11633-022-1410 URL https:tt////.. 1007/s11633- 022 1410-8.
mi rww.errcee...................ae. http:///aattteeee//oo.oo.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le. Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023b. URL https: : /ete:://pppeeeee hppp//eeeeeewweeee//ee/e nnt for wtaee//foommmmddeeeee wiw.:mmmddddiIiidNA
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07666 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoe V Le, Denny Zhou, et al. th-tuutt prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022b.
Gail Weiss, Yoav Goldberg, and Eran Yahav. Thinking like transformers. In International Conference on Machine Learning. pp. 11080-110999 PMLR, 2021.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In International Conference on Learning Representations.
Yixuan Weng and Bin Li. Visual answer localization with cross-modal mutual knowledge transfer. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEEE, 2023.
Yixuan Weng, Bin Li, Fei Xia, Minjun Zhu, Bin Sun, Shizhu He, Kang Liu, and Jun Zhao Large language models need holistically thought in medical conversational qa. arXiv preprint arXiv:2305.05410. 2023a.
Yixuan Weng, Zhiqi Wang, Huanxuan Liao, Shizhu He, Shengping Liu, Kang Liu, and Jun Zhao Lmtuner: An friendiy and iighly-integrable training framework for fine-tuning large language models. arXiv preprint arXiv:2308. 10252, 2023b.
Yixuan Weng. Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with velf-verification. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 2550-2575, 2023c.
Yixuan Weng, Shizhu He, Kang Liu, Shengping Liu, and Jun Zhao. Controllm: Crafting diverse personalities for language models. arXiv preprint arXiv:2402.
2024.
Fei Xia, Bin Li, Yixuan Weng, Shizhu He, Kang Liu, Bin Sun, Shutao Li, and Jun Zhao. Medconqa:
Medical conversational question answering system based on knowledge graphs. In Proceedings of the The 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 148-158, 2022.
Kaiyu Yang and Jia Deng. Learning symbolic rules for reasoning in quasi-natural language. arXiv preprint arXiv:2 12008,, 2021.
Zhen Yang, Ming Ding, Qingsong Lv, Zhihuan Jiang, Zehai He, Yuyi Guo, Jinfeng Bai, and Jie Tang. Gpt solve mathematical problems without calculator, 2023.
15

---

Published as a conference paper at ICLR 2024
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre- pre-trained trained transformer language 2022.
models. arXiv preprint arXiv:2205.01000
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations, 2023. URL https:/ / openreview. net/ for forum?dddd5TT &GE JUHkI.
Yang Zhao, Jiajun Zhang. and Chengqing Zong. Transformer: A general framework from ma- chine translation to others. Machine Intelligence Research, 20(4):514-538, 2023. ISSN 2731- 538X. doi: 10. article/doi////ic 110.
1393-3. URL https::/ww// mi aell-research net/en/
022 022-0222--1111
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. Lcast-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205. 2022a.
Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, and Hanie Sedghi. Teaching algorithmic reasoning via in-context learning. arXiv preprint 0rXiv:2211.09066 2022b.
Minjun Zhu, Yixuan Weng. Shizhu He, Kang Liu, and Jun Zhao. Reasonchainga: Text-based complex question answering with explainable evidence chains. arXiv preprint arXiv:2210.08765: 2022.
Minjun Zhu, Yixuan Weng, Shizhu He, Cunguang Wang, Kang Liu, Li Cai, and Jun Zhao. Learning to build reasoning chains by reliable path retrieval. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
A DISCUSSION OF LIMITATIONS AND FUTURE WORK
We have presented a novel framework that integrates Compiled Neural Networks (CoNNs) with existing language models to bolster their rule understanding abilities. Although our approach has shown promising performance improvements on symbolic and arithmetic reasoning tasks, there are several limitations and potential avenues for future research that warrant further exploration.
A significant limitation of our current framework lies in the more efficient and natural incorporation of CoNNs into language models. Currently, our method employs a sparse neural network that treats the pretrained language model and CoNNs as separate modules. A more desirable solution is to leverage a dense neural network, simultaneously utilizing the benefits both components. Examining the large-scale applicability of CoNNs is a beneficial endeavor. Although our experiments have been conducted on a relatively small scale (up to five stacked CoNNs), the advancements and abilities language models may gain from larger-scale combinations of CoNNs remain unclear. Exploring the scalability of our method and the performance advantages of deploying more complex CoNN architectures in language models could provide valuable insights into their potential.
Another promising area of research is the inclusion of explicit knowledge into CoNNs. While the current implementation centers encoding rules into CoNNs, future work could exploit techniques from knowledge graphs to compile explicit knowledge into language models. This may significantly enhance language models' interpretability and knowledge representation capabilities, potentially resulting in improved performance on an even broader range of tasks.
In conclusion, our work on enhancing language models' rule understanding capabilities through CoNN integration has yielded promising results, albeit with some limitations and remaining chal- lenges. By addressing these areas and extending our approach, we believe that it can ultimately lead to the development of more powerful, interpretable, and knowledge-rich language models.
B COMPILED NEURAL NETWORKS
In this section, we will discuss the concept, implementation, and potential of Compiled Neural Networks (CoNN), type of neural network inspired from previous works transformers. CoNNs can perform diverse tasks such as computer arithmetic and linear algebra, demonstrating a wide range of applications in LMs and beyond.
16

---

Published as a conference paper at ICLR 2024
B.1 INTRODUCTION
Transformers have garnered significant attention due to their ability to capture high-order relations and manage long-term dependencies across tokens through attention mechanisms. This enables transformers to model contextual information effectively. Pretrained language models, such as GPT-3 (Brown et al., 2020), exploit contextual learning to invoke various modules for different tasks, like performing arithmetic upon receiving arithmetic prompts. To further enhance rule comprehension in such models, CoNN-based modules are introduced as a part of Neural Comprehension.
Distinct from common models like BERT (Devlin al., 2018), CoNNs leverage transformer structure and derive their weights from specialized design rather than pretraining. Each Attention layer and Multilayer Perceptron (MLP) layer a CoNN represents specific sequence transformation, leading to neural network module embodying explicit and interpretable operations.
RASP (Weiss et al., 2021) is a Restricted Access Sequence Processing Language that abstracts the computational model of Transformer-encodee by mapping its essential components, such as attention and feed-forward computation, into simple primitives like select, aggregate, and zipmap. This language enables RASP programs to perform various tasks like creating histograms, sorting, and even logical inference, as demonstrated by Clark et (2020).
Tracr (Lindner et al., 2023) serves as a compiler that converts human-readable RASP code into weights for GPT-like transformer architecture with only decoder module. The Traer framework uses JAX transform RASP-defined code into neural network weights. Our neural reasoning framework employs weights generated by Tracr, which are then converted into PyTorch weights to be compatible with the pretrained language model.
Looped Transformers as Programmable Computers (Giannou et al., 2023) introduces a novel trans- former framework that simulates basic computing blocks, such edit operations input sequences, non-linear functions, function calls, program counters, and conditional branches. This is achieved by reverse engineering attention and hardcoding unique weights into the model, creating looped structure. The resulting CoNN can emulate general-purpose computer with just 13 layers of trans- formers, and even implement backpropagation-bait context learning algorithms, showcasing the approach's 's vast application prospects.
Overall, the potential applications of CoNNs are extensive, given their capacity to perform wide array of tasks beyond natural language processing. CoNNs offer increased interpretability and transparency through explicitly defined operations, which is vital fields such as medical diagnosis and legal decision- decision-making. Additionally, CoNNs can lead to more efficient and effective neural network architectures by reducing pretraining requirements and facilitating improved optimization of network parameters.
B.2 EXAMPLE
In this subsection, we briefly describe how computational processes can be represented using trans- former code and demonstrate how new CoNN weights can be obtained with the aid of the Tracr compiler.
B.2.1 PARITY CONN
In the introduction, we tried to introduce how to perform parity checking sequence containing [0 []] using a CoNN. Whenever we need to check the sequence, this CoNN can output the completely correct answer.
Figures 8 and present two distinct input sequences, and illustrate the corresponding hidden state and final output obtained after passing through the internal layers of the Parity CoNN architecture.
B.2.2 REVERSE CONN
Figures 10 and 11 show the hidden state and output of Reverse CoNN when inputting text. The embedding of CoNN can be customized, so tokens can be either words like 'hello' or individual letters.
17

---

Published as a conference paper at ICLR 2024
THE TRACR CODE OF PARITY CONN
parity (sop) raap.3op:
*MMMultiply the Zength of each Loken, sop rasp. Sesp.Seuuncceapp
x,y: sop, length) n'medd hww.emdmmm__eerriiiiiaa map_length"")
*dddd each
out raap comericall (rasp, Aggregate (rasp.Selec) Select (raap, indicea, indices, rasp. Compar comperlaon laon.
I3UE) aamed (" Select' rasp. numericce (rasp Map (lambda x: X, sop) ('map )) default=0)
Aggregate")
zemainder dividing by out = rasp. Mop (lambda x! x == alse o,t named ( Zipmap')
cncaacuiate
even. e*y
return out
NUP
Jet-
MLP3
HLP
DeaD
gaoood
Figure 8: Input the [1,0,0,0,1] (target output = 0) for Parity CoNN.
Figure 9: Input the [1,0,1,011] (target output = 1) for Parity CoNN.
B.2.3 ADDITTON CONN
Due to the high complexity of the model, we decided to omit the hidden state transformation for the Addition CoNN. However, we have provided code later in the text that will allow for easy implementation of this CoNN. The code includes add_in. iin_th__same, pposiiit itt.tttt tion and add_carry functions, which are used to calculate the addition and carry of pairs the CoNN respectively. We divide the entire operation into two models. For the add_carry model, we refer to the approach ALBERT. After the output of the add iiniinntthhh Se_ttee___ same same_posit__ www.iissiiooooo.p model, we eyclically the add. carry model L times, where L is the length of the text, to ensure that all digits can carry. is important to note that this particular Addit addition CoNN is only capable of performing addition operations natural numbers.
18

---

Published as a conference paper at ICLR 2024
THE TRACR CODE OF REVERSE CONN
def reverse (sop) -> rasp.Sup:
s*agat the indices from back front. e** opp_idx (langth indccindices) .tw.indiiesenaaaaa. .named i"
opo idx 80 that the fizat digit of indices 0, nne opp_ids (opp_idx mammedd "opp_idx
*eauae opp idx to query des,,e, get the Select, geverse_selecten rasp, Select (rasp. ced, opp_idk, rasp, Compari teverse_selectore
*"nggreyale the reverse selector and ворa*я
return rasp.Aggregate (reverse_selectone sop) .zemed ( reverss")
THE TRACR CODE OF ADDITTON CONN
def split (sap, taken, index):
*"*Match the position of Lazget Loken*"*
perget_position rasp Aggregate (razp. (sop, rasp.Map (1ambbd token, sop), rasp. Compariscn. Ful rasp.indices)
*engf need to match the front position, index out Aggregate (tasp. target_goeition) rasp, aesp.oomarrieen.20 sop) Move the right.
rasp.indices (leagth
the left side of the taken the far
raap. SequenceMap (lambda x, reap, Bep.eeceaeeM [lambda x, 1: 2 if x 5- else 0, zasp, indices, length fill empty position on left.
else "_", rasp.categorioal
parget__osition)) )
wonig need to match the finally number.
alse:
return rasp. SaquenceMap (lambda x, 1: if else "_,, 50p, raap. SecuenceMau (lambda x, .named(
else 0, indices,
E"shift") Use target_ empty position
left.
def atoi (sopl:
converss all text numbor, and 6 for strings of types other then mumbers, It may be sixed with fatrt or cint..
return rasp Segueneeia (lambda x, int (x) x.indigst sep, rasp.indices).
named
"atoi")
def shift (sop):
sonGet target indices. e*e idx = (rasp.indizes (siedd(""))
saause opp idx to query indices, get the Solect, ector rasp.Select (idx, raap.inclos. Casp.Compaiies isun. isua.Bqi named "shiiit_seeector"
**gggrggttes the sepa and seiectors (converted from indexes) *** shift Regregate (selector, sop) ("amedd"ehiiiu return shift
def in__itt siit__sae__eee poste_oositinn
stoi (split +', 20) atoi (split (sop, +*, (111))
return x
def (sop):
weight shift trasp. Kap (lambda n:l n>S else sop))
weight - zasp azaspp
(zasp. queryl weight, defaul:=0;
(rasp indices, rasp. indicen, lambda key, query:key
Map (lambda nin-10 if n>9 else sopt
return x weight
19

---

Published as a conference paper at ICLR 2024
Figure 10: Input the [ ('hello", v. 'wolldd] for Reverse CoNN.
Figure 11: Input the ("r''e"""""""""""" for Reverse CoNN.
THE TRACR CODE OF SUBTRACTION CONN
def split (sop, Loken, index):....
def atoi (sop) :...
def shift (sop):....
def sub_in the the__tteeeeeeeeeeee same wtwesaaaeeeesseeeee tome (splii return x
atpllt (sop,'
daf carry (sap):
weight
trasp. (lambda
0, sop))
weight zasp Aggregate (zasp. Select (ras..iiiicac indices, indice,, lambda key, query:kep queryl weight, defaulcm0;
Map (lambda nint10 if n<0 alse sepl
return x weight
B.2.4 SUBTRACTION CONN
The subtraction CoNN is similar to the addition CoNN. First, each digit is subtracted from its corresponding digit, and then it is determined whether to carry over. For ease of experimentation, this subtraction CoNN only supports subtraction of natural numbers where the minuend is greater than the subtrahend.
B.3 CONN MODEL PARAMETERS
The parameter sizes of all CoNN models used in this work are listed Table It is noteworthy that for GPT-3, which has parameters that are orders of magnitude larger, it remains challenging to solve symbolic problems. However, with the use of compiled neural networks, only a small number of parameters needed to achieve Neural Comprehension.
20

---

Published as a conference paper at ICLR 2024
Model
Layers
llends
Vucabulary Size
Window Size
Hlidden Size
MLP [6akeen Sizr
MLP[iiäien
# Parameters
Compared to GPT-3
Farilly
40
1959
Reverse
1/50.000
"ast fetter
16
1/30000,000
copy
1
28
16
69
26
3.8K
1/200000,00 D00
7
40
535
6422
51 51.8M
2x1/5008
Add Carry
3
122
40
130
52
117K
=010,,00,000
in the position
7
13
40
535
6422
51 11.MM
#31/5000
Sab_Carry
3
122
40
130
52
117K
=:/11,500000
Table 3: We reported CoNN with single function, including its actual parameter size and comparison with the parameters of GPT-3.
B.4 ENVIRONMENTAL AND HUMAN-CENTRIC BENEFITS OF COMPILED NEURAL NETWORKS
Compiled Neural Networks (CoNNs) address concerns related to the environmental impact of training large models and the need for human-centre computing. CoNN models can reduce energy consump- tion and carbon emissions by minimizing extensive pretraining and decreasing parameter size, as seen in Table 3. This reduction computational power and energy requirements makes both the training and inference processes more environmentally friendly. Additionally, Neural Comprehension offers more interpretable and transparent alternative to conventional deep learning models. CoNN's explicit operation definitions and specialized architecture enable users to comprehend the reasoning behind model decisions, fostering trust and facilitating human-Al collaboration. Increased interpretabiliity also allows for scrutiny of model behavior, promoting the development of fair, accountable, and transparent systems aligned with ethical considerations and human values.'
C EXPERIMENT FOR AUTOCONN
C.1 METHOD
For experts, they may need spend a lot of time writing code suitable for CoNN, while non-expert users find it hard to obtain or modify CoNN. These issues limit the efficient combination of CoNN and L.M, so we utilized the few-shot ability of language models to make the AutoCoNN toolkit (Weng et al., 2023b). In this section, we will show a series of detailed exper- iments on AutoCoNN to demonstrate this. First, It is the Demo of Auto- CoNN code:
Model
Example=
Example=2
Example=5
Parity Model
5/10
10/10
10/10
Reverse Model
10/10
10/10
10/10
Tast Letter Model
9/10
10/10
10/10
Copy Model
10/10
Table 4: For each CoNN model, we selected ten groups of models that were judged to be correct by AutoCoNN. We manually evaluated whether these models were indeed correct. The 'Example=x' means that x Examples were provided in the validation stage.
DEMO OF AUTOCONN
from NeuralCon Com, AutoCoRN import AutoCoMN
INSTRUCT Create, Stp_Sppp_eeee that,,,s,,,.... ies. VOCAB ['a', 'b', 'e', "d', 'e' (''"g")
EXAMPLE - (b' /ef Me /ttfee///eelllllleeeeeee Ir',,, (tar,
(UtoroNN()
model, Lokenizer auto (inst:ttttTTsteteee vocab=WOCAB, example-EXMPPA
Table 5 shows the efficiency comparison between experts and AutoCoNN. This demonstrates that the AutoCoNN toolkit can generate various CoNNs faster. But we also found that for more difficult ones like Addition and Subtraction, it fails to successfully generate, which becomes one of the limitations of AutoCoNN. On the other hand, we tried providing only "Instruct" or "Example" for AutoCoNN to generate", and often "Instruct" can generate CoNN with higher accuracy, while "Example" cannot. This shows that giving explicit operational instructions performs better than directly observing data in AutoCoNN.
In this experiment, the few-shot samples also contained only of them
21

---

Published as a conference paper at ICLR 2024
CONN Model
Expent's Working Time
Suceess by AutoCaNN
Can AutoCoNN solve
AuroCaNN (w: Instruet)
AutoCYNN Example)
Facity Model
Revezse Model
0,5 hour
15/20
//
16/20
11/20
Last. Lettez Model
0,5 hour
13/20
//
12/20
10/20
Copy Model
0.2
17/20
JV
17/20
15/30
Additt Model
48
Subt nact for Model
48 hours
0/00
(/20
0/20
Table 5: Comparison between AutoCoNN and Expert Built CoNN. The column 'Expert's Working Time' refers the time required for trained engineer to write the CoNN code; 'Success by AutoCoNN' refers to the accuracy of 20 results generated by using GPT-3.5 for diverse decoding; 'Can AutoCoNN solve' refers to whether AutoCoNN can identify suitable CoNN code from the 20 results through validation. It is worth noting that in this experiment, we use sampling decoding with temperature=0.7 to generate 20 different CoNNs codes, which we convert Pytorch versions of CoNNs models. We report the accuracy of the CoNNs codes through manual (expert) evaluation.
It is difficult for non-expert users to assess the accuracy of the generated code, automatically utilize the Example information to verify the accuracy of the CoNN model - checking whether the output result of the input sequence is exactly consistent with the Example. The results shown in Table 4 demonstrate that generally 2 Examples are sufficient to select an accurate CoNN model, which means it is very easy for different tasks, we still suggest non-expert users provide more Examples to ensure the accuracy of the generated CoNN.
and demonstrate. However, considering the varying difficulty of
D EXPERIMENTAL SETTINGS
In this study, we primarily explore the capacity of language models to address symbolic reason- ing tasks, concentrating three areas: symbolic operations, symbolic reasoning, and arithmetic reasoning.
Building upon the approaches developed by Anil et al. (2022) and Qian et al. (2022), we examine the following tasks: Parity, Reverse, Addition and Subtraction. These tasks do not require complex text understanding, but only require faithfully implementing symbolic operations and outputting the corresponding results.
Symbolic Operations
Symbolic Reasoning We employ the experimental framework of Wei et al. (2022b) for the two tasks, Last Letter Concatenation and Coin Flip. These tasks require a combination of language understanding and rule comprehension abilities.
Arithmetic Reasoning To evaluate the method's generalization ability from symbolic operations arithmetic reasoning addition and subtraction tasks,
five established arithmetic reasoning datasets: AddSub (Hosseini et al., 2014), SingleEq (Koncel-Kee -ennel--eddiorski et al., 2015), MultiArith (Roy Roth, 2016), GSM8K (Cobbe al., 2021). 1), and SVAMP (Arkil al., 2021). Additionally, we introduce the AddSub dataset, containing tasks of varying complexity based on the number of digits
involved in arithmetic operations, ranging from diigitt addition to 20-digit addition/subtraction tasks.
E SUPPLEMENTARY EXPERIMENT
E. THE EFFECT OF TRAINING DATA SCALE ON LENGTH GENERALIZATION OF GRADIENT-BASED MODELS
To investigate the impact of training data scale on out-of-distribution (OOD) performance, conducted experiments using the T5-lerge model with varying amounts of in-distribution training data. The experimental setup closely followed that of Main Figure 3, utilizing numbers with 10 to 20 digits as the training set but varying the number of training examples between million and 15 million. The peak validation set performance for each experiment is reported in Figure 12.
22

---

Published as a conference paper at ICLR 2024
Trastig Dur
Tert Leag
Tert
(a) Parity
(b) Reverse
Accurac
Accurati (6)
Trainin Data
4he Training Data
Teah
(c)
Addition
(d) Subtraction
Figure 12: Length Generalization Performance of Language Models with Different Dataset Sizes.
The results in Figure 12 show that increasing the scale of the In-Dist training data leads only marginal improvements in OOD performance. This finding is discouraging, suggesting that gradient- based language models face challenges capturing the underlying meaning of symbols and their transformation rules based on the data distribution alone.
E.2 REAL-WORLD ARITHMETIC REASONING TASKS
Method
GSMSK
SingleGi
AddSub
MultiAnih
SVAMP
Average
(175)
CoT
13.84
62.02
57.22
45.85
38.42
43.47
oodie davinci 001 CoT Neural Comprehension
13,95 13.95 (1)
83 (00.0)
60.25 (.000))
45.85
38.62
44.30)
GPT33 (1755)
Col
61.20
91.01
82.78
96.13
75.87
$1.20
OoT Noural Comprihemion
60.42,
01
62.78
96, 15,00
76.09
40.222
-0000)
Table Problem solve rate (%) on arithmetic reasoning datasets. The previous SoTA baselines are obtained from: (a) GPT-3 175B finetuned (Cobbe et al., 2021); 1);1);; (b) GPT-3 175B finetuned plus an additional 175B verifier(Cobbe et al., 2021); (c) al. (2019); (d) Roy Roth (2016); (e) Roy Roth (2016); (f) Amini et al. (2019); (f) Pi et al. (2022)
23

---

Published as a conference paper at ICLR 2024
As model parameters, training calculations, and dataset sizes have increased, language models have gained new capabilities (Srivastava et al., 2022; Wei et al., 2022a; Weng et al., 2024), such as Machine Translation (Zhao et al., 2023; Li et al., 2024), complex QA (Zhu et al., 2022; Daull et al., 2023; Zhu et al., 2023), Multimodal QA (Wang et al., 2023a; Li et al., 2023; Weng & Li, 2023), coding (Li al., 2022; Nijkamp et al., 2022), few-shot learning (Brown et al., 2020; Perez et al,, 2021), medical diagnosis (Li et al., 2021a; Xia et al., 2022; Weng et al., 2023a), and chain of thought (Wei et al., 2022b; Weng al., 2023c).
In Table 6, we compared Vanilla CoT with the Neural Comprehension framework for arithmetic reasoning tasks. We integrated the Addit and Subtract on CoNNs with LLMs and observed improved performance across several tasks. This suggests that the proposed Neural Comprehension framework can compensate for the difficulties faced by large-scale language models in computational tasks. Nevertheless, the performance improvement is not as significant due to the choice of specific CoNN models to ensure clarity our experiments. Designing CoNN models to support more general arithmetic tasks could potentially yield more substantial improvements. In addition, since the Neural Comprehension framework improves the gap between the data distribution learned by the language model during training through gradient descent and the real rules, it can also be combined with some existing logical improvements to language models, including celficonns (Wang et al., 2023b), least-to-most (Zhou al., 2022a), self-improve (Huang et al., 2022), and self-verification (Weng et al., 2023c). It can also be combined with some zero-shot methods (Kojima et al., 2022; Zhang al., 2023).
To further evaluate the effectiveness of the Neural Comprehension framework, Table 7 presents the results of fine-tuning T5 models with Addition and Subtraction CoNN on the GSM8K train- ing dataset. The comparison of three different-sizer models reveals that the framework can model deter- ministic rules defined by humans, thus avoiding the uncertainty associated with gradient descent learning from data distribution.
Method
T5-mmal
TS-buse
TS-lage
Origin
1.74
1.52
3.87
Neural Comprebeesiio
1.82
1.59
4,02
Ours Improve
+0.15
Table 7: The test set problem-solving rate (% of the T5 model on the GSM8K dataset.
E.3
THE EFFICIENCY OF NEURAL COMPREHENSION
Vanilla
Neural Comprehension
éyine
Model
Params
Task
GPU
CPU
GPU
CPU
GPU
CPU
T5
Coin Flip
5.280s
5.720s
5.431s
5.872s
0.151s (2.86%)
0.152s (2.66%)
T5-base
220M
Coin Flip
7.865s
13.767s
8.010s
13.939s
0.145s 848%)
0.172s (1.25%)
TS-large
770M
Coin Flip
14.055s
32.953s
14.194s
33.120s
0.139s (0.99%)
0.167s (0.51%)
T5-small
60M
Last Letter Concatenation
16,233s
28.309s
16.744s
28.720s
0.511s (3.15%)
0.41 (1.45%)
T5 hase
220M
Last Letter Concatenation
28. 28.912s
55.660s
29.426s
56.087s
0,514s (1.78%)
0.427s (0.77%)
T5-large
770M
Last Letter Concatenation
49.584s
103.739s
50.066s
104.134s
0.482s (0.97%)
0.395s (0.38%)
Table 8: In Neural Comprehension framework, the inference latency comparison of the T5 model.
To evaluate the efficacy of Neural Comprehension, we conducted further experiments comparing the inference latency of both the Vanilla and Neural Comprehension frameworks on an equal number of sequences and equal sequence lengths using GPU and CPU configurations. We employed a batch size of and assessed the inference latency of Neural Comprehension in conjunction with various T5 model sizes across two symbolic inference tasks to ascertain efficiency. The full results are detailed in Table 8.
Our findings reveal that implementing Neural Comprehension increases computational requirements, primarily attributed to the supplementary parameter volume and computational demands introduced by CoNNs. However, the scale of pretrained language models expands, the proportion of 6rime within the Neural Comprehension framework progressively diminishes, particularly for larger language models.
24

---

Published as a conference paper at ICLR 2024
F IMPLEMENTATION AND DETAILS
Model
Model Creator
Modality
Version
# Parameters
Tokenizer
Window Size
Access
T3-small
Google
Text
T5.1.0
60M
T5
512
Open
T5-buse
Google
220M
Open
13-large
Google
TTxx
15.100
770M
T5
512
Open
GLM-130B
Tsinghua University
Text
GLM-130B
130B
ICE
2048
open
GPT-3
OpenAl
ToxeCode Text, Codc.Instruct
code-davinci-001
175B
GPT-2
2048
limited
GP-3.5
OpenAl
code-davinci-002
1758
GPT-2
8096
limited
GPT-4
OpenAl
Text, Lode..nstruct....
gpt-4
1750B*
GPT-2
8000
limited
Table 9: Models. Description of the models evaluated in this effort: provenance for this information is provided in models. indicates that we believe the associated OpenAl models this size, but this has not been explicitly confirmed to our knowledge.
In this section, we provide a detailed description of the experimental setup from a model and dataset perspective, ensuring repeatability of our experiments.
F.1 MODEL
Our experiments primarily involve the T5, GPT, and GLM-130B families of models. Neural Compre- hension framework supports seamless integration with language models having decoder structures, regardless of the scale of the language model. We fine-tune the T5 models, while the larger mod- els with over 10 billion parameters are used for few-shot In-context learning. Table presents comprehensive list of all models used in our experiments".
F.1.1 FINI-TUNING
For the T5 models, we employ the standard tne-tuning approach using the pretrained models as a starting point. We follow the pre- pre-processing steps in the T5 original paper, which involves set the input text max length 150 and using the tokenizer to process the data. We use batch size of 64 for all models and the Adafactor optimizer (Shazeer & Stern, 2018) with learning rate of 10-1 The models are trained for a maximum of 20 epochs. We use a cosine learning rate schedule with a warm-up phase comprising 5% of the total number of training steps. We employ dropout rate of 0.1 during training to mitigate overfitting. Our experiments utilize the PyTorch framework (Paszke et al., 2019) for training and inference. Table 5 displays the parameter settings for the T5 models during training, which is conducted on four NVIDIA A6000 GPUs with 48GB of memory each.
In Table 10, list the hyperparameters used to train the T5 model. We carefully selected these parameters ensure that the within-disrrbbution validation accuracy roughly converged. We re- port all peak validation set results, and in ev- ery experiment we ran, we found that within- distribution validation accuracy monotonically increased during training iterations (until it ap- proached 100%), and we never observed over- fitting. This may be due to the regular nature of the tasks we considered in the paper. We fol- lowed the Anil et al. (2022)'s setup and did not use OOD performance in model selection, as this would constitute "peaking the test conditions". Regarding the number of training iterations, also tried training the T5 model with more iterations in the addition task, but this did not lead to substantial differences in OOD performance (i.e., it was still equally poor).
Training Setting
Configuration
optimizer
Adafactor
base learning rate
1 x 10
weight decay
2 x 10-5
decay rate
-0.8
optimizer eps
(1 x 10
.2 x 10
batch size
64
training epochs
20
gradient clip
1.0
Table 10: Training Setting
F.1.2 FEW-SHOT IN- IN-CONEXT LEARNING
For the few-shot context learning on GPT and GLM-130B models, we employ an approach inspired by the recent work on GPT-3 (Brown et al., 2020). In this methodology, the models are provided
"The specific configurations of the GPT series of models can be found at com/ docs /moee 1 -1 inde-- for- resear cheracchhhrrre. mode/ ndex- For-researchers
openai.
25

---

Published as a conference paper at ICLR 2024
PARITY DATASET
saieeale_paaily__atee
data = []
xange|100000):
input_atr (ste (random..andg rand" 1)) for in range (n))
label sum (int 1x) for x input_str) 2 data.eppend input': input_str, 'label': label))
return data
parity_data -) in range(1, 41): parity_data[n] panerate_parlty_daaaa (n)
context consisting of few examples the input-output pairs in natural language format. Following this, the models expected to generalize and perform well the task without any explicit fine- tuning. We carefully design the context to include diverse examples that represent the range of input types and required model reasoning. Importantly, maximum token limits of the models. For instance, GPT-3 has a token limit of 2048. Due to limited access to the GPT f'amlly of models, we utilize the official API for these experiments 5, .For the BLMM-330B we employ the Faster Transforneff framework to set up local inference with INT4 on eight NVIDIA GeForce RTX 3090 GPUs with 24GB of memory each.
limit the context length be within the
To compare with CoT and PAL which experimented on GPT-3 series models, we simulated Neural Comprehension (NC) within the constraints of the API access. We treated the API's output as it were part of the Neural Comprehension structure. This involved a simulated gating mechanism, where the output from the API was corrected using CoNNs form left to right, and then the adjusted response (Truncate the text after the altered text.) changed into the API input for continued generation. This simulation was to ensure that the benefits NC could be compared fairly with the existing results from PAL.
F.2 TASKS AND DATASET
In this paper, all data sets related length generalization consist of independent data sets with the same number of digits but different lengths, and each digit in the test set is unique. Therefore, there may be slight fluctuations between data sets of different lengths, but the overall trend is generally clear. To further illustrate the differences between data sets of different lengths, the following examples provided:
Parity:
110
101100
010001110101 0
Length =3
Length =
Length 12
Reverse:
abe Length =3
cba
figure erugif Length
accomplished dehsilpmocca
Length
Addition:
1 2
=
18 + 245 263
48864 + 964316 1013179
Length =3
Length
Length 12
Arithmetic Reasoning:
Joan found 65466345746$8499 seashells and Jessica found
Length 16
385519660206362 seashells on the beach. How many seashells did they find
Length 16
together ?
F.2.1
DATA GENERATION DETAILS
Synthetic Parity Dataset:
We sample instances of lengths 1-40 from uniform Bernoulli distribution. We first uniformly sample the number of ones, and then randomly shuffle the positions of each one within a fixed-length bitstring. For the experiments in Figure 5.1, we train T5 on lengths 10-20, with 99000 training samples per bit. For all methods, we test each bit using 1000 samples.
'OpenAl's APl:httaa /tppaaiippiap//////////e// hom/app//
26

---

Published as a conference paper at ICLR 2024
REVERSE DATASET
zeverse_dale [:
for in range (1, 41):
reverse_dztale
for in (ange(100c00)
word /oin join (random. choice iisssriig.sseii lowercase) for looe-eecaee svrese_dataa[ee aapendd (f' Laput' sword, 'label': :f joinjjnnnn llist (wcrd))
range (n)
ADDITION DATASET
Generate tNo randos n-digit numbers and their sum
def samerata_dddriiveeeee
data [] 1182 ia
else
for in range: (100000)
a random. randint (10.. (ark (ark-1), 1), (0tk(
b random. randint (10+ (1-1), 10+en
data a ((' input'; tfe "label'; b))
return data
Generate an additive data set for digits ranging from 3 to
additive_dat = =()
for in range (3, 41):
coditive_dataastae
generate_adelt teeeeooo
Synthetic Reverse Dataset: We selected a dataset of instances with lengths ranging from 1 to 40. For the experiments in Figure 1, the training set consists 99000 samples each for lengths 10-20. Each input is randomly generated word string of specified length, where the letters are selected uniformly at random from a set of 26 letters using a Bernoulli distribution (without necessarily having any actual meaning), and all letters converted to lowercase. The test set for lengths 1-40 contains at least 1000 test samples.
Synthetic Addition and subtraction Dataset: Addition and subtraction are fundamental arithmetic operations that are commonly taught in primary school. To generate the dataset. we takes as input the number of digits n and returns a list of 100000 examples. The function first calculates the remainder k when (n 1) is divided by 2, and then divides (n 1) by 2 if is greater than 2, else it sets to itself. This ensures that the length of the first number is either equal to or one digit longer than the length of the second number. The function then generates 100000 examples using randomly generated numbers. Specifically, it generates two numbers and b where is random integer between 10(*+*-1) and 10(*+*) - 1, and b is random integer between 10(n-1) and 10"h 1. It then appends each example to the list data in the form of dictionary with the input as the string "a+b" and the label as the sum a+b.
For the experiments in Figure Figure 1, we provided 99000 training data examples for addition with numbers ranging from to 10 digits in length for the T5 model. For the GPT-3.5 and GPT-4 models, we provided 8 few-shot samples within 10 digits. We evaluated the performance of all three models on numbers ranging from 3 to 30 digits in length, with 1000 test samples per digit. On the other hand, for the experiments in Figure Figure 5.1, we provided 99000 training data examples for addition with numbers ranging from 10 to 20 digits in length for the T5 model. For the GPT-3.5 and GPT-4 models, provided 8 few-shot samples within the range of 10 to 20 digits. We evaluated the performance of all three models on numbers ranging from 3 to 30 digits in length, with 1000 test samples per digit.
For subtraction,
similar approach.
F.2.2 SYMBOLIC REASONING DATASET
Coin Flip: We followed Wei et al. (2022b)'s setup and randomly concatenated first and last names from the top 1000 names in the census data (https: : / /namecensus com/) to create the <NAME> token. In work, flipping coin corresponds to 1 and not flipping coin corresponds to O. To make the inputs as close to English as possible without using too many symbols, we used the sentence models "<NAME> flips the coin." and "<NAME> does not flip the coin." " represent
27

---

Published as a conference paper at ICLR 2024
SUBTRACTION DATASET
Generate randos a digit numbers and theiz def generate_mingg eeaplus__eaapeee_eeee (3):
in 1) 2
ta 11 // 2 else
for in :
random. randint (10* (10** (a-k-1), (ntk) -
cendom.randinn (10.a (a 1),, 10vMn
if a b:
dsta.append H input'" stx(s)++/+aatr 'labal': a b11
else:
append H" L'pput": str (istr(a), "labelf: b all
return data
Generate an subcraction
digits zanging
mimus_data = ()
in range (3, 41):
minus_datá [stria)]
COIN FILP DATASET
dataset
for in zange (500):
randomly choose names name_ list
for o in range
sentence
label = for time in (o)
sandom.sample (names, k=11 [0]
not flip = random. random.chaice (True, False])
randomly choose whether flip
generate the statement and label based whether the coin was Elipped if flip:
sentence p" www.aae cammeccapitallieett theps, coin." agpel.ap)end())
else:
(1 http:/lloooooooo....t.thtetoeeet sentence 1- ses__lle__conn_ttiil_..e
sentence +s f"
aabel.append 10)
daLaset. append question' isentence, 'answer' y0s nl:fno' (Label)
whether the coin was flipped or not. This task is similar to the parity task, but requires further semantic understanding. We constructed training set of 1500 samples, with 500 samples for each of 2-4 coin flips. For the test set, we selected 100 non-overlapping samples for each of 2-4 coin flips, and evaluated the model every 5 steps.
Last Letter Concatenation: We followed Wei et al. (2022b)'s setup and randomly concatenated first and last names from the top 1000 names in the census data to create the <NAME> token. This task requires the model to connect the last letter of each word in a concatenated name. This task requires Neural Comprehension of rules in two aspects. First, it requires the model to correctly identify the last letter each word. Second, it requires the model to concatenate all the last letters of the words together. We concatenated 2-5 first or last names, and constructed training set of 1500 samples, with 500 samples for each name length of 2-4. For the test set, we selected 100 non-overlapping samples for each name length of 2-4, and evaluated the model every steps.
F.2.3 ARITHMETICAL REASONING DATASET
In Table 11, we summarize the information of all arithmetic reasoning datasets used in this work. We provide the links to access these datasets:
GSM8K: Gtt.k//ttsss.///tt..cc/ooo/////a
AddSub: httpbb/wwttpp.........aaaaaaa//p
MultiArith:
98
28

---

Published as a conference paper at ICLR 2024
Dataset
Number of samples
Average words
Answer Format
Lience
GSMSK
1319
46.9
Number
MIT License
SingleEq
508
27.4
Number
MIT License
AddSub
395
31.5
Number
Unspecified
MultiArith
600
31.8
Number
Unspecified
SVAMP
1000
31.8
Number
MIT License
Table 11: Arithmetical Reasoning Dataset Description.
SVAMP: SSW.P//rtpp///ghhht....///a//a/p
G SOME EXAMPLES OF NEURAL COMPREHENSION
In this section, we will use gray font to represent the task input, yellow font to represent the neural network output during training, and blue background to represent the neural network output during generated.
G.1 SYNTHETIC SYMBOLIC
Q: 1011001010
Q: 01111011 1000 A: 0
Q: 101001 1001 1 10 A:
Q: 10000001001001 A:
Q: 1110001111000 A:
Q: 111001 1001011110
Q:
A:
Q: 01 1000000110100100000000000 A: 0
(I.LM's few-shot prompt)
Q: 011111111 10011000011111101111 A:
Table 12: The example of Parity
Q: neofascism A: msicsafoen
Q: betaquinine A: eniniuqateb
Q: corediastasis A: sisatsaideroc
Q: ferroelectronic A: cinortceleorref
Q: cryoprecipitation A: noitatipicerpoyre
Q: cryofibrinogenemia A: aimenegonirbifoyre
Q: chemocareinngeenssis A: sisenegomiicracomehe
Q: ponipedqjuuhvviojnnyy ybmjoivihuujjgdepioo
(LLM's few shot prompt)
Q: helloworldhellochhnaa A: anihcollehadloowollech
Table The example of Reverse
29

---

Published as a conference paper at ICLR 2024
Q: 82637+3058 A: 85695
Q: 58020+96632 154652
Q: 717477+55874 A: 776175
Q: 29830097008888 A: 1001167
Q: 106146242633780 A: 3685242
Q: 587209700661609004 A: 1203300M
Q: 3649200799788864800 443781959
Q: 60503300022211065224 A: 6261395326
(LLM's shw-shot prompt)
Q: 20021011220021000 A: 40042016
Table 14: The example of Addition
Q: 82637-3058 A: 79579
Q: 96632-58020 A: 38612
Q: 717471-58704 A: 658767
Q: 702858-298309 404549
Q: 1067746--161622222 A: 1562318
Q: 68720990--61609004 A: 11936
Q: 1364920479-788614800 A: 286058999
Q: 1350330002-1100553222 A: 393967676
(LLM's fow-shot prompt)
Q: 20021012-20022004 A: 8
Table 15: The example of Subtraction
G.2 SYMBOLIC REASONING
A coin is heads up. Devin flips the coin. Maxwell does not flip the coin. James flips coin. Kenneth flips coin. coin still heads up? 101 1
coin is heads up. Ira flips the coin. Danny does not flip the coin.
Horace flips the coin. Is the coin still heads up?
101 > 0
Table 16: The example of Coin Flip
30

---

Published as a conference paper at ICLR 2024
Take the last letters of the words in Elias Earnest Milton and concatenate them. The last letter of Elias -> The last letter of Earnest +>
The last letter of Milton >> n The answer is stn
Take the last letters of the words in Randolph Weldon Olin Robbicand concatenate them.
The last letter of Randolph
The last letter of Weldon
The last letter of Olin >> The last letter of Robbie >> e The answer is hnne
Table 17: The example of Last Letter Concatenation
G.3 ARITHMETICAL RESONING
Joan found 65466645546888999 seashells and Jessica found 38551966020636213 seashells on the beach. How many seashells did they find together ?
Joan started 65466345746884999 seashells. She gave some to Sam.
65466345746888999 3855196602063661 = 2691437972624878
The answer is 2691437976624878
Table 18: The example of Arithmetirc Reasoning
31


Reviews:
[{"review_id": "qbqhFBoho6", "reviewer": "Reviewer_NkHa", "summary": "- This paper proposes a new method called \"Neural Comprehension\" to enhance the symbolic reasoning capabilities of LLMs by integrating compiled neural networks (CoNNs), which are transformer-based networks with artificially designed attention weights to explicitly implement rules and transformations, thus can accurately perform symbolic computations.\n- Neural Comprehension outperforms baselines like fine-tuning and few-shot learning on length generalization, and achieves near perfect accuracy on symbolic operations like parity, reverse, addition, and subtraction.", "questions": "- In table 5 of appendix, it seems that you plug the neural comprehension into GPT-3. How did you achieve that when GPT-3 can only be accessed through APIs?\n- Have you ever tried the [GSM-hard](https://huggingface.co/datasets/reasoning-machines/gsm-hard) dataset? [1] I think the proposed method is helpful for problems that have more digits. I expect that this method will be useful for GSM-hard as it is intended to includes more digits in the answers. Can you try experiments on it and compare the results with PAL?\n\n\n\n[1] PAL: Program-aided Language Models. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, Graham Neubig https://arxiv.org/abs/2211.10435", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The gating mechanism naturally incorporates CoNN into LLMs and allows gradient optimization for the whole network. This feature makes the proposed method different from the existing tool-use papers. \n- Near perfect accuracy on symbolic tasks, significant gains on arithmetic reasoning, compared to the regular LLMs like GPT-3/3.5, especially when the number of digits goes up.\n- The proposed method results in less than 3% increase in the inference time, showing the potential to be applied to the existing models efficiently.", "weaknesses": "- Marginal performance improvement on real-world arithmetic reasoning tasks such as GSM8K. The result suggests that it may not be necessary for us to apply such neural compression into LLMs if the task is not related to strict symbolic operations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "osjYK8VJbu", "reviewer": "Reviewer_68he", "summary": "This paper addresses the limitations of language models (LMs) in handling deterministic symbolic reasoning and rule-based tasks due to their implicit learning from textual data. To enhance their rule comprehension ability, the authors introduce \"Neural Comprehension,\" which incorporates Compiled Neural Networks (CoNNs) into LMs. CoNNs are transformer-based neural networks that execute rules using artificially generated attention weights. This method enables LMs to effectively tackle rule-intensive challenges and improves their performance in symbolic reasoning tasks and real-world calculations.", "questions": "1. What does $I_{d_{L}}$ in Equation 1 mean? The author seems to have not provided an explanation. This seriously impedes my understanding of the crucial part of the proposed method;\n\n2. What does \"ICL\" mean? Which term is the abbreviation for?\n\n3. How are CoNNs compiled? How is the gradient $I_{d_{2}}$ for rule-based in Equation 2 obtained? Does it involve first training a specific task's CoNNs and then extracting the rule-based gradient, which is merged with the LM's gradient to influence the final gradient? If so, how are CoNNs trained? What is the training dataset like?\n\n4. From all the equations, I haven't seen an explanation of how the LM-based gradient and Rule-based gradient are combined. I feel very confused about this and need a more detailed explanation from the authors.\n\nTypos:\n\n1. ’=’  ->  ‘=’，  ’Select’  ->  ‘Select’\n\n2. “Neuralresulting”\n\n3. As depicted in Figure 5.4  ->  As depicted in Figure 7", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper introduces a novel approach, \"Neural Comprehension\" by incorporating Compiled Neural Networks (CoNNs) into language models. This approach is highly original as it addresses the limitations of LMs in handling deterministic symbolic reasoning, offering an innovative solution that combines neural networks and explicitly coded, interpretable CoNNs;\n\n2. The research methodology is well-founded, with empirical experiments showing superior performance compared to existing techniques. The study also includes a thorough analysis of combining multiple CoNNs, offering insights into model fusion. Besides, the AutoCoNN toolkit for automatic CoNN generation adds to the quality and scalability of the approach.", "weaknesses": "1. Even though the use of CoNNs to handle symbolic reasoning tasks is meticulously explained, the authors lack a well-structured explanation of \"Neural Comprehension\" and CoNNs, making it not very accessible to readers;\n\n2. As I understand it, the authors need to generate a corresponding CoNN for each rule-based task (such as Parity, Reverse, Subtraction, and so on) and integrate it with the LM, is that correct? If so, this approach seems to lack generality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3cIMFxqwoh", "reviewer": "Reviewer_M9Rq", "summary": "This work introduces Neural Comprehension, a method for combining language models with compiled neural networks (CoNNs) which allow explicit rules to be encoded as transformers. CoNNs can generalize more effectively on symbolic problems, but do not have the linguistic flexibility of LMs. My understanding is that Neural Comprehension works \"similar to MoE\" (mixture of experts), in that the token at a given position is decoded either by the LM or by a CoNN, depending on whether explicit reasoning is needed. The authors demonstrate that Neural Comprehension indeed generalizes better than a pure LM, especially for problems that might have very long length (e.g Figure 4).", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- LMs often do not generalize well in symbolic reasoning problems, so finding ways to improve this is very important\n- experiments are quite extensive\n- Neural Comprehension seems to work reasonably well on many of the tasks the authors look at", "weaknesses": "- Presentation is overall quite hard to follow. Authors should correct me if I misinterpret any parts of the method below\n- It is not clear when aspects of CoNNs are learned or if they are just compiled as fully operational symbolic functions. Gradients are mentioned, but the authors do not elaborate on exactly what these apply to\n- The combination of CoNNs and LMs seems quite simplistic here based on Figure 3 and section 4.1. For example, it seems that Neural Comprehension takes the output of the Addition CoNN when encountering '='. It is not clear whether there is a more sophisticated process used to decide when to apply CoNNs. If not, why is using CoNNs better than simply using external tools? \n- In a similar vein, it would strengthen the results to also compare to networks which use external tools. The authors frame the work in contrast to that, but it is not clear how those methods would compare", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7YuSrPxrLZ", "reviewer": "Reviewer_uMuv", "summary": "This paper proposes a method called \"Neural Comprehension\" to augment language models with the ability to perform symbolic reasoning and operations more robustly. The key idea is to incorporate compiled neural networks (CoNNs) that encode rules explicitly into the language model architecture.\n\nThe authors argue that standard language models struggle with symbolic tasks like arithmetic due to relying on statistical patterns learned implicitly from data. In contrast, CoNNs directly compile rules like addition into neural network modules using specialized attention weights. By integrating CoNNs in a plug-and-play manner, language models can leverage both their natural language understanding and the symbolic reasoning of CoNNs.\n\nThe proposed Neural Comprehension framework uses a gating mechanism to determine when to invoke a CoNN module versus the original language model decoder. Experiments on symbolic operations and arithmetic tasks demonstrate substantially improved performance over baseline methods.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(1) Proposes a novel way to impart symbolic reasoning abilities in language models using CoNNs. Addresses a clear limitation of standard LMs.\n\n(2) Achieves strong performance improvements on multiple symbolic and arithmetic datasets over baselines. Shows the benefit of combining statistical and symbolic knowledge.\n\n(3) CoNN integration does not require tool APIs. Comparable to tool-based methods while being end-to-end differentiable and not needing code generation.\n\n(4) Reasoning process is more interpretable compared to black-box LM predictions. Outputs show step-by-step reasoning.", "weaknesses": "(1) CoNN design and integration could be non-trivial for more complex reasoning tasks, e.g. compositional arithmetic reasoning tasks. Gating mechanism may work fine if we just want to add one skill. However, it will be hard to scale to thousands of skills for a specific language model and this exploration remains limited. Developing a specific model for a specific task involves significant human efforts and we want to build a model that can have multiple skills.\n\n(2) Although this paper claims it does not require tool API, in my opinion, this is not free and has very large costs. If a calculator is there, I think we should use it instead of building neural networks mimicking these tools. API is also required for internet-based retrieval and data analysis (OpenAI Code Interpreter ), etc. I think adding a python api is simpler than adding a neural module.\n\n(3) Experimental results shown in Table 5 on real-world reasoning tasks, e.g. GSM8K, are similar to baselines without significant improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)