Title: Mastering Symbolic Operations: Augmenting Language Models with Compiled Neural Networks
Venue: NeurIPS 2023
Accepted: no
Keywords: Language Models

Compiled Neural Networks

Neural Comprehension

Symbolic Operations

Length Generalization
Abstract: Language models (LMs) proficiency in handling deterministic symbolic reasoning and rule-based tasks remains limited due to their dependency implicit learning on textual data. To enable fully rule comprehension ability, we explore how to incorporate compiled neural networks (CoNNs) which weight is specially designed into the architecture of LMs, to achieve high accuracy and robust performance. CoNNs are transformer-based neural networks that execute rules through artificially generated attention weights. Our method, which call "Neural Comprehension", by incorporating CoNN modules into the LM, the framework effectively tackles rule-intensive challenges. Our experiments on symbolic reasoning tasks and real-world arithmetic reasoning tasks demonstrate the superior performance of our method compared to existing techniques. Furthermore, our LM achieves flawless execution on symbolic operations tasks, highlighting the potential of our method in enabling LMs to possess true symbolic comprehension capabilities.

1

Abstract

Language models (LMs) proficiency in handling deterministic symbolic reasoning and rule-based tasks remains limited due to their dependency implicit learning on textual data. To enable fully rule comprehension ability, we explore how to incorporate compiled neural networks (CoNNs) which weight is specially designed into the architecture of LMs, to achieve high accuracy and robust performance. CoNNs are transformer-based neural networks that execute rules through artificially generated attention weights. Our method, which call "Neural Comprehension", by incorporating CoNN modules into the LM, the framework effectively tackles ruleintensive challenges. Our experiments on symbolic reasoning tasks and real-world arithmetic reasoning tasks demonstrate the superior performance of our method compared to existing techniques. Furthermore, our LM achieves flawless execution on symbolic operations tasks, highlighting the potential of our method in enabling LMs to possess true symbolic comprehension capabilities.

1 Introduction

Language models (LMs), particularly large language models (LLMs), have exhibited impressive performance on complex reasoning tasks [Brown et al., 2020, Zhang et al., 2022a, Chowdhery et al., 2022, Wei et al., 2022d,a, Suzgun et al., 2022]. Despite this, the proficiency of LMs in tackling deterministic symbolic reasoning and rule-based tasks is still limited [Welleck et al., Razeghi et al., 2022]. For example, GPT-3's arithmetic performance declines with higher digit numbers [Brown et al., 2020], and its mathematical accuracy is influenced by word frequency in training data [Razeghi et al., 2022]. Moreover, length generalization [Anil et al., 2022] remains a challenge even for 100-billion-parameter models, such as GPT4 [Bubeck et al., 2023]. We hypothesize that these limitations stem from LMs' dependency on implicitly learning rules from textual data. During the training process, the primary objective of implicitly learning

based on gradient Updating is to minimize the loss associated with the given textual dataset. As 33

illustrated in Figure 1, a simple length generalization experiment using addition tasks with varying 34 numbers of digits highlights this limitation. Performance deteriorates as test length increases, indicat35 ing that these models strongly rely on statistical patterns in the data rather than capturing fundamental 36 logical structures. This reliance on implicit learning constrains LMs' accuracy in executing symbolic 37 operations tasks. As a result, their performance suffers when confronted with out-of-distribution and 38 rule-intensive tasks that require a more profound understanding of abstract rules. 39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

Neural

Comprehension

Comprehension Neural We propose a transformer-based language model framework, termed "Neural Comprehension", which synergistically integrates a pre-trained LM [Li et al., 2021b] and compiled neural networks (CoNNs) [Weiss et al., 2021] to achieve high accuracy and robust performance. CoNNs are neural networks but the rules are explicitly coded through transformer-liked structures and attention. Therefore CoNN is human-controllable, executing rules through artificially generated attention weights, and can achieve perfect accuracy once compiled network is done. Neural Comprehension relying solely on neural networks without requiring additional tools. It employs a token-bytoken generation method, analogous to GPT-3, where each token can be generated by either the pre-trained LM or one of the CoNNs. We comprises a pre-trained LM and multiple sets of CoNNs. The implementation of the Neural Comprehension framework facilitates the integration of rule-intensive abilities and reasoning capabilities into LMs, endowing them with genuine symbolic comprehension skills.

In this work, we conduct extensive experiments to evaluate the performance of our proposed Neural Comprehension method on a variety of rule-intensive tasks. Our experimental results demonstrate the effectiveness of our approach in comparison with existing state-of-the-art techniques, such as vanilla fine-tuning, few-shot learning, and Chain-of-Thought reasoning. Specifically, Neural Comprehension outperforms these methods in terms of accuracy, efficiency, and interpretability, showcasing its superiority in handling rule-intensive tasks. Our study presents a strong case for the deployment of Neural Comprehension in language models, highlighting its potential to transform the landscape of symbolic reasoning and language understanding capabilities.

Contributions

Our main contributions are as follows:

We pioneer the development and implementation of flawless execution rule-intensive symbolic operations for language models that rely on neural networks. By employing a versatile and interpretable method, we successfully integrate CoNNs, which are explicitly coded and human-controllable, into the language model. Our method facilitates direct rule deduction without the need for learning from conditional probabilities, leading to a more robust and effective approach. ( Section 3)

To expand the application field, we leverage the In-context learning ability of large language models to auto generate CoNN. Our method can be easily extended to various symbolic operations tasks. ( Appendix C )

Our experimental results on controllable symbolic reasoning tasks and real-world numerical calculation tasks demonstrate the superior performance of our method in comparison to existing techniques. Notably, our language model achieves flawless execution on symbolic reasoning tasks. ( Section 5.1 5.2 5.3)

We also studied the potential of combining multiple CoNNs and found that adding correlated CoNNs can continuously increase performance, while adding uncorrelated CoNNs rarely leads to performance degradation. This provides a new approach for model fusion, enabling the model to easily acquire new knowledge. ( Section 5.4)

2 Related Works

As model parameters, training calculations, and dataset sizes have increased, language models have gained new capabilities [Srivastava et al., 2022, Wei et al., 2022a], such as coding [Li et al., 2022b, Nijkamp et al., 2022], medical diagnosis [Li et al., 2021a, Xia et al., 2022], complex questionanswering [Zhu et al., 2022, Daull et al., 2023], cross-language translation [Fan et al., 2021, Li et al., 2022a], few-shot learning [Brown et al., 2020, Perez et al., 2021], and thought chaining [Wei et al., 2022c, Weng et al., 2022]. However, these models also exhibit limitations as they generally learn superficial patterns rather than the innate logic and rules of language. Consequently, humans often find it challenging to trust the results provided by language models [Sarker et al., 2021, Moore, 2022].

Pre-trained Language Models encompass those trained on general-purpose corpora [Lewis et al., 89 2019, Scao et al., 2022] and specialized symbolic tasks [Geva et al., 2020, Lewkowycz et al., 2022]. 90 They primarily aim to capture statistical patterns in language, which limits their capacity for symbolic 91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

reasoning. Symbolic reasoning involves manipulating abstract symbols and logical rules to derive new knowledge [Shindo et al., 2021, Yang and Deng, 2021] and necessitates the ability to extrapolate to novel situations and reason about concepts absent in the training data [Fujisawa and Kanai, 2022]. Due to the constraints of gradient learning, neural networks face challenges in wholly solving symbolic reasoning problems.

In-Context Learning has emerged as a promising approach to address these challenges [Dong et al., 2022] and closely approximate the predictors computed by gradient descent [Akyürek et al., 2022]. By prompting the language model to generate an explanation before generating an answer, the chain of thought [Wei et al., 2022c, Kojima et al., 2022, Zhang et al., 2022b, Zhou et al., 2022a] encourages the model to think sequentially. This technique has been employed in various numerical and symbolic reasoning tasks, such as scratchpad prompting [Nye et al., 2021] for length generalization [Anil et al., 2022] and utilizing the chain of thought to perform arithmetic operations like summing pairs of single digits with carry [Zhou et al., 2022b]. However, this approach often necessitates substantial computational resources, and achieving perfect accuracy remains challenging.

Augmented Language Models have been proposed as an alternative, supplementing language models with external tools [Mialon et al., 2023]. Examples include generating Python code for numerical reasoning [Gao et al., 2022, Chen et al., 2022] or incorporating tool usage as a pre-training task [Schick et al., 2023]. However, using external tools lacks a unified framework with language models and instead relies on the normativity of program generation. Consequently, if a task demands higher-level abstraction or intricate and robust capabilities, such as Redefine [Wei et al., 2022b], Autoformalization [Wu et al., 2022], and Theorem Proving [Wu et al., 2020], the language model may struggle to solve it, even if it possesses the ability to operate external tools [Zhou et al., 2022b].

3 Methods

3.1 Preliminaries

In-Context Learning (ICL) , Recent studies on ICL algorithms have shown that the learning process of language models within the ICL framework is analogous to gradient descent [Akyürek et al., 2022]. Specifically, transformer-based in-context learners implicitly implement standard learning algorithms by encoding smaller models in their activations and updating these implicit models as new examples appear in the context. However, these models face challenges in rule-intensive questions, as the rules represent abstract, high-dimensional knowledge that cannot be directly learned from the data, resulting in difficulties with implicit learning.

Compiled Neural Network (CoNN) . The flexibility of neural networks to adjust their weights is a unique characteristic not found in the human brain. We propose incorporating CoNNs into LLM architectures to leverage this feature. The CoNN is a transformer-based neural network leveraging artificially compiled attention weights to execute rules. A transformer model comprises multiple attention layers and Multi-Layer Perceptron (MLP) layers. Each attention layer facilitates interactions between tokens, with the multiplication of query and key elements representing a " Select " operation in CoNN. Subsequent multiplication with value elements indicates an " Aggregate " operation. The MLP layer is responsible for the token itself and is referred to as the " Zipmap " operation [Weiss et al., 2021]. Utilizing the three operations (Select, Aggregate, and Zipmap) to represent the sequenceto-sequence process, we can convert this information into transformer weights [Lindner et al., 2023]. By stacking multiple attention layers, CoNN can address various human-defined rule understanding problems, such as mathematical calculations and symbol operations 1 .

3.2 Neural Comprehension

Language models excel in language understanding tasks, while CoNNs achieve absolut accuracy in rule-intensive operation tasks using attention weights guided by abstract rules. To combine the language understanding capabilities of existing language models with accurate problem-solving for rule-based tasks (e.g., computation), we propose the Neural Comprehension, which integrates the language model's implicit learning parameters and CoNNs' explicit learning parameters. In Neural

…

…

To find out how much farther Stanley ran than walked, we

need to subtract the distance he walked from the distance he ran

364425 - 216582 = 147843

meters. Therefore, Stanley ran 147843

meters farther than he walked.

Compiled

Neural Network

Language Models with Compiled Neural Networks

Pre-training

Neural Network

...                                               ...

Both neural networks are similar to the decoder structure of transformer

…

…

216582

=

147843

meters

-

=

147843

216582

Output

Neural

Comprehension

Neural

Comprehension

Neural

Comprehension

Neural

Comprehension

Input

--

--

--

--

147843

meters

=

216582

-

148843

216582

-

=

147843

meters

…

…

Comprehension, CoNNs represent high-dimensional rules explicitly using multiple attention matrices 141 and incorporate these with the original LM's attention matrix. 142

143

144

145

146

147

148

149

150

151

As illustrated in Figure 2, we maintain the use of a decoder architecture to iteratively generate the subsequent context step by step. In particular, the language model encodes the context and produces the textual and reasoning process context D ( x ) step by step, while CoNNs handle sequence transformations involving rules. When a rule-required operation emerges, CoNN's attention is utilized to calculate specific values. The structure of Neural Comprehension is similar to MoE [Shazeer et al., 2017]. For example, when calculating 364425-216582 , the pre-trained language model output 148843 , which is incorrect. However, the Subtraction CoNN can correct the result to 147843 in the neural comprehension framework. This process encoded into context dynamically, improving intermediate results interpretability and final result accuracy.

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

Neural Comprehension combines LM and CoNNs in a piecewise function to perform gradient update.

LLM hidden state output is H L = ( H L 1 · · · H L d L ) ⊤ ∈ R d L , H L i ∈ (0 , 1) , and CoNN output is H C = ( H C 1 · · · H C d C ) ⊤ ∈ R d C , H C i ∈ (0 , 1) 2 . Specifically, we perform model fusion by adding the mapping from the last hidden layer representation to the vocabulary.

Within the Neural Comprehension, CoNNs manage sequence transformations involving rules. When the model encounters a rule-required operation, a gating mechanism determines whether to use CoNN's attention for computation. The gating mechanism assesses whether to maintain the initial output, provided by the pretrained language model, or modify it using the CoNN. where the model corrects the answer by applying a gradient to the in-context learning function through β . In Equation 1, since the hidden state output H C i elements of CoNN are { 0 , 1 } , when β = 0 , the model adopts the original decoding token of LM. When encountering a rule calculation problem, β = 1 , the model calculates the result by taking the maximum value of CoNN's hidden layer output H C and decodes the result from CoNN's vocabulary. Regarding the selection of β , since the CoNN involved in this paper is relatively simple, it is determined by the forward computation results of CoNN. For example, when we set up an Addition CoNN, we specify that the final result should be output when

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

encountering '=', so when encountering '=', β = 1 . However, for larger-scale CoNN, we recommend that a learnable gating network determine β .

3.3 Gradient Modification in Neural Comprehension

To better appreciate the benefits of our method in handling rule-intensive tasks and improving accuracy, it is crucial to understand the gradient perspective of ICL. The optimization process in ICL can be viewed as a search for suitable gradients to minimize the loss function. Due to the implicit learning nature of standard ICL methods, gradients learned from data may not always be ideal for addressing rule-intensive tasks. Therefore, our proposed method introduces an explicit learning component to provide more appropriate gradient updates for such tasks, ultimately leading to enhanced overall performance. In this section, we focus on elucidating the changes in the gradient introduced by the Neural Comprehension model.

The gradient of the model during the execution of ICL can be partitioned into two categories based on the origin of the gradients:

Here, I d 1 represents the gradients derived implicitly from the language model (LM) and corresponds to the text-based learning aspect of the model. Conversely, I d 2 represents the gradients explicitly derived from the CoNNs, encoding rule-based knowledge. The Neural Comprehension model integrates both gradient sources to optimize the ICL process.

In linear regression problems, the loss function can be expressed as a piecewise function according 184 to 1, here P 1 ( x ) is the LLM and P 2 ( x ) is CONN, the In-context-learner can be separate into two 185 process : 186

Based on the partitioned gradient as defined in Equation 2, the overall gradient of the Neural 187 Comprehension model can be obtained by computing their individual gradients concerning the 188 respective β : 189

This partitioning allows the Neural Comprehension model to specifically address the gradient require190 ments of both implicit learning via LM and explicit learning via CoNNs. It is crucial to note that 191 CoNNs are designed to minimize the loss associated with rule-based tasks, essentially providing an 192 optimal gradient for tasks involving rule-intensive operations. This leads to a substantial improvement 193 in the model's accuracy for rule-based tasks, as the gradient updates provided by CoNNs are more 194 suitable for rule learning compared to the initially available gradients from the LM. By amalgamating 195 the both of gradient sources, the Neural Comprehension model achieves a more refined optimization 196 of in-context learning. Additionally, from the perspective of gradients, our approach surpasses 197 conventional data-driven implicit learning techniques as it integrates explicit rule-based learning 198 mechanisms that exhibit more suitable gradient updates for rule-intensive questions. The Neural 199 Comprehension model effectively balances the need for implicit and explicit learning within the ICL 200 framework, leading to an enhanced overall performance in terms of accuracy and interpretability. 201

202

203

204

205

4 Experimental Settings

In this study, we primarily explore the capacity of language models to address symbolic reasoning tasks, concentrating on three areas: symbolic operations, symbolic reasoning, and arithmetic reasoning.

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

Symbolic Operations Building upon the approaches developed by Anil et al. [2022] and Qian et al. [2022], we examine the following tasks: Parity, Reverse, Addition and Subtraction. These tasks do not require complex text understanding, but only require faithfully implementing symbolic operations and outputting the corresponding results.

Symbolic Reasoning We employ the experimental framework of Wei et al. [2022c] for the two tasks, Last Letter Concatenation and Coin Flip. These tasks require a combination of language understanding and rule comprehension abilities.

Arithmetic Reasoning To evaluate the method's generalization ability from symbolic operations to arithmetic reasoning in addition and subtraction tasks, we use five established arithmetic reasoning datasets: AddSub [Hosseini et al., 2014], SingleEq [Koncel-Kedziorski et al., 2015], MultiArith [Roy and Roth, 2016], GSM8K [Cobbe et al., 2021], and SVAMP [Arkil et al., 2021]. Additionally, we introduce the AddSub + dataset, containing tasks of varying complexity based on the number of digits involved in arithmetic operations, ranging from 1-digit addition to 20-digit addition/subtraction tasks.

5 Ecperiment and Result

5.1 Symbolic Tasks



































  



















































  

















































  



















































  















In this study, we conduct a length generalization experiment [Anil et al., 2022] to examine the 221 distinctions between the Neural Comprehension and learning-based methods, as depicted in Figure 3. 222 Our experimental design encompasses 1000 × 40 independent test sets, comprising problems with 223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

varying digit lengths from 1 to 40 digits. 10 to 20 digits within the range are provided by us for methods based on implicit learning for training; during the testing phase, this range is called In-Dist. Furthermore, we present results for both Scratchpad [Anil et al., 2022] and Algorithmic [Zhou et al., 2022b] approaches.

The results of our experiment demonstrate that the Vanilla Fine-tune (red lines) method performs optimally on the in-domain (10-20 digit) training set, while its performance deteriorates for both more simplistic and more intricate. This finding suggests that the absence of relevant samples in the training set may cause gradient descent-based language models to underperform on both simpler and more complex tasks. As further discussed in the appendix D.1 , this phenomenon can be attributed to the inherent generalization limitations of statistical models and the position bias of language models.

Considering the Vanilla Few-shot method (green lines), we determine that its performance is less impacted by the prompt sample range compared to Vanilla Fine-tune. Large language models, which are trained on extensive text corpora, excel at solving more straightforward problems such as symbolic operations within a ten-digit range. Nevertheless, performance remains below par for test sets with more than ten digits, even when prompted with 10-20 digit samples.

Observing CoT-like methods (we use GPT-3.5), including Scratchpad and Algorithmic, unveils their robust length generalization capabilities. Scratchpad works by requiring large language models to record intermediate steps, while Algorithmic employs a similar approach to record the carry operations involved in the addition process. This can be primarily attributed to their proficiency in decomposing complex problems into smaller incremental steps and maintaining intermediate states. However, these methods necessitate substantial computational resources, and extending the length beyond the input limit of the model becomes challenging.

Our study reveals that Neural Comprehension attains remarkably high accuracy in symbolic operations. This implies that Neural Comprehension, unlike conventional methods, does not rely on training data and remains unaffected by discrepancies in input lengths for in-distribution and out-of-distribution data. Consequently, it alleviates the requirement for step-by-step work tracking, and language models with CoNNs only need relatively fewer computational steps to execute sequence operations directly. Encoding rules into neural network modules endows us with greater interpretability, enabling language models to flawlessly perform purely symbolic operation tasks.

5.2 Symbolic Reasoning

In this section, we investigate the performance of Neural Comprehension in terms of symbolic reasoning capabilities. Our hypothesis is that, although pretrained Language Models (LMs) demonstrate strong language understanding abilities, they lack the capacity to deduce and comprehend rules regarding symbolic reasoning tasks. Thus, we aim to evaluate whether the incorporation of compiled neural networks in the form of CoNNs can address this limitation and improve the LM's symbolic reasoning abilities.

To assess the performance of the rule comprehension component (CoNNs) in symbolic reasoning, we devise an experiment that measures the model's accuracy using intermediate processes and represents them in a "Chain of Thought"-like manner. In doing so, the experiment decomposes language understanding and rule comprehension explicitly into simpler outputs, avoiding the complexities of reasoning and additional error propagation in the models. Example outputs from this approach can be found in Appendix F . We observed that neural com-

0

6K

12K

18K

24K

0%

20%

40%

60%

80%

100%

Coin Flip

Iterations

T5 Small:60M

0

6K

12K

18K

24K

0%

20%

40%

60%

80%

100%

Iterations

0

6K

12K

18K

24K

0%

20%

40%

60%

80%

100%

Iterations

0

6K

12K

18K

24K

0%

20%

40%

60%

80%

100%

Last Letter Concatenation

Iterations

0

6K

12K

18K

24K

0%

20%

40%

60%

80%

100%

Iterations

0

6K

12K

18K

24K

0%

20%

40%

60%

80%

100%

Iterations

Neural Comprehension

Direct Fine-tune

Improve Performance

prehension improves the symbolic reasoning capabilities of pre-trained language models in most 278 cases (Neural Comprehension almost always outperforms Vanilla Fine-tune in Figure 4), and can fit 279 faster. This observation suggests that the introduction of compiled neural networks has a positive 280 impact on pretrained LMs, addressing rule comprehension limitations in symbolic reasoning tasks. 281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

5.3 Arithmetic Reasoning





























































































Arithmetic reasoning serves as a suitable testbed for evaluating language models and their ability to address real-world problems. In this study, we examine the AddSub + dataset variants that involve different digit lengths, utilizing the Addition and Subtraction models from the CoNNs family. Notably, the capabilities of Neural Comprehension extend beyond these tasks, as CoNNs can also simulate calculators that support multiplication and division operations, and potentially perform linear algebra computations or even in-context learning algorithms that employ backpropagation [Giannou et al., 2023].

To evaluate the impact of Neural Comprehension on arithmetic reasoning, we compare the output of vanilla CoT language models and those incorporating Neural Comprehension, using the vanilla CoT baseline as a reference. As demonstrated in Figure 5, the vanilla CoT model struggles to extrapolate and solve arithmetic problems involving longer digit lengths. However, integrating Neural Comprehension significantly improves the performance of language models on such complex arithmetic tasks. Since we only incorporated the Addition and Subtraction CoNNs, we attribute the observed performance enhancement to the increased computational accuracy of the language model. For further evidence, we present additional experimental results on widely-used arithmetic reasoning datasets in Appendix D.2 , which reinforce the benefits of using Neural Comprehension over the vanilla CoT model.

In comparison to language models employing external tools like PAL [Gao et al., 2022], our findings suggest that generating accurate code for the less code-trained GLM-130B model might be challenging for PAL, resulting in performance levels inferior to those of the vanilla CoT. This outcome indicates that language models offer greater flexibility, whereas external tools may have difficulties in more complex or unique situations. The integration of compiled neural networks appears to be a more promising approach, as evidenced by the performance improvements observed in our experiments.

Specifically, when language models encounter intricate arithmetic tasks that involve nested operations or multi-step calculations, the integrated CoNNs can efficiently handle these operations, allowing the language model to focus on higher-level reasoning. In contrast, the use of external tools often requires explicit coding and may not generalize effectively to more complicated scenarios. In conclusion, our results demonstrate that incorporating compiled neural networks into language models provides a more robust and versatile solution for arithmetic reasoning and related challenges, underlining the superiority of this approach over external tools such as PAL.

5.4 Ablation and Analyses: Module Combination for Neural Comprehension

Efficiently deploying multiple CoNNs is crucial for achieving exceptional Neural Comprehension performance. As depicted in Figure 4, the amalgamation of distinct CoNNs, tailored for both symbolic

GLYPH(cmap:d835)

+ 1 GLYPH(cmap:d835)

GLYPH(cmap:d835)

+ GLYPH(cmap:d835) GLYPH(cmap:d835)

A Pre-trained Model with a CoNN

A Pre-trained Model with some CoNNs

(Increase from left to right)



























































































































































































































































and arithmetic reasoning tasks within the language model framework, can lead to remarkable benefits. 316 It is observed that integrating pertinent CoNNs bolsters the performance of the initial language model, 317 whereas the inclusion of unrelated language models rarely causes detrimental effects, regardless of 318 whether single or multiple CoNNs are combined. 319

320

321

322

323

324

325

326

This can be ascribed to the refined design of the Neural Comprehension framework, which ensures the precise execution of assigned tasks by CoNNs without interference from irrelevant modules. Each CoNN module is adept at generating the appropriate output when needed, thereby preventing the emergence of erroneous results from unrelated components. Importantly, as seen in Appendix B.3 , the parameter count for each CoNN module ranges from 1/1000 to 1/1000000 of that for GPT-3, and the experiments in Appendix D.3 show that the inference latency in the neural understanding framework only increases by 1%-3% compared to Vanilla.

327

328

329

330

331

332

This observation underscores the remarkable scalability of the Neural Comprehension framework, which possesses the capability to not only accommodate existing knowledge concepts but also assimilate novel ones as the number of CoNNs expands. Theoretically, the integration of tens of thousands of CoNN modules within language models holds the potential to foster a comprehensive understanding of concepts.

6 Conclusion

We have observed that pretrained language models lack an intrinsic comprehension of rule-based 333 concepts and explored how Neural Comprehension can integrate compiled neural networks into the 334 language model framework in a simple and generic manner. We demonstrated the superiority of our 335 approach over existing learning-based method, Without external tools, our approach enables language 336 models to perform nearly perfect symbolic operations and can be applied to more realistic arithmetic 337 reasoning tasks. 338

Our study opens new avenues for language models, such as the investigation of more complex CoNNs 339 related to higher-order abstract reasoning, the development of more advanced gating mechanisms for 340 smoother integration, and the exploration of other domains in which Neural Comprehension could 341 exhibit significant advantages. Furthermore, our framework provides a foundation for future work on 342 unifying both implicit and explicit learning in language models and facilitating the seamless. 343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

References

Akyürek, D. Schuurmans, J. Andreas, T. Ma, and D. Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661 , 2022.

Anil, Y. Wu, A. J. Andreassen, A. Lewkowycz, V. Misra, V. V. Ramasesh, A. Slone, G. Gur-Ari, E. Dyer, and B. Neyshabur. Exploring length generalization in large language models. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=zSkYVeX7bC4 .

Arkil, B. Satwik, and G. Navin. Are nlp models really able to solve simple math word problems? 2021.

Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.

Chen, X. Ma, X. Wang, and W. W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588 , 2022.

Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

Cobbe, V. Kosaraju, M. Bavarian, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021.

Daull, P. Bellot, E. Bruno, V. Martin, and E. Murisasco. Complex qa and language models hybrid architectures, survey. arXiv preprint arXiv:2302.09051 , 2023.

Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui. A survey for in-context learning. arXiv preprint arXiv:2301.00234 , 2022.

Fan, S. Bhosale, H. Schwenk, Z. Ma, A. El-Kishky, S. Goyal, M. Baines, O. Celebi, G. Wenzek, V. Chaudhary, et al. Beyond english-centric multilingual machine translation. The Journal of Machine Learning Research , 22(1):4839-4886, 2021.

Fujisawa and R. Kanai. Logical tasks for measuring extrapolation and rule comprehension. arXiv preprint arXiv:2211.07727 , 2022.

Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. Pal: Program-aided language models. arXiv preprint arXiv:2211.10435 , 2022.

Geva, A. Gupta, and J. Berant. Injecting numerical reasoning skills into language models. arXiv preprint arXiv:2004.04487 , 2020.

Giannou, S. Rajput, J. yong Sohn, K. Lee, J. D. Lee, and D. Papailiopoulos. Looped transformers as programmable computers, 2023.

J. Hosseini, H. Hajishirzi, O. Etzioni, and N. Kushman. Learning to solve arithmetic word problems with verb categorization. empirical methods in natural language processing , 2014.

Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=e2TBb5y0yFf .

Koncel-Kedziorski, H. Hajishirzi, A. Sabharwal, O. Etzioni, and S. D. Ang. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics , 2015.

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 , 2019.

Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858 , 2022.

Li, E. Chen, H. Liu, Y. Weng, B. Sun, S. Li, Y. Bai, and M. Hu. More but correct: Generating diversified and entity-revised medical response. arXiv e-prints , pages arXiv-2108, 2021a.

Li, Y. Weng, B. Sun, and S. Li. A multi-tasking and multi-stage chinese minority pre-trained language model. In T. Xiao and J. Pino, editors, Machine Translation , pages 93-105, Singapore, 2022a. Springer Nature Singapore. ISBN 978-981-19-7960-6.

Li, T. Tang, W. X. Zhao, and J.-R. Wen. Pretrained language models for text generation: A survey, 2021b.

Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al. Competition-level code generation with alphacode. Science , 378(6624): 1092-1097, 2022b.

Lindner, J. Kramár, M. Rahtz, T. McGrath, and V. Mikulik. Tracr: Compiled transformers as a laboratory for interpretability. arXiv preprint arXiv:2301.05062 , 2023.

Mialon, R. Dessì, M. Lomeli, C. Nalmpantis, R. Pasunuru, R. Raileanu, B. Rozière, T. Schick, J. Dwivedi-Yu, A. Celikyilmaz, et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842 , 2023.

Moore. Language models understand us, poorly. arXiv preprint arXiv:2210.10684 , 2022.

Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474 , 2022.

Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114 , 2021.

Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. 2022.

Perez, D. Kiela, and K. Cho. True few-shot learning with language models. Advances in neural information processing systems , 34:11054-11070, 2021.

Qian, H. Wang, Z. Li, S. Li, and X. Yan. Limitations of language models in arithmetic and symbolic induction. arXiv preprint arXiv:2208.05051 , 2022.

Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research , 21(1):5485-5551, 2020.

Razeghi, R. L. Logan IV, M. Gardner, and S. Singh. Impact of pretraining term frequencies on few-shot reasoning. arXiv preprint arXiv:2202.07206 , 2022.

Roy and D. Roth. Solving general arithmetic word problems. arXiv: Computation and Language , 2016.

Sarker, L. Zhou, A. Eberhart, and P. Hitzler. Neuro-symbolic artificial intelligence: Current trends. arXiv: Artificial Intelligence , 2021.

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ili´ c, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 , 2022.

Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761 , 2023.

Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. CoRR , abs/1701.06538, 2017. URL http://arxiv.org/abs/1701.06538 .

Shindo, D. S. Dhami, and K. Kersting. Neuro-symbolic forward reasoning. arXiv preprint arXiv:2110.09383 , 2021.

A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, A. Kluska, A. Lewkowycz, A. Agarwal, A. Power, A. Ray, A. Warstadt, A. W. Kocurek, A. Safaya, A. Tazarv, A. Xiang, A. Parrish, A. Nie, A. Hussain, A. Askell, A. Dsouza, A. Slone, A. Rahane, A. S. Iyer, A. Andreassen, A. Madotto, A. Santilli, A. Stuhlmüller, A. Dai, A. La, A. Lampinen, A. Zou, A. Jiang, A. Chen, A. Vuong, A. Gupta, A. Gottardi, A. Norelli, A. Venkatesh, A. Gholamidavoodi, A. Tabassum, A. Menezes, A. Kirubarajan, A. Mullokandov, A. Sabharwal, A. Herrick, A. Efrat, A. Erdem, A. Karaka{ ¸s}, B. R. Roberts, B. S. Loe, B. Zoph, B. Bojanowski, B. Özyurt, B. Hedayatnia, B. Neyshabur, B. Inden, B. Stein, B. Ekmekci, B. Y. Lin, B. Howald, C. Diao, C. Dour, C. Stinson, C. Argueta, C. F. Ramírez, C. Singh, C. Rathkopf, C. Meng, C. Baral, C. Wu, C. Callison-Burch, C. Waites, C. Voigt, C. D. Manning, C. Potts, C. Ramirez, C. E. Rivera, C. Siro, C. Raffel, C. Ashcraft, C. Garbacea, D. Sileo, D. Garrette, D. Hendrycks, D. Kilman, D. Roth, D. Freeman, D. Khashabi, D. Levy, D. M. González, D. Perszyk, D. Hernandez, D. Chen, D. Ippolito, D. Gilboa, D. Dohan, D. Drakard, D. Jurgens, D. Datta, D. Ganguli, D. Emelin, D. Kleyko, D. Yuret, D. Chen, D. Tam, D. Hupkes, D. Misra, D. Buzan, D. C. Mollo, D. Yang, D.-H. Lee, E. Shutova, E. D. Cubuk, E. Segal, E. Hagerman, E. Barnes, E. Donoway, E. Pavlick, E. Rodola, E. Lam, E. Chu, E. Tang, E. Erdem, E. Chang, E. A. Chi, E. Dyer, E. Jerzak, E. Kim, E. E. Manyasi, E. Zheltonozhskii, F. Xia, F. Siar, F. Martínez-Plumed, F. Happé, F. Chollet, F. Rong, G. Mishra, G. I. Winata, G. de Melo, G. Kruszewski, G. Parascandolo, G. Mariani, G. Wang, G. Jaimovitch-López, G. Betz, G. Gur-Ari, H. Galijasevic, H. Kim, H. Rashkin, H. Hajishirzi, H. Mehta, H. Bogar, H. Shevlin, H. Schütze, H. Yakura, H. Zhang, H. M. Wong, I. Ng, I. Noble, J. Jumelet, J. Geissinger, J. Kernion, J. Hilton, J. Lee, J. F. Fisac, J. B. Simon, J. Koppel, J. Zheng, J. Zou, J. Koco´ n, J. Thompson, J. Kaplan, J. Radom, J. Sohl-Dickstein, J. Phang, J. Wei, J. Yosinski, J. Novikova, J. Bosscher, J. Marsh, J. Kim, J. Taal, J. Engel, J. Alabi, J. Xu, J. Song, J. Tang, J. Waweru, J. Burden, J. Miller, J. U. Balis, J. Berant, J. Frohberg, J. Rozen, J. Hernandez-Orallo, J. Boudeman, J. Jones, J. B. Tenenbaum, J. S. Rule, J. Chua, K. Kanclerz, K. Livescu, K. Krauth, K. Gopalakrishnan, K. Ignatyeva, K. Markert, K. D. Dhole, K. Gimpel, K. Omondi, K. Mathewson, K. Chiafullo, K. Shkaruta, K. Shridhar, K. McDonell, K. Richardson, L. Reynolds, L. Gao, L. Zhang, L. Dugan, L. Qin, L. ContrerasOchando, L.-P. Morency, L. Moschella, L. Lam, L. Noble, L. Schmidt, L. He, L. O. Colón, L. Metz, L. K. { ¸S}enel, M. Bosma, M. Sap, M. ter Hoeve, M. Farooqi, M. Faruqui, M. Mazeika, M. Baturan, M. Marelli, M. Maru, M. J. R. Quintana, M. Tolkiehn, M. Giulianelli, M. Lewis, M. Potthast, M. L. Leavitt, M. Hagen, M. Schubert, M. O. Baitemirova, M. Arnaud, M. McElrath, M. A. Yee, M. Cohen, M. Gu, M. Ivanitskiy, M. Starritt, M. Strube, M. Sw{ ˛e}drowski, M. Bevilacqua, M. Yasunaga, M. Kale, M. Cain, M. Xu, M. Suzgun, M. Tiwari, M. Bansal, M. Aminnaseri, M. Geva, M. Gheini, M. V. T, N. Peng, N. Chi, N. Lee, N. G.-A. Krakover, N. Cameron, N. Roberts, N. Doiron, N. Nangia, N. Deckers, N. Muennighoff, N. S. Keskar, N. S. Iyer, N. Constant, N. Fiedel, N. Wen, O. Zhang, O. Agha, O. Elbaghdadi, O. Levy, O. Evans, P. A. M. Casares, P. Doshi, P. Fung, P. P. Liang, P. Vicol, P. Alipoormolabashi, P. Liao, P. Liang, P. Chang, P. Eckersley, P. M. Htut, P. Hwang, P. Mi{ł}kowski, P. Patil, P. Pezeshkpour, P. Oli, Q. Mei, Q. Lyu, Q. Chen, R. Banjade, R. E. Rudolph, R. Gabriel, R. Habacker, R. R. Delgado, R. Millière, R. Garg, R. Barnes, R. A. Saurous, R. Arakawa, R. Raymaekers, R. Frank, R. Sikand, R. Novak, R. Sitelew, R. LeBras, R. Liu, R. Jacobs, R. Zhang, R. Salakhutdinov, R. Chi, R. Lee, R. Stovall, R. Teehan, R. Yang, S. Singh, S. M. Mohammad, S. Anand, S. Dillavou, S. Shleifer, S. Wiseman, S. Gruetter, S. R. Bowman, S. S. Schoenholz, S. Han, S. Kwatra, S. A. Rous, S. Ghazarian, S. Ghosh, S. Casey, S. Bischoff, S. Gehrmann, S. Schuster, S. Sadeghi, S. Hamdan, S. Zhou, S. Srivastava, S. Shi,

Singh, S. Asaadi, S. S. Gu, S. Pachchigar, S. Toshniwal, S. Upadhyay, S. S. Debnath, S. Shakeri, 491

Thormeyer, S. Melzi, S. Reddy, S. P. Makini, S.-H. Lee, S. Torene, S. Hatwar, S. Dehaene, 492

Divic, S. Ermon, S. Biderman, S. Lin, S. Prasad, S. T. Piantadosi, S. M. Shieber, S. Misherghi, 493

Kiritchenko, S. Mishra, T. Linzen, T. Schuster, T. Li, T. Yu, T. Ali, T. Hashimoto, T.-L. Wu, 494

Desbordes, T. Rothschild, T. Phan, T. Wang, T. Nkinyili, T. Schick, T. Kornev, T. Telleen-Lawton, 495

496

T. Tunduny, T. Gerstenberg, T. Chang, T. Neeraj, T. Khot, T. Shultz, U. Shaham, V. Misra, V. Dem-

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

berg, V. Nyamai, V. Raunak, V. Ramasesh, V. U. Prabhu, V. Padmakumar, V. Srikumar, W. Fedus,

W. Saunders, W. Zhang, W. Vossen, X. Ren, X. Tong, X. Zhao, X. Wu, X. Shen, Y. Yaghoobzadeh,

Lakretz, Y. Song, Y. Bahri, Y. Choi, Y. Yang, Y. Hao, Y. Chen, Y. Belinkov, Y. Hou, Y. Hou,

Bai, Z. Seid, Z. Zhao, Z. Wang, Z. J. Wang, Z. Wang, and Z. Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. 2022.

Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261 , 2022.

Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022a.

Wei, Y. Tay, and Q. V. Le. Inverse scaling can become u-shaped. 2022b.

Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 , 2022c.

Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. H. Chi, Q. V. Le, D. Zhou, et al. Chain-ofthought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems , 2022d.

Weiss, Y. Goldberg, and E. Yahav. Thinking like transformers. In International Conference on Machine Learning , pages 11080-11090. PMLR, 2021.

Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston. Neural text generation with unlikelihood training. In International Conference on Learning Representations .

Weng, M. Zhu, S. He, K. Liu, and J. Zhao. Large language models are reasoners with selfverification. arXiv preprint arXiv:2212.09561 , 2022.

Wu, A. X. Jiang, J. Ba, and R. Grosse. Int: An inequality benchmark for evaluating generalization in theorem proving. arXiv: Artificial Intelligence , 2020.

Wu, A. Q. Jiang, W. Li, M. N. Rabe, C. Staats, M. Jamnik, and C. Szegedy. Autoformalization with large language models. 2022.

Xia, B. Li, Y . Weng, S. He, K. Liu, B. Sun, S. Li, and J. Zhao. Medconqa: Medical conversational question answering system based on knowledge graphs. In Proceedings of the The 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 148-158, 2022.

Yang and J. Deng. Learning symbolic rules for reasoning in quasi-natural language. arXiv preprint arXiv:2111.12038 , 2021.

Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022a.

Zhang, A. Zhang, M. Li, and A. Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493 , 2022b.

Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, O. Bousquet, Q. Le, and E. Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625 , 2022a.

Zhou, A. Nova, H. Larochelle, A. Courville, B. Neyshabur, and H. Sedghi. Teaching algorithmic 538 reasoning via in-context learning. arXiv preprint arXiv:2211.09066 , 2022b. 539

Zhu, Y. Weng, S. He, K. Liu, and J. Zhao. Reasonchainqa: Text-based complex question answering 540 with explainable evidence chains. arXiv preprint arXiv:2210.08763 , 2022. 541

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLarge language models lack expertise skills and this is reflected in their limited capability for arithmetic etc.\n\nThe paper proposes a method to integrate a CoNN (compliled neural networks) into an LLM via gating. Such integrations allow better performance for rule intensive tasks such as symbolic reasoning, arithmetic reasoning etc. To perform this, the paper proposes a gating mechanism although the implementation seems rule based triggering (line 167).\n\nWith the proposed mechanisms, the authors evaluate on arithmetic tasks (5.1) where the model achieves consistently 100% accuracy. On symbolic reasoning (5.2) and arithmetic reasoning, the method also performs better than finetuning appraoches in terms of performance or data efficiency.\n\nSTRENGTHS:\nThe paper correctly identifies the limitations of LLMs and propose a novel approach to tackle the problem. The solution consists of a neural machine that serves as expert for symbolic/arithmetic operations.\n\nEmpirically, the paper demonstrates strong results over competitive baselines on arithmetic/symbolic reasoning approaches.\n\nWEAKNESSES:\nThe novelty introduced in the paper doesn't quality a Neurips paper. The novelty of the paper concretely is using the rule based trigger to combine a CoNN with an LLM; neither components consist of the paper novelty. \n\nThere are various presentation issues that make the paper quite hard to follow:\n- The second contribution item is put in Appendix, I read it but please note that even reviewers are not obliged to read those materials in Appendix to judge the paper. Related to this remark, CoNN is only introduced and referenced, for people not familiar with the technology, there is nowhere in the paper to know how it works.\n- Section 3.2 introduces the gating between LLM and CoNN, I think equation (1) has a flaw since it involves choose argmax from a matrix which seems ill defined and I think the authors mean to concatenate HL and HC instead. The gradient flow (3.3) doesn't give further insight rather than it is a gating mechanism. \n- I am confused by the illustrative figures in the paper. Figure 2 has the input text on top which is easily confused to a paper title and I feel Figure 1 is not relevant to the paper.\n\nQUESTIONS:\nFor experiments other than 5.1, the model doesn't achieve 100% accuracy (even for tasks like the last letter). How do authors explain this difference please? Is it because some arithemetic CoNN has been implemented while the last letter one has to be learnt?\n\nLine 162, when beta equals 1, left side of equation one is an identity matrix, so the model seems to choose between HL and HC max, but the authors say that it will take the HC max, why?\n\nLIMITATIONS:\nUnless Neural Comprehension machines are widely used, I don't see why this approach is particularly useful: I don't see the advantage compared to API calling approach adopted by today's industry. Particularly, as shown in this paper, many operations have to be implemented individually (subtraction, addition, etc.)", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors have proposed a novel way to augment large language model called Neural Comprehension to improve symbolic reasoning in tasks where rule-based execution is required by design such as numbers summation. The core idea behind their method is to augment the LM with compiled NN (CoNN) for a specific task is a manner of mixture of experts where they design a policy which determines LM or CoNN will be executing the next token prediction at each time step. In addition, they described how their method could be used with in-context learning (ICL). Authors perform set of experiments in symbolic operations (parity, reverse. addition, subtraction), symbolic reasoning (concat, coin flip) and arithmetic reasoning. They show empirically how their method outperforms stand-alone LMs finetuned on corresponding tasks data. Finally they show a potential of combining multiple CoNN with the given LM to increase the task capability of the final Neural Comprehension model.\n\nSTRENGTHS:\nThis work proposes an original way to combine LMs with CoNN networks and analyze the performance using multiple correlated or uncorrelated CoNNs together. \nThe wide range of symbolic experimental tasks show that authors performed high quality experimental investigation.\n\nWEAKNESSES:\nI think major weaknesses of this work is (1) a hardcoded structure of CoNNs under consideration and (2) hard coded policy of choosing the LM vs CoNN component by connecting that to task-based properties. I believe the most interesting part would be to learn the beta factor which also seems to be very challenging. \n\nAuthors claimed that their work suggests potential improvements from using Neural Comprehension in other tasks, but they did not mention how to get CoNNs for these tasks? In general, the discussion about CoNN design and implementation is somewhat skipped in the paper while it seem to be a crucial factor in this paper's impact.\n\nQUESTIONS:\nThere are some grammar and syntax typos in the paper, even in the section titles, please fix that.\n\nA clarification question: from my understanding you have done some hard coding of the beta policy such that i.e. it only triggers ICL gradient when it sees \"=\" token. Does this mean that this approach will not work at all if you use sequence \"4 + 4 equals 8\"? If so, do you have any ideas in mind how to make it work? In general I like this idea of plug-in CoNNs, but they need to be more seamless and fluent without requiring such hard coding in the task level.\n\nLIMITATIONS:\nAuthors discuss some statistical limitation aspects of LMs in the appendix and how their proposed method alleviates that. However, I did not find explicit discussion of limitations of their own approach except for describing future work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWhile Large Language Models show promise for a wide swath of tasks, they are lacking when applied to symbolic reasoning tasks. To overcome this limitation, the authors propose to employ Compiled Neural Networks (CoNNs). They create networks specialised to arithmetic and symbolic tasks and propose a mechanism by which an LLM can propagate the gradient through CoNNs to better learn to solve symbolic reasoning tasks. They demonstrate improvement in pure symbolic manipulation (parity and reverse), arithmetic (addition and subtraction), and more complex symbolic reasoning (coin flip and last letter concatenation). They demonstrate better generalisation to out-of-distribution examples (proxied by digit length or sequence length for the first four tasks), a considerable improvement on LLC, and parity with a larger LLM when augmenting a smaller one (T5 small + NC v vanilla T5 large). The improvements are comparable to external tool-based approaches, however, hold a promise of better integration (as gradients can propagate without surrogates), and the mixture-of-experts style combination can be learnt rather than rule-based.\n\nSTRENGTHS:\n- By construction, CoNNs are interpretable from their basic building blocks, ensuring that paths that do go through them can be interpreted according to the rules they encode.\n- The reduced number of parameters holds promise for reducing the cost of language models (relative to GPT-3)\n- Even with simple gating, there is a non-trivial improvement on tasks when multiple CoNNs are employed (Section 5.4)\n\nWEAKNESSES:\n- There is an implicit assumption on practitioners to know what rules to expect and generate appropriate networks that then get used MoE style.\n- Expanding on the previous, practitioners should be able to translate their rules, from, for example, regular expressions, to RASP to enable compilation to a NN.\n\nQUESTIONS:\n- While orthogonal to the paper, perhaps a remark on the difficulty of creating CoNNs appropriate for a task should be briefly discussed. For example, Section C.1 does discuss employing an LLM and ICL to obtain networks that approximate specific rules, how difficult is it to source examples for ICL? How many examples are needed before the output is of high enough quality? Can the correctness of the proposed CoNN be assessed by a non-expert? (as this last would be the use case, it is safe to assume an expert could write the network directly)\n\n### Discussion Phase\n\nThe authors have clearly addressed the concerns raised during the rebuttal with clear and, as necessary, additional data. As long as such data gets exposed into the paper, via appendix, for example, I feel the paper has further improved.\n\nLIMITATIONS:\nThe authors have address most limitations that arose as questions during the reading of the paper spare the one I listed under Questions with regards to the cost of producing CoNNs.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper shows a strategy to improvise ICL by including CoNNs in the learning pipeline, which enable the LM to learn symbolic operations in addition to standard autoregressive LM generation. The resulting model is trained by a hand designed gradient accumulation technique and results are compared on symbolic tasks.

STRENGTHS:
The paper shows how symbolic tasks can be included with general autoregressive training and therefore provides a way to train models following a few fixed symbolic tasks in mind. The paper provides a solid training recipe with proper mathematical justification. The results correlate with the claims and justify using the method.

WEAKNESSES:
It is unclear from the paper how different gating mechanisms are being derived in this network and how they are being included in the training framework. The authors say that \beta is not learned in the algorithm and essentially rule calculations are assigned to CoNNs. If that is true, the applicability seems a bit ad-hoc as different rules will then need to be hand written and not derived, and the benefits of the network will only be applicable to scenarios that are symbolically encoded. In other words, this seems to be a scalability challenge in terms of letting LMs learn rules. The experimental evaluation to justify the benefits seem very limited.

The paper also suffers from poor presentability with multiple grammatical errors, spelling mistakes etc. Also more context on training the overall CoNN+DNN system is missing from the paper or appendix.

QUESTIONS:
1. How does the method scale to new symbolic tasks?
2. How do we train a joint model and let the model learn the rules from data?

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-09