Title: Learning to Add, Multiply, and Execute Algorithmic Instructions Exactly with Neural Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: neural networks

arithmetic

instruction execution
Abstract: Neural networks are known for their ability to approximate smooth functions, yet they fail to generalize perfectly to unseen inputs when trained on discrete operations. Such operations lie at the heart of algorithmic tasks such as arithmetic, which is often used as a test bed for algorithmic execution in neural networks. In this work, we ask: can neural networks learn to execute binary-encoded algorithmic instructions exactly? We use the Neural Tangent Kernel (NTK) framework to study the training dynamics of two-layer fully connected networks in the infinite-width limit and show how a sufficiently large ensemble of such models can be trained to execute exactly, with high probability, four fundamental tasks: binary permutations, binary addition, binary multiplication, and Subtract and Branch if Negative (SBN) instructions. Since SBN is Turing-complete, our framework extends to computable functions. We show how this can be efficiently achieved using only logarithmically many training data. Our approach relies on two techniques: structuring the training data to isolate bit-level rules, and controlling correlations in the NTK regime to align model predictions with the target algorithmic executions.


Peer reviews:
comments: SUMMARY:
This paper gives theoretical results showing that an MLP with 1 hidden layer can learn various bitwise algorithms, using the NTK infinite-width framing. The general idea is to partition the neurons into "blocks", each of which functions effectively as a memory slot, and to assume training examples which illustrate how each block transforms independently. Provided that those training examples are provided, and that there are not inter-block correlations, they show that various algorithms can be learned, given step-by-step examples of each block transformation, and with an acceptable sample complexity (logarithmic in the input space, so polynomial in the number of bits). They support their claims with numerical simulations.

STRENGTHS AND WEAKNESSES:
Strengths:
1. It's important to understand the computational abilities of neural networks, and while much past work has addressed what neural networks can represent, this work further addresses what is actually learnable
2. The work appears rigorous, although I am not an expert in the relevant theoretical areas
3. I found the work to be original and creative. 

Weaknesses:
1. The empirical validation is explained well in the appendix, but given that the NeurIPS crowd has a more empirical leaning, it would be wise to migrate more of this discussion to the main text (for example, I wanted to know how many hidden nodes you use in order to approximate the "infinite width" limit)
2. I'm not so sure about the Turing-completeness claim, see question

QUESTIONS:
I am very confused by the claim that the results extend to arbitrary algorithms simply because SBN is Turing complete. It seems that your setup assumes a fixed amount of memory. How can it be Turing complete in that case?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-04

comments: SUMMARY:
The paper proves that NTK-trained networks can learn algorithms solvable by local template matching.

STRENGTHS AND WEAKNESSES:
Strengths:
- The paper presents a rigorous proof that NTK-trained neural networks can learn template matching with high probability.
- The authors give several examples of algorithms that can be solved via template matching, and hence, prove exact learnability of such algorithms with NTK-trained models.

Weaknesses:
- The paper suffers from some notational ambiguity. For example, in Section 4 the block configuration is represented by $\mathbf{x}$, but the dependence on the block $B_i$ is suppressed.

QUESTIONS:
- In the caption for Fig. 2, what do the authors mean by "although the method is iterative, this example completes in one step"? Also, can the authors clarify if template matching can only provide such iterative solutions?
- Could the authors replot the two curves in each subplot of Fig. 5 such that they coincide? Also I'd like to know what the constant factors are in each case.
- How does Turing completeness allow Assumption 5.1 to hold? 
- Since the experiments were done with finite-sized networks, do the authors have any insight for asymptotics on $n_h$ - the width of the network?

LIMITATIONS:
There are no important limitations that were not highlighted by the authors.
TITLE: Official Review
CLARITY: 2
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates whether neural networks, specifically two-layer fully connected ReLU networks in the infinite-width limit (analyzed via the Neural Tangent Kernel or NTK), can learn to execute algorithmic instructions exactly. The authors demonstrate that by structuring training data into orthogonal bit-level templates and analyzing training dynamics in the NTK regime, such models can provably learn four core algorithmic tasks: binary permutations, addition, multiplication, and Subtract and Branch if Negative (SBN) instructions—the latter being Turing-complete. They prove that exact learning is possible using a logarithmic number of training examples, and provide ensemble complexity bounds to ensure high-probability correctness after rounding.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. Provides rigorous theoretical results showing exact learnability of discrete algorithmic tasks.\n\n2. Uses NTK formalism effectively to derive precise guarantees, going beyond simulation-based evidence.\n\n3. Covers Turing-complete instruction sets (via SBN), showing generality.\n\n4. Results are supported by illustrative examples, ensemble bounds, and practical template encodings.\n\n## Weakness\n\n1. Main result theorem 5.1 requires assumption 5.1. Also the paper assume orthogonal training data and explicit template access, which may limit applicability in more realistic, data-driven settings.\n\n2. Does not address noisy or partial supervision, which is relevant for LLMs and real-world execution.\n\n3. Empirical validation is limited; most claims rely on infinite-width limits and concentration results. Practical significance may be limited unless these theoretical insights can be extended to finite-width or real-world training settings.\n\nQUESTIONS:\n1. Can the authors clarify how to obtain the orthogonal training set in practical scenarios? This assumption is central to the theoretical results but may not hold for real-world training datasets. Could approximate orthogonality suffice?\n\n\n2. Your setup leads to exact learning under NTK dynamics with small datasets. How do your findings relate to grokking phenomena, where models trained with standard datasets undergo sudden generalization?\n\n\n3. The analysis assumes infinite width and NTK regime. Have you tested the fidelity of your predictions with finite-width networks trained with SGD?\n\n\n4. Assumption 5.1 is central to the correctness of the NTK predictor. How sensitive are your results to this assumption? Can the authors provide more intuition (beyond the bound of 2 unwanted correlations) or empirical plots showing how performance degrades as it is violated?\n\nLIMITATIONS:\nYes. The authors explicitly discuss their limitations in Section 7, including the assumptions of orthogonal template training and direct instruction access. They also propose several promising directions to relax these constraints.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the learnability of a particular class of boolean functions (expressed as a disjunction of simpler blocks) by two-layer ReLU networks with large width. Appealing to NTK theory, they suggest that the NTK limit should converge to approximate functions of this form satisfying a particular margin-like assumption (Assumption 5.1).\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The paper addresses the problem of learning primitive discrete operations with feedforward networks that, when composed, could be computationally powerful. This is tangentially related to methods like chain of thought, and theoretical understanding of the learning difficulty could be interesting and important.\n2. Section 3 on NTK (an existing result) and Section 4 on templates seem rigorous and correct\n\n## Weaknesses\n\n1. The problem considered here could be much better motivated. The authors confusingly allude to Turing completeness but are not particularly careful or clear. The point is that iteratively composing a sequence of boolean functions of the form considered here is quite expressive. You consider the learning problem for one step in this chain - not the end-to-end learning problem. It’s important to clarify this both so that readers do not get the wrong impression and to make it clear what the motivation of the paper is in the first place.\n2. Related to the motivation, the authors should cite and engage with significant existing work on on the expressivity and learnability of iteratively applying simple neural functions to compute more complex ones (e.g., attaining Turing completeness via chain of thought):\n    1. https://arxiv.org/abs/2309.06979\n    2. https://arxiv.org/abs/2107.13163\n3. I am not able to verify the correctness of Sections 5 and 6, which constitute the core new results. In particular, I do not even understand Assumption 5.1, which is central the message here. Why is w0 non-positive if it’s defined as the proportion of training samples that do not match a block? Shouldn’t a proportion always be positive? This part of the paper should be clarified so that it is easier to understand and verify the results.\n\nQUESTIONS:\nRather than saying logarithmic sample complexity (implicitly, in the size of the number of possible inputs, which is confusing), say sample complexity polynomial in the input size (i.e., in the number of bits needed to code an input)\n\nWhat is the convergence error of the large-width network to the NTK limit as a function of the hidden dimension?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}