Title: Probing the Inductive Bias of Neural Networks through Learning Random Cellular Automata
Venue: NeurIPS 2025
Accepted: no
Keywords: Cellular Automata

Generalization

Inductive Bias

Deep Learning

Coarse-Graining

Discrete Dynamical Systems

Physics for Deep Learning
Abstract: In this paper, we empirically examine whether the inductive bias of deep networks can be linked to structural properties of dynamical systems inspired by physics, such as symmetry, locality, and coarse‑grained observation of outcomes. 
To explore this question, we generate “toy universes” by sampling random cellular‑automaton rules that satisfy these constraints, and train convolutional neural networks (CNNs) to predict their evolution under three experimental factors: temporal coarse‑graining, spatial pooling, and a structured (low‑entropy) initial state. Throughout, we measure each network’s average generalization performance relative to a baseline. 
While classical constraints such as symmetry and locality are necessary, they alone are not sufficient for learnability. However, when we account for the perturbation sensitivity of the target function, we observe a strong negative correlation with learnability. Further, using a structured (low‑entropy) initial state leads networks to favor coarser macroscopic patterns over details.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates whether the inductive bias in neural networks is sufficient to ensure the learnability of physical systems that follow corresponding structural principles. The main finding is that inductive bias alone is not sufficient in the 2D cellular automata examples; moreover, the authors observe a negative correlation between learnability and perturbation sensitivity.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nWhether an appropriate inductive bias in learning models is sufficient to capture the underlying physics in real data is a fundamental question in scientific machine learning. This paper addresses the issue by using cellular automata to design well-controlled experiments for systematic evaluation. The paper is well-written, and the main ideas are clearly presented.\n\n**Weaknesses**\n\n1. *Comparison with Prior Work*. Similar experiments were conducted by Wolfram on 1D cellular automata, where long-term prediction was found to be infeasible due to computational irreducibility. See, for example, his blog post “Can AI Solve Science”. Based on those results, it is intuitive that long-time prediction in 2D systems may also be infeasible. The authors should clarify what new insights or conclusions this work offers beyond previous studies, especially in light of those known limitations. \n\n2. *Interpretation of Results*. Although computational irreducibility is a deep theoretical issue that may resist simple conclusions, the paper would benefit from offering more intuitive interpretations of the results. Even informal explanations could help readers better grasp the implications of the findings.\n\nQUESTIONS:\n1. *Clarity in Figure 3 (Spatial Coarse-Graining)*. In the spatial coarse-graining experiments (Figure 3), the authors state that accuracy increases with spatial scale. However, this trend is not visually obvious. While it appears that more points cluster near 1, the heavy overlap of data points makes it hard to discern the average behavior. The authors may consider a better representation of the trend.\n\n2. *Coarse-Graining Scales*. As the spatial coarse-graining scale increases, the density of while and black pixels (1s and 0s) can change. At a large scale, one color can domiante which makes the prediction task has a higher null-accuracy. This is analogous to the renormalization group flow in the Ising model, where only at the critical point do we obtain a nontrivial fixed point. The authors could discuss for example how the density of a pixel, the spatial frequency of patterns changes with coarse-grain scales, and whether a possible trivialization affects the interpretation of the results.\n       \n3. *Clarification of Section 4.3 Experiments (Interior vs. Exterior Regions)*. In Section 4.3, the exterior region appears as uniformly black pixels, making the prediction task relatively easy in that area. The authors should clarify the motivation behind this experiment. Specifically, whether the higher accuracy in the exterior region is merely a consequence of the low-frequency (or low-complexity) signal present there, or if it reflects a deeper connection to the use of regularized or structured initial conditions.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates how structural constraints inspired by physics (i.e., locality, symmetry, and coarse-grained observations) can affect the generalization ability of neural networks, especially for the convolutional neural network (CNN) structure. They apply the CNN models to the cellular automata (CA) for predicting their future state and study how much spatiotemporal locality or localized initial conditions affect the prediction accuracy.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1) The use of randomly sampled CAs with physical constraints as testbeds is creative and allows a controlled study of neural network priors\n2) The extensive numerical simulations are performed to validate their claims.\n\n\n**Weaknesses**\n1) Only 2D binary CAs are considered, which are abstract and don’t directly reflect real-world systems.\n2) The paper is entirely empirical; no theoretical justification is provided for the observed relationships.\n3) The goal and the actual contribution of this paper are not clear. I guess that this paper studies how the physical constraints affect the neural network performance, where the network structure intrinsically embodies such constraints as inductive bias. However, the provided evidence is not sufficient to support their claims or seems trivial.\n\nQUESTIONS:\n**Questions & Suggestions**\n\n***Major***\n1) I am not convinced that the specific CNN architecture proposed in Section 3.5 is necessary or sufficiently justified for the prediction task. The authors construct a deep CNN with $T$ stacked blocks, one per timestep, but provide no ablation or comparison against simpler alternatives. As I understand it, the key structural priors relevant to this task are spatial locality and translation invariance, both of which are naturally captured by standard CNNs regardless of depth. Therefore, it is unclear whether the depth and timestep-aligned design contribute meaningfully to the learning performance. I believe the authors should provide evidence that their findings are robust across different CNN architectures, or at least clarify whether the proposed architecture is critical to the observed results.\n\n2) In Section 4.1, the authors present a correlation between model accuracy and perturbation sensitivity (PS), showing that accuracy decreases as PS increases, and this effect is amplified with longer time horizons $T$. However, it is unclear what novel insight this provides about the inductive bias of neural networks. The observed trend appears to be a direct and expected consequence of the nature of dynamical systems: greater $T$ naturally allows more time for small perturbations to propagate and amplify, especially in systems exhibiting chaotic behavior. Therefore, while the result is valid, it does not, in itself, reveal anything specific about the network’s inductive bias beyond the trivial fact that sensitive systems are harder to predict over time. I would encourage the authors to clarify what new understanding—if any—about inductive bias is gained from this observation.\n\n3) Following the previous question, a similar concern arises in Sec. 4.2 regarding spatial coarse-graining.  As in the temporal case, this result appears to reflect a trivial reduction in task complexity: coarse-graining removes fine-scale information and effectively simplifies the learning target. It is therefore unclear how this observation supports any specific claim about the inductive bias of neural networks.\n\n4) In Section 4.4, the authors analyze the accuracy of predictions under localized initial conditions and observe that CNNs perform better near the boundary of the evolved structure, with accuracy dropping significantly in the interior. However, this outcome seems to follow directly from the nature of spatial information propagation in cellular automata: boundary regions are less entangled and influenced by fewer interactions, whereas interior regions result from many complex, compounded updates. Thus, the declining accuracy with distance from the edge appears to be a trivial effect of spatial depth in dynamical evolution, rather than a meaningful insight into the neural network’s inductive bias.\n\n***Minor***\n1) Page 5: In the description of localized initialization, is the border width $T$ the same as the prediction timestep $T$? Please clarify.\n2) Figures 2 and 3: The scatter plots are visually cluttered. Consider using transparency to better reveal overlapping points.\n3) Please specify how many steps are used to evolve a configuration before it is used as input. Additionally, how many steps are used to evolve a configuration, and how many steps are required to reach a stationary state?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper links neural network inductive bias to three structural traits of discrete dynamical systems, locality, symmetry, and input state's entropy, by training a fully convolutional network to predict random outer-totalistic 2-D Cellular-Automaton rules defined on a $3\\times3$ Moore neighbourhood with $2^{18}$ possibilities. The authors vary three experimental factors: (i) temporal coarse-graining, (ii) spatial pooling via majority vote, and (iii) the entropy of the initial state (fully random, ``naturalised,'' or localised borders).\nKey findings are: (1) convolutional locality and symmetry alone are insufficient for learnability once temporal depth or spatial pooling compound the rule; (2) a rule’s Perturbation Sensitivity (average output change when a single input bit is flipped) is strongly and negatively correlated with predictive accuracy, making it a useful hardness indicator; and (3) low-entropy inputs can mask rule complexity, so input statistics critically shape apparent learnability.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1. Carefully constructed “toy universe.”  \n   Random outer-totalistic 2-D cellular automata on a 3 × 3 Moore neighbourhood\n   (2^18 possible rules) give a large yet tractable rule space that isolates the\n   effects of locality and symmetry.\n\n2. Motivation of using CNN architecture.  \n   A fully convolutional network with 3 × 3 and 1 × 1 kernels mirrors the CA’s\n   local, translation-invariant update rule and is expressive enough for\n   single-step prediction.\n\n3. Comprehensive discussion on experiment factors.  \n   The study varies three factors: (i) temporal depth (T = 2–7), (ii) spatial\n   pooling, and (iii) initial-state entropy, revealing how each\n   dimension shapes learnability.\n\nWeaknesses:\n\n1. While the empirical study is convincing, the paper would be significantly stronger with at least a sketch of a capacity bound, sample-complexity bound, which are classical concepts related to learnability.\n\n2. Limited real-world connection.  Experiments are confined to 2-D binary cellular automata; higher-dimensional or continuous-valued systems with physical constraints (such as, conservation\n   laws) remain unexplored.\n\n3. Only a CNN is evaluated. Exploring models with different spectral or\n   global-context biases, such as Fourier-feature layers (like the work by Tancik et al., 2020 [1]) could reveal whether the observed limitations are CNN-specific.\n\n4. Missing training dynamics.  \n   The study reports accuracy after the training; the absence of learning\n   curves obscures optimisation behaviour, convergence speed, and potential\n   over- or under-fitting.\n\n[1] Tancik, Matthew, et al. \"Fourier features let networks learn high frequency functions in low dimensional domains.\" Advances in neural information processing systems 33 (2020): 7537-7547.\n\nQUESTIONS:\n1. Can you share training-loss curves or variance across random seeds to show whether failures arise from optimization getting stuck or from fundamental representational limits?\n2. Would you add accuracy vs iteration (or vs number-of-examples) plots for further clarification?\n\nLIMITATIONS:\nSome of the limitations are:\n\n1. Only a single CNN variant is tested. A short discussion or comparison experiment with different type of layer could strengthen the paper.\n\n2. Accuracy is reported only after a fixed 4 096 gradient steps. Presenting learning curves or sample-efficiency plots, and noting that conclusions might change with other optimizers or longer training could clarify the scope of the findings.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a dataset based on cellular automata (CAs) as a toy model to probe the structures convoluational neural networks (CNNs) learn.\nThey sweep over of a large number of rules, vary the prediction horizon, and toggle three knobs—temporal coarse-graining, spatial pooling, and the entropy of initial conditions.  \nPerformance is then correlated with a perturbation-sensitivity (PS) score that measures how fast a single-bit flip spreads through the CA.\nTheir main result is a sharp learnability cliff: once PS crosses a threshold, accuracy collapses beyond a short horizon, even though locality and symmetry still hold; when the initial state is localized, the network recovers the coarse boundary of emerging patterns but misses interior detail, hinting at a simplicity bias toward low-frequency structure.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\nTo my knowledge, this is the first systematic link between a concrete complexity metric (PS) and learnability across such a large family of dynamical rules; the empirical sweep is thorough and well documented.\nExperiments are extremely well documented: hyperparameters, runtimes, and error bars are clearly laid out, and the dataset is open-sourced.\n\n**Weaknesses**:\nAll experiments use a single, vanilla CNN; there is no discussion about whether the PS cliff is architecture-dependent.\nThe study is purely empirical—there is no theoretical account of why PS governs the cliff, nor an attempt to tie the phenomenon to NTK or spectral-bias analyses.  \nBinary CAs are a long way from real-world signals, so the immediate impact feels limited unless a similar story holds for more realistic dynamics.  \nNo baselines beyond majority/logistic regression are reported; a simple k-NN or kernel predictor could help calibrate how hard the task really is.\n\nQUESTIONS:\n1. How robust is the PS–accuracy cliff to the choice of model?  Have the authors tried a transformer, an RNN, or even a shallow neural operator? Given that the current grid search reportedly took only $\\sim 15$ GPU-hours, a quick sweep over a few additional architectures feels tractable and would clarify whether the cliff is CNN-specific.  \n2. Can the authors sketch a principled argument—e.g.\\ via NTK eigenvalues, spectral-bias calculations, or a Lyapunov-style bound—that explains \\emph{why} high-PS rules break generalisation beyond a short horizon? Even a toy calculation for a simplified CA would strengthen the empirical story.  \n3. Besides majority and logistic regression, could the authors add a simple k-nearest-neighbour or kernel predictor? Such baselines would calibrate how much of the performance gap is due to deep learning versus the intrinsic difficulty of high-PS rules.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-03"}