Title: Continuous Thought Machines
Venue: NeurIPS 2025
Accepted: yes
Keywords: Biologically Inspired Neural Network

Neural Dynamics

Neural Synchronization

Reasoning Model
Abstract: Biological brains demonstrate complex neural activity, where neural dynamics are critical to how brains process information. Most artificial neural networks ignore the complexity of individual neurons. We challenge that paradigm. By incorporating neuron-level processing and synchronization, we reintroduce neural timing as a foundational element. We present the Continuous Thought Machine (CTM), a model designed to leverage neural dynamics as its core representation. The CTM has two innovations: (1) neuron-level temporal processing}, where each neuron uses unique weight parameters to process incoming histories; and (2) neural synchronization as a latent representation. The CTM aims to strike a balance between neuron abstractions and biological realism. It operates at a level of abstraction that effectively captures essential temporal dynamics while remaining computationally tractable. We demonstrate the CTM's performance and versatility across a range of tasks, including solving 2D mazes, ImageNet-1K classification, parity computation, and more. Beyond displaying rich internal representations and offering a natural avenue for interpretation owing to its internal process, the CTM is able to perform tasks that require complex sequential reasoning. The CTM can also leverage adaptive compute, where it can stop earlier for simpler tasks, or keep computing when faced with more challenging instances. The goal of this work is to share the CTM and its associated innovations, rather than pushing for new state-of-the-art results. To that end, we believe the CTM represents a significant step toward developing more biologically plausible and powerful artificial intelligence systems. We provide an accompanying [interactive online demonstration](https://pub.sakana.ai/ctm/) and an [extended technical report](https://pub.sakana.ai/ctm/paper).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new model for deep learning inspired by two important (and underexplored) ideas in neuroscience: (i) neuron-specific dynamics, and (ii) across-neuron synchronization. The resulting model is essentially a recurrent neural network, where each \"neuron\" is parameterized by a complex neural network, namely one feedforward network to compute its current potential as a function of its neighbors' activity, and a second recurrent network to compute its output as a function of its historical potential (this captures idea (i)). Property (ii) arises in that the model's actual output is (a projection of) the similarity matrix of the trajectories of its neurons -- in other words, their synchronization. The model is designed to adapt its computational time, particularly through a loss function that incentivizes it to simply make its prediction its most confident output, regardless of timestep. The model is evaluated through a diverse variety of tasks with both sequential and non-sequential character, including image classification and solving mazes.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nS1. The writing and presentation throughout are excellent. The model is fairly complex so this is very important. The supplementary website with animations is a nice bonus.\nS2. The implementation of synchronization is original, creative, and thought-compelling.\nS3. The experiments are diverse, described thoroughly, and seem to make a compelling case for the model's effectiveness.\n\nWeaknesses\nW1. It's difficult to tell the how important the different components of this model are to its effectiveness. It's also, frankly, a bit unintuitive to reason about. Is synchronization helpful to the model, or is it implicitly regularizing? More experimental probing of this would be helpful, especially ablation studies.\n\nQUESTIONS:\nQ1. Of course, it would be very interesting to see a language experiment in a model like this.\nQ2. Why is it necessary to include both the loss on the best tick and the loss on the most confident tick in the loss function? Intuitively, it seems like only the second term is important.\nQ3. The \"Emergent Phenomena\" section of the appendix is interesting and it would be nice to see a few of these observations empirically fleshed out in the main body.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a deep neural network architecture which uses timing and synchrony between neuronal units to compute. The architecture is end-to-end differentiable, and the authors demonstrate that it can be optimized to perform interesting and computationally challenging tasks, such as 2D mazes and ImageNet 1K classification. The authors also provide interactive examples of their network performing these tasks.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths:\n\nThe paper is well-written, and the technical contributions are clearly described. Also, the authors go above and beyond by providing interactive examples of their network solving a 2D maze task, which this reviewer found charming and fun to play with. \n\n# Weakness:\n\nUnfortunately, I have major concerns about the originality of this work. Although the authors repeatedly claim that they are the first to use neural synchronization in a deep learning context, this is simply untrue (details below in the \"Questions\" section).\n\nQUESTIONS:\n## Question on Novelty and Originality\n\nMy main question concerns the novelty and originality of the work. The authors claim multiple times that they are the first to incorporate neural synchronization within a deep learning context. For example:\n\n- **Lines 46–47**:  \n  *\"This use of neural synchronization yields a fundamentally new type of biologically-motivated latent representation...\"*\n\n- **Line 65**:  \n  *\"The CTM introduces neural timing and synchronization as core computational principles.\"*\n\nHowever, this appears to overlook previous work that has explicitly explored similar ideas, most notably:\n\n- **[Reichert & Serre (2014)](https://arxiv.org/abs/1312.6115)**  \n  *Neuronal synchrony in complex-valued deep networks*  \n  This work directly addresses the use of neural synchrony in deep networks.\n\n- **[Lee et al. (2022)](https://ieeexplore-ieee-org.libproxy.mit.edu/stamp/stamp.jsp?tp=&arnumber=9849162)**  \n  *Complex-valued neural networks: A comprehensive survey*  \n  This survey covers numerous architectures that utilize neural timing and synchronization as core computational principles.\n\nGiven the existence of this prior work, was the omission of this literature an honest oversight, or was there a specific reason for its exclusion?  If the latter, could the authors clarify how their approach differs from or advances beyond these previous efforts?\n\nThe authors also claim, in the abstract, that \"Most artificial neural networks simplify neurons by abstracting away dynamics\". I am not sure what this means--can the authors clarify? Do they mean abstracting away the complex biophysics dynamics of single-neuron Hodgkin-Huxley models? Because taken at face value, their sentence is certainly false: many hundreds of papers are written every year which use artificial neural networks with dynamical neurons. This has been going on for many years in neuroscience, for example see [Computation Through Neural Population Dynamics – Yas S, Golub MD, Sussillo D, Shenoy KV. *Annual Review of Neuroscience*, 2020.](https://doi.org/10.1146/annurev-neuro-092619-094115)\n review from five years ago. \n\n\n**To be clear, I believe this could be a strong and valuable paper, provided the inaccurate novelty claims are removed and the related work section is substantially expanded to more accurately contextualize the contribution within existing literature. I would consider increasing my score if the authors were to address my concerns here.**\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 1, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Continuous Thought Machine (CTM), a novel artificial neural network model designed to take into account the neural timing and interactions for decision making. The proposed model uses neuron-level processing and synchronization as foundational elements. The proposed method has been tested on several datasets (2D Maze navigation, Imagenet 1K classification, cumulative parity tasks, cifar 10 etc. ) and shows promising results.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\nIntroducing neural timings and synchronization as the core principle is a novel idea and moves closer to biologically inspired learning algorithms. CTM has been tested on wide datasets and shows promising results. The paper provides extensive details on architecture, hyperparameters, datasets, and optimization settings in the appendices. \n\nWeakness\n\nFrom Figure 6 (Page 8), CTM's training seems sensitive to random seed, initialization. For internal tricks varying from 75 to 100, the variance is comparatively higher.\n\nThe longer training time seems to go against biological plausibility. Would like to know the response of the authors.\n\nQUESTIONS:\nDo the neuron level models used in CTM increase the parameters compared to the conventional ANN’s? \n\nFrom Figure 6 (Page 8), CTM's training seems sensitive to random seed, initialization. For internal tricks varying from 75 to 100, the variance is comparatively higher.\n\nHave the authors experimented on trying biological neuronal models such as Hindmarsh and rose neuronal model, Hodgkin and Huxley neuronal model etc.? Any specific reason on why biological neuronal models are not used? \n\nThe longer training time seems to go against biological plausibility. Would like to know the response of the authors.\n\nLIMITATIONS:\nI would like the authors to respond to the questions raised.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Continuous Thought Machines (CTMs), a novel neural network architecture that emphasizes temporal processing at the single-neuron level and population-level synchronization as core computational principles. CTMs replace standard pointwise nonlinearities with neuron-level models that process a history of pre-activations, aiming to better capture the complexity of biological neural dynamics. Furthermore, CTMs use neural synchronization (defined through correlations in neuron activity across internal “ticks”) as a central representational mechanism for attention and downstream readout. The authors propose that this architecture offers a step toward bridging the gap between biological and artificial intelligence, and present a diverse suite of experiments across different domains, focusing on the qualitative behavior and flexibility of CTMs rather than competitive benchmark performance.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n- Conceptual originality: The paper proposes a novel architectural shift, where both temporal integration and inter-neuronal synchronization are fundamental to how representations are formed. This is a creative departure from conventional feedforward or RNN-based models.\n- Biological inspiration: The authors ground their design choices in neuroscience, particularly in emphasizing the importance of temporal structure and population dynamics, which are often abstracted away in standard deep learning.\n- Breadth of experimentation: The authors test CTMs across a range of tasks, demonstrating versatility, even if not aiming for SOTA performance.\n- Modularity: The design of neuron-level models, decaying influence kernels, and synchronization mechanisms is well-structured, suggesting future extensibility.\n\n### **Weaknesses**\n\n- Clarity issues:\n    - Some key figures (e.g., Figures 1 and 2) are introduced abruptly and not clearly contextualized in the main text. What does “teleporting” mean in Figure 1?\n    - Figure 3 is convoluted and could be made clearer.\n    - Terms like “neural synchronization as representation\" are used repeatedly but never precisely defined, why use the term “synchronization” instead of “temporal correlation”?\n- Quantitative rigor: There is a lack of controlled comparisons in the main text with baseline models on shared tasks. It is unclear whether CTMs offer any advantage under equivalent parameter or FLOP budgets.\n- Biological grounding: While motivated by neuroscience, some modeling choices (e.g., neurons retaining a full history of pre-activations) lack biological justification. It is unclear whether such memory mechanisms exist at the single-neuron level in biological systems.\n- Limited ablation: No ablation experiments are presented to isolate the importance of neuron-level models versus the synchronization mechanism.\n\nQUESTIONS:\n1. Synchronization as representation: You emphasize synchronization as a key latent representation for attention and output. Could you clarify what precisely is being represented here? Is “synchronization” equivalent to temporal correlation? If not, how does it differ?\n2. Decaying influence kernel: In the computation of synchronization, earlier ticks appear to have greater influence due to the exponential decay term $\\exp(-r_{ij}(t - \\tau))$. This seems counter-intuitive. Can you clarify or justify this design choice?\n3. Biological plausibility: Do real neurons retain a history of their pre-activations in a way that could support a mechanism like your neuron-level models? If not, how do you justify this as a biologically inspired component?\n4. Subsampling neuron pairs: How sensitive is the model to the random sampling of neuron pairs for computing $\\mathbf{S}^t$? How many samples are sufficient, and do you resample at every tick?\n5. Ablations: It would be very helpful to see ablation studies isolating the contributions of (a) the neuron-level models and (b) the synchronization mechanism. Could you explore these separately?\n6. \"Looking Around\" Rationale: While an interesting emergent property, what are the specific desired computational benefits or motivations for the CTM to \"look around\" images during classification, especially for static inputs?\n\nScore could increase if the authors:\n\n- Clarify terminology and figures,\n- Provide ablations to better understand the architecture,\n- Offer more rigorous comparisons or parameter-controlled baselines, highlighting the trade-offs of CTM vs SOTA models.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}