Continuous Thought Machines
Authors: Luke Nicholas Darlow
Ciaran Regan
Sebastian Risi
Jeffrey Seely
Llion Jones
Venue: NeurIPS 2025 spotlight ()
Keywords: Biologically Inspired Neural Network
Neural Dynamics
Neural Synchronization
Reasoning Model
Primary area: deep_learning
TL;DR: Neurons in brains use timing and synchronization in the way that they compute, so we built a model that does the same.

Abstract: Biological brains demonstrate complex neural activity, where neural dynamics are critical to how brains process information. Most artificial neural networks ignore the complexity of individual neurons. We challenge that paradigm. By incorporating neuron-level processing and synchronization, we reintroduce neural timing as a foundational element. We present the Continuous Thought Machine (CTM), a model designed to leverage neural dynamics as its core representation. The CTM has two innovations: (1) neuron-level temporal processing}, where each neuron uses unique weight parameters to process incoming histories; and (2) neural synchronization as a latent representation. The CTM aims to strike a balance between neuron abstractions and biological realism. It operates at a level of abstraction that effectively captures essential temporal dynamics while remaining computationally tractable. We demonstrate the CTM's performance and versatility across a range of tasks, including solving 2D mazes, ImageNet-1K classification, parity computation, and more. Beyond displaying rich internal representations and offering a natural avenue for interpretation owing to its internal process, the CTM is able to perform tasks that require complex sequential reasoning. The CTM can also leverage adaptive compute, where it can stop earlier for simpler tasks, or keep computing when faced with more challenging instances. The goal of this work is to share the CTM and its associated innovations, rather than pushing for new state-of-the-art results. To that end, we believe the CTM represents a significant step toward developing more biologically plausible and powerful artificial intelligence systems. We provide an accompanying [interactive online demonstration](https://pub.sakana.ai/ctm/) and an [extended technical report](https://pub.sakana.ai/ctm/paper).

Continuous Thought Machines
Luke Darlow!
Ciaran Regan"
Sebastian Risi!,3
Jeffrey Seely!
Llion Jones'
Sakana AI, Tokyo, Japan
University of Tsukuba, Japan
TT University of Copenhagen, Denmark
(luke, ciaran, sebastianrisi, jeffrey, 1lion) @sakana. ai
Abstract
Biological brains demonstrate complex neural activity, where neural dynamics critical how brains process information. Most artificial neural networks ignore the complexity individual neurons We challenge that paradigm. By incorporating neuron-level processing and synchronization, timing as a foundational element. We present the Continuous Thought Machine (CTM), model designed to leverage neural dynamics its core representation. The CTM has two innovations: (1) neuron-level temporal processing, where each neuron uses unique weight parameters process incoming histories; and (2) neural synchronization as latent representation. The CTM aims to strike balance between neuron abstractions and biological realism. It operates at a level of abstraction that effectively captures essential temporal dynamics while remaining computationally tractable. We demonstrate the CTM's s performance and versatility across range of tasks, including solving 2D mazes, ImageNet- IK classification, parity computation, and more. Beyond displaying rich internal representations and offering natural avenue for interpretation owing to its internal process, the CTM is able to perform tasks that require complex sequential reasoning. The CTM can also leverage adaptive compute, where it can stop earlier for simpler tasks, or keep computing when faced with more challenging instances. The goal this work is to share the CTM and its associated innovations, rather than pushing for new state-of-the-arr results. To that end, we believe the CTM represents significant step toward developing more biologically plausible and powerful artificial intelligence systems. We provide an accompanying interactive online demonstration and an extended technical report
reintroduce neural
Perfect solution (trained up 100 steps).
(Emergent) further than the training route.
(c) Generalizing to larger maze.
Figure 1: Solving 100 steps down 39 39 mazes. (a. Observing using attention (no positional encoding, weights overlaid), imagining route (arrows) from red to green pixels, (b) attending beyond 100 steps, and (c) generalizing to 99 99 via sequential re-applications of the same model.
39th Conference on Ncural Information Processing Systems (NcurIPS 2025).

---

(a) Each candom-colored subplot is (b) The CTM looks around to build up its prediction, effectively tracing an single neuron's 's activity.
intuitive path by synchronizing its neurons to attend dynamically.
Figure 2: ImageNet- IK demonstration. (a) Complex neural dynamics whose synchronization are the representation with which the CTM observes and predict. (b) CTM's attention process, showing all 16 attention heads (left) and average thereof (middle). Arrows trace the average weighting over internal ticks, exemplifying complex path that emerges without any training signal. We discuss more interesting emergent properties of the CTM in Appendix [ Video demonstrations are here.
Introduction
Biological brains exhibit complex time-dependent neural dynamics, but artificial neural networks (NNs) intentionally abstract away the precise timing and interplay of neuron interactions to facilitate large-scale deep learning 2 3. While enabling significant advancements over the years, these simplifications deviate from fundamental biological neural computation principles. Emulating the temporal aspects of neural dynamics present in brains remains challenging. Consequently, modern NNs prioritize simplicity and computational efficiency over strict emulation. This abstraction, though lask-performant, contributes to a gap between flexible human cognition and current AI capabilities, suggesting missing fundamental components, potentially related to temporal processing 5]
Despite its outstanding performance, modern Al lacks the flexibility, efficiency, fluidity, generalization capabilities, and common sense of human intelligence, which operates an open world where learning and adaptation are tied to the arrow of time 7 6 8) We argue that incorporating time as part of neural computation is crucial for advancing AI 9 10 We introduce the Continuous Thought Machine (CTM), model explicitly incorporating neural dynamics over time. Our contributions are:
1.
The CTM architecture using an internal dimension for modeling the temporal evolution of neural activity, neuron-level models (NLMs) as a more biologically plausible mid-level abstraction of neurons that unfold neural dynamics, and the use of neural synchronization directly as the representation (implemented via temporal correlations between neuron-level activity; Section 3.4) for observation and prediction, making neural dynamics the core operating principle.
2. An exposition of the capabilities unlocked by the CTM, including strong performance sequential reasoning tasks (Figure [ native adaptive compute time, natural and interpretable behaviors such as 'looking around' images before predicting (Figure 2 and learning algorithmic solutions, opening up opportunities to the AI community for new research.
The CTM learns use neural synchronization its latent representation, distinguishing it from existing work that explores synchrony as emergent properties for post-hoe use 11 12]. This representation is distinct from the common static 'snapshot' representations used in most modern NNs as it directly encodes the temporal interplay of neural dynamics.
Recurrence and Reasoning. Recurrence is a strong contender for extending model complexity beyond current scaling limitations [3
14 15). We posit that recurrence, while essential, merely one piece of the puzzle. The temporal dynamics unlocked by recurrence are equally crucial. We demonstrate in this paper that neural dynamics can be leveraged to build new kind of neural network with surprising capabilities. We show how the CTM navigates complex 2D mazes by forming internal maps without positional encodings (Section 4.. learns to 'look around' (without any signal to do so) when classifying images and exhibits native adaptive computation time as a side-effect

---

and utilizes its dynamic representations for tasks requiring memory and sequential reasoming (Section(6). These capabilities emerge from the same core architecture applied different tasks, showcasing its versatility and trainability. We believe that the CTM represents a step towards bridging the gap between powerful modern Al and biological plausibility.
(Section 5),
The remainder of this paper details related work (Section 2 describes the CTM (Section 3) evaluates core capabilities on 2D mazes, ImageNet- classification, and parity computation (Sections to summarizes further experiments and applications (Section 7 and discusses findings (Section 8
2
Related Work
The CTM uses neural timing and synchronization as core computational principles. This positions it relative to, yet distinct from, several lines of research.
Adaptive Computation. Many approaches achieve adaptive computation via explicit mechanisms. Early-exit networks [16] use intermediate classifiers for early termination. PonderNet [17] and Adaptive Computation Time (ACT) [88] introduce learnable halting modules governing recurrent steps. More recent methods like Ada Aape [99] dynamically extend input sequences, while Sparse Universal Transformers (SUT) (20] combine recurrent weight sharing with dynamic halting and Mixture-of offExxerts. In contrast, the CTM's adaptive processing (varying internal ticks per input based certainty and loss dynamics; Section 3.5 emerges naturally from its core architecture, driven by the unfolding of its internal neural dynamics without dedicated halting components.
Iterative and Recurrent Reasoning. The CTM's internal ticks facilitate iterative refinement, akin to models promoting internal computational steps. For instance, Quiet-STaR (2 uses hidden rationale generation in language models, and Recurrent Independent Mechanisms (RIMs) 22 employ modular, asynchronous sub-networks for multi-step reasoning. While Recurrent Models of Visual Attention (RAM) (23 also leveraged recurrence for sequential processing of visual glimpses, the CTM's novelty lies in generating internal neural dynamics from neuron-level histories across a decoupled time dimension and then utilizing the explicit temporal patterns of neural synchronization as its primary representation. This contrasts with RAM's s focus on perceptual decision-makinn from external glimpses or models relying solely final recurrent state.
Biologically Inspired Neural Dynamics. There is growing interest in more biologically plausible neural computation [24]. Examples include Liquid ConettCntt Networks (LTCNs) (2 with neurons governed by time-varying differential equations, and various Spiking Neural Network (SNN) paradigms that inherently use discrete, timed events, with recent work also exploring synchronization mechanisms [26 27]. The CTM draws inspiration from temporal coding and neural synchrony, but uses: (1) neuron-level models (NLMs) to process a history of continuous-valued pre-activations to generate complex dynamics, and (2) neural synchronization as the primary latent representation for attention and output. While inspired by principles like timke-timing and synchrony, CTM abstracts these-focusing on local temporal integration and population-leved synchronization- into a tractable, differentiable framework suitable for gradient- based deep learning, rather than replicating detailed biophysics. This situates the CTM alongside. yet distinct from, extensive work on models such as Liquid State Machines (28). and diverse SNNs that exploit precise spike timing for computation
32 33 These latter models often emphasize event-driven dynamics, explore non-differentiable computation, or focus on online learning. The CTM offers a complementary direction, retaining inspiration from biological timing while ensuring compatibility with established deep learning training paradigms.
or employ specialized learning rules 29 30
31
Synchronization. Reichert & Serre [1] proposed a model where synchronization emerges from in- teractions among complex-valued neurons, serving as a gating mechanism that modulates information flow and enables post-hoc grouping of neurons for tasks like object segmentation. Unlike CTM, how- ever, their model does not use synchrony as a learned latent representation during computation. Other approaches in complex-valued neural networks [2] employ synchronization from a control-theoretic perspective, aiming to stabilize or coordinate networks via externally enforced synchrony. In contrast, CTM integrates synchronization intrinsically, optimizing neural phase relationships during training to encode task-relevant representations. This positions CTM as a computationally grounded model of synchrony, fundamentally distinct from prior works that treat synchrony control objective.
3

---

Method
SYNAPSE
NEURON-LEVEL MODELS
MODEL
time/thought
LOOT
m
Z
a
θ
HISTORY (pre-activations)
next internal tick
10
SYNCHRONISATION
DATA
6
S'
OUT
ATTN
Z
W... W
LATENT REP
8
HISTORY (post-activttions
Figure 3: CTM architecture overview. Key components include:
Synapse model generating
pre-activations from prior post-activations z and attention output of History of pre-activations
History of post-activations Z Neural synchronization matrix St computed from Z O Selected neuron pairs from S form latent representations used for outputs y' and attention queries q' U Attention output O' concatenated with z for the next internal tick. Owing to the inherent difficulty in visualizing a dynamic, time-based architecture, we include the supplementary video 'arch.mp4' (hosted here too) that visualizes functional data flow.
A'. 3 Neuron-level models (NLMs) processing A to produce ! post-activations Zd
The Continuous Thought Machine (CTM) is neural network architecture that explicitly incorporates neural dynamics as a core component. Figure 3 ! + ! and pseudocode in Listing illustrate the CTM's flow. The CTM differs from other recurrent architectures
[8] 37 in ways: (1) it applies neuron-level models (NLMs), each with private weights, to histories of pre-activations to produce complex neuron-level activity (Section 5p; and (2) uses neural synchronization directly as the latent representation for modulating data and producing outputs (Section 3.4).
[4 35
36
3.1
Continuous Thought: The Internal Sequence Dimension
The CTM uses internal dimension E {1..... 7), decoupled from data dimensions. This timeline of internal ticks 34 35 36 37] enables iterative refinement of representations, even for static data. Unlike conventional sequential models that process data-inherent sequences, the CTM along self-generated timeline of "thought steps' that unfolds neural dynamics for downstream use.
3.2 Recurrent Weights: Synapses
A ! synapse model, fe... interconnects neurons in shared D-dimensional latent space, z' R. We found a U-NET-esque 38 MIP (details in Appendix C.I performs best, suggesting benefit from deeper and more flexible synaptic computation. It produces pre-activations, a":
fo... (concat(z', oo)) E R.D.
where o' is attention output (Section 3.4
The M most recent pre-activations form 2 history A at] E DxM
A Initial pre-activation history and z'= are learnable parameters. We found that setting M ~ 10 100 effective during initial exploration.
[
M+1
1-M+2
(2)

---

3.3
Dwwately-Parameteriiee Neuron-Level Models (NLMs)
Each neuron d E fl...., D) has @ privately parameterized NLM, 9e, (depth MLP of width dhidden), processing its -ddimensional pre-activation history A to produce ! post-activations:
gos (A²).
The full set of post-activations z ① concatenated with attention output, o', and fed into the synapse model fo.g for the next internal tick, t + See Listing 2 for pseudo-code.
3.4 Neural Synchronization: Modulating Data and Outputs
Synchronization is inspired by biological brains 39 The CTM modulates data via the synchroniza- tion of neural activity" We first collect post-activations into (non-fixed length) history:
Ze
z
z'
R
Dxt
We define neural synchronization is defined as the inner product of the histories of each neuron:
S' Z (Z²)T E RDD
3.4.1 Neuron Pairing: A Sub-sampling Approach
Since S' scales with O(D²) it can grow very large. We sample (i.j)) neurons at the start of training by randomly selecting Dout and Daction pairs for two synchronization representations, S E RR Pm action E Dusis, These are projected by Wau and Win for outputs y' and attention queries q²:
and St
y Wout outr
(6)
q Win S
Sationn
(7)
We use standard cross attention 40 for o':
0' = AAtenntion(Q = K' = FeatureExtrractor((attt))
(8)
where a FeatureExtraatto (e.g., ResNet 41) provides keys/values. of E BR is then concatenated
with z #1 This process, including learnable temporal scaling, is shown in Listing
Scaling Temporal Dependency. To modulate the influence of past activity on S we introduce leamablee exponential decay factors ray for each neuron pair éj. The rescaling vector over is:
R (exp(-ray(t 1)) exp(-rap 2))
exp (0)) R
The rescaled synchronization is (see Appendix for efficient recursive computation):
St (Z)) diag(R, (Z))
(10)
E R,,]
Higher rij bias towards recent ticks (rij means no decay). Learnable decay rates riy allow the CTM modulate synchronization across multiple time scale Details on neuron-pair sub-sampling strategies, including recovering snapshot dependencies, are in Appendix C.2
3.5 Loss Function: Optimizing Across Internal Ticks
The CTM produces outputs y" C (c.g., class probabilities) each internal tick t. We compute a loss L CrossEntropyy M.true) and certainty C* (1 normalized entropy) per tick. For each forward pass we select ticks:
1. the point of minimum loss: t = argmin L), to optimize the 'best' prediction; and
'We found that 'suapshot" representations downstream task thereby limits the types of dynamics it can produce, whereas synchronization decouples it. Forr full disclosure, we found that the CTM barely leveraged this for ImageNet (Section 5) but more so for 2D mazes (Section suggesting task-dependent temporal sensitivities
too constraining: projecting from z' strongly ties it to the
5

---

2. the point of maximum certainty: to = argmax(C), to ensure certainty aligns with correctness.
The final loss for optimizing Osyn and O d==1... D is:
L C4 + C
(11)
2
Since t and f2 are dynamically defined per data point, the CTM can attribute variable compute (internal ticks) to different data points as needed without explicit restrictions on which tick should be used in the loss function. This effectively implements native adaptive computation [8 as
opposed a post-hoc addition. We give pseudo-code in Listing
Experimental Evaluation
The following sections present a focused evaluation of the CTM on tasks that highlight its core principles: neuron-level temporal processing and neural synchronization as a direct latent representation. We aim to demonstrate how neural dynamics enables the CTM to implement complex reasoming or adaptive processing, while yielding interpretable strategies. We prioritize depth in three key experiments: 2D maze navigation, ImageNet- KK classification, and parity computation. We also summarize and highlight additional experiments demonstrating the CTM's broader capabilities.
2D Mazes: Complex Sequential Reasoning and Internal World Models
In this section we analyze the CTM's capacity for sequential reasoning, planning, and spatial understanding using a challenging phrasing of the 2D maze navigation task. Solving mazes can be easy with the right inductive bias. For example, matching the output dimensions to the input space, model can perform binary classification at each location. Such a setup amenable to machines by design, as they can learn iterative algorithmic solutions [37 42). but this is not how humans solve mazes.
Setup. The setup of our maze task deviates from the norm, specifically to necessitate the forma- tion of an internal world model 43 by (1) requiring direct sequence-of-action output and (2) disallowing positional embeddings in the visual input. This requires a model to build its own spatial representation via observation (see Appendix [D6 for further discussion). We compare the CTM against LSTM and feed-forward (FF) baselines. For the results that follow, trained a CTM, LSTMs (1, and layers), and a FF baseline to predict up to 100 steps down the path of 39 x 39 mazes, where predictions took the form of sequence of classes for left, right, up, down, and wait, using 'wait' to pad instances shorter than 100 steps. For the CTMs and LSTM baselines, we used 75 internal ticks, but LSTM stability issues meant that using 50 internal ticks yielded superior performance, so we report these too. In each case we used a automatic curriculum approach when training (see details in Appendix D.3) Appendices D.2 and D.4 detail hyperparameters for the CTM and baselines.
4.1 Results
The CTM significantly outperforms the baselines solving these mazes, demonstrating superior trainability and generalization to longer paths (Figure 4.. The FF model and LSTMs struggled to learn effectively or overfit (see Appendix [D.5), whereas the CTM achieved high accuracy. This suggests that the CTM's architecture, particularly its use of neural dynamics and synchronization, is well-suited for tasks requiring robust internal state maintenance and planning.
4.2 Demonstrations and Generalization
Qualitative analysis shows the CTM methodically tracing paths (Figures aalan and 1b supplementary video 'mazes. exhibiting emergent behavior such as continuing to explore paths beyond its training horizon. This suggests the CTM learns general procedure rather than merely memorizing. Furthermore, the CTM, trained on 39 39 mazes, generalizes effectively to longer paths and larger 99 x 99 mazes (Figure(4c) by re-applying its learned policy, shown in Figure (see supplementary videos mazz--arrgel.me Impp to 'maze-large4. mpp4 for examples). Crucially, this CTM not using any positional embedding, meaning that in order for it to follow path through the maze it must craft the

---

Smale relve
0.8
18t943
CTM. 75 Boc
Puth
Pouto
(a) Accuracies versus path length. (b) Accuracies when generalizing
(c) Mean re-application count.
Figure 4: CTM versus baselines on 2D mazes. The CTM demonstrates superior trainability compared baselines, yielding higher accuracy for longer paths. Using iterative re-applications, we show in (b)
that the CTM can generalise to longer paths and bigger mazes. See Appendix D.5 for loss curves.
cross-attention query by 'imagining' the future state of the maze: process known 'episodic future thinking' 44 in humans. Appendix discuss some of the emergent properties we observed.
5 ImageNet-1K Classification: Adaptive Processing and Emergent Dynamics
We evaluate the CTM on ImageNet-IK to understand its internal processing dynamics when trained to solve standard classification task. We are not yet aiming for state-of-the-ard accuracy (with 50 internal ticks and ResNet-152 backbone: 72.47% top-1, 89.89% top-5 on uncropped data). Since the CTM uses neural computation principles it would require thorough hyperparameter search to find the optimal settings, and that is outside the scope of this work. Instead, we focus on how the CTM leverages neural dynamics (setup details in Appendix E.T as a new mechanism for reasoning.
5.1
Adaptive Computation and Calibration
1.0
After all ticks
0.8
0.8
40
% of
0.6
0.6
30
of
Internal
0.4
0.4
Positive
0.2
10
Negative
0.0
0.0
50
0.2
0.4
0.6
0.8
Internal tick
Mean predicted probabiliies
(a) Top-5 accuracies using a certainty threshold of 0.8.
(b) Calibration plot per internal tick.
Figure ImageNet-IK results: (a) Native adaptive compute potential based on a 0.8 certainty threshold, showing performance expected at each internal tick. (b) Excellent model calibration when
averaging probabilities up to each tick shown. See Appendix E.3 for further analysis.
The CTM exhibits adaptive computation: it can halt internal ticks based on prediction certainty. For instance, setting certainty threshold of 0.8 (Figure 5a) means that user could halt compute for the majority of instances after fewer than 10 of 50 internal ticks. This is a consequence of internal recurrence couple with our novel loss function. The CTM also demonstrates excellent calibration (Figure 5b) as an emergent property of its iterative refinement process (Appendix F.3)
5.2 Reasoning sequentially about static images
The CTM exhibits diverse temporal dynamics (Figure 2ap. the synchronization which is the representation with which it observes data and forms predictions. We show in Figure 2b how the CTM learns to 'look around' an image in order to gather information and make a prediction. does this entirely without prompting any guide, implementing computationally beneficial adaptive
7

---

compute in an intuitive fashion. This internal process can even manifest emergent phenomena like low-frequency traveling waves 45 across UMAP-projected neuron activations (see supplementary video wmnp.mp44"). Unpacking every interesting facet of these attention map progressions is simply infeasible in a static form; we encourage viewing supplementary video menagenet.mpt for demon- strations of the CTM 'gazing' in manner not quite entirely unlike how humans might look around images. Appendix [h.4has further demos and UMAP visualizations. These observations underscore that the CTM solves classification by leveraging internal, dynamic reasoning process, departure from typical feed-forward approaches.
Parity: Learning Sequential Algorithms and Interpretable Strategies
To test the CTM's ability learn algorithmic procedures and develop interpretable strategies, use a cumulative parity task: given a 64-length binary sequence, predict the parity at each position (Figure (a). Unlike prior work focusing final parity [18), our setup requires the model to output sequences each internal tick, enabling us to examine how the full output evolves across ticks and throughout training. Setup details in Appendix F.
6.1 Results and Learned Strategies
100
Iters
10 Baess
25 Dars.
Iters
AccurSo
Rars
70
CTM
LSTM
50
50000
100000 Thaining theations
150000
200000
Input
Target
(a) Example:
ve parity;
vc.
(b) Accuracies during training.
0.25
90
50
0..00
Attention Weight
Accuracy (%)
80
Sequence
70
60
0.05
50
LSTM
0.00
25
50
75
100
0
20
40
60
80
Internal Ticks
Interall Tick
(c) Accuracies ticks.
(d) Example attention strategy.
Figure 6: CTM performance on the parity task: (a) example; (b) training accuracy comparisons; (c) Impact of internal ticks on accuracy; and (d) an example showing how this CTM uses at least one attention head to scan the input sequence from start to end. Error bars (b, shaded) represent standard deviation over seeded runs; Appendix discusses the implications of seed variations.
The CTM's 's accuracy improves with more internal ticks, significantly outperforming parameter- matched LSTMs, which struggled with stability and performance (Figure CTMs with 75 and 100 ticks could achieve perfect accuracy in some seeded runs. Figure 6d shows how the attention shifts over the input data, and Figure 7 shows a specific demonstration of 8 attention heads), revealing a distinct and interpretable strategy. Which specific 'style' of solution depends on the configuration and seed, so we show other examples and analyses in Appendix
[22 Crucially, this experiment demonstrates that the CTM can learn to form and follow an internal strategy for an algorithmic task.

---

a
(a) f 5
(b) 30
tc) 75
(d) Target
Figure 7: Determining parity: (a, b. e) are the trajectories of the argmax of attention for 4 heads and the corresponding prediction at different internal ticks, and (d) is the target (perfectly predicted here). See supplementary material "parity.mp4' for video format.
Other Experiments and Analyses
We also evaluated the CTM in number of other settings in order to probe its functionality and versatility, Owing to space constraints, we provide the details these additional experiments the appendices (referenced below). In summary, these additional experiments investigated:
CIFAR-10 Classification Compared Humans (Appendix G. Ip: The CTM, feed-forward, and LSTM baselines were evaluated on CIFAR-10, 10, with results compared against human data for difficulty and uncertainty. The CTM demonstrated good model calibration and alignment with humans.
We investigated the impact of model width and the number of internal ticks. We found that the diversity of neural activity are functions of these. Wider models tended to exhibit more varied neural dynamics. Using more internal ticks allowed the CTM to engage in extended processing, sometimes revealing distinct computational phases.
CIFAR-100 Ablation Studies (Appendix G.2
Neuron-Level Models and Synchronization Ablations (Appendix G.3)
We compared the CTM parameter-matched variants without NLMs and without synchronization, as well an LSTM with synchronization. The results show that the combination of neuron-level models and synchronization as a representation is key to the success of the CTM.
Sorting Real Numbers (Appendix G.4|; The CTM was tasked with sorting sequences of 30 real numbers, outputting sorted indices sequentially using Connectionist Temporal Classification (CTC) loss [46]. This experiment showed that the CTM could learn an algorithmic sorting procedure and exhibited adaptive computation by varying its internal processing duration ("wait times") based on characteristics of the input sequence, such as the difference between successive values.
Q&A MNIST (Appendix G.5); : In this task, the CTM processed sequences of MNIST digits fol- lowed by index and operator embeddings to perform multi-step modular arithmetic. This investigation highlighted the CTM's capacity for memory and retrieval, using its synchronization mechanism to recall digit information beyond the immediate history window of individual neuron-level models, and to generalize to longer computational sequences than seen during training.
Reinforcement Learning (Appendix G.6): The CTM was adapted for reinforcement learning in several partially observable Markov decision processes (POMDPs), including classic control (CartPole, Acrobot) and grid-world navigation (MiniGrid Four Rooms). This demonstrated the CTM's applicability to sequential decision-making continuous interaction settings, where it achieved performance comparable to LSTM baselines while developing richer internal state dynamics.
Discussion and Conclusion
The Continuous Thought Machine (CTM) represents new perspective, where the temporal dynamics of neural activity are central to artificial cognition. Its core innovations neuron-level models and -fairaattiioeeeeeiiieelllll enable it to both unfold and leverage neural dynamics to solve problems. We showed in this work that such an approach is not only feasible but also leads to unique computational capabilities and emergent properties.
synchronization
latent
Our experiments demonstrate that the CTM can effectively solve challenging tasks. We trained CTM to observe, plan, and implement routes through 2D mazes using setup that necessitated the

---

formation of an internal world model. On ImageNet, the CTM exhibited native adaptive computation, naturally tailoring processing time to imput difficulty, and achieved strong calibration desirable property often requiring specialized techniques. On algorithmic tasks like parity checking, the CTM developed interpretable, sequential problem-solving strategies. Notably, the core architecture remained consistent across tasks, highlighting its robustness.
The CTM's NLMs are inspired by the complexity of biological neurons, but are implemented with a level of abstraction appropriate for modern deep learning. The direct use of neural synchronization as a representation is, to our knowledge, a novel approach this scale, offering benefits such as a high-cardinality representational space and the potential to capture the temporal aspects of 'thought'. While traditional deep learning has abstracted away neural timing for computational efficiency, the CTM shows that reintroducing such dynamics in structured way can unlock new functionalities.
Limitations. The CTM uses an internal sequence, meaning training times are extended. NLMs also increase parameter counts compared to standard activation functions, but also provide a new avenue for scaling. The experiments in this paper are preliminary and not intended beat state-of of-the-art models tailored for performance, therefore a limitation of this paper is its relatively limited depth of comparison since we favored breadth to investigate the CTM's internal functionality.
Future Work. We plan to apply the CTM to language modeling, self-supervised video understand- ing, lifelong-learning, diologicallyiispireee memory and plasticity, multi-modal systems, and more. We believe that, conceptually, synchronization representations have high widespread potential.
References
[1] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436-444, 2015.
[2] Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning. MIT press Cambridge, 2016.
[3] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprini w-xXiv:2206076666 2022.
[4] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
[5] Gary Marcus. Deep learning: A critical appraisal. arXiv preprint arXiv:1801.006:3 2018.
[6] François Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547, 2019.
[7] Neil C Thompson, Kristjan Greenewald, Keeheon Lee, Gabriel F Manso, et al. The computa- tional limits of deep learning. arXiv preprint arXiv:2007.05558, 10, 2020.
[8] Patrick Hohenecker and Thomas Lukasiewiez. Ontology reasoning with deep neural networks. Journal of Artificial Intelligence Research, 68:503-540, 2020.
[9] Peter Cariani and Janet M Baker. Time is of the essence: neural codes, synchronies, oscillations, architectures. Frontiers in Computational Neuroscience, 16:898829, 2022.
[10] Wolfgang Maass. On the relevance of time in neural computation and learning. Theoretical Computer Science, (61(1):771) 2001.
[11] David P Reichert and Thomas Serre. Neuronal synchrony in complex-valued deep networks. arXiv preprint arXiv: 132.61331...15,.000 2013.
[12] Yhi Lee, Hideyuki Hasegawa, and Shangce Gao. Complex-valus neural networks: A comprehensive survey. IEEE/CAA Journal of Automatica Sinica, 106-:146611426 2022.
[13] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pages 4651-4664, PMLR, 2021.
10

---

[14] Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartold- son. Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up rest-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.0517,, 2025.
[15] Liu Yang, Kangwook Lee, Robert Nowak, and Dimitris Papailiopoulos. Looped transformers are better learning learning algorithms. arXiv preprint arXiv:2311.12424, 2023.
[16] Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. In International Conference on Machine Learning pages 527-536. PMLR, 2017.
[17] Andrea Banino, Jan Balaguer, and Charles Blundell. Pondernet: Learning to ponder. arXiv preprint arXiv:2107.05407, 2021.
[18] Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint orXiv:160:: 08983, 2016.
[19] Fuzhao Xue, Valerii Likhosherstov, Anurag Arnb,, Neil Houlsby, Mostafa Dehghani, and Yang You. Adaptive computation with elastic input sequence. In International Conference on Machine Learning, pages 38971-38988 PMLR, 2023.
[20] Shawn Tan, Yikang Shen, Zhenfang Chen, Aaron Courville, and Chuang Gan. Sparse universal transformer. arXiv preprint 1000022310.07096 2023.
[21] Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. Quiet-star: Language models can teach themselves to think before speaking. arXiv preprint arXiv:2403.09629, 2024.
[22] Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. arXiv preprint arXiv: 10893, 2019.
[23] Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent models of visual attention. Advances in neural information processing systems, 27, 2014.
[24] Samuel Schmidgall, Rojin Ziaei, Jascha Achterberg, Louis Kirsch, S Hajiseyedrazi, and Jason Eshraghian. Brain-inspired learning in artificial neural networks: a review. APL Machine Learning, 2(2), 2024.
[25] Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, and Radu Grosu. Liquid time-constant networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7657-7666, 2021.
[26] Anand Gopalakrishnan, Aleksandar Stanié, Jürgen Schmidhuber, and Michael Mozer. Recur- rent complex-weighted autoencoders for unsupervised object discovery. Advances in Neural Information Processing Systems, 37: 10787-117841 2024.
[27] Matei-loan Stan and Oliver Rhodes. Learning long sequences in spiking neural networks. Scientific Reports, 14(1):21957, 2024.
[28] Wolfgang Maass. Liquid state machines:
motivation, theory, and applications. Computability in context: computation and logic in the real world, pages 275-296, 2011.
[29] Friedemann Zenke and Surya Ganguli. Superspike: Supervised learning in multilayer spiking neural networks. Neural computation, 30(6):1514-1544 2018.
[30] Larry F Abbott, Brian DePasquale, and Raoul-Martin Memmesheimer. Building functional networks of spiking model neurons. Nature neuroscience, 19(3)350-3355, 2016.
[31] David Sussillo and Larry Abbott. Generating coherent patterns of activity from chaotic neural networks. Neuron, 63(4):544-557, 2009.
[32] Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake A Richards, and Richard Naud. Burst-dependent synaptic plasticity can coordinate learning hierarchical circuits. Nature neuroscience, 24(7):1010-109, 2021.
11

---

[33] Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Leg- enstein, and Wolfgang Maass. solution the learning dilemma for recurrent networks of spiking neurons. Nature communications, 11(1):3625, 2020.
[34] Louis Kirsch and Jürgen Schmidhuber. Meta learning backpropagation and improving it. Advances in Neural Information Processing Systems, 34: 111442222-11444 2021.
[35] Joachim Pedersen, Erwan Plantec, Eleni Nisioti, Milton Montero, and Sebastian Risi. Struc- turally flexible neural networks: Evolving the building blocks for general agents, In Proceedings of the Genetic and Evolutionary Computatioon Conference, pages 1119-1127, 2024.
[36] Louis Kirsch, Sehastian Flennerhag, Hado Van Hasselt, Abram Friesen, Junhyuk Oh, and Yutian Chen. Introducing symmetries to black box meta reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7202-7210, 2022.
[37] Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. Advances Neural Information Processing Systems, 34:6695-6706, 2021.
[38] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[39] Peter Uhlhaas, Gordon Pipa, Bruss Lima, Lucia Melloni, Sergio Neuenschwander, Danko Nikolić, and Wolf Singer. Neural synchrony in cortical networks: history, concept and current status. Frontiers in integrative neuroscience, 3:543, 2009.
[40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[41] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[42] Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. End-to-end algorithm synthesis with recurrent networks: Extrapolation without overthinking. Advances in Neural Information Processing Systems, 35:20232-20242, 2022.
[43] David Ha and Jürgen Schmidhuber. World models. arXiv preprint 183. 2018.
[44] Cristina M Atance and Daniela K O'Neill. Episodic future thinking. Trends in cognitive sciences, 5(12):533-539 2001.
[45] Lyle Muller, Frédéric Chavane, John Reynolds, and Terrence J Sejnowski. Cortical travelling waves: mechanisms and computational principles. Nature Reviews Neuroscience, 19(5):255- 268, 2018.
[46] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376, 2006.
[47] Ulya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1711..1711.05 101, 2017.
[48] James Goreet and Matt Thomson. Automated construction of cognitive maps with visual predictive coding. Nature Machine Intelligence, 6(7):820-83) 833, 2024.
[49] Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-00-27. Open Review, 62(1):1-62, 2022.
[50] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.034223 2018.
12

---

[51] Takeru Miyato, Sindy Löwe, Andreas Geiger, and Max Welling. Artificial kuramoto oscillatory neurons, arXiv preprint arXiv:2410.13821 18221, 2024.
[52] Mozes Jacobs, Roberto C Budzinski, Lyle Muller, Demba Ba, and Anderson Keller. Traveling waves integrate spatial information into spectral representations. arXiv preprint arXiv:2502.06034, 2025.
[53] Tien Ho-Phuoc. Cifar10 to compare visual recognition performance between deep neural networks and humans. arXiv preprint
2018
[54] Joshua C Peterson, Ruairidh M Battleday, Thomas L Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9617-9626, 2019.
[55] Robin Manhaeve, Sebastijan Dumancie, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. Deepproblog: Neural probabilistic logic programming. Advances in neural informa- tion processing systems, 31, 2018.
[56] Imanol Schlag and Jürgen Schmidhuber. Augmenting classic algorithms with neural components for strong generalisation ambiguous and digeediimnsional data. In Advances in Programming Languages and Neurosymbolic Systems Workshop, 2021.
[57] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11)2278-2334, 1998.
[58] Andrew G. Barto, Richard Sutton, and Charles W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cyberneties,
846, 1983.
[59] Richard S Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding, Advances in neural information processing systems, 8, 1995.
[60] Maxime Sievvalie-BBoiveert Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, abs/2306. 13831, 2023.
[61] Mark Towers, Ariel K Kwiatkowski, Jordan Terry, John U. Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, Rodrigo Vicente Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Hannah Tan, and Omar G. Younis. Gymnasium: standard interface for reinforcement learning environments, 2024.
[62] Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In AAAI fall symposia, volume 45, page 141, 2015.
[63] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International conference on machine learning, pages 933-941. PMLR, 2017.
[64] Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 1- 2022.
[65] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint N00.::1777.03444 2017.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Our main claim is that introducing way of using neural dynamics, more biologically plausible mechanism, for neural computation and the core representation of the model. The entire paper serves to explain and justify this perspective.
Justification:
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide a limitations paragraph at the end Section g honest about the preliminary nature of this work, favoring exploration and sharing with the Al community over depth of analysis or hyperparameter searches. Furthermore, we also discussed the core limitation of the CTM: its sequential nature.
We were entirely
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: no theoretical results.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide extensive hyperparameter settings for all experiments in Appen- dices D.2 D.4 E.
G.2.3 G.5.2 and G.6. multiple pseudo-code listings to aid implementation in listings to and Section 3 fuuly describes the novel architecture.
F4 F.5
G.I.I
We also provide (and will release) the working code.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We include a zipped code repository with scripts to run all experiments in the paper, and README files describing experiments. Our hyperparameter settings in Appendices D.2 D.4 E. F. F.5 G.I.I G.2.3 G.5.2 and G.6. also aid here. We use publicly accessible data, with the maze data as an exception, which we can also provide google drive link (it is too large for the appendix) and will release publicly.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Hyperparameters in Appendices D.2 D.4
G.2.3
G.5.2
G.6.I including training details. Owing to the highly experimental nature of this work, did not go through extensive hyperparameter tuning and used default settings wherever possible. We also discussed the limited tuning and serach in the limitations paragraph in Section
7. Experiment statistical significance
14

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In most experiments when comparing other models (e.g., Section and Ap- pendices [. G.2 G.5 a.6]..6). The error bars are always standard deviation over seeded runs, as discussed in captions in these sections. For the maze task in Section 4 the computational overhead meant that we did not have the time or resources to compute error bars, but we did report multiple settings for the main comparison method. For Imagenet-IK in Section 5 we also did not compute error bars owing to computational constraints, and also because this was meant as a demonstration as opposed to a strict performance evaluation.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: While do mention in the hyperparmeter appendices that used NVIDIA H100's, we did not track or report all the detail of computational requirements. This paper is meant initial piece of research change to existing architectures, which would warrant more in-depth compute analysis.
type of NN, instead of evaluation of
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with NeurlPS Code of Ethics httpp/////eurpppp.cccooolttt//t/
Answer: [Yes]
Justification: We made sure to preserve our anonymity in this submission.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Tw we discussed how recurrence and reasoning are current focuses in Al research, and how this is driving towards more powerful AL. Throughout this paper discuss the potential for adaptive compute and how it reduce resource use, and the interpretability of our method, which can aid in model trustworthiness. While we do not specifically intentionally discuss the societal impacts of these perspectives and findings, moving Al forward in a tangible way is discussed throughout.
Justification: In Section [
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not release data models that have a high risk for misuse, such as pre-trained language models, image generators scraped datasets.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We used publicly available data and cited appropriately.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation
provided alongside the assets?
15

---

Answer: [Yes]
Justification: We submit code for our new architecture as supplementary and will release this publicty. There is extensive documentation inside that code.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: never used crowdsourcing.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We never used crowdsourcing and the CIFAR human data was publicly available from prior cited research.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [No]
Justification: We did LLMs for any component above.
16

---

Appendices
The following appendices provide further details about the architecture, experimental setup, and describe additional experiments we ran while undertaking this research. Specifically, these appen- dices are structured follows: we begin with glossary of key terms (Appendix A followed by collection of code listings to support our main descriptions (Appendix [B. We then present a more detailed explanation of the core method (Appendix (). including the synapse model (Appendix (C.I) and our approach to neuron sampling for synchronization (Appendix C.2). Next, we outline the experimental setups for the maze (Appendix D), ImageNet-1K IK (Appendix E and parity tasks (Ap-
pendix F. This is followed by in-depth descriptions of further experiments (Appendix [ including results on CIFAR- 10 versus humans (Appendix [..I.. ablations using CIFAR 100 (Appendix G ablating the use of NLMs and synchronization
representation in maze task (Appendix G.3 sorting (Appendix G.4). question-ansee answering and memory (Appendix G.3), and reinforcement learning (Appendix G.6). We provide details for fast recursive formulation computing synchronization (Appendix Hp. We conclude by providing insights into some of the emergent phenomena we observed while training working with the CTM in Appendix[]
A Glossary
We provide glossary containing the terminology and symbols used throughout this paper, in Table and Table respectively.
Term
Description
Internal tick Memory length
One step internal computation.
Length of rolling history of pre-activations, updated in rolling FIFO fashion
Synapse model
Recurrent model that takes z' and of as input produce pre-activations, a'.
rreaactvationss
Output of recurrent synapse model, input to NLMs.
Post-activations
Output of NLMs, neuron states at time t.
(Pre/Post) activation history
Sequentially ordered history of activations over time. Per-neuron MLP over pre-activation history.
Neuron-Level Model (NLM)
Synchronization
Dot product of post-activation histories.
Self-pair
Diagonal synchronization matrix entries (i. i).
Action synchronization
Synchronization representation for attention queries.
Output synchronization
Synchronization representation for predictions.
Decay Tij
Learnable time decay for synchronization (action or output).
Feature extractor
Task-specift input encoder (c.g.. ResNet). Output after synchronization, and keys/values from data. Table I: Glossary of terms.
Attention output
using queries, q', computed from action
B Listings
In this section, we describe the core aspects of the CTM using pseudocode. Specifically, Listing | gives simplified overview of the CTM code. Sestinggshowcase how the NLMs can be written shows how we compute synchronization. Listing shows how the certainty-batt loss described in Section 3.5 is computed.
using Einstein summation. Listing shows
Method details
C.1
Synapse models
Figure g show the synapse model which is the recurrent structure that shares information across neurons the CTM. is implemented by choosing depth of & (always even), where each subsequent
17

---

Symbol
Meaning
Number of internal ticks
M
Memory length
maddel
Dimensionality latent state in the CTM
disput
Dimensionality of attention output
dnidden
Hidden size in each newron's 's private MLP (NLM)
k
Depth of the synapse MLP or U-Net
Pdropout
Dropout probability in the synapse model
Theads
Number of heads in multi-head attention
Jaction
Number of neurons used for action synchronization Number of neurons used for output synchronization Dimensionality of action synchronization vector
Jout
Daction
Dout
Dimensionality of output synchronization vector
Thell
Number of self-pairs (i, i) used in synchronization sampling Learnable decay parameter for synchronization between neuron i and Full synchronization matrix at internal tick /.
rs
St
q'
Query vector projected from action synchronization Output vector (e.g., logits) projected from output synchronization
y
Table 2: Glossary of symbols.
layer width is chosen to linearly reduce the dimensionality until a width of 16 reached, and then increase thereafter, using skip connections to retain information. The synapse model also takes of (the output of attention) as input.
C.2 Sampling synchronization neurons
The CTM operates using recurrence on a latent representation, z, that is D-dimensional. z unfolds over time and the synchronization between some chosen neurons form the new kind of representation that the CTM enables.
There are Dx(++1) unique pairs of neurons, making for substantially larger set of neuron synchronization pairs than there are neurons themselves. This motivates the need for a selection process. Over the development of the CTM we came up with three approaches to selecting neurons:
1. Dense pairing: in this setup select J neurons and compute synchronization for every possible (i.j) pair of the J neurons. For S we choose Jout neurons and for St Sueioon we choose non-overlapping Jaction neurons. Selecting Jont neurons for dense pairing results in an output synchronization representation of D Jon X and similarly for the action representation.
2
This approach essentially creates a strong bottleneck where all gradients must flow through the selected neurons, which can be advantageous for some tasks.
2. Semi-dense pairing: in this setup open up the aforementioned bottleneck twofold by selecting two different subsets, J1 and J2, such that the left neurons of the synchronization dot product, i, are taken from J1 and the right neurons, are taken from J2. The same dense computation as before is then applied.
The bottleneck width in this case is 2x as wide as before. Output and action selections are. once more, not overlapping.
3. Random pairing: in this setup we randomly select Doat or Dastion pairs of neurons and compute the synchronization between each pair as opposed doing densely between all selected neurons. We also intentionally compute the (i, i) dot products between fislrr neurons in each case in order ensure that the CTM could recover snapshot representation if it wanted to.
This opens up the bottleneck much more than before. We allow overlapping selections in this case.
18

---

#сссesввaReS6ssR DEPINITIONS Oyypar paranetara not shown for aimplicity) 66A66065SАллс044 Backbone can be. Resket for images
PestureBncoder
projectors, standard attention nodule s-projecter, kv. kv.projactor Lisear(), Lineaz
ahtn Nulssllandttsmntin)))
Syваpaи
linear MLP D-MET
аyлараея KLP()
e Nearon lavel models (sne Listing 2)
lerro__leel__oddll. KLMB()
Uutput prosestor
eynchronteation
lieting
output_proj Linear
* Initialise pre-activations and x as lsarsable parazeters the model vidth
the nsuron nenory langth
srr_acte.history..ntt Decnettrr(eize-GC K))
Featurise inputa vith bactdone and compate KY takens
ky wr.projactor (hackbane (inputa))
In each ninibatch, initislise learnable be_act_bietory pro_acte_.hsster pre_aess..aatery..ntt unague.nee inneueueee (0).
start the post_acts_history with learnable x_inis post_acte, bistor ouspata.histon historr initial
wnngutteeeeueeeeeC (0), re).re din-0)]
eynchrozicarion query
synch,a coxpute, (ttttr type="action") e Other initialiaations including learnable start histories and first pre-acta round
thought_eteps))
Project attention query symchronseation
q g-projector (synch..)
atta_out attalq, kr, kr)
Concatemate attention
pre. pre_acts (oaapes
t Keep history of pre activations. This is FIFO structure:
concat concat hprre (i, :-1], pre_acte), dim=-1) pre_acte_hister,
histories (see Listing z poairon_oorel..oeeefffe (0t.
post
post_acts_ htt.aactt.hitttry.pppec.. appeery..apeedd())
t Compute syachronisations (see Listiag
syach_a compate_eysch wwt.cccpost..tt...eeeee... acts_bistory, type="action") synch_o compste_aysca actt t Projecte prodiction/ ootioo/ off synchrostsatios outpute_hiee history history.appead append (pppru (ontput.proo (eyach,_)_
typa="ontpus"))
Retura
thought step loss fuaction
zeturn patputs_aletor
Simplified overview of the CTM code. Features are encoded using a backbone (e.g., ResNet layers for images), data is attended by @ projecting query from neural synchronization, information is shared across neurons using an MLP synapse model to produce pre-activations, ! private neuron-level models are applied synchronization is computed from a ① tracked history of post-activations (see Listing 3). and outputs projected off of synchronization.
Listing 1:
tracked history of pre-activations (see Listing
D 2D Mazes
D.1 Dataset
We used the maze-dataset repository to generate mazes for this work. We generated mazes of size 19 19, 39 39, and 99 99. In each case we generated 50000 mazes and split them into train sets of size 45000 and test sets of size 5000. We used the 39 x 39 for training in this technical report and tested generalization on the 99 x 99. We provide all three maze datasets in the CTM code repository made available upon publication.
D.2 Architecture details
We used the following hyperparameters:
39 x 39 mazes and ResNet-34 backbone, where the keys and values were taken features after the second hyper- block, resulting in down-sample to 10 10
() = 2048 (the width of z' and a²)
We found the 19 19 beneficial for debugging, hence provide it too.
19

---

Initialiaations
weights.1 Paraneter (K, d_hidden, d_model))
(shape-(1
d_sodel))
weighta_2 Paraneter (shape (d_hidden, d_model))
bisa_2 zeros (shape-(1, d_nodel))
Foreard
bbebatch, Nezenory, ded_nodel, h-d_hidden
inputa shape (b, d, inputa hit.at..
nut winzun( ("bdN, Hhd-zbdb'., imputa, weights_1) biax_1 bdd ous, veighte_2)
Listing 2: Neuron-leved models: ① Using einsum greatly simplifies and speeds up the application of neuron-level models as their outputs can be computed in parallel. The first einsum computes h-dimension latent for each neuron from the (truncated to M most recent) incoming history, @. The second einsum then computes the single activation per-neuron (ignore the '1" dimension here for simplicity).
v AT IDITIALISATION:
choose D_chesen
paira total
D_chesen can be D_out D_action
other neuron selection strategies exist, but here se show randon selection
idza_left (lov-,, high-D, size-D_chosen)
high-D, size-D_chosea) laarnable azponential dacay acaling factora per sauras pair r Parasater(zereal (1, D_choaws, 1))
idxs_right randiat
overlap
8 INITIAL19ATIO OTER.
shape [3, T-history length)
8 decay BACK is time
t_back (7-1- -1, reahaaph,
Compute per NEDRON PAIR expozential decays, and expand
exp_decay =rp(-t_bbckk r). (zpand(1 7, b_choamn)
Compute weighted inner dot products using differet subseta of nsurona
s.auttiplied a (is.i,ixx_loft
exp_decay =[i,s, righht e [B, I, [_chosen]
San the repeasaopraattattoon (8_sultiplied) aum (1)/mma/aggg) aggt (exp_decay. (1))
dixanzion normalise by agrt of
dacays
D_chosen]
Listing 3: Neural synchronization to create the latent representations used in Listing Careful reshaping and broadcasting enables the of per neuron-pair learnable exponential decays for synchronization, which lets the CTM learn complex timing dependencies. The decay parameters are initialized as zeros (i.e., no decay). This process repeated for output and action (see Section 3.4.1) In practice we use recursive approach that greatly reduces compute overhead; see Appendix
k = 16 (synapse depth, layers down and 8 layers up) danpuu 512 (the width of attention output, o²)
Theads
16
Dense pairing for neuron selection (see Appendix C.2
Jout 32 (width of Stt. synchronization representation)
Jaction 32 (width of S synchronization representation)
T 75 (internal ticks)
M 25 (FIFO rolling memory input to NLMs)
duidden 32 (width of MLPs inside NLMs)
Paropout = 0.1 (dropout probability for synapse model)
No positional embedding
We used the following settings for optimization:
Trained using a batch size of 64 on H100 Nvidia GPU
1000000 iterations for training using AdamW 47
A learning rate of le-4 with linear warmup of 10000 iterations and decaying to zero using cosine annealing learning rate scheduler
20

---

def stm, (logit,,, targata):
8, c, T logits.shape
Ceclasses, T-thought steps
Targata shapa: [8]
4 Conpute cartainties на
sorsalised entropy
P.aoftmax
log-p tor (logits,
eatropy (L log-2. din-1)
maz maz_entropy sorch.log(C0
certainties -1- (entropp maar.antrop))
thape:
Repand targets over thought staps targets_atp torch. iewa.,, "aaggtt.uuquuuzaeee 7, -1) Loss function could other thinge,
cress without reduction Losaux m..Cravfatrayloon (nnerrryio((tlltttooooonn)
shaps [3, losses loss_.n
targets_exp)
for each ites in the minibatch a Get indices of lowext loxa thought
1oveet_1dx losses.argoing argoin(-1)
Get indicer of nost certain steps for each iten in the ninibatch
cartain_idr irraainties.argan (-1)
l0st (lasses[;, loxest_idx] losses sertain_16x)>22
Listing 4: CTM loss function, enabling the CTM to be flexible regarding the number of internal ticks used for any given data point.
DROP
DROP
DROP
DROP a'
LINEAR
EAR
LINEAR
LINEAR
LINEAR
O
Z
LN
LN
LN
LN
LN
SiLU
SiLU
SILU
SILU
SILU
h
h
h
LN
LN
Figure Overview of UNET style 'synapse" recurrent models. z are the post-activations from the previous step, o' the attention outputs from observing data, and the synapse model yields a' pre-activations for the NLMs to process. The UNET structure is set up so that the innermost bottleneck layer is 16 units wide, with linear scaling for each layer in between. Skip connections with layer-norm implement the classic UNET structure to maintain information. Layers that produce lower dimensionality shown in blue, while layers that produce higher dimensionality shown in orange.
No weight decay
This resulted in model with 31,998,330 parameters.
D.3 Maze curriculum
We adapted the loss function for solving mazes to include curriculum element. Before computing t and t2 for the loss in Equation (IT) we first altered the loss at each internal tick to only account for those steps in the maze that were correctly predicted with plus 5 additional steps along the path. This effectively lets the model (CTM or LSTM baselines) slowly learn to solve the maze from start to finish. Listing shows how this is implemented when computing the loss and is used for both CTM and LSTM training.
21

---

def singecclasiifiasionn losa (owdepraicciionn cersaintinz, targats, uww.mott_cartainnTrra
Computes
auto-estendingg
Predictions are of shape: (B, class, internal, ticks),
Gertainties of shape: (B, intarnal_ttcke)
dinassion [normalized 1-nprnallasddenttrryy)
Tazgeta are shape:
sse__oxt_cerraai will sslect wither the nest cartain point the final paist. can sargets_arpandee tarch. tarch-rapeat, intarllnt (arrraaaacc(rrgeeesneeenncc)))) unaguanza(-1), prediationa. & Lessez are of shape [3, internel_tien ticka]
losees Croseeintropproo (reduction= (predictions,, torgete.ospazéed
loss_indes__ loaasss argeis..ggnnnddee (dix-1))
loss_inder. certaintien (-1)
van_noot_certait & Revart final loxx if
1ose_indox_
badch_indezer tarch. torch. arange (pradiittiona..iiaaa
deviceepradeetionnn.deiii lose_index, 11.
lose_nininse.ce 1ossee rols_tel (batch, lose_indes_2].
loss
lose_eelectess/22
Listing Loss function implementation of Section 3.5 for maze route prediction, including an auto curriculum approach for both CTM and LSTM.
D.4 Baselines details
We tested a number of LSTM baselines for solving this task, but struggled with stability during training (see Figure particularly beyond a single LSTM layer or with a higher number of internal ticks. Hence we tested three LSTM configurations depths 1,2, and 3. For each model we tested with 75 internal ticks to match the CTM, and 50 internal ticks for stability. We also tested feed-forward model, projecting feature space (before average pooling) into a hidden layer of the same width as the CTM, yielding slightly higher parameter count. We kept all hyperparameters constant, yielding following setups:
LSTM, layer, T 50 and T 75: 42,298,688
LSTM, layers, 50 and
75: 75,869 75.869,504 parameters
LSTM, layers, 50 and = 75: 109,440,320 parameters
Feed-forward, with a hidden layer width of 2048 (and GLU activation thereon): 54,797,632 parameters
D.5
Maze loss curves
Figure D gives the loss curves for the maze solving models in Section 4.1 more stable and performant when trained on this task.
showing how the CTM is
D.6
Discussion: the need for a world model and cognitive map
Internal models of the world and cognitive maps represent crucial aspects of intelligent systems In this case, we consider world model to be an internal representation of the external environment, encapsulating an agent's 's knowledge about the world's structure, its dynamics, and its actionable place therein. A good world model should enable an agent to reason about the world, plan, and predict the consequence of its actions. Cognitive maps [48] specifically focus on spatial relationships and navigation. The ability to construct and utilize these internal representations strong indicator, and arguably prerequisite, for sophisticated intelligence. The notion of 'episodic future thinking' [44 is even considered a hallmark feature of human intelligence. An agent devoid a world model would be limited to reactive behaviors. Similarly, lacking cognitive map would severely restrict an agent's is ability to navigate and interact effectively within complex spatial environments. Therefore, the presence and sophistication of world models and cognitive maps can serve benchmark for evaluating intelligence.
43
48
22

---

FF
Loss (average cross entropy over path)
LSTM, depth=1. 50 ticks
LSTM, depth=2, 50 ticks
1.0
LSTM, depth=3, ticks
LSTM, depth=1, 75 ticks
0.8
LSTM. depth=2, ticks
LSTM. depth=3, 75 ticks
0.6
CTM
0.4
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Training iterations
1e6
Figure 9: Loss curves when training the CTM and baselines.
To this end, we designed the maze task such that it would require good internal world model to solve. This was achieved by (1) requiring the model to output a route directly, as opposed to solving the maze with local algorithm (37]. and (2) forgoing any positional embedding in the image representation, meaning that the model must build its own spatial cogmitive map in order to solve the task [48). Indeed, saw that the NLMs and synchronization components of the CTM enables it to solve our 2D maze task, far surpassing the best baselines we trained. Our results suggest that the CTM is more capable of building and utilizing internal model of its environment.
E ImageNet-1K
This section provides additional details and results for the ImageNet-1K experiments.
E.1
Architecture details
We used constrained version of the classic ResNet architecture [41] for this task, which adapted from https://githut: www.comm/uu/aa http///aaaaaa/yorcchh CIFAR10 It differs from the standard implementation for ImageNet in that the first convolution constrained to a kernel size of 3 x as opposed to 7 x 7. We used ResNet-152 structure and took the output prior to the final average pooling and projection to class logits. We used input images of size 224 224 which yielded 14 14 features for the keys and values used in cross attention.
We used the following hyperparameters:
= 4096 (the width of z' and a²)
k 16 (synapse depth, layers down and 8 layers up)
dinguut 1024 (the width of attention output, o²)
Theads 16
Random pairing for neuron selection (see Appendix
Dou = 8196 (width of st synchronization representation)
Daction 2048 (width of St St synchronization representation)
Nseli 32 (for recovering snapshot representation)
T 50 (internal ticks)
M 25 (FIFO rolling memory input NLMs)
daideca 64 (width of MLPs inside NLMs)
23

---

Parooout = 0.2 (dropout probability for synapse model)
No positional embedding
We used the following settings for optimization:
Trained using a batch size of 64 across 8 H100 Nvidia GPUs
500000 iterations for training, using a custom sampling such that each minibatch was sampled with possible replacement
AdamW 47
A learning rate of 5e-4 with linear warmup of 10000 iterations and decaying to zero using cosine annealing learning rate scheduler
Gradient norm clipping set norm of 20
No weight decay
E.2
Loss function
Listing 6 shows the python code for the image classification loss function used to train the CTM on ImageNet-1k IK. This accompanies the loss defined in Section 3.5
def cla.ageccaasiticaiiio.ooo (pw.diriitiiio certainties, targeto, use_nest ue__oss__crrtaaanTze)
Computes
with oute-oxtending cirriculum.
Prodictions are of chapo: (B, class, internal internal.ticse))
Certainties are of shape: (B, Intermal_ciekon (normalised, entropy, L-noraalied_eeatrpp
inside dimozsion
Targeta are shape:
use uec_nost_certain will solect either the nost certain point or the final point. e*n targets_
torch. torch- repeat, interioave (targets unsqueeze (-1), prediccioss.et2s t Loeses are of chape [3, internal ticks]
losses as Cr.Crrssnttopyyooo (reduction ieciiionnooooo 'sone (pradictioss, tapgets))xpanded)
jose_indox__ losses argnin(dix=1)
loss_indes__ certaintien (: [1] (-1)
if not use_aost_certaiz $ Revert to final less if set iosa_indes__[:))
batch, indozer terch. arange arange loss_sininsn_ce leasea (batch, indexer, hewp_indds______ me).nnn))))))))) losa_selected leases (batch, indexer, Josa_indes,_ .eann)
dericenpredietions.due device)
leas (Less_ninisun.ce loss_aeletted)/2 return loss
Listing 6: Loss function implementation of Section 3.5 for standard classification tasks, used for ImageNet-1K.
E.3
Further analysis
note how compute calibration. For Figure 5 we showed the calibration plots per internal tick. To compute these we averaged the probability of the predicted class (at a given tick) over all ticks preceding that. This approach aligned with the way that the CTM reasons and builds up its predictions. For example, an uncertain prediction (e.g., Figure [5] would have low certainty over all ticks, and typically exhibit only gradual increase certainty, whereas certain prediction (e.g., Figure 2b) would show a sharp increase in certainty early on, yielding an average predictive probability close to as thought progressed.
We also give accuracies over internal ticks in Figure 10a for four styles
of predictions:
1. Instant: by taking the prediction a given internal tick.
2. Most certain: by taking the prediction at the most certain tick up given tick.
3. Average logits: taking the average logits up given tick.

---

4. Average logits weighted by certainty: by first re- re-weighting logits prior to aggregating up a given tick.
This shows how the process of aggregating or using the predictions from the CTM is non-trivial. hope that this opens avenues for future research.
90.0
Top-k=5 accuracy
87.5
85.0
82.5
Instant
80.0
Most certain
Average logits
77.5
Logits weighted by certainty
75.0
20
40
Internal ticks
(a) Accuracy vs. steps for different prediction mechanisms.
10a
Counts cortainty
10
10a
10
10a
Internal ticks
(b) Histogram of certainty of at least 0.8.
Figure 10: Exploration of the performance and utility of the CTM. showing the relationship between internal ticks and top-5 ImageNet- IK accuracy. (a) we show the accuracy versus internal ticks when determining the output prediction in different ways, showing how taking the prediction at given internal tick is sensible until approximately 15 steps, where it becomes better to consider the certainty measure of success. In (b) show histogram of data counts exceeding certainty of 0.8 for each internal tick; color denotes class indices. In (c) we show calibration plots, where predicted probabilities for the CTM are considered be the average probability up to given internal tick, showing how this results in good model calibration.
In Figure 10b we show the distribution of classes (in different colours) over each internal tick.
E.4
Additional demonstrations
We showcase series of additional demonstrations of CTMs classifying images from ImageNet in Figures 12 to 15 We encourage the reader to view similar demonstrations as videos (we will provide links upon publication; see supplementary video 'imagenet.mp4 m
To visualize the dynamics of the neuron activations we used UMAP [50] to project the neurons to 2D feature space. We used the history of post-activations for 200 different ImageNet images as the high dimensional inputs to UMAP. Specifically, this input was 200 x T 200 x 5 1000 dimensional. The resulting 2D projection assigns each neuron to position in space determined by its activation 'profile' its response pattern both over time and over multiple stimuli. Visualizing this mapping over internal ticks reveals low frequency structures propagating across the feature space. We show snapshots of this visualization over the internal ticks Figure II noting that these visualizations best viewed in video form (see supplementary video 'umap.mp4")
Importantly, the CTM generates this structure in an emergent fashion, without any explicit driving signal. Analogous phenomena occur networks of Kuramoto oscillators [5]] in our case, waves
25

---

Figure 11: Observation of the neurons in the CTM as observes and thinks about an image. The colors indicate activations that range from low (blue) to high (red). We show the progression of the neural activity over internal ticks from top left to bottom right. Upon careful inspection, one can discover clear structures multiple scales. This visualization is best viewed in video form. (Sce supplementary video 'umap.mp4").
propagate across learned feature map in an all-to-all network. Concurrent work also explores explicitly encoding traveling waves for long-range communication [52]. We do not assign functional meaning to these observed waves but highlight their distinct presence during the CTM's thought process.
aatometed
Figure 12: Validation image index 1235, showing incorrect and uncertain prediction.
CEL TRIT TNIM (1.001)
SPV
Figure 13: Validation image index 15971, showing correct prediction and plausible 2nd most probable class.
26

---

Figure 14: Validation image index 21202, incorrect prediction after passing by correct prediction, showing 'over-thinking'.
deni's dami
Figure 15: Validation image index 39275, correct but uncertain prediction.
F
Parity
F1
Task Details
The parity of binary sequence is given by the sign of the product of its elements. When processing a sequence element by element, an RNN could conceivably compute parity by maintaining an internal state, flipping an internal 'switch' whenever negative number is encountered. However, if the entire sequence is provided simultaneously, the task increases in difficulty due to the increasing number of distinct patterns the input. Previous work [88 has addressed this challenge using recurrent models, which can learn sequential algorithms for statically presented data. Computing parity, posed in this manner, is well-suited for testing the capabilities of the CTM.
We apply the CTM to the task of computing the parity of 64-length sequence containing the values 1 and -1 random positions. Unlike [18], we set up the task such that the model computes the cumulative parity at every index of the sequence, not just the final parity. An example is shown in Figure (6a] The values -1 and are embedded as learnable vectors combined with positional embeddings, using attention to ingest input data. We train the CTM with the loss function described in Section 3.5 As a baseline we also trained an LSTM, but set 12 to be the final iteration since this gave the best results and stability for LSTM training.
F.2 Additional Results and Analysis
In this section, we describe some additional results for the parity experiments.
The CTM learns sequential algorithm. To analyze how the CTM learns to solve the parity task, Figure (howss the accuracy for each the 64 elements in the input sequence different stages of training, for three different internal tick configurations. The models first learn to predict the parity of the initial elements, and as training proceeds, learn to predict later and later positions. With more internal ticks, the model can accurately predict more elements in the target sequence.
27

---

1076
15%
Sequrnce
(a) 10 internal tick CTM.
(b) 25 internal tick CTM.
(c) 75 internal tick CTM.
Figure 16: Accuracy across the 64-element sequence at different training stages (indicated by color) for various internal tick configurations. Early in training, all CTMs accurately predict parity only for initial sequence elements, gradually improving for later elements as training progresses. Models with more internal ticks achieve higher accuracy, with the 10-step model (a) correctly predicting approximately half the sequence and the 75-step model (c) correctly predicting the entire cumulative parity sequence.
Train Prog. = 5%
Train Prog. 25%
Train Prog. 100%
Attention
Soa 10 telox
10.20
Wenght
Smg Seguenooe Iedexx
40
600
40
20
50.10
20
00.12
20
00.30
50.00
20
20
-00
4.0
Time
Time Stop
Timme Stop
Accuracy
Soqueraa indox
Sequence intex
Hin: Entropy
Min Entropy
40
Acounny
80
Sequerce
40
-5.00
50.0
50.0
20
20
-0
40
(a) CTM trained with 100 internal ticks.
Train Prog. 5%
Train Prog. 25%
Train Prog. 100%
Atrention
Seqerno tedox
60
20.20
Sequeroo tetex
00:23
Sequerce Nesghh
0.15
40
60
600
20.10
00.13
-3.00
20
120
0.29
50.00
3.00
60
Time
Time
Time
Accuracy
Soquerna Invox
60
Gequence Intex
Hin: Entropy
Min
40
80
Sequence
40
-7.0
70.0
70.0
60.0
50.0
50.0
50.0
(b) CTM trained with 75 internal ticks.
Figure 17: Attention patterns (top) and accuracy (bottom) at different points in training, for CTM trained with 100 internal ticks (a) and 75 internal ticks (b). The black points the accuracy plots denote the internal tick which the model reached maximum certainty, with the error bars denoting one standard deviation across samples.
28

---

To gain insight into how the model solves the cumulative parity task, we visualize the CTM's attention patterns, accuracy, and points of highest certainty across all 64 elements multiple stages of training in Figure 17 for two different models. The attention and certainty patterns evidence that these CTMs are leveraging different algorithms to solve the cumulative parity task. When using 100 internal ticks, attention moves from the beginning to the end of the sequence, and with it, the model increases its certainty of the prediction at that position. The CTM with 75 iterations, on the other hand, learns to attend to the sequence in reverse order, accurately predicting the parity of the majority of the sequence simultancously during the final internal ticks. This reverse search through the data suggests that the CTM is carrying out form of planning. building up its understanding of the observed data before making a final decision on the cumulative parity of the sequence. These results highlight that although multiple strategies exist for solving this task, some of which are more interpretable than others, the CTM clearly demonstrates the ability to form and follow a strategy.
F.3 Dataset details
The input data for the parity task is vector of length 64, where at each position is either -1 1. The target for each sample is a vector of the same size, where at each position is the parity of the sequence up to that position, which we refer the cumulative parity. This data is generated on the fly, each time a new batch is fetched.
E4 Architecture details
The following architecture is used for the experiments in the parity task. Inputs to the model are of shape (B, 64), where the size of the minibatch and sequence length are B and 64, respectively. Each of the values in the 64-1ength sequence are either -1 or 1, at random. First, the values of 1 and I are converted into embeddings in dembed dinpui and positional embeddings are added. The resulting embeddings are passed through linear layer with layer normalization to form (identical) attention keys and values. As described in section 3] the synchronization between Jaction neurons is computed and from this representation an attention query is formed. This query is then used to compute the attention values, which are concatenated to the activated state to be processed by the synapses and the neuron-level models. For the synapses we use a shallow feedforward network. This process repeats for T internal ticks, where at each internal tick t, the synchronization can be computed between Jeut neurons and projected to the logit space.
In the parity task, we experimented with how the model performs with varying number of internal ticks and memory length. As a baseline, we use single-layer L.TTMs which are both parameter matched and use the same number of internal ticks as the CTM.
All CTM models share common set of architectural hyperparameters, listed below. The Table shows the subset of hyperparameters that vary across experimental configurations.
dmodel 1024
dipputt = 512
duidden = 4
k == 1
Paropout 0
Theads
Jaction 32
Jun 32
Semi-dense pairing was used for selecting neurons for synchronization
Absolute positional encoding was added to the input features
F.5
Optimization details
The CTM was trained using the certainty-based loss function described in section 3.5 whereas the LSTM basclines utilized the cross-entropy loss computed at the final internal tick. This choice was
29

---

Model
T
M
dmodel
Total Parameters
CTM
1024
4908706
LSTM
669
4912710
CTM
10
1024
5043874
LSTM
10
686
5050716
CTM
10
1024
5212834
LSTM
25
706
5224386
CTM
50
25
1024
5719714
LSTM
50
765
5722374
CTM
75
25
1024
5719714
LSTM
75
765
5722374
CTM
100
50
1024
6564514
LSTM
100
857
6567486
Table 3: Model hyperparameters for the parity task (that vary across configurations
made due initial difficulties training the LSTM effectively with the certainty- -ased loss. In Figure [8] we compare training accuracy curves for the LSTM baselines with 10 and 25 iterations, trained with either the final or certainty-based loss. Generally, both loss functions lead to unstable training for LSTMs with multiple internal ticks.
We used the following settings for optimization:
Trained using batch size of 64 H100 Nvidia GPU.
200000 iterations for training using Adam W 47
A learning rate le-4 with a linear warmup 500 iterations and decaying to zero using a cosine annealing learning rate scheduler.
Accuracy (86)
56
54
Certain 10 Iters, Final
52
50
25 Iters. Final
50000
100000
150000
200000
Triiiing Iterations
Figure 18: Test accuracies for LSTM baselines, trained with either the certainty- based loss the cross-entropy loss at the final internal tick. Both loss functions lead to unstable learning.
F.6 Results
Model performance varies significantly between seeds Figure 6b shows the accuracy over training for various CTM and LSTM configurations, with each configuration averaged over three independent runs. These training curves exhibit considerable variance due to significant differences in performance between runs, strongly influenced by the initial random seed. For example, Figure 19 shows individual training curves for the CTM trained with 75 internal ticks and a memory length of 25. Runs and reach perfect accuracy, while run 2 converges to suboptimal solution.
Furthermore, three of these models display significantly different behaviors, with each CTM attending to very different parts of the input sequence over the 75 internal ticks. These attention patterns over the internal ticks are shown Figure 20 Run results in a model that attends from the beginning to the end of the entire sequence, while run attends in reverse order.
30

---

100
Run
0.7
Run 3
Run
Pun
Run
(h)
0.5
80
0.4
70
0.2
0.0
50000
100000
1500RD
200000
50000
100000 Theining lteration
156030
200000
Theining lherelimm
Accuracy.
(h) Loss.
Figure 19: Training curves for three CTMs trained with three random seeds. Run and 3 converge to a loss of zero, while the other run converges a non-zero loss.
60
00.23
60
Attendeon Wogght
Sequiric Irdise
60
100.5
40
00.15
80
00 22
00.12
00.1
20
12
00.05
20
20
50
23
40
50
20
-00
50
Time Stop
Time Stop
Time Stop
(a) Run
(b) Run 2.
(c) Run 3.
Figure 20: Attention patterns for each the three runs after training. For videos of these patterns emerging during training for (a) and (c) see supplementary materials sewriy--ttenionnbbawwaard and sonrit-attention-ooraddd
respectively.
G Additional Experiments
G.1 CIFAR-10: CTM versus Humans and Baselines
In this section we test the CTM using CIFAR-10, 10, comparing it to human performance, feed- forward (FF) baseline, and LSTT baseline. For the model-based baselines, we used constrained featurization backbone in order to emphasize the differences owing to the model structure post- featurization (i.e., CTM versus LSTM versus FF). We also used internal ticks to give the CTM and LSTM 'time to think'. The human and model baselines were set up follows:
Human baseline. We used two datasets of human labels for CIFAR-I0; we call these CIFAR-IOD [53] owing to its calibration of difficulty levels, and CIFAR-IOH [54] originally used to quantify human uncertainty We used CIFAR- to determine easy versus difficult samples, and CIFAR-10H
direct human baseline.
FF baseline. A feed-forver -eed-dor only baseline (denoted FF). An MLP was applied to ResNet features after average pooling, where the width of the hidden layer was set to match the parameter count of the CTM for this experiment.
LSTM baseline. An LSTM set up to unroll with an internal thought dimension, with hidden width image at each step and used the same loss as the CTM for valid comparison.
match the parameter count of the CTM. The LSTM could attend the
Figure 21 shows the training curves of the CTM, FF, and LSTT models, and calibration plots for each, including an estimation of human calibration using CIFAR-105 10H. The FF baseline reaches high training accuracy early on, but also demonstrates poor generalization gap. The LSTM less stable during training (we had to set the learning rate to 0.0001 for all experiments because of this) and yields a marginally improved test accuracy. The CTM more stable and performant.
For the human calibration we used the probabilities provided CIFAR- 10H, which were computed using guesses from multiple humans. We computed calibration here as we did for ImageNet- K (see Figure 5b in the main paper): we compute the predictive probability as the average probability for
(CIFAR-IOD can be found at htttp:////sttee..ggll..oo//////s htw.:p/www.....eeeoeooooo////eee 1ifar10-classificaat CIFAR- 10H be found at httpps///gttaio.cccm///ptttrrrrt
31

---

95.0
92.5
40.0
Rete of postties
87.5
0.8
82.5
train
80.2
CIM, train
Perfect Calloraion
FT, tent: 84-4
D2
CTM
77.5
LSTM, test: 85 54%
Harras (egg)
CTM. test: 86.03%
LSTH
75.0
0.0
Thiniin
promensiitt
(a) Training curves.
(b) Calibration plot.
Figure 21: CIFAR-10 training curves (average over 3 seeds) and calibration plots for the CTM, a forw-forward only baseline, and LSTM baseline. The CTM is slower than the LSTM per forward pass (+224x) but is also more stable during learning. The CTM has the best test performance. The calibration plot shows that even human baseline [54] is poorly calibrated, and that the CTM demonstrates good calibration, failing in a way that is strikingly similar to humans.
the chosen class over all internal ticks. None of the models are perfectly calibrated, but the CTM demonstrates the best calibration, even when compared humans. Strikingly, the CTM has even better calibration than humans, while the LSTM follows the human under-confidence.
2400
0.24
2400
Uncer memelleet wimspp))
Aecursy [S)
60
Time
+0.20
1800
1800
- CIFR-11D (Berchmari)
CITAR-TCH Otumans!
(lug)
1600
(Aug)
CTM
00.44
difficulll
difficlly
cifficulty
(a) Performance vs. difficulty.
(b) CTM uncertainty vs. difficulty. (c) LSTM uncertainty vs. difficulty.
Figure 22: Analysis model and human performance versus difficulty on CIFAR-IO. We used the difficulty calibration from [ and compared human predictions from CIFAR- [54]. We assume that human reaction times are a reasonable proxy for uncertainty and compare this to the trend in
uncertainty for the CTM and a parameter-matched LSTM baseline. The eπror visualized here is scaled standard deviation.
Figure 22a compares models and CIFAR-10H against the difficulty determined using the CIFAR- dataset. Each model and humans have similar trends this case, although the CTM follows most closely to CIFAR- 10H. Figures 22b and 22c compare the uncertainties of the CTM and LSTM to the uncertaintics of humans (using reaction times from CIFAR- 10H as proxy for uncertainty). We compute the CTM and LSTM uncertainties using the normalized entropies (see Section 3.5 in the main paper) averaged over internal ticks as this approximates the total uncertainty each model has regarding the observed data. Both the CTM and LSTM exhibit trends similar to human reaction times.
Figure 23 shows the neural activities for the CTM and the LSTM baseline. The CTM yields rich, diverse, and complex dynamics with multiple interesting features, including periodic behavior (there is no periodic driving function). The distinct difference between the CTM and LSTM neural activities evidence that the two novel elements of the CTM (NLMs and synchronization as a representation) enable neural dynamics as a fundamental computational tool.
32

---

(a) CTM neural activity.
(b) LSTM neural activity.
Figure 23: Neuron traces for the CTM and an LSTM baseline on CIFAR-10, showing how the CTM produces and uses complex neural dynamics. The LSTM yields some dynamic behavior in the post-activation histories shown here, but not nearly to the same degree. Each subplot (in a random color) shows the activity of single neuron over internal ticks, where multiple examples for different images are shown as faint background lines, and the foreground line is from a randomly chosen example.
G.1.1 Implementation details
For this experiment, we used the first hyper-block of constrained ResNet-18 backbone (see Ap- pendix E.I 5 convolutional layers in total and a downsample factor of 2x. We used the following hyperparameters for the CTM:
256 (the width of 'z' and a²)
k 10 (synapse depth, 5 layers down and 5 layers up)
dinpur 64 (the width of attention output, o²)
Ticaads 16
Random pairing for neuron selection (see Appendix C.2
Dam 256 (width of St synchronization representation)
Daction 512 (width of secioon synchronization representation)
Nselt
T 50
M 15 (FIFO rolling memory input to NLMs)
daidten 64 (width of MLPs inside NLMs)
Péropout = 0.0
Weight decay 0.0001
No positional embedding
We used the following settings for optimization:
33

---

Trained using a batch size of 512 on 1 H100 Nvidia GPU
600000 iterations for training using AdamW [47]
A learning rate of le-4 with linear warmup of 2000 iterations and decaying to zero using cosine annealing learning rate scheduler
For the LSTM baseline a 2-layer LSTM was used as this performed better than single layer LSTM setup and was relatively stable in training (compared to the maze task). The CTM synapse depth, memory length, and NLM hidden width were chosen such that the model width was kept constant (256) while parameter counts were closely matched between the CTM and LSTM. For the feed-forward model kept the model width constant.
G.2 CIFAR-100: Ablation Analysis
In this section we explore two aspects of the CTM: (1) width (i.e., number of neurons), and (2) number of internal ticks. We used CIFAR-IO0 in the experiments discussed below as is a more challenging dataset than CIFAR-10, 10, while remaining relatively low-demand regarding compute.
G.2.1
Varying the number of neurons
625
020
9.30
0.23
11181-2243
0.25
61.
625
9.25
0.04
620
8.66
1830
2000
-1.3
23
1.3
(a) CIFAR-IO0 accuracies.
(b) Neuron similarity across data.
(c) Neuron similarity across neurons.
Figure 24: CIFAR- 100 accuracies and neuron similarities for different model widths. For (b) neuron similarity across data, we computed the average (over neurons) cosine similarities between matched neurons for all pairings across sample of 128 images cach bar is the proportion of image pairings that have this average neuron similarity. For (c) neuron similarity across neurons, we compute the average (over data) cosine similarities for all pairs of neurons within each model each bar is the proportion neurons having that average cosine similarity. Cosine similarity absolutely closer to zero indicates dissimilarity, and hence improved neuron diversity.
Figure 24a shows CIFAR-100 100 accuracy versus model width (i.e., the number of neurons) for fixed backbone network (details which in Appendix G.2.3). evidencing improved test performance to a point, and then reduction in performance. The performance drop-off might be related to overfitting, but it might also be that a wider model requires more training (we set fixed number of training iterations).
Figures 24b and 24c show relationship between model width and the diversity neural activity. Intuitively, expect that with more neurons would observe greater degree of neural activity, and these distributions show exactly that. In Figure 24b we see that when measuring cosine similarity neuron-level across data points (averaged over all neurons), a wider model results in tighter distribution around zero. This means that a wider model results less similar neurons, indicating that the CTM encode more information about data point in its neural dynamics when there are more neurons to work with. Figure 24c shows a similar quantity, where we measure the cosine similarity across neurons for the same data points (averaged over many different data points). In this case the wider model only results in a slightly tighter distribution.
G.2.2 The impact of longer thinking
Figure 25 exxllores the impact internal ticks on the CTM, showing (a) accuracies versus internal ticks and (b) the distributions over internal ticks where the CTM is most certain. The accuracies
34

---

64.4
66.
0.30
accurayy
64.0
0.28
0.08
Steps
Stop0-50
Sto90-100
50
1.0 30
70
90
Thought
(a) CIFAR-100 accuracies.
(b) Distribution where the CTM most certain.
Figure 25: CIFAR- 100 accuracies and internal tick analysis. The distributions and accuracies in (b) are computed for those internal ticks (x-axis) where the CTM's were the most certain (see Section 3.5 in main paper). In each case the CTM has two regions of certainty, early and later, regardless of how many internal ticks are used.
in Figure 25a close, although the CTM using 50 internal ticks the most performant. This suggests once more that with more internal ticks more training might be warranted.
The emergence of two regions of high certainty in Figure 25b is interesting as it indicates that these CTMs do indeed benefit from having more 'time to think", perhaps following two different processes internally depending on the data. Although it is difficult to say exactly why this emerges, the fact that these distributions are far from uniform indicates a more complex process than simply computing the result in strictly feed-forward nature; more analysis is required in future work.
G.2.3 Implementation details
we used a the first two hyper-blocks constrained ResNet-34 backbone (see Appendix E.I.: a downsample factor of 4x. For this experiment we varied D but kept all other hyperparameters as:
For Appendix G
k- = 8 (synapse depth, 4 layers down and 4 layers up)
dinput 512 (the width of attention output, o²)
Theads
Random pairing for neuron selection (see Appendix C.2
Dom 2048 (width of St synchronization representation)
Daction 1024 (width of Sietion synchronization representation)
nielff 32
T 50
M 25 (FIFO rolling memory input to NLMs)
duiddea 32 (width of MLPs inside NLMs)
Péropout = 0.2
No weight decay
No positional embedding
For Appendix G.2.2 we used the first two hyper-blocks of constrained ResNet-19 backhone (see Appendix E.I: p: a downsample factor of 4x. For this experiment we varied '' but kept all other hyperparameters as:
512
k 4 (synapse depth, 2 layers down and layers up)
dinpur 256 (the width of attention output, o²)
35

---

Nbaaas 4
Random pairing for neuron selection (see Appendix C.2
Dont 256 (width of St synchronization representation)
Daction 256 (width of S action synchronization representation)
Nseli
M 25 (FIFO rolling memory input to NLMs)
dnideen 16 (width of MLPs inside NLMs)
Paropout 0.0
Weight decay of 0.001
No positional embedding
This model was set up to be more constrained compared to the other CIFAR-100 ablation because of the overhead induced by using with longer ticks could benefit from more training, and this disparity would be greater should we use bigger models for this experiment.
ticks. We explained in Appendix G.2.2 that the variants trained
G.3 Neuron-Level Models and Synchronization: Ablation Analysis
Core to the design of the CTM are the NLMs and the use of synchronization as a representation. To understand how both of these components contribute to the performance of the CTM, carried out ablations using a variant of the maze task of Section with 15 x 15 sized mazes. Our ablations include four model configurations:
1. The standard CTM detailed in Section
2. A CTM without NLMs, where additional synapse model layers are introduced to maintain an equivalent number of parameters.
3. A CTM without synchronization, in which attention queries and output projections formed directly from the post-activations z'.
4. An LSTM with synchronization, where the synchronization computed using the time series of each dimension of the LSTM's hidden state.
Each configuration was parameter matched to approximately 9M parameters and trained for 100000 iterations. A full list of hyperparameters is shown Table
The training curves for each of the four variants are shown in Figure 26 with the final results summarized Table 3 We find that the two augmented variants of the CTM and the LSTM with synchronization were unable to achieve solve rate above 50%. The standard CTM, on the other hand, achieves a solve rate of 66% and a per-step accuracy 95%. These findings suggest that the combination of neuron-level models and synchronization as representation is key to the success of CTM.
Model
Num Params
dmodel
djnput
k
Dout
D Daction
Iterations
CTM
8967160
1024
256
8
1024
1024
50
CTM (No NLMs)
8948885
1045
256
10
1024
1024
50
CTM (No Synch)
8965112
1024
256
8
50
LSTM + Synch
8967454
775
256
8
1024
1024
50
Table 4: Model hyperparameters for the maze ablation experiments.
G.4 Sorting Real Numbers
In this section, we apply the CTM to the task of sorting 30 numbers drawn from the normal distribution. Sorting real numbers was a task explored by [18] when designing RNNs for adaptive compute, and provides test bed for understanding the role of compute for adaptive-compute system, such as
36

---

(TV
079 Du (0N
CT 1IM)
19900
1000
1000
(a) Test accuracy per step
(b) Solve rate
Figure 26: Ablation analysis of NLMs and synchronization. (a) The test accuracy across training steps. (b) The solve rate across training steps. The shaded areas represent one standard deviation across seeds.
Method
Test Accuracy (%)
Test Solve Rate (%)
CTM
94.6 + 0.7
65.9 + 5.7
CTM (No NLMs)
82.9 + 4.4
35,0 + 7.2
CTM (No Synch)
85.1 + 0.5
37.5 + 0.7
LSTM Synch
82.4 + 0.9
33.8 + 3.3
Table 5: Ablation results for NLMs and synchronization.
the CTM. In this case the CTM does not use attention, but rather ingests the randomly shuffled input data (30 real numbers) directly. This is implemented by replacing the attention mechanism with straightforward concatenation, replacing 10 in Figure
Can the CTM produce sequential outputs? For this experiment we set the CTM up to output sequence over its internal ticks. This is a more standard approach to modeling sequences and we wanted to understand whether the CTM could trained in this fashion. At each internal tick the CTM output vector of length 31, including 30 indices for sorting and the 'blank' token used for the well-known connectionist temporal classification (CTC) loss [46.. We then applied this CTC loss over the full output of the CTM over its internal ticks.
0.8
12.5
0.6
warcme
10.0
7.5
0.4
0.2
2
10
15
20
25
30
0.0
0.5
100
1.5
2.0
2.5
3.0
0.0
0.5
1.0
3.5
2.0
2.5
3.0
Indo In soquence
Dora delta prevtous value
Std. dev. normal datribution
(a) Mean wait times per sequence (b) Wait times index (measured ticks).
change in se- (c) Generalizing beyond the distribu- tion used for training.
quence values.
evident pattern in the average wait times, where the initial wait time (number of internal ticks) is high, goes to its lowest point, and has slightly higher bump toward the end of the sequence. In (b) various wait times, but that the difference between the previous output value and the current output value ('data delta') impacts wait time. In (c) see how this CTM scale data drawn from different normal distributions.
Figure 27: Results when sorting (0, Iso). In (a)
that the CTM employs
Figure 27 gives the results of the CTM on the sorting task. There is clear pattern to the process follows, as evidenced by correlation between wait times and both the current sequence index (a) and the difference between the previous value and the current value being output (b). A similar
37

---

task was explored by [18), who sorted 15 numbers using an adaptive compute RNN. In their case, they observed similar wait times before beginning output (analogous our first sequence element) and also near the end of the sequence. Our analysis of the relationship between wait times and the difference between current and previous data values (what we call 'data delta' in Figure 27b constitutes evidence that the CTM is using an internal algorithm that depends the layout of the data. We also show that this CTM generalizes to distributions outside of the training data.
Valt maan
-1.5
-1.0
0.5
0.0
0.5
1.0
2.0
Number
Figure 28: Sorting demonstration. The input data is represented vertical lines whose colors denote their original shuffled position (from purple through to red in the 'rainbow' colormap). The red and green bars show positive and negative deviation from the mean wait time (Figure 27a for each index in the sequence), respectively.
demonstrates the CTM's wait times in real use-case. The red bars indicate longer than average wait times for a given index, and green bars indicate shorter than average wait times. Longer wait times tend to be related to bigger gaps between data points ('data delta' in Figure 27b).
Figure 28
G.S Q&A MNIST: Memory and Arithmetic
To assess the CTM's capabilities for memory, retrieval, and arithmetic computation, we devise Question and Answering (Q&A) MNIST task, reminiscent of [55] or (56). In this task, the model sequentially observes a series MNIST digits [57). followed by an interwoven series of index and operator embeddings that determine which of the observed digits to select and which modular operation to perform over them. This allows us to probe whether the CTM can simultancously recognize hand-drawn digits, recall previous observations, and perform logical computation them without any prior knowledge of the digits depicted in the images or the relationships between them. operations at inference time than observed during training time,
Furthermore, by applying can test the generalizability of the CTM.
Specifically, the model first observes N. MNIST digits sequentially for la internal ticks each. Next, model receives an interwoven sequence Niax index embeddings (indicating which digit to select) and Nop operator embeddings (specifying either modular addition or subtraction, where each intermediate result is taken modulo 10 to keep answers within the range 0-9), each presented for tux and top internal ticks, respectively. Finally, the model observes signaling the model to produce its answer. The target, between and 9, results from the composition of all specified modular arithmetic operations. An example is shown in Figure 29
tensor for tans internal ticks,
Input
Target
Index
index
index
anzwer flag
9
(((9 + 5)%10) - 53%110)
Figure 29: Overview of the Q&A MNIST task. The model observes series of digits followed by series of index and operator embeddings, each repeated for several internal ticks. The model is then shown an answer flag and must predict the result of the modular operations.
We trained CTMs and parameter-matchen L.TTMs with two different configurations, varying how many internal ticks were used to process each input. Digits and embeddings were observed for either
38

---

1 or 10 internal ticks, with corresponding answering times of or 10 internal ticks. The number of digits and the number of operations were sampled uniformly between 1 and 4. Memory lengths for the I and 10 internal ticks per input CTMs were set to 3 and 30 steps, respectively. We highlight that with these observation and memory length configurations that the digit observations will always lie outside of the memory length-sized window during the answering stage. In this way. the CTM must organize its activations such that it can recall the digits at later time steps. The CTM is trained with the loss defined in Section 3.5 (main paper), computed only over the final las steps. Once more, we set &2 to be the final iteration for the LSTM for stable training.
G.5.1 Results
100
Iters.
80
10 Iters.
Accuracy (%)
CTM
60
LSTM
40
20
50000
100000
150000 Training Iterations
200000
250000
Figure 30: Training curves for the CTM and LSTM the Q&A MNIST task. The shaded areas represent one standard deviation across seeds. With single internal tick, the LSTM outperforms the CTM. However, the performance of the CTM increases with the number of internal ticks, while the LSTM becomes increasingly unstable.
Memory via synchronization. Training curves for three seeded runs for CTMs and parameter- matched LSTMs are shown Figure With single internal tick, the LSTM initially outperforms the CTM. As the number of internal ticks increases, the LSTM's performance degrades and learning becomes considerably more unstable, In contrast, the CTM consistently improves performance with additional thinking time. Specifically, all three seeded runs for the CTM with 10 internal ticks per input achieved over 96% accuracy on the most challenging in-distribution task (performing four operations after observing four digits). In contrast, the corresponding 10-internal tick LSTM performed at or below 21% accuracy across all seeded runs. The strong performance of the single-tick LSTMs highlights the effectiveness of the LSTM's complex gated update. however, this mechanism does not scale effectively to multiple internal steps, unlike the CTM, which effectively utilizes internal ticks to build up synchronization representation.
The CTM performs well even when the observed digits are outside of the memory window, indicating that it has learned to memorize what it has observed to some degree, purely via the organization and synchronization of neurons. The strong performance of the CTM indicates that processing timing information through the synchronization of neuron activations may be powerful mechanism for memorization and recall.
The CTM can generalize. We examine generalization by measuring the accuracy of the models when given more digits or index-operator embeddings than used during training. Figure 3T shows the accuracy of the CTM and LSTM as a function of the number of digits shown and operations to perform, with the training regime highlighted in red. We find that both the CTM and LSTM baselines can generalize to an increased number of operations. To understand how the model is capable of generalizing out of distribution, we illustrate an example thought process the CTM in Figure 32 which shows sample sequence of inputs and snapshot of the output logits. We find that the CTM sequentially computes the modular computation as the embeddings are observed, instead of waiting for the final answer flag to determine the final solution once. A similar behavior can be seen in the
39

---

100
100
80
80
Number of Operations
Number Operations
60
Accurocy (8)
60
Accuracy (%)
10
10
Number of Digits
Number of Digits
(a) CTM with internal tick.
(b) LSTM with internal tick.
100
100
Number Operations
Number Operations
Accuracy [%]
Accuracy 0%)
10
10
Number of Digits
Number of Digits
(c) CTM with 10 internal ticks.
(d) LSTM with 10 internal ticks.
Figure 31: Generalizability on the Q&A MNIST task for CTM and LSTM models with and 10 internal ticks. The x-axis denotes the number of MNIST digits input to the model, the y-axis denotes the number of operations the model must perform, and the color corresponds the test accuracy.
1-internal tick LSTM baseline. We are not claiming that the CTM can do something that the LSTM cannot, but instead that it can learn to use synchronization tool to solve this task, and that the result both effective and scales to longer task requirements.
G.5.2 Implementation details
Unlike other tasks, the Q&A MNIST task processes multiple input types: MNIST digit images, embeddings for operator and index markers, and zero tensors answer flags. MNIST images undergo preprocessing through a convolutional backbone consisting of two convolutional blocks, each containing convolutional layer, batch normalization, ReLU activation, and max pooling layer. Outputs from this backbone form attention keys and values, which the CTM queries using projections from the synchronization representation. The resulting attention outputs are concatenated with the CTM's s activated state before synaptic processing. In contrast, operator and index embeddings, as well answer flags, bypass the convolutional backbone and attention mechanism, being directly concatenated to the CTM's activated state. Operators use learned embeddings, indices utilize sinusoidal embeddings 40.. and answer flags are zero vectors matching the embedding dimension.
For comparison, parameter and internal tick matched single-layer LSTM baselines were used. The common parameters used in the experiment follows:
dmodel 1024
dinput 64
duiddca 16
40

---

8
9
Lndex O
Index
(
maxir
DEg max(()
Index 0
Index
Index
Angwer Flag
ang maxii
nrg maxd( -9
ang muaa)
ang maa
Figure 32: Example CTM thought process from the Q&A MNIST task. Shown are the inputs the model (MNIST digit, index and operator embeddings) as well as the argmax of the output logits I, at different snapshots. Each input is repeated for 10 internal ticks. In this case, the model is compute ((((((1 9) % 10) 1) 1)%10 + 8)%10 8))10). We find that the model computes each part of this composition sequentially the embeddings are observed, with outputs of 2. and finally, the correct answer of 1, projected from the synchronization representation.
k 1
Puropout 0
Theads 4
J aatioon 32
Jout = 32
Semi- dense pairing was used for selecting neurons for synchronization.
Positional encoding was not used.
Detailed hyperparameters of the CTM are provided in Table 6
Model
T
M
Repeats/Input
Answering Steps
Total Parameters
CTM
3
2,501,388
LSTM
2,507,218
CTM
10
30
10
10
3,413,772
LSTM
10
10
10
3,418,954
Table 6: Differing model hyperparameters and total parameters for the Q&A MNIST experiments.
The column Repeats/Input refers to the number of internal ticks the model used process unique input. For example, Repeats/Input= = 10 implies that 10 internal ticks are used to process each MNIST digit and each index or operator embedding. The Answering Steps refers the number of internal ticks the answering flag is observed for.
The CTM was trained using the certainty-based loss function described in section 3.5 while the LSTM baselines were trained using the cross-entropy loss at the final internal tick. We used the following settings for optimization:
Trained using a batch size 64 on H100 Nvidia GPU.
300000 iterations for training using AdamW
47
A learning rate of le-4 with a linear warmup of 500 iterations and decaying to using cosine anncaling learning rate scheduler.
G.6 Reinforcement Learning
We have previously shown that the CTM can process sequentially on non-sequential tasks via its decoupled internal recurrence. Here, we extend the CTM to sequential decision-making tasks involving interactions with external environments. Specifically, train CTMs using reinforcement
41

---

learning (RL), where the model learns action- selection policies based on environmental observations and tndal-and-rrror interactions. In this setting, the CTM processes one or more internal ticks before producing an action that transitions the environment to the next state. To achieve this, we continuously maintain the neuron dynamics across these internal ticks over successive environment steps, allowing previous environmental observations to influence current internal states via the NLMs. A central goal of this section is to provide evidence that the CTM can be set up to learn a continuous environment.
G.6.1 Implementation Details
Environments. We test the CTM on two classic control tasks and one navigation task, namely, CartPole, Acrobot and MiniGrid Four Rooms, implemented in Gymnasium [58] of these tasks are shown in Figure 33 Because the CTM maintains environment transitions, functions as a stateful recurrent neural network. Therefore, we specifically evaluate the CTM in partially observable settings, where RNNs are effective [62]. Partial observability introduced by masking the positional and angular velocity observation components the control tasks and restricting the field of view in the navigation task. This masking converts these tasks into partially observable Markov decision processes (POMDPs), requiring the CTMs to develop policies that recall past observations. For instance, in the Acrobot task, selecting the correct action depends on recalling past positions and inferring the velocity to increase arm elevation.
[59 60 61 Examples
activation history across
(a) CartPole
(b) Acrobot
(c) MiniGrid Four Rooms
Figure 33: Reinforcement learning environments used for CTM evaluation.
Architecture. The configuration of the CTM for training with PPO is as follows. First, observations are processed by a backbone, such as a feedforward network, and are concatenated to the current activated state of the CTM, without an attention mechanism, for processing over fixed number of internal ticks. After this fixed number of internal ticks, the synchronization between the output neurons is calculated and passed to the actor and critic heads for selecting the next action and estimating the state value.
Unlike the other tasks, which calculate synchronization across the entire activation history, in the RL setting we use sliding window of size memory length M. This approach prevents the buildup of very long activation histories, which may grow into the thousands for these tasks. Additionally, this allows for maintaining tensors of the same shape across all stages of the episode rollouts. To facilitate this, the CTM is initialized with both learned initial state trace and learned initial activated state trace, which are supplied to the model on the initialization of each episode. After a single forward pass of the model (corresponding to single environment step), these state traces will be maintained and provided to the model on the next environment step. This allows the CTM to process continuous history of activity, enabling activations from many environment states in the past to impact the present.
For the classic control tasks, the observation backbone is composed of two blocks containing linear layer, gated linear unit (GLU) 633 and layer normalization. similar input processing carried out for the navigation task, however, each of the object, color and state IDs are first embed in dembed 8. While for the CTM the output of the backbone concatenated to the current activated state, for the LST baseline, the output is instead processed for the same number of internal ticks as the CTM. The actor and critie heads are implemented as two-layer multilayer perceptrons (MI.ss,, each comprising two hidden layers of 64 neurons with ReLU activations. For the CTM, these heads
42

---

receive the synchronization of the output neurons as inputs, while for the LSTM baselines, they receive the hidden state of the LSTM after T' internal tick. Dense pairing was used to select neurons for synchronization.
Unlike other tasks such image classification (Section 5). which use UNet-style synapse model, the RL tasks employ a two-layer feedforward synapse, where each layer consists of a linear transformation, GLU and LayerNorm. Empirically, we found that of these layers significantly outperformed single layer, particularly in the navigation task, where a single-layer synapse consistently failed to match the LSTM's s average episode length.
The model hyperparameters used for the experiments for CartPole, Acrobot and MiniGrid Four Rooms can be found in Tables to 9
The PPO implementation is based on 664
Model
T
M
dmedel
diupuu
daidden
J Jout
Total Parameters
CTM
10
128
128
4
175437
LSTM
1
118
128
175855
CTM
2
20
128
128
4
188237
LSTM
2
126
128
188863
CTM
5
50
128
128
4
16
226637
LSTM
5
148
128
227275
Table 7: Model byperparameters for the CartPole experiments.
Model
T
M
dmodel
dinput
daidden
J out
Total Parameters
CTM
256
64
4
350094
LSTM
1
243
64
350118
CTM
2
10
256
64
4
362894
LSTM
2
249
64
364290
CTM
5
25
256
64
4
16
401294
LSTM
5
265
64
403490
Table Model hyperparameters for the Acrobot experiments.
Model
T
M
dmodel
dinppu
daidden
J out
Total Parameters
CTM
10
512
128
16
7802690
LSTM
1
294
128
7813697
CTM
2
20
512
128
16
7976770
LSTM
2
300
128
7979304
Table 9: Model hyperparameters for the MiniGrid Four Rooms experiments.
Opimization. The models were trained with Proximal Policy Optimization [65] on single H100 Nvidia GPU. The same set of PPO hyperparameters were used for both the CTM and the LSTM bascline, and are shown in Table [0
G.6.2 Results
The CTM can continuously interact with the world. Training curves for the reinforcement learning tasks are shown Figure 33 In all tasks, we find that the CTM achieves a similar performance to the LSTM baselines.
Figure 33 compares the neuron traces of the CTM and the LSTM baselines for the CartPole, Acrobot and MiniGrid Four Rooms tasks. In the classic control tasks, the activations for both the CTM and LSTM feature oscillatory behavior, corresponding to the back-and-forth movements of the cart and arm. For the navigation task, rich and complex activation pattern emerges in the CTM. The I.STM on the other hand, features a less diverse set of activations. The L.STMs trained in this section have more dynamic neural activity than what can be seen when trained CIFAR- 10 (Figure 23
43

---

Hyperparameter
CartPole
Acrobot
MiniGrid Four Rooms
Learning Rate (LR)
1 x 10-3
5 x 10-1
x 10-4
Total Environment Steps
10M
2M
300M
Rollout Length
50
100
50
Number of Environments
256
12
256
Max Environment Steps per Episode
200
500
300
Update Epochs
4
1
Minibatches
4
4
Discount Factor (γ)
0.99
0.99
0.99
GAE Lambda (A)
0.95
0,95
0.95
Clip Coefficient
0.1
0.1
0.1
Entropy Coefficient
0.1
0.1
0.1
Value Function Coefficient
0.25
0.25
0.25
Value Function Clipping
No
No
No
Max Gradient Norm
0.5
0.5
0.5
Table 10: PPO hyperparameters for each task.
CTM,
500
LSTM, 1 iters.
Epiondee Lergth
150
400
CTM. Itors.
CTM, Iters.
LSTM,
Episode
300
CTM, Iters.
1.00
CTM, Iters
200
LSTM, 5 Iters.
LSMM, Iters.
LSIM, Iters.
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
Steps
Environment
1e6
(a) Cartpole
(b) Acrobot
300
LSTM, Iters.
200
L.TM,
150
50
1.5
2.5
Ervironment Steps
(c) MiniGrid 4-Rooms
Figure 34: Training curves for reinforcement learning tasks. Each curve depicts a moving average of the episode length during training, averaged over three training runs. The shaded region represents one standard deviation across seeds. For Cartpole, higher is better. For Acrobot and MiniGrid 4-rooms, lower is better.
This is likely due to the sequential nature of RL tasks, where the input to the model changes over time owing its interaction with the environment, inducing a feedback loop that results in the model's latent representation also evolving over time.
H Recursive computation of the synchronization matrix
In Section 3.4 we defined the synchronization matrix internal tick as
=
Z''TT, Z E Bg Dxt
(12)
44

---

CTM, CartPole, 2 Internal Ticks
(b) LSTM, CartPole, 2 Internal Ticks
NIMAA w www
(c) CTM, Acrobot, 1 Internal Tick
(d) LSTM, Acrobot, Internal Tick
(e) CTM, MiniGrid Four Rooms,
Internal Ticks
(1) LSTM. MiniGrid Four Rooms, Internal Ticks
Figure 35: Neural activities over the course of a single episode for the CTM and LSTM on CartPole, Acrobot and MiniGrid Four Rooms tasks. The CTM features richer neuron dynamics than the LSTM.
where the d-th row of Z' stores the post-activation trace of neuron d up to tick / (cf. Eq. Because Eq. [12) recomputes all D2 inner products from scratch at every tick, its time complexity O(D()) over a roll-out of length I. Below we show that, with the exponentially-deeays of Eq. (TO), the same quantity can be obtained from pair of first-order recursions that require only ((Duub) work per tick, where Daub < D is the number of subsampled neuron indices actually used for the output and action projections.
rescaling
For notational clarity we first consider a single (i,j) neuron pair and omit the subsampling; the extension to batch of pairs is immediate. Recall that the rescaled synchronization entry is defined as
ro (t-r) 23
r=1
(13)
(t-T))
where riy 2 0 is the learnable decay rate for the pair (i. Define the following auxiliary sequences
a
r.y (t-7) 27 23,
at 2//2..
(14)
Bly Σ (t)(t=))
B/ = 1.
(15)
Then St, at// / and both at, and B obey simple first-order difference equations:
a 1+1 = -rg at + 2 ++1 (+1
2)
(16)
at+1 = e Tu 31 +1
(17)
45

---

The rank-1 1 update in Eq. (T6 makes it unnecessary to store the full activation history or to repeatedly form large outer products. During forward simulation we maintain dly and Big for each selected pair and update them in (1) time.
In practice we store {aiz- figt only for the two disjoint subsamples that form Saa and S action (Section 3.4]. Both memory footprint and compute overhead therefore scale linearly with the number of retained pairs, i.e. (Dauh) = ( + Daction) per tick.
I Emergent phenomena
We found that there were several emergent behaviors during learning and when the CTM was put under strong constraints (c.g., limited internal ticks), and we encourage the reader to train their own CTMs (code included in supplementary material) in order to make the same observations. These emergent behaviors include:
1. At initialization the neural dynamics are poor and the neurons do not exhibit any periodic nature. The periodic nature of the dynamics (as have shown in this paper) only emerge as training progresses, and the dynamics become richer, more complex and diverse, and periodic training continues. One can apply layer-norm pre-activation histories (inputs to NLMss in order to drive more periodic behavior at the outset, but the performance is generally worse (there is an option for this in the code).
2. Regarding neural dynamics, a useful emergent property of the CTM architecture is that visualizing neural dynamics enables understanding whether there are any 'dead' neurons (there is actually 1 of these in Figure 2a evidenced by having extremely little variation across data instances). This enables new perspective regarding the utilization of weights in a NNN and we hope that future work can leverage this emergent utility.
3. For the maze task we found that relatively early on in training the CTM would exhibit a 'double take' approach where it would rapidly and approximately solve the maze (as shown by the attention progression) only to start again and do so more slowly. We think that this is a naturally bootstrapping technique as disappears as the CTM gets better at solving the maze (it is wasted compute, after all).
4. Sometimes the CTM begins down the incorrect path for the maze task but soon changes its mind and follows the correct path. This is a consequence of the freedom afforded by our loss function. Supplementary video 'maze-change.mpp mp44 demonstrates an instance of this.
5. Upon careful inspection the attention heads for the maze task we noticed that some heads were dedicated to a broader, more global perspective of the maze, while other heads were dedicated to following the path from start to end points. A similar behavior emerged in the parity task, where some heads would be dedicated to finding positive (or negative) values (see supplementary video "parity.mp4".
6. On ImageNet-IK IK we found that the degree to which the CTM looks around' also increases with training time, becoming more diverse.
7. Also on ImageNet-1K IK we observe some evidence that the CTM tends to shift between broad perspectives (spread out attention) to narrow views (tight attention) as it thinks.
8. Also on ImageNet- K, even without positional embeddings the CTM could learn to follow directional regions over real images in much the same way as it did for mazes (see supplementary video 'imagenet.mp4".
9. Under heavy constraints, such as when using limited internal ticks (c.g., 50) and requiring long solutions (e.g., 150 200 steps in a maze; we showed only 100 steps in this paper), we found that the CTM would implement unusual or alternative strategies. These strategies would still remain interpretable but could be more computationally efficient. We include a supplementary video satee-bbcwwad.s.ee mpp4" demonstrating an interesting maze solution where a constrained CTM ended up learning a highly-performant strategy that looked ahead by small chunks before then tracing backwards, jumping forwards again, and repeating until the maze was solved. We find this behavior fascinating and evident of what a machine could do that might be strange to humans, yet still understandable.
46

---

10. CTMs that can perfectly solve the parity task consistently learned two types of strategies: either attending to the data from the beginning end, or reverse order. We show two examples in supplementary videos 'parity-ateenion- prgiienioonnorwwrd.mpm and 'parity-attentoo- backwards. which highlight how these behaviors emerge during training.
11. In the Q&A MNIST task (Appendix mediate modular result after each answering flag to produce the final output. Although LSTMs can display this behavior with a single internal tick, they struggle to maintain it as the number of internal ticks grow.
G.5 found that the CTM would emit the inter- tuple, rather than waiting for the
47


Reviews:
[{"review_id": "NSvDIohIvD", "reviewer": "Reviewer_iujh", "summary": "This paper introduces Continuous Thought Machines (CTMs), a novel neural network architecture that emphasizes temporal processing at the single-neuron level and population-level synchronization as core computational principles. CTMs replace standard pointwise nonlinearities with neuron-level models that process a history of pre-activations, aiming to better capture the complexity of biological neural dynamics. Furthermore, CTMs use neural synchronization (defined through correlations in neuron activity across internal “ticks”) as a central representational mechanism for attention and downstream readout. The authors propose that this architecture offers a step toward bridging the gap between biological and artificial intelligence, and present a diverse suite of experiments across different domains, focusing on the qualitative behavior and flexibility of CTMs rather than competitive benchmark performance.", "questions": "1. Synchronization as representation: You emphasize synchronization as a key latent representation for attention and output. Could you clarify what precisely is being represented here? Is “synchronization” equivalent to temporal correlation? If not, how does it differ?\n2. Decaying influence kernel: In the computation of synchronization, earlier ticks appear to have greater influence due to the exponential decay term $\\exp(-r_{ij}(t - \\tau))$. This seems counter-intuitive. Can you clarify or justify this design choice?\n3. Biological plausibility: Do real neurons retain a history of their pre-activations in a way that could support a mechanism like your neuron-level models? If not, how do you justify this as a biologically inspired component?\n4. Subsampling neuron pairs: How sensitive is the model to the random sampling of neuron pairs for computing $\\mathbf{S}^t$? How many samples are sufficient, and do you resample at every tick?\n5. Ablations: It would be very helpful to see ablation studies isolating the contributions of (a) the neuron-level models and (b) the synchronization mechanism. Could you explore these separately?\n6. \"Looking Around\" Rationale: While an interesting emergent property, what are the specific desired computational benefits or motivations for the CTM to \"look around\" images during classification, especially for static inputs?\n\nScore could increase if the authors:\n\n- Clarify terminology and figures,\n- Provide ablations to better understand the architecture,\n- Offer more rigorous comparisons or parameter-controlled baselines, highlighting the trade-offs of CTM vs SOTA models.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### **Strengths**\n\n- Conceptual originality: The paper proposes a novel architectural shift, where both temporal integration and inter-neuronal synchronization are fundamental to how representations are formed. This is a creative departure from conventional feedforward or RNN-based models.\n- Biological inspiration: The authors ground their design choices in neuroscience, particularly in emphasizing the importance of temporal structure and population dynamics, which are often abstracted away in standard deep learning.\n- Breadth of experimentation: The authors test CTMs across a range of tasks, demonstrating versatility, even if not aiming for SOTA performance.\n- Modularity: The design of neuron-level models, decaying influence kernels, and synchronization mechanisms is well-structured, suggesting future extensibility.\n\n### **Weaknesses**\n\n- Clarity issues:\n    - Some key figures (e.g., Figures 1 and 2) are introduced abruptly and not clearly contextualized in the main text. What does “teleporting” mean in Figure 1?\n    - Figure 3 is convoluted and could be made clearer.\n    - Terms like “neural synchronization as representation\" are used repeatedly but never precisely defined, why use the term “synchronization” instead of “temporal correlation”?\n- Quantitative rigor: There is a lack of controlled comparisons in the main text with baseline models on shared tasks. It is unclear whether CTMs offer any advantage under equivalent parameter or FLOP budgets.\n- Biological grounding: While motivated by neuroscience, some modeling choices (e.g., neurons retaining a full history of pre-activations) lack biological justification. It is unclear whether such memory mechanisms exist at the single-neuron level in biological systems.\n- Limited ablation: No ablation experiments are presented to isolate the importance of neuron-level models versus the synchronization mechanism.", "was_revised": true, "final_justification": "This is a technically solid paper and the authors have adequately addressed my remaining questions and concerns in their rebuttal. Therefore, I believe it should be accepted, and I am maintaining my score of 5.", "rebuttal": "Thank you for your in-depth review and comments. It is clear that you unpacked and understood both the motivation and efforts of our work.\n\n---\n\n## Regarding weaknesses\n### Clarity issues.\n1. We have removed the word ‘teleport’. Its use was to capture the process of re-applying the same model sequentially by shifting the start location to the predicted (valid) end location (determined by following the steps predicted by the model). We propose replacing the end of the caption with:\n> ‘... (c) generalizing to 99 x 99 via sequential re-applications of the same model.’\n\n2. Re Figures 1 and 2.  \n**We will contextualize those figures** in the second contribution, providing pointers appropriately. We apologize for this oversight.\n\n3. Re Figure 3.  \nWe appreciate this feedback. Unfortunately, visualizing a time-based architecture in a static figure is inherently difficult. This is why we created the supplementary animated video ‘arch.mp4’. In our revision, **we will explicitly reference this video** in the caption, framing it as an essential companion for understanding the model's operational flow.\n\n4. Re the use of the word “**synchronization**” instead of “**temporal correlation**”.  \nWe chose “synchronization” because it describes the emergent, coordinated functional behavior we aim to model, a biologically-inspired concept central to population coding in neuroscience. “Temporal correlation”, on the other hand, is the underlying *mathematical mechanism* we use to implement it. To make this point clear, we propose the following change when first introducing the term (in the contributions):\n> “...the use of neural synchronization directly as the representation (implemented via temporal correlations between neuron-level activity; Section 3.4)…”\n\n5. Re terminology, particularly referring to “**synchronization as a representation**”.  \nWe explicitly define the meaning of synchronization as a representation using equations in Section 3.4. We now see the need for a slightly more exhaustive descriptive introduction - the above response addresses this.\n---\n### Regarding quantitative rigor:\nThank you for pushing for greater rigor. To that end, **we have taken two direct actions**:\n1.  We have run ablations (details to follow).\n2.  We will clarify that our baselines **are already** parameter-matched where applicable.\n\nWith that said, we would also like to share the perspective that guided our experimental design. A traditional approach is to immediately position a new idea against SOTA benchmarks. However, we are cognizant that today's leading models are the product of years of compounding engineering gains and fine-tuning on those specific tasks. Given this, we believe that for a fundamentally new architecture like the CTM, a premature focus on SOTA performance would distract from this paper's core scientific contribution: understanding *what* the CTM does and *how* its temporal mechanisms operate.\n\nOur primary goal, therefore, was to establish the *what* and *how* of the CTM's unique operation first, providing a stronger foundation for the community. This required designing novel experiments to specifically probe its capabilities. For example, finding the right “hello world” task was a challenge in itself; our phrasing of the maze task - which requires forming an internal world model by disallowing positional embeddings and outputting a direct action sequence - was a deliberate choice to truly showcase the nuanced differences between the CTM and recurrent baselines like LSTMs. We believe this setup reveals more about our architecture's potential for complex reasoning than a raw performance number on a standard classification benchmark might.\n\nWe understand your desire for more rigorous quantitative metrics and hope this response clarifies our rationale and strengthens our contribution.\n\n---\n### Regarding biological grounding:\nYou have raised an important point about the level of abstraction. Our approach is one of **biological inspiration, not simulation**. The NLM is a functional abstraction of the complex, time-dependent integration that occurs in biological neurons, rather than a literal model of a specific subcellular mechanism. We use a truncated FIFO history (as noted in line 116) as a computationally tractable way to provide this temporal context. We propose the following change to the first paragraph of line 271:\n> “The CTM's NLMs are inspired by the complexity of biological neurons, but are implemented with a level of abstraction appropriate for modern deep learning.”\n\n---\n### Regarding limited ablations:\nWe have run an ablation study on a 15x15 maze task, using 100K training steps and parameter-matched (+-9M) models that we will add in a new appendix. The results are as follows:\n\n| Method | Test Accuracy (%) | Test Solve Rate (%) |\n| :--- | :---: | :---: |\n| CTM | 94.6 $\\pm$ 0.7 | 65.9 $\\pm$ 5.7 |\n| CTM (No NLMs) | 82.9 $\\pm$ 4.4 | 35.0 $\\pm$ 7.2 |\n| CTM (No Synch) | 85.1 $\\pm$ 0.5 | 37.5 $\\pm$ 0.7 |\n| LSTM + Synch | 82.4 $\\pm$ 0.9 | 33.8 $\\pm$ 3.3 |\n---\n## Regarding questions\n### Synchronization representation.\nThank you for asking for this clarification. Section 3.4 defines synchronization mathematically as the temporally-decayed correlation of neuron activity histories. This representation is crucial for three reasons:\n1.  **Decoupling**: It decouples the representation used for downstream tasks from the raw neuron states ($z_t$), allowing the neurons to develop richer internal dynamics without being solely constrained by the immediate prediction loss.\n2.  **Hebbian Inspiration**: It captures the idea of “neurons that fire together, wire together.” By making the correlation of activity the representation, we directly model the interplay within the neural population over time.\n3.  **Multi-scale Temporal Sensitivity**: The learnable decay parameter ($r_{ij}$) for each neuron pair allows the model to learn a representation that is sensitive to phenomena at different timescales simultaneously, which is critical for complex sequential reasoning.\n\nOwing to the lack of space in the main paper, expanding on these points in the main text is not possible. That being said, we can potentially write a new appendix that goes into more detail about the nature of the synchronization representation, should the reviewer request that.\n\n---\n### Decaying influence kernel.\nThank you for this question. The decaying influence kernel gives more weight to recent ticks. As defined in Equation 9, the influence of a past tick, $\\tau$, on the current tick, $t$, is scaled by $\\exp(-r_{ij}(t-\\tau))$. Because the difference $(t-\\tau)$ is large for distant ticks and small for recent ones, the negative exponent correctly implements an exponential decay of influence over time.\n\nFor example, for $t=3$, the numerator of Equation 10 expands to $z_i(1)z_j(1)\\exp(-2r) + z_i(2)z_j(2)\\exp(-r) + z_i(3)z_j(3)\\exp(0)$ which correctly accounts for a larger influence of the current tick ($z_i(3)z_j(3)$) compared to the previous ticks.\n\n---\n### Biological plausibility.\nAs discussed in our response to the weaknesses, our goal is **biological inspiration, not direct simulation**. The CTM’s NLMs are an abstraction that enables learning more complexity on a neuron-level, as opposed to an attempt at emulation. Please see the clarification and proposed adjustment to the text (in the weaknesses section, above).\n\n---\n### Subsampling neuron pairs.\n* **Sensitivity**: The model is not highly sensitive to the specific random sample of pairs, provided the number of pairs ($D_{out}$, $D_{action}$) is sufficiently large for the task's complexity. For our experiments, we generally chose these values to be on the same order of magnitude as the model's width or the task's output dimensionality.\n* **Resampling**: **The pairs are sampled once at initialization and then fixed.** Resampling at each tick is not feasible, as the projection weights ($W_{out}$, $W_{in}$) are learned specifically for the chosen pairs. **We will clarify this in Section 3.4.1.**\n\n---\n### Ablations.\nAs mentioned in our response to weaknesses, we have now completed an ablation study that shows NLMs and synchronization are both beneficial. \n\n---\n### “Looking around” rationale.\nThis is a fantastic question about the functional role of this emergent behavior. When we built the CTM there was never any explicit objective for “looking around”, so when we observed this behaviour we were struck by its parallels to how we expect an intelligent agent to observe a scene. Thus, we believe its emergence points to both a compelling parallel with observable intelligence and a tangible computational benefit.\n\nFrom a conceptual standpoint, we often identify intelligence functionally through action. A crow solving a puzzle is deemed intelligent because it undertakes actions to achieve a goal. Similarly, the CTM's emergent “gaze” behaviour is an active process of information gathering. That this behavior arises without any direct supervision is a fascinating result because it parallels an intuitive aspect of how intelligent agents interact with their environment.\n\nMore crucially, from a computational perspective, this “looking around” process is the mechanism that enables **native adaptive computation**. By iteratively attending to different parts of the image over its internal ticks, the CTM can refine its understanding and adjust its processing time based on the input's complexity. As shown in Figure 5a, this allows it to spend more “thought” (more internal ticks) on difficult images compared to simpler ones. This input-dependent allocation of compute is a direct, beneficial consequence of the architecture's temporal design.\n\nWe have added clarification to Section 5.2. as follows:\n> “...It does this entirely without prompting or any guide, implementing computationally beneficial adaptive compute in an intuitive fashion…”\n\n---\nThank you again for your thorough review. Please feel free to ask more questions."}, {"review_id": "RRpJrxmqZa", "reviewer": "Reviewer_EBTG", "summary": "This paper introduces the Continuous Thought Machine (CTM), a novel artificial neural network model designed to take into account the neural timing and interactions for decision making. The proposed model uses neuron-level processing and synchronization as foundational elements. The proposed method has been tested on several datasets (2D Maze navigation, Imagenet 1K classification, cumulative parity tasks, cifar 10 etc. ) and shows promising results.", "questions": "Do the neuron level models used in CTM increase the parameters compared to the conventional ANN’s? \n\nFrom Figure 6 (Page 8), CTM's training seems sensitive to random seed, initialization. For internal tricks varying from 75 to 100, the variance is comparatively higher.\n\nHave the authors experimented on trying biological neuronal models such as Hindmarsh and rose neuronal model, Hodgkin and Huxley neuronal model etc.? Any specific reason on why biological neuronal models are not used? \n\nThe longer training time seems to go against biological plausibility. Would like to know the response of the authors.", "limitations": "I would like the authors to respond to the questions raised.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n\nIntroducing neural timings and synchronization as the core principle is a novel idea and moves closer to biologically inspired learning algorithms. CTM has been tested on wide datasets and shows promising results. The paper provides extensive details on architecture, hyperparameters, datasets, and optimization settings in the appendices. \n\nWeakness\n\nFrom Figure 6 (Page 8), CTM's training seems sensitive to random seed, initialization. For internal tricks varying from 75 to 100, the variance is comparatively higher.\n\nThe longer training time seems to go against biological plausibility. Would like to know the response of the authors.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your thoughtful feedback and are encouraged by the positive assessment of our work. The questions raised are insightful and touch upon some of our core design principles. We hope that addressing these points will strengthen the paper.\n\n## Weaknesses and key questions\n\nYou have highlighted two primary concerns: (1) the CTM's sensitivity to initialization in the parity task and (2) the apparent tension between long training times and biological plausibility. We will address these crucial points first.\n\n### 1. Training stability/sensitivity to seed\nYou are correct that different random seeds can lead to varied final performance. We discuss and analyze this in Appendix F.6.\n\nCrucially, we argue that this variance is not owing to arbitrary instability but rather a direct and fascinating consequence of the CTM discovering **qualitatively different algorithmic strategies** to solve the task. Our analysis in Figure 20 shows that different runs converge to distinct, interpretable algorithms:\n* One successful run learns a **forward-pass strategy**, scanning the sequence from start to end.\n* Another successful run learns a **reverse-pass strategy**, attending to the sequence from end to start.\n* Sub-optimal runs often learn incomplete or less efficient hybrid strategies.\n\nThis emergent property actually highlights a **strength of the CTM**: its representational space, built on neural synchronization, is rich enough to enable the discovery of distinct computational solutions. While this can lead to variance in performance depending on the “quality” of the discovered algorithm, it also makes the CTM a powerful tool for investigating *how* models learn to solve procedural tasks, not just *if* they can. The interpretability benefits of the CTM are noteworthy in this case. In the future, we hope to develop techniques for driving towards and/or selecting from the distinct computational solutions the CTM can learn.\n\nTo ensure this point is not missed, **we will add a pointer to the main text in Section 6.1** that explicitly references the detailed analysis of these emergent strategies in Appendix F.6.\n\n---\n\n### 2. Biological plausibility and training times\nYou have made an insightful, and also deeply philosophical point. The comparison between training in deep learning and learning in biology begets a careful consideration of timescales. Consider the following:\n* **Evolutionary and developmental “training”**: The priming of biological brains has occurred over millions of years of evolution, shaping architecture, learning rules, etc. This is followed by years of childhood learning. Taken in totality, such a protracted process is somewhat analogous to the use of millions of gradient descent iterations to train a model’s internal machinery from scratch.\n* **In-situ, rapid learning**: The remarkable few-shot or one-shot learning that humans exhibit (e.g., learning a new object class) is possible only because it builds upon this vast foundation of prior “training”.\n\nOur work aims to capture some functional dynamics of an **“already trained” brain**, rather than simulating its entire evolutionary and developmental trajectory, hence the long training times. Furthermore, a key feature of the CTM is its **native adaptive compute** (Section 5.1). This principle directly aligns with that of biological efficiency at inference time. The CTM naturally learns to “think longer” for challenging inputs, which is to say biologically-plausible ‘rapid’ timescale intelligent resource allocation.\n\n---\n\n### 3. Other questions\n\n### Re: NLM parameter count\nYes, they do. We acknowledge this trade-off directly in our limitations paragraph (Section 8, line 277).\n\nWe view this not as a simple drawback, but as a justified design choice that provides a new axis for model scaling and expressivity. **Per-neuron MLPs** is what enables each neuron to learn a unique, complex internal model, which in turn is fundamental to generating the neural dynamics at the core of the CTM.\n\n### Re: Neuron model types\nThis is an excellent query of our modeling philosophy. Our approach is one of **biological inspiration, not emulation**. We briefly discuss this choice in the conclusion on line 271, and will expand on this as per another reviewer's request.\n\nModels like that of Hodgkin-Huxley are architecturally more realistic representations of the underlying functionality of neurons. These types of neuron models are described by complex, non-linear differential equations. Integrating these models within large-scale, gradient-based deep learning frameworks is, with current methods, computationally prohibitive and often non-differentiable. Taking this approach would have drastically limited the types of experiments we could undertake and would most likely not have yielded the observations we share in this paper.\n\nInstead, we chose to abstract the functional role of a neuron in a dynamic system: to produce a complex temporal output based on its history of inputs. A simple MLP enables the CTM to learn the optimal dynamic response for the task at hand. Crucially, this choice ensures that learning remains tractable, end-to-end differentiable, and compatible with current deep learning paradigms. Our intentional choice in this matter is what allows us to bridge principles (not emulations) from neuroscience with findings and techniques of modern AI.\n\n---\n\n## Additional note: Ablations\nIn response to other reviewers we have performed an ablation study to elucidate the impact of NLMs and synchronization. We ran this ablation study on a 15x15 maze task, using 100K training steps and parameter-matched (+-9M) models that we will add in a new appendix:\n\n| Method | Test Accuracy (%) | Test Solve Rate (%) |\n| :--- | :---: | :---: |\n| CTM | 94.6 $\\pm$ 0.7 | 65.9 $\\pm$ 5.7 |\n| CTM (No NLMs) | 82.9 $\\pm$ 4.4 | 35.0 $\\pm$ 7.2 |\n| CTM (No Synch) | 85.1 $\\pm$ 0.5 | 37.5 $\\pm$ 0.7 |\n| LSTM + Synch | 82.4 $\\pm$ 0.9 | 33.8 $\\pm$ 3.3 |\n\n---\nWe hope these responses have clarified our design choices and the novel implications of our work. Please feel free to ask any additional questions or propose changes. Thank you."}, {"review_id": "k9nrrJiVHN", "reviewer": "Reviewer_tk25", "summary": "This paper introduces a deep neural network architecture which uses timing and synchrony between neuronal units to compute. The architecture is end-to-end differentiable, and the authors demonstrate that it can be optimized to perform interesting and computationally challenging tasks, such as 2D mazes and ImageNet 1K classification. The authors also provide interactive examples of their network performing these tasks.", "questions": "## Question on Novelty and Originality\n\nMy main question concerns the novelty and originality of the work. The authors claim multiple times that they are the first to incorporate neural synchronization within a deep learning context. For example:\n\n- **Lines 46–47**:  \n  *\"This use of neural synchronization yields a fundamentally new type of biologically-motivated latent representation...\"*\n\n- **Line 65**:  \n  *\"The CTM introduces neural timing and synchronization as core computational principles.\"*\n\nHowever, this appears to overlook previous work that has explicitly explored similar ideas, most notably:\n\n- **[Reichert & Serre (2014)](https://arxiv.org/abs/1312.6115)**  \n  *Neuronal synchrony in complex-valued deep networks*  \n  This work directly addresses the use of neural synchrony in deep networks.\n\n- **[Lee et al. (2022)](https://ieeexplore-ieee-org.libproxy.mit.edu/stamp/stamp.jsp?tp=&arnumber=9849162)**  \n  *Complex-valued neural networks: A comprehensive survey*  \n  This survey covers numerous architectures that utilize neural timing and synchronization as core computational principles.\n\nGiven the existence of this prior work, was the omission of this literature an honest oversight, or was there a specific reason for its exclusion?  If the latter, could the authors clarify how their approach differs from or advances beyond these previous efforts?\n\nThe authors also claim, in the abstract, that \"Most artificial neural networks simplify neurons by abstracting away dynamics\". I am not sure what this means--can the authors clarify? Do they mean abstracting away the complex biophysics dynamics of single-neuron Hodgkin-Huxley models? Because taken at face value, their sentence is certainly false: many hundreds of papers are written every year which use artificial neural networks with dynamical neurons. This has been going on for many years in neuroscience, for example see [Computation Through Neural Population Dynamics – Yas S, Golub MD, Sussillo D, Shenoy KV. *Annual Review of Neuroscience*, 2020.](https://doi.org/10.1146/annurev-neuro-092619-094115)\n review from five years ago. \n\n\n**To be clear, I believe this could be a strong and valuable paper, provided the inaccurate novelty claims are removed and the related work section is substantially expanded to more accurately contextualize the contribution within existing literature. I would consider increasing my score if the authors were to address my concerns here.**", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 1, "strengths_and_weaknesses": "# Strengths:\n\nThe paper is well-written, and the technical contributions are clearly described. Also, the authors go above and beyond by providing interactive examples of their network solving a 2D maze task, which this reviewer found charming and fun to play with. \n\n# Weakness:\n\nUnfortunately, I have major concerns about the originality of this work. Although the authors repeatedly claim that they are the first to use neural synchronization in a deep learning context, this is simply untrue (details below in the \"Questions\" section).", "was_revised": true, "final_justification": "The authors clearly responded to all my points.", "rebuttal": "Thank you for your insightful and directed review. We appreciate the important points that you have raised regarding novelty and prior work.\n\n## Re: Clarifying novelty\nWe agree that our novelty claim, regarding synchronization, was **insufficiently posited within these prior works**. This lack was an honest oversight, but the specific details of what makes our use of synchronization novel still stand. Specifically, and crucially, we use synchronization as **learnable representations**. These representations are used downstream via weighted projections (e.g., for attention queries and class predictions, although many such representations can be extracted from the singular dynamical system that the CTM defines).\n\n**Reichert & Serre (2014)** introduced a model where synchronization emerges from the interaction of complex-valued neurons, where the mechanism is the phased interaction of complex weighted-sums of inputs, and the role of their synchronization is as a **modulator of information flow**, effectively binding neurons in a form of gating. They make use of these emerging ‘bound’ neuron groups for tasks such as object segmentation, using a post-hoc processing step to use the synchronized neurons. While interesting and useful, this is quite distinct from our direct use of synchronization as a latent representation.\n\nIn the context of complex-valued neural networks, there are a variety of different architectures that utilize neural timing and synchronization (Lee et al. 2022). These use “synchronization” in the traditional control and dynamical systems sense: they develop controllers or conditions to make complex-valued neural network models synchronize with each other or reach stable synchronized states. By contrast, the CTM uses neural synchronization in a **computation sense**: it treats the synchronization of neuron activities as a feature representation within a neural network model. There is no external “controller” in CTM enforcing synchronization; instead, CTM’s training optimizes how neurons’ activation patterns synchronize so that this synchrony encodes useful information for tasks. Essentially, CTM repurposes the concept of synchrony as a tool for thinking, whereas the other works treat synchrony as a behavior to be achieved for system stability or control objectives. We believe this makes the CTM fundamentally different from those prior synchronization studies in how the concept is applied.\n\n**Actions:**\n* We will completely clarify that our novel use of synchronization is as a latent, learnable, and internally constructed representation that the model builds and uses downstream. To that end, we propose to rephrase the paragraph starting on line 46 as:\n    > “The CTM learns to use neural synchronization as its latent representation, distinguishing it from existing work that explores synchrony as emergent properties for post-hoc use [citations, including Reichert and Serre]. This representation is distinct from the common static `snapshot' representations used in most modern NNs as it directly encodes the temporal interplay of neural dynamics.”\n* Regarding line 65: this was perhaps a poor phrasing of what we intended to say, as we did not mean to claim that the CTM is the introduction to the use of neural timing, but rather that it uses timing and synchronization as computational tools for its learning. Thank you for pointing that out. We will also rephrase Line 65 to read as:\n    > “The CTM uses neural timing and synchronization as core computational principles...”\n* We propose to include the following paragraph (subject to minor tweaks due to space constraints) in the related work section, dedicated to earlier explorations in synchrony (please feel free to offer critique and/or make requests):\n    > “Reichert & Serre (2014) proposed a model where synchronization emerges from interactions among complex-valued neurons, serving as a gating mechanism that modulates information flow and enables post-hoc grouping of neurons for tasks like object segmentation. Unlike CTM, however, their model does not use synchrony as a learned latent representation during computation. Other approaches in complex-valued neural networks (Lee et al., 2022) employ synchronization from a control-theoretic perspective, aiming to stabilize or coordinate networks via externally enforced synchrony. In contrast, CTM integrates synchronization intrinsically, optimizing neural phase relationships during training to encode task-relevant representations. This positions CTM as a computationally grounded model of synchrony, fundamentally distinct from prior works that treat synchrony as a control objective.”\n\n---\n\n## Re: abstracting away dynamics\nThank you for pointing out this important distinction. We do, indeed, mean single-neuron models, hence the introduction of NLMs, enabling a more complex (learnable) neuron model. We propose to clarify this in the abstract, replacing the sentence you pointed out with the following:\n> “Most artificial neural networks ignore the complexity of individual neurons.”\n\n---\n## Additional note: Ablations\nIn response to other reviewers we have performed an ablation study to elucidate the impact of NLMs and synchronization. We ran this ablation study on a 15x15 maze task, using 100K training steps and parameter-matched (+-9M) models that we will add in a new appendix:\n\n| Method | Test Accuracy (%) | Test Solve Rate (%) |\n| :--- | :---: | :---: |\n| CTM | 94.6 $\\pm$ 0.7 | 65.9 $\\pm$ 5.7 |\n| CTM (No NLMs) | 82.9 $\\pm$ 4.4 | 35.0 $\\pm$ 7.2 |\n| CTM (No Synch) | 85.1 $\\pm$ 0.5 | 37.5 $\\pm$ 0.7 |\n| LSTM + Synch | 82.4 $\\pm$ 0.9 | 33.8 $\\pm$ 3.3 |\n---\nWe hope that this sufficiently clarifies our claims and better positions our paper in the full breadth of prior art. Thank you for your excellent suggestions and nuanced questions. Please let us know if we can do anything more, or if you have any further questions."}, {"review_id": "VzGKCDLLDw", "reviewer": "Reviewer_Fj9J", "summary": "This paper proposes a new model for deep learning inspired by two important (and underexplored) ideas in neuroscience: (i) neuron-specific dynamics, and (ii) across-neuron synchronization. The resulting model is essentially a recurrent neural network, where each \"neuron\" is parameterized by a complex neural network, namely one feedforward network to compute its current potential as a function of its neighbors' activity, and a second recurrent network to compute its output as a function of its historical potential (this captures idea (i)). Property (ii) arises in that the model's actual output is (a projection of) the similarity matrix of the trajectories of its neurons -- in other words, their synchronization. The model is designed to adapt its computational time, particularly through a loss function that incentivizes it to simply make its prediction its most confident output, regardless of timestep. The model is evaluated through a diverse variety of tasks with both sequential and non-sequential character, including image classification and solving mazes.", "questions": "Q1. Of course, it would be very interesting to see a language experiment in a model like this.\nQ2. Why is it necessary to include both the loss on the best tick and the loss on the most confident tick in the loss function? Intuitively, it seems like only the second term is important.\nQ3. The \"Emergent Phenomena\" section of the appendix is interesting and it would be nice to see a few of these observations empirically fleshed out in the main body.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nS1. The writing and presentation throughout are excellent. The model is fairly complex so this is very important. The supplementary website with animations is a nice bonus.\nS2. The implementation of synchronization is original, creative, and thought-compelling.\nS3. The experiments are diverse, described thoroughly, and seem to make a compelling case for the model's effectiveness.\n\nWeaknesses\nW1. It's difficult to tell the how important the different components of this model are to its effectiveness. It's also, frankly, a bit unintuitive to reason about. Is synchronization helpful to the model, or is it implicitly regularizing? More experimental probing of this would be helpful, especially ablation studies.", "was_revised": true, "final_justification": "I believe this is an insightful and strong paper which will introduce a few interesting ideas to the community. I'm doubtful that any of those ideas will be transformative but they are well worth sharing. I have no concerns with the rigor with which the paper's claims were evaluated. Another reviewer raised an issue over the authors' novelty claim regarding synchronization -- however, the authors immediately recognized the issue and were willing to adjust the language. The claim was overblown but the method remains novel. In sum, this paper is a clear 5, accept for me.", "rebuttal": "Thank you for your insightful review. Your summary, in particular, was helpful in understanding whether our work was sufficiently well communicated.\n\n## Re: difficulty in reasoning about the model owing to its moving parts.\nThis is a very useful comment, thank you. To this end we ran an ablation study on a simplified maze task in order to elucidate any performance differences when ablating NLMs and synchronization. We ran this ablation study on a 15x15 maze task, using 100K training steps and parameter-matched (+-9M) models that we will add in a new appendix:\n\n| Method | Test Accuracy (%) | Test Solve Rate (%) |\n| :--- | :---: | :---: |\n| CTM | 94.6 $\\pm$ 0.7 | 65.9 $\\pm$ 5.7 |\n| CTM (No NLMs) | 82.9 $\\pm$ 4.4 | 35.0 $\\pm$ 7.2 |\n| CTM (No Synch) | 85.1 $\\pm$ 0.5 | 37.5 $\\pm$ 0.7 |\n| LSTM + Synch | 82.4 $\\pm$ 0.9 | 33.8 $\\pm$ 3.3 |\n\nReferring specifically to the question of the role of synchronization, it serves three important functions:\n1.  It **decouples the representation** for downstream use (i.e., attention and projections to predictions) from the raw neuron states ($z^t$), which enables neurons to develop and enhance dynamics without being constrained by the immediate expectations of the representation itself. In other words, the synchronization representation lets a rich representation unfold and stabilize while still allowing the neurons themselves to oscillate and develop patterns over long time scales.\n2.  It enables **multi-scale temporal sensitivity**, by way of the learnable decay parameters associated with neuron pairs ($r_{ij}$).\n3.  It enables **higher cardinality representations** since the number of neuron pairs is far greater than the number of neurons. Using pairwise decay parameters unique for each synchronization representation (i.e., there are multiple, and can be many multiple in the future), this further differentiates the functionality of each pairing in time, not just space.\n\nWe explored several techniques for extracting usable representations from the latent space of the CTM ($z^t$), including but not limited to: direct linear projections therefrom, MLP-based projections therefrom, gated projections, all of the aforementioned but into some ‘holder’ representation that updated slowly, and even 1D convolutions. None of these offered the same level of simplicity, flexibility, stability, and representational capacity that synchronization representations could. As soon as we started using synchronization we observed an ‘**unlocking**’ to the complexity of the neural dynamics that we had yet to see to such a degree.\n\n---\n\n## Re: questions\n\n### Re: language model.\nWe agree with that sentiment. However, the space of language modelling is extremely competitive and not necessarily well-suited as a test-bed for a truly novel modelling approach. The CTM can certainly benefit from the community effort required to transition a new model to a truly competitive one, but we simply do not have the resources to do that on our own. Hence, we focused on interesting experiments that could yield observations and findings that might eventually serve to understand artificial intelligence on a deeper level.\n\n### Re: why the best tick in the loss function?\nThis is an insightful question. You are right in thinking that it is not strictly necessary. The effect of including the best tick is that it promotes certainty in that it drives the best prediction to be even better. We found that this helped bootstrap learning, although it could benefit from further exploration in the future.\n\n### Re: Emergent phenomena.\nThere simply is no additional space in the main text to flesh these out. Further, many of these observations were as a consequence of the “journey” taken to develop this work, and would be difficult to re-explore after having learnt the lessons the observations had on offer. That said, we agree that future experiments into some of these phenomena would be fascinating and are currently investigating related concepts.\n\n---\n\nWe hope that this response addresses your outstanding concerns and answers your questions. It would be extremely helpful to engage with you further, particularly if there is anything you think we could do that would prompt you to consider improving your review score."}]

Author rebuttal:

Meta-review: The paper received recommendations for acceptance across the board. The biggest issue raised during reviewing was that the authors did a poor job in citing prior work. To be clear, I think their response to the reviewer regarding this issue was poor and surprising. This is not the first work to even use learnable synchrony, and I encourage the authors to do a deeper dive into the literature to better understand where the field is on this. Indeed, it seems that the authors didn't follow the citations Reichert & Serre 2014 until now, or else they would have done a better job at this. I recommend the authors fix this for their camera ready version.

With that said, the results and presentation are excellent, and the method is interesting and well-suited for NeurIPS. This is an easy accept.
Decision: Accept (spotlight)