Artificial Kuramoto Oscillatory Neurons
Authors: Takeru Miyato
Sindy Löwe
Andreas Geiger
Max Welling
Venue: ICLR 2025 Oral (2025)
Keywords: Oscillatory neurons
Feature binding
Object-centric learning
Reasoning
Adversarial robustness
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Oscillatory neurons strongly bind object features, can reason, and are robust to adversarial and natural perturbations

Abstract: It has long been known in both neuroscience and AI that ``binding'' between neurons leads to a form of competitive learning where representations are compressed in order to represent more abstract concepts in deeper layers of the network. More recently, it was also hypothesized that dynamic (spatiotemporal) representations play an important role in both neuroscience and AI. Building on these ideas, we introduce Artificial Kuramoto Oscillatory Neurons (AKOrN) as a dynamical alternative to threshold units, which can be combined with arbitrary connectivity designs such as fully connected, convolutional, or attentive mechanisms. Our generalized Kuramoto updates bind neurons together through their synchronization dynamics. We show that this idea provides performance improvements across a wide spectrum of tasks such as unsupervised object discovery, adversarial robustness, calibrated uncertainty quantification, and reasoning. We believe that these empirical results show the importance of rethinking our assumptions at the most basic neuronal level of neural representation, and in particular show the importance of dynamical representations. Code: https://github.com/autonomousvision/akorn Project page: https://takerum.github.io/akorn_project_page/

Published as a conference paper at ICLR 2025
ARTIFICIAL KURAMOTO OSCILLATORY NEURONS
Takeru Miyato', Sindy Löwe?, Andreas Geiger', Max Welling University of Tübingen, Tübingen Al Center University of Amsterdam
ABSTRACT
It has long been known in both neuroscience and Al that "binding" between neu- rons leads to form of competitive learning where representations are compressed in order to represent more abstract concepts in deeper layers of the network. More recently, it was also hypothesized that dynamic (spatiotemporal) representations play an important role both neuroscience and AI. Building on these ideas, we introduce Artificial Kuramoto Oscillatory Neurons (AKOrN) as dynamical alter- native to threshold units, which can be combined with arbitrary connectivity de- signs such fully connected, convolutional, or attentive mechanisms. Our gener- alized Kuramoto updates bind neurons together through their synchronization dy- namics. We show that this idea provides performance improvements across wide spectrum of tasks such as unsupervised object discovery, adversarial robustness, calibrated uncertainty quantification, and reasoning. We believe that these em- pirical results show the importance of rethinking our assumptions at the most ba- sic neuronal level of neural representation, and in particular show the importance of dynamical representations. Code:
Project page: https://takerumm hwwpmmuibhuubbbooooooooeoeee
INTRODUCTION
Before the advent of modern deep learning architectures, artificial neural networks were inspired by biological neurons. In contrast to the McCulloch-Pitt neuron (McCulloch & Pitts 1943) which was designed as abstraction of an integrate-and-fire neuron (Sherrington] 1906), recent building blocks of neural networks are designed to work well on modern hardware (Hooker 2021). As our understanding of the brain is improving over recent years, and neuroscientists are discovering more about its information processing principles, we can ask ourselves again if there are lessons from neuroscience that can be used as design principles for artificial neural nets.
In this paper, we follow a more modern dynamical view of neurons as oscillatory units that are cou- pled to other neurons (Muller et al. 20188). Similar to how the binary state of McCulloch-Pit neu- ron abstracts the firing of a real neuron, we will abstract an oscillating neuron by an N-dimensional unit vector that rotates on the sphere (Löwe al. 2023). We build a new neural network architec- ture that has iterative modules that update N --dimensional oscillatory neurons via generalization of the well-known non-linear dynamical model called the Kuramoto model (Kuramoto) 1984).
The Kuramoto model describes the synchronization of oscillators; each Kuramoto update applies forces to connected oscillators, encouraging them to become aligned or anti-aligned. This process similar to binding in neuroscience and can be understood as distributed and continuous clustering. Thus, networks with this mechanism tend to compress their representations via synchronization.
We incorporate the Kuramoto model into an artificial neural network, by applying the differential equation that describes the Kuramoto model to each individual neuron. The resulting artificial Ku- ramoto oscillatory neurons (AKOrN) can be combined with layer architectures such as fully con- nected layers, convolutions, and attention mechanisms.
We explore the capabilities of AKOrN and find that its neuronal mechanism drastically changes the behavior of the network. AKOrN strongly binds object features with competitive performance to slot-based models in object discovery. enhances the reasoning capability of self-attention, and increases robustness against random, adversarial, and natural perturbations with surprisingly good calibration.

---

Published as a conference paper at ICLR 2025
45000
5000
4 5500
50
100
150
200
250
Figure 1: Our proposed artificial Kuramoto oscillatory neurons (AKOrN). The series of pictures on the left are 64x 64 oscillators evolving by the Kuramoto updates (Eq. (2), along with plot of the energies computed by Eq. 3 Each single oscillator x, is an N-dimensional vector the sphere and is influenced by (1) connected oscillators through the weights Jij. (2) conditional stimuli c, and (3) S2; that determines the natural frequency of each oscillator. See Fig. for details on and J.
MOTIVATION
It was recognized early on that neurons interact via lateral connections (Hubel & Wiesel, 1962. Somers et al., 1995). In fact, neighboring neurons tend to cluster their activities (Gray et al. 1989 Mountcastle| 1997). and clusters tend to compete to explain the input. This "competitive learning" has the advantage that information is compressed as we move through the layers, facilitating the process of abstraction by creating information bottleneck (Amari & Arbib 1977). Additionally, the competition encourages different higher-level neurons to focus on different aspects of the input (i.e. they specialize). This process is made possible by synchronization: like fireflies in the night, neurons tend to synchronize their activities with their neighbors', which leads to the compression of their representations. This idea has been used in artificial neural networks before to model "binding" between neurons, where neurons representing features such as square, blue, and toy are bound by synchronization to represent a square blue toy (Mozer al., 1991 Reichert & Serre 2013 Löwe et al. 2022). In this paper, we will use an N-dimensional generalization of the famous Kuramoto model (Kuramoto] 1984) model this synchronization.
Our model has the advantage that it naturally incorporates spatiotemporal representations in the form of traveling waves (Keller al., 2024), for which there is ample evidence in the neuroscien- tific literature. While their role in the brain remains poorly understood, it has been postulated that they are involved in short-term memory, long-range coordination between brain regions, and other cognitive functions (Rubino al. 2006 Lubenov & Siapas 2009 Fell & Axmacher] 2011 Zhang et al. 2018 Roberts et al. 2019 Muller et al. 2016)] Davis et al. 2020 Benigno et al. 2023). For example, Muller et al. (2016) finds that oscillatory patterns in the thalamocortin
network during sleep are organized into circular wave-like patterns, which could give an account of how memories consolidated in the brain. Davis et al. (2020) suggest that spontaneous traveling waves in the vi- sual cortex modulate synaptic activities and thus act as gating mechanism in the brain. In the gen- eralized Kuramoto model, traveling waves naturally emerge as neighboring oscillators start syn- chronize (see on the left in Fig. and Fig. T0 in the Appendix).
Another advantage of using dynamical neurons is that they can perform a form of reasoning. Ku- ramoto oscillators have been successfully used to solve combinatorial optimization tasks such as k- SAT problems (Heisenberg) 1985 Wang Roychowdhury 2017). This can be understood by the fact that Kuramoto models can be viewed as continuous versions of discrete Ising models, where phase variables replace the discrete spin states. Many authors have argued that the modern architec- tures based on, c.g., transformers lack this intrinsic capability of "neuro-symbolic reasoning" (Dzini ctt al., 2024 Bounsi et al.) 2024). We show that AKOrN can successfully solve Sudoku puzzles, il- lustrating this capability. Additionally, AKOrN relates models in quantum physics and active mat- ter (see appendix B.I..
In summary, AKOrN combines beneficial features such competitive learning (i.e., feature bind- ing), reasoning, robustness and uncertainty quantification, as well as the potential advantages of trav- eling waves observed in the brain, while being firmly grounded in well-understood physics models.
2

---

Published as a conference paper at ICLR 2025
3 THE KURAMOTO MODEL
The Kuramoto model (Kuramoto 1984) is a non-linear dynamical model of oscillators, that exhibits synchronization phenomena. Even with its simple formulation, the model can represent numerous dynamical patterns depending on the connections between oscillators (Breakspear et al. 2010: Heit- mann et al. 2012).
In the original Kuramoto model, each oscillator is represented by its phase information 0; E [0, 2x). The differential equation of the Kuramoto model is
0. w. + Ej sin(O, 0,),
where wi is the natural frequency and J,y IR represents the connections between oscillators: if Jij > 0 the and j-th oscillator tend to align, and if Jij < O, they tend to oppose each other.
While the original Kuramoto model describes mieedimeniiona oscillators, we use a multi- dimensional vector version of the model (Olfati-Saber) [006)] Zhu 2013 Chandra et al. 2019 Lip- ton et al. 2021 Markdahl et al. 2021) with a cenameetr--reakinng term into neural networks. We de- note oscillators by X J where each x; is vector on hypersphere: x; E RR, ||x: 1. N is each single oscillator dimension called rotating dimensions and C is the number of oscillators. While each x; is time-dependent, we omit t for clarity. The oscillator index i may have multiple di- mensions: if the input is an image, for example, each oscillator is represented by Xa.b, with e, h, w indicating channel, height and width positions, respectively.
The differential equation of our vector-valued Kuramoto model is written as follows:
x = S2 + Projx, (c, + E1e where Projx, (y,) = y. - (y x4) X2
Here, S2 an N x N anti-symmetric matrix and S2 is called the natural frequency term that determines each oscillator's own rotation frequency and angle. The second term governs interactions between oscillators, where Proix. is an operator that projects an input vector onto the tangent space of the sphere at x,. We show a visual description of Proig, and a relation between the vector valued Kuramoto model and the original one in the Appendix A. C {c:){_ C2 RR- is data- dependent variable, which is computed from the observational input or the activations of the previous layer. In this paper, every C; is set to be constant across time, but it can be a time-dependent variable. c, can be seen as another oscillator that has unidirectional connection to X, Since C; is not affected by any oscillators, C; strongly binds x; to the same direction as C₁ i.e. it acts as bias direction (see Fig. 10 in the Appendix). In physics lingo, C is often referred to as "symmetry breaking" field.
The Kuramoto model Lyapunov if we assume certain symmetric properties in J.j and 22 (Aoyagi
1995
Wang & Roychowdhury 2017). For example, if Júj I, Jıj Jjs E IR, Ω. = $2, and 0, each update is guaranteed to minimize the following energy (proof is found in F
0c.
E E= E Σet Ef'x
Fig. on the left shows how the oscillators and the corresponding energy evolve with a simple Gaussian kernel as the connectivity matrix. Here, we set C as silhouette of fish, where c; on the outer silhouette and c; = 0 on the inner silhouette. The oscillator state is initially disordered, but gradually exhibits collective behavior, eventually becoming spatially propagating wavy pattern. We include animations of visualized oscillators, including oscillators of trained AKOrN models used in our experiments, the Supplementary Material.
We would like to note that we found that even without symmetric constraints, the energy value de- creases relatively stably, and the models perform better across all tasks we tested compared to mod- els with symmetric J. A similar observation is made by [ffenberger et al. (2022) where heteroge- neous oscillators such as those with different natural frequencies are helpful for the network to con- trol the level of synchronization and increase the network capacity. From here, we assume sym- metric constraints on J and Ω. Having asymmetric (a.k.. non nonrreciprocal) connections is aligned with the biological neurons in the brain, which also do not have symmetric synapses.
3

---

Published as a conference paper at ICLR 2025
X(4)
xT
xT
Kuramoto Layer
Kuramoto Layer Layer
Preprocess
Readout
Readout
C(O)
C(L)
Figure 2: Our proposed Kuramoto-based network (here, for image processing). Each block consists of a Kuramoto-layer and readout module described in Sec [ C(1) is used to make the final prediction of model. Similar network structures proposed in (Bansal al. 2022
Geiping
et al. 2025).
NETWORKS WITH KURAMOTO OSCILLATORS
We utilize the artificial Kuramoto oscillator neurons (AKOrN) basic unit of information process- ing in neural networks (Fig. 2 First, we transform an observation with a relatively simple function to create the initial conditional stimuli ( Next, ( is initialized by either C(O), a fixed learned embedding, random vectors, mixture of these initialization schemes. The block is composed of two modules: the Kuramoto layer and the readout module, which together process the pair (X, C). The Kuramoto layer updates X with the conditional stimuli C, and the readout layer extracts fea- tures from the final oscillatory states to create new conditional stimuli. We denote the number of layers by L, and I-th layer's output by (x()) C().
Kuramoto layer Starting with (1.0) := (1-1)) as initial oscillators, where the second superscript denotes the time step, update them by the discrete version of the differential equation (2)
Ax (1.)) 52.00 (1.4) + Proizao(c) (1-1) + Σ (1.0)
(4)
(--1 11
x (1.0) + Axxx (11)
(5)
where is the normalizing operator x/ | that ensures that the oscillators stay on the sphere.
0 is scalar controlling the step size of the update, which is learned in our experiments. We call this update a Kuramoto update or a Kuramoto step from here. We optimize both (2 and J(!)
given the task objective.
We update the oscillators T times. We denote the oscillators by (I.T), This oscillator state is used the initial state of the next block: ( (1.T)
Readout module We read out patterns encoded in the oscillators to create new conditional stimuli ((I) for the subsequent block. Since the oscillators are constrained onto the (unit) hyper-sphere, all the information is encoded in their directions. In particular, the relative direction between oscillators an important source of information because patterns after certain Kuramoto steps only differ in global phase shifts (see the last patterns in Fig. 10 in the Appendix). To capture phase invariant patterns, we take the norm of the linearly processed oscillators:
( g(m) E CR C' m& = |z11|2, 2₁ E, X (1.7) N.
(6)
where Usi N'N' is learned weight matrix, is learned function, and [m_, ..., mx |T
R N' is typically set to the same value as N. In this work, g is just the identity function, a
linear layer, or most a three-layer neural network with residual connections. Because the module computes the norm of (weighted) (ITT), this readout module includes functions that are invariant to the global phase shift in the solution space. Unless otherwise specified, we set C" = C and K C x N in all experiments.
4

---

Published as a conference paper at ICLR 2025
4.1 CONNECTIVITIES
We implement artificial Kuramoto oscillator neurons (AKOrN) within convolutional and self- attention layers. We write down the formal equations of the connectivity for completeness, however, they simply follow the conventional operation of convolution or self-attention applied to oscillatory neurons flattened w.r.t the rotating dimension N. In short, convolutional connectivity local, and attentive connectivity is dynamic input-dependent connectivity.
Convolutional connectivity To implement AKOrN in convolutional layer, oscillators and condi- tional stimuli are represented as (x,,,,w, Ce,b,wr where c, h, are channel, height and width posi- tions, and the update direction is given by:
Yc,h, = Cc,h,w +
Σ
Σ
Je, Je.d,h', /ttee (x
d Is w ER(H.WW
where R[H', W = [I...,, H''] x [l.... W is the H' x W' rectangle region (i.e. kernel size) and J Je,d s² E NxN are the learned weights in the convolution kernel where (c, d), (h', w() are output and input channels, and height and width positions.
Attentive connectivity Similar to Bahdanau et al. (2014); Vaswani et al. (2017), we construct the internal connectivity in the V-Vateentio manner. In this case, oscillators and conditional stimuli represented by {x,,;; ) where and are indices of tokens and channels, respectively. The
update direction becomes:
Vış C1. Σ
Ju Jl.mii -i,i+ mid E.h
ΣΣ
W R.t. A) R.R. Xmm.
(8)
m.j
(dillim)
An( = Em edult.m) da dl(,m))= Σ a
Ewo w X1,s, Σ wK WK.a..mm
(9)
where Wo h.i.k W.. K.k.j. W Na. i W h.o.t ERNN are learned weights of head h. Since the connec- tivity is dependent on the oscillator values and thus not static during the updates, it unclear whether the energy defined in Eq. 33) is proper. Nonetheless, in our experiments, the energy and oscillator states are stable after several updates (see the Supplementary Material, which includes visualiza- tions of the oscillators of trained AKOrN models and their corresponding energies over timesteps).
5 RELATED WORKS
Many studies have historically incorporated oscillatory properties into artificial neural net- works (Baldi Pineda 1991 Wang & Terman 1997] al. [0133] Neil et al. 2016|[Cher al.,
202 Rusch & Mishra 2020 Laboriiux & Zenke 20022; Rusch et al. 2022 van Gerven & Jensen, 2024 The Kuramoto model, a well-known oscillator model describing synchronization phenom- ena, has been rarely explored in machine learning, particularly in deep learning. However, several works motivate us to use the Kuramoto model as a mechanism for learning binding features. For example, although tested only in fairly synthetic settings, Liboni al. (2023) show that cluster fea- tures emerge in the oscillators of the Kuramoto model with lateral connections without optimization. Ricci et al. (2021) studies how data-dependent connectivity can construct synchrony on synthetic amples. Nguyen et al. (2024) relates the over-smoothing to the notion of phase-synchrony and uses the model to mitigate ovor-smogthing phenomena in graph neural networks. Also, line of works on neural synchrony (Reichert & Serre 2013] I Löwe et al. 2022 Stanic et al. 2023 [Zheng et al. 2023) Löwe al. 2023 Gopalakrishnan et al. (024)) shares the same philosophy with AKOrN. [hhng al. (2023) model synchrony by using temporal spiking neurons based on biological neuronal mecha- nisms. Löwe et al. (2023) extend the concept of complex-valus
neurons used by Reichert Serre 2013): Löwe et al. 2022) to abstract temporal neurons innoonn-innoooo multidimensional neurons. They show that, together with specific activation function called x-binding that implements the 'winner-take- all' mechanism at the single neuron level (Löwe et al., 2024), the multidimensional neurons learn to encode binding information in their orientations. Those synchrony-basee models are shown to work well relatively synthetic data but have been struggling scale natural images. Löwe et al. (003) show that their model can work with a large pre-trained self-super learning (SSL) model feature extractor, but its performance improvement is limited compared to slot-based models.
5

---

Published as a conference paper at ICLR 2025
FG-ARI
FG-ARI
75
50
MBO
40
Tetros
dSprites
CLEVR
Shapes
CLEVRiex
OOD
CAMO
CLEVRTOX
0OD
CAMO
BrCone
IrSA
It/SA (L 1)
AKOr (L
RISA [1-1]
AKDIN 9T &
AKOrN cone
(1-21
Figure 3: Object discovery performance on synthetic datasets.
Input
ItrSA
AKOrN
GTmask
Input
ItrSA
AKOrN
GTmask
Figure 4: AKOrN learns more object-bound features than the non- nuramoooooto model counterpart.
CLEVRTex
OOD
CAMO
Model
FG-ARI
MBO FG-ARI
MBO
FG-ART
MBO
"MONet (Burgess et al. 2019
19.811.
37.3110
31.510.9
SLATE (Singh et
2022)
44.21NA 50.91NA
Slot-Attetion Locatello et al.
2020
62.412.3
58.511.9
57.511.0
Slot-diffusion Wu el al. 2023
69.7=NA
61.9+NA
Slot-diffusion+ [Wu et al.
2023
78.5 68.7+NA
"DTI [Mnnier et al.
2022
79.9+1.4
73.7+1.0
72.9+1.9
"I-SA CCang et al.
2022
79.0+3.9
83.7740.9
57.2+13.3
BO-SA (Jia et al.
2023)
80.542.5
86.510.2
63.746.1
ISA-TS Biza
2023
92.940.4
84.410.8
86.2
AKOryatto
88.510.9
59.7.0.9 87.7 60.810.6
77.010.5 53.4.0.7
Object discovery performance on CLEVRTex and its variants (OOD, CAMO), AKON is compared among models trained from scratch. *Numbers taken from Jia et al. (203).
Table I:
Slot-based models (Le Roux et al. 2011 Burgess et al. 2019 Greff al. 2019 Locatello et al,, 2020) are the most-used model for object-centric (OC) learning. Their discrete nature of represen- tations is shown to be good inductive bias to learn such OC representations. However, similarly to synchrony-based models, these models struggle on natural images and therefore often combined with powerful, pre-trained SSL models such as DINO (Caron et al. [202T) Our proposed contin- Kuramoto neurons be building block of the SSL network itself, and show that they learn better object-centric features than well-known SSL models. Our work is the first work that demonstrates that synchrony-
model solely scaled up to natural images.
AKOrNs perform particularly well on object discovery tasks when implemented in self-attention layers. Self-attention updates with normalization have been shown mathematically cluster token features (Geshkovski et al.) 2024). Our work combines this clustering behavior of transformers with the clustering induced by the synchronization of the Kuramoto neurons, resulting in AKOrN being the first competitive method to slot-based approaches.
Finally, there exist several works on interpreting self-attention the context of the Hopfield net- works (Ramsauer et al. 2020; Hoover et al.,(2022) (223).. Energy transformer (Hoover et al. 2023) intro- duces a symmetrized attention mechanism to guarantee the update minimizing certain energy. How- we find such symmetric models worsen the performance in reasoning task. Our Kuramoto- based models differ from these approaches:
neurons with asymmetric con- nections in J, and their symmetry-breakin term C. These elements contribute to performance im- provement over the approach by (Hoover et al. 2023) and conventional self-atee attention in the reason- ing task of our experiments.

---

Published as a conference paper at ICLR 2025
Input
DINO
AKOrN
GTMask
Input
DINO
AKO
GTmask
Figure 5: Visualization of clusters on (Left) Pascal and (Right) COC02017.
EXPERIMENTS
6.1 UNSUPERVISED OBJECT DISCOVERY
Unsupervised object discovery is the task of finding objects in an image without supervision. Here, we test AKOrN on five synthetic datasets (Tetrominoes, dSprites, CLEVR (Kabra et al. 2019). Shapes, CLEVRTex (Karazija et al., 202 and two real image datasets (PascalVOC (Everingham jet al. 2010). COCO2017 (Lin et al., 2014)) (see the Appendix C for details). Among the five synthetic datasets, CLEVRTex has the most complex objects and backgrounds. We further evaluate the models trained on the CLEVRTee dataset on two variants (OOD, CAMO). The materials and shapes of objects in OOD differ from those in CLEVRTex, while CAMO (short for camouflage) features scenes where objects and backgrounds share similar textures within each scene.
As baselines, eet al. 2021), but iterate the convolution or self-attention layers multiple times with shared parame- ters. This allows evaluate the impact of our proposed, Kuramoto-based iterative updates. We denote these basclines as Iterative Convolution (ItrConv) and Iterative Self-Attention (ItrSA), re- spectively. Fig. in in the Appendix shows diagrams of each network. In AKOrN, C is initialized by the patched features of the images, while each is initialized by random oscillators sampled from the uniform distribution on the sphere. We train the AKON model and baselines with the self- supervised SimCLR (Chen et al. 2020) objective.
train models that similar to ResNet (He et al. 2016) and VIT (Dosovitskiy
We train each model from scratch on the five synthetic datasets. For the two real image datasets, we first train AKOrN on ImageNet (Krizhevsky et al.) 2012) and directly evaluate that ImageNet- pretrained model on both datasets without fine-tuning. When evaluating, we apply clustering to final block's output features (In AKOrN, it is C(D)). We use agglomeration clustering with average linkage, which we found to outperform K-means for both the baseline models and AKOrN. We evaluate the clustering results by foreground adjusted rand index (FG-ARI) and Mean-Best-Overt (MBO). FG-ARI measures the similarity between the ground truth masks and the computed clusters, only for foreground objects. MBO first assigns each cluster to the highest overlapping ground truth mask and then computes the average intersection-over over-union (IoU) of all pairs. For Pos and COCO2017, we show instance-level MBO (MBO,) and class- (MBO.) segmentation results.
for details.
7

---

Published as a conference paper at ICLR 2025
PascalVOC COC02017
Model
MBO, MBO, MBO, MBO,
(slot-based models)
Slot-attention
22.2
23.7
24.6
24.9
SLATE (Singh et al. 2021
35.9
41.5
29.1
33.6
(DINO + slot-based model)
DINOSAUR (Seitzer et al. 2023
44.0
51.2
31.6
39.7
Slot-diffusion (Wu et al. 2023
50.4
55.3
31.0
35.0
SPOT Kakogeorgiou et al.
2024)
48.3
55.6
35.0
44.7
(transformer SSL))
MAE (He et al. 2022
34.0
38.3
23.1
28.5
MoCoV3 (Chen al. 2021a)
47.3
53.0
28.7
36.0
DINO (Caron et al. 2021)
47.2
53.5
29.4
37.0
AKOrN
52.0
60.3
31.3
40.3
Table 2: Object discovery on PascalVOC and COCO2017.
Because the patched feature resolution is small due to patchification, the obtained cluster assign- ments are coarse. compute finer cluster assignments, we introduce an upsampling method called up-tiling. This involves shifting the input image slightly along the horizontal and/or vertical axes to generate multiple feature maps. These feature maps are then interleaved to create a higher-resolutiin feature map. See Section 1.2 in the Appendix for the methodological details of this up-tiling.
AKOrN binds object features Fig. 3 shows that AKOrNs improve the object discovery performance over their non- non-Kurann Kuramoto counterparts in every dataset. Interestingly, we observe that convolution less effective than attention. In Fig. 4 we see that the Kuramoto models' clusters are well- aligned with the individual objects, while clusters of the ItrSA model often span across objects and background, and are sensitive to the texture of the background and the specular highlight on the floor (more clustering results shown in Figs 32 34 in the Appendix).
Tab. I shows a comparison to existing works on CLEVRTex and its variants. All other methods are slot-based. Among the distributed representation models, AKOrN is the first method that is shown to be competitive with slot-based models on the complex CLEVRTex dataset.
AKOrN scales to natural images Fig. shows AKOrN binds object features natural images much better than DINO (Caron et al., 202T). We show a benchmark comparison on Pascal VOC and COC02017 in Tab. 2 The proposed AKOrN model outperforms existing SSL models including DINO, MoCoV3, and MAF on both datasets, showing that it learns more object-bound features than conventional transformer-basee including models trained from scratch and models trained on features of a pretrained DINO model. On COCO, AKOrN again outperforms methods that are trained from scratch and is competitive to DINOSAUR and Slot-diffusion, but is outperformed by the recent SPOT model.
models. On Pascal, AKOrN considerably better than other models
6.2 SOLVING SUDOKU
To test AKOrN's reasoning capability, we apply it on the Sudoku puzzle datasets (Wang et al. 2019
Palm et al. 2018). The training contains boards with 31-42 given digits. We test models
distribution (ID) and
(OOD) scenarios. The ID test set contains 1,000 boards sampled from the same distribution, while boards in the OOD set contain much fewer given digits (17-34) than the train set. To initialize C, we use embeddings of the digits 0-9 (0 for blank, 1-9 for given digits). The initial x; takes the value c ////c///2 2 when digit is given, and is randomly sampled from the uniform distribution on the sphere for blank squares. The number of Kuramoto steps during training is set to 16. We also train a transformer model with 8 blocks.
AKOrN solves Sudoku puzzles AKOrN perfectly solves all puzzles in the ID test set, while only Recurrent Transformer (R- Transfforrerr (Yang et al., (203)) achieves this (Tab. B}. On the OOD set, AKOrN achieves 89.5+2.5 which is better than all other existing approaches including IRED (Du ct al. 2024). an energy-based diffusion model. AKOrN again strongly outperforms its non- Kon-Kurann Kuramoto counterparts, ItrA and Transformer.
8

---

Published as a conference paper at ICLR 2025
(a)
-10000
13 Test
11660
(c)
Board Acc (%)
50-
2:500
11000
-20000
22.700
-00
AKONN
tr
-20000
20
-20200
22250
27000
25
50
50
Steps
Stepss
16
32
64
128
Steps
Tevu
Figure 6: (a) Transition of the energy in Eq. (3) over # Kuramoto steps the Sudoku datasets. The semi traniirannprrentt lines are actual energy values averaged across examples, and the solid ones connect the troughs. The dotted vertical line indicates # Kuramoto steps set during training. (b) A zoomed- in version of each plot. (c) The effect of test-time extension on # Kuramoto steps.
Model
ID
OOD
Board Accuracy (%)
90
SAT-Net (Wang et al. 2019
98.3
3.2
Diffusion Duu et al. 2024)
66.1
10.3
80
IREM Du al. 2022)
93.5
24.6
70
RRN (Palm et al.
2018
99.8
28.6
R-Transtormer
Yang al. 2023
100.0
30.3
60
IRED (Du et al.
2024)
99.4
62.1
50
Transformer
98.6+0.3
5,2+0.2
1
16
256 4096
ItrSA AKOrNatta
95.7 18.5
34.415..
# Random samples
100.0#0.0
89.5+2.5
Figure Improvement of board ac- Table 3: Board accuracy on Sudoku Puzzles. We show the curacy by the post-selection of predic- mean and std of the accuracy of models with 5 different ran- tions based on the E values described dom seeds for the weight initialization. The AKOrN results obtained with Teval 128 and the energy- based voting with 4096 samples of initial oscillators.
in Sec 6.2
Teval is set to 128.
Test-time extension of the Kuramoto steps Just as we humans use more time to solve harder prob- lems, AKOrN's performance improves as we increase the number of Kuramoto steps. As shown in (a, on the ID test set, the energy fluctuates but roughly converges to minimum after around 32 steps. On the OOD test set, however, the energy continues to decrease further. Fig. 6 (c) shows that increasing the number of Kuramoto steps at test time improves accuracy significantly (17% to 52%), while increasing the step count of standard self-attention provides a limited improvement on the OOD set (14% to 34%) and leads to lower performance on the ID set (99.3% to 95.7%).
The energy value tells the correctness of the boards The energy value defined Eq 3 is good indicator of the solution's correctness. In fact, we observe that predictions with low-energy oscillator states tend to be correct (see Fig. 28). We utilize this property to improve the performance. sample multiple predictions with different initial oscillators and select
For each given board, the lowest-energy prediction as the model's answer, which we call Energy-based voting (F-vote). We in Fig. that by increasing the number of sampled predictions, the model's board accuracy improves. This result implies that the Kuramoto layer behaves like energy-based models, even though its parameters are optimized solely based on the task objective. We found that just averaging the predictions of different states (i.e., majority voting) does not give better answers.
6.3 ROBUSTNESS AND CALIBRATION
We test AKOrN's robustness to adversarial attacks and uncertainty quantification performance on CIFARIO and CIFARIO with common corrppiions (CC, Hendrycks & Dietterich| (2019)). We train two types networks: a convolutional AKOrN (AKOrNeomy, ) and AKOrN with both convolution and self-attention (Ax.r.ymix). The former has three convolutional Kuramoto layers. The latter replaces the last block with an attentive Kuramoto block. We use AutoAttack (Caron et al. 2021 to evaluate the model's 's adversarial robustness.
AKOrNs are resilient against gradient- brase attacks The model heavily regularized and achieves both good adversarial robustness and robustness natural corruptions (Tab. 4 This is

---

Published as a conference paper at ICLR 2025
Accuracy (%)
ResNet-18 Diffenderfer 21 J=
Q-0 0 No Full
random esc.
Figure 8: Robustness performance on random noise examples. Each bar plot shows classification accuracy on CIFARIO with strong random noise (I|ell>> 64/255). The left two pictures are examples of images with that c. Green bars show accuracy when we ablate each element of AKOrN.
t Accuracy Adv
J. JCE
Bartoldson 24
Diffenderfer 21
Model
Clean
CC
CC
100
Bartoldson et al. (2024)
93.68
73.71 75.9
20.5
Diffenderfer al. (2021)
96.56
0.00 92.8
4.8
50
50
ViT
91.44
0.00
81.0
9.6
ResNet-18
94.41
0.00
81.5
8.9
AKOrNconv
88.91
*58.91 83.0
1.3
50
100
50
100
AKOrNmix
91.23 *5.55 86.4
1.4
ResNet-18 18
AKOrymix
100
100
Accuracy
Table 4: Robustness to adversarial examples by Au-
toAttack (Adv) and common corruptions (CC) on CI- FARI0. "The attack is done by AutoAttack with
50
50
EoT (Athalye al. 2018). ||e||
8/255. Ex-
pected Calibration Error (ECE) measures the alignment between confidence of the prediction and accuracy. The top two methods are selected from the highest-
100
Confidence
Figure 9: Confidence vs Accuracy plots on
ranked methods on https://robusthench.h github.ii
CIFAR10 with common corruptions.
markable, since conventional neural models need additional techniques such as adversarial training and/or adversarial purification to achieve good adversarial robustness. In contrast, AKOrN is robust by design, even when trained on only clean examples.
K-Nets are well-calibrated and robust to strong random noise We found that AKOrNs are robust to strong random noise (Fig, 8 and give good uncertainty estimation (on the bottom right in Fig. 9 Surprisingly, there is almost perfect correlation between confidence and actual accuracy. This similar to observations in generative models (Grathwohl et al. 2020) Jaini et al.) 2024), where conditional generative models give well-calibrated outputs. Since AKOrN's energy not learned to model input distribution, we cannot tightly relate ours to such generative models. However, we speculate that AKOrNs' energy roughly approximates the likelihood of the input examples, and thus the oscillator state fluctuates according to the height of the energy, which would result in good calibration.
DISCUSSION & CONCLUSION
We propose AKOrN, which integrates the Kuramoto model into neural networks and scales to com- plex observations, such as natural images. AKOrNs learn object-binding features, can reason, and are robust to adversarial and natural perturbations with well-calibrated predictions. We believe our work provides a foundation for exploring a fundamental shift the current neural network paradigm.
In the current formulation of AKOrN, each oscillator is constrained onto the sphere and each single oscillator cannot represent the 'presence" of the features like the rotating features in Löwe et al. 2023). Because of that, AKOrN would not perform well memory tasks, where the model needs to remember the presence events. This norm constraint also does not align with real biological neurons that have firing and non-firing states. Relaxing the hard norm constraint of the oscillator would be an interesting future direction in terms both biological plausibility and applicability to much wider range of tasks such long-term temporal processing.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGEMENT
Takeru Miyato and Andreas Geiger were supported by the ERC Starting Grant LEGO-3D (850533) and the DFG EXC number 2064/1 project number 390727645. We thank Andy Keller, Lyle Muller, Terry Sejnowski, Bruno Olshausen, Christian Shewmake, Yue Song, Pietro Perona, Andreas Tolias, Masanori Koyama, Jun-nosuke Teramae, Bernhard Jaeger, Madhav Iyengar, and Daniel Dauner for their insightful feedback and comments. We thank Vladimir Fanaskov for providing more general proof of the Lyapunov property of our generalized Kuramoto model. Max Welling thanks the Cal- ifornia Institute of Technology for hosting him in January and February 2024. Takeru Miyato ac- knowledges his affiliation with the ELLIS (European Laboratory for Learning and Intelligent Sys- tems) PhD program.
REFERENCES
Shun-Ichi Amari and Michael Arbib. Competition and cooperation neural nets. Systems neuroscience, pp. 119-165, 1977.
Toshio Aoyagi. Network of neural oscillators for retrieving phase information. Physical review letters, 74(20): 4075, 1995.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In Proc. of the International Conf. on Machine learning (ICML), pp. 274-283, 2018.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv.org, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv.org, 2014.
Pierre Baldi and Fernando Pineda. Contrastive learning and neural oscillations. Neural Computation, 3(4):
225, 1991.
Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. End-to-end algorithm synthesis with recurrent networks: Extrapolation without overthinking. Advances Neural Information Processing Systems, 35:20232-20242, 2022.
Brian Bartoldson, James Diffenderfer, Konstantinos Parasyris, and Bhavya Kailkhura. Adversarial robustness limits via scaling-law, law and human-alignnn aligmmtt studies. In Proc. of the International Conf. on Machine learning (ICML), 2024.
Gabriel B Benigno, Roberto C Budzinski, Zachary W Davis, John Reynolds, and Lyle Muller. Waves travel- ing over map of visual space can ignite short-term predictions of sensory input. Nature Communications, 14(1):3409, 2023.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
Ondrej Biza. Sjoerd Van Steenkiste, Mehdi SM Sajjadi, Gamaleldin F Elsayed, Aravindh Mahendran, and Thomas Kipf. Invariant slot attention: Ohject discovery with slot-centric reference frames. Proc. of the International Conf. on Machine learning (ICML), 2023.
Wilfried Bounsi, Borja Ibarz, Andrew Dudzik, Jessica Hamrick, Larisa Markeeva, Alex Vitvitskyi, Razvan Pascanu, and Petar Veličković. Transformers meet neural algorithmic reasoners. arXiv.org, 2024.
Michael Breakspear, Stewart Heitmann, and Andreas Daffertshofer. Generative models of cortical oscillations: neurobiological implications of the kuramoto model. Frontiers in human neuroxcience, 4:190, 2010.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised scene decomposition and representation. arXiv.org. 2019.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, In Proc. of the IEEE International Conf. on Computer Vision (ICCV), pp. 9650-9660, 2021.
Sarthak Chandra, Michelle Girvan, and Edward Ott. Continuous versus discontinuous transitions in the d- dimensional generalized genneaiieeleeeeerliieeeed kuramoto model: Odd d is different. Physical Review 91 1002, 2019.

---

Published as a conference paper at ICLR 2025
Michael Chang. Tom Griffiths, and Sergey Levine. Object representations fixed points: Training iterative refinement algorithms with implicit differentiation. In Advances Neural Information Processing Systems (NeurIPS), pp. 22694-32708, 2022.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geofiryy Hinton. A simple framework for contrastive learning of visual representations. In Proc. of the International Conf. on Machine learning (ICML), pp.
1597-1607, 2020.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proc. of IEEE International Conf. on Computer Vision (ICCV), pp. 9640-9649, 2021a.
Yi Chen, Hong Qu, Malu Zhang, and Yuchen Wang. Deep spiking neural network with neural oscillation and spike-phase information. In Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI 202/), pp. 7073-7080, 2021b.
Zachary W Davis, Lyle Muller, Julio Martinez-Trujj Trujillo, Terrence Sejnowski, and John H Reynolds. Sponta- ncous travelling cortical waves gate perception in behaving primates. Nature, 587(7834):432 2020.
Bhishma Dedhia and Niraj K Jha, Neural slot interpreters: Grounding object semantics in emergent slot repre- sentations. arXiv.org 2024.
James Diffenderfer, Brian Bartoldson, Shreya Chaganti, Jize Zhang, and Bhavya Kailkhura. A winning hand: Compressing deep networks can improve -fitt--distribution robustness. In Advances in Neural Information Prucessing Systems (NeurlPS), pp. 664-676. 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Proc. of the In- ternational Conf. on Learning Representations (ICLR), 2021.
Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordateh. Learning iterative reasoning through energy minimization. In Proc. of the International Conf. on Machine learning (ICML), pp. 5570-5582, 2022.
Yilun Du, Jiayuan Mao, and Joshua B Tenenbaum. Learning iterative reasoning through energy diffusion. In Proc. of the International Conf. on Machine learning (ICML), 2024.
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, et al. Faith and fate: Limits of transformers on compositionality In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Felix Effenbegger, Pedro Carvalho, Igor Dubinin, and Wolf Singer. The functional role of oscillatory dynamics in neocortical circuits: a computational perspective. bioRxiv, pp. 2022-11, 2022.
Mark Everingham, Lue Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision (ICV), 88:303-338, 2010.
Juergen Fell and Nikolai Axmacher: The role of phase synchronization in memory processes. Nature reviews neuroscience,
2011.
Michel Fruchart, Ryo Hanai, Peter Littlewood, and Vincenzo Vitelli. Non-reciprocal phase transitions. Nature, 592(7854):363 369, 2021.
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv, 2025.
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. The emergence of clusters in self- attention dynamics. In Advances Neural Information Processing Systems (NeurIPS), 2024.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Proc. of the International Conf. on Learning Representations (ICLR), 2014.
Anand Gopalakrishnan, Aleksandar Stanić, Jürgen Schmidhuber, and Michael Curtis Mozer. Recurrent complex-weighted autoencoders for unsupervised object discovery. arXivorrg, 2024.
Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. Uncovering the limits of adversarial training against norm-hounded adversarial examples. arXivorg, 2020.
12

---

Published as a conference paper at ICLR 2025
Sven Gowal, Sylvestre-Alvise Rebufli, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A Mann. Improving robustness using generated data, Advances Neural Information Processing Systems (NeurIPS), pp. 4218 4233, 2021.
Will Grathwohl. Kuan- Chich Wang. Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi. Kevin Swersky. Your classifier is secretly an energy based model and you should treat like one. Proc. of the International Conf. on Learning Representations (ICLR), 2020.
Charles M Gray, Peter König, Andreas K Engel, and Wolf Singer. Oscillatory responses in cat visual cortex exhibit inter-columan synchronization which reflects global stimulus properties. Nature, 338(6213))334 337, 1989.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerehner. Multi-object representation learning with iterative variational inference. In Proc. of International Conf. on Machine learning (ICML), pp. 2424-2433, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Proc. of the European Conf. on Computer Vision (ECCV), pp. 630-645, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 16000-16009, 2022.
Werner Heisenberg. Zur theorie des ferromagnetismus, Springer, 1985.
Stewart Heitmann, Pulin Gong, and Michael Breakspear. A computational role for bistability and traveling waves motor cortex. Frontiers computational neuroscience, 6:67, 2012.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In Proc. of International Conf. Learning Representations (ICLR), 2019.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Aug- mix: A simple data processing method to improve robustness and uncertainty. In Proc. of International Conf. on Learning Representations (ICLR), 2020.
Sara Hooker. The hardware lottery. Communications of the ACM, 64(12):58 65, 2021.
Benjamin Hoover, Yuchen Liang, Bao Pham, Rameswar Panda, Hendrik Strobelt, Duen Homg Chau, Mo- hammed Zaki, and Dmitry Krotov. Energy transformer. Advances in Neural Information Processing Sys- tems (NeurIPS), 2023.
John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Pro- ceedings of the national academy of sciences, 79(8):2554-22 2558, 1982.
David Hubel and Torsten N Wiesel, Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of physiology, 160(1):106, 1962.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Pruc. of the International Conf. on Machine learning (ICML), 2015.
Priyank Jaini, Kevin Clark, and Rohert Geirhos. Intriguing properties generative classifiers. Proc. of the International Conf. on Learning Representations (ICLR), 2024.
Baoxiong Jia, Yu Liu, and Siyuan Huang. Improving object-centric learning with query optimization. In Proc. of the International Conf. on Learning Representations (ICLR), 2023.
Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. In Advances Neural Information Processing Systems (NeurIPS), 2023.
Whie Jung, Jachoon Yoo, Sungjin Ahn, and Seunghoon Hong. Learning to compose: Improving object centric learning by injecting compositionality. In Proc. of the International Conf. on Learning Representations (ICLR), 2024.
Rishabh Kabra, Chris Burgess, Loic Matthey, Raphael Lopez Kaufman, Klaus Greff, Malcolm Reynolds, and Alexander Lerchner. Multi-object datascts. http://hww.ccccceeeeemmnmnmttttt
2019.
Ioannis Kakogeorgiou, Spyros Gidaris, Konstantinos Karantzalos, and Nikos Komodakis. Spot: Self-training with patch-order permutation for object-centric learning with autoregressive transformers. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 22776-22786, 2024.
13

---

Published as a conference paper at ICLR 2025
Laurynas Karazija, Iro Laina, and Christian Rupprecht. Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation. arXiv.org, 2021.
T Anderson Keller, Lyle Muller, Terrence J Sejnowski, and Max Welling. A spacetime perspective dynamical computation in neural information processing systems. arXiv.org 2024.
Nicholas Ketz, Srinimisha G. Morkonda, and Randall C. O'Reilly. Theta coordinated error-driven learning in the hippocampus. PLoS Computational Biology, (6))e1003067 2013.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. of the International Conf. on Learning Representations (ICLR), 2015.
Klim Kireev, Maksym Andriushchenko, and Nicolas Flammarion. On the effectiveness of adversarial training against common corruptions. In Unceriainty in Artificial Intelligence, pp. 1012-1021. PMLR, 2022.
Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2012.
Yoshiki Kuramoto. Chemical turbulence. Springer, 1984,
Alina Kuznetsova, Hassan Rom, Neil Alidrin, Jasper Uijlings. Ivan Krasin, Jordi Pont- Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov. et al. The open images dataset Unified image classification, object detection, and visual relationship detection at scale. International journal of computer vision, 128(())1956-19)) 2020.
Axel Laborieux Friedemann Zenke. Holomorphic equilibrium propagation computes exact gradients through finite size oscillations. Advances in neural information processing systems, 35:1 29500-12963, 2022.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Nicolas Le Roux, Nicolas Heess, Jamie Shotton, and John Winn. Learning generative model of images by factoring appearance and shape. Neural Computation, 23(3):593-650, 2011.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibratee classifiers for detecting out-of-distribution samples. arXiv.org, 2017.
Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Decpak Pathak. Your diffusion model is sccretly a zero-shot classifier. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), pp. 2206 2217,222222222222 2023.
Luisa HB Liboni, Roberto C Budzinski, Alexandra N Busch, Sindy Löwe, Thomas A Keller, Max Welling, and Lyle E Muller. Image segmentation with traveling waves in exactly solvable recurrent neural network. arXiv.org, 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Ziticck. Microsoft coco: Common objects in context. In Proc. of the European Conf. on Computer Vision (ECCV), pp. 740-755. Springer, 2014.
Max Lipton, Renato Mirollo, and Steven H Strogatz. The kuramoto model sphere: Explaining its low- dimensional dynamics with group theory and hyperbolic geometry. Chaos: An Interdisciplinary Journal of Nonlinear Science, 31(9), 2021.
Francesco Locatello, Dirk Weissenbom, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Advances in Neural Information Processing Systems (NeurIPS), pp. 11525-11538, 2020.
Sindy Löwe, Phillip Lippe, Maja Rudolph, and Max Welling. Complex-valued autoencoders for object discov- ery. arXiv preprini arXiv:2204.0200 2022.
Sindy Löwe, Phillip Lippe, Francesco Locatello, and Max Welling. Rotating features for object discovery. In Advances in Neural Information Processing Systems (NeurlPS). 2023.
Sindy Löwe, Francesco Locatello, and Max Welling. Binding Dynamics in Rotating Features. ICLR 2024 Workshop: Bridging the Gap Between Practice and Theory in Deep Learning, 2024.
Evgueniy V Lubenov and Athanassios G Siapas. Hippocampal theta oscillations travelling waves. Nature, 459(7246):5345533 2009.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. arXivorg, 2017.
14

---

Published as a conference paper at ICLR 2025
Johan Markdahl, Daniele Proverbio, La Mi, and Jorge Goncalves. Almost global convergence practical synchronization in the generalized kuramoto model networks over the n-sphere. Communications Physics, 4(1):187, 2021.
Daniel C Mattis. The theory of magnetism Statics and Dynamics, volume 17. Springer Science & Business Media, 2012.
Warren S McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5:115-132 1943.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regulariza- tion method for supervised and semi-supervised learning. IEEE transactions pattern analysis and ma- chine intelligence, 41(8):1979-1993, 2018.
Takeru Miyato, Bernhard Jaeger, Max Welling, and Andreas Geiger. Gta: A geometry-aware attention mecha- nism for multi-view transformers. Proc. of the International Conf. on Learning Reprexentations (ICLR), 2024.
Tom Monnier, Elliot Vincent, Jean Ponce, and Mathicu Aubry. Unsupervised layered image decomposition into object prototypes. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 8640- 8650, 2021.
Vernon B Mountcastle, The columnar organization of the neocortex. Brain: journal of neurology, 120(4): 701-722, 1997.
Michael C Mozer, Richard Zemel, and Marlene Behrmann. Learning to segment images using dynamic feature binding. Advances Neural Information Processing Systems, 1991.
Lyle Muller, Giovanni Piantoni, Dominik Koller, Sydney Cash, Eric Halgren, and Terrence J Sejnowski. Rotating waves during human sleep spindles organize glohal patterns of activity that repeat precisely through the night. Elife, 5:e17267, 2016.
Lyle Muller, Frédéric Chavane, John Reynolds, and Terrence J Sejnowski. Cortical travelling waves: mecha- nisms and computational principles. Nature Reviews Neuroscience, 19(5):255-28) 268, 2018
Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu. Phased Istm: Accelerating recurrent network training for long event-based sequences. In Advances in Neural Information Processing Systems 29 (NIPS 2016), pp. 3889-3897, 2016.
Andrew Ng and Michael Jordan. On discriminative generative classifiers: A comparison of logistic regres- sion and naive bayes. Advances Neural Information Processing Systems (NeurIPS), 2001.
Tuan Nguyen, Hirotada Honda, Takashi Sano, Vinh Nguyen, Shugo Nakamura, and Tan Minh Nguyen. From coupled oscillators to graph neural networks: Reducing over-smoothing via a kuramoto model-based ap- proach. In International Conference on Artificial Intelligence and Statistics, pp. 27 2710-2718. PMLR, 2024.
Reza Saber. Swarms on sphere: programmable swarm with synchronous behaviors like oscillator networks. In Proceedings of the 45th IEEE Conference on Decision and Control, pp. 1060-5066. IEEE, 2006.
Rasmus Palm, Ulrich Paquet, and Ole Winther, Recurrent relational networks. Advances in neural information processing systems, 31, 2018.
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleimer, Milena Pavlović, Geir Kjetil Sandve, Hopfield networks all you need. arXiv.org, 2020.
David P Reichert and Thomas Serre. Neuronal synchrony in complex-valued deep networks. arXiv preprint 1311v:131..6115, 2013.
Matthew Ricci, Minju Jung, Yuwei Zhang, Mathieu Chalvidal, Aneri Soni, and Thomas Serre. Kuranet: sys- tems of coupled oscillators that learn to synchronize. arXiv.org, 2021.
James A Roberts, Leonardo L Gollo, Romesh G Abeysuriya. Gloria Roberts, Philip B Mitchell, Mark W Woolrich, and Michael Breakspear. Metastable brain waves. Nature communications, 10(1):1056, 2019.
Doug Rubino, Kay A Robbins, and Nicholas G Hatsopoulos. Propagating waves mediate information transfer in the motor cortex. Nature neuroscience, 1549)1559-15557 2006.
15

---

Published as a conference paper at ICLR 2025
T Konstantin Rusch and Siddharthh Mishra. Coupled uscillatory recurrent neural network (cornn); An accurate and (gradient) stable architecture for learning long time dependencies. arXiv preprint arXiv:2010.00951, 2020.
Konstantin Rusch, Ben Chamberlain, James Rowbottom, Siddhartha Mishra, and Michael Bronstein. Graph- coupled oscillator networks, In International Conference on Machine Learning, pp. 18888-18909, PMLR, 2022.
Bruno Sauvalle and Arnaud de La Fortelle. Unsupervised multi-object segmentation using attention and soft- argmax. In Proc. of the IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 3267- 3276, 2023.
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon- Gabriel, Tong He, Zheng Zhang, Berhaard Schölkopf, Thomas Brox, al. Bridging the gap real-world object-centric learning Proc. of the International Conf. on Learning Representations (ICLR), 2023.
Charles Scott Sherrington. The integrative action of the nervous system. Yale University Press, 1906.
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e Iearns to compose. arXiv.org, 2021.
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. In Advances in Neural Information Processing Systems (NeurlPS), pp. 18181-18196, 2022.
David C Somers, Sacha B Nelson, and Mriganka Sur. An emergent model of orientation selectivity in cat visual cortical simple cells. Journal neuroscience, 15(8):5448 5465, 1995.
Aleksandar Stanić. Anand Gopalakrishnan, Kazuki Irie, and Jürgen Schmidhuber. Contrastive training of complex-valued autoencoders for object discovery. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Jianlin Su. Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced trans- former with rotary position embedding, arXivorrg 2021.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. Proc. of the International Conf. on Learning Representations (ICLR), 2014.
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. In Advances in Neural Information Processing Systems (NeurIPS), pp. 1633-1645, 2020.
Marcel van Gerven and Ole Jensen. Oscillations in an artificial neural network convert competing inputs into temporal code. PLoS Computational Biology, :009):1 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NIPS), pp. 5998 00088,60882000 2017.
Deliang Wang and James T. Terman. Image segmentation based on oscillatory correlation. Neural Computation, 944):055-336, 1997.
Po-Wei Wang, Priya Donti, Bryan Wilder, and Zico Kolter. Satnet: Bridging deep learning and logical reasoning using differentiable satistiability solver. In Proc. of the International Conf. on Machine learning (ICML), pp. 6545 6554, 2019.
Tianshi Wang and Jaijeet Roychowdhury, Oscillator-based ising machine. arXiv.org, 2017.
Yuxin Wu and and Kaiming Hc. Group normalization. In Proc. the European Conf. on Computer Vision (ECCV), pp. 3-19, 2018.
Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Animesh Garg. Slotdiffusion: Object-centric generative modeling with diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), pp. 50932-50958, 2023.
Zhun Yang, Adam Ishay, and Joohyung Lee. Learning to solve constraint satisfaction problems with recurrent transformer. arXivorg, 2023.
Honghui Zhang, Andrew J Watrous, Ansh Patel, and Joshua Jacobs. Theta and alpha oscillations traveling waves in the human neocortex. Neuron, 1869)126611188 2018.
16

---

Published as a conference paper at ICLR 2025
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In Proc. of International Conf. on Machine learn- ing (ICML), pp. 7472-7482, 2019.
Hao Zheng, Hui Lin, and Rong Zhao. Gust: combinatorial generalization by unsupervised grouping with neuronal coherence, Advances in Neural Information Processing Systems (NeurIPS), 36, 2023.
Jiandong Zhu. Synchronization of kuramoto model in high-dimensional linear space. Physics Letters A, 377 (41):2939- 2013.
17

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
Model analysis
20
A.
Projection operator
20
A.2
Replacing the Kuramoto model with the conventional residual update
21
A.3
Number of rotating dimensions
21
A.4
The bias term C and norm form-taking term
22
A.5
Training & inference time
23
B
Additional Discussion on Motivation and Related work
23
B.1
Relation to physics models
23
Related on the NN robustness
23
Experimental settings
23
Unsupervised object discovery
................................
24
Sudoku solving
................................
28
Robustness and calibration CIFARTO
................................
29
D
Additional experimental results
31
D.1
Positional encoding for the attentive connectivity
................................
31
D.2
Unsupervised object discovery
32
D,3
Sudoku solving
36
D.4
Symmetric constraint
37
D.5
Robustness and calibration on CIFARIO
................................
38
Additional cluster visualizations
39
Proof of the Lyapunov property of our generalized Kuramoto model
44
More general proof
45
18

---

Published as a conference paper at ICLR 2025
C
X
Figure 10: The transition of the 64 oscillator neurons (N = 4). (Left) Visualiaztion of C. c, on the white region is set to 1 and the black region is set to O. (Right) Oscillators' time evolution. Sim- ilar colors indicate oscillators directing similar directions. The connectivity J a 9 x 9 convolution kernel with random filters. The oscillators the white region of C are aligned with the conditional stimuli and almost stay constant across time. The oscillators on the black region are largely influ- enced by the neighboring oscillators and exhibit wavy patterns.
RO
MLP
MLP
X T
X T
x
Conv
SA
K-Layer Conv/Attn
GELU
GN
LN
(a) Iterative conv
(b) Iterative self-attention
(c) AKOrN
Figure 11: Block diagrams of (a) ItrConv (b) ItrSA, and (c) AKOrN. GN and LN stand for Group Normalization (Wu & He 2018) and Layer normalization (Ba et al. 2016), respectively. The MLP in (a) or (b) is composed of a stack of GN or LN followed by Linear, GELU, and Linear layers. The hidden dim of MLP is set to 2x (channel size). The number of heads in SA and the K-Layer with attentive connectivity is set to 8 throughout our experiments.
19

---

Published as a conference paper at ICLR 2025
A
MODEL ANALYSIS
In this section, provide an extensive comparison of the architectural designs. Specifically, show:
A visual description of the projection operator and its effect on the performance (Sec. A.1
The Kuramoto model vs conventional residual update (Sec. A.2)
The effect of the number of rotating dimensions N (Sec. A.3)
The efficacy of and m in AKOrN (Sec. A.4).
Additionally, we show run-tima comparisons between AKOrNs and their non-Kuramoto counter- parts on different datasets in Sec. A.S
A.I PROJECTION OPERATOR
Projected update
J Jij
Projx, (J,z,)) J, - (J,زX,, x//x,
Xi
Figure 12: Visual description of Proix, (J, 0' is the angle difference between x, and J,y Given the length of JıyXj, the length of the projected update in Eq. (2) and the negative energy in Eq. 33) are inversely proportional.
Fig. 12 illustrates visual representation of the projection operator Projx, defined in Eq. (2). Note that this operator plays key role in Riemannian optimization on the sphere, ensuring that the updated direction lies within the tangent space at the point x; on the sphere.
Relation between the vectorized Kuramoto model and the original one The vectorized Ku- ramoto model includes the original one special case. Suppose the case of having a scalar connection for Jiz: J.j Jip where Jıj E IR and I is the 2 x identity matrix. Then we have = 0, 0; where 0,, 02 = arg(x:), arg(x;). From the definition of trigonometric functions, we get
2, c, = 0, and
(J,3,. x x,)x = J,,00s 0,)x:
(10)
Proix, (x²) J,yx, (J x,)x, = Jsysin(O, 0,)x!'
(11)
where x is the unit vector perpendicular to and its direction is increasing 0, Thus the Eq.
an extension of Eq. ( This proof just rephrased version of [handra et al. (2019) and Proposition I in Olfati- 2006). Please refer to them for details.
Note that with or without Proj only changes the length of the update direction of each neuron. The updated x; stays the sphere since we normalize each updated neuron to be the unit vector in Eq. (5). We test AKOrN without Proj operators and summarize the results Tab. 5 We see almost identical and bit degraded performance the CLEVR-TEs object discovery and Sudoku solving, respectively. Interestingly, without projection, the adversarial robustness and uncertainty quantification get worse than the original AKOrN.
20

---

Published as a conference paper at ICLR 2025
t Accuracy
4 ECE
Projx
FG-ARI
MBO
Proix
ID
OOD
Projx
Clean Adv CC
CC
x
79.012.5 56.2+1.0
X
99.9.0.0
45.0
X
89.9 0.1 82,4
4.5
80.5.1.5 54.910.6
100.0 51.7
84.6 64.9 78.3
1.8
(a) CLEVR-Tex
(b) Sudoku
(c) CIFARIO
Table 5: Ablation of Projx.
A.2 REPLACING THE KURAMOTO MODEL WITH THE CONVENTIONAL RESIDUAL UPDATE
Here, conduct ablation study of the Kuramoto updates. Specifically, train pro- posed AKOrN architecture on CLEVRTex and Sudoku, but without projection and normalization (Proj and II in Eqs (4) and (5) The update results in the conventional residual update. Tab, shows that the ablated model degrades both the object discovery performance and Sudoku solving signifi- cantly, which clearly shows the large contribution of the Kuramoto update to the performances.
Kuramoto FG-ARI MBO
Kuramoto
ID
OOD
66.211.6 01.4
59.8 154.6 17.1 16.66
80.5 54.9=0.6
100.0
51.7
(a) CLEVR-Tex
(h) Sudoku
Table 6: Performance with or without the Kuramoto update rule.
A.3 NUMBER OF ROTATING DIMENSIONS
Tab. and Fig. show AKOrN with = 2, which is close to the original Kuramoto model as shown in A.I significantly underfit in the object discovery experiments and the Sudoku solving. We do not observe significant improvement by increasing above and a sudden drop when N exceeds certain number depending datasets (See Fig. 16
Training Loss CLEVRTex
Training Loss Sudoku
N FG-ARI
MBO
N
ID
OOD
AKOON -4
2 44.612.5 28.0=1.
2
0.3
0.0.1.0
Loss
AKOHNIN
Loss
10-4
4 80,5+1.5 54.9=0.6
4
100.0:0.00 51.743..3
10
10-4
(a) CLEVR-Tex
(b) Sudoku
10a
Table 7: The effect of the number of rotating di- mensions. The inferior performance with N 2
10:1
50
Epoch
Epoch
comes from the model's underfitting (See the next figures)
(c) CLEVR-Tex
(d) Sudoku
Figure 14: Loss curve comparison between models with N 2 and N =
Input
Oscillators
t Accuracy Clean
+ ECE
Model
Adv
CC
CC
ResNet 0.2)
85,2 22.3 75.1
2.3
ResNet (a = 0.225)
83,9 25.5
73.6
2.6
Noisy oscillators when The three panels show the states of the os- cillators at consecutive time steps. The ani-
Figure 15:
AKOrN (N 2)
84.6 64.9 78.3
1.8
Table 8: Robustness comparison with ResNets mation file cifar10 block gif in the Supple- trained to resist Gaussian noises, a indicates the mentary Material provides more clear visual- standard deviation of the noise added during train- ing.
ization of the fluctuations.
21

---

Published as a conference paper at ICLR 2025
Figs 16 and 17 show the performance dependence the choice of N. We here test the object discovery task and the sudoku solving. We observe slight improvement as N increases, followed by sudden drop in performance in both tasks (except the Shapes dataset in the object discovery task).
55
70-
50
FGAR
Accuracy OOD test set
AODIN
50
26 32
64
126256
8
16
129256
40
Accuracy (%)
Ni
(a) CLEVR-Tex
30
N=#2
N=64
N=128
PG-00)
MBO
N=2
10-
N-512
MOON
64
Teval
(b) Shapes
Figure 17: Sudoku board accuracy on the OOD test set. The models' channel C is set to 512/N.
Figure 16: FG-ARI and MBO oscillator dimensions N. Here, the models' channel C is 256/N #layers and #(Kuramoto up- dates) are set to and 8, respectively.
A.4 THE BIAS TERM C AND NORM TAKING TERM m
AKOrN employs a two-stream architecture to process observations, which helps stabilize training. Additionally, the norm-taking part m in Eq. (6) plays key role in improving the model's fitness to the data. Fig. 18 presents a performance comparison with a model stacking only K-layers (Staking K-Layers) and AKOrN without Here, 'Stacking K-Layers' removes the term C in Eq. (4) and instead processes observation into ( by using single 3x3 convolution applied the RGB input. We see the use of C and m significantly contributes to the loss minimization. The final test accuracies of Stacking K-Layers, AKOrN wo/ and the original AKOrN are 62.9, 77.8, and 84.6, respectively.
Training loss on CIFAR10
2.0
Stacking K-Layers AKOrN wo/ m AKOrN
1.8
1.6
Loss
1.4
1.2
1.0
0.8
0
100
200
300
400
Epoch
Figure 18: Effect of the use of C and m in Eq.
22

---

Published as a conference paper at ICLR 2025
A.5 TRAINING & INFERENCE TIME
60.
1281
44..22
33.59
40-
33.40
65.50
234.00
60-
80
400
Time (ma
2.0
26.06
40
20D
20-
40
20-
200
10
IrrCome AxDeN
IrSA
AKGHN
tr5A
trrOam AKOH)
br5A
AKOV
1:54
ARGHN
DIN
AKOIN
(a) Training time (ms)
(b) Inference time (ms)
Figure 19: Training and inference time in different tasks. Training time is the time taken to complete a single gradient step (excluding data loading). Inference time is the time taken for single forward pass with a mini-batch size of 100.
B ADDITIONAL DISCUSSION ON MOTIVATION AND RELATED WORK
B.1 RELATION TO PHYSICS MODELS
Similar to how the Ising model is the basis for recurrent neural models, such as the Hopfield model (Hopfield 1982). the Kuramoto model with symmetric lateral interactions can also stud- ied by viewing it as model from statistical physics called the Heisenberg model (Mattis) (2012). In fact, we use more general version of the Kuramoto model which involves symmetry-breakin term (akin to a magnetic field interaction) and asymmetric connections between the neurons. This not only is biologically plausible (synapses are not symmetric), it also leads to much better results in our experiments.
Non-equilibrium soft matter physics has studied models with nonreciprocal interactions, for instance in the field of "active matter". They have developed accurate coarse- coarse-grained hydrodynamics models to approximate the microscopic dynamics and observed very interesting behavior, such as symmetry- breaking phase transitions and resultant traveling waves representing so-called Goldstone modes (Fruchart et al. 202T). We hope that this opens the door to a deeper understanding of these models when employed as neural networks.
B.2 RELATED WORKS ON THE NN ROBUSTNESS
Experimental proof of the conventional NNs' limited OOD generalization is represented by the
vulnerability to adversarial examples (Szegedy et al.] 2014: Goodfellow et al.] 2014). The most effective way resist such examples training the model adversarial examples generated by the model itself, which called adversarial training (Goodfellow et al., 2014 Madry et al.) 2017] Miyato al. 2018 Zhang al. 2019). Many other defenses have been proposed, but most of them were found to be not fundamental solution (Tramer et al. 2020).
One framework that can produce more human-algined predictions is a generative classifier (Ng & Jordan 2001 Bishop & Nasrabadi 2006), where we train a model with both generative and discriminative objectives or turn label conditional generative model into discriminative model based on Bayes theorem. Interestingly, different generative classifiers trained with different methods share similar robust and calibration properties (Lee et al., 2017;; Grathwohl al.) 2020 et al., 2023 Jaini et al., 2024). Generative classifiers are robust but involve costly generative training such as denoising diffusion et al.] 20233Jaini Jaini et al., 2024), MCMC (Grathwohl et al. 2020 to generate negative samples, or unstable min-max optimization as GANs training (Lee et al. 2017). AKOrN shares similar robustness properties but without any generative objectives.
C EXPERIMENTAL SETTINGS
We observe that both the readout module and conditional stimuli C are essential for stable training, especially when N 2. We also that AKOrN with N 2 exhibits strong regularity, which
23

---

Published as a conference paper at ICLR 2025
Tetrominoes
dSprites
CLEVR
Shapes
Training examples
60,000
60,000
50,000
40,000
Test examples
320
320
320
1,000
Image size
32
64
128
40
Max. #objects
6
6
4
Patch size
4
4
8
2
Patch resolution
16
16
20
Channel size
128
128
256
256
#internal steps (T)
8
8
8
4
#Epochs
50
50
300
100
Batchsize
256
Learning rate
0.001
Augmentations
Random resize and crop + color jittering
#clusters set for eval
4
7
5
Table 9: Experimental settings on Tetrominoes, dSprites, CLEVR, and Shapes.
acts positively on robustness performance while having negative effects unsupervised object dis- covery and the Sudoku-solving experiments. We show results of AKOrN with N = 4 in those ex- periments. We do not observe significant improvement by increasing N above 4 (See Fig. [[6]. Fur- ther experimental and mathematical analysis is needed to understand why this occurs, which could provide insights into how we can leverage both advantages.
Tabs 9 12 show experimental settings each dataset (e.g. hyperparameters models and opti- mization, the number of training and test examples, dataset statisties, etc...). For AKOrN, the channel size is set to (the channel size shown in the table)/N, so that the memory consumption and FLOPs are effectively the same between AKOrNs and their nor-Kuramoto counterpart baselines. All mod- els trained with Adam (Kingma & Ba 2015) without weight decay.
C.I UNSUPERVISED OBJECT DISCOVERY
We test on synthetic benchmark datasets (Tetrominoes, dSprites, CLEVR, CLEVRTex), one synthetic dataset created by (Shapes), and real image benchmark datasets (PaS COCO2017). The Shapes dataset consists of images with 2-4 objects that are randomly sampled from four basic shapes (triangle, square, circle, and diamond). Note that each image can have mul- tiple objects of the same shape together.
The kernel size of convolution layers in AKOrNconv and ItrConv noes, dSprites, and CLEVR, respectively. In addition to ItrCony and ItrSA, we also train ViT model (Dosovitskiy et al. 202T) as another baseline.
5, 7, and 9 Tetromi-
All networks process images similarly to ViT (Dosovitskiy et al. 2021). First, we patch each image into H/P W/P patches where H, W are the height and width of the image and the patch size. We then apply the stack of blocks. The output of the final layer is further processed by global max-pooling followed by a single hidden layer MLP, whose output is used to compute the SimCLR loss. We used conventional set of augmentations for SSL training: random resizing, cropping, and color jittering. We also apply horizontal flipping for the ImageNet pretraining. All models including baseline models have roughly the same number of parameters and are trained with shared hyperparameters such as learning rates and training epochs. See Tabs 9 II for those hyperparameter details.
24

---

Published as a conference paper at ICLR 2025
CLEVRTex
OOD
CAMO
Training examples
40,000
Test examples
5,000
10,000
2,000
Image size
128
Max. #objects
10
Patch size
8
Patch resolution
16
Channel size
256
# internal steps (T)
8
# epochs Batchsize
500
256
Learning rate Augmentations
0.0005
Random resize and crop + color jittering
#clusters set for eval
11
Table 10: Experimental settings on CLEVRTex and its variants (OOD, CAMO). We also train large AKOrN model that is trained with the doubled channel size, and epochs. We denote that model by Large AKOrN.
In AKOrN, ((0) computed by the patched features of images, while each x, is initialized by random oscillators sampled from the uniform distribution the sphere. We use the identity function for g in each readout module. In multi-block models, we apply Group Normalization (Wu He 2018 to except for the last block's s output C(L).
For the Tetrominoes, dSprites, and CLEVR datasets, train single-block models with T = 8. We observe that stacking multiple blocks does not yield improvements on those three datasets. CLEVRTex, we train single- and two-block models with attentive connectivity and T = 8, while on ImageNet, we train a three-block AKOrN model with attentive connectivity and =
C.1.1 METRICS
We use FG-ARI and MBO to evaluate cluster assignments, both of which are well-used metrics in object discovery tasks. Below the summaries of the FG-ART and MBO metrics.
The Adjusted Rand Index (ARI) computes how well the clusters align with ob- ject masks compared random cluster assignments. The foreground ARI (FG-ARI) only considers foreground objects and a well-used metric in object discovery tasks. The max- imum value of 100 indicates perfect alignment between the obtained clusters and the ob- ject masks. If the cluster assignment is completely random or all features are assigned to the same cluster, the value is O.
FG-ARI:
MBO: The Mean Best Overlap (MBO) first assigns each cluster to the highest overlapping ground truth mask and then computes the average intersection-over-uiiie
(IoU) of all pairs. The value takes 100 at maximum. Following the literature, we exclude the background mask from the MBO evaluation. Since MBO computes IoU, tightly aligned object masks give a higher value than FG-ARI (FG-ARI does not penalize the mask extending into the background region).
25

---

Published as a conference paper at ICLR 2025
ImageNet
PascalVOC
COC02017
Training examples
1,281,167
Test examples
1,449
5,000
Image size
256
256
256
Patch size
16
Patch resolution
Channel size
768
Blocks
# internal steps (T)
epochs
400
Batchsize
512
Learning rate
0.0005
#clusters set for eval
7
Table 11: Experimental settings on ImageNet pratraining and the PascalVOC and COC02017 evaluation. For SimCLR training augmentations, we use random resize and crop, color jittering, and horizontal flipping-
Sudoku(ID) (Wang et al.
2019
SudD (Palm et 2018
9
6
3
6
8
4
3
2
7
6
9
4
5
9
1
4
Training examples
9,000
Test examples
1,000
18,000
Channel size
512
epochs
100
Batchsize
100
Learning rate
0.0005
Table 12: Sudoku puzzle datasets.
26

---

Published as a conference paper at ICLR 2025
1
2
Network
Tiling
3
3
4
Figure 20: 2x up-tiling. First, we create horizontally or/and vertically shifted images with stride equal to (patchsize/2) and compute the model's s output each shifted image. We then interleave each token feature to make 2x upsampled feature map.
C.1.2 UPSAMPLE FEATURES BY UP-TILING
When we compute the cluster assignment, we upsample the output features by up-tiling. this ap- proach, the model processes set of slightly shifted versions of the input image along the horizon- tal and/or vertical axes. These shifted images allow the model to generate slightly different feature maps for each shifted position. The final higher-resolution feature map is then created by interleav- ing these feature maps, effectively combining the shifted perspectives into a single, more detailed representation. This up-tilling enables us get finer cluster assignments and substantially improves the object discovery performance our AKOrN. We show a pictorial explanation in Fig. 20 and PyTorch code in Code 1. In Fig. 21 compare up-tiled features with the original features and features with bilinear upsampling. Fig. 22 shows some examples of up-tiled features. We apply up- tiling with the scale factor of 4 for producing numbers on Tabs | and well for cluster visual- ization in Figs 5 and 3 36 Unless otherwise stated, no upsampling is performed when com- puting the cluster assignment.
27

---

Published as a conference paper at ICLR 2025
Code 1: PyTorch code for up-tiling
def Create_shifter jings (img, psize, stride):
H, w sing.shape. [-2:]
img = F. interpolate (img,
(#+psize-stride, R-psize-stride),
mode=
align_corners-aalsg
imgs = []
for h in range (0, psize, stride):
for w in range (0, psize, stride):
mmgs.appent (ingl:, ;, h:htH, w:w+H]
return imgs
def uptiling (model, images, psize=16, s=2): в**
Args:
model: function that takes [B, C, H, W] -shaped tensor
and outputs [B, C, C/pssize,, W/psize] -saaped tensor,
images: a tensor of shape [B, C, H, W].
psize: the patch size of the model.
s: scale factor. The resulting features will be upscaled to [s*H/psize, s*R/psize] where (H, W) are the original image size. Must be equal to or less than the patch size.
Returns:
nimgs: tensor of shape [B, C, s*H/psize, s*N/paize]
images. shapee [0]
stride = psize //
Create shifted images.
shifted imgs create
greate_shifted shifted imgs (images, psize, stride)
Compute a feature map on each shifted image.
outputs
for i in range (len with torch no. norch.no_grad.e ():
output = model (shifted_inge [i] ]. cuda ()
outputs.appent wottpu..detaci cpu )))
# Tile the output feature maps.
oh, оw outputs shape[-2:] [shap-s]
nimgs = borch.zeroo (B, outputs [0] shape(1), oh, ow, s)
for in range (s)
for in range (s)
nimgs!:: ;, :, h, :, w] outputs [h*stw] Reshape into [B, C, (H/psize), s (N/psize) nimgs = nimgs.view view (, -1, oh+nh, ow=nw)
return nimgs
C.2 SUDOKU SOLVING
The task to fill 9x9 grid, given some initial digits from 9, so that each row, column, and 3x3 subgrid contains all digits from to 9, While the task may be straightforward if the game's 's rules are known, the model must learn these rules solely from the training set. Example boards are shown in Tab. 12
We train AKOrN with attentive connections, the ItrSA model, and a conventional transformer model. We denote them by Aarr ItrSA, and Transformer, respectively. AKOrAmttn has almost the same architecture used in the object discovery task except for g in the readout module, which is composed of the norm computation layer followed by stack of ReLU, and linear layer.
28

---

Published as a conference paper at ICLR 2025
PCA1-3
PCA4-6
PCA7-9
Bi-linear features
Input
Up-tiled features
(a) The original features (top) and features upsampled by up-tiling (bottom).
PCA1-3
PCA4-6
PCA7-9
Original
features
Up-tiled features
(h) Features upsampled by bilinear upsampling (top) and by up-tiling (bottom).
Figure 21: Comparison of AKOrN's output features upsampled by different methods. PCA(i j} indicates that the corresponding column's panels represent the features' i-th to j-th PCA compo- nents. The scaling factor of up-tiling is set to 8.
The input for each model is 9x9 digits from to 9 (0 for blank, 1-9 for given digits). We first embed each digit into 312-dimensional token vector. The 9x9 tokens then flattened into 81 tokens. We apply each model to this token sequence and compute the prediction on each square by applying the softmax layer to each output token of the final block. All models trained minimize cross- entropy loss for 100 epochs.
The number of blocks of both ItSA and AKOrN is set to one. We tested models with more than one block but found no improvement on the ID test set and a decline in OOD performance. Similar the object discovery experiments, transformer results in even worse performance than the ItrA model (Tab. 19).
C.3 ROBUSTNESS AND CALIBRATION ON CIFARIO
We train two types of networks for this task: convolution-based AKOrN and AKOrN with a combi- nation of convolution and attention. The former has three proposed blocks, and all of the Kuramoto layer's 's connectivities are convolutional connectivity. The kernel sizes are 9,7, and from shallow to deep, and T' is set to 3 for all blocks. Between consecutive blocks, single convolution with stride being is applied to each of and X. Thus, the feature resolution the final block's output is 8 8. Each readout module's g Batch Normalization (loffe & Szegedy 2015) followed by ReL.U,
29

---

Published as a conference paper at ICLR 2025
PCA1-3
PCA4-6
PCA7-9
Input
(a) CLEVRTex
PCA1-3
PCA4-6
PCA7-9
Input
(b) PascalVOC
Figure 22: Up-tilied feature maps on CLEVRTex and VOC, The scale factors are set to 8 and 16 for CLEVRTex and VOS respectively.
30

---

Published as a conference paper at ICLR 2025
shot noise
gaussian noise
impulse noise speckle noise
gaussian blur
mation blur
defocus blur
glass blur
zoom blur
frost
pixelate
fog
brightness
jpeg compression
elastic transform
contrast
saturate
spatter
Figure 23: Example images the Common Corruption dataset (CIFARIO-C). The top right image is the original clean image.
and 3x3 convolution. C33) is average-pooke pooaleee-poeedddd followed by the softmax layer that makes category predictions. The latter network is identical to the former one except for the third block, which we replace with the block with attentive connectivity. For this attentive model, different timesteps T set across different blocks, which are [6, 4, 2] from shallow to deep.
For ResNet-18 18 and AKOrN, first conduct pre-training the Tiny-imagenet (Le & Yang 2015) dataset with the SimCLR loss for 50 epochs with batchsize of 512. We observe that this pre-training effective for AKOrN and improves the CIFARIO clean accuracy compared training from scratch (from 87% to 91 %). The ImageNet pretraining slightly improves ResNet's clean accuracy (from 94.1% to 94.4%). Each model is then trained on CIFAR10 for 400 epochs. We apply augmenta- tions, including random scaling and cropping, color jittering, and horizontal flipping, along with AugMix (Hendrycks al. 2020), as commonly used robustness benchmarks. Both models are trained to minimize the cross-entropy loss.
We also train an ItrConv model as our-moooooto counterpart for this robustness experiment. To construct the ItrConv model, We replace each block of AKOrNeonv with the ItrCovv block shown in Fig. IT and set the same kernel size to each layer as AKOrNsonv (i.e. 9, 7, and 5 from shallow to deep layers). Hyperparameters such as the number of channels, learning rate, and others are shared with AKOrAcoov
D ADDITIONAL EXPERIMENTAL RESULTS
D. POSITIONAL ENCODING FOR THE ATTENTIVE CONNECTIVITY
We need positional encoding (PE) for AKOrN with attentive connectivity. We found GTA-type PE (Miyato al. 2024 effective and used for AKOrN throughout our experiments. Comparison to absolute positional encoding (APE) Vaswani et al. 2017) and RoPE (Su et al. 202T) is shown in Tab. 13 GTA does not improve the baseline InSA models.
31

---

Published as a conference paper at ICLR 2025
CLEVRTex
PE
Sudoku(OOD)
PE
FG-ARI
MBO
APE
34.415..
APE
66.9
42.2
ItrSA
ItrSA
GTA
24.3
GTA
66.1
43.4
APE
48.1
APE
72.0
51.4
AKOrN
RoPE
48.415.6
AKOrN
RoPE
65.7
50.2
GTA
51.713..
GTA
75.6
57.7
(b) Sudoku (OOI) Test)
CLEVRTex
Table 13: Comparison of positional encoding schemes. The number of blocks is one for all mod- els. The Sudoku results of AKOrNs obtained with test-time extensions of the Kuramoto steps (Teval = 128) but without the energy-based voting.
D.2
UNSUPERVISED OBJECT DISCOVERY
D.2.1 MBO ON THE SYNTIEETIC DATASETS
Fig 24 shows AKOrNzon and AKOrNatto outperform their counterparts on almost every dataset in terms of MBO, as well as FG-ARI shown in Fig 3
60
MBO
40
20
Tetros.
dSprites
CLEVR
Shapes
ItrConv
ltrSA
AKOrN cony
AKOrNeitn
Figure 24: MBO on Tetrominoes, dSprites, CLEVR, and Shapes.
D.2.2 MBO; vs CLUSTERS
Fig. 25 shows AKOrN outperforms the other SSL methods across wide range of the numbers of clusters, demonstrating AKOrN's robustness to variations the number of clusters on object discovery performance.
0.300
0.50
0.48
0.295
MBO
0.44
MBO-
0.285
0.42
0,280
0.40
AKOrN
0.275
AKO
0.38
DINO
DINO
MoCoV3
0.270
MoCoV3
0.34
0.265
15 45
10
20 Number of clusters
7 10
15
Number of clusters
(a) PaC
(b) COC02017
Figure 25: MBO; vs the number clusters used for evaluation.
32

---

Published as a conference paper at ICLR 2025
D.2.3
FULL TABLES OF OBJECT DISCOVERY PERFORMANCE
Tabs 14 18
show extended comparisons between AKOrN and existing models.
Tetrominoes
dSprites
CLEVR
Model
FG-ARI MBO FG-ARI MBO
FG-ARI
MBO
ItrConv
59.012.9 51.612.2
29.1 38.515.2 49.315.1 29.713.0
AKOrNcouv
76.4 51.9+1.5
63.8+7.7 50.7+4.7 59.0+4.3 44,4+2.0
ItrAA
85.8.0.8 54.913.4
68.1 18.4
63.0
82.511.7
39.411.
AKOnNatta
88.6+1. 56.4
78.3+1.3
63.0
91.0+0.5
45.5+14
(tup-tiling (x4))
AKOrNattn
93.1 56.3+v.0
87.1
60.2+1. 94.6+0.7 44.7
Synchrony-based models)
CAE (Löwe et al. 2022
78
51
27
CICAB (Sanic al. 2023)
8449
56=11
54+2
SynCx (Gopalakrishaan et al.
2024)
8941
82+1
59+3
Rotating Features (Löwe al. 2023)
4219
88.811.5
86.3
66.411.. 60.8
(Slot-based model)
Slot-Attnetion (Locatello et al. 2020) 99.5+0.2
91.3 #0.3
98.8.0.3
Table 14: Object discovery results on synthetic datasets, We show the mean and std of the metrics of models with different randomm seeds for the weight initialization.
Input
ItrSA
AKOrN
GTmask
Input
ItrSA
AKOrN
GTmask
Db
8
(b) Failure cases
(a) Comparision between ItrA and AKOrN
Figure 26: Cluster visualization on Shapes. (b) Both ItrA and AKOrN sometimes fail at separating overlapping objects with complex configurations.
L
Model
2
Model
FG-ARI
MBO
IttSA
48.9
49.3+2.5
57.0+5.5
ItrConv
21.0+0.8
18.7=0.5
AKOrNattn
52.5
65.5+2.
71.3
AKOrNcomv
46.6+2.1
30.0=0.6
(c) FG-ARI
ItrA
57.0+5.5
34.8=1.6
L
AKOrNattn
71.3144.2
46.9=1.4
Model
2
Table 15:
Object discovery performance on
ItrSA
38,8+1.3 37.2+2.1 34.8++1.6
Shapes. We show the mean and std of the met- rics of models with different random seeds for the weight initialization.
AKOrAattn 40.2
44.6:1.2
46.9
(d) MBO
Table 16: Object discovery performance on
Shapes varying the number of layers L.
33

---

Published as a conference paper at ICLR 2025
CLEVRTex
OOD
CAMO
Model
FG-ARI
MBO FG-ARI
MBO
FG-ARI
MBO
VIT (I. = 8, T
46.410.6
25.1 44.1 20.5 27.2
32.5.0.6 16.1
ItrSA (I, =1 1)
65.7+0.3
44.6=0.9
64.620.8 45.1
49.0.0.7
30.240.8
ItrSA (L 2)
76.3+0.4
48.5=0.1
=4.9 46.4
61.9113 37.1
AKOrNattn (L 1)
75.610.2
55.0
73.4 56.1.1.1
59.9.0.1
44.310.9
AKOrNatta (L = 2)
80.5+15
54.9=0.6
79.2#1.2 55.7=0.5
67.7+15 46.2+0.9
(+up-tiling (x4))
AKOrNatta (L = 2)
87.7+1.0
55.3=2.1
85.2
55.6+1.5
74.5
45.6+3.4
Large AKOnNattn (L = 2)
88.5
59.7=0.9
87.7 60.8
77.010.5 53.4
*MONet Burggss
2019)
19.8+1.0
37.3+1.0
31.5+09
"SLATE Singh et al. 2022
44.2+NA N
Sloot-Atetion locatello et al.
2020)
62.412.3
58.5
57.5110
Slot-diffusion Wu et al. 2023
69.7 61.91NA
SSLAEE+ (Singh et al. 2022
70.7 54.9=NA
$1.SD (Jiagg et al.
2023)
76.4NNA 72.4+NA
Slot-diffusio+-BBO
et
2023
78.5+NA 68.7+NA
*DTI Monnierr et a 20211)
79.9+1.37
73.7=1.0
72.9+19
*I-SA Chang al.
2022
79.013.9
83.710.9
57.2.13.3
BO-SA (Ja et al. 2023)
80.5
-6..50.2
63.716.1
NNS (Dedhia & Jhal
2024)
89.9+0.0 46.6=0.0
ISA-TS Bizza
2023)
92.9+0.4
84.4
86.2408
Jung et al. (2024)
93. NA N5,4+NA
Sauvalle & de La Fortelle (2033)
94. 40.5
83.1
87.3+3.8
Table 17: Object discovery on CLEVRTex (Karazija et al. 2021 Use Openimages (Kuznetsova
et al. 2020) -pretraine encoder. Numbers are from Jung et al. 2024). *Use prrerane- DINO, "Numbers taken from Jia et al. (2023). PUse Imagenet
maagenct-preerained pretrained backbone models. We show the mean and std of the metrics of models with 3 different random seeds for the weight initial- ization.
34

---

Published as a conference paper at ICLR 2025
PascalVOC COC02017
Model
MBO, MBO, MBO; MBO,
(slot-based models)
Slot-attention (Locatello et al. 2020
22.2
23.7
24.6
24.9
SLATE (inghh al. 2021
35.9
41.5
29.1
33.6
(DINO + synchrony-based models)
Rotating Features (Löwe et al. 2023
40.7
46.0
(DINO slot-based model)
NSI (Dedhia Jha 2024)
28.1
32.1
DINOSAUR (Seitzer 2023
44.0
51.2
31.6
39.7
Slot-diffusion (wu et al. 2023
50.4
55.3
31.0
35.0
SPOT (Kakggorrgiou et
2024)
48.3
55.6
35.0
44.7
(SSL models)
MAE (He et al. 2022)
33.8
37.7
22.9
28.3
DINO Caron et al. 2021
44.3
50.0
28.8
35.8
MoCo 3 Chen et al. 2021a)
47.3
53.0
28.7
36.0
AKOrN rasth
50.3
58.2
30.2
38.2
(SSL models + up-tiling (x4))
MAE
34.0
38.3
23.1
28.5
DINO
47.2
53.5
29.4
37.0
MoCoV3
44.6
50.5
29.0
35.9
AKOrNatin
52.0
60.3
31.3
40.3
Table 18: Object discovery on PascalVOC and COCO2017.
D.2.4 TRAINING EPOCHS VS MBO
Fig. 27 shows that MBO, and MBO, scores on Pascal and COCO improve as ImageNet pretraining progresses. Similar observations made CLEVRTex datasets, where larger AKOrNs give better object discovery performance (see Figs 32 34 and Tab. [7]. These results indicate that there alignment between the SSL training with AKONN and learning object-binding features and that increasing parameters and computational resources can further enhance the object discovery performance.
50
30
MBO,
MBO-
47
29
46
45
50
44
48
28
35
100 200
300 400
100 200
300 400
100
300 Epoch
200
400
0
100
200 Epoch
300
400
Epoch
Epoch
AKOrNattn
DINO
MoCov3
Figure 27: MBO, and MBO, vs. training epochs. (Left) Pascal VOC (Right) COC02017.
35

---

SAT-Net WWang et al. 2019) Published as a conference paper at ICLR 2025
D.3
SUDOKU SOLVING
D.3.1
FULL TABLE OF BOARD ACCURACY
Model
ID
OOD
Energy Transformer (Hoover et 2023
1.0+
0.040.0
"Symmetrized AKOrN (1.00((1== 1,7
T6)
84.6114.2
1.4
Transformer
98.6+0.3
5,2+0.2
ItSA (L = 1, 16)
99.7=0.3
14.1=2.7
AKOrNattn wo 5 [L 1,7 16)
99.810.1
16.612.2
AKOrNatta (L 1,7 16)
99.8+0.1
16.6#2.1
(+Test time extensions of internal steps)
ItrSA (Teval = 32)
95.7+8.5
34.4
AKOrNattn wo 5 (Teval = 128)
100.0:0.0
49.6#3.3
AKOrNattn (Teval 128)
100.0
51.743.3
(Teval 128, Energy- voting (K 100)
AKOrNattn wo S2, EToval
100.0
46.8=9.0
AKOrNattn Expnaa AKOrNatta E, Er
100.010.0
7..0
100.0+0.0
81.6+1.5
(Teval 512, Energy-based voting (K 1000))
AKOrNattn, E. Fr
100
89.5
98.3
3.2
Diffusion (Du et al. 2024)
66.1
10.3
IREM (Du
93.5
24.6
RRN Palm et al.
2018
99.8
28.6
R-Transtormer Yang el 2023
100.0
30.3
IRED
(Du et al.
2024)
99.4
62.1
Table Board accuracy on Sudoku Puzzles. The harder dataset (OOD) has fewer conditional dig- its per example than the train set (17-34 in the harder dataset while 31-42 in the train set). We show the mean and std of the accuracy of models with different random seeds for the weight initializa- tion. "Numbers calculated with excluding energy-based voting, we found the sum of energy values across timesteps (E; Eg) indicate board correctness better than the energy at the last time step (Erwaa).
trained model that has stuck during training. For
36

---

Published as a conference paper at ICLR 2025
D.3.2
EFFECT OF THE NATURAL FREQUENCY TERM IN ENERGY- BASED VOTING
Waryy
1009
(a) without &
(b) with 22
Figure 28: Energy distribution of the K-Net with or without the Z term. In each panel, given a single board, compute energies of the final oscillatory states that start from different random oscillators and show the histogram of these energies, color-coded by the correctness of the predictions made on the corresponding final oscillatory states.
Interestingly, the model without the term does not give improvement with the energy vote, as the energy value and correctness
inconsistent (Fig. 28). This implies the asymmetric term Ω prevents the oscillators from being stuck in bad minima. We show the AKOrN's 's board accuracy without the N term in Tab. 19 and see that the model's performance degrades with the energy vote (49.6 46.8).
D.4 SYMMETRIC CONSTRAINT
Fig. 29 shows a comparison of AKOrN to Energy Transformer (Hoover et al.) 2023) and symmet- ric version of AKOrN. The symmetric version AKOrN constructed by using the same weight to compute query and key vectors and symmetric weight for value vectors, Board accuracies of these symmetrized models are shown in Tab. 19 We observe a similar tendency the two symmetric models: both models underfit the data (See Fig. 29). Energy Transformer is not able to solve even in-distribution boards. Symmetrized AKOrN also gets stuck depending on the seed for the weight initialization.
Training Loss on Sudoku
101
Energy Transformer
Symmetrized AKOrN
10
AKOrN
Loss
10-1
10-2
10-3
25
75
100
Epoch
Figure 29: A training curve comparison with symmetric transformer models.
37

---

Published as a conference paper at ICLR 2025
D.5 ROBUSTNESS AND CALIBRATION ON CIFARI0
1 Accuracy
ECE
Model
Clean
Adv
CC
CC
Gowal ct al.
2020)
85.29
57.14
69.1
13.2
Gowal et 2021
88.74
66.10
70.7
5.6
Bartoldson et al. (2024
93.68
73.71
75.9
20.5
Kireev et 2022
94.75
0.00 83.9
9.0
Diffenderter et al. (2021)
96.56
0.00 89.2
4.8
VIT
91.44
0.00
81.0
9.6
ResNet- 18
94.41
0.00 81.5
8.9
ItrCnn
93.46
0.00 83.6
5.9
AKOrNCoDV (N = 2)
88.91
*58.91
83.0
1.3
AOXXX (N = 2)
91.23
*51.56 86.4
1.4
AKOrNmix (N 4)
93.51
*0.00 84.0
6.4
Table 20: (The extended table of Tab. 4 Robustness to adversarial attack (Adv) and Common Cor- ruptions (CC) on CIFARIO with the most severe corruption level (5). "The adversarial attack is done by AutoAttack with HoT (Athalye et al. (0118) The max norm constraint of the adversrial per- turbtions is set 8/255. With N = 4, the performance tendency of AKOrN is almost the same as ResNet except for the accuracy and uncertainty calibration on CIFAR10 with natural corruptions, which are moderately better with AKOrNmix
d
predsplane
pred:bird
pred:car
pred:frog
pred:ship
pred:horse
pred:frog
predsdog
Figure 30: AKOrN's adversarial examples are interpretable. Each pair of images is an original and the adversarially perturbed image ([|e||o> 64/255). The text above each image indicates the class prediction made by the AKOrN model.
38

---

Published as a conference paper at ICLR 2025
E ADDITIONAL CLUSTER VISUALIZATIONS
Input
shallow
deep
GTmmask
(a) Shapes
Input
shallow
deep large
GTmask
(b) CLEVRTex (Ist and 2nd row), CLEV (3rd and 4th row), and CLEVRTex-CAME (the row)
Figure 31: Deeper, wider, and more epochs make the models learn more binding features in AKOrN. (a): comparing a single-layer model (shallow) and a 3-layer model (deep), (b): comparing a single- layer model (shallow) and model with doubled layers, channels, and epochs (deep+large).
39

---

Published as a conference paper at ICLR 2025
Input
ItrSA
AKOrN
Large AKOrN
GTmask
Figure 32: Visualization of clusters on CLEVRTex. The number of blocks L is set to two for all models.
Input
ItrA
AKOrN
Large AKOrN
GTmask
Figure 33: Visualization of clusters on CLEVRTex-OOD. The number of blocks L is set to two for all models.
40

---

Published as a conference paper at ICLR 2025
Input
ItrSA
AKOrN
Large AKOrN
GTmask
Figure 34: Visualization of clusters CLEVRTex-CAMO. The number of blocks L is set two for all models.
41

---

Published as a conference paper at ICLR 2025
Input
DINO
MoCoV3
AKOrN
GTmask
Figure 35: Visualization of clusters on VOC. The number of clusters is set to 4.
42

---

Published as a conference paper at ICLR 2025
Input
DINO
MoCoV3
AKOrN
GTmask
FRIOS
Figure 36: Visualization of clusters COCO2017. The number of clusters is set
43

---

Published as a conference paper at ICLR 2025
F PROOF OF THE LYAPUNOV PROPERTY OF OUR GENERALIZED KURAMOTO MODEL
Under the assumptions: J., = J,JI, = J3 E = Ω and Qc. = 0, we will prove below that Eq (3) is Lyapunov function for the dynamics defined in Eq (2).
First, we compute the time derivative of E: dE
dt -- -
0E
Σ JojKy + Ci. xi)
Ox;
By substituting x; from the model, dF -
Σ
Jajx, cc, Sx + Proix: Edio + tci))
dt
which splits into two sums:
d. = = - Mi + cc, sxx:)
-E Jip + c Projx. ( xy + +c,))
dt
(I)
For the term (I), consider
[ Jiy xj + ci, s2x.)
We separate the c, part:
E(e sxx;) = - cef Ωx,
Since Z c2 = O by assumption, follows that ca Ω = Hence those terms vanish. Next, for the E; JiyX, part:
ΣΣ J x x Ux
Since Jij = Jje is symmetric but S2 is skew-symmetric (i.c. S2 -S2), one has Jip x sxx, = Jij x3 Ω sx-
x!" Ωx, - x Ωx,,
Thus the double sum cancels term by term:
ΣΣ Ju sxx 0.
Therefore, The term (I) is 0.
We are left with
Term (II) - ( Juj c,, Projx, +
Recall that Projx. (y) is (by definition) the projection of y onto the subspace orthogonal to x,. In particular, if ||x]|| 1, one has
y (y) |roijs (y)]l 2 0 for all y. Hence each inner product (y, Proix, (y)) 2 0.
Since there is minus sign in front of this term in the expression for dE we conclude Term (II) 0.
(see the original equation), dr
Putting Term (I) and Term (II) together,
dE
Term (I) + Tenn (II) < 0.
dt
Therefore, dE < 0. proving that E is indeed Lyapunov function for the given dynamics.
44

---

Published as a conference paper at ICLR 2025
G MORE GENERAL PROOF
One can obtain natural sufficient conditions for energy function that ensure it is non-increasing on trajectory.
We define the block matrices J and as follows:
J
J12
Jic
Ω
ONNN
J21
J22
J20
==
O ONXN
Zc
Jen Jcz
Joa_
We similarly define e and ac Proposition G.1 (sufficient conditions, non-constructive). Suppose that the following conditions hold
1. Block matrix symmetric, i.e., Ju J J. JJ.
2. Block matrix Q is block-diagonal and antisymmetric;;
3. Block vector с is in the kernel of block matrix Q,, i.e., 24c; 0.
Energy function non-increasing on trajectories of dynamical system 2
JΩ DJ 2 0.
(12)
that is commutator [J is positive semi-definite matrix.
Proof. Let P is block-diagonal matrix with elements Pa 1,-1--0 x, Using block matrices we
can rewrite dynamical system (2) and energy function (3) as fx + (c Jæ) and E(x)
x Jx - c. æ. Using this notation we obtain for the derivative of the energy
dE((w) dt
(OE) 0x
2 x Já c x Jfe D. JP (c J ax P Jæ).
The red term is zero by assumption. Blue terms can be combined together to the expression y Py where y = (c + Jæ). This expression is always non-negative given that P is an orthogonal projec-
The remaining term xx SΩx can be transformed f (JR+ æ. Using Ω =
and J we obtain x T J f2x J T [J, .2] æ. If this commutator is positive-defiitte matrix can be sure that the energy is non-increasing.
The condition above is not especially convenient since it does not tells directly which matrices J and ! are allowed. An example of more direct result given below.
Proposition G.2 (sufficient conditions, constructive). Commutator [J, S2] 0 and energy function
3) non-increasing on trajectories of dynamical system 22) if J S normal matrix. One
specific example be f I & Z and J I.
Proof. We know that J + S2 is normal matrix. Using standard definition of normality we obtain
= (J + 22) [ 12) - (J- (2) (J + .2) = 2 [
matrices and J clearly commute since they are non-trivial in different factors of Kronecker product.
For special choice I @ ! and J
The special case from Proposition G.2 is worth writing explicitly in the original notation. All oscil- lators have the same natural frequencies S, Ωt and coupling matrices J realise multiplication by scalar, i.e., Joj kig
45


Reviews:
[{"review_id": "CbCjclI4cf", "reviewer": "Reviewer_uqq5", "summary": "In this work, the authors proposed to use Kuramoto oscillatory neurons (AKOrN) to replace the usual thresholding units in deep neural networks. The basic neuron architecture can be applied to layers with different connectivity patterns, including convolutional and self-attention layers. Models with Kuramoto oscillatory neurons show promising performance in visual discovery and reasoning tasks, they are also more robust to adversarial attacks and input corruptions.", "questions": "1. Judging from the method, AKOrN seems to be much more computationally costly than the usual thresholding units. I noticed that the authors controlled the number of channels in AKOrN models, but showing the training/inference time would be helpful.\n2. It would be helpful to include some explicit discussion on the weakness of the method. i.e., what tasks are probably more suitable for this method and why?\n3. As mentioned above, more analysis might help establish the advantage of AKOrN. For example, the paper shows that the AKOrN models show better robustness, but it is not clear from the design that they are more robust. Thus analysis of the network dynamics on some toy tasks might be helpful (e.g., synthesized classification tasks on low-dim vectors).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The neuronal architecture draws inspiration from physics and neuroscience and is very novel in deep learning.\n2. Model performance is impressive across a range of tasks.\n3. Extensive experiments show that the method applies to mainstream architectures including convolutional neural networks and transformers.", "weaknesses": "1. While AKOrN shows promising performance on the selected tasks, the model might not work as well on more \"classical tasks\" such as image classification. \n2. Although the motivation of AKOrN is clear from a neuroscience perspective, how, and why the model shows superior performance in the tested task is not well understood.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RwnZxKfHUz", "reviewer": "Reviewer_ynsZ", "summary": "the authors propose a neural ntwork architecture in which units are high-dimensional Kuramoto oscillatory neurons. This architecture leads synchronization or 'binding' between neurons which enables competitive performance in object discovery, reasoning and robustness.", "questions": "- The format of figure references could be improved. I would recommend references as 'Fig. X'. Currently there is either the period missing or there's no space (e.g. 'Fig1' on line 96). Same with Tab X -> Table X. Should also be Figs for multiple figures (not Fig as in line 368)\n- line 36: 'we follow a more modern dynamical view of neurons as oscillatory units'. Is this correct? Oscilliations have been studied in the brain for a long time; moreover, I would say that the majority of computational neuroscientists currently model neurons as scalar values.\n- For table 3 results are presented with error bars over 5 seeds (which I favour), why is this not the case for any of the other results presented?\n- line 121: 'we introduce a multi-dimensional vector version of the model' - this line is confusing as this was already employed by Chandra et al. and Lipton et al. (as cited). Perhaps replace with 'incorporate a mutli-dimensional model...'. On this subject could you clarify to me the exact novelty of the model? Is it that it's not been introduced in deep neural nets before? Or is it about the fact connections between oscillators J is not symetric in AKOrN? Relatedly, line 308 'our..models differ from these approaches in various aspects such as...' should really provide all fundemental differences beyond some examples. In fact, in any case what's the difference between 'asymmetric connections' and 'their symmetry breaking term'? \n- line 123 (and other places): what's C? the size of the neural population? this should be clarified\n- line 245: what is 'proper' energy?\n- line 323: I don't understand how these convolution/self-attention layers are iterated upon. What are the shared parameters?\n- I didn't understand the implementation for the sudoku task. 1. is the entire 9x9 soduku grid fed all at once, or one digit at a time? and 2. how come x_i takes the value c_i when a digit is given, does the model no longer follow equation 2? Or is this just the initial value taken before the Kuramoto steps?\n- line 425: 'the energy is a good indication of the solution's correctness'. Why? I can believe that stability is often a desirable thing in tasks but think this line deserves some elaboration\n- line 464: would it be possible to give a brief description/overview of what 'common corruptions' are?\n- line 506: why is it surprising that confidence and accuracy are linearly correlated? \n- line 527: 'the oscillator...'. I didn't understand the logic of this sentence, could you explain it to me please?\n\nOther comments:\n\n- the notation x_{chw} is confusing, I think x_{c,h,w} is better\n- on line 201 the terms should be switched around (the initial state of the next block is the thing being defined). But also, how does this fit in with line 189 where X^(l,0) = X^l?\n- the metrics in all of the tables should be clarified. I presume the unit is %, but e.g. in the case of Table 1 I'm not entirely sure what the metric is. % objects correctly identified?\n- in the results section the language can be overly strong. For example 'far better' (line 377) and 'vastly better' (line 416) do not appear to me as very scientific language. \n- in the discussion there is talk of small N resulting in a strongly regularized model. Given that the size of N nor regularization was mentioned in the prior text, this comes rather abruptly as is confusing to the reader\n\ntypos: \n- line 150: 'set C a silhouette' -> 'set C as a silhouette'", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- the model is interesting and well defined\n- the quantitative results appear impressive, though I am have some issues with their presentation (see below)\n- the many numerical results presented, and comparisons to other models, suggests that overall a good deal of thought/time has been put in the manuscript", "weaknesses": "- there are numerous minor issues with the text/presentation (see question below) which impacts the clarity of the paper \n- This not a field I have experience in, but I am not entirley convinced a fair comparison was made to other models for object discovery. Is the idea that AKOrN is a highly competitive model for models 'trained from scratch', and that comparing it to models with some pretrained parameters (e.g. Lowe et al. 2024) is unfair? I think this could be better clarified if so\n- there is a notion from the abstract and sections 1/2 that this work is relevant to neuroscience, but the link feels quite tenuous to me. Is it reasonable to think of neurons as high-d points on a sphere?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TL7uSW9l6z", "reviewer": "Reviewer_AtLx", "summary": "- This paper developed a feature-rotation-like neuron models (high-dim feature rotation encodes synchrony binding), a promissingly neuron-level building block that can be combined with diverse network architectures, like CNN and transformer. \n- The capability of the new architecture is validated on three independent tasks: object discovery (synthetic & natural image), reasoning on Sudoku task, robustness w.r.t adversarial attack in CI\nFAR10. \n- On each task, the proposed model shows SOTA performance and interesting shining points: (1) For object discovery, the model scales to natural image without pretrained DINO model; (2) for reasoning, the model's correctness is naturally implied by the \"internal coherence\", indicated by the task-independent energy measure; (3) for adversarial attack, robustness seems a built-in property of the dynamics and again, the energy could imply the accuracy.", "questions": "(1) For the claim in the paper: \n> “binding” between neurons leads to a form of competitive learning where representations are compressed  in order to represent more abstract concepts in deeper layers of the network. \n\n> In fact, neighboring neurons tend to cluster their activities, and clusters tend to compete to explain the input. This “competitive learning” has the advantage that information is compressed as we move through the layers, facilitating the process of abstraction by creating an information bottleneck. Additionally, the competition encourages different higher-level neurons to focus on different aspects of the input (i.e., they specialize). This process is made possible by synchronization: like fireflies in the night, neurons tend to synchronize their activities with their neighbors’, which leads to the compression of their representations.\n\nCan author clarify the reference if such claim is well-documented? If it is the novel claim of the author, the author should modify the tone to avoid overclaim or provide rationale or experiments to make sense of such claims.\n\n(2) (Weakness section). What is the contribution of the \"Proj\", and will there be synchronization, e.g. given input image, if we replace Proj into general linear sum (ablation of Proj)? And what is the difference between eq(6) and binding mechanism in Lowe's work? Since the binding mechanism itself can generate synchronization, how to disentangle the contribution from (6) to contribution from the \"Kura idea\".\n\n(3) Can author shows the segmentation results on binary datasets, like Zheng2024. The different color of object may make it eaiser to break symmetry compared with binary images. e.g. in Fig.4 the segmentation is more likely based on color instead of object-instance (the highlight is treated as an object). Besides, what is the actual input dimmension of the image to the model? To what extent it is downscaled?\n(4) Why the energy function still make sense even if the J and $\\Omega$ is not constraint by symmetry?\n(5) What is the rationale of robustness? Is it the general outcome of dynamical models or is it the outcome of this Kura model? I suggest compare with related non-synchrony dynamical ANN models to disentagle the causal-effect.\n(6) Why energy function, or the synchrony level in the network, can indicate the task performance? Can the author explain that more solidly?\n(7) For the Sudoku task, can the author compare with those Ising-based models? e.g. Hopfield network or modern hopfield network. So as to disentagle the contribution for the \"Kura idea\" from that generally comes from attractor dynamics.\n\nIn general, I would like to vote for the accept of the paper, but I will also revisit the evaluation based on the author's response and other reviewer's comments.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I enjoyed reading this paper, impressed by the achievements on diverse tasks and also the authors's thoughts dispersed over the paper.  In brief, the strength of the paper includes:\n- Soft grouping/clustering in neural network is an important question and often missing in main-stream ANNs.\n- The proposed neuron model is neuroscientific motivated yet has concise fomula. The physical origin of the model makes it plausible to construct an \"energy function\", which turns out to be quite informative on the performance.\n- While previous work on binding by synchrony mainly focuses on object discovery, the author extends these insights into reasoning tasks, and adversarial attack. And the performance on these new tasks is very successful.\n- Lastly, the author provides a different conceptual angle for the binding by synchronization: the synchronization enables competitive learning, and therefore compresses representation to construct a bottle neck, which facilitate abstraction. As far as I know, it is a novel argument not presented in previous works and it also makes sense to me.", "weaknesses": "For the weakness:\n- It is not clear what is the actual conceptual contribution / nolvelty of the Kura model proposed in this paper compared with Lowe's Rotating Feature (and recent updates). For example, equation (6) is very similiar to the \"binding mechanism activation\" in  [1] and I have the intuition that this is the essential mechanism for the binding ability of model, similiar to [1]. If I am wrong, please correct me.\n- When generalizing the original Kura model to high-dim, it is not clear whether the Proj in eq(2) is an 'equivalent' generalization of $sin(\\theta_j-\\theta_i)$. So it is not clear how much this term actually contribute to the computation of the network. Note that $sin(\\theta_j-\\theta_i)$ is a essential mechanism in original Kura model to guarentee the synchrony as a stable state and sin acts among each pair of neurons before the $\\sum$, but Proj only acts after the $\\sum$. (The first two concerns are mainly about how \"Kura\" the model is ?)\n- Since the activity is constraints on the sphere, it loses the representational ability for encoding feature presence, which is an important conceptual diaviation from synchrony binding idea and is likely to limit its ability to infer in certain cases, e.g. when the feature is uncertain or weakly related to the task.\n- On the experimental side, it mostly ignored the Lowe's model for comparison (Fig.3, Fig.4,Fig.5,Tab.1,Tab2, Tab3...) and other synchrony based model, e.g. [2].\n- The code is not provided, so it is hard to evaluate the the reproducibility of the results.\n- The author misses several essential recent works: e.g. [2] shows how oscillation emerges in spike-based model to bind features; [3] combine Kura model with ANNs and also achieved synchrony binding.\n- The first two sentences in Abstract is quite confusing and the first paragraph in Introduction is not very informative.\n\n[1] Löwe S, Lippe P, Locatello F, et al. Rotating features for object discovery[J]. Advances in Neural Information Processing Systems, 2024, 36.\n[2] Zheng H, Lin H, Zhao R. GUST: combinatorial generalization by unsupervised grouping with neuronal coherence[J]. Advances in Neural Information Processing Systems, 2024, 36.\n[3] Ricci M, Jung M, Zhang Y, et al. KuraNet: systems of coupled oscillators that learn to synchronize[J]. arXiv preprint arXiv:2105.02838, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BQekMHxPRJ", "reviewer": "Reviewer_DAWQ", "summary": "The paper proposes a novel type of neural network where neurons’ activities are defined on the unit sphere (in N-dimensional space, where N is typically 2 or 4), and are updated based on Kuramoto equations. This can model oscillatory dynamics that can be leveraged for binding activations across neurons. The model is applied to various settings (unsupervised object discovery, image classification adversarial or natural noise robustness, Sudoku problem solving) and compared against relevant baselines (e.g. slot-based models, Transformers), each time demonstrating competitive performance. The model has clear potential to improve the state-of-the-art across domains.", "questions": "* on Lines 140-141, you argue that “we found that the use of C is necessary for stable trainings […] as a symmetry-breaking field”. But isn’t C also –and more simply—required for the network to receive inputs, and thus to perform any useful function?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* A biologically inspired architecture, well described and motivated by neuroscientific findings\n* A thorough set of experiments across diverse machine learning domains\n* Meaningful comparisons with baselines revealing competitive performance", "weaknesses": "* The dependence of the model on the choice of dimensionality N, although it is explicitly acknowledged in the paper, warrants further investigations (which can be the goal of future studies)\n* Although baselines are roughly matched for memory/parameter size and flops, it is possible that GPU optimizations will make them more efficient than the AKOrN versions. There is unfortunately no discussion of this issue nor any report of absolute time required for training or inference. This could be added to the Appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)