ARTIFICIAL KURAMOTO OSCILLATORY NEURONS

## Abstract

Abstract It has long been known in both neuroscience and AI that “binding” between neurons leads to a form of competitive learning where representations are compressed in order to represent more abstract concepts in deeper layers of the network. More recently, it was also hypothesized that dynamic (spatiotemporal) representations play an important role in both neuroscience and AI. Building on these ideas, we introduce Artificial Kuramoto Oscillatory Neurons ( AKOrN ) as a dynamical alternative to threshold units, which can be combined with arbitrary connectivity designs such as fully connected, convolutional, or attentive mechanisms. Our generalized Kuramoto updates bind neurons together through their synchronization dynamics. We show that this idea provides performance improvements across a wide spectrum of tasks such as unsupervised object discovery, adversarial robustness, calibrated uncertainty quantification, and reasoning. We believe that these empirical results show the importance of rethinking our assumptions at the most basic neuronal level of neural representation, and in particular show the importance of dynamical representations † † Code: https://github.com/autonomousvision/akorn . .

### 1 Introduction

Before the advent of modern deep learning architectures, artificial neural networks were inspired by biological neurons. In contrast to the McCulloch-Pitts neuron (McCulloch & Pitts, ) which was designed as an abstraction of an integrate-and-fire neuron (Sherrington, ), recent building blocks of neural networks are designed to work well on modern hardware (Hooker, ). As our understanding of the brain is improving over recent years, and neuroscientists are discovering more about its information processing principles, we can ask ourselves again if there are lessons from neuroscience that can be used as design principles for artificial neural nets.

In this paper, we follow a more modern dynamical view of neurons as oscillatory units that are coupled to other neurons (Muller et al., ). Similar to how the binary state of a McCulloch-Pitts neuron abstracts the firing of a real neuron, we will abstract an oscillating neuron by an $N$-dimensional unit vector that rotates on the sphere (Löwe et al., ). We build a new neural network architecture that has iterative modules that update $N$-dimensional oscillatory neurons via a generalization of the well-known non-linear dynamical model called the Kuramoto model (Kuramoto, ).

The Kuramoto model describes the synchronization of oscillators; each Kuramoto update applies forces to connected oscillators, encouraging them to become aligned or anti-aligned. This process is similar to binding in neuroscience and can be understood as distributed and continuous clustering. Thus, networks with this mechanism tend to compress their representations via synchronization.

We incorporate the Kuramoto model into an artificial neural network, by applying the differential equation that describes the Kuramoto model to each individual neuron. The resulting artificial Kuramoto oscillatory neurons (AKOrN) can be combined with layer architectures such as fully connected layers, convolutions, and attention mechanisms.

We explore the capabilities of AKOrN and find that its neuronal mechanism drastically changes the behavior of the network. AKOrN strongly binds object features with competitive performance to slot-based models in object discovery, enhances the reasoning capability of self-attention, and increases robustness against random, adversarial, and natural perturbations with surprisingly good calibration.

### 2 Motivation

It was recognized early on that neurons interact via lateral connections (Hubel & Wiesel, ; Somers et al., ). In fact, neighboring neurons tend to cluster their activities, and clusters tend to compete to explain the input. This “competitive learning” has the advantage that information is compressed as we move through the layers, facilitating the process of abstraction by creating an information bottleneck. Additionally, the competition encourages different higher-level neurons to focus on different aspects of the input (i.e. they specialize). This process is made possible by synchronization: like fireflies in the night, neurons tend to synchronize their activities with their neighbors’, which leads to the compression of their representations. This idea has been used in artificial neural networks before to model “binding” between neurons, where neurons representing features such as square, blue, and toy are bound by synchronization to represent a square blue toy (Reichert & Serre, ; Löwe et al., ). In this paper, we will use an $N$-dimensional generalization of the famous Kuramoto model (Kuramoto, ) to model this synchronization.

Our model has the advantage that it naturally incorporates spatiotemporal representations in the form of traveling waves (Keller et al., ), for which there is ample evidence in the neuroscientific literature. While their role in the brain remains poorly understood, it has been postulated that they are involved in short-term memory, long-range coordination between brain regions, and other cognitive functions (Rubino et al., ; Lubenov & Siapas, ; Fell & Axmacher, ; Zhang et al., ; Roberts et al., ; Muller et al., ; Davis et al., ; Benigno et al., ).
For example, Muller et al. () finds that oscillatory patterns in the thalamocortical network during sleep are organized into circular wave-like patterns, which could give an account of how memories are consolidated in the brain. Davis et al. () suggest that spontaneous traveling waves in the visual cortex modulate synaptic activities and thus act as a gating mechanism in the brain. In the generalized Kuramoto model, traveling waves naturally emerge as neighboring oscillators start to synchronize (see on the left in Fig 1, and Fig 10 in the Appendix).

Another advantage of using dynamical neurons is that they can perform a form of reasoning. Kuramoto oscillators have been successfully used to solve combinatorial optimization tasks such as k-SAT problems (Heisenberg, ; Wang & Roychowdhury, ). This can be understood by the fact that Kuramoto models can be viewed as continuous versions of discrete Ising models, where phase variables replace the discrete spin states. Many authors have argued that the modern architectures based on, e.g., transformers lack this intrinsic capability of “neuro-symbolic reasoning” (Dziri et al., ; Bounsi et al., ). We show that AKOrN can successfully solve Sudoku puzzles, illustrating this capability. Additionally, AKOrN relates to models in quantum physics and active matter (see appendix A).

In summary, AKOrN combines beneficial features such as competitive learning (i.e., feature binding), reasoning, robustness and uncertainty quantification, as well as the potential advantages of traveling waves observed in the brain, while being firmly grounded in well-understood physics models.

Figure: Figure 1: Our proposed artificial Kuramoto oscillatory neurons (AKOrN). The series of pictures on the left are 64$\times$ 64 Kuramoto oscillators evolving by the Kuramoto updates (Eq. (2)), along with a plot of the energies computed by Eq. (3). Each single oscillator $\mathbf{x}_{i}$ is an $N$-dimensional vector on the sphere and is influenced by (1) connected oscillators through the weights $\mathbf{J}_{ij}$, (2) conditional stimuli $\mathbf{c}_{i}$, and (3) $\bm{\Omega}_{i}$ that determines the natural frequency of each oscillator. See Fig 10 for details on $\mathbf{C}$ and $\mathbf{J}$.
Refer to caption: x1.png

### 3 The Kuramoto model

The Kuramoto model (Kuramoto, ) is a non-linear dynamical model of oscillators, that exhibits synchronization phenomena. Even with its simple formulation, the model can represent numerous dynamical patterns depending on the connections between oscillators (Breakspear et al., ; Heitmann et al., ).

In the original Kuramoto model, each oscillator $i$ is represented by its phase information $\theta_{i}\in[0,2\pi)$. The differential equation of the Kuramoto model is

$$ $\displaystyle\dot{\theta}_{i}=\omega_{i}+{\textstyle\sum}_{j}J_{ij}\sin(\theta _{j}-\theta_{i}),$ (1) $$

where $\omega_{i}\in\mathbb{R}$ is the natural frequency and $J_{ij}\in\mathbb{R}$ represents the connections between oscillators: if $J_{ij}>0$ the $i$ and $j$-th oscillator tend to align, and if $J_{ij}<0$, they tend to oppose each other.

While the original Kuramoto model describes one-dimensional oscillators, we introduce a multi-dimensional vector version of the model (Cumin & Unsworth, ; Chandra et al., ; Lipton et al., ) with a novel symmetry-breaking term into neural networks. We denote oscillators by $\mathbf{X}=\{\mathbf{x}_{i}\}_{i=1}^{C}$, where each $\mathbf{x}_{i}$ is an $N$-dimensional unit vector $\mathbf{x}_{i}\in\mathbb{R}^{N},~{}\|\mathbf{x}_{i}\|_{2}=1$. While each $\mathbf{x}_{i}$ is time-dependent, we omit $t$ for clarity.
The oscillator index $i$ may have multiple dimensions: if the input is an image, for example, each oscillator is represented by $\mathbf{x}_{chw}$ with $chw$ indicating channel, height and width positions, respectively.

The differential equation of our vector-valued Kuramoto model is written as follows:

$$ $\displaystyle\dot{\mathbf{x}}_{i}=\bm{\Omega}_{i}\mathbf{x}_{i}+{\rm Proj}_{ \mathbf{x}_{i}}(\mathbf{c}_{i}+\sum_{j}\mathbf{J}_{ij}\mathbf{x}_{j})~{}{\rm where }~{}{\rm Proj}_{\mathbf{x}_{i}}(\mathbf{y}_{i})=\mathbf{y}_{i}-\left\langle \mathbf{y}_{i},\mathbf{x}_{i}\right\rangle\mathbf{x}_{i}$ (2) $$

Here, $\bm{\Omega}_{i}$ is an $N\times N$ anti-symmetric matrix and $\bm{\Omega}_{i}\mathbf{x}_{i}$ is called the natural frequency term that determines each oscillator’s own rotation frequency and angle. The second term governs interactions between oscillators, where ${\rm Proj}_{\mathbf{x}_{i}}$ is an operator that projects an input vector onto the tangent space of the sphere at $\mathbf{x}_{i}$.
$\mathbf{c}_{i}\in\mathbb{R}^{N},\mathbf{C}=\{\mathbf{c}_{i}\}_{i=1}^{C}$ is a data-dependent variable, which is computed from the observational input or the activations of the previous layer. In this paper, every $\mathbf{c}_{i}$ is set to be constant across time, but it can be a time-dependent variable. $\mathbf{c}_{i}$ can be seen as another oscillator that has a unidirectional connection to $\mathbf{x}_{i}$. Since $\mathbf{c}_{i}$ is not affected by any oscillators, $\mathbf{c}_{i}$ strongly binds $\mathbf{x}_{i}$ to the same direction as $\mathbf{c}_{i}$, i.e. it acts as a bias direction (see Fig 10 in the Appendix). Usually, the Kuramoto model is studied without such conditional stimuli, but we found that the use of $\mathbf{C}$ is necessary for stable training. In physics lingo, $\mathbf{C}$ is often referred to as a “symmetry breaking” field.

The Kuramoto model is Lyapunov
if we assume certain symmetric properties in $\mathbf{J}_{ij}$ and $\bm{\Omega}_{i}$ (Aoyagi, ; Wang & Roychowdhury, ).
For example, if $\mathbf{J}$ is symmetric and different oscillators share the same natural frequencies: $\mathbf{J}_{ij}=\mathbf{J}_{ji}^{\rm T}$, $\bm{\Omega}_{i}=\bm{\Omega}$, and $\bm{\Omega}\mathbf{c}_{i}={\bf 0}$, each update is guaranteed to minimize the following energy:

$$ $\displaystyle E=-\sum_{i,j}\mathbf{x}_{i}^{\rm T}\mathbf{J}_{ij}\mathbf{x}_{j} -\sum_{i}\mathbf{c}_{i}^{\rm T}\mathbf{x}_{i}$ (3) $$

Fig 1 on the left shows how the Kuramoto oscillators and the corresponding energy evolve with a simple Gaussian kernel as the connectivity matrix.
Here, we set $\mathbf{C}$ a silhouette of a fish, where $\mathbf{c}_{i}={\bf 1}$ on the outer silhouette and $\mathbf{c}_{i}={\bf 0}$ on the inner silhouette.
The oscillator state is initially disordered, but gradually exhibits collective behavior, eventually becoming a spatially propagating wavy pattern. We include animations of visualized oscillators, including oscillators of trained AKOrN models used in our experiments, in the Supplementary Material.

We would like to note that we found that even without symmetric constraints, the energy value decreases relatively stably, and the models perform better across all tasks we tested compared to models with symmetric
$\mathbf{J}$. A similar observation is made by Effenberger et al. () where heterogeneous oscillators such as those with different natural frequencies are helpful for the network to control the level of synchronization and increase the network capacity.
From here, we assume no symmetric constraints on $\mathbf{J}$ and $\bm{\Omega}$.
Having asymmetric (a.k.a. non-reciprocal) connections is aligned with the biological neurons in the brain, which also do not have symmetric synapses.

### 4 Networks with Kuramoto oscillators

We utilize the artificial Kuramoto oscillator neurons (AKOrN) as a basic unit of information processing in neural networks (Fig 2). First, we transform an observation with a relatively simple function to create the initial $\mathbf{C}$. Next, $\mathbf{X}$ is initialized by either $\mathbf{C}$, a fixed learned embedding, random vectors, or a mixture of these initialization schemes.
The block is composed of two modules: the Kuramoto layer and the readout module, which together process the pair $\{\mathbf{X},\mathbf{C}\}$. The Kuramoto layer updates $\mathbf{X}$ with the conditional stimuli $\mathbf{C}$, and the readout layer extracts features from the final oscillatory states to create new conditional stimuli.
We denote $l$-th layer’s output of the $l$-th block by $\{\mathbf{X}^{(l)},\mathbf{C}^{(l)}\}$.

Figure: Figure 2: Our proposed Kuramoto-based network (here, for image processing). Each block consists of a Kuramoto-layer and a readout module described in Sec 4. $\mathbf{C}^{(L)}$ is used to make the final prediction of our model.
Refer to caption: x2.png

###### Kuramoto layer

Starting with $\mathbf{X}^{(l,0)}:=\mathbf{X}^{(l)}$ as initial oscillators, where the second superscript denotes the time step, we update them by the discrete version of the differential equation (2):

$$ $\displaystyle\Delta\mathbf{x}_{i}^{(l,t)}=\bm{\Omega}_{i}\mathbf{x}_{i}^{(l,t) }+{\rm Proj}_{\mathbf{x}_{i}^{(l,t)}}(\mathbf{c}_{i}^{(l)}+\sum_{j}\mathbf{J}_ {ij}\mathbf{x}_{j}^{(l,t)})$ (4) $\displaystyle\mathbf{x}_{i}^{(l,t+1)}=\Pi\left[\mathbf{x}_{i}^{(l,t)}+\gamma \Delta\mathbf{x}_{i}^{(l,t)}\right],$ (5) $$

where $\Pi$ is the normalizing operator $\mathbf{x}/\|\mathbf{x}\|_{2}$ that ensures that the oscillators stay on the sphere. $\gamma>0$ is a scalar controlling the step size of the update, which is learned in our experiments. We call this update a Kuramoto update or a Kuramoto step from here.
We optimize both $\bm{\Omega}$ and $\mathbf{J}$ given the task objective.

We update the oscillators $T$ times. We denote the oscillators at $T$ by $\mathbf{X}^{(l,T)}$. This oscillator state is used as the initial state of the next block: $\mathbf{X}^{(l,T)}:=\mathbf{X}^{(l+1,0)}$.

###### Readout module

We read out patterns encoded in the oscillators to create new conditional stimuli $\mathbf{C}^{(l+1)}$ for the subsequent block. Since the oscillators are constrained onto the (unit) hyper-sphere,
all the information is encoded in their directions. In particular, the relative direction between oscillators is an important source of information because patterns after certain Kuramoto steps only differ in global phase shifts (see the last two patterns in Fig 10 in the Appendix).
To capture phase invariant patterns, we take the norm of the linearly processed oscillators:

$$ $\displaystyle\mathbf{C}^{(l+1)}=g(\mathbf{m})\in\mathbb{R}^{C^{\prime}\times N },m_{k}=\|\mathbf{z}_{k}\|_{2},~{}\mathbf{z}_{k}=\textstyle\sum_{i}\mathbf{U}_ {ki}\mathbf{x}_{i}^{(l,T)}\in\mathbb{R}^{N^{\prime}},$ (6) $$

where $\mathbf{U}_{ki}\in\mathbb{R}^{N^{\prime}\times N}$ is a learned weight matrix, $g$ is a learned function, and $\mathbf{m}=[m_{1},...,m_{K}]^{\rm T}\in\mathbb{R}^{K}$. $N^{\prime}$ is typically set to the same value as $N$. In this work, $g$ is just the identity function, a linear layer, or at most a three-layer neural network with residual connections. Because the module computes the norm of (weighted) $\mathbf{X}^{(l,T)}$, this readout module includes functions that are invariant to the global phase shift in the solution space. Unless otherwise specified, we set $C^{\prime}=C$ and $K=C\times N$ in all our experiments.

#### 4.1 Connectivities

We implement artificial Kuramoto oscillator neurons (AKOrN) within convolutional and self-attention layers. We write down the formal equations of the connectivity for completeness, however, they simply follow the conventional operation of convolution or self-attention applied to oscillatory neurons flattened w.r.t the rotating dimension $N$.
In short, convolutional connectivity is local, and attentive connectivity is dynamic input-dependent connectivity.

###### Convolutional connectivity

To implement AKOrN in a convolutional layer, oscillators and conditional stimuli are represented as $\{\mathbf{x}_{chw},\mathbf{c}_{chw}\}$ where $c,h,w$ are channel, height and width positions, and the update direction is given by:

$$ $\displaystyle\mathbf{y}_{chw}:=\mathbf{c}_{chw}+\sum_{d}\sum_{h^{\prime},w^{ \prime}\in R[H^{\prime},W^{\prime}]}\mathbf{J}_{cdh^{\prime}w^{\prime}}\mathbf {x}_{d(h+h^{\prime})(w+w^{\prime})},$ (7) $$

where $R[H^{\prime},W^{\prime}]=[1,...,H^{\prime}]\times[1,...,W^{\prime}]$ is the $H^{\prime}\times W^{\prime}$ rectangle region (i.e. kernel size) and $\mathbf{J}_{cdh^{\prime}w^{\prime}}\in\mathbb{R}^{N\times N}$ are the learned weights in the convolution kernel where $(c,d),(h^{\prime},w^{\prime})$ are output and input channels, and height and width positions.

###### Attentive connectivity

Similar to Bahdanau et al. (); Vaswani et al. (), we construct the internal connectivity in the QKV-attention manner. In this case, oscillators and conditional stimuli are represented by $\{\mathbf{x}_{li},\mathbf{c}_{li}\}$ where $l$ and $i$ are indices of tokens and channels, respectively. The update direction becomes:

$$ $\displaystyle\mathbf{y}_{li}:=\mathbf{c}_{li}+\sum_{m,j}\mathbf{J}_{lmij} \mathbf{x}_{mj}=\mathbf{c}_{li}+\sum_{m,j}\sum_{k,h}\mathbf{W}^{O}_{h,ik}A_{h} (l,m)\mathbf{W}^{V}_{h,kj}\mathbf{x}_{mj}$ (8) $\displaystyle A_{h}(l,m)=\frac{e^{d_{h}(l,m)}}{\sum_{m}e^{d_{h}(l,m)}},~{}d_{h }(l,m)=\sum_{a}\left\langle\sum_{i}\mathbf{W}^{Q}_{h,ai}\mathbf{x}_{li},\sum_{ i}\mathbf{W}^{K}_{h,ai}\mathbf{x}_{mi}\right\rangle$ (9) $$

where $\mathbf{W}^{O}_{h,ik},\mathbf{W}^{V}_{h,kj},\mathbf{W}^{Q}_{h,ai},\mathbf{W}^{
K}_{h,ai}\in\mathbb{R}^{N\times N}$ are learned weights of head $h$.
Since the connectivity is dependent on the oscillator values and thus not static during the updates, it is unclear whether the energy defined in Eq, (3) is proper. Nonetheless, in our experiments, the energy and oscillator states are stable after several updates (see the Supplementary Material, which includes visualizations of the oscillators of trained AKOrN models and their corresponding energies over timesteps).

### 5 Related works

The Kuramoto model
is rarely seen in machine learning, especially in deep learning. However, several works motivate us to use the Kuramoto model as a mechanism for learning binding features. For example, although tested only in fairly synthetic settings, Liboni et al. () show that cluster features emerge in the oscillators of the Kuramoto model with lateral connections without optimization. Also, a line of works on neural synchrony (Reichert & Serre, ; Löwe et al., ; Stanić et al., ; Löwe et al., ; Gopalakrishnan et al., ) shares the same philosophy with AKOrN. Löwe et al. () extend the complex-valued neurons used by Reichert & Serre (); Löwe et al. () to multidimensional neurons and shows that, together with a specific activation function called $\chi$-binding that implements the ‘winner-take-all’ mechanism at the single neuron level (Löwe et al., ), the multidimensional neurons learn to encode binding information in their orientations.
The mechanism itself is intriguing in its own right but struggles to scale to natural images without pre-trained models. Additionally, its integration beyond linear and convolution layers, such as into attention mechanisms, remains unclear.

Slot-based models (Le Roux et al., ; Burgess et al., ; Greff et al., ; Locatello et al., ) are the most-used model for object-centric (OC) learning. Their discrete nature of representations is shown to be a good inductive bias to learn such OC representations, but these models struggle on natural images, and are therefore often combined with powerful, pre-trained self-supervised learning (SSL) models such as DINO (Caron et al., ). Our proposed continuous Kuramoto neurons can be a building block of the SSL network itself, and we show that they learn better object-centric features than well-known SSL models.
AKOrNs perform particularly well on object discovery tasks when implemented in self-attention layers. Self-attention updates with normalization have been shown mathematically to cluster token features (Geshkovski et al., ).
Our work combines this clustering behavior of transformers with the clustering induced by the synchronization of the Kuramoto neurons, resulting in AKOrN being the first competitive method to slot-based approaches.

Finally, there exist several works on interpreting self-attention as energy-based models (Ramsauer et al., ; Hoover et al., ). Our Kuramoto model-based models differ from these approaches in various aspects such as the implementation of unit-norm-constrained neurons with asymmetric connections, and their symmetry breaking term.

Figure: Figure 3: Object discovery performance on synthetic datasets. $B$ indicates the number of blocks.
Refer to caption: x3.png

### 6 Experiments

#### 6.1 Unsupervised object discovery

Figure: Figure 5: Visualization of clusters on (Left) PascalVOC and (Right) COCO2017.
Refer to caption: extracted/5935374/gfx/multi_objects/cluster_imgs_high/pascal_real.jpg

Unsupervised object discovery is the task of finding objects in an image without supervision.
Here, we test AKOrN on four synthetic datasets (Tetrominoes, dSprites, CLEVR (Kabra et al., ), CLEVRTex (Karazija et al., )) and two real image datasets (PascalVOC (Everingham et al., ), COCO2017 (Lin et al., )) (see Appendix C for details). Among the four synthetic datasets, CLEVRTex has the most complex objects and backgrounds. We further evaluate the models trained on the CLEVRTex dataset on two variants (OOD, CAMO). The materials and shapes of objects in OOD differ from those in CLEVRTex, while CAMO (short for camouflage) features scenes where objects and backgrounds share similar textures within each scene.

As baselines, we train models that are similar to ResNet (He et al., ) and ViT (Dosovitskiy et al., ), but iterate the convolution or self-attention layers multiple times with shared parameters. This allows us to evaluate the impact of our proposed, Kuramoto-based iterative updates. We denote these baselines as Iterative Convolution (ItrConv) and Iterative Self-Attention (ItrSA), respectively.
Fig 11 in the Appendix shows diagrams of each network.

In AKOrN, $\mathbf{C}$ is initialized by the patched features of the images, while each $\mathbf{x}_{i}$ is initialized by random oscillators sampled from the uniform distribution on the sphere. We train the AKOrN model with the self-supervised SimCLR (Chen et al., ) objective.

We train each model from scratch on the four synthetic datasets. For the two real image datasets, we first train AKOrN on ImageNet (Krizhevsky et al., ) and directly evaluate that ImageNet-pretrained model on both datasets without fine-tuning.
When evaluating, we apply clustering to the final block’s output features (In AKOrN, it is $\mathbf{C}^{(L)}$). We use agglomeration clustering with average linkage, which we found to outperform K-means for both the baseline models and AKOrN. We evaluate the clustering results by foreground adjusted rand index (FG-ARI) and Mean-Best-Overlap (MBO).
FG-ARI measures the similarity between the ground truth masks and the computed clusters, only for foreground objects. MBO first assigns each cluster to the highest overlapping ground truth mask and then computes the average intersection-over-union (IoU) of all pairs.
For PascalVOC and COCO2017, we show instance-level MBO (MBOi) and class-level (MBOc) segmentation results.

###### AKOrN binds object features

Fig 4 shows that AKOrNs improve the object discovery performance over their non-Kuramoto counterparts in almost every dataset (except for Tetrominoes).
Interestingly, we observe that convolution is less effective than attention in most datasets.
In Fig 4, we see that the Kuramoto models’ clusters are well-aligned with the individual objects, while clusters of the ItrSA model often span across objects and background, and are sensitive to the texture of the background (more clustering results are shown in Fig 20-22 in the Appendix).

Tab 1 shows a comparison to existing works on CLEVRTex and its variants. All other methods are slot-based. Among the distributed representation models, AKOrN is the first method that is shown to be competitive with slot-based models on the complex CLEVRTex dataset.

###### AKOrN scales to natural images

Fig 5 shows AKOrN binds object features on natural images much better than DINO (Caron et al., ).
We show a benchmark comparison on Pascal VOC and COCO2017 in Tab 2. The proposed AKOrN model outperforms existing SSL models including DINO, MoCoV3, and MAE on both datasets, showing that it learns more object-bound features than conventional transformer-based models.
On Pascal, AKOrN is far better than other models trained from scratch and is better than methods trained on features of a pretrained DINO model.
On COCO, AKOrN again outperforms methods that are trained from scratch and is competitive to DINOSAUR and Slot-diffusion, but is outperformed by the recent SPOT model.

**Table 2: Object discovery on PascalVOC and COCO2017.**
|  | PascalVOC | COCO2017 |  |  |
| --- | --- | --- | --- | --- |
| Model | MBOi | MBOc | MBOi | MBOc |
| (slot-based models) |  |  |  |  |
| Slot-attention (Locatello et al., ) | 22.2 | 23.7 | 24.6 | 24.9 |
| SLATE (Singh et al., ) | 35.9 | 41.5 | 29.1 | 33.6 |
| (DINO + slot-based model) |  |  |  |  |
| DINOSAUR (Seitzer et al., ) | 44.0 | 51.2 | 31.6 | 39.7 |
| Slot-diffusion (Wu et al., ) | 50.4 | 55.3 | 31.0 | 35.0 |
| SPOT (Kakogeorgiou et al., ) | 48.3 | 55.6 | 35.0 | 44.7 |
| (transformer + SSL) |  |  |  |  |
| MAE (He et al., ) | 34.0 | 38.3 | 23.1 | 28.5 |
| MoCoV3 (Chen et al., ) | 47.3 | 53.0 | 28.7 | 36.0 |
| DINO (Caron et al., ) | 47.2 | 53.5 | 29.4 | 37.0 |
| AKOrN | 52.0 | 60.3 | 31.3 | 40.3 |

#### 6.2 Solving Sudoku

To test AKOrN’s reasoning capability, we apply it on the Sudoku puzzle datasets (Wang et al., ; Palm et al., ). The training set contains boards with 31-42 given digits. We test models in in-distribution (ID) and out-of-distribution (OOD) scenarios. The ID test set contains 1,000 boards sampled from the same distribution, while boards in the OOD set contain much fewer given digits (17-34) than the train set.
To initialize $\mathbf{C}$, we use embeddings of the digits 0-9 (0 for blank, 1-9 for given digits). $\mathbf{x}_{i}$ takes the value $\mathbf{c}_{i}$ when a digit is given, and is randomly sampled from the uniform distribution on the sphere for blank squares. The number of Kuramoto steps during training is set to 16. We also train a transformer model with 8 blocks.

###### AKOrN solves Sudoku puzzles

AKOrN perfectly solves all puzzles in the ID test set,
while only Recurrent Transformer (R-Transformer  (Yang et al., )) achieves this (Tab 3). On the OOD set, AKOrN achieves 61.1±14.7 accuracy which is on par with IRED (Du et al., ), an energy-based diffusion model, and vastly better than all other existing approaches (including the R-Transformer). AKOrN again strongly outperforms its non-Kuramoto counterparts, ItrSA and Transformer.

###### Test-time extension of the Kuramoto steps

Just as we humans use more time to solve harder problems, AKOrN’s performance improves as we increase the number of Kuramoto steps.
As shown in Fig 6 (a,b), on the ID test set, the energy fluctuates but roughly converges to a minimum after around 32 steps. On the OOD test set, however, the energy continues to decrease further. Fig 6 (c) shows that increasing the number of Kuramoto steps at test time improves accuracy significantly (17% to 52%), while increasing the step count of standard self-attention provides a limited improvement on the OOD set (14% to 34%) and leads to lower performance on the ID set (99.3% to 95.7%).

###### The energy value tells the correctness of the boards

The energy value is a good indicator of the solution’s correctness. In fact, we observe that predictions with low-energy oscillator states tend to be correct (see Fig 18).
We utilize this property to improve the performance. For each given board, we sample multiple predictions with different initial oscillators and select the lowest-energy prediction as the model’s answer, which we call Energy-based voting ($E$-vote). We see in Fig 7 that by increasing the number of sampled predictions, the model’s board accuracy improves.
Interestingly, just averaging the predictions of different states (i.e., majority voting) does not give better answers.

Figure: Figure 6: (a) Transition of the energy in Eq. (3) over # Kuramoto steps on the Sudoku datasets. The semi-transparent lines are actual energy values averaged across examples, and the solid ones connect the troughs. The dotted vertical line indicates # Kuramoto steps set during training. (b) A zoomed-in version of each plot. (c) The effect of test-time extension on # Kuramoto steps.
Refer to caption: x6.png

Figure: Figure 7: Improvement of board accuracy by the post-selection of predictions based on the $E$ values described in Sec 6.2. $T_{\rm eval}$ is set to 128. ‘$E$-vote’ and ‘Avg’ stand for energy-based voting and majority voting, respectively.
Refer to caption: x7.png

#### 6.3 Robustness and CalibrationImage Classification

o
bustness to adversarial attacks and uncertainty quantificits ation performance onthe network with the and CIFAR10 with common corruptions (CC,  Hendrycks & Dietterich ()).
We train two types of networks: a convolutional AKOrN (AKOrN^conv) and AKOrN with both convolution and self-attention (AKOrN^mix).
The former has three convolutional Kuramoto blocks. The latter replaces the last block with an attentive Kuramoto block.
We use AutoAttack (Caron et al., ) to evaluate the model’s adversarial robustness.

###### AKOrN s are resilient against gradient-based attacks

The model is heavily regularized and achieves both good adversarial robustness and robustness to natural corruptions (Tab 4). This is remarkable, since conventional neural models need additional techniques such as adversarial training and/or adversarial purification to achieve good adversarial robustness. In contrast, AKOrN is robust by design, even when trained on only clean examples.

Figure: Figure 8: Robustness performance on random noise examples. Each bar plot shows classification accuracy on CIFAR10 with strong random noise ($\|\epsilon\|_{\infty}=64/255$). The left two pictures are examples of images with that $\epsilon$. Green bars show accuracy when we ablate each element of AKOrN.
Refer to caption: x8.png

Figure: Table 4: Robustness to adversarial examples by AutoAttack (Adv) and common corruptions (CC) on CIFAR10. ^∗The attack is done by AutoAttack with EoT (Athalye et al., ). $\|\epsilon\|_{\infty}$ is set to 8/255. Expected Calibration Error (ECE) measures the alignment between confidence of the prediction and accuracy. The top two methods are selected from the highest-ranked methods on [https://robustbench.github.io/](https://robustbench.github.io/).
Refer to caption: x9.png

###### K-Nets are well-calibrated and robust to strong random noise

We found that AKOrNs are robust to strong random noise (Fig 8) and give good uncertainty estimation (on the bottom right in Fig 9). Surprisingly, there is an almost linear relationship between confidence and actual accuracy.
This is similar to observations in generative models (Grathwohl et al., ; Jaini et al., ), where conditional generative models give well-calibrated outputs. Since AKOrN’s energy is not learned to model input distribution, we cannot tightly relate ours to such generative models. However, we speculate that AKOrNs’ energy roughly approximates the likelihood of the input examples, and thus the oscillator state fluctuates according to the height of the energy, which would result in good calibration.

### 7 Discussion & Conclusion

We propose AKOrN, which integrates the Kuramoto model into neural networks and scales to complex observations, such as natural images. AKOrNs learn strongly object-binding features, can reason and are robust to adversarial and natural perturbations with well-calibrated predictions.
We believe our work provides a foundation for exploring a fundamental shift in the current neural network paradigm.

Our models still have a lot of phenomena that are not fully uncovered. For example, AKOrN exhibits quite different behaviors depending on the rotating dimension $N$. AKOrN with $N=2$ is strongly regularized, which positively influences its robustness, but negatively impacts optimization. Additionally, the performance with $N=2$ for object discovery and Sudoku solving is much worse than $N=4$. Further experimental and mathematical analysis is needed to understand why this occurs, which could provide insights into how we can leverage both advantages.

The oscillator is constrained onto the sphere and each single oscillator cannot represent the ‘presence’ of the features like the rotating features in Löwe et al. (). Because of that, AKOrN would not perform well on memory tasks, where the model needs to remember the presence of events. This norm constraint also does not align with real biological neurons that have firing and non-firing states. Relaxing the hard norm constraint of the oscillator would be an interesting future direction in terms of both biological plausibility and applicability to a much wider range of tasks such as long-term temporal processing.

### 8 Acknowledgement

Takeru Miyato and Andreas Geiger were supported by the ERC Starting Grant LEGO-3D (850533) and the DFG EXC number 2064/1 - project number 390727645.
We thank Andy Keller, Jun-nosuke Teramae, Bernhard Jaeger, Madhav Iyengar, and Daniel Dauner for their insightful feedback and comments.
Takeru Miyato acknowledges his affiliation with the ELLIS (European Laboratory for Learning and Intelligent Systems) PhD program.

## Appendix

Figure: Figure 10: The transition of the 64$\times$64 oscillator neurons ($N=2$). (Left) Visualiaztion of $\mathbf{C}$. $\mathbf{c}_{i}$ on the white region is ${\bf 1}$ and the black region is ${\bf 0}$. (Right) Oscillators’ time evolution. Similar colors indicate oscillators directing similar directions. The connectivity $\mathbf{J}$ is the following $9\times 9$ Gaussian kernel: $\mathbf{J}_{cdhw}=k_{hw}\mathbf{I}\in\mathbb{R}^{2\times 2},k_{hw}=\exp(-\frac {\|h-4.5\|_{2}^{2}+\|w-4.5\|_{2}^{2}}{3.0^{2}})$. The oscillators on the white region of $\mathbf{C}$ are fixed in the same direction as the conditional stimuli and almost stay constant across time. The oscillators on the black region are largely influenced by the neighboring oscillators and exhibit wavy patterns.
Refer to caption: x10.png

Figure: (a) Iterative conv
Refer to caption: x11.png

### Appendix A Relation to physics models

Similar to how the Ising model is the basis for recurrent neural models, such as the Hopfield model (Hopfield, ), the Kuramoto model with symmetric lateral interactions can also be studied by viewing it as a model from statistical physics called the Heisenberg model (Mattis, ). In fact, we will be using a more general version of the Kuramoto model which involves a symmetry-breaking term (akin to a magnetic field interaction) and asymmetric connections between the neurons. This not only is biologically plausible (synapses are not symmetric), it also leads to much better results in our experiments.

Non-equilibrium soft matter physics has studied models with nonreciprocal interactions, for instance in the field of “active matter”. They have developed accurate coarse-grained hydrodynamics models to approximate the microscopic dynamics and observed very interesting behavior, such as symmetry-breaking phase transitions and resultant traveling waves representing so called Goldstone modes (Fruchart et al., ). We hope that this opens the door to a deeper understanding of these models when employed as neural networks.

### Appendix B Related works on the NN robustness

Experimental proof of the conventional NNs’ limited OOD generalization is represented by the vulnerability to adversarial examples (Szegedy et al., ; Goodfellow et al., ).
The most effective way to resist such examples is training the model on adversarial examples generated by the model itself, which is called adversarial training (Goodfellow et al., ; Madry et al., ; Miyato et al., ; Zhang et al., ).
Many other defenses have been proposed, but most of them were found to be not a fundamental solution (Tramer et al., ).

One framework that can produce more human-algined predictions is a generative classifier (Ng & Jordan, ; Bishop & Nasrabadi, ), where we train a model with both generative and discriminative objectives or turn a label conditional generative model into a discriminative model based on Bayes theorem.
Interestingly, different generative classifiers trained with different methods share similar robust and calibration properties (Lee et al., ; Grathwohl et al., ; Li et al., ; Jaini et al., ).
Generative classifiers are robust but involve costly generative training such as denoising diffusion (Li et al., ; Jaini et al., ), MCMC (Grathwohl et al., ) to generate negative samples, or unstable min-max optimization as GANs training (Lee et al., ). AKOrN shares similar robustness properties but without any generative objectives.

### Appendix C Experimental settings

We observe that both the readout module and conditional stimuli $\mathbf{C}$ are essential for stable training, especially when $N=2$.
We also see that AKOrN with $N=2$ exhibits a strong regularity, which acts positively on robustness performance while having negative effects on unsupervised object discovery and the Sudoku-solving experiments. We show results of AKOrN with $N=4$ in those experiments. We do not observe improvement by increasing $N$ above 4.

Tab 5 - Tab 8 show experimental settings on each dataset (e.g. hyperparameters on models and optimization, the number of training and test examples, dataset statistics, etc…).
For AKOrN, the channel size is set to $\text{(the channel size shown in the table)}/N$, so the memory consumption and FLOPs are effectively the same between AKOrNs and their non-Kuramoto counterpart baselines we test.
All models are trained with Adam (Kingma & Ba, ) without weight decay.

**Table 5: Experimental settings on Tetrominoes, dSprites, and CLEVR.**
|  | Tetrominoes | dSprites | CLEVR |
| --- | --- | --- | --- |
|  |  |  |  |
| Training examples | 60,000 | 60,000 | 50,000 |
| Test examples | 320 | 320 | 320 |
| Image size | 32 | 64 | 128 |
| Maximum number of objects | 3 | 6 | 6 |
| Patch size | 4 | 4 | 8 |
| Patch resolution | 8 | 16 | 16 |
| Channel size | 128 | 128 | 256 |
| # internal steps ($T$) | 8 |  |  |
| # Epochs | 50 | 50 | 300 |
| Batchsize | 256 |  |  |
| Learning rate | 0.001 |  |  |
| Augmentations | Random resize and crop + color jittering |  |  |
| #clusters set for eval | 4 | 7 | 11 |

**Table 6: Experimental settings on CLEVRTex and its variants (OOD, CAMO). We also train a large AKOrN model that is trained with the doubled channel size, epochs, and epochs. We denote that model by Large AKOrN.**
|  | CLEVRTex | OOD | CAMO |
| --- | --- | --- | --- |
|  |  |  |  |
| Training examples | 40,000 | - | - |
| Test examples | 5,000 | 10,000 | 2,000 |
| Image size | 128 |  |  |
| Maximum number of objects | 10 |  |  |
| Patch size | 8 |  |  |
| Patch resolution | 16 |  |  |
| Channel size | 256 |  |  |
| # internal steps ($T$) | 8 |  |  |
| # epochs | 500 | - | - |
| Batchsize | 256 | - | - |
| Learning rate | 0.0005 | - | - |
| Augmentations | Random resize and crop + color jittering |  |  |
| #clusters set for eval | 11 |  |  |

**Table 7: Experimental settings on ImageNet pratraining and on the PascalVOC and COCO2017 evaluation. For SimCLR training augmentations, we use random resize and crop, color jittering, and horizontal flipping.**
|  | ImageNet | PascalVOC | COCO2017 |
| --- | --- | --- | --- |
|  |  |  |  |
| Training examples | 1,281,167 | - | - |
| Test examples | - | 1,449 | 5,000 |
| Image size | 256 | 256 | 256 |
| Patch size | 16 |  |  |
| Patch resolution | 16 |  |  |
| Channel size | 768 |  |  |
| # Blocks | 3 |  |  |
| # internal steps ($T$) | 4 |  |  |
| # epochs | 400 | - | - |
| Batchsize | 512 | - | - |
| Learning rate | 0.0005 | - | - |
| #clusters set for eval | - | 4 | 7 |

**Table 8: Sudoku puzzle datasets.**
|  | Sudoku(ID) (Wang et al., ) | Sudoku(OOD) (Palm et al., ) |
| --- | --- | --- |
|  |  |  |
| Training examples | 9,000 | - |
| Test examples | 1,000 | 18,000 |
| Channel size | 512 |  |
| # epochs | 100 | - |
| Batchsize | 100 | - |
| Learning rate | 0.0005 | - |

#### C.1 Unsupervised object discovery

We test on 4 synthetic datasets (Tetrominoes, dSprites, CLEVR, CLEVRTex) and 2 real image datasets (PascalVOC, COCO2017).
The kernel size of convolution layers in AKOrN^conv and ItrConv is set to 5, 7, and 9 on Tetrominoes, dSprites, and CLEVR, respectively.
In addition to ItrConv and ItrSA, we also train a ViT model (Dosovitskiy et al., ) as another baseline model.

All networks process images similarly to ViT (Dosovitskiy et al., ). First, we patch each image into $H/P\times W/P$ patches where $H,W$ are the height and width of the image and $P$ is the patch size. We then apply the stack of blocks. The output of the final block is further processed by global max-pooling followed by a single hidden layer MLP, whose output is used to compute the SimCLR loss. We used a conventional set of augmentations for SSL training: random resizing, cropping, and color jittering. We also apply horizontal flipping for the ImageNet pretraining.
All models including baseline models have roughly the same number of parameters and are trained with shared hyperparameters such as learning rates and training epochs. See Tab 5-7 for those hyperparameter details.

In AKOrN, $\mathbf{C}$ is initialized by the patched features of images, while each $\mathbf{x}_{i}$ is initialized by random oscillators sampled from the uniform distribution on the sphere. We use the identity function for
$g$ in each readout module. In multi-block models, we apply Group Normalization (Wu & He, ) to $\mathbf{C}$ except for $\mathbf{C}^{(L)}$.

For the Tetrominoes, dSprites, and CLEVR datasets, we train single-block models with $T=8$. We observe that stacking multiple blocks does not yield improvements on those three datasets.
On CLEVRTex, we train single- and two-block models with attentive connectivity and $T=8$, while on ImageNet, we train a three-block AKOrN model with attentive connectivity and $T=4$.

Following the literature, we exclude the background mask from the MBO evaluation.

##### C.1.1 Upsample features by up-tiling

When we compute the cluster assignment, we
upsample the output features by up-tiling where we let the model see a set of pictures that are slightly shifted both on the horizontal or/and vertical axes and make the higher resolution feature map by interleaving those features. This up-tiling enables us to get finer cluster assignments and substantially improves the object discovery performance of our  AKOrN.
We show a pictorial explanation in Fig 12 and PyTorch code below. We also show a comparison to the original features and bilinear upsampling in Fig 13 and examples of up-tiled features in Fig 14.
We apply up-tiling with the scale factor of 4 for producing numbers on Tab 1 and 2 as well as for cluster visualization in Fig 4,5 and Fig 20-24. Unless otherwise stated, no upsampling is performed when computing the cluster assignment.

Figure: Figure 12: 2$\times$ up-tiling. First, we create horizontally or/and vertically shifted images with stride equal to (patchsize/2) and compute the model’s output on each shifted image. We then interleave each token feature to make a 2$\times$ upsampled feature map.
Refer to caption: x16.png

Figure: Code 1: PyTorch code for up-tiling

Figure: (a) The original features (top) and features upsampled by up-tiling (bottom).
Refer to caption: x17.png

Figure: (a) CLEVRTex
Refer to caption: x19.png

#### C.2 Sudoku solving

The task is to fill a 9$\times$9 grid, given some initial digits from 1 to 9, so that each row, column, and 3$\times$3 subgrid contains all digits from 1 to 9. While the task may be straightforward if the game’s rules are known, the model must learn these rules solely from the training set. Example boards are shown in Tab. 8”.

We train AKOrN with attentive connections, the ItrSA model, and a conventional transformer model. We denote them by AKOrN^attn, ItrSA, and Transformer, respectively. AKOrN^attn has almost the same architecture used in the object discovery task except for $g$ in the readout module, which is composed of the norm computation layer followed by a stack of BatchNormalization, ReLU, and linear layer.

The input for each model is 9$\times$9 digits from 0 to 9 (0 for blank, 1-9 for given digits). We first embed each digit into a 512-dimensional token vector. The 9$\times$9 tokens are then flattened into 81 tokens. We apply each model to this token sequence and compute the prediction on each square by applying the softmax layer to each output token of the final block. All models are trained to minimize cross-entropy loss for 100 epochs.

The number of blocks of both ItrSA and AKOrN is set to one. We tested models with more than one blocks but found no improvement on the ID test set and a decline in OOD performance. Similar to the object discovery experiments, a transformer results in even worse performance than the ItrSA model (Tab 13).

The readout module is composed of the norm computation followed by the Batch Normalization layer, ReLU, and a linear layer.

#### C.3 Robustness and calibration on CIFAR10

We train two types of networks: a convolution-based AKOrN and AKOrN with a combination of convolution and attention.
The former has three proposed blocks, and all the Kuramoto layer’s connectivities are convolutional connectivity. The kernel sizes are 9,7, and 5 from shallow to deep, and $T$ is set to 3 for all blocks. Between consecutive blocks, a single convolution with a stride being 2 is applied to each of $\mathbf{C}$ and $\mathbf{X}$. Thus, the feature resolution of the final block’s output is $8\times 8$. Each readout module’s $g$ is Batch Normalization (Ioffe & Szegedy, ) followed by ReLU, and a 3$\times$3 convolution. $\mathbf{C}^{(3)}$ is average-pooled followed by the softmax layer that makes category predictions.
The latter network is identical to the former one except for the third block, which we replace with the block with attentive connectivity. Different timesteps $T$ are set across different blocks, which are $[6,4,2]$ from shallow to deep.

For ResNet-18 and AKOrN, we first conduct pre-training on the Tiny-imagenet (Le & Yang, ) dataset with the SimCLR loss for 50 epochs with batchsize of 512. We observe that this pre-training is effective for AKOrN and improves the CIFAR10 clean accuracy compared to training from scratch (from 87% to 91%). The ImageNet pretraining slightly improves ResNet’s clean accuracy (from 94.1% to 94.4%).
Each model is then trained on CIFAR10 for 400 epochs. We apply augmentations, including random scaling and cropping, color jittering, and horizontal flipping, along with AugMix (Hendrycks et al., ), as commonly used in robustness benchmarks. Both models are trained to minimize cross-entropy loss.

### Appendix D Additional experimental results

#### D.1 Positional encoding for the attentive connectivity

We need a positional encoding (PE) for AKOrN with attentive connectivity. We found GTA-type PE (Miyato et al., ) is effective and used for AKOrN throughout our experiments. Comparison to absolute positional encoding (APE) (Vaswani et al., ) and RoPE (Su et al., ) is shown in Tab 9. GTA does not improve the baseline ItrSA models.

**(a) CLEVRTex**
|  |  | CLEVRTex |  |
| --- | --- | --- | --- |
|  | PE | FG-ARI | MBO |
| ItrSA | APE | 66.87 | 42.15 |
| GTA | 66.07 | 43.41 |  |
| AKOrN | APE | 71.96 | 51.35 |
| RoPE | 65.70 | 50.22 |  |
| GTA | 75.79 | 54.08 |  |

#### D.2 Unsupervised object discovery

Figure: Figure 15: MBO on Tetrominoes, dSprites, and CLEVR.
Refer to caption: x21.png

##### D.2.1 MBO i vs # clusters

Figure: (a) PascalVOC
Refer to caption: x22.png

##### D.2.2 Full tables of object discovery performance

**Table 10: Object discovery results on synthetic datasets.**
| Model | Tetrominoes | dSprites | CLEVR |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | FG-ARI | MBO | FG-ARI | MBO | FG-ARI | MBO |
| ItrConv | 55.56 | 48.82 | 20.46 | 31.25 | 56.41 | 33.98 |
| AKOrN^conv | 75.37 | 53.88 | 73.35 | 56.80 | 65.08 | 47.03 |
| ItrSA | 86.81 | 51.74 | 69.42 | 64.67 | 80.15 | 36.76 |
| AKOrN^attn | 86.19 | 55.06 | 79.98 | 65.57 | 90.93 | 43.55 |
| (+up-tiling ($\times$4)) |  |  |  |  |  |  |
| AKOrN^attn | 92.72 | 56.40 | 88.57 | 62.81 | 93.64 | 43.76 |
| (Distributed representation models) |  |  |  |  |  |  |
| CAE (Löwe et al., ) | 78 | - | 51 | - | 27 | - |
| CtCAE (Stanić et al., ) | 84 | - | 56 | - | 54 | - |
| SynCx (Gopalakrishnan et al., ) | 89 | - | 82 | - | 59 | - |
| Rotating Features (Löwe et al., ) | 42 | - | 88.8 | 86.3 | 66.4 | 60.8 |
| (Slot-based model) |  |  |  |  |  |  |
| Slot-Attnetion (Locatello et al., ) | 99.5 | - | 91.3 | - | 98.8 | - |

**Table 11: Object discovery on CLEVRTex (Karazija et al., ). ^†Use Openimages (Kuznetsova et al., )-pretrained encoder. Numbers are from Jung et al. (). ^‡Use ImageNet-pretrained DINO. ^∗Numbers taken from Jia et al. (). ^pUse Imagenet-pretrained backbone models.**
| Model | CLEVRTex | OOD | CAMO |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | FG-ARI | MBO | FG-ARI | MBO | FG-ARI | MBO |
| ViT | 46.37 | 23.77 | 43.60 | 27.01 | 31.40 | 15.75 |
| ItrSA ($B=1,T=8$,) | 66.07 | 43.41 | 65.70 | 44.50 | 49.02 | 29.48 |
| ItrSA ($B=2,T=8$) | 75.33 | 48.44 | 73.91 | 45.69 | 60.38 | 36.72 |
| AKOrN^attn ($B=1,T=8$) | 75.79 | 54.94 | 73.11 | 55.05 | 59.70 | 43.28 |
| AKOrN^attn ($B=2,T=8$) | 81.50 | 54.08 | 80.15 | 55.02 | 68.73 | 44.98 |
| (+up-tiling ($\times$4)) |  |  |  |  |  |  |
| AKOrN^attn ($B=2,T=8$) | 87.28 | 55.40 | 86.41 | 56.32 | 74.85 | 45.95 |
| Large AKOrN^attn ($B=2,T=8$) | 89.24 | 60.02 | 88.00 | 60.96 | 77.18 | 53.43 |
| ^∗MONet (Burgess et al., ) | 19.78 | - | 37.29 | - | 31.52 | - |
| SLATE (Singh et al., ) | 44.19 | 50.88 | - | - | - | - |
| ^∗Slot-Attetion (Locatello et al., ) | 62.40 | - | 58.45 | - | 57.54 | - |
| Slot-diffusion (Wu et al., ) | 69.66 | 61.94 | - | - | - | - |
| ^†SLATE^+ (Singh et al., ) | 70.71 | 54.90 | - | - | - | - |
| ^†LSD (Jiang et al., ) | 76.44 | 72.44 | - | - | - | - |
| Slot-diffusion+BO (Wu et al., ) | 78.50 | 68.68 | - | - | - | - |
| ^∗DTI (Monnier et al., ) | 79.90 | - | 73.67 | - | 72.90 | - |
| ^∗I-SA (Chang et al., ) | 78.96 | - | 83.71 | - | 57.20 | - |
| BO-SA (Jia et al., ) | 80.47 | - | 86.50 | - | 63.71 | - |
| ^‡NSI (Dedhia & Jha, ) | 89.89 | 44.86 | - | - | - | - |
| ISA-TS (Biza et al., ) | 92.9 | - | 84.4 | - | 86.2 | - |
| ^†Jung et al. () | 93.06 | 75.36 | - | - | - | - |
| ^pSauvalle & de La Fortelle () | 94.77 | - | 83.14 | - | 87.27 | - |

**Table 12: Object discovery on PascalVOC and COCO2017.**
|  | PascalVOC | COCO2017 |  |  |
| --- | --- | --- | --- | --- |
| Model | MBOi | MBOc | MBOi | MBOc |
| (slot-based models) |  |  |  |  |
| Slot-attention (Locatello et al., ) | 22.2 | 23.7 | 24.6 | 24.9 |
| SLATE (Singh et al., ) | 35.9 | 41.5 | 29.1 | 33.6 |
| (DINO + slot-based model) |  |  |  |  |
| DINOSAUR (Seitzer et al., ) | 44.0 | 51.2 | 31.6 | 39.7 |
| Slot-diffusion (Wu et al., ) | 50.4 | 55.3 | 31.0 | 35.0 |
| SPOT (Kakogeorgiou et al., ) | 48.3 | 55.6 | 35.0 | 44.7 |
| (transformer + SSL) |  |  |  |  |
| MAE (He et al., ) | 33.8 | 37.7 | 22.9 | 28.3 |
| DINO (Caron et al., ) | 44.3 | 50.0 | 28.8 | 35.8 |
| MoCoV3 (Chen et al., ) | 47.3 | 53.0 | 28.7 | 36.0 |
| AKOrN^attn | 50.3 | 58.2 | 30.2 | 38.2 |
| (transformer + SSL + up-tiling ($\times$4)) |  |  |  |  |
| MAE | 34.0 | 38.3 | 23.1 | 28.5 |
| DINO | 47.2 | 53.5 | 29.4 | 37.0 |
| MoCoV3 | 44.6 | 50.5 | 29.0 | 35.9 |
| AKOrN^attn | 52.0 | 60.3 | 31.3 | 40.3 |

##### D.2.3 Training epochs vs MBO

Fig. 17 shows that MBOi and MBOc scores on Pascal and COCO improve as ImageNet pretraining progresses. Similar observations are made on CLEVRTex datasets, where larger AKOrNs give better object discovery performance (see Fig 20-22 and Tab 11).
These results indicate that there is an alignment between the SSL training with AKOrN and learning object-binding features and that increasing parameters and computational resources can further enhance the object discovery performance.

Figure: Figure 17: MBOi and MBOc vs. training epochs. (Left) PascalVOC (Right) COCO2017.
Refer to caption: x24.png

#### D.3 Sudoku solving

**Table 13: Board accuracy on Sudoku Puzzles. The harder dataset (OOD) has fewer conditional digits per example than the train set (17-34 in the harder dataset while 31-42 in the train set). We show the mean and std of the accuracy of models with different random seeds for the weight initialization.**
| Model | ID | OOD |
| --- | --- | --- |
| ItrSA $(B=1,T=16)$ | 99.7$\pm$0.3 | 14.1$\pm$2.7 |
| Transformer | 98.6$\pm 0.3$ | 5.2$\pm$0.3 |
| AKOrN^attn wo $\bm{\Omega}$ $(B=1,T=16)$ | 99.8$\pm 0.1$ | 16.6$\pm$2.2 |
| AKOrN^attn $(B=1,T=16)$ | 99.8$\pm 0.1$ | 16.6$\pm$2.1 |
| (+Test time extensions of internal steps) |  |  |
| ItrSA ($T_{\rm eval}=32$) | 95.7$\pm$8.5 | 34.4$\pm$5.4 |
| AKOrN^attn wo $\bm{\Omega}$ ($T_{\rm eval}=128$) | 100.0$\pm 0.0$ | 49.6$\pm$3.3 |
| AKOrN^attn ($T_{\rm eval}=128$) | 100.0$\pm 0.0$ | 51.7$\pm$3.3 |
| ($T_{\rm eval}=128$, Energy-based voting ($K=100$)) |  |  |
| AKOrN^attn wo $\bm{\Omega}$ | 100.0$\pm 0.0$ | 46.8$\pm$9.0 |
| AKOrN^attn | 100.0$\pm 0.0$ | 61.1$\pm$14.7 |
| SAT-Net (Wang et al., ) | 98.3 | 3.2 |
| Diffusion (Du et al., ) | 66.1 | 10.3 |
| IREM (Du et al., ) | 93.5 | 24.6 |
| RRN (Palm et al., ) | 99.8 | 28.6 |
| R-Transformer (Yang et al., ) | 100.0 | 30.3 |
| IRED (Du et al., ) | 99.4 | 62.1 |

##### D.3.1 Effect of the natural frequency term in energy-based voting

Interestingly, the model without the $\bm{\Omega}$ term does not give improvement with this post-selection, as the energy value and correctness are inconsistent (Fig 18). This implies the asymmetric term $\bm{\Omega}$ prevents the oscillators from being stuck in bad minima.

Figure: (a) without $\bm{\Omega}$
Refer to caption: x25.png

#### D.4 Robustness and calibration on CIFAR10

**Table 14: (An extended version of Tab 4) Robustness to adversarial attack (Adv) and Common Corruptions (CC) on CIFAR10 with the most severe corruption level (5). ^∗The adversarial attack is done by AutoAttack with EoT (Athalye et al., ). The max norm constraint of the adversrial perturbtions is set to 8/255. With $N=4$, the performance tendency of AKOrN is almost the same as ResNet except for the accuracy and uncertainty calibration on CIFAR10 with natural corruptions, which are moderately better with AKOrN^mix.**
|  | $\uparrow$ Accuracy | $\downarrow$ ECE |  |  |
| --- | --- | --- | --- | --- |
| Model | Clean | Adv | CC | CC |
| Gowal et al. () | 85.29 | 57.14 | 69.1 | 13.2 |
| Gowal et al. () | 88.74 | 66.10 | 70.7 | 5.6 |
| Bartoldson et al. () | 93.68 | 73.71 | 75.9 | 20.5 |
| Kireev et al. () | 94.75 | 0.00 | 83.9 | 9.0 |
| Diffenderfer et al. () | 96.56 | 0.00 | 89.2 | 4.8 |
| ViT | 91.44 | 0.00 | 81.0 | 9.6 |
| ResNet-18 | 94.41 | 0.00 | 81.5 | 8.9 |
| AKOrN^conv ($N=2$) | 88.91 | ^∗58.91 | 83.0 | 1.3 |
| AKOrN^mix ($N=2$) | 91.23 | ^∗51.56 | 86.4 | 1.4 |
| AKOrN^mix ($N=4$) | 93.51 | ^∗0.00 | 84.0 | 6.4 |

Figure: Figure 19: AKOrN’s adversarial examples are interpretable. Each pair of images is an original and the adversarially perturbed image ($\|\epsilon\|_{\infty}=64/255$). The text above each image indicates the class prediction made by the AKOrN model.
Refer to caption: x27.png

Figure: Figure 20: Visualization of clusters on CLEVRTex. The number of blocks in all models is two.
Refer to caption: extracted/5935374/gfx/multi_objects/cluster_imgs_high/clevrtex/img_10.png

Figure: Figure 21: Visualization of clusters on CLEVRTex-OOD. The number of blocks in all models is two.
Refer to caption: extracted/5935374/gfx/multi_objects/cluster_imgs_high/clevrtex_ood/img_7.png

Figure: Figure 22: Visualization of clusters on CLEVRTex-CAMO. The number of blocks in all models is two.
Refer to caption: extracted/5935374/gfx/multi_objects/cluster_imgs_high/clevrtex_camo/img_5.png

Figure: Figure 23: Visualization of clusters on PascalVOC. The number of clusters is set to 4.
Refer to caption: extracted/5935374/gfx/multi_objects/cluster_imgs_high/long_pascal_real.jpg

Figure: Figure 24: Visualization of clusters on COCO2017. The number of clusters is set to 7.
Refer to caption: extracted/5935374/gfx/multi_objects/cluster_imgs_high/long_coco_real.jpg

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Judging from the method, AKOrN seems to be much more computationally costly than the usual thresholding units. I noticed that the authors controlled the number of channels in AKOrN models, but showing the training/inference time would be helpful.
2. It would be helpful to include some explicit discussion on the weakness of the method. i.e., what tasks are probably more suitable for this method and why?
3. As mentioned above, more analysis might help establish the advantage of AKOrN. For example, the paper shows that the AKOrN models show better robustness, but it is not clear from the design that they are more robust. Thus analysis of the network dynamics on some toy tasks might be helpful (e.g., synthesized classification tasks on low-dim vectors).
soundness: 3
strengths: 1. The neuronal architecture draws inspiration from physics and neuroscience and is very novel in deep learning.
2. Model performance is impressive across a range of tasks.
3. Extensive experiments show that the method applies to mainstream architectures including convolutional neural networks and transformers.
summary: In this work, the authors proposed to use Kuramoto oscillatory neurons (AKOrN) to replace the usual thresholding units in deep neural networks. The basic neuron architecture can be applied to layers with different connectivity patterns, including convolutional and self-attention layers. Models with Kuramoto oscillatory neurons show promising performance in visual discovery and reasoning tasks, they are also more robust to adversarial attacks and input corruptions.
weaknesses: 1. While AKOrN shows promising performance on the selected tasks, the model might not work as well on more "classical tasks" such as image classification. 
2. Although the motivation of AKOrN is clear from a neuroscience perspective, how, and why the model shows superior performance in the tested task is not well understood.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 2
questions: - The format of figure references could be improved. I would recommend references as 'Fig. X'. Currently there is either the period missing or there's no space (e.g. 'Fig1' on line 96). Same with Tab X -> Table X. Should also be Figs for multiple figures (not Fig as in line 368)
- line 36: 'we follow a more modern dynamical view of neurons as oscillatory units'. Is this correct? Oscilliations have been studied in the brain for a long time; moreover, I would say that the majority of computational neuroscientists currently model neurons as scalar values.
- For table 3 results are presented with error bars over 5 seeds (which I favour), why is this not the case for any of the other results presented?
- line 121: 'we introduce a multi-dimensional vector version of the model' - this line is confusing as this was already employed by Chandra et al. and Lipton et al. (as cited). Perhaps replace with 'incorporate a mutli-dimensional model...'. On this subject could you clarify to me the exact novelty of the model? Is it that it's not been introduced in deep neural nets before? Or is it about the fact connections between oscillators J is not symetric in AKOrN? Relatedly, line 308 'our..models differ from these approaches in various aspects such as...' should really provide all fundemental differences beyond some examples. In fact, in any case what's the difference between 'asymmetric connections' and 'their symmetry breaking term'? 
- line 123 (and other places): what's C? the size of the neural population? this should be clarified
- line 245: what is 'proper' energy?
- line 323: I don't understand how these convolution/self-attention layers are iterated upon. What are the shared parameters?
- I didn't understand the implementation for the sudoku task. 1. is the entire 9x9 soduku grid fed all at once, or one digit at a time? and 2. how come x_i takes the value c_i when a digit is given, does the model no longer follow equation 2? Or is this just the initial value taken before the Kuramoto steps?
- line 425: 'the energy is a good indication of the solution's correctness'. Why? I can believe that stability is often a desirable thing in tasks but think this line deserves some elaboration
- line 464: would it be possible to give a brief description/overview of what 'common corruptions' are?
- line 506: why is it surprising that confidence and accuracy are linearly correlated? 
- line 527: 'the oscillator...'. I didn't understand the logic of this sentence, could you explain it to me please?

Other comments:

- the notation x_{chw} is confusing, I think x_{c,h,w} is better
- on line 201 the terms should be switched around (the initial state of the next block is the thing being defined). But also, how does this fit in with line 189 where X^(l,0) = X^l?
- the metrics in all of the tables should be clarified. I presume the unit is %, but e.g. in the case of Table 1 I'm not entirely sure what the metric is. % objects correctly identified?
- in the results section the language can be overly strong. For example 'far better' (line 377) and 'vastly better' (line 416) do not appear to me as very scientific language. 
- in the discussion there is talk of small N resulting in a strongly regularized model. Given that the size of N nor regularization was mentioned in the prior text, this comes rather abruptly as is confusing to the reader

typos: 
- line 150: 'set C a silhouette' -> 'set C as a silhouette'
soundness: 3
strengths: - the model is interesting and well defined
- the quantitative results appear impressive, though I am have some issues with their presentation (see below)
- the many numerical results presented, and comparisons to other models, suggests that overall a good deal of thought/time has been put in the manuscript
summary: the authors propose a neural ntwork architecture in which units are high-dimensional Kuramoto oscillatory neurons. This architecture leads synchronization or 'binding' between neurons which enables competitive performance in object discovery, reasoning and robustness.
weaknesses: - there are numerous minor issues with the text/presentation (see question below) which impacts the clarity of the paper 
- This not a field I have experience in, but I am not entirley convinced a fair comparison was made to other models for object discovery. Is the idea that AKOrN is a highly competitive model for models 'trained from scratch', and that comparing it to models with some pretrained parameters (e.g. Lowe et al. 2024) is unfair? I think this could be better clarified if so
- there is a notion from the abstract and sections 1/2 that this work is relevant to neuroscience, but the link feels quite tenuous to me. Is it reasonable to think of neurons as high-d points on a sphere?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: (1) For the claim in the paper: 
> “binding” between neurons leads to a form of competitive learning where representations are compressed  in order to represent more abstract concepts in deeper layers of the network. 

> In fact, neighboring neurons tend to cluster their activities, and clusters tend to compete to explain the input. This “competitive learning” has the advantage that information is compressed as we move through the layers, facilitating the process of abstraction by creating an information bottleneck. Additionally, the competition encourages different higher-level neurons to focus on different aspects of the input (i.e., they specialize). This process is made possible by synchronization: like fireflies in the night, neurons tend to synchronize their activities with their neighbors’, which leads to the compression of their representations.

Can author clarify the reference if such claim is well-documented? If it is the novel claim of the author, the author should modify the tone to avoid overclaim or provide rationale or experiments to make sense of such claims.

(2) (Weakness section). What is the contribution of the "Proj", and will there be synchronization, e.g. given input image, if we replace Proj into general linear sum (ablation of Proj)? And what is the difference between eq(6) and binding mechanism in Lowe's work? Since the binding mechanism itself can generate synchronization, how to disentangle the contribution from (6) to contribution from the "Kura idea".

(3) Can author shows the segmentation results on binary datasets, like Zheng2024. The different color of object may make it eaiser to break symmetry compared with binary images. e.g. in Fig.4 the segmentation is more likely based on color instead of object-instance (the highlight is treated as an object). Besides, what is the actual input dimmension of the image to the model? To what extent it is downscaled?
(4) Why the energy function still make sense even if the J and $\Omega$ is not constraint by symmetry?
(5) What is the rationale of robustness? Is it the general outcome of dynamical models or is it the outcome of this Kura model? I suggest compare with related non-synchrony dynamical ANN models to disentagle the causal-effect.
(6) Why energy function, or the synchrony level in the network, can indicate the task performance? Can the author explain that more solidly?
(7) For the Sudoku task, can the author compare with those Ising-based models? e.g. Hopfield network or modern hopfield network. So as to disentagle the contribution for the "Kura idea" from that generally comes from attractor dynamics.

In general, I would like to vote for the accept of the paper, but I will also revisit the evaluation based on the author's response and other reviewer's comments.
soundness: 3
strengths: I enjoyed reading this paper, impressed by the achievements on diverse tasks and also the authors's thoughts dispersed over the paper.  In brief, the strength of the paper includes:
- Soft grouping/clustering in neural network is an important question and often missing in main-stream ANNs.
- The proposed neuron model is neuroscientific motivated yet has concise fomula. The physical origin of the model makes it plausible to construct an "energy function", which turns out to be quite informative on the performance.
- While previous work on binding by synchrony mainly focuses on object discovery, the author extends these insights into reasoning tasks, and adversarial attack. And the performance on these new tasks is very successful.
- Lastly, the author provides a different conceptual angle for the binding by synchronization: the synchronization enables competitive learning, and therefore compresses representation to construct a bottle neck, which facilitate abstraction. As far as I know, it is a novel argument not presented in previous works and it also makes sense to me.
summary: - This paper developed a feature-rotation-like neuron models (high-dim feature rotation encodes synchrony binding), a promissingly neuron-level building block that can be combined with diverse network architectures, like CNN and transformer. 
- The capability of the new architecture is validated on three independent tasks: object discovery (synthetic & natural image), reasoning on Sudoku task, robustness w.r.t adversarial attack in CI
FAR10. 
- On each task, the proposed model shows SOTA performance and interesting shining points: (1) For object discovery, the model scales to natural image without pretrained DINO model; (2) for reasoning, the model's correctness is naturally implied by the "internal coherence", indicated by the task-independent energy measure; (3) for adversarial attack, robustness seems a built-in property of the dynamics and again, the energy could imply the accuracy.
weaknesses: For the weakness:
- It is not clear what is the actual conceptual contribution / nolvelty of the Kura model proposed in this paper compared with Lowe's Rotating Feature (and recent updates). For example, equation (6) is very similiar to the "binding mechanism activation" in  [1] and I have the intuition that this is the essential mechanism for the binding ability of model, similiar to [1]. If I am wrong, please correct me.
- When generalizing the original Kura model to high-dim, it is not clear whether the Proj in eq(2) is an 'equivalent' generalization of $sin(\theta_j-\theta_i)$. So it is not clear how much this term actually contribute to the computation of the network. Note that $sin(\theta_j-\theta_i)$ is a essential mechanism in original Kura model to guarentee the synchrony as a stable state and sin acts among each pair of neurons before the $\sum$, but Proj only acts after the $\sum$. (The first two concerns are mainly about how "Kura" the model is ?)
- Since the activity is constraints on the sphere, it loses the representational ability for encoding feature presence, which is an important conceptual diaviation from synchrony binding idea and is likely to limit its ability to infer in certain cases, e.g. when the feature is uncertain or weakly related to the task.
- On the experimental side, it mostly ignored the Lowe's model for comparison (Fig.3, Fig.4,Fig.5,Tab.1,Tab2, Tab3...) and other synchrony based model, e.g. [2].
- The code is not provided, so it is hard to evaluate the the reproducibility of the results.
- The author misses several essential recent works: e.g. [2] shows how oscillation emerges in spike-based model to bind features; [3] combine Kura model with ANNs and also achieved synchrony binding.
- The first two sentences in Abstract is quite confusing and the first paragraph in Introduction is not very informative.

[1] Löwe S, Lippe P, Locatello F, et al. Rotating features for object discovery[J]. Advances in Neural Information Processing Systems, 2024, 36.
[2] Zheng H, Lin H, Zhao R. GUST: combinatorial generalization by unsupervised grouping with neuronal coherence[J]. Advances in Neural Information Processing Systems, 2024, 36.
[3] Ricci M, Jung M, Zhang Y, et al. KuraNet: systems of coupled oscillators that learn to synchronize[J]. arXiv preprint arXiv:2105.02838, 2021.

---

confidence: 4
contribution: 4
initial_rating: 10
presentation: 4
questions: * on Lines 140-141, you argue that “we found that the use of C is necessary for stable trainings […] as a symmetry-breaking field”. But isn’t C also –and more simply—required for the network to receive inputs, and thus to perform any useful function?
soundness: 4
strengths: * A biologically inspired architecture, well described and motivated by neuroscientific findings
* A thorough set of experiments across diverse machine learning domains
* Meaningful comparisons with baselines revealing competitive performance
summary: The paper proposes a novel type of neural network where neurons’ activities are defined on the unit sphere (in N-dimensional space, where N is typically 2 or 4), and are updated based on Kuramoto equations. This can model oscillatory dynamics that can be leveraged for binding activations across neurons. The model is applied to various settings (unsupervised object discovery, image classification adversarial or natural noise robustness, Sudoku problem solving) and compared against relevant baselines (e.g. slot-based models, Transformers), each time demonstrating competitive performance. The model has clear potential to improve the state-of-the-art across domains.
weaknesses: * The dependence of the model on the choice of dimensionality N, although it is explicitly acknowledged in the paper, warrants further investigations (which can be the goal of future studies)
* Although baselines are roughly matched for memory/parameter size and flops, it is possible that GPU optimizations will make them more efficient than the AKOrN versions. There is unfortunately no discussion of this issue nor any report of absolute time required for training or inference. This could be added to the Appendix.

Meta-review: This paper takes on a long-standing idea from the computational neuroscience community, namely, that binding of different features in an input can be done using oscillations in unit activity that synchronize to indicate binding. The authors propose Artificial Kuramoto Oscillatory Neurons (AKOrN), which employs oscillatory units based on the Kuramoto (1984) model.

The authors claim that their system is capable of binding together features in an input (e.g. objects in an image). They further claim that this provides performance improvements across various tasks such as unsupervised object discovery, adversarial robustness, calibrated uncertainty quantification, and reasoning. They demonstrate the vailidity of these claims using a range of benchmarks, such as CLEVRTex, PascalVOC, and Sodoku puzzles.

The primary strength of this paper is that it manages to solve a long-standing technical issue (how to bind features) with a brain-inspired solution that no one else has previously been able to get working really well. It also provides strong empirical support for the claims, and illuminating analyses of the model. There were some weaknesses with clarity on the model's computational overhead and critical parts, but those were addressed in rebuttal. Altogether, it is a very strong paper. As such, a decision to accept for an oral was reached.

Decision: Accept (Oral)