SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We train the GUST on 54000 images composed of three random generated shapes (squares / up-triangle / down-triangle) located at different locations. We add salt&pepper noise to the input and train the GUST to denoise the image by its averaged firing rate during the convergent phase (last  $\tau_{2}$  steps). In this section, we show the simulation result of the trained GUST when perceiving a randomly selected image. The GUST is simulated 3-times longer than training to confirm the stability of the grouping. The SynScore and AMI is evaluated every  $\tau_{2}$  steps based on latest spiking patterns of length  $\tau_{2}$  during the simulation.  $\tau_{2} = 10$
The Synscore in Fig.4 gradually converge to 0.9 from 0, showing that the neurons in SCS evolve into a coherent state from random firing. Besides, the similar convergent curve of the AMI score further shows that the neuronal coherence successfully reflects the grouping assignment.
From the visualization result (Fig.4
top,bottom), it is more evident that in the convergent phase neurons related to the same object fires synchronously within a very short time window  $(\tau_{1} = 2)$  while neurons relevant to different objects fires out of phase. Therefore, the neurons and features are grouped in the time dimension softly, which means that features are not forced to fit into a discrete slot but seat softly in the neighbourhood of a gamma-like temporal coordinate (Fig.1b, Fig.4black curve of LFP in the convergent phase)[46]. Notably, the temporal coordinates are not predefined as in slot models but self-emerged as an emergent phenomenon during the simulation, providing the potential to group objects more flexibly and dynamically.
However, why GUST can learn to group single objects in the first place? It seems intriguing because the GUST is only trained to denoise the multi-object images so that no information about single objects or grouping is leaked to GUST during training. To unravel the mystery, we dive into details into the training process in the next section.
Figure 4: The simulation of a well-trained GUST on a randomly selected image. Top: coloring results based on the temporal structure of the spiking patterns. Middle: SynScore and AMI along the simulation, evaluated every 10 time steps, averaged over 5 random seeds. Bottom (in the black box): zoomed in spiking patterns during the initial 30 steps and last 30 steps. Upper pattern: snapshot of firing neurons at each time step (from left to right, from up to bottom); Middle pattern: spiking plot of neurons (similar to Fig.1b). The coloring is based on the ground truth grouping. Lower pattern (the back curve): the averaged population activity (LFP, similar to Fig.1b).
Figure 5: (a) AMI score and SynScore during the training epochs. (b): training / testing loss; (c) DAE reconstruction loss of single / multiple objects during the training process. Note the reconstruction losses of single / multiple objects have a ten-fold difference. (d) Visualization of SCS representation and DAE feedback along training epochs. From top to bottom: 4 examples selected in the 0/10/100/300 epoch during the training process, marked as I/II/III/IV in (a). Left: coloring of SCS neurons based on their spike firing time in the convergent phase. Right: spike firing sequence  $(s(t))$  and feedback sequence  $(\gamma'(t))$  (in the same convergent phase). In epoch 10 (marked as II), the feedback is a vague attention map while in epoch 100/300, the feedback is the precise reconstruction.
4.2 Learning to group without supervision
The AMI score and SynScore in Fig.5a is evaluated at the end of each epoch during the training, averaged over all testing images. The scores are based on the convergent phase in the simulation. Simulation in testing is 3-times longer than that during training. Surprisingly, even though the GUST is only trained to reconstruct the superimposed input by the averaged firing rate (eq.8), it seems to gradually discover a way to segregate the scene into a self-organized temporal structure and represent the building blocks by the neuronal coherence, which is illustrated by the simultaneously increasing AMI and SynScore along the training epochs (Fig.5a).
As expected, the denoising loss is decreased during the training (Fig.5b). Notably, it seems that the 3-times-longer testing process further improves the denoising performance since the denosing loss in the testing phase is much lower than that of the training phase (Fig.5b). Therefore, the GUST can stably generalize when using much longer iteration steps at test time.
To figure out what the DAE learns to reconstruct, we evaluate the reconstruction loss of the isolated DAE separated from the GUST architecture during the training (Fig.5c). The reconstruction loss is evaluated as the MSE loss between the image inputs and the denoising outputs of the DAE. Two types of images are used. One only contains a single randomly generated shape and the other contains three randomly generated shapes as used in training and testing. Surprisingly, the performance of reconstructing a single object improves with the training process (Fig.5c, single reconstruction loss). On the contrary, the reconstruction of multiple-object is not increased as saliently (ten-fold difference!). Denoising single objects by DAE is an important support to group single objects from multi-object images[37] (Also see Appendix). However, given that no explicit information about individual objects is provided to the DAE, how does it achieve the identification of a single object at all?
To this end, we further visualize the spike representation in the SCS  $(s(t))$  and the top-down attention given by the DAE  $(\gamma'(t))$ .
It can be seen that the DAE of GUST learns in a phasic manner (Fig.5d I to IV). At the beginning of the training process, it first learns to capture the global feature of the object (position) and feeds back coarse attention maps focusing on local regions covering each single object  $(\gamma^{\prime}(t)$  in Fig.5d II). Despite of the vague feedback, the gating effect of input (eq.2) further refines the firing pattern so that
each clear-cut shape fires in synchrony ( $s(t)$  in Fig.5d II) because irrelevant neurons in the attended local regions are kept from firing by zero driving signals. However, if objects come closer or even overlap, such rough attention mechanism may lead to incorrect groupings.
In the second stage, the DAE learns to further capture the local shape information to provide more precise denoising reconstructions (Fig.5d III,IV). Notably, the higher-quality feedback also improves the neuronal coherence and grouping performance in SCS (Fig.5d II to IV).
How can we understand the phasic learning phenomenon? Here we provide a heuristic explanation which highlights the role of the biological constraints. Although the learning target is the superimposed reconstruction by the averaged firing rate, the DAE needs to extract "adequate" information for top-down prediction. However, the biological constraint of limited temporal window CD (eq.6) makes it difficult at the beginning because neurons fire randomly (eq.3) and sparsely (eq.5) so that only a small part of firing can be observed at any temporal window (Fig.5d I). Thus, to achieve the objective with such limited observations, the DAE chooses to learn a detour, which is easier: Inferring a local location based on very sparse firings. This is the attention map in the first phase. However, this unexpected by-product has profound influence on the subsequent learning process: the local attention map breaks the symmetry of the neuronal responses to the input and the refractory dynamics(eq.5) thereby separate neuronal groups at alternative timings, albeit with lower precision. Thus, the original learning objective is "decomposed" into reconstructing each observed single object in the following training epochs. In turn, more precise top-down inference modulates the spike timing into a more structured synchrony. The ablation of biological constraints are shown in Fig.6a and detailed discussion of their role in grouping can be found in Appendix A.3.2.
4.3 Combinatorial generalization to novel scenes
Finally, we can ask: Does the GUST generalize the grouping mechanism in a systematical way? Specifically, does it generalize to novel scenes with more or fewer objects than ever encountered during training? For this purpose, we alter the number of objects both in the training data and testing data. The AMI results are shown in Fig.6b,c,d. For comparison, we compare the grouping performance with a similar benchmark model RC-bind[47] and Slot Attention[48], that also utilize autoencoder architectures to achieve grouping. Background is considered when computing the AMI.
The AMI results show that the GUST generalizes to scenes of varied number of objects (more than 0.5 in all cases and achieve 0.7 in 4 cases), outperforming the RC-bind (around 0.3) and slot-attention (failure). This indicates that the learned symbol-like entities can retain themselves and be composed according to the combinatorial nature of a novel scene.
Slot grouping. There is a broad range of research for object grouping in the artifi
cial neural network[47, 14, 49, 50, 51, 52]. Most works use a particular representation format, the slot. These representational slots are predefined into the network architecture to explicitly separate the objects with fixed capability. On the contrary, the temporal grouping in GUST assigns each group implicitly as an emergent property of the network dynamics. Without explicit human knowledge, the grouping is more general and flexible.
Figure 6: (a) Ablation of biological constraints. Compared with original GUST (all), removing stochasticity (-noise), refractoriness (-rfr), feedback delay (-delay), coincidence detectors (-CD), spiking non-linearity (-spike) will all harm the grouping performance (AMI) and even cause the failure of the grouping. (b)(c)(d) AMI score of GUST, RC-bind, and Slot Attention, trained / tested with 2/3/4 objects.
Complex-value grouping. Grouping based on complex-value approach[53] abstracts away the firing rate and firing timing into the amplitude and phase of the complex number, and readout the grouping information by specific complex-value activation functions. The approach has the desirable property of soft grouping. However, due to the limited range of the phase value, the number of objects is limited for grouping[54]. In contrast, the range of temporal dimension is potentially unlimited.
DASBE[37] is a recently developed hybrid model[55] to bind features through spike timing synchrony and generative attention. However, DASBE requires the isolated training of the DAE to denoise single objects. In contrast, instead of showing the existence of binding solution by construction, GUST shows that the grouping can even be (1) learned (2) in fully unsupervised way (3) in a single neural network directly perceiving (4) multiple objects through the (5) structural bias of biological constrains and the learned blocks can (6) generalize combinatorially.
Binding problem in neuroscience. How to bind distributed features in the neural system is one of the fundamental questions in neuroscience and cognitive literature. Towards the binding problem, two mainstream theories have been proposed. On the one hand, in the temporal binding theory (TBT)[17], it is reported that the temporal coherence states are related to the binding of features and is highly related to the constructive nature of the top-down feedback in a processing hierarchy[32]. On the other hand, feature integration theory (FIT)[56] is based on attention mechanism. It assumes a two-stage processing. In the first phase, features are extracted into a location map. In the second phase, a top-down attention "spotlight" searches the location map to group the features. It is still open which theory captures the nature of binding in the brain[57] and it is likely that brain uses both strategies[7]. Since the GUST minimally integrates both neuronal synchrony and attention mechanism in a closed way, it is promising that GUST can also serve as a unified data-driven modeling framework for bridging both TBT and FIT theories to explain the binding process in neural circuits in the future.
In this paper, we develop GUST, a brain-inspired unsupervised grouping system that dynamically finds a coherent grouping solution by the iterative processing between spiking dynamics and denoising feedback. The grouping is learned with a succinct denoising loss and can be generalized to represent scenes of combinatorially different structures.
This work was supported by the National Key Research and Development Program of China (no. 2021ZD0200300), the National Nature Science Foundation of China (nos. 62088102, 61836004).
[1] Paul C Quinn, Sherry Burke, and Amy Rush. Part—whole perception in early infancy: Evidence for perceptual grouping produced by lightness similarity. Infant Behavior and Development, 16(1):19-42, 1993.
[2] Hei-Rhee Ghim and Peter D Eimas. Global and local processing by 3-and 4-month-old infants. Perception & Psychophysics, 43(2):165-171, 1988.
[3] Allen E Milewski. Visual discrimination and detection of configurational invariance in 3-month infants. Developmental Psychology, 15(4):357, 1979.
[4] Eliane Vurpillot, Josette Ruel, and Anne-Yvonne Castrec. L'organisation perceptive chez le nourrisson: Réponse au tout ou à ses éléments. Bulletin de psychologie, 30(327):396-405, 1977.
[5] Johan Wagemans, James H. Elder, Michael Kubovy, Stephen E. Palmer, Mary A. Peterson, Manish Singh, and Rüdiger von der Heydt. A century of gestalt psychology in visual perception: I. perceptual grouping and figure-ground organization. Psychological bulletin, 138 6:1172-217, 2012.
[6] Christoph von der Malsburg. The correlation theory of brain function. In Models of neural networks, pages 95-119. Springer, 1994.
[46] Catherine Tallon-Baudry and Olivier Bertrand. Oscillatory gamma activity in humans and its role in object representation. Trends in cognitive sciences, 3(4):151-162, 1999.
[47] Klaus Greff, Rupesh Kumar Srivastava, and Jürgen Schmidhuber. Binding via reconstruction clustering. ArXiv, abs/1511.06418, 2015.
[48] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33:11525-11538, 2020.
[49] Klaus Greff, Sjoerd Van Steenkiste, and Jurgen Schmidhuber. Neural expectation maximization. Advances in Neural Information Processing Systems, 30, 2017.
[50] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, pages 2424–2433. PMLR, 2019.
[51] Christopher P Burgess, Loic Matthew, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901.11390, 2019.
[52] Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In International Conference on Learning Representations, 2020.
[53] David P. Reichert and Thomas Serre. Neuronal synchrony in complex-valued deep networks. CoRR, abs/1312.6115, 2014.
[54] Sindy Lowe, Phillip Lippe, Maja R. Rudolph, and Max Welling. Complex-valued autoencoders for object discovery. ArXiv, abs/2204.02075, 2022.
[55] Rongzhen Zhao, Zheyu Yang, Hao Zheng, Yujie Wu, Faqiang Liu, Zhenzhi Wu, Lukai Li, Feng Chen, Seng Song, Jun Zhu, Wenli Zhang, Haoyu Huang, M. Xu, Kaifeng Sheng, Qianbo Yin, Jing Pei, Guoqi Li, Youhui Zhang, Mingguo Zhao, and Luping Shi. A framework for the general design and computation of hybrid neural networks. Nature Communications, 13, 2022.
[56] Anne Treisman and Garry A. Gelade. A feature-integration theory of attention. Cognitive Psychology, 12:97-136, 1980.
[57] Pieter R. Roelfsema. Solving the binding problem: Assemblies form when neurons enhance their firing rate—they don't need to oscillate or synchronize. Neuron, 111:1003–1019, 2023.