GUST: Combinatorial Generalization by Unsupervised Grouping with Neuronal Coherence
Authors: Hao Zheng
Hui Lin
Rong Zhao
Venue: NeurIPS 2023 poster (2023)
Keywords: neuronal coherence
combinatorial generalization
perceptual grouping
unsupervised learning
TL;DR: grouping with emergent synchronization and the representation can generalize combinatorially

Abstract: Dynamically grouping sensory information into structured entities is essential for understanding the world of combinatorial nature. However, the grouping ability and therefore combinatorial generalization are still challenging artificial neural networks. Inspired by the evidence that successful grouping is indicated by neuronal coherence in the human brain, we introduce GUST (Grouping Unsupervisely by Spike Timing network), an iterative network architecture with biological constraints to bias the network towards a dynamical state of neuronal coherence that softly reflects the grouping information in the temporal structure of its spiking activity. We evaluate and analyze the model on synthetic datasets. Interestingly, the segregation ability is directly learned from superimposed stimuli with a succinct unsupervised objective. Two learning stages are present, from coarsely perceiving global features to additionally capturing local features. Further, the learned symbol-like building blocks can be systematically composed to represent novel scenes in a bio-plausible manner.

guST: Combinatorial Generalization by Unsupervised Grouping with Neuronal Coherence
Hao Zheng! Hui Lin' Rong Zhao"
Center for Brain-Inspired Computing Research,
Optical Memory National Engineering Research Center,
Tsinghua University China Electronics Technology HIK Group
Co. Joint Research Center for Brain-Inspired Computing.
IDG / McGovern Institute for Brain Research at Tsinghua University,
Department of Precision Instrument,
Tsinghua University, Beijing 100084, China.
hzheng-h17, (anhuii21)@aaill tsinghua. www.gaa.eu....oomoammmmom edu. cn r_zhao@mail. tsinghua. edu.
Abstract
Dynamically grouping sensory information into structured entities is essential for understanding the world of combinatorial nature. However, the grouping ability and therefore combinatorial generalization are still challenging artificial neural net- works. Inspired by the evidence that successful grouping is indicated by neuronal coherence in the human brain, we introduce GUST (Grouping Unsupervisely by Spike Timing network), an iterative network architecture with biological constraints to bias the network towards dynamical state of neuronal coherence that softly flects grouping information the temporal structure its spiking activity. We evaluate and analyze the model on synthetic datasets. Interestingly, the segregation ability is directly learned from superimposed stimuli with a succinet unsupervised objective. Two learning stages are present, from coarsely perceiving global features to additionally capturing local features. Further, the learned building blocks are systematically composed represent novel scenes in bio-plausible manner.
Introduction
Humans are able to effortlessly group the world in terms of abstract and reusable 'building blocks', and leverage those learned blocks to understand novel contexts. For example, there is a list of evidences that infants young 3-month old able group information from individual visual elements into wholistic perceptual units 2 3 4 This phenomenon referred as perceptual grouping (5) or binding in neuroscience and psychology literature 6 and such ability is believed to be fundamental for the compositional generalization of neural network models (7].
However, grouping ability challenges artificial neural networks (ANNs) for several reasons. Firstly, grouping from raw sensory input is in principle an unsupervised task with multiple possible solutions. For example, how to segment scene 8 how to chunk sensory stream into episodic memories 9 do not have a golden criterion. This excludes the solution through massive labeling and supervised learning [0]. Secondly, the representation of symbol-like content (the pattern of a single object) should be able to retain itself and be reused front various situations with a common format. Without proper constraints, simple network models can fall into ambiguities when trying to compose
EEqual contribution.
"Corresponding author.
The code is publicly available at: http:::://github...oo/oonsstette
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

those learned patterns, referred to as the binding problem Thirdly, the cost of representing all possible combinations statically and locally is exponentially high. limiting the capability of the local code to deal with increasingly complex scenes 12 13.. Lastly, the grouping ability should not be constrained to an explicit prior of separation [14] but can organize itself flexibly and dynamically according to the statistical structure of the input.
In this paper, we take insights from the neural system of the human brain to provide a brain-inspired grouping mechanism that mitigates the aforementioned challenges in ANNs. Neurons in the brain have a spiking nature. The firing activity in the high- highddimensionl neuronal space can have both spatial and temporal structures. The spatial structure of spike firing, usually referred to as receptive field, is related to certain objects[[5] or certain feature content of objects (orientation, color, texture, depth etc. [6). Occasionally, the neuronal firings evolve into coherent state, where large amounts of neurons interrelate in the millisecond timescale (Fig rigght)) Specifically, neurons encoding features of the same entity tend to fire synchronously and out of phase with neurons that encode features of different entities 18. At this time, the temporal structure of the spike firing emerges as the neuronal oscillation arises 19 20 In the cognition literature, this coherent state is highly related to awareness and successful percepts 22 23 24). According to these evidences, humans appear to use the spatial and temporal structure of spikes for representing contents and the grouping information among contents, respectively. Salient coherent states are indicators of the successful grouping, which enable better readout of contents by downstream neurons with non -on-linear dendrites sensitive to coincident events[25]. These insights have been developed as temporal binding theory [26] 27 or correlation brain theory 66]
in the neuroscience literature.
Compared with grouping mechanisms in ANNs, grouping with temporal neuronal coherence has several promising advantages. First, the grouping is a flexible and dynamical self-organized process without supervision or explicit prior of the separation. Second, the coherence level may help to encode the grouping uncertainty. Third, grouping in time dimension will naturally maintain common format Lastly, the (spatial) representational cost is reduced to a single pool of perceptual elements 7]. However, despite the advantages of grouping with temporal coherence, as far as we know, such grouping mechanism is seldom present in ANNs field due to the gap between temporal binding models and muddient-based ANNs. To be more specific, precise temporal codes rely on spiking neurons, which are non non-diffcrentiable and also require simulating the dynamics of neurons. Besides, the connection between neurons is required to be built-up by hebbian-like plasticity rule(28) instead of gradient-basse learning rules which necessitates a completely different training framework from that of ANNs. Lastly, it is not clear how neuronal coherence emerge within ANNs framework.
In this paper, we introduce the GUST network (short for Grouping Unsupervisely by Spike Timing network) that assigns the grouping information implicitly in spiking synchrony property of the network. The coherent state is achieved by iterative bottom-up top-down processing between a spike coding space (SCS) biased by biological constraints and denoising autoencoder (D 31) to provide delayed feedback. The top-down delayed feedback architecture for the temporal grouping is closely related to the constructive nature of the brain perception (2). Notably, the GUST is trained end-to-end by gradient-basee method and therefore compatible with ANNs framework. Besides, the grouping ability is learned implicitly with general unsupervised objective to directly denoise multi-object visual inputs. Lastly, once the GUST "sees" a given number of objects during training, the acquired grouping ability can be successfully generalized to visual inputs of different number of objects, showing its combinatorial generalization capability to understand scenes of different structures (eg. number of objects).
emergent
Designing GUST is non-trivial for four challenges. First, stochastic dynamics based on spike firing is non-differentiable and gradients cannot be naturally back-propagated as standard artificial networks. Additional biological constraints like refractoriness, non-linear dendrites, transmission delay and timescale difference aggravate the problem. Second, temporal grouping requires relatively high temporal resolution, which in turn implies a long simulation period, Such long simulations challenge the efficiency of backpropagation. Third, as far as know, there objective functions to train temporal grouping network directly from superimposed inputs. Lastly, the representation needs be explainable in terms of its precise temporal structure. To overcome challenges for pack-propagation, we approximate gradients in non-differentiable parts and short-cut the proppaaapant by the transmission delay (Sectior3.3). For unsupervised learning, surprisingly, a super succinct objective function to denoise superimposed visual inputs is shown to enable the GUST to learn segregate 3 once biased by list of biological constraints (c
trivial unsupervised
2

---

For evaluation, we develop methods to quantitatively evaluate neuronal coherence and its relation the grouping (Section3.6) based on non-euclidean metric of the precise temporal coding the visual cortex 33 By quantitative evaluation and qualitative visualization, the representation in GUST clearly explained.
In sum, the contribution of GUST is threefold. First, a gradient-based. unsupervised training framework developed to enable the GUST to group objects with neuronal coherence. Since the framework is compatible with ANNs, contributes to bridging the temporal binding theory and the ANNs framework. Second, the GUST the first temporal grouping model that learns group directly from multi-object inputs and is able to systematically generalize the mechanism to inputs of different number of objects. Third, we develop clustering method evaluate the neuronal coherence through a non-lineat metric that measures the distance between arbitrary spike trains directly based on their precise temporal structures.
Grouping in the temporal dimension
Grouping. In this work, grouping is formalized as a pixel-level segmenta- " tion of visual scenes composed of multi- ple objects (T). It is notable that while only visual inputs used for clarity, the general mechanism not restricted to certain modality. Simi- larly, we take the pixel-level elements more abstract features that dis- tributed in a high-dimensional represen- tation space, the grouping mechanism is not restricted to the superficial features as well.
grouping
neuronal coberence
no temporal stracture
emeeggen semporal coordinos
Grouping with neuronal coherence. Two separated timescales consid- ered: 7i < T2 (Fig During period of T2, assume that the accumulated firing rate of each neuron reflects certain features of an object. In other words, the spatial structure of a population of neu- rons encode which feature are present in the current scene. In this work, features formalized object-related pixels in an image. When the neuronal coherence state arrived, the firing of neurons is synchronized within a much narrower temporal window 71 so that the tempo-
firing zate
F2
firing rate
f2
combinatorial
generalization
phase
festing phose
Figure 1: (a) Grouping segmentation. (b) Grouping with neuronal coherence. The spike plot shows the emer- gence of the neuronal coherence (x-axis: time; y-axis: neuronal index). (c) The task scheme for combinatorial generalization
ral structure of spike firing emerges (Fig rigghht)) The temporal structure is reflected in the regular rhythmic pattern the population activity (LFP), which act as an emergent coordinate (2i; # Ri-2 in [p to assign the additional grouping information. It is notable that the temporal coordinate is not an explicit prior from human experts but an emergent property from the evolution of the network dynamics, Therefore, the grouping is flexible and dynamic.
Combinatorial generalization by grouping. we formalize "seeing" scene as successfully grouping the visual objects with neuronal coherence, the combinatorial generalization of the grouping in turn defined as the following question: Given that the GUST has only "seen" scenes composed of certain number of objects during training, does it generalize its grouping ability to "see" scenes composed different number objects, which structurally change the input distribution(Fig) [+) In other words, can GUST form coherent neuronal groups even in unseen situations? This question requires the GUST not only learns to group certain types of scenes into building blocks, but also manages to compose the building blocks to understand novel scenes.

---

(b)
(6)
4(x)
DAE
T
p P=
CD(C)
CD(9)
CD6
CD13
CDLs
,,
x
((t)
t+d+1
Figure 2: (a) The spiking neuron model. (b) The general architecture of GUST. (c) The unrolling of GUST through time.
3
Method
In this section, we first introduce the general architecture of GUST (Section 3.1 and provide details of each module (Section 3.2 Then, we introduce how three essential challenges are solved including designing the unsupervised objective function (Section 3.4). pack-propagation strategy to train the GUST (Section 3.5 and how grouping with neuronal coherence is evaluated (Section 3.6
3.1 General architecture GUST
How can the coherent states emerge from random firing? While it is still intricate question without a unified answer, we take inspiration from anticipating nature of the neural activity in processing hierarchy. The top-down feedback the primary sensory cortices carries predictions
about the features of the environmental stimuli and such modulatory top-down effects might influence the temporal structure coherence level of the neuronal responses [32 34 35). Therefore, the predictive feedback acts as positive cues to guide the grouping and coherence and is realized as the reconstructive output of autoencoder in GUST. Besides, spiking dynamics, pointed out in neuronal sampling theory (36), provides biological constraints that facilitate bypassing the energy barrier to switch between alternative stable states. In this way, the contents of different groups can get separated in time (See Appendix A.3.1).
The general architecture of GUST is composed of a spike coding space (SCS, green square in Fig 2 and denoising autoencoder (DAE, blue shadow in Fig 2p). The grouping solution represented as neuronal coherence in the SCS and is found by iterative bottom-up neuronal sampling in the SCS and top-down attentional feedback y from the DAE during the whole simulation period T. The feedback delayed by d steps.
3.2
Spike coding space (SCS)
The SCS layer of the same dimension as input and is composed of two-compartment pyramidal neuron models (h). Each pyramidal neuron receives the driving signals from the noisy visual inputs ă (during training") and the delayed modulatory feedback from the DAE output.
(t) (t d).
(1)
where 7 the delayed modulatory feedback to neuron i, 74 is the output DAE and is the delay period. Two sources of inputs multiplied determine the noiseless membrane potential p: (t):
px(t) (t) 44.
(2)
The stochastic neuron fires spike if it is not refractory and its membrane potential exceeds given threshold. More formally,
s; H (p.(t) + (t).
where e is the random noise drawn from uniform distribution U 1) and H is the non-linear activation function of the spiking neurons (threshold = 1):
1, 6; 0 u; 2 1.
Hus; 0z)
0. 6, 0 u, < 1.
(4)
0. 6, 0.
"During testing and visualization, it receives original inputs æ, Appendix A.7 (Salt&Pepper noise).

---

where (t) is the refractory variable for each neuron, which is non-zero if the neuron is refractory. A neuron falls into refractory period of length ő if it fires a spike. More formally.
{
0. 8 1)
s;(t) = 0 ^ 6, (t - 1) 0. 0. s, 0 A 6,
(((1)
(5)
8,
(1 1.
To sum up, the SCS integrates the driving input and the delayed modulatory feedback multiplicatively to determine the spike firing of neurons that are not in the refractory period. Once a neuron fires spike, it gets into the refractory period that is unable give another spike firing. The refractory dynamics provides the essential temporal competition to separate groups of different objects and acts as structural bias for grouping solution 337
3.3 Denoising autoencoder (DAE)
The coherent activities in SCS are readout by coincident detectors (CD) that integrate adjacent activities within very narrow time window (Tw 2) feed subsequent bottom-up processing.
(((() = CD((s((),...... ... ,i(t ,..
Twr + 1))
1,
Σ
8:(1-1 (()) 2 1.
(6)
F 0
0, st(t -t) <1
The bottom-up and top-down communication in a processing hierarchical formulated denoising autoencoder that is composed of an encoder f. latent layer @ and decoder (Fig@b). The encoder extracts more abstract features into the latent representation 0 and the decoder computes the top-down feedback to SCS. More formally,
(1)) = (((((())))
The dimension of input and output of DAE are kept the same as SCS and the dimension the latent space much smaller to form information bottle (eck(38] of the hierarchical processing.
3.4 Loss function
The first challenge is how grouping with neuronal coherence can be learned an unsupervised way similar to young infants. For this motivation, the objective function should leak any explicit grouping information.
We train the GUST to actively predict and denoise the multi-object input patterns at timescale 72- Only a simple unsupervised loss function identified:
L(x)
Σ 7t)) (=T-12
(
L(x) is the MSE distance between the original inputs æ and the averaged denoising predictions by the DAE (averaged over Ta time steps). 7 is the simulation length and we select the last segment of length 72 where is more possible to reach a convergent state (b) Z is a normalization factor.
Notably, the objective of the training process is consistent with the constructive nature of the perception and only related to multi-object inputs as well as the firing rate neurons. Therefore, grouping of objects and the neuronal synchrony are not explicitly guided by the objective function itself. Instead, they arise "effortlessly" as an emergent phenomenon along with anticipating the world 32), like an infant[4).
5

---

(x, mlny(
costycu)
Step 1:
Σ
Deo (5)55))))))
Step II:
((s)) argmineee Dyp(s" .5)
delete
cillet 1
22
S2
add
corlage
S
S4
c
shift
$6
S5
S6
56
Figure 3: The illustrative scheme of SynClu. Gray box illustrates how VP metric(Dv is computed.
8//5 is the spike train of neuron i/j. T is an arbitrary list of transformation to transform to 8j.
T(u) E {difta Dvr is defined as the minimal cost to transform one spike train s; to another 8y. The right side shows how SynClu cluster the neurons (exemplified for a single clustering iteration). Step I: from clustering assignment [(cx) to updated k cluster centers s T(cx) is the set of spike trains in the cluster cg. Step II: from k clustering centers to updated cluster assignment C(s)).
3.5 Gredients and unrolling
The second challenge is how to train the GUST in a gradient-bascl manner, given that the end-to-end training of the GUST is not naturally compatible with the standard deep learning framework due to the binary and dynamical nature of the SCS.
The challenge comes from two parts. One is the non-differential elements in SCS, like spiking
non-linearity (H, eq 4h. coincidence detector (CD, eqfō) and refractoriness (6,, ey Another is the long simulation period (> 100), which makes it harder backpropagate the gradients due to stability issues [39]. For the spiking non-linearity we approximate the Heavistep function H as a piece-wise
linear function to compute a surrogate gradient (4, 41). The same strategy is used to pass the gradient through the threshold of coincidence detectors. As for refractoriness, we cut off the gradient through the refractory variable ó; to simplify the computation graph. As a result, the refractory variable acts as a gating variable so that the gradients can only be passed through neurons that are not in their refractory periods in the unrolling graph (green dots in the SCS in Fig 2c). Lastly, the transmission delay naturally shortcuts the depth of the backpropagation for d times so that the backpropagation of
errors along the simulation period is much more efficient (green lines in Fig D The green lines and green dots in ( stand for the shortcuts neurons where the gradients can be back suck-propagated.
3.6 Evaluation
The last challenge is how to quantitatively measure the neuronal coherence in a high dimensional spike coding space and evaluate the grouping performance based on neuronal synchrony.
The evaluation of the coherent neuronal groups requires clustering method directly based on the temporal structure of the spike firings. Therefore, we develop SynClu (Fig a clustering method that combines a spike-timing-based metric and K-medoids clustering together. The Victor-Purpun metric (Dvp) that we use is a classical non-Euclidean metric for analyzing the precise temporal coding in the visual cortex (33) The K-medoids [42] thereby iteratively assign K clustering centers(sf) among given neurons (s,) to minimize the inner-cluster VP distance. The convergent clustering assignment is compared with the ground truth grouping by the adjusted mutual information (AMI) to evaluate the grouping quality (1 is perfect grouping while 0 is the chance level). The Silhouette score [c4re/44] indicates the inner-cluster coherence level of the spiking pattern, therefore is denoted as the synchrony score (SynScore).
4 Experiments and results
Since the grouping is formalized as a pixel-level segmentation of visual scenes composed of multiple objects (Section 2 Grouping), synthetic images composed of multiple Shapes[45] are generated for evaluation. By generating a different number of shapes [potom,, Section 2 Combinatorial generalization by grouping), we can control the statistical structure of the training data and testing data, as well as access to the ground truth of grouping assignment. This allows us to evaluate the grouping performance and further evaluate the combinatorial generalization ability of the approach.

---

In the following sections, we unfold the grouping property of GUST step by step. We first demonstrate the emergence of neuronal coherence during the simulation of a well-trained GUST, which softly indicates the grouping assignments (Section 4.1 To figure out why the intriguing grouping capability learned without explicit supervision, we dive into the details of the training process. Notably, it is shown that the DAE of GUST gradually learns to denoise single objects instead of multiple Further visualization of the oulput attention map Y shows that the DAE
objects (Section[4.2) learns in phasic manner, from coarsely predicting the global features (position) to additionally capturing local features (shapes) of single objects, which may provide heuristic explanation of the mechanism. Lastly, given that GUST can learn to segregate multi-object scenes into grouped single-object representations during training. we study whether the single-object representation can be composed to understand novel scenes with more or fewer objects than ever encountered during training (Section 4.3). In other words, we ask whether the GUST generalizes its grouping mechanism in combinatorial manner. For visualization and analysis in diverse cases, see Appendix A.9 A.11.
4.1 Emergence of neuronal coherence for grouping
We train the GUST on 54000 images composed of three random generated shapes (squares up-triangle/ down- triangle) located at different locations. We add salt&pepper noise to the input and train the GUST to denoise the im- age by its averaged firing rate during the convergent phase (last 72 steps). In this section, we show the simula- tion result of the trained GUST when perceiving a randomly selected image. The GUST is simulated 3-times longer than training to confirm the stability of the grouping. The SynScore and AMI is evaluated every 72 steps based on latest spiking patterns of length T2 during the simulation. T2 = 10.
1.0
AMI
Tran until
SynScore
0.0
100
150 Time Step
200
250
Input
Convenge
Trun
Neuronal
MA A 2
The simulation of well-trained GUST on a randomly selected image. Top: coloring results based on the temporal structure of the spiking patterns. Middle: SynScore and AMI along the simulation, evaluated every 10 time steps, averaged over 5 random seeds. Bottom (in the black box): zoomed in spiking patterns during the initial 30 steps and last 30 steps. Upper pattern: snapshot of firing neurons at each time step (from left to right, from up to bottom); Middle pattern: spiking plot of neurons (similar to Fig [T)). The coloring is based on the ground truth grouping. Lower pattern (the back curve): the averaged population activity (LFP, similar [ig[Tp).
Figure
The Synscore in Fig 4 gggaduully con- verge to 0.9 from O, showing that the neurons in SCS evolve into coher- ent state from random firing. Besides, the similar convergent curve of the AMI score further shows that the neu- ronal coherence successfully reflects the grouping assignment.
From the visualization result (Fig bottoo),, it is more evident that the convergent phase neurons related to the same object fires synchronously within a very short time window (r) 2) while neurons relevant to different objects fires out of phase. Therefore, the neurons and features are grouped in the time dimension softly, which means that features are not forced to fit into discrete slot but seat softly in the neighbourhood of a gamma-like temporal coordinate (Fig [p. Fig 4lack curve of LFP in the convergent phase) (46] Notably, the temporal coordinates are not predefined in slot models but self-emerged as an emergent phenomenon during the simulation, providing the potential to group objects more flexibly and dynamically.
However, why GUST can learn to group single objects in the first place? seems intriguing because the GUST is only trained to denoise the multi-object images so that no information about single objects or grouping leaked to GUST during training. To unravel the mystery, we dive into details into the training process in the next section.
7

---

(a)
0.5
Synocoe
0.1
100
150
200
230
300
epoch
(b)
(c)
mihlo
III
opoct
apachh
Figure 5: (a) AMI score and SynScore during the training epochs. (b): training / testing loss; (c) DAE reconstruction loss of single / multiple objects during the training process. Note the reconstruction losses of single multiple objects have a ten-fold difference. (d) Visualization of SCS representation and DAE feedback along training epochs. From top to bottom: examples selected in the 0/10//00/300 epoch during the training process, marked as I/IL/III/VV in (a). Left: coloring of SCS neurons based on their spike firing time the convergent phase. Right: spike firing sequence (s()) and feedback sequence (7/ (4)) (in the same convergent phase). In epoch 10 (marked as II), the feedhack vague attention map while in epoch 100/300, the feedback the precise reconstruction.
4.2 Learning to group without supervision
The AMI score and SynScore in Fig 5p evaluated at the end of each epoch during the training, averaged over all testing images. The scores are based on the convergent phase in the simulation. Simulation in testing is 3-times longer than that during training. Surprisingly, even though the GUST only trained reconstruct the superimposed input by the averaged firing rate (eq(). seems to gradually discover way to segregate the scene into self-organized temporal structure and represent the building blocks by the neuronal coherence, which is illustrated by the simultaneously increasing AMI and SynScore along the training epochs (F)
As expected, the denoising loss is decreased during the training (p). Notably, it seems that the 3-times-longer testing process further improve the denoising performance since the denosing loss in the testing phase is much lower than that of the training phase (Fig 5p). Therefore, the GUST can stably generalize when using much longer iteration steps at test time.
To figure out what the DAE learns reconstruct, evaluate the reconstruction loss of the isolated DAE separated from the GUST architecture during the training (Fig The reconstruction loss is evaluated the MSE loss between the image inputs and the denoising outputs of the DAE. Two types of images are used. One only contains a single randomly generated shape and the other contains three randomly generated shapes used in training and testing. Surprisingly, the performance of reconstructing single object improves with the training process (Fig single reconstruction loss). On the contrary, the reconstruction of multiple-object is not increased saliently (ten-fold difference!). Denoising single objects by DAE is an important support to group single objects from multi-object images (37] (Also see Appendix). However, given that explicit information about individual objects is provided to the DAE, how does it achieve the identification of single object at all?
To this end, we further visualize the spike representation the SCS (s(t) and the top-down attention given by the DAE (r'(t) (t)).
can be seen that the DAE of GUST learns in a phasic manner (Fig 1 to IV), At beginning
of the training process, it first learns to capture the global feature of the object (position) and feeds back coarse attention maps focusing on local regions covering each single object (7'(1) in 5 II). Despite of the vague feedback, the gating effect of input (cq 2 further refine the firing pattern that

---

each clear-cut shape fires in synchrony (s(t) in 5 II) because irrelevant neurons in the attended local regions are kept from firing by zero driving signals. However, if objects come closer or even overlap, such rough attention mechanism may lead to incorrect groupings.
In the second stage, the DAE learns to further capture the local shape information to provide more precise denoising reconstructions 5 III,IV). Notably, the higher-quality feedback also improves the neuronal coherence and grouping performance in SCS (Fig 5 II IV).
How can we understand the phasic learning phenomenon? Here we provide a heuristic explanation which highlights the role of the biological constraints. Although the learning target is the superim- posed reconstruction by the averaged firing rate, the DAE needs to extract "adequate" information for top-down prediction. However, the biological constraint of limited temporal window CD (eq 6 makes difficult at the beginning because neurons fire randomly (eq) 3 and sparsely (eq so that only small part of firing can be observed any temporal window (Fig I). Thus, to achieve the objective with such limited observations, the DAE chooses to learn a detour, which is easier: Inferring local location based on very sparse firings. This the attention map the first phase. However, this unexpected by-product has profound influence on the subsequent learning process: the local attention map breaks the symmetry of the neuronal responses the input and the refractory dynamics(eq)) 5 thereby separate neuronal groups at alternative timings, albeit with lower precision. Thus, the original learning objective is "decomposed" into reconstructing each observed single object in the following training epochs. In turn, more precise top-down inference modulates the spike timing into more structured synchrony. The ablation of biological constraints are shown in Fig 6 and detailed discussion of their role in grouping can be found in Appendix A.3.2.
4.3 Combinatorial generaization novel scenes
Finally, we can ask: Does the GUST generalize the grouping mechanism in a systematical way? Specifically, does generalize to novel scenes with more fewer objects than ever encountered during training? For this purpose, we alter the number of objects both in the training data and testing data. The AMI results are shown in Fig For comparison, we compare the grouping performance with a similar benchmark model RC- RC-bind (47) and Slot Attention that also utilize autoencoder architectures to achieve grouping. Background considered when computing the AMI.
The AMI results show that the GUST generalizes scenes of varied number of objects (more than 0.5 in all cases and achieve 0.7 in 4 cases), outperforming the RC-bind (around 0.3) and slot-attention (failure). This indicates that the learned symbol-like entities can re- tain themselves and be com- posed according to the com- binatorial nature of a novel scene.
(8)
(b)
Train #obj
0.6
a1
spike
dalay
-CD
noloc
Tlest #obj.
(c)
Train #abj. =
(d)
Train #obj.
Test eab).
Test Robj,
Figure 6: (a) Ablation of biological constraints. Compared with original GUST (all), removing stochasticity (-noise), refractoriness (-rfr), feedback delay (-delay), coincidence detectors (-CD), spiking non-linearity (-spike) will all harm the grouping performance (AMI) and even cause the failure of the grouping. (b)())d) AMI score of GUST, RC-bind, and Slot Attention, trained / tested with 2/3/4 objects.
5 Related work
Slot grouping. There is a broad range of research for object grouping the artifi-
cial neural network [47 [4 49 50
Most works use particular representation format, the slot. These representational slots are predefined into the network architecture to explicitly separate the objects with fixed capability. On the contrary, the temporal grouping in GUST assigns each group implicitly as an emergent property of the network dynamics. Without explicit human knowledge, the grouping is more general and flexible.
51 52

---

Complex-value grouping. Grouping based on complex-value approach [53] abstracts away the firing rate and firing timing into the amplitude and phase of the complex number, and readout the grouping information by specific complex-value activation functions. The approach has the desirable property of soft grouping. However, due to the limited range of the phase value, the number of objects is limited for grouping(54) In contrast, the range of temporal dimension is potentially unlimited.
DASBE 37 is recently developed hybrid m55e to bind features through spike timing synchrony and generative attention. However, DASBE requires the isolated training of the DAE to denoise single objects. In contrast, instead of showing the existence of binding solution by construction, GUST shows that the grouping can even be (1) learned (2) in fully unsupervised way (3) in a single neural network directly perceiving (4) multiple objects through the (5) structural bias of biological constrains and the learned blocks can (6) generalize combinatorially.
Binding problem in neuroscience. How to bind distributed features in the neural system is of the fundamental questions in neuroscience and cognitive literature. Towards the binding problem, two mainstream theories have been proposed. On the one hand, in the temporal binding theory (TBT) it is reported that the temporal coherence states are related to the binding of features and is highly related to the constructive nature of the top-down feedback in processing hierarchy (32). On the other hand, feature integration theory (56) is based on attention mechanism. It assumes a two-stage processing. In the first phase, features are extracted into location map. In the second phase, top-down attention "spotlight" searches the location map to group the features. It is still open which theory captures the nature of binding in the brain [57] and is likely that brain uses both strategies 77]. Since the GUST minimally integrates both neuronal synchrony and attention mechanism in closed way, promising that GUST can also serve as a unified data-driven modeling framework for bridging both TBT and FIT theories to explain the binding process in neural circuits in the future.
6 Conclusion
In this paper, develop GUST, a brain-inspired unsupervised grouping system that dynamically finds a coherent grouping solution by the iterative processing spiking dynamics and denoising feedback. The grouping is learned with a suceinct denoising loss and can be generalized to represent scenes of combinatorially different structures.
between
7 Acknowledgements
This work supported by the National Key Research and Development Program of China (no. 2021ZZ00000000), the National Nature Science Foundation of China (nos. 62088102, 61836004).
References
[1] Paul C Quinn, Sherry Burke, and Amy Rush. Part-whole perception in early infancy: Evidence for perceptual grouping produced by lightness similarity. Infant Behavior and Development, 16(1):19 42, 1993.
[2] Hei-Rhee Ghim and Peter D Eimas. Global and local processing by 3-and 4-month-old infants. Perception & Psychophysics, 1355)11 1988.
[3] Allen E Milewski. Visual discrimination and detection of configurational invariance in 3-month infants. Developmental Psychology, 15(:3357, 1979.
[4] Eliane Vurpillot, Josette Ruel, and Yvonne Castrec. L'organisation perceptive chez le nourrisson: Réponse au tout ou à ses éléments. Bulletin de psychologie, 306327):396-455, 1977.
[5] Johan Wagemans, James H. Elder, Michael Kubovy, Stephen E. Palmer, Mary A. Peterson, Manish Singh, and Rüdiger von der Heydt. A century of gestalt psychology in visual perception: I. perceptual grouping and figure-ground organization. Psychological bulletin, 138 6: 172-211, 2012.
[6] Christoph von der Malsburg. The correlation theory of brain function. Models of neural networks, pages 95-119. Springer, 1994.
10

---

[7] Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. ArXiv, abs/2012.05208, 2020.
[8] Venkata Satya Sai Ajay Daliparthi. Ikshana: A theory of human scene understanding mechanism. ArXiv, abs/2101. 2021.
[9] Arnaud D' Argembeau, Olivier Jeunehomme, and David Stawarczyk. Slices of the past: how events temporally compressed in episodic memory. Memory, 30:43 48, 2021.
[10] Ross B. Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. 2014 IEEE Conference on Computer Vision and Pullern Recognition, pages 580-587, 2014.
[11] Christoph Von der Malsburg. The what and why of binding: the modeler's perspective. Neuron, 24(1):95-104, 1999.
[12] Paul Smolensky. Tensor product variable binding and the representation of symbolic structures connectionist systems. Artificial intelligence, 46(1-2):159-211 1990.
[13] Kumihiko Fukushima. Neural network model for mechanism of pattern recognition unaffected by shift in position-neeoconniton. IEICE Technical Report, A, 62(10))658-66) 665, 1979.
[14] Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. Advances in Neural Information Processing Systems, 29, 2016.
[15] David C. Plaut and James L. McClelland. Locating object knowledge in the brain: comment on bowers's 's (2009) attempt to revive the grandmother cell hypothesis. Psychological review, 117 1:284 8, 2010.
[16] Elie Bienenstock, Leon N. Cooper, and Paul W. Munro. Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. In The Journal of neuroscience: the official journal of the Society for Neuroscience, 1982.
[17] Andreas Karl Engel and Wolf Singer. Temporal binding and the neural correlates of sensory awareness. Trends in Cognitive Sciences, 5: 6-25, 2001.
[18] Wolf Singer. Distributed processing and temporal codes in neuronal networks. Cognitive neurodynamics, (()):189-196, 2009.
[19] Wolf Singer. Neuronal oscillations: unavoidable and useful? European Journal of Neuroscience, 48(7):2389-2398, 2018
[20] Gabriele Nase, Wolf Singer, Hannah Monyer, and Andreas K Engel. Features of neuronal synchrony in mouse visual cortex. Journal of Neurophysiology, 1152):1 2003.
[21] Andreas Karl Engel, Pascal Fries, Peter König, Michael Brecht, and Wolf Singer. Temporal binding, binocular rivalry, and consciousness. Consciousness and Cognition, 8:128 151, 1999.
[22] Marius Usher and Nick Donnelly. Visual synchrony affects binding and segmentation in perception. Nature, (14466689)179-1198 1998.
[23] Peter Uhlhaas and Wolf Singer. Neural synchrony in brain disorders: relevance for cognitive dysfunctions and pathophysiology. neuron, 52(1):155-168, 2006.
[24] Wolf Singer. Large scale temporal coordination of cortical activity as prerequisite for conscious experience. Pragmatics & Cognition, 18(3):570-583, 2010.
[25] Moshe Abeles. Role of the cortical neuron: integrator or coincidence detector? Israel journal of medical sciences. 18 1:83-92, 1982.
[26] Wolf Singer and Charles M Gray. Visual feature integration and the temporal correlation hypothesis. Annual review of neuroscience, 18(1):555-586 1995.
11

---

[27] Wolf Singer. Neuronal synchronization: A solution to the binding problem. The mind-mri bbaain continuum: Sensory processes, pages 101-130, 1996.
[28] Sen Song, Kenneth D. Miller, and L. F. Abbott. Competitive hebbian learning through spike- timing-dependent synaptic plasticity. Nature Neuroscience, 3:919-926, 2000.
[29] Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, R. Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with back proppaattn network. In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 2. Morgan- Kaufmann, 1989.
[30] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre- Anerre-Attiine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096-1103, 2008.
[31] Sven Behnke. Learning iterative image reconstruction in the neural abstraction pyramid. International Journal of Computational Intelligence and Applications, 1(04):427-438 2001.
[32] Andreas K Engel, Pascal Fries, and Wolf Singer. Dynamic predictions: oscillations and synchrony in top down processing, Nature Reviews Neuroscience, 2(10):744-716, 2001.
[33] Jonathan D Victor and Keith P Purpura. Nature and precision of temporal coding in visual cortex: a metric-space analysis. Journal of neurophysiology, 76(2):1310-1326, 1996.
[34] Antonio Damasio. The brain binds entities and events by multiregional activation from convergence zones. Neural computation, 1(1):123-1322 1989.
[35] Antonio R Damasio. Synchronous activation multiple cortical regions: a mechanism for recall. In Seminars in the Neurosciences, volume 2, 1990.
[36] Lars Buesing, Johannes Bill, Bernhard Nessler, and Wolfgang Maass. Neural dynamics as sampling: A model for stochastic computation in recurrent networks of spiking neurons. PLoS Computational Biology, 7, 2011.
[37] Hao Zheng, Hui Lin, Rong Zhao, and Luping Shi. Dance of snn and ann: Solving binding problem by combining spike timing and reconstructive attention. S. Koyejo, Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Process- ing Systems, volume 35, pages 31430-3 1330001145 31443. Curran Associates, Inc., 2022.
[38] Naftali Tishby and Noga Zaslavsky. Deep learning and the Information Bottleneck principle. In IEEE Information Theory Workshop (ITW), 2015.
[39] Seol-Hyun Noh. Analysis of gradient vanishing of mns and performance comparison. Inf., 12:442, 2021.
[40] Emre O. Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. JEEE Signal Processing Magazine, 36:51-63, -6, 2019.
[41] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. training high-performance spiking neural networks. Frontiers in Neuroscience, 12, 2018.
backpropagation for
[42] Leonard Kaufman and Peter J. Rousseeuw. Clustering by means of medoids. 1987.
[43] Nguyen Xuan Vinh, Julien Epps, and James Bailey. Information theoretic measures for cluster- ings comparison: Variants, properties, normalization and correction for chance. The Journal of Machine Learning Research, 11:2837-2854, 2010.
[44] Peter Roussecuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65, 1987.
[45] David Reichert, Peggy Seriès, and Amos Storkey. A hierarchical generative model of recurrent object-based attention in the visual cortex. In ICANN, 2011.
12

---

[46] Catherine Tallon- -allryy and Olivier Bertrand. Oscillatory gamma activity in humans and its role object representation. Trends in cognitive sciences, 3(4):151-162, 1999.
[47] Klaus Greff, Rupesh Kumar Srivastava, and Jürgen Schmidhuber. Binding via reconstruction clustering. ArXiv, 0oos/1511.6441 2015.
[48] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centrie learning with slot attention. Advances Neural Information Processing Systems, 33:1 1521-125111 11538, 2020.
[49] Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. Advances in Neural Information Processing Systems, 30, 2017.
[50] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object repre- sentation learning with iterative variational inference. In International Conference on Machine Learning, pages 2424-2433. PMLR, 2019.
[51] Christopher Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa- tion. arXiv preprint arXiv:1901.11399 2019.
[52] Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Genera- tive scene inference and sampling with object-centric latent representations. In International Conference on Learning Representations, 2020.
[53] David P. Reichert and Thomas Serre. Neuronal synchrony in complex-valued deep networks. CoRR, ftbb//311..113. 2014.
[54] Sindy Lowe, Phillip Lippe, Maja R. Rudolph, and Max Welling. Complex-valued autoencoders for object discovery. ArXiv, abs/2204.02075. 2022.
[55] Rongzhen Zhao, Zheyu Yang. Hao Zheng. Yujic Wu, Faqiang Liu, Zhenzhi Wu, Lukai Li. Feng Chen, Seng Song, Jun Zhu, Wenli Zhang, Haoyu Huang, M. Xu, Kaifeng Sheng, Qianbo Yin, Jing Pei, Guoqi Li, Youhui Zhang, Mingguo Zhao, and Luping Shi. A framework for the general design and computation of hybrid neural networks. Nature Communications, 13, 2022.
[56] Anne Treisman and Garry A. Gelade. A feature-inteeraaioo theory of attention. Cognitive Psychology, 12:97-136. 1980.
[57] Pieter R. Roelfsema. Solving the binding problem: Assemblies form when neurons enhance their firing rate-they don't need to oscillate or synchronize. Neuron, 111: 111:1003-1019, 2023.
13


Reviews:
[{"review_id": "SMqzYkYRL2", "reviewer": "Reviewer_dadF", "summary": "This paper tackles the challenge of grouping information from individual visual elements into whole perceptual units, i.e., how compositional generalization is achieved, through proposing a GUST (Grouping Unsupervisely by Spike Timing network) that leverages spiking synchrony for grouping. This framework introduced a spike coding space (SCS) and a denoising autoencoder (DAE) to control for temporal grouping, enabled representing scenes of combinatorially different structures in simulated datasets.", "questions": "See corresponding questions and suggestions in the Weaknesses section. ", "limitations": "This paper proposed a brain-inspired algorithm and assumed representations of combinatorially different structures happen within the same spiking neural network for classification. However, we already know this is not entirely true in the mammalian brain circuits. Often there is a clear hierarchical and modular structure, where low-level features were represented in early-stage cortical regions, which further enables compositional encoding. From this aspect, the paper seems limited in bringing more insights for biological interpretations. From the aspect of computational algorithms, the paper does not demonstrate superior performance compared to other SOTA unsupervised learning through deep neural networks. Therefore, this paper seems limited in terms of bringing higher impacts for both neuroscience and machine learning fields. I would suggest the authors to elaborate the limitations from biological interpretations, and the future directions of this work.  ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper combined insights from the neuroscience literature including correlation brain theory and temporal coding to enable grouping mechanisms in unsupervised learning. \n2. The paper tackles a non-trivial challenge wrt designing a spike timing neural network properly, given that spiking is non-differential, requirement of high temporal precision, and interpretations of representations. Specifically, the proposed algorithm enables learning directly from multi-object inputs. It also enables a gradient-based training framework. The paper further leveraged non-linear metrics to evaluate grouping performance based on neuronal synchrony. Overall, the paper proposed elegant solutions for these challenges. \n3. The paper examined the effect of biological constraints in their algorithm by providing an ablation experiment, and demonstrated  superior grouping performance in novel scenes than baseline models RC-bind and Slot Attention. ", "weaknesses": "1. I'd love to see stronger evaluation benchmarking with more metrics, given that unsupervised clustering often gives different results based on how they are tuned and often is biased by noise. The authors leveraged the adjusted mutual information (AMI) to evaluate the grouping quality, and used Silhouette score (SC) to evaluate inner-cluster coherence level. For example, SC is generally higher for convex clusters, and it is known that different metrics sometimes choose different preferred method/algorithm in noisy clustering cases.   Despite AMI adjusts for chance, AMI is high when there are pure clusters in the clustering solution (see [ref](https://jmlr.csail.mit.edu/papers/volume17/15-627/15-627)). It is unclear of the dataset clustering distribution to evaluate potential bias in their metrics. Specifically, I'd like to see more analysis on how their benchmarking performance varies across different random seeds, number of clusters, etc. Moreover, I'd recommend to add benchmarking results using ARI (adjusted rand index), to provide additional information. \n2. The grouping evaluation focused on clustering different objects. This is very limited to persuade the readers on how their algorithm demonstrated superior compositional grouping. I'd recommend to add more evaluations on different aspects of compositions. For example, instead of classify different shape objects, is the proposed algorithm able to learn compositional structures in an object. For example, can different body parts of a complicated object being learned and segmented? \n3. A major challenge of unsupervised learning on multi-object inputs is when objects are overlapped with different degrees, or collapsed with different noise levels. There is no analysis to quantify the distribution in datasets. Although image in Figure 4 indicates a partial overlap of triangle and square shapes, Figure 5 suggests those shapes are mostly clearly separated in space. It is thus unclear if the proposed algorithm can deal with grouping and unsupervised learning in these scenarios. \n4. The whole idea of leveraging temporal coherence as a way for learning generalization and classification is not new, and has been explored in the literature. For example, this [paper](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005137) shows that a spiking network driven by input timing dependent plasticity (ITDP) could perform visual classification task well and to generalize to unseen datasets. This [paper](https://arxiv.org/pdf/1807.10936.pdf) shows a hierarchical spiking architecture in which motion can selectively emerges in an unsupervised fashion. Additionally, how combinatorial coding can emerge in the cortex also has been discussed in [paper 1](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2693376/), [paper 2](https://www.princeton.edu/~wbialek/our_papers/osborne+al_08.pdf), and [paper 3](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4605134). This paper did not cite these relevant work, nor discussed how their proposed algorithm outperforms or links with these previously proposed combinatorial coding mechanisms. It is thus difficult to evaluate the novelty and whether this paper brings additional new insights in bio-inspired algorithms (see more comments in Limitations).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__A1 Benchmarking analysis and selective bias__: See __A1, A2 in global response__.\n\n__A2. Concerns on hierarchical case (brain organization / hierarchical grouping)__.\n\nFirst,  we consider combinatorial generalization of _single-level grouping_ in this paper, where varying object number seems to be complete for composition, instead of _compositional grouping_. \n\nSecond, the GUST architecture in this paper serves as the basic building block of more complex systems for more complex binding (bigger picture). Therefore, the GUST architecture is consistent with hierarchical and modular structure of mammalian brain and can be generalized to account for hierarchical grouping of ‘compositional structures’: GUST is each ‘cortical column’ to group features that belong to body / part of respective levels.\n\nThird, we are actually pursuing this idea as an ongoing work: hierarchical organization of multiple GUSTs for hierarchical grouping like the brain ('early stage for low-level features' as reviewer suggested), which we term as _representing the part-whole hierarchy of the visual scene_. Specifically, each GUST module serves as 'column' of each level, and different levels of GUSTs (characterized by their own time scale constants, fast for low-level GUST and slow for high-level GUST) are organized in a hierarchy (__Fig 6 a,b in global response__). The emerged synchrony (to represent part-whole) in turn is also of different time scales, fast one for part and slow one for whole, nested with each other like gamma-theta coupling. Preliminary results are shown in __Fig.6 c in global response__. However, as argued in A.1 (line108) in SI, this paper prefers to keep architecture minimal and general and leave ‘composition grouping’ (reviewer suggested) as future works. For other aspects of grouping / generalization results, see A.12 in SI\n\n__A3 Concerns on overlap__.\n\nThe grouping is robust w.r.t. the overlap or noise since the attractor dynamics provides a completion process. For example, top-down feedback is a completion of the bottom-up noisy firing (Figure 11 in SI). See __A3 ~ A6 in global response for more__.\n\n__A4 Concerns on  contribution to machine learning / neuroscience literature__.\n\nFirst of all, unsupervised grouping is a hard problem in deep learning / CV field (as reviewer fSip correctly points out, also stressed in ref[7] in main text) and it is also a fundamental operation for human brain, still lacking a clear mechanism. Therefore, combining the two sides on the one hand provides a general solution of basic computational problems in machine learning, and on the other hand, provides a systematic algorithmic level understanding of various biological structures/phenomenon/factors (like top-down feedback in cortex, delay-coupling, refractory, synchronized assembly, gamma oscillation, etc), bridging the levels (See A.5 in SI). As far as we know, both contributions bear the novelty. \n\nSecondly, for related work on bio-algorithms, the reviewer seems missed the core problem in this paper, grouping, which motivated the whole story. We do not exploit coherence for general learning / classification / generalization (as the IDTP paper and optical flow paper shows), but to solve the grouping problem in neural network, which is a problem at more basic level: representation (learning and generalization is relatively secondary). Therefore, they are totally different algorithms. Besides, aforementioned two papers use feedforward architecture without feedback and therefore coherence is not an emergent property through recurrent dynamics. Also, the dynamical nature of their ‘coherence’ is different from ours (though apply the same term). Our model has switching synchronized groups, which is non-trivial (need symmetry-breaking, __new insight__) and essential for grouping (__new insight__), while the two related papers only consider a single synchronized group for classification. Therefore, they differ from ours on at least (1) why coherence (2) what coherence (3) how coherence. \n\nThirdly, for combinatorial code, actually it is an elegant abstraction of spike code beyond rate code. Here, we specifically focus on synchrony code itself. The presence of synchrony code in cortex and their contribution to the grouping function has been cited as [17~24,26-28] in main text. There is such a wide range of supports in neuroscience that we cited the most relevant ones. Besides, the 2 works by William Bialek are appealing theoretical assessment of information contained in the combinatorial code, with a _descriptive_ model to formulate the probability,  instead of providing a _computational mechanism_ of how combinatorial code can emerge (as the reviewer improperly stated) and how they function to solve problems. Our work focuses on the latter two sides (__new insight__). Even so, we share a common spirit on additional information beyond rate code and they may potentially inspire our future works like binding by polychronization.\n\nFourthly, the CTC theory (old or new) takes the existence of coherence for granted and directly focuses on its contribution to communication, instead of the other way around:  communication also contributes to coherence. Therefore, it is not a closed framework. In our work, we provide a closed framework of (1) how coherence contributes to communication between DAE/SCS and (2) how communication between DAE/SCS also contributes to the emergence of coherence (Section 4.2 in main text, A.4.1,A.4.2 in SI). It is an iterative process instead of a one-way process in CTC (__new insight__). \n\nLastly, we discussed limitation/future direction in A.1 of SI. As reviewer suggested, we will add more on biological limitations when we revise the paper, including the correlation-based plasticity like IDTP, more general combinatorial coding like polychronization, and hierarchical grouping in a hierarchical architecture. For concern of SOTA, 'slot attention' is one such recent SOTA on color image (See A.9.6 in SI)."}, {"review_id": "EqnFfiOCEq", "reviewer": "Reviewer_fSiP", "summary": "The research topic of this study is development of a model that can learn to group (segment) the pixels in an image into objects in an unsupervised manner and in a way to enable systematic (combinatorial) generalization with respect to the number of objects. Toward this goal, the authors proposed a model, named GUST, by taking inspirations from the mechanism of human brain. The proposed model consists of two types of neural networks, spiking neural network and denoising autoencoder, that implement an iterative bottom-up/top-down processing together with several other brain-inspired components. The authors selected a simple loss function corresponding to image denoising for unsupervised training, and proposed a strategy for end-to-end gradient-based traing and also a clustering method for grouping (segmentation). The effectiveness of the proposed method was tested with synthetic images containing simple 2D objects. The proposed method has shown superior generalization performance over two competitors with respect to the difference in the number of objects during training and testing. The characteristics of the learning process and role of each component were also empirically studied.", "questions": "Question\n\nAre there any other differences between your study and Zheng et al [38] than the six aspects stated in Section 5  (Related work)? \n\nMajor suggestions\n\n1. Describing further differences from  Zheng et al [38] (if any)   \nIf the answer to the above question is yes, it is beneficial to describe the additional differences in the paper (either in the main body or in the Appendix). The differences are not necessarily to be about GUST and DASBE but can be about other aspects of the studies (papers). If any of the already stated six aspects can be further detailed to emphasize the novelty of this work, it would also be beneficial.  \n\n1. Clarifying explainability issue  \nIn the paragraph staring at line 73, four challenges are stated; the last one is explainability of the representation. However, it seems (to me) that the answer/discussions about this one is not clearly stated in the paper, at least compared to the other three. It would be better if  this foreshadowed issue could be highlighted with explicit keywords like \"explainability\"or \"explanable\" in the latter part of the main body of the paper.\n\n\n1. Adding pointers to the Appendix in the main body of the paper  \nAlthough the Appendix of this paper contains rich contents, currently it is mentioned only three times in the main body (without section numbers).  Additional appropriate pointers to the Appendix in the main body would be beneficial for the readers.  (For example, I had a concern about the limited variety of inputs in the experiments when I first read the main body, but later I realized that results with other inputs are provided in the Appendix.)\n\nPlease also refer to the Weaknesses section above and Limitations section below.\n\nMinor questions and suggestions\n\n1. Figure 2 (a) lacks $x$ and Figure 2 (b) lacks $\\tilde{x}$. It is better to connect these two in figures. \n1. Is $\\tau_w$ in line 177 a typo of $\\tau_1$? If not, what is $\\tau_w$? \n1. The code availability is better to be stated in the main body of the paper.\n1. Please review the descriptions in References. For example,  von der Malsburg 1994 [6 in References for the main body] and von der Malsburg 1994 [11  in References for the main body] lack information about what kind of publications they are,  and Engelcke et al [53 in References for the main body] was accepted at ICLR 2020.", "limitations": "The limitations are detailed in Appendix A.1, but the existence of this section is not indicated in the main body of the paper. It would be better if the authors mention the section at an appropriate place in the main body when they revise the paper. \n\nIn addition to the limitations already stated, I think the current study is also limited in the following two points. \n\n1. The objects used in the experiments are all 2D. (Experiments with images resulted from 3D objects would be preferable to be used in the future work.)\n1. The background is void. (If the background also consists of objects, the task becomes more difficult. This situation would be also preferable to be tackled in the future work.)\n\nIf these are correct, it would be nice to mention them as well.\n\n  \n\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Unsupervised segmentation (grouping) itself is a challenging computer vision task, and also can be the important first step in more complex visual tasks. Systematic (combinatorial) generalization to new situations is a hallmark of human intelligence yet to be achieved by machine learning models. The result that a model consists of spiking neural network and denoising autoencoder with several other brain-inspired components can be effectively trained to solve segmentation task in a systematic generalization setting in an end-to-end unsupervised manner will be of interest to the NeurIPS audience. \n\nThe ideas and implementations of the brain-inspired components are explained fairly well, and the actual code is accessible via the URL provided in the Appendix. The authors provide empirical analysis about the learning process and the effect of the brain-inspired components (ablation study) in addition to the simple performance report, which gives additional values to this study. Although the generalization performance is still not perfect and the experiments are conducted with simple synthetic images, this study shows an interesting research direction worth pursuing.  \n\\# I took additional experiments provided in the Appendix into consideration when I evaluated the variety of  experiments. ", "weaknesses": "A relatively weak point of this study is the difference from Zheng et al [38, in References for the main body of the paper]. Although the proposed model (GUST) is advanced compared to DASBE [38] in multiple aspects as stated in Section 5 (Related work), the core idea of combining spiking neural network and denoising autoencoder in brain-inspired manner and using the temporal coherence for grouping is proposed in DASBE.  \n\n[38] Hao Zheng, Hui Lin, Rong Zhao, and Luping Shi. Dance of SNN and ANN: Solving binding problem by combining spike timing and reconstructive attention. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 31430–31443. Curran Associates, Inc., 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__A1. Major response__ to the concern about the difference with Zheng’s work.\n\nWe will fist detail the six aspects in Section 5 and then point out more technical differences.\n\nFirst of all, the two papers are __answering different questions__ of different levels, not just advanced. Intuitively, the Zheng’s work focuses on how to __design__ a clustering program (like EM) while ours focuses on how to __learn__ the program from blank slate. Therefore, Zheng's focuses on dynamics only, but we take learning process into account. To some extent, from design to learning is an important step made by machine learning/deep learning, which is not obvious at all. It is especially the case here. While the architecture and final phenomenon is similar between the two works, which is actually desirable, the underlining mechanism is largely different, which is discussed in Section 4.2,4.3 in the main text (also Figure 11 in appendix). For example, _during training_, how blank slate DAE evolves so that: initial global random guess breaks the symmetry and tends to attend on a local spatial area, followed by further capturing the texture details. _The new insight_ we provide is that, the _symmetry-breaking behavior_ (grouping each single object into alternative synchronized assemblies, given multiple objects) can be learned with a succinct _symmetry-preserving_ loss function (reconstructing the whole multi-object scene). The Zheng’s work did not provide insight at this level. Additionally, evaluation of the __generalization__ is also not accounted for in Zheng’s work, since it does not have a complete learning scheme.\n\nSecond, due to the difference of answered questions, the __interpretation of the biological factors__ (or network architecture) is different. In Zheng’s work, factors like delayed coupling and spiking dynamics function only as _shaping the network dynamics_ so that the convergent state is the synchrony. However, in our work, we provide additional new insights that the factors like narrow time window, refractoriness, etc, could even act as inductive bias to _bias the learning_ process (Figure2 in Appendix). The Zheng’s work does not provide insight about network architecture at this level. \n\nThird, due to different interpretation, the __underlining hypothesis about how brain could implement such a model__ is totally different. Zheng’s work must assume a _pre-wired_ cortical bottom-up/top-down pathway as a pre-trained single-object DAE, which is constructed during development, determined by gene or evolution. But how each potential single object be accounted for by pre-wiring? In contrast, our work relaxes this assumption and suggests that the brain could _gradually learns_ such a model in a way consistent with predictive coding during the life even after the development.\n\nForth, to some extent, the DAE module (Section 3.3) in our work is not 'really' a DAE (it is not trained separately/explicitly to denoise), but a general encoder-decoder structure. The _denoising_ is achieved by the _whole system_ during iteration instead of the DAE module its own (Zheng's). We term it as DAE to provide a more intuitive picture of the architecture. However, __the nature of the architecture__ is different between Zheng’s and ours.\n\nWe believe these aspects conceptually distinguish this work from Zheng’s, not just technically advanced. In addition to these conceptual-level novelty, there are also technical difference. Some are introduced in Section 3.4,3.5 in the main text, including the __novel loss function__ and dealing with the __gradient__ along delayed path and refractory period. \n\nBesides, one major difference on study is that we provide a __new quantitative evaluation scheme__ of the grouping representation (more efficient). In Zheng’s work, it uses K-means to cluster the spike trains (to compute the AMI), which is hard to explicitly distinguish time coding from rate coding (A.13.2 SI). In our work, we resolve this problem by explicitly taking timing code into account when clustering. That is why we introduce K-medoids and VP-metric in Section 3.6. SynScore also differs and is more rational. Therefore, the quantitative scores have different interpretations between the two works. This is the technical novelty of the evaluation.\n\nIn sum, while the basic idea of combining spiking neural network and denoising autoencoder is shared between Zheng’s and ours. They differ in (1) motivation (representation/dynamics or learning/generalization) (2) nature/interpretation of the architecture (3) underlining mechanism/bio-picture (4) evaluation scheme. These stress the nolvety of this paper.\n\n__A2. For explainability__, actually, to analysis the grouping, the representation must be explainable. As a result, by clustering/visualizing the spiking pattern in Section 4, we make the representation in SCS explainable.\n\n__A3. For advices on revision__, including explainability/pointers/figures/code/reference/limitation, we will take these helpful advices into account when we revise the paper. \n\n__A4.__  $\\tau_w$ is the internal model parameter while $\\tau_1$ describe the externally observed synchrony (Figure2 in main text). They are relevant but not limited to be identical\n\n__A5. For future work on 3D and background__. Actually, this is one of our ongoing works (group in clever dataset, which is 3D objects with varied background). We find that the two limitation is firstly caused by the binary SCS in the pixel space because 3D objects should be grey-scale at least and general background is also difficult to be accounted for in binary image. As a result, we decide to implement SCS as a __binary hidden layer__ so that there are no constraints of the input type (binary/gray-scale/colored). The preliminary results of the ongoing work show that such change of network architecture preserve the basic mechanism and grouping/synchrony in hidden layer, still emerge. We will further study this situation as future work\n"}, {"review_id": "CYTcGwUxhL", "reviewer": "Reviewer_WxQh", "summary": "The paper introduces GUST (Grouping Unsupervisely by Spike Timing network), a network architecture inspired by the human brain that aims to address the challenges of grouping sensory information in artificial neural networks (ANNs). The network incorporates biological constraints to bias the network towards a state of neuronal coherence, reflecting grouping information in the temporal structure of spiking activity. GUST is evaluated on synthetic datasets and demonstrates the ability to learn and represent grouping information from unsupervised objectives. The model progresses from perceiving global features to capturing local features and systematically composes learned building blocks to represent novel scenes. The paper highlights the advantages of grouping with temporal coherence, such as flexibility, dynamism, and reduced representational cost. GUST overcomes challenges related to non-differentiable dynamics, high temporal resolution, unsupervised learning, and explainability. It contributes to bridging the temporal binding theory with ANNs and enables the grouping of objects directly from multi-object inputs. The paper also presents a clustering method to evaluate neuronal coherence based on precise temporal structures.", "questions": "-The modeling of the CD seems a bit confusing to me. The authors call it a coincidence detector but it appears that s' is 1 when a given neuron has rapidly tired between two time steps (this is also confusing to me since the neurons have refactory periods so my understanding is if s_i(t) = 1 then s_i(t-1) can't be 1 . If this is the case, the term coincidence detector seems a bit misleading if it is considering the coincidence of one neuron across time? Or is it the coincidence of multiple neurons? Which would made sense in the neural coherence picture, but if this is the case then should it be s' = CD(s_i(t),s_j(t-1))? Could the authors clarify this? I'm not sure if I am misunderstanding something or if it is a confusion of notation.\n\n-In section 3.2, the authors state: \"The refractory dynamics provides the essential temporal competition to separate groups of different objects and acts as a structural bias for a grouping solution\". Could the authors better clarify what they mean by this? How does the refractory period provide competition?\n\n- Small typo in header of 3.5. Gredients should be Gradients. \n\n- Could the authors elaborate and clarify why \"The GUST is simulated 3-times longer than training to confirm the stability of the grouping.\". This is a technical thing I am missing, but if the authors stop training at a particular time, then what is it that continues to change - is it the weights in the DAE are frozen but the model continues to receive inputs to output a score?\n\n- In figure 5d right, where it shows the spike firing sequence, to be very clear, are the individual columns the same neuron firing across time after that particular epoch? Or are they individual neurons (so 10 neurons) firing.\n\n", "limitations": "The authors thoroughly address limitations in the appendix.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper very clearly presents the formalism in that feels very intuitive and tractable. It is well written and clear. Despite how long the appendix is, the authors did an excellent job of keeping things self contained where limited reference to the appendix is needed to understand the paper\n\n-The paper incorporates really interesting biologically inspired structures into the architecture. Most interestingly, they incorporate the udse of coincidence detectors in a spiking model convolved with feedback information from the DAE to build a model which can group representations of object shapes in an unsupervised way. It is a really interesting architecture and I feel the model is an insightful architecture. \n\n- The background presented is very well motivated. It discusses experimental findings and key results in the field on the experimental and the modeling end which motivate the work in the rest of the paper.\n\n\n", "weaknesses": "- While this paper is very technically involved (and I did not read the whole appendix as a 26 page appendix is a bit unreasonable), I did thoroughly enjoy this paper and the ideas it integrates into the model. \n\n- The authors do not appear to compare to some comparable architectures on this task with the same metric. But this is a minor weakness. It is a little difficult to know how to orient oneself when reading this model relative to the literature. However, the novelty of the architecture is more interestng.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer to spend the valuable time for carefully reviewing the paper and providing the inspiring comments and helpful suggestions. We appologize for the confusions the paper raise to the reviewer, which will be clarified as followings.\n\n__A1. The response to concerns on related works__.\n\nIn this paper we provide the RC-bind and slot attention model as the benchmark, which is of comparable recurrent architecture (attentional feedback) and of similar metric (AMI). As discussed in the related work (Section 5), the general computational problem is related to object-centric representation or object discovery in CV. Therefore, the general architecture is more or less related to this line of models (‘slot grouping’ in Section 5). RC-bind and slot-attention are two representative models along this line. However, as the reviewer points out, the architecture of this work bears the novelty compared w.r.t this line of works in deep learning, partly because they have different representation assumptions (pre-designed slots vs emergent neuronal synchrony), which is central for binding problem.\n\n__A2. The response to the concern about CD__.\n\nThe coincidence detector in neuroscience literature stresses two things: First, the decay is very fast so that only coincident arriving spikes within a very narrow time window could be integrated; Second, the threshold is low so that the detector neuron can be activated by only a few spikes within the integration window. Here we take these ideas: the integration time window is narrow ($\\tau_w=2$) and the threshold is low (one spike is able to trigger the neuron). That is partially why we term this non-linearity as CD.\n\nHowever, as the reviewer concerns, in our model, each single CD (for $s_i$) is applied in such a simplified case that the original interpretation degrades into a short-term integrator of $s_i$ itself, with narrow time window and low threshold. However, all CDs (taken together) still acts as a coincidence detector / coincidence filter of the _population activity_ of SCS. The inputs to the DAE (for non-linear processing), is not a single slice of spiking patterns but narrowly integrated spikes (by CD). So that only coincident spikes $s_i(t), s_j(t-1)$ is taken into account by the downstream DAE. This is exactly the function of coincidence detector in the cortex, to distinguish the coincident spikes from averaged dispersed spikes for downstream processing.\n\nIn sum, we term eq 6 as (generalized) CD because it captures the core feature of CD (narrow window and low threshold) and also function as a coincidence filter to encourage the emergence of synchrony during learning (Section 4.2). Therefore, it captures more about the picture, either for model itself or bio-correlates of the model. Besides, the simple formulation of eq6 could naturally be extended to consider a local region of neurons (but it is not necessary in this model), so that the coincidence of multiple neurons is explicitly taken into account. For example, in redundant/coarse coding scheme, there could be a column of neurons of similiar response property at each location of SCS, instead of a single neuron at each location. In this case, each CD can be extended to be a non-linear low-threshold function of this column of neurons instead of a single $s_i$ so that multiple coincident spikes is taken into account (coincidence spikes indicate the confidence of the presence of objects/features). \n\n__A3. The response to the refractory period__.\n\nWhile the top-down modulation by DAE provides the positive feedback (reinforce the original pattern), which is needed to construct a synchrony state. But to group multiple objects, we need different synchronized neuronal groups to alternate. Without competition (negative feedback), the winner pattern will dominate forever. Here, refractory period enforces a hard temporal competition of neuron itself: if the neuron fires at some time, it cannot fire later. Therefore, different possible ‘timings’ compete with each other to fire the spike for each neuron. Since the objects are grouped at different timings, each timing point is a _possible_ candidate grouping ‘slot’. Refractoriness provides competition among these ‘candidate timing slots’, which we call temporal competition. Since refractory neuron is a structural constraint, it acts as structural bias to avoid WTA pattern and encourage synchronized neuronal groups to alternate. It is notable that, here, refractoriness is a special type of self-inhibition. In the brain, similar temporal competition/self-inhibition could possibly be provided by inhibitory neurons, not restricted to refractory mechanism, which is also plausible to induce various types of oscillations.\n\n__A4. The response to the concern on simulation period__. \n\nWe provide the detailed explanation of simulation length in A.9.1 of the Appendix: ‘simulation length’ and ‘additional results’ (also see Figure5/6 and Table2 in the appendix). \n\nDuring training, the simulation length is 100 (Table2 in appendix). We keep the simulation length relatively small because it is more efficient to backpropagate the gradient and it saves the training time. During testing/visualization, the model indeed tends to converge during the first 100 time-steps (Figure 5 in the appendix), but even longer simulation time could allow the model to converge more completely. That is why we use 3-times longer length for testing and visualization. On the other hand, it also confirms the stability of the model.\n\n__A5. The response to the concern on figure5__.\n\nEach small-square is a snapshot of SCS population activity (marked as ‘s’) or the attention feedback of the same dimension (marked as ‘$\\gamma$’). Each row of small squares is last-10 time-step simulation of the population activity (after the epoch). More detailed discussion can be found in A.9.3 of the appendix."}, {"review_id": "zYmRtdfSSc", "reviewer": "Reviewer_9PtM", "summary": "This paper uses findings from vision sciences research to provide an image-segmentation model that could be used in ANNs. The “grouping with temporal coherence” assumes that grouping information emerges via spiking synchrony. This is implemented in their network, called GUST. The network is composed of two main parts: a spike-coding space (SCS) and a denoising auto-encoder (DAE). The DAE provides delayed feedback to the SCS. Unsupervised training enables GUST to detect objects with neuronal coherence. The neuronal coherence is evaluated through a metric that measures distance between spike trains of populations of artificial units conceptualized as neurons. The system is iterative and is shown to generalize as well.", "questions": "What exact computer-vision problem is this resolving? It should be clear that this paper is about image segmentation and not perceptual grouping. Perhaps the most relevant literature in neuroscience is object recognition in cluttered scenes. \n\nWhy could this not be accomplished by deep neural network models with a lot of training data? A simple intuitive answer is lacking. \n\nUnder what circumstance does the grouping by temporal coherence fail in realistic cases? For example, choices like tau2 might not be wise or it might fail in speedy segmentation.\n\nIn Figure 6 (b-d), it seems like all models, including GUST, slightly perform better for larger number of test images. This seems counter-intuitive to me. Can the authors explain why the performance does not drop in their model?  \n\nI find it very interesting that the “neuronal” coherence could encode the grouping uncertainty. This feature is not exploited in any of the simulations though. Do you think that you can emphasize more this?", "limitations": "The limitations are not deeply or explicitly elaborate. This is problematic. Moreover, existing alternative algorithms are mentioned, but they are not systematically compared against the new work presented by the authors.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper attempts to address a very important question in computer vision/neurosci. The network is brain-inspired and does not require extensive training. GUST shows through training that it gradually learns to segregate the scene into a self-organized temporal structure and represent the building blocks of a scene by “neuronal” coherence. This can have major implications for computer-vision models and possibly be used to resolve the difficult problem of identifying objects in cluttered scenes.", "weaknesses": "Although this work has a broad audience, the framing of the question is rather poor. The authors start by talking about the binding problem, but then they constantly give examples that are about cluttered visual scenes. The binding problem is more general and can be applied to e.g., grouping of points that belong to a circle. \nSome assessment of the sensitivity of their model vs. for example RC-bind is missing (e.g., showing the performance as a function of %overlap). The comparison is only made for generalization to a different number of segments (Figure 6). Also, they don’t explain whether/how this can be combined with deep-net models of visual object recognition.\nFrom the neuroscience point of view, the authors seem to take for granted that the neuroscience evidence supporting perceptual grouping or binding by synchronized activity of neurons, but in neuroscience this is far from being accepted. It is indeed controversial. While it is Ok to present it as a neuroscience theory with credit and evidence in favor, it should also be mentioned that this theory is controversial and that there are alternative theories. Better balance would be highly beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__A1. Framing of the question__. \n\nThe major problem this paper focuses on is indeed the binding / grouping problem (in ANN / neuroscience literature), which is clearly framed in Section 2 in main text. The review paper [7] in main text frames binding problem as representation, segregation and composition \n (__Fig 3 in global pdf__) and this paper focuses on the first two, which is central to the binding problem. \n\nIt seems that what really concerns the reviewer is the relevance between the grouping problem (‘more general’ as reviewer argues) and the task we demonstrated (‘cluttered visual scenes’/’image segmentation’/'classification' as reviewer suggested). On the one hand, the binding problem _in neuroscience_ is famously exemplified as grouping features of cluttered visual objects, so called superposition catastrophe (See ref [6][7][11] in main text, copied in __Fig 3 in global pdf__); On the other hand, studying binding / grouping problem via cluttered visual scenes is common in a list of works in _ANN field_ on binding (See [14][48][7] in the main text). While the task seemingly bears the similarity with image segmentation / object discovery in ANN field, they focus on different challenges (we design GUST for general grouping problem challenges [unsupervise, multistable, common format...] with desirable features of general sense [denoising mechanism], instead of to conquer specific static image segmentation). Since the architecture in this work do not have explicit constraints / expert knowledge on the modality/semantics of data (eg. classical image segmentation algorithms like MRF or supervised segmentation), the mechanism could be generalized to other modalities or more abstract high-dimensional representational space (argued in Section 2 in main text). Thus, the visual task in this paper is for vividness and should not loss the generality. For a systematic review, please see ref [7] in main text, which could resolve most of the concern about the background knowledge (how to ground binding problem in ANNs field, w.r.t related topics like segmentation and object discovery, in neuroscience, and in Gestalt psychology). For example, as to the reviewer's __concern__: ’ _grouping of points that belong to a circle_ is also binding', it is actually specific principles (proximity, similarity, closure, symmetry, etc) in Gestalt Psychology. However, in Gary’s work [1], it is argued that various Gestalt Laws are all special cases of a single information-theoretic grouping principle, which is what we focus on.\n\n[1] Gary Hatfield and William Epstein. The status of the minimum principle in the theoretical analysis of visual perception. Psychological Bulletin, 1985.\n\n\n__A2. Relation to deep learning__.\n\n__The intuitive answer of why grouping challenges deep-nets__ is explained in the second paragraph of Section2 in main text and reviewed in detail in ref [7] in main text. On the specific question of why large data can not solve grouping is mentioned in the paper (line28): ‘the cost of representing all possible combinations statically and locally is exponentially high’. While the specific task seems easy to be conquered by specific ANN models with supervised training, we focus on solving the task with the model of desirable feature that is of general sense, so that it has potential to solve a family of tasks( see A1), which is a hard problem.\n\n__The way GUSTs could be combined with deep-nets__ is diverse and is explicitly discussed in A.1 of the appendix (line 76, 99). Briefly, generative model and denoising is very basic idea in deep learning, indicating possibility to link to more advanced models (eg. diffusion model). Also, the spiking activity in SCS could be readout to ANNs (to solve downstream tasks like _recognition_) by HU proposed in ref [56] in main text.\n\n__A3. Concern for more results__\n\nFor analysis of __overlap effect, see A.3 in global response__. Besides, as far as we understand, for single level grouping, varying 'object number' is complete for composition. For more grouping results, see A.12 in appendix.\n\n__A4. Controversial of binding by synchrony__. \n\nActually, two main-streams of binding theories are synchrony-based and attention-based (FIT theory), which is briefly mentioned in appendix (line123). We actually combine the two sides in our model. We will balance the two sides better when we revise the paper.\n\n__A5. Failure case__. \n\nFailure case has been shown in Figure 12 in appendix, when overlap is sickly heavy. Moving case is also challenging (dual use of time, See A.12.5, Figure 18 in appendix). In A.12.5, we also discussed the _speedy_ consideration (line 538, line545, line 574). For $\\tau_2$ , as Figure 6 in main text shows, same $\\tau_2$ accounts for grouping varied number of objects. It is the grouping dynamics that has the 'wisdom' (flexibility). \n\n__A6 Counter-intuitive common fate of all models__:\n\nObject number could have slight effect on overall overlap ratio among objects(__Fig1 in global pdf__). Heavier overlap tends to cause more unbalanced cluster size. Since AMI favors unbalanced situation, it tends to have slightly better score. Despite of slight bias, it is still clear that GUST generalizes in different cases (~0.6) and outperforms benchmarks in each case. \n\n__A7 ‘Grouping uncertainty’__. \n\nThe coherence level can be readout by a coincidence detector (HU in ref[56]) to help downstream tasks. In general, such coherence acts as an internal indicator of 'good' solutions so as to enable communication / transmission to downstream processes. For concrete example, see __A.8 in global response__.\n\n__A8. Response to limitation__. \n\nThe limitation is explicitly elaborated in A.1 in appendix. We compared our work with RC-bind and slot attention as benchmark in this paper. In A.9.6 in appendix, we provided more detailed discussion of their relevance. While Slot attention is one recent SOTA on color image, we can add more related models when we revise the paper."}]

Author rebuttal:
We thank all the reviewers for spending their valuable time to carefully read the paper and writing reviews. The comments and suggestions are very valuable and insightful, which helps to revise the paper and motivate future directions. We take all suggestions carefully when we revise the paper.

Specific responses to each reviewer are provided in separate rebuttal panel and in this global response, we address several concerns that requires new figures or related to concerns of multiple reviewers. 

___Response to questions on evaluation/benchmarking___

__A.1 (from reviewer dadF) Rationality of AMI and SC for evaluation, based on data distribution__.

On the one hand, _AMI / ARI are common-practice evaluation metrics to evaluate grouping_ (ref [14, 38, 48, 50, 51, 52, 53] in main text, discussed in A.13.3 in SI). On the other hand, we include background during both K- Medoids clustering and AMI evaluation (K=object number +1) since removing background introduces explicit prior of foreground-background segmentation (eg. what is object), which should also be considered for general grouping ability. Actually, during experiment, we found that confusing near-boundary region of Shapes with background is one reason that challenges the evaluation scores. In this case, the cluster size is relatively unbalanced because background tends to be the bigger cluster. Since AMI is more suitable for unbalanced situation, AMI is a more suitable metric than ARI, which fits more for balanced case (eg. if background is explicitly removed for evaluation). Also see __Fig 2__ right in pdf.

For SC, we visualize the low-dimensional structure of clusters through PCA in the metric space of spike trains induced by VP-metric (__Fig 5__ in pdf). It is shown that in both non-overlap and overlap cases, clusters are well-centered and seperated ('convex' in metric space), so that SC is suitable to evaluate the coherence level.

__A.2 (from reviewer dadF) More benchmarking results w.r.t random seed, number of clusters, AMI vs ARI__.

First, we have used 5 random seeds to evaluate the averaged AMI score in figure 6 in main text and the error bar is neglectable (Table3 in Appendix). Thus, the averaged AMI within large dataset is not influenced by uncertainty of AMI on each single sample (shown in __Fig 2__ in pdf), and there is no bias.

Second, the benchmarking in figure 6 in main text (of different object number) already indicate different number of clusters (K=object number+1). As far as we understand, it is exactly what reviewer asked for. __Fig 4__ in pdf is a more detailed benchmarking with varied object / cluster number.

Third, as explained in A1, AMI is a more suitable metric than ARI in our case. We also provide benchmarking on ARI (__Fig 4__ in pdf) for supplement.

Lastly, we provides additional benchmarking results w.r.t different overlapping degree in __Fig 4__ in pdf (both AMI and ARI).


__A.3 (from reviewer 9PtM). Compare the sensitivity among models (Performance vs Overlap ratio)__.

Please see __Fig 4__ in pdf (upper), where comparison is made for '3 train objects and 3 test objects' case (major case). Due to page limit, it is implausible to provide all 9 combinations (2/3/4 train objects X 2/3/4 test object) as Figure 6 in main text.

___Response to questions on Overlap / Noise___

__A.4 (from reviewer dadF) Analysis of the distribution of overlap in dataset__.

Figure 5 in main text do not suggest shapes are mostly clearly separated (the reviewer argued), because this image is just one randomly selected example. Besides, the example in Figure 4 in main text, in contrast, has partial overlap, which is much more common (__Fig1__ in pdf). We visualize the distribution of overlap in __Fig 1__ in pdf.

__A.5 (from reviewer dadF) Whether GUST can deal with overlaps__.

Figure 4 is one example, which suggests that GUST can deal with partial overlap through its denoising mechanism. Actually, in Shapes dataset, overlap is quite common (__Fig 1__ in pdf). We provide systematic quantitative analysis of overlap effect in __Fig 4__ in pdf.

__A.6 (from reviewer dadF) Whether GUST can deal with noise__.

The DAE in GUST can denoise the stochastic firing pattern in SCS through its feedback. However, input noise will linearly affect the spike firing, since in current implementation, input _hard gates_ the spiking neurons. If grouping is decoded from SCS ($s(t)$), the grouping will be linearly affected, but if grouping is decoded from feedback ($\gamma(t)$), it is robust to noise. It is plausible that adding horizontal spatial connection in SCS (future work) will provide pattern completion in SCS based on proximity law (Gestalt principle), when given noisy input.

___Preliminary results on hierarchical grouping___

__A7. (reviewer dadF) How GUST can be extended to hierarchical grouping__.

Schematic architecture and preliminary result of hierarchical grouping (part-whole hierarchy) is shown in __Fig 6__ in pdf.

__A8. (reviewer 9PtM) How uncertainty encoded in neuronal coherence can be used  (for hierarchical grouping)__.

Here we provide an example of one ongoing work (part-whole hierarchy). 

The overall architecture is shown __Fig6__ a,b in pdf. To represent the part-whole hierarchy of a visual scene, it is efficient to seach for solutions in a divide-and-conquer way: whole first and then part (possible brain strategy). Beginning with high temporature, high-level grouping achieves the high-level coherence (slow oscillation, coarser spatiotemporal structure) while low-level still remains unstructured/ random; __this high-level coherence (slow oscillation) indicates high certainty in whole-level grouping and is readout by inhibitory neurons to lower the temporature (decrease the excitability) so that low-level grouping is further achieved__ (faster oscillation, finer spatiotemporal structure), _conditioned by high-level grouping_. Overal, coherence acts as __internal indicator__ of a good solution with high confidence.

Meta-review: This paper addresses the complex task of aggregating information from individual visual elements into coherent perceptual units, a phenomenon known as compositional generalization. The proposed solution, GUST (Grouping Unsupervised by Spike Timing network), harnesses spiking synchrony to achieve grouping. The framework introduces a spike coding space (SCS) and employs a denoising autoencoder (DAE) to regulate temporal grouping, thus enabling the representation of scenes with diverse structures in simulated datasets.

Drawing from neuroscience literature, the paper integrates concepts such as correlation brain theory and temporal coding to establish grouping mechanisms in unsupervised learning. The challenge of effectively designing a spike timing neural network is tackled, considering the non-differential nature of spiking, the need for precise temporal encoding, and interpretability of representations. The study examines the algorithm's resilience to biological constraints through an ablation experiment, showcasing superior grouping performance in novel scenes compared to baseline models RC-bind and Slot Attention.

We think there is a good novelty and contribution to the community.
Decision: Accept (poster)