Dance of SNN and ANN: Solving binding problem by combining spike timing and reconstructive attention
Authors: Hao Zheng
Hui Lin
Rong Zhao
Luping Shi
Venue: NeurIPS 2022 Accept (2022)
Keywords: Perceptual grouping
Binding problem
Time coding
Neuronal synchrony
Top-down attention
Compositional generalization
Object learning
Hybrid neural network
Spiking neural network
Artificial neural network

Abstract: The binding problem is one of the fundamental challenges that prevent the artificial neural network (ANNs) from a compositional understanding of the world like human perception, because disentangled and distributed representations of generative factors can interfere and lead to ambiguity when complex data with multiple objects are presented. In this paper, we propose a brain-inspired unsupervised hybrid neural network (HNN) that introduces temporal binding theory originated from neuroscience into ANNs by integrating spike timing dynamics (via spiking neural networks, SNNs) with reconstructive attention (by ANNs). Spike timing provides an additional dimension for grouping, while reconstructive feedback coordinates the spikes into temporal coherent states. Through iterative interaction of ANN and SNN, the model continuously binds multiple objects at alternative synchronous firing times in the SNN coding space. The effectiveness of the model is evaluated on five artificially generated datasets of binary images. By visualization and analysis, we demonstrate that the binding is explainable, soft, flexible, and hierarchical. Notably, the model is trained on single object datasets without explicit supervision on grouping, but can successfully bind multiple objects on test datasets, showing its compositional generalization capability. Further results show its binding ability in dynamic situations.

Dance of SNN and ANN: Solving binding problem by
combining spike timing and reconstructive attention
Hao Zheng Hui Hii Rong Zhao Luping Shij
Department of Precision Instrument
Center for Brain- Brain-Inspired Computing Research Tsinghua University, Beijing 100084, China {zheng-h17, (illlnniiiimaaa. damails.tsingee edu. cn {r ziao, 1pshi /ttsinghua. edu. cn
Abstract
The binding problem is one of the fundamental challenges that prevent the artificial neural network (ANNs) from compositional understanding of the world like hu- man perception, because disentangled and distributed representations of generative factors can interfere and lead to ambiguity when complex data with multiple objects are presented. In this paper, we propose a brain-inspired hybrid neural network (HNN) that introduces temporal binding theory originated from neuroscience into ANNs by integrating spike timing dynamics (via spiking neural networks, SNNs) with reconstructive attention (by ANNs). Spike timing provides an additional dimension for grouping, while reconstructive feedback coordinates the spikes into temporal coherent states. Through iterative interaction of ANN and SNN, the model continuously binds multiple objects at alternative synchronous firing times in the SNN coding space. The effectiveness of the model evaluated on synthetic datasets of binary images. By visualization and analysis, we demonstrate that the binding explainable, soft, flexible, and hierarchical. Notably, the model is trained grouping, but successfully
single object datasets without explicit supervision binds multiple objects on test datasets, showing its compositional generalization capability. Further results show its binding ability in dynamic situations.
Introduction
Learning disentangled and distributed representation of generative factors of the world is believed to benefit compositional generalization, because those invariant features can be reused as symbols
to build exponentially larger amounts of objects with higher complexity [I] 2 B 4. However, disentangled representation can suffer from fundamental difficulty when the inputs are composed of multiple objects. Since the features are disentangled and invariant to specific objects, multiple coexist feature representations interfere and lead to superposition catastrophe (Fig This conceptualized as the binding problem by von der Malsburg (1981)[5] 6) and extensively debated in neuroscience and psychology literature [7]. Recently, the binding problem is also documented as a fundamental reason behind various inabilities of artificial neural networks (ANNs) like data inefficiency and limited transfer capability 3
Segregating and representing symbol-like entities from raw data is challenging and often requires supervision 8 10
[3] or task task-speeific design [4 [5] 16 I7]. Thus, the problem usually ignored by assuming there is only single object of interest or mediated by using static localized binding [18 19 the cost of expensive computational demand. All the above methods in ANNs
cannot fundamentally solve the binding problem. In contrast, slot models 3 20 21
22
23
24 25
*Corresponding author, i Contribute equally
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

26 27 28 29
31 32] in ANNs seems to provide a promising approach, which uses various types of computational slots inform the object label. However, current slot models are not so flexible and dynamic as human perception partly because they require pre-defined, fixed, discrete number of slots to place a hard label to each object. Besides, the slot is usually entangled with certain dimension of features [3 and the set of features often need to be duplicated for each slot 25).
30
To overcome this challenge, we brought ideas from neuroscience. In neuroscience, temporal coordi- nation is an alternative solution to support the binding in a human brain. The solution is based on spike timing correlation in a network of spiking neurons (SNNs). referred to as temporal binding theory [S] or correlation brain theory [34). The theory assumes hybrid code. Each neuron has its receptive field of firing rate tuned to certain features. Its spike timing can additionally be modulated to express grouping information, so that assemblies encoding features belonging to the same object fire coherently in time (Fig [b). Grouping in time dimension has five important advantages. First, object representation based on additional dimension of time will naturally be in a common format R essen- tial for systematic generalization to unseen situations or more objects. Second, unlike slot models, a single set of features are shared among all object representations. Third, the grouping information is usually continuous thus may help to encode uncertainty about the grouping. Fourth, unlike supervised method, temporal binding is an unsupervised process, consistent with the multi-stability property of the grouping. Fifth, the number of groups is not needed to be explicitly pre-defined, but self-organized dynamically according to temporal coherence. Despite the virtue of temporal binding, however, how to realize spike timing correlation in various computational demanding situation remains an open and challenging problem. As far as we know, the temporal binding ability of traditional SNN models is only tested on very simple stimulus patterns [35 36 37
39 40), not as complex as datasets in the field of ANNs. Besides, since the concept of spike and spike timing is incompatible with the ANN framework, very few studies have incorporated the promising SNN solution into ANNs B
38
In this work, we introduce hybrid neural network architecture (HNN), combining spike timing code within the SNN module with delayed reconstructive feedback from the ANN module (Fig Tp). First, the augmented spike coding space exploits time dimension to softly group multiple objects with natural common format and a single set of features. Second, the denoising autoencoder (DAE) [4 42
feedback important modulatory signals to coordinate the spike timing into coherent states. Third, the spiking dynamics induces temporal competition among multiple objects to group them at separate timings. Taken together, SNNs expand the representation space of ANNs, and ANNs provide the complex computations necessary for flexible binding. Two parts interact iteratively. Thus, we name the resulting architecture as DASBE (short for Dance of ANN and SNN Binding nEEwork)). We also provide an informal proof of the convergence of the model.
Evaluating binding is intrinsically challenging due to multi-stability and afford multiple grouping states, some of which are even unknown. Thus, we combine quantitative and qualitative analysis for evaluation. The former is mainly to provide partial measure of the effectiveness based on pre-defined ground truth while the latter provide important additional proof of the binding. Specifically, (1) We quantitatively evaluate the binding in DASBE by the mean adjusted mutual information (AMI) score [4) between clustering result of spiking patterns and ground truth segmentation. The DASBE achieves more than 0.50 all dataset and 0.80 on 3 out of 5 datasets, demonstrating its grouping capability on different types of stimuli. (2) We further visualize the spike representation and analyze the properties of the representation on the Shape dataset, showing that the grouping is continuous and based on spike synchrony, which is measured based on Victor-Purpun metric 44 of spiking patterns. Interestingly, the bound assemblies are seated on the peak of emerged gamma-like like oscillations, analogous to the brain [45). (3) Besides binding in pixel-level features, we also show that features of hierarchical levels can be bound in DASBE. (4) Lastly, is demonstrated that DASBE can even track multiple moving objects and detect pop-up objects in the dynamic situation without explicit supervision. Taken together, we hope integrating ANNs and SNNs properly can provide new possibilities to solve binding problem in brain-inspired manner.
3 scenes can be ambiguous
2 Contribution
(1) We introduce the spiking dynamics into ANNs to solve the binding problem, so that the binding is unsupervised, soft, in a common format and keeps single set of features. (2) We introduce reconstructive attention from trainable DAE into SNNs so that the capability of temporal binding is enlarged beyond that of traditional fixed connection models (3) We develop neuroscience-inspired

---

loo1
ANN layers
DAE
loca
loc2
Top down
Roc-
loc2
Low lovel features
loo2
(a)
(b)
(c)
(d)
(v] Attention map ?(t) for delayed feedback
Spike firing pe(t)
peried)
Y
sc(()
P(( x7((
0
(V)
0
DAE
(M
Proximal dendrite
Stimulus x,
Context
t
7((1) Pi(t Teelny>
r(t)
Temporal coding
Pyramidal neuron model
x
(o)
(n)
Figure 1: (a) The binding problem. (b) Temporal binding theory. (c) Overview of DASBE
architecture. (d) Biological relevance. (e) Single neuron model in DASBE. (f) Detailed information flow of DASBE. (i) input image (ii) SNN coding space (iii) coincidence detector (iv) Encoder (v) latent space (vi) attention map given by decoder. Feedback flow not shown for clarity
analysis method to evaluate and visualize the property of spiking representation in single level or hicrarchical levels, in static or dynamic situations. Gamma-like oscillation is observed.
Model
How can temporal binding be incorporated in ANNs to enhance the compositional representation ability of ANNs? We provide an answer by augmenting an ANN with spike coding space (by SNNs), that the time dimension can possibly be used group different objects. On the contrary, how can each spiking neuron "know" its grouping companions to fire with? We provide an answer by top-down modulation from a denoising autoencoder (DAE). Lastly, how can different object be grouped at separate timings? We provide a minimal answer with neural refractory dynamics to induce mandatory temporal competition.
The model composed of four parts: a spike coding space (SCS) of low-level features, an ANN encoder, latent space of higher-level features, and an ANN decoder (Fig [b). Four parts are connected end end, so that the ANNs and SNNs interact dynamically in an iterative way (with delay). The biological interpretation of each part is shown in Fig TH.
3.1 Encode binary stimulus by topographical mapping
Inspired by topographical mapping in primary cortical layers [46. the dimension of spike coding space (SCS) is set to be of the same as the input image so that pixels in the image have one-to- one correspondence to neurons in the SCS (Fig The encoding is binary. x E {0. 1 and s E (0, 13 where the image stimulus and s the spiking response of neurons SCS, N the size of image.

---

3.2 Spike coding space (SCS)
The single neuron model SCS (Fig ((i)) the two-compartment model of pyramidal cells [P) The proximal dendrite [48] of neuron receives binary driving signals from external
stimuli (t). The distal dendrite 49 neuron receives smooth modulatory signals (t) from
delayed top-down feedback.
Since the distal and proximal current usually interact in non-linear manner in vivo 47 50 the product two terms determines the firing rate of the active neurons (not in refractory period).
pi(t) - x; (t))
where is the neuron index; (t(t) is the firing rate. If the neuron generates spike, 1, otherwise 8; After firing a spike, the neuron undergoes a period of refraction Tyfr E N. In total,
(1) Ber (p:())), A.
where is the set of active neurons that are not in refractory periods Tr and Ber() is a sampling step according Bernuuli distribution with mean p.(t). Thus, the firing event 8; reflects the integrative effect of external stimuli, top-down modulation from ANNs and spiking dynamics in SNNs.
3.3 Bottom-up processing by the encoder of DAE with coincidence detectors
The spiking events in SCS are readout by coincidence detectors (CD, Fig ( [51], which only response to very limited temporal window Tw < 3 with stochastic decay (H.p)) {0, 1]. The CD result is further fed to the autoencoder.
(t) θ
s:(t 7) Vaa)
is step function with threshold Via. standing for the activation function of the CD; stochastic decay Ti, Ber(p"),p < 1. Then the DAE encoder extracts out higher-level features of the objects in the images the latent representation 0(t) (*)))))) Here, f is the encoder and 0 is the latent vector (iv.)v. T is an additional spike sampling operation latent representation is binary, otherwise it just an identity map.
3.4 Delayed modulation of attention map given by the decoder of DAE
The output of DAE decoder is fed back to the pyramidal neuron in SCS through distal dendrite as top-down modulation, but with delay period.
(t) g(O(t Tdelay)) = A(t T.elay).
(4)
g is the decoder of DAE. y refers to the attention map given by the DAE (Fig and Taetay E N refers to the delay period.
For clarity, the overall binding process is shown in Algorithm | We identify three key properties of the DASBE binding process: First, the positive feed-back structure of DAE, which is termed as the "folded" DAE here, embeds each single object its attractive fixed phonts.Those Those fixed points act as associative memories of symbol-like entities. Second, The superposed state of multiple objects is not such fixed point. Thus, superposition is not stable representation of "folded DAE". However, "folded" DAE may focus on one object and ignore others. Third, if augmenting "folded" DAE with refractory dynamies of SNN, and delayed feedback, then each object becomes only transient states due to temporal competition. However, alternations of objects in turn is possible stable trajectory. More formally,
Proposition Assume the mapping SDAE: objects {z,}. In E [0, 1]d perfectly. Then, (1) (Fixed point) Vz,, E {2n), In fDAE(E,) and
Rd - Rd has learned to reconstruct vectors single

---

Algorithm Temporal binding process in DASBE. The input x is binary vector of dimension D immege The attention map (attn) is real-valued vector of dimension (T + Tdelay) Dimage T is the length of binding process. We initialize attn [t], t E I-deelay: 0] as independent positive samples (attn 0) from standard Gaussian distribution (0, f). Refractory variable rfr is initialized as In our experiments, we set Tdelay E [20, 60),
0. Tdelay- Tws fr are time-scale
Tw =3, 3, Trfr E (0, 15), T x: 1000. The hyper-parameters for CD: VnA = 1,p 0.5
Input: inputs E {0, 1} Dimage, attn Tdelay 0) [O T)| E [R [Raaiiay x Dimage
2: Layer params: MLP, MLPg- encoder and decoder of DAE.
3: Other layer operations: Norm; Softmax; CD; Ber, T: Bernoulli sampling operation for binary latent space or identity map for real-valued latent space.
4: attn -Tittta 0] = Norm(attn
-delay
0]);
normaalizatonn of initial attention map
5: for t = 0...
6:
context attn it - Tdelayl:
7:
firing_rate = inputs O context @ (rfr == =) 0);
# @: element_ wise product
8:
spike [t] Ber Ber
# Ber: sampling according Bernoulli distribution # set spiking neuron into refractory period update refractory variable
9:
rfr+ = spike -
10:
rfr = max(rfr- 1,0;
11:
i = cotnslkke [t Tu :():
# CD: coincidence detector
12:
encode = Sof (o (input2dae)),
13:
latent T(encode);
# latent space can be real-valued or binary
14:
attn [t] Sof tMax tmax (latent)):
return spike
(2) (Attractive) Under dynamics z(t + 1) = fDAE(r(()): 36 > 0, E (0,1), s.t., Ve E 1J, s.t. S 6, then, IIfDAE(E, c) - znll a (I -) znll, "-" is constraint in the binary space of Zn
Proposition 2. = In. (lfDAE(x') z'll > 0 (superposed state is not a fixed point), "E" constraint in the binary space
Proposition 3. Under delayed dynamics: x(t + 1) = fDAa(=(tt - Tdelay)), (x(()= = Fn, E [0, possible attractive solution.
Proposition 4. Taking refractory dynamics and delayed feedback together: x(1 + 1) SDA (=:( Taclog) @ [rrfr 0)) then "xx(() = In E [0, TJ" is not a solution. Given I' -K In, Bxtra [0, T] - {0, 1]d (t) {an,3 (0) is a possible stable trajectory. (I(x) is an indicator function)
The proof is in the Appendix. Note that the propositions are idealized and therefore may deviate from situations in practice. However, the feasibility of above properties is very likely to hold as can be seen in the experiments (Also see Appendix), which heuristically imply the binding ability of the model.
4 Experiment and results
4.1 Experiment setup
We test the binding performance on benchmark datasets[2], including Bars, Shapes, Corners, Multi- MNIST and SNiST+ All these datasets are synthetic for convenience of evaluation and analysis. Further visualization are only analyzed in Shapes dataset for clarity, while results on other datasets can be found in Appendix.
5

---

Table 1: AMI score (%)
Dataset
DASBE
Folded DAE
PCNN
Bars
96.5 + 0.03
9.6+
9.3+0.04
Shapes
90.2 0.07
47.5=0.04
44.7=:0.15
Corners
80.6 + 0.08
54.2=0.15
28.5
Multi-MNISS
55.5 0.06
43.0 0.08
23.0 0.11 0.11
ShapT+
53.3 + 0.26
38.3+0.03
12.9
4.2 Results
4.2.1
Quantitative evaluation of binding
We evaluate the DASBE binding performance on a list of benchmarked artificially generated datasets consisting binary images of different complexity. For each dataset, a DAE trained remove the salt&pepper noise on images of single objects. The best trained DAE is used for binding on images containing multiple objects. The binding quality evaluated based on ground truth object identity.
After the simulation, we use the K-means clustering to cluster spiking neurons into different groups according to their spike train pattern. Since the ground-truth truth segmentation is available for generated datasets, we compare the clustering result and ground-truth by adjusted mutual information (AMI)
43 as measure for temporal feature binding performance. The AMI score 1 for perfect grouping, and O for chance level. Although K-means uses Euclidean distance to compare each spike train with an averaged firing density, thus, in principle, do not explicitly evaluate timing difference, it will be shown that the clustering is actually based on spike timing code instead of firing rate difference (See Sed4.2.3 Fig 3. The model achieves more than 0.50 all datasets and 0.80 on 3 out of 5 datasets (5 random seeds, 6000 samples), implying their successful binding ability (Table I
To show the synergistic effect of combing ANN and SNN part, reduce the hybrid DASBE into SNN (PCNN) and ANN ("folded" DAE) version as baseline. Pulse-coupled neural network (PCNN) is traditional temporal binding model in neuroscience [52 53 [54). and can be considered replacing the trainable ANN(DAE) part by the fixed recurrent connections. Since original PCNN binds objects based connectivity only, it gets into trouble when objects overlap (Bars, MNIST+ unconnected parts compose a whole (Corners). The "folded" DAE (as defined in Proposition 3 and Appendix) can be considered as removing the spiking dynamies replacing the SNN coding space by a non-spiking layer (z, I; (t), where z(t) is the non-spiking input to the DAE module). As shown in Proposition "folded" DAE may focus too much one object and ignore others. As result, both control models perform much worse than the DASBE (Table
when
4.2.2 Qualitative analysis
Visualization of spike timing pattern SCS on Shapes dataset shown in Fig selected image consists of square and two triangles located at different positions. Starting from random scattered firings, feature neurons responsive to the same object gradually synchronize themselves and keep out of phase to that of other objects. Object-related assemblies are seated on the peak of emergent gamma-like oscillatory population activities 45) (Fig 2).. d). More details can be found in Appendix.
Fig2b.c.d b.c.d. The randomly
4.2.3 Evaluating synchrony, coding scheme and convergence
To get more insights of spiking code, we develop method to evaluate synchrony level. Here, we define synchrony as a measure of temporal coherence of spike firing in each group. Such measure requires two things. (1) coherence measure of clusters and (2) metric explicitly dependent on spike timing difference. In this spirit, we define synchrony score as the silhouette score (55) of clusters based on Victor-Purpy SictorPurrpura metric [44] of spike trains. The Victor-Purpura metric is non Euclidean and has a time-scale parameter q. At its extreme, the metric explicitly measures the spiking rate when q + and spiking synchrony when g + +oo (see Appendix). We define synchrony score and rate score by large and small y value. It can be observed that, along the temporal binding process on Shapes, synchrony score is increased from 0.2 0.9, indicating the neurons converge to an object-dependent synchronous state from random guess (F In contrast, rate score remains

---

900 044
synchrony
rate
-02
-04
time
Temporal binding process in SCS. (a) Evolution of averaged synchrony score and rate score in five random initial condition. (b) left most: input image; right most: ground truth; middle five: firing phase each feature ncuron during simulation, indicated by color. The initial phase indicated by red box and bound phase indicated by green box (c) Zoomed in firing pattern of initial (d) spike recording (color dots) of each neuronal group in last 120 time steps. Shallow shadow is the firing rate of each group. Black line is the total population activity, implying gamma-lite like oscillation.
Figure 2:
phase (red box) and bound phase(green box) in (b), respectively.
(0)
16
(4)
synchnory scone
Uincrrong SCOr0
3.5
Tata SCOre
Palie score
6.0
-25
-15
10
22
10
22
iteration steps
iteration steps
Clustaring score
Clustering score
Figure 3: Synchrony score, rate score and clustering score K-means, (a) Temporal evolution of synchrony score (b) Rate score (c) Synchrony score vs clustering score. (d) Rate score clustering score.
around zeros. Due to consistency between synchrony score and clustering score during the binding 33 green linc), spike timing is indeed used by both binding process and clustering evaluation. These properties are roughly consistent across all datasets (Fig
It is interesting to note that the grouping information is continuous (Fig 2 c.d, indicated by yellow circles): each feature (spikes of feature neurons) not hard-paired to a discrete slot, but "softly" bound to one multiple limited contingent regions in time dimension. Thus, firing dispersion seems to continuously encode the uncertainty of binding measure of grouping. The number of coherent states is not fixed by pre-defined number K, but as an emergent phenomenon during the dance ANN and SNN parts. These are special features of temporal binding both in DASBE and in the brain.
natural way, acting like a coherence

---

deoode
DAE
Low-level
Position
Hierarchical temporal binding in DASBE. (a) Feature neurons whose receptive field related to certain features and independent of opposite type of features (b) Firing activity in hidden layer, firing time indicated by grouping color. Exemplar of found feature neurons circled. (c) DASBE with hierarchical spike coding space (d) Spiking recording of low-level (bottom) and high-level (top) feature neurons (e) Decoding of synchronous firing neurons in the hidden layer
Figure 4:
4
A
4
A
A
A
4
4
A
A
A
4
0
8
A
AD
40
4
2
0
4
4
0
4
g
4
AD
A
4
0
4
2
A
24
34
&
Y
%
A
A
B4
0A
D
Figure 5: Binding in moving situation. Black-white image is the input and color image is the clustering result of latest spikes (1 delay period). Time evolves left-to-right and up-to-bottom.
4.2.4 Hierarchical binding in DASBE
Features in both the brain and ANNs has a hierarchical organization. Thus, we further study whether DASBE can bind hierarchical features in time. To verify it in shape dataset, we realized spiking latent coding space (big in the DAE by making T Bernoulli sampling process (Sed3.3) AlgT). Besides, we train the DAE with an additional supervised contrastive loss(56) only learn disentangled and distributed latent representation the original binding problem. Feature neurons are identified if they invariantly response to certain generative factors (shape and position in this simple case, 4a) necessary for hierarchical binding in general cases.
that the result is more comparable to that studied
4a). Note that, such supervision is not
The results show that the shape and position, two generative factors of the shape dataset, is represented in an approximately distributed and disentangled manner the latent spike coding space of DAE (F.g....b). After the binding converges, both the low-level features(object-relln and high-level features (shape and position neurons) synchronize together (Fig d.e). is interesting to note that hierarchy tree of features of different levels can be readout by following such spiking synchrony.
pixels neurons)
4.2.5 Binding in moving situation
We further study whether the simple DASBE can bind in dynamic situation. To verify, we generate the moving shape dataset (Fig5). where each object can move along a fixed direction during the experiment with constant speed (1 pixel per delay period). Besides, new DAE with a recurrent hidden layer trained to predict next step of single object. Surprisingly, the DASBE can not only track multiple moving objects robustly even in overlap situation but also detect pop-up objects. The result shows that temporal binding in DASBE flexible since the number of groups is not fixed, and dynamic since the model re-correct itself based on the on-going situation.

---

5
Related work
The concept of synchrony was first illustrated in the ANNs by complex-valued Boltzmann machine [57], and latest developed into complex- valued auto-encoder(58). However, these works abstract away firing rate/firing timing to the amplitude/phase of complex number. Due to the limited range of phase values, approaches using complex-va -vauud activation can only bind small number of objects. On the contrary, the DASBE can flexibly use temporal correlation to bind large number of objects (See Appendix, examples for binding Bars).
Most work attempting to solve the binding problem in ANNs focus on one particular representational
27). These slots are predefined in the network structure and explicitly separate the latent space for different objects. In contrast, the DASBE implicitly and softly assign grouping information to objects in a self-organized process.
format: slots 2 21 22
23 24 25
Some earlier works solve temporal binding problem by attractor dynamics 36 40 that memorizes
the single objects. Here, the DASBE is also a dynamical system whole. The interesting point is that the pre-trained DAE seems to parameterize the attractor dynamics and explicitly plant the fixed point of this system (Proposition)) in an equivariant and structured way. Equivariance refers that retrieved memories can adapt itself in feature space along with the change of stimulus, instead of a discrete converged prototype. Besides, the structure of the attractor is encoded in latent layers of the DAE, like shape and position. Thus, the capacity of SNN binding model get expanded so that they can be tested beyond toy patterns.
The top-down modulation may remind us of the attention mechanisms in solving the binding problem 20 26 [9]. It also has been argued that temporal binding and attentional binding may be the two sides of the same coin and it seems the brain uses both methods 3 60). Thus, this model can be considered bridging the temporal binding and the attention mechanism together.
Discussion
Can the DASBE provides insights to the temporal binding process in the brain? In point of view
of modeling, the model can be imagined as a kind of simplified cortical circuits (Fig Td): The SCS
composed of refractory pyramidal neurons with proximal and distal dendritic sites [47 48 49 mimics
the layer II/III. The delayed loop of bottom-up/top- down processing [61 62] in higher cortical layers
(IV, V, VI) is formulated as folded denoising autoencoder [63]. In this way, DASBE suggests a theory of temporal binding, which highlights the role of top-down modulation from pre-wint pre-wired cortical circuits: the pre-wired cortical pathway, from thousands of years of evolution, encodes priors of prototypes of plenty of single objects. This prior is the essential cue for actively sampling the input and providing preparing signals for grouping. This prior is encoded in the decoder of DAE in DASBE during training. The preparing modulatory signals and driving signals act on pyramidal neurons at different dendritic locations in a non-linear manner [50]. The driving signals at the proximal sites dominate as gating signals while modulatory signals select which neuron to finally be activated [49] The refractory period provides important competition in temporal dimension, thus firing alternates among different groups. Coincidence detectors [5T] readout the synchrony events and feed to the higher cortical areas, where higher-level features are extracted. The feedback, carrying the prior cue, delayed due to signal transmission along the pathway, of tens of milliseconds (64). This delay period provides window where grouping can alternate on gamma waves. Thus, the model highlights the role of the priors in pre-wired cortical circuits, and the interaction between refractory dynamics and the delay of feedback. In our model, it is the dance between SNNs and ANNs.
The aim in this paper is to demonstrate that the combination of top-down modulation and spike timing in an HNN framework (65) is promising way to solve the binding problem. Since this framework is conceptually consistent with many essential properties of the binding capability, like unsupervision, common format, and multi- multi-stability(33. while structurally compatible with both ANNs and SNNs, DASBE can be either enhanced by advancing ANN architectures to reach more competitive binding capabilities or added with more biological ingredients for modeling binding phenomenon in the neural system. We are convinced that DASBE can help to bridge insights from various aspects to solve binding problem in the future.

---

7 Conclusion
We propose a hybrid neural network architecture to combine top-down modulation and spike timing representation to solve the visual binding problem. The top-down feedback from the output of DAE modulates the spike timing of spiking neurons. Therefore, the solution to the binding problem is formed through iterative top-down/bottom-up processing and represented by neuronal synchrony.
Acknowledgments and Disclosure of Funding
We would like to thank Mingkun Xu, Wenli Zhang and Faqiang Liu for general advice and feedback on the paper. We would like to acknowledge the anonymous reviewers for valuable suggestions and comments. This work was supported by National Nature Science Foundation of China (No. 61836004, No. 62088102), National Key Research and Development Program of China (Grant No. 202 2021ZD0200300), IDG/ DG/McGovvrn Institute for Brain Research at Tsinghua University, CETC Haikang Group- Group-Brain Inspired Computing Joint Research Center.
References
[1] Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint 1rXiv: 2018.
[2] Klaus Greff, Rupesh Kumar Srivastava, and Jürgen Schmidhuber. Binding via reconstruction clustering. arXiv preprint arXiv:1511.06411 2015.
[3] Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05206 2020.
[4] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
[5] Christoph Von der Malsburg. The what and why of binding: the modeler's perspective. Neuron, 2441):95-104, 1999.
[6] Geoffrey Hinton. Distributed representations. 1984.
[7] Jerome Feldman. The neural binding problem (s). Cognitive neurodynamics, 7(1):1111, 2013. 2013.
[8] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv: 2019.
[9] Chen Sun, Per Karlsson, Jiajun Wu, Joshua B Tenenbaum, and Kevin Murphy. Stochastic pre- diction of multi-agent interactions from partial observations. arXiv preprint arXiv: 1e..1 9994.0964,, 2019.
[10] Nicholas Watters, Daniel Zoran, Theophane Weber, Peter Battaglia, Razvan Pascanu, and Andrea Tacchetti. Visual interaction networks: Learning physics simulator from video. Advances in neural information processing systems, 30, 2017.
[11] Boris Knyazev, Harm de Vries, Cätälina Cangea, Graham W Taylor, Aaron Courville, and Eu- gene Belilovsky. Graph density-aware losses for novel compositions in scene graph generation. arXiv preprint arXiv:2005.082. 2020.
[12] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580-587, 2014.
[13] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
10

---

[14] Tejas D Kulkarni, Pushmeet Kohli, Joshua B Tenenbaum, and Vikash Mansinghka. Picture: A probabilistic programming language for scene perception. Proceedings of the ieee conference on computer vision and pattern recognition, pages 4390-4399 2015.
[15] Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin P Murphy, and Honglak Lee. Unsupervised learning of object structure and dynamics from videos. Advances Neural Information Processing Systems, 32, 2019.
[16] Anand Gopalakrishnan. Sjoerd van Steenkiste, and Jürgen Schmidhuber. Unsupervised object keypoint learning using local spatial predictabilityy arXiv preprint arXiv:201 2020.
[17] Coline Devin, Pieter Abbeel, Trevor Darrell, and Sergey Levine. Deep object-centric represen- tations for generalizable robot learning. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7111-7118. IEEE, 2018.
[18] Maximilian Riesenhuber and Tomaso Poggio. Hierarchical models of object recognition in cortex. Nature neuroscience, 2(1 11:10111-111001 1999.
[19] Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with back-propagation network. Advances in neural information processing systems, 2, 1989.
[20] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances Neural Information Processing Systems, 33:1 1125-115388 2020.
[21] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object repre- sentation learning with iterative variational inference. In International Conference on Machine Learning, pages 2424-2422442 2433. PMLR, 2019.
[22] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa- tion. arXiv preprint arXiv: 11390, 2019.
[23] Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Gener- ative scene inference and sampling with object-centric latent representations. arXiv preprint 1rXiv:1907.1305, 2019.
[24] Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. Advances in Neural Information Processing Systems, 29, 2016.
[25] Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. Advances in Neural Information Processing Systems, 30, 2017.
[26) SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances Neural Information Processing Systems, 29, 2016
[27] Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint arXiv:1802.1033: 2018.
[28] Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. Advances in Neural Information Processing Systems, 31, 2018
[29] Karl Stelzner, Robert Peharz, and Kristian Kersting, Faster attend-iifer-reeeatt with tractable probabilistic models. In International Conference on Machine Learning, pages 5966-5975. PMLR, 2019.
[30] Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with con- volutional neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3412-3420, 2019.
11

---

[31] Jindong Jiang, Sepehr Janghorbani, Gerard De Melo. and Sungjin Ahn. Scalor: Generative world models with scalable object representations. arXiv preprint 1.IV:1910.00284, 2019.
[32] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. arXiv preprint arX1v:2001.02407, 2020.
[33] Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. In International conference on learning representations. 2018.
[34] Christoph von der Malsburg. The correlation theory of brain function. Models of neural networks, pages 95-119. Springer, 1994.
[35] DeLiang Wang. The time dimension for scene analysis. IEEE Transactions on Neural Networks, 1616):400-1146, 2005.
[36] DeLiang Wang and Xiuwen Liu. Scene analysis by integrating primitive segmentation and associative memory. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 32(3):254-268) 2002.
[37] David Hom and Irit Opher. Collective excitation phenomena and their applications. Pulsed neural networks, pages 297-320, 1999.
[38] DeLiang Wang and David Terman. Image segmentation based on oscillatory correlation. Neural Computation, 94): 836, 1997.
[39] G Kuntimad and Heggere S Ranganath, Perfect image segmentation using pulse coupled neural networks. IEEE Transactions on Neural networks, 10(3):591-598, 1999.
[40] DeLiang Wang, Joachim Buhmann, and Christoph von der Malsburg. Pattern segmentation in associative memory. Neural Computation, 2(1):94-106, 1990.
[41] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoing Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096-1103, 2008
[42] Sven Behnke. Learning iterative image reconstruction in the neural abstraction pyramid. International Journal of Computational Intelligence and Applications, 1(04):427-438 2001.
[43] Nguyen Xuan Vinh, Julien Epps, and James Bailey. Information theoretic measures for cluster- ings comparison: Variants, properties, normalization and correction for chance. The Journal of Machine Learning Research, 11:2837-2854, 2010.
[44] Jonathan D Victor and Keith P Purpura. Nature and precision of temporal coding in visual cortex: a metric-space space analysis. Journal of neurophysiology. 76(2):1300-1326, 1996.
[45] Andreas K Engel and Wolf Singer. Temporal binding and the neural correlates of sensory awareness. Trends in cognitive sciences, 5(1):16-25, 2001.
[46] Jon H Kaas. Topographic maps are fundamental to sensory processing. Brain research bulletin, 44 1997.
[47] Nelson Spruston. Pyramidal neurons: dendritic structure and synaptic integration. Nature
Reviews Neuroscience, 93):206-221, 2008
[48] Charles C Lee and S Murray Sherman. On the classification of pathways in the auditory midbrain, thalamus, and cortex. Hearing research, 276(1-2):99-77, 2011.
[49] Charles C Lee and S Murray Sherman. Drivers and modulators in the central auditory pathways. Frontiers in neuroscience, 4:14, 2010.
[50] S Murray Sherman and RW Guillery. On the actions that one nerve cell can have another: distinguishing "drivers" from "modulators". Proceedings of the National Academy of Sciences, 95(12):111--1226 1998.
12

---

[51] Peter König, Andreas K Engel, and Wolf Singer. Integrator or coincidence detector? the role of the cortical neuron revisited. Trends in neurosciences, 19(4):130-137 1996.
[52] Reinhard Eckhorn, Herbert J Reitboeck, MT Ardt, and P Dicke. Feature linking via synchro- nization among distributed assemblies: Simulations of results from cat visual cortex. Neural computation, 2(3):293-307, 1990.
[53] Reinhard Eckhorn, Roman Bauer, Wolfgang Jordan, Michael Brosch, Wolfgang Kruse, Matthias Munk, and HJ Reitboeck. Coherent oscillations: A mechanism of feature linking in the visual cortex? Biological cybernetics, 60(2):21-130,, 1988.
[54] Kun Zhan, Jinhui Shi, Haibo Wang. Yuange Xie, and Qiaoqiao Li. Computational mechanisms of pulse-coupled neural networks: comprehensive review. Archives of Computational Methods in Engineering, 24(3):573-588, 2017.
[55] Peter Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65, 1987.
[56] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. ArXiv, abs/2004. 1362, 2020.
[57] David P Reichert and Thomas Serre. Neuronal synchrony in complex valued deep networks. arXiv preprint arXiv: 2013.
(58) Sindy Lowe, Phillip Lippe, Maja R. Rudolph, and Max Welling. Complex-valued autoencoders for object discovery. ArXiv, abs/2204. 02075,, 2022.
[59] Jeremy M Wolfe. Forty years after feature integration theory: An introduction to the special issue in honor of the contributions of anne treisman. Attention, Perception, & Psychophysics, 82(1):1-6, 2020.
[60] Shifei Ding, Lingheng Meng, Youzhen Han, and Yu Xue. A review on feature binding theory and its functions observed in perceptual process. Cognitive Computation, 9(2): 206, 2017.
[61] Sayak Bhattacharya, Matthieu BL Cauchois, Pablo A Iglesias, and Zhe Sage Chen. The impact of a closed-loop thalamocortical model on the spatiotemporal dynamics of cortical and thalamic traveling waves. Scientific reports, 11(1):1-99, 2021.
[62] Andreas J Keller, Morgane M Roth, and Massimo Scanziani. Feedback generates second receptive field in neurons of the visual cortex. Nature, 582(7813):545-544 2020.
[63] Rajesh PN Rao and Dana Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79-87, 1999.
(())))99
[64] Lionel G Nowak and Jean Bullier. The timing of information transfer in the visual system. In Extrastriate cortex primates, pages 205-241. Springer, 1997.
[65] Rongzhen Zhao, Zheyu Yang. Hao Zheng. Yujie Wu, Faqiang Liu, Zhenzhi Wu, Lukai Li, Feng Chen, Seng Song, Jun Zhu, Wenli Zhang. Haoyu Huang, M. Xu, Kaifeng Sheng, Qianbo Yin, Jing Pei, Guoqi Li, Youhui Zhang, Mingguo Zhao, and Luping Shi. A framework for the general design and computation of hybrid neural networks. Nature Communications, 13, 2022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section
(b) Did you describe the limitations of your work? [Yes] See Section 6 And Appendix
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Please Appendix. We have separate Section for broader impact
13

---

(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Full set of assumptions, discussions and proofs can be found in Appendix.
(b) Did you include complete proofs of all theoretical results? [Yes] We provide in the
Appendix
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as a URL)? [Yes] The code for results in this paper can be found on Github:
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We provide clear details in Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Table and Appendix (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix.
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] For example, we cite the Dataset[], Silhouette score[55], and AMI score
(b) Did you mention the license of the assets? [Yes] See Appendix.
(C) Did you include any new assets either in the supplemental material or as a URL2 [Yes] Relevant code is provided in supplementary file
(d) Did you discuss whether and how consent obtained from people whose data you'le 're using/curating? [Yes] See Appendix.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Appendix. The data is synthetic and do not contain personally identifiable information or offensive content
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "kFqvZ_dFjTe", "reviewer": "Reviewer_2YkF", "summary": "This paper proposed a hybrid neural network of SNN and ANN to solve the binding problem. This hybrid network on the one hand expands the representation space with additional temporal dimension, and on the other hand holds the computational power of ANNs. The effectiveness of the model is evaluated on several artificially generated datasets, showing its availability, though not concrete enough. The limitations of the current paper were well illustrated (in the appendix), however, the proposed model could be a significant step towards a broader work.", "questions": "The current model only deals with the binary images, which is quite compatible with the binary spike in the spike coding space (SCS), when this input extends to real-valued pixels, how would it influence the performance of this model, in particular, how would this influence the SCS (I guess you will then need an additional coding mechanism which will then violate the one-to-one correspondence described in line 107)?\n\nBased on the paper, the SNNs provide an additional temporal domain by inducing the neuron dynamics, what is the function of the “spikes” in this model, i.e., what if we replace the spiking neuron models with real-valued dynamic neuron models, which still provide the temporal dimension but then remove the binary coding mechanism? \n", "limitations": "please see above", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The idea of combing SNN and ANN in binding problem is quite novel. The theory (although with idealized conditions) is built very solidly. The experiments show the advances of this combined model compared to pure ANN or SNN. \n\nWeakness: As mentioned above, the complexity of the implemented experiments are not significant enough. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "drE7lLw4ACz", "reviewer": "Reviewer_Ybbh", "summary": "This paper proposes a neural network architecture for solving the visual binding problem. The proposed architecture is comprised of a single layer of spiking neurons and a denosing autoencoder. Importantly, the top-down feedback from the output of the autoencoder modulates the spike timing of the spiking neurons. The solution to the binding problem for every visual input is formed through the iterative top-down and bottom-up processing.\n", "questions": "Major comments: \n\n1- It is not clear in the paper how the k-means clustering is applied to the pure-ANN model. For DASBE, as mentioned in the paper, the clustering is applied to the spike trains. Without the spiking output of the first layer, how is the clustering method implemented? Also, for sake of clarity, it would be helpful to include an example result of the pure-ANN-based clustering in the figures. Moreover, it seems that pure-ANN and folded-DAE are used interchangeably throughout the paper referring to the same type of model. I suggest that the authors use a single term to avoid confusion. \n\n2- In the formulation of DASBE, the output of the DAE decoder is used to module spike timing and it's called $\\gamma$ in equation (4). In folded-DAE what seems to be the same variable is called X'. If they're referring to the same variable, and there is no difference between the two in the two models, I strongly suggest that the authors use the same notation in the two models to avoid any confusion. If there is any difference in how the output of the DAE decoder is used in the two models, it should be explained.\n\n3- In section 4.2.4, the contrastive loss that is used for training the model is a *supervised* contrastive loss, as explained in the supplementary materials. Given that contrastive losses are mostly associated with self-supervised methods, it'd be more clear if the authors would clearly explain that in the case of this problem, a supervised contrastive loss is used to learn a disentangled representation of object shapes. Also, How does the results compare with the results of using only a reconstruction loss?\n\nMinor comments: \n\n1- How are the number of clusters set in k-means? \n\n2- Figure 2b is too small and the text is not readable. It's not clear what the rows and columns and each square are showing in the zoomed-in views of figure 2c. \n\n3- In the last paragraph of page 6, pulse-coupled neural network (PCNN) is introduced, but it's referred to as pure-SNN in the rest of the paper. I suggest that the authors use only one term for referring to this model to avoid confusion.\n\n4- In line 212, \"It is interesting to note that the grouping information is continuous (Fig.2 d.g)\" How are the plots in Figures 2 d-g related to what is stated in this sentence? ", "limitations": "Given that the authors are mostly relying on qualitative comparisons to present the performance of the model, it's important to also include cases where the model doesn't perform very well. Ideally, one would like to see some statistics on the number of cases in each dataset where the model doesn't outperform other approaches or even possibly performs worse. In the absence of such statistics, at least I need to see a comparison between successes and failures of the proposed model. \n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: I enjoyed reading this paper and very much liked the proposed idea. A combination of top-down modulation and spike timing is an intriguing idea as a solution to the binding problem. I believe that both neuroscience and AI communities would, in general, benefit from the publication of this paper as a proof of concept for the potential role of spike timing in binding different properties of an object.\n\nWeakness: The paper could benefit a lot from some clarification in the method. Also, some of the figures need further explanation and editing. For example, it's not clear what the rows and columns are representing in the zoomed in views in figure 2c. Moreover, the details in some figures are not readable, e.g. in figure 3. I have a few concerns and comments that I elaborate below. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Y-QIxIza85S", "reviewer": "Reviewer_QMxx", "summary": "In the paper \"Dance of SNN and ANN: Solving binding problem by combining spike timing and reconstructive attention\" the authors propose a hybrid architecture consisting of a spiking neural networks and an attention map modulating denoising autoencoder to address the binding problem. They test their approach by encoding input images consisting of binary images and evaluate binding properties of the network. Notably, they observe gamma-like oscillations. Moreover, they prove four propositions pertaining to convergence of the network.", "questions": "Line 50: The statement is too strong. Continuous does not necessarily imply representation of uncertainty.\n\nWhat value was used for V_th?\n\nAlgorithm 1, line 2: misspelled Bernoulli.\n\nFig 2a: Why is the rate score approximately zero? When the units start firing, the rate should increase, shouldn't it?\n\nFig 2, panels a, b and h: The authors used an extremely tiny font size. It is not legible even when zooming in a bit.\n\nFig 3: This figure wastes a lot of whitespace and unnecessarily repeats the legend.\n\nTable 1: It should be explained that the values are percentages. The main text describes values between 0 and 1.", "limitations": "The authors discuss potential societal impacts in the supporting information. The authors also discuss limitations with respect to missing biological components and competitiveness when compared to segregation methods.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper describes an interesting idea to address the binding problem: the combination of a spiking neural network with a denoising autoencoder. To the best of my knowledge, this is a novel approach in the context of the binding problem.\n\nHowever, the significance of the paper is lacking. The authors compare their approach to a pure-ANN and a pulse-coupled neural network (PCNN) as a pure-SNN, which do not perform well. However, in the PCNN the authors do not tune all hyperparameters (lines 206ff in the supporting information). Indeed, in the Discussion the authors note that their approach does not compete with a dedicated segmentation algorithm. Hence, significance to the machine learning community is clearly limited. On the other hand, with a denoising autoencoder as an ANN and missing biological ingredients such as STDP, relevance to the neuroscience community is limited too.\n\nRegarding the proven propositions, the authors admit that their propositions do not guarantee convergence of their network in practice. Therefore, their propositions seem to be of limited value.\n\nI could not find conceptual or technical mistakes in the main paper. I did not check the proofs in the supporting information.\n\nThe clarity of the main text is fine. The figure quality could be improved. The labels are often small and sometimes not legible (see Questions).\n\nEDIT: The authors clarified some misunderstandings and I increased my score from 3 to 5.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9yB_oOeX8s", "reviewer": "Reviewer_1EXz", "summary": "In this article, the authors propose to combine Artificial Neural Network (ANN) and Spiking Neural Network (SNN) to solve the binding problem. In their model the ANN is a Denoising Autoencoder (DAE) that has been trained to remove salt and pepper noise on images of single object. The SNN is a coincidence detector that is feeding the encoder of the DAE. In return, the decoder of the DAE is providing an attention map used by the SNN. This is defining a dynamical loop between SNN and ANN. The authors show that the combination of both (ANN and SNN) allows to solve the binding problem on 5 different datasets (Bars, Shapes, Corners, Multi-MNIST, MNIST+Shape).\n", "questions": "* The training procedure is never really detailed. I have understood that only the ANN was trained (as a DAE trained to remove salt and pepper noise in images of single objects), not the SNN. Is it correct ? If so could you explain if the ANN is trained separately ? Including a proper mathematical formulation of the loss function might improve the understanding of the training procedure\n\n* In general, the value of the hyper-parameters are not detailed. Sometimes, I don't even know if they are parameters (updated through a learning process) or hyper-parameters (i.e. set by hand). For example, what about p in eq 3 (Is it learned ) ? I would suggest the authors to update their article to make this specific point clearer.\n\n* In figure 3 (and in table 1), we observe that the performances are lower for Multi-MNIST (and MNIST + shape). This is not properly discussed in the article. \n\n* In section 5: [1] should be added in the reference (this is one of the most recent model that solve the binding problem)\n\n* The proposed network should be compared to other state of the art networks. For exemple other complex auto encoder [1] are known to also solve the binding problem.\n\nMinor:\nLine 107 : N is not defined (I guess this is the size of the image, but it should be explicitly said)\n\nLine 115: peroid —> period\n\nLine 142 dynamcs —> dynamics\n\nLine 209: On the contrast —> \"In Contrast\" or \"On the contrary\"  \n\nLine 255 Equivarince —> Equivariance\n\n\n[1] : Löwe, Sindy, et al. \"Complex-Valued Autoencoders for Object Discovery.\" arXiv preprint arXiv:2204.02075 (2022).", "limitations": "The authors have properly discussed the limitations and the societal of their work in their article.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clear and well written. To the best of my knowledge the type of model presented in this article is novel. Thus, this is a valuable contribution to the field. Another strength of the article lies in the method to evaluate the synchrony level (which is also a novelty)\n \nThe main weakness of this article is the lack of comparison with other related models. Even if the principle behind the presented model is novel, other models also claim to solve the binding problem (e.g. complex autoencoders in [1]...). \n\nOverall, I would tend to accept the article but it is borderline because there is no comparison with other algorithms solving a similar tasks (so it is hard to say if the proposed model is bringing a better binding ! )\n\n[1] : Löwe, Sindy, et al. \"Complex-Valued Autoencoders for Object Discovery.\" arXiv preprint arXiv:2204.02075 (2022).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept