Continuous Spatiotemporal Events Decoupling through Spike-based Bayesian Computation
Authors: Yajing Zheng
Jiyuan Zhang
Zhaofei Yu
Tiejun Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: Bayesian Computation
Spiking Neural Network
Event Cameras
Motion Segmentation
Winner-Take-All
Spike-timing-dependent plasticity
Primary area: neuroscience_and_cognitive_science

Abstract: Numerous studies have demonstrated that the cognitive processes of the human brain can be modeled using the Bayesian theorem for probabilistic inference of the external world. Spiking neural networks (SNNs), capable of performing Bayesian computation with greater physiological interpretability, offer a novel approach to distributed information processing in the cortex. However, applying these models to real-world scenarios to harness the advantages of brain-like computation remains a challenge. 
Recently, bio-inspired sensors with high dynamic range and ultra-high temporal resolution have been widely used in extreme vision scenarios. Event streams, generated by various types of motion, represent spatiotemporal data. Inferring motion targets from these streams without prior knowledge remains a difficult task. The Bayesian inference-based Expectation-Maximization (EM) framework has proven effective for motion segmentation in event streams, allowing for decoupling without prior information about the motion or its source. 
This work demonstrates that Bayesian computation based on spiking neural networks can decouple event streams of different motions. The Winner-Take-All (WTA) circuits in the constructed network implement an equivalent E-step, while STDP achieves an equivalent optimization in M-step. Through theoretical analysis and experiments, we show that STDP-based learning can maximize the contrast of warped events under mixed motion models. Experimental results show that the constructed spiking network can effectively segment the motion contained in event streams.

Continuous Spatiotemporal Events Decoupling through Spike-based Bayesian Computation
Yajing Zhengles lee
Jiyuan Zhang' Zhaofei Yu 1.20 Tiejun Huang! 4.2
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University Institute for Artificial Intelligence, Peking University {yj zheng, , yuzf12, tjhuang) 0pku. edu. cn Jyzhangestu pku. edu.
Abstract
Numerous studies have demonstrated that the cognitive processes of the human brain can modeled using the Bayes theorem for probabilistic inference of the external world. Spiking neural networks (SNNs), capable of performing Bayesian computation with greater physiological interpretability, offer a novel approach distributed information processing in the cortex. However, applying these models to weal-world scenarios to harness the advantages of brain-like computation remains challenge. Recently, bio-inspired sensors with high dynamic range and ultra-high temporal resolution have been widely used in extreme vision scenarios. Event streams, generated by various types of motion, represent spatiotemporal data. Inferring motion targets from these streams without prior knowledge remains difficult task. The Bayesian inference-based Expectation-Maximizatee framework has proven effective for motion segmentation in event streams, allowing for decoupling without prior information about the motion or its source. This work demonstrates that Bayesian computation based on spiking neural networks can decouple event streams of different motions. The Winner-Take- circuits in the constructed network implement an equivalent E-step, while Spike Timing Dependent Plasticity (STDP) achieves an equivalent optimization in the M-step. Through theoretical analysis and experiments, we show that bbseed learning can maximize the contrast of warped events under mixed motion models. Experimental results show that the constructed spiking network can effectively segment the motion contained in event streams.
(EM)
(WTA)
Introduction
Bayesian computation is a fundamental concept in statistics, machine learning, and computational neuroscience [35, 3J. The Bayesian brain hypothesis suggests that the brain functions as a probabilistic generative model, simultaneously inferring hidden causes of sensory inputs and refining its model parameters [22, 8, 21J. Bayesian computation requires a generative model to predict observations, formulated as the joint probability P(æ, y) of observations and hidden states y. This joint probability can decomposed into the prior Pvy) and the likelihood P(æly). which are combined using the Bayes theorem to update the before posterior probability P(y|z).
The brain's 's inferential processes use probabilistic models to represent and update perceptions based on new sensory input, framing perception as an active test of predictions against sensory data [11, 34).
Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Events
Segmented E Events
Likelihood Assignment
Psp
Event Input
Spike uuput
P1
-2
Motion Estimation
0:
Fe f
P(s, fiee spila)) P.
argmax,PPC
Motion-cormene collencomppensteed Events
P(a,= 1)= Update with STDP rule
I
STDP
(a) EM-based clustering methad
(b) basee Bayesian computation
Figure I: Comparison of EM-based Clustering and the proposed Spike-based Bayesian Computation Methods for Event Decoupling.
Bayesian inference in the brain operates through neural circuits, where spikes encode activity, and synaptic weights facilitate information integration. Soft-WTA circuits, found in cortical microcir- cuits [7], support selective activation of neurons, allowing the most active neurons to suppress less active ones, which improves computational efficiency. This competition-driver mechanism works in tandem with STDP [4, 9]. which strengthens synapses based precise spike timing, aligning with Bayesian principles to refine predictions. This Bayesian framework has promising applications in train-maching interfaces and neuromorphic computing [25]. While existing frameworks demon- strate feasibility, their application to complex, spatiotemporal tasks remains a rich area for further
exploration.
Recent advances in neuromorphic vision sensors [20, 13, 18] mimic retinal function, generating asynchronous event streams based light intensity changes. Unlike traditional RGB cameras, these sensors capture motion effectively and reduce blur with high temporal resolution. Event cameras generate events independently at each pixel, complicating data with various object movements and camera motion. This "chicken -anckenn-nddegg" problem of distinguishing and solving different motions can be addressed using a Bayesian framework combined with the EM algorithm [1], iteratively updating motion model probabilitics.
Previous work [38, 44] has demonstrated that jointly optimizing motion parameters and updating event-cluster membership probabilities in an EM fashion can solve motion segmentation problems. This approach achieves per-event segmentation rather than rough region segmentation and supports multiple motion models. The optimization of motion parameters is mainly achieved using the Contrast Maximization (CM) algorithm [14], aligning events to a reference time similar to motion compensation video processing.
How can an SNN model implementing Bayesian computation decouple and segment event streams generated by different motion patterns? Earlier spike-based EM algorithms primarily focused on generating models for mixture distributions, such as Gaussian mixture models, where the probability of each sample belonging each sub-disti distrribbbtt equal. contrast, in motion segmentation, the probability of each event stream belonging to a specific motion parameter distribution is variable. Additionally, optimizing for mixture distributions involves maximizing the likelihood function of each distribution given the observed samples, which differs from the objective of maximizing contrast in event-based motion segmentation. Therefore, applying WTA circuits combined with STDP for motion segmentation in event streams presents several challenges.
This work proposes a spike-based Bayesian computation framework for event segmentation. We theoretically prove that the proposed framework is equivalent to previously implemented event- based motion segmentation algorithms via motion compensation. Experimental results demonstrate that the constructed spiking network can effectively segment motions in event streams. This work provides a theoretical foundation for applying the Bayes theorem using spiking neural networks to event stream decoupling tasks. We aim to provide a theoretical foundation for applying Bayesian inference using SNNs for event stream decoupling tasks, leveraging the energy efficiency of SNNs. This is particularly relevant for neuromorphic hardware (NMHW), e.g., Loihi [5]. SpiNNaker [12] and Tianjic [6], known for their low latency and power consumption. Neuromorphic computing emulates the brain's 's low-power yet complex visual processin capabilities. Previous research

---

validated the hypothesis that SNNs can implement Bayesian inference [29]. However, its application neuromorphic computing hardware remains unverified. Our work aims to validate the capability of a spike-based Bayesian computation framework applied to neuromorphic sensors.
Contributions of this work be summarized follows:
Develop a spiking Bayesian computation framework for continuous motion segmentation in event streams, demonstrating its ability to perform similarly to previous EMM algorithm-based models.
Show that the WTA circuit can implement the E-step and that STDP rules can achieve the M-step for contrast maximization, validating these components" theoretical and practical efficacy.
Verify the proposed network can effectively learn online from continuous input, enabling accurate motion segmentation through the firing of output neurons.
2
Related Works
Spike-based Bayesian Computation. Spike-based Bayesian computation employs various methods achieve probabilistic inference and optimization in neural circuits, such maximum likelihood posterior function. For example, SNNs implement Bayesian inference using belief propagation on binary MRFs and tree-based reparameterization for exponential family distributions [31, 40, 41, 42], which approximate the posterior probability. Spiking neuron-base based neural sampling [2, 40] uses non-reversible process, like Markov chain Monte Carlo (MCMC) sampling. A particularly suitable method for complex tasks like motion segmentation in event streams involves using WTA circuits combined with STDP [29, 28] learning rules to implement the EM algorithm for decoupling mixture distribution. This approach leverages the competitive dynamics of WTA circuits to estimate joint probability distributions, offering high accuracy and robustness in learning and inference. The WTA- STDP framework is well-suited for handling the asynchronous and event-driven nature of spiking data, making it effective for segmenting complex motion patterns event streams.
Event-based motion segmentation. Recent methods [26, 16, 39, 33, 27, 38] in event- motion segmentation leverage clustering, probabilistic models, motion compensation, and deep learning. Cluster- based techniques group events by similar motion patterns and are simple to implement, but may struggle with complex scenes. Probabilistic approaches, such as those using Bayesian frameworks and the EM algorithm [38, 44], iteratively estimate motion parameters and provide robust segmentation, though they can be computationally intensive. Motion compensation aligns events to a reference frame to enhance sharpness (contrast) of warped events [38], offering high accuracy, but may be sensitive to parameter initialization. Deep learning methods employ neural networks to predict motion segments directly from raw event streams, providing high performance and adaptability [19. 37, 27], but requiring large datasets for training. Some works [32, 43] also attempt to SNN for estimating motion in event streams, thereby achieving motion segmentation or object detection. The main idea of these methods is to first convert the event stream into time- event-like inputs, rather than directly distinguishing the event stream in the spatiotemporal dimension. Consequently, these methods exhibit a certain lag and can easily be disturbed by camera self-motion, which interferes with the analysis of moving objects.
3 Preliminaries
3.1 Event Cameras.
Dynamic Vision Sensors (DVS) [23] in event cameras detect brightness changes independently each pixel, generating events instead of capturing images at fixed intervals. An event eg = (Irg, Lks (s) occurs when the intensity change (L(x,, at pixel 2y exceeds threshold e The change in intensity is expressed as:
AL t4) L(x tx) L( to At4) = 9
(1)
where [(,,, 1) is the logarithmic intensity at pixel 4 is the event timestamp, Ale is the interval since the last event at the same pixel, and qe E [1] indicates the polarity of the intensity change. Since our work does not use event polarity, this property will not be referenced further. This
3

---

asynchronous mechanism enables event- based cameras to capture dynamic scenes with high temporal resolution and low latency, making them ideal for real-time applications.
3.2 Event-based Motion Segmentation in EM Fashion.
In previous event-based motion segmentation algorithms, the probability that an event stream e belongs different motion models is denoted as P. The motion model that generates the event is then used to warp the event positions, resulting in a sharp Image of Warped Events (IWE), The IWE 1, (x) for given motion parameter 03 is typically calculated by warping events onto specific time plane, where z represents the pixel location, and refers to the j-th motion model:
N. Σ Pkjō(z +z)))
(x
where 5()) denotes the Dirac function, and Pkj represents the probability that event eg belongs to motion model Zj. The Dirac function can also be replaced by smoother kernel function, such as the Gaussian function. T'ss is the transformed event position obtained using motion parameters 0y:
Kij (I Lx; 0j).
The objective of the entire model is to find the motion parameters 0* and the event-cluster probability P that maximize the variance of all IWEs corresponding to different motions:
N
(0" P*) = arg max (0.P)
Var(I,).
3=1
In EM fashion, first perform the E-step calculate the posterior probabilities of the samples belonging to the distribution based on the assumed model and parameters. The initial motion parameters are used calculate the IWE for different motions. The responsibility Pky for each event belonging to a motion model is then computed using the formula:
(O,)
Pkj = Σ (ok))
(5)
Next, in the M-step, these responsibility values used to optimize the motion parameters, maximiz- ing the objective function f0). In this work, we use a different variance form to [38], but the effect is the same. The objective function is computed
N₂
N2
((0)
Σ Var(Iy)
Σ
E2
E, [x]
(6)
j=1
j=1
where E[x] represents the expectation process as:
[x] [ / (x(z) dx,
E, [10 [
(xx)² dr.
(8)
Ω denotes the image plane. The gradient of the variance Var(1,) given by:
@ar
[x
OE [x]
AOz
2E,
[z]]
(9)
a0
00
30
This method effectively distinguishes event streams generated by different motions and produces sharp IWE. Other optimization strategies, such as the conjugate gradient method [30], can be applied to update the motion parameters 0,-
4 Spike-based Bayesian Computation for Motion Segmentation
The main procedure for event-based motion segmentation involves three key steps: 1) obtaining the IWE. 2) assigning probabilities for events corresponding to different motion models, and 3) optimizing the motion parameters to maximize the contrast of the IWE. In our model, we use a network with a WTA circuit to compute the IWE and the probabilities P, and we update the motion parameters using the STDP rule. Below, we will demonstrate that our constructed model and learning method approximates the event-based motion segmentation approach.

---

WTA
pike O Outuut
Warp
Event Input
3/1
21
Global Inhibition
3/2
22
Pkj
H
exi tx:
y
23
Event Stream
22N,
Update through STDP
Figure 2: Architecture of the spike- based motion- nootionsegmmnnatioon network.
4.1 Model Construction and Learning
E-step. The first step of event-based motion segmentation involves obtaining the IWE for different motion parameters and then determining the event-cluster responsibility values Ptj by comparing the contrast of the IWE at the corresponding positions after warping. This operation is analogous to neuron competition, where the motion model that produces the highest contrast gains ownership of the events mapped that location. This process can be implemented using WTA eircuit, which is how our proposed network updates the responsibility values for different events.
As shown in Fig. 2, in network, neurons y representing different motions are responsible for warping events and transmitting them to the output neurons z. There is a one-to-one correspondence between y and The output neuron zj follows an integrate-ang and-fire (IF) model [15], with the membrane potential uy expressed as:
u, (t = Σ Wylek Pky:
(10)
k=1
where W, represents the operation of the motion neuron y, It is evident that by accumulating the warped event streams, equivalent to obtaining the IWE in Eq. 2.
The output neuron 2, receives feedback from global inhibition neuron H. The value of the global inhibition neuron is the sum of all IWE values, i.e., H(tt) = E, uy (t). Here, we use stochastic firing model for zj. where the firing probability depends on the membrane potential in conjunction with the inhibition neuron H. This computation can be formulated as:
uy (t)
piz; fire at time f)
(11)
H())
Unlike the WTA circuits constructed by Nessel al. [29, 28], our output neurons form tensor corresponding to the event space dimensions. The synaptic parameters updated by STDP do not represent the connection weights between scalar neurons but rather the coefficients of functions representing the receptive fields of different motion neurons (e.g., functions that linearly transform along the direction of motion). The concept of neurons representing tensor values has also been applied in various works, such as in capsule networks [36].
M-step. After obtaining the network's output, we can update the network parameters using the STDP rule based on the relationship between the firing of 2 and the input events. In our previous definitions, the network parameters include only the motion parameters 0,. The probability Pkj is derived from the firing probability of the output neurons and is also used as the input for the next step the network training process.
According to the STDP rule, if the firing of the presynaptic neuron leads to the firing of the post- synaptic neuron, their synaptic weight is increased, corresponding long-term potentiation (LTP). Otherwise. the synaptic weight is decreased, corresponding to long-term depression (LTD). The synaptic weight updated as follows:
0 + p(presynaptic neuron fired within [t' [L' o, ['] postsynaptic neuron fires t'). (12)
5

---

(a)
(b)
trajectory of 0
trajectory of 6i
300
6
200
AB,- a Varlu;(t)) Ps:
38
100
100
200
300
-log(p(z) 1|e 1))
300
200
-000
0
100
200
300
Figure 3: Illustration of learning through STDP. (a). Learning curves of STDP for motion parameters 0. (b). Optimization trajectory of parameters during STDP learning. The heat map shows the gradient of f0) in Eq. 6 as motion parameters change.
In this context, the probability of firing between the presynaptic event and the postsynaptic neuron is positively correlated with their association probability Pkj- Consequently, the update direction of the motion parameter 0j is also positively correlated with Pkj. Combining this with the concept of motion compensation, the goal of updating 0, to maximize contrast Var(T,), so the gradient direction is related to the contrast of (t) (where uy (t)(t) is equivalent to IWE Ij). Therefore, in our proposed network, the gradient update for the motion parameter 02 is given by:
aw2
30 17. Var(u,())) Pkj
(13)
00;
Fig. 3(a) shows the learning curve of STDP. We do not explicitly show the use of LTD for updates because the presence of P&j in the update formula ensures that if the event stream does not belong to the motion model, will not significantly affect the update of its motion parameters. This approach is reasonable, as low Pkj value indicates that the current input event stream does not maximize the contrast for the motion model. Hence, it should not influence the learning of the motion neuron parameters, thereby preventing interference with the detection of other motion patterns in the event stream.
Our objective is to maximize the sum of the variances of the IWE across different motion parameter distributions (Eq. (4)). As described in Fig. S7 of the Appendix, the variance indicates that correct motion patterns concentrate the event flow distribution along the object's ' edges, resulting in higher variance. In contrast, incorrect motion patterns disperse the event flow, leading to lower variance. In the network we designed, which includes a WTA mechanism, the parameters of the motion neurons, denoted as 0, are optimized using the STDP rule to achieve this goal. As shown in Eq. (12) and Fig. 3, when P(zz l|e+ 1). only event streams that match motion pattern will activate the corresponding motion neuron to adjust its parameters, thereby maximizing the variance of IWE (encoded by u) associated with motion pattern j.
In our network, applying STDP optimize the parameters of motion neurons does not strictly follow the gradient of the objective function for each parameter. However, the angle between the update direction and the gradient of the objective function is less than 90 degrees. Previous work has demonstrated that synaptic updates need only maintain an angle less than 90 degrees with the error function to achieve optimization [24]. We will also show that under this update rule, the contrast of the IWE can gradually increase.
4.2 Equivalence of STDP-Rule Updates to M-Step in Event-Based Motion Segmentation
In the M-step of event-based motion segmentation, the goal is to optimize the parameter values by maximizing the contrast of the IWE. The objective is to increase the contrast of all IWE values during the optimization process, and various optimization strategies can employed. Our proposed STDP rule effectively increases the value of Var(u,).

---

(a)
parameter search on 11th-patch events
parameter search on 1st-patch events
2,055
1f
200
-3.10
00045
of
-2.10
0.055
-200
200
202
122
0%
0:
11th-patch events
warped with Bj.
1st-pates patch events
warped with 81
component 1 of SVD
Figure 4: Initialization parameters 0 through sampling parameters with the contrast of IWE as the objective function. (a). SVD components of parameters of different patches. (b). Sampling process of parameters of different patches. (c). Warping events with the best sampling parameters 8".
The variance formula is expanded as follows:
N N
N
1
1
Var(u,) = u (B))
Σ n
(k))
(14)
k=1
where N |/2/. Assume that after the STDP update, the parameter becomes f9, = =, A8,, resulting in output u' with variance Var(u')). The updated variance formula:
Var(u'))
1 N = Σ
(1))
1
Σ
N
u1(())
(15)
N
N
The updated output is
uk((k) uA(k) (k,(k),
(16)
where (ky)k) is the change induced by 20,. The change in variance is calculated as:
1 Var(u))-Var(u,)) N Σ (4)
Σ (k) (K))))()uu))) Au,
-Var(uy) (17)
Since (k) a small change relative to 0j. we can use a first-order approximation:
Au, ~ ou, (k) 00,
A0,.
(18)
The parameter update increases the variance. According to the STDP update rule, the direction of A0, consistent with @Vrr(u,)/00, Therefore, the variance will increase after the update.
Fig 3(b) illustrates the optimization trajectory of the motion parameters when applying STDP to maximize the objective function Var(u, (u, (t)) for motion segmentation. The results show that under the STDP learning rule, influenced by the WTA circuit, the input events most relevant to the motion are used for optimization. This allows to progress in a direction that maximizes corresponding event stream. Additionally, this process does not interfere with the learning of other motion patterns 0p. The WTA eircuit combined with the STDP rule ensures that the contrast of event streams not belonging to the motion pattern is not forcefully maximized.
contrast of the
7

---

Experiments
5.1 Implementation Details.
The proposed spike-hased Bayesian Computation framework and its corresponding event-based clustering framework are both locally convergent algorithms. The STDP rule adjusts the weights based the firing patterns of pre- and post-synaptic neurons, and it does not guarantee convergence to global solution. To ensure both algorithms converge to a better solution, we adopt strategy similar to the event-based layered algorithm for initializing 0 and P. The number of motion models Ne and the type of motion models (e.g., linear motion, affine, or rotation) are hyperparameters that need to be predefined. Considering the ultra-high temporal resolution advantage of event cameras, we primarily set the motion model linear motion, i.e., 0 = {Uz, By After setting the number of motion models Ny and the form of I, we take subset of events for parameter initialization.
Parameter Initialization.
Specifically, we divide the events into different patches and use the specified motion parameters to maximize the contrast of the IWE corresponding to these patch events. To search for parameters more efficiently instead of using brute force methods (e.g.. grid search), we employ combination of random sampling and Bayesian optimization using the Tree-structured Parzen Estimator (TPE) [10]. After completing the parameter search, we obtain a parameter set corresponding to the number of patches Nup (N > N2). Since different patches may have similar motion parameters, we use Singular Value Decomposition (SVD) [17] analyze the components of the returned parameter set and select the top N₁ parameters with the most significant differences as the initialization parameters for the algorithm.
Fig. shows the parameter initialization process sequence from the Extreme Event Dataset (EED) [26], which includes both camera self-motion and moving object. The initial parameter search identifies two significantly different motion parameters corresponding to the event streams from the 1st- and 11th patches, respectively.
Network Learning and Inference. After initializing the parameters, we select a fixed number
of events in chronological order, dividing all events into different packets {e" J as inputs to the network over time. During online learning, we also split the n-th events packet {e") into different patches and feed them into the network. After optimizing the parameters 0 for several epochs, we obtain the optimized motion parameters 0", and then input all events into the network to get the responsibilities P of all events belonging to different motion parameters. The motions estimated by clustering (e") can be propagated in time to predict an initialization for the clusters of the next packet {e" [e"+] ). All steps of the proposed method are summarized in Algorithm. 1.
Algorithm 1 Spike-based Bayesian Computation for Event Motion Segmentation
Input: Events packet {eg N. k= number of clusters Nę.
Output: Cluster parameters
(0, (0 event-cluster assignments P.
1:
procedure
2: Initialize the unknowns (8, P) by sampling potential motion patterns of different patches of events based on the TPE [10] and SVD [17].
3: while not converged do
4:
E-step Compute the event-cluster assignments Pkj based on the WTA circuit using Eq. 10 and Eq. 11.
5: M-step Update the motion parameters of all clusters using Eq. 13 based on the STDP rule.
6: end while
7: end procedure
Network Learning and Inference:
1: Divide events into packets { based on fixed event count.
2: for each packet e do
3: Split events into patches and feed into the network.
4: Optimize parameters for several epochs to obtain optimized motion parameters 0*.
5: Feed all events into the network to obtain event responsibilities P.
6: end for

---

Accumulated events (w/ polarity)
Before learning
Event streams
height
IWE1
IWEz
time
width
Spike-inferenced Events (w/o polarity)
After learning
Figure 5: Examples of motion segmentation through the proposed spike-based EM models.
Our model is event-driven and operates with parallel computations across different patches. The primary focus is on CPU-based verification due to minimal graphical operations, ensuring fast processing as an online learning algorithm. Additionally, the speed on both GPU and CPU is comparable. Our network structure is compact, resulting in low memory consumption.
5.2 Evaluation on Event Motion Segmentation Datasets
Fig. 5 shows the optimized motion parameters 6* and the spike firing rates Pkj of the output neurons for the scene described in Fig. 4. The figure also presents the IWEs and the events warped and fused according the corresponding motion parameters. After training with the proposed network, the IWEs for different motion parameters show higher contrast compared to before learning. This allows for effective separation of the camera's 's self-motion from the motion of the ball. The Spike-inference Events in Fig. 5 are accumulated based on the spike activity of different neurons represented by colors. verify that the proposed spiking neural network can learn the parameters of motion neurons online from continuous input, we continuously feed the event stream into the network and observe firing state of the output neurons z.
Fig. 6 shows the network's 's performance in segmenting three scenarios in the EED that involve mixed camera self-motion and high-speed moving objects. In these scenes, the background event streams vary in density and shape. Despite these varying backgrounds, the output neurons can still distinguish the motion parameters of moving objects. This demonstrates that the proposed network can learn to suppress irregular input spike patterns and, through local plasticity learning, identify the motion parameters that maximize contrast in specific regions, thereby accurately locating different moving objects.
Conclusions & Discussions
This paper proposes spike Bayesian computation framework for continuous motion segmentation in event streams. We demonstrate that the constructed network can achieve the same effect as previous event-based motion segmentation models using the Expectation Sittation-MMxiizzatioon (EM) algorithm. Specifically, the WTA circuit the network implements the equivalent E-step, and the STDP rule for adjusting network parameters realizes the equivalent M-step for contrast maximization. Both theoretical analysis and experimental results show that STDP-based learning can optimize the contrast of mapped images under mixture motion parameter model. Using the Extreme Event Dataset, we validate the network's ability to learn online from continuous input and perform motion segmentation through the firing of output neurons.

---

What is background?
Accumulated events (w/ polarity)
Occlusions
Fast drone
time
Figure 6: Motion segmentation results for continuous event streams. Different colors represent the firing different output neurons z the proposed spike-based network.
Limitations. This work primarily aims to demonstrate that a biologically inspired network frame- work implicitly does Bayesian computation can decouple spatiotemporal data, proposing a prototype framework applicable to neuromorphic cameras. The validation focuses motion segmentation in event streams rather than pursuing state-of-the-ard performance. Therefore, there is limited quan- titative performance evaluation and comparison with existing event-based motion segmentation algorithms. This limitation arises because the network relies on local learning rules and may not converge to the optimal solution, being highly dependent on parameter initialization.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (62306015, 62176003, 62088102), the China Postdoctoral Science Foundation (2023T160015), the Young Elite Scientists Sponsorship Program by CAST (2023QNRC001) and the Beijing Nova Program (20230484362).
References
[1] Christopher Bishop. Pattern recognition machine learning. Springer google schola, 2:1 1128.
2006.
[2] Lars Buesing, Johannes Bill, Bernhard Nessler, and Wolfgang Maass. Neural dynamics sampling:
model stochastic computation recurrent networks of spiking neurons. PLoS computational biology.
111)22100 2011.
[3] Andy Clark. Whatever next? predictive brains, situated agents, and the future of cognitive science. Behavioral brain sciences. 36(3):181-208 2013.
[4] Yang Dan and Mu-ming Poo. Spike timing- dependent plasticity of neural circuits. Neuron, 44(1):23 30, 2004.
[5] Mike Davies, Narayan Srinivasa, Tsung- Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. leee Micro. 38(1):82-93 99, 2018.
[6] Lei Deng. Guanrui Wang, Guoqi Li, Shuangchen Li. Ling Liang. Maohua Zhu, Yujie Wu, Zheyu Yang. Zhe Zou, Jing Pei, et al. Tianjic: A unified and scalable chip bridging spike-bases and continuous neural computation. IEEE Journal of Solid-Sh State Circuits, 55(8):2228 2246, 2020.
10

---

[7] Rodney Douglas and Kevan AC Martin. Neuronal circuits of the neocortex. Annu. Rev. Neurosci. 27:419-451, 2004.
[8] Kenji Doya. Bayesian bruin: Probubilistic approaches to neural coding. MIT press, 2007.
[9] Daniel E Feldman. The spike-timing dependence of plasticity. Neuron, 75(4):556-571, 2012.
2012.
[10] Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient- based hyperparameter optimization. In International Conference on Machine Learning, pages 1 165--173.
PMLR, 2017.
[11] Karl Friston. The history of the future of the bayesian brain. Neurolmage. 62(2): 2012.
[12] Steve B Furber. David R Lester, Luis A Plana, Jim D Garside, Eustace Painkras. Steve Temple. and Andrew D Brown. Overview of the spinnaker system architecture. IEEE transactions on computers. ((12):2454-2467 2012.
[13] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al. Event-based vision: A survey,
IEEE TPAMI, 44(1):154-180, 2020.
[14] Guillermo Gallego, Henri Rebecq, and Davide Scaramuzza. A unifying contrast maximization framework for event cameras, with applications to motion, depth, and optical flow estimation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3867-3876, 2018.
[15] Wulfram Gerstner. Integrate-and-firg neurons and networks. The handbook of brain theory and neural networks, 2:577-581, 2002.
[16] Arren Glover and Chiara Bartolozzi. Event-driven ball detection and gaze fixation clutter. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2203- 2208. IEEE, 2016.
[17] Nathan Halko, Per-Gumnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53(2):217-288, 2011.
[18] Tiejun Huang, Yajing Zheng, Zhaofci Yu, Rui Chen, Yuan Li, Ruiqin Xiong, Lei Ma, Junwei Zhao, Siwei Dong, Lin Zhu, et al. 1000 faster camera and machine vision with ordinary devices. Engineering,
25:110-119 11910-1-,2201001111 2023.
[19] Xueyan Huang, Yueyi Zhang, and Zhiwei Xiong. Progressive spatio-temporal alignment for efficient event-based motion estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1537-1546, June 2023.
[20]
Giacomo Indiveri and Rodney Douglas. Neuromorphic vision sensors. Sciemce,,88(5486))) 288(5469): 188-11190,
2000.
[21] Friston Karl. A free energy principle for biological systems. Entropy. 14(11):2100 2121, 2012.
[22] David C Knill and Alexandre Pouget. The bayesian brain: the role of uncertainty in neural coding and computation. TRENDS in Newrosciences, 27(12):722-71 719, 2004.
[23] Patrick Lichtensteiner, Christoph Posch, and Delbruck. A 128x128 120db 15ps latency asynchronous
temporal contrast vision sensor. IEEE Journal of Solid-State Circuits, (2):566-576, 2008.
[24] Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7(1x:3276, 2016.
[25] Danijela Marković, Alice Mizrahi, Damien Querlioz, and Julie Grollier. Physics for neuromorphic
computing. Nature Reviews Physics, 2(9):499-51) 510, 2020.
[26] Anton Mitrokhin, Cornelia Fermüller, Chethan Parameshwara, and Yiannis Aloimonos. Event-based moving object detection and tracking. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1-9. IGEE, 2018.
[27] Anton Mitrokhin, Chengxi Ye, Cornelia Fermüller, Yiannis Aloimonos, and Tobi Delbruck. Ev-imo: Motion segmentation dataset and learning pipeline for event cameras. in 2019 ieec. In RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6105-6112.
[28] Bernhard Nessler, Michael Pfeiffer, Lars Buesing, and Wolfgang Maass. Bayesian computation emerges in generic cortical microcircuits through spike-timing dependent plasticity. PLoS computational biology. 1003:377 2013.
[29] Bernhard Nessler, Michael Pfeiffer, and Wolfgang Maass. Stdp enables spiking neurons to detect hidden causes their inputs. Advances in neural information processing systems, 22, 2009.
[30] Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer. 1999.
[31] Thomas Ott and Ruedi Stoop. The neurodynamics belief propagation on binary markov random fields.
Advances in neural information processing systems, 19, 2006.
[32] Chethan M Parameshwara, Simin Li. Comelia Fermüller, Nitin J Sanket, Matthew S Evanusa, and Yiannis Aloimonos. Spikems: Deep spiking neural network for motion segmentation. In 202/ IEEERS) International Conference on Intelligent Robots and Systems (IROS), pages 3414-3420. IEFF, 2021.
[33] Chethan M Parameshwara, Nitin J Sanket, Arjun Gupta, Comelia Fermuller, and Yiannis Aloimonos. Moms with events: Multi-object motion segmentation with monocular event cameras. arXiv preprint arXiv:2006.0m 06108. 2(3):5. 2020.
[34] Alexandre Pouget, Jeffrey M Beck, Wei Ji Ma, and Peter E Latham. Probabilistic brains: knowns and unknowns. Nature neuroscience, 16(9): 1100 1178, 2013.

---

[35] Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: functional interpretation of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79-87, 1999.
[36] Sara Sabour, Nicholas Frosst, and Geoffrey Hinton. Dynamic routing between capsules. Advances in neural information processing systems, 30, 2017.
[37] Nitin Sanket, Chethan Parameshwara, Chahat Deep Singh, Ashwin Kuruttukulam, Cornelia Fermüller, Davide Scaramuzza, and Yiannis Aloimonos. Evdodgenet: Deep dynamic obstacle dodging
with event cameras. In 2020 IEEE International Conference Robotics and Automution (ICRA), pages 10651-10657. IEEE, 2020.
[38]
Timo Stoffregen, Guillermo Gallego, Tom Drummond, Lindsay Kleeman, and Davide Scaramuzza. Event-
based motion segmentation by motion compensation. In ICCV, pages 7244-7253, 2019.
[39] Valentina Vasco, Arren Glover, Elias Mueggler, Davide Scaramuzza, Lorenzo Natale, Chiara Bartolozzi.
Independent motion detection with event-driven cameras. In 2017 18th International Conference on
Advanced Robotics (ICAR), pages 530-536. IEEE, 2017.
[40] Rajkumar Vasudeva Raju and Zachary Pitkow. Inference by reparameterization in neural population codes. Advances Neural Information Processing Systems, 29, 2016.
[41] Zhaofei Yu. Feng Chen, and Jianwu Dong. Neural network implementation of inference binary markov random fields with probability coding. Applied Mathematicx and Computation, 301:193-200, 2017. [42] Yajing Zheng, Shanshan Jia, Zhaofei Yu, Tiejun Huang. Jian K Liu, and Yonghong Tian. Probabilistic inference of binary markov random fields in spiking neural networks through mean-field approximation. Neural networks, 126:42-51, 51, 2020.
[43] Yajing Zheng. Zhaofei Yu, Song Wang. and Tiejun Huang. Spike-based motion estimation for object tracking through bio-inspired unsupervised leamning. IEEE TIP, 32:335-349, 2022.
[44] Yi Zhou, Guillermo Gallego, Xiuyuan Lu, Siqi Liu, and Shaojie Shen. Event-based motion segmentation
with spatio-temporal graph cuts. IEEE Transactions on Neural Networks and Learning Systems, 34(8):8868-
4880, 2021.
12

---

A Appendix
A.1 Explanation Var(1;)
As shown in Fig. S7, correct motion models concentrate event distributions at object edges, resulting in higher variance, while incorrect models disperse events, leading to lower variance. This helps in validating the accuracy of motion pattern detection.
Event Stream Space- Time
Verience: 562.87
Veriance: 55.16
00
40
20
100
25-
30
50
20
Figure S7: Explanation of Var(4)) From left to right: events, IWEs of different motion patterns.
A.2 Joint probability density
From the perspective of a probabilistic model, we can consider that events conforming to different motion patterns emerge over time. However, in the task of dividing event streams based on motion patterns, it is hard to directly generate a model (e.g., mixture distribution model) using a generative model. In previous work [34], the authors adopted method based on Eq. (2), which is more suitable for traditional mixture distribution models (e.g., fuzzy mixture models and k-means clustering) to divide event streams by motion.
In fuzzy mixture density and k- k-means methods, the motion- compensated IWEs do not include the event cluster associations P, which means that sharper object boundaries appear some IWEs compared to others. The key difference between the EM model in this paper and traditional mixture distribution models lies in the fact that not all motion parameters are mixed. Instead, specific one-to-one relationship is established between the event eg and the motion neuron 2j. resulting in more precise correspondence.
Therefore, we can define the joint probability distribution as:
P(ek, 2k () P(zz 0) P(ex 24,, 0)
where the definition of P(ex =k, 0) relates the IWE value. Specifically, we can define the conditional probability event en belonging to motion pattern žj as:
P(ex 2). x 1,(ak, 10,)
where 1,(4,, 10,) represents the IWE value calculated based on the position and time event ex using the parameters 8, corresponding to motion pattern Zz-
A.3 Motion parameter initialization
our work, we were inspired by the layered method for event stream motion segmentation described in the EM-based approach and the SOFADS algorithm [s1]. The SOFADS method iteratively refines optical flow estimates through the Track Plane and Flow Plane modules. The Track Plane contains projections of different flow hypotheses, updated based incoming events. Similarly, adopted
13

---

SVD Result
KMeans Clustering of Optimized Paramet
180
16
Guater
140
Component 2
130
Parameter
100
80
200
Component 1
Parameter 1
Figure S8: Comparison of selection methods for parameter initialization (SVD vs. KMeans).
patch-based approach perform importance sampling the event stream to identify potential motion parameters based on optimization objective (contrast of IWE).
Our method involves a search process, with Fig. 4 illustrating this search method combined with SVD analysis. We select the Nı representative parameters with the highest variance as initial values. is essential to note that the parameter search process is crucial, and the use of SVD can be substituted with K-means for initial parameter selection. As shown Fig, S8, the parameter points obtained using K-means clustering are similar to those obtained with SVD.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We reflect the paper's contributions in the abstract and list them in the intro- duction. The claims include motivations and match our experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations at the end of the paper in Sec.6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We get theoretical results with careful deductions by formulas and proofs. The formulas are properly numbered and referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All the information for achieving the result specifically described in the methods and experiments of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will release the code and testing data after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. httppsccccbbliccceceeeeeececsece /tteggdeeeeooodedmiiiieeeeeeeee /CodeSuumiissteeeeee wwsisssoiiicyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We describe the implementation details and experimental settings for all the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
The experimental results of this paper are not related to the error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have detailed the computational resources used in our experiments within the relevant sections of the paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //tpr//eerppp.eeeee cc cc/ c/pppbicc/thiiicsuileeeeeee
Answer: [Yes]
Justification: The research conducted in the paper conforms, every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification:
The paper has no societal impacts. Our work shows Bayesian computation based on spiking neural networks can decouple event streams of different motions, which is foundational research.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite all the referenced assets and use them properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. con/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "oXc7ucunlE", "reviewer": "Reviewer_WNgv", "summary": "This paper demonstrates that WTA circuits along with STDP learning resembles EM algorithm-like Bayesian inference and could be used for motion segmentation from event streams by contrast maximization of warped events.", "questions": "Here are some general questions/comments about the framework:\n1. Full form of the abbreviation STDP missing in abstract/introduction.\n2. In Eq. (1), why is $\\Delta L(x_k,t_k) = q_k\\Theta$, since according to line 107, event $e_k$ corresponds to when the intensity change \\textit{exceeds} $\\Theta$ (noting that $|q_k|=1$). In line 109, add \"where $L(x,t)$ is the ... at pixel $x$ \\textit{at time $t$}\". \n3. Line 118: what integration is used? Eq. (2) only describes $I_j(x)$ as a mixture of Dirac measures. Add the definition of $N_e$ (possibly the number of observed events). In Eq. (2), does $x$ represent a pixel? What does the suffix $j$ capture? Based on the description, it suggests that it represents the different \\textit{motion models} --  it would be better to explain both the model and the optimization problem in slightly more detail for clarity. \n4. The EM framework: while the updates for the model resemble E and M steps in EM, is it actually related? Can you show that this method indeed improves some form of likelihood of the model (recall that EM is most commonly used for MLE in mixture models or other latent variable models)? Can the authors discuss how their method is related to EM (the E and M steps in the current paper are more closer to the hard clustering type algorithms e.g. K-Means rather than EM, particularly the E-step Eq. (5))\n5. In Eq. (6), extra $dx$ at the end, also might be better to keep the two terms in a parenthesis.\n6. More on the model Eq (2): according to line 119, $p_{kj}$ represents the probability that event $e_k$ belongs to motion model $z_j$, in that case $\\sum_j p_{kj}=1$, is that correct? However, in that case, Eq. (2) does not represent a mixture -- i.e., $\\sum_k p_{kj}$ might not be 1, can the authors clarify this? Furthermore, the Dirac function in Eq (2) allows the IWE to only pick up values at pixels $x$, where at least one event $e_k$ has been observed (through the transformed position). This does not allow any spatial relation across the pixels - why can the Dirac function not be replaced by some other smooth kernel (like Gaussian)?\n7. More on the optimization problem Eq (4): When writing $\\text{Var}(I_j)$, whose randomness are we taking the variance (or other expectation operations) with respect to? It seems like $\\theta, P$ are parameters (hence fixed) and $x_{kj}'$ is some deterministic transform of the observed events. Can the authors clarify the underlying probability structure of the model?\n8. The authors might provide a brief description of the STDP method (and its connections to spiking neural networks) and WTA circuit, which might clarify some of the paragraphs e.g., lines 170-173. It is also unclear why $u_j (t)$ (defined in Eq (10)) is equivalent to $I_j$ (in Eq (2)) as claimed in line 178 - is the $W_j$ in Eq (10) same as the WTA $W$ in Eq (3) -- if so, why is the second input $t_k$ in the latter while $p_{kj}$ in the former?\n9. Can the authors explain lines 205-206. It seems like they argue that gradient update increases the variance -- however, this is only the M-step (i.e., conditional on the current values of $p_{kj}$ I am guessing). \n10. Why is $u_j$ expressed as a function of time $t$ in Eq (10)? Can the authors clarify how the temporal dependence is captured in the model?", "limitations": "See the questions.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper proposes an interesting approach for event motion segmentation based on observations from event-based dynamic vision sensors, utilizing a EM-like framework for identifying various motion models from event streams and clustering them into motion patterns. This is achieved using WTA circuits together with STDP-based learning.", "weaknesses": "The main weakness of the paper is that the proposed method lacks proper justification of the presented approach, which seems like a heuristic hard clustering method, together with gradient based learning. The experiments also lack depth and the authors demonstrate the high dependence of the performance of the method on the parameter initialization. A more careful writing of the underlying model, the optimization framework and the proposed methodology would be good (see the questions below). Furthermore, the paper lacks more details regarding the choice of $N_{\\ell}$ (number of motion models) and the specific forms of the warping functions $W_j$ used. Several steps in the entire methodology, although intuitive, are presented in a heuristic fashion without detailed description and clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our **innovative** approach to event motion segmentation using event-based dynamic vision sensors and an EM-like framework. We appreciate your acknowledgment of our method's use of WTA circuits combined with STDP-based learning.\n\nThe following are the main issues addressed:\n\n### **A. Justification of the Approach**\n**1). Theoretical Background:**\nOur research aims to explore the application of Bayesian Inference for motion segmentation by decoupling events using a Spiking Neural Network (SNN)-based framework. This addresses the question, \"**How can an SNN model implementing Bayesian computation decouple and segment event streams generated by different motion patterns?**\" Building on previous studies [34] that successfully utilized EM-based methods for similar tasks, our work demonstrates both theoretically and experimentally that this method can be implemented using an SNN.\n\n**2). Clarification:**\nWe aim to provide a theoretical foundation for applying Bayesian inference using SNNs for event stream decoupling tasks, leveraging the energy efficiency of SNNs. This is particularly relevant for neuromorphic visual sensors and computing hardware (e.g., `Loihi [s1] and SpiNNaker [s2]`), known for their low latency and power consumption. Neuromorphic computing emulates the brain's low-power yet complex visual task-processing capabilities. Previous research validated the hypothesis that SNNs can implement brain-like Bayesian inference. However, its application to neuromorphic computing hardware remains unverified. Our work aims to validate the capability of a spike-based Bayesian computation framework applied to neuromorphic sensors.\n\n**References:**\n\n[s1] M. Davies et al., “Loihi: A neuromorphic manycore processor with on-chip learning,” IEEE Micro, vol. 38, no. 1, pp. 82–99, Jan. 2018.\n\n[s2] Furber, Steve B., et al. \"Overview of the SpiNNaker system architecture.\" IEEE transactions on computers 62.12 (2012): 2454-2467.\n\n### **B. Detailed Model and Methodology Comprehensive Writing**\nThe underlying model, optimization framework, and proposed methodology are detailed thoroughly. We have included pseudo-algorithm code `(please see in the supplementary PDF)` to provide step-by-step explanations, ensuring clarity. This will be added to the revised version for a more comprehensive description.\n\n**Examples and Case Studies:** We have included examples and case studies to illustrate each step of the methodology (Fig. 4-6 in the main text). These practical illustrations help clarify the application of the method and its effectiveness.\n\n### **C. Choice of Parameters**\n**1). Rationale for $𝑁_l$**:\nThe number of $𝑁_l$ can be set based on the experience with the datasets, making sense when trying to separate motion patterns contained in the events. A more intelligent approach can be used after obtaining different motion patterns of other patches. As shown in Fig. 4, clusters of different motions are identified using a non-parametric clustering approach for motion patterns of various patches.\n\n**2). Forms of $𝑊_𝑗$:**\nThe forms of $𝑊_𝑗$ depend on the selected motion patterns. A general method uses an affine transformation matrix, but we mainly consider optical flow changes as motion patterns, using $(vx, vy)$ as the motion parameters. Detailed descriptions and justifications for the specific forms of the warping functions (𝑊𝑗) used are provided, explaining their derivation and role in the overall model.\n\n### **D: Specific Clarifications:**\n**1). STDP, WTA, and Eq. (6):**\nThank you for the reminder! We will remove dx from equation (6) and add the full name of STDP in the abstract. Descriptions of the STDP method and WTA circuit will be included to clarify related paragraphs.\n\n**2). Line 109 Addition:**\nThe formula Δ𝐿(𝑥𝑘,𝑡𝑘)=𝑞𝑘Θ represents changes between consecutive events. Event streams are generated when a threshold is reached, followed by a reset. Thus, the log intensity change between event streams is 𝑞𝑘Θ.\n\n**3). IWE Clarification:**\nWe will clarify that IWE results from warping events to the same image plane, where x represents the pixel location. The term \"integration\" will be removed to avoid confusion. \"j\" represents the $j-th$ motion model.\n\n**4). EM Algorithm:**\nThe EM algorithm, a framework for finding parameters in probabilistic models with latent variables, is used in our approach to iteratively improve motion parameter estimates for event stream motion segmentation. Unlike the Kalman filter, which estimates the state of dynamic systems through prediction and update steps, our approach alternates between the E-step and M-step within the EM framework to achieve event stream motion segmentation.\n\n**5). Var(𝐼𝑗) Explanation:**\nThe variance (Var) of the IWE under different motion models shows `(Fig. S1 in the supplemented PDF)` that correct motion models concentrate event distributions at object edges, resulting in higher variance, while incorrect models disperse events, leading to lower variance. This helps in validating the accuracy of motion pattern detection.\n\n**6). Gradient Update Impact:**\nThe designed spiking model, combined with the STDP rule, updates motion neuron weights, effectively maximizing the variance of different IWEs, thereby validating the feasibility of the spike-based Bayesian inference framework for event stream motion segmentation both theoretically and experimentally.\n\n**7). Temporal Dependence:**\nThe value $u_j(t)$ represents the value of neuron $u_j$ at time $t$. Thank you for pointing it out. We will change this notation in future versions to avoid ambiguity. Our IF spiking neuron model mainly considers input event streams without simulating other temporal decay characteristics. \n\nThank you again for your valuable feedback. We hope our clarifications and additional details address the concerns raised and further demonstrate the robustness and potential of our proposed method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NG919XDC7y", "reviewer": "Reviewer_pCA1", "summary": "The paper proposes to address motion segmentation at very high temporal resolution via an event-based or spiking implementation of expectation-maximization in a generative model. It demonstrates the performance of the resulting spiking neural networks on example experiments.", "questions": "The authors have addressed my questions, though I would still like to see discussion of how this framework for EM in spiking networks could be generalized beyond motion detection.", "limitations": "I am not confident that I can identify the specific limitations of this paper as opposed to the limitations of spiking neural networks generally.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The strength of the paper is its deep engagement with the spiking neural network literature, as well as its use of spiking networks for the specific type of problem to which they are most suited: event-based computation.", "weaknesses": "The paper's major weakness is its lack of clarity, which the authors have discussed and addressed in the review period.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of our approach to high-resolution motion segmentation using an event-based implementation of the EM algorithm and acknowledge **our deep engagement with the spiking neural network literature**.\nHere, we aim to provide further clarification on the EM framework and our model's specific application.\n\n### **Clarification on the EM Algorithm Framework**:\n\nThe EM algorithm is a versatile framework used to find parameters in probabilistic models with latent variables. While traditionally, the M-step in the EM algorithm maximizes the likelihood function, it can also maximize other objective functions, such as the Evidence Lower Bound (ELBO) [s1] in variational inference or the Penalized Likelihood [s2] in sparse regression models like Lasso regression.\n\nIn previous work on event stream motion segmentation [34], the M-step focused on optimizing the contrast of different IWEs, effectively projecting the event distribution variance using motion parameters. Our current approach uses the EM framework to iteratively perform the E-step and the M-step to refine the estimates of motion parameters, achieving event stream motion segmentation. `The relationship between them is shown in Fig. 1 of the main text`. Therefore, our method does not involve the common concept of maximizing log densities in mixture distribution models.\n\n**References:**\n\n[1s] Blei, D. M., Kucukelbir, A., & McAuliffe, J. D. (2017). Variational Inference: A Review for Statisticians. Journal of the American Statistical Association, 112(518), 859-877.\n\n[2] Friedman, J., Hastie, T., & Tibshirani, R. (2010). Regularization Paths for Generalized Linear Models via Coordinate Descent. Journal of Statistical Software, 33(1), 1-22.\n\n### **Model's Nature and Application:**\n\nThank you for pointing out the potential confusion regarding our model's nature. *Our proposed approach is not a generative model. Instead, it is designed to learn motion parameters embedded in event streams generated by different motions.* These learned motion parameters help segment the event streams according to their respective motion distributions. The term \"motion distribution\" here refers to a projection plane where the variance is maximized if the events follow the motion distribution.\n\nBy implementing this EM-based motion segmentation using SNNs, we can leverage the low-latency and low-power characteristics of **neuromorphic hardware (NMHW)**, such as `Loihi and Spinnaker`. This combination enhances the performance of event-based cameras and neuromorphic computing systems. We recognize the importance of this aspect and will include more detailed discussions in future revisions of our paper.\n\nThank you again for your comments. We hope this response clarifies our approach and addresses your concerns comprehensively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C2bSq1Le6O", "reviewer": "Reviewer_wrk7", "summary": "This work proposes a spike Bayesian computational framework for continuous motion segmentation in event streams and demonstrates that the constructed network can implement an EM-based event stream motion segmentation model. The proposed model uses WTA circuits in the network to achieve an equivalent E-step, while the STDP rules for an M-step for contrast maximization. Experimental results demonstrate the network's online learning effectiveness for continuous inputs on extreme event camera datasets.", "questions": "This method primarily targets optical flow motion estimation. For more complex motion patterns, how to design the parameters? How robust is this method against noise in the evaluation of such motion models? The authors should clarify it.", "limitations": "The authors have stated the limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed network's effectiveness for motion segmentation has been validated on event datasets featuring challenging scenarios that involve mixed camera self-motion and high-speed moving objects. The proposed spike Bayesian inference framework is highly interpretable and applicable to various neuromorphic vision chips and computing hardware, representing a promising research direction.", "weaknesses": "The authors mainly use SVD to find different patches' motion patterns for initialization. Why is this method used, and can other methods be employed for selection? It is recommended that the authors conduct ablation experiments to explore further.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your acknowledgment of our validation of event datasets featuring challenging scenarios, including mixed camera self-motion and high-speed moving objects, which is highly valued. We are pleased that you find our spike Bayesian inference framework to be **highly interpretable** and **applicable to various neuromorphic vision chips and computing hardware**. Your appreciation of our work's potential as a **promising research direction** is encouraging.\n\n### **Detailed Response**\n\n**1. Motion Parameter Initialization:**\n\nIn our work, we were inspired by the layered method for event stream motion segmentation described in the EM-based approach and the SOFADS algorithm [s1]. The SOFADS method iteratively refines optical flow estimates through the Track Plane and Flow Plane modules. The Track Plane contains projections of different flow hypotheses, updated based on incoming events. Similarly, we adopted a patch-based approach to perform importance sampling on the event stream to identify potential motion parameters based on the optimization objective (contrast of IWE).\n\nOur method involves a search process detailed in lines 226-234 of our paper, with Fig. 4 illustrating this search method combined with SVD analysis. We select the $N_l$ representative parameters with the highest variance as initial values. It is essential to note that the parameter search process is crucial, and the use of SVD can be substituted with K-means for initial parameter selection. As shown in Fig. S2 `(please see in the supplementary PDF)`, the parameter points obtained using K-means clustering are similar to those obtained with SVD.\n\n**2. Types of Motion Neuron Parameters:**\n\nThe parameters of motion neurons in our framework can accommodate much more complex motion patterns by simply modifying the neuron model. For instance, our model can be extended to handle rotational motion [s2], 4-DOF motion [23], or even 8-DOF homographic motion [s3]. This flexibility underscores the versatility of our framework in adapting to various motion types.\n\n**3. Noise Handling:**\n\nRegarding noise, its randomness means it generally does not conform to any initialized motion model. During motion segmentation, noise tends to have similar probabilities across different categories, effectively filtering it out. This inherent filtering capability enhances the robustness of our segmentation method in real-world scenarios where noise is prevalent.\n\nThank you once again for your valuable feedback. We hope these clarifications address your concerns and further illustrate the robustness and potential of our proposed method for motion segmentation using event-based neural networks.\n\n**References:**\n\n[s1] Stoffregen, Timo, and Lindsay Kleeman. \"Simultaneous optical flow and segmentation (SOFAS) using dynamic vision sensor.\" arXiv preprint arXiv:1805.12326 (2018).\n\n[s2] Guillermo Gallego and Davide Scaramuzza, “Accurate angular velocity estimation with an event camera,” IEEE Robot. Autom. Lett., vol. 2, no. 2, pp. 632–639, 2017.\n\n[s3] Guillermo Gallego, Henri Rebecq, and Davide Scaramuzza, “A unifying contrast maximization framework for event cameras, with applications to motion, depth, and optical flow estimation,” in IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 3867–3876, 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N8Pd3yH7BG", "reviewer": "Reviewer_LG4A", "summary": "This paper presents a spike-based Bayesian inference framework for motion segmentation with event cameras. By designing neurons that utilize STDP for online learning of motion patterns, the framework can perform the M-step of the EM algorithm in motion segmentation of event streams. Additionally, the WTA circuit implements the E-step, allowing for the online partitioning of event streams into different motion patterns. The authors provide theoretical proof and experimental results to demonstrate the network's spatiotemporal decoupling capabilities for mixed motion patterns of event streams.", "questions": "Please see weaknesses.", "limitations": "There is a need for quantitative evaluations and an assessment of the dependency on parameter initialization.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors demonstrate that the SNN framework based on WTA is equivalent to the EM algorithm for motion segmentation of event streams. This online learning approach is compatible with neuromorphic data and beneficial for deployment on low-power, low-latency neuromorphic computing platforms.\n \n• The work is based on the Bayesian brain hypothesis, using a more physiologically interpretable SNN for Bayesian inference. Applying this to spatiotemporal data from neuromorphic cameras represents a promising research direction.", "weaknesses": "• The experimental results lack quantitative evaluations. Can the authors further perform object detection and tracking based on the motion segmentation, providing metrics such as object detection success rates and comparisons with other methods?\n \n• The proposed algorithm lacks the analysis of time complexity or processing speed. Can it leverage the low-latency advantage of event cameras?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive suggestions and have supplemented our work with further results on object detection based on motion segmentation. Specifically, we calculated the detection success rate on the EED dataset, corresponding to Fig. 6 in the main text. Our detection success rates across three test scenarios are `100%`, comparable to many current state-of-the-art algorithms ([s1], [s2]).\n\n**References:**\n\n[s1] Kepple, Daniel R., et al. \"Jointly learning visual motion and confidence from local patches in event cameras.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16. Springer International Publishing, 2020.\n\n[s2] Zheng, Yajing, et al. \"Spike-based motion estimation for object tracking through bio-inspired unsupervised learning.\" IEEE Transactions on Image Processing 32 (2022): 335-349.\n\n**2. Parameter Efficiency and Computational Resource Requirements:**\n\nOur method requires minimal parameters and computational resources. Specifically, the model uses neurons corresponding to different motion models and a single global inhibitory neuron to perform WTA. This parameter efficiency and low computational requirement make our approach particularly advantageous for hardware or neuromorphic hardware deployment. It can fully leverage the low latency and low power consumption characteristics of neuromorphic cameras and computing platforms.\n\nThank you once again for your invaluable feedback. We hope these additions and clarifications address your concerns and further demonstrate the robustness and potential of our proposed method for event-based motion segmentation and object detection using spiking neural networks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the valuable feedback from all reviewers. Thank you for recognizing the strengths of our work. Reviewer **WNgv** praised our innovative approach to event motion segmentation using event-based dynamic vision sensors and an EM-like framework, highlighting the use of Winner-Take-All (WTA) circuits combined with Spike-Timing-Dependent Plasticity (STDP)-based learning. Reviewer **pCA1** appreciated our high-resolution motion segmentation using an event-based implementation of the EM algorithm and noted our engagement with the spiking neural network literature. Reviewer **wrk7** recognized the effectiveness of our network for motion segmentation validated on challenging event datasets, including mixed camera self-motion and high-speed moving objects, and commended our spike Bayesian inference framework's interpretability and applicability to neuromorphic vision chips and computing hardware. Lastly, Reviewer **LG4A** endorsed our SNN framework based on WTA as equivalent to the EM algorithm for motion segmentation and supported our application of the Bayesian brain hypothesis using a physiologically interpretable SNN for Bayesian inference on spatiotemporal data from neuromorphic cameras as a promising research direction.

In response to the questions raised, we provide the following clarifications and updates, and have provided some materials and results figures in the uploaded supplementary PDF.

### **A. Theoretical Foundation:**
Our research explores Bayesian Inference for motion segmentation by decoupling events using an SNN-based framework, addressing how SNNs implementing Bayesian computation can decouple and segment event streams generated by different motion patterns. Inspired by previous EM-based methods, we demonstrate the theoretical and experimental viability of this approach with SNNs. We aim to provide a theoretical foundation for applying Bayesian inference using SNNs for event stream decoupling tasks, leveraging their energy efficiency, which is particularly relevant for neuromorphic visual sensors and computing hardware like Loihi and SpiNNaker.

### **B. Model and Methodology:**
Our model, optimization framework, and methodology are thoroughly detailed. We will include pseudo-algorithm code for step-by-step explanations in the revised version for clarity. Examples and case studies illustrate each step of the methodology. We discussed the rationale for setting the number of motion patterns (𝑁ℓ) and provided justifications for the forms of warping functions (𝑊𝑗), explaining their derivation and role. Additionally, we will add the full name of STDP in the abstract and include descriptions of the STDP method and WTA circuit to clarify related paragraphs.

### **C. EM Algorithm Framework:**
In previous work on event stream motion segmentation [34], the M-step focused on optimizing the contrast of different IWEs, effectively projecting the event distribution variance using motion parameters. Our current approach uses the EM framework to iteratively perform the E-step and the M-step to refine the estimates of motion parameters, achieving event stream motion segmentation. Thus, our method does not involve the common concept of maximizing log densities in mixture distribution models.

### **D. Technical Clarifications:**
We clarified various technical details, such as the formula Δ𝐿(𝑥𝑘,𝑡𝑘)=𝑞𝑘Θ, the IWE concept, the use of the EM algorithm, variance explanations for motion models, and gradient update impacts. Our proposed model is not a generative model but is designed to learn motion parameters embedded in event streams generated by different motions, helping segment the event streams according to their respective motion distributions. Implementing this EM-based motion segmentation using SNNs leverages the low-latency and low-power characteristics of neuromorphic hardware, enhancing the performance of event-based cameras and neuromorphic computing systems.

### **E. Supplementary Results:**
We have supplemented our work with further results on object detection based on motion segmentation. Specifically, we calculated the detection success rate on the EED dataset, corresponding to Fig. 6 in the main text. Our detection success rates across three test scenarios are `100%`, comparable to many current state-of-the-art algorithms.

### **F. Parameter Efficiency and Resource Requirements:**
Our method is highly efficient, requiring minimal parameters and computational resources. The model uses neurons corresponding to different motion models and a single global inhibitory neuron to perform WTA. This parameter efficiency and low computational requirement make our approach particularly advantageous for deployment on hardware or neuromorphic hardware, leveraging the low latency and low power consumption characteristics of neuromorphic cameras and computing platforms.

We hope these additions and clarifications address your concerns and further demonstrate the robustness and potential of our proposed method for event-based motion segmentation and object detection using spiking neural networks.

Meta-review: This paper describes a variety of novel applications of Bayesian computation in spiking neural neural computations.  All four reviewers felt that it was above the threshold for acceptance, and I'm pleased to report that it has been accepted to NeurIPS.  Congratulations!  Please revise the manuscript according to the reviewer comments and discussion points, particularly the critical comments of reviewers WNgv and pCA1.
Decision: Accept (poster)