Continuous Spatiotemporal Events Decoupling through Spike-based Bayesian Computation

Authors: yajing  zheng
jiyuan  zhang
zhaofei  yu
tiejun  huang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 12:20:09

Abstract: Numerous studies have demonstrated that the cognitive processes of the human brain can be modeled using the Bayesian theorem for probabilistic inference of the external world. Spiking neural networks (SNNs), capable of performing Bayesian computation with greater physiological interpretability, offer a novel approach to distributed information processing in the cortex. However, applying these models to real-world scenarios to harness the advantages of brain-like computation remains a challenge. 
Recently, bio-inspired sensors with high dynamic range and ultra-high temporal resolution have been widely used in extreme vision scenarios. Event streams, generated by various types of motion, represent spatiotemporal data. Inferring motion targets from these streams without prior knowledge remains a difficult task. The Bayesian inference-based Expectation-Maximization (EM) framework has proven effective for motion segmentation in event streams, allowing for decoupling without prior information about the motion or its source. 
This work demonstrates that Bayesian computation based on spiking neural networks can decouple event streams of different motions. The Winner-Take-All (WTA) circuits in the constructed network implement an equivalent E-step, while STDP achieves an equivalent optimization in M-step. Through theoretical analysis and experiments, we show that STDP-based learning can maximize the contrast of warped events under mixed motion models. Experimental results show that the constructed spiking network can effectively segment the motion contained in event streams.

Main paper: 
 introduction: 
 Bayesian computation is a fundamental concept in statistics, machine learning, and computational neuroscience [35, 3] . The Bayesian brain hypothesis suggests that the brain functions as a probabilistic generative model, simultaneously inferring hidden causes of sensory inputs and refining its model parameters [22, 8, 21] . Bayesian computation requires a generative model to predict observations, formulated as the joint probability P (x, y) of observations x and hidden states y. This joint probability can be decomposed into the prior P (y) and the likelihood P (x|y), which are combined using the Bayes theorem to update the before a posterior probability P (y|x). The brain's inferential processes use probabilistic models to represent and update perceptions based on new sensory input, framing perception as an active test of predictions against sensory data [11, 34] . validated the hypothesis that SNNs can implement Bayesian inference [29] . However, its application to neuromorphic computing hardware remains unverified. Our work aims to validate the capability of a spike-based Bayesian computation framework applied to neuromorphic sensors.• Develop a spiking Bayesian computation framework for continuous motion segmentation in event streams, demonstrating its ability to perform similarly to previous EM algorithm-based models. • Show that the WTA circuit can implement the E-step and that STDP rules can achieve the M-step for contrast maximization, validating these components' theoretical and practical efficacy. • Verify the proposed network can effectively learn online from continuous input, enabling accurate motion segmentation through the firing of output neurons.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We reflect the paper's contributions in the abstract and list them in the introduction. The claims include motivations and match our experimental results. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related works: 
 Spike-based Bayesian Computation. Spike-based Bayesian computation employs various methods to achieve probabilistic inference and optimization in neural circuits, such as maximum likelihood or posterior function. For example, SNNs implement Bayesian inference using belief propagation on binary MRFs and tree-based reparameterization for exponential family distributions [31, 40, 41, 42] , which approximate the posterior probability. Spiking neuron-based neural sampling [2, 40] uses a non-reversible process, like Markov chain Monte Carlo (MCMC) sampling. A particularly suitable method for complex tasks like motion segmentation in event streams involves using WTA circuits combined with STDP [29, 28] learning rules to implement the EM algorithm for decoupling mixture distribution. This approach leverages the competitive dynamics of WTA circuits to estimate joint probability distributions, offering high accuracy and robustness in learning and inference. The WTA-STDP framework is well-suited for handling the asynchronous and event-driven nature of spiking data, making it effective for segmenting complex motion patterns in event streams.Event-based motion segmentation. Recent methods [26, 16, 39, 33, 27, 38] in event-based motion segmentation leverage clustering, probabilistic models, motion compensation, and deep learning.Cluster-based techniques group events by similar motion patterns and are simple to implement, but may struggle with complex scenes. Probabilistic approaches, such as those using Bayesian frameworks and the EM algorithm [38, 44] , iteratively estimate motion parameters and provide robust segmentation, though they can be computationally intensive. Motion compensation aligns events to a reference frame to enhance sharpness (contrast) of warped events [38] , offering high accuracy, but may be sensitive to parameter initialization. Deep learning methods employ neural networks to predict motion segments directly from raw event streams, providing high performance and adaptability [19, 37, 27] , but requiring large datasets for training. Some works [32, 43] also attempt to use SNN for estimating motion in event streams, thereby achieving motion segmentation or object detection. The main idea of these methods is to first convert the event stream into timeevent-like inputs, rather than directly distinguishing the event stream in the spatiotemporal dimension. Consequently, these methods exhibit a certain lag and can easily be disturbed by camera self-motion, which interferes with the analysis of moving objects.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We have discussed the limitations at the end of the paper in Sec.6. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 3.1 Event Cameras.Dynamic Vision Sensors (DVS) [23] in event cameras detect brightness changes independently at each pixel, generating events instead of capturing images at fixed intervals. An event e k = (x k , t k , q k ) occurs when the intensity change ∆L(x k , t k ) at pixel x k exceeds a threshold Θ. The change in intensity is expressed as:EQUATIONwhere L(x, t) is the logarithmic intensity at pixel x, t k is the event timestamp, ∆t k is the interval since the last event at the same pixel, and q k ∈ {-1, +1} indicates the polarity of the intensity change. Since our work does not use event polarity, this property will not be referenced further. This asynchronous mechanism enables event-based cameras to capture dynamic scenes with high temporal resolution and low latency, making them ideal for real-time applications.In previous event-based motion segmentation algorithms, the probability that an event stream e belongs to different motion models z is denoted as P . The motion model that generates the event is then used to warp the event positions, resulting in a sharp Image of Warped Events (IWE). The IWE I j (x) for a given motion parameter θ j is typically calculated by warping events onto a specific time plane, where x represents the pixel location, and j refers to the j-th motion model:EQUATIONwhere δ(•) denotes the Dirac function, and p kj represents the probability that event e k belongs to motion model z j . The Dirac function can also be replaced by a smoother kernel function, such as the Gaussian function. x ′ kj is the transformed event position obtained using motion parameters θ j :x ′ kj = W (x k , t k ; θ j ).(3) The objective of the entire model is to find the motion parameters θ * and the event-cluster probability P * that maximize the variance of all IWEs corresponding to different motions:EQUATIONIn EM fashion, we first perform the E-step to calculate the posterior probabilities of the samples belonging to the distribution based on the assumed model and parameters. The initial motion parameters are used to calculate the IWE for different motions. The responsibility p kj for each event belonging to a motion model is then computed using the formula:EQUATIONNext, in the M-step, these responsibility values are used to optimize the motion parameters, maximizing the objective function f (θ). In this work, we use a different variance form to [38] , but the effect is the same. The objective function is computed as:EQUATIONwhere E[x] represents the expectation process as:EQUATIONEQUATIONΩ denotes the image plane. The gradient of the variance Var(I j ) is given by:EQUATIONThis method effectively distinguishes event streams generated by different motions and produces a sharp IWE. Other optimization strategies, such as the conjugate gradient method [30] , can be applied to update the motion parameters θ j .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We will release the code and testing data after the paper is accepted.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
spike-based bayesian computation for motion segmentation: 
 The main procedure for event-based motion segmentation involves three key steps: 1) obtaining the IWE, 2) assigning probabilities for events corresponding to different motion models, and 3) optimizing the motion parameters to maximize the contrast of the IWE. In our model, we use a network with a WTA circuit to compute the IWE and the probabilities P , and we update the motion parameters using the STDP rule. Below, we will demonstrate that our constructed model and learning method approximates the event-based motion segmentation approach. E-step. The first step of event-based motion segmentation involves obtaining the IWE for different motion parameters and then determining the event-cluster responsibility values p kj by comparing the contrast of the IWE at the corresponding positions after event warping. This operation is analogous to a neuron competition, where the motion model that produces the highest contrast gains ownership of the events mapped to that location. This process can be implemented using a WTA circuit, which is how our proposed network updates the responsibility values for different events.As shown in Fig. 2 , in our network, neurons y representing different motions j are responsible for warping events and transmitting them to the output neurons z. There is a one-to-one correspondence between y and z. The output neuron z j follows an integrate-and-fire (IF) model [15] , with the membrane potential u j expressed as:EQUATIONwhere W j represents the operation of the motion neuron y j . It is evident that by accumulating the warped event streams, equivalent to obtaining the IWE in Eq. 2.The output neuron z j receives feedback from a global inhibition neuron H. The value of the global inhibition neuron is the sum of all IWE values, i.e., H(t) = j u j (t). Here, we use a stochastic firing model for z j , where the firing probability depends on the membrane potential in conjunction with the inhibition neuron H. This computation can be formulated as:EQUATIONUnlike the WTA circuits constructed by Nessel et al. [29, 28] , our output neurons form a tensor corresponding to the event space dimensions. The synaptic parameters updated by STDP do not represent the connection weights between scalar neurons but rather the coefficients of functions representing the receptive fields of different motion neurons (e.g., functions that linearly transform along the direction of motion). The concept of neurons representing tensor values has also been applied in various works, such as in capsule networks [36] .M-step. After obtaining the network's output, we can update the network parameters using the STDP rule based on the relationship between the firing of z k and the input events. In our previous definitions, the network parameters include only the motion parameters θ j . The probability p kj is derived from the firing probability of the output neurons and is also used as the input for the next step in the network training process.According to the STDP rule, if the firing of the presynaptic neuron leads to the firing of the postsynaptic neuron, their synaptic weight is increased, corresponding to long-term potentiation (LTP). Otherwise, the synaptic weight is decreased, corresponding to long-term depression (LTD). The synaptic weight is updated as follows: In this context, the probability of firing between the presynaptic event e and the postsynaptic neuron z is positively correlated with their association probability p kj . Consequently, the update direction of the motion parameter θ j is also positively correlated with p kj . Combining this with the concept of motion compensation, the goal of updating θ j is to maximize contrast Var(I j ), so the gradient direction is related to the contrast of u j (t) (where u j (t) is equivalent to IWE I j ). Therefore, in our proposed network, the gradient update for the motion parameter θ j is given by:EQUATIONEQUATIONFig. 3 (a) shows the learning curve of STDP. We do not explicitly show the use of LTD for updates because the presence of p kj in the update formula ensures that if the event stream does not belong to the motion model, it will not significantly affect the update of its motion parameters. This approach is reasonable, as a low p kj value indicates that the current input event stream does not maximize the contrast for the motion model. Hence, it should not influence the learning of the motion neuron parameters, thereby preventing interference with the detection of other motion patterns in the event stream.Our objective is to maximize the sum of the variances of the IWE across different motion parameter distributions (Eq. ( 4)). As described in Fig. S7 of the Appendix, the variance indicates that correct motion patterns concentrate the event flow distribution along the object's edges, resulting in higher variance. In contrast, incorrect motion patterns disperse the event flow, leading to lower variance. In the network we designed, which includes a WTA mechanism, the parameters of the motion neurons, denoted as θ, are optimized using the STDP rule to achieve this goal. As shown in Eq. ( 12) and Fig. 3 , when P (z j = 1|e k = 1), only event streams that match motion pattern j will activate the corresponding motion neuron to adjust its parameters, thereby maximizing the variance of IWE (encoded by u) associated with motion pattern j.In our network, applying STDP to optimize the parameters of motion neurons does not strictly follow the gradient of the objective function for each parameter. However, the angle between the update direction and the gradient of the objective function is less than 90 degrees. Previous work has demonstrated that synaptic updates need only maintain an angle less than 90 degrees with the error function to achieve optimization [24] . We will also show that under this update rule, the contrast of the IWE can gradually increase.In the M-step of event-based motion segmentation, the goal is to optimize the parameter values by maximizing the contrast of the IWE. The objective is to increase the contrast of all IWE values during the optimization process, and various optimization strategies can be employed. Our proposed STDP rule effectively increases the value of Var(u j ). The variance formula is expanded as follows:EQUATIONwhere N = |Ω|. Assume that after the STDP update, the parameter becomes θ ′ j = θ j + ∆θ j , resulting in a new output u ′ j with variance Var(u ′ j ). The updated variance formula:EQUATION)The updated output is :EQUATIONwhere ∆u j (k) is the change induced by ∆θ j . The change in variance is calculated as:EQUATION)Since ∆u j (k) is a small change relative to θ j , we can use a first-order approximation:EQUATION)The parameter update increases the variance. According to the STDP update rule, the direction of ∆θ j is consistent with ∂Var(u j )/∂θ j . Therefore, the variance will increase after the update. The results show that under the STDP learning rule, influenced by the WTA circuit, the input events e most relevant to the motion are used for optimization. This allows θ 1 to progress in a direction that maximizes the contrast of the corresponding event stream. Additionally, this process does not interfere with the learning of other motion patterns θ 0 . The WTA circuit combined with the STDP rule ensures that the contrast of event streams not belonging to the motion pattern is not forcefully maximized. 
experiments: 
 5.1 Implementation Details.Parameter Initialization. The proposed spike-based Bayesian Computation framework and its corresponding event-based clustering framework are both locally convergent algorithms. The STDP rule adjusts the weights based on the firing patterns of pre-and post-synaptic neurons, and it does not guarantee convergence to a global solution. To ensure both algorithms converge to a better solution, we adopt a strategy similar to the event-based layered algorithm for initializing θ and P. The number of motion models N ℓ and the type of motion models (e.g., linear motion, affine, or rotation) are hyperparameters that need to be predefined. Considering the ultra-high temporal resolution advantage of event cameras, we primarily set the motion model to linear motion, i.e., θ = {v x , v y }. After setting the number of motion models N ℓ and the form of θ, we take a subset of events for parameter initialization.Specifically, we divide the events into different patches and use the specified motion parameters to maximize the contrast of the IWE corresponding to these patch events. To search for parameters more efficiently instead of using brute force methods (e.g., grid search), we employ a combination of random sampling and Bayesian optimization using the Tree-structured Parzen Estimator (TPE) [10] .After completing the parameter search, we obtain a parameter set corresponding to the number of patches N np (N np > N ℓ ). Since different patches may have similar motion parameters, we use Singular Value Decomposition (SVD) [17] to analyze the components of the returned parameter set and select the top N ℓ parameters with the most significant differences as the initialization parameters for the algorithm.Fig. 4 shows the parameter initialization process in a sequence from the Extreme Event Dataset (EED) [26] , which includes both camera self-motion and a moving object. The initial parameter search identifies two significantly different motion parameters corresponding to the event streams from the 1st-and 11th-patches, respectively.Network Learning and Inference. After initializing the parameters, we select a fixed number of events in chronological order, dividing all events into different packets {e n } Ng n=1 as inputs to the network over time. During online learning, we also split the n-th events packet {e n } into different patches and feed them into the network. After optimizing the parameters θ for several epochs, we obtain the optimized motion parameters θ * , and then input all events into the network to get the responsibilities P of all events belonging to different motion parameters. The motions estimated by clustering {e n } can be propagated in time to predict an initialization for the clusters of the next packet {e n+1 }. All steps of the proposed method are summarized in Algorithm. 1.Input: Events packet {e n k } Ne k=1 , number of clusters N ℓ . Output: Cluster parameters θ = {θ j } N ℓ j=1 , event-cluster assignments P. 1: procedure 2:Initialize the unknowns (θ, P) by sampling potential motion patterns of different patches of events based on the TPE [10] and SVD [17] .while not converged do 4: E-step Compute the event-cluster assignments p kj based on the WTA circuit using Eq. 10 and Eq. 11.M-step Update the motion parameters of all clusters using Eq. 13 based on the STDP rule. Split events into patches and feed into the network.Optimize parameters θ for several epochs to obtain optimized motion parameters θ * .Feed all events into the network to obtain event responsibilities P. 6: end for Our model is event-driven and operates with parallel computations across different patches. The primary focus is on CPU-based verification due to minimal graphical operations, ensuring fast processing as an online learning algorithm. Additionally, the speed on both GPU and CPU is comparable. Our network structure is compact, resulting in low memory consumption.Fig. 5 shows the optimized motion parameters θ * and the spike firing rates p kj of the output neurons z for the scene described in Fig. 4 . The figure also presents the IWEs and the events warped and fused according to the corresponding motion parameters. After training with the proposed network, the IWEs for different motion parameters show higher contrast compared to before learning. This allows for effective separation of the camera's self-motion from the motion of the ball. The Spike-inference Events in Fig. 5 are accumulated based on the spike activity of different neurons represented by colors. To verify that the proposed spiking neural network can learn the parameters of motion neurons online from continuous input, we continuously feed the event stream into the network and observe the firing state of the output neurons z. Fig. 6 shows the network's performance in segmenting three scenarios in the EED that involve mixed camera self-motion and high-speed moving objects. In these scenes, the background event streams vary in density and shape. Despite these varying backgrounds, the output neurons can still distinguish the motion parameters of moving objects. This demonstrates that the proposed network can learn to suppress irregular input spike patterns and, through local plasticity learning, identify the motion parameters that maximize contrast in specific regions, thereby accurately locating different moving objects. 
conclusions & discussions: 
 This paper proposes a spike Bayesian computation framework for continuous motion segmentation in event streams. We demonstrate that the constructed network can achieve the same effect as previous event-based motion segmentation models using the Expectation-Maximization (EM) algorithm. Specifically, the WTA circuit in the network implements the equivalent E-step, and the STDP rule for adjusting network parameters realizes the equivalent M-step for contrast maximization. Both theoretical analysis and experimental results show that STDP-based learning can optimize the contrast of mapped images under a mixture motion parameter model. Using the Extreme Event Dataset, we validate the network's ability to learn online from continuous input and perform motion segmentation through the firing of output neurons. Limitations. This work primarily aims to demonstrate that a biologically inspired network framework implicitly does Bayesian computation can decouple spatiotemporal data, proposing a prototype framework applicable to neuromorphic cameras. The validation focuses on motion segmentation in event streams rather than pursuing state-of-the-art performance. Therefore, there is limited quantitative performance evaluation and comparison with existing event-based motion segmentation algorithms. This limitation arises because the network relies on local learning rules and may not converge to the optimal solution, being highly dependent on parameter initialization.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We describe the implementation details and experimental settings for all the results.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The experimental results of this paper are not related to the error bars.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We have detailed the computational resources used in our experiments within the relevant sections of the paper.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: The paper has no societal impacts. Our work shows Bayesian computation based on spiking neural networks can decouple event streams of different motions, which is foundational research.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 
licenses for existing assets: 
 Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We cite all the referenced assets and use them properly.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Explanation of V ar(I j )As shown in Fig. S7 , correct motion models concentrate event distributions at object edges, resulting in higher variance, while incorrect models disperse events, leading to lower variance. This helps in validating the accuracy of motion pattern detection. From the perspective of a probabilistic model, we can consider that events conforming to different motion patterns emerge over time. However, in the task of dividing event streams based on motion patterns, it is hard to directly generate a model (e.g., a mixture distribution model) using a generative model. In previous work [34] , the authors adopted a method based on Eq. ( 2), which is more suitable for traditional mixture distribution models (e.g., fuzzy mixture models and k-means clustering) to divide event streams by motion.In fuzzy mixture density and k-means methods, the motion-compensated IWEs do not include the event cluster associations P , which means that sharper object boundaries appear in some IWEs compared to others. The key difference between the EM model in this paper and traditional mixture distribution models lies in the fact that not all motion parameters are mixed. Instead, a specific one-to-one relationship is established between the event e k and the motion neuron z j , resulting in a more precise correspondence. Therefore, we can define the joint probability distribution as:P (e k , z k | θ) = P (z k | θ) • P (e k | z k , θ)where the definition of P (e k | z k , θ) relates to the IWE value. Specifically, we can define the conditional probability of event e k belonging to motion pattern z j as:P (e k | z j , θ) ∝ I j (x ′ kj | θ j )where I j (x ′ kj | θ j ) represents the IWE value calculated based on the position and time of event e k using the parameters θ j corresponding to motion pattern z j .In our work, we were inspired by the layered method for event stream motion segmentation described in the EM-based approach and the SOFADS algorithm [s1]. The SOFADS method iteratively refines optical flow estimates through the Track Plane and Flow Plane modules. The Track Plane contains projections of different flow hypotheses, updated based on incoming events. Similarly, we adopted a patch-based approach to perform importance sampling on the event stream to identify potential motion parameters based on the optimization objective (contrast of IWE).Our method involves a search process, with Fig. 4 illustrating this search method combined with SVD analysis. We select the N l representative parameters with the highest variance as initial values. It is essential to note that the parameter search process is crucial, and the use of SVD can be substituted with K-means for initial parameter selection. As shown in Fig. S8 , the parameter points obtained using K-means clustering are similar to those obtained with SVD. 


Decision: This paper describes a variety of novel applications of Bayesian computation in spiking neural neural computations.  All four reviewers felt that it was above the threshold for acceptance, and I'm pleased to report that it has been accepted to NeurIPS.  Congratulations!  Please revise the manuscript according to the reviewer comments and discussion points, particularly the critical comments of reviewers WNgv and pCA1.

Reviews: review: paper_summary: paper_summary: This paper demonstrates that WTA circuits along with STDP learning resembles EM algorithm-like Bayesian inference and could be used for motion segmentation from event streams by contrast maximization of warped events.
strength_weakness: strength_weakness: The main weakness of the paper is that the proposed method lacks proper justification of the presented approach, which seems like a heuristic hard clustering method, together with gradient based learning. The experiments also lack depth and the authors demonstrate the high dependence of the performance of the method on the parameter initialization. A more careful writing of the underlying model, the optimization framework and the proposed methodology would be good (see the questions below). Furthermore, the paper lacks more details regarding the choice of $N_{\ell}$ (number of motion models) and the specific forms of the warping functions $W_j$ used. Several steps in the entire methodology, although intuitive, are presented in a heuristic fashion without detailed description and clarity.
questions: questions: Here are some general questions/comments about the framework:
1. Full form of the abbreviation STDP missing in abstract/introduction.
2. In Eq. (1), why is $\Delta L(x_k,t_k) = q_k\Theta$, since according to line 107, event $e_k$ corresponds to when the intensity change \textit{exceeds} $\Theta$ (noting that $|q_k|=1$). In line 109, add "where $L(x,t)$ is the ... at pixel $x$ \textit{at time $t$}". 
3. Line 118: what integration is used? Eq. (2) only describes $I_j(x)$ as a mixture of Dirac measures. Add the definition of $N_e$ (possibly the number of observed events). In Eq. (2), does $x$ represent a pixel? What does the suffix $j$ capture? Based on the description, it suggests that it represents the different \textit{motion models} --  it would be better to explain both the model and the optimization problem in slightly more detail for clarity. 
4. The EM framework: while the updates for the model resemble E and M steps in EM, is it actually related? Can you show that this method indeed improves some form of likelihood of the model (recall that EM is most commonly used for MLE in mixture models or other latent variable models)? Can the authors discuss how their method is related to EM (the E and M steps in the current paper are more closer to the hard clustering type algorithms e.g. K-Means rather than EM, particularly the E-step Eq. (5))
5. In Eq. (6), extra $dx$ at the end, also might be better to keep the two terms in a parenthesis.
6. More on the model Eq (2): according to line 119, $p_{kj}$ represents the probability that event $e_k$ belongs to motion model $z_j$, in that case $\sum_j p_{kj}=1$, is that correct? However, in that case, Eq. (2) does not represent a mixture -- i.e., $\sum_k p_{kj}$ might not be 1, can the authors clarify this? Furthermore, the Dirac function in Eq (2) allows the IWE to only pick up values at pixels $x$, where at least one event $e_k$ has been observed (through the transformed position). This does not allow any spatial relation across the pixels - why can the Dirac function not be replaced by some other smooth kernel (like Gaussian)?
7. More on the optimization problem Eq (4): When writing $\text{Var}(I_j)$, whose randomness are we taking the variance (or other expectation operations) with respect to? It seems like $\theta, P$ are parameters (hence fixed) and $x_{kj}'$ is some deterministic transform of the observed events. Can the authors clarify the underlying probability structure of the model?
8. The authors might provide a brief description of the STDP method (and its connections to spiking neural networks) and WTA circuit, which might clarify some of the paragraphs e.g., lines 170-173. It is also unclear why $u_j (t)$ (defined in Eq (10)) is equivalent to $I_j$ (in Eq (2)) as claimed in line 178 - is the $W_j$ in Eq (10) same as the WTA $W$ in Eq (3) -- if so, why is the second input $t_k$ in the latter while $p_{kj}$ in the former?
9. Can the authors explain lines 205-206. It seems like they argue that gradient update increases the variance -- however, this is only the M-step (i.e., conditional on the current values of $p_{kj}$ I am guessing). 
10. Why is $u_j$ expressed as a function of time $t$ in Eq (10)? Can the authors clarify how the temporal dependence is captured in the model?
limitations: limitations: See the questions.
paper_summary: paper_summary: The paper proposes to address motion segmentation at very high temporal resolution via an event-based or spiking implementation of expectation-maximization in a generative model. It demonstrates the performance of the resulting spiking neural networks on example experiments.
strength_weakness: strength_weakness: The paper's major weakness is its lack of clarity, which the authors have discussed and addressed in the review period.
questions: questions: The authors have addressed my questions, though I would still like to see discussion of how this framework for EM in spiking networks could be generalized beyond motion detection.
limitations: limitations: I am not confident that I can identify the specific limitations of this paper as opposed to the limitations of spiking neural networks generally.
paper_summary: paper_summary: This work proposes a spike Bayesian computational framework for continuous motion segmentation in event streams and demonstrates that the constructed network can implement an EM-based event stream motion segmentation model. The proposed model uses WTA circuits in the network to achieve an equivalent E-step, while the STDP rules for an M-step for contrast maximization. Experimental results demonstrate the network's online learning effectiveness for continuous inputs on extreme event camera datasets.
strength_weakness: strength_weakness: The authors mainly use SVD to find different patches' motion patterns for initialization. Why is this method used, and can other methods be employed for selection? It is recommended that the authors conduct ablation experiments to explore further.
questions: questions: This method primarily targets optical flow motion estimation. For more complex motion patterns, how to design the parameters? How robust is this method against noise in the evaluation of such motion models? The authors should clarify it.
limitations: limitations: The authors have stated the limitations.
paper_summary: paper_summary: This paper presents a spike-based Bayesian inference framework for motion segmentation with event cameras. By designing neurons that utilize STDP for online learning of motion patterns, the framework can perform the M-step of the EM algorithm in motion segmentation of event streams. Additionally, the WTA circuit implements the E-step, allowing for the online partitioning of event streams into different motion patterns. The authors provide theoretical proof and experimental results to demonstrate the network's spatiotemporal decoupling capabilities for mixed motion patterns of event streams.
strength_weakness: strength_weakness: • The experimental results lack quantitative evaluations. Can the authors further perform object detection and tracking based on the motion segmentation, providing metrics such as object detection success rates and comparisons with other methods?
 
• The proposed algorithm lacks the analysis of time complexity or processing speed. Can it leverage the low-latency advantage of event cameras?
questions: questions: Please see weaknesses.
limitations: limitations: There is a need for quantitative evaluations and an assessment of the dependency on parameter initialization.

Comments: title: Comments
Thank you very much!
I must be misunderstanding something.
comment: Thank you very much for your feedback and for increasing the score. We appreciate your observation regarding the similarities and differences between our framework and EM.

To clarify, our method does not implement a hard assignment of events to specific motion patterns. Instead, the probability of an event belonging to a particular motion pattern is proportional to its IWE value (or encoded as membrane potential $u$ in the proposed model) under different motion patterns, as outlined in Eq. (5) and Eq. (11) of the main text. Specifically, the conditional probability $P(e_k \mid z_j, \theta) \propto I_j(x_{k_j}' \mid \theta_j)$, where $I_j(x_{k_j}' \mid \theta_j)$ represents the IWE value associated with the motion pattern $z_j$. This approach allows for a more nuanced assignment that considers the probabilistic contributions from multiple motion patterns rather than a strict, binary assignment.

In the revised version of the paper, we will explicitly describe the conditional likelihood during the Expectation step to better articulate the similarities with EM, ensuring that these differences are clear and avoiding any further ambiguity.

Thank you again for your valuable suggestions, which will help us improve the clarity and accuracy of our work!
We're deeply grateful for your understanding and your willingness to revisit your review. Your support and consideration mean a lot to us!
Thank you for your valuable feedback! We're glad our responses addressed your concerns, and we appreciate your consideration of a higher rating. We will continue to refine our work based on the reviewers' suggestions.
Thank you for the response. Some of the ambiguity have been dealt with and I increase my score to 5. However, although the overall framework is *similar* to EM, it is not clear why it *actually is*. To be precise, it would be better to explicitly write the conditional likelihood to describe the Expectation step. As I mentioned, it seems that it is closer to a hard-clustering algorithm rather than an EM (they are similar but fundamentally different).
The authors responded to my earlier questions about the quantitative comparison of their proposed method with SOTA and addressed my concerns regarding delay and time complexity. They provided detection success rates (100%) for the three extreme scenarios in Fig. 6, which are comparable to SOTA methods, but with significantly lower algorithm complexity. After reviewing the authors' replies to all the reviewers, I'm convinced that incorporating the reviewers' feedback, such as suggestions on WNgv descriptions, will greatly enhance the potential of the spike-based Bayesian Computation framework presented in this paper. This framework has enormous potential for low-latency, low-power applications in linking neuromorphic sensors and NMHW. Therefore, I am considering giving it a higher rating.
Thank you for including this explanation in the revised paper.  With that I think I can revisit my review.
Thank reviewer for the valuable feedback and taking the time to review our work. We're glad the clarifications were helpful!
We sincerely appreciate the reviewer’s comments and the opportunity to further clarify our work. We are grateful for the chance to provide additional details regarding our optimization goals and the joint probability density.
### **1. What objective are the authors claiming their application of EM optimizes?**
Our objective is to maximize the sum of the variances of the IWE across different motion parameter distributions (Eq. (4) in the main text). As described in the supplementary PDF (Fig. S1), the variance indicates that correct motion patterns (illustrated by the circles in the middle subfigure) concentrate the event flow distribution along the object's edges, resulting in higher variance. In contrast, incorrect motion patterns disperse the event flow, leading to lower variance. In the network we designed, which includes a WTA mechanism, the parameters of the motion neurons, denoted as $\theta$, are optimized using the STDP rule to achieve this goal. As shown in Eq. (12) and Fig. 3 of the main text, when $P(z_j=1|e_k=1)$, only event streams that match motion pattern $j$ will activate the corresponding motion neuron to adjust its parameters, thereby maximizing the variance of IWE (encoded by $u$) associated with motion pattern $j$.
### **2. Joint Probability Density**
From the perspective of a probabilistic model, we can consider that events conforming to different motion patterns emerge over time. However, in the task of dividing event streams based on motion patterns, it is hard to directly generate a model (e.g., a mixture distribution model) using a generative model. In previous work [34], the authors adopted a method based on Eq. (2), which is more suitable for traditional mixture distribution models (e.g., fuzzy mixture models and k-means clustering) to divide event streams by motion.

In fuzzy mixture density and k-means methods, the motion-compensated IWEs do not include the event cluster associations $P$, which means that sharper object boundaries appear in some IWEs compared to others. The key difference between the EM model in this paper and traditional mixture distribution models lies in the fact that not all motion parameters are mixed. Instead, a specific one-to-one relationship is established between the event $e_{k}$ and the motion neuron $z_j$, resulting in a more precise correspondence.

Therefore, we can define the joint probability distribution as:

$$
P(e_k, z_k \mid \theta) = P(z_k \mid \theta) \cdot P(e_k \mid z_k, \theta)
$$

where the definition of $P(e_k \mid z_k, \theta)$ relates to the IWE value.

Specifically, we can define the conditional probability of event $e_k$ belonging to motion pattern $z_j$ as:

$$
P(e_k \mid z_j, \theta) \propto I_j(x_{k_j}' \mid \theta_j)
$$

where $I_j(x_{k_j}' \mid \theta_j)$ represents the IWE value calculated based on the position and time of event $e_k$ using the parameters $\theta_j\$ corresponding to motion pattern $z_j$.

We will include this explanation in the revised version of our paper. Thank you again for your valuable comments!
I thank the authors for their close technical engagement here, and would request that they guide me through just a couple more steps of reasoning.

I understand that the EM algorithm is ultimately a coordinatewise optimization of the ELBO, first on latent variables and then on model parameters (alternating until convergence).  But what objective, then, are the authors claiming their application of EM optimizes?

Similarly, I understand that the authors are claiming to perform EM rather than implement a generative model, but EM is an inference algorithm for probabilistic generative models with latent variables.  If you cannot write down a joint probability density, that is a generative model, to parameterize the ELBO, then the EM algorithm, as I understand it, cannot be used for inference.

Could the authors help me through my confusion here?
Thank the authors for the clarifications on the motion pattern initialization and noise handling. I have no more questions.