Title: Continuous Spatiotemporal Events Decoupling through Spike-based Bayesian Computation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Bayesian Computation

Spiking Neural Network

Event Cameras

Motion Segmentation

Winner-Take-All

Spike-timing-dependent plasticity
Abstract: Numerous studies have demonstrated that the cognitive processes of the human brain can be modeled using the Bayesian theorem for probabilistic inference of the external world. Spiking neural networks (SNNs), capable of performing Bayesian computation with greater physiological interpretability, offer a novel approach to distributed information processing in the cortex. However, applying these models to real-world scenarios to harness the advantages of brain-like computation remains a challenge. 
Recently, bio-inspired sensors with high dynamic range and ultra-high temporal resolution have been widely used in extreme vision scenarios. Event streams, generated by various types of motion, represent spatiotemporal data. Inferring motion targets from these streams without prior knowledge remains a difficult task. The Bayesian inference-based Expectation-Maximization (EM) framework has proven effective for motion segmentation in event streams, allowing for decoupling without prior information about the motion or its source. 
This work demonstrates that Bayesian computation based on spiking neural networks can decouple event streams of different motions. The Winner-Take-All (WTA) circuits in the constructed network implement an equivalent E-step, while STDP achieves an equivalent optimization in M-step. Through theoretical analysis and experiments, we show that STDP-based learning can maximize the contrast of warped events under mixed motion models. Experimental results show that the constructed spiking network can effectively segment the motion contained in event streams.


Peer reviews:
comments: SUMMARY:
This paper presents a spike-based Bayesian inference framework for motion segmentation with event cameras. By designing neurons that utilize STDP for online learning of motion patterns, the framework can perform the M-step of the EM algorithm in motion segmentation of event streams. Additionally, the WTA circuit implements the E-step, allowing for the online partitioning of event streams into different motion patterns. The authors provide theoretical proof and experimental results to demonstrate the network's spatiotemporal decoupling capabilities for mixed motion patterns of event streams.

STRENGTHS:
The authors demonstrate that the SNN framework based on WTA is equivalent to the EM algorithm for motion segmentation of event streams. This online learning approach is compatible with neuromorphic data and beneficial for deployment on low-power, low-latency neuromorphic computing platforms.
 
• The work is based on the Bayesian brain hypothesis, using a more physiologically interpretable SNN for Bayesian inference. Applying this to spatiotemporal data from neuromorphic cameras represents a promising research direction.

WEAKNESSES:
• The experimental results lack quantitative evaluations. Can the authors further perform object detection and tracking based on the motion segmentation, providing metrics such as object detection success rates and comparisons with other methods?
 
• The proposed algorithm lacks the analysis of time complexity or processing speed. Can it leverage the low-latency advantage of event cameras?

QUESTIONS:
Please see weaknesses.

LIMITATIONS:
There is a need for quantitative evaluations and an assessment of the dependency on parameter initialization.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

comments: SUMMARY:
This work proposes a spike Bayesian computational framework for continuous motion segmentation in event streams and demonstrates that the constructed network can implement an EM-based event stream motion segmentation model. The proposed model uses WTA circuits in the network to achieve an equivalent E-step, while the STDP rules for an M-step for contrast maximization. Experimental results demonstrate the network's online learning effectiveness for continuous inputs on extreme event camera datasets.

STRENGTHS:
The proposed network's effectiveness for motion segmentation has been validated on event datasets featuring challenging scenarios that involve mixed camera self-motion and high-speed moving objects. The proposed spike Bayesian inference framework is highly interpretable and applicable to various neuromorphic vision chips and computing hardware, representing a promising research direction.

WEAKNESSES:
The authors mainly use SVD to find different patches' motion patterns for initialization. Why is this method used, and can other methods be employed for selection? It is recommended that the authors conduct ablation experiments to explore further.

QUESTIONS:
This method primarily targets optical flow motion estimation. For more complex motion patterns, how to design the parameters? How robust is this method against noise in the evaluation of such motion models? The authors should clarify it.

LIMITATIONS:
The authors have stated the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

comments: SUMMARY:
The paper proposes to address motion segmentation at very high temporal resolution via an event-based or spiking implementation of expectation-maximization in a generative model. It demonstrates the performance of the resulting spiking neural networks on example experiments.

STRENGTHS:
The strength of the paper is its deep engagement with the spiking neural network literature, as well as its use of spiking networks for the specific type of problem to which they are most suited: event-based computation.

WEAKNESSES:
The paper's major weakness is its lack of clarity, which the authors have discussed and addressed in the review period.

QUESTIONS:
The authors have addressed my questions, though I would still like to see discussion of how this framework for EM in spiking networks could be generalized beyond motion detection.

LIMITATIONS:
I am not confident that I can identify the specific limitations of this paper as opposed to the limitations of spiking neural networks generally.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper demonstrates that WTA circuits along with STDP learning resembles EM algorithm-like Bayesian inference and could be used for motion segmentation from event streams by contrast maximization of warped events.\n\nSTRENGTHS:\nThe paper proposes an interesting approach for event motion segmentation based on observations from event-based dynamic vision sensors, utilizing a EM-like framework for identifying various motion models from event streams and clustering them into motion patterns. This is achieved using WTA circuits together with STDP-based learning.\n\nWEAKNESSES:\nThe main weakness of the paper is that the proposed method lacks proper justification of the presented approach, which seems like a heuristic hard clustering method, together with gradient based learning. The experiments also lack depth and the authors demonstrate the high dependence of the performance of the method on the parameter initialization. A more careful writing of the underlying model, the optimization framework and the proposed methodology would be good (see the questions below). Furthermore, the paper lacks more details regarding the choice of $N_{\\ell}$ (number of motion models) and the specific forms of the warping functions $W_j$ used. Several steps in the entire methodology, although intuitive, are presented in a heuristic fashion without detailed description and clarity.\n\nQUESTIONS:\nHere are some general questions/comments about the framework:\n1. Full form of the abbreviation STDP missing in abstract/introduction.\n2. In Eq. (1), why is $\\Delta L(x_k,t_k) = q_k\\Theta$, since according to line 107, event $e_k$ corresponds to when the intensity change \\textit{exceeds} $\\Theta$ (noting that $|q_k|=1$). In line 109, add \"where $L(x,t)$ is the ... at pixel $x$ \\textit{at time $t$}\". \n3. Line 118: what integration is used? Eq. (2) only describes $I_j(x)$ as a mixture of Dirac measures. Add the definition of $N_e$ (possibly the number of observed events). In Eq. (2), does $x$ represent a pixel? What does the suffix $j$ capture? Based on the description, it suggests that it represents the different \\textit{motion models} --  it would be better to explain both the model and the optimization problem in slightly more detail for clarity. \n4. The EM framework: while the updates for the model resemble E and M steps in EM, is it actually related? Can you show that this method indeed improves some form of likelihood of the model (recall that EM is most commonly used for MLE in mixture models or other latent variable models)? Can the authors discuss how their method is related to EM (the E and M steps in the current paper are more closer to the hard clustering type algorithms e.g. K-Means rather than EM, particularly the E-step Eq. (5))\n5. In Eq. (6), extra $dx$ at the end, also might be better to keep the two terms in a parenthesis.\n6. More on the model Eq (2): according to line 119, $p_{kj}$ represents the probability that event $e_k$ belongs to motion model $z_j$, in that case $\\sum_j p_{kj}=1$, is that correct? However, in that case, Eq. (2) does not represent a mixture -- i.e., $\\sum_k p_{kj}$ might not be 1, can the authors clarify this? Furthermore, the Dirac function in Eq (2) allows the IWE to only pick up values at pixels $x$, where at least one event $e_k$ has been observed (through the transformed position). This does not allow any spatial relation across the pixels - why can the Dirac function not be replaced by some other smooth kernel (like Gaussian)?\n7. More on the optimization problem Eq (4): When writing $\\text{Var}(I_j)$, whose randomness are we taking the variance (or other expectation operations) with respect to? It seems like $\\theta, P$ are parameters (hence fixed) and $x_{kj}'$ is some deterministic transform of the observed events. Can the authors clarify the underlying probability structure of the model?\n8. The authors might provide a brief description of the STDP method (and its connections to spiking neural networks) and WTA circuit, which might clarify some of the paragraphs e.g., lines 170-173. It is also unclear why $u_j (t)$ (defined in Eq (10)) is equivalent to $I_j$ (in Eq (2)) as claimed in line 178 - is the $W_j$ in Eq (10) same as the WTA $W$ in Eq (3) -- if so, why is the second input $t_k$ in the latter while $p_{kj}$ in the former?\n9. Can the authors explain lines 205-206. It seems like they argue that gradient update increases the variance -- however, this is only the M-step (i.e., conditional on the current values of $p_{kj}$ I am guessing). \n10. Why is $u_j$ expressed as a function of time $t$ in Eq (10)? Can the authors clarify how the temporal dependence is captured in the model?\n\nLIMITATIONS:\nSee the questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}