Unifying Spike Perception and Prediction: A Compact Spike Representation Model using Multi-scale Correlation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 07:13:14

Abstract: The widespread adoption of bio-inspired cameras has catalyzed the development of spike-based intelligent applications.
Despite its innovative imaging principle allows for functionality in extreme scenarios, the intricate nature of spike signals poses processing challenges to achieve desired performance.
Traditional methods struggles to deliver visual perception and temporal prediction simultaneously, and they lack the flexibility needed for diverse intelligent applications.
To address this problem, we analyze the spatio-temporal correlations between spike information at different temporal scales.
A novel spike processing method is introduced for compact spike representations that utilizes intra-scale correlation for higher predictive accuracy.
Additionally, we propose a multi-scale spatio-temporal aggregation unit (MSTAU) that further leverages inter-scale correlation to achieve efficient perception and precise prediction.
Experimental results show noticeable improvements in scene reconstruction and object classification, with increases of **3.49dB** in scene reconstruction quality and **2.20%** in accuracy, respectively.
Besides, the proposed method accommodate different visual applications via switching analysis models, offering a novel perspective for spike processing.

Main paper: 
 introduction: 
 With the popularization of autonomous driving and Industry-4.0, an innovative biologically inspired spike camera has emerged, which emulates the fovea of mammalian retina [31] [3] . Breaking the limitation of exposure [18] , the spike camera emits spikes at each position asynchronously and achieves a maximum temporal resolution of up to 40k frames per second (FPS). This can effectively mitigate motion blur and content missing, supporting visual tasks in scenarios with high-speed motion, such as scene reconstruction and classification for fast moving objects. In practical usages, intelligent applications are usually deployed in an end-cloud collaborative architecture due to the weak computational ability of cameras [38] [25] . The camera captures information and emits spikes, which are compressed and transmitted to a cloud server for further processing according to specific task. On the one hand, it is essential to extract sufficient semantic information from spike sequences to satisfy the precision requirements of visual applications at different granularities [23] [13] [16] . On the other hand, accurate prediction of future spatio-temporal states is also crucial for proactive decision-making ahead of time [21] [20] . This imposes demands on perception and prediction of spike signals simultaneously. Given the analyzed results, an intuitive strategy for simultaneous perception and prediction of spike signals is to first process spikes to meet demands of visual applications and then perform temporal predictions on the result sequences. However, this struggles to effectively address these challenges. On one hand, although this approach can achieve satisfactory performance in specific task, it lacks versatility for scenarios with multiple tasks, especially when these applications involve different semantic levels [36] [37] . On the other hand, inconsistency between the target of perception and prediction lead to distortion, which accumulates gradually over temporal interval, making it increasingly difficult to effectively perceive after predictions [15] . For instance, in scene reconstruction, spike sequence can be rebuilt into a series of scenes initially, upon which video prediction methods are applied for forecasting. However, results in Fig. 1(b) show that the predictive quality is less efficient enough. One issue is that the reconstructed scenes exhibit severe distortion compared with ground truth, which escalates with the extension of prediction interval. Moreover, prediction in pixel-level accuracy is limited so that the abundant temporal information contained in the spike sequence is used to compensate for the lack of spatial information (e.g., calculating the average firing rate and estimating the light intensity at the current moment based on spike generation over a period of time). In contrast, prediction on scenes rebuilt from spikes requires joint modeling of spatio-temporal information from historical observations. With temporal details discarded in reconstructed scenes and only spatial information preserved, achieving high-accuracy forecasts becomes challenging. Besides, this method is not directly applicable for other visual tasks and retraining is required for all modules. Therefore, there is an urgent necessity to propose an effective method for spike processing with perception and prediction.Building upon the above discussion, we introduce an innovative spike processing method considering multi-scale correlations, which are categorize as intra-scale and inter-scale one. Compared with previous approaches, information within spikes can be predicted in temporal domain with higher efficiency and fidelity, which can further serve for downstream applications with better performance. The major contributions are summarized as follows:• We present a novel spike data processing method utilizing compact spike representations through multi-scale correlations. Our method offers perception and prediction at multiple semantic levels, providing a fresh perspective on spike visual intelligence.  
related work: 
 Scene reconstruction and object classification are two important visual applications in spike processing. The former utilizes the principle of spike firing, which is stimulated by luminance intensity, to rebuild scenario content with high pixel-level fidelity. The latter involves categorizing objects captured by spike cameras based on higher-level semantic information, resulting in superior accuracy. Scene reconstruction. Chen et al. [28] introduced a tokenizer and spatio-temporal attention significantly enhances the accuracy and stability of reconstruction quality. Zhu et al. [42] put forward a framework for retina-like visual image reconstruction, harnessing bio-realistic Leaky Integrate and Fire (LIF) neurons along with synapse connections governed by spike-timing-dependent plasticity (STDP) rules. Additionally, Spk2ImgNet proposed in [39] stood out for its ability to reconstruct dynamic scenes from continuous spike streams through the use of deep learning techniques. Wei et al. [19] also introduces a retinal spike train decoder to enhance the accuracy of reconstructing visual scenes from retinal spike trains, demonstrating its potential to advance brain-machine interface technologies by improving how visual scenes are decoded from spike data. Xiang et al. [35] contributed to the field by proposing a learning-based super-resolution reconstruction method tailored for high temporal resolution spike streams. [8] presented a novel coarse-to-fine method utilizing region-adaptive-based spike distinction, which effectively differentiates between dynamic and static spikes to enhance the reconstruction of natural scenes.Object classification. Felix et al. [10] employed linear filters for enhanced signal representation and signal-to-noise ratio, and used well-defined thresholds for simultaneous spike detection and classification. Wilson et al. [33] utilized multiple monotonic neural network to group spikes automatically via hierarchical clustering, which is visually compared with hand marked grouping on a single record. Recent years, Zhao et al. [40] introduced a modeling algorithm to assess the detection capability of spiking cameras under various scenarios, such as different brightness intensities and camera lens configurations. This algorithm helped determine appropriate camera settings for capturing high-speed objects. SpiReco mentioned in [41] discussed the potential of spike cameras in highspeed object recognition. The study highlighted the advantages of spike cameras, such as their high temporal resolution and dynamic range, while also acknowledging the challenges posed by their physical limits. Besides, a accelerator was implemented on FPGA for high-speed moving objects detection and tracking with a spike camera. This research explored hardware acceleration techniques to enhance the real-time processing capabilities of spike cameras, enabling them to detect and track high-speed objects more effectively. 
methodology: 
 Building upon the discussion in Sec. 1, we propose to convert the spike sequence {𝑆 𝑡 } into compact spike representations {𝐹 𝑡 } according to the biological visual mechanism, which are rich in spatio-temporal information. Based on observations of historical representations {𝐹 𝑡 -𝑖 }, we predict the representation at the current moment F𝑡 and utilize it to generate the result R𝑉 𝑡 for applications, formulating asEQUATIONEQUATIONEQUATIONwhere FE and 𝑃𝑟𝑒𝑑 denote the feature extraction and prediction module respectively, and A 𝑉 is the analysis component for task 𝑉 . On the one hand, {𝐹 𝑡 } are profuse with spatial information at different granularities, which can be reused for multiple downstream tasks at various semantic levels through different A 𝑉 . On the other hand, {𝐹 𝑡 } are abundant with temporal information at different scales, enabling efficient and accurate forecast at multiple extents. Moreover, the dimensions of {𝐹 𝑡 } can be compacted to enhance the expressive capability of {𝐹 𝑡 } through increasing the density of information, while the computational complexity can be decreased in the meanwhile. Consequently, the compact representation-based spike processing approach can efficiently meet the demands of various applications and achieve high-precision temporal predictions.As spike cameras are bio-inspired, the processing for spike signals should also mimic biological visual mechanism to extract features abundant with spatio-temporal information [17] [2] . We decouple the persistent light stimuli into a sequence of transient luminance stimulus, each of which lasts for an extremely short time and thus the luminance can be approximated as constant. Taking the LIF as an example of neural dynamics [34] [14], for a neuron with time constant 𝜈, under the influence of transient luminance stimulus 𝐼 defined byEQUATIONwe derive the temporal extent of this impact reaching up to 𝑡 𝜃 .The entire process can be divided into two dependent procedures, which are the charging during [0, 𝑡 0 ) and the discharging during [𝑡 0 , 𝑡 𝜃 ]. The schematic diagram of the membrane potential changing over time is shown in Fig. 2(a) . The neuro-dynamic differential equation can be expressed asEQUATIONwhere 𝑉 𝑡 and 𝐸 denote the membrane potential at moment 𝑡 and the resting potential, 𝑅 is the membrane resistance and 𝛼 represents the photovoltaic conversion efficiency [4] [27] . By joint solving equations for both two processes, 𝑡 𝜃 can be formulated asEQUATIONwhere 𝑉 𝜃 illustrates the threshold that can distinguish between electrical signals or disturbances. Noticing that 𝑡 0 is an infinitesimal, which means the equation above can be approximated via Taylor expansion asEQUATIONwhere 𝑃 0 denotes the amount of photons received by the neuron per unit time [22] . For any moment 𝑡, 𝑉 𝑡 is a nonlinear superposition of the effects of all luminance stimuli during [𝑡 -𝑡 𝜃 , 𝑡). After substituting the relevant parameters, the PLCC (Pearson Linear Correlation Coefficient) between 𝑡 𝜃 and 𝜈 reaches 0.995 [7] , indicating that their correlation is extremely high. Since the mammalian visual system consists of neurons with different time constants [11] [1] [24] , it is essential to take into account the impact of various 𝜈 on neural dynamics in a comprehensive manner. 11 Detailed derivations are shown in the supplementary material.From perspective of implementation, MTFs with various temporal receptive fields (TRFs) are required as a combination to extract features with different temporal granularities, which are subsequently associated to support intelligent applications. We further illustrate the necessity of using MTFs through a scene reconstruction task in both spatial (Fig. 2 (b∼e)) and temporal (Fig. 2 (f∼i)) domain. As for spatial texture, reconstructed results are significantly influenced by the stochastic nature of spikes (pixel-level noise) if TRF is too small, while an excessively large TRF makes it challenging for alignment, leading to severe motion blur. In terms of temporal characteristic, movement exhibits strong disorder and randomness for small TRF. In contrast, motion is more consistent and the boundaries between foreground and background are more pronounced for large TRF.We assert that features produced by filters with varying TRFs exhibit unique temporal scales. Therefore, composing features 𝑓 1:𝐾 𝑡 at different scales into 𝐹 𝑡 is crucial to maintain the richness and continuity of spatio-temporal information. Eq. 1 can be adapted asEQUATIONEQUATIONwhere MFE and FC denote the multi-scale feature extraction and feature composition module respectively, and 𝐾 represents the number of temporal scales.To explore the discrepancies and correlations between features at different scales, we statistically analyze the spatio-temporal characteristics through TFP reconstruction results with various TRFs [26] [32], as shown in Fig. 3(a) . The pixel smoothness in spatial domain is negatively correlated with TRF, while the motion intensity in temporal domain is positively associated with TRF. This strong correlation indicates that visual information corresponding to different TRFs have significant disparities. Therefore, the representation composed from features with varying scales possess more complex spatio-temporal characteristics, making it challenging to be precisely predicted.Despite the different texture contents and motion trends extracted from features at various temporal scales, the spatio-temporal information within the same scale exhibits significant stability in temporal domain, referred as intra-scale correlation. We calculate the L1-norm of the residual between adjacent frames for features at each scale [12] , as shown in Fig. 3(b) . Results indicate that features at the same scale demonstrate strong temporally continuity with fewer noticeable fluctuations, denoting stable characteristics within scale. In contrast, although the representation is composed from these features, the emphatic temporal randomness and volatility make it challenging for prediction. Therefore, we propose an ICP method to model spatio-temporal characteristics of features at each scale separately for higher prediction accuracy, pipeline of which is illustrated as Fig. 4 . With historical observations from last 𝜏 moments, each feature is predicted separately as f 𝑘 𝑡 , all of which are then composed as F𝑡 . Therefore, Eq. 9 and Eq. 2 can be adapted as1 1 𝐶𝑒𝑙𝑙 1 2 𝐶𝑒𝑙𝑙 1 𝐾 𝑓 𝑡 1 𝑓 𝑡 2 𝑓 𝑡 𝐾 𝐶𝑒𝑙𝑙 𝑁 1 𝐶𝑒𝑙𝑙 𝑁 2 𝐶𝑒𝑙𝑙 𝑁 𝐾 𝐹 𝑡+1 ⋮ 𝑇 𝑡-1 1,𝐾 𝑇 𝑡-1 1,2 𝑇 𝑡-1 1,1 𝑇 𝑡 1,𝐾 𝑇 𝑡 1,2 𝑇 𝑡 1,1 𝑇 𝑡-1 𝑁,𝐾 𝑇 𝑡-1 𝑁,2 𝑇 𝑡-1 𝑁,1 𝑇 𝑡 𝑁,𝐾 𝑇 𝑡 𝑁,2 𝑇 𝑡 𝑁,1 መ 𝑓 𝑡+1 1 መ 𝑓 𝑡+1 2 መ 𝑓 𝑡+1 𝐾 𝑆 𝑡 1,1 𝑆 𝑡 1,2 𝑆 𝑡 1,𝐾 𝑆𝑝𝑘 Multi-scale Feature Extraction ⋮ ⋮ 𝑆 𝑡 𝑁-1,1 𝑆 𝑡 𝑁-1,2 𝑆 𝑡 𝑁-1,𝐾 Representation CompositionEQUATIONEQUATIONwhere 𝑓 {1:𝐾 } {𝑡 -𝜏:𝑡 -1} are historical features extracted according to Eq. 8.Although there is a wide variation between features with different scales shown in Fig. 3 (b), we observe a relatively consistent trend in their changes. Therefore, we calculate the PLCC between optical flows at various scales, as shown in Temporal Regression Module. Based on the temporal relevance at each scale, this module regresses historical states towards the current moment and generate multiple regression results corresponding to each scale. Prior works transferred the relevance between (𝑡 -𝜏 ∼ 𝑡 -1) and 𝑡 in the 𝑛 -1 𝑡ℎ layer to the 𝑛 𝑡ℎ layer to predict the state at 𝑡 [6] [5] . However, these approaches performed poorly due to the lack of consideration for intra-scale and inter-scale correlations. Our module extends this concept to multiple scales, transferring the temporal relevance of all scales in the 𝑛 -1 𝑡ℎ layer to the current scale in the 𝑛 𝑡ℎ layer for regression. For scale 𝜅, similarities between spatial states at 𝑡 and 𝑡 -𝑖 in the 𝑛 -1 𝑡ℎ layer is calculated as guidance on regression of temporal states, formulating asEQUATIONEQUATIONwhere 𝑞 𝜅 𝑖 is the similarity score and 𝑇 𝑛,𝜅 𝑟𝑒𝑔 denotes the regression result at scale 𝜅 in the 𝑛 𝑡ℎ layer, which is regarded as a long-term regression for temporal state at moment 𝑡.Multi-scale Aggregation Module. The major challenge addressed by this module is how to adaptively aggregate long-term regressions 𝑇 𝑛,{1:𝐾 } 𝑟𝑒𝑔 corresponding to various scales and combine them with momentary state 𝑇 𝑛,𝑘 𝑡 -1 to obtain the most accurate estimation of the temporal state. As for long-term aggregation, we transfer the correlation between all scales and the current scale from spatial states to temporal ones. Attention scores within scales 𝑟 𝑘 {1:𝐾 } are generated according to pixel-wise similarity, which then serve as navigation for fusion of 𝑇 𝑛,𝑘 {𝑡 -𝜏:𝑡 -𝑖 } , formulating asEQUATIONEQUATIONwhere 𝑇 𝑛 𝑎𝑔𝑔 is respected as the comprehensive long-term aggregation for temporal state. As for long and short-term combination, 𝑇 𝑛,𝑘 𝑡 -1 is utilized for providing momentary information and maintaining content consistency. We employ an adaptive gating mechanism that generates fusion ratio, based on which to fuse trending and momentary information, formulating asEQUATION)EQUATIONwhere 𝑢 𝑓 represents the ratio generated from fusion gate 𝑈 𝐹 and 𝑇 𝑒𝑠𝑡 denotes the estimated temporal state. Spatio-Temporal Injection Module. During information propagation, spatio-temporal states should be processed with injection bidirectionally to ensure the consistency. Drawing inspiration from the dual-gate mechanism, two gates 𝑈 𝑆 and 𝑈 𝑇 are employed to generate injection ratios respectively, guiding the bi-directional interaction of spatio-temporal information. As shown in Fig. 5(c ), the temporal and spatial injection sub-modules are formulated asEQUATION)EQUATIONEQUATIONEQUATIONwhere 𝑢 𝑠 and 𝑢 𝑡 are ratios generated from 𝑈 𝑆 and 𝑈 𝑇 , guiding the injection of information form temporal to spatial domain and its reversal. This constrains the output to maintain consistency between two dimension. 
experimental settings: 
 We select 𝐾 = 4 MTFs in ICP methods with 𝑇 𝑅𝐹 = (7, 9, 11, 13) .MSTAUs are stacked with 𝑁 = 4 layers, each of which takes 𝜏 = 5 historical states as observations. We consider scene reconstruction and object classification as representatives of spike-based visual applications at distinct semantic levels. The former is validated on S-VIMEO dataset [9] , while the latter is evaluated on S-MNIST, S-CIFAR, and S-CALTECH datasets [41] . For performance comparison, we utilize two state-of-the-art (SOTA) temporal prediction models, PredRNN [30] and MAU [6] , as backbones to assess the reconstruction precision and classification accuracy. 2 Effectiveness of intra-scale correlation. Through comparing the predictive accuracy with and without incorporating ICP method, we validate the effectiveness of intra-scale correlation in guiding spike prediction. Without ICP method, the predictor forecasts representations for future moments directly, corresponding to Eq. 1, 2 and 3. In contrast, with ICP method composed of Eq. 8, 10, 11 and 3, the predictor forecasts multi-scale features, which 2 Details are illustrated in the supplementary material. are then composed as visual representations to support intelligent applications. As shown in Table . 2, the prediction error measured by MSE on representation { F𝑡 } decreases by 7.3% for PredRNN and 8.9% for MAU when the ICP method is applied. This improvement is attributed to the ICP method's capability to effectively utilize intra-scale correlations to predict spatio-temporal contents with high fidelity. The enhanced accuracy is evident when comparing the prediction errors of { f {1:4} 𝑡 } and { F𝑡 } in Table . 2, demonstrating the more complex spatio-temporal characteristics of 𝐹 𝑡 make it more challenging to predict accurately compared to 𝑓 {1:4} 𝑡 , which is consistent with the conclusion in Fig. 3(b) . ICP method, by composing accurate f {1:4} 𝑡 as F𝑡 , simplifies a complex prediction into multiple sub-predictions and reaches high precision through the strong relevance within each scale.Precise scenes can be reconstructed subsequently from this accurate predicted representation, with statistical results shown in Table . 3, demonstrating the efficacy of intra-scale correlation for visual perception. Compared to the absence of ICP method, employing ICP method leads to an average increase of 2.7% in the PSNR, with the gain growing as the prediction time interval extends (0.32dB, 0.59dB, 0.99dB, 1.19dB ). This suggests that ICP can exploit the stronger temporal continuity of features at different scales, resulting in more significant performance enhancements for long-term predictions of scene sequences. We also visualize reconstructed scenes based on predicted features through ICP method, as shown in Fig. 6 . Without ICP method, local low-intensity features are often misinterpreted as noise, causing uncontrolled blurring in the reconstructed content. Moreover, these noisy features tend to propagate over time, leading to accumulation of blurring and other artifacts that significantly deteriorate scene quality. On the other hand, implementing the ICP method stabilizes the predicted features, preventing high-frequency information from being overshadowed by low-frequency noise. Consequently, scenes reconstructed with ICP exhibit distinct edges and rich texture details.Effectiveness of inter-scale correlation. We demonstrate the significant enhancement in perception and prediction achieved through further incorporation of inter-scale correlations by comparing MSTAU with other ICP methods. According to Table. 2, the MSTAU improves accuracy of each feature sequence by 4.9%, 7.5%, 8.6% and 7.0% compared to ICP methods, which is consistent with the PLCC in Table . 1 approximately. Feature corresponding to TRF=11 exhibit the strongest correlation with others and guiding it with other features can therefore maximize the prediction accuracy of f 3 𝑡 , while f 1 𝑡 corresponding to TRF=7 demonstrates opposite characteristics. This indicates that content with stronger correlations to features at other scales can be guided by inter-scale information more effectively, resulting in more precise predictions. Moreover, the accuracy of predicted representation F𝑡 can also improved with assistance of MSTAU, as illustrating in Table . 2. Compared to ICP approaches only considering intra-scale correlations, MSTAU enhances the precision of composed representations by an average of over 8.9%. This improvement exhibits a roughly decreasing trend with the increase in prediction temporal interval (13.2%, 10.5%, 6.8%, 5.2%). Although features are correlated, the spatio-temporal characteristics corresponding to different TRFs are distinct essentially. As a result, the correlation between contents within different features significantly decreases over time, and the temporal estimation generated with the help of prior knowledge shows noticeable distortion. These factors collectively contribute to the decrease in accuracy gain.For visualization, we showcase the predictive reconstruction scenes of MSTAU in Fig. 7 . MSTAU outperforms methods with no inter-scale correlation by predicting and generating higher quality scenes in short-term scenarios (e.g. the spoon with more distinct edges and the flour with a grainier texture). However, we note that as the prediction interval extends, the quality of the reconstructed scenes deteriorates, characterized by overly high local contrast and a noticeable light-dark grid effect (Fig. 7(d) ). This decline in quality stems from the decreasing correlation of features across different scales, resulting in content inconsistencies and misalignments. The reliance on the pixel-wise L2-norm as the only constraint causes the model to overlook the rationality of local textures and their correlation with adjacent frames during processing. Future work can address this issue by incorporating inter-frame constraints. S-CALTECH respectively as shown in Table . 4. This demonstrates that introducing inter-scale correlation for prediction can enhance the semantic fidelity of visual representations, providing support for spike-based intelligent applications in perception and understanding. Additionally, we visualize the last layer of latent features from S-MNIST via t-Distributed Stochastic Neighbor Embedding (t-SNE) [29] , as shown in Fig. 8 . Results indicate that predicted labels for samples within the same category remain tightly clustered in the high-dimensional space, thereby ensuring high fidelity in classification accuracy. However, the degree of congregation slightly decreases over time, leading to a small number of samples being misclassified and a slight reduction in classification precision essentially.Versatility for multiple tasks. We further validate the versatility and robustness of our method for multiple intelligent tasks on S-MNIST. Fig. 9 shows the results processed by the reconstruction and classification network respectively from the predicted visual representation F𝑡 . The proposed ICP method and MSTAU can reconstruct precise scenes while ensuring accurate classification with high confidence. Moreover, we note that although the distortion in reconstructed scenes becomes more noticeable as the prediction interval extends, the classification accuracy consistently remains high. This indicates that for visual tasks at different semantic levels, performance exhibits varying levels of temporal decay with the prediction interval, suggesting that our method can be paired with different analysis models to achieve diverse intelligent applications with varying granularity and precision requirements.Effectiveness of feature-domain processing. We take scene reconstruction to demonstrate the rationality of feature-domain processing compared to that in scene-domain. Comparison of results in Table . 3 and 5 show that feature-domain methods outperform scenedomain ones by 8.4% and 6.4% for PredRNN and MAU respectively. The average PSNR gain of the reconstructed scenes is positively proportional to the prediction interval (1.0dB, 1.7dB, 1.7dB, 1.8dB), indicating that visual features have better temporal coherence and contain more abundant spatio-temporal information to ensure efficient prediction and authentic reconstruction compared to reconstructed scenes. As shown in Fig. 1 , results generated through feature-domain processing broadly eliminate content-irrelevant artifacts and blurring, while retain high-frequency texture details in the meanwhile . This indicates that visual features contain comprehensive information of various semantic granularities, which is utilized for reconstruction and prediction with higher accuracy.Effectiveness of multi-scale aggregation. We verify the rationality of proposed multi-scale aggregation through ablation experiments. In the aggregation process described as Eq.14 and 15, the inter-scale correlations between spatial states are utilized to obtain the weight 𝑟 . 1, suggesting that features with stronger correlations are more effectively guided and aggregated through information interaction, leading to enhanced prediction accuracy.Additionally, we further confirm the impact of aggregation process on the spatial and temporal states of the output, as shown in Fig. 11 . 𝑢 𝑓 , 𝑢 𝑠 , and 𝑢 𝑡 correspond to Eq. 16, 18 and 20 respectively. The sequence of 𝑢 𝑓 indicates that in the initial stage, the information within 𝑇 𝑛 𝑎𝑡𝑡 is not yet accurate, so the aggregation module obtain most content from 𝑇 𝑛,𝑘 𝑡 -1 . As time accumulates, 𝑇 𝑛 𝑎𝑡𝑡 gradually stabilizes. For stationary areas, the module tends to obtain information from long-term estimates, while only whose with intense motion aggregate momentary information. Similarly, sequences of 𝑢 𝑠 and 𝑢 𝑡 indicate that 𝑇 𝑎𝑔𝑔 obtained mainly from momentary content has a significant deviation from 𝑆 𝑛-1,𝑘 𝑡 , making it difficult to fuse and obtain spatio-temporal states with high accuracy. As the aggregation quality improves over time, the fusion module can gradually align 𝑆 𝑛-1,𝑘 𝑡 and 𝑇 𝑎𝑔𝑔 on content, thereby jointly generating 𝑆 𝑛,𝑘 𝑡 and 𝑇 𝑛,𝑘 𝑡 with high precision. This provides support for subsequent visual applications at various semantic granularities. 
conclusion: 
 We propose an innovative spike processing method for visual perception and temporal prediction utilizing compact spike representation with high versatility, providing a fresh perspective on spike intelligence. Taking multi-scale correlation into consideration, our method effectively models spatio-temporal information at different scales, facilitating proactive decision-making in scenarios with diverse semantic complexities. Experimental results confirm that our method significantly enhance predictive accuracy of spike perception, propelling the widespread adoption and application of spike cameras and further contributing for a tech-harmonious world.  
 
 Appendix:  


Decision: This paper introduces an innovative approach to processing spike signals from bio-inspired cameras, addressing challenges posed by the intricate nature of these signals. The authors analyze spatio-temporal correlations at different temporal scales and propose a new spike processing method for compact spike representations, utilizing intra-scale correlation for higher predictive accuracy. Additionally, they present a multi-scale spatio-temporal aggregation unit (MSTAU) that leverages inter-scale correlation for efficient perception and precise prediction. Experimental results show significant improvements in scene reconstruction quality and object classification accuracy, with increases of 3.49dB and 2.20%, respectively. The proposed method also accommodates various visual applications by switching analysis models, offering a novel perspective on spike processing. The proposed method for spike signal processing represents a significant advancement in the field of bio-inspired cameras and intelligent applications. The innovative use of multi-scale spatio-temporal correlations leads to notable improvements in performance metrics, while the flexibility of switching analysis models adds to its practical value. The thorough experimental evaluation provides strong evidence of the method's effectiveness. Although there are minor areas for improvement in explaining algorithmic complexity and discussing hardware dependence, these do not detract from the overall contribution and impact of the work.

Reviews: review: paper_summary: paper_summary: The paper addresses the functional requirements of bio-inspired cameras in extreme scenarios by proposing a novel spike signal processing method. The paper analyzes the spatio-temporal correlations of spike information across different time scales and introduces an innovative approach to spike processing. This method leverages intra-scale correlations to achieve a more compact representation of spikes and enhance predictive accuracy. Furthermore, the paper proposes a Multi-scale Spatio-Temporal Aggregation Unit (MSTAU) that further capitalizes on inter-scale correlations to realize efficient perception and precise prediction.

Experimental results demonstrate that the proposed method has achieved significant improvements in scene reconstruction and target classification. The method can adapt to different visual applications by switching analysis models, providing a new perspective for spike signal processing.
strength_weakness: strength_weakness: The MSTAU proposed in the paper is a novel innovation, and the paper analyzes its theoretical rationality and demonstrates its advantages through experiments; The paper thoroughly demonstrates the effectiveness of the proposed spike processing method through extensive experiments, and the visualization of multiple experimental results more intuitively showcases the efficacy of the proposed approach.
limitations: limitations: Can an analysis of the algorithmic complexity be added to evaluate the efficiency of the algorithm? Can the effectiveness of the innovation be further demonstrated by adding tests on other datasets and tasks in terms of multi-tasking? Can you provide the specific meanings of hyperparameters in experimental settings and the reasons for determining the values of hyperparameters?
paper_summary: paper_summary: Bio-inspired cameras have a wide range of applications, however, traditional methods are difficult to provide both visual perception and temporal prediction. In this paper, the authors propose a method for spike representation using intra-scale correlation by analyzing the spatio-temporal correlation of spike information at different time scales, in addition to which, the authors propose multi-scale spatio-temporal aggregation units to further achieve efficient perception and accurate prediction. The authors conducted relevant experiments on scene reconstruction and object classification to achieve higher network model performance.
strength_weakness: strength_weakness: 1. The paper is clearly motivated. The authors first summarize two problems with existing methods for simultaneous perception and prediction of impulse signals when applied to visual tasks, namely lack of generality and prediction distortion. Among them, prediction distortion is not only a lack of pixel-level accuracy, but also aggravates the distortion over time. Therefore, there is an urgent need to propose an effective pulse processing method with both sensing and prediction functions, which is the contribution of this paper.
2. The authors propose a spike processing method for multi-scale correlation, which is divided into intra- and inter-scale correlation. The method is able to predict intra-peak information with higher efficiency and fidelity in the time domain compared to previous methods, which further provides better performance for downstream applications.
3. The experiments are adequate and convincing. The authors compare with SOTA in scene reconstruction and object classification. I think the ablation experiment part is commendable and I clearly see the effectiveness of the two proposed methods, Intra-scale Correlation based Prediction and Multi-scale Spatio-Temporal Aggregation Unit.
limitations: limitations: I see no major flaws in this paper and I am inclined to accept it.

I believe that the method is valid, nevertheless, I am not sure about the extent to which it contributes to the field as a whole, i.e. how widely it is applied and how much it contributes to the field as a whole. I therefore need a reviewer who is more experienced in this area to help me make a judgement. My rating at this stage is ‘Weak Accept’.
paper_summary: paper_summary: This paper presents a novel method for spike signal processing that aims to unify spike perception and prediction through a compact spike representation model using multi-scale correlation. The introduction critiques existing methods for their lack of simultaneous visual perception and temporal prediction capabilities, and proposes a multi-scale spatio-temporal aggregation unit (MSTAU) to address these deficiencies. The method exploits intra-scale and inter-scale correlations to increase the efficiency of spike signal processing, demonstrating significant improvements in scene reconstruction and object classification. Specifically, the method achieved a 3.49dB increase in scene reconstruction quality and a 2.20% increase in classification accuracy.
strength_weakness: strength_weakness: The approach proposed in this paper is novel. The authors develop a new way of processing spike signals, the Multiscale Spatio-Temporal Aggregation Unit (MSTAU), by analyzing the spike signals and spatial and temporal correlations on different time scales.
limitations: limitations: 1. Does MSTAU incur higher computational costs? It is recommended that the authors provide more details on the computational cost and practicality of deploying this method in real applications.
2. An analysis of the limitations of existing methods is lacking in the descriptions in the related work in Section 2.
3. Many symbols in the main text lack corresponding meanings, which causes reading problems, although the appendix provides a table of symbol descriptions.
paper_summary: paper_summary: The paper introduces an innovative spike processing method, integrating perception and prediction using compact spike representations through multi-scale correlations. The proposed method, utilizing intra-scale correlation based prediction (ICP) and a Multi-scale Spatio-Temporal Aggregation Unit (MSTAU), aims to enhance predictive accuracy for visual tasks like scene reconstruction and object classification. The experiments demonstrate significant improvements in predictive accuracy, scene reconstruction quality, and classification accuracy, providing a robust foundation for spike processing applications.
strength_weakness: strength_weakness: Novel Methodology: The introduction of ICP and MSTAU to leverage both intra-scale and inter-scale correlations offers a fresh perspective on spike processing. This approach enhances the precision of temporal predictions and visual perception tasks.  
Experimental Validation: The paper provides extensive experimental results showcasing improvements in predictive accuracy, scene reconstruction quality (3.49dB increase in PSNR), and classification accuracy (2.20% improvement in Top-1 accuracy).  
Versatility: The proposed method is adaptable to both low-level and high-level spike visual tasks, indicating its potential applicability across a wide range of intelligent visual applications.  
Comprehensive Analysis: The paper includes a thorough analysis of spatio-temporal correlations and the effectiveness of the proposed modules, supported by detailed experimental results and visual comparisons.
limitations: limitations: The authors mentioned that the code is available in code.py, but it is currently an empty file.  
The description of the implementation details, especially for the multi-scale spatio-temporal aggregation unit, might be too complex for straightforward replication without an accompanying detailed pseudocode or source code. The mention of code.py being empty is a significant oversight in this respect.  
While the paper mentions using the S-MNIST, S-CIFAR, and S-CALTECH datasets for experiments, it lacks detailed descriptions of the specific experimental settings and data preprocessing. This may cause difficulties for readers trying to replicate the experiments.  
The paper selects PredRNN and MAU as comparison methods but does not provide detailed reasons for choosing these methods. It also does not cover other possible comparison methods, such as other models based on spatio-temporal correlations.