H-Direct: Homeostasis-aware Direct Spike Encoding for Deep Spiking Neural Networks

Authors: Yechan Kang, Mingyeong Seo, Jeonghee Jo, Hyun Jae Jang, Jongkil Park, Jaewook Kim, Suyoun Lee, Jinkyu Kim, Seongsik Park
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: spiking neural networks; direct encoding; neuromorphic computing

Abstract:
Deep spiking neural networks (SNNs) have been expected to enable energy-efficient artificial intelligence as a next-generation artificial neural network. Recently, with the development of various algorithms, such as direct spike encoding, many applications have been successfully implemented in deep SNNs. Notably, most state-of-the-art deep SNNs have greatly improved their performance by adopting direct spike encoding, which expresses input information as discrete spikes, thereby exerting substantial influence. Despite the importance of the encoding, efficient encoding methods have not been studied. As the first attempt to our knowledge, we thoroughly analyzed the conventional direct encoding. Our analysis revealed that the existing encoding restricts the training performance and efficiency due to inappropriate encoding. To address this limitation by maintaining an appropriate encoding, we introduced a concept of homeostasis to the direct spike encoding. With this concept, we presented a homeostasis-aware direct spike encoding (H-Direct), which consists of dynamic feature encoding loss, adaptive threshold, and feature diversity loss. Our experimental results demonstrate that the proposed encoding achieves higher performance and efficiency compared to conventional direct encoding across several image classification datasets on various architectures. We have validated that brain-inspired algorithms have the potential to enhance the performance and efficiency of deep SNNs.

Decision: reject
Reviewer scores: 5, 3, 6, 3

Reviews and discussion:
<thread>
creation date:
1729043928156

thread_type:
official_review

<thread_object>
creator: reviewer_n9cn

<content>
summary:
The paper proposes H-Direct, a homeostasis-aware spike encoding technique for deep Spiking Neural Networks (SNNs). It incorporates mechanisms like dynamic feature encoding loss, adaptive thresholding, and feature diversity loss to improve the performance and efficiency of deep SNNs. While the method shows potential in improving spike encoding performance, there are several concerns that prevent this work from being accepted at this stage.

strengths:
The concept of integrating homeostasis to regulate spike encoding is interesting and innovative. The proposed approach seems to achieve meaningful improvements in terms of classification accuracy and energy efficiency in the experiments conducted.

weaknesses:
While the introduction of homeostasis in spike encoding is intriguing, the overall novelty of the approach is limited. Many of the techniques applied, such as adaptive thresholds and feature diversity loss, have been explored in various forms in prior works on SNNs and neuromorphic computing. There is insufficient discussion on how this method truly differentiates itself from previous work like direct encoding approaches and surrogate gradients. The proposed method lacks a strong theoretical foundation. Although the paper discusses homeostasis from a biological perspective, the paper does not thoroughly justify the mathematical and theoretical basis for why introducing homeostasis in the encoding stage would lead to significant improvements in SNN performance. A deeper explanation of the relationship between the proposed encoding mechanisms and their impact on network learning dynamics is needed.

The experiments focus heavily on classification tasks using typical SNN benchmarks (CIFAR-10, CIFAR-100, etc.), which limits the generalization of the proposed approach. There are no experiments on more challenging neuromorphic datasets or tasks beyond classification, which could better validate the robustness and applicability of H-Direct. The performance gains, while present, are relatively marginal compared to existing state-of-the-art methods, especially when considering the complexity introduced by the additional loss functions and adaptive mechanisms. The paper’s presentation could be significantly improved. The descriptions of the core methods (like dynamic feature encoding loss and adaptive thresholds) are somewhat unclear and hard to follow without repeated reading. Additionally, the figures do not always effectively support the key points being made and could benefit from clearer labeling and explanations. The related work section does not adequately cover recent advances in energy-efficient SNN training or alternative spike encoding methods. Key works on the use of advanced encoding strategies and their implications on energy efficiency should be cited and discussed.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729126430634

thread_type:
official_review

<thread_object>
creator: reviewer_ktle

<content>
summary:
This study addresses the limitations of existing direct spike encoding by first analyzing traditional direct spike encoding methods and categorizing the encoded spike channels into four types: over-activated, under-activated, dynamically selective, and persistent encoding. Subsequently, the concept of homeostasis was introduced into direct spike encoding, and this paper proposes a homeostasis-aware direct spike encoding method called H-Direct. This method achieves stable and appropriate encoding by suppressing over- and under-activation while encouraging dynamic feature selection.

strengths:
1. Experiments show that the proposed steady-state mechanism improves the training performance and efficiency of deep SNNs on various datasets, models, and training algorithms, demonstrating the effectiveness and universality of the method.
2. A coding method inspired by the human brain is proposed.

weaknesses:
1. This paper significantly reduces the number of spike data, but lacks theoretical power consumption calculation. Can it give specific power consumption results and compare them with some advanced works [1]?
2. The experimental effect of this paper is limited, and it is not compared with some advanced works [2][3] on many datasets.
I hope the author can introduce why H-direct works from a theoretical perspective.

[1] Yao M, Hu J, Zhou Z, et al. Spike-driven transformer[C]. Advances in neural information processing systems, 2024, 36.
[2] He H. Adaptive Spiking Neural Networks with Hybrid Coding[J]. arXiv preprint arXiv:2408.12407, 2024.
[3] Hu J K, Yao M, Qiu X, et al. High-Performance Temporal Reversible Spiking Neural Networks with $\mathcal {O}(L) $ Training Memory and $\mathcal {O}(1) $ Inference Cost[C]. Forty-first International Conference on Machine Learning.

questions:
see weakness below.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730484060652

thread_type:
official_review

<thread_object>
creator: reviewer_ey4j

<content>
summary:
This paper presents an analysis of direct encoding and introduces a concept of homeostasis to direct spike encoding, including feature encoding loss, adaptive threshold, and feature diversity loss. The experimental results demonstrate that the proposed encoding improves performance and efficiency, but performance improvements are limited.

strengths:
The direct encoding issue of SNN that the article focuses on is valuable, and the experiments are also sufficient.

weaknesses:
1. Many statements in the paper lack sufficient persuasiveness,  such as those described in the abstract and introduction.
- 'Despite the importance of the encoding, efficient encoding methods HAVE NOT BEEN studied.' 
As far as I know, the time-to-first-spike (TTFS) method that utilizes the first spike time to encode information is regarded as a highly energy-efficient encoding method.
- 'Thus, the energy problem has become the MOST urgent issue to be addressed for sustainable development and utilization of AI in our lives.'
Aside from the energy problem, the sustainable development of AI also faces multiple challenges, such as data security, interpretability, model generalization, and regulatory and formal laws.
- 'Direct encoding learns encoding methods from data, which leads to superior performance over other encoding approaches.'
Can the author elaborate on this?

2. The proposed method lacks sufficient innovation, since the approach of setting dynamic neuronal thresholds and adjusting loss functions to enhance performance is widely used.

3. The experimental results do not demonstrate a substantial improvement in accuracy, and as model size and dataset scale increase, this improvement diminishes further. This raises concerns about the practical utility of the proposed method.

4. Does the performance improvement depend on a large number of training epochs? Could the authors provide more experimental details? Additionally, does the introduction of the proposed loss function add computational overhead?

questions:
In addition to the problem mentioned in weakness, I have the following questions:
1. In Figure 1, in the image to the left of (b), the red box denotes OFE (Over-fired Encoding). Why is the RED box filled with ORANGE blocks (Orange represents normal spike counts)?

2. The criteria for the divisions are not clearly articulated. For instance, what spike count corresponds to DSE? Furthermore, when the spike count is zero, how to distinguish between OFE and DSE?

3. Given that the performance improvements demonstrated in the experimental results are not substantial, the motivation behind this work raises questions. Intuitively, not all spiking neurons contribute to the output; some neurons at specific pixels may not respond or fire, which is also a reflection of the “sparseness” advantage of SNN. Therefore, is it necessary to activate all neurons associated with OFE and UFE?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730643295573

thread_type:
official_review

<thread_object>
creator: reviewer_mhwu

<content>
summary:
This paper proposes a homeostasis-aware direct spike encoding method (H-Direct) for spiking neural networks (SNNs) to improve encoding efficiency and stability. The authors introduce three novel components: Dynamic Feature Encoding (DFE) Loss, Adaptive Threshold (AT), and Feature Diversity (FD) Loss. Extensive experiments across multiple datasets demonstrate the effectiveness of H-Direct in enhancing the performance of deep SNNs, and the ablation study further shows the contribution of each component.

strengths:
The experimental validation is thorough, with tests conducted on multiple datasets and network architectures, providing strong evidence for the proposed method’s efficacy. The authors also conducted an ablation study that helps to highlight the importance of each component within H-Direct, particularly in enhancing encoding stability and feature diversity. The overall approach is well-motivated and provides an innovative solution to improve SNN encoding by borrowing the concept of homeostasis from biological systems.

weaknesses:
While H-Direct addresses important encoding issues in the encoding layer, similar issues may also exist in deeper layers, raising the question of why these corrections are not applied throughout the network. Additionally, the ablation study in Table 4 only shows improvements on CIFAR-10 with VGG16, while ResNet20 shows decreased performance for certain modules. This inconsistency brings the role of each module into question. DFE is also heavily dependent on BN, raising concerns about its applicability to networks without BN. Further clarification on why FD Loss fits a probability density function and how H-Direct performs on architectures like RNNs or Transformers is also needed. If the authors are unable to sufficiently address these concerns, I may consider lowering my score. However, should the authors provide thorough explanations and additional experiments that address these points, I would raise my score.

questions:
1.	Why are the corrective mechanisms not applied beyond the encoding layer, given that over-firing and under-firing issues may also exist in deeper layers?
2.	In Table 4, why do certain modules of H-Direct decrease performance in ResNet20 compared to the baseline? Does this suggest limitations in module effectiveness across architectures?
3.	How would DFE perform in networks without Batch Normalization?
4.	What is the purpose of fitting FD to a probability density function, and which PDF is used? What is the impact of different PDFs?
5.	Does H-Direct show improvements in other architectures, such as RNNs or Transformers?
6.	Could the authors analyze the robustness of hyperparameters for the DFE, AT, and FD losses?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a direct spike encoding for spiking neural networks. The main motivation of the authors is to improve the efficiency and stability of the encoding method. The authors introduced the concept of homeostasis to direct encoding and introduced three new components: Dynamic Feature Encoding (DFE) Loss, Adaptive Threshold (AT), and Feature Diversity (FD) Loss. Experimental results show that H-Direct can achieve higher performance and efficiency for SNN.

After the rebuttal stage, 4 reviewers rate 3, 3, 5, and 6, respectively. Most reviewers agree that H-Direct has conducted sufficient experiments and has good research motivation. This is the advantage of H-Direct. However, there are some criticisms. Reviewers N9Cn and ey4J believe that many of the techniques applied in the method have been widely explored in other papers, and the method lacks unique contributions. In addition, reviewers ey4J and KtLe also believe that the performance improvement of the paper is limited compared with previous work. Reviewer MhWu pointed out that the method is highly dependent on the batch norm module to achieve generalization ability. I think these criticisms are reasonable. I suggest that the author further improve the clarity of the paper, the proposed method, and the presentation of experimental results.

One important discussion during the rebuttal period was about the generalization problem. Reviewer MhWu was concerned about the generalization problem and believed that the proposed algorithm lacked sufficient evidence to prove its generalization ability on SNN. In the reply, the author pointed out that other deep learning techniques such as KL divergence can also achieve similar effects. At the same time, it was also acknowledged by the authors that the method may have unexpected situations as the network depth increases. I think the author's efforts in replying to the reviewer MhWu's questions are worthy of recognition. At the same time, I also think that some of Reviewer MhWu's criticisms are still reasonable. The author should consider the potential impact of the proposed algorithm on network performance in depth. This will help the dissemination of the author's proposed method in the SNN community. In addition, the author should improve the article based on the reviewer's comments.

Overall, I think this paper needs further revision. Therefore, the final decision is to reject this paper.