Reversing Structural Pattern Learning with Biologically Inspired Knowledge Distillation for Spiking Neural Networks

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-31 09:07:39

Abstract: Spiking neural networks (SNNs) have superb characteristics in sensory information recognition tasks due to their biological plausibility. However, the performance of some current spiking-based models is limited by their structures which means either fully connected or too-deep structures bring too much redundancy. This redundancy from both connection and neurons is one of the key factors hindering the practical application of SNNs. Although Some pruning methods were proposed to tackle this problem, they normally ignored the fact the neural topology in the human brain could be adjusted dynamically. Inspired by this, this paper proposed an evolutionary-based structure construction method for constructing more reasonable SNNs. By integrating the knowledge distillation and connection pruning method, the synaptic connections in SNNs can be optimized dynamically to reach an optimal state. As a result, the structure of SNNs could not only absorb knowledge from the teacher model but also search for deep but sparse network topology. Experimental results on CIFAR100, Tiny-imagenet and DVS-Gesture show that the proposed structure learning method can get pretty well performance while reducing the connection redundancy. The proposed method explores a novel dynamical way for structure learning from scratch in SNNs which could build a bridge to close the gap between deep learning and bio-inspired neural dynamics.

Main paper: 
 introduction: 
 Spiking Neural Networks (SNNs) are supposed to be one of the efficient computational models because of their biological plausibility, especially since the structures are dynamically malleable. Cognitive activities can be realized with the help of the complex structures of the human brain which is composed of billions of neurons and more neural connections between them. Similar to biological neural networks, SNNs transfer and process information via binary spikes. During the flow of the information, those fired spikes are transmitted on the synaptic connection between neurons with the structural plasticity rules.Compared to conventional artificial neural network models including convolutional neural networks (CNNs), definitely, SNNs are more biologically inspired and energy efficient. However, there is one fact that cannot be ignored the SNNs cannot achieve nearly as good performance as ANNs did. One of the key factors that we think some of the structures of current spiking based models are limited by training methods, that is SNNs cannot leverage the global backpropagation (BP) rules directly as CNNs did. This defect directly leads to unreasonable structures in SNNs. Besides, whether structures of CNNs or SNNs are both fixed, unlike biological neural systems, the structures and topologies should have been dynamical which means the connections could be discarded as needed.Aiming at tackling this problem, some studies focus on constructing more efficient structures to improve the efficiency of SNNs. Researchers proposed an approximate BP method named STBP (Spatio-temporal backpropagation) for training high-performance SNNs [25] . To avoid training SNNs directly, some studies [6, 11] proposed ANN-to-SNN get parameters from trained ANNs, then map them to SNNs. Although these methods could construct pretty deep structures, they bring additional computational power. Moreover, these BP based learning rules can only match the fixed structure, if changing the network topology, it would be retrained from scratch.Aiming at constructing more biologically flexible SNNs, this paper proposed biologically inspired structure learning methods with reverse knowledge distillation (KD). Based on the proposed training method, the wanted student SNN models could learn rich information from teacher ANN models [26] . Compared to traditional KD methods, one of the key differences of the proposed re-KD method for SNN training is that we think the structures play an important role in the training process, they are not only the final results, they could instruct themselves to train. Not limited to label smoothing regularization [29] , this paper proved that the proposed re-KD method could build more robust structures of SNNs. We evaluated the proposed methods on several pattern recognition datasets (CIFAR100, Tiny-imagenet and DVS-Gesture) experimental results show that the proposed methods can not only get good recognition performance but also show robust generation ability in time-series dynamic environments. The main contributions in this paper are summarized as follows:• This paper proposed reverse KD methods for constructing efficient structures of spiking neural networks. The proposed methods are emphasized to circumvent the no-differentiable of the spikes when using BP rules directly. • The proposed methods let sparse structures models as teachers help construct robust student SNN models. Besides, this paper provides a brandnew teacher-free KD method which could help student SNN models absorb useful information when the teacher is default. • Experimental evaluations showed that the proposed re-KD method could facilitate the performance of SNNs. Furthermore, this kind of KD construction for deep SNNs could allow teacher and student models to be homogeneous or heterogeneous, which shows great potential on neuromorphic hardware platforms. 
related work and motivation: 
 There lacking suitable structures for deep SNNs, which caused difficulties in training deep SNNs using BP rules directly. Based on the KD method, this paper proposed reverse-KD method to further facilitate the brain-inspired characteristics of dynamic spiking signals.The proposed reverse-KD methods include two ways, one is when the structures of the teacher model are sparse named sparse-KD, and the other is when the teacher is the default during KD training process named teacher default-KD. For further play and to verify the effectiveness of the proposed method, this paper embedded surrogated gradient method into the proposed method to train the deep SNNs.Compared to the structures in deep learning field, those of SNNs are various because there are no unified learning rules for deep SNNs training as BP did in deep learning. Deng et al. [5] , Xu et al. [27] proposed temporal efficient training methods to build deep SNNs. Chen et al. [2] dug out the state transition of dendritic spines to improve the training efficiency of sparse SNNs. Some studies focused on the activation function in SNNs [1, 31] to build high brain-inspired spiking neurons and neural circuits.Considering more detailed differences between ANNs and SNNs [3, 19] , some researchers are committed to improving the network structure so that to make SNNs could be applied to detection [13] and dynamic time-series data processing [12] . Although they can build efficient deep structures of SNNs, they also bring huge power consumption and additional computing resources. Especially, when they used ANN-to-SNN conversion [1, 21] . Besides, these structure construction methods have overlooked the flexibility of structure creation which means if the task changed, you must train another brand-new SNN from scratch.Different from ANN-to-SNN conversion [7, 11] , because of the huge success that BP gets in deep CNNs, some studies also want to utilize the BP-based rules to train deep SNNs. Neftci et al. [18] firstly introduced surrogated gradient training method to mimic the backpropagation process in CNNs. These types of methods aimed at the no-differentiable problem in binary spikes, by applying surrogated gradient to make spikes become differentiable so that the corresponding weights can be trained globally.Zenke and Vogels [30] analysed the robustness of surrogated gradient training in SNNs and instilled the complex function in SNNs. Combing the characteristic of membrane potential, some scholars want to incorporate learnable membrane time constant to enhance the surrogated gradient training [6] . Others made a further improvement to make it friendly to neuromorphic on-chip hardware systems [23] .Although these surrogated training methods consume little power consumption compared to ANN-to-SNN conversion, they still bring additional computational resources. More importantly, they still make the structures of SNNs fixed and cannot change with the change of tasks. To build more flexible structures of SNNs, combing with surrogated-gradient training, some studies proposed knowledge distillation-based methods to build more efficient SNNs [15, 16, 24] . They normally set a powerful ANN as a teacher and a shallow SNN as a student, it will make unrealistic assumptions and introduce more computing consumption when training a strong ANN additionally.Aiming at tackling the aforementioned problems in constructing efficient deep SNN models, this paper proposed a reverse KD method to construct deep SNNs. Through re-think the original KD in SNNs, this paper makes a systematic analysis of the knowledge transfer between teacher and student.This paper not only focused on the efficient structures of SNNs, but also care about the power consumption brought by KD process. Specifically, we build two re-KD methods for deep SNN construction, one is sparse-KD and the other is teacher default-KD. One of the key innovations is that we think the teacher is not always strong meanwhile, the student is not always weak. By building the relationship between weak teachers and strong students, this paper rethinks the KD process in SNN training and proved that based on the proposed method, the performance would be improved meanwhile the power consumption decreases. 
method: 
 In order to construct a bio-inspired training method for combining structural learning and knowledge distillation for SNNs, this paper proposes the re-KD method. Our method presents two approaches for knowledge distillation: sparse-KD and teacher default-KD. This knowledge distillation framework explores a method of knowledge distillation that utilizes a network with a sparse structure or a virtual network as the teacher network, which is different from conventional knowledge distillation methods.Spiking neuron model. We use IF (Integrate-And-Fire Models) spiking neuron models as the basic unit of the network. The IF neuron model is one of the commonly used spiking neuron models. It stimulates the process of action potential in biological neurons, where it receives spiking stimulation from presynaptic neurons and the state of membrane potential changes. The neural dynamics equation of the IF neuron can be represented by Eq. (1) .EQUATIONwhere 𝑉 (𝑡) is the membrane potential of IF neuron. 𝑋 (𝑡) denotes the current input at time 𝑡. Therefore, the current membrane potential of the IF neuron can be expressed as Eq. (2) .EQUATIONWhen the membrane potential reaches the threshold, the neuron will fire a spike, and then its membrane potential will be reset to the reset voltage 𝑉 reset . In addition to this, the framework trains SNN based on gradient surrogates.Overall framework of the reverse knowledge distillation method. Re-KD explores a novel way of knowledge distillation, which combines bio-inspired structure and distillation. A reverse knowledge distillation approach is adopted to train the SNN student network. As shown in Section 1, the pruned SNN model is used as a teacher network to guide the training of the student network, which can reduce the interference in the hidden information of the teacher network to better train the student network. As shown in Section 1, a 100% accuracy virtual teacher network is designed to avoid the impact of wrong classification better. Then this framework adopts a response-based knowledge distillation method to train a student network.Teacher sparsified. This method uses a sparse network as the teacher network which is called sparse-KD. We use a weight-based pruning method to obtain a sparse network structure as the teacher model to guide the training of the student model. This sparse method prunes a certain proportion of connections with low-weight values from a pre-trained SNN model. There is often redundancy in the connections within a network structure, which is similar to that of a biological neural network. Removing these redundant connections makes the network structure more robust. First, we train an SNN model and fix the weights of the model. Then, we sort the weights of all connections in the convolutional layers of the model and prune the weights with the smallest values according to a certain proportion. This method sets a mask that is multiplied by the weights, with the elements in the corresponding positions of the mask matrix set to 0 for the connections with weights that need to be pruned. The weight and pruning mask in 𝑙 layer is 𝑊 𝑙 and 𝑀 𝑙 and the weight after pruning can be expressed as Eq. ( 3):EQUATIONLoss function. This method uses a knowledge distillation method based on response, where the final layer output of the teacher network is used to guide the training of the student network. In this case, the teacher network refers to a pre-trained SNN model that has been pruned. The learning objectives for the student network are divided into soft labels and true labels. Soft labels refer to the output of the final layer of the teacher network, which contains hidden knowledge. In order to increase the entropy of the probability distribution of the network's output, a temperature parameter 𝑇 is introduced to smooth this probability distribution. This allows for better learning of the hidden similarity information in the probability distribution of the teacher network's output. The probability distribution is represented by 𝑍 𝑖 . The flatten output probability distribution 𝑞 𝑖 can be expressed as Eq. ( 4):EQUATIONThe loss function for knowledge distillation during training of a student network consists of two parts. One is the loss calculated using cross-entropy between the output of the student network and the true labels. 𝑄 𝑠 and 𝑄 𝑡 is the probability distribution of the student and teacher network's output. The other is the loss calculated using KL divergence between the output of the student network and the soft labels. In the 𝐿 𝑠𝑜 𝑓 𝑡 function, the probability distributions of the outputs of the student network and the teacher network are both flattened by parameter temperature. The soft loss is computed by the simplified KL divergence. In the 𝐿 ℎ𝑎𝑟𝑑 function, the loss is computed by the CrossEntropy. The student network can be trained using the loss function in Eq. ( 5), as referenced in paper [17] .𝐿 𝑠𝑝𝑎𝑟𝑠𝑒 -𝐾𝐷 = 𝛼 * 𝐿 𝑠𝑜 𝑓 𝑡 (𝑄 𝑠 , 𝑄 𝑡 ) + (1 -𝛼) * 𝐿 ℎ𝑎𝑟𝑑 (𝑄 𝑠 , 𝑦 true ) (5)where 𝛼 controls the importance of the two parts of the loss function. 𝑦 true denotes the true labels.Training algorithm. The first step is to train an SNN model and then prune the model based on the weights and save it as the teacher network. In the second step, we use an SNN model with the same structure as the teacher network as the student network. Then we use knowledge distillation based on response to train the student network. Through this method, the clearer hidden knowledge in the pruned teacher network is utilized to guide the training of the student network.Teacher default. This method uses a probability distribution designed manually as the teacher network, which allows the teacher network to have an accuracy of 100%. In normal knowledge distillation, the accuracy of the teacher network is usually not 100%, so there may be some interference in its hidden knowledge. Using a pruned teacher network for knowledge distillation can yield a more robust student network. In order to better guide the learning of the student network, a 100% accurate virtual teacher network can be designed. Assuming there are 𝐶 classification categories, this probability distribution sets the probability of the correct label 𝛼 (𝛼 > 0.9), and the probability distribution of this virtual teacher network can be represented by Eq. (6) .EQUATIONwhere 𝑐 represents each category and 𝑡 is the correct category. The probability distribution 𝑝 (𝑐) can be viewed as the probability distribution of the output of this virtual teacher network.Loss function. This method is similar to the response-based knowledge distillation method. The loss function of the student network training is divided into two parts. One is the cross-entropy loss between the output of the student network and the true label. The other is the KL divergence loss between the output and the virtual label. The probability distribution of this virtual teacher network can be flattened using the parameter temperature to obtain the soft label. The two parts are summed to obtain the loss function of this knowledge distillation, as shown in Eq. (7) .𝐿 𝑑𝑒 𝑓 𝑎𝑢𝑙𝑡 -𝐾𝐷 = 𝛼 * 𝐿 𝑣𝑖𝑟𝑡𝑢𝑎𝑙 (𝑄 𝑠 , 𝑄 𝑣 ) + (1 -𝛼) * 𝐿 ℎ𝑎𝑟𝑑 (𝑄 𝑠 , 𝑦 true ) (7)where 𝑄 𝑠 is the output of student network. 𝑄 𝑣 denotes the probability distribution designed manually after flatting by parameter temperature.Training algorithm. The first step is to select appropriate parameters to design this probability distribution to obtain a virtual teacher network with completely correct outputs. The second step is to use the virtual teacher network to guide the training of the student network through knowledge distillation. The virtual teacher networks have a 100% accuracy and do not have interference from incorrect classification, so they can better guide the training of student networks. 
experimental results: 
 We conducted experiments on the static dataset CIFAR100 and Tiny-imagenet and on the neuromorphic dataset DVS-Gesture to verify the effectiveness of this framework. Firstly, we analyze the impact of different pruning ratios of the teacher network on the performance of the student network. Then we verify the performance of knowledge distillation with a virtual teacher network. In addition, in order to validate the advantages of this re-KD method, we compare the experimental results of this framework with other SNN methods.We conduct the experiments on the server which is equipped with 16 cores Intel(R) Xeon(R) Gold 5218 CPU with 2.30GHz and 8 NVidia GeForce RTX 2080 Ti GPUs. The training of SNN is based on the spikingjelly framework, which is an SNN framework developed based on pytorch. Our experiments mainly focus on residual structures such as Resnet and WideResnet. In this experiment, the teacher network adopts the same structure as the student network in order to better verify the advantages of structure learning.Dataset CIFAR100 and Tiny-imagenet. CIFAR100 is a commonly used static dataset, it has three channels RGB and the image size is 32*32. The CIFAR100 dataset has 100 classes, and each class has 500 training sets and 100 test sets. The image size of Tinyimagenet is 64*64 and it has 200 classes, each class has 500 training sets and 50 test sets. Each image in both datasets has fine-labels and coarse-labels two labels. The dataset is relatively complex and can verify the performance of our proposed framework in a more realistic way. During training, we need to first encode the static images into spiking sequences and then input them into the SNN, here the first spiking neuron layer in the network is regarded as the encoding layer.Dataset DVS-Gesture. DVS-Gesture is a neuromorphic dataset that includes 11 gestures for recognition. The dataset is stored in the form of events and needs to be integrated into frame data before use. During training, the spiking sequences data obtained can be directly input into SNN. The dataset has 2 channels and the image size is 128×128.We use the SNN student network itself and the SNN student network with different pruning ratios as the teacher network and then analyze the experimental results. In this experiment, we used the spiking form of Resnet18, WRN16-4 and 5Conc-1FC ( [6] ) network structures to conduct the experiments.For the CIFAR100 dataset, as shown in Table 1 , using the SNN student network itself as its own teacher network for knowledge distillation improves the accuracy of the student network by 1.33% (Resnet18) and 2.46% (WRN16-4). The experiments also show that using models with different pruning ratios of the student network itself as the teacher network can also improve the accuracy of the student network. However, using a pruned model as a teacher network for knowledge distillation is more effective than using a non-pruned model as a teacher network. As can be seen in Table 1 , for Resnet18, when using a model pruned at 0.1 as the teacher network, the accuracy of the student network can be improved by 1.49%, which is greater than 1.33%. For WRN16-4, when using a model pruned at 0.1 as the teacher network, the accuracy of the student network can be improved by 2.60%, which is greater For the Tiny-imagenet dataset, as shown in Table 1 , using the SNN Resnet18 student network itself as its own teacher network for knowledge distillation improves the accuracy of the student network by 1.71%. The teacher network pruned at 0.1 imporves the accuracy of the student network better by 1.96%. When the teacher model is pruned at 0.1 and the accuracy is only 35.81%, it still can conduct knowledge distillation training on student networks. To better compare the effectiveness of knowledge distillation using different pruning ratios and unpruned teacher networks, we visualize a line graph of test accuracy. As shown in Figure 2 , the blue line represents the accuracy of the student network before any pruning, while the orange line represents the accuracy of the student network using the unpruned model as the teacher network. The other lines above the orange line represent the accuracy of the student network using different ratios of pruning as the teacher network.For the DVS-Gesture dataset, as shown in Table 1 , using the SNN student network itself as its own teacher network for knowledge distillation improves the accuracy of the student network by 1.04%. When pruning the teacher network at 0.1, the improvement effect on the student network is more obvious, and it improves by 1.39%.Using a pruned student network or the student network itself for knowledge distillation is different from the general perception of knowledge distillation. The accuracy of the pruned student network may be slightly lower than that of the student network, but experimental results show that it can still improve the accuracy of the student network. This shows that the pruned model, by reducing some of the interference from redundant connections, can better transfer effective hidden knowledge to guide the training of the student network.We use a virtual teacher network to perform knowledge distillation. The experiment is based on the spiking form of Resnet18, WRN16-4, and VGG16 network structures. The virtual teacher network is a probability distribution designed by humans, with an accuracy of 100%.The effectiveness of using virtual teacher networks for different models varies, as shown in Table 2 . On the CIFAR100 dataset, using knowledge distillation with a virtual teacher network on a WRN16-4 model improves the accuracy of the student network by 2.22%. For the Resnset18 model, the student network's accuracy is improved by 0.80%. On the Tiny-imagenet dataset, the student network's accuracy is improved by 4.35% which is greater than sparse-KD method. On the DVS-Gesture dataset, the student network's accuracy is also slightly improved over 0.34%.Using a virtual teacher network to perform knowledge distillation on a student network is an effective way to improve the accuracy of the student network. The virtual teacher network does not have any output errors and can better guide the training of the student network. Since the training of a greater teacher SNN is difficult, this method is also suitable for situations where it is difficult to find a model with better performance as a teacher network for knowledge distillation.We visualize the output distributions of the teacher network's last layer in different methods. For sparse-KD, as shown in Figure 3 , we can observe that the pruned teacher network's output (blue line) is relatively smoother compared to the unpruned teacher network (orange line), and it contains inter-class information (e.g., class 24 "clouds" and class 50 "mountains"). As for default-KD, we can see that the probability distribution designed by the virtual teacher network (green line) is mostly consistent with that of the original teacher network (orange line).As shown in Table 3 , in order to better analyze the effectiveness of the method, we compare it with some existing methods. For the CIFAR100 dataset, the sparse-KD and default-KD methods using a pruned model with a pruning rate of 0.3 as the teacher network on the Resnet model have an accuracy of 72.83% and 72.10%, respectively, which is better than the Rmp-snns [9] , Hybrid [20] , Opt.[4] and TSC [8] methods. For the Tiny-imagenet, the accuracy of our method is better than LTL [28] with less time steps. For the DVS-Gesture dataset, the sparse-KD method using a pruned model with a pruning rate of 0.1 as the teacher network on the 5Conv-1FC structure has an accuracy of 96.18%, and the default-KD method has an accuracy of 95.13%, both of which are better than the SLAYER [22] and Com. [10] methods. Compared to the PLIF [6] and STBP-TdBN [32] methods, the accuracy is slightly lower, but the time step used is 16, which is shorter than them. 
conclusion: 
 Inspired by the biological plausibility of neural systems in human brain, this paper proposed efficient structure learning methods with reverse knowledge distillation for SNNs. This kind of method could help to build a deep but sparse structure under the guidance of the pruning method which could not only discard the redundancy of the complex spiking neural dynamics but also save power consumption and memory storage in SNNs.Considering the abnormal KD cases such as the proposed sparse-KD and teacher-default KD methods, experimental results showed that we can expand our work to broader conditions in SNNs especially when the teacher model is weak. It also showed that the proposed models not only get good performance on a relatively large dataset (CIFAR100 and Tiny-imagenet) but are also suitable for the dynamic scene (DVS-Gesture). Under strict timesteps, the proposed method can help SNNs get good performance compared to other spiking based models. That is to say, the proposed methods could give full play to the advantages of low power consumption of SNNs.In our future work, we will expand the structure learning methods to utilize the advantages of spiking signals, not limited by the proposed two teacher-weak conditions, we will consider more situations when the teacher model is ill or disabled and evaluate them on more types of datasets. 
 
 Appendix:  


Decision: From the final rating, four reviewers gave a Weak Accept, while one gave a borderline reject. Overall, there is broad agreement that this paper presents a novel method for improving the efficiency and performance of SNNs. It introduces a structure construction method that dynamically optimizes synaptic connections in SNNs, making them more biologically plausible and efficient. Experimental results on datasets like CIFAR100, Tiny-ImageNet, and DVS-Gesture demonstrate that the proposed method enhances performance while reducing connection redundancy. Therefore, I recommend the acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes an efficient structure learning methods for Spiking Neural Networks (SNNs) with reverse knowledge distillation. By integrating knowledge distillation and connection pruning techniques, the synaptic connections within SNNs can be dynamically optimized to achieve an optimal state. This allows the SNN structure to not only learn from a teacher model but also to explore deep yet sparse network topologies. Experiments conducted on datasets such as CIFAR100, Tiny-ImageNet, and DVS-Gesture demonstrate that the proposed structure learning method achieves competitive performance while reducing connection redundancy. This innovative dynamic approach to structure learning in SNNs helps bridge the gap between deep learning and bio-inspired neural dynamics.
strength_weakness: strength_weakness: 1. The authors introduce advanced techniques such as structural learning and knowledge distillation into Spiking Neural Networks (SNNs), making this a novel, interesting, and worthwhile topic to explore.
2. The motivation, relevant background, and methodological details are presented by the authors with great clarity and thoroughness.
limitations: limitations: 1. Although the authors conducted experiments on image datasets (i.e., CIFAR100 and Tiny-ImageNet) and an event dataset (DVS-Gesture), these datasets are relatively small in scale. Specifically, for the DVS-Gesture dataset, most existing methods already achieve over 0.95 accuracy, making it challenging to differentiate between methods. It is suggested that the authors experiment with larger-scale event datasets, such as the ASL-DVS dataset.
2. Further ablation studies are needed, for instance, on the hyperparameter 𝑎 in the loss function, as different values might impact the final performance.
3. The authors should report the inference time or algorithmic complexity of different methods.
4. One of the main advantages of Spiking Neural Networks is their low power consumption when deployed on neuromorphic chips. Currently, most algorithms include power consumption analyses. The authors should theoretically analyze the feasibility of deploying their operations on neuromorphic chips and provide theoretical or experimental statistics on power consumption.
5. The authors are encouraged to further improve the writing of the paper, including the text, figures, and tables. Additionally, the official limit allows for up to 8 pages of main content, so the authors have room to include more details in the paper.
paper_summary: paper_summary: This paper designs a novel method for improving the efficiency and performance of SNNs. Here are the key points from the paper:

 **Proposed Solution**: The paper proposes an evolutionary-based structure construction method that combines knowledge distillation and connection pruning. This approach aims to optimize synaptic connections in SNNs, aligning them more closely with the efficient neural topologies observed in the human brain.

**Benefits**: By integrating these methods, the structure of SNNs can be refined to learn from teacher models and maintain efficient sparse network topology. This reduces computational redundancy and enhances performance.

**Experimental Results**: Tests on CIFAR100, Tiny-Imagenet, and DVS-Gesture datasets demonstrate that the proposed learning method achieves significant performance improvements while reducing connection redundancy.
strength_weakness: strength_weakness: 1. Dynamic Structure Learning: The approach allows SNNs to dynamically adjust their structure during the learning process.
2. Reverse Knowledge Distillation: Unlike traditional knowledge distillation, this method trains the student SNNs by learning from the teacher ANNs in a reverse manner, focusing on structural efficiency.
limitations: limitations: The main limitation is there's no experiments on large dataset like ImageNet-1K. While there are already lots of SNNs working on ImageNet with direct training method, this new KD method could make a comparison.
paper_summary: paper_summary: This paper proposes sparse-KD and teacher default-KD to enhance the performance of spiking neural networks, and conducts extensive experiments on static and neuromorphic datasets to validate the effectiveness of the proposed methods.
strength_weakness: strength_weakness: Inspired by biological neural systems, sparse-KD attempts to construct a deep but sparse SNN by using the pruned model as the teacher to guide the training of the unpruned model. The teacher default-KD allows for direct distillation without additional teacher models by manually defining soft labels. The authors conducted extensive experiments on three datasets to demonstrate the advantages of the proposed methods in improving the performance of SNNs.
limitations: limitations: 1. In sparse-KD, the authors prune the trained SNN as the teacher and take the SNN model with the same structure as the student for distillation (line 352). However, it is not clear from the paper whether the student model is randomly initialized or has the same weights as the trained but unpruned model, and it is hoped that the authors will clarify this.
2. Table 1 shows that using a model with a pruning rate of 0 as the teacher can also improve the performance of the SNN model. The teacher and student at this point have exactly the same architecture and parameters, and their output logit is the same. In this case, a distillation loss of 0 would not cause the error gradient to propagate, so why would it improve performance?
3. In the experimental section, the authors should compare the proposed method with recent SOTA methods.
paper_summary: paper_summary: This paper presents a novel method for structuring Spiking Neural Networks (SNNs) using biologically inspired knowledge distillation. The method proposes an evolutionary-based structure construction to reduce redundancy and dynamically optimize SNNs. It integrates knowledge distillation with connection pruning, allowing SNNs to learn from a teacher model and search for a deep yet sparse network topology. Experimental results show the method enhances performance while reducing connection redundancy.
strength_weakness: strength_weakness: This paper has a new insight on the training process that jointly considering structural learning and distillation learning.

This paper introduces a structure construction method that dynamically optimizes synaptic connections in SNNs, making them more biologically plausible and efficient.

Experimental results on datasets like CIFAR100, Tiny-imagenet, and DVS-Gesture demonstrate that the proposed method enhances performance while reducing connection redundancy.
limitations: limitations: In Section 3.2, why the teacher model needs to be sparse is not clearly explained. The authors claimed that removing these redundant connections makes the network structure more robust. While the authors are not mentioning what pruning methods they are using.

In Section 3.3, the authors introduce a way to construct a teacher model that is 100% correct. I am confused about how to achieve this goal using the technique in Section 3.3 and how to integrate this technique into the distillation framework.

Texts in Figures 2 and 3 are not readable.

One extra question: if using a sparse teacher model for distillation, will the student SNN be sparse?
paper_summary: paper_summary: This paper proposes an evolutionary-based structure construction method for constructing more reasonable and high performance SNNs. The reverse-KD framework explores a method of knowledge distillation that utilizes a network with a sparse structure or a virtual network as the teacher network, including two approaches: sparse-KD and teacher default-KD. This method offers a new approach to knowledge distillation in the SNN field by training student networks to enhance their performance under resource-constrained conditions.
strength_weakness: strength_weakness: The paper introduces a novel method for training SNNs through reverse knowledge distillation. It utilizes a weak teacher network to train a student network, effectively enhancing the student model's performance. This approach is particularly beneficial in scenarios where it's challenging to find a higher-performing teacher network under resource-constrained conditions. It leverages the low-power and biologically interpretable advantages of SNNs and presents a new way of absorbing hidden knowledge in teacher networks through learning.
limitations: limitations: The paper has some shortcomings as follows:
1. In the paper, the motivation behind proposing this reverse knowledge distillation method is not sufficiently clear. It needs to elaborate on the benefits of introducing this distillation approach for training SNNs.
2. The paper fails to provide a reasonable explanation for the biological basis mentioned in the abstract. Are there any related references that could elucidate this aspect?
3. There are some spelling and grammar errors in the paper. Some of the illustrations are also rather blurry.