Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Pros:
1.	A novel coding scheme is proposed based on Discrete Cosine Transform (DCT) for efficient information expression in place of conventional Poisson distribution method. The required time-steps are 2-14x reduced compared with other conversion-SNNs or hybrid trained SNNs.
2.	DCT is data-independent while performing at par with PCA.
Cons:
1.	The experimental results are not convincing enough.
2.	Correctness Problem. I am afraid that the descriptions about the reconstruction of input image is wrong. The reverse transform should be a matrix multiplication instead of Hadamard product of the coefficients and the basis. For example, in Fig 1., X=Y_1*T_1+Y_2*T_2+…+Y_5*T_5. It results in conceptual errors in Fig1 & Fig2.
3.	The specific equation of DCT should be with the discussion of the desirable properties and constraints in the section of encoding scheme to provide a clear picture of the method. 
About the experimental results:
1.	Could the authors try to provide some explanations for why DCT is able to outperform Poisson method or directly exposing the original image to the input spike neuron, since DCT’s reverse transform is a reconstruction of the original image over time?
2.	Fewer time-steps with relatively lower accuracy is kind of confusing. I would like that the authors could further show the required time-steps for reaching strictly equal (or better) accuracy results with other SNN works (especially those directly trained). It is because that the trade-off between accuracy and the number of time-steps is natural in SNNs. For example, Rathi et al. (2020) could increase their VGG16’s performance from 91.13% to 92.02% by adding 100 time-steps on CIFAR-10. 
Rathi, Nitin, et al. "Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation." arXiv preprint arXiv:2005.01807 (2020).

3.	I notice that you cite Wu's article in related works, but there is a lack of comparison to it in later experiment part. In my opinion, the results of the paper and its sequel on time-steps for training SNNs from scratch are worth-noticing. I would like a more comprehensive and fair comparison of results.
Wu, Yujie, et al. "Spatio-temporal backpropagation for training high-performance spiking neural networks." Frontiers in neuroscience 12 (2018): 331. 
Wu, Yujie, et al. "Direct training for spiking neural networks: Faster, larger, better." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019. 

Clarity: The paper is fairly well-written.
Originality: DCT is a widely used transformation technique in signal processing and data compression, but this paper creatively explores it as a coding scheme in SNNs.
Significance: The proposed coding scheme might provide a new resolution to the high inference latency bottleneck in SNNs.

review_title: A new coding scheme based on DCT for SNNs, but not convincing enough.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The scheme proposed breaks down the information in a block of an image into orthogonal basis functions (DCT is used) to make a progressively better reconstruction of the original image block with the addition of more basis functions used (like an nth order Taylor expansion).  The increasing spatial frequency components are known to be perceptually less sensitive (they need to include this) in images, so the low freq components can be presented first.  Each freq component is encoded into spikes sequentially, thereby staging the more perceptually important information first, with less important info coming later.  This reorders the presentation of information to allow a tradeoff of image quality with time/latency.

I think the solution proposed is well-founded and will indeed mitigate the latency problem for spiking neural networks.  However, this feels a bit more like an engineering solution to a specific problem rather than a new concept.  I do like the injection of methods from other fields like image/video compression; it often feels that the deep learning field rediscovers things that have been uncovered years ago in other fields.  I see that as the main value of the paper in addition to helping to make spiking neural networks a POSSIBLE viable solution to edge deployment.  

Section 1: I don’t think it’s a strongly supported claim that deep learning architectures are unsuitable for edge deployment.  There are plenty in deployment and there are new processors (Movidius, Mythic, etc) that can handle these computations for real-time applications.  I’d suggest a softer language there.  This does weaken the motivation for the paper though.

Section 1, second paragraph:  typo: Thy -> The

Section 3.2: On constraints for the transforms.  Did the authors consider Integer Transform (IT)?  This is used in MPEG/AVC.  It is a reversible transform that is an integer simplification of the DCT.  Given that the point of the paper is to decrease latency and computing requirements for edge deployments, this could help.

Section 3.2: The authors do a good job of sweeping performance for different block sizes.

Figure 5:  Isn’t it an obvious result that more time steps are required for Poisson vs DCT?  There simply aren’t enough bins to sum over to have a result until a certain point.  

review_title: This paper discusses a method to make spiking networks more relevant to latency-sensitive applications on the edge. I believe the authors' method is relevant to this problem, but doesn't feel like it uncovers anything fundamentally new.  It is an engineering solution to a specific problem; and spiking networks are not a widely used method at the edge currently.  

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes an encoding method based on the Discrete Cosine Transform (DCT) for Spiking Neural Network (SNN). The key idea is to decompose an image into different frequency components and feed them to the SNN sequentially. Compared to the Poisson coding method used in most SNN studies, the proposed encoding method significantly decreases the latency that the SNN needs for image classification while having minimal accuracy decease. 
 
Highlights:

1. The idea of using DCT for input spike encoding is novel and has great potential. One of the problems that prevent the SNN from using fewer inference timesteps is the ineffectiveness of encoding input information. Using DCT, the method can potentially filter out less important information and more effectively encode the information in limited timesteps (as shown in Fig. 6 and Fig. 8 in the paper).

2. The paper doesn't directly learn in the frequency domain generated from DCT. Instead, it reverse transforms the DCT result back to the spatial domain and spreads it into different timesteps of the SNN. By doing so, the spike encoding gives more importance to the low-frequency information in the image. This is desirable because low-frequency information is more important than high-frequency information in the image for classification.
 
Concerns:

1. The paper lacks experiments to show that DCT directly contributes to the decrease of timesteps for classification. Although comparisons with earlier SNN works that use Poisson encoding are shown, there is a lack of comparison with any SNN methods that directly convert pixel values into spikes using IF neurons and threshold selection. Thus, the existing experiments are not sufficient to exclude the possibility that the latency decrease is not due to DCT. The reviewer suggests conducting additional experiments for this.

2. While the proposed method only focuses on the input encoding of SNN, many recent papers target new training methods (such as [Jibin Wu et al, 2019], [Sen Lu et al, 2020]) that also result in significant latency decrease of SNN for image classification. The paper lacks experiments to compare the performance with these more recent results. The reviewer suggests conducting additional experiments for this.

3. The paper claims that the proposed method has better performance than ANNs trained on DCT coefficients. However, this is not a fair comparison since their encodings are different. The ANNs trained on DCT coefficients (such as [Max Ehrlich et al, 2019]) follow the same procedure as JPEG compression. The encoding uses non-overlapping 8x8 blocks, and the ANNs directly learn from the JPEG transformed domain. If the paper wants to compare with these ANNs, it needs experiments using the same encoding input.

4. The example in Fig. 1 for the reverse transformation is not the same as the source code (spike_model_vgg9_submit.py: Line 233). In section 3.1, the paper performs inverse transform by doing an element-wise multiplication between the transformed vector Y and each frequency basis in the transformation matrix, and claims the same method generalizes to the 2D case. However, the source code performs inverse transformation using the particular element in the transformed matrix Y (it's now a matrix but not a vector) corresponding to the specific frequency basis. Thus, the explanation in the paper contradicts the implementation. The reviewer thinks the example given in Fig. 1 is mathematically incorrect.

5. The proposed method spreads the reverse transformed image into different timesteps, and each timestep corresponds to a particular frequency. However, the paper doesn't explore other possible approaches for spreading the information. For example, the encoding method can use multiple subsequent timesteps for a particular frequency or only present intermittent frequencies. The lack of in-depth analysis of the proposed method possibly prevents the paper from fully exploring the potential of the use of DCT encoding for SNN.

Minor Comments:

1. What is the meaning of "ov" in Fig. 4? The reviewer thinks it means "overlap". However, it needs to be explained in the text or figure caption.

2. In Table 2, it's not clear whether DNN-d uses DCT coefficients or the reverse transformed image. If the DNN-d uses the DCT coefficient, is there any change to the ConvNet since the DCT destroys the block's spatial relationships?

3. In Table 2, the SNN-d results for TinyImageNet are missing. Is there any reason for that?


Jibin Wu et al, 2019, A Tandem Learning Rule for Effective Training and Rapid Inference of Deep Spiking Neural Networks

Sen Lu et al, 2020, Exploring the Connection Between Binary and Spiking Neural Networks

Max Ehrlich et al, 2019, Deep residual learning in the jpeg transform domain

 Since most of my primary concerns are resolved, I have updated my rating based on the revised version.
review_title: Novel input spike encoding method for SNN but lacks crucial experiments to support the conclusion

final_score: rating: 6: Marginally above acceptance threshold
review_content: Authors applied this algorithm into several datasets such as CIFAR 10, CIFAR 100 and TinyImagenet, they argued the image classification accuracies onto these datasets were comparable. The experiments were sufficient but not well designed. The reported experimental results show some novelty and good performance.

In the early stage of the spiking neural networks, the encoding methods are very important, especially for the training process. As the authors said that the rated based encoding method brings much more time latency which is time consuming. Therefore, the topic of this work is critical. However I have some worries about the proposed methods, from my point of view, this work is just combing the DCT and ANN-SNN method, the novelty is significantly limited, but the idea is interesting. Then, the experimental results reported by this paper were not well designed. The authors argued that the rated based encoding methods are time consuming, but they just did not compare the much more temporal encoding methods in Table 3. And for me, CIFAR10, CIFAR 100 and MNIST are in the same quantity level, which means that you used a VGG net is waste of resource (VGG is too deep). I even did not know what parts of the final results works, the deep CNN based network architecture? The DCT encoding method? Or the ANN-SNN methods. Actually, ANN-SNN method is not a typical bio-inspired way to construct a SNN, I prefer to see you adopt the proposed method into a Tempotron or STDP based learning rule not a surrogate-gradient based rule. The computational efficiency is nice; especially the authors calculated the spike rate of each single layer, but if you just argued the proposed the method is energy consumption, you should at least consider the ANN training process, it is not a single trade-off between inference accuracy and latency. I have run the code from authors provided, the reproducibility is reliable.  

Also there are some writing errors, such as thy->they, -s, etc. and reference missing, such as these important works:
1.	An FPGA Implementation of Deep Spiking Neural Networks for Low-Power and Fast Classification.
2.	Deep CovDenseSNN: A hierarchical event-driven dynamic framework with spiking neurons in noisy environment 

review_title: his work utilizes DCT based encoding method into spiking neural networks. This encoding method combines the mathematics of DCT and bio-plausibility of spiking neurons (LIF) to enrich the correlations in the spatio-temporal domain of spikes, which is quite inspiring.

Meta-review: This paper provides a method of encoding inputs to a spiking neural network (SNN) using the discrete cosine transform (DCT). The goal is to create a more energy and time efficient means of doing inference with SNNs. The authors provide a description of the method, then show accuracy results on a variety of standard benchmarks. They also compare to a number of other methods for ANN and SNN based inference in the literature. Altogether, they show that their method allows for accurate inference using fewer spikes than other approaches, which can potentially reduce the energy used for inference.

The paper is fairly clearly written, and the results well articulated. The reviewers had a number of concerns, most notably related to questions of (1) clarity about the actual benefits of this approach, and (2) fair comparisons to other models. Altogether, the authors did try to address the reviewers comments, and at least one reviewer increased their score. 

However, the actual scores for this paper remained very close to the acceptance threshold, and the first point was difficult to rebut without a lot more added to the paper. Ultimately, this paper is using a classic signal processing strategy to improve SNN run time, and the reviewers asked for some reason as to why that is desirable/novel. The author's answer was effectively that SNNs provide promise for low-energy edge computing, and their method could make SNNs for edge computing even more efficient.  This is potentially of interest for edge computing, but the paper could do a lot more to demonstrate that. Specifically, some consideration of how this would actually operate on spiking chips or a more robust estimate of energy efficiency than that given at the end of section 4 would be required to make this paper a clear accept. Notably, the paper does not demonstrate that this technique could be used to significantly reduce the energy requirements for spiking chips, relative to other SNNs, just that this is more energy efficient than ANNs, which is already known for other spiking neural network approaches. Given this, and the scores relative to other papers at ICLR, a "reject" is recommended. However, the AC notes that this was a difficult decision, and this paper was right at the threshold.
Decision: Reject