Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: ## Edit on second review

I apologize again for the tone of my first review, I sincerely tried to understand the paper but I could not when I first read it. A re-read the paper and finally understood it during the review. I left a comment to the authors in the discussion below and they appropriately addressed my new recommendations. With the new equation (1) the paper is hopefully more understandable now. 

I increase my grade from 3 to 5. The findings are quite interesting but I still believe that the paper is not well written: the equations are interesting but the explanations between the equations are often unclear. One has to understand each equation and be quite imaginative to finally identify the contributions of the paper (even for somebody only "very slightly" off from the research topic).

## Summary

The authors suggest a relationship between a leaky relu and a spiking integrate and firing neuron model.
This relationship suggests a mapping between the two models which is imperfect, a loss seems to be derived to reduce this mismatch along the network training. The method is tested on CIFAR-10 and CIFAR-100, and compared with some other methods for converting ANNs to SNNs.

## Critical review

This topic is potentially important since spiking neural network are gaining popularity. But this paper is clearly badly written and it is extremely hard to understand, both in the math and in the text. I don't think it would help the progress of the field to publish the article in the current form.

I tried to read that carefully and got lost after equation (4), the transition to equation (5) and (6) are not clear at all. I do not understand what is an approximation, what is a definition and what is a derivation.

Also (5) seems wrong in itself, the authors are trying to approximate a rectified linear network but it suggest that the activity will be equivalent to a linear network (at least when v(T) is small) ? And magically this changes in (6), and a clip non-linearity is introduced ?

The Figure 1 seems very encouraging at first, because it suggests that there is a clear and easy mapping between accumulating the spikes and computing a relu. I did not understand where this is appearing in the math and I cannot check whether the intuition conveyed by the figure is correct or not.

I was therefore hoping to see an empirical study of the difference between the SNN and the ANN: do the activity of the spiking neural network match the activity of artificial network? This is not shown.  I do not even understand if it is necessary to re-train the network to go back from the SNN to ANN or vice versa.

Since I had not understood the basics of the paper, it was impossible for me to understand the later section about the conversion error. My only take is that it seems wrong at first sight: how minimizing the error in the loss would minimize the mismatch between the network activity?

review_title: Interesting (second review)

final_score: rating: 7: Good paper, accept
review_content: The authors seek a mechanism to train a spiking neural net to duplicate the function of a non-spiking one. This is desirable for energy-efficient inference, although the training process becomes challenging due to the discrete nature of the spiking process.

To achieve their goal, they described the spiking neuron non-linearity by a "staircase" function of the input (spiking output increases by 1 each time the input gets big enough to reach the next stair), and related that to the ReLu function used in the non-spiking neural net. They then determined parameters for the modified ReLU that would minimize the deviation between these activation functions, and computed the minimum conversion error (for converting ANN -> SNN). This scales with the square of the threshold voltage for spiking, divided by the simulation time. As one might expect, lower thresholds, and longer simulation times, both of which lead to potentially higher spike counts and thus lower discretization errors, lead to smaller conversion errors.

Using this, they defined their procedure for training SNN to mimic ANN as follows: they trained the ANN with their modified ReLU (which is closer to the SNN activation function but more readily differentiable), and then used the weights from that ANN in their SNN.

Next, the authors evaluated their procedure on several different image categorization networks. Nice performance was obtained in all cases: better than using a normal ReLU, or other comparison activation functions, in the "target" ANN.

Overall, this is a reasonably nice piece of work. I'd like to see this applied to recurrent neural nets: there, the dynamics of the SNN could be used more naturally, and the results might be more meaningful.

review_title: Training spiking neural nets to copy non-spiking ones

final_score: rating: 7: Good paper, accept
review_content: Strength: 
(1) This paper proposes a layer-wise optimization method for ANN-SNN conversion. I appreciate the theoretical analysis of how to minimize the conversion error.

(2) This work significantly reduces the simulation time since long simulation time is usually required for converted SNN to reduce error. 

(3) It's interesting that conversion to SNN actually improves rather than damage the accuracy on ResNet.

Weakness:
(1) Although the proposed method is much more efficient, it does not show obvious performance improvement compared to existing methods.

(2) In the experiment, the ResNet consistently show better performance. I hope the authors can provide more comments on this.

(3) I'm not sure if I missed anything. The threshold RELU is not defined in the paper which may cause confusion.

(4) I hope the authors can summarize the whole steps by formula or algorithm to help readers understand the entire process. 
review_title: This paper proposes a layer-wise decomposition of the conversion error method to optimize the ANN-SNN conversion. 

Meta-review: The work tackles the task to convert an artificial neural networks (ANN) to a spiking neural network (SNN). The topic is potentially important for energy-efficient hardware implementations of neural networks. There is already quite some literature available on this topic. 
Compared to these, the manuscript exhibits a number of strong contributions: It presents a theoretical analysis of the conversion error and consequently arrives at a principled way to reduce the conversion error. The authors test the performance of the conversion on a number of challenging data sets. Their method achieves excellent performances with reduced simulation time / latency (usually, in order to achieve comparable performance to ANNs, one needs to run the SNN for many simulated time steps- this simulation time is reduced by their model). 
One reviewer criticized that the article was hard to read, but this opinion was not shared by other reviewers and the authors have improved the readability in a revision.

In summary, I believe that this manuscript presents a very good contribution to the field.
Decision: Accept (Poster)