Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: I wanted to first thank the authors for their work on connecting biologically inspired CWA to deep learning. The paper overall reads nicely. 

The authors make the assumption that reviewers are fully aware of biological terminology related to CWA, however, this may not be the case. It took a few hours of searching to be able to find definitions that could explain the paper better. I suggest the authors add these definitions explicitly to their paper (perhaps in supplementary), also an overview figure would go a long way. I enjoyed reading the paper, and before making a final decision (hence currently weak reject), I would like to know the answer to the following questions:

1. Have the authors considered the computational burden of Equation 3? In short, it seems that there are two summations (one for building the probability space over measure h) and one right before e_j. This is somewhat important, if this type of neural network is presented as a competitor to affine mapped activations. 

2. It would be nice to have some proof regarding universal approximation capabilities of CWA. In my opinion it is, but a proof would be nice (however redundant or trivial - simply use supplementary). 

3. I was a bit confused to see CWA+BN in the Table 1. In introduction, authors write “But CWA networks are by definition normalized and range-limited. Therefore, one can conjecture that CWA plays a normalization role in biological neural networks.” Therefore, I was expecting CWA+BN to work similarly as CWA for CIFAR10. Please elaborate further on this note. 

4. Essentially, the CWA changes the definition of a layer in a neural net. Do authors see a path from “CWA works” to “CWA works better than affine?”. If so, please elaborate. Specifically, I am asking this question “Why should/must we stop using affine maps in favor of CWA?”. Now this may or may not be the claim of the paper. It’s ok if it is not; still showing competitive performance is somewhat acceptable, but certainly further insight would make the paper stronger. 

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper focuses on non-spiking Hudghkin-Huxley model, which is different from existing works on spiking neural network-based Hudghkin-Huxley model. 

There are many ways of using neuron firing model as unit to construct neural networks. They choose a specific way (mentioned above). I think the most interesting part would be the CWA method which achieves the normalization. 

They have a fair list of literature in spiking neural networks. But I find the way they illustrate the difference between their model and other models is insufficient. They should focus on the model-wise difference, instead of focusing on whether it’s applied to MNIST or not or what’s the accuracy. 

They don’t include any other SNN model in the paper for experimental comparison. They also mention a few SNN works that work well on MNIST in the related work section which actually have better accuracies than their model. So it is inappropriate to say this proposed method is a state-of-art neuro-inspired method, Because others perform well on MNIST as well, and their limited experiments only investigate MNIST and CIFAR-10, which are less interesting generally. 

CWA cannot outperform Affine+BN. 

Overall, the idea is somehow interesting, but the experiments are weak. Applying the method to MNIST and CIFAR-10 is far from being called either “interesting computer vision applications” or “difficult perceptual tasks”. They only use perceptual in the title, but the applications are MNIST and CIFAR-10. It feels like they want to learn something big, but they only focus on benchmark datasets. 

They compare nothing with other SNN type of model on other truly difficult perceptual tasks.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes a novel neural network architecture inspired by the analysis of a steady-state solution of the Hodgkin-Huxley model. Using a few simplifying assumptions, the authors use conventional backpropagation to train DNN- and  CNN-based models and demonstrate that their accuracies are not much lower than the state-of-the-art results.

The paper is well-written, sufficiently detailed and understandable. Derived self-normalizing Conductance-Weighted Averaging (CWA) mechanism is interesting in itself, especially contrasting CWA results with those obtained for the non-Batch-Normalized networks. It is also inspiring to see that this model can be derived based on a relatively accurate biological neuron model.

My main question is actually related to the potential impact of this study. I am curious about the implications and the ways in which these results can inspire other researchers.

After reading the paper, I got an impression that:

(a) From the point of view of a machine learning practitioner, these results may not be particularly impressive. They do hint at the importance of self-normalization though, which could potentially be interesting to explore further.

(b) From the point of view of a neuroscientist, the proposed model might be too simplistic. It is my understanding, that neural systems (even at "rest") are inherently non-equilibrium (and I assume the presence of simple feedback loops could also dramatically change the stead-state of the system). Is it possible that something similar to this "steady-state inference" mode could actually take place in real biological neural systems?

(c) Presented results appear to be important from the point of view of someone who wants to transfer insights from biology into the field of deep learning. But there might be an extent to what is achievable given a simple goal of optimizing a supervised accuracy of an artificial neural network trained using gradient descent (especially considering limitations imposed by hardware). I am optimistic about the prospect of knowledge transfer between these disciplines, but it is my feeling that the study of temporal dynamics, emergent spatiotemporal encodings, "training" process of a biological neural system, etc. have potentially much more to offer to the field of machine learning. These questions do appear to be incredibly complex though and the steady-state analysis is definitely a prerequisite.
review_title: Official Blind Review #3

Meta-review: The paper studies non-spiking Hudgkin-Huxley models and shows that under few simplifying assumptions the model can be trained using conventional backpropagation to yield accuracies almost comparable to state-of-the-art neural networks. Overall, the reviewers found the paper well-written, and the idea somewhat interesting, but criticized the experimental evaluation and potential low impact and interest to the community.  While the method itself is sound, the overall assessment of the paper is somewhat below what's expected from papers accepted to ICLR, and I’m thus recommending rejection.
Decision: Reject