LEARNING DIFFICULT PERCEPTUAL TASKS WITH HODGKIN-HUXLEY NETWORKS
Authors: Alan Lockett
Ankit Patel
Paul Pfaffinger
Venue:  (2020)
Keywords: conductance-weighted averaging
neural modeling
normalization methods

Abstract: This paper demonstrates that a computational neural network model using ion channel-based conductances to transmit information can solve standard computer vision datasets at near state-of-the-art performance. Although not fully biologically accurate, this model incorporates fundamental biophysical principles underlying the control of membrane potential and the processing of information by Ohmic ion channels. The key computational step employs Conductance-Weighted Averaging (CWA) in place of the traditional affine transformation, representing a fundamentally different computational principle. 
Importantly, CWA based networks are self-normalizing and range-limited. We also demonstrate for the first time that a network with excitatory and inhibitory neurons and nonnegative synapse strengths can successfully solve computer vision problems. Although CWA models do not yet surpass the current state-of-the-art in deep learning, the results are competitive on CIFAR-10. There remain avenues for improving these networks, e.g. by more closely modeling ion channel function and connectivity patterns of excitatory and inhibitory neurons found in the brain.  

Under review as a conference paper at ICLR 2020
LEARNING HIGH-DIMENSIONAL PERCEPTUAL TASKS WITH BODGKIN-HUXLE NETWORKS
Anonymous authors Paper under double-bling review
ABSTRACT
This paper demonstrates that computational neural network model using ion channel-based conductances to transmit information can solve standard computer vision datasets at near state-of-the-ar performance. Although not fully biologically accurate, this model incorporates fundamental biophysical principles underlying the control of membrane potential and the processing of information by Ohmic ion channels. The key computational step employs Conductance-Weige (CWA) in place of the traditional affine transformation, representing fundamen- tally different computational principle. Importantly, CWA based networks are nelf-normalizing and range-limited. We also demonstrate for the first time that a network with excitatory and inhibitory neurons and nonnegative synapse strengths can successfully solve computer vision problems. Although CWA models do not yet surpass the current state-of-the-ar in deep learning, the results are competitive on CIFAR-10. There remain avenues for improving these networks, egg. by more closely modeling ion channel function and connectivity patterns found in the brain.
Averaging
INTRODUCTION
Deep learning has achieved groundbreaking advances in computer vision, but modern artificial neural networks have largely developed independent of neuroscience research on biological neuronal networks. Many of the core features found in biological neural networks have never been integrated successfully into artificial neural networks, raising serious questions of whether the success of deep learning is relevant to neuroscience research vice versa.
This paper presents an artificial neuronal network incorporating features found in the classic Hodgkin- Huxley equations that relate neuronal membrane potential to the activity of ion channel conductance pathways. A primary feature of this approach is that the values that the neuronal elements of the network can take are limited and self- neff-normalizing. In addition, each neuron is classified as either excitatory or inhibitory, an aspect of biological neural networks that until now has never been effectively utilized in artificial neural networks. Here we demonstrate that more biologically accurate networks can learn complex computer vision datasets such as CIFAR- 10 competitively with the state-of-the-are just a few years prior. Biological neurons do not compute an affine transform of their inputs. Rather, the synaptic inputs compete against other active conductance pathways in the neuron for the control of the membrane potential and thus the firing state of the neuron. The valence of these different pathways is reflected by the zero-current potentials of the types of channel conductances being regulated, which determines whether the pathway excites or inhibits the neuron. The combination of these conductances into a single membrane potential is analogous to performing a conductance weighted averaging (CWA) of these zero-current potentials.
The concept of neuron valence is related to the types of synaptic channels that a neuron regulates in its post-synaptic neurons. Because neurons primarily release one type of neurotransmitter, they functionally fall into one of two classes: Excitatory Inhibitory neurons.
In this paper, CWA similarly uses two valences, (+1) for neurons forming excitatory connections and (-1) for neurons forming inhibitory connections. Importantly, the combining of these pathways into single neuron using CWA is constrained to produce neuronal outputs that lie somewhere between these two extremes, and thus the output of CWA is bound by the range of the valences.

---

Under review as a conference paper at ICLR 2020
Normalization and regularization strategies, such BatchNorm (offe & Szegedy 2015) or DropOut (Srivastava et al. 2014), have been repeatedly shown to improve the quality of artifi- cial neural networks. But CWA networks are by definition normalized and range-limited. Therefore, one can conjecture that CWA plays a normalization role in biological neural networks. As seen in Table [??] bare CWA without additional normalization regularization is competitive with standard affine networks that employ these strategies, especially in the case of fully connected networks (FCN), whose neurons, like biological neurons, are characterized by having thousands of inputs.
In sum, neural networks with CWA provide a more biologically accurate model with excitatory- inhibitory neurons and conductance- based normalization that succeeds on computer vision datasets without extra normalization and regularization, with implications for neuroscience and deep learning.
2 ABSTRACTING FROM HODGKIN-HUXLEY
Hodgkin and Huxley described the change in voltage within a neuron as a function of the ion channels active in the neuron (Hodgkin & Huxley, 1952), for which they received Nobel prize. Their model remains the foundation of modern understanding of biological neurons, but deep learning was never based directly on this model. In this section, a steady- abstraction of the Hodgkin Huxiey model is developed, compared to standard (i.e., affine) artificial neural networks, and used to derive the CWA neuron as an abstraction.
2.1 STEADY-STATE CONDUCTANCE MODEL
The information state of neuron is encoded by the voltage on its surface membrane. The surface membrane is generally analogous to a capacitor; on its surface are many ion channels that transfer charge across this capacitance, allowing the movement of ions into or out of the cell. Depending on the ion species of channel, the membrane potential will move in a direction determined by the equilibrium potential of that species. Ion flow changes the voltage across the membrane and is the key mechanism of information transfer within the brain. The strength of a synapse depends on the number of ion channels at the synapse, the ion species of the channel, and the overall activity of other pathways. Electrochemically, the net effect is a conductance-wwiijt weighted average of the equilibrium potentials for all the ion channel pathways in the neuron (see e.g. Kandel al. (2013). Ch. 5).
In the Hodgkin-Hux always moves in the direction that will reduce the net ionic current towards zero. The voltage and current are time-varying quantities governed by a differential equation, but they can be converted iime-invariant, steady-state model by presuming that the the inputs the network are constant. Because ion movement is linked to changes in the charge of the membrane capacitance, the steady- state membrane potential is found by letting the net ionic current go to zero.
model, the membrane potential (the voltage difference at the membrane)
For linear Ohmic conductance pathways, this stable membrane potential is determined by the conductance-wette weighted average of the zero-current potentials of the ionic pathways present in the neuron. For single-chamber neuron i, and letting ~ i range over presynaptic neurons j, the voltage V; of can written in simplified form as:
V, E, + Ejn Gisp,Ej
VV-- - E
V
where =o=
Gr + Eyrs GisP3
This steady-state equation is the expected value of probabilistic model driven by neural spike trains. The value Pj represents the probability that the pathways leading out of neuron are open, that is, the synapses out of j are releasing transmitter onto the post-synaptic ion channels. The exact computation of Pj is the only element of Equation not well defined by biophysics; in steady-state (non-spiking) models, it is commonly modeled as a logistic function or a ReLU. The logistic function in Equation] has spike threshold E/ and is scaled by fixed scaling constant kj. The G.,j terms describe the maximum conductance possible along the pathway from neuron j to i, which occurs when p,- = 1.
The term G, represents the resting ceff-conductance of the neuron, and E,- the self celf-potential absent synaptic activity (see e.g. [Gabbiani & Cox (211)) Chs. 2,11; Koch & Segev 1999), Chs. 1,2). These interactions are shown in Figure]]
2

---

Under review as a conference paper at ICLR 2020
eutiallory
Vi
Inhilitorr Irftamnee
Exciattory Infloceue
BedducaanceWeiihhee
GB
GO
GB
GO
GD
Ei
E
Firing Probability
P
PH
P
BV
P
Contingent
Inhibitory (B) Ros)
Excitatory Ea))
Neurans
Neurons
The steady state approximation of biological neuron i with excitatory valence. The neuron has its voltage VÚ determined as in Equation by the firing probabilities Py of the contingent neuron, each of which has an influence gated by the conductance Gij of the synapse ij and the firing probability Py- This influence is either inhibitory (negative) or excitatory (positive) depending on the inherent, fixed state of the contingent neuron. There is also resting inhibitory bias.
Figure 1:
The F. terms the zero-current potentials for each pathway, specific to the type of neuron For example, pyramidal neurons produce excitatory glutamatergic synapses yielding larger Eú values, but lateral inhibitory interneurons produce GABA-ergic synapses with Eg values below the threshold.
At behaviorally relevant time scales, changes in are solely responsible for neuronal membrane potential dynamics. Key factors causing changes in Pj include changes in synaptic activity for neurotransmitter gated channels changes in gating of the other membrane channels due to voltage or intracellular messengers. For neurons, the importance these changes in membrane potential that they regulate neuronal spiking and hence neurotransmitter release onto follower neurons.
At larger time scales, the Gt,j adapt to control how the neuron processes information based on the addition or removal of channels; the method of this adaptation is as yet not well understood.
All of the terms in Equation have fixed sign: G Py. E,. G,,, E,, E and kj. In this model, synapse is inhibitory if the incoming zero-current potential F. is less than the outgoing threshold Eth in which case j has an influence on that tends to push the spike probability below 1 Thus neuron types with E, less than the threshold are described inhibitory, and those with E, above the threshold are excitatory. Biological neural networks have several types of inhibitory and excitatory neurons, but every outgoing synapse of an excitatory neuron has an excitatory effect on post- synaptic neuron, and conversely every outgoing synapse of inhibitory neuron has an inhibitory effect. This partitions the network into classes of neurons with valences E, determined by the class.
2.2 COMPARISON AFFINE NEURAL NETWORKS
Artificial neural networks were originally modeled on biological neurons (Rosenblati 1958) but with simplifications. particular, they assume that the total conductance is fixed (Shriki al. 2003). In analogy to Equation] [ the typical artificial neuron is described by
2; b; + Σ Wt,Pi where Pj = o
(2)
where o is an activation function, typically sigmoid such as the logistic or tanh. Because z; is an affine transformation of the inputs Pj. such networks will be referred to affine networks below.
Due to the long history of affine networks within machine learning, their connection with neuroscience is now obsolete, with advances in neuroscience having relatively impact the machine learning community. The disconnection is further enhanced by the success of deep learning in computer vision and natural language processing, in comparison neuroscience simulations which typically just seek
3

---

Under review as a conference paper at ICLR 2020
to mimic some aspect of experimental observations. One of the goals of this paper is to demonstrate experimentally that networks of CWA neurons are potentially as powerful as affine neural networks.
Equation has similarities to Equation
! but two aspects of Hodgkin-Huxley model are missing:
1. A conductance-weiggtee average (CWA) controlling neuronal activity.
2. zero-current potential making neurons either excitatory or inhibitory.
With regard (1), conductance-weighter averaging has normalizing influence, limiting voltage to a range determined by the zero-current potentials E,. Numerous methods deep learning impose normalization neural activation, but this biological mechanism has not been extensively explored.
The nature of the CWA operation is fundamentally distinct from the operation of an affine net. In Equation 1 the output Pj of pre-synaptic neurons is only used in order to determine the conductance weighting, and the post-synaptic voltage the weighted average of the incoming zero-current potentials Ej. which divides the neurons into excitatory and inhibitory classes, in point (2) above.
Previous attempts to implement Equation] have been limited in scope and results. We demonstrate that abstraction incorporating points (1) and (2) above can solve computer vision problems
level that exceeds the state-of-the-20 in affine networks from just a few years prior.
2.3
CONDUCTANCE- WEIGHTED AVERAGING
From Equation the Bodgkin-Huuxlee neuron computes weighted sum of the fixed Ej values, i.e.
Gi.pP
V
Σ Wi gm
where W1.j = EES GIAPE
Here wiy reflects both inputs and parameters. For an input vector p this equation computes probability distribution u(p) and takes the expected value over vector of fixed values, labeled
h(96)) h(gik) pk
= By(p) [e] =
( g022
The parameters 9ty R vary across the real line, and the function h R [0, o0) enforces nonnegativity. Equation is obtained by setting Gt,, = h(gg)) and supposing a connection to a bias node with h(grr) G, with p, 1. Given that P5 E [0, 1], it follows that 2; is necessarily range-limited, falling between the minimum and maximum values of ej. For simplicity, set
e = 2 Ej Emin
-1
Emax Emin
so that e, and 2, fall in the interval [-1, 1]. Equation 3 is Conductance Weighted Averaging or CWA.
Notice that Equation 3 is self-normalizing. The output necessarily lies in the range spanned by the ej. so if e, 1, 1]. then 2, E [-1, The factor μ(p), imposes normalization. The dependence on Pj determines which of the e, to listen to, and the output is a weighted average of these values.
Affine neural networks alternate between affine transformation and non-linear activation. Equation is already non-linear transformation due to the denominator, and one might wonder whether non-linear activation is then necessary. In preliminary experiments, omitting a further activation function was sufficient to solve the MNIST digits problem with 98% accuracy with fully connected architecture, but obtaining good performance with more complex architectures and datasets did in fact require non-linear activation to silence minimally activated neurons, providing the benefit of sparse neural activation, known to contribute to good generalization affine networks.
Rather than implementing P; as sigmoid based on Eth and ky, we observe that the biophysics is not specific here and employ a simpler scaling and clipping function:
if otherwise
P: = (z))
This equation justified by the fact that as an expected value 2; is bounded by the fixed values of the vector so that E 1). Clipping off the negative values yields p; E [0, 1] as desired;
4

---

Under review as a conference paper at ICLR 2020
it also results in ReLU activation (Nair & Hinton 2010). The biological appropriateness of this simplification would depend on (1) the extent to which the spiking probability of biological neurons approximates clipped linear function of membrane potential, and (2) whether there is functional significance to the balance of positive and negative ej with respect to the spike threshold in biological systems. These questions will not be resolved here, but different biological activation function would likely not substantially change the overall computations resulting from Equations 3 and 4
Now suppose in the extreme that rather than choosing ej anywhere in the interval [-1, 1]. one enforces either ej = +1 or
-1. If +1 neuron j is excitatory or posilive, and if ej -1, neuron inhibitory or negative. This restriction loses no mathematical power of the system. As long as u()) can represent (nearly) arbitrary probability distributions, then 2 can result in any point in the interval [-1, 1] using just ej E [-1,+1} Of course, in biological neural networks, there are several distinct values for ej, not just two. It is not clear yet whether this distinction is computationally significant or merely a result of biological accident. Nonetheless, in implementing CWA, the experiments below divide the neurons up into two groups of positive/excitatory and negative/inhibitoyy neurons.
To explain these networks, Equation can be rewritten as
1
Σ positive
Pj (94)
Σ negutive
Pÿ h (9rs)
(S)
(:(D)
where (.(p) = hy is the normalizing value. From an intuitive perspective, the left term as sums up the positive evidence and the right term sums the negative evidence. the negative evidence predominates, then z; is negative. If the positive evidence predominates, then is positive. The parameters Pú control which evidence is listened to by neuron i.
Once 2, is thought of computing evidence for something, the role of neuron within larger network is to represent that something as discrete feature. If 2; > O, the feature is observed to degree determined by the magnitude of z4. If 24 S O, then on balance the feature has not been observed.
To summarize, CWA is an abstraction based on the Hodgkin-Huxll model. Its purpose to demonstrate that networks using cenduuttanceweigghee averaging and partitioned into excitatory and inhibitory neurons can solve complex computer vision datasets. This is a more realistic model of biological neural networks than the affine model, but abstracted from reality in the following ways:
1) Non-synaptic channels (i.e. G,) are not dynamically gating and so their p values are unversally 1. They behave similarly to the bias term in affine networks but also provide mathematical anchor point for the normalizing sum.
2) Values for the zero-current potentials are scaled to lie in the range of +1, and the channel conductances Gi, are represented by real-valued proxies Pij-
3) Synaptic channel activity probability for neuron is controlled by the membrane potential of the presynaptic neuron, and is modeled by a typical computational neuronal network activity function as in Equation facting on the presynaptic neuron's 's membrane potential.
4) Only two types of zero-current potentials are used, +1 for excitatory neurons and -1 for inhibitory neurons.
The next sections discuss related work and the provide experimental results for CWA on computer vision tasks using classic fully-comnected and convolutional architectures.
3 RELATED WORK
Spiking Neural Networks. Spiking neural network (SNN) models simulating the membrane poten- tial as a differential equation have been applied to classification problems Pfeiffer & Pfeil (2018). The best supervised test accuracy for SNNs on MNIST is 99. % with convolutional SNN trained using backpropagation Lee et al. (016)) Several methods for SNNs achieve 98- 98.99% on MNIST (ulkammi & Rajendran| 2018 Diehl et al. 2015; Kheradpisheh et al. 2018). SNNs have also been CIFAR-IO 10 with test accuracy of up to 92.37% (Hu et al. 2018) Sengupta et al. 2018). However, although SNNs compute with biologically realistic spikes, of these models apply conductance
5

---

Under review as a conference paper at ICLR 2020
averaging is present in biological neurons, and they rely on Batch inssiire layers. Huh & Sejnowski (2018) published a method for supervised training in SNNs with a conductance based rule, but they did not test the model on computer vision datasets. Diehl & Cook (2015) simulated SNNs with conductances and excitatory-inhibitory neurons on MNIST, reaching 95% accuracy on MNIST with unsupervised technique. Diehl al. (203)) used spiking affine networks, but with a renormalization scheme that divided all weights in layer by the maximum positive activity to prevent unbounded spiking. This method is loosely reminiscent but distinct from CWA. achieved 99.15% test accuracy on MNIST in convolutional SNN, similar to the 99.02% reported below.
ExcitatoryLnnhiiioryy Networks. Few classification results have been reported using neurons divided into excitatory and inhibitory groups. Diehl & Cook (2015) has already been mentioned. Delahunt & Kutz (2018) modeled the olfactory bulb of moths and applied to MNIST with 75-85% accuracy. /hhu et al. (2017) introduced a method that modeled excitatory neurons as ReI.U activated and inhibitory neurons as activated with a negative-valuee saturating ReLU, attaining 94.22% on CIFAR-10 with networks containing both excitatory and inhibitory activations inside of an otherwise standard CNN; this result slightly exceeds that obtained with their model for excitatory-only networks, suggesting that there is value in using multiple neuron types.
Normalization and Regularization Methods. CWA keeps neural activity within a fixed range independent of the synapse strengths. Within deep learning, is well known that large weights and activations can be a symptom of overfitting the training data. Techniques to control this include 1.-regularrzzaton (Ng) 2004) of weights (i.e., weight decay) and activities. An older technique, the Neocognitron, explicitly normalizes neural activities, inspired by the concept of complex vs. simple cells in the brain Fukushima et al. (988). Dropout Srivastava al. (2014) was proposed of balancing the dependence of a neuron over its inputs, which also helps to prevent large variations in weight and activation magnitude. Krizhevsky et al. (2012) introduced Local Response Normalization as a method of locally smoothing neural activity in the hidden layers of a CNN. It was superseded by BatchNorm (Toffe& & Szegedy 2015), which explicitly centers and rescales neural activities to a standard Gaussian. Most frae-offthe-arr results in deep learning rely heavily on BatchNorm to keep neural activities well-conditioned and to improve the curvature of the gradicnts during learning.
method
One hypothesis regarding conductance networks is that by imposing finite range the neural activities, CWA may play a role in the brain similar to that BatchNorm within artificial neural networks. An advantage of CWA over BatchNorm is that it does not require extra parameters and does not impose requirements on how the training data are sampled in order to train the network. This last point is relevant since BatchNorm makes importance sampling difficult, which has prevented its use, for example, within deep reinforcement learning, where priority sampling is commonly applied. CWA could be used to fill the role of BatchNorm in these cases; best as is currently known, the human brain is trained via reinforcement learning.
EXPERIMENTAL RESULTS
In order to assess its experimental value, CWA can be used as drop-in replacement for the classical affine transformation inside of existing fully-connected and convolutional architectures. The resulting artificial neural networks and some variants have been tested on two image classification tasks, MNIST and CIFAR- 10. The architectures, datasets, and results are described in this section.
4.1 COMPUTER VISION DATASETS
MNIST (LeCun & Cortes 2010) consists of a training set of 60,000 images of handwritten digits from to 9, with a test set of 10,000 images. All images are 28 x 28 pixels in size, and the dataset is balanced among the ten digits, so that each digit has 6,000 labeled training examples and 1,000 labeled test examples.
CIFAR-10 10 (Krizhevsky et al. 2009) contains 50,000 images for training and 10,000 for test, where the images are divided into 10 balanced classes such as airplanes, cars, trucks, and dogs. Each image in CIFAR10 10 has 32* x 32 pixels.

---

Under review as a conference paper at ICLR 2020
In the experiments below, the only data augmentation applied to CIFAR-10 was random shifting by up to pixels followed by cropping, and random horizontal flipping of the image. This augmentation regime is sufficient to train common architectures to over 90% accuracy.
Preprocessing. Ordinarily, images are preprocessed before presentation to neural network. This processing typically includes some form of whitening contrast normalization, which makes the images more consistent and easier to classify. In the experiments below, only minimal preprocessing was performed. For CWA, the color channel values were scaled lie between [-0.3, 1] and then clipped to [0, 1]. Much like in Equation 4 this clipping was done to promote sparsity and had a positive effect on learning speed and generalization accuracy. For affine networks, the color channels were mean-centers centered and variance-scaled. This whitening could not be used for CWA because CWA assumes that the inputs lie in the interval [0, 1].
Although these datasets are not sufficient to establish state-of- oftt-offthe-ar performance for CWA, they do comprise strong bank of tests to establish that CWA can solve computer vision tasks the level of the ofate-of-the-art for affine networks from just 2-3 years prior.
4.2 ARCHITECTURES
One of the key advances in neural networks in the past decade has been the development of novel architectures, including the layout of artificial neurons and the introduction of components to address Cnormattio-processing concerns. For CWA, three common architectures have been adapted for experiments the datasets.
Fully Connected Network (FCN), FCNs consist of several layers of neurons with ferw-forward computation. For CWA, the experiments partition each layer into excitatory and inhibitory neurons, with half the neurons in the layer being assigned to each category. The neurons in the input layer doubled, with one excitatory copy and one inhibitory.
Neural Network (CNN), Convolutional neural networks adapt a feed-forward net- work to incorporate local connections (receptive fields) and shared weights. The experiments below use sequence of four or five convolutional layers with stride 2 and no pooling. To implement CWA-CNNs, the simplest adaptation is assign each presynaptic channel valence (i.e., zero- current potential) in the context of the receptive field. For the experiments, half of the channels were considered excitatory and half inhibitory. To adapt the input image, the input channels were doubled from three to six, with two channels for each color, one excitatory and one inhibitory. Using tensor notation, for the standard convolutional filter one writes
Convolutional
i = b W p.
which is adapted to the conductance network as
8 G,e, + (G@c)
(6)
G++
where Gijay ((ggtay) is the max conductance parameter, X indicates componentwise multiplica- tion (also known as the Hadamard product) and
Cijzy +1 if < J/2
otherwise.
where is the number of presynaptic channels to layer i, i.e, half the channels are excitatory and half inhibitory. Here G @ is modeled rather than p & to reflect the concept that the neuron valence determined by the receptive field, but the two concepts are of equivalent significance given the definition of e above.
Residual Networks (ResNet). Residual Networks (He et al., 2016) are variant of convolutional neural networks characterized by blocks of successive convolutions with shortcut connections. This paper uses the ResNet-18 architecture without BatchNorm but with CWA convolutions as in Equation T and activated according to Equation
4.3 TRAINING RESULTS
Each of the three architectures was tested on both datasets for both affine networks and CWA. For clarity, the affine networks used ReL.U activation (Nair & Hinton 2010) and were initialized with
7

---

Under review as a conference paper at ICLR 2020
MNIST
CIFAR-TO AFFINE No BN
Architecture
CWA
AFFINE BN
CWA
CWA+BN
AFFINE + BN
FCN
98.65
98.72
61.98
64.93
65.10
63.93
CNN
98.89
99.42
78.09
73.40
88.60
RESNET- 18
99.02
99.59
85.06
79.60
91.25
Table 1: Test Accuracy for CWA and Affine Networks on Two Datasets.
Glorot initialization (Glorot & Bengio, 2010). Two variants of affine networks were tested. The first variant (Affine, No BN) took the same unwhitened, scaled inputs as CWA and did not use batch norm. In the second variant (Affine + BN), image inputs were whitened and each ReLU activation was preceded by an application of BatchNorm. The purpose of training affine networks is to obtain meaningful baselines for comparison. In one case, CWA was tested with BatchNorm applied similarly (CWA+BN) The training procedures are described below, followed by the results.
Initialization. The 9ij 's and 9ijay's 's comprise all of the parameters in conductance network. These parameters were Glorot-initialized, so that the initial G.,, parameters varied tightly around 1. Unlike with standard neural networks, CWA parameters can be initialized to zero without preventing learning, although we did not do so. To enforce the nonnegativity of Gt.j. h(g:t) = (1 + 9.5]+ was used in Equation 3 Although this function is discontinuous, by initializing the Ptj close zero, a wide range is available for optimization, and the linear scaling of the gradient proved to be advantageous from a numerical perspective. By contrast, h exp could have been used, but it caused overflow later stages of training in some cases during preliminary experiments.
Bias Term. The bias conductance G, h(g.) and its valence e,,, representing the neuron's zero-current potential, were fixed and not adapted during training. Specifically, G, = was used, and e, -0.4 was used for FCN but e, 0 for CNN and ResNet. By fixing these values, the denominator in Equations 3 and [ could never produce division by zero. The inhibitory bias for FCN was determined experimentally order produce a tendency towards not firing in the zero-input state. In convolutional architectures, setting to any value other than zero caused training to fail in preliminary experiments for unknown reasons.
The output of an affine network on categorical networks is traditionally determined by applying softmax activation, P(c,)) x exp(2:), in place of the standard activation. Biological networks do not seem to have categorical outputs of this form. Thus in order to compute a categorical output, final affine layer with softmax activation was added as the top layer of each CWA architecture to extract the categories from the CWA network.
Categorical Output.
Parameter Fitting. All networks, both CWA and affine networks, were trained with backprop using mini-batch stochastic gradient descent with the Adam optimizer (Kingma & Ba 2014) to control per- per-parameter learning rates (31 = 0.9, 32 = 0.99). Lower initial learning rates (1) Ie 5) were found to be necessary in most cases for CWA, and the resulting parameter learning
noticeably
slower than for affine networks, which were trained with initial v = 0.001. Backprop was used purely mathematical method of demonstrating that CWA networks do indeed have the capacity to fit the
data.
Training Procedures. Each network was trained until it had fit the training set to high accuracy. In all cases the training accuracy was > 99.4%. At each epoch of training, every training image was presented to the network in minibatches. At the end of each epoch, the training loss was computed, and if 25 epochs elapsed with no improvement in training loss for CWA, the learning rate was decreased by a factor of 10. For affine networks, only 5 epochs without decrease were allowed. The test accuracy was computed after each epoch, and the lowest test accuracy during training is reported for each network. In general, test accuracy did not substantially decrease at any point during training. No validation sets were used; all networks were trained on all training data.
layers were of the same size for both CWA and affine networks in all cases, but varied according dataset. For MNIST, the FCN had two hidden layers of 2, 000 neurons each. For CIFAR- 10, a five-layer FCN with three hidden layers of 2000, 1000, 500 was used. The CNN for MNIST had four convolutional layers with 32, 64, 128, and 256 channels, respectively. The CNN for CIFAR- 10 added a fifth convolutional layer with 512 channels. ResNet- 18 consists of four blocks of two convolutions each. The initial input was projected to 64 channels, and the four
Network Sizes. The
8

---

Under review as a conference paper at ICLR 2020
blocks internally used 64, 128, 256, and 512 channels, respectively. The ResNet-18 architecture was not changed across tasks.
Results. The results are shown in Table ?? Focusing first on the fully connected architecture, CWA is seen to be competitive with affine networks with BatchNorm. On MNIST, the two are nearly equal at 98.65% and 98. 72%, and CIFAR-10 10 they differ by less than 2% 61.98% and 63.93. Adding BatchNorm to CWA does however improve performance to 64.93%. There are many opportunities to close the gap between CWA and Affine networks. The input and output procedures are not biologically driven and may not be optimal for CWA. For example, the affine networks have the benefit of using whitened inputs, whereas the CWA inputs are merely scaled and clipped. Different connectivity patterns between excitatory and inhibitory inputs may also help as well.
In the case of the CNN and ResNet-18 18, there is a greater distance between the results for CWA and affine networks, but the results for CWA on CIFAR-10 10 are still comparable to published results from approximately 2012 to 2014 (see e.g. mitt::/rodrrgoht wewrrrioo.giiiii github..hhh...
yet/buillc It is also noteworthy, that without BatchNorm, affine networks underperform CWA. One reason why the convolutional version of CWA performs worse might have to do with the low number of incoming connections. The first convolutional kernel in CNN has merely 6 x 5 x 5 = 150 connections for each neuron in the second layer. This number should be compared with 32 x 32 6 = 6,144 connections for each neuron in the second layer of the FCN and tens of thousands of connections neurons in the brain. Higher connectivity may be important to good performance with CWA, because it allows the denominator of Equation 3 to depend there are fewer connections, handful of input features can dominate the denominator, reducing generalization performance. Wider convolutional kernels might help, but are impractical on small images such as those of CIFAR and MNIST. Better performance might be possible on datasets of larger images. It must added that as an architecture, CNNs uses weight sharing in order to achieve economy with respect to data and memory. Biological neural networks are organized differently, and may be that a new architectural paradigm is needed for CWA.
many more input features. When
Regardless of any deficiencies, the results above demonstrate that network with conductance averaging and excitatory-inn -iciittoryinhiiittrry neurons such as CWA can solve real problems. To our knowledge, this is the first time that these two biological features have been successfully combined inside of an artificial neural network.
4.4 CWA VARIANTS: ABSNORM AND LATERAL INHIBITION
Networks with excitatory and inhibitory neurons and strictly positive weights are a consequence of Equation I but there is a generalization of conductance averaging that can be applied to affine nets as an ablation study on CWA. The basic idea is to make the valence ej +1 a property of the synapse (ei.) rather than the neuron and then assign Wi, = Gaycis so that [W,ll I = Gt.j- Then Equation becomes
23 Σ Σ JmH
WisPJ
E W Pk
which is just Equation 2 with divisive norming based on the absolute value of the weights. This equation will be called AbsNorm. As with CWA, it forces 2, E [-1, 1]. so that applying a ReLU afterwards results in E [].
AbsNorm characterizes the two main differences between CWA and affine nets, namely, the divisive averaging based on total input activity and the pattern in which outgoing connections from a neuron all share the same valence. Interestingly, AbsNorm failed to perform as well as CWA in preliminary experiments with the FCN architecture on CIFAR-10; these results are shown in Table ??] The difference between CWA and AbsNorm substantial, and was consistently worth about 2% test accuracy in four separate trials, the highest of which is shown for each in the table. This result suggests that there is value in separating the neurons into distinct valence classes. Distinct valences delineate of features as having an inherently positive or negative significance, though it is not clear why this yields better performance at this time.
One major area in which the architectures tested in this paper not biologically faithful is the connectivity pattern among different neuron types. This is an interesting aspect for future work. As first step in this direction, extra inhibitory neurons were added to each layer of CWA and

---

Under review as a conference paper at ICLR 2020
ARCHITECTURE
TEST ACCURACY
CWA
61.98
ABSNORM
59.95
CWA LATERAL INHIBITION
62.03
Table 2: CWA Variants, FCN Accuracy on CIFAR-10
activated recurrently temporal rollout, simulating the effect lateral inhibitory interneurons. which potentially regulate the overall activity level within cortical regions. As seen in Table CWA with this addition did achieve the highest accuracy among CWA variants tested, but only marginally so. More work is needed to establish the effect of different connectivity patterns.
CONCLUSIONS
Limitations & Future Work. There are several ways improve CWA. Most prominently, the decision to split layers into half excitatory and half inhibitory neurons connected in a cerd-forward fashion was somewhat arbitrary and does not reflect biology. It would be interesting approximate biological valence ratios and connectivity patterns to understand better how these features contribute information processing neural networks.
Realistic connectivity patterns would require the use of a recurrent activation pattern rather than feed-forward. Although substantial work has been done simulating the dynamics of spiking neural networks, less research has been directed towards discrete time slice approximations as used in recurrent neural networks (RNNs). The same principles of this paper could be applied to that case as well.
Separately, it is notable that the fully connected architecture obtained closer results to the affine baseline than convolutional architectures. Convolutional neurons in deep learning generally have fewer incoming connections than biological neurons do, and in may be that high connectivity is needed for CWA to work well. This issue needs further exploration.
Finally, parameter fitting with CWA was based on backprop. There is no conclusive evidence that animal brains adapt their conductances using backprop, and it would be worthwhile to explore alternatives for parameter fitting that do not use backprop but work well with CWA.
Applications to Deep Learning. CWA inherently implements features that have been found to be useful in deep learning, such as sparsity and limiting the output range of neurons. CWA does so by dynamically scaling the output marginalize out the level of input activity. It would be of interest to understand how this dynamic scaling affects other properties of affine networks, such as susceptibility adversarial images. Furthermore, the network architectures tested in this paper well as the training procedures were developed for affine networks. It may be that with specially adapted training procedures architectures, CWA could still achieve state-of-the-art results.
Applications to Neuroscience. In contrast to many biologically motivated neuronal simulations, the CWA experiments in this paper solve serious computer vision problems, suggesting that the value of neuroscience research can be illustrated by application to deep learning problems. However, non- biological adaptations were required to process the inputs and outputs the network and to control parameter adaptation. It would be interesting to understand the extent to which more biologically accurate settings can still obtain similar results.
The experimental setting of this paper also offers miniature lab in order to test questions such as whether multiple neuron valences promote better learning, or how the balance and connection among these valence types affect information processing and learning, Although such experiments could not conclude anything about what biological systems actually do, they could be invaluable in identifying possibilities for consideration.
To conclude, CWA implements biologically accurate understanding of neuronal activity and for the first time achieves competitive performance at computer vision datasets in network with conductance averaging and socitatory-inibbion neurons. This research provides a pattern for identifying and abstracting basic biophysical knowledge demonstrate relevance machine learning.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Charles B. Delahunt and J. Nathan Kutz. A moth learns to read MNIST. In The Seventh International Conference on Learning Representations, Workshop Track, 2018.
P. U. Diehl, D. Neil, J. Binas, M. Cook, S. Liu, and M. Pfeiffer. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In 2015 International Joint Conference Neural Networks (IJCNN), pp. 1-8, July 2015. doi: 10. 10.110//
Peter Diehl and Matthew Cook. Unsupervised learning of digit recognition using spike-timing- dependent plasticity. Frontiers in Computational Neuroscience, 9:99, 2015. ISSN 1662-5188. doi: 10.3389/fncom.20000 2015.00099, URL polttps://ww..r www. wwwnt wwt.:/rgg/ggg..o.//o htt.:///wwaaaaaaooooooooooooorrr 3389/ wwtps//nco..ommm 2015 00099
Kunihiko Fukushima, Sei Miyake, and Takayuki Ito. Artificial neural networks: Theoretical concepts. chapter Neocognitron: A Neural Network Model for Mechanism of Visual Pattern Recognition, pp. 144. IEEE Computer Society Press, Los Alamitos, CA, USA, 1988. ISBN 0-8186-0855-2. URL http: acm. org/ citati on cfm?i 6= 821 104123
Fabrizio Gabbiani and Steven James Cox (eds.). Mathematics for Neuroscientists (Second Edition).
Academic Press, San Diego, second edition edition, 2017. ISBN 978-0-12-8018955 doi: https//db..og//101101//997/////
00037-3. URL http://tt////tt sc encedi rect
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In In Proceedings of the International Conference on Artificial Intelligence and Statisties (AISTATSIO). Society for Artificial Intelligence and Statistics, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778, 2016. doi: 2LL1109CCVR..220..00
A. L. Hodgkin and A. F. Huxley. A quantitative description of membrane current and its application to conduction and excitation in nerve. The Journal of Physiology, 117(4):500-44, August 1952. doi: doi: 11111 13/jphysiol.. httiippyyioo.1192..pp.... sp22,744477
Yangfan Hu, Huajin Tang, Yueming Wang, and Gang Pan. Spiking deep residual network. CORR, abs/1 880.001551... 2018. URL htt.: //apxia.aavvv///////a org/ abs/ abs/1805, 1805. 01352
Dongsung Huh and Terrence Sejnowski. Gradient descent for spiking neural net- works. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Resa-Bianchi
and R. Garnett (eds.), Advances Neural Information Processing Systems 31, pp. 1440-1450. Curran Associates, Inc., 2018. URL http:://papprrs nips www..niis..ccc.. cttpssscc/pppe// 1417-gradient desceet-eeettttee for stor-spikiig spiking neural cectrallneettee www.wrks...eeoooooooo.o....... pdf]
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 448 456, Lille, France, 07-09 Jul 2015. PMLR.
Eric R. Kandel, James H. Schwartz, Thomas M. Jessell, Steven A. Siegelbaum, A. J. Hudspeth, and Sarah Mack (eds.). Principles of Neural Science. 5th Edition. 2013. ISBN 978-0-07-13901 1-8..
Saeed Reza Kheradpisheh, Mohammad Ganjtabesh. Simon J. Thorpe, and Timothie Masquelier. Stdp-based spiking deep convolutional neural networks for object recognition. Neural Networks, 99:56 67, 2018. ISSN 0893-6080. doi: https://do..or//10. httpg///....1....neeeeeee mww..nnuntt.207.11222
Diederik Kingma and Jimmy Ba. abs/1412.6980. 2014.
Adam: A method for stochastic optimization. CoRR,
Christof Koch and Idan Segev (eds.). Methods in Neuronal Modeling, Second Edition. MIT Press, 1999.
11

---

Under review as a conference paper at ICLR 2020
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). 2009. URL
cs http::/w.....ottttteee//ttaooae. -triz/iifarrraa .html
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In F. Pereira, C. J. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097-1105. Curran Associates, Inc., 2012.
Shruti Kulkarni and Bipin Rajendran. Spiking neural networks for handwritten digit recognition supervised learning and network optimization. Neural Networks, 103, 04 2018. doi: 10.1016/j. neunet.2018 .3.0999.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010.
Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. Frontiers in Neuroscience, 10:508, 2016. ISSN 1662-453X. doi: 10.3389/fnins. 2016.00508.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML' 10, pp. 807-814, USA, 2010. Omnipress. ISBN 978-1-60558-907-7. URL
attp::/dda acm. wwwpor/gggttttt.t citatittaiiiono cfm?id 310422231122...0 3104425
Andrew Y. Ng. Feature selection, vs. 12 regularization, and rotational invariance. In Proceedings of the Twenty-first International Conference Machine Learning, ICML '04, pp. 78-, New
York, NY, USA, 2004. ACM. ISBN 1-58113-838-5. doi: 00.145/0015330001553355 URL http://doi. acm. wttp/1..14./111113.30 TTW.//11333..11131333 1015435
Michael Pfeiffer and Thomas Pfeil. Deep learning with spiking neurons: Opportunities and challenges. Frontiers in Neuroscience, 12:774, 2018. 8. ISSN 1662-453X. doi: 10.3389/fmins.200 2018.00774.
F. Rosenblatt. The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, pp. 365-386, 1958.
Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: VGG and residual architectures. CoRR, abs/1802.02667, 2018. URL
02627
Oren Shriki, David Hansel, and Haim Sompolinsky. Rate models for conductance-bused cortical neu- ronal networks. Neural Computation, 15(8):1809-188 2003, doi: 10. 10.1162208997766666555533. URL /ttp:p//.o.o.ooooo.....666666663
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929-1958, 2014.
Guibo Zhu, Zhaoxiang Zhang, Xu-Yao Yao Zhang, and Cheng-Lin Liu. Diverse neuron type selection for convolutional neural networks. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pp. 3560 3566, 2017. doi: 10.24963/jjcai.21117 2017)//.i..21///999 URL
2211///
12

---

Under review as a conference paper at ICLR 2020
SUPPLEMENTARY MATERIAL
6.1 CODE IMPLEMENTING CWA
The following code implements a fully connected CWA layer in python as PyTorch module. As can be CWA requires two matrix multiplies instead of the usual one.
class CWALayer (torch. nn. Modulle)
def ___iiiit_see___ neurons_im, (eurons_out))
super (CWALayer, self) self)._init__ iniit_()) ()
self self.neurons_in = necrons_in
self.neurons_out
neurons_out
self.ga torch. nn P
torch. zeros (neurons_in, neurons_out)
self.es forch.zerowe ft. requizes_grad-Falls
self.es [ [ineurons_ni//22 -1.0
def fix(s gs) return torch. clamp (1+gs, 0.0, float ('inf')
def (sewa ps)
Gs self.fix (self.gs) normalizer 1
(ps, Gs)
numerator = -0.4 + torch.matmuj (ps=self waa.sellfees... es. unsqueeze Gs)
activity
numerator / torch. cl clamp (normalizer, 1e-6, 6, float (' winf)))
return activity
The following code implements convolutional CWA layer. As is below, implementing CWA
requires two convolutions instead of one, but is otherwise similar to the affine version.
class ConvCWALayez (torch. nn aodolll
def Linit_((elll in_channels, out_channels,
kernel_size, stride=1, padding=0) super (ConvCWALayer, self).._nit___
self.gs (out_channels, in_channels, torch. zeros (out kernel_size, kernel_size)
self.es = torch. nn. www.ch...n.aaaeeer......
torch. (in_channels, Kernel_size kernel_size).fie fill_ll111111 (1.0),
requizes_grad-false
selff.es[:in_chmmell] self.stride = stride
-1.0 self.padding padding
nn. self.grest = torch.nn. Parameter (torch. zeros )1))
def fself, gs):
return torch. cl amp (1+gs, 0.0, np.inf)
13

---

Under review as a conference paper at ICLR 2020
def forward ps)
Gs sell.fix normalizer = 1.0 + torch.nn. nn. forch.nn.functe. functionall conv2d
ps, Gs, stride=self. padding=self, padding
numerator www.h.nn. con. ps, Gs*self. es unsqueeeeee (0), stride=self. stride, padding=self. pac.sssiil.....inngggggga. padding
return
numerator / torch. cl clamp (normalizer, 1e-6, float (1 ('inf))) )
14


Reviews:
[{"review_id": "HJgyLhgI9r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I wanted to first thank the authors for their work on connecting biologically inspired CWA to deep learning. The paper overall reads nicely. \n\nThe authors make the assumption that reviewers are fully aware of biological terminology related to CWA, however, this may not be the case. It took a few hours of searching to be able to find definitions that could explain the paper better. I suggest the authors add these definitions explicitly to their paper (perhaps in supplementary), also an overview figure would go a long way. I enjoyed reading the paper, and before making a final decision (hence currently weak reject), I would like to know the answer to the following questions:\n\n1. Have the authors considered the computational burden of Equation 3? In short, it seems that there are two summations (one for building the probability space over measure h) and one right before e_j. This is somewhat important, if this type of neural network is presented as a competitor to affine mapped activations. \n\n2. It would be nice to have some proof regarding universal approximation capabilities of CWA. In my opinion it is, but a proof would be nice (however redundant or trivial - simply use supplementary). \n\n3. I was a bit confused to see CWA+BN in the Table 1. In introduction, authors write “But CWA networks are by definition normalized and range-limited. Therefore, one can conjecture that CWA plays a normalization role in biological neural networks.” Therefore, I was expecting CWA+BN to work similarly as CWA for CIFAR10. Please elaborate further on this note. \n\n4. Essentially, the CWA changes the definition of a layer in a neural net. Do authors see a path from “CWA works” to “CWA works better than affine?”. If so, please elaborate. Specifically, I am asking this question “Why should/must we stop using affine maps in favor of CWA?”. Now this may or may not be the claim of the paper. It’s ok if it is not; still showing competitive performance is somewhat acceptable, but certainly further insight would make the paper stronger. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgeFBvRYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on non-spiking Hudghkin-Huxley model, which is different from existing works on spiking neural network-based Hudghkin-Huxley model. \n\nThere are many ways of using neuron firing model as unit to construct neural networks. They choose a specific way (mentioned above). I think the most interesting part would be the CWA method which achieves the normalization. \n\nThey have a fair list of literature in spiking neural networks. But I find the way they illustrate the difference between their model and other models is insufficient. They should focus on the model-wise difference, instead of focusing on whether it’s applied to MNIST or not or what’s the accuracy. \n\nThey don’t include any other SNN model in the paper for experimental comparison. They also mention a few SNN works that work well on MNIST in the related work section which actually have better accuracies than their model. So it is inappropriate to say this proposed method is a state-of-art neuro-inspired method, Because others perform well on MNIST as well, and their limited experiments only investigate MNIST and CIFAR-10, which are less interesting generally. \n\nCWA cannot outperform Affine+BN. \n\nOverall, the idea is somehow interesting, but the experiments are weak. Applying the method to MNIST and CIFAR-10 is far from being called either “interesting computer vision applications” or “difficult perceptual tasks”. They only use perceptual in the title, but the applications are MNIST and CIFAR-10. It feels like they want to learn something big, but they only focus on benchmark datasets. \n\nThey compare nothing with other SNN type of model on other truly difficult perceptual tasks.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJl5fg_6FS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel neural network architecture inspired by the analysis of a steady-state solution of the Hodgkin-Huxley model. Using a few simplifying assumptions, the authors use conventional backpropagation to train DNN- and  CNN-based models and demonstrate that their accuracies are not much lower than the state-of-the-art results.\n\nThe paper is well-written, sufficiently detailed and understandable. Derived self-normalizing Conductance-Weighted Averaging (CWA) mechanism is interesting in itself, especially contrasting CWA results with those obtained for the non-Batch-Normalized networks. It is also inspiring to see that this model can be derived based on a relatively accurate biological neuron model.\n\nMy main question is actually related to the potential impact of this study. I am curious about the implications and the ways in which these results can inspire other researchers.\n\nAfter reading the paper, I got an impression that:\n\n(a) From the point of view of a machine learning practitioner, these results may not be particularly impressive. They do hint at the importance of self-normalization though, which could potentially be interesting to explore further.\n\n(b) From the point of view of a neuroscientist, the proposed model might be too simplistic. It is my understanding, that neural systems (even at \"rest\") are inherently non-equilibrium (and I assume the presence of simple feedback loops could also dramatically change the stead-state of the system). Is it possible that something similar to this \"steady-state inference\" mode could actually take place in real biological neural systems?\n\n(c) Presented results appear to be important from the point of view of someone who wants to transfer insights from biology into the field of deep learning. But there might be an extent to what is achievable given a simple goal of optimizing a supervised accuracy of an artificial neural network trained using gradient descent (especially considering limitations imposed by hardware). I am optimistic about the prospect of knowledge transfer between these disciplines, but it is my feeling that the study of temporal dynamics, emergent spatiotemporal encodings, \"training\" process of a biological neural system, etc. have potentially much more to offer to the field of machine learning. These questions do appear to be incredibly complex though and the steady-state analysis is definitely a prerequisite.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies non-spiking Hudgkin-Huxley models and shows that under few simplifying assumptions the model can be trained using conventional backpropagation to yield accuracies almost comparable to state-of-the-art neural networks. Overall, the reviewers found the paper well-written, and the idea somewhat interesting, but criticized the experimental evaluation and potential low impact and interest to the community.  While the method itself is sound, the overall assessment of the paper is somewhat below what's expected from papers accepted to ICLR, and I’m thus recommending rejection.
Decision: Reject