Training stochastic stabilized supralinear networks by dynamics-neutral growth
Authors: Wayne WM Soo
Máté Lengyel
Venue: NeurIPS 2022 Accept (2022)
Keywords: cortical circuit
visual cortex
recurrent neural network
stabilized supralinear network
Gaussian scale mixture
Bayesian inference
Markov chain Monte Carlo

Abstract: There continues to be a trade-off between the biological realism and performance of neural networks. Contemporary deep learning techniques allow neural networks to be trained to perform challenging computations at (near) human-level, but these networks typically violate key biological constraints. More detailed models of biological neural networks can incorporate many of these constraints but typically suffer from subpar performance and trainability. Here, we narrow this gap by developing an effective method for training a canonical model of cortical neural circuits, the stabilized supralinear network (SSN), that in previous work had to be constructed manually or trained with undue constraints. SSNs are particularly challenging to train for the same reasons that make them biologically realistic: they are characterized by strongly-connected excitatory cells and expansive firing rate non-linearities that together make them prone to dynamical instabilities unless stabilized by appropriately tuned recurrent inhibition. Our method avoids such instabilities by initializing a small network and gradually increasing network size via the dynamics-neutral addition of neurons during training. We first show how SSNs can be trained to perform typical machine learning tasks by training an SSN on MNIST classification. We then demonstrate the effectiveness of our method by training an SSN on the challenging task of performing amortized Markov chain Monte Carlo-based inference under a Gaussian scale mixture generative model of natural image patches with a rich and diverse set of basis functions -- something that was not possible with previous methods. These results open the way to training realistic cortical-like neural networks on challenging tasks at scale.

Training stochastic stabilized supralinear networks by dynamics-neutral growth
Wayne W.M. Soo Department of Engineering University of Cambridge wmws20cam. ac. uk
Máté Lengyel
Department of Engineering University of Cambridge m. m.lengyel@eng. cam uc.uk
Department of Cognitive Science Central European University
Abstract
There continues to be a trade-off between the biological realism and performance of neural networks. Contemporary deep learning techniques allow neural networks to be trained to perform challenging computations at (near) human-level, but these networks typically violate key biological constraints. More detailed models of biological neural networks can incorporate many of these constraints but typically suffer from subpar performance and trainability. Here, we narrow this gap by developing an effective method for training canonical model of cortical neural circuits, stabilized supralinear network (SSN), that previous work had be constructed manually or trained with undue constraints. SSNs are particularly challenging train for the same reasons that make them biologically realistic: they are characterized by cronnllyconnected excitatory cells and expansive firing rate non- -onliincasssies that together make them prone to dynamical instabilities unless stabilized by appropriately tuned recurrent inhibition. Our method avoids such instabilities by initializing a small network and gradually increasing network size via the setamics-neutra addition of neurons during training. We first show how SSNs can be trained to perform typical machine learning tasks by training an SSN MNIST classification. We then demonstrate the effectiveness of method by training an SSN on the challenging task of performing amortized Markov chain Monte Carlo-based inference under Gaussian scale mixture generative model of natural image patches with a rich and diverse set of basis functions something that was not possible with previous methods. These results open the way to training realistic cortical-like neural networks on challenging tasks at scale.
1 Introduction
Biological neural network models of the brain have long been studied to understand some of the fundamental network mechanisms employed by the brain [14 However, these models were not capable of actually achieving the brain's 's performance on its wide range of complex computations. Conversely, artificial neural networks have been achieving competitive performance image, language, and speech-relate tasks, but typically without regard to biological realism 85 In systems neuroscience, neural networks have also been trained on laboratory tasks employed in typical experiments [6]. These approaches provided important insights into the contributions of experimentally found macroscopic neural dynamics and representations to the successful performance of such tasks. However, these networks did not incorporate some of the most salient constraints on the detailed organization of cortical circuits. For example, they were purely feedforward [0], utilized neuronal transfer functions that were either outright saturating (c.g. tanh) at best lacked the superlinear characteristics of cortical neurons (as ReLUs, or the rectified softplus) [9-T] 15 [6].
multitude of
had noiseless dynamics [12], no separation of excitatory (E) and inhibitory (I) cells [ 10 [5] and
sometimes even employed artificial gating mechanisms such as LSTMs 13
Due to these properties,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

these networks remained abstracted away from biological neurons in key aspects, and hence offered limited insight into the neuron-level mechanisms that drive their computations.
To bring the modeling of neural circuit mechanisms underlying challenging computations at (or near) single-cell resolution within reach, develop novel method to train experimentally- supported neural network model, the stabilized supralinear network (SSN) [17]. Critically, the SSN satisfies many of the key constraints of cortical circuits: it has separate but recurrent excitatory and inhibitory populations, expansive (rectified power-law) single neuron non-linearities, realistic single-neuron time constants, and no reliance on artificial gating mechanisms. its original form, the SSN continues a venerable tradition of handcrafted excitatory-inhiiioon
(E-I) networks whose dynamics have been extensively studied [18] [9]. and that helped reveal key consequences of the cortex operating in a balanced E-I regime G 20 [2]. In particular, the SSN accounts for the experimentally-ooseen
effects of stimulus tuning, sublinear response summation and surround suppression of neural responses in sensory cortices 4 (23,, as well as oscillations, bistable and persistent responses 24 Furthermore, when extended to include noise, the resultant stochastic SSN
produces realistic, stimulus-modulateed population patterns of noise variability (25)
However, the same features that make the SSN an attractive substrate for its biological realism also make it particularly challenging to train it. This is because, in SSNs, excitatory cells are typically strongly connected with motthe in order to implement non-trivial nonlinear transformations of their inputs, such as divisive normalization, which underlie many of the experimentally observed phenomena for which they account. In addition, single neuron non-linearities are expansive reflect the experimental finding that cortical neurons almost exclusively use the convex (non-saturating) part of their firing rate non-linearities under physiological conditions, including for the strongest stimuli 26 (even if they can be trivially driven to saturation by direct current injection, due to the refractory period of the mechanism generating action potentials). These properties make SSNs particularly susceptible to dynamical instabilities resulting in run-away excitation, thus rendering their training highly challenging. Indeed, in the few cases in which the training of SSNs was attempted, either noiseless neurons were used [27 28). or the network was so heavily under-parameterizer that it substantially limited its expressivity [29].
Here, we develop a new method to train SSNs that avoids dynamical instabilities during training and is able to train SSNs at scale on a variety of tasks. While standard approaches to training networks only optimize weights and keep the architecture of the network fixed throughout network training (but see recent work Refs. 32), our method alternates between optimizing network weights with fixed architecture, and changing the architecture by growing the network. Importantly, in each network growth step, new neurons are added such that they do not affect the dynamics of the network before weights are optimized again. We first demonstrate the effectiveness of our method by training an SSN on standard machine learning benchmark, MNIST classification [33]. We achieve only slightly lower accuracies than state-of- the-art, despite restricting our network to be biologically plausible. For a direct comparison, we also train an SSN on a probabilistic inference task that has been shown to have neurobiological relevance but has proved to be challenging target (29. Our approach successfully trains an SSN that has three orders of magnitude more parameters than what was previously possible to optimize. The trained SSN performs accurate inference under Gaussian scale mixture-model (GSM) [34] with a set of basis functions that is sufficiently rich for allowing high-quality reconstructions of CIFAR-IO 10 (35] images. Inference under this GSM represents an unachievable target for previous, heavily constrained approaches for training SSNs. These results validate our approach and open the way training SSNs large variety of different tasks, and to using the resulting networks study the circuit mechanisms of neural computations at a single cell resolution.
2 Methods
2.1 Stochastic supralinear stabilized network
The SSN is a canonical model of cortical circuits that has been shown to account for a wealth of neural response properties in the primary visual cortex [7 25]. Its dynamics are described by
du df
= - -ff(() + Wr+n
(1)
2

---

where T collects the neurons' time constants, u denotes their membrane potentials, W is the symaptic weight matrix, 77 represents noise, correlated across time with time constant Tys as well as across neurons with covariance E., f(h) represents a non-linear transformation of the external input h,
[1,h)) 0, (h+ 02)
with parameters 0 = {01, 02, 83), and denotes neural firing rates, given by rectified expansive non-linearity,
T = k [uas7
with 7 > 1 and A > Note that there is no built-in upper saturation to the model, which presents potential instability problems - excitations within the network can exponentially grow unbounded through mutual and self-excitation. The network relies purely on inhibition to stabilize, which therefore restricts the parameter regime in which the network is stable. The network comprises of distinct populations of excitatory and inhibitory neurons, imposing restrictions on the polarity of the elements of W. Specifically, the elements of each column of W must have the same sign (positive for excitatory neurons, negative for inhibitory neurons).
2.2 Theoretical motivation
We note that under the assumption of reasonable noise characteristics, it is always possible to add neuron to an existing stable SSN while preserving stability in non-trivial manner. We begin by defining the newly-added neuron as a "twinn" of an existing neuron. The new twin has the same incoming recurrent weights as its original counterpart. This way, the newly-added neuron is guaranteed to receive the same input as the original neuron. Second, the newly-added neuron also inherits the outgoing weights of its original counterpart, but such that all outgoing weights of both neurons are halved. Thus, disregarding noise and external input, all other neurons in the network receive the same total input from these twin neurons as they used to receive from just the original neuron before its duplication. Therefore, still disregarding noise and external input, the activity of every neurons remains unchanged, and the newly-added neuron simply repeats the activity of its original counterpart. In other words, if the network was stable before adding the new neuron, it will also be stable after having added We refer to this process as the "dynnamics-nuuraal" addition of neuron to the network. This idea forms the mathematical foundation of our proposed training method.
Formally, let neuron be the cloning candidate. The membrane potential vector u, and thus the firing rate vector r now each have an additional element:
uun
r-m
r-n I'
Un
and
Un
I'n
where -n represents the indices of all neurons that are not Similarly, we split the weight matrix W into four blocks, {W( i, a), W m,n): W(n, m), W W(nn)). We update the network weights according to
W (m) (n.-n)
W W(-n.))
W (-n,-w)
W (mm.n)
7
W
(-n.a)
W (n.n)
W (n.
W (n.m)
W
(n,n)
(5)
W (m.m))
W (n.))
W
(n.n)
2
2
The synaptic input into all neurons (the equivalent of the third term the RHS of Eq. is now given by
W (-n,-M)
-n)
W (-mm)
W (-),))
r
W m)) P-n + W
ma) Tn
W
(n))m)
W (n,))
W (n,))
W (m W (n.n)
(6)
W
(n,m))
W (n,))
W (m.))
Tn
W ((n-n) P-n (n.n) Tn
where the top rows of this vector are
W W I.n, m) I-n + W (nn,n) Tn
Wr
(n) Fon + W
(n.n) Pn
which is the same input as that received by the previously existing neurons, before the addition of the new neuron, claimed (cf. Eg. 1. Note that during optimization, symmetry between the twin
3

---

initialization phase
growth phase
unstable
incomplete
random initialization
network stability
stable
dynamics- neuatall neuron addition
network optimization
building status
complete
end
Figure I: Method sketch for training SSNs by dynamics-neutral growth. As randomly initializing large SSNs yields unstable or trivially stable (silent) networks with high probability [29], we start with a small network two neurons, comprising of single excitatory and inhibitory neuron, by generating network parameters until set of parameters corresponding to stable network is found (initialization phase). We subsequently add neurons a dynamics- neutral manner while concurrently optimizing for the training objective until the desired network size is achieved (growth phase).
neurons broken due them receiving statistically equal but nonetheless umiquely-generatee noise inputs. As our test cases are constructed to have the same number of inputs as the number of neurons in the network, we also choose not duplicate incoming weights from external inputs for newly added neurons, but instead to connect each newly added neuron to a single new input. (In future work, we will consider all-to-all connections between external inputs and existing neurons, with trainable weights, which are duplication at network growth.) These factors wear down the absolute guarantee of stability, but is still reasonable to build on this intuitively high chance of stability.
2.3 Training method
Even with a reliable way of growing a network, we still explicitly require an existing (smaller) network a starting point. By "backward induction", recognize that the simplest and most reliable solution is to initialize the whole process by generating the smallest SSN possible network with only one excitatory and one inhibitory neuron. This can be achieved by frute-force or by some targeted initialization specific to the training objective in order to get head-start on performance optimization. The full network can then be gradually constructed by adding neurons in nenamics-neutral way as previously described. The entire training method is summarized in Fig.
2.4 Training reliability
We calculate three different metries that together characterize how reliably our method protects against instability problems typically encountered in training SSNs.
1. Proportion of unstable networks during training. We repeat the entire training method for a total of 100 separate networks and count the number of networks that become unstable over training (i.e. runaway activity for any of the training inputs).
2. Proportion of unstable trials with shuffled weights. For each of the 100 fully trained networks (see above), we shuffle the elements within each quadrant of their weight matrix, defined by the excitatory (E) or inhibitory (I) nature of the pre- and postsynaptic neurons (E-E, E-I, I-E and I-I). We repeat this shuffling 100 times, run each shuffled network for 100 trials, each with a different input (from either MNIST or CIFAR-10, see below), and count the number of trials in which the network becomes unstable. We report the mean and standard error of the proportion of unstable trials across the 10,000 shuffled networks.
3. Proportion of unstable randomly generated networks. For each of the 100 fully trained networks (see above), we generate 100 random networks, such that each element of their weight matrix is generated from a zero-mean Gaussian with variance equal to the variance of all weights in the corresponding trained stable network. We then set the sign of each weight to comply with the E/I identity of the presynaptic neuron. We each random network for 100 trials with different inputs, as above, count the number of unstable trials, and report the mean and standard error of the proportion of unstable trials across the 10,000 random networks.

---

autoencoder
output
decoder weights
readout layer
readout weights
encoder weights
network weights noise covariance
SSN
U4
input function
input image
Figure 2: Training an SSN perform the MNIST classification task. Right: the network consists of 40 excitatory and 10 inhibitory neurons (not shown). Left: before training the SSN, we optimize 3-layer autoencoder (green) to reduce the dimensionality each MNIST image to match the number of excitatory neurons in the network, receiving the encoded image input after transformation by the input function, S. For each input, SSN activities, M/ evolve over time until reaching their stationary distribution, after which the mean membrane potentials are passed through feedforward readout layer to make prediction (red). Trained parameters are highlighted on the two sides of the figure.
MNIST classification task
We first train SSNs MNIST digit classification 33 Due to its recurrent dynamics (Eq. Ip. neural activities an SSN evolve over time even in response to fixed external input, instead of instantaneously producing fixed output. Moreover, due to the presence of noise in the dynamics, a stochastic SSN never settles onto a single response. While these features are ever-present in experimental neural recordings, and thus contribute the biological realism of stochastic SSNs, they also make their training challenging. Therefore, to define the network's 's output for the purposes of training (and, more generally, for measuring its performance), we evolve neural activities until they reach a steady-state distribution, and pass the mean membrane potentials into readout (softmax) layer.
We train an SSN with 80 excitatory and 20 inhibitory neurons (a biologically-reale realistic E:1 ratio). We also train an SSN with 50 excitatory and 50 inhibitory neurons to compare with the performance of previous models (9).. Only the excitatory neurons receive external input (i.e. MNIST images) and are decoded in the readout layer, although activities in the entire network (including inhibitory neurons) evolve over time. In order to transform the MNIST images into inputs for the SSN, we optimize an autoencoder (also comprising of layers) to reduce input dimensionality from 784 (number of pixels in the MNIST images) to 50 (number of excitatory neurons in the network). As base comparison, also train multi-layer perceptrons (MLPs) with layers, and perform logistic regression on the encoded data from the autoencoder.
An overview of the training approach is shown in Fig. 2
Training results summarized in Table None of our 200 trained networks encountered instability problems. This is highly non-trivial: at both E:I ratios, it is virtually impossible to initialize networks randomly that they are stable, and in instability in nearly 80% of cases. In addition, our trained SSNs at both E:I ratios perform at only slightly lower levels than the corresponding MIPs, and also better than logistic regression despite having noise injected at every time step and also having to maintain dynamic stability. SSNs constrained to have a "ring" architecture (such that each E-I quadrant of the weight matrix is circulant) with Gaussian weight profiles, as in previous work [29], are not able to perform the task competently because of their highly-constrained parameterization designed exploit rotational symmetries in their training images, which do not exist in the MNIST dataset. Nevertheless, their performance is still above chance (0.1).
shuffling the weights of trained stable networks results
4 Amortized probabilistic inference
We next train an SSN on a challenging probabilistic inference task. This task requires the SSN to act as recognition model for probabilistic generative model of natural image patches, the Gaussian scale mixture (GSM) model 34
36 by performing amortized approximate Markov chain Monte Carlo
(MCMC) inference produce trajectories in the state space of its neurons (encoding latent variables of the GSM) such that
Fig. 3). That is, in response to an input (observation), the network needs to
5

---

Table I: Stability and performance of SSNs trained on MNIST classification, Numbers separated by colons indicate the number of excitatory and inhibitory neurons in the SSN, respectively. The encoded data are labeled according to their reduced dimensionality (see SageFig..1 2 For comparison, we separately train MLPs (with 3 layers) and perform logistic regression to classify the encoded data. Despite the presence of noise, our trained SSN performs only slightly worse than the MLP and better than logistic regression.
stability
SSN E:1 ratio
networks
proportion unstable
trained networks
0.0
50:50
networks w/ shuffled weights
0.790 (+ 0.007) 1.0 (# 0.0)
random networks
trained networks
0.0 0.794 (#: 0.008)
80:20
networks w/ shuffled weights random networks
1.0 (+ 0.0)
performance
model
MNIST data
test accuracy
Logit
encoded-50
0.914
MLP
encoded-50
0.974
Gaussian ring SSN [29] (50:50)
encoded-50
0.223
SSN (50:50)
encoded-50
0.949
Logit
encoded-80
0,922
MLP
encoded-80
0.976
SSN (80:20)
encoded-80
0.952
the distribution of neuton response patterns sampled by these trajectories approximately matches the joint posterior distribution of the GSM for corresponding input (hence performing approximate MCMC inference). Notably, the network needs achieve this with single set of parameters (prominently, synaptic weights) for a large number of inputs (hence the inference is "amortized"). This a challenging task because the objective requires the network to maintain finite levels of variability while modulating the co-variability of its neurons in a stimulus-dependen way, which is only possible stochasticity of neural noise not suppressed and recurrent connections are sufficiently strong combination that pushes the network towards instability. This task has also been suggested to have neurobiological relevance. Previous work indicated that the stationary statistics of neural responses in the primary visual cortex (V1) may be accounted for by model assuming these responses represent statistical samples from a GSM posterior [36] 37]. and that SSNs producing such samples also account for other dynamical aspects of VI responses such oscillations and transient overshoots [29].
4.1 Gaussian scale mixture model
The GSM 34 36
was proposed as generative model of natural image patches suitable for image processing, and as an internal model that the visual system (in particular, V1) may implicitly use to process images. According to the GSM, image is generated "projective fields" (columns of a matrix, A, see below), with overall scaling ("global contrast") determined by scalar, 2, and with some additive zero-mean, Gaussian white noise pixel noise, 7x ~ (0, o I) (where I is the identity matrix):
linear combination of so-called
(8) where y are latent variables scaling the individual contribution of each projective field in A. The projective fields 2D Gabor filters, each characterized by four parameters determining the orienta- tion, size, and 2D location of the filter (see also Appendix A in the Supplementary material) such that each column of A corresponds to the pixels of single vectorized 2D Gabor filter. An ideal observer, when presented with an image x, computes a posterior distribution over and = by incorporating their respective priors:
x = Ay + +7
y Nyy; O, C). and ~ Gamma(a, 3))
(9)

---

GSM
stochastic SSN neural network
latent activations
network weights noise covariance
latent prior
y/
Yj
Ui
Uj
projective fields
contrast
input function
contrast prior
2
X
receptive fields
noise variance
noise
Figure 3: Training an SSN to perform inference under GSM. The GSM (left) is generative model natural image patches proposed underlie computations in the visual cortex. According to it, image is constructed each contributing according its corresponding latent "activation", Vi. scaled by a global scalar contrast level, z, and corrupted by white zero-mean Gaussian pixel noise, 1/x- We train SSN to perform approximate amortized MCMC inference under the GSM (right). Input the the SSN an image patch linearly filtered by oriented receptive fields (taken to be identical to the projective fields of the GSM), elementwise transformed by a nonlinear "input function", f. Excitatory neurons SSN are taken correspond to latent variables of the GSM, such that their responses, u. given some input image, represents statistical samples from the posterior over the latent variables of the GSM given the same observed image. Trained parameters are highlighted on the two sides of the figure.
linear combination of (oriented Gabor filter-based) projective fields,
so that (by Bayes' rule)
=(y, x A A N(y:0, 0, C) Gamma(z;o,)
(10)
after which the contrast level z is marginalized, resulting a posterior distribution over y:
P P(y, 2/x)
(11)
We construct our GSM by first maximizing the fraction of variance explained by the projective fields on CIFAR-10 images and then optimizing its remaining parameters by maximizing model (marginal) likelihood on the same images. Additional details can be found in Appendix A.
4.2 Sampling-based probabilistic inference
Given an observed image as input, the membrane potential responses of the SSN, u, as determined by. I are required to (approximately) represent statistical samples from the posterior distribution over the latent variables of the GSM as computed by an ideal observer using Eq. 11 (assuming one-to-one correspondence between the excitatory neurons of the SSN and the latent variables of the GSM). This inference method is therefore described as "sampling-basee based". We achieve this by optimizing the SSN parameters with respect to the mean-squared errors (averaged over 50,000 training images) between the mean and (co)ane of its response distributions, Esss u1]. Vsss[1) and CssN [u). and of the corresponding target posterior distributions, PGSM- a and EGSM:
L Ap EssN [u] MGSM| 11 >>a ||Vsss[u oásMll 112 As [[Cssv[u]- EesM
(12)
constant coefficients Ay. Ag² and As (and where || denotes the Frobenius norm). In this case, both excitatory and inhibitory neurons receive external input corresponding to their respective receptive fields (such that an E-I pair shares the same receptive field) but, as before, only excitatory neurons considered in the computation of the cost function.
7

---

Table 2: Stability and performance of SSNs trained on probabilistic inference under the GSM. SSNs have been previously been trained for the same task [29]. which serves as comparison. We train two SSNs, one for the ring GSM in 29] (labeled as ring SSN), and another for our GSM which consists of a diverse set of basis functions optimized from CIFAR-10 images (general SSN). Since training targets are different for the two GSMs, we compute a modified cost (Eq. 13 in order to directly compare optimization performance.
stability
model
networks
proportion unstable
trained networks
0.0 0.446 (+ 0.008)
general SSN
networks w/ shuffled weights
random networks
1.0 (+ 0.0)
performance
network
GSM target
normalized cost
Gaussian ring SSN (2
ring 29
1.0 0.158
ring SSN
ring [29
Gaussian ring SSN
29
optimized for CIFAR-10
~ 104
general SSN
optimized for CIFAR-10
0.745
4.3
Results
We train SSN with circulant quadrants in the weight matrix ("ring SSN") for a GSM whose projective fields only differ in their orientation (and thus form a ring topology), used in earlier work [2]]. We also train unconstrained SSN ("general SSN") for the GSM with richer set of projective fields we described above (Section 4. For both GSMs, we also train ring SSNs that are constrained have Gaussian weight profiles (Gaussian ring), as in previous work [29]. As target moments are different in the two GSMs, we compute a normalized cost function so that we can compare networks trained across different GSMs:
C' - Ap [[ssN[u] [ MosMil2 + Ag² Vsss [u] cdsM 12 + As
[SSN [u]
EGSM 2
(13)
|aSM 2
EGSMIE
In order fully appreciate these results, briefly describe the GSM-SSN pairs in each row of Table and how they are trained. A full comparison can be found in Appendix B.
1. The first row describes the Gaussian ring-SSN from 29 optimized for the ring GSM from the same study. This SSN is highly constrained; for example, the entire 100 x 100 weight matrix is parameterized by only 8 parameters. The same 5 crtationall--symmettt training and testing, due to the high computational costs associated with the method. Optimization is first done using Adam for small number of iterations, followed by a veroinar but biased semi-analytical mean- mean-field method described in (38 (and Appendix B). The normalized cost of this model is set to by definition.
images are used for both
2. The ring SSN in the second row is trained by method, with the same 5 images used the training and test set (as above). The weight matrix constrained to be blotksymmerrric-cicclaal in order to fully exploit the rotational symmetry of the images. Each symmetrii--crcullnt 50 50 block is parameterized by 26 parameters, giving a total of 104 parameters for the weight matrix. The additional degrees of freedom has resulted in nearly an order of magnitude lower cost.
3. The third row shows the Gaussian ring SSN from 29 trained for our GSM. Due to its highly- constrained parameterization (see above) and built-in rotational symmetry assumptions, the model is not able succeed in matching the moments of the GSM posterior moments when observing CIFAR- 10 natural images and fails to converge during training.
4. The general SSN in the fourth row has all 10,000 parameters of its 100 x 100 weight matrix individually trained on our GSM. Despite the increase in difficulty of the task (matching more complex GSM), our method produces a network that achieves normalized cost lower than that achieved by Gaussian ring SSN specifically designed to work for ring GSM 29].
8

---

A
ms
ms
40 ms
60 ms
80 ms
100 ms
B
network
mean
0
network mean
network network variance
variance
0
0
0
0
0
targets
targets
targets
targets
targets
targets
Figure 4: Response statistics in the trained network rapidly adapt after a stimulus change. The network initially in steady state, with responses producing (approximate) samples from the posterior corresponding to some stimulus (red) input. At 0 ms, new stimulus (blue) replaces the old one as the input to the network. As a result, network responses evolve over time to represent the new posterior distribution. Each column represents the state of the network some time & after new stimulus onset (top). A. Input image into the network (top left) and current image "perceived" by the network, reconstructed from the neural responses (bottom right). B-C. Membrane potential means (B) and variances (C) of excitatory neurons (dots) versus target GSM posterior means corresponding the old (blue) and new (red) stimuli, respectively.
In summary, our method provides two advantages. First, it allows training networks with large number of free parameters, which in turn achieves much lower cost than the more highly constrained networks to which previous training methods were restricted (first two rows). Second, focused approach successfully trains networks to perform inference under a complex GSM of natural images, feat which was previously impossible (last two rows). Once again, in all 100 independent training attempts, not a single network became unstable. Full-sized stable networks are also impossible obtain by random generation, further justifying our approach.
stability-
4.4 Analysis of the trained network
It is interesting analyze the trained SSN and explore its dynamics. For this, reverse-engineer the network in order to obtain the image that it "perceives" at a given time by interpreting the mean membrane potentials of its excitatory neurons as the latent activations of the GSM, from which an image is reconstructed using the GSM's projective fields and generative process (Eq. 8). We observe the phenomenon of percept morphing (Fig. 4/)), driven by membrane potential moments initially matching the GSM targets corresponding to the old stimulus, and gradually evolving over time instead match those corresponding to the new stimulus (Fig. 4B-C). While the target means are matched with high precision and within 440 ms (Fig. 48). variances are slightly less precisely matched and take longer converge (Fig. 4C), demonstrating the difficulty of matching higher-order moments. Furthermore, membrane potential means transition between the two posteriors faster than membrane potential variances From an MCMC perspective, this suggests a difference in mixing times for the two quantities.

---

5 Discussion
Network stability is problem faced in both artificial [3 40 and biological neural networks [17] Typically, artificial neural networks (ANNs) can implement quick-fix solutions that will not impact their performance negatively, such applying batch and layer normalizations [41. Furthermore, artificial neurons may be designed to be stable, e.g. by simply choosing a saturating activation function (tanh, sigmoid) or by implementing other souudeddinput-bbuude---oom
designs.
In general, the concept of growing networks during training has recently been proposed ANNs. Previous work combined both network growing and pruning to produce efficient networks [ allowed the splitting of an existing neuron into multiple neurons [30]. and used principles similar to ours for adding nodes without impacting the operation of the existing network (32). Broadly speaking, the two primary goals of these algorithms were to reduce training load and improve performance. Here, given its particular relevance for biological networks, instead focused on the even more basic requirement of staying within a stable dynamical regime.
Building network gradually during optimization may adversely afftect the speed and scalability of our algorithm because requires computationally expensive function retracing operations every time the network grows and thus changes in architecture. We justify this with two reasons. First, there presently no other effective way of training SSNs. Second, the computational resources spent on optimization are still much greater than those spent on tracing, especially number of tracing operations required (until the network reaches full size), but a much larger number of gradient computations and optimization iterations (which may need to continue even once the network reaches full size). Better balancing network building and its associated computational costs is an important future direction.
there is only a finite
Our method may be further improved by fine-tuning its hyperparameters, such as the number of neurons to duplicate at the same time, or the criteria for selecting which neuron(s) to duplicate. In addition, our method may also be combined with methods based on fundamentally different approaches, such as adding regularization terms to the cost function explicitly encouraging networks to stay within a stable regime [27].
Throughout our analysis, we use backpropagation through time with Adam [44] to train SSNs (see Appendix C for additional information on the training procedure). This is because our primary objective is to build biologically-realitt characteristics after they have been optimized. Future work will need to study how biologically plausible plasticity mechanisms may achieve the same goal.
networks so as to study their dynamics and network
Finally, as example of a computationally particularly challenging and (for V1) biologically relevant task, we chose to train SSNs to perform sampling- inference [29]. Our method is readily applica- ble training SSNs other cognitive tasks engaging a number of different cortical areas, including the prefrontal cortex [16]. opening the way to studying SSNs beyond primary sensory cortices.
Conclusion
We present an effective method for training SSNs, a canonical model of cortical circuits highly prone to exhibit dynamic instabilities. To demonstrate the generality of our method, we train SSNs to perform MNIST image classification. We also train SSNs to perform aappling-based inference under a GSM optimized for CIFAR- 10 images, a previously unattainable task. Our contribution also makes it possible to train SSNs other neurobiologically relevant cognitive tasks and thus study the network dynamics that underlie the complex computations performed by the brain.
Acknowledgments and Disclosure of Funding
This work was supported by the Wellcome Trust (Investigator Award in Science 212262/7/18/2 M.I.) and the Human Frontiers Science Program (Research Grant RGP0044/2018 to M.L.).
Author contributions
W.S. developed the algorithm and performed the experiments. Both authors designed the study, interpreted the results, and wrote the paper.
10

---

References
[1] McCulloch, W. & Pius, W. A logical calculus of the ideas immanent in nervous activity. The Bulletin of Mathematical Biophysics 5, 115-133 (1943).
[2] Hopfield, J. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences 79, 2554-2558 (1982).
[3] Van Vreeswijk, C. Sompolinsky, H. Chaos in neuronal networks with balanced excitatory and inhibitory activity. Science 274, 1724-17277 1726 (1996).
[4] Rubin, D. B.. Hooser, S. D. V. & Miller, K. D. The stabilized supralinear network: A unifying circuit motif underlying multi-in multi-input integration in sensory cortex. Neuron 85, 402-417 (2015).
[5] Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. Communications of the ACM 60, 84-90 (2017).
[6] Goodfellow, et al. Generative adversarial nets. In Advances in Neural Information Processing Systems, vol. 27 (2014).
[7] Vaswani, A. al. Attention is all you need. In Advances in Neural Information Processing Systems, vol. 30 (2017).
[8] Ramesh, A. et al. Zero-shot text-to- tonttto-mmaee generation. Proceedings of the 38th International Conference on Machine Learning, vol. 139 of Proceedings of Machine Learning Research, 8821-8831 (2021).
[9] Mante, V., Sussillo, D., Shenoy, K. V. & Newsome, T. Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature 503, 78-84 (2013).
[10] Yamins, D. L. K. et al. Performance-opiinizze hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences 111, 8619-24 (2014).
[11] Song, H. F., Yang, G. & Wang, X.-.. Training excitatory-inhibitonn recurrent neural networks for cognitive tasks: A simple and flexible framework. PLOS Computational Biology 12, e1004792 (2016).
[12] Orhan, A. & Ma, W. Efficient probabilistic inference in generic neural networks trained with non- probabbbbiiiittcc feedback. Nature Communications 8 (2017).
[13] Banino, A. et al. Vector-based navigation using grid-like representations in artificial agents. Nature 557, 419-4294432 (2018).
[14] Cueva, J. & Wei, X. Emergence of grid-like representations by training recurrent neural net- works to perform spatial localization. In International Conference on Learning Representations (2018).
[15] Remington, E., Narain, D., Hosseini, E. Jazayeri, M. Flexible sensorimotor computations through rapid reconfiguration of cortical dynamics. Neuron 98, 1005-1019 (2018).
[16] Yang, G. R., Joglekar, M. R., Song, H. F., Newsome, W. T. & Wang, X.-. Task representations in neural networks trained to perform many cognitive tasks. Nature Neuroscience 22, 297-306 (2019).
[17] Ahmadian, Y., Rubin, D. B. Miller, K. D. Analysis of the stabilized supralinear network. Neural Computation 25, 1994-2037 (2013).
[18] Wilson, H. R. & Cowan, J. D. Excitatory and inhibitory interactions in localized populations of model neurons. Biophys 12, 1-24 (1972).
[19] Li. Z. & Hopfield, J. J. Modeling the olfactory bulb and its neural oscillatory processings. Biol Cybern 61, 379-392 (1989).
[20] Adini, Y., Sagi. D. & Tsodyks, M. Excitatory-nniibtonn network in the visual cortex: Psy- chophysical evidence. Proceedings of the National Academy of Sciences 94, 10426-10431 (1997).
11

---

[21] Litvak, V., Sompolinsky, H., Segev, I. & Abeles, M. On the transmission of rate code in long feedforward networks with excitatory-inhibiic balance. Journal of neuroscience 23. 3006-3015 (2003).
[22] Ahmadian, Y. & Miller, K. D. What is the dynamical regime of cerebral cortex? Neuron 109, 3373-3391 (2021).
[23] Obeid, D. & Miller, K. D. Stabilized supralinear network: Model of layer 2/3 of the primary visual cortex. bioRxiv 10.1101, 2020.12.30.424899 (2021).
[24] Kraynyukova, N. & Tchumatchenko. T. Stabilized supralinear network give rise to bistable, oscillatory, and persistent activity. Proceedings of the National Academy of Sciences 115, 1664-3499 (2018).
[25] Hennequin, G., Ahmadian, Y., Rubin, D. B.,, Lengyel, M. & Miller, K. D. The dynamical regime of sensory cortex: Stable dynamics around a single stimulus-tuned attractor account for patterns of noise variability. Neuron 98, 846-860.e5 (2018).
[26] Priebe, J. J. & Ferster, D. Inhibition, spike threshold, and stimulus selectivity in primary visual cortex. Neuron 57, 482-97 (2008).
[27] Festa, D., Hennequin, G. & Lengyel, M. Analog memories in a balanced rate-based network of
E-I neurons. Advances in Neural Information Processing Systems 27 2231-2239 (2014).
[28] Eckmann, S. & Gjorgjieva, Synapse-type-speeifie competitive Hebbian learning forms functional recurrent networks. bioRxiv 10.1101, 1....00..11.48899 (2022).
[29] Echeveste, R., Aitchison, L., Hemnequin, G. & Lengyel, M. Cortical-like dynamics in recurrent circuits optimized for sampling-bass -based probabilistic inference. Nature Neuroscience 23, 1138- 1149 (2020).
[30] Wu, L., Wang, D. & Liu, Q. Splitting steepest descent for growing neural architectures. In Advances in Neurul Information Processing Systems, vol. 32 (2019).
[31] Yuan, X., Savarese, P. H. P. Maire, M. Growing efficient deep networks by structured continuous sparsification. In International Conference on Learning Representations (2021).
[32] Evci, U.. van Merrienboer, B.. Unterthiner, T., Pedregosa, F. & Vladymyrov, M. Gradmax: Growing neural networks using gradient information. In International Conference on Learning Representations (2022).
[33] Deng, L.
The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine 29, 141-142 (2012).
[34] Wainwright, M. J. & Simoncelli, E. Scale mixtures gaussians and the statistics of natural images. In Advances in Neural Information Processing Systems, vol. 12 (1999).
[35] Krizhevsky, A. & Hinton, G. Learning multiple layers of features from tiny images. Tech. Rep. 0, University of Toronto, Toronto, Ontario (2009).
[36] Orbán, G., Berkes, P., Fiser, J. & Lengyel, M. Neural variability and sampling-based probabilis- tic representations in the visual cortex. Neuron 92, 530-543 (2016).
[37] Festa, D., Aschner, A., Davila, A., Kohn, A. & Coen-Cagli, Neuronal variability reflects probabilistic inference tuned to natural image statistics. Nature Communications 12 (2021).
[38] Hennequin, G. & Lengyel, M. Characterizing variability in nonlinear recurrent neuronal networks. arXiv 10.48550, 1610.03110 (2016).
[39] Bai, S., Kolter, 7. Koltun, V. Deep equilibrium models. Advances in Neural Information Processing Systems 32, 699 (2019).
[40] Bai, S., Koltun, V. & Kolter, J. Z. Stabilizing equilibrium models by jacobian regularization. In Proceedings of the 38th International Conference on Machine Learning, vol. 139, 555-565 (2021).
12

---

[41] Ioffe, S. & Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, 448 456 (2015).
[42] Cooijmans, T., Ballas, N., Laurent, C., Güilçehre, C. & Courville, A. Recurrent batch normaliza- tion. arXiv 10.48550, 1603.09025 (2016).
[43] Santurkar, S., Tsipras, D., Ilyas, A. & Madry, A. How does batch normalization help optimiza- tion? In Advances in Neural Information Processing Systems, vol. 31 (2018).
[44] Kingma, D. P. & Ba, Adam; A method for stochastic optimization. arXiv 10.48550, 1412.6980 (2014).
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] In our abstract and introduction, we claim that we have developed a method to train supralinear stabilized networks. This method can be found in in[Section 2] We also claim that we have successfully trained the network to perform campling-based inference on an internal generative model of natural images. These results can be found summarized in [Section 4, Finally, we claim that our method is an order magnitude better than previous approaches. This can be found in Table 2) (b) Did you describe the limitations of your work? [Yes] We have highlighted some potential limitations in /Section 5
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We have not found any potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Section 2.2
(b) Did you include complete proofs of all theoretical results? [Yes] Section 2.2
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] The link to our code can be found in the supplementary material. Additional information required to replicate our results are also included.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All parameters of the model are clearly introduced early in the text. Numerical values assigned to all parameters can be found in the supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Tables I] and 2
(d) Did you include the total amount compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] Computational resources and related information can be found in the supplementary material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We have used CIFAR-10 and MNIST and have cited the creators.
(b) Did you mention the license of the assets? [Yes] We have linked the licenses in the supplementary material.
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] CIFAR-IO and MNIST are publicly available datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] This can be found in the supplementary.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "a7-9HTwuywi", "reviewer": "Reviewer_PiTG", "summary": "This paper studies stochastic stabilized supralinear networks (SSNs), a firing rate recurrent neural network model of interest in computational neuroscience. The parameters of these models are typically set by hand, at random, or using semi-analytical methods. The current work approaches the problem of learning the parameters of stochastic SSNs, aiming at developing a method that can scale to large numbers of parameters. The authors focus on the problem of learning to do amortized inference for a well-known Bayesian model for vision, by optimizing a moment-matching objective function.", "questions": "Two things weren't very clear to me:\n- Why is the stochastic SSN particularly well-suited to (approximately) sample from a GSM?\n- How were the target moments obtained, for the objective function (11)?", "limitations": "The discussion is sufficient for the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clear, well written, and pleasing to read. It does a very good job in summarily explaining the current interest in the SSN in neuroscience, and the challenges (and interests) in optimizing it with deep learning techniques.\n\nThe one weakness I see in this paper is that it is purposefully written as a somewhat \"niche\" paper. The problem of learning to do amortized inference on a GSM is both interesting and challenging enough (thus well chosen) for the current paper, but the exclusive focus on it may deter many from reading the paper.\n\nI think it would be possible to write a paper that would be far more reaching within the NeurIPS community and beyond, by abstracting what I understood to be the key challenge here: training recurrent neural networks with the expansive nonlinearity of eq. 10. There has been interest throughout the years in training recurrent neural networks with unbounded nonlinearities (typically rectified linear units, see e.g. the path-SGD paper by Neyshabur et al., 2015. The present paper fits nicely along this line of work, while focusing on the even more challenging case of an expansive nonlinearity. The paper could also discuss more explicitly previous work on neural networks that grow throughout training.\n\nA small section on the easier case of deterministic SNs could be nice too, as one could make direct contact with an exciting line of recent work on so-called deep equilibrium models, which also face stability issues, and see what the current method brings there. It seems to me that some of the techniques investigated in this line of work to remedy instability (see e.g. Bai et al, ICML 2021) could be brought directly to deterministic SNs, and perhaps adapted to stochastic SNs?\n\nThere is a small number of typos in the paper. Also, there is no confusion, but perhaps it's better to denote a minimizer by arg min instead of min? I guess the F subscript can be dropped in eq. 16, as the argument of the norm is a vector?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "V16a02WvkT", "reviewer": "Reviewer_rMNf", "summary": "Stabilized supralinear networks (SSN) offer a biologically plausible model of cortical neural circuits, however, previous works have had difficulty training these models in a stable way due to the model's construction. This work focuses on how to effectively train these models using gradient-based techniques and relatively few constraints, specifically in application to the task of using SSNs as a recognition model that performs probabilistic inference under a Gaussian scale mixture (GSM) generative model.  The training method consists of initially starting with a small SSN that is easier to initialize stably, and then gradually building the network up during optimization. An iterative algorithm is used where the randomly initialized network is simulated to check for convergence. If it converges, statistics are collected and used in a differentiable loss function to generate a new set of parameters. After a certain number of iterations, more neurons are carefully added to the network. The procedure repeats until the network is fully built out and the optimization of parameters is complete.\n\nThe results of the new method are compared to the existing setup in the previous work [35], and are shown to significantly reduce the cost. The network is also used to perform inference in a GSM optimized for CIFAR-10. \n\n", "questions": "1. The paper would benefit from a more thorough explanation of the first experiment in section 4.1 and the \"existing setup in [35]\" referenced in line 209. What is the training and test data? The training size is 5? What is the biased semi-analytical method used by the previous method? What are the previous method's free parameters?  A few related details are sprinkled throughout the text (lines 80-81, 146-147), however, a consolidated high-level description in Section 4.1 (and perhaps a more detailed description in the appendix) would help the reader better understand the comparison and superiority claims. \n\n2. Partly related to the point above, I do not understand the evaluation method used for the CIFAR-10 experiment. It seems the previous method was not evaluated on CIFAR-10, and instead some kind of correction is applied? Why was the previous method not just evaluated on CIFAR-10 also? Perhaps it is not feasible to train the previous method on the larger dataset? If the latter is the case, stating that explicitly would be helpful. What should I take away from comparing the different methods with different targets and why is this valid? Is 0.788 vs 1 a significant improvement? These are questions I have that currently make it hard to evaluate the claims related to this experiment. \n\n3.  I think the point of the overfitting discussion in Section 4.2 is that it is an accomplishment just to be able to train this relatively unconstrained model and get good performance on the training set, and the overfitting can be easily handled with regularization. But why not just directly show this by applying some of the referenced regularization methods? Is there something about the current setup that makes this nontrivial?\n\n4. Generality: The entire paper is geared around training SSNs as a recognition network for a GSM model. A general reader may come away from the paper wondering if this is the only way this method can be used? The answer appears to be no and the authors mention their desire to apply the approach to cognitive tasks as in [16] for future work. But the current organization of the paper makes it unclear how well this method can be adapted to these other problems. I would perhaps recommend presenting the general approach first outside of the context of the recognition network setting, and then show how the probabilistic inference method can be used in the context of the recognition network for the GSM model and any specifics related to that. Ideally, there might also be another experiment, even if simple or synthetic, where this approach is used for a different setup. This is just a suggestion that I think could strengthen the presentation.\n\nMinor points:\n\n- Typos in lines 95-96 (\"is neural noise is\")  lines 123 (\"the some\") \n- I think a better explanation of coding vs non-coding neurons in Section 2.3 would help readers less familiar with these models. \n- I would recommend using a different letter than $n$ for the variable in equation 10 and line 126. This same letter $n$ is later used to refer to specific neurons in section 3.3, and this could lead to confusion for the $n=2$ point in lines 181. \n- I think parenthesis are missing in Equation 16? It seems like it should be $\\mathbb{E}[u] - (\\mathbb{E}[f(h)] + W\\mathbb{E}[r])$.\n\n\n\n\n ", "limitations": "The authors adequately discuss limitations such as concerns around how well this method will scale computationally to larger problems. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Originality: The idea to use the evolutionary approach to gradually build the network up to avoid stability problems appears to be novel for SSNs. The approach of interleaving gradient optimization steps with the evolutionary steps also appears to be original. In addition, this appears to be the first practical method to train SSNs without the constraints previous approaches required. However, the paper would benefit from a more detailed discussion of evolutionary approaches such as this (mentioned briefly in lines 246-248) as well as the previous attempts to train SSNs (mentioned briefly in the introduction) to help the reader understand the literature landscape and the specific contributions of this paper.\n\n- Quality: The general approach and mathematical details appear to be sound. The approach seems to outperform the previous method on the existing task (though see Questions section below for concerns about the explanation of this setup).  The approach was also used for a more challenging task of optimizing the GSM on CIFAR-10 images. However, clarity issues related to the explanation of the comparison for this task currently make the claims of improved performance on this task hard to evaluate (See detailed questions concerning this below).\n\n- Clarity: The paper is generally written well, but various typos in the text and equations do make it less smooth to read at times. See the questions section below for some specific examples, but thorough proofreading would help the presentation. In addition, as mentioned in the Quality section above (and in more detail in the Questions section below), there seem to be important details related to the experiment setup that are missing which makes it confusing and hard to evaluate the empirical claims.\n\n- Significance: The ability to effectively train SSNs at scale could have an important impact on the computational neuroscience community. I think this paper makes a step in that direction. The ideas of incorporating gradient-based optimization along with potential topological changes during training could be of interest to the broader machine learning community.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LRXvqBoH8MI", "reviewer": "Reviewer_2UAi", "summary": "This paper presents a method for training a neuro-plausible artificial neural network.  _Stochastic supralinear networks_ (SSNs) have been shown to have favourable parallels to real cortical neural networks and dynamics.  This is in-part owing to the network's flexible parameterization and reduced constraints.  However, this flexibility and lack of constraints leads SSNs to be prone to instability (from recurrent feedback loops) and difficult to train at scale, without (re-)introducing additional constraints.  The authors present a method where the size of the network is grown during training, and is able to reliably reach far larger network sizes and parameter counts than preceding methods.  The method is benchmarked on an existing and new task, and appears to perform favourably.   \n", "questions": "I raise several questions in my main review.  ", "limitations": "I don’t see any particular unaddressed limitations of the work as presented.  That said, it is ostensibly an empirical study of a design heuristic, and therefore the robustness and scalability is uncertain.   \n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Review Summary\n\nI will preface my review by saying that I am not a domain expert.  While I have a feel for neuro-plausible networks, I am not too familiar with the general architecture or existing work.  \n\nThe paper itself is fairly well written, seems to be thoroughly referenced, and there are helpful figures and diagrams that help to take some explanatory burden off of the text.  The work also appears theoretically sound.  The model/architecture itself is clearly and succinctly described.  The description of the method is a little lacking, and the description of the experimental setup and objectives is also a little lacking.  I do think a method like this could be well received, and would definitely have an audience at NeurIPS.  \n\nThat said, I am not going to endorse publication at this stage.  The thrust of my criticisms are (1) the foundations of the method, (2) some aspects of the presentation, and (3) the evaluation.\n\n\n# Major Comments:\n\n### 1.  The Method Itself\nThe core concept of the method is that it is easier to grow a stable network into a slightly larger stable network, than to outright train a large and stable network.  However, this is essentially a heuristic.  It is not clear to me that this is a robust method for generating a network.  There are two phases to the method:  initialisation and growth.  The initialisation is essentially trial-and-error in the base case of the “induction”.  The growth phase is then hoping to expand the network capacity in such a small step that it remains stable – but it is also a heuristic, that is essentially trial-and-error, in which the chance of “error” is minimised.  It is appealingly simple, but it lacks the rigour for me to be super excited / confident in the methodology.  \n\nThere is also little exploration of \"hyper parameters\"/choices, such as how frequently the network should be grown, further heuristics for which neurons are split, the implications of splitting highly active versus occasionally active neurons etc.  I feel like the method has not been fleshed out enough for me to be confident that it is a general, robust and extensible framework.  \n\nThere are also unanswered questions about how the choice of coding and non-coding neurons is made?  Is it 50-50?  Is there are more reasoned way of choosing this?  How are neurons allocated to c/nc during growth?  How is the network initialised in time, and how long is the network run for? \n\n1.1.  Is it reasonable to expect the latent states of an RNN to reflect samples from a posterior?  I can't quite wrap my head around this, but I think that there should be some kind of decoder that maps the RNN state onto the samples?  Especially for neuro-plausibility, right?  I struggle to see why an SNN is the right architecture / amortised MCMC is the right test for this.  I invite the authors to comment.\n\n1.2.  I also think ~80% of neurons are excitatory.  It would be nice if the SNN reflected that.  \n\n1.3.  I'm not sure how this would work, but it would be nice if there was a method for adding neurons in a better way.  I.e. if the network activity is low, add excitatory, if it is high, add inhibitory.  If a particular neuron is too active, add an inhibitory connection etc.\n\n1.4.  Is the weight matrix dense?  Again, it would be nice if connections were sparse with some target sparsity / L1 regularisation.  \n\n\n### 2.  Presentation\nA major point I am struggling with is the presentation of the method.  I believe the method itself is the idea of starting small and growing the network by adding new neurons.  I don’t fully understand why the presentation is so tied up in the GSM – which I think is just an example of where this could be applied, or indeed, a method for generating targets for supervised learning.  Sampling-based inference is also just an example application.  I feel like the intermingling of application and model/method in the exposition is maybe indicative of them being mingled in the authors minds?  It makes it very difficult for a reader to tease them apart, or indeed, how the authors intended them to be teased apart.  I wish the method was clearly presented in terms of inputs and outputs -- irrespective of how they were generated, their spaces, types, interpretation etc.  \n\n2.1.  Similar to this, but also related to (1) and (3), can you explain how the network actually performs inference, and how this inference is evaluated?  I understand it as the visual stimuli is repeatedly presented to the learned network as in (7).  Each timestep is then treated as a sample of the mean and covariance?  How do you then train this against the single and static values of $\\mu_{GSM}$ and $\\Sigma_{GSM}$?  How is this inference shown on Figure 2 & 3?  What are the different dots?  How are there different “Target” values?  What is the difference between red, blue and purple?  (And can we use more visually distinct colours please!)  Basically, F2, F3, the experimental task and the results need much more thorough explanation.  \n\n2.2.  I would also like to see an algorithm block or block-diagram of the method?  I think it is an appealing simple method, but a visualisation of this would still be nice.  \n\n2.3.  I also think calling it a “toolbox” is misleading – is at most a “method”, and, at least, a “heuristic”.  \n\n2.4.  If I understand correctly, the GSM is trained ahead of time to generate the supervised targets for learning the SSN?\n\n\n### 3.  Empirical Evaluation\nI find the empirical evaluation difficult to judge.  I am not too familiar with what a “standard” task is in this domain, but the task itself seems fairly abstract.  The performance metrics are also a little obscure and don’t have much explanation or justification.  \n\n3.1.  As suggested in (1), I’d like to see some ablations/variants of the method.  I would also like to see how repeatable the results are across random seeds.  I would like to see some form of quantitative error metric with error bounds to at least indicate how reliable this method is.  How many runs went unstable for each method during training?  \n\n3.2.  I also feel like the range of experimental evaluation is quite limited.  I understand that the network operates slightly differently to, for instance, AlexNet.  That said, i would still like to see how this network/method applied to slightly more “standard” NN tasks.  You certainly don’t need to achieve state of the art, but, if all i can reasonably use this for is MCMC in a particular model, then the scope and impact of the method is limited (which is a shame!).  This should be a method for generating a neural encoding for performing a task (be it classification off of the final value, average value in time, the raw samples etc).  It would also be interesting to see if this network can learn an encoding for time-series tasks. \n\n3.3.  Unless I have misunderstood something, I feel like saying it is performing amortised MCMC is also a bit of a stretch.  It is essentially doing moment-matching using supervised learning.  \n\n\n# Minor Comments.\n\na.  Figure S2 is a cool figure, and I think would add real, concise impetus to the motivation of your method if included in the main.  \n\nb.  Why is the training loss so much higher for the larger network?\n\nc.  Why do you do moment matching in (11)?  I would have thought you’d have done maximum likelihood, given that you can get out the distributions under the GSM.  \n\nd.  How are the Gabor filters computed and applied?  From Figure 1 it looks like there are multiple Gabor filters that are used, but there appears to only be one $A$ matrix.  It isn;t even really clear to me how (2) is used.  Where is $\\phi$ used?\n\ne.  I think the introduction can be slashed in length.  There is a lot of information in there that isn’t really relevant – get to the nub of the contribution:  a method for training neuro-plausible SSNs.  I also think Section 6 can be cut to make space.  \n\nf.  I don’t understand the “old stimulus” and “new stimulus” rows of Figure 3.  Can you explain them please? \n\n\n# Very Minor / Typographical Comments.\ni. Section headers should be in capital case, as opposed to sentence case. \n\nii. The abbreviation of excitatory and inhibitory to E and I makes those sentences harder to read.  I wouldn’t use the abbreviation.  \n\niii. Putting the variable at the end of the square brackets for an expectation is non-standard and makes it harder to read.  They should be placed after the expectation, e.g. $\\mathbb{E}_{z} \\left[ \\cdot \\right]$.  \n\niv. It would be handy if you linked the equations together more.  For instance, explicitly reference the pseudo-inverse in (12) to when it was introduced in (8).  It just helps tie the relevant math together.  \n\nv.  Line 98:  “optimise” -> “optimize”  (you have used US English elsewhere).\n\nvi.  What is the subscript $F$ in the norms?  Frobenius norm?  \n\nvii.  In (12), how do you define the variance of a vector?  You take the variance within the vector?  Or the variance of elements between vectors?  Basically, $\\mathbb{V}$ is undefined.\n\nviii.  In (11), $\\mathbb{C}$ is normally reserved for the space of complex numbers, as opposed to covariance.  Maybe use $\\mathbb{C}\\mathrm{ov}$ to be explicit. \n\n\n# Summary.\nAlthough my review comes off as quite negative, I do generally like the paper – I just do not think it is ready for publication at this point in time.  That said, if the authors can address my comments, then I would consider upgrading my review.  \n\nI implore the authors to explore theoretically justifying aspects of the method, or, performing further ablations/variations/experimental types to explore the method.  With these enhancements, this could be a very strong conference submission.  \n\nGood work, and good luck.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept