Noise Transforms Feed-Forward Networks into Sparse Coding Networks
Authors: Trenton Bricken
Bruno Olshausen
Gabriel Kreiman
Venue: Submitted to ICLR 2023 (2023)
Keywords: Sparse Coding
Sparsity
Top-K Activation
Noise
Biologically Inspired

Abstract: A hallmark of biological neural networks, which distinguishes them from their artificial counterparts, is the high degree of sparsity in their activations. Here, we show that by simply injecting symmetric, random, noise during training in reconstruction or classification tasks, artificial neural networks with ReLU activation functions eliminate this difference; the neurons converge to a sparse coding solution where only a small fraction are active for any input. The resulting network learns receptive fields like those of primary visual cortex and remains sparse even when noise is removed in later stages of learning.

Under review as a conference paper at ICLR 2023
NOISE TRANSFORMS FEED-FORWARD NETWORKS INTO SPARSE CODING NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
A hallmark of biological neural networks, which distinguishes them from their artificial counterparts, is the high degree of sparsity their activations. Here, we show that by simply injecting symmetric, random, noise during training in recon- struction or classification tasks, artificial neural networks with ReLU activation functions eliminate this difference; the neurons converge tion where only small fraction are active for any input. The resulting network learns receptive fields like those of primary visual cortex and remains sparse even when noise is removed in later stages of learning.
sparse coding solu-
INTRODUCTION
The brain is highly sparse with an estimated 15% of neurons firing at any given time (Attwell &
Laughlin| 2001). ) The most immediate answer for why is metabolic efficiency: action potentials
consume ~0% of the brain's energy (Sterling & Laughlin 2015) Attwell & Laughlin 2001 Sen- gupta et al. 2010). However, there are further advantages to sparsity in the brain (Olshausen &
Field, 2004). J One significant advantage is improving the signal to noise ratio (SNR) of neural sig- nals. Sparsity improves SNR by (i) turning off any weakly firing neurons activated by noise; (ii)
increasing the separability of data points (Ahmad & Scheinkman 2019 Xie et al. 2022).
Inhibitory interneurons that suppress all but the most active neurons from firing are an important mechanism for enforcing this sparsity (Haider et al.) 2010). Theoretical and empirical results sup- port their involvement in both silencing noise and separating neural representations. Examples in-
clude horizontal interneurons in the retina (Sterling & Laughlin 2015) and Golgi interneurons in cerebellar-lin like structures (Fleming et al. 2022 Lin et al. [014]] Xie al. 2022).
Biologically, as depicted in Fig. M these inhibitory interncurons implement a negative feedback loop whereby the more active the excitatory neurons are, the more active the interneuron becomes, and hence the more it inhibits these excitatory neurons. Simplified models the circuit written as ordinary differential equations (ODEs) show convergence an approximate k of the most active neurons remaining on Goozel & Gerstner, 2021). This will be referred to as Top-K activation function (also known Winners Take All). There is empirical support for number of interneuron circuits approximating the Top-K operation (Sterling & Laughlin 2015 Fleming et al. 2022
al. 2014).
By contrast, in the field of deep learning, while inhibition is possible, analogous interneuron circuits that enforce sparsity across layer have not been widely adopted. The only truly sparse activation function is the ReLU (Glorot et al., 2011). Moreover, mechanisms that enforce sparse neuronal activity are also rarely used and when given the choice, networks will prefer to dense. This be-
cause sparsity can limit model capacity, resulting in information bottlenecks that harm performance Goodffllow et al. 2015).
Here, we find that by simply introducing isotropic, symmetric noise centered about zero during train- ing, a layer of artificial neurons will converge to a sparse coding solution. This solution mimicks simplified version of the biological inhibitory interneuron circuit. The network gradually imple- menting this inhibitory interneuron also results better performance than explicitly enforcing any sort of inhibition at the start of training.
Concretely, the network synchronizes every neuron's 's bias term, setting them to be approximately the same negative value, and also every neuron's weight vector, setting them have the same Tzg norm.

---

Under review as a conference paper at ICLR 2023
This results in the activity of every with the negative bias term they all agree upon, results a sparse neurons remaining active.
existing within a particular range that, when combined
Because assume that & is small, Top-K networks that always have an approximate neurons on for any input. They also implement this sparsity by approximating the functionality of inhibitory interneuron. The sparsity that comes from inhibition allowing a small subset of neurons to fire from the total number that otherwise would distinct from sparsity where many of the neurons in the network are dead and never fire for any input. This latter form of sparsity is misleading and equivalent to having a pruned, smaller network that is densely firing, we indicate when this alternative form of sparsity occurs.
special class of sparse coding networks
The degree to which our network becomes a sparse coding, Top-K network is proportional the amount of noise applied, up noise limit. To validate that the network approx- imates an inhibitory interneuron, we replace each neuron's bias term with a single, shared bias term. This results in identical model performance and very similar levels of sparsity. Further investigating the degree to which this network uses sparse coding, learns receptive fields similar to those of mammalian VI with Gabor filters and retinal gan- glion cells with on/off center surround (Sterling & Laughlin 2015 Olshausen & Field 1997).
find that it
We find this Top-K K network formation is particularly evident for reconstruction tasks but also exists for classification tasks and the intermediate MLP lay- ers of a Transformer architecture. Our results hold across variety of datasets, noise distributions, and numbers of neurons. We also observe that the net- work retains its Top-K approximation even after the noise is removed, making it an effective pre-training task for sparsifying activations. This increase
Excitatory Neurons
Inbibitory. Interneuron
Input
sparsity could reduce FLOPs when run on hardware that can take advantage of (Wang. 2020: Gale
Exodiforeant Enchatary
bookback iniibboy
ett al. 2020 Davies et al. 2018).
We first review related work (Section before out- lining our experimental setup and presenting empiri- cal observations (Section() B}. Finally, we analyze the network's learning dynamies, providing intuition for our results and highlighting avenues for future work (Section 4h
Figure 1: An inhibitory interneuron circuit implementing a negative feedback loop to si- lence all but the & most active neurons.
RELATED WORK
Injecting noise into training data for statistical models was previously proposed by (Sietsma Dow 1991]. Gaussian noise injection has been interpreted form of model regularization, to help avoid overfitting (Zur et al. 2009) and improve generalization (Sietsma & Dow 1991; Matsuokal 1992).
was later shown that noise injection is in fact equivalent to having regularization term that seeks to minimize the L2 norm of the network's ' Jacobian (Bishop) 1995 Rifai et al.) 2011 Alain & Bengio 2014). From this perspective, it makes sense that sparsity can used to turn off neurons and minimize this Jacobian, providing potential explanation for our results.
Training with noise became particularly prominent in the form of de-noising autoencoders (Good- fellow et al. 2015). Before deep neural networks with many layers could be trained end-to-end, it was popular to train each layer of the network one a time with an unsupervised, de-noising recon- struction loss (Goodfellow et al. 2015). With small amounts of noise, these de-noising autoencoders were shown to equivalent to contractive auto-encoders that were regularized to be robust to local perturbations of the training data (Alain & Bengio 2014).
Approaches to introduce activation sparsity have included explicitly using Top- activation func- tions (Razzato al. 2007] Makhzani & Frey 2014 Ahmad & Scheinkman, 2019), novel regular- ization terms (Kurtz et al. 2020] Yang et al., 2020) and other approaches (Schwarz et al., [02TT]
Molchanov al. 2017).
However,
unaware of any existing work which finds that noisy
2

---

Under review as a conference paper at ICLR 2023
training with ReLU activation function not only results in sparsity but also converges to sparse coding, Top-K network.
Learning reconstructions of training data with sparse activations is the mainstay of sparse coding (Olshausen & Field 1997). Sparse coding uses an overcomplete basis to learn a reconstruction of data with an L1 regularization penalty neuron activity. The image reconstruction can be written as x Σx a, where de basis vector from the matrix or "dictionary" d; a, the activity of this basis vector, and the activation weighted sum of basis vectors produces the output x. The objective function aims to minimize the mean squared-error between the reconstructed image and the original image while having sparse neuron activity: arg ming, a (x - x)² + A Σ las]- This sparse coding task is example of statistical model converging with neuroscience because, in addition to producing sparse representations, the basis functions learn to become Gabor filters like those found in area VI of the visual system (Olshausen & Field 1997).
Interestingly, the ReL.U activation function that is crucial for the results of this paper can be derived the optimal solution simplified sparse coding problem when we assume that the basis vectors are orthogonal (Bal 2020 Gregor & LeCun 2010). This results in the optimal solution for activity a, be given by ReLU (u; A) ReLU un A) or, if we constrain a, to be non-negative, ReLU((; - A), where u, is the value obtained by projecting the data onto basis Φ,. Intuitively, if the values of our target image x reconstructed by a, are too small, it better for a, than predict x, where the threshold for this tradeoff is given by X, which behaves an activation threshold.
Top-K activation funetions inspired by inhibitory interneurons have been implemented in a number of deep learning models for greater computational efficiency (Ahmad & Scheinkman| (019)) adver- sarial robustness (Xiao et al., 2020 Tramer et al. 2020), interpretability (Nelson et al. 2002), and continual learning (Shen et al. 202 Iyer et al. 2022). However, in all of these instances there are explicit mechanisms built into the network to enforce Top-K and handle repercussions such as creating dead neurons.
These Top-K networks, when trained with noise, are also closely related to associative memory models such as Sparse Distributed Memory (SDM) and Modern Hopfield Networks that activate a sparse subset of neurons and were developed to handle noisy queries (Kanerva 1988 Krotov & Hopfield, 2016). SDM in particular can be viewed
de-noising autoencoder and has been developed into an MLP that uses a Top-K activation function (Keeler 1988). Finally, de-noising
autoencoders are single step versions of diffusion models (Song & Ermon 2019
3 RESULTS
Experimental Setup - By default, we train a single hidden layer ReLU network with different amounts of random Gaussian noise applied to the input. The objective is to remove the noise and reconstruct the original input. Formally, noise is applied from an isotropic Gaussian with variance the input The corrupted image is then put through our encoder and decoder. For the single hidden layer case of Eq. the encoder and decoder weight matrices and bias terms are: W. E pmxa b E Rm Wa Roxm ba E Rº where n is the input dimension, o is output dimension, and m is the number of neurons in the hidden layer. We will use the term "key vector" to refer to rows of W. and "value vector" refer to the rows of Wa that correspond to each neuron. Our loss function uses the mean squared error between the original image and reconstruction across our full dataset, X.
x x+e,, eN N(0,
Wa ReLUUW.x b.)) + ba
loss Σ ()- xeX
We use Kaiming randomly initialized weights (He et al.) 2015) and train until the fraction of active neurons converges. We primarily use the CIFARIO dataset of 50,000 images of 32x32x3 dimen- sions, training either the raw pixels (flattening them into 3072 dimensional vector) or latent embeddings of 256 dimensions, produced by a ConvMixer pretrained on ImageNet (Trockman & Russakovsky et al. 2015).
Kolter 2022
3

---

Under review as a conference paper at ICLR 2023
To ensure results robust, vary number of settings, finding that in every case the rela- tionship between noise and network sparsity holds. All code and training parameters can be found at: pettps://githen
cwm aaaam/anon83711 A
Symmetrie Mean Zero Noise Induces Top-K Networks Figure 2 shows that a single hidden layer ReLU network with 10,000 neurons learns to become Top-K network with smaller & values as the amount noise increases. This network was trained on the CIFARIO embeddings to mimick training within deeper network layer and where the manifold smoother than when training on raw image pixels.
While Fig. 2 only shows the mean number of neurons that are active across all 50,000 CIFARIO inputs per epoch, Figure 3 shows the fraction of active neurons for every input. This reveals that as noise increases, the variance for how many neurons are active shrinks, so that the mean is an accurate summary statistic. Note that there are no dead neurons. Fig. 3] also shows the bias terms for each neuron and the L2 norm of their key vectors. It is clear that these values all synchronize to value decided on by the network and are responsible for the sparse k neurons remaining active after the ReLU nonlinearity is applied.
As noise increases, the bias terms become more negative to raise the activation threshold and in- crease sparsity. The Ly norms of the key vectors also become smaller. We hypothesize this is a counter-response the L2 norms of the data increasing with noise and explains why in high noise settings (a 2 0.8) neuron activations linger around ~50% sparsity before rapidly becoming very sparse (see Fig. 2. Before the key vector L2 norms sufficiently small, noisy data with large L2 norms produce very large neuron activations (both positive and negative) and the bias terms initial- randomly
active. Because the key vectors initialized from a symmetric distribution around 0, in expectation half of the neurons are active after the ReLU nonlinearity is applied. This means the network must wait to first shrink its key vector L2 norms before the bias terms can affect activation sparsity.
ized close to 0 have real effect on which neurons
Fraction of Active Neurons function of Gaussian Noise during training
18
1.5
0.8
0.3
=00.0.1
0.05
0.6
Fraction 0.4 Active
Increase
Noise
200
400
600
Epochs
There is positive relationship between activation sparsity and noise. The average fraction of neurons active during each CIFAR1O latent input over the course of training for 800 epochs. Each line corresponds to training randomly initialized networks with different noise levels denoted by different colors. We show the average of three different random seeds and their standard error of the mean (not visible as the variance is low). Interestingly, the higher noise levels take longer to sparsify but upon convergence become the sparsest.
Figure 2:
To gain more intuition for the amounts noise applied and reconstruction quality, we train networks directly CIFAR10 pixels and demonstrate their results in Fig. 4 Unsurprisingly, the more noise that is applied, the worse reconstruction performance is. This is especially for noise levels large enough to make competing image closer to the noisy input than the noise-free target image.
4

---

Under review as a conference paper at ICLR 2023
Active Neurons
Neuron Blas Terms
Key Vector L2 Norms
0.0
03
306
20A
204
15K4
26
306
Noise Amounts
0.3
03
03
4.5
54
306
15A
5A
308 358
26
04
4.1
04
02
Figure 3: Noise induces the formation of Top-K networks. Shown column-wise are the fraction of neurons that are active for each training input (Left), the values of each of the 10,000 neurons bias terms (Middle) and the L2 norms of their key vectors (Right), as a function of number of training epochs. Each is shown for three different representative noise levels (rows). These plots show each input the network and use the density of blue to represent how many times a particular value occurs.
Original Image
0.1
0.3
0.8
1.5
3.0
Recon.
Recon.
Recon.
Noise
Recon.
Recon.
Figure 4: Example CIFAR10 reconstructions. Shown example reconstructions obtained at different noise levels for five randomly selected images from the test data. The network at a 2 0.8 qualitatively transitions from fuzzy reconstructions to general image details.
Shared Bias Terms and Inhibitory Interneurons To test the observed transition into Top-K network, we re-train our networks, replacing each neuron's 's bias term with global shared bias term. This results in identical model performance and similar & values. Using this shared bias term is interesting because, in conjunction with the synchronization the L2 weight norms, it analogous to dynamically learning k value for the network.

---

Under review as a conference paper at ICLR 2023
We compared this network with explicitly creating sparse Top-K networks. These Top-K networks sort the neuron activations and keep only the most active. They were either given fixed values could learn the k values using either continuous relaxation that is differentiable the REIN- FORCE algorithm (see Appendix [ for details) (Williams) 2004). These approaches all performed worse than the shared bias (and default ReLUU) networks, with the best networks getting similar validation accuracies as the much smaller 100 neuron network and only when they were not sparse, using 3, 000 neurons. Even the approaches learning & were insufficiently dynamic and de- pended too much upon the k initialization value.
Biological Receptive Fields Looking more closely the sparse solutions found by our ReLU networks, we discovered that for 0.3, neurons begin to form receptive fields reminiscent of VI Gabor filters and on/off center surround retinal ganglion cells (Sterling & Laughlin 2015 Olshausen shoows the receptive fields of the 425 neurons most activated (sorted from
& Field 1997). left-to-right and top-to-bottone by a noisy image of car when a 0.8. This similar result to that of sparse coding (Olshausen & Field 1997). however, here we noise with the ReLU activation function instead sparsity penalty and train on full images instead 8x8 patches. See Appendix B for more receptive fields from models trained at every noise level.
The longer training progressed, the more similar the receptive fields appeared to those found in biology. These receptive fields also unique to the noisy, reconstruction task. When tried training the network without any noise but instead with an I, activation penalty motivated by the original sparse coding setup (Olshausen & Field 1997), biological receptive fields were observed. In addition, this approach resulted in unstable training, more dead neurons, less sparsity, and no synchronization of key vector norms bias terms]
Figure 5: Biological receptive fields form with noise. Networks trained on CIFAR10 pixels with
0.8 Gaussian noise. The active 125 neuron receptive fields are reshaped into 32x32x3 and re-scaled so their values span the range of pixel values [0, 255]. Neurons are sorted by their activity levels (top left is the most active, going across rows then down columns) for random noisy input of car. Starting at a 2 0.1 and particularly for a = 0.8 (shown here) we observe the formation of many Gabor filters and on/off center surround receptive fields that are unique to noisy training.
Explaining Away and Facilitation Motivated by the similarity of our results with the sparse coding solution, we adopt more ideas from sparse coding, specifically the ability to "explain away" and "facilitate" (Olshausen & Field 1997). Explaining away reduces the activity of neurons that are redundant with others that are already active, while facilitation encourages co-activation of neurons encoding a common structure such as a contour. We present our implementation and evaluation of these models in the Appendix
With these models we find that our networks become even more sparse and do so more quickly, as would expect due to the explaining away of redundant neurons (see Appendix C). The network also converges faster and overfits to the training data. Using optimal stopping, we get the same performance with explaining away with the best ReLU model. This suggests that the default ReLU network already surprisingly effective at de-noising.
Noisy Pre- Fre-Training Retains Model Sparsity without Performance Costs Investigating our noise trained sparse networks, we find that even after the noise is removed, they remain sparse. We train our models in the same way as in Fig. 2 but at epoch 800 linearly anneal each noise level down to 0.0 over the next 800 epochs. Fig. shows that the sparsity levels remain largely unperturbed.
We did not implement the full sparse coding framework, only the T- penalty, however, this shows that it is insufficient implement sparse coding while noisy training is.

---

Under review as a conference paper at ICLR 2023
For classification, it is clear that while the correlation between noise and sparsity holds, it is not quite as strong. We believe this is due to the fact that classification only needs to cluster the data into the ten CIFARIO labels, creating less interference between neural representations.
There is also no reduction in reconstruction or classification performance. In fact, for classification {0.1. 0.3) both get 94.3% validation accuracy instead of the 93.4% that the baseline gets, showing slightly better generalization while also being more sparse (66% for baseline versus 55% for a - 0.1 and 45% for a 0.3). Note that for the networks with a 2 0.8 for reconstruction and a 2 3.0 for classification, annealing the noise results in dead neurons, presumably because the data manifold no longer spans as large a space. However, these dead neurons do not harm task performance or entirely explain the sparsity levels. See Appendix D for data on the fractions of dead neurons.
Reconstruction Task
Classification Task
Start of noise anneal
All mocels
Start of nolse arneal
All models
0=0.0
a=0.0
0.0
0.6
Mean
Fraction
0.4
0.4
Active
Neurons
0.2
0.2
1.5k
Epochs
1.5k
Epochs
Figure 6: Noisy pretraining retains network sparsity. The vertical purple lines denote the start of noise annealing where each noise level is linearly decayed down to a O. For both reconstruction (left) and classification (right) tasks the networks remain highly sparse even after all noise is re- moved. Moreover, there is no noticeable performance difference between any of the networks while some of those trained with moderate noise perform slightly better than the noise free baseline. We truncate the x-axis to start at epoch 600 for the sake of clarity. The classification networks continue to sparsify even when noise annealing has started because unlike in the reconstruction task, the spar- sity level did not converge within the first 800 epochs.
Model Ablations Figure 7 shows the mean neuron sparsity network convergence across a num- ber of different experimental conditions: (i) three datasets: CIFARIO pixels, CIFARIO latent em- beddings, MNIST; (ii) two noise distributions: Gaussian and Laplace; (iii) four numbers of neurons: 100, 1,000, 10,000 and 100,000; (iv) three training tasks: reconstruction, classification, and Trans- former next token prediction.
We ran hyperparameter sweeps of different learning rates for Adam, RMSProp, SGD, and SGD with Momentum (SGDM). We found that every optimizer with high enough learning rate converged to a sparse Top-K solution without creating dead neurons 2 activation functions. While neither of these functions is truly sparse in setting activations to 0, we counted neurons with absolute activation values < 0.0001 as "off". We found for the latent CIFARIO dataset, both sigmoid and GELU (Hendrycks & Gimpel 2016) had no sparsity but while sigmoid performs worse, GELU does just as well as ReLU. If raised our arbitrary activation threshold to < 0.01 we found that GELU does become sparse with respect to noise but still not as sparse as ReLU while sigmoid remains fully dense (see Appendix F Interestingly, for the raw CIFARIO 10 dataset, both sigmoid and GELU using the lower < 0.0001 threshold did become sparse and perform as well as ReLU. While still neither activation function becomes as sparse as ReLU, the GELU networks implement the same Top-K convergence as ReLU shown in Fig. 3 and form biologically plausible receptive fields like those of Fig. 5
We also tested the sigmoid and GELU
2 'Interestingly for SGD, with low learning rate and enough epochs the model could learn non-sparse solutions with 500 neuron activity and no updating of hias terms that performed almost as well as their sparse counterparts.
7

---

Under review as a conference paper at ICLR 2023
As part of ablating tasks, we test Transformer next token prediction. We train small GPT2 models (Radford et al.] 2019) with three blocks of interleaved Attention and Feedforward MLP layers. In front of each MLPs of each network, we inject one of the following noise levels E {00.0, 0.05, 0.1, 0.8, 1.5, 3.0, 8.0) and use the WikiText- 103 dataset. At global training step 200k, remove all noise to also investigate the longer term pretraining effects.
The correlation between noise and activation sparsity remains and is again retained even after noise removed. Furthermore, aside from the highest noise level of 8.0, pre-training has no effect on model performance. Figure d)) shows the final sparsity amounts of the first model layer with each of the other layers giving similar results with ~3x to ~-xx more sparsity when o 3.0, compared to a = 0.0.
(a) Varying the number of neurons
(b) Varying the dataset
1.0
C40330
0.8
22,000 No.rood
228 000 Nou onu
mrddoo of
0.8
0.2
044
0.2
0.3%
Noos
(c) Varying the noise distribution
(d) Varying the task
Cauusinn
Recansam ctice
Laglace
8.6
8.5
83
0.4
01
0.0
Figure 7: Training ablations. The positive correlation between noise and sparsity is robust to (a) neuron counts, (b) datasets, (c) noise distributions, and (d) training tasks.
4 DISCUSSION
Why does noisy training induce our ReLU network to become a biologically plausible Top-K net- work? Fundamentally, the network should want to use many neurons it can to give the best reconstruction of the input data as possible. However, it must trade-off the increase in accuracy gained by pooling more neurons with increased interference. This interference is caused by both noise and orhoooooogyyalityy of the learned weight vectors (Elhage et al. 2022).
An increase in noise will create more erroneous low-level activations in neurons that must be ig- nored to maximize signal to noise ratio. Therefore, during noisy training, the model must learn weight vectors to encode information about the data distribution that can be separated from the noise using only ReLU functions. In natural images, for example, certain projections (e.g., with Gabor filters) have high kurtosis which allows them be easily distinguished from noise via thresholding or "coring" (Simoncelli & Adelson 1996). Similarly, we hypothesize that the encoding layer in our network is learning such projections, so that thresholding performs noise rejection while also pro- ducing sparse activations. In theory, incorporating the "explaining away" of sparse coding networks should additionally remove interference due to the non-orthogonality of weight vectors. Our initial attempt doing did not produce performance gains, though it did increase sparsity. However further investigation is warranted to understand its potential benefit.
The positive correlation between noise and sparsity is opposite to Transformer Attention and sociative memory models such as SDM and Hopfield Networks (Bricken & Pehlevan| (00T)). For Transformer Attention, the difference is that Attention has direct access to every input within its
8

---

Under review as a conference paper at ICLR 2023
receptive field but no further. This provides for high accuracy when dealing with these inputs (keys) but an inability to store long term memories. As a result, in the low noise regime, it is optimal for Attention implement form of nearest neighbour lookup, activating few keys. When there is more noise, especially noise that moves the query so far from its target key that it is no longer the nearest, it is optimal activate more keys and average over them, losing accuracy but giving a solution in the correct neighbourhood.
By contrast, SDM can store long term memories. However, has the same negative correlation between noise and sparsity as Attention because it attempts to store full patterns with maximum fidelity within each neuron instead of distributing the features of each pattern across neurons (Kan- erva 1988). This results in SDM attempting to also perform a nearest neighbour lookup, activating only as many neurons around the noisy query as necessary to activate those storing the target pattern.
We have shown the extent to which feedforward ReLU MLP is capable of converging with a biological inhibitory interneuron circuit. Using deep learning to give insights into neuroscience, this result suggests that the presence of noise, the biological sparse coding solution is optimal. Moreover, the fact that noise alone results in sparse coding rather than having a sparsity penalty in the loss function suggests that the primary driver behind sparse coding may be the handling of noise with metabolic efficiency as added benefit, rather than the other way around. In addition, the fact that the reconstruction loss converges to this solution hints that the brain may be trying to maximize mutual information of its sensory stimuli (Sterling & Laughlin 2015).
Our work is largely empirical. An important goal of future work will be to develop rigorous mathematical theory for modelling how the amount of noise affects sparsity for given dataset, rather than having to test a number of noise levels to find one that works (analogous to tuning hyper-parameteryy
Limitations
In addition, while we show that our de-noising autoencoder converges with the biological sparse coding network, conventional hardware accelerators are unable to benefit from the increased levels of unstructured sparsity. Mcanwhile, the advantages of noisy training for generalization, continual learning, and adversarial robustness have already been documented. However, sparse networks have recently been shown to aid mechanistic interpretability by removing polysemantism and encourag- ing specialization (Nelson et al. 2022) and additional future applications may arrive.
Conclusion We have shown that the realistic assumption of noisy training data is sufficient for forwaro layers to organically become highly sparse and retain their sparsity, even after noise is removed. This result is particularly striking because of the way the networks converge with sparse coding, producing biological receptive fields and approximating an inhibitory interneuron that implements Top-K operation. Combining noisy training with other approaches to induce sparsity, including taking additional ideas from sparse coding, may result in even higher degrees of sparsity. More broadly, this work builds new bridge between artificial and biological neural networks by showing how they can be more similar than they may otherwise appear.
REFERENCES
Ahmad and Luiz Scheinkman. How can we be so dense? the benefits of using highly sparse representations. ArXiv, abs/1903. 2019.
Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data-generating distribution. J. Mach. Learn. Res., 15:3563-3593, 2014.
David Attwell and Simon B. Laughlin. An energy budget for signaling in the grey matter of the brain. Journal of Cerebral Blood Flow & Metabolism, 21:1133 1145, 2001.
Demba E. Ba. Deeply-sparse signal representations (ds2p). IEEE Transactions on Signal Process- ing, 68:4727-4742. 2020.
Charles M. Bishop. Training with noise is equivalent tikhonov regularization. Neural Computa- tion, 7:108-116, 1995.
Trenton Bricken and Cengiz Pehlevan, Attention approximates sparse distributed memory. NeurIPS, 2021.

---

Under review as a conference paper at ICLR 2023
Mike Davies, Narayan Srinivasa, Tsung- -aang-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning, leee Micro, 38(1):82-99)) 99, 2018.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of super- position. Transformer Circuits Thread, 2022. URL
pub/2022/ 2ow2202222yyaaeee tttp:/tttmmmodd/l/// mode_ potrdel/inndee htt///idee.htte html
Elizabeth Fleming, Michael R. Tadross, and Court Hull. Local synaptic inhibition mediates cerebel- lar pattern separation necessary for learned sensorimotor associations. bioRxiv, 2022.
Trevor Gale, Matei A. Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. In SC, 2020.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In AISTATS, 2011.
lan Goodfellow, Yoshua Bengio, and Aaron C. Courville. Deep learning. Nature, 521:436-444 2015.
Olivia Gozel and Wulfram Gerstner. A functional model of adult dentate gyrus neurogenesis. eLife, 10, 2021.
Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In ICML, 2010.
Bilal Haider, Matthew R Krause, Alvaro Duque, Yuguo Yu, Jonathan Touryan, James A Mazer, and David A McCormick. Synaptic and network mechanisms of sparse and reliable visual cortical activity during nonclassical receptive field stimulation. Neuron, 65(1):107-121, 2010.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. 2015 IEEE International Conference on Computer Vision (ICCV) pp. 1026-1034, 2015.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv: Learning, 2016.
Abhiram Iyer, Karan Grewal, Akash Velu, Lucas O. Souza, Jeremy Forest, and Subutai Ahmad. Avoiding catastrophe: Active dendrites enable multi-task learning dynamic environments. In Frontiers in Neurorobotics, 2022.
Pentti Kanerva. Sparse Distributed Memory, MIT Pr., 1988.
James D. Keeler. Comparison between kanerva's sdm and hopfield-type neural networks. Cognitive Science, 12(3):299 329, 1988. ISSN 0364-0213. doi: hups://doi.oo org//
10.1016603544-01134488990222 URL
rect. con/cc.osccc ence/
article//11/03340213388
Dmitry Krotov and John Hopfield. Dense associative memory for pattern recognition. Advances in neural information processing systems, 29, 2016.
Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William M. Leiserson, Bill Nell, Nir Shavit, and Dan Alistarh. Inducing and exploiting activation sparsity for fast neural network inference. 2020.
Andrew C. Lin, Alexei M. Bygrave, Alix de Calignon, Tzumin Lee, and Gero Miesenbock. Sparse, decorrelated odor coding the mushroom body enhances learned odor discrimination. Nature neuroscience, 17:559 568, 2014.
Alireza Makhzani and Brendan Frey. k-sparse autoencoders. CoRR, abs/1312.5663, 2014.
Kiyotoshi Matsuoka. Noise injection into inputs in pack-propagation learning. IEEE Trans. Syst. Man Cybern., 1992.
10

---

Under review as a conference paper at ICLR 2023
Dmitry Molchanov, Arsenii Ashukha, and Dmitry P. Vetrov. Variational dropout sparsifies deep neural networks. In ICML, 2017.
Elhage Nelson, Hume Tristan, Olsson Catherine, Nanda Neel, Henighan Tom, Johnston Scott, ElShowk Sheer, Joseph Nicholas, DasSarma Nova, Mann Ben, Hernandez Danny, Askell Amanda, Ndousse Kamal, Jones, Drain Dawn, Chen Anna, Bai Yuntao, Ganguli Deep. Lovitt Liane, Hatfield-Dodds Zac, Kernion Jackson, Conerly Tom, Kravec Shauna, Fort Stanislav, Kada- vath Saurav, Jacobson Josh, Tran-Johnson Eli, Kaplan Jared, Clark Jack, Brown Tom, McCandlish Sam, Amodei Dario, and Olah Christopher. Softmax linear units. Transformer Circuits Thread, 2022. URL https:
pub/ 2022 swww/indes.... html
Bruno A. Olshausen and David J. Field. Sparse coding with an overcomplete basis set: A strategy employed by vI? Vision Research, 37:3311-3322, 1997.
Bruno Olshausen and David Field. Sparse coding of sensory inputs. Current opinion in neuro- biology, 14(4):411-487, 2004.
Alec Radford, Jeffrey Wu, Dario Amodei, Daniela Amodei, Jack Clark, Miles Brundage, and Ilya Sutskever. Better language models and their implications. OpenAl Blog https://openai. combblog/betee-laggggggmddddddd 2019.
Marc' Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun. A unified energy-based framework for unsupervised learning. In AISTATS, 2007.
Salah Rifai, Xavier Glorot, Yoshua Bengio, and Pascal Vincent. Adding noise to the input of model trained with regularized objective. ArXiv, abs/1 104.3250, 2011.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211-252, 2015.
Jonathan Schwarz, Siddhant M. Jayakumar, Razvan Pascanu, Peter E. Latham, and Yee Whye Teh. Powerpropagation: A sparsity inducing weight reparameterisation. In NeurIPS, 2021.
Biswa Sengupta, Martin B Stemmler, Simon B. Laughlin, and Jeremy E. Niven. Action potential en- ergy efficiency varies among neuron types in vertebrates and invertebrates. PLoS Computational Biology, 6, 2010.
Yang Shen, Sanjoy Dasgupta, and Saket Navlakha. Algorithmic insights on continual learning from fruit flies. ArXiv, abs/2107.07617, 2021.
Jocelyn Sietsma and Robert F. Dow. Creating artificial neural networks that generalize. Neural Networks, 4:67-79, 1991.
Eero P Simoncelli and Edward H Adelson. Noise removal via bayesian wavelet coring. In Proceed- ings of 3rd IEEE International Conference on Image Processing, volume pp. 379-382. IEEE, 1996.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. ArXiv, abs/1997.00600, 2019.
Peter Sterling and Simon Laughlin. Principles of neural design. 2015.
Florian Tramèr, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. ArXiv, 0tss/2002.08447, 2020.
Asher Trockman and J. Zico Kolter. Patches are all you need? ArXiv, abs/2201.09799. 2022.
Ziheng Wang. Sparsert: Accelerating unstructured sparsity on gpus for deep learning inference. Proceedings of the ACM International Conference Parallel Architectures and Compilation Techniques, 2020.
Ronald Williams. Simple statistical foollwent--flowiigggg algorithms for connectionist reinforcement learning. Machine Learning, 8:229-256, 2004.
11

---

Under review as a conference paper at ICLR 2023
Chang Xiao, Peilin Zhong, and Changxi Zheng. Enhancing adversarial defense by k-winners-take- all. arXiv: Learning, 2020.
Marjorie Xie, Samuel P. Muscinelli, Kameron Decker Harris, and Ashok Litwin-Kumar. Task-
dependent optimal representations for cerebellar learning. bioRxiv, 2022.
Huanrui Yang. Wei Wen, and Hai Helen Li. Deephoyer: Learning sparser neural network with differentiable scale-invariant sparsity measures. ArXiv, abs/ abs//908..em 1ess/998..9999, 2020.
Richard M. Zur, Yulei Jiang, Lorenzo L. Pesce, and Karen Drukker. Noise injection for training artificial neural networks: a comparison with weight decay and early stopping. Medical physics, 36 10:4810-5 2009.
Appendix
Table of Contents
A Learning Top-K Values
12
Receptive Fields
12
Explain Away
D Dead Neurons
17
E Activation Function Ablations
18
A LEARNING TOP-K VALUES
There two different approaches taken to learn the k values in the explicit Top-K networks. The first to use a continuous relaxation the & value that otherwise a non-differentiable integer. We create window of size w and apply sigmoid weighting to all values between k w. We can then let l be a continuous value that is responsible for shifting the center point of this sigmoidal activation window.
Another solution to create probability distribution over the possible values, sample discrete l integers from it, and the REINFORCE algorithm (Williams) 2004) update the sufficient statistics of our distribution. Intuitively, we can think of this approach as trying a bunch of different l values and updating to see more of those that lead lower loss.
Both of these implementations can be found in our github repo at the following URL: https:
cwm/ http://....oooooon//////nonggge
B RECEPTIVE FIELDS
Here show additional images of the receptive fields for networks trained with different amounts of noise (Fig. the queries used to activate these neurons (Fig. 9 and how the receptive fields look more biological as the network trained for longer (Fig. I0).
12

---

Under review as a conference paper at ICLR 2023
Noise 0.05
Noise 0.1
Noise
Noise 0.8
Noise 1.5
Noise
Noise 10.0
Figure 8: Receptive fields across noise levels. We show for all noise levels the 625 most active neurons for the truck query (middle of Fig. D). For 0.8 > a which activate fewer neurons, we also show the cat query below it. Biological like receptive fields start to appear for 0.1 and can be seen most clearly for a E {0.3, 1.5)
13

---

Under review as a conference paper at ICLR 2023
Figure 9: Queries used activate the receptive fields. These queries randomly selected.
We use the cat (left) and truck images (middle) with the relevant noise levels to activate the neurons shown in Fig. We the car image (right) for the main text Fig. and receptive field changes across epochs in Fig. 10
3000 Epochs
9000 Epochs
19000 Epochs
Figure 10: Receptive fields become more biological across epochs. We show how the most active receptive fields for the a 0.3 network evolve over the course of training to become more biolog- ically similar. The specific number of epochs used (3,000, 9,000 and 19,000) were chosen simply because of when the models were checkpointed and ro-loaded for additional training. We use the car image as our noisy query here (rightmost in Fig. 9).
14

---

Under review as a conference paper at ICLR 2023
C
EXPLAIN AWAY
We can model explaining away and facilitation with an additional weight matrix G E Rm (recall m is the number of neurons) that we run in a recurrent inner loop. Our one hidden layer network becomes:
fg = W + be
ReLU((4)
AU (ND GZt) Fr.
Me+1 t4: + 7
X = Wa ReLU((;)) + ba.
(2)
where t increments from I as the number of iterations we want the inner loop optimize neural activity and 7 is the size of the update step. In the simplest case where 1* 1 and ? 1, we do a single forward pass through this circuit making analogous to residual connection that comes before the activation function. More generally, the value of t" can be thought of as the number of recurrent connections that all originate from ff and end after every update to z. The original Eq. is special case where t* O. This use of G' learnt with backpropagation was first explored in Gregor LeCun (0100).
Training the same networks before but using Eq. 2 with t" 1 and we present results in Table [
We show the sparsity amounts of the networks in Fig.
We can also replace Gz, with GaG.z, where Ge Rixm Ga E Rm implement single in- hibitory interneuron that uses 2m rather than m² parameters and must learn each of its afferent and efferent connections. When the neurons still have their bias terms this solution has effect. Re- moving the bias terms and relying just on the inhibitory interneuron, we found that it performs well (Table I and becomes sparse as function of noise but results in number of dead neurons and is not as sparse as the baseline ReLU or Explaining Away models. However, this shows that the full inhibitory interneuron solution is at least plausible.
Method
Val. Ace.
a 0.05
a 0.1
0.3
0.8
ReLU m =100
0.0047
0.0067
0.0130
0.0209
ReLU m =1,000
0.0019
0.0042
0.0112
0.0203
ReLU m = =10,000
0,0018
0.0041
0,0109
0,0203
ReLU m =100,000
0.0032
0.0053
0.0111
0.0203
Shared Bias
0,0017
0.0040
0,0110
0.0204
Explain Away
0.0018
0.0041
0.0110
0.0206
Inhib. Interneuron
0.0017
0.0041
0.0109
0.0204
Table 1: Reconstruction Validation Accuracy The baseline ReLU model varying the number of neurons, compared to the shared bias, explaining away, and inhibitory interneuron networks. We only show noise up to a = 0.8 because the higher noise levels present trivial solutions that all networks perform equally well on. Due to very high convergence between random seeds, we just present a single run for each model.
15

---

Under review as a conference paper at ICLR 2023
Fraction of Active Neurons with Explaining Away Network
0.4
Fraction of Active Neurons
Increase Noise
0.2
200
400
600
Epochs
Figure 11: Explaining Away network is sparser and sooner. For every level of noise, the explain- ing away network removes neurons redundantly active and makes them much more sparse.
16

---

Under review as a conference paper at ICLR 2023
D DEAD NEURONS
Here we show the number of dead neurons that are present when we anneal our noise value to 0 for reconstruction (Fig. 12 and classification (Fig. 13p.
Reconstruction
Noise Anneal Fraction of Dead Neurons
Start of noise anneal
All models g=0.0
Fraction
Dead Neurons
500
1k
1.5k
Epochs
Figure 12: Dead Neurons during the reconstruction task. During noise annealing, the higher noise levels a 2 0.8 see number of dead neurons. However. this does not full explain their sparsity levels and this trivial task it does not hurt their reconstruction performance.
Classification Noise Anneal Fraction of Dead Neurons
10
D0=3 =
0.8
0.3
0.1
Oa 0.05
Start of
All models 0-0.0
0.4
Fraction of Dead Neurons
Epochs
Figure 13: Dead Neurons during the classification Task. Here there fewer dead neurons, they only appear for a 2 3.0 and they only appear close to when the noise is entirely turned off. Again these results do not fully explain the sparsity amounts found.
17

---

Under review as a conference paper at ICLR 2023
E ACTIVATION FUNCTION ABLATIONS
For CIFARIO, we train each model for 3,000 epochs and present the mean number of active neurons for GELU (Fig. 14) and sigmoid (Fig. [5]. We don't show the latent CIFARIO results as sigmoid and GELU are fully dense with no sparsity when the 0.0001 activation threshold is used. Neither of these networks are as sparse as the ReLU network. Also neither network creates sparsity through dead neurons, but given that all neurons have non-zero absolute activity, having dead neurons is impossible.
While we just show the mean active neurons here, both solutions look like those of Fig. 3 with the neurons for all inputs converging Top-K solution where the bias terms become negative and synchronize along with the key vector L2 norms.
Fraction of Active Neurons with GELU Network
10
-0=0.1
8.05
Active
Neurons 0.4
1000
2000
3000
Epochs
Figure 14: The GELU network sparsifies with noise when trained on CIFAR10. The GELU network closely resembles the ReLU network by sparsifying (but not to the same degree). also forms biological receptive fields. We use the arbitrary 0.0001 absolute activation threshold to label a neuron is on off.
18

---

Under review as a conference paper at ICLR 2023
Fraction of Active Neurons with Sigmoid Network
18
1.5
0.00
0.8
Fraction Active Neurons
1000
2000
3000
Epochs
Figure 15: The sigmoid network also sparsifies with noise for CIFAR10. While sigmoid sparsifies proportion noise, does not form biological receptive fields and is less sparse than GELU or ReLU.
19


Reviews:
[{"review_id": "yF2Uug_Cjm", "reviewer": "Reviewer_39gB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper shows that when adding noise to the input, sparse and \"biologically plausible\" features emerge in a 1-hidden layer autoencoder. Moreover, using the ReLU-activation function seems to important for this pattern to emerge. ", "summary_of_the_review": "Overall, the paper tackles an important problem but there are some issues regarding novelity, clarity and some errornous claims. Therefore, I cannot recommend acceptance of the paper at this stage. ", "strength_and_weaknesses": "Strengths:\n* The paper tackles the important problem of bridging the gap between biological plausible learning and current artificial neural networks.\n\nWeaknesses:\n* The paper mostly focuses on 1-hidden layer networks. It is not too clear why and how these results are relevant for practice, where typically much deeper networks are considered. For classification tasks, the observed sparsity seems to be much less pronounced. \n* Unlike claimed in the paper, using the L1-norm can lead to biologically plausible and sparse features even for classification tasks, see e.g. Figure 4 in \"Towards Learning Convolutions from Scratch\" (https://papers.nips.cc/paper/2020/file/5c528e25e1fdeaf9d8160dc24dbf4d60-Paper.pdf)\n* There are some issues regarding novelty/clarity (see next section)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ws7dKBoip5h", "reviewer": "Reviewer_AnUn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper demonstrates that adding noise to a single network layer causes the activations in that layer to become sparse when the output is given by a ReLU nonlinearity. They first apply this idea to a one-layer autoencoder and show that the degree of sparsity increases as the magnitude of the noise increases (Figure 2). This sparsity is created by a negative bias that is shared across all units coupled with a constrained L2 norm on the encoding weights. The result is a “Top-K” activation profile in which for a given stimulus only the top K units have non-zero values. The authors report that the presence of noise produces center-surround and oriented receptive fields partially reminiscent of the classic paper from Olshausen & Field (1997) (which they report not being able to replicate). They report that this sparsity is maintained when the noise is gradually annealed to zero and is present to a lesser degree for a classification task on the CIFAR dataset. ", "summary_of_the_review": "The paper reports a phenomenon that is novel to knowledge. The relevance to multilayer networks is unclear which limits the impact of the study to the ML community. The relevance to biology is also unclear for the reasons noted above. As a consequence, I am doubtful that the study will have a substantial impact on the ML or neuroscience community. \n\nResponse to rebuttal\n\nI apologize for my slow response and thank the authors for responding to each of my points. I still find the paper underwhelming, because of the limitation to shallow models (though I appreciate the authors are working on extending to deeper models) and the lack of any quantitative model comparisons with biological data (which is of course challenging, but needed if one wishes to make claims about relevance to biology).  ", "strength_and_weaknesses": "Strengths\n\nThe paper is simple and easy to follow. \n\nThe sparsity effects shown appear relatively large and robust. \n\nAdding noise is a simple manipulation that can have benefits beyond sparsity, and so there is some appeal to using this manipulation as a way to generate sparsity. \n\nWeaknesses\n\nIt is odd that the authors focus so much on a one-layer model. For this finding to be broadly relevant, it is important to test whether this effect occurs in large-scale DNNs and whether networks can generate sparsity while maintaining good performance on real-world tasks. The only thing that is done along these lines is with a single, small GPT2 model. \n\nIt appears you need fairly high levels of noise to generate substantial sparsity (e.g., sigma=0.3 to get below 20% activation in Figure 2, which based on Figure 4 is quite a lot of noise). It seems this might limit the practical utility of the method, though it is hard to say in part because experiments are mostly limited to the single-layer training regime.\n\nThe link with biological receptive fields is tenuous. There are multiple ways of generating center-surround and oriented/bandpass receptive fields (e.g., whitening for center-surround RFs, sparsity for oriented RFs, CNNs often exhibit similar structure at the first layer). It is not clear this approach provides a better fit to biological data than these alternatives and there is no quantification of the match to biology or model comparison. Moreover, there are so many differences between the models trained here and biological networks, that it is hard to know the relevance of this finding, particularly given that the results can vary with the activation function (e.g., the sigmoid activation function does not generate sparsity) and the optimization algorithm (e.g., SGD with a low learning rate apparently does not generate sparsity). \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aRpsWd6K1g", "reviewer": "Reviewer_6F1u", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this empirical paper, the authors observe the correlation between the sparsity of the learned representations and the level of noise injected into the input of Neural Networks with the ReLU activation function. Experiments have been carried out on CIFAR10 both on pixel data and on a neural representation, MNIST, and on the WikiText-103 dataset. They report a positive correlation between the magnitude of the injected noise and the sparsity of the representation. They observe Gabor-like filters when reconstructing with noise on pixel data. For the classification task, the performance of the model did not decrease with sparsity. Also, compared with using only the top-K activations of a model learned without noise, the models learned with noise gave better validation accuracies.", "summary_of_the_review": "While I think that the experiments presented are interesting, I think that adding baselines to control for the quality of the sparse representation obtained is necessary to allow for acceptance of the work.", "strength_and_weaknesses": "The principal strength of this paper is that the set of experiments presented in this paper is interesting and is worth sharing with the community.\n\nThe main weakness of the paper is the fact that the focus is on observing the obtained sparsity. Reconstruction accuracies are only compared using a top-K network. I think that comparing with an actual sparse coding model is important to control for the quality of the solution learned.\n\nSimilarly, about the classification accuracy comparison, it would be important to report the accuracies for all noise levels as well as all number of hidden units and compare with an equivalent sparse coding representation.\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VDATW4uIbfS", "reviewer": "Reviewer_wvBd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The present work explores the relationship between input noise and network sparseness.  Such sparseness is known to exist in biological networks (brains), and has been hypothesized to arise as a solution to increase the signal-to-noise ratio of the network and to keep energy consumption low. Here, the authors find that adding noise to ANNs effectively decreases the biases of the constituent neurons, resulting in only the top k neurons being active. The authors further find that such noise results in receptive fields that are similar to those found in V1.  The results suggest that sparse neuronal activity in brains may arise to better deal with noisy inputs rather than (or in addition to) minimizing energy consumption.  Such sparse, top-k networks may also be useful in future, low-energy ANN applications. ", "summary_of_the_review": "While the authors demonstrate that injecting noise into feedforward networks sparsifies their responses, such results can easily be explained and recapitulated--and as noted by the authors--by simply globally reducing the neurons' biases.  Thus the presented results comprise a minor advance in the area", "strength_and_weaknesses": "Strengths\n\n1. The authors find a simple, implicit manner to create top-k networks that compares favorably to explicit methods\n\n2. The method is tested in several different circumstances, with different tasks, numbers of neurons, and noise levels. Simple characteristics of the network, such as the fraction of active neurons and their biases, are well described.\n\nWeaknesses\n\nMajor\n\n1. As the authors observe, the results are recapitulated using a global negative bias term, acting as common inhibition that requires greater input values to activate neurons and resulting in top-k networks.  Thus, while such an observation may indeed be useful for those interested in such networks, the advance it comprises is a minor one.\n\nMinor\n\n1. While the presented results may bear on biological considerations, as suggested therein, caution is advised, as noise in biological networks is dynamic and thus not perfectly analogous to simply adding noise to inputs of a feedforward network.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject