Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this empirical paper, the authors observe the correlation between the sparsity of the learned representations and the level of noise injected into the input of Neural Networks with the ReLU activation function. Experiments have been carried out on CIFAR10 both on pixel data and on a neural representation, MNIST, and on the WikiText-103 dataset. They report a positive correlation between the magnitude of the injected noise and the sparsity of the representation. They observe Gabor-like filters when reconstructing with noise on pixel data. For the classification task, the performance of the model did not decrease with sparsity. Also, compared with using only the top-K activations of a model learned without noise, the models learned with noise gave better validation accuracies.
strength_and_weaknesses: The principal strength of this paper is that the set of experiments presented in this paper is interesting and is worth sharing with the community.

The main weakness of the paper is the fact that the focus is on observing the obtained sparsity. Reconstruction accuracies are only compared using a top-K network. I think that comparing with an actual sparse coding model is important to control for the quality of the solution learned.

Similarly, about the classification accuracy comparison, it would be important to report the accuracies for all noise levels as well as all number of hidden units and compare with an equivalent sparse coding representation.




clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. 
About novelty, the fact that Denoising Auto-Encoders produces Gabor-like filter has already been observed in previous work (e.g. in [1])
Reproducibility is allowed thanks to a github repository. 

[1] : Extracting and composing robust features with denoising autoencoders, P. Vincent et. al. ICML 2008
summary_of_the_review: While I think that the experiments presented are interesting, I think that adding baselines to control for the quality of the sparse representation obtained is necessary to allow for acceptance of the work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper shows that when adding noise to the input, sparse and "biologically plausible" features emerge in a 1-hidden layer autoencoder. Moreover, using the ReLU-activation function seems to important for this pattern to emerge. 
strength_and_weaknesses: Strengths:
* The paper tackles the important problem of bridging the gap between biological plausible learning and current artificial neural networks.

Weaknesses:
* The paper mostly focuses on 1-hidden layer networks. It is not too clear why and how these results are relevant for practice, where typically much deeper networks are considered. For classification tasks, the observed sparsity seems to be much less pronounced. 
* Unlike claimed in the paper, using the L1-norm can lead to biologically plausible and sparse features even for classification tasks, see e.g. Figure 4 in "Towards Learning Convolutions from Scratch" (https://papers.nips.cc/paper/2020/file/5c528e25e1fdeaf9d8160dc24dbf4d60-Paper.pdf)
* There are some issues regarding novelty/clarity (see next section)

clarity,_quality,_novelty_and_reproducibility: Clarity:
* It was difficult to follow what  precisely is meant by "Top-K" networks. Adding a more rigorous definition to the introduction will benefit the paper. 

Novelty:
* The fact that a 1-hidden layer denoising auto-encoder learns "sparse-coding"-like features is well-known.  See e.g., Figure 2 in the paper "Marginalized Denoising Auto-encoders for Nonlinear Representations" .

Reproducibility:
* The results in the paper seem reproducible thanks to the provided code.
summary_of_the_review: Overall, the paper tackles an important problem but there are some issues regarding novelity, clarity and some errornous claims. Therefore, I cannot recommend acceptance of the paper at this stage.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The present work explores the relationship between input noise and network sparseness.  Such sparseness is known to exist in biological networks (brains), and has been hypothesized to arise as a solution to increase the signal-to-noise ratio of the network and to keep energy consumption low. Here, the authors find that adding noise to ANNs effectively decreases the biases of the constituent neurons, resulting in only the top k neurons being active. The authors further find that such noise results in receptive fields that are similar to those found in V1.  The results suggest that sparse neuronal activity in brains may arise to better deal with noisy inputs rather than (or in addition to) minimizing energy consumption.  Such sparse, top-k networks may also be useful in future, low-energy ANN applications. 
strength_and_weaknesses: Strengths

1. The authors find a simple, implicit manner to create top-k networks that compares favorably to explicit methods

2. The method is tested in several different circumstances, with different tasks, numbers of neurons, and noise levels. Simple characteristics of the network, such as the fraction of active neurons and their biases, are well described.

Weaknesses

Major

1. As the authors observe, the results are recapitulated using a global negative bias term, acting as common inhibition that requires greater input values to activate neurons and resulting in top-k networks.  Thus, while such an observation may indeed be useful for those interested in such networks, the advance it comprises is a minor one.

Minor

1. While the presented results may bear on biological considerations, as suggested therein, caution is advised, as noise in biological networks is dynamic and thus not perfectly analogous to simply adding noise to inputs of a feedforward network.
clarity,_quality,_novelty_and_reproducibility: The present work adds to the literature on the effects of adding noise in neural models. However, the advance is minor, and easily realized, as the authors observe, by imposing a negative bias to the neurons instead.  The communication of the methods and results are clear, and the code to reproduce the results has been shared as well (note: this code should be listed in a separate "Reproducibility" section before the references if possible).
summary_of_the_review: While the authors demonstrate that injecting noise into feedforward networks sparsifies their responses, such results can easily be explained and recapitulated--and as noted by the authors--by simply globally reducing the neurons' biases.  Thus the presented results comprise a minor advance in the area
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper demonstrates that adding noise to a single network layer causes the activations in that layer to become sparse when the output is given by a ReLU nonlinearity. They first apply this idea to a one-layer autoencoder and show that the degree of sparsity increases as the magnitude of the noise increases (Figure 2). This sparsity is created by a negative bias that is shared across all units coupled with a constrained L2 norm on the encoding weights. The result is a “Top-K” activation profile in which for a given stimulus only the top K units have non-zero values. The authors report that the presence of noise produces center-surround and oriented receptive fields partially reminiscent of the classic paper from Olshausen & Field (1997) (which they report not being able to replicate). They report that this sparsity is maintained when the noise is gradually annealed to zero and is present to a lesser degree for a classification task on the CIFAR dataset. 
strength_and_weaknesses: Strengths

The paper is simple and easy to follow. 

The sparsity effects shown appear relatively large and robust. 

Adding noise is a simple manipulation that can have benefits beyond sparsity, and so there is some appeal to using this manipulation as a way to generate sparsity. 

Weaknesses

It is odd that the authors focus so much on a one-layer model. For this finding to be broadly relevant, it is important to test whether this effect occurs in large-scale DNNs and whether networks can generate sparsity while maintaining good performance on real-world tasks. The only thing that is done along these lines is with a single, small GPT2 model. 

It appears you need fairly high levels of noise to generate substantial sparsity (e.g., sigma=0.3 to get below 20% activation in Figure 2, which based on Figure 4 is quite a lot of noise). It seems this might limit the practical utility of the method, though it is hard to say in part because experiments are mostly limited to the single-layer training regime.

The link with biological receptive fields is tenuous. There are multiple ways of generating center-surround and oriented/bandpass receptive fields (e.g., whitening for center-surround RFs, sparsity for oriented RFs, CNNs often exhibit similar structure at the first layer). It is not clear this approach provides a better fit to biological data than these alternatives and there is no quantification of the match to biology or model comparison. Moreover, there are so many differences between the models trained here and biological networks, that it is hard to know the relevance of this finding, particularly given that the results can vary with the activation function (e.g., the sigmoid activation function does not generate sparsity) and the optimization algorithm (e.g., SGD with a low learning rate apparently does not generate sparsity). 

clarity,_quality,_novelty_and_reproducibility: I am not aware of prior studies showing that noise can induce sparsity. 

The text of the paper is clear overall. I was able to follow what they did. 

The authors perform many analyses on single-layer networks, which gives one a sense of the robustness of the results in that setting. However, much less is done with multilayer networks, which is a substantial limitation.

The authors provide code implementing their models and analyses. 

When the authors train on the output of CIFAR10 embeddings it is unclear whether the noise is applied to the embeddings or the input. It would be useful to do both and report the results.

I assume that when measuring the sparsity of the network, this is done on testing data without noise?

I wondered if the sparsity results were robust to the scale of the weight initializations?

summary_of_the_review: The paper reports a phenomenon that is novel to knowledge. The relevance to multilayer networks is unclear which limits the impact of the study to the ML community. The relevance to biology is also unclear for the reasons noted above. As a consequence, I am doubtful that the study will have a substantial impact on the ML or neuroscience community. 

Response to rebuttal

I apologize for my slow response and thank the authors for responding to each of my points. I still find the paper underwhelming, because of the limitation to shallow models (though I appreciate the authors are working on extending to deeper models) and the lack of any quantitative model comparisons with biological data (which is of course challenging, but needed if one wishes to make claims about relevance to biology).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Reviewers agree that while interesting, the paper is not ready for publication. Several major issues were raised with respect to missing baselines (sparse coding), limited novelty (Denoising Auto-Encoders and related RBMs are well known to yield Gabor-like filters [1,2]) and weak experimental results (small models, datasets, and no clear sign that the proposed method can yield sparsity while retaining performance on tasks and models of interest).

Perhaps even more important are the points raised by [AnUn]. There are many ways to induce sparse activations and Gabor-like filters. Unfortunately for ML practitioners, the authors do not convincingly establish why their method should be preferred or why it should be integrated into ML pipelines. On the other hand, the paper will also be of limited interest to the neuroscience community as no effort is made to establish fit to biological data. I would encourage the authors to pursue one of these directions before resubmitting.

[1] https://www.jmlr.org/papers/volume11/vincent10a/vincent10a.pdf, see Fig 6 for DAEs applied to natural image patches.
[2] https://www.cs.toronto.edu/~fritz/absps/reluICML.pdf (RBMs applied to natural images of faces, Fig 8)
justification_for_why_not_higher_score: Reviewers were unanimous in their decision, with which I agree 100% after reading the paper and reviews. Furthermore, the rebuttal did not significantly change the main points of criticism.
Decision: Reject