Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The main point of this paper is that we can rest easy about the biological plausibility of convolutional networks, because a simple hebbian mechanism - along with a “sleep phase” - provides  a method for making initially random weights match up, i.e., makes a locally connected network convolutional. The hebbian learning mechanism moves weights at every kth interval to the same value, while keeping them from moving very far from their initial value. 

Through experiments on several small datasets, the authors show that this mechanism works better than simply using lots of translations of the image to make the weights similar. In addition, the performance of the hebbian-regularized model is not far from a standard convnet, and on Tiny ImageNet, it actually performs a bit better than a standard convnet. 

On ImageNet, the mechanism results in performance similar to a regular convolutional network.

I have read the authors' response, and I am going to stick with my rating. If the paper does not get into NeurIPS, I encourage the authors to continue their work and submit it somewhere else, e.g., WACV or Vision Research.
main_review: Originality: As far as I know, this is original.

Quality: The work appears sound. There is a proof of the dynamics of the hebbian part in the appendix that I didn’t check. The claims in the abstract are well supported by the results.

 Clarity: Is the submission clearly written? Yes: the paper is quite clear.

Significance: Are the results important? To the extent that they give a biologically-plausible mechanism for generating nearly-convolutional networks, yes. One has to care about such things; I do. The work provides a justification of layered convolutional networks as models of the visual system. 

Minor comments:
line 227: it is to frequent -> it is too frequent.
limitations_and_societal_impact: The authors are clear about the limitations; the model works best when the sleep phase is implausibly frequent. The model requires fairly precise connectivity, and there is a somewhat implausible sequential application of the hebbian part, where layer l is regularized before layer l+1. While the authors suggest that this could be done simultaneously, but would take longer, this is not demonstrated. 

In terms of societal impact, I don’t see any way that this would have societal impact, and neither do the authors.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper designed a convolutional neural network that relaxed the weight-sharing constraint by adding local lateral connections to implement dynamic weight sharing as a "sleep phase". The idea was inspired by the biological neural network, where a strict weight-sharing mechanism does not exist. They further used a Hebbian learning rule for weight update during the sleep phase. Their results suggest such a design may have comparable performance as the traditional CNN on image recognition tasks, and it is more computationally efficient than using data augmentation (e.g., translation) to force weight-sharing.
main_review: Questions and Comments:
1. Is each dynamic weight sharing update (the "sleep phase") an iterative process? Or, more specifically, is it a one-step update as described in Eq. (4), or does it converge to Eq. (5) during one intermediate "sleep phase" before starting the next "training phase"?
2. Fig. 3A illustrates the assumption that the model design will reconcile weight divergence during separate learning and weight convergence during the sleep phase. Has the author verified whether the proposed model actually exhibits this learning behavior during training?
3. The result in Table 1 suggests that more frequent weight sharing (e.g., once every batch) shows better visual task performance. Is this because "equalizing" the weights every batch may force the learning process to avoid large divergence that cannot be well-compensated by the "sleep phase"?
4. In this work, the authors remove the weight-sharing constraint in the CNN but introduce a "sleep phase" as a weight regularization process. As the original motivation is to make the model more "brain-like", what is the analogy of this "sleep phase" in the biological neural network? Is there any inspiration about how and when this dynamic weight-sharing process should happen during the learning process?
limitations_and_societal_impact: N/A.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a biologically plausible substitute for convolutional networks, which uses locally connected networks with data augmentation and a weight-sharing procedure. The experiments on some benchmark datasets show that the proposed algorithm achieves good results close to convolutional networks.
main_review: Strength

The paper proposes the weight sharing procedure that is the key point of the algorithm, which enforces the weights of each neuron to be the same and close to their average. This idea is interesting and novel. It shows from simulations the weights could converge with a relatively small number of iteration, and also gives a realistic neural dynamic model to justify the weight sharing update. The experiments on CIFAR and ImageNet data show adding weight sharing can improve the LC network performance. 

Weakness

1. The weight sharing update rule is given in (4), which is also Hebbian's learning rule and effectively drives all the weights to the mean. Here z_i should be taken as the input signal from previous layer x, and (z_i - 1/N\sum z_j) should be the activation of the neuron (fire rate). It is a little bit strange that in the lateral connections, the neuron j sends its input signal z_j directly to the neuron i, instead of the activations. Maybe there are some steps omitted, but I think the authors should make the dynamics clearer.
2. I'm not sure that in the real data experiments, whether the authors implement the update rule (4) or simply compute the final solution (5), or even just take the mean of the weights. Also, if using (4) or (5), does the covariance matrix C computed from current batch or all previous batch, I was not able to find these details in the paper.
3. It can be observed from table 3 in Appendix B that the results are much worse when removing the padding, and also increase the frequency of weight sharing operation is not very helpful in improving accuracy. It shows padding plays a more important role in the model and improvements from the weight sharing procedure are relatively marginal.


--- after rebuttal ---

I appreciate the author's feedback. After reading the response and other reviews, I would like to keep my original rating for this paper.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents a way to increase biological plausibility in convolutional neural networks. In particular, this paper focuses on two alternatives to weight sharing that aim at the same regularization principle. In particular, the authors propose data augmentation via multiple image translations and, much more relevant, dynamic weight sharing via lateral connections, which allows neurons to share weight information during a sleep-like phase. The paper is clear, well-structured, and I think it has value and interest for the community. 
main_review: The paper presents a way to increase biological plausibility in convolutional neural networks. In particular, this paper focuses on two alternatives to weight sharing that aim at the same regularization principle. In particular, the authors propose data augmentation via multiple image translations and, much more relevant, dynamic weight sharing via lateral connections, which allows neurons to share weight information during a sleep-like phase. The paper is clear, well-structured, and I think it has value and interest for the community. 

My main concern, or question/comment, is related with the main motivation of this paper: What is the problem with the fact that ConvNets are biologically implausible or that could be incompatible with current understandings in neuro-biology? I consider that all proposals to improve deep nets training or design are welcome, but I do not see the need of explicitly searching per se for biologically realistic algorithms. As Yann LeCun would say (https://cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf): "Let's be inspired by nature, but not too much". It is indeed nice to be able to emulate/imitate nature, but we mainly need to understand what is actually relevant for our practical purposes: "How do we know which details are important? How do we know which details are merely the result of evolution, and the constraints of biochemistry? For airplanes, we developed aerodynamics and compressible fluid dynamics, and we figured out that feathers and wing flapping were not crucial". From this point of view, I think that any research endeavor is worth of trying, but I would like to see a better contextualization and/or justification of why we need more biologically plausible models. For instances, the authors also remark that "we trained networks using backpropagation, which is not biologically plausible". Why is that problematic? In other words, from my point of view, being inspired by nature is fine, but that inspiration should be translated into meaningful and justified design of the method in question, which not only should make sense, but also be qualitatively novel and quantitatively advantageous with regard to previous approaches. 

Related to the aforementioned comment, there are some statements in the paper that remain a bit unclear to me:
- "Such models can guide medical applications such as brain machine interface and neurological rehabilitation. However, that could also lead to the design of potentially harmful adversarial attacks on the brain". How the introduced approach could be used in neurological rehabilitation? What is an adversarial attack on the brain?  
- "it achieves near convolutional performance even on hard tasks, such as ImageNet classification, making it a much more likely candidate than data augmentation for the brain". What does it mean to be a "more likely candidate [...] for the brain"?

From the methodological point of view, I have several comments/questions that, in my humble opinion, if taken into account would make the paper stronger: 
- In the context of ConvNets for image classification problems, locally connected networks, if I'm not mistaken, according to Figure 1, imply the use of different filters for different regions of the image. Is that correct?
- Does this dynamic weight sharing via lateral connections slow down the training procedure? By how much?
- "The lateral connections are used only for dynamic weight sharing, and not for actual computation". What does this mean exactly?
- What is the actual difference in terms of parameters to train between the ConvNets and the LC networks employed in the experiments?
limitations_and_societal_impact: Yes, they do, at the end of Section 4.
review_title: null

Meta-review: This paper received 3 marginal accepts and 1 accept. The reviewers stated that their lack of enthusiasm had to do with the somewhat niche aspect of the work (not because of technical or experimental limitations). Many NeurIPS attendees do care about biology and the work should be of interest to this community as noted by one of the reviewers. The AC thus recommends acceptance.

Decision: Accept (Poster)