Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors present Direct ONE-shot learning with Hebbian imprinting (DONE) a method for one-shot learning inspired by Hebbian learning in the brain. The method uses neural activations from the final layer of an “encoder” network (such as a vision transformer or EfficientNet) on a single example from an unseen class to create a weight vector for the class of the new class. The presented method is closely related to and improves upon a that of a 2018 paper by Qi et al, which the authors cite. 

The authors present experiments in which DONE is compared to Qi’s method. Specifically, the authors one-shot-learn 1 or 8 classes, evaluate the performance on those classes, and measure the degree to which the new classes interfere with initially trained classes. The authors also present results on k-shot learning. 

strengths_and_weaknesses: Strengths:
1. The paper presents a novel and simple method for few-shot learning. 
2. While the paper does not claim to model the brain, it is exciting to see that one-shot learning with brain-line Hebbian imprinting can work so well. 
3. This paper is a pleasure to read. It is well-structured, and the writing is mostly clear. 

Weaknesses
1. The authors compare their method with only one method, when it would have been helpful for readers to compare to a broader range of existing one-shot learning approaches for image classification. The authors state “It is meaningless to compare the above three approaches with weight imprinting, because weight imprinting does not contain any optimization algorithm. “ . I disagree with this, for two reasons. First, papers such as the one about one-shot learning with siamese networks (https://www.cs.cmu.edu/~rsalakhu/papers/oneshot1.pdf), which the authors cited) are quite similar to DONE in that a big model gets trained on large amounts of data in a time-consuming process, and later one-shot classification becomes cheap. In other words, both were optimized at some point. Second, even if two models fall into different categories (e.g. because one is much more computationally expensive than the other), it’s useful for readers to know how much accuracy they lose (if any) by using a more flexible methodology. 
2. While the algorithm from the paper is novel and undoubtedly very elegant, it is very similar to Qi’s method and other papers that the authors cite. In this sense, the paper is not a must-read for most researchers in the community. 

questions: 1. The authors write on line 31 that “The human brain does not necessarily have more complex processes than DNNs…”. I recommend to remove this statement. Many researchers would disagree with the statement, and the statement is not essential to the paper. 

2. The authors write on line 33 that “a series of simple processes such as linear filtering followed by a nonlinearity can describe the function of lower visual cortex”. I recommend to rephrase this slightly, as learning in the lower visual cortex is part of the lower visual cortex’s function, but that aspect is not described by a series of simple processes such as linear filtering followed by a nonlinearity. 

3. “modify” on line 48 should be “modifies”

4. On line 61, the authors write “neural activity and synaptic strength are different in dimension.” Do the authors mean  “neural activity and synaptic strength are different in scale.”? Line 157 also mentions “different dimension” when the authors may have meant “different scale”. 

5. On line 135, the authors write “the backbone DNN model is a very good model as a heritage of mankind”. I did not understand this sentence. Could the authors rephrase it?

6. Figure 2 is impossible to understand without referring to explanations of data point markers in section 4.1, which means that a reader needs to jump back and forth between the Figure 2 and the text on the page after. I recommend explaining the figure markers in Figure 2’s caption instead of in section 4.1. 

7. On line 185, the authors write “(not 1008 classes here)”. At this point, the reader has not yet read about the 1008-class experiments. How about removing the comment in parentheses?

8. The EfficientNet architecture is misspelled in a number of places in the paper, e.g. as “EfficinetNet” and “EfficientNnet”

9. Koray Kavukcuoglu’s name should be upper-cased on line 377. 
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a one-shot learning mechanism that adds a new class to the network's output using quantile normalization of the new input (to the last layer) w.r.t. weights in the last layer that correspond to the old classes.
strengths_and_weaknesses: ## Strengths:
1. The proposed one-shot classification method is simple and, unlike previous work, doesn't require layer normalization in the second last layer.
2. The method works reasonably well.
3. The method is novel, as far as I know.

## Weaknesses:
1. This work has "Hebbian" in the title and some part of the text, but as far as I can tell has nothing to do with Hebbian learning. (Elaborated in the Questions section)
2. Performance of the method is not adequately compared to previous work. (Again, elaborated below) 

## Summary

My current score is 4 (borderline reject). The paper proposed an interesting method, but it doesn't properly explain why it works, and it doesn't justify the lack of comparison (in terms of performance and computational complexity) with previous methods. I'm willing to increase the rating if my points are addressed, however.
questions: ## Large concerns
### Hebbian learning
This work does not describe a Hebbian mechanism of learning. Hebbian means $\Delta W_{ij} = f(x_i, y_j)$ for input neuron $x_i$ and output neuron $y_j$ (in the classical Hebbian sense, $f(x, y)=xy$). Eq. 3 is fundamentally not Hebbian because it uses information about other weights. Lines 154-155 say quantile normalization "is suitable for implementing Hebbian theory", but this claim is not backed up. The authors should either thoroughly explain what makes their method Hebbian, or remove mentions of Hebbian learning.

### Comparison with previous methods
In lines 104-109 the authors try to explain why they’re not comparing their method to previous work directly:
> It is meaningless to compare the above three approaches with weight imprinting, because weight imprinting does not contain any optimization algorithm. Therefore, in principle, there is no reason for weight imprinting methods to outperform other methods by themselves in accuracy. The performance of weight imprinting methods is uniquely determined by the backbone DNN without any randomness, hence its performance is suitable as a reference baseline for other methods. Thus, weight imprinting does not aim for the highest accuracy but for practical convenience and reference role as a baseline method.

– I can agree with not chasing the best accuracy, but then there must be a discussion of concrete practical benefits of the method. (No optimization != faster, as quantization takes some time too.)

### Overall reason behind the method
Section 3.1: I did not understand why quantile normalization helps. Shouldn’t it make the response in each neuron $y$ similar to that of the new class? Either way, the authors should spend more time explaining their method and why it works, as it is the core contribution.

## Unclear parts in the text

Lines 135-136: "the backbone DNN model is a very good model as a heritage of mankind and should not be changed as much as possible (especially for many non-expert users)"
– this is not an explanation.

153: what probability distributions? Everything has been deterministic so far, so it's worth explaining what distributions are discussed here. Is it over individual weights? Individual neurons?

166: "The range of applicable models is yet unclear, but in principle it is wider than Qi’s method."
– what does it mean? The line before said that you need a network with a dense final layer, which clearly defines the range of models.

223-224: what’s good accuracy for practical uses? It’s never explained how the authors came up with that value

## Small corrections
**Note**: the paper contains a lot of grammatical errors. One simple way to fix it is to copy-paste the source code into a google doc and go through all suggested corrections.

Line 15 (abstract): performs "at a", not "a", practical-level accuracy?

Also next sentence: Can write as "DONE overcomes ...mentioned issues of DNNs..."; Currently the sentence doesn’t read well.

92: uses "a" classification

93: aim"s"

145: applies to, not apply into

212-214: black and orange said twice

Why is the supplementary material just a corrected version of the main paper?

limitations: The authors can mostly addressed the limitations of their work (although see below), but they did not account for the potential negative impacts. I'm not flagging it for an ethics review as it's just a small correction though.

Checklist 1C: the described model can be easily adapted for face recognition. This does imply potential negative societal impacts and should be discussed by the authors, even though previous work can be used in a similar way.

Related, the authors refer to practical uses throughout the whole paper, in particular when talking about performance of DONE. But it’s never discussed what those are, and why the achieved performance of the method is good enough for them.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents a method for the one-shot classification of novel inputs. For the classification of novel inputs, the method uses representations obtained by the final dense layer of a pretrained backbone model to create novel weight vectors of new classes. Importantly, DONE does not optimize or change the backbone model in any way, making it applicable to various backbone models and computationally efficient.
strengths_and_weaknesses: Strengths:
1) Introducing quantile normalization in machine learning, a method that was unknown to me.

Weaknesses:
1) The link to Hebbian learning is weak and not meaningful for this paper goals.
2) The idea that the last dense layer of a DNN includes representations that allow novel classifications is not novel, without any model modification, is not novel (e.g., [1] analyzed it theoretically).
3) There are many writing problems. Too many to mention them all. Examples: a. talking about the heritage of mankind in line 135 (unprofessional); b. figure 2 has an irrelevant screen cursor at the right-most data point. More importantly, essential legend details of this figure appear in the text instead of the caption, which is confusing; c. “the simplest” in line 324 is also not professional (I think that Qi’s method is simpler). 
4) The method does not improve upon previous works Figure 2 shows that Qi’s method works similarly or better. I am not convinced that the backbone modifications it does are detrimental.
5) Accepting the updated paper that was submitted via the supplementary materials, at a later deadline, is unfair to other authors.

[1] Sorscher, B., Ganguli, S., & Sompolinsky, H. (2021). The geometry of concept learning. bioRxiv.‏
questions: My main suggestion is that you would make a further effort to convince this method's benefits over Qi's. In the current paper, the benefits are argued in text. Try to argue using a test where Qi's modification to the backbone model impairs performance, while DONE does not.

Also, I suggest adding more details about the practical implementation of quantile normalization.

Finally, the paper needs to be thoroughly edited.
limitations: The paper adequately discusses the method limitations.
review_title: null

Meta-review: metareview: ## Summary
Humans can learn a new task just from a couple of examples whereas often supervised deep learning models require lots of labeled samples to learn a task.  This paper proposes a one-shot learning method inspired by Hebbian learning by adding a new class to the output layer of the network with quantile normalization of the new inputs based on the weights of the last layer that corresponding to the other classes. The proposed approach uses features of a penultimate layer of an “encoder” network (for example EfficientNet). The paper presents results in the k-shot classification setting.

## Decision
This paper studies an important problem and introduces interesting ideas such as quantile normalisation into deep learning models.  Nevertheless the paper requires a revision as discussed with other reviewers and as a result it would benefit from another round of reviews. However, during the discussion period none of the reviewers were willing to nominate this paper for acceptance with confidence. 

* *Reviewer j4P6* claimed that the authors claims regarding to connections to Hebbian algorithm is flawed and  during the discussion period authors agreed their algorithm is not Hebbian. It seems like this is the biggest problem reviewer j4P6 has with theispaper. The reviewer also complained that the paper only has a handwavy explanation of the algorithm and some limited evaluations. It’s a decent contribution overall, but in its current form it does not meet the bar of NeurIPS.
* *Reviewer 8B2d* thinks that this paper needs another revision. The method presented in the article is interesting, but the current manuscript evaluates it with only two models, and the results are better in just one of them. It is therefore not clear whether the improvement is anecdotal or general. Furthermore, the manuscript relates the work to Hebbian learning in the brain, which is irrelevant for presenting their work. The authors agreed to that point when discussing with reviewer j4P6. The reviewer thinks that the article is interesting, but the authors should revise the manuscript, remove the Hebbian analogy and focus on better evaluations, making their statements more sound.
* *Reviewer RJBn*’s score is a “6 - Weak Accept”, but fine with this paper getting rejected. After reading the reviews of other reviewers, RJBn agrees with the other reviewers that the connections drawn to Hebbian learning in the brain are weak, and that the evaluation could be better.

confidence: Certain
Decision: Reject