Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work presents the use of a hybrid CNN-Transformer model for few-shot image classification. Specifically, the paper applies encoding and encoding-decoding transformers between convolution layers to increase the learning capacity.  The paper also includes a Conv-based hybrid model and uses Omniglot, miniImageNet, and tieredImageNet for evaluations. 
main_review: Strengths: 

– The overall idea of the paper to increase the learning capacity looks interesting. 


Weaknesses:

– I think the novelty of the current version of the paper is incremental, and adding transformer layers between Conv4 would likely increase the Conv4’s capacity due to the over-parameterization. 

– While respecting the effort of the author(s) on evaluation of their idea using tieredImageNet, I think we need some additional large datasets instead of Omniglot or miniImagaNet in the case of measuring the effectiveness of a model with higher capacity. Here, deep backbones (such as ResNet) are required in the evaluations. Of course, we should also consider an appropriate regularization technique while increasing the model capacity of deeper backbones.  

– The presentation of the paper is not clear in some parts of the paper. For example, there are some grammatical errors in the paper (e.g. opening sentence of section 3). I also had problems with understanding some of the statements. For example, the first sentence of the second paragraph in section 1 “, relies on pretraining to a low-dimensional embedding...“. We know that some metric-based methods (like ProtoNet) can be applied to the flatten output of ResNet12 (resulting in high dimensionality) and still work well.   
summary_of_the_review: I think the current version of the paper needs to be strengthened from novelty, evaluation, and presentation perspectives.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method for solving few-shot image classification that generates all the weights of a very small CNN model. This has the advantage that the generated model can be very small+compact, compared against for example some embedding methods that might require large image classification networks to run as a pre-process. The method is also able to handle unlabeled samples in a fairly natural way.
main_review: Overall I think the method is fairly interesting and a useful contribution to the "small model" few-shot image classification problem. I think the main weakness of the problem is that peformance seems fairly comparable to existing methods if one is willing to use a "test time" model of any model complexity. While there are several "meta network" approaches that directly generate the weights of another, simpler network, I think there are a few useful/generalizable insights that the paper presents that will be of use to other methods; in particular, the two-stage transformer visualized in Fig 1. The separation into a high-capacity model for parsing the support set, and a low-capacity model for executing, is a natural one for this problem. However in most practical cases I have worked with the cost of a larger backbone network is fairly minimal on modern hardware, it's still nice to have smaller models where possible and the paper demontrates that this can be achieved with comparable accuracy.

When the paper initially claims "the transformer model is invariant to sample permutations" I was slightly confused as this depends on how exactly the transformers are used (in NLP the token order is certainly relevant.)  Some earlier clarity about how the paper intends to use transformers would help here.

Reasserting early on that this is focused entirely on few-shot image classification will also help here, there are plenty of other relevant few shot problems (value regression, image regression/translation, document classification, and so on.)  The paper doesn't clarify this at all in the abstract or early part of the intro.

I am familiar with this general topic but as it is not my primary area of research, I may have missed some prior work approach here.
summary_of_the_review: 
The paper shows an interesting way to apply "high capacity" transformers to generate the weights of a low-capacity CNN model for few-shot image classification. While results are comparable to SOTA high-capacity classifier networks, the meta-network generation is interesting and able to produce very low-capacity classifiers.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper suggests HyperTransformer, a transformer-based model that produces weights of CNN models in meta-learning setup. Experimental results show that the proposed approach improved the performance of CNN models below a certain size in few-shot classification and semi-supervised few shot classification tasks.

main_review: 1. It is quite interesting to see a self-attention computational circuit can be interpreted as applying a single gradient descent step with the cross-entropy loss.
2. I hate to say, but experimental results are far from state-of-the-art results. why not experimenting with larger CNN and transformers networks?
3. the replacement of existing CNNs with transformers might not be a meaningful contribution to the community.
4. Which encoder architecture did you use eventually? (a) or (b)? I personally prefer to focus on the final architecture you used, and focus on arguing more about it.
5. Any comparisons against other ‘weight generating’ approaches?

[minor comments]
- Fonts in the Figure 1 and 2 are too small, hard to see
- Could you explain about ‘placeholders’ a bit in detail? I think I get the idea, but it's good to be wordy for a broader audience.

summary_of_the_review: This paper shows that the recent transformer architecture works for meta-learning setup as a hypernetwork. Although I believe it is a good information, but I am not fully convinced that this has enough contribution to be a full ICLR paper. Experimental results shows some improvements, but it is not convincing enough so that researchers will follow up on this work later. Therefore, I tend to reject this submission.
review_title: null

Meta-review: The paper uses a transformer model to generate CNN models and use it for few shot learning.

Although the reviewers appreciate the ideas and the good benchmarking results presented in the paper they are find the paper somewhat incremental compared to previous work in the hyper network literature. This also despite the authors thorough rebuttal with additional results. This shows that the authors could have done a better job in presenting their work.

Rejection is therefore recommended with a strong encouragement to rework the paper to counter future reviewers having similar reservations.
Decision: Reject