HyperTransformer: Attention-Based CNN Model Generation from Few Samples
Authors: Andrey Zhmoginov
Max Vladymyrov
Mark Sandler
Venue: ICLR 2022 Submitted (2022)
Keywords: few-shot learning
transformer model
weight generation
supervised learning
semi-supervised learning

Abstract: In this work we propose a HyperTransformer, a transformer based model that generates all weights of a CNN model directly from the support samples. This approach allows to use a high-capacity model for encoding task-dependent variations in the weights of a smaller model. We show for multiple few-shot benchmarks with different architectures and datasets that our method beats or matches that of the traditional learning methods in a few-shot regime. Specifically, we show that for very small target models, our method can generate significantly better performing models than traditional few-shot learning methods. For larger models we discover that applying generation to the last layer only, allows to produce competitive or better results while being end-to-end differentiable. Finally, we extend our approach to semi-supervised regime utilizing unlabeled samples in the support set and further improving few-shot performance in the presence of unlabeled data.

Under review as a conference paper at ICLR 2022
HYPERTRANSFORMER: ATTENTION-BASED CNN MODEL GENERATION FROM FEW SAMPLES
Anonymous authors Paper under double-blind review
ABSTRACT
In this work we propose a HyypeTTrnssfrret a transformer based model that gen- erates all weights of a CNN network directly from support samples. This approach uses a high-capacity model for encoding task-dependent variations in the weights of smaller model. We show on multiple few-shot image classification bench- marks with different model sizes and datasets that our method beats or matches the performance of many traditional learning methods. Specifically, we show that for very small target architectures, our method can generate significantly better per- forming models than traditional few-shot learning approaches. For larger models we discover that generating the last layer alone allows produce competitive or better results while being end-to-end differentiable. Finally, we extend our ap- proach to semi-supervised regime utilizing unlabeled samples in the support set and further improving few-shot performance in the presence of unlabeled data.
INTRODUCTION
In few-shot learning, a conventional machine learning paradigm of fitting a parametric model to training data is taken to limit of extreme data scarcity where entire categories are introduced with just one or few examples. A generic approach to solving this problem uses training data to identify parameters & of learner ag that given small batch of examples for particular task (called support set) can solve this task on unseen data (called a query set).
One broad family of few-shot image classification methods frequently referred to as metric-based learning, relies on pretraining an embedding e. and then using some distance in the embedding space to label query samples based on their closeness to known labeled support samples. These methods proved effective on numerous benchmarks sian et al. (2020) for review and references) however the capabilities of the learner limited by the capacity of the architecture itself, these methods try to build a universal embedding function.
On the other hand, optimization--assen based methods such seminal MAML algorithm (Finn et al. 2017) can fine-tune the embedding e& by performing additional SGD updates on all parameters of the model producing it. This partially addresses the constraints of metric-based methods by learning new embedding for each new task. However, in many of these methods, all the knowledge extracted during training on different tasks and describing the learner as still has "fit" into the same number of parameters as the model itself. Such limitation becomes more severe as the target models get smaller, while the richness of the task set increases.
In this paper we propose a new few-shot learning approach that allows us to decouple the complexity of the task space from the complexity of individual tasks. The main idea is to use the transformer model (Vaswani et al.] 2017) that given few-shot task episode generates an entire inference model by producing all model weights in single pass. This allows us to encode the intricacies of the avail- able training data inside the transformer model, while still producing specialized tiny models that can solve individual tasks. Reducing the size of the generated model and moving the computational weight generator,
overhead to the transformer-base images. This can dramatically reduce the overall computation cost in cases, where the tasks change infrequently and hence the weight generator is only used sporadically.
lower the cost of the inference new
We start by observing that the self-attention mechanism is well suited to be an underlying mech- anism for few- CNN weight generator. In contrast with earlier CNN- (Zhao et al. 2020) or

---

Under review as a conference paper at ICLR 2022
BiLSTM-based approaches (Ravi & Larochelle 2017), the vanille transformer model is invariant to sample permutations and can handle unbalanced datasets with a varying number of samples per category. Furthermore, we demonstrate that single-layer self-attention model can replicate sim- plified gradient-descent-basee learning algorithm. Using a transformer-base model to generate the logits layer on top of conventionally learned embedding, achieve competitive results on several common few-shot learning benchmarks. Varying transformer parameters we demonstrate that this high performance can be attributed additional capacity of the transformer model that decouples its complexity from that of the generated CNN.
We then extend our method to support unlabeled samples by using a special input token concate- nated to unlabeled samples to encode unknown labels. In our experiments, we observe that using transformers with two or more layers, we achieve better performance by adding unlaheled data into the support set. We explain our results in Section 4.3 where we show that such a transformer with at least two layers can encode the nearest-neighhor style algorithm that associates unlabeled sam- ples with similarly labeled examples. In essence, by training the weight generator to produce CNN models with best possible performance on a query set, we teach the transformer to utilize unlabeled samples without having to manually introduce additional optimization objectives.
We also show that by generating all layers of the CNN model we can improve both the training and the test accuracies of CNN models below a certain size. The training accuracy can be viewed as a capability of the generated CNN model to adapt to tasks seen at training time, whereas the test accuracy computed on unseen categories characterizes the generalization capability of this model adaptation mechanism. We empirically demonstrate the expected increase of the model training and test accuracies with the increase of the layer size and the number of generated layers (see Figure 3 Interestingly, generation of the logits layer alone appears to be "sufficient" above a certain model size threshold. This threshold is expected to depend on the variability and the complexity of the training tasks. We conjecture that this might reflect the fact that models are sufficiently expressive for the benchmarks we considered, or that additional regularization is needed to prevent overfitting of the meta-learning models.
Finally, in addition to being able to decouple the complexity of the task distribution from the com- plexity of individual tasks, another important advantage of our method is that it allows to do learning end-to-end without relying on complex nested gradients optimization and other meta-learning ap- proaches, where the number of unrolls steps is large.
The paper is structured as follows. In Section 2, we discuss the few-shot learning problem sctup and highlight related work. Section 3 introduces our approach, discusses the motivation for choosing an attention- based model and shows how our approach can be used to meta-learn semi-supervised learning algorithms. In Section 4, discuss our experimental results. Finally, in Section 5, provide concluding remarks.
2 PROBLEM SETUP AND RELATED WORK
2.1 FEW-SHOT LEARNING
The main goal of a few-shot learning algorithm to use a set of training tasks Tiaain for finding a learner as parameterized by & that given new task domains can train to recognize novel classes using just a few samples per each class. The learner as can be thought of as a function that maps task description T = (z), containing k labeled input samples {z,, from n classes, the weights 0 = (T) of a trained model fax: 0). The parameters & are meta-optimized to maximize the performance of the model far; as (T))) generated using support set Ts with x drawn from a query set To. Each task 7' = (T's, T2) is randomly drawn from space of training tasks Traiin Typically, T's and To are generated by first randomly choosing several distinct classes from the training set and then sampling examples without replacement from these classes to generate T's and To- In a classical -o-woy-k-shot" setting, is the number of classes randomly sampled in each episode, and l is the number of samples for each class in the support set Ts.
The quality of a particular few-shot learning algorithm is typically evaluated using a separate test space of tasks Test. By forming Trest from novel classes unseen at training time, we can evaluate
without attention masking or positional encodings
2

---

Under review as a conference paper at ICLR 2022
generalization of different learners ag. Best algorithms are expected to capture the structure present in the training set and to perform well on novel concepts. This structure may, for example, include certain properties of the distributions Pa(z)) with being the class label, or the presence of particular discriminative (or alternatively invariant) features in the tasks from Terain-
2.2 RELATED WORK
Few-shot learning received a lot of attention from the deep learning community and while there are hundreds of few-shot learning methods, several common themes emerged in the past years. Here we outline several existing approaches, show how they relate to our method and discuss the prior work related it.
Metric-Based Learning. One family of approaches involves mapping input samples into an em- bedding space and then using some nearest neighbor algorithm that relies on the computation of distances from a query sample embedding to the embedding computed using support samples with known labels. The metric used to compute the distance can either be the same for all tasks, or can be task-dependent. This family of methods includes, for example, such methods as Siamese networks (Koch et al. 2013), Matching Networks (Vinyals et al. 2016), Prototypical Networks (Snell et al.) 2017]. Relation Networks (Sung et al., 2018)) and TADAM (Oreshkin et al. 2018). It has recently been argued (Tian al.] (020)) that methods based on building powerful sample representation meth- ods. However, such approaches essentially amount to the "one-model solves all" approach and thus require larger models than needed to solve individual tasks.
can frequently outperform numerous other approaches including many optimization-base
Optimization-Base Learning. An alternative approach that can adapt the embedding to a new task is incorporate optimization within the learning process. A variety of such methods based on the approach called Model-A Model-Agnostic Meta-Learning, or MAML (Finn et al.) 2017). In MAML, aG obtained by initializing a DNN at On- = Φ and then performing one or more gradient de- scent updates on a classification loss function L, i.e., computing 0x+1 0₂ y (OL/00)(T) 0x)- This approach was later refined (Antoniou et al. 2019) and built upon giving rise Reptile (Nichol et al. 2018). LEO (Rusu et al., 2019) and others. One limitation of various M MAML-inspired meth- ods is that the knowledge about the set of training tasks Trrain distilled into parameters & that have the same dimensionality as the model parameters O. Therefore, for very lightweight model fz; 0) the capacity of the task-adaptation learner ag is still limited by the size of 0. Methods that use parameterized preconditioners that otherwise do not impact the model fæz 0) can alleviate this issue, but as with MAML, such methods can be difficult to train (Antoniou et al. 2019).
Weight Modulation and Generation. The idea of using a task specification to directly generate or modulate model weights has been previously explored in the generalized supervised learning context
(Ratzlafi & Li 2019) and in specific language models (Mahabadi et al., 2021 Tay et al. 2221 Ye
& Ren 2021). Some few-shot learning methods described above also employ this approach and
task-specific generation or modulation of the weights of the final classification model. For example, in LGM-Net (Li et al., 2019b) the matching network approach is used generate a few layers on top of a task-agnostic embedding. Another approach abbreviated as LEO (Rusu et al.) (2019) utilized similar weight generation method to generate initial model weights from the training dataset in a few-shot learning setting, much like what is proposed in this article. However, in (Rusu et al.) 2019, the generated weights were also refined using several SGD steps similar to how it is done in MAML. Here we explore a similar idea, but largely inspired by the HypERNETwOND (Ha et al. 2017). we instead propose to directly generate an entire task-specific CNN model. Unlike LEO, we do not rely on pre-computed embeddings for images and generate the model in a single step without additional SGD steps, which simplifies and stabilizes training.
approach
Transformers in Computer Vision and Few-Shot Learning. Transformer models (Vaswani et al. 2017) originally proposed for natural language understanding applications had since become a useful tool in practically every subfield of deep learning. In computer vision, transformers have recently seen explosion of applications ranging from state-of-the to-tte--rtttttttt image classification results
(Dosovitskiy et al.] 2021: Touvron et al. 202I) to object detection (Carion et al.) 2020 Zhu et al.)
2021), segmentation (Ye et al. 2019), image super-resolution (Yang et al. 2020). image generation
3

---

Under review as a conference paper at ICLR 2022
02
TranSfrmerr
T
f2
Genersted Laye
Autivations
di
Tranformer
Local Feature
fi
Layer
Staree Features
(x2)
3
212
373
C1
C2
C3
Feature Estrattoo
Support Samples
Support Labels
Figure 1: A diagram of our model showing generation of two CNN layers: transformer-basee weight generators receive global and local features along with sample labels as their inputs and produce CNN layer weights (0, and 02). After being generated, the CNN model used to compute the loss on the query set. The gradients of this loss are then used to adjust the weights of the entire weight generation model.
(Chen al. 2021 and many others. There also several notable applications in few-shot image classification. For example, in Liu et al. (2021). the transformer model was used for generating universal representations in the multi-domaii few-shot learning scenario. And closely related to our approach, in Ye et al. (2020). the authors proposed to accomplish embedding adaptation with the help of transformer models. Unlike our method that generates an entire end-to-end image clas- sification model, this approach applied task-dependent perturbation to an embedding generated by an independent task-agnostic feature extractor. In (Gidaris Komodakis 2018), simplified attention-bae based model was used for the final layer generation.
3
OUR APPROACH
In this section, we describe our approach to few-shot learning that we call a HYPERR/RANSFORMER (HT) and justify the choice of the self-attention mechanism as its basis.
3.1 FEW-SHOT LEARNING MODEL.
A learner as (as introduced in Section 2.1 is the core of a few-shot learning algorithm and in this paper, we choose ag be a transformer- based model that takes task description T' { c 4) as input and produces weights for some or all layers [0z]e E [1, L|} of the generated CNN model. For layers with non- non-generated weights, they are learned in the end-to-end fashion as ordinary task- agnostic variables. In our experiments generated CNN models contain set of convolutional layers and final fully-connected logits layer. Here Og are the parameters of the f-th layer and is the total number of layers including the final logits layer (with index L). The weights are generated layer by layer starting from the first layer: (T) (2(01:T) + ... - 0 TT) Here we use 0 B short notation for (0., Ba+1. 0b).
Shared and local features. The local features at layer are produced by a convolutional feature extractor h/, (zf) applied to the activations of the previous layer of = fi-1( O, 01....(-11 for e and 2 = Ti- In other words, the transformer for layer receives
I= {( S6. (I:)). h,,, (ft-1 01...e1)),, Gi }) }
The intuition behind the local feature extractor is that the choice of the layer weights should primarily depend the inputs received by this layer. The shared features, the other hand, are the same
4

---

Under review as a conference paper at ICLR 2022
Weight
Transformer
Class Embeeddings Sample Features
Weight Embeddiigs
Figure 2: Structure of the tokens passed to and received from transformer model.
for all layers and are produced by separate trainable convolutional neural network $6, (35). Their purpose is to modulate each layer's weight generator with a global high-level sample embedding that, unlike the local embedding, is independent of the generated weights and is also fully shared between all layer generators.
Encoding and decoding transformer inputs and outputs. our experiments we considered several different architectures of the transformer model, different ways of feeding I't into it and different ways of converting transformer outputs into the CNN weights.
Input samples were encoded by concatenating local and shared features from I' to trainable label embeddings E(c) with & : [1, n] -> IRd Here n the number of classes per episode and d a chosen size of the label encoding. Note that the label embeddings do not contain semantic information, but rather act as placeholders to differentiate between distinct classes.
Along with the input samples, the sequence passed to the transformer was also populated with special learnable placeholder tokens] each associated with a particular slice of the to-be-generated weight tensor. After the entire input sequence was processed by the transformer, we read out model outputs associated with the weight slice placeholder tokens and assembled output weight slices into the final weight tensors (see Fig. 2 In our experiments we also considered two different ways of encoding k x k x Rinput x Moutput convolutional kernels: (a) generating Routput weight slices with each output token having a dimension of X2 x f'input (we call it "output allocation"), (b) 1-2 weight slices of size Pinput noutput ("spatial allocation"). We show these results in Supplementary Materials.
Transformer model. The input discussed above was passed through a sequence of transformer encoder layers and the output weight tokens were then concatenated into full weight tensor (see Fig. 2). Experiments with alternative architectures employing transformer decoders are discussed in Supplementary Materials.
Training the model. The weight generation model uses the support set to produce the weights of some all CNN model layers. This generated CNN model then processes the query set samples and the cross cross-entropy classification loss computed. The weight generation parameters & are then learned by optimizing this loss function using stochastic gradient descent.
3.2 REASONING BEHIND THE SELF-ATTENTION MECHANISM
The choice of self-attention mechanism for the weight generator is not arbitrary. One motivating reason behind this choice is that the output produced by generator with the basic self-attention is by design invariant input permutations, i.e., permutations of samples in the training dataset. This also makes it suitable for processing unbalanced batches and batches with a variable number of samples (see Sec. 4.3). Now we show that self-attention can express several intuitive algorithms, thus further motivating its utility.
"ach taken is learnable d-dimensional vector padded with zeros the size of the input sample taken
5

---

Under review as a conference paper at ICLR 2022
Supervised learning. Self-attention in its rudimentary form implement method similar to cosine-sime caaiee-imillait-bbaaieee -bsse sample weighting encoded in the logits layer with weights W:
Way
Σ m1=
(m) y
(m)
(I)
which can also be viewed as result of applying single gradient descent step on the cross-entropy
loss (see Appendix A Here n. is the total number of support-set samples (xx(">)/mr E [[, and e(m)) ()m)) are the embedding vector and the one-hot label corresponding to x (m()))
The approach can be outlined (see more details Appendix A as follows. The self-attention opera- tion receives encoded input samples Z = (E(cs),es) and weight placeholders (a(i).0) 0) as its input. If each weight slice W,, represented by a particular token (u(i), 0) produces query Q, that only attends to keys K3 corresponding to samples I with labels cæ matching i and the values of these samples are set to their embeddings Cg. then the self-attention operation will essentially average the embeddings of all samples assigned label i thus matching the first term in W in equation 1
Semi-supervised learning. similar self-attention mechanism can also be designed to produce logits layer weights when the support set contains some unlabeled samples. The mechanism first propagates classes of labeled samples to similar unlabeled samples. This can be achieved by choos- ing the queries and the keys of the samples to be proportional to their embeddings. The attention map for sample would then be defined by a softmax e; Ej. or in other words would pro- portional to exp(e; eÿ). Choosing sample values to be proportional to the class tokens, we can then propagate a class of a labeled sample e, to a nearby unlabeled sample with embedding e,, for which e; ej is sufficiently large. If the self-attention module is "residual", i.e., the output of the self-attention operation is added to the original input, like it is done in the transformer model, then this additive update would essentially "mark" unlabeled sample by the propagated class (albeit this term might have a small norm). The second self-attention layer can then be designed similarly the supervised case. If label embeddings are orthogonal, then even small component of class label can be sufficient for weight slice to attend to it thus adding its embedding to the final weight.
EXPERIMENTS
In this section, we present HYpeR'TRANSFORMET (HT) experimental results and discuss the impli- cations of our empirical findings.
4.1 DATASETS AND SETUP
Datasets. For our experiments, we chose several most widely used few-shot datasets including OMNIGLOT, MINIIMAGENET and TIEREDIMAGENET. MINIIMAGENET contains relatively small set of labels and arguably the simplest to overfit to. Because of this and since in many recent publications MINIMMGGENET was replaced with more realistic TIEREDIMAGENEi datasets, we conduct many of our experiments and ablation studies using OMNIGLOT and TIRREDIMAGENET datasets.
Models. While HT can support generation of arbitrarily large weight tensors by flattening and slicing the entire weight tensor, in this work, we limit our experiments to HT models that gener- ate slices encoding individual output channels directly. For the target models we focus on 4-layer architectures identical to those used in MAML++ and numerous other papers. Generating larger architectures such as RESNET and NIDERESNET will be the subject of our future work. More specifically, used sequence of four 3 x 3 convolutional layers with the same number of output channels followed by batch normalization layers, nonlincarities and max max-pooling stride-2 layers. All BN variables were learned and not generated

---

Under review as a conference paper at ICLR 2022
Table 1: Comparison of HT with MAML+ on models of different sizes and different datasets: (a) 20-way OMNIGLOTand (b) 5-way MINTMAGENET.. Results for MAML++ were obtained us- ing GitHub code accompanying Antoniou et al. (2019), those marked with t are from Antoniou et al. 2019). HT outperforms MAML++ on many few-shot tasks. Accuracy confidence intervals:
OMNIGLOT between 0.1% and 0.3%, MINIIMAGENET between 0.2% and 0.5%.
1-shot (channels)
5-shot (channels)
Approach
16
32
48
64
16
32
48
OMNIGLOT:
MAML++
81.4
88.6
95.6
95.8
97.7
83.2
94.9
98.6
98.8
99.31
HT
87.2
93.7
95.5
95.7
96.2
94.7
98.0
98.6
98.8
98.8
MINE
MAML++
43.9
46.6
49.4
52.21
59.0
64.6
66.8
68.31
HT
45.5
50.2
53.8
55.1
58,5
63.8
67.1
68.1
4.2 SUPERVISED RESULTS WITH LOGITS LAYER GENERATION
Our first experiments compared the proposed HT approach with MAML++ OMNIGLOT, MINI- IMAGENET and TIEREDIMAGENET datasets (see Table Interestingly, for models that had more than 8 channels per layer, the results obtained with HT generating the final logits layer proved to be nearly identical to those where HT was used to generate all CNN layers (see Section 4.4). In our experiments the number of local features was chosen to be the same as the number of model channels and the shared feature had a dimension of 32 regardless of the model size. The shared fea- ture extractor was a simple 4-layer convolutional model with batch normalization and stride-2 3 x convolutional kernels. Local feature extractors were two-layer convolutional models with outputs of both layers averaged over the spatial dimensions and concatenated to produce the final local feature. For all tasks except 5-shot MINUIMAGENET our transformer had 3 layers, used simple sequence of encoder layers (Figure - and used the "output allocation" of weight slices (Section (.T.. Exper- iments with the -ncoder-decoder transformer architecture can be found in Appendix D The 5-shot MINIMMAGENET results presented in Table were obtained with a simplified transformer model that had layer, and did not have the final cully-connected layer and nonlinearity. This proved necessary for reducing model overfitting of this smaller dataset. Other model parameters are described in detail in Appendix B
Results obtained with our method in few-shot setting (see Table I are frequently better than MAML++ results, especially on smaller models, which can be attributed to parameter disentangle- ment between the weight generator and the CNN model. While the improvement over MAML++ gets smaller with the growing size of the generated CNN, our results on MINIIMAGENET and TIEREDIMAGENES appear to be comparable to those obtained with numerous other advanced meth- ods (see Table 2). Discussion of additional comparisons to IGM-Nett (Li et al., 2019h) and LEO (Rusu et al.) 2019) using a different setup (which is why they could not be included in Table 2 and showing an almost identical performance can be found in Appendix C While the learned HT model could perform relatively simple calculation on high-dimensional sample features, perhaps not too different from that in equation [ our brief analysis of the parameters space (see Appendix D) shows that using simpler 1-layer transformers leads to modest decrease of the test accuracy and a greater drop in the training accuracy for smaller models. We observed that the results in Table [ could be improved even further by increasing the feature sizes (see Appendix D but we did not pursue an exhaustive optimization in the parameter space.
It is worth noting that overfitting leading gories, but poor generalization to unseen categories, may still have practical applications. Specifi- cally, if the actual task relies on classes seen at the training time, we can generate model customized to particular task in a single pass without having to perform any SGD steps to fine-tune the model. This is useful if, for example, the client model needs to be adjusted to a particular set of known classes most widely used by this client. We also anticipate that with more complex data augmen- tations and additional synthetic tasks, more complex transformer- models can further improve their performance on the test set and a deeper analysis of such techniques will be the subject of our future work.
good performance tasks composed of seen cate-
'here we assume that the embeddings e are unbiased, i.c., (et) = 0
EEperiments with generated BN variables did not show much difference with this simpler approach.
7

---

Under review as a conference paper at ICLR 2022
Table 2:
Comparison of MINTIMAGENET and TIEREDIMAGENET 1-shot (1-S) and 5-shot (5-S)
5-way results for HT (underlined) and other widely known methods with a 64-64-64- 6444-6444444 model
including (Tian al., 2020): Matching Networks (Vinyals al., 2016). IMP (Allen et al.
2019)
Prototypical Networks (Snell al. (0177). TAML (Jamal & Qi 2019). SAML (Hao et al.
2019
GCR (Li et al.) 2019a), KTN (Peng al. 2019). PARN (Wu et al. 2019), Predicting Parameters
from Activations (Qiao al.) (018), Relation Net (Sung et al. 2018). MELR (Fei et al.
2021 We
also include results for CNNs with fewer channels ("-32" for 32-channel models, etc.).
MINIIMAGENET
TIEREDIMAGENET
Method
1-8
5-8
Method
1-8
5-S
Method
1-S
5-S
HT
54.1
68.5
HT-48
55.1
68.1
HT-32
52.7
69.9
MN
43.6
55.3
SAML
52.2
66.5
MAML--32
51.7
70.3
IMP
49.2
64.7
GCR
53.2
72.3
HT
56.1
73.3
PN
49.4
68.2
KTN
54.6
71.2
PN
53.3
72.7
MELR
55.4
72.3
PARN
55.2
71.6
MELR
56.4
73.2
TAML
51.8
66.1
PPA
54.5
67.9
RN
54.5
71.3
4.3 SUPE-VUDE RESULTS WITH LOGITS LAYER GENERATION
In our approach, the weight generation model is trained by optimizing the loss calculated on the query set and therefore any additional information about the task, including unlabeled samples, can be provided as part of the support set to the weight generator without having to alter the optimiza- tion objective. This allows us to tackle a semi-supervised few-shot learning problem without making any substantial changes to the model or the training approach. In our implementation, we simply added unlabeled samples into the support set and marked them with an auxiliary learned "unlabeled" token & in place of the label encoding E(c).
Since OMNIGLOT is typically characterized by very high accuracies in the 97% 99% range, conducted all our experiments with TIEREDIMAGENE! Results of our experiments presented in 3 that adding unlabeled samples leads substantial increase of the final test accuracy. Furthermore, notice that the model achieves its best performance when the number of transformer layers is greater than one. This is consistent with the basic mechanism discussed in Section 3.2 and requiring two self-attention layers to function.
is worth noticing that adding more unlabeled samples into the support set makes our model more difficult to train and it gets stuck producing CNNs with essentially random outputs. Our solution was introduce unlabeled samples incrementally during training. This was implemented by masking out some unlabeled samples in the beginning of the training and then gradually reducing the masking probability over time.
4.4 GENERATING MORE MODEL LAYERS
We demonstrated that HTT model can outperform MAML++ on common few-shot learning datasets by generating just the last logits layer of the CNN model. But under what conditions can it be advantageous to generate additional CNN layers? As we show here, generating all CNN layers with multi-layer transformer can lead to significant performance improvement on CNN models below a particular size.
Table 3: Test accuracy on TIEREDIMAGENET supervised 1-shot and 5-shot models and semi- supervised 1-shot models with additional unlabeled samples per class. The weight generation transformer model uses Lr encoder layers. Notice a performance improvement of semi-supervised learning over the 1-shot supervised results. Accuracy is seen to grow with the number of unlabeled samples and the maximum accuracy is reached when the encoder has at least two layers.
(u. Lr)
I-shot
5-shot
(2,3) 3)
(4.
(4.
(4,3) 3)
(9.3)
Accuracy
56.0
69.9
58.3
56.6
59.9
59.9
61.5
8

---

Under review as a conference paper at ICLR 2022
100
8-Channel
95
Training Accuracies
96
90
lest accuracy
6-channel
85-
channell
loght loghs (test)
75-
all
all
70
logits logits
(test) oracle
Training steps (million)
Number Channals
Figure 3: (Left) Test accuracies for the generated 4., 6- and 8-channel CNN models on the 5-shot- 20-way OMNIGLOT task. Models with only the last logits layer generated (red) are characterized by lower test accuracies compared to the models with some or all convolutional layers also being generated (blue, green). Similar plot for TIFREDIMAGENET can be found in Appendix (Fig. [3]. (Right) 5-shot-20-way OMNIGLOT training/test accuracies as function of the CNN model com- plexity: only the final logits layer being generated (logits), all layers being generated (all), training the model on all available samples for a random set of few classes (oracle). A model that generates CNN weights by memorizing all samples (being able to determine their classes) and also memoriz- ing optimal trained weights for any selection of classes would reach the oracle accuracy, but would not generalize.
We demonstrated this by conducting experiments, in which some layers generated and some layers were learned (usually the first few layers of the CNN). For OMNIGLOT dataset, we saw that both training and test accuracies for a 4-channel and a 6-channel CNNs increased with the number of generated layers (see Fig. and Table in Appendix) and using more complex transformer models with 2 or more encoder layers improved both training and test accuracies of fully-generated CNN models of this size (see Appendix DD. However, as the size of the model increased and reached 8 channels, generating the last logits layer alone proved to be sufficient for getting the best results on OMNIGLOT and TIEREDIMAGENET.
The positive effect of generating convolutional layers can also be observed shallow models with large convolutional kernels with large strides where the model performance can be much more sensi- tive to a proper choice model weights. For example, a 16-channel model with two convolutional kernels of size 9 and the stride of 4, the overall test accuracy for a model generating only the final convolutional layer was about 1% lower than the accuracies of the models generating at least one additional convolutional filter. We also speculate that as the complexity of the task increases, gener- ating some or all intermediate network layers should become more important for achieving optimal performance. Verifying this hypothesis and understanding the "boundary" in the model space be- tween two regimes where a static backbone is sufficient, or no longer sufficient will be the subject of our future work.
5 CONCLUSIONS
In this work, we proposed a HlsperTransformer (HT), a novel transformer-based model that gener- ates all weights of a CNN model directly from few-shot support set. This approach allows us to use a high-capacity model for encoding task-dependent variations in the weights of a smaller model. We demonstrate that generating the last logits layer alone, the transformer-basee or matches performance of multiple traditional learning methods on several few-shot benchmarks. More importantly, we showed that HT can be straightforwardly extended to handle unlabeled sam- ples that might be present in the support set and our experiments demonstrate considerable few- shot performance improvement in the presence of unlabeled data. Finally, we explore the impact of the sunsformer-encodder model diversity in CNN models of different sizes. We use HT to generate some or all convolutional kernels and biases and show that for sufficiently small models, adjusting all model parameters further improves their fho-shot learning performance.
weight generator beats

---

Under review as a conference paper at ICLR 2022
REFERENCES
Kelsey R. Allen, Evan Shelhamer, Hanul Shin, and Joshua B. Tenenbaum. Infinite mixture proto- types for few-shot learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceed- ings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 232-241. PMLR, 2019.
Antreas Antoniou, Harrison Edwards, and Amos J. Storkey. How to train your MAML. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.). Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part volume 12346 of Lecture Notes in Computer Science, pp. 213-229. Springer, 2020.
Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chun- jing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In IEEE Conference on Computer Vision and Putlern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 12299- 12310. Computer Vision Foundation/ / IEEE, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Nanyi Fei, Zhiwu Lu, Tao Xiang, and Songfang Huang. MELR: meta-learning via modeling episode-level relationships for few-shot learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1126-1135, PMLR, 06-11 Aug 2017.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting, In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp- 4367-4375. IEEE Computer Society, 2018. doi: 10. 109//R.. 2018.00459.
David Ha, Andrew M. Dai, and Quoc Le. HyperNetworks. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017
Fusheng Hao, Fengxiang He, Jun Cheng, Lei Wang, Jianzhong Cao, and Dacheng Tao. Collect and select: Semantic alignment metric learning for few-shot learning. In 2019 IEEE/CVF In- ternational Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 8459-8468. IEEE, 2019. doi: 10. 10.1109/CCV2209.000
Muhammad Abdullah Jamal and Guo-Jun Qi. Task agnostic meta-learning for few-shot learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 11719-11727. Computer Vision Foundation / IEEE, 2019. doi: 10. 10.1109/CVPR.2019.000 199.
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2. Lille, 2015.
Aoxue Li, Tiange Luo, Tao Xiang, Weiran Huang, and Liwei Wang. Few-shot learning with global class representations. In 2019 IEEE/CVF International Conference Computer Vision, ICCV 2019, Seoul. Korea (South), October 27 November 2, 2019, pp. 9714-9723. IEEE, 2019a. doi: 10.1 119//CCC.22199000981
10

---

Under review as a conference paper at ICLR 2022
Huai- Yu Li, Weiming Dong, Xing Mei, Chongyang Ma, Feiyue Huang, and Bao-Gang Gang Hu. Lgm- net: Learning to generate matching networks for sho-shot learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 3825-3834 PMLR, 2019b.
Lu Liu, William L. Hamilton, Guodong Long, Jing Jiang, and Hugo Larochelle. A universal repre- sentation transformer layer for few-shot image classification. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Chongyang Ma. LGM-Net. httppa///gittub.comm//
wetkeesswellll www.e//ee...welLL--eeeeeeee 2019.
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetoors. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the Ith International Joint Conference on Natural Language Processing, ACL/JCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 565-576, Association for Computational Linguistics, 2021. doi: 10.18.53//1/222. acl-long.47.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. CoRR, abs/1803.02999, 2018
Boris N. Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. TADAM:
task dependent adaptive metric for improved few-shot learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Informa- tion Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 719-729, 2018.
Zhimao Peng. Zechao Li, Junge Zhang, Yan Li, Guo-Jun Qi, and Jinhui Tang. Few-shot image recognition with knowledge transfer. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27- November 2, 2019, pp. 441-449. IEEE, 2019. doi: 10. 1,.1109/CC..2219.00000 .00053..
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L. Yuille. Few-shot image recognition by predicting parameters from activations. In 2018 IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. pp. 7229-7238. Computer Vision Foundation IEEE Computer Society, 2018. doi: 10. 10.11000 (1199//VV..00..000000
Neale Ratzlaff and Fuxin Li. Hypergan: A generative model for diverse, performant neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML, 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 5361-5369 PMLR, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In 5th Interna- tional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-leaminng with latent embedding optimization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4077-4087 2017.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang. Philip H. S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. In 2018 IEÉE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City. UT, USA. June 18-22, 2018, pp. 1199-1208. IEEE Computer Society, 2018. doi: 10009//V...008.000.111

---

Under review as a conference paper at ICLR 2022
Yi Tay. Zhe Zhao, Dara Bahri, Donald Metzler, and Da- Da-Cheng Juan. Hypergrid transformers: To- wards A single model for multiple tasks. In 9th International Conference on Learning Represen- tations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola. Rethinking few- shot image classification: good embedding is all you need? In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.). Computer Vision ECCV 2020 16th European Conference, Glasgow, UK. August 23-28. 2020. Proceedings, Part XIV, volume 12359 of Lecture Notes in Computer Science, pp. 266- Springer, 2020. doi: 10. 00.1007/978-3-030-555566
16. Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Ma- rina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. ICML 2021. 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 10347-10357. PMLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp- 2017.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 3630-3638, 2016.
Ziyang Wu, Yuwei Li, Lihua Guo, and Kui Jia. PARN: position- relation networks for few-shot learning. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 -November 2, 2019. pp. 6658-6666. IEEE, 2019, doi: 10.1 109//V// 2019.00676.
Fuzhi Yang, Huan Yang. Jianlong Fu, Hongtao Lu, and Baining Guo. Learning texture transformer In 2020 IEEE/CVF Conference on Computer Vision and
network for image Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. pp. 5790-5799. Computer Vision Foundation / IEEE, 2020. doi: 10.1 109/CVPR42000022000000 20300,003...
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adapta- tion with set-to-set functions. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 8805-8814, IEEE, 2020. doi: D0.100//CVPR42600000000 00883.
Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self-attention network for referring image segmentation. In IEEE Conference on Computer Vision and Puttern Recogni- fion, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 10502-10511. Computer Vision Foundation IEEE, 2019, doi: 10. (09//CVP.2019.000777
Qinyuan Ye and Xiang Ren. Learning to generate task-specific adapters from task description. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the I 1ith International Joint Conference on Natural Language Processing, ACIJJJNNLP 2021, (Volume 2: Short Papers), Virtual Event, August 1-6, 2021, pp. 646-653, Association for Computational Linguistics, 2021. doi: 1/28655/1/22221.acc-hhoo
Dominic Zhao, Johannes von Oswald, Seijin Kobayashi, João Sacramento, and Benjamin F Grewe. Meta-learning via hypernetworks. 2020.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: deformable transformers for end-to-end object detection. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
12


Reviews:
[{"review_id": "rkzheQhPwxg", "reviewer": "Reviewer_As45", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\n– The overall idea of the paper to increase the learning capacity looks interesting. \n\n\nWeaknesses:\n\n– I think the novelty of the current version of the paper is incremental, and adding transformer layers between Conv4 would likely increase the Conv4’s capacity due to the over-parameterization. \n\n– While respecting the effort of the author(s) on evaluation of their idea using tieredImageNet, I think we need some additional large datasets instead of Omniglot or miniImagaNet in the case of measuring the effectiveness of a model with higher capacity. Here, deep backbones (such as ResNet) are required in the evaluations. Of course, we should also consider an appropriate regularization technique while increasing the model capacity of deeper backbones.  \n\n– The presentation of the paper is not clear in some parts of the paper. For example, there are some grammatical errors in the paper (e.g. opening sentence of section 3). I also had problems with understanding some of the statements. For example, the first sentence of the second paragraph in section 1 “, relies on pretraining to a low-dimensional embedding...“. We know that some metric-based methods (like ProtoNet) can be applied to the flatten output of ResNet12 (resulting in high dimensionality) and still work well.   ", "summary_of_the_paper": "This work presents the use of a hybrid CNN-Transformer model for few-shot image classification. Specifically, the paper applies encoding and encoding-decoding transformers between convolution layers to increase the learning capacity.  The paper also includes a Conv-based hybrid model and uses Omniglot, miniImageNet, and tieredImageNet for evaluations. ", "summary_of_the_review": "I think the current version of the paper needs to be strengthened from novelty, evaluation, and presentation perspectives.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ozyCYG4Annx", "reviewer": "Reviewer_1KF1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. It is quite interesting to see a self-attention computational circuit can be interpreted as applying a single gradient descent step with the cross-entropy loss.\n2. I hate to say, but experimental results are far from state-of-the-art results. why not experimenting with larger CNN and transformers networks?\n3. the replacement of existing CNNs with transformers might not be a meaningful contribution to the community.\n4. Which encoder architecture did you use eventually? (a) or (b)? I personally prefer to focus on the final architecture you used, and focus on arguing more about it.\n5. Any comparisons against other ‘weight generating’ approaches?\n\n[minor comments]\n- Fonts in the Figure 1 and 2 are too small, hard to see\n- Could you explain about ‘placeholders’ a bit in detail? I think I get the idea, but it's good to be wordy for a broader audience.\n", "summary_of_the_paper": "This paper suggests HyperTransformer, a transformer-based model that produces weights of CNN models in meta-learning setup. Experimental results show that the proposed approach improved the performance of CNN models below a certain size in few-shot classification and semi-supervised few shot classification tasks.\n", "summary_of_the_review": "This paper shows that the recent transformer architecture works for meta-learning setup as a hypernetwork. Although I believe it is a good information, but I am not fully convinced that this has enough contribution to be a full ICLR paper. Experimental results shows some improvements, but it is not convincing enough so that researchers will follow up on this work later. Therefore, I tend to reject this submission.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Qze2W2W-Kvw", "reviewer": "Reviewer_2iqR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I think the method is fairly interesting and a useful contribution to the \"small model\" few-shot image classification problem. I think the main weakness of the problem is that peformance seems fairly comparable to existing methods if one is willing to use a \"test time\" model of any model complexity. While there are several \"meta network\" approaches that directly generate the weights of another, simpler network, I think there are a few useful/generalizable insights that the paper presents that will be of use to other methods; in particular, the two-stage transformer visualized in Fig 1. The separation into a high-capacity model for parsing the support set, and a low-capacity model for executing, is a natural one for this problem. However in most practical cases I have worked with the cost of a larger backbone network is fairly minimal on modern hardware, it's still nice to have smaller models where possible and the paper demontrates that this can be achieved with comparable accuracy.\n\nWhen the paper initially claims \"the transformer model is invariant to sample permutations\" I was slightly confused as this depends on how exactly the transformers are used (in NLP the token order is certainly relevant.)  Some earlier clarity about how the paper intends to use transformers would help here.\n\nReasserting early on that this is focused entirely on few-shot image classification will also help here, there are plenty of other relevant few shot problems (value regression, image regression/translation, document classification, and so on.)  The paper doesn't clarify this at all in the abstract or early part of the intro.\n\nI am familiar with this general topic but as it is not my primary area of research, I may have missed some prior work approach here.", "summary_of_the_paper": "The paper proposes a method for solving few-shot image classification that generates all the weights of a very small CNN model. This has the advantage that the generated model can be very small+compact, compared against for example some embedding methods that might require large image classification networks to run as a pre-process. The method is also able to handle unlabeled samples in a fairly natural way.", "summary_of_the_review": "\nThe paper shows an interesting way to apply \"high capacity\" transformers to generate the weights of a low-capacity CNN model for few-shot image classification. While results are comparable to SOTA high-capacity classifier networks, the meta-network generation is interesting and able to produce very low-capacity classifiers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper uses a transformer model to generate CNN models and use it for few shot learning.

Although the reviewers appreciate the ideas and the good benchmarking results presented in the paper they are find the paper somewhat incremental compared to previous work in the hyper network literature. This also despite the authors thorough rebuttal with additional results. This shows that the authors could have done a better job in presenting their work.

Rejection is therefore recommended with a strong encouragement to rework the paper to counter future reviewers having similar reservations.
Decision: Reject