HyperTransformer: Attention-Based CNN Model Generation from Few Samples

1 INTRODUCTION . In few-shot learning , a conventional machine learning paradigm of fitting a parametric model to training data is taken to a limit of extreme data scarcity where entire categories are introduced with just one or few examples . A generic approach to solving this problem uses training data to identify parameters φ of a learner aφ that given a small batch of examples for a particular task ( called a support set ) can solve this task on unseen data ( called a query set ) . One broad family of few-shot image classification methods frequently referred to as metric-based learning , relies on pretraining an embedding eφ ( · ) and then using some distance in the embedding space to label query samples based on their closeness to known labeled support samples . These methods proved effective on numerous benchmarks ( see Tian et al . ( 2020 ) for review and references ) , however the capabilities of the learner are limited by the capacity of the architecture itself , as these methods try to build a universal embedding function . On the other hand , optimization-based methods such as seminal MAML algorithm ( Finn et al. , 2017 ) can fine-tune the embedding eφ by performing additional SGD updates on all parameters φ of the model producing it . This partially addresses the constraints of metric-based methods by learning a new embedding for each new task . However , in many of these methods , all the knowledge extracted during training on different tasks and describing the learner aφ still has to “ fit ” into the same number of parameters as the model itself . Such limitation becomes more severe as the target models get smaller , while the richness of the task set increases . In this paper we propose a new few-shot learning approach that allows us to decouple the complexity of the task space from the complexity of individual tasks . The main idea is to use the transformer model ( Vaswani et al. , 2017 ) that given a few-shot task episode generates an entire inference model by producing all model weights in a single pass . This allows us to encode the intricacies of the available training data inside the transformer model , while still producing specialized tiny models that can solve individual tasks . Reducing the size of the generated model and moving the computational overhead to the transformer-based weight generator , we can lower the cost of the inference on new images . This can dramatically reduce the overall computation cost in cases , where the tasks change infrequently and hence the weight generator is only used sporadically . We start by observing that the self-attention mechanism is well suited to be an underlying mechanism for a few-shot CNN weight generator . In contrast with earlier CNN- ( Zhao et al. , 2020 ) or BiLSTM-based approaches ( Ravi & Larochelle , 2017 ) , the vanilla1 transformer model is invariant to sample permutations and can handle unbalanced datasets with a varying number of samples per category . Furthermore , we demonstrate that a single-layer self-attention model can replicate a simplified gradient-descent-based learning algorithm . Using a transformer-based model to generate the logits layer on top of a conventionally learned embedding , we achieve competitive results on several common few-shot learning benchmarks . Varying transformer parameters we demonstrate that this high performance can be attributed to additional capacity of the transformer model that decouples its complexity from that of the generated CNN . We then extend our method to support unlabeled samples by using a special input token concatenated to unlabeled samples to encode unknown labels . In our experiments , we observe that using transformers with two or more layers , we achieve better performance by adding unlabeled data into the support set . We explain our results in Section 4.3 where we show that such a transformer with at least two layers can encode the nearest-neighbor style algorithm that associates unlabeled samples with similarly labeled examples . In essence , by training the weight generator to produce CNN models with best possible performance on a query set , we teach the transformer to utilize unlabeled samples without having to manually introduce additional optimization objectives . We also show that by generating all layers of the CNN model we can improve both the training and the test accuracies of CNN models below a certain size . The training accuracy can be viewed as a capability of the generated CNN model to adapt to tasks seen at training time , whereas the test accuracy computed on unseen categories characterizes the generalization capability of this model adaptation mechanism . We empirically demonstrate the expected increase of the model training and test accuracies with the increase of the layer size and the number of generated layers ( see Figure 3 ) . Interestingly , generation of the logits layer alone appears to be “ sufficient ” above a certain model size threshold . This threshold is expected to depend on the variability and the complexity of the training tasks . We conjecture that this might reflect the fact that models are sufficiently expressive for the benchmarks we considered , or that additional regularization is needed to prevent overfitting of the meta-learning models . Finally , in addition to being able to decouple the complexity of the task distribution from the complexity of individual tasks , another important advantage of our method is that it allows to do learning end-to-end without relying on complex nested gradients optimization and other meta-learning approaches , where the number of unrolls steps is large . The paper is structured as follows . In Section 2 , we discuss the few-shot learning problem setup and highlight related work . Section 3 introduces our approach , discusses the motivation for choosing an attention-based model and shows how our approach can be used to meta-learn semi-supervised learning algorithms . In Section 4 , we discuss our experimental results . Finally , in Section 5 , we provide concluding remarks . 2 PROBLEM SETUP AND RELATED WORK . 2.1 FEW-SHOT LEARNING . The main goal of a few-shot learning algorithm is to use a set of training tasks Ttrain for finding a learner aφ parameterized by φ that given new task domains can train to recognize novel classes using just a few samples per each class . The learner aφ can be thought of as a function that maps task description T = { ( xi , ci ) } ti=1 containing k labeled input samples { xi , ci } from n classes , to the weights θ = aφ ( T ) of a trained model f ( x ; θ ) . The parameters φ are meta-optimized to maximize the performance of the model f ( x ; aφ ( TS ) ) generated using a support set TS with x drawn from a query set TQ . Each task T = ( TS , TQ ) is randomly drawn from a space of training tasks Ttrain . Typically , TS and TQ are generated by first randomly choosing several distinct classes from the training set and then sampling examples without replacement from these classes to generate TS and TQ . In a classical “ n-way-k-shot ” setting , n is the number of classes randomly sampled in each episode , and k is the number of samples for each class in the support set TS . The quality of a particular few-shot learning algorithm is typically evaluated using a separate test space of tasks Ttest . By forming Ttest from novel classes unseen at training time , we can evaluate 1without attention masking or positional encodings generalization of different learners aφ . Best algorithms are expected to capture the structure present in the training set and to perform well on novel concepts . This structure may , for example , include certain properties of the distributions pc ( x ) with c being the class label , or the presence of particular discriminative ( or alternatively invariant ) features in the tasks from Ttrain . 2.2 RELATED WORK . Few-shot learning received a lot of attention from the deep learning community and while there are hundreds of few-shot learning methods , several common themes emerged in the past years . Here we outline several existing approaches , show how they relate to our method and discuss the prior work related to it . Metric-Based Learning . One family of approaches involves mapping input samples into an embedding space and then using some nearest neighbor algorithm that relies on the computation of distances from a query sample embedding to the embedding computed using support samples with known labels . The metric used to compute the distance can either be the same for all tasks , or can be task-dependent . This family of methods includes , for example , such methods as Siamese networks ( Koch et al. , 2015 ) , Matching Networks ( Vinyals et al. , 2016 ) , Prototypical Networks ( Snell et al. , 2017 ) , Relation Networks ( Sung et al. , 2018 ) and TADAM ( Oreshkin et al. , 2018 ) . It has recently been argued ( Tian et al. , 2020 ) that methods based on building a powerful sample representation can frequently outperform numerous other approaches including many optimization-based methods . However , such approaches essentially amount to the “ one-model solves all ” approach and thus require larger models than needed to solve individual tasks . Optimization-Based Learning . An alternative approach that can adapt the embedding to a new task is to incorporate optimization within the learning process . A variety of such methods are based on the approach called Model-Agnostic Meta-Learning , or MAML ( Finn et al. , 2017 ) . In MAML , θ = aφ is obtained by initializing a DNN at θ0 = φ and then performing one or more gradient descent updates on a classification loss function L , i.e. , computing θk+1 = θk − γ · ( ∂L/∂θ ) ( T ; θk ) . This approach was later refined ( Antoniou et al. , 2019 ) and built upon giving rise to Reptile ( Nichol et al. , 2018 ) , LEO ( Rusu et al. , 2019 ) and others . One limitation of various MAML-inspired methods is that the knowledge about the set of training tasks Ttrain is distilled into parameters φ that have the same dimensionality as the model parameters θ . Therefore , for a very lightweight model f ( x ; θ ) the capacity of the task-adaptation learner aφ is still limited by the size of θ . Methods that use parameterized preconditioners that otherwise do not impact the model f ( x ; θ ) can alleviate this issue , but as with MAML , such methods can be difficult to train ( Antoniou et al. , 2019 ) . Weight Modulation and Generation . The idea of using a task specification to directly generate or modulate model weights has been previously explored in the generalized supervised learning context ( Ratzlaff & Li , 2019 ) and in specific language models ( Mahabadi et al. , 2021 ; Tay et al. , 2021 ; Ye & Ren , 2021 ) . Some few-shot learning methods described above also employ this approach and use task-specific generation or modulation of the weights of the final classification model . For example , in LGM-Net ( Li et al. , 2019b ) the matching network approach is used to generate a few layers on top of a task-agnostic embedding . Another approach abbreviated as LEO ( Rusu et al. , 2019 ) utilized a similar weight generation method to generate initial model weights from the training dataset in a few-shot learning setting , much like what is proposed in this article . However , in ( Rusu et al. , 2019 ) , the generated weights were also refined using several SGD steps similar to how it is done in MAML . Here we explore a similar idea , but largely inspired by the HYPERNETWORK approach ( Ha et al. , 2017 ) , we instead propose to directly generate an entire task-specific CNN model . Unlike LEO , we do not rely on pre-computed embeddings for images and generate the model in a single step without additional SGD steps , which simplifies and stabilizes training . Transformers in Computer Vision and Few-Shot Learning . Transformer models ( Vaswani et al. , 2017 ) originally proposed for natural language understanding applications had since become a useful tool in practically every subfield of deep learning . In computer vision , transformers have recently seen an explosion of applications ranging from state-of-the-art image classification results ( Dosovitskiy et al. , 2021 ; Touvron et al. , 2021 ) to object detection ( Carion et al. , 2020 ; Zhu et al. , 2021 ) , segmentation ( Ye et al. , 2019 ) , image super-resolution ( Yang et al. , 2020 ) , image generation ( Chen et al. , 2021 ) and many others . There are also several notable applications in few-shot image classification . For example , in Liu et al . ( 2021 ) , the transformer model was used for generating universal representations in the multi-domain few-shot learning scenario . And closely related to our approach , in Ye et al . ( 2020 ) , the authors proposed to accomplish embedding adaptation with the help of transformer models . Unlike our method that generates an entire end-to-end image classification model , this approach applied a task-dependent perturbation to an embedding generated by an independent task-agnostic feature extractor . In ( Gidaris & Komodakis , 2018 ) , a simplified attention-based model was used for the final layer generation .

Summary: The paper proposes a method for solving few-shot image classification that generates all the weights of a very small CNN model. This has the advantage that the generated model can be very small+compact, compared against for example some embedding methods that might require large image classification networks to run as a pre-process. The method is also able to handle unlabeled samples in a fairly natural way.