Hierarchical Variational Memory for Few-shot Learning Across Domains

1 INTRODUCTION . Few-shot learning with an external memory is known to learn new concepts quickly with only a few samples , especially when embedded in a meta-learning setting ( Santoro et al. , 2016 ) . A common tactic is to store short-term memory ( Munkhdalai & Yu , 2017 ; Munkhdalai et al. , 2018 ; Kaiser et al. , 2017 ) as obtained from the support set of the current task , and to empty it at the end of a task . Another tactic is to let the memory store long-term knowledge distilled from all the training tasks ( Zhen et al. , 2020a ) , which provides a conceptual context to learn a new task . The long-term memory has demonstrated effectiveness in enabling a model to quickly learn new few-shot learning tasks within domains ( Zhen et al. , 2020a ) . Nonetheless , for both the short- and long-term memory tactics , the learning ability is limited when facing a task from an unseen domain ( Tseng et al. , 2020 ; Guo et al. , 2020 ; Cai & Shen , 2020 ; Du et al. , 2021 ) . Cross-domain few-shot learning ( Tseng et al. , 2020 ; Guo et al. , 2020 ) is even more challenging than the conventional few-shot learning problem as it also has to take distribution shift into account . In cross-domain few-shot learning , directly using traditional external memory , which often only stores high-level semantic features ( Zhen et al. , 2020a ) , is unable to find the suitable semantics in memory due to the domain shift causing side effects . Nonetheless , low-level features like textures , shapes , edges , etc . from training domains are often still meaningful for a new domain ( Adler et al. , 2020 ) . We hypothesize , features at different levels would play distinctive roles in generalizing across domains . It remains unexplored to investigate hierarchical features in memory that can enable quick adaptation , as humans do , when faced with a few samples from new domains . In this paper , we make three contributions . First , we propose hierarchical variational memory , which accrues and stores the long-term semantic knowledge of multiple network layers from past experiences . Each entry stores features at different hierarchical levels by summarizing the feature representations of class samples , thus generalizing well in presence of a domain shift between the training and test distributions . We formulate the memory recall at different hierarchical levels as a variational inference of the latent memory , which is an intermediate stochastic variable . Second , to better deploy the hierarchical memory , we introduce a hierarchical variational prototype model to obtain prototypes at different levels as well . We also develop the hierarchical variational prototype in a latent model by treating the prototypes at different levels as the latent variable . In doing so , the model is endowed with the ability to flexibly rely on low-level textures or high-level semantic features if the domain shift circumstances so demand . As the third contribution , we propose to learn the weights associated with the prototype at each level in a data-driven way , which can explore and exploit the importance of the different feature levels for domain generalization . We leverage the few training samples in the new domain to generate reasonable weights for each prototype , since these samples already carry sufficient appearance cues about the new domain , which enables the model to adaptively choose the most generalizable features . Two different hierarchical variational inference problems are included in our optimization objective : 1 ) the inference of the latent memory , which conditions on the latent memory of previous layers and the memory contents of the current layer ; 2 ) the inference of prototypes , which treats the support set , the latent memory of the current layer and the prototype of previous layers as conditions . We demonstrate the effectiveness of the proposed hierarchical memory by extensive experiments on the cross-domain few-shot learning classification tasks introduced by ( Guo et al. , 2020 ) and the more traditional within-domain few-shot learning classification tasks ( Snell et al. , 2017 ) . We conduct an extensive ablation study to demonstrate the contributions of different components in our model . We also analyze how hierarchical memory adaptively chooses the weights in the face of domain change and identifies the appropriate layer of the prototype that should be used . An extensive evaluation of cross-domain few-shot and traditional few-shot classification benchmarks reveals that our hierarchical memory consistently achieves results that are better , or at least competitive , compared to other methods . 2 METHODOLOGY . 2.1 PRELIMINARIES . In the following , we present the relevant background on few-shot classification and its cross-domain setting , as well as the preliminaries of the ( variational ) prototypical network and variational semantic memory ( Zhen et al. , 2020a ) . Few-shot classification In the few-shot classification scenario , we define the N -way K-shot classification problem , which is comprised of the support sets S and query set Q . Each task also called an episode , is a classification problem sampled from a task distribution p ( T ) . The ‘ way ’ of the episode refers to the number of classes in the support , while the ‘ shot ’ of the episode refers to the number of examples per class . Tasks are drawn from a dataset by randomly sampling a subset of classes , sampling points from these classes , and then partitioning the points into support and query sets . Episodic optimization ( Vinyals et al. , 2016 ) iteratively trains the model by taking one episode update at a time . Cross-domain setting For few-shot classification , the test tasks are typically assumed to come from the same task distribution p ( T ) as the training tasks . The recently introduced task of cross-domain few-shot learning ( Guo et al. , 2020 ) considers the few-shot classification challenge under domain shift . Concretely , the training task comes from a single source domain p ( T0 ) , while the test tasks come from several unseen domains { T1 , · · · , TM } . The goal of cross-domain few-shot learning is to learn a meta-learning model using a single source domain that generalizes to several unseen domains . Prototypical network Our model builds upon the prototypical network ( Snell et al. , 2017 ) , which is widely used for few-shot image classification . It computes a prototype zk= 1K ∑ k fθ ( xk ) for each class through an embedding function fθ , which is realized by neural networks . It computes a distribution over classes for a query sample x given a distance function d ( · , · ) as the softmax over its distances to the prototypes in the embedding space : p ( yi = k|x ) = exp ( −d ( fθ ( x ) , zk ) ) ∑ k′ exp ( −d ( fθ ( x ) , zk′ ) ) ( 1 ) Variational prototype network The variational prototype network ( Zhen et al. , 2020a ) is a powerful model for learning latent representations from small amounts of data , where the prototype z per class is treated as a distribution . Given a task with a support set S and query set Q , the ELBO takes the following form : log [ |Q|∏ i=1 p ( yi|xi ) ] ≥ |Q|∑ i=1 [ Eq ( z|S ) [ log p ( yi|xi , z ) ] −DKL ( q ( z|S ) ||p ( z|xi ) ) ] , ( 2 ) where q ( z|S ) is the variational posterior over z , p ( z|xi ) is the prior , and Lz is the number of Monte Carlo samples for z. Variational semantic memory Variational semantic memory ( Zhen et al. , 2020a ) is proposed to accumulate and store the semantic information from previous tasks for the inference of prototypes of new tasks . It consists of two main processes : memory recall , which retrieves relevant information that fits with specific tasks based on the support set of the current task ; memory update , which effectively collects new information from the task and gradually consolidates the semantic knowledge in the memory . Variational semantic memory formulates the memory recall as a variational inference of the latent memory , which is an intermediate stochastic variable . The objective is written as follows : LVSM = |Q|∑ i=1 [ − Eq ( z|S , m ) [ log p ( yi|xi , z ) ] +DKL [ q ( z|S , m ) ||p ( z|xi ) ] +DKL [ |M |∑ i γip ( m|Mi ) ||p ( m|S ) ] ] , ( 3 ) where p ( m|S ) is the introduced prior over latent memory m and M denotes the memory content . The above networks and memory only focus on the problem of few-shot classification within domain and their performance would be degrade for cross domain few-shot learning , because of the domain shift problem . In this paper , we propose hierarchical memory to address few-shot classification under domain shift . 2.2 HIERARCHICAL VARIATIONAL PROTOTYPE . In the variational prototype network ( Zhen et al. , 2020a ) , the probabilistic prototype is obtained by feeding the last layer of features to an amortization network . Hence , it is not possible to obtain the prototypes of different layers when addressing the domain shift problem , making it relying on the high-level prototype only , which is not necessarily shared between domains . We introduce the hierarchical variational prototype network to generate different layer prototypes . Compared to its flat counterpart , it introduces q ( zl|zl−1 , S ) instead of q ( z|S ) , so we approximate the true posterior by minimizing the KL divergence : DKL [ q ( z l|zl−1 , S ) ||p ( zl|x , y ) ] , ( 4 ) where zl is the prototype of layer l. By applying the Baye ’ s rule , we then obtain the following ELBO : log [ |Q|∏ i=1 p ( yi|xi ) ] ≥ |Q|∑ i=1 [ [ 1 L L∑ l=1 Eq ( zl|S ) [ log p ( yli|xi , zl ) ] −DKL ( q ( zl|zl−1 , S ) ||p ( zl|zl−1 , xi ) ) ] . ( 5 ) In practice , the variational posterior q ( zl|zl−1 , S ) is implemented by an amortization network ( Kingma & Welling , 2013 ) that takes the concatenation of the average feature representations of samples in the support set S and the upper layer prototype zl−1 and returns the mean and variance of the current layer prototype zl . The hierarchical probabilistic prototype provides both a more informative representation than the deterministic prototype and the ability to capture different representation levels , making it more suitable for cross-domain few-shot learning . More importantly , the hierarchical variational prototype also provides a principled way of incorporating the prior knowledge of the different levels from the past experienced tasks . Therefore , we introduce the external hierarchical memory to enhance the prototype at different levels . observed during meta-training and meta-test observed during meta-training only latent variable hierarchical memory Figure 1 : Computational graph of hierarchical variational memory for prototypical few-shot learning . The memory consists of L layers of the hierarchy : each layer M l is used to build up the corresponding latent prototype zl . ST and Qt are the support set and the query set . x and y are the sample and label . The latent memory ml is inferred from the memory content M l and the upper latent memory ml−1 , and the latent prototype zl is constructed from the support set S , latent memory ml , and the latent prototype zl−1 in the upper layer .

Summary: This paper proposes a hierarchical memory to store features at different semantic levels for few-shot learning across domains. It introduces a hierarchical prototype model, where each level of the prototypes fetches the corresponding information from the hierarchical memory.  The authors follow the hyper network design to learn the weights when combining predictions from multiple levels.  The overall model is optimized using a variational inference framework.   The proposed method is evaluated on 4 various datasets which have a domain gap between the training data.  The authors also show that this method is competitive on the commonly used few-shot image classification benchmarks.