Hierarchical Variational Memory for Few-shot Learning Across Domains

1 INTRODUCTION . Few-shot learning with an external memory is known to learn new concepts quickly with only a few samples , especially when embedded in a meta-learning setting ( Santoro et al. , 2016 ) . A common tactic is to store short-term memory ( Munkhdalai & Yu , 2017 ; Munkhdalai et al. , 2018 ; Kaiser et al. , 2017 ) as obtained from the support set of the current task , and to empty it at the end of a task . Another tactic is to let the memory store long-term knowledge distilled from all the training tasks ( Zhen et al. , 2020a ) , which provides a conceptual context to learn a new task . The long-term memory has demonstrated effectiveness in enabling a model to quickly learn new few-shot learning tasks within domains ( Zhen et al. , 2020a ) . Nonetheless , for both the short- and long-term memory tactics , the learning ability is limited when facing a task from an unseen domain ( Tseng et al. , 2020 ; Guo et al. , 2020 ; Cai & Shen , 2020 ; Du et al. , 2021 ) . Cross-domain few-shot learning ( Tseng et al. , 2020 ; Guo et al. , 2020 ) is even more challenging than the conventional few-shot learning problem as it also has to take distribution shift into account . In cross-domain few-shot learning , directly using traditional external memory , which often only stores high-level semantic features ( Zhen et al. , 2020a ) , is unable to find the suitable semantics in memory due to the domain shift causing side effects . Nonetheless , low-level features like textures , shapes , edges , etc . from training domains are often still meaningful for a new domain ( Adler et al. , 2020 ) . We hypothesize , features at different levels would play distinctive roles in generalizing across domains . It remains unexplored to investigate hierarchical features in memory that can enable quick adaptation , as humans do , when faced with a few samples from new domains . In this paper , we make three contributions . First , we propose hierarchical variational memory , which accrues and stores the long-term semantic knowledge of multiple network layers from past experiences . Each entry stores features at different hierarchical levels by summarizing the feature representations of class samples , thus generalizing well in presence of a domain shift between the training and test distributions . We formulate the memory recall at different hierarchical levels as a variational inference of the latent memory , which is an intermediate stochastic variable . Second , to better deploy the hierarchical memory , we introduce a hierarchical variational prototype model to obtain prototypes at different levels as well . We also develop the hierarchical variational prototype in a latent model by treating the prototypes at different levels as the latent variable . In doing so , the model is endowed with the ability to flexibly rely on low-level textures or high-level semantic features if the domain shift circumstances so demand . As the third contribution , we propose to learn the weights associated with the prototype at each level in a data-driven way , which can explore and exploit the importance of the different feature levels for domain generalization . We leverage the few training samples in the new domain to generate reasonable weights for each prototype , since these samples already carry sufficient appearance cues about the new domain , which enables the model to adaptively choose the most generalizable features . Two different hierarchical variational inference problems are included in our optimization objective : 1 ) the inference of the latent memory , which conditions on the latent memory of previous layers and the memory contents of the current layer ; 2 ) the inference of prototypes , which treats the support set , the latent memory of the current layer and the prototype of previous layers as conditions . We demonstrate the effectiveness of the proposed hierarchical memory by extensive experiments on the cross-domain few-shot learning classification tasks introduced by ( Guo et al. , 2020 ) and the more traditional within-domain few-shot learning classification tasks ( Snell et al. , 2017 ) . We conduct an extensive ablation study to demonstrate the contributions of different components in our model . We also analyze how hierarchical memory adaptively chooses the weights in the face of domain change and identifies the appropriate layer of the prototype that should be used . An extensive evaluation of cross-domain few-shot and traditional few-shot classification benchmarks reveals that our hierarchical memory consistently achieves results that are better , or at least competitive , compared to other methods . 2 METHODOLOGY . 2.1 PRELIMINARIES . In the following , we present the relevant background on few-shot classification and its cross-domain setting , as well as the preliminaries of the ( variational ) prototypical network and variational semantic memory ( Zhen et al. , 2020a ) . Few-shot classification In the few-shot classification scenario , we define the N -way K-shot classification problem , which is comprised of the support sets S and query set Q . Each task also called an episode , is a classification problem sampled from a task distribution p ( T ) . The ‘ way ’ of the episode refers to the number of classes in the support , while the ‘ shot ’ of the episode refers to the number of examples per class . Tasks are drawn from a dataset by randomly sampling a subset of classes , sampling points from these classes , and then partitioning the points into support and query sets . Episodic optimization ( Vinyals et al. , 2016 ) iteratively trains the model by taking one episode update at a time . Cross-domain setting For few-shot classification , the test tasks are typically assumed to come from the same task distribution p ( T ) as the training tasks . The recently introduced task of cross-domain few-shot learning ( Guo et al. , 2020 ) considers the few-shot classification challenge under domain shift . Concretely , the training task comes from a single source domain p ( T0 ) , while the test tasks come from several unseen domains { T1 , · · · , TM } . The goal of cross-domain few-shot learning is to learn a meta-learning model using a single source domain that generalizes to several unseen domains . Prototypical network Our model builds upon the prototypical network ( Snell et al. , 2017 ) , which is widely used for few-shot image classification . It computes a prototype zk= 1K ∑ k fθ ( xk ) for each class through an embedding function fθ , which is realized by neural networks . It computes a distribution over classes for a query sample x given a distance function d ( · , · ) as the softmax over its distances to the prototypes in the embedding space : p ( yi = k|x ) = exp ( −d ( fθ ( x ) , zk ) ) ∑ k′ exp ( −d ( fθ ( x ) , zk′ ) ) ( 1 ) Variational prototype network The variational prototype network ( Zhen et al. , 2020a ) is a powerful model for learning latent representations from small amounts of data , where the prototype z per class is treated as a distribution . Given a task with a support set S and query set Q , the ELBO takes the following form : log [ |Q|∏ i=1 p ( yi|xi ) ] ≥ |Q|∑ i=1 [ Eq ( z|S ) [ log p ( yi|xi , z ) ] −DKL ( q ( z|S ) ||p ( z|xi ) ) ] , ( 2 ) where q ( z|S ) is the variational posterior over z , p ( z|xi ) is the prior , and Lz is the number of Monte Carlo samples for z. Variational semantic memory Variational semantic memory ( Zhen et al. , 2020a ) is proposed to accumulate and store the semantic information from previous tasks for the inference of prototypes of new tasks . It consists of two main processes : memory recall , which retrieves relevant information that fits with specific tasks based on the support set of the current task ; memory update , which effectively collects new information from the task and gradually consolidates the semantic knowledge in the memory . Variational semantic memory formulates the memory recall as a variational inference of the latent memory , which is an intermediate stochastic variable . The objective is written as follows : LVSM = |Q|∑ i=1 [ − Eq ( z|S , m ) [ log p ( yi|xi , z ) ] +DKL [ q ( z|S , m ) ||p ( z|xi ) ] +DKL [ |M |∑ i γip ( m|Mi ) ||p ( m|S ) ] ] , ( 3 ) where p ( m|S ) is the introduced prior over latent memory m and M denotes the memory content . The above networks and memory only focus on the problem of few-shot classification within domain and their performance would be degrade for cross domain few-shot learning , because of the domain shift problem . In this paper , we propose hierarchical memory to address few-shot classification under domain shift . 2.2 HIERARCHICAL VARIATIONAL PROTOTYPE . In the variational prototype network ( Zhen et al. , 2020a ) , the probabilistic prototype is obtained by feeding the last layer of features to an amortization network . Hence , it is not possible to obtain the prototypes of different layers when addressing the domain shift problem , making it relying on the high-level prototype only , which is not necessarily shared between domains . We introduce the hierarchical variational prototype network to generate different layer prototypes . Compared to its flat counterpart , it introduces q ( zl|zl−1 , S ) instead of q ( z|S ) , so we approximate the true posterior by minimizing the KL divergence : DKL [ q ( z l|zl−1 , S ) ||p ( zl|x , y ) ] , ( 4 ) where zl is the prototype of layer l. By applying the Baye ’ s rule , we then obtain the following ELBO : log [ |Q|∏ i=1 p ( yi|xi ) ] ≥ |Q|∑ i=1 [ [ 1 L L∑ l=1 Eq ( zl|S ) [ log p ( yli|xi , zl ) ] −DKL ( q ( zl|zl−1 , S ) ||p ( zl|zl−1 , xi ) ) ] . ( 5 ) In practice , the variational posterior q ( zl|zl−1 , S ) is implemented by an amortization network ( Kingma & Welling , 2013 ) that takes the concatenation of the average feature representations of samples in the support set S and the upper layer prototype zl−1 and returns the mean and variance of the current layer prototype zl . The hierarchical probabilistic prototype provides both a more informative representation than the deterministic prototype and the ability to capture different representation levels , making it more suitable for cross-domain few-shot learning . More importantly , the hierarchical variational prototype also provides a principled way of incorporating the prior knowledge of the different levels from the past experienced tasks . Therefore , we introduce the external hierarchical memory to enhance the prototype at different levels . observed during meta-training and meta-test observed during meta-training only latent variable hierarchical memory Figure 1 : Computational graph of hierarchical variational memory for prototypical few-shot learning . The memory consists of L layers of the hierarchy : each layer M l is used to build up the corresponding latent prototype zl . ST and Qt are the support set and the query set . x and y are the sample and label . The latent memory ml is inferred from the memory content M l and the upper latent memory ml−1 , and the latent prototype zl is constructed from the support set S , latent memory ml , and the latent prototype zl−1 in the upper layer .

Summary: This paper presents a hierarchical version of the variational memory approach of [Zhen et al., 2020] for few-shot learning.  The core technical methodology follows that developed by Zhen et al., with the difference being in terms of the model: this work utilizes a deep model with per-layer memory, whereas Zhen et al. utilize a single memory for high-level concepts (i.e., deepest layer only).  Motivating the choice of per-layer memory is the desire to better handle few-shot learning tasks with domain shift, as the representations in earlier layers may be more relevant in such scenarios; the proposed framework learns a hypernetwork to predict attention weights over the per-layer prototypes.