Hierarchical Variational Memory for Few-shot Learning Across Domains
Authors: Yingjun Du
Xiantong Zhen
Ling Shao
Cees G. M. Snoek
Venue: ICLR 2022 Poster (2022)
Keywords: Meta-learning
Variational hierarchical memory
Variational hierarchical prototype
Cross-domain few-shot learning

Abstract: Neural memory enables fast adaptation to new tasks with just a few training samples. Existing memory models store features only from the single last layer, which does not generalize well in presence of a domain shift between training and test distributions. Rather than relying on a flat memory, we propose a hierarchical alternative that stores features at different semantic levels. We introduce a hierarchical prototype model, where each level of the prototype fetches corresponding information from the hierarchical memory. The model is endowed with the ability to flexibly rely on features at different semantic levels if the domain shift circumstances so demand. We meta-learn the model by a newly derived hierarchical variational inference framework, where hierarchical memory and prototypes are jointly optimized. To explore and exploit the importance of different semantic levels, we further propose to learn the weights associated with the prototype at each level in a data-driven way, which enables the model to adaptively choose the most generalizable features. We conduct thorough ablation studies to demonstrate the effectiveness of each component in our model. The new state-of-the-art performance on cross-domain and competitive performance on traditional few-shot classification further substantiates the benefit of hierarchical variational memory.

Published as a conference paper at ICLR 2022
HIERARCHICAL VARIATIONAL MEMORY FOR FEW-SHOT LEARNING ACROSS DOMAINS
Yingjun Du', Xiantong Zhen' Ling Shao', Cees G. M. Snoek!
AIM Lab, University of Amsterdam "Inception Institute of Artificial Intelligence
National Center for Artificial Intelligence, Saudi Data and Artificial Intelligence Authority
ABSTRACT
Neural memory enables fast adaptation to new tasks with just a few training sam- ples. Existing memory models store features only from the single last layer, which does not generalize well in presence of a domain shift between training and test distributions. Rather than relying on a flat memory, we propose hierarchical alternative that stores features at different semantic levels. We introduce a hierar- chical prototype model, where each level of the prototype fetches corresponding information from the hierarchical memory. The model is endowed with the ability to flexibly rely on features at different semantic levels the domain shift eircum- stances so demand. We meta-learn the model by newly derived hierarchical variational inference framework, where hierarchical memory and prototypes are jointly optimized. To explore and exploit the importance of different semantic levels, we further propose to learn the weights associated with the prototype at each level in data-driven way, which enables the model to adaptively choose the most generalizable features. We conduct thorough ablation studies to demonstrate the effectiveness of each component in our model. The new state-of-the-arr per- formance on cross-domain and competitive performance on traditional few-shot classification further substantiates the benefit of hierarchical variational memory.
INTRODUCTION
Few-shot learning with an external memory is known to learn new concepts quickly with only few samples, especially when embedded in meta-learning setting (Santoro et al., 2016). A common tactic is to store short-term memory (Munkhdalai & Yu, 2017; Munkhdalai et al., 201 8; Kaiser et al., 2017) as obtained from the support set of the current task, and to empty it at the end of a task. Another tactic is to let the memory store long-term knowledge distilled from all the training tasks (Zhen et al., 2020a), which provides a conceptual context to learn a new task. The long-term memory has demonstrated effectiveness in enabling a model to quickly learn new few-shot learning tasks within domains (Zhen et al., 2020a). Nonetheless, for both the short- and long-term memory tactics, the learning ability is limited when facing a task from an unseen domain (Tseng al., 2020; Guo et al., 2020: Cai & Shen, 2020; Du et al., 2021).
Cross-domain few-shot learning (Tseng et al., 2020; Guo al., 2020) is even more challenging than the conventional few-shot learning problem as it also has to take distribution shift into account. In cross-domain few-shot learning, directly using traditional external memory, which often only stores high-level semantic features (Zhen et al., 2020a), is unable to find the suitable semantics in memory due the domain shift causing side effects. Nonetheless, low-level features like textures, shapes, edges, etc. from training domains are often still meaningful for a new domain (Adler et al., 2020). We hypothesize, features different levels would play distinctive roles generalizing across domains. It remains unexplored to investigate hierarchical features in memory that can enable quick adaptation, as humans do, when faced with a few samples from new domains.
In this paper, make three contributions. First, we propose hierarchical variational memory, which accrues and stores the long-term semantic knowledge of multiple network layers from past experiences. Each entry stores features at different hierarchical levels by summarizing the feature representations class samples, thus generalizing well in presence of a domain shift between the training and test distributions. We formulate the memory recall at different hierarchical levels

---

Published as a conference paper at ICLR 2022
variational inference of the latent memory, which is intermediate stochastic variable. Second, to better deploy the hierarchical memory, we introduce a hierarchical variational prototype model to obtain prototypes at different levels well. We also develop the hierarchical variational prototype in a latent model by treating the prototypes at different levels as the latent variable. In doing so, the model is endowed with the ability to flexibly rely low-level textures high-level semantic features if the domain shift circumstances so demand. As the third contribution, we propose to learn the weights associated with the prototype at each level in data-driven way, which can explore and exploit the importance of the different feature levels for domain generalization. We leverage the few training samples in the new domain to generate reasonable weights for each prototype, since these samples already carry sufficient appearance cues about the new domain, which enables the model to adaptively choose the most generalizable features. Two different hierarchical variational inference problems are included in our optimization objective: 1) the inference of the latent memory, which conditions on the latent memory of previous layers and the memory contents of the current layer; 2) the inference of prototypes, which treats the support set, the latent memory of the current layer and the prototype of previous layers as conditions.
We demonstrate the effectiveness of the proposed hierarchical memory by extensive experiments on the cross-domane few-shot learning classification tasks introduced by (Guo et al., 2020) and the more traditional within-domain few-shot learning classification tasks (Snell et al., 2017). We conduct an extensive ablation study to demonstrate the contributions of different components in our model. We also analyze how hierarchical memory adaptively chooses the weights in the face of domain change and identifies the appropriate layer of the prototype that should be used. An extensive evaluation of cross-domain few-shot and traditional few-shot classification benchmarks reveals that our hierarchical memory consistently achieves results that are better, or at least competitive, compared to other methods.
METHODOLOGY
2.1 PRELIMINARIES
In the following, we present the relevant background on few-shot classification and its cross-domain setting, as well the preliminaries of the (variational) prototypical network and variational semantic memory (Zhen et al., 2020a).
Few-shot classification In the few-shot classification scenario, we define the N N-way K-shot classi- fication problem, which is comprised of the support sets S and query set Q. Each task also called an episode, classification problem sampled from task distribution p(T). The 'way" of the episode refers to the number of classes in the support, while the "shot" of the episode refers to the number of examples per class. Tasks are drawn from a dataset by randomly sampling a subset of classes, sampling points from these classes, and then partitioning the points into support and query sets. Episodic optimization (Vinyals et al., 2016) iteratively trains the model by taking one episode update at time.
Cross-domain setting For few-shot classification, the test tasks are typically assumed to come from the same task distribution p(T) the training tasks. The recently introduced task of cross-domain few-shot learning (Guo et al., 2020) considers the few-shot classification challenge under domain shift. Coneretely, the training task comes from single source domain (TT)), while the test tasks come from several unseen domains (Tı ... Tm }. The goal of cross- domain few-shot learning is to learn meta-learning model using single source domain that generalizes to several unseen domains.
Prototypical network Our model builds upon the prototypical network (Snell et al., 2017), which is widely used for few-shot image classification. It computes a prototype 2 2 R Σn fo(x;) for each class through an embedding function fe. which is realized by neural networks. It computes a distribution over classes for a query sample x given distance function d(-,-) as the softmax over its distances to the prototypes in the embedding space:
exp(-d((faxx),x))
ply: = k|x)
(1)
Σx R exp(- 2xr))
The variational prototype network (Zhen et al., 2020a) is a powerful model for learning latent representations from small amounts of data, where the prototype z per class
Variational prototype network
2

---

Published as a conference paper at ICLR 2022
treated distribution. Given task with support set S and query set Q, the ELBO takes the following form:
IQI
Q
log p(y)).))
where (z) is the variational posterior over 2, p()]x;) is the prior, and Lg is the number of Monte Carlo samples for z.
Variational semantic memory Variational semantic memory (Zhen et al., 2020a) is proposed to accumulate and store the semantic information from previous tasks for the inference prototypes of new tasks. It consists of two main processes: memory recall, which retrieves relevant information that fits with specific tasks based on the support set of the current task; memory update, which effectively collects new information from the task and gradually consolidates the semantic knowledge in the memory. Variational semantic memory formulates the memory recall as a variational inference the latent memory, which is an intermediate stochastic variable. The objective is written as follows:
LVSM Σ
gx S. (lgg(y DKL [q(z]S, m) [[z]x]
M
+ DKL
Σ
(mp(m MM M)]lp([|s]]]
where p(m[S) is the introduced prior over latent memory m and M denotes the memory content.
The above networks and memory only focus on the problem of few-shot classification within domain and their performance would be degrade for cross domain few-shot learning, because of the domain shift problem. this paper, we propose hierarchical memory to address few-shot classification under domain shift.
2.2 HIERARCHICAL VARIATIONAL PROTOTYPE
In the variational prototype network (Zhen et al., 2020a), the probabilistic prototype is obtained by feeding the last layer of features to an amortization network. Hence, it is not possible to obtain the prototypes of different layers when addressing the domain shift problem, making it relying the high-level prototype only, which is not necessarily shared between domains.
We introduce the hierarchical variational prototype network to generate different layer prototypes. Compared to its flat counterpart, it introduces q(z" zz' ! instead q( SS), so we approximate the true posterior by minimizing the KL divergence:
DKLÍg(z' S))p((z(xxx (x,)].
(4)
where z' is the prototype of layer I. By applying the Baye's rule, we then obtain the following ELBO:
QQ
IQI
log [I] [(y] > Σ
E 19t [logp///xx 21)] DDxr - (1)p((z'22 x.))
(5)
In practice, the variational posterior q(z'|z'-1,,S) is implemented by an amortization net- work (Kingma & Welling, 2013) that takes the concatenation of the average feature representations of samples in the support set S and the upper layer prototype z and returns the mean and variance of the current layer prototype z'. The hierarchical probabilistic prototype provides both a more informa- tive representation than the deterministic prototype and the ability to capture different representation levels, making it more suitable for cross-domant few-shot learning. More importantly, the hierarchical variational prototype also provides principled way of incorporating the prior knowledge of the different levels from the past experienced tasks. Therefore, we introduce the external hierarchical memory to enhance the prototype different levels.
3

---

Published as a conference paper at ICLR 2022
Computational graph of hierarchical variational memory for prototypical few-shot learning. The memory consists of L layers of the hierarchy: each layer M is used to build up the corresponding latent pro- totype z'. S and Q' are the support set and the query set. x and y are the sample and label. The latent memory m' is inferred from the memory con- tent M and the upper latent memory m' and the latent prototype z is constructed from the support set S, latent memory m and the latent pro- totype z in the upper layer.
Figure 1:
M1
M2
Mt
T
S
Q
x
m
m
m
X
2
z
y
observed during mela-training only
hierarchical memory
observed during meta-trairing and meta-test
Intent variablee
2.3 HIERARCHICAL VARIATIONAL MEMORY
We introduce the hierarchical variational memory to accrue and store the long-term semantic knowl- edge at different levels from previously experienced tasks for the hierarchical inference of prototypes for new tasks. By seeing more examples objects at different semantic feature levels, the knowledge in the memory is enriched and consolidated, which allows for quick learning new tasks.
Concretely, our model has T. layers of memory, each having M storage units which store key- value pair in each of the memory array. The key in each memory stores the average feature representations of images from the same class and the value is the corresponding label. We adopt similar memory mechanism (Zhen et al., 2020a): memory recall and memory update, but introduce a hierarchical version of memory recall.
We apply hierarchical variational memory into the hierarchical variational prototype network to generate the prototypes at different layers. From a Bayesian perspective, the prototype posterior can be inferred by:
g( (S) q(2 5Spmm S))m',
(6)
where m' is the latent memory of layer I. Different from the variational semantic memory (Zhen et al., 2020a), we design a hierarchical variational approximation q(m mmmm the latent memory m' of layer by inferring from M' conditioned on S. The hierarchical variational inference of prototypes by a hierarchical Bayesian framework is written as
to the posterior over
M
/ ((z2)zz--,_6 m mm)p(mm Ma m' m''',S)dmm
(
M.
Σ plalM,
where a is the addressed categorical variable, M, represents the corresponding memory content at address a, and |M indicates the memory size. We first leverage the support set S and memory M to generate the addressed categorical variable a, and then use the fetched memory content M,, support set S, and the latent memory m of upper layer I to infer the latent memory m' of layer I. For the prototype z², we use the upper layer prototype -11 the support set S, and the latent memory m' conditions.
By combining the ELBO in Eq. (5) for the hicrarchical variational network and Eq. 7, we obtain the following ELBO for hierarchical variational inference:
Q
[@|
log
ply x
B S.m xt log z')]
- -KL [22
x [g( m'
|lp2))lp(2212-- xxi] - DKL Σ
p( (m -1 M))lp((m |mn- S)]]
(
4

---

Published as a conference paper at ICLR 2022
The overall computational graph of our approach is shown in Fig. 1. Directly optimizing the above objective does not consider the role of the different feature levels for domain generalization. Thus, we propose to learn the weights associated with the prototype at each level to achieve adaptive prototypes, which enables the model to adaptively choose the most generalizable features.
2.4 LEARNING TO WEIGH PROTOTYPES
Features from different layers contain different levels of semantic abstraction about an object and should contribute differently to the generalization of new tasks. Therefore, the importance of established prototypes from different feature levels need to treated distinctively by depending on the domain from which a new task originates. To this end, we propose learning to weigh prototypes of different levels in a data-driven way.
Assume we have L prototypes: {z',.... 2²), and we can obtain L different prediction logits {y'.... y based on Eq. (1). We can produce the final logit ý by bagging (Breiman, 1996) to vote which category it should be categorized into. However, this approach does not allow the model to adaptively choose the weight that different level prototypes should occupy based on the new domain.
I.
In general, the final logit can be represented as a weighted sum: y - - an where we learn [=1
the weight og for each prototype, by conditioning the support set of the task. For cross-domain few-shot learning, we have a few samples S when testing on new domains, we can generate a reasonable weight for prototypes at different levels of S, because, intuitively, these samples already carry sufficient domain information. To generate the weight ar of layer 1, we deploy hypernetwork (+() that takes the average gradients (Munkhdalai et al., 2018) of all the support sets input, and returns the weight or by softmax:
aq ==ftmaax (V's),
(9)
where V's are the average gradients on layer 1 of all the support sets. The hypernetwork (1()) is first learned at veeainraining time and then directly used as the support set S from the test domain at meta-test time. Note that on the test domain we do not learn the parameter of fa(:); instead, we simply rely on the support gradient of each layer to generate its weight for the final result. We propose to learn the weights associated with the prototype at each level of the hierarchical variational prototype and hierarchical variational memory to adaptively unify different prototypes at different levels, which enables the model to adaptively choose the most generalizable features, thus achieving generalization for cross-domain few-shot learning.
3 RELATED WORKS
Memory Several works augment neural networks with an external memory module to improve learning (Santoro et al., 2016; Pritzel et al., 2017; Weston et al., 2014; Graves et al., 2016; Kaiser et al., 2017; Munkhdalai & Yu, 2017: Munkhdalai et al., 2018; Ramalho & Garnelo, 2019). (Santoro al., 2016) equip neural networks with neural Turing machine for few-shot learning. The external memory stores samples in the support set in order to quickly encode and retrieve information from new tasks. (Bornschein et al., 2017) proposed augmenting generative models with an external memory, where the chosen address is treated as a latent variable. Sampled entries from training data are stored in the form of raw pixels in the memory for few-shot generative learning. In order to store minimal amount of data, (Ramalho & Garnelo, 2019) proposed surprise- suprise-based memory module, which deploys memory controller to select minimal samples write into the memory. An external memory was introduced to enhance recurrent neural network in (Munkhdalai et al., 2019), in which memory is conceptualized as an adaptable function and implemented as a deep neural network. Semantic memory has recently been introduced by (Zhen et al., 2020a) for few-shot learning to enhance prototypical representations of objects, where memory recall cast as a variational inference problem. In contrast to (Zhen et al., 2020a), our memory adopts hierarchical structure, which stores the knowledge at different levels instead of the last layer only.
The prototypical network for shw- learning is first proposed by (Snell et al., 20.7). It learns to project the samples into a common metric space where classification is conducted by computing the distance from query samples to prototypes of classes. Infinite Mixture Prototypes was
Prototypes
5

---

Published as a conference paper at ICLR 2022
proposed by (Allen et al., 2019), which improved the prototypical network by infinite mixture of prototypes that represents each category of objects by multiple clusters. (Triantafillou et al., 2020) combines the complementary strengths of Prototypical Networks and MAML (Finn et al., 2017) by leveraging their effective inductive bias and flexible adaptation mechanism for few-shot learning. (Zhen et al., 2020a) proposed the variational prototypical network improve the prototypical network by probabilistic modeling of prototypes, which provided more informative representations of classes. Different from their work, we propose hierarchical variational prototype networks generate different layer prototypes instead of using one prototype, which could capture information at different levels for few-shot learning across domains.
Meta-learning or learning to learn (Schmidhuber, 1987; Bengio et al., 1991; Thrun & Pratt, 1998), is a learning paradigm where model is trained on distribution of tasks so as to enable rapid learning on new tasks. Meta-learning approaches to few-shot learning (Vinyals et al., 2016; Ravi & Larochelle, 2017; Snell et al., 2017: 7 Zhen et al., 2020a; Finn et al., 2017; Du et al., 2021) and the domain generalization (Du et al., 2020; Xiao al., 2021; 2022) differ in the way they acquire inductive biases and adapt to individual tasks. They can be roughly categorized into three groups. Those based metric-learning generally learn shared embedding space in which query images matched to support images for classification ( (Vinyals et al., 2016; Snell et al., 2017; Santoro et al., 2016; Oreshkin et al., 2018: Allen al., 2019). The second, optimization-based group learn an optimization algorithm that is shared across tasks, which can be adapted to new tasks for efficient and effective learning (Ravi& & Larochelle, 2017: Andrychowicz et al., 2016; Finn al., 2017: 2018; Grant et al., 2018; Triantafillou et al., 7017; Rusu et al., 2019; Zhen et al., 2020b). The memory- meta-learning group leverages an external memory module to store prior knowledge that enables quick adaptation to new tasks (Santoro et al., 2016; Munkhdalai & Yu, 2017; Munkhdalai et al., 2018; Mishra al., 2018; Zhen et al., 2020a). Our method belongs to the first and third group, as it is based on prototypes with external hierarchical memory, with the goal to perform cross-domain few-shot classification.
Cross-domain few-shot learning The problem is formally posed by (T'seng et al., 2020), who attack it with feature-wise transformation layers for augmenting mage features using affine transforms to simulate various feature distribution. Then, (Guo et al., 2020) proposed the cross-domain few-shot learning benchmark, which covers several target domains with varying similarities to natural images. (Phoo & Hariharan, 2020) introduce a method, which allows few-shot learners to adapt feature representations to the target domain while retaining class grouping induced by the base classifier. performed cross-doman low-level feature alignment and also encodes and aligns semantic structures in the shared embedding space across domains. (Wang & Deng, 2021) proposed a method which improved the cross-domain generalization capability in the cross-domain few-shot learning through task augmentation. In contrast, we first address cross-domain few-shot learning by a variational inference approach, which enables us to better handle the prediction uncertainty on the unseen domains. Moreover, we propose a hierarchical variational memory for cross-domain few-shot learning, and to better deploy the hierarchical memory, we introduce a hierarchical variational prototype model to obtain prototypes at different levels.
4 EXPERIMENTS
4.1
EXPERIMENTAL SETUP
We apply our method to four cross-domain few-shot challenges and two within-domain few-shot image classification benchmarks. Sample images from all datasets are provided in the appendix A.
Cross-domain datasets The 5-way 5-shot cross-domain few-shot classification experiments use minilmagenet (Vinyals et al., 2016) as training domain and test on four different domains, i.e., CropDiscase (Mohanty et al.. 2016) containing plant discase images, EuroSAT (Helber et al., 2019) consisting of a collection of satellite images, ISIC2018 (Tschandl et al., 2018) containing dermoscopic images of skin lesions, and ChestX (Wang et al., 2017), set of X-ray images. Results on the 5-way 20-shot and 5-way 50-shot are provided in the appendix.
Within-domain datasets The traditional few-shot miniImagenet (Vinyals et al., 2016) which consists of 100 randomly chosen classes from ILSVRC- 2012 (Russakovsky et al., 2015), and tieredlmagenet (Ren et al., 2019) which is composed of 608
experiments conducted on

---

Published as a conference paper at ICLR 2022
Table 1: Benefit of hierarchical variational prototype in (%) on four cross-domain challenges under the 5-way 5-shot setting. Hierarchical variational prototype achieves slighly better or comparable performance to the variational prototype on all domains.
Method
CropDiseases
EuroSAT
ISIC
ChestX
Variational prototype (Zhen et al., 2020a)
80.72 + 0.37
73.21 2041
40.53
0.35
25.03 + 0.41
Hierarchical variational prototype
83.75 + 0.33
74.29 1.4
41.21 0.3
26.15 0.5
Table 2: Hierarchical vs. flat variational memory in (%) on four cross-domain challenges under
the 5-way 5-shot setting. Hierarchical variational memory is more critical than the flat variational memory for cross-domain few-shot learning.
Method
CropDiseases
EuroSAT
ISIC
ChestX 22.15 + 1.00
Flat variational memory (Zhen et al., 2020a) Hierarchical variational memory
79.12 0.65
72.21 +: 0.70
38.97 0.53
87.65 + 0.35
74.88 + 0.45
42.05 = 0.34
27.15 + 0.45
classes grouped in 34 high-level categories. In the test stage, we measure the accuracy of 600 tasks sampled from the meta-test set. In this paper, we focus on 5-way 1-shot/5-shot tasks following the prior work (Snell al., 2017; Finn et al., 2017: Zhen et al., 2020a).
Implementation details We extract image features using ResNet- backbone, which is commonly used for cross drosinn few-shot classification (Ye al., 2020; Guo et al., 2020). For the computation of features at each level, we first input the last convolutional layer feature map of each residual block through a flattening operation and then input the flattened features into the two fully connected layers. For the within-domain experiments, Conv-4 was first used for few-shot classification by (Vinyals et al., 2016), and is widely used (Snell et al., 2017; Finn al., 2017: Sun al., 2019; Zhen et al., 2020a). To gain better performance, ResNet- 12 (Bertinetto et al., 2019; Gidaris & Komodakis, 2018; Yoon et al., 2018) is also widely reported for few-shot classification. Following the prior works, we configure the ResNet- 12 backbone as 4 residual blocks. Each prototype is obtained in the same way as the cross-domain few-shot classification. We two-layer inference network for the few-shot cross domain experiments. Following the VSM (Zhen et al., 2020a), we use the same three-layer inference network for the few-shot within domain experiments. Our code will be publicly released.
use two backbones: Conv-4 and ResNet-12. The
Metrics The average cross-domain/ within-domain few-shot classification accuracy (%, top-1) along with 95% confidence intervals are reported across all test images and tasks.
4.2 RESULTS
Benefit of hierarchical prototypes To show the benefit of the hierarchical prototypes, we compare hierarchical variational prototypes with variational prototypes (Zhen et al., 2020a), which obtains probabilistic prototypes of the last layer only. Table shows the hierarchical prototype achieves slightly better or comparable performance to the single prototypes on all cross-domain few-shot classification tasks. More importantly, the experiments confirm that hicrarchical prototypes can capture features different levels, which allows for better deployment of hierarchical memory. As it may fetch the corresponding information for each level of prototype, which we will demonstrate next.
Hierarchical vs. flat variational memory We first show that flat memory as used in (Zhen et al., 2020a) is less suitable for cross-domain few-shot learning in Table 2. With flat memory, performance even degrades a bit compared variational prototypes (see Table 1). One interesting observation is that on ChestX - with the largest domain gap- - the performance of flat memory degrades the most compared to the variational prototype. This reasonable since features vary for the domains, resulting in the tested domains not finding the appropriate semantic information. Our hicrarchical variational memory consistently surpasses flat memory on all test domains. We attribute the improvements to our model's 's ability to leverage memory of different layers to generate prototypes. The hierarchical memory provides more context information at different layers, enabling more information to be transferred to new domains, and thus leads to improvements over flat memory.
Benefit of learning to weigh prototypes We investigate the benefit of adding the learning to weigh prototypes to the hierarchical variational memory. The experimental results are reported in Table 3. Hicrarchical variational memory with bagging produces the final result by bagging (Breiman, 1996)
httpp::/ thub. wwt.://oo///..aaae hwt.://eeweeeeerreg.ee.emo erMemory.
7

---

Published as a conference paper at ICLR 2022
Table 3: Benefit of learning to weigh prototypes in (%) on four cross-domain challenges. Hierarchical variational memory with learning to weigh prototypes achieves better performance than with bagging.
Method
CropDiseases
EuroSAT
ISIC
ChestX
Hierarchical variational memory with bagging
85,43 + 0.35
74.02 0.41
40.39 = 0.32
25.98 0445
Hierarchical variational memory
87.65 0.35
74.88 0.45
42.05 0.34
27.15 0.45
5-way 5-shot
5-way 20-shot
5-way 50-shot
0.8
0.8
6.6-
Value of
Value of a
0.4
0.4
0.-
0.-
BUnDAT
ISC
Chalk
0.0
DAVSAT
ISC
Cheak
Creploceat: BurotAr
Cheste
Figure 2: Visualization of weights a' at different residual blocks. The weight a' varies per test- domain, the larger the domain shift between the training and test domain, the larger the weight the low-level prototype.
to vote which category it should be categorized into. Adding the learning to weigh prototypes on top of the hierarchical variational memory leads to small but consistent gain under all test domains. Thus, the learned adaptive weight acquires the ability to adaptively choose the most generalizable features for the new test domain, resulting in improved performance.
Importance of prototypes at different levels We present visualization of the influence of the prototypes at different levels, using a varying number of shots, in Fig. 2. We sample 600 test tasks on each domain to show the value of a' at different blocks, and average a' among all the test tasks. We choose the last three blocks of ResNet-10 to generate the memory and prototype. As presented in (Guo et al., 2020), the degree of domain shift between minilmageNet and target domains is ordered by: CropDiseases EuroS < ISIC- < ChestX. From Fig. 2, we can see that the value of a increases along with the increase in domain shift. Another interesting observation is that as the training samples (shots) in each class increase, the weight of the high-level prototypes also increases. This is expected as high-level features can already better represent the information of the new domain due to the increase in the number of training samples. However, the weight of the low-level prototypes is still larger than the weight of the high-level ones. This experiment demonstrates that the lower-level features may be more useful across different domains than the more specialized high-level concepts.
Visualization of hierarchical variational prototypes To understand the empirical benefit of using prototype at different levels, we visualize the distribution of prototypes obtained by different blocks on the most challenging ChestX domain under the 5-way 5-shot setting in Fig. 3. The level (low-level) prototypes enable different classes to be more distinctive and distant from each other, while the prototypes obtained by level (high-level) have much more overlap among classes. This suggests that with large domain shift between the original source domain and the new target domain, low-level textures are more useful and distinctive. This again demonstrates that our hierarchical variational memory to leverage prototypes different levels is suitable for cross-domain few-shot learning.
Table 4: Comparative results of different algorithms on four proposed cross-domain few-shot chal- lenges. The results of other methods are provided by (Guo et al., 2020). Runner-up method is underlined, Our hierarchical variational memory is consistent top top-performer.
Ciseasse S-way
EuroSAT 5-way
ISTC 5-way
ChestX S-way
Method
5-shot
20-shot
5-shot
20-shot
5-shot
20-shot
5-shot
20-shot
MatchingNet
66.39 0.7
76.38 0007
64.45
200
77,10
0.57
36.74 :: 0.5
45.72 0m5
22,40
= 0.00
23.61
NatchingNet-FWT
62.74 0.0
74.90 0.71
56.04
63.38
= 0.69
30.40
= 0.48
32.01
0:4
21.26
231
23.23
t.37
MAML
78.05 0.6
89.75
0:4
71.70
0.72
81.95
0.35
40.13
0.5
$2.36
055
23.48
096
27.53
ProtoNet
79.72 1:060
88.15 0.51
73.29 1:071
82.27
0:3.7
39.57
0:57
49.50
a.s
24.05
101
28.21 1.1
FoT
72.72
85.82
67.34
75.74
38.87
43.78
23.77
26.87
RelationNct
68.99 0.75
80.45 064
61.31 0:022
74.43
= 066
39.41 038
41.77
22.96 D48
26.63 = 192
FeT
64.91
78.43
61.16
69.40
35.54
43.31
22.74
26.75
MetaOpt
68.41 + 0.73
0.54 95.13
82.89
64.44
0:73
79.19
+ 0/6
36.28 = 030
49.42 3:6
22.53
= 181
25.53 = 1.02
Ours
87.65
74.88 0.45
84.81
42.05
54.97
27.15
30.54
8

---

Published as a conference paper at ICLR 2022
Level |
Level
Level 3
Figure Prototype distributions different levels with our hierarchical variational memory on ChestX under the 5-way 5-shot setting, where different colors indicate different categories. The prototypes of Level are more distinctive and distant from each other, which reflects the lower-level feature are more crucial when the domain shift is large.
Few-shot across domains We evaluate hierarchical variational memory on the four different datasets under B-way 5-shot and 5-way 20-shot in Table 4, the results for 50-shot are provided in the appendix B.4. Our hierarchical variational memory achieves state-of-the-an ttteeo---tt--rttpefrrmaaeee cross-domain few-shot learning benchmarks under each setting. On CropDisease (Mohanty et al., 2016), our model achieves high recognition accuracy under various shot configurations, surpassing the second best method, i.e. ProtoNet (Snell et al., 2017), by a large margin of 7.93% on the 5-way 5-shot. Even on the most challenging ChestX (Wang al., 2017), which has large domain gap with the minilmageNet, delivers 27.15% on the 5-way 5-shot setting, surpassing the second best ProtoNet (Snell al., 2017) by 3.10%. The consistent improvements all benchmarks under various configurations confirm that our hierarchical variational memory is effective for cross-domain few-shot learning.
all four
Few-shot within domain We also evaluate our method on few-shot classification within domains, in which the training domain is consistent with the test domain. The results using ResNet-12 12 are reported in Table 5, the results using Conv-4 are reported in the appendix B.5. Our hierarchical variational memory achieves consistently better performance compared the previous methods on both datasets. The results demonstrate that hierarchical variational memory also benefits performance for few-shot learning within domains.
Table 5: Comparative results for few-shot learning minilmagenet and mieredI using ResNet-12 12 backbone, Runner-up up method is underlined. The proposed hierarchical variational memory can also improve performance for few-shot learning within domains.
mini magenet 5-way
fiered Imagenneeenet 5-way
Method
1-shot
5-shot
1-shot
5-shot
SNAIL (Mishra et al., 2018)
55.71 + 0.99
68.88 + 0.2
Dynamic FS (Gidaris& & Komodakis, 2018)
55.45 0.89
70.13 1: 0.68
TADAM (Oreshkin al., 2018)
58,50
76.70
MTL (Sun al., 2019)
61.20
75.50 + 0.80
VariationalFSI. (Zhang et al., 2019)
61.23 + 0.26
77.69 +: 0.17
TapNet (Yoon et al., 2019)
61.65 + 0.15
76.36 4: 0.10
63.08 0.15
80.26 à 0.12
MctaOptNet (Loc et al., 2019)
62.64 066
78.63 +: 0,46
65.81
= 0.74
81.75 = 0.53
CTM (Li al., 2019)
62.05 0.55
78.63 0.06
64.78
81.05
0.52
CAN (Hou et al., 2020)
63.85 0.88
79.44 +
69.89
84.23
+ 86.77
VSM (Zhen et al., 2020a)
65.72 + 0.57
82.73 + 0.51
72.01
+ 0.71
+ 0.44
Ours
67 83 4 0.32
83.85 0:025
73.67 0.34
88 05.05.. 0.14
5
CONCLUSION
In this paper, we propose hierarchical variational memory for few-shot learning across domains. The hierarchical memory stores features at different levels, which incorporated as an external memory into the hierarchical variational prototype model to obtain prototypes at different levels. Furthermore, to explore the importance of different feature levels, we propose learning to weigh prototypes in data-driven way, which further improves generalization performance. Extensive experiments six benchmarks demonstrate the efficacy of each component in our model and the effectiveness of hierarchical variational memory in handling both the domain shift and few-shot learning problems.

---

Published as a conference paper at ICLR 2022
6 ACKNOWLEDGEMENT
This work is financially supported by the Inception Institute of Artificial Intelligence, the University of Amsterdam and the allowance Top consortia for Knowledge and Innovation (TKIs) from the Netherlands Ministry of Economic Affairs and Climate Policy.
REFERENCES
Thomas Adler, Johannes Brandstetter, Michael Widrich, Andreas Mayr, David Kreil, Michael Kopp, Günter Klambauer, and Sepp Hochreiter. Cross-domain few-shot learning by representation fusion. arXiv preprint arXiv:2010.06498, 2020.
Kelsey Allen, Evan Shelhamer, Hanul Shin, and Joshua Tenenbaum. Infinite mixture prototypes for few-shot learning. In ICML., pp. 232 241, 2019.
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In NeurlPS, pp. 3981-3999, 2016.
Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning synaptic learning rule. In IJCNN, 1991.
Luca Bertinetto, Joao Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differen- tiable closed-form solvers. In ICLR, 2019.
Jörg Bornschein, Andriy Mnih, Daniel Zoran, and Danilo Rezende. Variational memory addressing in generative models. In NeurlPS, 2017.
Leo Breiman. Bagging predictors. Machine learning, 24(2):123 140, 1996.
John Cai and Sheng Mei Shen. Cross-domain few-shot learning with meta fine-tuning, arXiv preprint arXiv:2005.10544 2020.
Yingjun Du, Jun Xu, Huan Xiong, Qiang Qiu, Xiantong Zhen, Cees GM Snock, and Ling Shao. Learning to learn with variational information bottleneck for domain generalization. In ECCV, pp. 200-216, 2020.
Yingjun Du, Xiantong Zhen, Ling Shao, and Cees G M Snoek. Metanorm: Learning to normalize few-shot batches across domains. In ICLR, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, pp. 1126-11333 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-lcarning. In NeurIPS. pp. 9516-9527, 2018.
Spyros Gidaris and Nikos Komodakis. Dynamie few-shot visual learning without forgetting. In CVPR, pp. 4367-4375, 2018.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning as hierarchical bayes. ICLR, 2018
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwiñska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626): 471-476 2016.
Yunhui Guo, Noel C Codella, Leonid Karlinsky, James V Codella, John R Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris. A broader study pp. 124-14) 141, 2020.
few-shot Icarning. ECCV,
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):221--2266, 2019.
10

---

Published as a conference paper at ICLR 2022
Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Cross attention network for few-shot classification. In NeurlPS, pp. 4005-4016, 2020.
Lukasz Kaiser, Ofir Nachum, Aurko Roy, and Samy Bengio. Learning to remember rare events, In ICLR, 2017.
Diederik Kingma and Max Welling. arXiv:13/2.61 14, 2013.
Auto-encoding variational bayes. arXiv preprint
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In CVPR, pp. 10657-10665, 2019.
Hongyang Li, David Eigen, Samuel Dodge, Matthew Zeiler, and Xiaogang Wang. Finding task-
relevant features for few-shot learning by category traversal. In CVPR, pp. 1-10, 2019.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. In ICLR, 2018.
Sharada P Mohanty, David P Hughes, and Marcel Salathé. Using deep learning for image-based plant disease detection. Frontiers plant science, 7:1419, 2016.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In JCML, pp. 2554-2563, 2017.
Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mehri, and Adam Trischler. Rapid adaptation with conditionally shifted neurons. In ICML, 2018.
Tsendsuren Munkhdalai, Alessandro Sordoni, Tong Wang, and Adam Trischler. Metalearned neural memory. In NeurlPS, 2019.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint STt:::8803002999 2018.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In NeurlPS, pp. 721-731, 2018.
Cheng Perng Phoo and Bharath Hariharan. Self-training for few-shot transfer across extreme task differences. arXiv preprint arXiv:2010.07734. 2020.
Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. ICML, 2017.
Tiago Ramalho and Marta Garnelo. Adaptive posterior learning: few-shot learning with a surprise- based memory module. In ICLR, 2019.
S. Ravi and H. Larochelle. Optimization as model for few-shot learning. In ICLR, 2017.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classification. In ICLR, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bemsteein, al. Imagenet large scale visual recognition challenge. IJC,, 115(3):211-25) 2015.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In ICLR, 2019.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with
neural networks. In ICML, pp. 1842-18500 2016.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, learning how to learn: the meta-meta... hook. PhD thesis, Technische Universität München, 1987.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurIPS, pp. 4077-4087 2017.
11

---

Published as a conference paper at ICLR 2022
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot
learning. CVPR, pp. 403 412,412,2222 2019.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning compare: Relation network for few-shot learning. In CVPR, pp. 1199-1208, 2018.
Sebastian Thrun and Lorien Pratt (eds.). Learning to Learn. Kluwer Academic Publishers, USA, 1998.
Eleni Triantafillou, Richard Zemel, and Raquel Urtasun. Few-shot learning through an information retrieval lens. arXiv preprint arXiv:1707.02610 2017.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta- dataset: A dataset of datasets for learning to learn from few examples. In ICLR, 2020.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham00000 dataset, large collection of multi-source dermatoscopic images common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
Hung- Yu Tseng, Hsin- Ying Lee, Jia-Bin Huang, and Ming- Hssan Yang. Cross-domain few-shot classification via learned feature-wise transformation. arXiv preprint 1XXi:20001008335 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. NeurIPS, 29:3630-3638, 2016.
Haoqing Wang and Zhi-Hong Deng. Cross-domain few-shot classification via adversarial task augmentation. arXiv preprint arXiv: 104. 14385, 2021.
Xiaosong Wang. Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Sum- mers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly classification and localization of common thorax diseases. In CVPR, pp. 2097-2106, 2017.
Jason Weston, Sumit Chopra, and Antoine Bordes.
Memory networks.
arXiv preprint
arXiv:1410.3916, 2014.
Zehao Xiao, Jiayi Shen, Xiantong Zhen, Ling Shao, and Cees Snoek. A bit more bayesian: Domain- invariant learning with uncertainty. In ICML, 2021.
Zehao Xiao, Xiantong Zhen, Ling Shao, and Cees Snock. Learning to generalize across domains on single test samples. In ICLR, 2022.
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In CVPR, pp. 8808-8817, 2020.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In NeurlPS, pp. 7343-7353, 2018.
Sung Whan Yoon, Jun Seo, and Jaekyun Moon. Tapnet: Neural network augmented with task-adaptive projection for shw-shot learning. In ICML, pp. 7115-7123, 2019.
Baoquan Zhang, Xutao Li, Yunming Ye, and Shanshan Feng. Prototype completion for few-shot learning. arXiv preprint arXiv:2108.08 05010, 2021.
Jian Zhang, Chenglong Zhao, Bingbing Ni, Minghao Xu, and Xiaokang Yang. Variational few-shot learning. ICCV, pp. 1695-1694, 2019.
Xiantong Zhen, Yingjun Du, Huan Xiong, Qiang Qiu, Cees GM Snoek, and Ling Shao. Learning to learn variational semantic memory. In NeurIPS, 2020a.
Xiantong Zhen, Haoliang Sun, Yingjun Du, Jun Xu, Yilong Yin, Ling Shao, and Cees Snoek. Learning to learn kernels with variational random features. In ICML. PMLR, 2020b.
12

---

Published as a conference paper at ICLR 2022
A
DATASETS
We apply our method to four cross-domain few-shot challenges and two within-domain few-shot image classification benchmarks. Sample images from each dataset are provided in Figure
mintmaggNeet
doredimageNiet
CropDisease
EuroSAT
ISIC
ChasiX
8
S
Figure 4: Examples from each dataset. The color of the border of each image represents the category of this image in this task. S and Q indicate the support and query sets for each task.
B MORE RESULTS
The more ablation results the 5-way 20-shot and 5-way 50-shot for cross-domain few-shot classification are shown in Sec. B.1. Sec. B.2 and Sec. B.3. The results on four proposed cross- domain few-shot challenges under 5-way 50-shot setting are shown in Sec. B.4. The results of few-shot learning methods on the two benchmark datasets minilmagenet and migred by the Conv-4 are shown in the Sec. B.5.
B. BENEFIT OF HIERARCHICAL PROTOTYPE
To show the benefit of the hierarchical prototypes, we compare hierarchical variational prototypes with variational prototypes (Zhen et al., 2020a), which obtains the probabilistic prototypes of the last layer. Table 10 and Table 11 show the the hierarchical variational prototypes consistently outperform variational prototypes on all cross-domain few-shot classification tasks under the 5-way 20-shot and 5-way 50-shot setting.
Table 6: Benefit of hierarchical variational prototype in (%) on four cross-domain challenges under 5-way 20-shots setting.
Method
CropDiseases
EuroSAT
ISTC
ChestX
Variational prototype (Zhen et al., 2020a)
89.73 a 0.35
83.21 + 0.37
50.64 a 0.38
29.12 a 0.46
Hierarchical variational prototype
93.15 033
85.13 0.34
52.65 0.39
30.14 + 044
Table Benefit of hierarchical variational prototype (%) on four cross-domain challenges under 5-way 50-shots setting.
Method
CropDiseases
EuroSAT
ISIC
ChestX
Variational prototype (Zhen al., 2020a)
92.01 0.3
81.96
54.56
30.98
Hierarchical variational prototype
94.25 + 031
84.33 0.35
57.74 0.32
31.27 043
13

---

Published as a conference paper at ICLR 2022
B.2 HIERARCHICAL VS. FLAT VARIATIONAL MEMORY
We show that a flat memory as used in (Zhen et al., 2020a) is less suitable for cross-domain few-shot learning in Table and 9 under the 5-way 20-shot and 5-way 50-shot setting. Hierarchical variational memory consistently achieves the best performance among all the datasets and settings.
Table 8: Hierarchical vs. flat variational memory in (%) on four cross-domain challenges under 5-way
20-shot setting.
Method
CropDiseases
EuroSAT
ISIC
ChestX
Flat variational memory (Zhen et al., 2020a)
87.27
81.25 0.37
49. 10
26.21
Hierarchical variational memory
95.13 0.35
84.81 :: 0.34
54.97 0.35
30.54 :: 0.47
Table 9: Hierarchical vs. flat variational memory in (%) on four cross-domain challenges under 5-way 50-shot setting.
Method
CropDiseases
EuroSAT
ISIC
ChestX
Flat variational memory (Zhen et al., 2020a)
91.59 0.32
80.28 +: 0.35
50.15 0.31
28.12 + 0.42
Hierarchical variational memory
97.83 0.33
87.16 + 0.3
61.71 0.32
32.76 + 0.46
B.3 BENEFIT OF LEARNING TO WEIGH PROTOTYPES
Table 10 and 11 show the results of hierarchical variational memory with bagging vs. hierarchical variational memory for the cross-domain few-shot classification under the 5-way 20-shot and 5-way 50-shot setting. hierarchical variational memory with learning to weigh prototype performs best overall.
Table 10: Benefit of learning to weigh prototypes (%) on four cross-domain challenges under 5-way 20-shot setting.
Method
CropDiseases
EuroSAT
ISIC
ChestX
Hierarchical variational memory with bagging
93.39
83.35 0.35
53.13
0.38
29.17 0.45
Hierarchical variational memory
95.13 : 0.35
84.81 0.34
54.97 = 0.3
30.54
B.4
FEW-SHOT ACROSS DOMAIN
We evaluate hierarchical variational memory on the four different datasets under 5-way and 50-shot configurations in Table 12. Our hierarchical variational memory achieves the new state-of-the- art performance on all the cross-domain few-shot learning benchmarks under 5-way and 50-shot setting.
B.5 FEW-SHOT WITHIN DOMAIN
We report the results of few-shot learning methods on the two few-shot within domain benchmark datasets minilmagenet and tieredImagenet by the Conv-4 backbones in Table 13, Under both datasets, hierarchical variational memory consistently outperforms the previous approaches.
C TRAINING SPEED
We plot the training loss versus training iterations for different algorithms in Figure 5. From Figure and Table 4, we conclude our hierarchical variational memory achieves best training efficiency and classification accuracy, which demonstrates that our hierarchical variational memory is effective for cross-domain and within domain few-shot learning.
D RESULTS ON DATA AUGMENTATION
We also provide results for few-shot within domain using a ResNet-12 backbone under data augmenta- tion in the meta-training stage following (Zhang al., 2021). The results are shown in Table 14. With
14

---

Published as a conference paper at ICLR 2022
Table 11: Benefit of learning weigh prototypes in (%) on four cross-domain challenges under 5-way 50-shot setting
Method
CropDiseases
EuroSAT
ISIC
ChestX
Hierarchical variational memory with bagging
95.91 + 0.35
85,90 0.34
58.97 = 0.33
30.95 045
Hierarchical variational memory
97.83 0.33
87.16 0.35
61.71 0.32
32.76 0.46
Table 12: Comparative results of few-shot learning methods on four proposed cross-domain few-shot challenges under 5-way 50-shot setting.
Method
CropDiseases
EuroSAT
ISIC
ChestX
MatchingNet
58.53 0.73
54.44 0.67
54.58 = 0.6
22.12 0.88
MatchingNet+FWT
75.68 = 0.78
62.75 = 0.76
33.17 = 043
23.01 0.34
ProtoNet
90.81
80.48 0.57
41.05 0.52
29.32
1.12
ProtoNet+FWT
87.17 = 0.50
78.64 1: 0.57
49.84 0.51
30.12
0.46
RelationNet
85.08 = 0.53
74.91 0.58
49.32 1051
28.45
FeT
81.14 + 0.56
73.84 +: 0.60
46.38 0.53
27.56 0.40
MetaOpt
91.76 0.38
83.62 0.68
54.80 + 0.54
29.35 0.99
Ours
97.83 0.33
87.16 0.35
61.71 = 0.32
32.76 0.46
Hierarchical variational memory Variational semantic memory
10a
Variational prototype
ProtoNet
Training Loss
10a
10000
20000
30000
40000
50000
60000
70000
80000
Iterations
Figure 5: Training loss versus iterations for different algorithms. Our hierarchical variational memory achieves fastest training convergence.
data augmentation for few-shot within domain, our model also consistently achieves thecompetitive performance.
E BENEFITS OF HIERARCHICAL STRUCTURES
We first show the benefit of hierarchical structures for few-shot within domain in Table 15. To show the effect of our hierarchical formulation, compare with VSM, which does not use the hierarchical structure for the prototype, and with hierarchical variational memory (last level). We found performance improvement by using the hierarchical formulation for the prototype. Also, by
15

---

Published as a conference paper at ICLR 2022
Table 13:
Comparative results of few-shot learning methods on minilmagenet and tieredImagenet using a Conv-4 backbone. Runner-up method is underlined.
minilmagenet 5-way
fieredImagenet 5-way
Method
1-shot
5-shot
1-shot
5-shot
Matching (Vinyals et al., 2016)
03.56+0.844
55.31+0.73
Meta-I.STM (Ravi & Larochelle, 2017)
43.44+0.777
60.60+0.71
MAML (Finn al.. 2017)
48.70:1.84
63.11:0.92
51.671181
70.3011.75
ProtoNets (Snell et al., 2017)
49.42:10.78
68.20:1066
48.5810.87
69.5710.75
Reptile (Nichol et al., 2018)
47.07+0.26
62.74+0.37
48.97+0.21
66.47+0.21
RelationNet (Sung et al., 2018)
50.44.10.22
65.32.1.770
54.4810.993
71.3210.78
IMP (Allen al., 2019)
49.60-10.80
68.1010..00
FEAT (Ye al., 2020)
55.15:0.220
71.61=0.16
VSM (Zhen et al., 2020a)
1.673+1..6
68 0..00
56.88 1.71
74.65 + 0.81
Ours
57 $7.04-10.92
72.65-10.20
59.01 +0.83
77. 77.76+10.22
Table 14: Comparative results for few-shot learning on minilmagenet and tieredImagenet using ResNet-12 backbone under same data augmentation as Zhang al. (2021). With data augmentation, our model also achieves competitive performance.
miniImagenet S-way
Imaeen 5-way
Method
1-shot
5-shot
1-shot
5-shot
Zhang al. (2021)
69,68
0.76
81.65
+ 054
74.19
0.00
86.09 + 0.60
Ours
71.05
84.54 0.23
75.43
0.29
88.97 0.24
comparing the hierarchical variational memory (last level) and hierarchical variational memory, found that the performance has further improved, which shows the effect of ensembling.
F BENEFITS OF MODEL SIZE
We also ablate our approach in terms of model size, by varying the backbone. We report few-shot classification within domain by using Conv-4, WRN-28-10, ResNet-12 12 and ResNet-188 in Table 16. The performance of our method increases along with the increase in backbone capacity. As our memory is much smaller than the backbones, the trend in experimental results affected by the change of backbone.
16

---

Published as a conference paper at ICLR 2022
Table 15: Benefit of learning hierarchical structure for few-shot learning using a Conv-4 backbone. The hierarchical structure benefits performance compared to the flat VSM.
minilmagenet 5-way
Imagenele 5-way
Method
1-shot
5-shot
1-shot
5-shot
VSM (Zhen al., 2020a)
5:1.73:11.60
68.01::0.90
56.88 1.71
74.65 = 0.81
Hierarchical variational memory (last level)
56.72 56.79
70.11::.21
57.89 0.81
76.01 + 0.60
Hierarchical variational memory
57.04::0.92
72.65::..20
59.0111.883
77.7610.62
Table 16: Comparative results for few-shot learning on minilmagenet and tieredImagennt using different backbones.
minilmagenet 5-way
tieredImagenet 5-way
Backbone Conv-4
1-shot
5-shot
1-shot
5-shot
57.04 + 092
72.65
+: 0.20
59.01
= 0.83
77.76 #: 0.62
WRN-28810 10
66. 19 : 0.3
81.61 0.20
69.81
0.33
84.76 : 0.12
ResNet-12 12
67.01 : 0.3
81.75
0.21
71.70
0.35
85.13 : 009
ResNet-18 18
68.94 0.30
82.75 0.25
73.13
0.88
85.06 0.14
17


Reviews:
[{"review_id": "ae7Ki0mnUC", "reviewer": "Reviewer_g1Bf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "$\\textbf{Strengths}$\n1. The paper is well written and easy to follow. The authors do a good job of introducing their model elements, and contrasting them with previous works.\n2. The improvements obtained over the baselines are impressive. Specifically, on the task of cross-domain few shot classification, the gap between the proposed method and the most competitive baseline is significant.\n3. Additionally, the ablation experiments shown are extensive and do a good job of highlighting the importance of each component in the proposed framework. \n\n$\\textbf{Weaknesses}$\n1. Table 13 highlights that the proposed method is not that effective when using shallow feature extractors like Conv-4. This probably leads me to believe that ensembling is providing the major improvements and the proposed hierarchical formulation isn't actually that important. To this end, I have two questions:\n\na. What happens if you don't use an ensemble and rather just use the logits from the last level (keeping the rest of the architecture as is)? As the later latent variables depend on the earlier ones (Figure 1), if the hierarchical framework works well, you should still see improvements over the baselines.\n\nb. What happens if you train $L$ instances of VSM (Zhen et al. 2020), where each instance is trained on the output of a residual block. That is, if you remove the connections between $\\mathbf{z}$ and $\\mathbf{m}$ in Figure 1. How crucial is it to have these dependencies between the latent variables.\nIt would be great if the authors could comment on this.\n\n2. What is the memory overhead of using the proposed hierarchical method over the baseline (Zhen et al. 2020)? I can imagine having $L$ layers of memory to considerably increase the memory overhead. Additionally, due to the increase in the number of latent variables, is convergence slower as well?\n\n3. The VSM numbers shown in Table 5 are lower than what is reported in (Zhen et al. 2020). The numbers in (Zhen et al. 2020) show that the proposed method is inferior when compared to VSM on within-domain few shot classification. I understand that the authors re-implemented their model, but is there an explanation as to why the reimplemented numbers are considerably lower?\n\n", "summary_of_the_paper": "The authors propose a novel model that focuses on improving cross domain few shot classification. To this end they introduce a hierarchical extension to the work proposed in (Zhen et al. 2020), wherein the latent variables at different levels capture distinct semantic information. The proposed framework enables generating class specific prototypes at different hierarchical levels, which are then used to make predictions at each level. These predictions are ensembled using domain specific weights obtained from the support set via a gradient based method. Through experiments on various cross-domain and in-domain tasks, the authors show considerable improvements over the baselines.", "summary_of_the_review": "This work provides a logical extension to the existing work in (Zhen et al. 2020) by introducing a hierarchical variational memory framework. Through the experiment results it is evident that the proposed method provides considerable improvements over existing approaches. I have some concerns regarding the actual importance of dependencies within the latent variables. I'm still inclined to accept this paper, and would be willing to increase my rating if the authors address my concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "HQJZ-rJIg9g", "reviewer": "Reviewer_2Ajk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n- The paper is well-written and easy to follow. The paper structure is very clear. \n- The idea of leveraging multi-level semantic information is interesting for few-shot learning across domains.  Although using multi-level features is not something new in the space in general, using it as a condition for the prototypes is novel for few-shot learning.  It seems to be a good extension to the previous memory networks. \n\nConcerns\n- Would the proposed method with external memory significantly increase the model size?  The external memory might be something unfair to approaches without explicit external memory. \n- The baselines in the result tables (Table 4 and 5) may not be the current SODA methods. According to the public leader board https://paperswithcode.com/sota/few-shot-image-classification-on-mini-2. the `prototype completion for few-shot learning` (Zhang et al.) work achieves 69.68% +- 0.76% on the 5-way 1-shot experiment using ResNet-12, which is higher than the reported numbers in Table 5. \n- More ablation study in terms of model size (e.g., different backbones), training efficiency (e.g., how fast is the variational inference-based approach?). \n\n\n", "summary_of_the_paper": "This paper proposes a hierarchical memory to store features at different semantic levels for few-shot learning across domains. It introduces a hierarchical prototype model, where each level of the prototypes fetches the corresponding information from the hierarchical memory.  The authors follow the hyper network design to learn the weights when combining predictions from multiple levels.  The overall model is optimized using a variational inference framework. \n\nThe proposed method is evaluated on 4 various datasets which have a domain gap between the training data.  The authors also show that this method is competitive on the commonly used few-shot image classification benchmarks. \n\n", "summary_of_the_review": "The overall idea is interesting but there are concerns regarding the experiments as well as the introduction of external memory. \n\n\n=== Post-rebuttal Comments===\nThe authors addressed most of my concerns in the feedback.  I kept my score and learned towards acceptance. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BSMsyNbk06q", "reviewer": "Reviewer_YEXZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "From a technical standpoint, the work appears to be a relatively straightforward extension of [Zhen et al., 2020].  Hence, experimental validation of the impact of the proposed per-layer memory model is especially important.  The paper presents results on the same few-shot tasks as Zhen et al. (mini-Imagenet and tiered-Imagenet), as well as comparison to other meta-learning methods on few-shot cross-domain tasks.  Here, ablation experiments also show learned weighting of per-layer prototypes to be useful.\n\nHowever, the experimental results leave open a critical question about the comparison of the proposed approach to the baseline variational semantic memory (VSM) of Zhen et al.  Specifically, the results quoted for VSM in Table 5 are worse than the results in [Zhen et al., 2020] for this same experiment.  In fact, the results reported in Table 6 of [Zhen et al., 2020] are better in 3 out of 4 settings than the results reported for the proposed system.  The discrepancy is:\n\nMethod mini-ImageNet (1-shot/5-shot) tiered-ImageNet (1-shot/5-shot)\n\nVSM  65.72 82.73 72.01 86.77\n\nVSM* 64.21 79.69 69.58 83.28\n\nOurs 67.01 81.75 71.70 85.13\n\nwhere VSM is as reported in [Zhen et al., 2020], VSM* is the \"re-implementation\" by this submission, and Ours is the submission's approach.  Since the proposed system is an extension of Zhen et al., it is quite detrimental to actually perform worse than the baseline.  These results are also on a central experimental setting (few-show learning with deep models).  It is not acceptable to present a \"re-implementation\" that flips the ranking of the methods; at minimum, some extensive explanation is required about differences between the original and \"re-implementation\" as well as why the original published results were not replicated.\n", "summary_of_the_paper": "This paper presents a hierarchical version of the variational memory approach of [Zhen et al., 2020] for few-shot learning.  The core technical methodology follows that developed by Zhen et al., with the difference being in terms of the model: this work utilizes a deep model with per-layer memory, whereas Zhen et al. utilize a single memory for high-level concepts (i.e., deepest layer only).  Motivating the choice of per-layer memory is the desire to better handle few-shot learning tasks with domain shift, as the representations in earlier layers may be more relevant in such scenarios; the proposed framework learns a hypernetwork to predict attention weights over the per-layer prototypes.\n", "summary_of_the_review": "While this work appears to be a promising extension of the variational semantic memory of [Zhen et al., 2020] to models with multi-level prototypes, it omits proper comparison to [Zhen et al., 2020] on a central experiment, which, when included shows the proposed approach produces worse performance than the baseline.  The author response should address this discrepancy.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a hierarchical memory for cross domain and few shot classification problems. The paper is well written, tackles an important topic, and the proposed approach which is an extension of VSM is interesting. Reviewer YEXZ has some concerns regarding comparison to a more proper baseline. I believe that the authors have adequately addressed this. Reviewer 2Ajk and g1Bf also have suggestions that the authors have incorporated in the revision. I recommend accepting this paper.
Decision: Accept (Poster)