Meta-GMVAE: Mixture of Gaussian VAE for Unsupervised Meta-Learning
Authors: Dong Bok Lee
Dongchan Min
Seanie Lee
Sung Ju Hwang
Venue: ICLR 2021 Spotlight (2021)
Keywords: Unsupervised Learning
Meta-Learning
Unsupervised Meta-learning
Variational Autoencoders

Abstract: Unsupervised learning aims to learn meaningful representations from unlabeled data which can captures its intrinsic structure, that can be transferred to downstream tasks. Meta-learning, whose objective is to learn to generalize across tasks such that the learned model can rapidly adapt to a novel task, shares the spirit of unsupervised learning in that the both seek to learn more effective and efficient learning procedure than learning from scratch. The fundamental difference of the two is that the most meta-learning approaches are supervised, assuming full access to the labels. However, acquiring labeled dataset for meta-training not only is costly as it requires human efforts in labeling but also limits its applications to pre-defined task distributions. In this paper, we propose a principled unsupervised meta-learning model, namely Meta-GMVAE, based on Variational Autoencoder (VAE) and set-level variational inference. Moreover, we introduce a mixture of Gaussian (GMM) prior, assuming that each modality represents each class-concept in a randomly sampled episode, which we optimize with Expectation-Maximization (EM). Then, the learned model can be used for downstream few-shot classification tasks, where we obtain task-specific parameters by performing semi-supervised EM on the latent representations of the support and query set, and predict labels of the query set by computing aggregated posteriors. We validate our model on Omniglot and Mini-ImageNet datasets by evaluating its performance on downstream few-shot classification tasks. The results show that our model obtain impressive performance gains over existing unsupervised meta-learning baselines, even outperforming supervised MAML on a certain setting.

Published as a conference paper at ICLR 2021
META-GMVAE: MIXTURE OF GAUSSIAN VAES FOR UNSUPERVISED META-LEARNING
Dong Bok Lee', Dongchan Min', Seanie Lee', and Sung Ju Hwang 1.2 KAIST', AITRICS", South Korea
{markhi, 95 lsnfami Ly02, sjjwanng22)kaiist.ccs
ABSTRACT
Unsupervised learning aims to learn meaningful representations from unlabeled data which can capture its intrinsic structure, that can be transferred to down- stream tasks. Meta-learning, whose objective is to learn to generalize across tasks such that the learned model can rapidly adapt to novel task, shares the spirit unsupervised learning in that the both seek to learn more effective and effi- cient learning procedure than learning from scratch. The fundamental difference the two is that the most meta-learning approaches are supervised, assuming full access the labels. However, acquiring labeled dataset for meta-training not only is costly as it requires human efforts in labeling but also limits its ap- plications to pre-defined task distributions. In this paper, we propose princi- pled unsupervised meta-learning model, namely Meta-GMVAE, based on Varia- tional Autoencoder (VAE) and set-level variational inference. Moreover, we in- troduce mixture of Gaussian (GMM) prior, assuming that each modality repre- sents each class- concect a randomly sampled episode, which we optimize with Expectation-Me -ititattoonMMimizzatiion (EM). Then, the learned model can be used for down- stream few-shot classification tasks, where we obtain task-specific parameters by performing semi-supervised EM on the latent representations of the support and query set, and predict labels of the query set by computing aggregated posteriors. We validate our model on Omniglot and Mini- Mini--ImgeeNet datasets by evaluating its performance on downstream few-shot classification tasks. The results show that model obtains impressive performance gains existing unsupervised meta- learning baselines, even outperforming supervised MAML on a certain setting.
INTRODUCTION
Unsupervised learning is one of the most fundamental and challenging problems in machine learn- ing, due to the absence of target labels to guide the learning process. Thanks to the enormous research efforts, there exist many unsupervised learning methods that have shown promising results on real-world domains, including image recognition (Le) 2013) and natural language un- derstanding (Ramachandran et al.) (017). The essential goal of unsupervised learning is obtaining meaningful feature representations that best characterize the data, which can be later utilized to im- prove the performance of the downstream tasks, by training a supervised task-specific model on the top of the learned representations (Reed et al. 2014 Cheung et al.) 2015 Chen et al., 2016 or fine-tuning the entire pre-trained models (Erhan et al.) 2010).
Meta-learning, whose objective is to learn general knowledge across diverse tasks, such that the learned model can rapidly adapt novel tasks, shares the spirit of unsupervised learning in that both seek more efficient and effective learning procedure over learning from scratch. However, the essential difference between the two is that most meta-learning approaches have been built on the supervised learning scheme, and require numan-crafted task distributions to be applied in few- shot classification. Acquiring labeled dataset for meta-tras meta-training may require a massive amount of human efforts, and more importantly, meta-learning limits its applications to the pre-defined task distributions (e.g. classification of specific set classes).
Two recent works have proposed unsupervised neta-learning that can bridge the gap between unsu- pervised learning and meta-learning by focusing on constructing supervised tasks with pseudo-labels from the unlabeled data. To do so, CACTUs (Hsu al. 2019) clusters data in the embedding space

---

Published as a conference paper at ICLR 2021
Latent space
EM algorithm
Unlabeled examples
Labeled examples
Ma::ss s. J
A
N(.
12
N(a P3.0P. of)
(u) Unsupervised Mctu-training
(b) Supevised Mela-test
Figure I: During meta-training, Meta-GMVAE learns multi-modal latent space that can best explain the unla- beled data using EM algorithm. At meta-test time, semi-supervised EM map both the support (labeled data) and queries (unlabeled data) each mode learned during meta-training.
learned with several unsupervised learning methods, while UMTRA (Khodadadeh et al. 2019) sumed that each randomly drawn sample represents a different class and augmented each pseudo- class with data augmentation (Cubuk et al. [2018). After constructing the meta-training dataset with such heuristics, they simply apply supervised meta-learning algorithms as usual. Despite the suc- cess of the existing unsupervised meta-learning methods, they fundamentally limited, since 1) they only consider unsupervised learning for heuristic pseudo-labeling of unlabeled data, and 2) the two-stage approach makes it impossible recover from incorrect pseudo-class assignment when learning the unsupervised representation space.
In this paper, we propose a principled unsupervised meta-learning model based on Variational Au- toencoder (VAE) (Kingma & Welling 2014) and set-level variational inference using self self-attention (Vaswani et al., 2017). Moreover, we introduce multi-modal prior distributions, a mixture of Gaus- sians (GMM), assuming that each modality represents each class-concept in any given tasks. Then parameter of GMM optimized by running Expectation Minitiionn-uxiiizatioin (EM) on the observations sampled from the set-dependent variational posterior. In this framework, however, there is no guar- antec that each modality obtained from EM algorithm corresponds to label. To realize modality as label, we deploy supervisde EM at meta-test time, considering the support set and query set as labeled and unlabeled observations, respectively. We refer to our method as Meta -eeta-Gasssian Mixture Variational Autoencoders
(See Figure[fac for high-level concept). While our method can be used as a full generative model for generating the samples (images), the ability to general- ize generate samples may not be necessary for capturing the meta-knowledge for non-generative downstream tasks. Thus, we propose another version of Meta-GMVAE that reconstructs high-level features learned by unsupervised representation learning approaches (e.g. Chen al. (2020)).
To investigate the effectiveness of our framework, we run experiments on two benchmark few- shot image classification datasets, namely Omiglot (Lake et al. 2011) and Mini-Imagenet (Ravi & Larochelle 2017). The experimental results show that our Meta-Gin Geta-GMVAE obtains impressive per- formance gains over the relevant unsupervised meta-learning baselines on both datasets, obtaining even better accuracy than fully supervised MAML (Finn et al.) (017)) while utilizing as small as 0.1% of the labeled data on one-shot settings Omniglot dataset. Moreover, our model can gen- eralize to classification tasks with different number of ways (classes) without loss of accuracy. Our contribution is threefold:
We propose novel unsupervised meta-learning model, namely Meta-GMVAE, which meta- learns the set-conditioned prior and posterior network for VAE. Our Meta-GMVAE is princi- pled unsupervised meta-learning method, unlike existing methods on unsupervised meta-learning that combines heuristic pscudo-labeling with supervised meta-learning.
We propose to learn the multi-modal structure of a given dataset with the Gaussian mixture prior, such that it can adapt to novel dataset via the EM algorithm. This flexible adaptation new task, is not possible with existing methods that propose VAEs with Gaussian mixture priors for single task learning.
We show that Meta-GMVAE largely outperforms relevant unsupervised meta-learning baselines two benchmark datasets, while obtaining better performance than supervised meta- learning model under specific setting. We further show that Meta- Meta-GMVAE can generalize to classification tasks with different number of ways (classes).
2

---

Published as a conference paper at ICLR 2021
2 RELATED WORK
Unsupervised learning Many prior unsupervised learning methods have developed proxy objec-
tives which is either based on reconstruction (Vincent al. 2010, Higgins et al., 2017). adver-
sarially obtained image fidelity (Radford et al. 2016 Salimans et al. [0166] Donahue et al.
2017
Dumoulin al. 2017). disentanglement (Bengio et al. 2013 Reed et al. 2he4nC al. 2015
Chen et al., 2 Mathieu et al. 2016;; Denton & Birodkar 2017; Kim & Mith,2013 Ding et al.,
2020), clustering (Coates & Ng. 2012: Krähenbühl et al. 2016 [Bojanowsii & Joulin 2017]] Caron
et al. 2018), or contrastive learning (Chen et al.) 2020). In the unsupervised learning literature, the most relevant work to ours are methods that use Gaussian Mixture priors for variational autoen- coders. Dilokthanakul et al. (2016); Jiang et al. (2017) consider single task learning and therefore, the learned prior parameter is fixed after training. and thus cannot adapt to new tasks. CURL (Rao) et al. 2019) learns network that outputs Gaussian mixture priors over sequence of tasks for un- supervised continual learning. However CURL cannot adapt to a new task without training on it, while our framework can generalize a new task without any training, via amortized inference with a dataset (task) encoder. Also, our model does not learn Gaussian mixture priors but rather obtain them on the fly using the expectation expectation-maximizttiin
maximization algorithm.
Meta- Meta-learning Meta-Icarning (Thrun & Pratt 1998) shares the intuition of unsupervised learning in that it aims to improve the model performance on an unseen task by leveraging prior knowledge, rather than learning from scratch. While the literature on meta-learning is vast, we only discuss relevant existing works for few-shot image classification. Metric-based meta-learning (Koch et al.) 2015 Vinyals et al., 2016; [Snell et al.) 2017; Oreshkin et al. 20188) Mishra et al.) 2018) is one of the most popular approaches, where it learns to embed the data instances of the same class to be closer in the shared embedding space. One can measure the distance in the embedding space by cosine similarity (Vinyals et al., 2016). or Euclidean distance (Snell et al.) (017)) On the other hand, gradient- -badd metaalcarning (Finn al. 2017| 2018 Li al. 2017 & Choi 2018 Ravi & Beatson 2019: Flennerhag et al.) (020) aims at learning a global initialization of parameters, which rapidly adapt to novel task with only few gradient steps. Moreover, some previous works (Hewitt et al.) [2018: Edwards & Storkey| [017]] Garnelo et al. (018)) tackle meta-learning by modeling the set-dependent variational posterior with a single global latent variable, however, we model the variational posterior conditioned on each data instances. Moreover, while all of these works assume supervised learning scenarios where one has access to full labels in meta-training stage, we focus on unsupervised setting in this paper.
Unsupervised meta-learning One of the main limitations of conventional leaanne methods that their application is strictly limited to the tasks from a pre-defined task distribution. A few works (Hsu et al. 2019; Khodadadeh et al. 2019) have been proposed to resolve this issue by combining unsupervised learning with meta-learning. The main idea is to construct meta-training dataset in an unsupervised manner by leveraging existing supervised meta-learning models. CACTUs (Hsu et al. 2019) deploy several deep metrie learning (Berthelot et al. [019 Donahue et al. 2017: Caron ctt al. 2018 Chen et al.) (016)) to episodically cluster the unlabeled dataset, and then train MAML (Finn al. 2017) and Prototypical Networks (Snell et al.) 2017)
the constructed data. UMTRA Khodadadeh et al. 2019) assumes that each randomly drawn sample is from a different class from others, and use data augmentation (Cubuk et al. (0188) to construct synthetic task distribution for meta-training. Instead of only deploying unsupervised learning for constructing meta-training task distributions, we propose an unsupervised learning model that meta-learns set-level variational posterior by matching the modti-modal prior distribution representing latent classes.
3 UNSUPERVISED META- WITH META-GMVAES
In this section, we describe our problem setting with respect to unsupervised meta-learning, and demonstrate our approach. The graphical illustration of our model for unsupervised meta-training and supervised meta-test is depicted in Figure
3.1 PROBLEM STATEMENT
Our goal is to learn unsupervised feature representations which can be transferred to wide range of downstream few-shot classification tasks. As suggested by Hsu et al. (2019); Khodadadeh et al.)
3

---

Published as a conference paper at ICLR 2021
Y
X
Unsupervised Meta-training
(b) Supevised Meta-test
Figure 2: The graphical illustration of Meta-GMVAE. The dotted lines denote either variational inference or Expectation Maximization. (a): We introduce the multimodal distribution pv (z.(z) into prior distribution, and its optimal spec-fpe parameter oे, is obtained by EM in an episodic manner. (b): For meta-test, we obtain specifcc parameter oे by semi-supervised EM using Xx: y. and Xq.
2019), we only assume unlabeled dataset D.. = {x,] in the meta-training stage. We aim toward applying the knowledge learned during unsupervised meta-training stage to novel tasks in meta-test stage, which comes with modest amount of labeled data (or as few single example per class) for each task. As with most meta-learning methods, we further assume that the labeled data drawn from the same distribution Specifically, the goal of a K K-way S-shot classification task is to correctly predict the labels of query data points Q = {x using S support data points and labels = y,)]== per class, where S is relatively small (i.e. between I and 50).
that of the unlabeled data, with different set of classes.
3.2 MEVA-LEVEL GAUSSIAN MIXTURE VAE
Unsupervised meta-training We now describe the meta-learning framework for learning unsuper- vised latent representations that can transferred to human-designed few-shot image-llasiffcattin tasks. In particular, aim toward learning multi-modal latent spaces for Variational Autoen- coder (VAE) in an episodic manner. We use the Gaussian mixture for the prior distribution py (z) = > (y = k)ps (zly = k), where &b is the parameter of the prior network. Then the generative process can described as follows:
Ps (y), where corresponds to the categorical L.V. for single mode.
Pa (zly), where z corresponds to the Gaussian L.V. responsible for data generation.
(x where 0 is the parameter of the generative model.
The above generative process is similar to those from the previous works (Dilokthanakul et al,, 2016 Jiang al. 2017) modeling the VAE prior with Gaussian mixtures. However, they target single-task learning and the parameter of the prior network is fixed after training such as equation lc Dilokthanakul al. (2016) and equation 5 in Jiang et al. (2017). which is suboptimal since meta-learning model should be able to adapt and generalize to a novel task.
To learn the set-dependent multi multi-modalities, each episodic dataset D; {x, 1 which is randomly drawn from the unlabeled dataset D,, Then derive the variational lower bound for the marginal log- pikeeiilddood of D, follows:
further assume that there exists a parameter v for
log dogPo (D,)
log po (x,) Σ log po(xx)1,) po (z,) j=l
2o Xj. Di) dz,
(z, x,: D.)
[ (x [logpo(x,) 22,) ) log Pv. (2z) log (z X D.))1]
(2)
M
logpo(x,lz])) (m)) + log logpo, (z)))) - log ()) Di)
(3)
(n) lid
L( d, wh. D.).
(4)
Here the lower bound for each datapoint is approximated by Monte Carlo estimation with the sample size N. Following the convention of the VAE literature, we assume that the variational posterior (s(2; xx, D:) follows an isotropic Gaussian distribution.
4

---

Published as a conference paper at ICLR 2021
Algorithm 1 Meta-training
Algorithm 2 Meta-test for an episode
Require:
An unlabeled dataset D.,
Require: A test task = SUQ
I: Initialize parameters 8,
I: Set D [x.] 1 U {x,]
2: while not done do
2: Draw n MC samples from 2o (z, D)
3: Sample episode datasets {= from De
4: for all i E [1. B3] do
3: Initialize μA _ =
and 0 =
5: Draw v. MC samples from yo (x,22)2;, |xj, D.)
6: Initialize T.s as 1/ K and randomly choose
K
4: Compute optimal parameter wi using Eq 10
different points for fag-
5: Compute p(y, |x D) using Rq 11
7: Compute optimal parameter tbς using Eq 7
6: Infer the label yo = arg maxply, xx,, D)
8: end for
9: Update 0, Φ using C(0, o, (2)) in Eq
9
7:
10: end while
8:
Set-dependent variational posterior Our derivation of the evidence lower bound in Eq 4 is similar to that of the hierarchical VAE framework, such as equation 3 in Edwards & Storkey (2017) and equation 4 in Hewitt et al. (2118), in that we use the i.i.d assumption that the log likelihood of dataset equals the sum over the log-lik log-likelihoods of each individual data point. Yet, previous works assume that each input set consists of data instances from single concept (e.g. class), therefore, they encode the dataset into a single global latent variable (e.g. go (z(D)). This is not appropriate for unsupervised meta-learning where labels are unavailable. Thus we learn a set-conditioned vari- ational posterior (z, (x,, Di), which models latent variable to encode each data X, within the given dataset D, into the latent space. Specifically, we model the variational posterior go (xy, D,) using the self-attention mechanism (Vaswani et al. 2017) as follows:
H (ransformme(nncodee((()))
μ1 Wn H, + b of exp(W H, + br²) (0(2)|x, D.) = NN2;
(5)
Here we deploy TransformerEncooder/ -) a neural network based on the multi-head self-attention mechanism proposed by Vaswani et al. (017). to model the dependency between data instances, and
If is a convolutional neural network (or an identity function for the Mini-ImageNet) which takes each
data in D; an input. Moreover, we use the reparameterization trick (Kingma & Welling 2014) to
train the model with backpropagation since the stochastic sampling process 2, is non-differentiable
(n) 1.ddd (x( D;)
Expectation Maximization As discussed before, we assume that the parameter t/ of the prior Gaussian Mixture is task-specific and characterizes the given dataset D. To obtain the task-specific parameter that optimally explain the given dataset, we propose to locally maximize the lower bound in Eq 4 with respect to the prior parameter eh, We can obtain the optimal parameter vे by solving the following optimization problem:
M.N Σ
4 arg maxC((f the, Dz) arg max S
log (z (,) z
(n) fid go(z))z,, (6)
where we only consider the term related to the task-specific parameter vs. and eliminate the nor- malization term & since it does not change the solution of the optimization problem. The above formula implies that the optimal parameter maximizes the log- likelihood of observations which can be drawn from the variational posterior distribution. However, we do not have an analytic solution for Maximum Likelihood Estimation (MLE) of a GMM.
The most prevalent approach for estimating the parameters for the mixture of Gaussian is solving it with Expectation Maximization (EM) algorithm. To this end, we propose optimize the task- specific parameter of GMM prior distribution using EM algorithm as follows:
(z)(2) (n), MK. I)
(E-step)
Qj,(k))= == p(y = =(ke"")
En N (z) (1) 1k6 I)
Σ ENNN (k) (m)
ΣMA Qj,n
(M-step)
p::
Σ M.N. Qj.n (1))
Th =
Σk Eソ (k)n(k)
w1 = {Mu, I, 1,
5

---

Published as a conference paper at ICLR 2021
where Tk. Me. and N-) denote the mixing probability of k-thh component, mean parameter, and normal distribution, respectively. We assume that the covariance matrix of Gaussian distribution is fixed with the identity matrix I, following the assumption of original VAE the prior distribution. k=1 and {14x k== as K and randomly drawn K' different points, respectively. We obtain MLE solution for the parameters of GMM, by iteratively performing E-step and M-step until the log-likelihood converges, We found that using fixed number of iterations for the EM algorithm does not degrade the performance, and consider it as hyperparameter of our framework.
We initialize {
Training objective Note that we want to maximize the variational lower bound of the marginal log- likelihood over all the episode datasets D, that can be sampled from D.. We use stochastic gradient ascent with respect to the variational parameter ø and the generative parameter 8, to maximize the following objective:
B
C(0,0, (P.) =
1
Σ
max C(0, o, the Di)
(8)
B
1=1
M
1
1xgpo(x;12)))) + log 1 log (2))
|xj, D.)
B
Here we use B mini-batch of episode datasets, where each dataset consists of M datapoints. The task-specific parameter 3i for each episode dataset D; is obtained by EM algorithm in Eg
Supervised meta-test By introducing the multi-modal prior distribution into a generative learning framework, our model learns pseudo-clast concepts by clustering latent features with EM algorithm. However, there no guarantee that each modality obtained by EM algorithm corresponds to the label we are interested in at the meta-test stage. To realize modality as label in downstream few- shot image classification tasks, we deploy semi-supervised EM algorithm instead. Given a task T consisting of support set S {(x.. y,)) and query set Q = {x-]-m1 we use both the support set and query set as an episode dataset D {x.)s=] { 1 and draw latent variables from the variational posterior ( xx,, D). Note that we abbreviate the index since we consider a single task for now. We then perform semi-supervised EM algorithm as follows:
(E-step) Qg.((k) = ply(")) =k(z = N(z) pk o
ExN(z)) (m), sak. o
SNN 1 - =x + Q.N Q.N
(n)
(k)z
q,
Pk =
S
(1())
+ Lin= Qu,
(10)
(M-step)
(r) (2(")
)2 + Egne Qq (k)
(2)
(n)
=s,==
-S.N
-x1" +
> mm Qn
QQN
1 = (11 ox R 1)
where denotes an indicator function. We fix the mixing probability as K since the labels in each task are uniformly distributed. Moreover, we utilize diagonal covariance of to obtain more accurate statistics for the inference. We initialize μı and of the average value of support latent representations and the identity matrix I, respectively. Similar to the meta-training stage, we obtain the MLE solution for the parameters of GMM, by performing E-step and M-step for fixed number of iterations. Finally, we compute the conditional probability of ply, Xy. D) using the obtained parameters v as follows:
N
p(yglxg: D) = Egg D) [p (y 12))
1 N
Σ Pe- (2)))()),
iid go (o Xny D). (11)
R
Here we compute pvr (y, (z)"))) (n)) with Bayes rule, and we reuse different Monte Carlo samples that drawn for Eq [0] where the prediction query yr = arg maxply, = k|xg, D). We present the
pseudo-code of the algorithm for training and inference of
in the Algorithm T and

---

Published as a conference paper at ICLR 2021
Visual feature reconstruction While method is generative model that can generate samples from output distribution, the ability to generate samples may not be necessary for discriminative downstream tasks (Chen et al. 2020). Morcover, we found that VAEs almost fail to learn in Mini- ImageNet dataset with the architecturally limited constraints of the meta-learning literature. Thus, propose high-level feature reconstruction objective instead for Mini-ImageNet dataset. We experimentally find that the recently proposed constrastive learning framework, namely SimCLR (Chen al. 2020), is the most effective for our settings. Specifically, SimCLR learns high-level representation by performing constrastive prediction task on pairs of augmented examples derived from a minibatch. We train SimCLR on the unsupervised dataset Du = {xulum) 1 and use high-level features extracted by SimCLR as an input for our framework.
4 EXPERIMENT
In this section, we now validate the effectiveness of our Meta-GMVAE on several downstream few- shot classification tasks. The source codes are available at htttt:://ggtttaiiee cwm/cbobccc..... has//bbbeeee Lee/ Meta-
4.1 EXPERIMENTAL SETUPS
Baselines and ours We now describe two supervised meta-learning approaches which we consider as "oracles", unsupervised meta-lcarning baselines, and the proposed Meta-GMVAE. 1) MAML (oracle): Model Agnostic Meta Learning by Finn et al. (2017). We compare against its performance reported in Hsu et al. (2019). 2) ProtoNets (oracle): Euclidean distance-based meta-leand lett-nngnnnng ap- proach by [nell et al. 2017p. We also compare against it using its performance reported in |Hsu al.)
2019). 3) CACTUs: Clustering to Automatically Construct Tasks for Unsupervised meta-learning by Hsu et al. (201)). It automatically constructs tasks by clustering the unsupervised dataset in em- bedding space learned by ACAI (Berthelot et al. 2019)), BiGAN (Donahue et al.) (017), and Deep- Cluster (Caron et al. 2018). Then they train either MAML or ProtoNets using the cluster indices as pscudo-labels. 4) UMTRA: Unsupervised Meta- Meta-learning with Tasks constructed by Random sampling and Augmentation by Khhdadddeh et al. (2019). For constructing a K-way 1-shot task, it randomly samples K-way datapoints from unsupervised dataset and augments each datapoint. Then MAML is trained on the constructed tasks. 5) Meta-GMVARB Our proposed Meta-level Gaussian Mixture VAE. learns latent representation by matching set-level amortized variational posterior and task-specific multimodal prior optimized by EM algorithm.
Datasets We validate all the models on two benchmark datasets for few-shot classification. 1) Omniglot: This is a collection of 28 x 28 gray-scale hand-written characters that describe 1623 different alphabets, each of which contains 20 instances. Following the experimental setup of Hsu ett al. (201)) we use 1200 classes for unsupervised meta-training, 100 classes for meta- vela-valid and the remaining 323 classes for meta-test. We further augment each class by rotating the images 90, 180, and 270 degrees, such that the total number of classes is 1623 x 4, following the con- vention. 2) Neti- This is a subset of ILSVRC-2012 (Deng et al., 2009) introduced by Ravi & Larochelle (2017), consisting of 100 classes that comes with 600 images of size 84 x 84 that describe different instances. We use 64 classes for unsupervised meta- veeaa-raining, 16 classes for meta-validation, and the remaining 20 classes for meta-test, following the standard protocol.
Implementation details We now introduce the specific implementation details of Meta-GMVAE. on the standard
the two benchmark datasets. 1) Variational posterior network go (z|x,, D,): Conv4 architecture on Omniglot dataset for a fair comparison against relevant baselines. On top of the Conv4 architecture, we stack two TransformerEEcooder layers and an affine transformation layer to predict the mean and log-variance of Gaussian distribution. For Mini-ImageNee dataset, we only utilize two TransformerEncoderr layers and an affine transformation layer since the input used for Mini-Imagehi chitecture trained with SimCLR. For both datasets, we set the dimensionality of the latent variable to 64. 2) Generative network (x|z): For Omniglot dataset, the architecture of generative net- work is symmetric to the Conv4 architecture of variational posterior network. The last layer outputs the parameter of output Bernoulli distribution. For Mini-ImageNet dataset, we use 3-layer MLP with ReLU activation to predict the mean of output Gaussian distribution. 3) Other details: we utilize Adam optimizer (Kingma & Bal 2015) with constant learning rate of 0.001 and 0.0001 for Omniglot and MinilmageNett experiments, respectively. We set the number of iterations for EM algorithm as 10 for all the experiments. For the more details, please the Appendix.
is already a high-level visual representation extracted from the Conv5 ar-
7

---

Published as a conference paper at ICLR 2021
Omniglot (way, shot)
Mini-ImagNet (way, shot)
Method
Clustering
(5,1)
(5,5)
(20,1)
(20,5)
(5,1)
(5,5)
(5,20)
(5,50)
Training from Scratch
N/A
52,50
74.78
24.91
47.62
27.59
38.48
51.53
59.63
CACTUS-MAML
BiGAN
58.18
78.66
35.56
58.62
36.24
51.28
61.33
66.91
CACTUS- ProttoNess
BiGAN
54.74
71.69
33.40
50.62
36.62
50.16
59.56
63.27
CACTUS-MAML
ACAI/DC
68.84
87.78
48.09
73.36
39,90
53.97
63.84
69.64
CACTUS-ProtoNert
ACAI/DC
68.12
83.58
47.75
66.27
39.18
53.36
61.54
63.55
UMTRA
N/A
83.80
95.43
74.25
92.12
39.93
50.73
61.11
67.15
MeVa-GMVAE (ours)
N/A
94.92
97.09
82.21
90.61
42.82
55.73
63.14
68.26
MAML (oracle)
N/A
94.46
98.83
84.60
96.29
46.81
62.13
71.03
75.54
ProtoNets (oracle)
N/A
98.35
99.58
95.31
98.81
46.56
62.29
70.05
72.04
Table I: The few-shot classification results (way, shot) on the Omniglot and Mini-ImageNet datasets. DC denotes DeepCluster. We report the average of accuracies evaluated over 1000 episodes. All the values are based on the reported performance in Hsu et al. (2019) and Khodadadeh et al. 2019), except for ours.
2
A
X
X
X
8
1
B
m
6
X
X
X
X
M
P
0 c
8
H
in
F
Y
Y 1
Y
Y
D
1
7
P PMh
h B
A
1
1
4
A
A
(a) Meta-train (Real)
(b) Meta-train (Generated)
(c) Meta-test (Real)
(d) Meta-test (Generated)
Figure 3: The samples obtained and generated for each mode at unsupervised meta-training and supervised meta-test step of Meta-GMVAL Samples in each row in the same modality obtained by EM.
4.2 EXPERIMENTAL RESULTS
Few-shot classification Table T shows the few-shot classification results obtained by supervised meta-learning baselines (oracle), the two unsupervised meta-learning baselines, and our Meta- GMVAE. For the Omniglot dataset, the Meta-GMVAE outperforms all the baselines that only utilize desupervised-lamming for constructing meta- meta-training tasks, except for the UMTRA on the 20-shot Dieta-GMVAE-aG also outperforms baselines on Mini-ImageNi ImageNet 1-shot, and 5-
5-shot classification. shot settings which are the most widely used settings, while it matches the performance of baselines in 20-shot, and 50-shot settings. This shows that meta-learning the posterior network can capture the multi-modal distribution of any given tasks with Meta-GMVAF, is indeed more effective over unsupervised meta-learning baselines which simply trains supervised meta-learning models with pseudo-labels obtained from unlabeled data. Moreover, our Meta-GMVAE obtains better perfor- mance than supervised MAML on Omniglot 5-way 1-shot classification, while utilizing as small as 0.1% of the labeled data. This matches the observation in Chen et al. (2020) that well-calibrated unsupervised learning approaches with a modest amount of labels can obtain a performance compa- rable to even better than supervised approaches.
Visualization To better understand how our Meta-GMVAE learns and realizes class-concepts in few- shot classification tasks, we visualize the actual samples an episode classified by Meta-GMVAE and ones generated by generative network pe (x|z), during unsupervised meta-training and super- vised meta-testing. We visualize the actual samples and generated ones that have a same modality in same row. In Figure Bja, b, we can observe that our Meta-GMVAE captures the similar visual structure in each modality during vetaintaining, but the modalities are not the class-concepts. How- ever, as shown Figure 3 d, our Meta-GMVAR easily realizes each modality as each class-concept at meta-test time.
Ablation study Furthermore, we compare the performance of our model variants by eliminating each of the most important components for our model. We describe the each variant as follows: 1) LR (SimCLR): This performs the logistic regression using support set top of features pretrained by SimCLR. 2) Vanilla VAE: We train Vanilla VAE on D., and predict labels using semi semi-supervised EM with fixed identity covariance I. 3) Vanilla VAE (SimCLR): This is same 2) Vanilla VAE except that is trained on features pretrained by SimCLR. 4) Ep: Meta-GMVAE with an episodic training with task specific parameter vi obtained by EMM 5) Set: Meta-GMM whether having set-
8

---

Published as a conference paper at ICLR 2021
Method
Ep
Set
σ²
M
Training (way, shot)
Training From Scratch
24.91
27.59
Test way
(5, 1)
(5,
(20, 1)
(20, 5)
(SimCLR)
N/A
40.11
2 2-way
98.26
98.00
98.36
98.23
Vanilla VAE
69.68
N/A
Vanilla VAE (SimCLR)
N/A
38.40
5-way
94.92
94.57
93.93
94.01
10-way
89.87
89.99
89.10
89.30
78.64
40.51
15-way
85.11
85.12
85.36
85.33
81.65
41.13
Meta-GMVAN
80.94
40.92
20-way
81.38
81.11
82.21
81.98
82.21
42.82
30-way
77.80
77.42
78.40
77.24
40-way
73.76
73.15
74.03
73.56
MAML (oracle) ProtoNets (orucle)
84.60
46.81
95.31
46.56
50-way
70.92
70.85
69.86
70.02
Table 2: Left: The results of the ablation study on Meta-GMVAE (O: 20-way 1-shot classification on Om-
niglot, M: 5-way 1-shot classification on Mini-ImageNet) Right: The results of cross-way 1-shot experiments on Omniglot. The values in the parenthesis indicate that model is trained based on the (way, shot) setting.
level variational posterior (i.e. (z(z)x, D:)) or not (i.e. (=(z)x)). 6) a²: Meta-GMM performs semi-supervised EM algorithm whether using diagonal covariance matrix fixing it with identity matrix I. Table 2- shows that all the components we consider are critical for the performance on the few-shot classification tasks expected. The best performance gain comes from Ep, which supports our proposal on meta-learning the set-level variational posterior by matching it with the multi-modal prior, where the task task-specific parameter is obtained with EM.
Cross-Ny Cross-way classification We then experiment our Meta-GMVAE by varying the number of way (between 2 and 50) and fixing the number of shot 1. In particular, we set the number of component & as the Test way for the meta-test and perform semi-supervised EM algorithm in Eq 10. Table 2 Right shows that the differ- ence in the number way used for training and test does not significantly affect the perfor- mance, which demonstrates the robustness of Meta-GMVAE on varying number of way. We also visualize the latent space for the cross-shot experiment using t-SNE (Rauber et al. 2016). in Figure 4 which shows that Meta-GMVAB trained with 20-way can cluster 5-way meta-test task.
(a) 20-way Meta-training
(b) 5-way Meta-test
Figure 4: The visualization of latent space for cross-shot generalization experiment.
5 CONCLUSION
We proposed a novel unsupervised meta-learning model, namely Meta-GMVAE, which can gener- ate a task-dependent posterior for a given unseen task with multi-modal Gaussian Mixture priors. Given random episode that consists of samples from diverse classes, we optimize the task-specific parameter of the mixture of Gaussian prior with Expectation-Maxxiiiaatte each mode capture intrinsic groupings in the given data. We meta-train the variational posterior network over such data-driven prior obtained over large number of episodes. Then, at the meta- test step, we realize each modality with a label by deploying semi-supervised EM algorithm with both the support and the query set. We validate our method on two few-shot image classification benchmark datasets, and show that Meta- Veta-GMVAE largely outperforms the relevant unsupervised meta-learning baselines, even achieving better performance than supervised MAML on Omniglot 5-way I-shot experiments.
algorithm, such that
Acknowledgements This work was supported by Institute of Information & communications Technology Planning Evaluation (ITTP) grant funded by the Korea government (MSIT) (No.2019. 0-00075, Artificial Intelligence Graduate School Program (KAIST)). Samsung Research Funding Center Samsung Electronics (No. SRFC-ITI502-S1), Samsung Electronics (10201214-08) 145- 01), and the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF-201 8R ISAA1999211), We sincerely thank the anonymous reviewers for their constructive comments which helped us significantly im- prove our paper during the rebuttal period. We also appreciate D. Khuê Lē-Huu for the valuable discussion on Rao et al. 2019).

---

Published as a conference paper at ICLR 2021
REFERENCES
Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Representation learning; A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 35(8):1798-1828, 2013.
David Berthelot, Colin Raffel, Aurko Roy, and Ian Goodfellow. Understanding and improving interpolation in autoencoders via an adversarial regularizer. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA,, USA, May 6-9, 2019. OpenReview.net, 2019.
Piotr Bojanowski and Armand Joulin. Unsupervised learning by predicting noise. In Doina Pre- cup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML. 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp- 517-526. PMLR, 2017.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference, Munich, Ger- many, September 14, 2018, Proceedings, Part XIV, volume 11218 of Lecture Notes Computer Science, pp. 139-156. Springer, 2018.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. CoRR, abs/2002.05709, 2020.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 2172-2180, 2016.
Brian Cheung, Jesse A. Livezey, Arjun K. Bansal, and Bruno A. Olshausen. Discovering hidden fac- tors of variation in deep networks. In Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Work- shop Track Proceedings, 2015.
Adam Coates and Andrew Y. Ng. Learning feature representations with k-means. In Grégoire Mon- tavon, Genevieve B. Orr, and Klaus-Rohert Müller (eds.), Neural Networks: Tricks of the Trade Second Edition, volume 7700 of Lecture Notes in Computer Science, pp. 561-580. Springer, 2012.
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoe V. Le. Autoaugment: Learning augmentation policies from data. CoRR, ftos//805.00500 2018
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248-255. IEEE Computer Society, 2009.
Emily L. Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances Neural Information Pro- cessing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 4414-4423, 2017.
Nat Dilokthanakul, Pedro A. M. Mediano, Marta Garnelo, Matthew C. H. Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. CoRR, abs/161 111166111064, 2016.
Zheng Ding, Yifan Xu, Weijian Xu, Gaurav Parmar, Yang Yang, Max Welling, and Zhuowen Tu. Guided variational autoencoder for disentanglement learning. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp- 7917-7926. IEEE, 2020.
10

---

Published as a conference paper at ICLR 2021
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In 5th Interna- tional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martín Arjovsky, Olivier Mastropi- etro, and Aaron C. Courville. Adversarially learned inference. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Harrison Edwards and Amos J. Storkey. Towards a neural statistician. In 5th International Confer- ence on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Dumitru Erhan, Yoshua Bengio, Aaron C. Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? J. Mach. Learn. Res., 11: 625-660, 2010.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th Interna- tional Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1126-1133 PMLR, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic moodll-agnstte meta-learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa Dasa-Bianchi, and Ro- man Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, 3-8 December 2018, Montréal, Canada, pp. 9537-9548, 2018.
Sebastian Flennerhag, Andrei A. Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Had- sell. Meta-learning with warped gradient descent. 8th Internationall Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.nnt, 2020.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Teh. Neural processes. CoRR, abs/1807.00 01620, 2018
Luke B. Hewitt, Maxwell I. Nye, Andreea Gane, Tommi S. Jaakkola, and Joshua B. Tenenbaum. The variational homoencoder: Learning to learn high capacity generative models from few examples. In Amir Globerson and Ricardo Silva (eds.), Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence. UAI 2018, Monterey, California. USA, August 6-10, 2018, pp. 988-997. AUAI Press, 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In 5th International Conference on Learning Representattinns ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep em- bedding: An unsupervised and generative approach clustering. Carles Sierra (ed.), Proceed- ings of the Twenty-Sien SSxth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pp. 1965-1972, 2017.
Siavash Khodadadeh, Ladislau Bölöni, and Mubarak Shah. Unsupervised meta-learning for few- shot image classification. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buce Emily B. Fox, and Roman Garett (eds.), Advances in Neural Information Process- ing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 20132-10143 2019.
11

---

Published as a conference paper at ICLR 2021
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15. 2018, volume 80 of Proceedings of Machine Learning Research, pp. 2653-2663. PMLR, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In Proceedings of the 32th International Conference on Machine Learning, ICMI, 2015, 2015.
Philipp Krähenbühl, Carl Doersch, Jeff Donahue, and Trevor Darrell. Data-dependent initializations of convolutional neural networks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Brenden M. Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua B. Tenenbaum. One shot learning of simple visual concepts. In Laura Carlson, Christoph Hölscher, and Thomas F. Shipley (eds.), Proceedings of the 33th Annual Meeting of the Cognitive Science Society, CogSei 2011, Boston, 2011.
Massachusetts, USA, July 20-23, 201 cognittvesciencessciety.oom
Quoc V. Le. Building high-level features using large scale unsupervised learning. In IEEE Inter- national Conference on Acoustics, Speech and Signal Processing, ICASSP 2013, Vancouver, BC, Canada, May 26-31, 2013, pp. 8595-8598 IEEE, 2013.
Yoonho Lee and Seungjin Choi. Gradient-based meta-leaming with learned layerwise metric and subspace. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10- 15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 2933-2944 2942. PMLR, 2018.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few shot learning. CoRR, abs//107.00 1000000983,.000 2017.
Michaël Mathieu, Junbo Jake Zhao, Pablo Sprechmann, Aditya Ramesh, and Yann LeCun. Disen- tangling factors of variation in deep representation using adversarial training. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Gareett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Process- ing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 5041-5049, 2016.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Boris N. Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. TADAM: task dependent adaptive metric for improved few-shot learning. In Samy Bengio. Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò fion Processing Systems 31: Annual Conference Neural Information Processing Systems 2018, NeurlPS 2018, 3-8 December 2018, Montréal, Canada, pp. 719-729, 2018
and Roman Garnett (eds.), Advances in Neural Informa-
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Yoshua Bengio and Yann LeCun (eds.). 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
12

---

Published as a conference paper at ICLR 2021
Prajit Ramachandran, Peter J. Liu, and Quoc V. Le. Unsupervised pretraining for sequence to quence learning. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copen- hagen, Denmark, September 9-1 2017, pp. 10.383-391 Association for Computational I Linguistics, 2017.
Dushyant Rao, Francesco Visin, Andrei A. Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Had- sell. Continual unsupervised representation learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buss Buc, Emily B. Fox, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 7645-7655, 2019.
Paulo E. Rauber, Alexandre X. Falcão, and Alexandru C. Telea Visualizing time-dependent data using dynamic t-sne. In Enrico Bertini, Niklas Elmq and Thomas Wischgoll (eds.), 18th Euro- graphics Conference on Visualization, Euro Vis 2016 Short Papers, Groningen, The Netherlands, June 6-10, 2016, pp. 73-77. Eurographics Association, 2016.
Sachin Ravi and Alex Beatson. Amortized bayesian meta-learning. In 7th International Confer- ence on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In 5th Interna- tional Conference on Leurning Representations, ICLR 2017, Toulon, Frunce, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Scott E. Reed, Kihyuk Sohn, Yuting Zhang, and Honglak Lee. Learning to disentangle factors of variation with manifold interaction. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings. pp. 1431-1433 JMLR.org. 2014.
Oleh Rybkin, Kostas Danilidiss, and Sergey Levine. Simple and effective VAE training with cali- brated decoders. CoRR, abs/2006. 13202, 2020.
Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Anmual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 2234-2234, 2016
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4.9 December 2017, Long Beach, CA, USA, pp. 4077-4087, 2017.
Sebastian Thrun and Lorien Y. Pratt (eds.). Learning to Learn. Springer, 1998.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp- 5998-6008, 2017.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. J. Mach. Learn. Res., 1 11:3371-3408 2010.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Daniel D. Lec, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 3630-3638, 2016.
13

---

Published as a conference paper at ICLR 2021
A OMNIGLOT EXPERIMENTS
A.1 TRAINING PROCEDURE
Omniglot is collection of 28 28 gray-scale hand-written characters that describe 1623 different alphabets, cach of which contains 20 instances. Following the experimental setup of Hsu et al. (2099),
1200 classes for unsupervised meta-training. 100 classes for meta-validation and the remaining 323 classes for meta-test. We further augment each class by rotating the images 90, 180, and 270 degrees, such that the total number of classes is 1623 4, following the convention. We evaluate the trained model using 1000 randomly selected tasks from test set. During evaluation, K S data instances are used as support inputs and K 15 data instances are used query inputs. We use the Adam (Kingma & Ba 2015) optimizer with constant learning rate of 0.001 to train all models. All models are trained for 60,000 iterations. For the 5-way experiments (i.e. K 5), we set the mini-bater size, the number of datapoints, and Monte Carlo sample size 4, 200, and 32, respectively (i.e.
20), we set them as 4, 300, and 32 (i.e. B = 4, M 300, and N 32). We set the number of EM iterations as 10.
4, M 200, and N 32). For the 20-way experiments (i.e.
A.2 NETWORK ARCHITECTURE
We summarize the network architecture in the following Table and We assume that the out- put follows Bernouli distribution, therefore, the output generative network po(x(z) the mean parameter.
Set-level variational posterior network go (2(x), D.)
Output Size
Layers
28 x 64 x 14 x 14
Input Images
conv2d(3 x 3, stride 1, padding 1), BatchNorm22), ReLU, Maxpool(2 x 2, stride 2)
64 X 7 X 7
conv2d(3 x 3, stride 1, padding 1), BatchNorm22, ReLU, Maxpool(2 x 2. stride 2)
64 X 4
conv2d(3 stride 1. padding 1), BatchNorm22, ReLU, Maxpool(2 x 2, stride 2)
64 2
conv2d(3 stride 1. padding 1), BatchNorm22, ReLU, Maxpool(2 2, stride 2) Flatten TransfornerEnccoer((lness 256, dæ = 256, h = 4, ELU, LayerNorm = False)
256
256
256
TransfomeerEnccderrddess 256, de 256, h 4, ELU, LayerNorm Falsc) Linear(256, 64 x
64 x
Table 3: Set-level variational posterior network used for Omniglot dataset. We refer the hyperpa-
rameter notation of TransformerEncodeer to Vaswani al. 2017
Generative network po (x
Output Size
Layers
64
Latent code
Linear(64,256), ELU Lincar(256, 256), ELU Lincar(256, 256), ELU
256
256
64 x X
Unflatten
64 X x
deconv2d(1 4, stride padding 1), BatchNorm22,, ReLU
X x
deconv2d(3 3, stride 2, padding 1), BatchNorm22,, ReLU
64 x 14 x 14
deconv2d(1 x 4, stride 2, padding 1), BatchNorm21D, ReLU
1 28 x 28
deconv2d(4 4 x 4, stride 2. padding 1), Sigmoid
Table 4: Generative Network for px(x(2) for Omniglot dataset.
A.3 95% CONFIDENCE INTERVAL
We provide the standard errors of our model's performance at 95% confidence interval over 1000 episodes the Omniglot dataset in Table
14

---

Published as a conference paper at ICLR 2021
Omniglot
(5,1)
(5,5)
(20,1)
(20,5)
Meta-GMVAE 94.92 + 0.42
97.09 = 0.20
82.21 + 0.44
90.61 11 0.19
Table 5: The few-shot classification results (way, shot) with 95% confidence interval on the Omniglot.
B MINI-IMAGENET EXPERIMENTS
B.1 TRAINING PROCEDURE
Mini-ImageNet subset ILSVRC-2012 (Deng et al. 2009
introduced by Ravi Larochelle (017)) consisting of 100 classes that comes with 600 images of size 84 x 84 that describe different instances. We first train Conv5 feature extractor using SimCLR objective with temperature term = 0.5, on Mini-ImageNet unsupervised meta-training dataset. We train the feature extractor using Adam optimizer with learning rate of 0.0001 for 400 epochs. We use 64 classes for unsupervised meta-training, 16 classes for meta-validation, and the remaining 20 classes for meta-test, following the standard protocol. We evaluate the trained model using 1000 randomly selected tasks from test set. During evaluation, are used as query inputs. For all the experiments, we use the Adam (Kingma & Ba 2015) optimizer with a constant learning rate of 0,0001, and set the mini-batch size, the number datapoints, and Monte Carlo sample size as 16, 5, and 256, respectively (i.e. B = 16, M = 5, and N 256) for the 5, and 20-shot experiments. For the 50-shot experiment, we them 4, 200, and 256, respectively (i.e. B =.M 200, and N = 256). We train the models for 5K, 10K, 15k, 25K, and 30K for 1, 5, 20, and 50-shot experiments, respectively. We the number of EM iterations 10.
S data instances used support inputs and 5 x 15 data instances
B.2 NETWORK ARCHITECTURE
We summarize the network architecture the following Table output follows Gaussian distribution, therefore, the output of generative network (x(x)z) is the mean parameter. Moreover, the variance output Gaussian distribution is obtained as suggested in Rybkin et al. (2020).
We assume that the
Feature Extractor for SimCLR
Output Size
Layers
3 x 84 x 84
Input Images
64 x 42 x 42
conv2d(3 x 3, stride 1, padding 1), BatchNorm22, ReLU, Maxpool(2 x 2, stride 2)
64 x 21 21
conv2d(3 3, stride 1, padding 1). BatchNorm22, ReLU, Maxpool(2 x 2. stride 2)
64 x x 10
conv2d(3 x stride 1. padding 1), BatchNorm2D, ReLU, Maxpool(2 2, stride 2)
64 x 5
conv2d(3 x 3, stride 1, padding 1), BatchNorm22, ReLUU, Maxpool 2, stride 2)
64 2
conv2d(3 x 3, stride 1, padding 1), BatchNorm22, ReLUU, Maxpool(2 x 2, stride 2)
256
Flatten
Table 6: Feature Extractor trained on Mini- Mini-ImageNet dataset using SimCLR objective.
Set-level variational posterior network qo (2|x, D:)
Output Size
Layers
256
Input Features
256
TransformerEnccddrrdddaaa. 256, da = 256, h 4, ReLU, LayerNorm = False)
256
TransforeerEhcoderrdd... 256, da 256, 4. ReLU, LayerNorm False) Linear(256, 64 2)
61
Table 7: Set-level variational posterior network used for Mini- IinieNm dataset. We refer the hy- perparameter notation of TranssormerEncoder to Vaswani al. (2017).
15

---

Published as a conference paper at ICLR 2021
Generative network (x/2)
Output Size Layers
64
Latent code
512
Linear(64, 512), ReLU
512
Linear(512. 512). ReLU
256
Linear(512, 256). ReLU
Table 8: Generative Network for po (x|2) for Mini-ImageNNer dataset.
B.3 95% CONFIDENCE INTERVAL
We provide the standard errors at 95% confidence interval over 1000 episodes on the Mini-ImageNett dataset Table 9
Mini-Image-Net
(5,1)
(5,5)
(20,1)
(20,5)
Meta-GMVAE 42.82 + 0.56
55.73 + 0.48
63.14 + 0.47
68.26 + 0.42
Table 9: The few-shot classification results (way, shot) with 95% confidence interval on the Mini- Mini-ImageNet.
B.4 ADDITIONAL. COMPARISON USING SIMCLR
To further understand where the improvement of Meta-GMVAE on the Mini-ImageNett dataset, we experiment baselines with SimCLR pretrained features. For CACTUs, we cluster in the em- bedding space pretrained by SimCLR. For UMTRA, we follow the exact same procedure to generate training episode, which is proposed by the authors. Moreover, tures as the setting of Meta-GMVAE for the both baselines. Table shows that Meta-GMVAE outperforms the baselines with SimCLR, which supports the effectiveness of Meta-GMVAE com- bined with SimCLR pretrained features.
fix the pretrained SimCLR fea-
Mini-ImageNett
(5,1)
(5,5)
(20,1)
(20,5)
CACTUS-MAML (SimCLR)
40.39
52.35
61.09
64.89
UMTRA (SimCLR)
40.85
51.47
61.08
67.30
Meta-GMVAE
42.82
55.73
63.14
68.26
Table 10: The comparison on the few-shot classification results (way. shot) using SimCLR.
16


Reviews:
[{"review_id": "ye_j3FA3Jd", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper goal is to learn unsupervised feature representations that can be transferred between few-shot classification tasks.  The paper models the class-concepts with a Mixture of Gaussians prior, and uses Variational Autoencoders to model the latent representations between the tasks and the samples.  \n\nThe presentation is clear and straightforward.  The idea is to use a GMM and use an Expectation-Maximization (EM) approach to learn the mixture.  To tackle the intractability of the variational posterior $q_\\phi (z_j | x_j, \\mathcal{D}_i)$, the paper proposes to use a Monte Carlo approximation.  For the meta-test, the model is tuned using EM in a semi-supervised fashion.\n\nThe experiments show the superiority of the Meta-GMVAE and the compared methods on the Omniglot and Mini-ImageNet datasets.  \n\nNevertheless, I find the idea simple, yet compelling.  The idea of adding GMM to enhance the modeling capabilities is a well known fact, and that has been explored before.  For instance, some recent publications applying the GMM idea (not that the final application and overall implementation may differ from meta-learning---see my comment below):\n- Dilokthanakul et al., Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders, https://arxiv.org/abs/1611.02648\n- Zhao et al., Truncated Gaussian-Mixture Variational AutoEncoder, https://arxiv.org/abs/1902.03717\n- Guo et al., Variational Autoencoder With Optimizing Gaussian Mixture Model Priors, 10.1109/ACCESS.2020.2977671.\n- Yang et al., Deep Clustering by Gaussian Mixture Variational Autoencoders with Graph Embedding, 10.1109/ICCV.2019.00654\n\nIt seems from the presentation that the main difference is the application to the meta-learning approach.  The authors should explain better what the contribution is and how it contrast to the existing literature of mixture models applied in variational modeling.\n\nPros:\n- Simple and effective idea.\n- Use of well known methods with simple approximators.\n- Good results on the presented experiments.\n\nCons:\n- The contribution is not clear.  I'm on the fence of whether the usage of the GMM to a new task is enough to guarantee a publication.\n\nOverall rating: ~~I'm giving a 5 due to the lack of clarity in the contribution and added novelty.  However, the presentation is good, and the explanations are clear.~~\nI'm updating my rating to accept the paper due to the comments and updates on the paper.  The proposed flexible usage of the GMM is novel from the existing literature.  The changes in the paper improved its clarity, and the contribution is better presented in contrast to existing work.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "iNbqyKVecne", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission proposes an algorithm for the semi-supervised meta-learning (unsupervised meta-training + supervised meta-testing) setting of [1], which adapts the few-shot learning + evaluation setting of [2, 3] by omitting classification labels at meta-training time. The algorithm makes use of a variational auto-encoder (VAE) formulation defined over a hierarchical model that describes the decomposition of a dataset into tasks of datapoint-target pairs (i.e., the meta-learning setup). The prior distribution of the hierarchical VAE is taken to be a mixture of Gaussians to facilitate the construction of pseudo-labels at meta-training time. The algorithm is evaluated on the Omniglot and miniImageNet few-shot classification tasks (with labels unused at meta-training time).\n\n##### Strengths: \n\nThe semi-supervised meta-learning (unsupervised meta-training + supervised meta-testing) setting is interesting and worthy of study as an analogue of unsupervised learning. The datasets used in the empirical evaluation are appropriate, although they do not represent the most complex image datasets used in few-shot classification evaluations (cf. meta-dataset [8]).\n\nThe use of a Gaussian mixture model (GMM) for the prior distribution of a variational auto-encoder (VAE), which allows an analytic solution for a subset of the variational parameters, is conceptually interesting, although its use would not be restricted to the meta-learning setting. Using it to construct pseudo-labels (as well as incorporate labels when available) for the semi-supervised meta-learning setting is a nice development, although not a significant advance from the use of $k$-means in CACTUs.\n\nMeta-GMVAE attaints higher performance than semi-supervised few-shot classification setting comparison methods (CACTUs & UMTRA) on the Omniglot and miniImageNet benchmarks; moreover, it approaches the level of a supervised few-shot classification method (MAML) on the Omniglot benchmark (although this supervised comparator does not represent state-of-the-art performance on this benchmark).\n\n##### Weaknesses: \n\n1) **Clarity**: The algorithmic components of the submission were very difficult to get straight. In the development of the algorithm in Section 3.2, the submission does not adequately discuss why and how particular subcomponents are employed, and various points about the different algorithmic components are made in the text without sufficient explanation or integration; some examples are:\n\n    - The VAE formulation is introduced without precedent just above equation (1). It is also a bit of a red herring because it is not subsequently used, as is, in the algorithm.\n\n    - At \"The difference of our model from original VAE is that we utilize a set-level variational posterior $q_\\phi(\\mathbf{z}_j |\\mathbf{x}_j , D_i)$, for inferring isotropic Gaussian distribution, to encode characteristics of a given dataset $D_i$. Specifically, we utilize self-attention mechanism (Vaswani et al., 2017) on top of a convolutional neural network.\" This is the first time an \"isotropic Gaussian distribution\" is mentioned in the method, self-attention is not explained further, and there is no explanation of how the convolutional neural network (CNN) fits into the whole framework. For example, it is not clear from this section whether (and if so, how) a CNN is used in addition to the SimCLR feature representation.\n\n    - \"...we set the prior distribution as a mixture of Gaussians (GMM), where $y$ is a discrete random variable indicating the component of a latent variable $\\mathbf{z}$\". $y$ and $\\mathbf{z}$ are not yet defined except by reference to the VAE formulation in (1), but that was insufficiently explained as a part of the algorithm.\n\n    More specific details for reproducibility are not described in the text (e.g., how the GMM parameters are initialized for EM; what some of the variables ($\\mathbf{z}$, $\\mathbf{x}$, $\\phi$, $\\theta$) refer to in the implementation). On top of this, results would be extremely difficult to reproduce: While component architectures and experimental setups are detailed in the appendix, how everything fits together is not adequately described. \n\n    More broadly, the submission would benefit significantly from an algorithm box to convey how all the components interact and which components act episodically (at the task level) vs. at the level of the entire dataset.\n\n2) **Quality**: The experimental evaluation does not provide a measure of variance (e.g., 95% confidence interval) in Table 1, which should be provided to ascertain the significance of the reported improvement.\n\n  The algorithm uses the SimCLR representation learning objective to pre-train the feature extractor, while the comparison semi-supervised meta-learning approach use less performative methods as feature extractors (CACTUs: BiGAN, ACAI/DC; UMTRA: a simple, 4-layer CNN). An ablation study that ablates the use of SimCLR with Meta-GMVAE is necessary to ascertain whether the improvement is due to using SimCLR vs. using components attributable to Meta-GMVAE.\n\n3) **Originality**: Highly relevant work on GMM priors for VAEs is not cited in the submission: [4, 5]. The submission also does not discuss variations on the VAE that address the meta-learning setting (e.g., [6, 7]), which also demonstrate how the VAE formulation in (1) derives from a hierarchical model (cf. the non-hierarchical model on which the original VAE formulation is based).\n\n##### Minor points:\n\nThere are errors in reproducing the results from [1] in Table 2 of the submission (some percentages are incorrect); these errors do not affect the ranking of comparisons.\n\n##### References:\n\n[1] [Hsu, Kyle, Sergey Levine, and Chelsea Finn. \"Unsupervised learning via meta-learning.\" In ICLR, 2019.](https://arxiv.org/abs/1810.02334)\n\n[2] [Vinyals, Oriol, Charles Blundell, Timothy Lillicrap, and Daan Wierstra. \"Matching networks for one-shot learning.\" In Advances in neural information processing systems, pp. 3630-3638. 2016.](http://papers.nips.cc/paper/6385-matching-networks-for-one-shot-learning)\n\n[3] [Ravi, Sachin, and Hugo Larochelle. \"Optimization as a model for few-shot learning.\" In ICLR, 2017.](https://openreview.net/pdf?id=rJY0-Kcll)\n\n[4] [Dilokthanakul, Nat, Pedro AM Mediano, Marta Garnelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. \"Deep unsupervised clustering with gaussian mixture variational autoencoders.\" arXiv preprint arXiv:1611.02648 (2016).](https://arxiv.org/abs/1611.02648)\n\n[5] [Jiang, Zhuxi, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. \"Variational deep embedding: An unsupervised and generative approach to clustering.\" In IJCAI, 2017.](https://arxiv.org/abs/1611.05148)\n\n[6] [Hewitt, Luke B., Maxwell I. Nye, Andreea Gane, Tommi Jaakkola, and Joshua B. Tenenbaum. \"The variational homoencoder: Learning to learn high capacity generative models from few examples.\" In UAI, 2018.](https://arxiv.org/abs/1807.08919)\n\n[7] [Garnelo, Marta, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Eslami, and Yee Whye Teh. \"Neural processes.\" In ICML, 2018.](https://arxiv.org/abs/1807.01622)\n\n[8] [Triantafillou, Eleni, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin et al. \"Meta-dataset: A dataset of datasets for learning to learn from few examples.\" In ICML, 2020.](https://arxiv.org/abs/1903.03096)", "extra_scores": {}, "extra_text": {}}, {"review_id": "XMUe5m5XMA3", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for unsupervised meta-learning based on using a variational autoencoder (VAE). The variational autoencoder model they use differs from the typical one in that it considers episode-specific datasets, where the approximate posterior can be computed as a function of the set (using transformer architecture) rather than using an individual example. Additionally, they use a mixture of Gaussian distribution as a prior, whose parameters are learned per-episode using the EM algorithm. For the supervised evaluation phase, in order to adapt the learned prior to the few-shot dataset setting, semi-supervised EM is run using both support and query sets to adapt the mixture of Gaussian distribution to the evaluation dataset. Then, the query set predictions are obtained using the learned prior and posterior from the VAE model. Experimental evaluation is conducted on the Omniglot and Mini-ImageNet benchmarks and the proposed method is compared against other unsupervised meta-learning methods, mainly CACTUs and UMTRA. An interesting aspect about the Mini-ImageNet experiments are that because learning the VAE directly for this high-dimensional data may be difficult, the authors use features from a SimCLR-trained model as input for their VAE model. The proposed method seems to perform favorably across both of the benchmarks when varying the number of \"shots\". \n\nPros\n* Whereas previous work in unsupervised meta-learning involved creating unsupervised episodes for meta-training (via augmentations or clustering of unsupervised model features), this paper takes a very different route but still seems to achieve very good performance.\n* The authors were able to scale their model to the Mini-ImageNet dataset by using SimCLR-trained features and with this choice, the final model attains good performance on the benchmark compared to previous work.\n\nCons\n* This is a not necessarily a big con but a point that could be clarified further. How are the number of components for the GMM decided for meta-training? How does the choice of the number of components impact how the GMM is used at evaluation-time? Would it not pose an issue that during training we may have more/less components than are actually necessary at evaluation-time depending on the number of classes we are considering at evaluation-time? Is it the case that a separate model needs to be trained if number of evaluation classes is changed i.e. from 1-shot, 5-class to 1-shot, 10-class?\n* I believe the paper could be improved by adding an algorithm description of how exactly the model is trained and how evaluation takes place in terms of exact steps The algorithmic pseudocode can reference equations within the paper but I believe this would greatly help in terms of understanding how to recreate the exact training and evaluation procedure for the proposed model.", "extra_scores": {}, "extra_text": {}}, {"review_id": "As3dQ4cEIj", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The problem which the authors attempt to solve is unsupervised meta-learning (UML), ie. learning in an unsupervised way such a model of a dataset, as to be able to perform meta-learning (here: few-shot classification) later. I see their contribution as two-fold:\n1. Proposing a framework for solving UML consisting of sampling subsets $D_i$ of a full dataset $D_u$, training a generative model based on both datapoints ($x_j$) themselves and the particular subset $D_i$ and using it in a semi-supervised fashion.\n2. Implementing a model in this framework based on a VAE. Here, the latent variable $z$ doesn't just compress information about a datapoint (as in a classical VAE), but is also able to encode (in an abstract way) the position of this datapoint in the subset $D_i$ (ie. \"task-specific label\"). To be able to capture this (arguably richer than in classical VAEs) distribution, authors use a GMM to model the variational distribution.\n\nBecause MLE of GMM is intractable, authors have a two stage optimization process:\na) Finding a task-specific (ie. encoding info about $D_i$ \"classes\") parameter $\\phi^*$ via EM and b) Optimizing the ELBO given $\\phi^*$ as usual. During meta-testing, the $\\phi^*$ parameter is estimated in a similar way using the test-time samples $x_i$ (trying to embed the new task into the learned manifold) and then latent variable $z$ is sampled conditionally based on $x_i$ and the expected value of the constructed distribution $p_{\\phi^*}(y|z)$ estimated via Monte Carlo.\n\n1. While I am neither a VAE expert nor enthusiast, I consider the proposed model principled: while the two-stage optimization mechanism is not ideal (as may make it harder to optimize compared to end-to-end differentiable models), learning a single distribution describing both elements we care about: images and their placement within a dataset seem to match the problem better than previous pseudo-labels-based methods.\n2. I particularly like introduction of the general framework (1.) (which is not emphasized in the paper). I believe that it should be possible (not necessarily straight-forwardly) to extend the proposed model to other generative models. To make it clear, I wouldn't expect this extension from the paper under review (what I'm proposing is basically yet another paper), but the opening of this direction of research is a big plus.\n3. Paper is easy to understand.\n4. The presented results, while competitive compared to the previous UML SOTA, are only presented on somewhat toyish problems (Omniglot, mini-imagenet). While it is understandable that it'll be hard to train a Meta-GMVAE on more complex datasets (as it's only harder than classic VAEs, which are already struggling with higher-dimensional tasks), presenting the results only on small datasets (even if this is the current SOTA and other methods do it) somewhat undermines the overall motivation to UML: to be able to use vast amounts of unstructured data while building ML models.\n5. I am not able to comment on the novelty of the work: I am barely aware of the contemporary VAE/UML literature. I will be willing to modify my score based on other reviewers' opinions in that regard.\n\nQuestion/proposal:\nIn Sec. 3.2. authors write \"assuming that the modalities in prior distribution represent class-concepts of any datasets\". Why would this be the case? This seems intuitive; I feel like there could be a nice theoretical argument why it would be the case.\n\nI find the model principled and new. It solves an important problem in a natural way, improving over SOTA and opening the potential for follow-up research. I weakly question the use of VAEs, which feels like it is limiting the method (making hi-dim UML impossible), but am aware that is more of a complaint against a well-established research domain than the contribution of this paper itself.\n\nTypos:\n1. Abstract:\n... from unlabeled data which can capture ...\n... shares the spirit of unsupervised learning in that they both seek ...\n2. Sec. 1:\n... effectiveness of our framework, we run experiments on ...\n3. Sec. 2:\n... One of the main limitations of ...\n4. Sec. 3.2.:\n... inferring isotropic Gaussian distribution, to encode ...\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses a method for unsupervised meta-learning where a VAE with Gaussian mixture prior is used and set-level inference, taking episode-specific dataset as input, is performed to calculate its posterior. In the meta-testing phase, semi-supervised learning with the learned VAE is used to fast adapt to few-show learning. Reviewers are satisfied with the author responses, agreeing that the method is a principled way to tackle unsupervised meta-learning. 

Decision: Accept (Spotlight)