Unsupervised Meta-Learning through Latent-Space Interpolation in Generative Models
Authors: Siavash Khodadadeh
Sharare Zehtabian
Saeed Vahidian
Weijia Wang
Bill Lin
Ladislau Boloni
Venue: ICLR 2021 Poster (2021)
Keywords: Meta-learning
Unsupervised learning
GANs

Abstract: Several recently proposed unsupervised meta-learning approaches rely on synthetic meta-tasks created using techniques such as random selection, clustering and/or augmentation. In this work, we describe a novel approach that generates meta-tasks using generative models. The proposed family of algorithms generate pairs of in-class and out-of-class samples from the latent space in a principled way, allowing us to create synthetic classes forming the training and validation data of a meta-task. We find that the proposed approach, LAtent Space Interpolation Unsupervised Meta-learning (LASIUM), outperforms or is competitive with current unsupervised learning baselines on few-shot classification tasks on the most widely used benchmark datasets. 

Published as a conference paper at ICLR 2021
UNSUPERVISED META-LEARNING THROUGH LATENT- SPACE INTERPOLATION IN GENERATIVE MODELS
Siavash Khodadadeh
Sharare Zehtabian"
Saeed Vahidian²
Weijia Wang²
Bill Lin?
Ladislau Bölöni!
Dept. of Computer Science University of Central Florida
Dept. of Electrical & Computer Engineoring University of California San Diego
ABSTRACT
Several recently proposed unsupervised meta-learning approaches rely on synthetic meta-tasks created using techniques such as random selection, clustering and/or augmentation. this work, we describe a novel approach that generates meta- tasks using generative models. The proposed family of algorithms generate pairs of in-class and out-of-class samples from the latent space in a principled way, allowing us to create synthetic classes forming the training and validation data a meta-task. We find that the proposed approach, LAtent Space Interpolation Unsupervised Meta-learning (LASIUM), outperforms is competitive with current unsupervised learning baselines on few-shot classification tasks on the most widely used benchmark datasets.
INTRODUCTION
Few-shot meta-learning algorithms for neural networks such as Mishra et al. (2018); Finn et al. (0017): [Seell et al. (2017) prepare networks quickly adapt to unseen tasks. This is done meta-training phase that typically involves a large number of supervised learning tasks. Generating supervised tasks depends on large, labeled datasets and hand- -and-specified task distribution. Very recently, several approaches had been proposed that perform the meta-training by generating synthetic training tasks from an unsupervised dataset. This requires us to generate samples with specific pairwise information: in-class pairs of samples that are with high likelihood in the same class, and out-of-class pairs that are with high likelihood not in the same class. For instance, UMTRA by Khodadadeh et al. (2019) and AAL by Antoniou & Storkey (2019) achieve this through random selection from domain with many classes for out-of-class pairs and by augmentation for in-class pairs. |Hsu et al. (2019) propose CACTUs that creates synthetic labels through unsupervised clustering of the domain.
In this paper, we rely on recent advances in the field of generative models, such as the variants of generative adversarial networks (GANs) and variational autoencoders (VAEs), to generate the in-class and out-of-class pairs of meta-training data. The fundamental idea of our approach is that in-class pairs are close while out-of-class pairs are far away in the latent space representation of the generative model. Thus, we can generate in-class pairs by interpolating between two out-of-class samples in the latent space and choosing interpolation ratios that put the new sample close one of the objects. From this latent sample, the generative model creates the new in-class object. Our approach requires little domain- we need to choose thresholds for latent space distance that ensure that classes are in different domains, as well as interpolation ratio thresholds that ensure that the sample is in the same class as the nearest edge. Another ad advantage of the approach is that we can utilize off- sfff,, pre-trained generative models. The main contributions of this paper can be summarized as follows:
tweaking, and the necessary tweaks are numan-comprehensitle For instance,
"Equal contribution. Correspondence ssaavash.hoodadane sharare.7chtabian @ @knightssucff.ddn

---

Published as a conference paper at ICLR 2021
We describe algorithm. LAtent Space Interpolation Unsupervised Meta- Meta-learning (LA- SIUM), that creates training data for a downstream meta-learning algorithm starting from an unlabeled dataset by leveraging interpolation in the latent space of generative model.
We show that on the most widely used few-shot learning datasets, LASIUM outperforms or performs competitively with other unsupervised meta-learning algorithms, significantly out- performs transfer learning in all cases, and a number cases approaches the performance of supervised meta-learning algorithms.
2 RELATED WORK
Meta-learning or "learning learn" in the field of neural networks is an umbrella term that covers a variety of techniques that involve training neural network over the course of a meta-training phase, such that when presented with the target task, the network is able learn it much more efficiently than a randomly initialized network would. Such techniques had been proposed since the 1980s (Schmidhuber) (1987); [Beniio et al. (9900); Naik & Mammone (1992); Thrun & Pratt (1998). In recent years, meta-learning has gained a resurgence, through approaches that either "learn to optimize" (Finn et al. (2017); Ravi & Larochelle M0s6)r et al. (2017); Nichol et al. (2018)
Rusu al. (2019) Rajeswaran et al. (2019) or learn embedding functions
non-parametric
setting (Snell et al. (017); Vinyals et al. (2016); Ren et al. (2018); Liu et al. (2019). ). Hybrids
between these two approaches had also been proposed (Triantafillou et al.
Wang et al. (2019))
Most approaches use labeled data during the meta-learning phase. While in some domains there an abundance of labeled datasets, in many domains such labeled data is difficult to acquire. Unsupervised meta-learning approaches aim to learn from an unsupervised dataset from a domain similar from that of the target task. Typically these approaches generate synthetic few-shot learning tasks for the meta-learning phase through variety of techniques. CACTUs (Hsu et al. (2019)) uses a progressive clustering method. UMTRA (Khodadadeh et al.) (2019) utilizes the statistical diversity properties and domain-specine augmentations to generate synthetic training and validation data. AAL (Antoniou Storkey| (2019) uses augmentation of the unlabeled training set to generate the validation data. The accuracy of these approaches was shown to be comparable with but lower than supervised meta-learning approaches, though with the advantage of requiring orders of magnitude less labeled training data. A common weakness of these approaches is that the techniques used to generate the synthetic tasks (clustering, augmentation, random sampling) are highly domain dependent.
Our proposed approach, LASIUM, takes advantage of generative models trained on the specific domain to create the in-class and out-of-class pairs of meta-training data. By creating new training data through interpolation between training samples, LASIUM-OC has similarities with mixup Zhang eet al. (2017). The most successful neural-network based generative models in recent years are variational autoencoders (VAE) (Diederik & Welling (2014) and generative adversarial networks (GANs) (Goodfellow et al. 20144). The implementation variants of the LASIUM algorithm described in this paper rely on the original VAE model and on two specific variations of the GAN concept, respectively. MSGAN (aka Miss-GAN) (Mao et al. (2119) aims to solve the missing mode problem of conditional GANs through a regularization term that maximizes the distance between the generated images with respect to the distance between their corresponding input latent codes. Progressive GAINS (Karras et al. (0018) are growing both the generator and discriminator progressively, and approach resembling the layer-wise training of autoencoders.
3 LATENT SPACE INTERPOLATION UNSUPERVISED META-LEARNING
Preliminaries: We define an N-way, (ttr supervised classification task, T, as a set Difer) composed of i E (I,..., Nx K k(tr)) data points (Ia, yt) such that there are exactly k(er) samples for each categorical label y E f1....,N] NN. During meta-learning, an additional set DDfuat) is attached
to each task that contains another x (vval) data points separate from the ones D(tr). We have exactly (vval) samples for each class in D(pal) well.
2

---

Published as a conference paper at ICLR 2021
Sampling
(b) In-dlass sampling
g
(c) Task geneeration
(())
5 (z))
9 (x2²)
King
K K
Figure 1: 3-way, -h(tr)-shot task generation with ((val) images for validation by pre-trained GAN generator G. a) Sample 3 random vectors. b) Generate new vectors by one of the proposed in-class sampling strategies. e) Generate images from all of the latent vectors and put them into train and validation set to construct a task. The images in this figure have been generated by our algorithm. The colored edge of each image indicates that it was generated from its corresponding latent vector.
It is straightforward to package N-way, (((tr) tasks with D( and Divaa) from a labeled dataset. However, in unsupervised meta-learning setting, a key challenge is how to automatically construct tasks from the unlabeled dataset U {ri).
3.1 GENERATING META-TASKS USING GENERATIVE, MODELS
We have seen that in order generate the training data for the meta-learning phase, we need to generate N N-way training tasks with (trr) training and K( validation samples. The label associated with the classes in these tasks is not relevant, as it will be discarded after the meta- learning phase. Our objective is simply generate samples of the type æi.j with i {1 {1... N) and
j E {1 k(tr) + K((aa)) with the following properties: (a) all the samples æi,5 are different (b) any two samples with the same index are in-class samples and (c) any two samples with different i index are out-of-class samples. the absence of human provided labels, the class structure of the domain is defined only implicitly by the sample selection procedure. Previous approaches to unsupervised meta-learning chose samples directly from the training data Ki, E U4, or created new samples through define the class structure of the domain by assuming that certain
augmentation. For instance, types of augmentations keep the samples in-class with the original sample. One challenge of such approaches is that the choice of the augmentation is domain dependent, and the augmentation itself can be a complex mathematical operation.
In this paper, approach the sample selection problem differently. Instead of sampling øi,j from U we use the unsupervised dataset to train a generative model p(x)). Generative models represent the full probability distribution of model, and allow us sample new instances from the distribution. For many models, this sampling process can be computationally expensive iterative process. Many successful neural network based generative models use the reparamettizzaion trick for the training and sampling which concentrate the random component of the model in a latent representation z. By choosing the latent representation z from simple (uniform or normal) distribution, we can obtain a sample from the complex distribution p(x) by passing through a deterministic generator (( + X. Two of the most popular generative models, variational autoencoders (VAEs) and generative adversarial networks (GANs) follow this model.
The idea of the LASIUM algorithm is that given a generator ((),, nearby latent space values 21 and 22 map to in-class samples Iι and Z2 that belong to the same class. Conversely, and 22 values that are far away from each other, map to out of class samples that belong to different classes. Naturally, still need to define what mean by "near" and "far" in the latent space and how choose the corresponding z values. However, this is a significantly simpler task than, for instance, defining the set of complex augmentations that might retain class membership.
Training a generative model Our method for generating meta-taske is agnostic to the choice of training algorithm for the generative model and can either VAE or GAN with the only
3

---

Published as a conference paper at ICLR 2021
(a) Sampling
Encoding
(c) in-class sampling
(d) Task generation
S
o
d
b
>
d/
(z)
K
K(vai
Figure 2: 3-way, -shtt task generation by VAE on Omniglot dataset with K(eal) images for validation set each task. a) Sample 3 images from dataset. b) Encode the images into latent space and check if they are distanced. e) Use proposed in-class sampling techniques generate new latent vectors. d) Generate images from the latent vectors and put them alongside with sampled images from step into train and validation set to construct task.
Algorithm 1: LASIUM for unsupervised meta-learning task generation
Unlabeled dataset U/ {æ;}. Pre-trained generator G require: : N: class-count, MB: meta-batch size, e; minimum threshold, a or as sampling
require:
hyperparameters
require: Lequire:Smmping Sampling Strategy S: could be N(o²), RO(a), or OC(a)
require: ((rr), K ((val); number of samples for train and validation during meta-learning B = [) // meta-batch of tasks
for in 1, ... NB do
Sample N class-vectors in latent space of G and add them to task-vectors until they least units away from each other in euclidean space
for w I,.... K(tr)_ + (val) do
Generate new-vectors S(class-vectors w) and add them to testors end
Generate N X (K((t)) + K(val) images by feeding task-vectors to generator G Construct task T, by putting the first N ((tr) images in task train set and the last x K (wal) images task validation set
B - BUT
10 end
return B
constraint of having appropriately structured latent space. In our VAE experiments, we used network trained with the standard VAE training algorithm (Diederik & Welling (2014)). For the experiments with GANs used two different methods mode seeking GANs (MSGAN) (Mao et al. (2019) and progressive growing of GANs (ProGAN) (Karras et al. (018)) MSGAN is trained for Omniglot and ProGAN trained for CelebA.
Algorithm I describes the steps of our method. We will delve into each step in the following parts of this section.
Sampling out of class instances from the latent space representation: Our sampling techniques differ slightly whether we are using a GAN or VAE. For GAN, we use rejection sampling to find N latent space vectors that are at pairwise distance of at least threshold see Figure When using a VAF, we also have an encoder network that allows us to map from the domain to the latent space. Taking advantage of this, and embed them into a latent space. the latent space representation of these N images are too close to each other, re- re-sample, otherwise we use the N images and their representations and continue the following steps exactly the same as GANs see Figure ((a) and (b). We will refer to the vectors selected here anchor vectors.
additionally sample data points from our unlabeled dataset
Generating in-class latent space vectors Next, having N sampled anchor vectors {z)..... 2 from the latent space representation, we aim generate N new vectors (z/s-.-; 2/y from the latent
4

---

in-class sample anchor Published as a conference paper at ICLR 2021
LASIUM-N
LASIUM-RO
LASIUM-OC
out-of-class
out-of-ciass
in-class
in-class
in In-class
out-of-class
validation
validation
train
validation
Figure 3: Latent space representation visualization of proposed strategies for generating in-class candidates. Left: LASIUM-N, adding random noise to the sample vector. Middle: LASIUM-RR interpolate with random out-of-class samples. Right: LASIUM-OC, interpolate with other classes' samples.
space representation such that the generated image (zz)) belongs to the same class as the one of (22)) for i E 1,..., N. This process needs to be repeated for (((rr) + K((al) - 1 times.
The sampling strategy takes as input the sampled vectors and number w E {1 K((*) (((+))++ (ratt) -1} and returns N new vectors such that 2; and z/ are an in-class pair for E (T..NN) This ensures that no two z! belong to the same class and creates N groups of (K((r) K K[(vl)) vectors in our latent space. We feed these vectors to our generator to get N groups (K(tr) + K(val) images. From each group we pick the first ((trr for Df and the last ((val) for D(rat)
What remains to define the strategy to sample the individual in-class vectors. We propose three different sampling strategies, all of which can be seen as variations of the idea of latent space interpolation sampling. This motivates the name of the algorithm LAtent Space Interpolation Unsupervised Meta-learning (LASIUM).
LASIUM-N (adding Noise): This technique generates in-class samples by adding Gaussian noise to the anchor vector z 2; +e where (O, o²) (see Figure Leftt. In the context of LASIUM, we can see this as an interpolation between the anchor vector and a noise vector, with the interpolation factor determined by σ. For the impact of different choices of see the ablation study in the supplemental material.
LASIUM-RO (with Random Out-of-class samples) To generate vector 2; we first find a random out-of-class sample U; and choose an interpolated version closer the anchor: z/ 24 + a X (w; - 2;) (see Figure Middle... Here, a is hyperparameter, which can be tuned to define the size of the class. As we are in a comparatively high-dimensional latent space (in our case, 512 dimensions), we need relatively large values of a, such a 0.4 define classes of reasonable size. This model effectively allows us to define complex augmentations (such as person seen without glasses, in a changed lighting) with only one scalar hyperparameter to tune. By interpolating towards another sample we ensure that we are staying the manifold that defines the dataset (in the case of Figure this being human faces).
LASIUM-OC (with Other Classes' samples) This technique is similar to LASIUM-RO, but instead of using a randomly generated out-of-class vector, we are interpolating towards vectors already chosen from the other classes in the same task (see Figure Rigght. This limits the selection of the samples to be confined to subspace the convex hull containing the initial anchor points. The intuition behind this approach is that choosing the samples this way focuses the attention of the meta-learner towards the hard to distinguish samples that are between the classes the few shot learning class (eg. they share certain attributes).
5

---

Published as a conference paper at ICLR 2021
Finally, failure cases exist for these sampling strategies. For example, if 21, 22, and 23 lie line, we end up generating poor out of class samples that belong to other classes in our task. However, as we are in high dimensional latent space(e.g. 512), the likelihood for three points being approximately colliniar is very low (as the volume fraction in which the middle point would have to be shrinks exponentially with the dimensionality). Note that meta-learning algorithms fraction of bad training examples in the meta-training phase (e.g. see Hsu et al. (2019); Khodadadch al. (2019).
not sensitive small
4 EXPERIMENTAL VALIDATION
We tested the proposed algorithms on three few-shot learning benchmarks: (a) the 5-way Om- niglot (Lake et al. (2011)), a benchmark for few-shot handwritten character recognition, (b) the 5-way CelebA few-shot identity recognition, and (c) the CelebA attributes dataset (Liu et al. (2015) proposed few-shot learning benchmark by (Hsu al. (2019)) The latter benchmark comprises binary classification (2-way) tasks in which each task is defined by selecting 3 different attributes and 3 boolean values corresponding to each attribute. Every image in certain task-specific class has the same attributes with each other while does not share any of these attributes with images in the other class.
We partition each dataset into meta-training, meta-validation, and meta-testing splits between classes. To evaluate our method, we use the classes in the test set to generate 1000 tasks as described in section 3 We set ((val) to be 15. Note that unlike meta-training, where we use synthetic generated tasks, for evaluation we use 1000 tasks from real unseen labeled data. Furthermore, we fix the random seed make sure we compare on the exact same tasks with other baselines. We average the accuracy on all tasks and report a 95% confidence interval. To ensure that comparisons are fair, we use the same random seed in the whole task generation process. For the Omniglot dataset, we report the results for (trr) E [1,5), and ((val) 15. For CelebA identity recognition, we report results for k((rr) E 51,5, 15) and ((val) 15. For CelebA attributes, we follow the K((t)) = 5 and K(val)= tasks as proposed by Hsu al. 2019). More ablation studies over the hyperparameters and result visualizations are provided in the supplemental material. Since excessive tuning of hyperparameters can lead to the overestimation of the performance of model (Oliver et al.) 2018 ), we keep the hyperparameters of the unsupervised meta-learning as constant as possible (including the MAML, and ProtoNets model architectures) in all experiments. Our model architecture consists of four stacked convolutional blocks. Each block comprises 64 filters that carry out 3 x 3 convolutions, followed by batch normalization, a Rel.U non-linearity, and 2 2 max-pooling. For the MAML experiments, classification is performed by fully connected layer, whereas for the ProtoNets model we compute distances based on the feature vectors produced by the last convolution module without any dense layers. The input size to our model is 84 x 84 x 3 for CelebA and 28 28 I for Omniglot. The detail of the neural networks architectures for each experiment is described in the supplemental material.
4.1 BASELINES
As baseline algorithms for our approach we follow the practice of recent papers in the unsupervised meta-learning literature. The simplest baseline to train the same network architecture from scratch with N x (trr) images. More advanced baselines can be obtained by learning an unsupervised
embedding on 24 and use it for downstream task training. We used the ACAI (Berthelot et al. (2019) ).
BiGAN (Donahue et al. (2017); Dumoulin al. (2077), and DeepCluster (Caron et al. (2018)
as representative of the unsupervised learning literature. On top of these embeddings, we report accuracy for nen-net neighbors, linear classifier, multi layer perceptron (MIP) with dropout, and cluster matching.
The direct competition for our approach are the current state-of-the-arr algorithms in unsupervised meta-learning. We compare our results with CAMT (2019) and UMTRA Khodadadeh et al. (2019). Finally, it is useful to compare our approach with algorithms that require supervised data. We include results for supervised standard transfer learning from VGG19 pre-trained on ImageNet (Simonyan & Zisserman (2013)) and two supervised meta-learning algorithms, MAMI. (Finn et al. (2017)) and ProtoNets (Snell et al. (017)).
(Hsu et al.

---

Published as a conference paper at ICLR 2021
Table I: Accuracy results on the Omniglot dataset averaged over 1000, 5-way, -hot downstream tasks with ((vai) 15 for each task. + indicates the 95% confidence interval. The top three
unsupervised results are reported in bold. The baseline results are from Hsu al. (2019) section 4.1.
Algorithm
Feature Extractor
K(*r) =1
k =
Training from scratch
N/A
51.64 = 0.65
71.44 + 0.53
K-nearest neighbors
ACAI
57.46 + 1.35
81.16
Linear Classifier
ACAI
61.08 - 1.32
81.82 0.58
MLP with dropout
ACAI
51.95 + 0.82
77.20 0.65
Cluster matching
ACAI
54.91 0.85
71.09
K-nearest neighbors
BiGAN
19.55 + 1.27
68.06
Linear Classifier
BiGAN
48,28 1.25
0.66
MLP with dropout
BIGAN
40.54 0.79
62.56 0.79
Cluster matching
BIGAN
43.96 = 0.80
58.62 - 0.78
CACTUS-MAML
BiGAN
58.18 1 0.81
78.66 0.65
CACTUS-MAML
ACAI
68.84 + 0.80
87.78 1: 0.50
UMTRA-MAML
N/A
81.91 1 0.58
94.58 1: 0.25
LASIUM-RO SEIUM-RO-GANAAN MO-GAA--AML
N/A
83.26 + 0.55
95.29 0.22
LASIUM-N --AIMM--------MA- VAE-MAML
N/A
76.11 0.64
94.42 + 0.26
PACTUS-ProtoNee
BiGAN
54.74 0.82
71.69 0.73
CACTUs-Protokes
ACAI
68.12 0.84
83.58 0.61
LASIUM-RO -E---RROO----CCCCECE
N/A
80.15 0.64
91.10 + 0.35
AASTUM-OC-VVAE-Prooooo
N/A
73.22 + 0.73
85.05 + 0.46
Transfer Learning (VGG-19)
N/A
54.49 0.90
89.57 0.44
Supervised MAML
N/A
94.46 - 0.35
98.83 +
Supervised ProtoNets
N/A
98.35 + 0.22
99.58 0.09
4.2 RESULTS ON OMNIGLOT
Table shows the results the Omniglot dataset. We find that the LASIUM-RO-GAN-MAMLL configuration outperforms all the unsupervised approaches, including the meta-learning based ones like CACTUs and UMTRA. Beyond the increase in performance, we must note that the (one- dimensional) search for the intra-class shift was much cheaper than fine-tuning the currently popular augmentation strategies. We also find that this benchmark, LASIUM outperforms transfer learning using the much larger VGG-19 19 network.
As expected even the best LASIUM result is worse than the supervised meta-learning models. However, we need to consider that the unsupervised meta-lcarning approaches use several orders of magnitude less labels. For instance, the 95.29% accuracy of MASIUMMRO-GAN--MAI was obtained with only 25 labels, while the supervised approaches used 25,000.
4.3 RESULTS ON CELEBA
Table shows our results on the CelebA identity recognition tasks where the objective is to recognize N different people given (tr) images for each. We find that on this benchmark well, the LASIUM- RO-GAN-MAME configuration performs better than other unsupervised leta-learning models as well as transfer learning with VGG-19 only falls slightly behind LASIUM-RO COOM--RO-ANNN GAN-ProtoNets on the one-shot case. As we have discussed in the case of Omniglot results, the performance remains lower then the supervised meta-learning approaches which use several orders of magnitude more labeled data.
Finally, Table 3 shows our results for CelebA attributes benchmark introduced (Hsu et al. (019)). A peculiarity of this dataset is that the way in which classes are defined based the attributes, the classes are unbalanced in the dataset, making the job of synthetic task selection more difficult. We find that LASIUM-N L-NNN--NNMANN GAN-MAML obtains performance of 75.07 07 + 1.08, within the confidence interval of the second best, CACTUs MAML with BiGAN 74.98 + 1.02. In this benchmark, transfer learning with the VGG-19 network performed better than all unsupervised meta-learning approaches,
7

---

Published as a conference paper at ICLR 2021
Table 2: Accuracy results of unsupervised learning on CelebA for different unsupervised methods. The results averaged over 1000, 5-way, sh(rr)-hot downstream tasks with ( 15 for each task. + indicates the 95% confidence interval. The top three unsupervised results are reported in bold.
Algorithm
K((*)) =1
K( =
<( = 15
Training from scratch
34.69 0.50
56.50 0.55
70.56 + 0.49
CACTUs
+.64
62.71 = 0.57
74.18 + 0.68
UMTRA
39.30 + 0.59
60.44 0.56
72.41 + 0.48
43.88 1 0.57
66.98 1 0.53
78.13 1 0.44
LASIUM-RO-VAE-MAN
41.25 1122:100 0.57
58.22 0.54
71.05 + 0.49
SELUM-RO-GAN-PrUn
44.39 0.61
60.83 + 0.58
66.66 + 0.53
LASIUM RO- Prt-Prrtosees
43.22 = 0.58
61.12 = 0.54
68.51 = 0.51
Transfer Learning (VGG-19)
33.28 0.57
58.74 0.62
74.04 + 0.49
Supervised MAML Supervised ProtoNets
85.46 + 0.55
94.98 = 0.25
96.18 = 0.19
84.17 + 0.61
90.84 + 0.38
90.85 = 0.36
Table 3: Results on CelebA attributes benchmark 2-way, 5-shot tasks with K((vat) 5. The results are averaged over 1000 downstream tasks and + indicates 95% confidence interval. The top three unsupervised results are reported bold. The bascline results are from Hsu et al. (2019) section 4.1.
Algorithm
Feature Extractor
Accuracy
Training from scratch
N/A
63.19 + 1.06
K-nearest neighbors
BiGAN
56.15 = 0.89
Linear Classifier
BiGAN
58.44 = 0.90
MLP with dropout
BiGAN
56.26 + 0.94
Cluster matching
BiGAN
56.20 + 1.00
K-nearest neighbors
DeepCluster
61.47 + 0.99
Linear Classifier
DeepCluster
59.57 = 0.98
MLP with dropout
DeepCluster
60.65 = 0.98
Cluster matching
DeepCluster
51.51 + 0.89
CACTUS MAML
BiGAN
74.98 1:: 1.02
CACTUs MAML
DeepCluster
73.79 1 1.01
N/A
75.07 1 1.08
CACTUs ProtoNets
BiGAN
65.58 +:: 1.04
CACTUs ProtoNets
DeepCluster
74.15 +::01 1.02
MASIUM--NGAAN-ProtooNe
N/A
73.41 +:: 1.10
Transfer Learning (VGG-19)
N/A
79.76 + 1.03
Supervised MAML.
N/A
87.10 + 0.85
Supervised ProtoNets
N/A
85.13 + 0.92
possibly due to existing representations of the discriminating attributes in that much more complex network.
4.4 RESULTS ON MINI-IMAGENET
Table shows the comparison results on the Mini- ImageNet benchmark. Mini-ImageNNet has large sample fidelity among the datasets considered in this paper, and it is hard for generative models to capture this diversity given a relatively small dataset such as Mini-ImageNNet train set. Thus, we used as the GAN pre-trained BigBiGAN! trained on the whole Imagenet dataset with supervision. Examples of meta- vecta-training tasks constructed by LASIUM-N with o² = 1.0 are shown in Figure 4 outperforms all other unsupervised learning algorithms for K=1, 5 and 20. For K=50, it is in the second place behind CACTUS MAML with DeepCluster, the accuracy difference being within the margin of error.
We notice that
8

---

Published as a conference paper at ICLR 2021
Train
Validation
Figure Meta-training tasks for Mini- Aani-mm constructed by LASIUM-N with o² 1.0
Table 4: Results on Mini-Imagent benchmark for 5-way, shot tasks with K(val) 15. The results are averaged over 1000 downstream tasks and = indicates 95% confidence interval. The top three unsupervised results reported in bold. The baselines from Hsu et al. (2019) section 4.1. Note that the BigBiGAN trained on the unlabeled Imagenet dataset which is larger than Mini-ImageNet training set that used by Hsu et al. (2019) for training.
Algorithm
Embedding
K
(( =5
K(t)) =
K(t*) =0
Training from scratch
N/A
27.59 +: 0.59
38.48 + 0.66
51.53 + 0.72
59.63
K-nearest neighbors
BiGAN
25.56 + 1.08
31.10 0.63
37.31 0.40
03.60
Linear Classifier
BiGAN
27.08 = 1.24
14.00=0.4:
50.41=0.37
MLP with dropout
BIGAN
22.91 = 0.54
19.06+0.66
4.70
48.36:0.71
Cluster matching
BIGAN
24.63 = 0.56
29.49 0.58
33.89
36.13
K-nearest neighhors
DeepCluster
28.90 + 1.25
42.25 0.67
0.44
63.90+0038
Linear Classifier
DeepCluster
29.44 + 1.22
39.79 0.64
0.44
65.28+00.4
MIPP with dropout
DeepCluster
29.03 + 0.61
39.67 + 0.69
52.71
00.95++0.63
Cluster matching
DeepCluster
22.20 = 0.50
23.50 1 0.52
24.97 + 0.54
20.87
CACTUs MAML
BiGAN
36.24 = 0.74
51.28 + 0.68
61.33 + 0.67
66.91
CACTUs MAML
DeepCluster
39.90 = 0.74
53.97 + 0.70
63.84 -3.84=00 0.70
69.64 + 0.63
UMTRA MAML
N/A
39.93
50.73
61.11
67.15
LASIUM-N-GAN-
N/A
40.19 + 0.58
54.56 1 0.55
65.17
0.49
69.13 0.49
MAML.
CACTUs ProtoNets
BiGAN
36.62 + 0.70
50.16 + 0.73
59.56 + 0.68
0.62
CACTUs ProtoNets
DeepCluster
39.18 + 0.71
53.36-10.70
61.54+0.68
03.6
LASIUM-N-GAN-
N/A
10.05-t0.60
52.53 + 0.51
59.45 + 0.48
61. 61.13+0.45
ProtoNets
Transfer Learning
N/A
44.06 + 0.66
70.11 + 0.67
86.12 0.36
92.67
Supervised MAMI.
N/A
46.81 + 0.77
62.13 + 0.72
71.03 0.69
75.54+0.62
Supervised ProtoNets
N/A
46.56 + 0.76
62.29 0.71
70.05 - 0.65
72.0410.60
DISCUSSION
We deseribed LASIUM, unsupervised meta-learning algorithm for few-shot classification. The algorithm creates synthetic meta-tasks using interpolation in the latent space of generative model with the general idea that points that are close in the latent space will likely generate in-class samples, while points far in the latent space will likely generate out-of-class samples. In this, LASIUM differs from techniques such as UMTRA and AAL that generate in-class samples through pixel- space augmentation and techniques such as CACTUs which uses clustering to separate unsupervised training data into in-class and out-of-class samples. We found that LASIUM outperforms or comes within the margin of error to state-of-the ohethe--atttt art unsupervised algorithms Omniglot, Mini Imageee,,,,et, CelebA attributes learning benchmark and the CelebA identity recognition benchmarks.
ACKNOWLEDCEMENTS
This work had been in part supported by the National Science Foundation under Grant Number I10-14,,,,3

---

Published as a conference paper at ICLR 2021
REFERENCES
Antreas Antoniou and Amos Storkey. Assume, augment and learn: Unsupervised few-shot meta- learning via random labels and data augmentation. arXiv preprint 1xXiv:1922.0988:1 2019.
Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning a synaptic learning rule. 1990.
David Berthelot, Colin Raffel, Aurko Roy, and lan Goodfellow. Understanding and improving inter- polation in autoencoders via adversarial regularizer. In Int'l Conf. on Learning Representations (ICLR), 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning visual features. In Proc. of the European Conf. on Computer Vision (ECCV), pp. 132-149, 2018.
P Kingma Diederik and Max Welling. Auto-encoding variational bayes. In Proc. of the Int'l Conf. on Learning Representations (ICLR), volume 1, 2014.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In Int'l Conf. on Learning Representations (ICLR), 2017.
Vincent Dumoulin, Ishmael Belghazi. Ben Poole, Olivier Mastropictro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned inference. In Int'l Conf. on Learning Representations (ICLR), 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agmostic meta-lcarning for fast adaptation of deep networks. Proc. of Int'l Conf. on Machine Learning (ICML). pp. 1126-1135, 2017.
Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), pp. 2672-2680, 2014.
Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning. In Int'l Conf. on Learning Representations (ICLR), 2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. Proc. of the Int'l Conf. on Learning Representations (ICLR), 2018
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In Proc. of the IEEE/CVE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
Siavash Khodadadeh, Ladislau Bölöni, and Mubarak Shah. Unsupervised meta-learning for few- shot image classification. In Advances in Neural Information Processing Systems (NeurIPS), pp. 10132-10142, 2019.
Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Proc. of the Annual Meeting of the Cognitive Science Society, volume 33, 2011.
Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang, Sungju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. In Int'l Conf. on Learning Representations (ICLR), 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proc. of Int'l Conf. on Computer Vision (ICCV), December 2015.
Qi Mao, Hsin- Hsin-Ying Lee, Hung- Yu Tseng, Siwei Ma, and Ming-Hsuan Yang. Mode seeking generative adversarial networks for diverse image synthesis. Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, pp. 1437, 2019.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen. and Pieter Abbeel. Meta-learning with temporal convolutions. arXiv preprint arv: 707.0314,, 2017.
10

---

Published as a conference paper at ICLR 2021
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A Simple Neural Attentive Meta-Learner. Int'l Conf. on Learning Representations (ICLR), 2018.
Devang K Naik and Richard J Mammone. Meta-neural networks that learn by learning. In [Proc. 1992] JCNN Int'l Joint Conf. on Neural Networks, volume pp. 437-442, 1992.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arvi 803.02999 2018.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In Advances Neural Information Processing Systems (NeurIPS), pp. 2235-3246, 2018.
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. In Advances in Neural Information Processing Systems (NeurIPS), pp. 113-124, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. Int'l Conf. on Learning Representations (ICLR), 2016.
Mengye Ren, Sachin Ravi, Eleni Triantafillou, Jake Snell, Kevin Swersky, Josh B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-Learning for Semi- Supervise Few-Shot Classification. In Int'l Conf. on Learning Representations (ICLR), 2018.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-Learning with Latent Embedding Optimization. Int'l Conf. Learning Representations (ICLR), 2019.
Jürgen Schmidhuber: Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. Int'l Conf. Learning Representations (ICLR), 2015.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 4077-4087, 2017.
Sebastian Thrun and Lorien Pratt. Learning to learn. Kluwer Academic Publishers. 1998.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pitrre-Antoine Manzagol, and Hugo Larochelle. Meta- Dataset: A Dataset of Datasets for Learning Learn from Few Examples. In Int'l Conf. Learning Representations (ICLR), 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 3630-3638, 2016.
Duo Wang, Yu Cheng, Mo Yu, Xiaoxiao Guo, and Tao Zhang. A hybrid approach with optimization- based and metric-based meta-learner for few-shot learning. Neurocomputing, 349:202-211, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 171vv:1710.09412 2017.
11

---

Published as a conference paper at ICLR 2021
7 HYPERPARAMETERS AND ABLATION STUDIES
In this section, we report the nyperpprameters of LASIUM-MAML in aad LASIUMMPPottNNE in Table 6 for Omniglot, CelebA, CelebA attributes and Mini Imaieee datasets. Our source code is also available on Github
Table LASIUM-MAML hyperparameters Lese summary
Hyperparameter
Omniglot
CelebA
CelebA attributes
Mini-ImageNNtt
Number of classes
5
2
5
Input size
28 x 28
84 84 X 3
84 x 84 X 3
84 x 84 x
Inner learning
0.4
0.05
0.05
0.05
learning rate
0.001
0.001
0.001
0.001
Meta-batch size
4
4
K (tr) meta-learning
I
5
(ual)) meta-learning
5
5
5
K (val)) evaluation
15
15
5
15
Meta-adaptate steps
5
5
5
Evaluation adaptation steps
50
50
50
50
Table 6: LASIUM-ProtoNee hyperparameters summary
Hyperparameter
Omniglot
CelebA
CelchA attributes
Mini- Mini-ImageNet
Number of classes
5
2
5
Input size
28 x 28 x 1
84 x 84 x 3
84 x 84 x 3
84
x 84 3
Meta learning rate
0.001
0.001
0.001
0.001
Meta-batch size
4
((rr) meta-learning
1
5
1
(val) meta-learning
5
((oa)) evaluation
15
15
15
We also report the ablation studies different strategies for task construction in Table We run all the algorithm for just 1000 iterations and compared between them. We also apply a small translation to Omniglot images.
Table 7: Accuracy of different proposed strategies on Omniglot. For the sake of comparison, we stop meta-learning after 1000 iterations. Results are reported 1000 tasks with a 95% confidence interval.
Sampling Strategy
Hyperparameters GAN-MAML
VAE-MAML
GAN Proto
Vro-Proto
LASIUM-N
=²=0.5
77.16+0.65
70.41 + 0.71
62.16 + 0.79
61.57 + 0.80
LASIUM-N
0²=1.0
71.10 0.70
68.26 = 0.71
60.95 + 0.78
62.17 0.80
LASIUM-N
o?=2.0
63.18 + 0.71
65.18 0.71
59.81 + 0.78
64.88110.78
LASIUM-RO
a=0.2
7.66210.64
75. 02.66260.66
62.2411.77
62.17 0.80
LASIUM-RO
a=0.4
75.7910.65
71.31:0.70
64.1910.76
62 02.2010.800
LASIUM-OC
0=0.2
74.70 0.68
74.98+0.67
61.79 + 0.79
62.16 + 0.78
LASIUM-OC
a=4.4
73.40 + 0.68
68.79 + 0.73
64.59110.76
63.0811.799
Besides, we perform a hyperparameter search on CelebA attributes benchmark. Table S demonstrates the results for experiments. We see that searching for hyperparameters for CelebA is almost as easy as doing the same thing for Omniglot. LASIUM-N with x² 0.25 outperforms state-of-the-a in this benchmark. We also a bad performance the case of LASIUM-OC. which we expected as the number of classes in this benchmark's 's tasks is N = 2. Thus samples generated during leaaning are limited only instances on the line connecting two anchor latent vectors. not the case for LASIUM-N and LÁSIUM-RO since we can sample latent codes in the neighborhood or any direction from anchor points the latent space.
Phtpss//gituu..com//iia http:////gtiiiimm/////oooooaaaod Lww.mnngmmmm......... -F2.0
12

---

Published as a conference paper at ICLR 2021
Table 8: Accuracy of different proposed strategies CelebA attributes task for GAN with 2-way, 5-shot tasks with K = 5. The results are averaged over 1000 downstream tasks and + indicates 95% confidence interval.
Sampling Strategy
Hyperparameters
GAN-MAML
GAN-Pron Proto
LASIUM-N
0==0.1
71.83 + 1.08
62.99 + 1.14
LASIUM-N
02=0.25
75.07 + 1.08
70.49 + 1.14
LASIUM-N
o²=0.5
71.41 1 1.13
69.96 + 1.15
LASIUM-N
o²=1.0
60.37 + 1.01
69.98 + 1.16
LASIUM-N
o²=2.0
50.00 1
70.33 + 1.14
LASIUM-RO
0=0.2
62.06 + 1.06
62.73 + 1.18
LASIUM-RO
x=0.4
67.57 1 1.11
68.19 + 1.12
LASIUM-RO
a=0.5
71.04 = 1.03
68.94 1 1.12
LASIUM-OC
0=0.2
59.69 1.11
53.67 + 1.02
LASIUM-OC
0=0.5
60.25 + 1.08
56.05 1 1.08
7.1 NEURAL NETWORK ARCHITECTURES
For Omniglot, our VAE model is constructed symmetrically. The encoder is composed of four convolutional blocks, with batch normalization and ReLU activation following each of them. A dense layer is connected to the end such that given an input image of shape 28 x 28, the encoder produces a latent vector of length 20. On the other side, the decoder starts from a dense layer whose output has length 7 x 7 64 3136. It is then fed into four modules each of which consists of a transposed convolutional layer, batch normalization and the ReLU non-linearity. We use 3 x 3 kernels, 64 channels and a stride of 2 for all the convolutional and transposed convolutional layers. Hence, the generated image has the size of 28 x 28 that is identical to the input images. This VAE model is trained for 1000 epochs with a learning rate of 0.001.
Our GAN generator gets an input of size / which is the dimensionality of the latent space and feeds it into a dense layer of size 7 x 128. After applying Leaky ReLU with a 0.2, we reshape the output of dense layer to 128 channels of shape 7 x 7. Then we feed it into two upsampling blocks, where each block has a transposed convolution with 128 channels, 4 4 kernels and 2 strides. Finally, we feed the outcome of the upsampling blocks into convolution layer with 1 channel and 7 x 7 kernel with sigmoid activaiton. The discriminator takes a 28 28 x input and feeds into three 3 x 3 convolution layers with 64, 128 and 128 channels and 2 2 strides. We apply leaky ReLU activation after each convolution layer with a = 0.2. Finally we apply a global 2D max pooling layer and feed it into dense layer with neuron to classify the output real or fake. We use the same loss function for training as MSGAN (aka Miss-GAN) described in Mao et al. (2019).
For the CelebA GAN experiments, of GANs (ProGAN), described in Karras et al.) (2018). For VAE, we use the same architecture as described for Omniglot VAE with one convolution block and more channels to handle the larger input size of 84 x 84 x 3. The exact architecture is described in the supplemental material.
the pre-trained network architecture, progressive growing
7.2 IMPACT OF GAN TRAINING ON LASIUM
Do we need a generative model that generates very high-quality images from data or can a premature trained GAN also work? We performed an ablation study to evaluate the impact of GAN training on LASIUM. First, we trained generative network on Omniglot dataset with adversarial training for 500 epochs and saved the corresponding weights at every epoch. Then we trained LASIUM with various generative networks at different epochs. For the sake of comparison, stopped LASIUM after 1000 iterations.
Figure demonstrates the impact of GAN training on LASIUM. We visualize image generated with the same exact latent code after different epochs. We can see that eventually, this latent code result in generating character "R" (after epoch 400 and 500). We that GAN stabilizes after 400 epochs while LASIUM stabilizes sooner around epoch 200. Nevertheless, the impact of training GAN for at least 50 epochs is correlated with LASIUM performance.
13

---

Published as a conference paper at ICLR 2021
1
M
R
80
Accuracy (%)
70
baseline
Generated image
60
103050
200
300
500
GAN training epochs
Figure 5: The impact of GAN training on LASIUM accuracy. The blue line shows test accuracy after 1000 iterations of LASIUM-N training with 95% confidence interval on 1000 one-shot tasks with K(val) 15. The images generated by GAN are shown epochs 0, 10, 30, 50, 100, 200, 300, 400, and 500. All of the images are generated from the same latent vector. The red line shows the training from scratch baseline.
7.3 ABLATION STUDY ON IMPACT OF ON LASIUM
In this section, we evaluate the accuracy of LASIUM with respect to the value of e. For the sake of comparison, consider LASIUM-N with o² 0.5 and stop the training after 1000 iterations Omniglot. The results are reported on the same 1000 one-shot tasks with ( 15 in Table 9 Furthermore, the last column shows the number of times resampling occurred since at least two the initial sampled latent codes were in distance smaller than from each other. We found that (within reasonable bounds) the choice of this hyperparameter has a small but not negligible impact on the performance of the algorithm.
7.4 TRAINING LASIUM ON FUNGI
We also tried LASIUM on Fungi dataset. We report LASIUM-N-GAN-MAMII accuracy over 1000 downstream tasks generated randomly from test dataset following Meta-dataset evaluation protocol proposed by Triantaillouu et al. (2020). For the choice of generative network, we used state-of-the-ar StyleGAN-v2 by Karras et al. (2020). and we trained on Fungi images. Figure shows some of the examples generated by StyleGAN-v2. Table 10 shows the results on LASIUM and some other relevant algorithms.
14

---

Published as a conference paper at ICLR 2021
Table Accuracy of LASIUM-N with x² 0.5 on Omniglot dataset with respect to different values of e. + indicates 95% confidence interval.
Accuracy (%)
Resampling Task
0.0
77.27 = 0.62
0
77.34 0.62
77.21 0.62
10.0
77.54 + 0.62
0
100.0
77.08 + 0.63
0
125.0
79.51 0.61
187.5
78.87 0.60
395
218.75
77.95 0.62
6012
234.375
77.15
27432
242.1875
77.15 0.64
61118
250.0
155499
253. 15625
78.48 0.62
238714
256,3125
78.05 0.62
378742
265.625
78.56 + 0.61
1472860
281.25
77.73 0.63
25936957
312.5
375
All
500.0
All
1000.0
All
Train
Validation
Figure 6: Meta-training tasks for Fungi constructed by LASIUM-N with 0² = 0.25
Table 10: Accuracy of 5-way 1-shot learning the Fungi dataset (part of the proposed Meta-dataset by Triantafillou et al. (2020)) For each system we indicate the dataset on which the meta-training phase was performed. The results for supervised first-order MAML are from Triantafillou al.
(2020). LASIUM-N was run with 0² = 0.25 and used the StyleGAN-v2 trained on the unsupervised version of the Fungi dataset, as discussed the text.
Method
Dataset
Accuracy (%)
Training from scratch fo-UMTRA MAML -OMMMLLL
26.10 1: 0.42
Unsupervised Fungi Unsupervised Fungi
28.27 1: 0.46
LASIUM-N-GAN-FE
29.43 0.49
DASIUM-N-GAN-MA MAML
Unsupervised Fungi
31.29 0.52
fo-MAML
Supervised Imagenet
32,10 1.10
fo-MAML
Supervised Meta-datase
33.54 + 1.11
15


Reviews:
[{"review_id": "XEE1qgpjEzv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\n- This paper proposes LASIUM, a framework for constructing tasks for unsupervised meta-learning for image classification via the use of a generative model on the unlabeled training data. Similar to Hsu et al. (2019), distance in the latent space of an unsupervised learner is taken to correspond to class-level information. Uniquely, LASIUM leverages generative modeling by proposing simple interpolation-based schemes to populate classes with latents, which are decoded by the generative model to produce image samples for the meta-learning stage. Three specific variants of LASIUM are proposed. The results indicate that the approach is competitive with respect to prior methods for the Omniglot and CelebA datasets, but curiously, results on miniImageNet are not presented.\n\n#### Strengths\n- The idea of using generative modeling and interpolation for unsupervised meta-learning for image classification is significantly different from prior works, which use data augmentation (Antoniou and Storkey 2019, Khodadadeh et al. 2019) and latent-space clustering (Hsu et al., 2019).\n\n- Reasonable variations of the core idea of latent-space interpolation and generation are explored and assessed. This includes a VAE-specific variant which additionally leverages the encoding capability of a VAE.\n\n- The experiments involve a reasonable variety of generative models, and the protocol seems to closely follow that of prior work.\n\n#### Weaknesses\n- Critically, experiments based on the miniImageNet dataset are completely missing. This is a benchmark that is present in all three prior works in unsupervised meta-learning for image classification ((Antoniou and Storkey 2019, Hsu et al. 2019, Khodadadeh et al. 2019), as well as the vast majority of meta-learning works. The diversity present in miniImageNet makes it a benchmark that assesses significantly different model capabilities than Omniglot or CelebA, and hence makes it an indispensable part of a comprehensive empirical evaluation.\n\n- Unfortunately, there is a fair amount of overclaiming. \"The proposed family of algorithms generate pairs\nof in-class and out-of-class samples from the latent space in a principled way\" -- the authors do not go beyond high-level intuition in justifying the latent interpolation schemes for task generation. \"Beyond the increase in performance, we must note that the competing approaches use more domain specific knowledge (in case of UMTRA augmentations, in case of CACTUs, learned clustering).\" -- how does CACTUs use more domain-specific knowledge than LASIUM? Both methods rely on an unsupervised pre-training stage to \"organize\" the unlabeled dataset into a latent space. \n\n- The use of generative models introduces a significant potential weakness: the performance of the downstream meta-learning is now additionally dependent on sample fidelity. CACTUs sidesteps this issue by only using samples from the original dataset, and AAL and UMTRA rely on carefully defined image transformations that do not compromise sample quality. This is touched upon in the discussion and described as a \"limitation\", but this key issue is not explored further.\n\n- There are significant clarity issues with the discussion section. \"A conceptual difference between clustering and LASIUM\" -- the difference is really between CACTUs and LASIUM; clustering doesn't entail encoding by itself. \"The former is shared with clustering based unsupervised learning\" -- what is clustering based unsupervised learning? Do you mean CACTUs? Also, do you mean \"latter\" instead of \"former\"? \"Interestingly, training of these feature extractors is based on augmentations like random crop and random flips (Caron et al. (2018)) or auto encoders (Berthelot et al. (2019))\" -- what precisely is interesting about this? Do you mean to make an explicit connection between CACTUs, UMTRA/AAL, and LASIUM here? \n\n#### Recommendation\n- I currently recommend rejection (4). The weaknesses outlined above outweigh the strengths. \n\n#### Questions\n- Why are results on miniImageNet not included?\n\n- Under what conditions can we expect LASIUM to be a favorable choice over CACTUs or UMTRA/AAL, if any?\n\n- \"Note that meta-learning algorithms are not sensitive to a small fraction of bad training examples in the meta-training phase (e.g. see Hsu et al. (2019); Khodadadehet al. (2019)).\" -- How do these works demonstrate this?\n\n#### Minor suggestions\n- There are minor typos, e.g. \"proGAN\" instead of \"ProGAN\".\n\n------------------------------------ Post-rebuttal comments ------------------------------------\n\nWith the authors' answers, the inclusion of miniImageNet results in the main text, and revisions to the writing, my concerns have largely been addressed and I have increased my rating from a 4 to a 6.\n\nI have a few further suggestions to make: first, make it clear in Table 4 via captioning and/or markings that UMTRA and LASIUM results for miniImageNet depend on the use of full unlabeled ImageNet data to train the AutoAugment and BigBiGAN models, respectively, while the CACTUs results do not. This results in an unfair comparison, and naturally raises the question of how LASIUM would do if we used a generative model trained only on the meta-training split of miniImageNet, which would be more in line with the protocol used for Omniglot and CelebA. It seems fair to assume that it would do worse than the current LASIUM results in Table 4, and probably the CACTUs results as well. This relates back to my point in the original review about sample fidelity: miniImageNet has the most diversity among the datasets considered, and it is hard for generative models to capture this diversity given a relatively small dataset. It would be good for this to be conveyed to the reader.\n\nThe usage of the extraneous ImageNet data for UMTRA and LASIUM does not conform to the definition of unsupervised meta-learning proposed in Hsu et al. (2019) and adopted in this work. Some discussion of the problem assumption may be warranted: in what practical circumstances would extraneous, relevant, unlabeled data be available? And when it is, why would one not use all of the data (e.g. the entirety of unlabeled ImageNet) to do unsupervised meta-learning and/or unsupervised representation learning, like in Table 12 of Hsu et al. (2019)?\n\nOverall, despite the unfair miniImageNet protocol, I still advocate for weak acceptance as the method does show competitive results for domains that are more suited to generative modeling. For the sake of clarity for readers, though, I strongly encourage the authors to implement my additional recommendations.", "extra_scores": {}, "extra_text": {}}, {"review_id": "P7xuAHYJ3d5", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a novel method for generating few-shot learning tasks from unlabeled data. The basic idea is to use a GAN or VAE to generate in-class (close in latent space) and out-of-class (far away in latent space) pairs of image data by linear interpolation in the latent space of the chosen generative model. A new in-class sample is generated by interpolating between two out-of-class samples in latent space with the interpolation weight chosen such that the generated sample is close to one of the out-of-class samples. The method is evaluated on common few-shot learning datasets with good results.\n\n**Pros:**  \n- The concept of generating few-shot learning tasks from an unlabeled dataset by performing interpolation in the latent space of a GAN or VAE is innovative and promising.\n- The experiments demonstrate that the proposed method is as good as or superior to competitive unsupervised methods in basic scenarios. \n- The paper is very well written, and the ideas are clearly presented.\n- The diagrams are very helpful to understanding the basic concepts.\n\n**Concerns:**  \n(1) The experiments are basic and do not explore potential limitations of the proposed method:\n- All the experiments were performed with a low, fixed value of way equal to 5 and a fixed number of shots on datasets with homogeneous content. It would be interesting to see the results of using LASIUM on the more challenging meta-dataset [1] which uses tasks of variable way and shot on 10 diverse datasets. In a brief examination of the source code supplied in the supplementary material, it seems that some experimentation on meta-dataset may have been done. Any reason that those results were not presented?\n- In particular, some of the datasets within meta-dataset (e.g. Fungi has a large number of classes of mushrooms that differ only subtly) require very fine-grained discrimination. It would be interesting to evaluate LASIUM in a higher way (> 25), fine-grained classification setting where classes are likely to be close in the latent space of the generator.\n\n(2) In section 2, it says: “Our proposed approach, LASIUM, takes advantage of generative models trained on the specific domain to create the in-class and out-of-class pairs of meta-training data.” However, the goal of a few-shot learning system after meta-training is that the system should be able to make predictions on unseen input data, potentially from a different domain. As pointed out in (1) above, LASIUM has only been evaluated on a single domain. How would LASIUM handle a setting with very diverse tasks? Would more than one pre-trained GAN be needed to generate the examples? Or perhaps a GAN that has be trained on a large quantity of diverse data?\n\n(3) LASIUM uses the latent space of a GAN or VAE to linearly interpolate between examples to generate few-shot learning training data. Were any ablation studies performed to ascertain that this is the best space for generating data? As a counter example, [3] uses linear interpolation in the raw data domain to augment training data in the supervised setting (i.e. they do not evaluate the method in the unsupervised setting) with particularly good results. The authors of [3] also consider doing the interpolation in an embedded feature space. Linear interpolation in alternative data spaces would make for a potentially interesting baseline or ablation study.\n\n(4) Some experiment details are missing:  \n- The main text (including Algorithm 1) makes no mention of using labeled data. However, in Section 4.2 it says: “For instance, the 95.29% accuracy of LASIUM-RO-GAN-MAML was obtained with only 25 labels, while the supervised approaches used 25,000.” Please elaborate on the use of labels in the various methods. Are some labels used to establish “anchor vectors”? How do the results change if you use no labels (i.e. fully unsupervised as opposed to semi-supervised)? Was labeled data also used by the competitive unsupervised approaches (i.e. CACTUs, UMTRA)?\n- No detail was given on how the transfer learning results were obtained (i.e. learning rates, number of iterations, regularization, augmentation, etc.). Specifically, the concern is that the transfer learning results seem to be quite low, while [2] shows that supervised transfer learning can outperform supervised few-shot learning methods. Also, the transfer learning results for miniImageNet seem to be missing.\n- What values for $\\alpha$ were used in the experiments? In table 6 in the supplementary material, there is a small ablation study for selecting a good value $\\alpha$ for the Omniglot experiments. It would be good to see this done for a greater range of $\\alpha$ and for datasets other than Omniglot. The key question is to know how difficult it is to set $\\alpha$ for each setting.\n- What values are used for the rejection sampling threshold $\\epsilon$ for the various methods? How sensitive is this parameter?\n- For the sake of reproducibility, details on how all the baseline experiments were carried out would be beneficial.\n \n(5) Some details are missing from Algorithm 1 including: a) how the unlabeled dataset gets encoded to the latent space in the VAE case and what role the unlabeled dataset plays in the GAN case; and b) the required hyperparameters for the sampling strategy (i.e. $\\alpha$, $\\sigma$, $\\epsilon$).\n\n**References:**  \n[1] Triantafillou, Eleni, et al. \"Meta-dataset: A dataset of datasets for learning to learn from few examples.\" arXiv preprint arXiv:1903.03096 (2019).  \n[2] Tian, Yonglong, et al. \"Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?.\" arXiv preprint arXiv:2003.11539 (2020).  \n[3] Zhang, Hongyi, et al. \"mixup: Beyond empirical risk minimization.\" arXiv preprint arXiv:1710.09412 (2017).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "G2WrAhG4bko", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes LASIUM (LAtent-Space Interpolation Unsupervised Meta-learning), a novel method of generating meta-tasks in an unsupervised way. It leverages recent advances in generative models. Specifically, it uses the fact that datapoints that are close in latent space are more likely to be from the same class. The method interpolates between samples in latent space, choosing interpolation ratios.\n\nI find this work compelling: it is built upon simple and intuitive ideas and leverages a rapidly growing research area. Its ability to use any pre-trained generative model to perform unsupervised meta-learning is promising.\n\nThe paper proposes three different variants of LASIUM, each of them compelling ideas: LASIUM -N, -RO, -OC. However, the paper does not comment on, which is best or what tradeoffs exist between the three methods. RO seems to win on Omniglot, but the paper only shows N for the hardest task (CelebA).\n\nIt would have been helpful to see how the generative model affects unsupervised meta-learning performance. For example, is being good at image generation always better for meta-learning? We cannot know from the experiments presented here because it only tries one model for each setting.\n\nMinor comments\n- the underlined phrases scattered throughout the paper helped comprehension. Nice touch.\n- (very minor) page 4: inconsistent indentation for paragraph \"Generating in-class...\". Other bolded paragraphs are not indented.\n- Table 3: Missing digit (probably 0) on MLP with dropout.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8zqE28JCDo", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of unsupervised meta-learning, where the goal is to generate tasks for meta-training without supervision. Whereas previous work generated training and test sets from the unlabeled set for meta-training via augmentations (UMTRA) or unsupervised clustering of embeddings (CACTUs), this paper considers doing this using interpolation of the latent space representations produced by generative models. Specifically, the idea is to first train a generative model on the unlabeled set and then produce training and test sets for meta-training by decoding the interpolation of latent space representations of multiple examples from the original unlabeled set. The authors discuss 3 specific ways to produce examples for the train and test sets for meta-training in this way. They first select an anchor example from the unlabeled set that will be representative of one class in the dataset. Then, the 3 methods involve:\n1. Adding noise: adding noise to the latent space representation of the anchor example to produce examples that make up the training and test set for a single class.\n2. Random out-of-class sample: selecting another example from the unlabeled set and finding new examples for the class by interpolating between the anchor's and this example's representations. \n3. With Other Classes' samples: same as (2) but instead of picking another random example, the example considered is another anchor example that was used to represent a different class.\n\nThe authors evaluate their method by considering 3 few-shot learning benchmarks: (1) Omniglot; (2) CelebA few-shot identity recognition; and (3) CelebA attribute prediction. On these 3 benchmarks, they show that their method performs favorably compared to UMTRA and CACTUs.\n\nPros\n* This paper proposes a simple yet very interesting idea for performing unsupervised meta-learning that is unique compared to previous work.\n* The big benefit of this method compared to previous work is that it seems to require less tweaking per dataset. Whereas previous methods required tuning per dataset (for example, in UMTRA, selecting which augmentations to use for a specific dataset), this method requires training a generative model on the unlabeled set and using the learned latent space interpolation (where the generative model can directly learn properties of the specific dataset that can be used during interpolation). However, there are still some choices to make in terms of hyperparameters for latent space interpolation).\n\nCons\n* I have minor concerns about the Mini-ImageNet experiments. Firstly, why are Mini-ImageNet experiments not discussed in the main paper but in the supplementary material? The difficulties of using Mini-ImageNet are mentioned, in that it is difficult to train a generative model on this more complex dataset using the limited examples in Mini-ImageNet. Thus, I believe it is a good idea to use whole ImageNet dataset as the unlabeled set, as the authors did, and the results for the method seem favorable compared to previous work. So, I think it's useful to include these results to show how this method extends to more complex images? I think a note just needs to be added that the unlabeled set this method uses is much larger than the ones used in previous work for the Mini-ImageNet comparison but I don't view this as a big negative because the data required for training is still unlabeled.\n* Details of the CelebA few-shot identity recognition benchmark seem to be lacking? I don't see this benchmark mentioned in previous work so I was curious how the metrics for other methods (such as CACTUs) were generated given that this benchmark wasn't discussed in those papers? I think more details about this benchmark would be useful in general. Additionally, some citations to previous results on this benchmark would also be helpful.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses a method for generating meta-tasks via latent space interpolation using a generative model, trained on the unlabeled dataset, to solve the unsupervised meta-learning. The method seems to be sound, but it lacks MiniImage-Net experiments, which is the main concern raised by most of reviewers. During the author responses, new empirical results on MiniImage-Net were added. During the discussion period with reviewers, I communicated with the reviewer with most negative comments. He/she was not fully satisfied, claiming that the protocol used for obtaining new MinImage-Net results was slightly unfair. It is suspected that the authors used a generative model trained on the ImageNet training set (of which miniImageNet is a subset) for LASIUM and  did not present results from only using miniImageNet meta-training data because it was not competitive with prior work. It should be clarified in the final paper. However, we arrived at the consensus that the paper is worth being presented. 


Decision: Accept (Poster)