Discrete InfoMax Codes for Meta-Learning
Authors: Yoonho Lee
Wonjae Kim
Seungjin Choi
Venue:  (2020)
Keywords: meta-learning
generalization
discrete representations

Abstract: This paper analyzes how generalization works in meta-learning. Our core contribution is an information-theoretic generalization bound for meta-learning, which identifies the expressivity of the task-specific learner as the key factor that makes generalization to new datasets difficult. Taking inspiration from our bound, we present Discrete InfoMax Codes (DIMCO), a novel meta-learning model that trains a stochastic encoder to output discrete codes. Experiments show that DIMCO requires less memory and less time for similar performance to previous metric learning methods and that our method generalizes particularly well in a challenging small-data setting.

Under review as a conference paper at ICLR 2020
DISCRETE INFOMAX CODES FOR META-LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
This paper analyzes how generalization works in meta-learning. Our core contri- bution is information-theoretie generalization bound for meta-leamiig, which identifies the expressivity of the task-specific learner as the key factor that makes generalization to new datasets difficult. Taking inspiration from our bound, we present Discrete InfoMax Codes (DIMCO), a novel meta-learning model that trains a stochastic encoder to output discrete codes. Experiments show that DIMCO requires less memory and less time for similar performance to previous metric learning methods and that our method generalizes particularly well in challenging small-data setting.
INTRODUCTION
Generalizing to unseen data is a problem of vital importance in machine learning. Many deep meta- learning methods optimize for generalization by directly minimizing the loss of held-out validation data. Recent works have used this framework to achieve impressive feats such as learning to classify using one labeled image per class (Snell al., 2017; Finn al., 2017), learning unsupervised update rules that generalize to different domains (Metz et al., 2018), 8), and accelerating training procedures that are millions of steps long (Flennerhag et al., 2018). However, the meta-learning setup introduces a new overfitting problem: the model may overfit to the distribution of tasks seen during training. In other words, the meta-learning framework decreases task-specific overfitting at the cost of introducing task-wise overfitting.
The primary aim of this work to elucidate this tradeoff between task-specific and task-wise overfitting in meta-learning Specifically, we tools from information theory to bound the overall generalization gap of meta-learner. Analogously to how generalization bounds for standard learning algorithms reveal the role of dataset size in generalizing to new datapoints, (n) in generalizing to new tasks. The specific
bound reveals the
roles of both dataset size (m) and number of datasets form of our generalization gap suggests that meta-learning models can be implicitly regularized by constructing them to express each datapoint with a small number of bits.
To this end, we propose Discrete InfoMax COdes (DIMCO), deep neural network model that outputs a discrete representation of each datapoint, Note that a continuous representation è E IR" (e.g. Snell et al. (2017)) uses 32n bits per datapoint. This wildly inefficient in terms of bit-efficiency: our experiments in Section 5 show that DIMCO's 's discrete representation requires roughly 10x less bits per datapoint to achieve similar performance compared to continuous methods. DIMCO generalizes well to novel datasets because its learning objective encourages the model to compactly use all of its degrees freedom, thus enabling it to effectively compare datapoints while only requiring a small number of bits per datapoint.
Our specific contributions are:
1. Derive a generalization bound for meta-learning that makes the tradeoff between task- specific and task-wise overfitting concrete.
2. Propose DIMCO, a neural network model that is designed have a low value of specific term in our generalization bound.
3. Empirically demonstrate that DIMCO generalizes better than previous meta-learnigg meth- ods when trained with small datasets, and that it is more memory- and time-efficient compared to previous image retrieval methods.

---

Under review as a conference paper at ICLR 2020
This paper is organized as follows. We detail our problem setup and derive a generalization bound for meta-learning in Section 2. Taking motivation from our bound, we propose our meta-learning model (DIMCO) in Section 3. We put analysis in the context of previous work in Section 4. Notably, suggest that certain previous meta-learning methods may have benefitted from implicit regularization. We present experiments in Section 5 and conclude the paper with discussion about limitations and future directions of our approach in Section 6.
2 A GENERALIZATION BOUND FOR META-
Throughout this section, we denote data, model outputs, and labels as x, x, and respectively, We use capital symbols X, X, Y to denote the random variables corresponding to x, x.
The (discrete) Mutual Information between two random variables X1, X2 is defined
X2) H(x)) H(,, H(x2) ((22(X))
[[x;; X22) is a symmetric quantity which measures the amount of information shared between X1 and X2. It has its lowest value O when X1 and X₂ are independent and increases with the correlation between X₃ and X2. We refer the reader to (Cover & Thomas, 2012) for further exposition.
2.1 PROBLEM SETUP
We begin by describing our meta-learning problem setup. Define a task to be a distribution over 2 = X x V Let tasks T'.... ..... T be sampled i.i.d. from distribution of tasks Associated with each task T, we define a dataset Dr = 21,-- 2T = wh).... (x7 uT) which is a set of m i.i.d samples from the data distribution (z) T).
We consider models with parameters that map datapoints X to representations (X, 9). Our objective is the expecteation of the negative mutual information between representations an labels across all tasks:
[(T,0) EE [1(((X,
(2)
This objective is closely related to both previous loss functions and evaluation metries in setups that involve testing on unseen classes (e.g. few-shot classification, image retrieval). We show that previous loss functions can be seen approximations to (2) in Appendix A, and that mutual information is strongly correlated with metrics such as few-shot accuracy and Recall@1 in Section 5.1.
important difference between (2) and previous objectives for few-shot classification is that we do not split each task into support/query (also called train/test) sets. This property bypasses the pesky issue of using batch normalization (BN) during meta-training. out, BN can leak information from the support set to the query set. This issue is not problem in setup since we do not assume a seperate "query set". Additionally, not using support/query splits enables meta-learning with tasks consisting of one image per class, Note that the standard meta-learning setup cannot learn from such tasks since it requires at least two images per class (one for support, one for query) to compute the loss function. Overall, demonstrates that the commonly used construction of a held-out test set within each task is not strictly necessary for meta-learning.
Nichol & Schulman (2018) points
demonstrate in Section 5.4.
model
2.2 GENERALIZATION BOUND
We bound the difference between expected loss and empirical loss:
Theorem 1. Let m. X, X, Y,0, L be defined above. Let de be the VC dimension of the encoder X(). Let (XT, Yr) be the empirical estimate of the mutual information using finite dataset Dr, and define empirical loss as
L( .0) = n
();
(3)
i=1
2

---

Under review as a conference paper at ICLR 2020
X
logit xdxp
P P(XI))
softmax
1(X;
000
0:01
6.2
encode
P(Y)
1 3 4
Y
Figure 1: A graphical overview of Discrete InfoMax COdes (DIMCO). A dataset D consists of pairs of images X and labels Y. DIMCO is a stochastic encoder that maps each image X distribution of discrete codes p (X) Each discrete code p(X[X) is a p-way code of length d. If = 4, d = 2 (as in the diagram), each code consists of 2 symbols and each symbol is {1, 3, 4). Inside the x 4 grid that represents the yd = 16 possible codes, the most likely row and column are colored. The most likely code in the diagram is (1, 2) with probability 30%. DIMCO is optimized by maximizing the mutual information between the discrete code and the label within each batch.
The following inequality holds with high probability:
,0) 50 (v de
| log(m) vm
Y
C(T, 0) - L(T -(
log +0
n2 de
+0
(4)
m
Proof. We use standard Chernoff bounds along with a finite sample bound for mutual information from Shamir et al. (2010); see Appendix B.
The generalization gap has three terms, two of which decrease m increases, and the other decreases as n increases. Typically for few-shot learning, n is very large while m is small: the typical minilmagenet 5-way 1-shot setup has 1010 and terms are the main difficulties for generalizing to new tasks. We see from Theorem that these terms can be reduced by using small | Therefore, in the context of meta-learning, using short representations (i.e. small X ) compensate for having small train set (i.e. small m).
We therefore claim that the latter two
Meta-Icarning is typically formulated as performing two levels of learning: task lask-general learning and task- sseeifee learning. Theorem 1 implies that in the tasks considered in recent literature, task-general learning should have almost no generalization gap, making task-specific learning the main source of (meta-)overfe Joverfiivv.,, This can be problematic since many works, the task-specific learning algorithm is usually just a byproduct of whatever clever meta-learning loss was proposed. Our theorem suggests that we should pay more attention to directly regularizing this task-specific learner, and our DIMCO model, described in the next section, can be seen as a minimal working example in this direction.
3 DISCRETE INFOMAX CODES (DIMCO)
We now present our model, Discrete InfoMax COdes (DIMCO), Motivated by Section 2, DIMCO produces short discrete code X and is trained by maximizing mutual information I(X;Y). Figure graphically shows the overall structure of DIMCO.
3.1 FACTORIZED DISCRETE CODES
We propose a factorized discrete representation scheme which enables us to represent discrete distributions with exponentially fewer parameters compared to listing the probability of each event. We represent each event as the product of d independent events, each of which consists of p different possibilities. We thus have nd events in total, but only require pd parameters represent the
3

---

Under review as a conference paper at ICLR 2020
probability of each event. Binary codes can be viewed special of this scheme where p 2. This factorization trick allows consider representations of size | 64256 (Section 5). This representation has the advantage of requiring only d logg P bits per datapoint, whereas D- dimensional continuous vector embedding requires 32d bits (assuming 32-bit floats).
3.2 MODEL
Recall that we represent given image using d independent discrete distributions, each of which has p possibilities. First, (convolutional) neural network enc(.) takes image X as input and outputs a vector of length dp, which we reshape into matrix of size d pt
l1
/12
113
lpp
l21
122
123
12p
enc(X)=
=
(5)
las
la2
Las
lap_
Each row of this matrix represents the logits of a discrete distribution. We apply the softmax function to each row to get probabilities.
softmax (4: lip)
p
The ith codeword is sampled according to the categorical distribution following these probabilites:
I3 ..
We simply concatenate each v, obtain the representation: x (#....., F.)-
3.3 TRAINING
Recall that X is discrete random variable and X its distribution. Instead of sampling x ~ X, we directly use X to compute the objective:
I(XY)) H(X) - H(X
(8)
The first term, H(X), be calculated by taking the average of all probabilities and computing the entropy:
d H(X) = Σ H(X)) = Σ
Ex=iph Ph
EØ p/e
Σ E/= Plp
Cat
m
m
1=1
- = k) where e is the number of classes. The marginal probability of Y (p(Y= = k)) is the frequency of l: in (y ym.. H(X = k) can be obtained by computing (9) using only x for which y k:
The second term (((())=Einl = =)
Though we have motivated the use of I(X; Y) as loss function throughout this paper, we provide yet another perspective using the decomposition in (8). Minimizing (X(Y) encourages discriminatory behavior. This term encourages the average embedding of each class be concentrated as possible. Maximizing H (X) incentivizes the model to overall use all possible values of X.
We emphasize that such closed-form computation f(X;Y) only possible because we are using discrete codes.
3.4 EVALUATION
We map all images to their probabilities Pij via (5) and (6) for all i 1,..., d and j = 1...., We map each training image its most likely code:
x
max Pij+
arg max Paj
arg max Paj
(10)
4

---

Under review as a conference paper at ICLR 2020
Fix a train image and test image, and let x be the most likely code for the train image. The similarity between train image and test image is measured by the probability of the test image producing X. This amounts to computing the product of the test image's 's probabilites using x; for each i = 1, ..... d:
d
II PRIS-
(11)
We use this as similarity metric for both few-shot classification and image retrieval. We perform few-shot classification by computing the most likely code for each class via (10) and classifying each test image by choosing the class that has highest value of (11). We similarly perform image retrival by mapping each support image to its most likely code (10) and for each query image retrieving the support image that has highest (11).
4 RELATED WORK
The ability to generalize to novel datasets is critical in meta-learning benchmarks, and even more in benchmarks such as a model is tested on datasets from an unseen domain. To the best of our knowledge, no works have proposed an explicit regularizer for generalizing to new tasks.
Regularizing Meta-Learnen
(Triantafillou al., 2019), where
Our analysis suggests that the success of some previous meta-learning methods can be attributed to being implicitly regularized by reducing the expressive power reducing task-specific learning. The following works have reported benefits from reducing the number of such task-specific parameters: Lee & Choi (2018) learns a subset of the full network to alter during task-specific learning, Rusu et al. (2018) explicitly represents each task with -ww-imensional latent space, and Zintgraf et al. (2018) alters only a pre- specified subset of the full network during task-specific learning. It may be surprising at first that these methods achieve higher accuracy than vanila MAML (Finn et al., 2017), which is more expressive since it alters all parameters during task-specific learning. Kim et al. (2018) 8) also reports better meta-generalization through approximate variational inference with respect to a learned prior, which can be seen as restricting the search space of the task-specific learner. We showed through Theorem I that restricting inner-loop expressivity reduces the generalization gap; this provides theoretical understanding to this consensus that meta-learning models with simple task-specific learners generalize new tasks more easily.
Information Bottleneck Theorem is close in spirit to the information bottleneck principle (Tishby et al., 2000; Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017). This principle states that to generalize, one should maximize T T(X; Y) while simultaneously minimizing T T(X; X). Likewise, our objective (2) is / [Xx; Y) while our bound (22) suggests that the representation capacity [X] should be low for generalization. Also related is the deterministic information bottleneck (Strouse & Schwab, 2017) which extends the information bottleneck by minimizing H ((X) rather than T(X; X). These three approaches to generalization are related via the chain of inequalities I(X; S H(X)= S log X) which is tight when X is an efficient code.
Representation Learning Previous works have applied information -thoorethc principles to analyze the objective of VAEs (Alemi al., 2017: Chen et al., 2018), derive an objective for GANs to learn disentangled features (Chen et al., 2016), and to directly learn representations (Alemi et al., 2016; Hjelm al., 2018; Oord et al., 2018; Grover & Ermon, 2018; Choi et al., 2019). Our work can also be viewed as information-theoree meta-learning setup and our main focus is the settaaeeneaalintioonn problem.
representation learning method, but we assume a supervised
Discrete Representations Discrete representations have been thoroughly studied in the context of information theory (Shannon, 1948). Recent deep learning methods directly lear discrete representa- tions, by learning variational autoencoders with discrete latent variables (Rolfe, 2016; van den Oord al., 2017; Razavi al., 2019) or maximizing the mutual information between representation and
In practice, we add log probabilities instead of multiplying probabilities for numerical stability.
5

---

Under review as a conference paper at ICLR 2020
data (Hu et al., 2017). DIMCO is related to but differs from these works as it assumes a supervised meta-learning setting and performs infomax using labels instead of data.
Similarly to setup, Jeong & Song (2018) uses labels to learn binary hash code. Their focus is on the speedup gained by using sparse codes, whereas DIMCO learns a dense discrete code to generalize better. Additionally, their method solves minimum cost flow problem within each batch to find the locally optimal code, whereas DIMCO is able to directly compute its loss function.
Factorized Representations The idea of using factorized representations has appeared the contexts of quantizing a continuous input (Jegou al., 2011), memory-efficient clustering (Norouzi Fleet, 2013), and constructing an expressive attention mechanism using few parameters ( (Vaswani et al., 2017). Likewise, DIMCO factorizes its discrete representatations to increase its representation power.
Metric Learning The structure and loss function of DIMCO is closely related to embedding based meta-learning Vinyals et al. (2016); Snell et al. (2017); Oreshkin et al. (2018) and image retrieval Hoffer & Ailon (2015); Sohn (2016); Wu al. (2017); Duan al. (2018) methods. We show in Appendix A that the loss functions of these methods can be seen approximation to the mutual information (X; Y)). While all of these previous methods require support/query split within each task, DIMCO simply optimizes an information-theoree quantity of each batch, removing the need for such structured batch construction.
EXPERIMENTS
We use the minilmageNet (Ravi & Larochelle, 2016) and CUB200 (Wah et al., 2011) datasets with standard splits for both in our experiments. The minilmageNNe dataset is subset of the Imagenet (Krizhevsky et al.. 2012) dataset that was made for few-shot classification. It consists of 100 classes each containing 600 images of size 84 x 84. The classes are split into 64 training, 16 validation, and 24 test classes. The Caltech-UCSD Birds-200-201
(CUB200) dataset consists of 1 1788 images of birds from 200 classes. The classes are split into 100 training and 100 test classes.
We use two different CNN backbones for our experiments: the 4-layer convnet commonly used for meta-learning (Finn et al., 2017; Sung et al., 2018; 8; Liu et al., 2018), and the Inception network (Szegedy et al., 2015) with batch normalization (loffe & Szegedy, 2015) which is commonly used for deep image retrieval (Sohn, 2016; Movshovitz- Attias et al., 2017: Wu et al., 2017). We randomly initialize weights for the 4-layer convnet and use pretrained weights for the Inception network
5.1 CORRELATION OF METRICS
This experiment empirically verifies whether mutual information / [X;; Y) is a reasonable metric for quality of representation. We trained DIMCO on the minilmagenet dataset with d = 64 for 20 epochs, for 8 independent runs. We plot the pairwise correlations between five different metrics ( (5, 10, 20)-way 1-shot accuracy, Recall@1 1, and T(X; Y) in Figure 2. We see that all five metries are very strongly correlated. We observed similar trends when training with with previously proposed loss functions: we visualize these results in Figure of the appendix due to space constraints. Alongside this empirical evidence, prove in Appendix A that previously used loss functions for few-shot classification and image retrieval approximations YX;Y).
5.2 WHAT DOES EACH CODE REPRESENT?
We inspected what each code represents in a DIMCO model (d = 16, p 64) trained on the minilmagenet dataset. Recall that each image produces a d x p probability matrix (5. 6). For each of these dp entries, we plotted the top 10 images in the test set that assigned highest probability to that entry. We show images corresponding to four such codes in Figure (right) and more in Figure 6 of the appendix.
We see that DIMCO learns distributed representation. For example, the top code in Figure 2 represents the high-level concept of a furry animal and the shown 10 images span 4 different classes. On the other hand, the bottom code seems to focus the color of the background. By aggregating

---

Under review as a conference paper at ICLR 2020
Bleary telet
300eey 1-bbet
9.16
220
31
0.2 104 Drayy T-dack
41 02 03 13wwy Lobat
54 0.3
1001
8:35
010
022
50-way 3-hk4
R901
MI
Figure 2: Left: Pairwise correlation between MI = T(X; X) and previous metrics. Right: Visualiza- tion of codes of small trained DIMCO model (d 16, p 64); we show the top test set images that assign highest probability to specific code, for 4 different codes.
Recal191
Recall01
I
DIMCO N-pair
DIMCO
N-pair
10
10
10
10-
10
Bits per representaionn
Time per query
Figure 3: Image retrieval performance of DIMCO and N-pair loss on the CUB-200 dataset. The y-axis for both figures are the Recall@1 metric, and bars reflect standard deviation computed from n = 5 runs per configuration. The x-axes represent (left) bits required to store one representation and (right) seconds required to perform retrieval for one query. Both x-axes log-scale.
such complementary features in each of its codewords, DIMCO is able to classify images that belong to previously unseen classes.
5.3 TIME- AND MEMORY EFFICIENT IMAGE RETRIEVAL
We conducted an image retrieval experiment using the CUB200 dataset, and used multiclass N- pair loss (Sohn, 2016) as baseline. As is standard for image retrieval, we use the Inception network as specified in the beginning this section. Using the same backbone, we trained DIMCO with (p. d) {64, 128, 256} (128, 256, 512} and multiclass N N-pair with embedding dimension {128, 256, 512). We measured the time per query for each method on a single Tesla P40 GPU by averaging the time required for 1 10000 batches of queries of size 32.
Results in Figure 3 show that the compact code of DIMCO takes roughly an order of magnitude less memory for similar performance to N-pair loss, and requires less query time well. This experiment also demonstrates that discrete representations can match the performance of modern methods that continuous embeddings this relatively large-scale task. We additionally note that DIMCO is
7

---

Under review as a conference paper at ICLR 2020
3 1-thet acctines
40
10 shot acrunacy
33
DIX00
DIXOO
16
64
36
Number of fmrges per class
Number of Images class
20 aay 1-goot aocurayy
10
DIMOO
Recalt)
DIMOO
10
Triplt
Triplct
16
64
Number imsgees pet class
Number of imaage pet class
Figure 4: Performance of methods trained using subsets of minilmageNet of varying size. The lowermost y axis value for each metric corresponds to the expected performance of random guessing.
able to train using large backbones without significantly overfitting, whereas experiments reported in Mishra et al. (2017) indicate that MAML (Finn et al., 2017) overfits tremendously when using deeper backbone.
5.4 GENERALIZATION TO NEW TASKS
This experiment measures how well DIMCO can generalize to new datasets after training with a small number of datasets. This challenging experimental setup measures how much generalizable information the model can extract from a limited set of datasets; it can be seen as the meta-learning analogue of measuring the performance of classifier trained with small dataset.
We trained each model using {1, 4, 16, 64) samples from each training class in the minilmageNet dataset. For example, by using samples, we are reducing the full train split of (64 classes 600 images per class) into (64 classes x images per class). We compare against three baseline methods: Triplet Nets(Hoffer Ailon, 2015), multiclass N N-pair loss (Sonn, 2016), and ProtoNets (nnell et al., 2017). We report the average and standard deviation of the top 5 results of random hyperparameter search (see appendix for details). We show {5, 10, 20y-way 1-shot accuracies and Recall@1 1 of the test set in Figure 4.
First note that DIMCO is the only method that can train using a dataset consisting of example class. This is because other methods require at least one train and test example per class within each batch, while DIMCO requires no such train/test (also called support/query) split and simply maximizes the mutual information within batch. Furthermore, Figure 4 shows that DIMCO learns much more effectively when the number of examples per class is low; this is because DIMCO uses fewer bits to describe each datapoint, which lowers its generalization gap (24) when applying to novel datasets.
6 CONCLUSION AND DISCUSSION
Summary We derived generalization bound for meta-learning using information-theoree princi- ples. Building on our bound, we proposed DIMCO, a model that learns a discrete representation of data by maximizing its mutual information with the label. DIMCO had benefits in time, memory, and generalization in our experiments.
Towards Explicit Meta-Regularte Rettaaegularizationn In Section 4, we have suggested with analogy to The- orem 1 that the benefits of some previous meta-learning methods can be attributed to implicitly being regularized by reducing the expressivity of their task-specific learners. While DIMCO has demonstrated better generalization by reducing this expressivity via hyperparameters (d, p), this is only applicable to DIMCO's s specific setup of few-shot classification through mutual information maximization. In future work, we would like explore explicit meta-regularization schemes that
8

---

Under review as a conference paper at ICLR 2020
can be applied to other problems (regression, reinforcement learning etc.) and algorithms (MAML, Neural Process etc.).
Meta-Learning Without Splits Our meta-learning problem setup in Section 2.1 does not assume a support/query (also called train/test) split within each dataset. Along with showing that the traditional support/query split not strictly necessary, we demonstrated in Section 5.4 that removing it has the benefit of enabling meta-learning in datasets having one image per class. We believe future work could benefit from further exploring this space of meta-learning algorithms that use datasets without splits.
REFERENCES
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv.1612.00410, 2016.
Alexander A Alemi, Ben Poole, Ian Fischer, Joshua V Dillon, Rif A Saurous, and Kevin Murphy. Fixing a broken elbo. arXiv preprint
2017.
Ron Amit and Ron Meir. Meta-leariigg by adjusting priors based on extended pac-bayes theory. arXiv preprini arXiv: 17X1::7711.02200 2017.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems, pp. 2610-2620, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel, Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016.
Kristy Choi, Kedar Tatwawadi, Aditya Grover, Tsachy Weissman, and Stefano Ermon. Neural joint source-channel coding, In International Conference on Machine Learning, pp. 1182-1192, 2019.
Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 2012.
Yueqi Duan, Wenzhao Zheng, Xudong Lin, Jiwen La, and Jie Zhou. Deep adversarial metric learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp.
2780-2789, 2018
Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. arXiv preprint arXiv:1710.11622, 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint
2017.
Sebastian Flennerhag, Pablo G Moreno, Neil D Lawrence, and Andreas Damianou. Transferring knowledge across learning processes. arXiv preprint arXiv:1812.01054 2018.
Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew Botvinick, Hugo Larochelle, Sergey Levine, and Yoshua Bengio. Infobot: Transfer and exploration via the information bottleneck. arXiv preprint 190. 2019.
Aditya Grover and Stefano Ermon. Uncertainty autoencoders:
Learning compressed representations via variational information maximization. arXiv preprint 1rXiv:1812.105.0 2018.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildee Karan Grewal, Adam Trischler, and
Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 08670 2018
Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. International Workshop on Similarity-Based Pattern Recognition, pp. 84-92. Springer, 2015.

---

Under review as a conference paper at ICLR 2020
Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self eeff-uugmeted training. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1558-1567. JMLR. org, 2017.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on patlern analysis and machine intelligence, 33(1):117-122) 2011.
Yeonwoo Jeong and Hyun Oh Song. Efficient end-to-end learning for quantizable representations. arXiv preprint meo::0000000000 2018.
Taesup Kim, Jaesik Yoon, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian -gnel-agi meta-learning. arXiv preprint arXiv:1806.03881 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69800 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Yoonho Lee and Seungjin Choi. Gradient-basee based meta-learning with learned layerwise metric and subspace. 2018.
Yanbin Liu, Juho Lee, Minseop Park, Sachoon Kim, and Yi Yang. Transductive propagation network for few-shot learning. arXiv preprint arXiv: 10002, 2018.
Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jaseha Sohl-Dickstein. Meta-learning update rules for unsupervised representation learning. arXiv preprint arXiv:1804.0022 1804.0022, 2018.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. arXiv preprint ar0i0v 2017.
Yair Movshovitz Attias, Alexander Toshev, Thomas K Leung, Sergey Ioffe, and Saurabh Singh. No fuss distance metric learning using proxies. Proceedings of the IEEE International Conference on Computer Vision, pp. 360-368, 2017.
2017.
Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint 18X1v:1803.02299, 2018.
Mohammad Norouzi and David Fleet. Cartesian k-means. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3017-3024, 2013.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807.0388, 2018
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, pp. 721-731, 2018.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. arXiv preprint arXiv: 004460 2019.
Jason Tyler Rolfe. Discrete variational autoencoders. arXiv preprint arXiv:/609.02200, 2016.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv: 180.:018770056/ 2018.
10

---

Under review as a conference paper at ICLR 2020
Ohad Shamir, Sivan Sabato, and Naftali Tishby. Learning and generalization with the information bottleneck. Theoretical Computer Science, 411029330:2696-222 2711,, 2010.
Claude Elwood Shannon. mathematical theory of communication. Bell system technical journal, 27(3):379-423, 1948.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint
2017.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, 4077-4087, 2017.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in Neural Information Processing Systems, pp. 1857-1865, 2016.
DJ Strouse and David J Schwab. The deterministic information bottleneck. Neural computation, 29 2017.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1199-1208, 2018.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Naftali Tishby and Noga Zaslavsky, Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5. IEEE, 2015.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004005,,00 2000.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: A dataset of datasets for learning learn from few examples. arXiv preprint arXiv: (xXivv9903.03090 2019.
Aaron den Oord, Oriol Vinyals, al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6306-6315, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-201. dataset. 2011.
Chao Yuao-Yuan Wu, Manmatha, Alexander Smola, and Philipp Krahenbuhl. Sampling matters in deep embedding learning. In Proceedings of the IEEE International Conference Computer Vision, pp. 2840-2848, 2017.
Luisa M Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Caml: Fast context adaptation via meta-learnin, arXiv preprint arXiv:/810.000 2018.
11

---

Under review as a conference paper at ICLR 2020
A PREVIOUS LOss FUNCTIONS ARE APPROXIMATIONS TO MUTUAL INFORMATION
Cross-entropy Loss The cross-entropy loss has directly been used for few-shot classification (Vinyals et al., 2016; Snell al., 2017).
Let g X: Φ) be parameterized prediction of given x, which tries to approximate the true conditional distribution q(y (). Typically in a classification network, is the parameters of learned projection matrix and g(-) is the final linear layer. The expected cross-entrope loss can be written as
xent(Y, X) Ey [- log q(y|x,,))
(12)
Assuming that the approximate distribution q(-) is sufficiently close ply|x), (x), minimizing (12) can be seen as
arg min xent(Y, arg min E y~Y | log ply [X]
(13)
arg H arg max ((;
(14)
where the last equality uses the fact that H(Y is independent of model parameters. Therefore, cross- entropy minimization approximate maximization of the mutual information between representation X and labels Y.
The approximation is that we parameterized q (1: Φ) as a linear projection. This structure cannot generalize to new classes because the parameters Φ are specific to the labels y seen during training. For model to generalize to unseen classes, one must amortize the learning of this approximate conditional distribution. (Vinyals et al., 2016; Snell et al., 2017) sidestepped this issue by using the embeddings for each class Φ.
Triplet Loss
The Triplet loss (Hoffer & Ailon, 2015) is defined as
Lupplt = =||X,, - x2 - ||ا x,
(15)
where N,, Xp, X,, Rdd the embedding vectors of query, positive, and negative images. Let Y
denote the label of the query data. Recall that the pdf function of unit Gaussian is log N ((x114, , 1) =
-C c2 /1 where C1, C2 are constants. Let (x) = N( 1) and Pa. (x) = N(x,, 1) be
unit Gaussian distributions centered Xp; X, respectively. We have
E
Lriple)]
x E [logP,(x) log P. (x))
(16)
~ [log log p())
(17)
((X (X( ((X;Y).
(18)
Two approximations were made in the process. We first assumed that the embedding distribution of images not in y, is equal to the distribution of all embeddings. This is reasonable when each class only represents a small fraction of the full data. We also approximated the embedding distributions y( (() with unit Gaussian distributions centered single samples from each.
N-pair Loss Multiclass N-pair loss (Sohn, 2016) was proposed as an alternative to Triplet loss. This loss function requires one positive embedding x and multiple negative embeddings x.,...> -- and takes the form
exp(x x++
log exp xx+) + EN exp((xxxx x,)
(19)
This can be seen as the cross-entropy loss applied to softmax x xi...., xx,xxxT x1).
Following the same logic as the cross-entropy loss, this also an approximation to I(X; This objective should have less variance than Triplet loss since it approximates p(x) using examples.
12

---

Under review as a conference paper at ICLR 2020
Adversarial Metric Learning Deep Adversarial Metric Learning (Duan al., 2018) tackles the problem most negative exmples being uninformative by directly generating meaningful negative embeddings. This model employs generator which takes input the embeddings of anchor, positive, and negative images. The generator then outputs a "synthetic negative" embedding that is hard to distinguish from a positive embedding while being close to the negative embedding.
This can be seen as optimizing
Po log Y(
(20)
by estimating px) using a generative network rather than directly from samples. Rather than modelling the marginal distribution p(x). this method conditionally models p(x; x, Xp, xn) so that X is hard to distinguish from Xp while sufficiently close to both X, and Xn.
B PROOF THEOREM
We restate and prove our main theorem.
Theorem 1. Let de be the VC dimension of the encoder X(). Let (X (Xr, ()x 0);Yr) be the empirical estimate of the mutual information using finite dataset Drr, and define empirical loss as
É(T -0== - (1)(Xr;0):
(21)
The following inequality holds with high probability:
L(T, 0) - L(T .0 S.00
(v de
log +
n de
+0
[X] log(m) mm
+0
[X | |
(22)
y
Proof. We use the following lemma from Shamir et al. (2010), which we restate using notation.
Lemma 1. Let X be random mapping of X. Let D be sample of size drawn from the joint probability distribution p(X, )). Denote the empirical mutual information observed from D between
X and Y as ()) For any ô E (0,1), 1), the following holds with probability at least 1 - ô:
(3 (3|X) + 2) log(m) 1og(4/8)
(YY + 1)( + 1)
-
(23)
V2m
We simplify this and plug our specific quantities of interest (X 0), Yr):
/ (x(X7. 0): (x(X7. ():K) 50 {0(1i [x]log(m)
) -
(24)
v m
We similarly bound the enror caused by estimating L with a finite number of tasks sampled from Denote the finite sample estimate L as
a n
L(r,
(x(
(25)
Let the mapping X be parameterized by 0 E θ and let this model have VC dimension dg. Using de. we can state that with high probability.
( log
de
-(),)) so
((T,0)
(26)
de
where de is the VC dimension of hypothesis class O.
13

---

Under review as a conference paper at ICLR 2020
Combining equations (26, 24), have with high probability
((T,0) (----121 (xx 0): 1):Y
(27)
$ C(T, -L(T, 0)
| log(m)
|
+0
(28)
vm
<0 (1 de n
[ log(m)
[
log
+0
+ 0
(29)
de
Vm
C EXPERIMENTS AND IMPLEMENTATION DETAILS
Every experiment was conducted on single Nvidia V100 GPU with CUDA 9.2. We used PyTorch version 1.0.1. Each experiment performed with different fixed initial seeds; manually fix seeds with manual __eed_eee________ () for python, pyt pytorch, and numpy.
Hardware
Optimizer For experiments with the 4-layer convnet, 2014) with learning rate 3e-4. For the Inception network, we use SGD with learning rate 3e-5 and momentum 0.9.
the Adam optimizer (Kingma & Ba,
We report the average of 500 batches of 1-shot accuracies and mutual information. I(X;Y) was computed using balanced batches of 16 images each from different classes, We additionally show in Figure 5 the correlation between 1-shot accuracies, Recall@1, and NMI using three previously proposed losses (triplet, npair, protonet).
Small Train Set Experiment For this experiment, we used the Adam optimizer and performed a log-uniform hyperparameter sweep for learning rate E [Ic-7, 1e-3] For DIMCO, we swept p e made the embedding dimension E [16, 32). For
(32, 128) and d E [16, 32). For other methods, each combination of loss and number of training examples per class, we ran the experiment 64 times and reported the mean and standard deviation of the top 5.
14

---

Under review as a conference paper at ICLR 2020
0.40
0.25
0.15
0.175
0.150
Joss
triplet
0.125
0.300
piotonet
0.35
0.30
0.25
0.20
0.30
0.26
0.20
0.15
0.50
5-waa
10-wag
20-way
R81
ami
Figure 5: Correlation between few-shot accuracy and retrieval measures.
15

---

Under review as a conference paper at ICLR 2020
Figure 6: Additional visualizations of codes.
16


Reviews:
[{"review_id": "ryeYXgyxjH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to maximize improve generalization in meta-learning by learning discrete codes via a mutual information maximization objective. I liked the motivation and presentation of the paper but see some critical shortcomings:\n\n- In Theorem 1, shouldn't there be a term that accounts for the complexity of the hypothesis class (VC-dim, Rademacher complexity etc.). Can we actually verify Theorem 1 in practice on synthetic distributions to get a sense of the constant terms?\n- The experiments do not compare with any mutual information related baselines. Eg, VIB [Alemi et al.]. This comparison is critical to stress the importance of discrete codes as opposed to continuous. \n- Can the authors shed more light on the tradeoffs between p and d? Empirical insights would lend more intuition and understanding.\n- Discreteness in activations is one form of regularization to reduce |tilde(x)|. Would regularizing the weights of the stochastic encoder (say by variational inference or minimizing say l2 norm) have the same/better/worse regularization effect (as done in Bayes by Backprop)?\n- There are a few missing references on stochastic encoders trained based on variational information maximization [1, 2]. \n\nReferences:\n[1] Uncertainty Autoencoders: Learning Compressed Representations via Variational Information Maximization. AISTATS 2019.\n[2] Neural Joint Source-Channel Coding. ICML 2019.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rkli6dJRYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents DIMCO, a meta-learner that is trained by maximizing mutual information between a discrete data representation and class labels across tasks. DIMCO is inspired by an information theoretic lower bound on the generalisation gap for meta-learning, which the authors argue identifies overfitting in the task learner as the bottleneck. \n\nThis work proposes to constrain a learner to output discrete codes that are learned to capture the mutual information with class labels. While the idea of using discrete codes is interesting, its presentation in the manuscript is not well motivated and at times hard to follow. This makes it challenging to evaluate the novelty, validity, and generality of the proposed approach. Meanwhile, the empirical evaluation is somewhat lacking. Thus, I do not believe this work is ready for publication in its current form.\n\nDetailed comments\n\nMy main concern is with respect to the primary contribution of this paper, a generalisation bound on meta-learning. The bound appears to be on a multi-task loss without task adaptation, and thus the claims made with respect to the theorem seem somewhat over-reaching. I also believe the VC-dimensionality of the encoder is missing in Eq. 4? If so, this changes the interpretation since the length of the code and the expressivity of the encoder are interrelated. Further, I would welcome a deeper analysis of the theorem and its implications. The current interpretation states that the number of tasks is independent of the size of each task, hence given many tasks, using minimal representations is an effective approach to meta-generalisation. Yet minimal representations is a well-known idea and has features in several works that use mutual information as a regularizer, most notably works on the Information Bottleneck. \n\nAnother reservation I have is the use of mutual information between encoder representations and class labels as a loss function (Eq. 1). It lacks context and a proper motivation, especially since the analysis of [1] shows that the loss function in Eq. 1 is the cross-entropy objective. The authors make a similar analysis in Appendix A and argue that Eq. 1 differs in that cross-entropy is an approximation because it adds a parametrized linear layer on top of \\tilde{X}. Thus, in the absence of that layer they collapse to the same objective. As DIMCO itself directly extract class label predictions from \\tilde{X}, I fail to see a difference between the loss in Eq. 1 and a cross-entropy objective. \n\nThe main motivation behind their loss objective is that it does not require a support / query set. This does not seem to be a feature of the mutual information objective itself, but rather a choice made by the authors. I would have liked a deeper discussion of this seeing as the authors make it a central tenet of DIMCO. Prior works use a support set as a principled means of doing meta-learning: meta-training explicitly takes into account that at test time, the learner will be given a small support set from which to learn how to query points. As far as I understand, DIMCO does not take this into account during meta-training.  At meta-test time however, DIMCO does use a support set to map query points (Eqs. 10 and 11). Why should we break protocols between meta training and testing? Are there any downsides to doing so?  \n\nEmpirically, I find the CUB experiment compelling but would welcome some ablations. What are the trade-offs between p and d? Can DIMCO outperform N-pair when number of bits are unconstrained?\n\nminiImagenet is a standard benchmark in few-shot learning, but I am unable to find a results table - could the authors please provide results on the standard setup so that the method can be compared against known baselines? Further, would the results currently presented hold in a N-way-5-shot setup?\n\nAs for the constrained version of miniImagenet that the authors propose, I am not convinced this is an interesting protocol. In general, the miniImagenet task distribution is created by N-way permutations of the classes in the meta-set (e.g. meta-training tasks are combinations of the 64 classes in the meta-training set). By keeping the number of classes constant but reducing the number of images per class, this protocol is not reducing samples per task: a task is always defined as 5/20-way-1-shot (Fig. 4). Instead, the effect should be that tasks are in (greater) violation of the task i.i.d. assumption. Thus, I question whether this setup demonstrates the trade-offs the authors present in Theorem 1 and whether the results can be interpreted in light of it. \n\nFinally, that both experiments are image-based raises questions as to the generality of the method. The paper could be considerably strengthened by evaluating DIMCO on a non-image task, or if not discuss the method’s limitations.  \n\nThe idea of discrete codes for few-shot classification is interesting and sufficiently novel, I am likely to increase my score if my concerns are addressed and the experimental section is strengthened.\n\nFurther questions and comments:\n\n- I am unable to parse Eq. 11 - what does the notation \\prod_i p_{\\tilde{x}_i, i} mean? \n- It is unnecessarily hard to follow the proof of theorem 1. It would help the reader if you restated relevant definitions, such as Eq. 1, since the difference with Eq. 23 is very subtle. It would also be helpful to explain how the summand in Eq. 25 differs from either, and Eq. 26 could be expanded or briefly explained after the derivation. \n-  Because DIMCO is trained with backpropagation, the interpretation of Eq. 5 as d independent events seems invalid. How does it affect the method if they are not independent?\n- Overloading X and Y as both random variables and mini-batch samples creates unnecessary confusion. I believe the objective in Eq. 1 is approximated, not calculated exactly? For instance, the mutual information in Eq. 8 is with respect to a mini-batch, so should it not be \\hat{I}? \n- p^j_{ik} in Eq. 9 is undefined.\n- Eq. 9 is interpreted as an exact entropy, however it appears to be a mini-batch approximation to the true entropy? \n\nReferences \n[1] Achille and Soatto. Emergence of Invariance and Disentanglement in Deep Representations. JMLR. 2018.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BklVIAW49B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to learn classifier outputs for meta learning in the form of factorized discrete codes by maximizing the mutual information between the model's outputs and the ground truth labels. The authors further present an information theoretic generalization bound for meta learning in terms of the number of tasks, number of training samples per task and the expressively of the model. They further show empirically that their approach does not need a separate query set during meta-training and can generalize better than a few of the other metric-learning based meta-learning approaches specifically at lower shot values. They show that their method requires less memory than the N-pair meta-learning method.\n\nThe paper addresses an important problem of generalization with very low shot values and proposes an interesting theoretical treatment of the problem in terms of deriving an information theoretic lower bound for it. I liked the authors' theoretical treatment of relating various metrics to the mutual information between the models' outputs and their labels. \n\nHowever, there have been many recent works attempting to address the problem of better generalization of meta learning models. Most notable among them is the work of Kim et al. \"Bayesian Model-Agnostic Meta-Learning\" (https://arxiv.org/pdf/1806.03836.pdf), which is not referenced or compared against in this paper at all.\n\nThe proposed method is also limited in its scope to classification tasks only and the authors make no attempt to address regression or reinforcement learning problem, which limits is widespread applicability.\n\nWhile the authors address the generalization of meta-learning methods for small values of M, they do not address how their model behaves viz-a-viz others when the number of tasks N available for training is small. Theoretically having more compact representations should also help in situations where the number of tasks available for training are small. I would like to see an empirical analysis of that as well.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were unanimous that this submission is not ready for publication at ICLR in its current form.

Concerns raised included that the method was not sufficiently general, including in choice of experiments reported, and the lack of discussion of some lines of significantly related work.
Decision: Reject