Meta Dropout: Learning to Perturb Latent Features for Generalization
Authors: Hae Beom Lee
Taewook Nam
Eunho Yang
Sung Ju Hwang
Venue:  (2020)

Abstract: A machine learning model that generalizes well should obtain low errors on unseen test examples. Thus, if we know how to optimally perturb training examples to account for test examples, we may achieve better generalization performance. However, obtaining such perturbation is not possible in standard machine learning frameworks as the distribution of the test data is unknown. To tackle this challenge, we propose a novel regularization method, meta-dropout, which learns to perturb the latent features of training examples for generalization in a meta-learning framework. Specifically, we meta-learn a noise generator which outputs a multiplicative noise distribution for latent features, to obtain low errors on the test instances in an input-dependent manner. Then, the learned noise generator can perturb the training examples of unseen tasks at the meta-test time for improved generalization. We validate our method on few-shot classification datasets, whose results show that it significantly improves the generalization performance of the base model, and largely outperforms existing regularization methods such as information bottleneck, manifold mixup, and information dropout.

Published as a conference paper at ICLR 2020
META DROPOUT: LEARNING TO PERTURB LATENT FEATURES FOR GENERALIZATION
Hae Beom Lee', Taewook Nam', Eunho Sang",,,ungy Ju Hwang
KAIST KAIST",ATTRICS' AITRICS" South Korea (haebeom. 1ee, namsan, eunhoy, sihwang32)ekaist acc
ABSTRACT
A machine learning model that generalizes well should obtain low errors on un- seen test examples. Thus, if we know how to optimally perturb training examples to account for test examples, we may achieve better generalization performance. However, obtaining such perturbation is not possible in standard machine Icarn- ing frameworks as the distribution of the test data is unknown. To tackle this challenge, we propose novel regularization method, meta-dropour, which learns to perturb the latent features of training examples for generalization in a meta- learning framework. Specifically, we meta-learn noise generator which outputs a multiplicative noise distribution for latent features, to obtain low errors on the test instances in an input-dependent manner. Then, the learned noise generator can perturb the training examples of unseen tasks at the meta-test time for improved generalization. We validate our method on few-shot classification datasets, whose results show that it significantly improves the generalization performance of the base model, and largely outperforms existing regularization methods such as in- formation bottleneck, manifold mixup, and information dropout.
INTRODUCTION
Obtaining a model that generalizes well is fundamental problem in machine learning, and is be- coming even more important in the deep learning era where the models may have tens thousands of parameters. Basically, a model that generalizes well should obtain low error unseen test ex- amples, but this is difficult since the distribution of test data unknown during training. Thus, many approaches resort to variance reduction methods, that reduce the model variance with respect to the change in the input. These approaches include controlling the model complexity (Neyshabur et al., 2017), reducing information from inputs (Tishby et al., 1999), obtaining smoother loss sur- face (Shirish Keskar et al., 2017; Neyshabur et al., 2017; Chaudhari et al., 2017; Santurkar et al., 2018), smoothing softmax probabilities (Pereyra et al., 2017) or training for multiple tasks with multi-task (Caruana, 1997) and meta-learning (Thrun & Pratt, 1998).
A more straightforward and direct way achieve generalization is simulate the test exam- ples by perturbing the training examples during training. Some regularization methods such as mixup (Zhang et al., 2017) follow this approach, where the training examples are perturbed the direction of the other training examples to mimic test examples. The same method could be also applied to the latent feature space, achieve even larger performance gain (Verma et al., 2019). However, these approaches are all limited in that they do not explicitly aim to lower the general- ization error on the test examples. How can we then perturb the training instances such that the perturbed instances will be actually helpful in lowering the test loss? Enforcing this generalization objective is not straightforward in standard learning framework since the test data unobservable.
To solve this seemingly impossible problem, we resort to meta-learning (Thrun & Pratt, 1998) which aims to learn a model that generalize over a distribution of tasks, rather than a distribution of data instances from single task. Generally, a meta-learner is trained a series of tasks with random training and test splits. While learning to solve diverse tasks, it accumulates the knoweee that is not specific to a single task, but is generic across all tasks, which is later leveraged when learning for a novel task. During this meta-training step, we observe both the training and test data. That is, can explicitly learn to perturb the training instances to obtain low test loss in this meta-learning

---

Published as a conference paper at ICLR 2020
framework. The learned noise generator then can be used to perturb instances for generalization at
meta-test time.
Yet, learning how much and which features to perturb is dif- ficult for two reasons. First of all, meaningful directions of perturbation may differ from one instance to another, and one task to another. Secondly, single training instance may need cover largely different test instances with its perturbation, since we do not know which test instances will be given at test time. To handle this problem, we propose learn an input- dependent stochastic noise; that is, we want to learn distribu- tion of noise, or perturbation, that is meaningful for given training instance. Specifically, we learn a noise generator for each layer features of the main network, given lower layer fea- tures as input. We refer to this meta-noise generator meta- dropout that learns to regularize.
After per perturbation
train
test
train
Lest
train
test
Figure 1: Concepts. In the feature space, each training instance stochas- tically perturbs so that the resultant decision boundaries (red linc) explain well for the test examples. Note that noise distribution does not have to cover the test instances directly.
Also the learned noise distribution is transferable, which is es- pecially useful for few-shot learning setting where only a few examples are given to solve novel task. Figure depicts such a scenario where the noise generator perturbs each input in- stance help the model predict better decision boundaries.
In the remaining sections, we will explain our model in the context of existing work and propose the learning framework for meta-dropout. We compare our method existing regularizers such as manifold mixup (Verma et al., 2019), information bottleneck, and information dropout (Achille & Soatto, 2018), which our method significantly outperforms. We further show that meta-dropout can be understood as meta-learning the variational inference framework for the graphical model in Figure 3. Finally, we validate our work on multiple benehmark datasets for few-shot classification.
Our contribution is threefold.
We propose novel regularization method called meta-dropout that generates stochastic input-dependent perturbations to regularize few-shot learning models, and propose a meta- learning framework train it.
We compare with the existing regularizers such as information bottleneck (Achille & Soatto, 2018; Alemi al., 2017) and manifold mixup (Verma et al., 2019). We also pro- vide the probabilistic interpretation of our approach as learning to regularize the variational inference framework for the graphical model in Figure 3.
We validate our method on multiple benchmark datasets for few-shot classification, which our model significantly improves the performance of the base models.
RELATED WORK
Meta learning While the literature on meta-learning (Thrun & Pratt, 1998) is vast, here dis- cuss a few relevant existing works for few-shot classification. One of the most popular approaches is metric-based meta-learning that learns shared metric space (Koch al., 2015; Vinyals et al., 2016; Snell et al., 2017; Oreshkin et al., 2018; Mishra et al., 2018 8) over randomly sampled few-shot classification problems, embed the instances to be closer their correct embeddings by some distance measure regardless of their classes. The most popular models among them are Match- ing networks (Vinyals et al., 2016) which leverages cosine distance measure, and Prototypical net- works (Snell et al., 2017) that make use of Euclidean distance. On the other hand, gradient-bart based approaches (Finn et al., 2017: 2018: Li al., 2017: Lee & Choi, 2018; Ravi & Beatson, 2019; Zint- graf et al., 2019) learns a shared initialization parameter, which can rapidly adapt to new tasks with only a few gradient steps. Recent literatures on few-shot classification show that the performance can significantly improve with larger networks, meta-level regularizers or fine-tuning (Lee et al., 2019; Rusu et al., 2019). Lastly, meta-leaming of the regularizers has been also addressed Balaji et al. (2018), which proposed to meta-learn & regularizer for domain adaptation. However, while this work focuses on the meta- meta-learning of the hyperparameter of generic regularizers, our model is more explicitly targeting generalization via input perturbation.
2

---

Published as a conference paper at ICLR 2020
Dropout Dropout (Srivastava et al., 2014) is regularization technique to randomly drop out neurons during training. In addition to feature decorrelation and ensemble effect, we could also interpret dropout regularization as variational approximation for posterior inference of the net- work weights (Gal & Ghahramani, 2016), in which case we can even learn the dropout rates with stochastic gradient variational Bayes (Kingma al., 2015; Gal et al., 2017). The dropout regular- ization could be viewed as noise injection process. In case of standard dropout, the noise follows the Bernoulli distribution, but we could also use Gaussian multiplicative noise to the parameters instead (Wang& & Manning, 2013). It is also possible to learn the dropout probability in an input- dependent manner as done with Adaptive Dropout (Standout) (Ba & Frey, 2013), which could be interpreted variational inference on input-dependent latent variables (Sohn et al., 2015; Xu et al., 2015; Heo et al., 2018; Lee et al., 2018). However, meta-dropout fundamentally different from adaptive dropout, as it makes use of previously obtained meta-knowlede in posterior variance in- ference, while adaptive dropout resorts only to training data and prior distribution.
Regularization methods There exist large number of regularization techniques for improving the generalization performance of deep neural networks (Srivastava et al., 2014; Ioffe & Szegedy, 2015: Ghiasi et al., 2018), but we only discuss approaches based on input-dependent perturbations that are closely related to meta-dropout, Mixup (Zhang et al., 2017) randomly pairs training instances and interpolates between them to generate additional training examples. Verma al. (2019) further extends the technique to perform the same procedure in the latent feature spaces at each layer of deep neural networks. While mixup variants are related to meta-dropout as they generate additional training instances via input perturbations, these heuristics may or may not improve generalization, while meta dropout perturbs the input while explicitly aiming to minimize the test loss a meta- learning framework. information-theored are also relevant to our work, where they inject input-dependent noise to latent features to forget some of the information from the inputs, resulting in learning high-level representations that invariant to less meaningful variations. Yet, meta-dropout has a more clear and direct objective to improve the generalization. Adversarial learning (Goodfellow al., 2015) is also somewhat related to our work, which perturbs the examples towards the direction that maximizes the training loss, as meta dropout also tend to pertub inputs to the direction of decision boundaries. In the experiments section, we show that meta dropout also improves adversarial robustness, which implies the connection between adversarial robustness and generalization (Stutz et al., 2019).
regularizers (Alemi et al., 2017; Achille & Soatto, 2018)
LEARNING TO PERTURB LATENT FEATURES
We now describe our problem setting and the meta-learning framework for learning perturb train- ing instances in the latent feature space, for improved generalization. The goal of meta-learning is to learn a model that generalizes over task distribution p(7). This is usually done by training the model over large number of tasks (or episodes) sampled from p(T), each of which consists of training set D {(x," y.! ) and test set Die ((x)))))) )
Suppose that we are given such a split of DDr and D Denoting the initial model parameter of an arbitrary neural network as 0, Model Agnostic Meta Learning (MAML) (Finn et al., 2017) aims to infer task-specific model parameter 0* with one or a few gradient steps with the training set Dir, such that 0* can quickly generalize to D with a few gradient steps. Let denote the inner-gradient step size, X and Y denote the concatenation of input data instances and their associated labels respectively for both training and test set. Then, have
1 M
log log PY 07) :
where 0-= - vVo (- 1og
min EP(T)
(1)
Optimizing the objective in Eq. 1 is repeated over many random splits of D and D such that the initial model parameter 0 captures the most generic information over the task distribution p(T).
3.1 META-DROPOUT
A notable limitation of MAML is that the knowledge transfer to unseen tasks is done only by sharing the initial model parameter over the entire task distribution. When considering few-shot classifica- tion task for instance, this means that given the initial 0, the inner-gradient steps the meta-test
3

---

Published as a conference paper at ICLR 2020
time only depends on few training instances, which could potentially lead to learning sub-optimal decision boundaries. Thus, it would be desirable we could transfer additional information from the meta-learning step, that could help the model to generalize better. Based on this motivation, propose to capture a transferable noise distribution from the given tasks at the meta-training time, and inject the learned noise the meta-test time, such that the noise samples would perturb the latent features of the training instances to explicitly improve the decision boundaries.
Toward this goal, we propose to learn an input-dependent noise distribu- tion, such that the noise is individually tailored for each instance. This because the optimal direction and the amount of perturbation for each input instance may vary largely from one instance to another. We empir- ically validate the effectiveness of this input-dependent noise generation in our experiments (Table 3). We denote the shared form of the noise dis- for
Main network
Noise generator
FC
Noise
Conv
Noise
X
tribution as p(z) x 0, Φ). with Φ as the additional meta-paramess the noise generator that does not participate in the inner-optimization. Note that the noise distribution has dependency on the main parameter 0, due to input dependency (see Figure 2).
Figure 2: Model archi-
tecture. Each bottom layer generates the noise for up- per layer with parameter &b
Recall that each inner- mener-rrdieet step requires to compute the gradient over the training marginal log- likelihood (Eq. 1). In our case of training with the input-dependent noise generator p(z]x"; 0, Φ), the marginal log- log-liielihood is obtained by considering all the plausible perturbations for each in- stance: log p( ; 0, Φ) = En= log Ez, ~p(x; (x)000)) [p(y"] |x] z;; 0)]. In this work, we instead consider its lower bound, which simply corresponds to the expected loss over the noise distribution (See section 3.2 for more discussion).
N
log D X": 0, +) 2
Ez peppplex))) [log [ogg(yy]xx z:; 0)]
(2)
We take the gradient ascent steps with this lower bound. Following Kingma & Welling (2014), use reparameterization trick to evaluate the gradient of the expectation w.r.. and Φ. such that z Softplus(p= E) and (O, I) (See Eq. 5). Then, the associated Monte-carlo (MC) samples allow to compute the gradients through the deterministic function μ parameterized by & and O. We use MC sample size S = for meta-training and S 30 for meta-testing.
N
0 = 0 +
Vo log plyplyl 0),
z
(a) ~
p(z, 6
(3)
N
S
By taking the proposed gradient step, the target learning process can consider all the plausible per- turbations of the training examples that can help explain the test dataset. Extension to more than one inner-gradee inner-gradient step also straightforward: for each inner-gradient step, we perform MC integration to estimate the model parameter at the next step, and repeat this process until we get the final 0"
Finally, we evaluate and maximize the performance of 0 on the test examples, by optimizing and & for the following meta-objective over the task distribution p(T). Considering that the test examples should remain as stable targets we aim to generalize to, we deterministically evaluate its tog-likelihood by forcing the variance of a to be zero in Eq. 5 (i.c. μ). Denoting as the one obtained from such deterministic a, we have
M
max 0.4
E PP(T)
log logply! x Z.; 0*)
(4)
M
By applying the same deterministic transformation μ both training and test examples, they can share the same consistent representation space, which seems important for performance, Lastly, to compute the gradient of Eq. w.r.t. d, we must compute second-order derivative, otherwise the gradient w.r.t. o will always be zero. See Algorithm and 2 for the pseudocode of Meta-dropout.
Form of the noise We apply deput-dep multiplicative noise to the latent features all lay- ers (Ba & Frey, 2013) (See Figure 2). Here we suppress the dependency on 9 and & for better readability. We propose simple Softplus transformation of Gaussian noise distribution. We
4

---

Published as a conference paper at ICLR 2020
could use other types of transformations, such exponential transformation (i.e. Log-Normal dis- tribution), but we empirically verified that Softplus works better. First, we generate input-dependent noise 2 given the latent features h(1-1) from the previous layer.
z (1) Softplus(a a Naaa (
(5)
where 10 : (h((1)) is parameterized by & and Note that although the variance of a is fixed as I, we can still adjust the noise scale at each dimension via the mean μ()) with the scale of the noise suppressed at certain dimensions by the Softplus function. While we could also model the variance in an input-dependent manner, we empirically found that this does not improve the generalization performance (Table 3). Then, we can obtain the latent features h(²) for the current layer by applying the noise to the pre-activation f(l) (h(-)) parameterized by
h (1) ReLU(F()) 20))
where o denotes the element-wise multiplication.
3.2 CONNECTION TO VARIATIONAL. INFERENCE
Lastly, explain the connection between the lower bound in Eq. 2 and the vari- ational inference with the graphical model Figure 3. Suppose we are given a training set D {(x, ((xx:;))] y=)] where for each instance x;, the generative process involves latent 2; conditioned on X;. Note that during the inner-gradient steps, the global parameter d² is fixed and only optimize O. Based on this specific context, we derive the lower bound L(0) as follows:
Food
N
Figure Graphical model
3:
log p Y Y|X; )b))
2 Σi 1= x plyp z,; 0)] K.lgfa//x. (7)
-E- N E 0x) [logply l. 0)]
T()))
where from Eq. 7 to Eq. 8 we let the approximate posterior g(z)x,yr y) share the same form with the conditional prior p(z)x 0, d²), such that the KL divergence between them becomes zero. By sharing the same form, we can let the training and testing pipeline be consistent (note that the label y not available for the test examples, see Sohn et al. (2015) and Xu al. (2015)). By maximizing (()), we obtain the task-specific model parameter O that can make more accurate predictions on the test examples, owing to the knowledge transfer from d², which regularizes the final form of the conditional prior p(z x: 0" +
4 EXPERIMENTS
We now validate our meta-dropout on few-shot classification tasks for its effectiveness.
Baselines and our models We first introduce the most important baselines and model. We compare against other few-shot meta-learning baselines, using their reported performances.
1) MAML. Model Agnostic Meta Learning by Finn al. (2017). First-order approximation is not considered for fair comparison against the baselines that use second-order derivatives.
2) Meta-SGD. A variant of MAML whose learning rate vector is element wisely learned for the inner-gradient steps (Li al., 2017).
3) Meta-dropout. MAML or Meta-SGi StD with our learnable input-dependent noise generator.
Datasets We validate method on the following two benchmark datasets for few-shot classifi- cation. 1) Omniglot: This gray-scale hand-written character dataset consists of 1623 classes with 20 examples of size 28 28 for each class. Following the experimental setup of Vinyals et al. (2016), we use 1200 classes for meta-training, and the remaining 423 classes for meta-testing. We further augment classes by rotating 90 degrees multiple times, such that the total number of classes is 1623 x 4. 2) minilmageNet: This is subset of ILSVRC-2012 (Deng et al., 2009), consisting of 100 classes with 600 examples of size 84 x 84 per each class. There are 64, 16 and 20 classes for meta- train huin/valiiation/test respectively.
5

---

Published as a conference paper at ICLR 2020
Table 1: Few-shot classification performance conventional 4-layer convolutional neural networks. All reported results are average performances over 1000 randomly selected episodes with standard errors for 95% confidence interval over tasks.
Omniglot 20-way
minilmageNee 5-way
Models
1-shot
5-shot
1-shot
5-shot
Meta- Learning LSTM (Ravi & Laruchelle, 2017)
43.44 43.74
60.60
Matching Networks (Vinyals et al., 2016)
93.8
98.7
43.56
$5.31
Prototypical Networks (Snell et al., 2017)
95.4
98.7
46. 14.1..
65.77
Prototypical Networks (Snell et al., 2017) (Higher way)
96.0
98.9
49. 49.442+077
68.20=
MAML (our reproduction)
95.23+0.17
98.38+a07
49.58=20.5
64.55
Meta-SGID (our reproduction)
96. 16:101114
98054
48.30.20.44
60.55
Reptile (Nichol al., 2018)
89.4
97.12+0.2
49.97
65.9
Amortized Bayesian MI (Ravi & Beatson, 2019)
20.00
Probubilistic MAML (Finn el al., 2018)
50. 1.86
MT-Net (Lee & Choi, 2018)
96.2+04
51.70
CAVIA (512) (Zintgraf et al., 2019)
51.82.10.55
65.8510.55
MAML Meta-dropout
96.6310.13
98.731006
51.93.0.07
6..42..052
Meta-SGD Meta-dropout
97.02+t.13
99.05=aus5
50.87
65.55
teat
decaton boundary
(a) MAML (minilmageNet)
(b) Meta-dropout (minilmageNet) (c) Meta-dropout (Omniglot)
Figure 4: Visualization of the task-specific decision boundaries in the last latent feature space. We use the model trained with 1-shot. The visualizations are the projection of the features after completing the last (5t)) inner-gradient step, where the sampled 4 examples (2 examples (O.A) x 2 classes) participate in the inner- optimization. (a) and (b) are drawn from the same task. See Appendix C for the details about this visualization.
Base networks. We use the standard network architecture for the evaluation of few-shot classifi- cation performance (Finn al., 2017), which consists of convolutional layers with 3 kernels ("same" padding), and has either 64 (Omniglot) or 32 (minilmageNet) channels. Each layer is fol- lowed by batch normalization, ReL.U, and max pooling ("valid"" padding).
Experimental setup. Omniglot: For 1-shot classification,, we use the meta-batchsize of = and the inner-gradient stepsize of 0.1. For 5-shot classification, we use B = 6 and 0.4. We train for total 40K iterations with meta-learning rate 10-3, mimilmageNett: We use B = 4 and 0.01. We train for 60K iterations with meta-learning rate 10 Both datasets: Each inner- optimization consists of 5 SGD steps for both meta- veta-ttrainng and meta-testie meta-testing. Each task consists of 15 test examples. Note that the testing framework becomes transductive via batch normalization, as done in Finn et al. (2017). We use Adam optimizer (Kingma & Ba, 2014) with gradient clipping of We used TensorFlow (Abadi et al., 2016) for all our implementations.
3).
4.1 FEW-SHOT CLASSIFICATION EXPERIMENTS
Table 1 shows the classification results obtained with conventional 4-layer convolutional neural net- works on Omniglot and mimilmageNNet dataset. The base MAML or Meta-SGD with Meta-dropout outperform all the baselines, except for the Prototypical Networks trained with "higher way" (20- way) on the minilmageNet 5-shot classification! Figure visualizes the learned decision boundaries of MAML and meta-dropout. We observe that the perturbations from meta-dropout generate dat- apoints that are close to the decision boundaries for the classification task at the test time, which could effectively improve the generalization accuracy. See Appendix section E for more visualiza- tions, including stochastic activations and input-level projections of perturbations.
Strictly, this result is not comparable with others as all other models are trained with 5-way classification problems during meta-training.

---

Published as a conference paper at ICLR 2020
Table 2: Comparison against existing perturbation-baset regularization techniques. The performances are obtained by applying the regularizers to the inner gradient steps MAMI..
Models
Noise
Hyper-
Omniglot w0-
minilnageNet 5-way
(MAML +)
Type
parameter
1-shot
5-shot
1-shot
5-shot
No perturbation
None
95.232017
98.38
49.58=006
64,55
Input& & Manifold Mixup
Y = 0.2
89.78
97.86.1008
48.62.00.6
63.8620.51
(Zhang et al., 2017)
Pairwise
Y = 1
87.00=
97.27
48.24=0.22
62.322004
(Verma et al., 2019)
7 2
87.2622028
97.1410.17
48.42:20.4
62.56:10.5
Variational Information Bottleneck
8 = 10 5
92.09+0.22
98.85 +o000
48.12
64.78++0.44
Add.
B 10
93.01
98. 80
46.75=0.6
64:07
(Alemi et al., 2017)
8 == 10
94.98:1016
98.75
47.59_2060
63.30:2051
Information Dropout
3= 10
94.492211
98.5012077
50.36=0.68
65.91 :0:55
(ReLU ver.)
Mult.
8 == 10
94.36220.17
98.53
19.1422..63
64.9622
(Achille & Soatto, 2018)
8 3= 10
94.28=a017
98.65
43.78=0.61
63.3644056
Meta-dropout
Add. Mult.
0.1 None
96.55-0144
99.04:1.05
50.25-0066
66.78+a.8
(See Appendix for Add.)
96.63
98.73
51.93=0067
67.42:10.1
Comparison against perturbation-baset regularization methods In Table 2, we compare with the existing regularization methods based on input-dependent perturbation, such as Mixup (Zhang et al., 2017; Verma et al., 2019) and Information-theoretic regularizers (Achille & Soatto, 2018; Alemi et al., 2017). We train the base MAML with the baseline regularizers added to the inner- gradient steps, and set the number of perturbations for each step to 1 for meta-training and 30 for meta-testing, as in the case of Meta-dropout. We meta-train additional parameters from the baselines by optimizing them in the inner-gradics steps for fair comparison.
We first compare meta-dropout against mixup variants. The perturbation for mixup regularization is defined as linear interpolation of both inputs and labels between two randomly sampled training datapoints. We interpolate at both the input and manifold level following Verma et al. (2019). The interpolation ratio À [0, 1] follows X Beta(7, 7)), and we use 7 E {0.2, 1, following the settings of the original paper. Table 2 shows that Mixup regularization significantly degrades the few-shot classification performance within the range of y we considered. This is because, in the meta-lcarning framework, the interpolations of each task-adaptation process ignores the larger task distribution, which could conflict with the previously accumulated mnta-knowledge.
Next, we compare with the coarmation-thooreti regularizers. Information Bottleneck (IB) method (Tishby& & Zaslavsky, 2015) is a framework for searching for the optimal tradeoff between forget- ting of unnecessary information in inputs (nuisances) and preservation of information for correct prediction, with the hyperparameter 3 controlling the tradeoff. The higher the 3, the more strongly the model will forget the inputs. We consider the two recent variations for IB-regularization for deep neural networks, namely Information Dropout (Achille & Soatto, 2018) and Variational Infor- mation Bottleneck (VIB) (Alemi et al., 2017). Those information thhooreti regularizers relevant to meta-dropout as they also inject input-dependent noise, which could be either multiplicative (In- formation Dropout) additive (VIB). However, the assumption of optimal input forgetting does not hold in meta-learning framework, because it will forget the previous task information as well. Table 2 shows that these regularizers significantly underperform ours in the meta-learning setting.
Adversarial robustness Lastly, we investigate if the perturbations generated from meta-dropout can improve adversarial robustness (Goodfellow et al., 2015) as well. Projected gradient descent (PGD) (Madry et al., 2017) attack of É, l2. and foo norm is considered. As adversarial learning baseline on MAML framework, we compare with a variant of MAML that adversarially perturbs each training example in its inner-grade inner-gradient steps (10 PGD steps with radius for each inner-gradient step²). Their outer objective maintains as test loss for fair comparison with other models. At meta- test time, we test all the baselines with PGD attack (200 steps) of all types of perturbations.
In Figure 5. we see that whereas the baseline regularizers such as manifold-mixup and information- theoretic regularizers fail to defend against adversarial attacks, meta-dropout not only improves
We choose as minimum and maximum value within the overall attack range for the test examples. For the hyperparameters of each baseline, we choose the one with the hest accuracy in Table 2. We verified that for those baselines, interestingly, adversarial robustness not much sensitive hyperparameter selection.
7

---

Published as a conference paper at ICLR 2020
Table 3: Ablation study on the noise types applied to the inner-gradien steps. (V) means that the baseline is core component of Mela-dropout.)
Models
Rand.
Learn. mult.
Input dep.
Omniglot 20-way
minilmageNNet 5-way
(MAML+)
samp.
1-shot
5-shot
1-shot
5-shot
None
X
95.23++0m7
98,38=ua
49.58+0.45
64.554u.52
Fixed Gaussian (v)
X
X
93.44
98.99 -0.06
49.39+006
66.8440..4
Weight Gaussian
O
X
X
94.32.10.18
98. 98.35
49.37=0.64
64,7810.4
Independent Gaussian
O
O
X
94,36+0018
98.26=u.0s
50.31
66.97+t0.4
MAML+ + More param
O
97.85 15.83
50.63
65.20.1..5
Determ. Meta-drop. (v)
95.9911014
97.78=u.0
50.75
65.62
Meta-drop. w/ learned var.
95.98 1.045
98.87_.0.0
50,93 10.68
66. 15.056
Meta-dropout
96.63 0.13
98. L..06
51.93 0.67
67.42...52
106-
300
200
60-
eccurncy
60-
26-
Vetudrip
PAYL
Fix
20
20
20
20
pn0.1
20
64
60
100
0.0
0.34
0.26
6:32
0.32
600
002
6:30
6:30
opstinn
epston
epalio
opstion
(a) Omniglot I-shot (fx) (b) Omniglot I-shot (f2) (c) Omniglot I-shot (foo) (d) Omniglot 5-shot (Eiso
Figure 5: Adversarial robustness against PGD attack with varying size of radius e. The region of clean accuracies are magnified for better visualization.
generalization (clean accuracies), but also improves adversarial robustness significantly. It means that the perturbation directions toward test examples can generalize to adversarial perturbations. This result is impressive considering neither the inner nor the outer optimization of meta-dropout involves explicit adversarial training. Although we need further research to clarify the relationship between the two, the results are exciting since existing models for adversarial learning achieve ro- bustness at the expense of generalization accuracy. See Stutz et al. (2019) for the discussion about disentangling between generalization and adversarial robustness. The results from minilmageNet dataset in Appendix D. Although the gap is not as significant as Omniglot dataset, the overall tendencies look similar.
Moreover, Figure 6(a), 6(b) and 6(c) shows that meta- mrop-ur can even generalize across multiple types of adversarial atlacks we considered. Generalization between multiple types of perturbations only recently investigated by Tramèr Boneh (2019), which a very important problem as most of the existing adversarial training methods target a single perturbation type at time, limiting their usefulness. We believe that the results in Figure can have a great impact in this research direction.
Ablation study In Table 3, we compare our model against other types of noise generator, to jus- tify the use of input-dependent multiplicative noise for meta-dropout. We describe the baseline noise generators as follows: 1) Fixed Gaussian: MAMI. with fixed multiplicative noise, such that a in Eq. 5 follows a ~ (0, I) without any trainable parameters. 2) Weight Gaussian: MAML with fully factorized Gaussian weights whose variances are meta-learned. 3) Independent Gaussian: MAML with input-independent multiplicative noise, such that a ~ (N,,)) where learnable pa- rameter μe has the same dimensionality to the channels (similarly channel-wise dropout). We see from the Table 3 that neither the fixed distribution (Fixed Gaussian) nor the input-independerr noise (Weight Gaussian and Independent) outperforms the base MAML, whereas our meta- dropout model significantly outperforms MAML. The results support our claim that the input-dependent noise gen- erators are more transferable across tasks, as it can generate noise distribution adaptively for each instance.
We also compare with the MAML with additional parameters & (deterministic a in Eq. 5) that par- ticipate in the inner-gradient steps well, which we denote as (MAML + More param). We get slight improvements from the base MAML by doing so. Deterministic Meta-dropout is MAML + More param with & not participating in the inner-gradient steps as with meta-dropout. Although there are some gain from deterministically meta-learning the representation space with Φ, we can significantly improve the performance by injecting stochastic noise (meta-dropout). Lastly, we com-
8

---

Published as a conference paper at ICLR 2020
pare against variant of meta-dropout that instead of using fixed I in Eq. 5, learns the variance in an input-dependent manner (meta-dropout w/ learned variance) but underperforms meta-dropout with fixed variance.
Note that the slightly good performance of Deterministic Meta-dropout and Fixed Gaussian does not hurt claim the effectiveness of multiplicative noise, because multiplicative noise, by definition, consists of two parts: deterministic multiplication and pure random noise. For example, Bernoulli dropout consists of Bernoulli retain probability 0 S p (deterministic multiplication) and actual random sampling (pure random noise). In this vein, we can say that Deterministic Meta- dropout demonstrates the effectiveness of meta-learning the probability p. and Fixed Gaussian shows the effectiveness of injecting (multiplying) pure random noise (0, I) on each feature location. Overall, our meta-dropout combines the two components, which are complementary each other, into a novel input-dependent form of perturbation function.
5 CONCLUSION
We proposed a novel regularization method for deep neural networks, meta-dropout, which learns to perturb the latent features of training examples for improved generalization. However, learning how to optimally perturb the input is difficult in standard learning scenario, as we do not know which data will be given at the test time. Thus, we tackle this challenge by learning the input- dependent noise generator in a meta-learning framework, to explicitly train it to minimize the test loss during meta-training. Our noise learning process could be interpreted as meta-learning of variational inference for specific graphical model in Fig. 3. We validate our method on benchmark datasets for few-shot classification, on which significantly improves the generalization perfor- mance of the target meta-learning model, while largely outperforming existing regularizers based on input-dependent perturbation. As future work, we plan to investigate the relationships between generalization and adversarial robustness.
Acknowledgements This work was supported by Google AI Focused Research Award, the Engi- neering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF-201 10--200811100 ASA 1059919,,, Institute of Information & commu- nications Technology Planning Evaluation (ITP)) grant funded by the Korea government (MSIT) (No.2019-0-00075, and Artificial Intelligence Graduate School Program (KAIST)).
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale Machine Learning on Heterogeneous Distributed Systems. arXiv: 663.04467, 2016.
Alessandro Achille and Stefano Soatto. Information Dropout: Learning Optimal Representations
Through Noisy Computation. In PAMI, 2018
Alex Alemi, Ian Fischer, Josh Dillon, and Kevin Murphy. Deep variational information bottleneck. In ICLR, 2017.
Jimmy Ba and Brendan Frey. Adaptive dropout for training deep neural networks. In NIPS. 2013.
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gener- alization using meta-regullarizatinn In NeurIPS. 2018
R. Caruana. Multitask Learning. Machine Learning. 1997.
Chaudhari, A. Choromanska, S. Soatto, Y. LeCun, C. Baldassi, C. Borgs, J. Chayes, L. Sagun, and R. Zecchina. Entropy-SGD: Biasing Gradient Descent Into Wide Valleys. In ICLR, 2017.
J. Deng, W. Dong, R. Socher, L.-J. Li. K. Li, and L. Fei-F ei. Imagenet: A Scrle-Scale Hicrarchical Image Database. In CVPR, 2009.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- veta-learning for fast adaptation of deep networks. In ICMI., 2017.

---

Published as a conference paper at ICLR 2020
Chelsea Finn, Kelvin Xu, and Sergey Levine. NeurIPS, 2018.
Probabilistic model-agnostic meta-learning In
Y. Gal and Z. Ghahramani. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In ICML, 2016.
Y. Gal, J. Hron, and A. Kendall. Concrete Dropout. In NIPS, 2017.
Golnaz Ghiasi, Tsung-Yi Yi Lin, and Quoc V Le. Dropblock: A regularization method for convolu- tional networks. In NeurIPS, 2018.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Heo, B. Lee, S. Kim, J. Lee, K. J. Kim, Yang, and Hwang. Uncertainty-Aware Attention for Reliable Interpretation and Prediction. In NewrIPS, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In JMLR, 2015.
D. Kingma, T. Salimans, and M. Welling. Variational Dropout and the Local Reparameterization Trick. In NIPS, 2015.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.
Diederik P. Kingma and Max Welling. Auto encoding variational bayes. In ICI.R. 2014.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML, 2015.
Hae Beom Lee, Juho Lee, Sachoon Kim, Eunho Yang, and Sung Ju Hwang. Dropmax: Adaptive variationial softmax. In NeurlPS, 2018.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In CVPR, 2019.
Yoonho Lee and Seungjin Choi. Gradient-based meta-leamnin with learned layerwise metric and subspace. ICMI., 2018
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few shot learning. arXiv preprint 17X1v:1707.09885, 2017.
2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. arXiv e-prints, 2017.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. In ICIRR 2018.
B. Neyshabur, S. Bhojanapalli, D. McAllester, and N. Srebro. Exploring Generalization in Deep Learning. In NIPS, 2017.
Alex Nichol, Joshua Achiam, and John Schulman. On First-Order Meta-Learning Algorithms. arXiv e-prints, 2018
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In NeurlPS, 2018.
Gabriel Pereyra, George Tucker, Jan Chorowski, Fukasz Kaiser, and Geoffrey Hinton. Regularizing Neural Networks by Penalizing Confident Output Distributions. arXiv e-prints, 2017.
Sachin Ravi and Alex Beatson. Amortized bayesian meta-learning ICLR, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In ICLR, 2017.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In ICLR, 2019.
10

---

Published as a conference paper at ICLR 2020
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? In NeurIPS. 2018.
N. Shirish Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. In ICLR, 2017.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NIPS, 2017.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. NIPS. 2015.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. JMLR, 15:1929-1958 2014.
David Stutz, Matthias Hein, and Bernt Schiele. Disentangling adversarial robustness and general- ization. In CVPR, 2019.
Sebastian Thrun and Lorien Pratt (eds.). Learning to Learn. Kluwer Academic Publishers, Norwell, MA, USA, 1998. ISBN 0-7923-8047-9.
Naftali Tishby and Noga Zaslavsky. Deep Learning and the Information Bottleneck Principle. In IEEE Information Theory Workshop, 2015.
Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. In Annual Allerton Conference on Communication, Control and Computing. 1999.
Florian Tramèr and Dan Boneh. Adversarial training and robustness for multiple perturbations. CoRR, 2019.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Aaron Courville, Ioannis Mitliagkas, and Yoshua Bengio. Manifold Mixup: Learning Better Representations by Interpolating Hidden States. In /CML, 2019.
Oriol Vinyals, Charles Blundell, Timothy P. Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NIPS, 2016.
Sida Wang and Christopher Manning. Fast dropout training. ICMI., 2013.
K. Xu, J. Ba. R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In ICML, 2015.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In ICLR, 2017.
Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In ICML, 2019.

---

Published as a conference paper at ICLR 2020
A ALGORITHM
We provide the pseudocode for meta-training and meta-testing of Meta-dropout.
Algorithm Meta-training
1: Input: Task distribution (7)), Number of inner steps K, Inner step size a, Outer step size B.
2: while not converged do
3: Sample (D", D p(T)
4: Oo
0
5: for k 0 to K do
6: Sample 2, p(z, x Φ. 0k) for i= i......,
7:
0x+1= + 02 + a Ve N Ei=1 logp(y")> 2,; 0x)
8:
end for
9:
0 + 0K
10:
0 - + 3 Σ j=
Ve log log 1x 22 = =2; 0')
11:
Φ - o + 3
V log ply 1x x2 = 2-;0")
j=1
12: end while
Algorithm 2 Meta-testing
1: Input: Number of inner steps K, Inner step size MC sample size S.
2: Input: Learned parameter and Φ from Algorithm
3: Input: Meta-test dataset (Du, DD)).
4: 0o 4- 0
5: for 0 K do
6: Sample { [ s=1 p(zi ; &o 0x) for i= I,,...,,
( iid
7: Ok+1 + O2 + aaVa N
ΣN ESm log poyl"" x ;,) 0x)
8: end for
9: 0" + θK
10: Evaluate ply"" xelf 2, =Z;; 0*) for M
B META-DROPOUT WITH ADDITIVE NOISE
In this section, introduce the additive noise version of our meta-dropout, introduced in Table 2. At each layer I, add Gaussian noise (!) whose mean is zero and covariance is diagonal, i.c. (1)) ~ (z) J0, diag(o")), to the preactivation features at each layer, i.e. f()) := ( (1()))) The diagonal covariance is generated from each previous layer's activation (t-1) in input-dependent manner. i.c. o() o (h(((1)) and we meta-learn such covariance networks ( (L) over the task distribution. Putting together,
( (22 J0, X2 diag(a?)),
(9)
h (1) ReLU(F()) z)).
(10)
where hyperparameter for controlling how far each noise variable can spread to cover nearby test examples. We fix À 0.1 in all our experiments, that seems to help stabilize the overall training process. Reparameterization trick is applied obtain stable and unbiased gradi- ent estimation w.r.t. the mean and variance of the Gaussian distribution, following Kingma and Welling Kingma & Welling (2014).
We empirically found that this additive noise version of meta-dropout performs well Omniglot dataset, but shows somewhat suboptimal performance on more complicated dataset such as minilm- ageNet (See Table 2). We may need more research different noise types, but for now, we suggest to use the multiplicative version of meta-dropout in Eq. 5 which seems to perform well on most of the cases.
12

---

Published as a conference paper at ICLR 2020
C HOw TO VISUALIZE DECISION BOUNDARY
Visualization in Figure 4 shows mappee latent features and binary classification decision bound- ary of two random classes in task. After parameters are adapted the given task, we make binary classifier of random classes e₁ and C2 by taking only two columns of final linear layer parameters W Wes: Wes] and b = [bea bea]- Then we compute the last latent features H of data points
in the classes
The decision hyperplane of the linear classifier in the latent space given as,
h (we, -we) + (be, be,) =
(11)
X-coordinates in our 2D visualization are inner product values between latent features and the nor- mal vector of the decision hyperplane i.e. latent features projected to orthogonal direction of the decision hyperplane.
W
Wea
c²
(12)
We
Woz
Y-coordinates are determined by t-distributed stochastic neighbor emgedding (t-SNE) to reduce all other dimensions.
(Wey Woz) )T Weal
ea = iSNE H
C2
(13)
Wes
The points on the decision hyperplane is projected to a vertical line with following x-coordinate in the 2D space.
T
We. Wez Wes
boz boz
(14)
Wea
We,
Wez
D ADVERSARIAL ROBUSTNESS EXPERIMENT WITH MINIIMAGENET
Himstrpp
AdV. 6m20
Apv. en0
Adv. enle
AdV: 50
apsilon
spsiken
(a) Img 1-shot (f1) (b) Net 1-shot (f2) (c) ImgNet 1-shot (Loo) (d) ImgNet 5-shot (Loo) Figure 6: Adversarial robustness against PGD attack with varying size of radius c. The region of clean accuracies are magnified better visualization.
Figure 6 plots the clean and adversarial accuracies over varying size of c. Although the difference between models are not as remarkable as in Figure 5, our meta-dropout seems to be the only model that improves both clean and adverarial accuracies. More research seems required investigate what makes difference between Omniglot and mimilmageNet dataset.
E MORE VISUALIZATIONS
In order to see the meaning of perturbation at input level, we reconstruct expected perturbed image from perturbed features. A separate deconvolutional decoder network is trained to reconstruct orig- inal image from unperturbed latent features, then the decoder is used to reconstruct perturbed image from perturbed features of 3rd layer.
13

---

Published as a conference paper at ICLR 2020
Original Image
Without Noise
With Naise!
With Noise2
Original image
Without Nake
With Noiset
With Nolse2
A
R
A
50
O
2
C-
64
E9
C4
H
m
9
9
5
9
5
4
(b)
Original Image
With Noise1
Original Image
With Noise1
Noise
Noise2
Noise
Noise2
JJ
H
7
4
4
e
6
0
0
H
4
л
R
R
(c)
(d)
Figure 7: Feature Visualization Layer activations and perturbed activations in Omniglot datset.
14

---

Published as a conference paper at ICLR 2020
Original Image
Channel Activation
Channel 1 Perturbed Activation
Channel 2 Activation
Channel 2 Perturbed Activation
V
7
A
7
K
R
J
J
3
3
31
Et
13
(a) omniglot, layer
Original Image
Channel Activation
Channel 1 Perturbed Activation
Channel 2 Activation
Channel 2 Perturbed Activation
mimilmageNet, layer
Figure 8: Visualization of stochastic features. We visualize the deterministic activations and the correspond- ing stochastic activations with perturbation.
15


Reviews:
[{"review_id": "rJxKsYxycS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to meta-learn, using MAML, the mean of an elementwise, input-dependent, multiplicative noise to improve generalization in few-shot learning.\nThe motivation is that meta-learning the noise allows to learn how to best perturb examples in order to improve generlization.  This claim is supported by ample experimental evidence and comparisons against many baselines, as well as additional ablation studies w.r.t design choices of the algorithm itself. The paper is well written and easy to read. Consequently, I think this is a nice paper and should be accepted. \n\nEdit (leaving everything else unchanged for now): After reading R3's assessment, I agree with them that it's worrying that the Deterministic Meta-Dropout performs better than baseline MAML - maybe it's an effect of a larger number of parameters in the model? \n\nEdit:\nThank you for your response.\n\nI will leave my score as is.\n I would strongly encourage the authors to incorporate the baseline \"(1)\" as proposed by R3 in a future version of the paper as I agree with them that this is a relevant baseline.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1e1sc_GqB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes meta dropout, which leverages adaptive dropout training for regularizing gradient based meta learning models, e.g., MAML and MetaSGD. Experiments on few shot learning show that meta dropout achieves better performance.\n\nOverally, I think this paper is well motivated and experiments on few shot learning are impressive. I have only two major concerns.\n\n1. Sec 3.2. According to my understanding, Meta dropout introduces a learnable prior for latent $z$, but the training objective does not require posterior inference and thus no variational inference is needed. I think it is ok to say that meta dropout tries to optimize a lower bound of log p(Y|X;\\theta,\\phi^*), but meta dropout does not regularize the variational framework because there is no variational inference framework.\n\n2. Experiments on adversarial robustness can be further improved. (1) the settings and the analysis of adversarial robustness experiment can be discussed in details. For example, how to build ''adversarial learning baseline'' in meta learning settings and why the result implies the perturbation directions for generalization and robustness relates to each other; (2) how other regularization methods (e.g., Mixup, VIB and Information dropout) perform on adversarial robustness? Does Meta dropout performs better than them? (3) FGSM is a quite weak adversarial attack method, which makes evaluating adversarial robustness on FGSM may be misleading. I suggest trying some other STOA attack methods (e.g., iterative methods).\n\nSome typos: \nPage 3, Regularization methods, 3rd line, ````wwwdiscuss\nPage 7, 2nd line from the bottom, FSGM->FGSM\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lRMzRpYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes learning to add input-dependent noise to improve the generalization of MAML-style meta-learning algorithm. The proposed method is evaluated on OmniGlot and miniImageNet. The paper reports improvements upon MAML, MAML with meta-learned parameter-wise learning rates, as well as a few regularization methods that are based on input/hidden state perturbations (Mixup, Variational Information Bottleneck). An ablation study also compares the proposed meta-dropout algorithm with a number of modifications, such as a fixed noise, input-independent noise, etc. It is furthermore shown that meta-dropout somewhat improves the model’s robustness against an adversarial attack. \n\nThe paper is somewhat incremental considering that Li et al, (2017) and Balaji et al, (2018) have already proposed meta-learning parameter-wise learning rates and parameter-wise regularization coefficient respectively. One difference from the methods above is that in the proposed method noise is controlled by the input. The ablation however shows that in 5-shot classification case simply adding non-trainable noise works quite well. \n\nIt seems like the choice of the particular method for adding the noise was performed using the test set. If it’s true, this is methodologically wrong: model selection should be performed on a development set (or meta-development) set. Futhermore, Table 2 contains some results named “Add.”, which I guess stands for additive noise. I did not find an explanation of what is the specific method for adding noise used in this case. Such additive noise is also missing from ablation experiments. \n\nOverall, it seems that paper falls short of clearly proving that back-propagating through MAML to the noise parameters is helpful. The “Deterministic Meta-Dropout” performs better than baseline MAML, and arguably, meaning that some part of the improvement upon MAML can be due to the architectural differences and not due to noise. “Independent Gaussian” and “Weight Gaussian” baselines perform worse than non-trainable noise (“Fixed Gaussian”). Learning the variance for the noise is shown to be detrimental. There is just too much confusion in the results, the improvements are not very robust. \n\nThe paper writing is okay, but there are serious issues. I am not sure I understand the argument in Section 3.2 that meta-dropout performs variational inference. It seems like Equation 7 is wrong because  y_i is missing from the second argument of the KL divergence term. The transition to Equation 8 is therefore also wrong, and as far as I can understand, the whole argument breaks down. Line 7 in Algorithm 1 in Appendix A (which by the way should really be in the main text) does not make sense.\n\nOther issues: \n- the second sentence of the abstract is not implied by the first, the usage of “thus” does not seem appropriate\n- the intro should probably mention L1 and L2 regularization as well\n- in Section 3.1 there is a forward reference to Equation 5, makes understanding the text quite hard\n- “meta-droput”, “robustenss”: typos in many places\n- Figure 4 visualization is not clear. \n- the architectural change required to add noise is not explained in the paper (i.e. what is \\phi and how it’s used) \n- no comparison to meta-learned L1 regularization \n- a baseline is missing in which \\phi is treated as a part of \\theta and trained with vanilla MAML\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a type of adaptive dropout to regularize gradient based meta-learning models. The reviewers found the idea interesting and it is supported by improvements on standard benchmarks. The authors addressed several concerns of the reviewers during the rebutal phase. In particular, revisions added results against other regularization mthods. We recommend that further attention is given to ablations, in particular the baseline proposed by Reviewer 1.
Decision: Accept (Poster)