Meta Dropout: Learning to Perturb Latent Features for Generalization

1 INTRODUCTION . Obtaining a model that generalizes well is a fundamental problem in machine learning , and is becoming even more important in the deep learning era where the models may have tens of thousands of parameters . Basically , a model that generalizes well should obtain low error on unseen test examples , but this is difficult since the distribution of test data is unknown during training . Thus , many approaches resort to variance reduction methods , that reduce the model variance with respect to the change in the input . These approaches include controlling the model complexity ( Neyshabur et al. , 2017 ) , reducing information from inputs ( Tishby et al. , 1999 ) , obtaining smoother loss surface ( Shirish Keskar et al. , 2017 ; Neyshabur et al. , 2017 ; Chaudhari et al. , 2017 ; Santurkar et al. , 2018 ) , smoothing softmax probabilities ( Pereyra et al. , 2017 ) or training for multiple tasks with multi-task ( Caruana , 1997 ) and meta-learning ( Thrun & Pratt , 1998 ) . A more straightforward and direct way to achieve generalization is to simulate the test examples by perturbing the training examples during training . Some regularization methods such as mixup ( Zhang et al. , 2017 ) follow this approach , where the training examples are perturbed to the direction of the other training examples to mimic test examples . The same method could be also applied to the latent feature space , to achieve even larger performance gain ( Verma et al. , 2019 ) . However , these approaches are all limited in that they do not explicitly aim to lower the generalization error on the test examples . How can we then perturb the training instances such that the perturbed instances will be actually helpful in lowering the test loss ? Enforcing this generalization objective is not straightforward in standard learning framework since the test data is unobservable . To solve this seemingly impossible problem , we resort to meta-learning ( Thrun & Pratt , 1998 ) which aims to learn a model that generalize over a distribution of tasks , rather than a distribution of data instances from a single task . Generally , a meta-learner is trained on a series of tasks with random training and test splits . While learning to solve diverse tasks , it accumulates the meta-knowledge that is not specific to a single task , but is generic across all tasks , which is later leveraged when learning for a novel task . During this meta-training step , we observe both the training and test data . That is , we can explicitly learn to perturb the training instances to obtain low test loss in this meta-learning framework . The learned noise generator then can be used to perturb instances for generalization at meta-test time . Yet , learning how much and which features to perturb is difficult for two reasons . First of all , meaningful directions of perturbation may differ from one instance to another , and one task to another . Secondly , a single training instance may need to cover largely different test instances with its perturbation , since we do not know which test instances will be given at test time . To handle this problem , we propose to learn an inputdependent stochastic noise ; that is , we want to learn distribution of noise , or perturbation , that is meaningful for a given training instance . Specifically , we learn a noise generator for each layer features of the main network , given lower layer features as input . We refer to this meta-noise generator as metadropout that learns to regularize . Also the learned noise distribution is transferable , which is especially useful for few-shot learning setting where only a few examples are given to solve a novel task . Figure 1 depicts such a scenario where the noise generator perturbs each input instance to help the model predict better decision boundaries . In the remaining sections , we will explain our model in the context of existing work and propose the learning framework for meta-dropout . We compare our method to existing regularizers such as manifold mixup ( Verma et al. , 2019 ) , information bottleneck , and information dropout ( Achille & Soatto , 2018 ) , which our method significantly outperforms . We further show that meta-dropout can be understood as meta-learning the variational inference framework for the graphical model in Figure 3 . Finally , we validate our work on multiple benchmark datasets for few-shot classification . Our contribution is threefold . • We propose a novel regularization method called meta-dropout that generates stochastic input-dependent perturbations to regularize few-shot learning models , and propose a metalearning framework to train it . • We compare with the existing regularizers such as information bottleneck ( Achille & Soatto , 2018 ; Alemi et al. , 2017 ) and manifold mixup ( Verma et al. , 2019 ) . We also provide the probabilistic interpretation of our approach as learning to regularize the variational inference framework for the graphical model in Figure 3 . • We validate our method on multiple benchmark datasets for few-shot classification , on which our model significantly improves the performance of the base models . 2 RELATED WORK . Meta learning While the literature on meta-learning ( Thrun & Pratt , 1998 ) is vast , here we discuss a few relevant existing works for few-shot classification . One of the most popular approaches is metric-based meta-learning that learns a shared metric space ( Koch et al. , 2015 ; Vinyals et al. , 2016 ; Snell et al. , 2017 ; Oreshkin et al. , 2018 ; Mishra et al. , 2018 ) over randomly sampled few-shot classification problems , to embed the instances to be closer to their correct embeddings by some distance measure regardless of their classes . The most popular models among them are Matching networks ( Vinyals et al. , 2016 ) which leverages cosine distance measure , and Prototypical networks ( Snell et al. , 2017 ) that make use of Euclidean distance . On the other hand , gradient-based approaches ( Finn et al. , 2017 ; 2018 ; Li et al. , 2017 ; Lee & Choi , 2018 ; Ravi & Beatson , 2019 ; Zintgraf et al. , 2019 ) learns a shared initialization parameter , which can rapidly adapt to new tasks with only a few gradient steps . Recent literatures on few-shot classification show that the performance can significantly improve with larger networks , meta-level regularizers or fine-tuning ( Lee et al. , 2019 ; Rusu et al. , 2019 ) . Lastly , meta-learning of the regularizers has been also addressed in Balaji et al . ( 2018 ) , which proposed to meta-learn ` 1 regularizer for domain adaptation . However , while this work focuses on the meta-learning of the hyperparameter of generic regularizers , our model is more explicitly targeting generalization via input perturbation . Dropout Dropout ( Srivastava et al. , 2014 ) is a regularization technique to randomly drop out neurons during training . In addition to feature decorrelation and ensemble effect , we could also interpret dropout regularization as a variational approximation for posterior inference of the network weights ( Gal & Ghahramani , 2016 ) , in which case we can even learn the dropout rates with stochastic gradient variational Bayes ( Kingma et al. , 2015 ; Gal et al. , 2017 ) . The dropout regularization could be viewed as a noise injection process . In case of standard dropout , the noise follows the Bernoulli distribution , but we could also use Gaussian multiplicative noise to the parameters instead ( Wang & Manning , 2013 ) . It is also possible to learn the dropout probability in an inputdependent manner as done with Adaptive Dropout ( Standout ) ( Ba & Frey , 2013 ) , which could be interpreted as variational inference on input-dependent latent variables ( Sohn et al. , 2015 ; Xu et al. , 2015 ; Heo et al. , 2018 ; Lee et al. , 2018 ) . However , meta-dropout is fundamentally different from adaptive dropout , as it makes use of previously obtained meta-knowledge in posterior variance inference , while adaptive dropout resorts only to training data and prior distribution . Regularization methods There exist large number of regularization techniques for improving the generalization performance of deep neural networks ( Srivastava et al. , 2014 ; Ioffe & Szegedy , 2015 ; Ghiasi et al. , 2018 ) , but we only discuss approaches based on input-dependent perturbations that are closely related to meta-dropout . Mixup ( Zhang et al. , 2017 ) randomly pairs training instances and interpolates between them to generate additional training examples . Verma et al . ( 2019 ) further extends the technique to perform the same procedure in the latent feature spaces at each layer of deep neural networks . While mixup variants are related to meta-dropout as they generate additional training instances via input perturbations , these heuristics may or may not improve generalization , while meta dropout perturbs the input while explicitly aiming to minimize the test loss in a metalearning framework . Information-theoretic regularizers ( Alemi et al. , 2017 ; Achille & Soatto , 2018 ) are also relevant to our work , where they inject input-dependent noise to latent features to forget some of the information from the inputs , resulting in learning high-level representations that are invariant to less meaningful variations . Yet , meta-dropout has a more clear and direct objective to improve on the generalization . Adversarial learning ( Goodfellow et al. , 2015 ) is also somewhat related to our work , which perturbs the examples towards the direction that maximizes the training loss , as meta dropout also tend to pertub inputs to the direction of decision boundaries . In the experiments section , we show that meta dropout also improves adversarial robustness , which implies the connection between adversarial robustness and generalization ( Stutz et al. , 2019 ) . 3 LEARNING TO PERTURB LATENT FEATURES . We now describe our problem setting and the meta-learning framework for learning to perturb training instances in the latent feature space , for improved generalization . The goal of meta-learning is to learn a model that generalizes over a task distribution p ( T ) . This is usually done by training the model over large number of tasks ( or episodes ) sampled from p ( T ) , each of which consists of a training set Dtr = { ( xtri , ytri ) } Ni=1 and a test set Dte = { ( xtej , ytej ) } Mj=1 . Suppose that we are given such a split of Dtr and Dte . Denoting the initial model parameter of an arbitrary neural network as θ , Model Agnostic Meta Learning ( MAML ) ( Finn et al. , 2017 ) aims to infer task-specific model parameter θ∗ with one or a few gradient steps with the training set Dtr , such that θ∗ can quickly generalize toDte with a few gradient steps . Let α denote the inner-gradient step size , X and Y denote the concatenation of input data instances and their associated labels respectively for both training and test set . Then , we have min θ Ep ( T ) [ − 1 M log p ( Yte|Xte ; θ∗ ) ] , where θ∗ = θ − α∇θ ( − 1 N log p ( Ytr|Xtr ; θ ) ) . ( 1 ) Optimizing the objective in Eq . 1 is repeated over many random splits of Dtr and Dte , such that the initial model parameter θ captures the most generic information over the task distribution p ( T ) .

Summary: The paper proposes learning to add input-dependent noise to improve the generalization of MAML-style meta-learning algorithm. The proposed method is evaluated on OmniGlot and miniImageNet. The paper reports improvements upon MAML, MAML with meta-learned parameter-wise learning rates, as well as a few regularization methods that are based on input/hidden state perturbations (Mixup, Variational Information Bottleneck). An ablation study also compares the proposed meta-dropout algorithm with a number of modifications, such as a fixed noise, input-independent noise, etc. It is furthermore shown that meta-dropout somewhat improves the model’s robustness against an adversarial attack.