# Amortized Bayesian Meta-Learning with Accelerated Gradient Descent Steps
###### Abstract  
Recent meta-learning models often learn priors from observed tasks using a network optimized via stochastic gradient descent (SGD), which usually takes more training steps to convergence. In this paper, we propose an accelerated Bayesian meta-learning structure with a stochastic inference network (ABML-SIN). The proposed model aims to solve the training procedure of Bayesian meta-learning to improve the training speed and efficiency. Current approaches of meta-learning hardly converge within a few descent steps, owing to the small number of training samples. Therefore, we introduce an accelerated gradient descent learning network based on teacher-student architecture to learn the meta-latent variable \(\theta_{t}\) for task \(t\). With this amortized fast inference network, the meta-learner is able to learn the task-specific latent \(\theta_{t}\) within a few training steps; thus, it improves the learning speed of the meta-learner. To refine the latent variables generated from the transductive amortization network of the meta-learner, SIN--followed by a conventional SGD-optimized network--is introduced as the student-teacher network to online-update the parameters. SIN extracts the local latent variables and accelerates the convergence of the meta-learning network. Our experiments on simulation data demonstrate that the proposed method provides generalization and scalability on unseen samples, and produces competitive/superior uncertainty estimations on few-shot learning tasks on two widely adopted 2D datasets with fewer training epochs compared to the state-of-the-art meta-learning approaches. Furthermore, the parameters generated by SIN act as perturbations on latent weights, enhancing the probability of accelerating the training efficiency of the meta-learner. Extensive qualitative experiments show that our method performs well across different meta-learning tasks in both simulated and real-world circumstances. Bayesian meta-learning amortized Bayesian network neural network structure optimization; knowledge distillation +
Footnote †: journal:  
there is a lack of general-purpose tools for flexible and data-efficient learning [7], and there is still room for optimization in the meta-training process. In addition to improving predictive accuracy for general tasks, it is also important for meta-models to provide good uncertainty predictions with a fast training process. Thus, we adopt the Bayesian method as a principled framework to provide uncertainty prediction. Approximate Bayesian methods are widely used to provide uncertainty prediction for deep learning models [8; 9]. Bayesian meta-learning methods output a posterior distribution that correctly indicates the confidence level for unseen samples [2]. Researchers formulate the notion of the _hierarchical Bayes_ (HB) model or _empirical Bayes_ (EB) model, with the EB model restricting the learning of meta-parameters to point estimates. In our work, we consider EB model variational inference for meta-learning to simplify the training process without the HB formulation strength. This work focuses on the efficient usage of amortized variational inference for the Bayesian model with fast training steps. As such, we propose a synthetic inference network (SIN) combined with transduction for meta-learning, where the synthetic network produces the learned latent features over the neural network. During the training process, SIN refines the output of the transductive network that accelerates network convergence. We also propose a teacher-student network structure for the online training of SIN, where the conventional SGD-optimized network is treated as the teacher network. If not specified, we will denote the regular network optimized by SGD as the SGD network in the contents hereafter. The proposed method can converge a few steps after knowing the overall direction of the gradient descent. As illustrated in Figure 1, with collaborative training through the teacher-student network, the meta-training process is accelerated and the latent weights are optimized.