# Amortized Bayesian Meta-Learning with Accelerated Gradient Descent Steps
## 2 Related Work  
The meta-learning literature usually interprets the meta-learning problem as Bayesian inference in a hierarchical graphical model, where ideas of HB or EB models for meta-learning have been widely used in the past few years. For example, [PERSON]. [2] considered an  
Figure 1: The computation graph of the ELBO computation for meta-training with the proposed synthetic inference network (SIN), where SIN is regarded as an online trainable student network and the conventional SGD network is treated as the teacher network.  
HB model that improved MAML [5] to compute posteriors via amortized variational inference. [PERSON]. [10] proposed a gradient-based algorithm, which minimized the objective function of meta-learning based on the HB model using the PAC-Bayesian Analysis. [PERSON]. [11] introduced a new and rigorously formulated PAC-Bayes few-shot meta-learning algorithm, which implicitly learns a prior distribution of the model of interest. [PERSON]. [12] used the EB model to construct a variational posterior by introducing a transduction network with a synthetic gradient. Other similar Bayesian VI-based HB methods are referred to [7; 13; 14]. Moreover, the empirical risk minimization (ERM) perspective has been successfully applied to a variety of few-shot learning models [15; 16; 17]. However, the HB model lacks the ability to handle large complex models or high-dimensional datasets due to its high computational costs for inference. Recent research developments have been used to scale Bayesian meta-learning into large datasets. For instance, [PERSON]. [4] inferred episode-specific latent variables, and [PERSON]. [10] learned the separated posterior weights for each task.  
Other works focused on few-shot or one-shot learning-based Bayes methods proposed different perspectives. For instance, [PERSON]. [18] proposed a Gaussian process, like the uncertainty neural process, which can be trainable by gradient descent. [PERSON]. [2] proposed a method that efficiently amortizes hierarchical variational inference across tasks, learning a prior distribution over neural network weights so that a few steps of Bayes by Backprop will produce a good task-specific approximate posterior. [PERSON]. [19] proposed a novel method that can adapt a pre-trained neural network to novel categories by directly predicting the parameters from the activations, which is regarded as amortized mean average precision (MAP) inference derived from the hyper-network [20]. [PERSON]. [21] adapted the Bayesian decision theory to formulate the use of an amortization network for meta-learning so that it improves the state-of-the-art results.  
Recent literature studies on PAC-Bayes meta-learning methods have focused on optimizing the generalization bounds for predictors with prior distributions. In [11], the authors extended previous bounds by optimizing the shared prior through the expectation-maximization (EM) algorithm. [PERSON] et al. [22] proposed a class of PAC optimal meta-learning algorithms with performance guarantees and a principled meta-level regularization by introducing Gaussian process priors with the SVGD [23] approximate optimization method. Other works [24; 25] focused on improving generalization performance by introducing gradient augmentation or the regulation method at the inner level. Although these efforts strive to improve generalization performance, there are significant challenges in generalization on unseen tasks for meta-learning. Moreover, optimization of the meta-learning training procedure is also important and challenging as a means of improving training efficiency.