# Amortized Bayesian Meta-Learning with Accelerated Gradient Descent Steps
## 3 Meta-Learning with Amortized Bayesian Inference
### Fast Transductive Inference with the Synthetic Inference Network
Hence,  
\[p(\theta)=\mathrm{Dir}(\mathbf{\pi}|\mathbf{\alpha}_{0})\prod_{t=1}^{T}\mathcal{N}(\mu _{t}|m_{0}^{(t)},(\beta_{0}^{(t)}\tau_{t})^{-1})\mathrm{Gamma}(\tau_{t}|a_{0}, b_{0}), \tag{12}\]  
where \(\beta_{0}^{(t)}\tau_{t}=\frac{1}{\sigma_{t}^{2}}\) and \(a_{0}\) and \(b_{0}\) are the alpha and beta parameters for the gamma distribution. Note that \(a_{0}=\frac{\upsilon}{2}\), \(b_{0}=\frac{\upsilon}{2}\), where \(S=1/N_{t}\sum_{i}r_{it}(x_{i}-\bar{x})^{2}\). In our work, \(\theta^{0}=\phi(d^{S})\in\mathbb{R}^{(T\times N)}\) represent \(T\)-way meta parameters as learnable initializations. The neural network \(\phi(\cdot)\) takes the feature network output \(f(x_{n}\in\mathcal{D}_{S})\) as input, and \(\phi(\cdot)\) is treated as the weight initialization network. Each task-specific parameter \(\theta_{t}^{0}\) is passed through SIN to acquire \(\mathcal{I}\). The proposed network structure is displayed in Figure 4 as a brief overview. It follows a multi-task learning structure, where the common network layers are denoted as the shared layers, while the last two separate layers output the two predicted groups of \(\mathcal{I}\). Observe that SIN follows the online supervised learning scheme, where its training data \(\mathcal{D}_{\mathit{SIN}}=\{\theta_{t}^{0},\mathcal{M}\}\). In Figure 1, one can find that \(\mathcal{M}\) is acquired through the variational Bayes inference with expectation maximization (VBEM). The EM method is introduced to maximize the lower bound of \(\mathrm{KL}(q(\theta^{K})||p(\theta^{K}))\), where \(\theta^{K}\) is obtained from the SGD network with \(K\)-descent steps. As for the loss function, we use the simple MSE loss:\[\mathcal{L}=\gamma\mathcal{L}_{0}+(1-\gamma)\mathcal{L}_{1}, \tag{13}\]  
where \(\mathcal{L}_{0}=\parallel\hat{m}_{t}-m_{t}\parallel_{2}+\parallel\hat{\beta}_{ t}-\beta_{t}\parallel_{2}\) and \(\mathcal{L}_{1}=\parallel\hat{a}_{t}-a_{t}\parallel_{2}+\parallel\hat{b}_{ t}-b_{t}\parallel_{2}\). \(\gamma\) is the weight factor whose value is set to 0.5 as default. A detailed description of the proposed amortized Bayesian meta-training is depicted in Algorithm 1. The algorithm starts with the initializations of some meta-models, such as \(\phi(d^{5})\), \(\phi(d^{5})\), and \(f\). For each task \(t\), it computes \(\theta_{t}^{k}\) via fast gradient descent steps, mixing SIN with the SGD network. It first uses the standard SGD network to obtain \(\theta_{t}^{k}\). Meanwhile, the SGD network is used as a teacher network for training SIN. When the training process is valid, the output variable \(\theta_{t}^{\prime}\) sampled from SIN can reach \(\theta_{t}^{*}\) through fewer descent steps. After \(T\) training tasks are processed as the epoch, the transductive amortization network \(\phi\) is updated with the feature network \(f\) that has been fine-tuned. Notice that \(\xi\) denotes an online training loss threshold, which indicates that the model can use the learned amortized network output \(\theta_{t}^{\prime}\) for fast training. When the training process begins, SIN introduces a teacher-student network architecture for the prediction of \(\theta_{k}\) (after \(k\) steps, the gradient descent of SGD). It uses a linear artificial neural network (ANN) to learn the specific latent parameters \(\mathcal{I}\) related to \(\theta_{k}\). The parameter set \(\mathcal{I}=\{\mathbf{m},\beta,\mathbf{L},v\}\) represents parameters for normal and Wishart distributions in Equation (8). Note that we use these latent parameters to represent the posterior distribution of \(q(\theta)\), which maximizes the following lower bound given in Equation (11). We first train both teacher and student networks in \(K^{\prime}\) steps (\(K^{\prime}\ll K\)), after the student network learns the representation of the teacher network, then we use the inference of the student network directly to output \(\mathcal{I}\). Note that \(\mathcal{I}\) is obtained by the VBEM method of \(\theta_{t}^{k}\).