# Amortized Bayesian Meta-Learning with Accelerated Gradient Descent Steps
## 3 Meta-Learning with Amortized Bayesian Inference
### Fast Transductive Inference with the Synthetic Inference Network  
We now introduce the proposed synthetic neural network for fast transductive inference. A side-by-side comparison between MAML and our method is displayed in Figure 3. In MAML, the global parameter \(\theta\) is given an initialization according to its previous update, and it optimizes for a representation of \(\theta\) that can quickly adapt to new tasks. In our method, we first introduce a transductive neural network \(\phi\) that outputs an initial distribution \(\theta_{t}^{0}\) for different tasks \(t\). Next, we propose a trainable synthetic inference network (SIN), which takes features of \(\theta_{t}^{0}\) as the input and outputs its inferred parameters \(\mathcal{I}\) for the refined variational distribution \(q_{\phi(d_{t}^{S})}^{\prime}\). Then, the refined latent variable \(\theta_{t}^{\prime}\) is formed by sampling from \(\mathcal{I}\). With a few iteration steps, \(\theta_{t}^{\prime}\) can reach \(\theta_{t}^{K}\), converging to \(\theta_{t}^{*}\). Figure 3: A flow comparison between MAML and our method (SIN). (**a**) MAML meta-learning diagram; (**b**) our method, which introduces the synthetic inference network (SIN). The details of SIN are given below. We let meta-parameter \(\theta=\{\mathbf{\mu}_{\theta},\mathbf{\sigma}_{\theta}^{2}\}\), where \(\mathbf{\mu}_{\theta}\in\mathbb{R}^{D}\), \(\mathbf{\sigma}_{\theta}^{2}\in\mathbb{R}^{D}\) represent the mean and variance, respectively. For _task-specific_ parameter \(\psi_{t}\), its weight follows the normal distribution in Equation (7):  
\[p(\psi_{t}|\theta)=\mathcal{N}\big{(}\psi_{t}|\mathbf{\mu}_{\theta},\mathbf{\sigma}_{ \theta}^{2}\mathbf{\mathrm{I}}\big{)}, \tag{7}\]  
and the prior \(p(\theta)\) is written in Equation (8):  
\[p(\theta)=\mathrm{Dir}(\mathbf{\pi}|\mathbf{\alpha}_{0})\prod_{t=1}^{T}\mathcal{N}(\bm {\mu}_{t}|\mathbf{\mathrm{m}}_{0}^{(t)},(\beta_{0}^{(t)}\mathbf{\Lambda}_{t})^{-1}) \mathrm{Wi}(\mathbf{\Lambda}_{t}|\mathbf{\mathrm{I}}_{0}^{(t)},\upsilon_{0}^{(t)}), \tag{8}\]  
where \(\mathbf{\pi}=[\pi_{1},\cdots,\pi_{t}]\) follows the Dirichlet distribution with a symmetric prior \(\mathbf{\alpha}_{0}=\alpha_{0}\mathbf{\mathrm{I}}\). \(\mathbf{\mathrm{m}}_{0}\), \(\beta_{0}\mathbf{\Lambda}_{t}\) denote the mean and precision matrix for the normal distribution, and \(\mathbf{\mathrm{L}}_{0}\) and \(\upsilon_{0}\) represent the scale matrix and its freedom degree for the Wishart distribution. Note that when \(dim(\theta_{t})=1\times N\), Wishart reduces to the Gamma distribution in Equation (9):  
\[\mathrm{Wi}(\lambda|s^{-1},\upsilon)=\mathrm{Ga}(\lambda|\frac{\upsilon}{2}, \frac{s}{2}). \tag{9}\]  
Note that the exact posterior \(p(\psi,\theta|\mathcal{D})\) is a mixture of \(T\) distributions. We use the standard variational Bayes (VB) approximation to the posterior:  
\[p(\theta,\mathbf{\psi}_{1:T}|\mathcal{D})\simeq q(\theta)\prod_{t=1}^{T}q(\psi_{t }). \tag{10}\]  
Meanwhile, it attempts to maximize the following lower bound:  
\[\mathcal{L}=\sum_{\psi}\int q(\psi,\theta)\log\frac{p(f(x),\psi,\theta)}{q( \psi,\theta)}d\theta\leq\log p(\mathcal{D}) \tag{11}\]  
For a detailed overview of the proposed meta-training with the SIN framework, we show a computational graph in Figure 1. A teacher-student network architecture is proposed for the fast convergence training of \(\theta_{t}\). SIN is treated as an online trainable neural network with the stochastic gradient descent (SGD) network as its teacher network. Note that the learnable parameter of SIN is the inference parameter \(\mathcal{I}\) for distribution \(q^{\prime}_{\phi(\delta^{\prime}_{t})^{\prime}}\), where \(\mathcal{I}=\{\mathbf{m},\beta,\mathbf{\mathrm{L}},\upsilon\}\) denote parameters for the normal and Wishart distributions in (8). In fact, when \(\theta_{t}^{0}\in\mathbb{R}^{1\times N}\), the Wishart distribution reduces to the Gamma distribution.