# Amortized Bayesian Meta-Learning with Accelerated Gradient Descent Steps
## 4 Experiments
### Data Posterior Inference Examples  
We begin the posterior inference example with emulated data generated from a Gaussian distribution whose mean value is different from the task in Equation (14):  
\[ \begin{cases}&p(\psi^{(t)}|\theta)=\mathcal{N}(\psi^{(t)};\mu,\sigma^{2})\\ &p(x_{n}^{(t)}|\psi^{(t)})=\mathcal{N}(x_{n}^{(t)};\psi^{(t)},\sigma_{x}^{2}),\end{cases}  \tag{14}\]  
where \(p(\theta)\) follows the delta distribution \(\delta(\theta)\). According to [PERSON] et al. [7], we set the sample number \(N\in\{5,10\}\) for the support set and \(N^{\prime}=15\) for the query set. The transductive amortized neural network \(\phi\) is defined as follows:  
\[ \begin{cases}&\mu^{(t)}=\sum_{n=1}^{N}x_{n}^{(t)}\omega_{\mu}^{n}+b_{\mu}\\ &\log\sigma^{(t)}=\sum_{n=1}^{N}x_{n}^{(t)}\omega_{\sigma}^{n}+b_{\sigma}, \end{cases}  \tag{15}\]  
where parameter sets \(\theta^{0}=\{\mathbf{\omega}_{\mu},\mathbf{\omega}_{\sigma},b_{\mu},b_{\sigma}\}\) are trained parameters for \(\phi\), and \(q_{\phi}(\psi|D^{(t)})=\mathcal{N}(\psi;\mu^{(t)},\sigma^{(t)2})\). In this case, since the dimensions of these trainable parameters are low, it is unnecessary to use VI to inference the refined parameters \(\mathcal{I}\) in SIN. Instead, we use a simple encoder-decoder network that takes \(\theta^{0}\) as the input and outputs \(\theta_{t}^{\prime}\) directly since few parameters with low dimensions are easy to converge. The objective log-likelihood function for optimization is given in Equation (16):  
\[\mathcal{L}(\theta,\phi)=\frac{1}{T}\sum_{T}\log\frac{1}{N}\sum_{n=1}^{N}p(x_{ n}^{(t)}|\psi^{(t)},\theta),\;\psi^{(t)}\sim q_{\phi}(\psi|\mathcal{D}^{(t)},\theta) \tag{16}\]  
For implementation details, we generate \(T=250\) tasks with 5 shots and 10 shots for training observations. The teacher network (SGD) and the student network (SIN) are both trained to convergence by using the Adam [28] optimizer, and the generated data samples are rearranged as mini-batches of tasks for training input. To generate this dataset, we use N random variables for \(\mu\) and \(\sigma\) to generate the support set (small training samples) and query set (unseen samples of \(\mu\) and \(\sigma\)). After the loss function is converged, \(q_{\phi}(\psi|\mathcal{D})\) is inferred by SIN with the learned amortization parameters. Figure 6 shows the approximate posterior distributions inferred from unseen test samples and the true posteriors for two-shot cases. We compare the output of SIN with Versa [7], where Versa uses an end-to-end stochastic training that optimizes the likelihood function corresponding with \(\phi\) and \(\theta\). We can see that SIN can output the approximate posterior that closely resembles its true posterior, and its inference results are close to the standard amortization network. Another simulation example is made for spinning-line regression of zero-shot learning. Denote \(\mathcal{D}=\{(x_{n},y_{n})\}_{n=1}^{N}\), the output for \(x_{n}\) is \(y_{n}=w\cdot x_{n}\). Similar to the previous example, we set \(\theta^{0}=\{\mathbf{\omega},b\}\) and \(q_{\phi}(\psi|D^{(t)})=\mathcal{N}(\psi;\mu^{(t)}_{\psi},\sigma^{(t)2}_{\psi})\). Specifically, we have \(\mu^{(t)}_{\psi}=\sum_{n}x_{n}^{(t)}\omega_{n}+b\). To generate the dataset, we first sample i.i.d. Gaussian random variables \(x_{n}\sim\mathcal{N}(\mu,\sigma)\). Next, we sample slope \(w=\frac{1}{N}\sum_{n}x_{n}+\mathcal{N}(\mu_{w},\sigma_{w}^{2})\) from \(p(\psi|\mathcal{D})\). The posterior distribution \(p(\psi|\mathcal{D})\sim\mathcal{N}(x_{n}+\mu_{\psi},\sigma_{\psi}^{2})\), and the log-likelihood function is defined in Equation (17):  
\[\mathcal{L}(\theta,\phi)=\frac{1}{T}\sum_{T}\log\frac{1}{N}\sum_{n=1}^{N}p(y_{ n}^{(t)}|x_{n}^{(t)},\psi^{(t)},\theta), \tag{17}\]where \(\psi^{(t)}\sim q_{\phi}(\psi|\mathcal{D}^{(t)},\theta)\). In this example, we initialize \(n=32,\mu=0,\sigma=1,\mu_{w}~{}=~{}1\), \(\sigma_{w}~{}=~{}0.1\). The number of tasks \(T\) is set to 240 for both the train and test sets, and the learning rate is given as 0.001 with the mini-batch size of 8.