# Locally Scaled and Stochastic Volatility Metropolis-Hastings Algorithms
## 2 Methods
### Random Walk Metropolis-Hastings Algorithm  
The Metropolis-Hastings (MH) algorithm is an example of a basic and simple MCMC method and forms the essence of more complex MCMC algorithms, which use more advanced proposal distributions compared to MH. Suppose were are interested in determining the posterior distribution of parameters \(\mathbf{w}\) from some model \(M\). The MH method produces proposed samples using a proposal distribution \(Q(\mathbf{w}^{*}|\mathbf{w})\). A new parameter state \(\mathbf{w}^{*}\) is accepted or rejected probabilistically given the current state \(\mathbf{w}\) based on the posterior likelihood ratio [11; 29]:  
\[P\big{(}\text{accept}\ \mathbf{w}^{*}\big{)}=\alpha(\mathbf{w}^{*}|\mathbf{w})= \min\bigg{(}1,\frac{\pi(\mathbf{w}^{*})Q(\mathbf{w}|\mathbf{w}^{*})}{\pi( \mathbf{w})Q(\mathbf{w}^{*}|\mathbf{w})}\bigg{)} \tag{1}\]  
where \(\pi(\mathbf{w})\) is the stationary or target distribution evaluated at \(\mathbf{w}\), for some arbitrary \(\mathbf{w}\).  
Random walk Metropolis is a version of MH that utilises a Gaussian distribution centred at the current state as the proposal distribution [11]. The transition density in random walk Metropolis is \(\mathcal{N}(\mathbf{w},e\mathbf{\Sigma})\), where \(e\) is the variance of the proposal distribution [11]. Note that \(\mathbf{\Sigma}\) is typically set to be the identity matrix in practice. In this work, we tune \(e\) to target an acceptance rate of \(25\%\) using the primal-dual averaging methodology outlined in Section 3.2.  
When the transition density is symmetric, it implies that \(Q(\mathbf{w}|\mathbf{w}^{*})=Q(\mathbf{w}^{*}|\mathbf{w})\) reducing Equation (1) to a ratio of posterior likelihoods as follows [11]:  
\[P\big{(}\text{accept}\ \mathbf{w}^{*}\big{)}=\alpha(\mathbf{w}^{*}|\mathbf{w})= \min\bigg{(}1,\frac{\pi(\mathbf{w}^{*})}{\pi(\mathbf{w})}\bigg{)} \tag{2}\]  
This proposal normally results in random walk behaviour, which leads to high autocorrelations between the generated samples as well as slow convergence [11]. Algorithm 1 presents the pseudo-code for the MH algorithm.  
```
Data:\(\mathbf{w}_{\text{init}},e,N\) and unnormalised target \(\pi(\mathbf{w})\) Result:\((\mathbf{w})_{i=0}^{N}\)\(\mathbf{w}_{0}\leftarrow\mathbf{w}_{\text{init}}\)for\(n\gets 1\)to\(N\)do \(\mathbf{\Sigma}=\mathbf{I}\) \(\mathbf{w}^{*}\sim Q\) with \(Q=\mathcal{N}(\mathbf{w},e\mathbf{\Sigma})\) \(\mathbf{w}_{n}\leftarrow\mathbf{w}^{*}\) with probability: \(\alpha(\mathbf{w}^{*}|\mathbf{w})=\min\bigg{(}1,\frac{\pi(\mathbf{w}^{*})Q( \mathbf{w}|\mathbf{w}^{*})}{\pi(\mathbf{w})Q(\mathbf{w}^{*}|\mathbf{w})}\bigg{)}\)  end for
```  
**Algorithm 1**The Metropolis-Hastings Algorithm  
A significant drawback of the MH seminal method is the high autocorrelations between the generated samples. The high autocorrelations lead to slow convergence in the Monte Carlo estimates and consequently the necessity to generate large sample sizes. Approaches that reduce the random walk behavior are those that enhance the classical MH algorithm and utilise more information about the target posterior distributions [4]. The most popular extension is to incorporate first-order gradient information to guide the search of the target, as well as second-order gradient information to consider the local geometry of the posterior [4]. Examples of methods that are able to leverage first-order gradient information are Metropolis Adjusted Langevin Algorithm and Hamiltonian Monte Carlo [1; 4; 30; 14]. We consider these two methods in more detail in the following sections. Algorithm 1 provides the pseudo-code for the MH algorithm.