# Locally Scaled and Stochastic Volatility Metropolis-Hastings Algorithms
## 2 Methods
### Hamiltonian Monte Carlo and the No-U-Turn Sampler  
The Hamiltonian Monte Carlo (HMC) algorithm employs first-order gradient data of the posterior distribution, similarly to MALA, to navigate the parameter space [2; 14]. However, unlike MALA, the HMC adds an extra momentum variable \(\mathbf{p}\) to the parameter space \(\mathbf{w}\) and uses Hamiltonian dynamics to evolve the system over time. Note that \(\mathbf{p}\) and \(\mathbf{w}\) have the same number of dimensions. This dynamic system results in a Hamiltonian \(\mathrm{H}(\mathbf{w},\mathbf{p})\) which is given as [1]:  
\[\mathrm{H}(\mathbf{w},\mathbf{p})=\mathrm{U}(\mathbf{w})+\mathrm{K}(\mathbf{p}) \tag{7}\]  
where \(\mathrm{U}(\mathbf{w})\) is the potential energy (which is given by the negative log-likelihood of the target) and \(\mathrm{K}(\mathbf{p})\) is the kinetic energy which is a Gaussian kernel with covariance matrix \(\mathbf{M}\)[3]:  
\[\mathrm{K}(\mathbf{p})=\frac{1}{2}\text{log}\Big{(}(2\pi)^{D}|\mathbf{M}| \Big{)}+\frac{\mathbf{p}^{\text{T}}\mathbf{M}^{-1}\mathbf{p}}{2}. \tag{8}\][PERSON]'s equations are used to generate the path of the Markov chain at time \(t\) as follows [1]:  
\[\frac{\mathrm{d}\mathbf{w}}{\partial t}=\frac{\partial\mathrm{H}(\mathbf{w}, \mathbf{p})}{\partial\mathbf{p}};\quad\frac{\mathrm{d}\mathbf{p}}{\partial t}=- \frac{\partial\mathrm{H}(\mathbf{w},\mathbf{p})}{\partial\mathbf{w}}. \tag{9}\]  
These Hamiltonian dynamics ensure that the Hamiltonian system conserves energy, is reversible, and preserves phase space volume. These dynamics typically have to be solved numerically, with the leapfrog integrator being the most common numerical scheme. The leapfrog integration scheme has the following update equations [3; 14]:  
\[ \begin{split}\mathbf{p}_{t+\frac{\xi}{2}}&=\mathbf{p }_{t}+\frac{\epsilon}{2}\frac{\partial H(\mathbf{w}_{t},\mathbf{p}_{t})}{ \partial\mathbf{w}}\\ \mathbf{w}_{t+\epsilon}&=\mathbf{w}_{t}+\epsilon \mathbf{M}^{-1}\mathbf{p}_{t+\frac{\xi}{2}}\\ \mathbf{p}_{t+\epsilon}&=\mathbf{p}_{t+\frac{\xi}{2 }}+\frac{e}{2}\frac{\partial H\Big{(}\mathbf{w}_{t+\epsilon},\mathbf{p}_{t+ \frac{\xi}{2}}\Big{)}}{\partial\mathbf{w}}.\end{split}  \tag{10}\]  
The leapfrog integration scheme does not give an exact solution to Hamiltonian dynamics and produces numerical integration errors. This results in the total energy not being conserved, resulting in the detailed balance condition no longer holding. To ensure that detailed balance hods, a Metropolis-Hastings acceptance step is utilised. The overall process for generating a single sample from HMC is a Gibbs sampling scheme. In this scheme, we first generate the momentum variable from the Gaussian distribution, after which we sample new parameters based on the newly drawn momentum using the leapfrog integration scheme. This procedure is repeated until the desired number of samples has been generated. We have yet to address how one selects the step size \(\epsilon\) and trajectory length \(L\) parameters of HMC. These parameters significantly influence the sampling performance of the algorithms [32]. The NUTS algorithm of [PERSON] [32] automates the tuning of the HMC step size and trajectory length parameters. A large step size typically results in most of the generated samples being rejected, while a small step size results in slow mixing [32]. When \(L\) is too tiny, then HMC depicts random walk behaviour, and when the \(L\) is large, the method consumes computational resources [32]. In the NUTS methodology, the step size parameter is tuned through primal-dual averaging during an initial burn-in phase. On the other hand, the trajectory length parameter is automatically tuned by iteratively doubling the trajectory length until the Hamiltonian becomes infinite or the chain starts to trace back [8; 33]. That is when the last proposed position state \(\mathbf{w}^{*}\) starts becoming closer to the initial position \(\mathbf{w}\). More information about the NUTS algorithm can be found in [32].