## 3 Methods: Bayesian Modelling
### Bayesian Model: Black Box  
Neural networks can be considered as BB models since they provide little explanatory insight into the relative influence of the independent variables in the prediction process [56]. From a statistical perspective, neural networks are a robust approximator to estimate real-valued (prediction) and discrete-valued (classification) target functions because they can mimic the non-linearity of the functions and their learning methods are well-developed [57]. Neural networks and their family have been used in a large number of applications [58], including air pollution research [24]. In order to enable for a fair comparison with the Bayesian WB approach, here a Bayesian method is implemented into neural networks, known as a Bayesian neural network (BNN) [45].  
A neural network, \(f(X,\mathbf{\beta})\), can be viewed as a probabilistic model, that follows a Gaussian distribution, given by:  
\[p(\mathbf{y}|X,\mathbf{\beta},\gamma)\sim\mathcal{N}(\mathbf{y}|f(X,\mathbf{\beta}), \gamma^{-1}) \tag{6}\]  
where the notations of \(X,\mathbf{\beta}\), and \(\gamma\) are the inputs, the neural network parameters (i.e., weights), and the precision of the Gaussian distribution, respectively. Equation (6) is also known as a likelihood function.  
In a Bayesian framework, a prior distribution needs to be assigned, where in this case, the prior follows a Gaussian distribution with mean zero and the precision of \(\alpha\), given by:  
\[p(\mathbf{\beta}|\alpha)\sim\mathcal{N}(\mathbf{\beta}|\mathbf{0},\alpha^{-1}\mathbf{I}) \tag{7}\]  
Using the prior distribution and likelihood function, the posterior distribution for the BNN can be computed based on Bayes theorem, shown in Equation (3), to give:  
\[\underbrace{p(\mathbf{\beta}|\mathbf{y},X,\alpha,\gamma)}_{\text{posterior dist}}\propto\underbrace{p(\mathbf{y}|X,\mathbf{\beta},\gamma)}_{\text{ likelihood func. prior dist}}\underbrace{p(\mathbf{\beta}|\alpha)}_{\text{prior dist}} \tag{8}\]  
The inclusion of the prior distribution leads to a regularization, which then counters over-fitting [59; 60]. Furthermore, BNN provides a degree of belief on the estimated output, which can be used to assess the quality of the predictions. However, due to the non-linear dependence of \(f(X,\mathbf{\beta})\) on \(\mathbf{\beta}\), the posterior distribution calculation is intractable.  
The first solution to estimate the BNN posteriors was proposed using a Laplace approximation [61; 62]. Solutions to compute more accurate posterior distributions have been developed, including variational inference [63], sampling-based variational inference [64], and expectation propagation [65]. In this case, we adopt the recent solution based on automatic differentiation to variational inference (ADVI) proposed by [PERSON] et al. [52], [PERSON] et al. [66]. This approach optimises the weights by minimising a compression cost, known as the variational free energy or the expected lower bound on the marginal likelihood.  
The structural details of the BNN can be found in [PERSON] et al. [59], whereas the posterior based Bayesian optimisation can be found in [PERSON] et al. [66]. As with the Bayesian implementation of WB modelling, the BNN implementation also uses pyMC3 [55].