## 4 Statistical Optimization and Model Validation
### Bayesian Inference  
The Bayesian approach to optimization extensively exploits [PERSON] theorem to fit a statistical model to the problem at hand. This is achieved by updating the prior knowledge on hyper-parameters \(\Theta\), defined by the conditional probability \(\boldsymbol{\theta}\sim P(\boldsymbol{\theta})\), with possibly few observation data points. These hyper-parameters are treated as random variables, whose subjective definition must disclose the available information _before_ any measurement of the quantity of interest, \(\mathbf{Y}\), is performed. Let us assume that a computational model \(\mathcal{M}\) predicts a certain quantity of interest in a vector \(\boldsymbol{y}\in\mathbb{R}^{N_{out}}\) as a function of input parameters \(\boldsymbol{\theta}=\{\theta_{i},i=1,..,M\}\). Then, the Bayesian theorem in the context of the statistical inference can be cast as the following:  
\[P(\boldsymbol{\theta}|\mathcal{Y})=\frac{p(\mathcal{Y}|\boldsymbol{\theta})P( \boldsymbol{\theta})}{P(\mathcal{Y})}, \tag{15}\]  
where \(P(\boldsymbol{\theta}|\mathcal{Y})\) denotes the posterior distribution of the hyper-parameters, \(p(\mathcal{Y}|\boldsymbol{\theta})\) the likelihood function and \(P(\mathcal{Y})\) is the probability of the data. Assuming that a measured data set of \(\mathcal{Y}=\left(\mathcal{Y}_{1},...,\mathcal{Y}_{N}\right)^{T}\) with the independent realization of \(\mathbf{Y}\sim p(\mathcal{Y}|\boldsymbol{\theta})\) is available,the probability of observing the data at hand can be defined by the likelihood function as the following:  
\[ \begin{split} p(\mathcal{Y}|\mathbf{\theta})&:=\prod_{i=1} ^{N}p(\mathcal{Y}_{i}|\mathbf{\theta})\\ &=\frac{1}{\sqrt{(2\pi)^{N}\det\Sigma}}\exp\biggl{(}-\frac{1}{2}( \mathcal{M}(\mathbf{\theta})-\mathcal{Y})^{T}\Sigma^{-1}(\mathcal{M}(\mathbf{\theta} )-\mathcal{Y})\biggr{)}.\end{split}  \tag{16}\]  
where \(\Sigma\) denotes the covariance matrix, which includes the measurement error as well as all other error sources (Section 4.4). \(P(\mathcal{Y})\) is a normalization factor, which ensures that the posterior probability sums up to one. It is also known as _marginal likelihood_ or _evidence_ and can take the following form:  
\[P(\mathcal{Y})=\int_{\mathcal{D}_{\Theta}}p(\mathcal{Y}|\mathbf{\theta})P(\mathbf{ \theta})d\mathbf{\theta}. \tag{17}\]  
The posterior distribution in Equation (15) provides a summary of the inferred information regarding the hyper-parameters after updating the prior knowledge with the observed data. However, the practical computation of the posterior distribution \(P(\mathbf{\theta}|\mathcal{Y})\) is nothing but trivial. There exist analytical expressions only for special choices of the prior distribution \(P(\mathbf{\theta})\). For more involved cases, one approach is to use sampling methods, such as Markov Chain Monte Carlo (MCMC) simulation methods to approximate the posterior distribution. One advantage of using the Bayesian inference in the task of model validation is that the uncertainty on hyper-paramters \(\mathbf{\theta}\) can be incorporated into prior and posterior evaluations of quantities of interest \(\mathbf{Y}\) by so-called _predictive distributions_. The prior predictive distribution can be obtained by the following expression:  
\[P_{pred}^{\prime}(y):=\int_{\mathcal{D}_{\Theta}}p(y|\mathbf{\theta})P(\mathbf{\theta} )d\mathbf{\theta}, \tag{18}\]  
which is average of the conditional distribution of the prior distribution over the prior distribution \(P(\theta)\).