## 2 Bayesian Optimization  
### Gaussian Process (GP) Regression  
In order to properly understand BO, the GP framework needs to be explained in detail. The complete treatment of GP can be found in [PERSON] and [PERSON] (2006) [51]. Most of the material in this section has been inspired by [PERSON] (2018) [40] and [PERSON] (2015) [52] and the articles by [PERSON] [53; 54], which provide visual and interactive explanations of the materials discussed here. Multivariate Gaussian distributions (MVN) [55] act as the building blocks of GP. They express the distribution of multiple random variables that are distributed normally and whose joint distribution is also Gaussian. They are defined by a mean vector \(\boldsymbol{\mu}\), which describes the expected value of the distribution, and a positive semidefinite covariance  
Figure 1: NPV evaluation for oil production with a single well [50]. matrix \(\mathbf{\Sigma}\), which models the variance along each dimension and determines how the different random variables are correlated. The diagonal elements of \(\mathbf{\Sigma}\) express the variance of each random variable. The off-diagonal elements \(\Sigma_{i,j}\) describe the correlation between the \(i\)th and \(j\)th random variables. The probability density of an MVN in \(d\) dimensions is shown in Equation (3). \[N(\mathbf{x}|\mathbf{\mu},\mathbf{\Sigma})=\frac{1}{(2\pi)^{d/2}|\mathbf{\Sigma}|^{1/2}} \exp\left[-\frac{1}{2}(\mathbf{x}-\mathbf{\mu})^{T}\mathbf{\Sigma}^{-1}(\mathbf{x}- \mathbf{\mu})\right] \tag{3}\]  
where \(\mathbf{\mu}\) is the \(d\times 1\) mean vector and \(\mathbf{\Sigma}\) is the \(d\times d\) covariance matrix. A GP is the extension of this MVN to the space of functions, defining a probability measure on a function space. This function space acts as the probabilistic surrogate objective function, which will eventually converge to the actual objective function through sampling and utilization of the GP regression algorithm. At first, the probability measure is interpreted as our prior knowledge about the unknown objective function before data are seen. A GP is defined via its mean and covariance just like the MVN; however, those now represent functions. Given a finite collection of \(q\) points \(\mathbf{x}_{1:q}\in\mathbb{R}^{d}\) that appropriately discretize the \(d\)-dimensional input space, the objective function's values \(\mathbf{f}(\mathbf{x}_{1:q})\) are assumed to be drawn by an MVN. \[\mathbf{f}(\mathbf{x}_{1:q})\sim N(\mathbf{m}(\mathbf{x}_{1:q}),\mathbf{K}( \mathbf{x}_{1:q},\mathbf{x}_{1:q})) \tag{4}\]  
Compared with a multivariate normal, we have the following:  
* A random function vector \(\mathbf{f}(\cdot)\) instead of a random vector \(\mathbf{x}\);
* A mean function vector \(\mathbf{m}(\cdot)\) instead of a mean vector \(\mathbf{\mu}\);
* A covariance (kernel) function matrix \(\mathbf{K}(\cdot,\cdot)\) instead of a covariance matrix \(\mathbf{\Sigma}\). Above, \(\mathbf{K}(\cdot,\cdot)=\{k(\mathbf{x}_{i},\mathbf{x}_{j})\}\) and \(k(\mathbf{x}_{i},\mathbf{x}_{j})\) is the kernel function discussed below. Equation (4) states that the vector of functions conditional on inputs \(\mathbf{x}_{1:q}\) and the mean and covariance functions follows a \(q\)-dimensional MVN distribution. To avoid confusion, it needs to be stated that no assumption has been made on the distribution of the inputs. So far, the prior Gaussian distribution of the objective function has been defined. Function values selected in the prior are commonly referred to in the literature as the test data. In the prior distribution, it is common to set the mean function to a constant value of zero. This is a reasonable assumption since no prior knowledge is assumed. If prior knowledge can be assumed to exist, then the mean function can be set to Equation (5). \[m(\mathbf{x})=\mu_{0}+\sum_{i=1}^{p}\beta_{i}\Psi_{i}(\mathbf{x}) \tag{5}\]  
where \(\Psi_{i}\) are often parametric, low-order polynomials which summarize our prior knowledge on the objective function.