# Graphical User Interface for the Development of Probabilistic Convolutional Neural Networks
## 2 Materials and Methods
### Probabilistic Models  
One of the main advantages of PMs over DMs is the ability to present the epistemic uncertainty of predictions. However, the development of these PMs is typically more complex, requiring additional knowledge on how to build them, frequently using VI approximation [16]. It allows approximating probabilistic inference as an optimization problem, searching for an alternative a posteriori distribution that minimizes the Kullback-Leibler divergence (KLD) [17] with the true a posteriori distribution. In this way, all the weights \(W\) of the network are replaced by a probabilistic distribution, and the distribution is optimized during network training. In this case, the Gaussian distribution [18] is often used to represent these distributions.  
Specifically, VI involves selecting a member \(q\) from a family of distributions \(Q\), which is similar to the target distribution using KLD to measure the fit. In essence, the idea behind variational inference is to reframe our approximate Bayesian inference problem as an optimization problem. This reframing enables the leverage of the full range of mathematical optimization techniques to tackle the problem. By optimizing the parameters, denoted as \(\theta\), for \(q\), the solution that is closest to the conditional interest \(\mathrm{P}(W\,|\,D)\) can be found using KLD, where \(D\) is the training data. The optimal solution can be obtained by solving [16]  
\[q^{*}(W|\theta)=_{q(W|\theta)\sim Q}^{arg\min}\mathrm{KLD}[q(W|\theta)||\mathrm{ P}(W|D)] \tag{1}\]  
Since \(\mathrm{P}(W\,|\,D)\) is unknown, a different objective needs to be optimized in order to minimize the KLD. The standard approach is to use the evidence lower bound (ELBO), which can be calculated by [16]  
\[-\mathrm{ELBO}=\mathrm{KLD}[q(W|\theta)||\mathrm{P}(W)]-E_{q(W|\theta)\sim Q} [\log[\mathrm{P}(D|W)]] \tag{2}\]  
The cost function can be seen as a composition of two parts. The first part of the function is typically viewed as the complexity cost, which leads to the search for solutions whose densities are close to the prior. The second part is dependent on the data; it is usually referred to as the likelihood cost because it describes the probability of the data given to the model, leading to solutions that better explain the observed data. As a result, this cost function interprets a compromise between satisfying the simplicity of the prior probability distribution \(\mathrm{P}(W)\) and the complexity of the data.  
Although this approximation can be achieved through VI, using a prior distribution of the model weights to produce a regularization effect, the exact calculation of the ELBO as presented is computationally prohibitive. Another relevant aspect is the usual approach of using a Gaussian distribution for the model weights, with two parameters, the mean and the standard deviation, as it facilitates implementation. An approximation of the ELBO is required to enable the optimization of the network with gradient descent methods based on this loss function. This approximation is given by [16]  
\[\mathrm{ELBO}\approx\sum_{i=1}^{n}\log[q(w^{i}|\theta)-\log[\mathrm{P}(W^{i}) ]-\log[\mathrm{P}(D|W^{i})] \tag{3}\]  
To decrease the time required to train the networks, the Flipout estimator [19] is used to minimize the ELBO. Therefore, the output of each layer for all examples in the subgroup \(X\), considering the activation function \(\phi\) and the average weights \(\bar{W}\), is given by [19]  
\[Y=\phi[X\bar{W}+[(X\cdot B)\widehat{\Delta W}]\cdot A \tag{4}\]  
Since \(A\) and \(B\) are independently sampled from \(\widehat{\Delta W}\) and \(\bar{W}\), it is possible to back-propagate with respect to \(X\), \(\widehat{\Delta W}\), and \(\bar{W}\), optimizing, in this way, the network parameters. In short, VI directly approximates the posterior distribution with a simpler distribution, avoiding the calculation of the likelihood of all occurrences [20].