## 2 Theoretical Background
### Conditional Random Fields as Recurrent Neural Networks  
CRFs achieved state-of-the-art results in the image segmentation tasks, both in the traditional benchmark [47], as well as in application to medical image analysis, such as in PCa segmentation [48], weakly supervised segmentation of PCa [49], GS grading [50] or PCa detection [51]. CRF functioning is based on the notion of energy \(\mathcal{E}(\cdot)\), i.e., the cost of assigning a label to a given pixel. A CRF is composed of two types of energy--namely, unary and pairwise--which must agree and can be described as:  
\[\mathcal{E}(x)=\sum_{i}\Psi_{u}(x_{i})+\sum_{p}\Psi_{p}(x_{i},x_{j}). \tag{1}\]  
The unary energy \(\Psi_{u}(x_{i})\) is the probability of a pixel \(i\) belonging to a given label \(x_{i}\) in this case extracted by a CNN. Conversely, \(\Psi_{p}(x_{i},x_{j})\) corresponds to the pairwise energy. It measures the cost of assigning the labels \(x_{i}\) and \(x_{j}\) to pixels \(i\) and \(j\) simultaneously. It ensures image smoothness and consistency; pixels with similar properties should have similar labels. Thus, CRFs promote regions of homogeneous predictions. The pairwise energy is defined according to [52]:  
\[\Psi_{p}(x_{i},x_{j})=\mu(x_{i},x_{j})\mathcal{K}(\mathbf{f}_{i},\mathbf{f}_{j }), \tag{2}\]  
where \(\mu(x_{i},x_{j})\) is a label compatibility function and the Gaussian kernel \(\mathcal{K}(\cdot,\cdot)\) applied on the feature vectors \(\mathbf{f}_{i}\) and \(\mathbf{f}_{j}\), with \(w^{(m)}\) being linear combination weights:  
\[\mathcal{K}(\mathbf{f}_{i},\mathbf{f}_{j})=\sum_{m=1}^{k}w^{(m)}\mathcal{K}^{ (m)}(\mathbf{f}_{i},\mathbf{f}_{j}). \tag{3}\]  
In our case, the features consider the positions \(p_{i}\) and \(p_{j}\) as well as the intensity values \(I_{i}\) and \(I_{j}\) of the pixels in the image:  
\[k(\mathbf{f}_{i},\mathbf{f}_{j})=w^{(1)}\underbrace{\exp\Bigg{(}-\frac{|p_{i} -p_{j}|^{2}}{2\theta_{\pi}^{2}}-\frac{|I_{i}-I_{j}|^{2}}{2\theta_{\beta}^{2}} \Bigg{)}}_{\text{appearance kernel}}+w^{(2)}\underbrace{\exp\Bigg{(}-\frac{|p _{i}-p_{j}|^{2}}{2\theta_{\gamma}^{2}}\Bigg{)}}_{\text{smoothness kernel}}, \tag{4}\]  
with \(\theta_{\pi},\ \theta_{\beta}\) and \(\theta_{\gamma}\) being hyper-parameters controlling the importance of the hyper-voxel distance in the feature space (i.e., the appearance kernel promotes pixels with similar intensity values to be in the same class, while the smoothness kernel removes small isolated regions [52])--a stronger penalty is given if nearby pixels have different labels. In this version, the Potts model is used, i.e., \(\mu(x_{i},x_{j})=[x_{i}\
eq x_{j}]\)[31; 52]. The training time for a CRF grows exponentially with respect to the number of input pixels \(N\), even when considering approximate training methods, like Markov Chain Monte Carlo (MCMC), pseudo-likelihood or junction trees [31; 53]. To solve this shortcoming, the Mean Field approximation (MFa) can be used [52]. MFa consists in approximating the distribution \(P(\mathbf{X})\),--where \(\mathbf{X}\) is the vector of the random variables \(X_{1},X_{2},\ldots,X_{N}\) denoting the \(N\) pixels composing the image--by a simpler distribution \(Q(\mathbf{X})\), which can be written as the product of independent marginal distributions: \(Q(\mathbf{X})=\prod_{i}Q_{i}(\mathbf{X}_{i})\), subject to: \(\sum_{x_{i}}Q_{i}(\mathbf{X}_{i})=1\). The function \(Q_{i}(x_{i})\) can be defined such that is updated iteratively [52]. Bearing this in mind, an MFa-trained CRF still cannot be trained by means of the backpropagation, making its end-to-end integration with CNN infeasible, as the CNN and CRF need to be trained separately. Further refining MFa, the authors of [31] formalized the CRFs as Recurrent Neural Networks (CRF-RNN). This work redefines the MFa as a series of convolutional and recurrent layers. The convolutional layers perform the Gaussian operations on the features via learnable filters, while the recurrent layers behave as several iterations of the MFa method.