# Automated Pixel-Level Deep Crack Segmentation on Historical Surfaces Using U-Net Models
## 3 The Proposed Deep Crack Segmentation Approach
### Semi-Supervised Pixel-Level Map Generation  
This phase aims to build up the training dataset for crack segmentation. As is known, pixel-level annotation is extremely expensive and labor-consuming, especially for critical domains such as structural damages, requiring expert dedication. This module only requires a bounding box around the object; then, GrabCut [22] will work iteratively to generate the mask instead of marking pixel by pixel. This saves time, effort, and cost, as shown in Figure 4.  
Figure 3: Samples of cracks in the primary dataset. GrabCut is an iterative segmentation method, which is an extension of the Graphcut method. Gaussian mixture models (GMMs) are used for color images instead of the model for monochrome images. GMMs are utilized to learn the color distributions of the background and foreground by assigning each pixel a probability to belong to a group of other pixels [22; 23]. Given a color image \(I\), let us consider the array \(z=(z_{1},\ldots,z_{n},\ldots,z_{N})\) of \(N\) pixels, where \(z_{i}=(R_{i},G_{i},B_{i}),i\in[1,\ldots,N]\) in the \(RGB\) space. The segmentation of the image is presented as an array of opacity values \(\alpha=(\alpha_{1},...,\alpha_{N})\) at each pixel with \(\alpha_{n}\in 0,1\), with 0 and 1 for background and foreground, respectively. Two GMMs for the background and the foreground are taken to be the full-covariance Gaussian mixture with \(K\) components. In the optimization framework, an extra vector \(k=k_{1},...,k_{n},...,k_{N}\) is introduced, with \(k_{n}\in 1,...K\), assigning each pixel a unique GMM component either from the foreground or the background model according to \(\alpha_{n}\) to easily deal with the GMM [22; 23]. The Gibbs energy function \(E\) defined by GrabCut for segmentation, such that its minimum value should agree with a good segmentation, and it is computed as follows:  
\[E(\underline{\alpha},\underline{\theta},z)=U(\underline{\alpha},k,\underline {\theta},z)+V(\underline{\alpha},z) \tag{1}\]  
The data term \(U\) evaluates the fit of the opacity distribution \(\alpha\) to the data \(z\), given the histogram model \(\theta\), and is defined to be  
\[U(\underline{\alpha},k,\underline{\theta},z)=\sum_{n}D(\alpha_{n},k_{n}, \underline{\theta},z_{n}) \tag{2}\]  
where \(D(\alpha_{n},k_{n},\underline{\theta},z_{n})=-\log p(z_{n}|\alpha_{n},k_{n}, \underline{\theta})-\log\pi(\alpha_{n},k_{n})\), \(p(\cdot)\) is a Gaussian probability distribution, and \(\pi(\cdot)\) are mixture weighting coefficients, so that (up to a constant)  
\[D(\alpha_{n},k_{n},\underline{\theta},z_{n})=-\log\pi(\alpha_{n},k_{n})+\frac {1}{2}\log\det\sum(\alpha_{n},k_{n})+\]  
\[\frac{1}{2}[z_{n}-\mu(\alpha_{n},k_{n})]^{T}\sum(\alpha_{n},k_{n})^{-1}[z_{n} -\mu(\alpha_{n},k_{n})] \tag{3}\]  
Therefore, the parameters of the model are as follows:  
Figure 4: Illustration of semi-supervised GrabCut pixel-level map generation. \[\underline{\theta}=\pi(\alpha,k),\mu(\alpha,k),\sum(\alpha,k),\alpha\in 0,1,k=1,...K \tag{4}\]  
where \(\pi,\mu,\sum\) are the weights, mean, and covariance values of the 2K Gaussian components for both background and foreground distributions. Meanwhile, the smoothness term \(V\) is computed as follows [22; 23]:  
\[V(\underline{\alpha},z)=\gamma\sum_{(m,n)\in C}\left[\alpha_{n}\
eq\alpha_{m} \right]\exp-\beta\|\;z_{m}-z_{n}\;\|^{2} \tag{5}\]  
The original GrabCut segmentation works as shown in Algorithm 1[22; 23]. ```
1: Obtain a bounding box \(b\) around the region of interest. 2: Initialize trimap \(T\) by supplying only \(T_{B}\). The foreground is set to \(T_{F}=0\); \(T_{U}^{\ast}=T_{B}\), complement of the background. 3: Make \(\alpha_{n}=0\) for \(n\in T_{B}\) and \(\alpha_{n}=1\) for \(n\in T_{U}\) as initial segmentation. 4: Initialize GMMs for each of background and foreground from sets \(\alpha_{n}=0\) and \(\alpha_{n}=1\), respectively.