# Weakly Supervised Conditional Random Fields Model for Semantic Segmentation with Image Patches
## 3 The Proposed Method
### The Patch Based CRF Model Construct  
Semantic inference has always been crucial to weakly supervised ISS as it directly affects the segmentation results. For the determined image patches, they are used as nodes of CRF to construct an undirected graph \(G(V,E)\), where \(V\) and \(E\) denote respectively the node and the edge connecting the nodes. Each image patch needs to be assigned with a unique category of labelled class, which is determined via the CRF model. On this basis, given the observed field \(x\) formed by the extracted patches, the conditional probability distribution of the CRF model for \(y\) is defined as:  
\[P(y\big{|}x)=\alpha\exp(-E(x,y)) \tag{4}\]  
where \(P(y\big{|}x)\) is a conditional probability, \(E(x,y)\) is an energy function, the final category label assignment result is \(\overline{y}\) which satisfies the maximum posteriori probability. \[\overline{y}=\text{argmax}_{y}P(y\big{|}x) \tag{5}\]  
The energy function of patch based CRF model can be defined as:  
\[E(x,y)=w_{1}\sum_{i\in p}\phi_{i}(y_{i},x_{i})+w_{2}\sum_{(i,j)\in E}\phi_{ij }\Big{(}y_{i},y_{j},x_{i},x_{j}\Big{)} \tag{6}\]  
Figure 2: Illustration diagram of the superpixel merging algorithm: (**a**) the original image, (**b**) the superpixel generated image, (**c**) the superpixel merged into image patches image. where \(w_{1}\) and \(w_{2}\) are weights, \(\phi_{i}(y_{i},x_{i})\) is a unary potential energy function, which measures the probability that a node \(i\) is labeled as \(y_{i}\) for a given \(x\). \(\phi_{ij}\Big{(}y_{i},y_{j},x_{i},x_{j}\Big{)}\) is a pairwise potential energy function between adjacent nodes \(i\) and \(j\). Obviously, the solution of \(y\) in Equation (5) is the minimum value in Equation (6), thus Equation (5) is equivalent to Equation (7) below. \[\overline{y}=\text{argmin}_{y}E(x,y) \tag{7}\]  
Let \(X=[x_{1},\cdots,x_{P}]\) be an image containing \(P\) image patches, and \(x_{i}\) is the \(i\)-_th_ image patch. Its corresponding semantic category is labelled as \(y=\Big{[}y_{1},\cdots y_{P}\Big{]}\) where \(y\in[1,\cdots L]\), and \(L\) denotes the total number of categories. However, we can be encoded the label information at the image level, \(l(x_{i})=\left[l_{1},l_{2},\cdots l_{L}\right]^{T}\), where \(l_{i}\in[0,1]\), and \(l_{i}=1\) means that the category appears in this image, \(l_{i}=0\) means not appearing. The PBCRF model is used to assign similar image patches with the same semantic categories and less similar patches to different semantic categories. In the process of assigning each image patch to an appropriate semantic label, the unary potential energy function of the CRF is formulated as (8)  
\[\phi_{i}(y_{i},x_{i})=\frac{1}{Z(x)}\exp\left(\sum_{x_{i}\in N(i)}\left(1-l_{i }(x_{i}^{*},x_{i})\right)\right) \tag{8}\]  
where \(Z(x)\) is the normalization factor, \(N(i)\) refers to the set of image patches adjacent to \(x_{i}\). \(l\) is the value of the image label, and \(l_{i}(\cdot)\) equals to only 0 or 1. \(l(x_{i}^{*})\in R^{L}\) is the true label, \(x_{i}\) is the element of \(l_{i}\). Furthermore, in order to assign an appropriate semantic label to each superpixel, category correlation and similarity function are added to the pairwise potential function. Pairwise potential energy functions are defined by  
\[\phi_{ij}\Big{(}y_{i},y_{j},x_{i},x_{j}\Big{)}=t\Big{(}y_{i},y_{j}\Big{)}\exp \left(-\frac{\left\|\Psi_{x_{i}}-\Psi_{x_{j}}\right\|^{2}}{\delta}\right)+ \left(1-\mu\Big{(}y_{i},y_{j}\Big{)}\right)\exp\left(-\frac{\left\|D_{x_{i}}- D_{x_{j}}\right\|^{2}}{\delta}\right)\Big{(}y_{i}\
eq y_{j}\Big{)} \tag{9}\]  
where \(\delta\) is used to adjust the width of the Gauss nucleus, which is set to \(\delta=1\) in the experiment. \(\Psi_{x_{i}}\) is the feature descriptor of \(x_{i}\), \(D_{x_{i}}\) is the distance feature of \(x_{i}\). It is very important to categorize association information for semantic label inference.