# Scene Understanding Based on High-Order Potentials and Generative Adversarial Networks

Authors: Xiaoli Zhao; Guozhong Wang; Jiaqi Zhang; Xiang Zhang
Journal: Advances in Multimedia
 (2018)

---

## Abstract

Scene understanding is to predict a class label at each pixel of an image. In this study, we propose a semantic segmentation framework based on classic generative adversarial nets (GAN) to train a fully convolutional semantic segmentation model along with an adversarial network. To improve the consistency of the segmented image, the high-order potentials, instead of unary or pairwise potentials, are adopted. We realize the high-order potentials by substituting adversarial network for CRF model, which can continuously improve the consistency and details of the segmented semantic image until it cannot discriminate the segmented result from the ground truth. A number of experiments are conducted on PASCAL VOC 2012 and Cityscapes datasets, and the quantitative and qualitative assessments have shown the effectiveness of our proposed approach.


---

## Body

## 1. Introduction

Scene understanding, based on semantic segmentation, is a core problem in the field of computer vision, which has been applied to 2D image, video, and even volumetric data. Its goal is to assign each pixel a label and then provide complete understanding of a scene. Two examples of scene understanding are shown in Figure1. The importance of scene understanding is highlighted by the fact that there are increasing applications, such as autonomous driving [1], human-computer interaction [2], robot technology, and augmented reality, to name a few.Figure 1
Examples of scene parsing: (a) image; (b) ground truth.
(a) (b)The earliest scene parsing [3] is to classify 33 scenes for 2688 images on LMO dataset, which adopts label transfer technology to establish dense correspondences between the input image and each of the nearest neighbors using SIFT flow algorithm. State-of-the-art scene parsing frameworks are mostly based on fully convolutional network (FCN) [4]. FCN transforms the well-known networks-AlexNet, VGG, GooLeNet, and ResNet into fully convolutional ones by replacing the fully connected layers with convolutional ones. The key insight of FCN is to build the “fully convolutional” networks that take input of arbitrary size and produce corresponding-sized output with efficient inference and learning and realize end-to-end and image-to-image system of deep learning. For all these reasons and other contributions, FCN is considered as the milestone of deep learning. Although amounts of pooling operations enlarge the receptive fields of the convolution kernel of FCN, they lose the detailed location information, resulting in coarse segmentation result, which hinders its further application.In order to refine the segmentation result, a postprocessing stage using conditional random field (CRF) is adopted after the output of system [5], which makes use of the fully connected pairwise CRF to capture the dependencies of pixels and achieve fine local details. Dilated convolution is a generalization of Kronecker-factored convolutional filters [6] which expand exponentially receptive fields without losing resolution by disposing of some pooling layers. The works [7] that make use of this technique allow dense feature extraction on any arbitrary resolution and then combine dilated convolutions of different scales to have wider receptive fields with no additional cost. Combined CRF with dilated convolution, Chen et al. [8] propose the “deeplab” system, which enlarges the receptive fields of filters at multiple scales and overcomes the disadvantage of location accuracy by using a fully connected CRF to response the final layer of network. In order to take the dense CRF with pairwise potentials as an integral part of the network, Zheng et al. [9] propose a model called CRFasRNN to refine the segmentation of FCN; they make it possible to fully integrate the CRF with a FCN and train the whole network end to end. Although CRF taking into account the correlation of pixels has improved the segmentation accuracy, it has also increased the computational complexity. To incorporate suitable global features, Zhao et al. [10] propose a pyramid scene parsing network (PSPNet), which extends the pixel-level feature to special designed pyramid pooling one in addition to traditional dilated convolution. This algorithm achieves the champion of ImageNet scene parsing challenge 2016.In the above-mentioned algorithms, a common property is that all label variables are predicted either using unary potentials such as FCN or using pairwise potentials such as methods based on CRF. Despite the fact that pairwise potentials refine the accuracy of semantic segmentation, they only consider the correlation of two pixels. In an image, many pixels have the consistency across superpixels; high-order potentials should be effective in refining the segmentation accuracy. Arnab et al. [11] have integrated specific classes of high-order potentials in CNN-based segmentation models. This specific class may be object or superpixel and so on, for which we need to design different energy function to calculate high-order potentials, whose computation is complicated.The generative adversarial nets (GAN) proposed by Goodfellow et al. [12] in 2014 can be characterized by training a pair of networks in competition with each other, in which an adversarial network can estimate the generative model without approximating many intractable probability computation. Because there is no need for any Markov chains or unrolled approximate inference network, GAN has drawn many researchers’ attention in the domains of superresolution [13], image-to-image translation [14, 15], and image synthesis [16, 17], etc. We are interested in higher-order consistency without confining to a certain class. We also do not want to have complex probability or inference computation. Motivated by all kinds of GAN, we proposed a semantic segmentation framework based on GAN, which consists of two components: generative network and adversarial network. The former one generates the segmented image, and the latter one encourages the segmentation model to improve continuously the semantic segmentation result until it cannot be distinguished from the ground truth according to the value of loss function. Different from the classic GAN, we take the original image as the input of the generative network and and the output of generative network or corresponding ground truth as the input of the adversarial network; then adversarial network discriminates the similarity of two inputs. If the value of loss function of the framework is large, backpropagation is performed to adjust the parameters of the network; if the value of loss function satisfies the termination criterion, the output of the generative network is the final semantic segmentation result. The semantic segmentation framework based on GAN is shown in Figure 2. This approach takes into account the high-order potentials of an image because it differentiates the similarity between the segmented image and the corresponding ground truth in the whole image.Figure 2
Overview of the proposed scene parsing framework.


## 2. The Proposed Semantic Segmentation Approach

The aim of the proposed framework is to generate the semantic imageG(xn) from an original image xn. To achieve this goal, we design a generator network G and an adversarial network D. The generator is trained as a network parameterized by θG. These parameters denote the weights and are obtained by minimizing the loss function; then the output G(xn) of generator and the ground truth yn are fed into the adversarial network parameterized by θD, in which the discriminator is trained to distinguish real or fake value. In order to achieve the desired result, it is important to design the architecture network and loss function.


### 2.1. The Architecture of Networks

Some works have shown that deeper network model can improve the performance of the segmentation and meanwhile make the architecture of the network complex, resulting in difficult training [18]. We make a compromise between the depth of the network and the performance of the algorithm.In the generative network, which is shown in the first row of Figure3, there are two modules of convolution and deconvolution. The role of convolution module is to extract the feature maps of an image, which consists of 10 layers. Each layer is composed of convolution, activation function, and batch normalization. The convolution is performed with 3×3 kernels and 64 feature maps followed by ReLU layer as the activation function, whose role is to conduct the nonlinear operation. Batch normalization is performed to avoid the network overfitting in each layer. Although pooling operations enlarge the receptive field of the network, they also reduce the accuracy of the segmentation. To improve the fine details of feature maps, the last three pooling outputs are integrated into one, on which deconvolution is performed to achieve the same size output with the original image.Figure 3
Architecture of generative and adversarial networks.To discriminate the ground truth from the segmented image, we train a discriminator network, which is illustrated in the second row of Figure3. This architecture follows literature [13] to solve (4) in an alternating manner along with the generator. It contains eight convolution layers and uses LeakyReLU as the activation function. The convolution is conducted by 3×3 kernels, resulting in final feature maps of size 512, which are followed by two dense layers and a final sigmoid activation function to achieve a probability for classification.


### 2.2. Loss Function

In terms of information theory, cross entropy denotes the similarity of two variables; the more similar the distribution of two variables, the smaller the cross entropy, so we adopt the cross entropy as the loss function. The definition of cross entropy is shown in the following:(1)CEp,p^=-∑ipilog⁡pi^.where p and p^ are the real value and predicted value. Equation (1) is Shannon entropy when p and p^ are equal. In the multiple classification task, we use one-hot encoding cross entropy. Equation (1) can be rewritten as follows:(2)CEy,p^=-∑iyilog⁡pi^=-log⁡pi^.where y specifies one pixel of ground truth and yi represents 0 or 1.The loss function of the proposed networks is a weighted sum of two terms. The first is a multiclass cross entropy term of a generator that encourages the segmented output similar to the input. We useG(x) to denote the class probability map over C classes of size H×W×C that the segmentation model generates given an input image x of size H×W×C. This segmentation model predicts the right class label at each pixel independently, which is described in the following:(3)GLθG=lmcey,Gx=-∑i=1H×Wlog⁡pi^.where lmce(G(x),y) represents the cross entropy loss function of multiple classification on an image of size H×W, in which the class probability of per-pixel is predicted as pi^.The second loss term represents the loss of the adversarial network. If the adversarial network can distinguish the output of generator from the ground truth, the loss value is large; otherwise, the loss is small. Because the loss is calculated based on the whole image or a large portion of it, this high-order statistics dissimilarity can be penalized by the adversarial loss term. We take the output of the adversarial network asD(·)∈[0,1]. Training the adversarial model is equivalent to minimizing the following binary classification loss:(4)ALθD=lbceDy,1+lbceDGx,0.where lbce denotes the binary cross entropy loss and D(y) and D(G(x)) represent the label maps of adversarial network when the network input is the ground truth y or the output of a generator G(x).Given a data set ofN original images xn and the corresponding ground truth yn, we define the total loss functions of the proposed semantic segmentation networks based on GAN as in the following:(5)TLθG,θD=GLθG+α×ALθD=∑n=1Nlmceyn,Gxn+α×lbceDyn,1+lbceDGxn,0.where α denotes weight factor. In this paper, we set it as 0.01.


## 2.1. The Architecture of Networks

Some works have shown that deeper network model can improve the performance of the segmentation and meanwhile make the architecture of the network complex, resulting in difficult training [18]. We make a compromise between the depth of the network and the performance of the algorithm.In the generative network, which is shown in the first row of Figure3, there are two modules of convolution and deconvolution. The role of convolution module is to extract the feature maps of an image, which consists of 10 layers. Each layer is composed of convolution, activation function, and batch normalization. The convolution is performed with 3×3 kernels and 64 feature maps followed by ReLU layer as the activation function, whose role is to conduct the nonlinear operation. Batch normalization is performed to avoid the network overfitting in each layer. Although pooling operations enlarge the receptive field of the network, they also reduce the accuracy of the segmentation. To improve the fine details of feature maps, the last three pooling outputs are integrated into one, on which deconvolution is performed to achieve the same size output with the original image.Figure 3
Architecture of generative and adversarial networks.To discriminate the ground truth from the segmented image, we train a discriminator network, which is illustrated in the second row of Figure3. This architecture follows literature [13] to solve (4) in an alternating manner along with the generator. It contains eight convolution layers and uses LeakyReLU as the activation function. The convolution is conducted by 3×3 kernels, resulting in final feature maps of size 512, which are followed by two dense layers and a final sigmoid activation function to achieve a probability for classification.


## 2.2. Loss Function

In terms of information theory, cross entropy denotes the similarity of two variables; the more similar the distribution of two variables, the smaller the cross entropy, so we adopt the cross entropy as the loss function. The definition of cross entropy is shown in the following:(1)CEp,p^=-∑ipilog⁡pi^.where p and p^ are the real value and predicted value. Equation (1) is Shannon entropy when p and p^ are equal. In the multiple classification task, we use one-hot encoding cross entropy. Equation (1) can be rewritten as follows:(2)CEy,p^=-∑iyilog⁡pi^=-log⁡pi^.where y specifies one pixel of ground truth and yi represents 0 or 1.The loss function of the proposed networks is a weighted sum of two terms. The first is a multiclass cross entropy term of a generator that encourages the segmented output similar to the input. We useG(x) to denote the class probability map over C classes of size H×W×C that the segmentation model generates given an input image x of size H×W×C. This segmentation model predicts the right class label at each pixel independently, which is described in the following:(3)GLθG=lmcey,Gx=-∑i=1H×Wlog⁡pi^.where lmce(G(x),y) represents the cross entropy loss function of multiple classification on an image of size H×W, in which the class probability of per-pixel is predicted as pi^.The second loss term represents the loss of the adversarial network. If the adversarial network can distinguish the output of generator from the ground truth, the loss value is large; otherwise, the loss is small. Because the loss is calculated based on the whole image or a large portion of it, this high-order statistics dissimilarity can be penalized by the adversarial loss term. We take the output of the adversarial network asD(·)∈[0,1]. Training the adversarial model is equivalent to minimizing the following binary classification loss:(4)ALθD=lbceDy,1+lbceDGx,0.where lbce denotes the binary cross entropy loss and D(y) and D(G(x)) represent the label maps of adversarial network when the network input is the ground truth y or the output of a generator G(x).Given a data set ofN original images xn and the corresponding ground truth yn, we define the total loss functions of the proposed semantic segmentation networks based on GAN as in the following:(5)TLθG,θD=GLθG+α×ALθD=∑n=1Nlmceyn,Gxn+α×lbceDyn,1+lbceDGxn,0.where α denotes weight factor. In this paper, we set it as 0.01.


## 3. Experiments

To evaluate the proposed scene understanding algorithm based on GAN, we conduct some experiments on two widely used datasets, including PASCAL VOC 2012 [19] and urban scene understanding dataset Cityscapes [1]. We train networks on a NVIDIA Tesla K40 GPU and Intel Xeon E5 CPU using 2000 iterations and the batch size of size 16.To quantitatively assess the accuracy of scene parsing, four performance indices are adopted: pixel accuracy (PA), mean pixel accuracy (MPA), mean intersection over union (MeanIoU), and frequency weighted intersection over union (FWIoU), whose formulations [20] are in (6)−(9). We assume a total of k+1 classes, and pij is the amount of pixels of class i inferred to belong to class j. pii denotes the number of true positives, while pij and pji are usually represented as false positives and false negatives, respectively:(6)PA=∑i=0kpii∑i=0k∑j=0kpij.(7)MPA=1k+1∑i=0kpii∑j=0kpij.(8)MeanIoU=1k+1∑i=0kpii∑j=0kpij+∑j=0kpji-pii.(9)FWIoU=1∑i=0k∑j=0k∑i=0k∑j=0kpijpii∑j=0kpij+∑j=0kpji-pii.We use adaptive estimates of first-order moments (ADAM) [21] to optimize the algorithm because it requires little parameter-tuning, in which β1 and β2 are set to 0.9 and 0.999, respectively. We have also compared the divergence of different learning rate on the algorithm to select the optimal value, which is shown in Figure 4. According to this figure, we select 10-3 as the rate learning in these experiments.Figure 4
Divergence of different learning rate.


### 3.1. Experiment 1: PASCAL VOC 2012

We carry out experiments on PASCAL VOC 2012 segmentation dataset, which contains 20 object categories and 1 background class. Its augmented dataset [22] includes 10582, 1449, and 1456 images for training, validation, and testing. We have compared our method with the classic FCN [4] and popular DeepLab [5]: the accuracy of every class is shown in Table 1. Except for bicycle class, our approach achieves the highest accuracy on other 20 classes. Table 2 illustrates the four performance indices of different algorithms, PA, MPA, MeanIoU, and FWIoU. It is obvious that, from the left to right column, the accuracy of the algorithm gradually increases. The proposed approach gets the highest accuracy on these four performance indices.Table 1
Each class accuracy.
Class Label FCN-32s FCN-16s FCN-8s DeepLab Our Method Background 92.8 92.8 91.2 92.6 93.8 aeroplane 75.4 76.2 76.8 83.5 86.1 bicycle 33.6 34.3 34.4 36.6 35.9 bird 67.7 68.2 68.9 82.5 87.7 boat 48.6 49.4 49.4 62.3 63.5 bottle 58.4 59.2 60.3 66.5 67.2 bus 73.4 74.6 75.3 85.4 87.1 car 74.2 73.2 74.4 78.5 82.3 cat 77.6 78.4 77.6 73.7 86.8 chair 21.8 22.5 21.4 30.4 32.3 cow 62.1 62.5 62.5 72.9 76.5 Dining-table 46.3 46.7 46.8 60.4 62.0 dog 68.4 69.8 71.8 78.5 81.1 horse 63.4 63.8 63.9 75.5 77.9 motorbike 76.2 76.4 76.5 82.1 84.3 person 72.3 72.4 73.9 79.7 82.4 Potted-plant 44.5 44.5 45.2 58.2 59.6 sheep 71.2 71.6 72.4 82.0 84.3 sofa 37.4 37.2 37.4 48.8 54.9 train 69.4 69.8 70.9 73.7 76.2 tv/monitor 54.3 54.5 55.1 63.3 64.2Table 2
Four accuracy indices using different algorithms.
Accuracy FCN-32s FCN-16s FCN-8s DeepLab Our Method PA(%) 82.6 83.7 85.4 87.4 88.2 MPA(%) 61.3 61.8 62.1 69.8 72.6 Mean IOU(%) 63.5 64.5 67.2 70.3 73.9 FW IOU(%) 83.6 84.1 84.7 86.9 88.4To qualitatively validate the proposed method, several examples are exhibited in Figure5. For “cat” in row one, our method gets the cat in accordance with the ground truth; however, FCN and DeepLab segment other noise regions. For “cow” and “child” in rows two and five, the details, such as leg, can be segmented in our method, while leg cannot be found in images using other two methods. In the fourth image, little cow and person are segmented in fine contour comparing with other two methods. In a word, the subjective quality of the segmented image using DeepLab is better than that using FCN; the segmented result using our method outperforms those using FCN and DeepLab.Figure 5
Comparison of different semantic segmentation algorithms on PASCAL VOC 2012: (a) original image; (b) FCN-8S; (c) DeepLab; (d) our method; (e) ground truth.
(a) (b) (c) (d) (e)


### 3.2. Experiment 2: Cityscapes

Cityscapes [1] is a dataset for semantic urban scene understanding which was released in 2016. It contains 5000 high quality pixel-level finely annotated images collected from 50 cities in different seasons. The images, which consists of 2975, 500, and 1524 images for training, validation, and testing, are divided into 19 categories. Because this dataset is recently released, previous algorithms have not issued code for this dataset. We only do subjective assessment for Cityscapes using our method and FCN.Several examples are shown in Figure6. It is clear that our proposed method outperforms FCN and can achieve more details and distinguish road, building, cars, etc.Figure 6
Comparison of different semantic segmentation algorithms on Cityscapes: (a) original image; (b) FCN-8S; (c) our method; (d) ground truth.
(a) (b) (c) (d)


## 3.1. Experiment 1: PASCAL VOC 2012

We carry out experiments on PASCAL VOC 2012 segmentation dataset, which contains 20 object categories and 1 background class. Its augmented dataset [22] includes 10582, 1449, and 1456 images for training, validation, and testing. We have compared our method with the classic FCN [4] and popular DeepLab [5]: the accuracy of every class is shown in Table 1. Except for bicycle class, our approach achieves the highest accuracy on other 20 classes. Table 2 illustrates the four performance indices of different algorithms, PA, MPA, MeanIoU, and FWIoU. It is obvious that, from the left to right column, the accuracy of the algorithm gradually increases. The proposed approach gets the highest accuracy on these four performance indices.Table 1
Each class accuracy.
Class Label FCN-32s FCN-16s FCN-8s DeepLab Our Method Background 92.8 92.8 91.2 92.6 93.8 aeroplane 75.4 76.2 76.8 83.5 86.1 bicycle 33.6 34.3 34.4 36.6 35.9 bird 67.7 68.2 68.9 82.5 87.7 boat 48.6 49.4 49.4 62.3 63.5 bottle 58.4 59.2 60.3 66.5 67.2 bus 73.4 74.6 75.3 85.4 87.1 car 74.2 73.2 74.4 78.5 82.3 cat 77.6 78.4 77.6 73.7 86.8 chair 21.8 22.5 21.4 30.4 32.3 cow 62.1 62.5 62.5 72.9 76.5 Dining-table 46.3 46.7 46.8 60.4 62.0 dog 68.4 69.8 71.8 78.5 81.1 horse 63.4 63.8 63.9 75.5 77.9 motorbike 76.2 76.4 76.5 82.1 84.3 person 72.3 72.4 73.9 79.7 82.4 Potted-plant 44.5 44.5 45.2 58.2 59.6 sheep 71.2 71.6 72.4 82.0 84.3 sofa 37.4 37.2 37.4 48.8 54.9 train 69.4 69.8 70.9 73.7 76.2 tv/monitor 54.3 54.5 55.1 63.3 64.2Table 2
Four accuracy indices using different algorithms.
Accuracy FCN-32s FCN-16s FCN-8s DeepLab Our Method PA(%) 82.6 83.7 85.4 87.4 88.2 MPA(%) 61.3 61.8 62.1 69.8 72.6 Mean IOU(%) 63.5 64.5 67.2 70.3 73.9 FW IOU(%) 83.6 84.1 84.7 86.9 88.4To qualitatively validate the proposed method, several examples are exhibited in Figure5. For “cat” in row one, our method gets the cat in accordance with the ground truth; however, FCN and DeepLab segment other noise regions. For “cow” and “child” in rows two and five, the details, such as leg, can be segmented in our method, while leg cannot be found in images using other two methods. In the fourth image, little cow and person are segmented in fine contour comparing with other two methods. In a word, the subjective quality of the segmented image using DeepLab is better than that using FCN; the segmented result using our method outperforms those using FCN and DeepLab.Figure 5
Comparison of different semantic segmentation algorithms on PASCAL VOC 2012: (a) original image; (b) FCN-8S; (c) DeepLab; (d) our method; (e) ground truth.
(a) (b) (c) (d) (e)


## 3.2. Experiment 2: Cityscapes

Cityscapes [1] is a dataset for semantic urban scene understanding which was released in 2016. It contains 5000 high quality pixel-level finely annotated images collected from 50 cities in different seasons. The images, which consists of 2975, 500, and 1524 images for training, validation, and testing, are divided into 19 categories. Because this dataset is recently released, previous algorithms have not issued code for this dataset. We only do subjective assessment for Cityscapes using our method and FCN.Several examples are shown in Figure6. It is clear that our proposed method outperforms FCN and can achieve more details and distinguish road, building, cars, etc.Figure 6
Comparison of different semantic segmentation algorithms on Cityscapes: (a) original image; (b) FCN-8S; (c) our method; (d) ground truth.
(a) (b) (c) (d)


## 4. Conclusion

In this paper, we propose a scene understanding framework based on generative adversarial networks, which trains the fully convolutional semantic segmentation network by adversarial network, and adopt high-order potentials to achieve the fine details and consistency of the segmented semantic image. We perform a number of experiments on two famous datasets, PASCAL VOC 2012 and Cityscapes. We analyze not only each class accuracy but also four accuracy indices by using different semantic segmentation algorithms. The quantitative and qualitative assessments have shown our proposed method achieves the best accuracy among all algorithms. In the future, we will do more experiments on Cityscapes dataset and address the misclassification caused by class imbalance.



---

*Source: 8207201-2018-08-05.xml*