# Relevancy Between Objects Based on Common Sense for Semantic Segmentation
###### Abstract  
Research on image classification sparked the latest deep-learning boom. Many downstream tasks, including semantic segmentation, benefit from it. The state-of-the-art semantic segmentation models are all based on deep learning, and they sometimes make some semantic mistakes. In a semantic segmentation dataset with a small number of categories, images are often collected from a single scene, and there is a close semantic connection between any two categories. However, in the semantic segmentation dataset collected from multiple scenes, two categories may be irrelevant. The probability of objects in one category appearing next to objects in other categories is different, which is the basis of the paper. Semantic segmentation methods need to solve two problems of positioning and classification. This paper is dedicated to correcting those clearly wrong classifications that are contrary to reality. Specifically, we first calculate the relevancy between different class pairs. Then, based on this knowledge, we infer the category of a connected component according to the relationships of this connected component with its surrounding connected components and correct the obviously wrong classifications made by a deep learning semantic segmentation model. Several well-performing deep learning models are experimented on two challenging public datasets in the field of semantic image segmentation. Our proposed method improves the performance of UPerNet, OCRNet and SETR from 40.7%, 43% and 48.64% to 42.07%, 44.09% and 49.09% mean IoU on the ADE20K validation set, and the performance of PSPNet, DeepLabV3 and OCRNet from 37.26%, 37.3% and 39.5% to 38.93%, 38.95% and 40.63% mean IoU on the COCO-Stuff dataset, which shows the effectiveness of the method. semantic segmentation; scene parsing; computer vision +
Footnote †: journal: Medical Image Analysis  
by the encoder. Feature representation learning is the most important model component, which usually is finished by a backbone pretrained on the ImageNet. CNN is famous for the extraction of local features. After removing the fully connected layer, each element in the feature map output by the fully convolutional network has a shortage of encoding long-range dependency information, which is momentous for semantic segmentation and results in coarse results. To overcome the lack of multi-scale information, many works enlarged receptive fields of convolution operations. DeepLab [8; 9; 10; 11] introduced artous convolution and developed the ASPP module that adopted atrous convolutions with different dilation rates. PSPNet [12] proposed the PPM module to obtain contextual information on different scales and built a feature pyramid. To balance the contradiction between semantic and spatial information, many works [13] aggregated the feature maps from different stages of the backbone, and multiple variants of the encoder-decoder architectures are developed. However, the aforementioned methods still do not know which part of context information is pivotal for segmentation. To address this question, some works integrate attention modules into FCN-based architectures. PSANet [14] proposed the point-wise spatial attention module for capturing the long-range contextual information. DANet [15] used two attention modules to model long-range dependencies from spatial and channel dimensions, respectively. CCNet [16] introduced a relatively low computation complexity attention module. OCNet [17] attached attention modules to each branch of the PPM module or the ASPP module. The aforementioned models are still FCN-based models that are decorated with attention modules. More recently, the success of transformer in natural language processing inspires researchers to treat computer vision tasks as sequence-to-sequence tasks and build pure transformers for computer vision tasks. They usually deploy a transformer [18] to encode an image as a sequence of patches.