# Relevancy Between Objects Based on Common Sense for Semantic Segmentation
## 2 Method  
In this section, two post-processing methods for deep learning-based semantic segmentation models are introduced. There are usually many objects in the image in complex application scenarios. The categories of objects that appear in images taken from similar scenes are highly correlated. For example, there are beds, pillows and bedside tables in a bedroom and tables and cutlery in a dining room. On roads, we can see cars, buildings and traffic signs. Under normal circumstances, it is impossible for cars, traffic signs, grass, etc., to appear around a bed. There is a high probability that pillows and quilts will appear on a bed. Therefore, the relationship between two classes may be close or distant. This is not significant for datasets such as cityscapes [30] that focus on a single scene. However, for datasets such as ADE20K that are collected from multiple scenes, the relevancy between classes A and B may differ greatly from the relevancy between classes A and C.  
Each annotation in a semantic segmentation dataset is a matrix in which each element is a number representing a category. Inside a thing or a patch of stuff, it is filled with the same number, which makes it a connected component. As shown in Figure 1, there are other connected components around each thing or stuff. As mentioned before, connected components of class B maybe often appear around connected components of class A, while connected components of class C do not. Therefore, we count the frequency of any two adjacent connected components appearing in annotations of the training dataset to find out the distance between any two classes. We denote the frequency of objects belonging to class \(j\) appearing adjacent to objects belonging to class \(i\) as \(f_{ij}\). Frequencies between some categories are shown in Table 1. If region \(S\) is adjacent to region \(C\), then we approximate the probability that \(C\) belongs to category \(j\) under the condition that \(S\) is known to belong to category \(i\) as \(f_{ij}\). \[P(C=j|S=i)=f_{ij} \tag{1}\]  
The same as common conception, the class most closely related to wall is floor, and the least closely related class to wall is lake. Lake never appeared next to wall in the training set. While ceiling and wall share the closest connection. The closeness between many class pairs is 0, such as building and bed, road and cabinet. Therefore, if a model infers a region of an image to be road and it is true, then it is probably wrong that a region surrounded by road is inferred to be a cabinet. This letter is inspired by these common senses. In the semantic segmentation task, a model needs to localize and classify. In popular methods based on deep learning, various network architectures are designed to extract features. Regardless of the difference between network structures, all these networks will generate feature maps with the same number of channels as the number of categories. Then, feature maps are fed into a softmax layer. The class with the highest confidence is chosen as the prediction for each pixel. Such wrong inferences mentioned earlier are made by bad classifications of a model. In this letter, we only address classification, not localization. In order to correct wrong classifications of objects that deep learning models may make, we propose two post-processing methods to correct such wrong classifications. For each connected component, the categories of all connected components around it have an impact on the classification of the region. Our proposed post-processing methods are based on connected components. As shown in Figure 2, we denote the central connected component as \(C\) and the \(i\)th connected component as \(S_{i}\). Figure 1: An image and its annotation from the ADE20K dataset.