# Relevancy Between Objects Based on Common Sense for Semantic Segmentation
## 2 Method
\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline  & **Wall** & **Floor** & **Bed** & **Pillow** & **Sea** & **Sand** & **Road** & **Car** & **Tree** & **Lake** \\ \hline wall & 1 & 0.6449 & 0.1535 & 0.0529 & 0.0032 & 0.0017 & 0.0294 & 0.0331 & 0.0706 & 0 \\ floor & 0.8023 & 1 & 0.1746 & 0.0017 & 0.0005 & 0.0003 & 0.0015 & 0.0108 & 0.0042 & 0 \\ bed & 0.9748 & 0.8910 & 1 & 0.4816 & 0 & 0 & 0 & 0 & 0.0005 & 0 \\ pillow & 0.6591 & 0.0172 & 0.9452 & 1 & 0 & 0 & 0.0011 & 0 & 0 & 0 \\ sea & 0.0569 & 0.0077 & 0 & 0 & 1 & 0.3 & 0.0123 & 0.0077 & 0.1631 & 0 \\ sand & 0.0645 & 0.0097 & 0 & 0.0032 & 0.6290 & 1 & 0.0258 & 0.0065 & 0.1935 & 0.0032 \\ road & 0.0855 & 0.0035 & 0 & 0 & 0.0020 & 0.0020 & 1 & 0.6170 & 0.2243 & 0 \\ car & 0.1212 & 0.0320 & 0 & 0 & 0.0016 & 0.0006 & 0.7789 & 1 & 0.3353 & 0 \\ tree & 0.1226 & 0.0058 & 0.0001 & 0 & 0.0159 & 0.0090 & 0.1341 & 0.1588 & 1 & 0.0031 \\ lake & 0 & 0 & 0 & 0 & 0 & 0.0192 & 0 & 0 & 0.4038 & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Relevancy between different classes. The numbers in the table indicate the proportion of connected components labeled as one class that is adjacent to connected components labeled as another class. For example, floor is the closest class to wall. A total of 64.49% of the connected components labeled wall are adjacent to the connected components labeled floor. Lake is the farthest class from wall. There are no connected components labeled wall adjacent to a connected component labeled lake. Figure 2: Illustration of the central connected component and its neighbors. We first calculate the average confidence for each class over all pixels in region \(C\). \[avg\_conf_{k}=\frac{\sum_{i,i\in C}conf_{ik}}{n} \tag{2}\]  
where \(conf_{ik}\) is the predicted confidence that pixel \(i\) belongs to class \(k\) and \(n\) is the number of pixels in region \(C\). The class with the highest average confidence is the class that model predicted for region \(C\). \[l_{C}=\operatorname*{argmax}_{k\in K}(\mathbf{avg\_conf}_{C}) \tag{3}\]  
where \(K\) is the number of categories in the dataset. If the highest average confidence of a connected component is below the predefined threshold, it means that the model's prediction for this region is not confident enough. These connected components are what we think needs to be processed. The following describes the first post-processing method. We use the frequencies as the relevancy matrix of all classes. Each category in \(S\) has a relevancy vector with other classes. We add the relevancy vectors of all adjacent connected components as the reference \(\mathbf{r}_{c}\) for the central connected component in which we are interested. \[\mathbf{r}_{c}=\sum_{j\in S}\mathbf{f}_{j} \tag{4}\]  
where \(\mathbf{f}_{j}\) is the frequency vector of the \(j\)th surrounding class with other classes, and \(S\) is the set of classes to which the connected components around the central component belong. The highest-scored class in the reference is the most likely class to which the region of the image belongs, and the lowest-scored class is the opposite. If we denote the \(k\)th highest score in \(\mathbf{r}\) as \(r\), then the set of the top \(k\) scored classes denoted as \(K\) can be expressed as:  
\[K=\{i,r_{i}>r\} \tag{5}\]  
We think that the right class for region \(C\) is in \(K\). Therefore, we keep the top k scored classes and discard the rest. If the class that the deep learning semantic segmentation model predicted for region \(C\) is not in \(K\), then we change the classification result and choose the class with the highest average confidence among the top k scored classes as the classification result for the region. To this end, we set the average confidence of classes not in \(K\) to 0. \[avg\_conf_{i,i\
otin K}=0 \tag{6}\]  
At this point, the class with the highest average confidence in \(\mathbf{avg\_conf}\) is the label that we think region \(C\) belongs to. The category of each pixel in region \(C\) predicted by the model is the same.