# Place Classification Algorithm Based on Semantic Segmented Objects
## 2 Proposed Classification Algorithm
### Design of Weighting Factors  
We build the weight matrix shown in Figure 3 by using 2017 validation images from the COCO-stuff dataset, including 5K images. These images are manually labeled as scene classes. After preparing dataset for constructing the matrix, the images are fed to the pre-trained image segmentation model one by one. We can get one or more objects as a result of segmentation.  
The COCO-stuff dataset [12] includes 80 \"thing\" classes and 91 \"stuff\" classes, and stuff classes are divided into two wide categories--indoor and outdoor. The outdoor classes contain various classes representing background of an image such as building, mountain, and so forth (Figure 2).  
The weight matrix \(W\) has size of \(M\times N\), \(M\) is the number of object classes, and \(N\) is the number of scene classes. Since we use COCO-stuff dataset and three scene labels, it turns out to 171 \(\times\) 3. It is initialized with zeros at first.  
As shown in Figure 3, assuming that we get classes of 'cow','mountain','sky', and 'grass' from an image. The image has a nature scene, so add 1 to nature column in weight matrix for each object class (cow, mountain, sky, and grass). After iteration of this process under 5 K images, the matrix would be constructed with various numbers. Therefore, we normalize it for each row. In the Equation (1), \(W^{\prime}\) denotes the normalized weight matrix. In addition, \(m\) is \(m\)-th object class in the dataset and \(n\) is \(n\)-th label in the place classes. The algorithm of constructing the weighting matrix is described in Algorithm 1 and the inference process with the model is shown in Algorithm 2.  
In the inference process, we perform semantic segmentation for each test image, and we compute the scores of scene or place classes of the image by multiplying bias value to background object weights and adding all of them of each scene or place class by using pre-constructed weight matrix as:  
\[W^{\prime}_{mn}=\frac{W_{mn}}{\sum_{n^{\prime}=1}^{N}W_{mn^{\prime}}}. \tag{1}\]Figure 2: COCO-Stuff label hierarchy.  
```
0:\(F\): pre-trained semantic segmentation model
1:Inputs:
2: scene-labeled images \(\{(x_{1},y_{1}),(x_{2},y_{2}),...,(x_{n},y_{n})\}\)
3:Initialize:
4:\(W[i][j]\gets 0\), \(i=1,\ldots,o,j=1,\ldots,p\)
5:\(\theta\): # of object classes, \(p\): # of scene classes
6:for t = 1 to n do
7:\(O_{t}\gets F(x_{t})\)
8:for\(o\in O_{t}\)do
9:\(W[o][y_{t}]\gets W[o][y_{t}]+1\)
10:endfor
11:endfor
12:for i = 1 to o do
13:for j = 1 to p do
14:\(W[i][j]\gets W[i][j]/(W[i][1]+\ldots+W[i][p])\)
15:endfor
16:endfor
```  
**Algorithm 1** Weighting Matrix \(W\)  
```
0:\(F\): pre-trained semantic segmentation model and \(W[i][j]\): weight matrix
1:Inputs:
2:Test images \(\{x_{1},x_{2},...,x_{n}\}\)
3:Initialize:
4:\(V[i]\gets 0\), \(i=1,\ldots,p\)
5:\(\theta\): # of object classes, \(p\): # of scene classes
6:for t = 1 to n do
7:\(O_{t}\gets F(x_{t})\)
8:for\(o\in O_{t}\)do
9:bias \(\gets 1\)
10:if o is in Background then
11:bias \(\gets 1.4\)
12:endif
13:\(V[t]\gets V[t]+W[o]\times\textit{bias}\)
14:endfor
15:endfor
16:\(\hat{y}\gets argmax(V[t])\)
```  
**Algorithm 2** Inference Process