# Salient Semantic Segmentation Based on RGB-D Camera for Robot Semantic Mapping
## 4 Experiments
### Training and Evaluation in the Training Data
#### 4.1.3 Model Evaluation  
This section uses the Holdout verification method to evaluate the model. In all, 70% of the training set data in Table 1 were randomly selected for training. Moreover, the remaining 30% of the data were used to verify and evaluate the effectiveness of the model. As shown in Table 4, the maximum value of the average F1 score of the neural network was 0.979 at the threshold of 0.7. Moreover, the F1 scores of persons, dogs, and cars were currently 0.977, 0.979, and 0.982. The F1 score is the calculation result of comprehensive consideration of model Precision and Recall. The larger the F1 value, the higher the model quality. The AP of persons, dogs, and cars were 0.970, 0.965, and 0.975, respectively, and their mAP (mean AP) was 0.970. The mAP is the index to measure the effect of the model, and it is the mean value of three kinds of AP. The larger the mAP value, the better the model performance. For object recognition, the AP value is the area under the P-R curve of a certain category, and the precision (P) and recall rate (R) can be calculated for the recognition of each object under different thresholds. Figure 2 is the P-R curve of the three categories' average precision (P) and average recall (R). The training set of the three objects proposed in this paper is not from the same dataset, but the AP and F1 scores of each class of objects were good when the threshold of the three objects' recognition was set at 0.7. In addition to the F1 score, AP, and the relevant evaluation indicators, as shown in Table 4, the precision, recall, and accuracy were both good. Furthermore, the model has a processing speed of 23 consecutive frames per second (FPS) based on the experimental configuration, as shown in Table 3, which can be further improved by improving the hardware configurations. The main evaluation indicators show that the neural network model trained in this paper can be used in theory. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Category** & **Persons** & **Dogs** & **Cars** & **Average Value** \\ \hline F1 score & 0.977 & 0.979 & 0.982 & 0.979 \\ AP & 0.970 & 0.965 & 0.975 & 0.970 \\ Precision & 0.983 & 0.987 & 0.988 & 0.986 \\ Recall & 0.971 & 0.972 & 0.976 & 0.973 \\ Accuracy & 0.913 & 0.925 & 0.952 & 0.930 \\ \hline \hline \end{tabular}  
* The model’s main evaluation indicators are when its threshold for recognizing the three objects is set at 0.7. Our evaluation is based on our manually labeled ground truth of object recognition instead of based on the quality of salient semantic segmentation because there is no standard salient semantic segmentation dataset. The involved salient segmentation quality of the cars dataset depends on the segmentation quality of the used salient object detection model. Please refer to reference [24] for model details.  
\end{table} 
Table 4: Model evaluation indicators of salient object recognition. \begin{table}
\begin{tabular}{c c} \hline \hline
**Experimental Configuration** & **Detail Information** \\ \hline \hline  & Pytorch \\  & Torchvision \\  & Opencv \\  & python 3.6 \\  & tensorflow 1.8.0 \\  & keras 2.3.6 \\  & anaconda \\  & pycharm \\ Memory & 8 GB \\ GPU & NVIDIA GeForce GTX 1650 \\ Processor & AMD Ryzen 5 4600H with Radeon Graphics \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Experimental configurations. F1 is calculated by the following formula:  
\[\text{F1}=\frac{2 PR}{P+R}. \tag{1}\]  
The basis of F1-score and AP value calculation are Precision, Recall. We take the human category as an example to illustrate the calculation of the indicators, Precision, Recall, and Accuracy. (The AP value is the area under the P-R curve of a certain category.) \[\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}}, \tag{2}\]  
where P is the Precision denotes the ratio of the real person sample among the total number of samples predicted to be the person (TP) to the total number of samples predicted to be the person (TP + FP) when at a certain threshold.