## 4 Experiments
### Experimental Results
#### 4.2.3 Dog Behavior Recognition Based on Multimodal Data  
The performance evaluation of the fusion model was performed on the fusion models CNN, LSTM, and CNN-LSTM for each dog detection method. In addition, a model that only concatenated the extracted features for each set of data were added to the performance evaluation to compare the results of single data and multimodal data models.  
Experimental results are shown in Table 7. Compared with the performance when using only a single data type in Sections 4.2.1 and 4.2.2, it was confirmed that the average performance was improved by about 90.3% when the data were integrated. Among the fusion models for each dog detection method, the CNN-LSTM model showed higher accuracy than other fusion models. In particular, in the case of YOLOv4, which had the highest behavior recognition accuracy using video data, the CNN-LSTM model showed the highest accuracy at about 93.4%. The performance of the CNN model and the LSTM model is expected to improve slightly if they had as deep layers as the CNN-LSTM model.  
Figure 10 shows the confusion matrix for each fusion model. The recognition rate for all behaviors is improved overall compared to the case of using single data. However,  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{
\begin{tabular}{c} **Dog Detection** \\ **Method** \\ \end{tabular} } & \multirow{2}{*}{**Fusion Model**} & \multicolumn{4}{c}{**Measure**} \\ \cline{3-6}  & & **Accuracy** & **Precision** & **Recall** & **F-Score** \\ \hline \multirow{4}{*}{FasterRCNN} & Only concatenate & 0.885 & 0.89 & 0.89 & 0.89 \\  & CNN & 0.855 & 0.88 & 0.85 & 0.86 \\  & LSTM & 0.885 & 0.91 & 0.89 & 0.89 \\  & CNN-LSTM & 0.924 & 0.94 & 0.92 & 0.93 \\ \hline \multirow{4}{*}{YOLOv3} & Only concatenate & 0.908 & 0.92 & 0.91 & 0.91 \\  & CNN & 0.926 & 0.94 & 0.93 & 0.93 \\ \cline{1-1}  & LSTM & 0.914 & 0.92 & 0.91 & 0.92 \\ \cline{1-1}  & CNN-LSTM & 0.920 & 0.93 & 0.92 & 0.92 \\ \hline \multirow{4}{*}{YOLOv4} & Only concatenate & 0.904 & 0.91 & 0.9 & 0.9 \\  & CNN & 0.898 & 0.91 & 0.9 & 0.9 \\ \cline{1-1}  & LSTM & 0.880 & 0.9 & 0.88 & 0.88 \\ \cline{1-1}  & CNN-LSTM & **0.934** & **0.94** & **0.93** & **0.93** \\ \hline \multicolumn{2}{c}{Average} & **0.903** & **0.92** & **0.90** & **0.91** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Results of behavior recognition performance evaluation by fusion models based on multimodal data.  
Figure 9: Confusion matrix of behavior recognition by selected features using sensor data: (**a**) Confusion matrix with no selected features; (**b**) Confusion matrix with selected features of \(f_{mean}+f_{var}+f_{std}+f_{amp}+f_{skew}\).  
among them, the behavior recognition rate for B5 and B7 is slightly low. FasterRCNN, which showed a very lower recognition rate for B5 and B7 when using only video data, improved when using multimodal data, but still shows low numbers as shown in Figure 10a. Even with the CNN-LSTM model, FasterRCNN had a low behavior recognition rate for B5 and B7 as shown in Figure 10d, whereas YOLOv3 and YOLOv4 significantly improved the behavior recognition rate with the CNN-LSTM model as shown in Figure 10e,f.  
Since some aspects of sniffing may look similar to standing or walking, and walking has similar appearances to running except for speed, it is thought that detailed feature extraction is necessary to improve the accuracy of behavior recognition. Since the model structure proposed in this paper is not an optimal model for behavior recognition of dogs, if the model is improved, such as by increasing the layer depth or finding an appropriate hyperparameter, the recognition rate of all behaviors will probably be improved.