# Enhanced Pet Behavior Prediction via S2 GAN-Based Heterogeneous Data Synthesis
## 4 Experiment
### Results of the Behavior Prediction Experiment  
Due to the lack of a clear benchmark for absolute performance and the absence of research on GAN models using heterogeneous data, especially video and sensor data, the impact of synthesizing heterogeneous data using the S2 GAN model on the accuracy of pet behavior prediction was evaluated. For this purpose, the data were divided into six dataset types: 100% real data, 50% real data, 50% synthesized data +50% real data, 30% synthesized data +70% real data, only 100% video data, and only 100% sensor data. This approach was devised to evaluate the synthetic data's contribution to enhancing the model's predictive accuracy. For the experiment, the training, validation, and test datasets for each of the six datasets were divided with a ratio of 7:2:1. Hyperparameter settings included a batch size of 16 using categorical cross entropy as the loss function, Adam as the optimization function, a learning rate of 0.001, and 100 epochs. Table 7 presents the experimental results according to the dataset used, displaying metrics such as Accuracy, Precision, Recall, and F1-Score. The best performance was observed when using 100% real data. Compared to the results from 50% real data, it was found that the performance improved as the proportion of real data increased, highlighting the positive impact of real data on the model's learning capabilities. Meanwhile, the results from using synthesized data, although slightly lower, did not significantly differ from those obtained with the 100% real dataset. In essence, the results obtained from this experiment confirm that the approach of synthesizing video skeleton features from sensor data using S2 GAN can adequately replace the missing data. Herein, we demonstrate the potential of synthetic data to supplement actual data and present a new methodology to overcome the limitations of datasets arising potentially because of missing information. To verify the efficacy of the proposed method, experiments were conducted using confusion matrices to analyze behavior prediction results from datasets comprising (a) only video data, (b) only sensor data, (c) 100% real data, and (d) a combination of 70% real and 30% synthetic data (Figure 13). First, a notable difference in the recognition of behaviors was observed while using video and sensor data independently. The video data revealed proficient prediction for behaviors 2, 3, 4, and 5, which are typically associated with less movement in pets, attributable to motion blur affecting the extraction of accurate skeleton coordinates. Moreover, sensor data usually yielded lower results than video data, with particularly poor recognition of behavior 0. Behaviors 7 and 8--often confused with behavior 1 due to similar sensor patterns--also demonstrated lower recognition rates, although sensor data were slightly better at capturing these behaviors than video data. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & **Accuracy** & **Precision** & **Recall** & **F1-Score** \\ \hline
**100\% real data** & **0.954** & **0.945** & **0.947** & **0.945** \\
50\% real data & 0.919 & 0.882 & 0.877 & 0.919 \\
50\% real data + 50\% synthetic data & 0.928 & 0.891 & 0.924 & 0.906 \\
70\% real data + 30\% synthetic  
When analyzing the results from 100% real data, there was an overall improvement in the recognition of behaviors that were not well detected by either video or sensor data alone, highlighting the advantage of integrating heterogeneous data to enhance performance. Meanwhile, the dataset with 70% synthetic data and 30% real data showed partially lower rates than that with 100% real data but demonstrated a general improvement across all behaviors. In summary, behavior prediction through data synthesis based on the proposed S2 GAN model demonstrated that synthetic data can effectively address missing data issues within multimodal behavior prediction systems.