# Smart Detection System of Safety Hazards in Industry 5.0
## 5 Experiments and Results
### Experimenting with Synthetic and Limited Amount of Real-World Data  
As described in Section 4.2, an AI model trained exclusively on a synthetic dataset cannot achieve as high performance as one trained on a real-world dataset. However, in many real-world scenarios, the acquisition of large-scale amounts of real data is infeasible, posing a significant challenge to the development of an effective AI model. Therefore, the combination of real and synthetic data that constitutes the training dataset seems like a promising solution, addressing the problem of limited real data. Therefore, it is crucial to define the optimal ratio of real-world and synthetic data to maximize the model's performance. Through a series of experiments, we aim to examine the potential benefits and limitations of combining real and synthetic images for the creation of a training set that can be used to train AI models for real-world applications. To achieve this, we design a series of experiments where we begin with a relatively small number of real images and progressively augment the dataset with synthetic images. The real images are sourced from the CHV dataset, of which 50 are allocated for the training set and 25 for the validation set. In each experiment, we trained the YOLOv5s model on the combined training dataset while we evaluated it on the CHV test set, comprising 133 images, to ensure a consistent benchmark for evaluation. Table 4 presents the number of real and synthetic images that constitute the training set for each experiment. In the first one, working under the constraints of limited data availability, the model is trained exclusively on a set of 50 real images. In all the following six experiments, the number of real images remains the same while we incrementally increase the number of synthetic images. YOLOv5s is trained on training datasets of the various experiments, and it is evaluated at the same CHV test set. Table 5 presents the mAP values of trained models for each experiment. In the initial experiment, the model, trained with only a small number of real images, achieved a mAP of 14.3%. In the second experiment, the incorporation of synthetic data led to a slight improvement in the model's performance, increasing overall mAP to 16.3%. In the third one, a significant rise in mAP is observed, escalating to 71.2%. The highest mAP value of 84.1% is achieved in the sixth experiment, in which the training set contains 50 real images and 600 synthetic ones. In the last experiment, where we doubled the number of synthetic images in the training set to 1200, it was observed that the mAP value slightly decreased to 81.0%. One explanation for this decrease in performance is the model's over-adaptation to the characteristics of the synthetic data. When exposed to a significant volume of synthetic data, the model develops a bias, becoming particularly adept at recognizing objects in virtual conditions. Consequently, its ability to generalize and recognize objects in real-world scenarios could be compromised. This trend suggests a potential saturation point beyond \begin{table}
\begin{tabular}{c c c} \hline
**Experiment ID** & **Real Images Number** & **Synthetic Images Number** \\ \hline E1\_50\_0 & 50 & 0 \\ E2\_50\_50 & 50 & 50 \\ E3\_50\_100 & 50 & 100 \\ E4\_50\_150 & 50 & 150 \\ E5\_50\_300 & 50 & 300 \\ E6\_50\_600 & 50 & 600 \\ E7\_50\_1200 & 50 & 1200 \\ \hline \end{tabular}
\end{table} 
Table 4: Number of real-world and synthetic images for each experiment. \begin{table}
\begin{tabular}{c c c c c c c c} \hline  & **E1\_50\_0** & **E2\_50\_50** & **E3\_50\_100** & **E4\_50\_150** & **E5\_50\_300** & **E6\_50\_600** & **E7\_50\_1200** \\ \hline mAP & 14.3\% & 16.3\% & 17.3\% & 71.2\% & 79.5\% & 84.1\% & 81.0\% \\ \hline \end{tabular}
\end{table} 
Table 5: mAP values of object detector for each experiment. which adding more synthetic data may not necessarily lead to performance gains and, in fact, could risk the model's efficacy in real-world conditions, leading to diminished returns.