# Searching for the Ideal Recipe for Preparing Synthetic Data in the Multi-Object Detection Problem
###### Abstract
The trained model will only be used over the ocean, so there is no need to diversify the environment so much. In the context of Virtual Worlds [30], synthetic data have found applications as proxies for multi-object tracking analysis. In the context of detecting cars and traffic, virtual crowd simulation is utilized to augment training data and expand datasets [23], particularly emphasizing highly reflective objects, with a focus on bathroom utilities. The video game environment [29] serves as a common tool, leveraging synthetic samples to achieve results comparable to models trained on real-world data. The integration of synthetic data generated within virtual simulations into training sets proves instrumental in significantly enhancing the performance of object detection algorithms. An approach involving the generation of synthetic objects on real backgrounds [28], featuring a high density of detectable objects, aims to emulate real-world clutter effectively. By incorporating multiple synthetic and real datasets alongside a simulation tool [32], there is potential to create large volumes of affordably annotated synthetic data. This approach can lead to the establishment of domain similarity among these datasets, contributing to more robust and comprehensive training datasets. \begin{table}
\begin{tabular}{l l l l l l} \hline
**Name** & **Year** & **Img** & **Obj** & **Sim** & **Object of Detection** \\ \hline
[28] & 2017 & - & - & ✓ & Vehicles \\
[29] & 2016 & - & - & ✓ & Various elements of a city \\
[30] & 2016 & - & - & ✓ & Vehicles \\ \hline \end{tabular}
\end{table} 
Table 1: _Cont._Several authors have emphasized the importance of achieving photorealism in simulated data, investigating the impact of synthetic datasets generated through photorealistic rendering techniques. This focus extends to areas such as street scene parsing [24] and transfer learning, with a predominant reliance on synthetic training data [27]. In the work in [15], a new algorithm was designed to enhance realism by initiating from a limited set of real images. Then, it estimates the rendering parameters needed to synthesize similar images when provided with a coarse 3D model of the target object. Furthermore, the generation of synthetic data [12] involves incorporating randomized illumination, blur, and noise to address the challenges of object detection in complex environments. This approach aims to overcome the limitations associated with existing methods, which heavily depend on large volumes of labeled real data. In the realm of cross-modality learning, a framework [26] employs terms utilizing a deep convolutional network to establish a non-linear mapping between RGB and thermal data. This enables the learning of features that are both discriminative and resilient to poor illumination conditions. Models trained exclusively on synthetic data often fall short in performance when tested on real-world datasets, and the process of data synthesis itself likely contributes to the observed domain gap [33]. To mitigate the disparity between real and synthetic data, two common strategies are employed. One approach involves mixing real data into the training set alongside synthetic data. Other strategies include conducting fine-tuning on mixed data after pretraining on larger uncorrelated with the given problem but in a similar domain dataset (like COCO) or enhancing the quality of synthetic data to align more closely with the target domain. While fine-tuning models with real data can lead to improvements, it does not address the fundamental issue of the domain gap. Addressing the domain gap for semantic segmentation [34] can be achieved by adapting the representations learned by segmentation networks across synthetic and real domains. Alternatively, domain randomization [25] can be applied together with fine-tuning on real data.