# Using Auto-ML on Synthetic Point Cloud Generation
## 3 Results and Discussion  
Table 2 states the results of the first experiment. It shows the parameters chosen for data generation as well as the corresponding performance with confidence. On system exit, the random optimizer displays the best domain parameter settings, the resulting model, and some statistics. The best found setting for the investigated domain parameters leads to an average model performance of 0.806 with respect to the class car IoU. Synthetic training data containing several objects in the simulated scenes are generally more useful than synthetic training data containing only cars. Setting the virtual LiDAR to a spiral trajectory is more appropriate than using a grid trajectory. The best setting of the Angular Loss parameter depends on the setting of the other domain parameters. Setting Objects in Environment to True will cause models to perform more differently than setting the parameter to False. The domain parameters under investigation have an impact on the performance of a model of up to 53% with respect to the class car IoU. Figure 6 shows the prediction results on a test scene of the Hessigheim3D dataset for two models trained on synthetic data generated with different domain parameter settings. Figure 6 highlights the need for a system that automatically finds suitable synthetic domain parameter settings from a given search space. It can be clearly seen that the selection of appropriate domain parameter settings for synthetic data generation is critical for achieving good model performance. Because of performance scatter among models trained with the same domain parameter settings, the optimizer can only relate domain parameter settings to a performance range. The range is determined by the Gaussian distribution of the performances of models trained with the same domain parameter settings. Figure 7 shows the mapping of domain parameter settings to a performance range. If domain parameter settings are mapped to overlapping performance ranges this can \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Stage** & **Objects in** & **Angular** & **Camera** & **Mean** & **Confidence** \\  & **Environment** & **Loss** & **Trajectory** & **Car-IoU** & **IoU (_p = 0.95_)** \\ \hline
1 & False & off & grid & 0.528 & \(\pm\)0.005 \\
2 & False & off & spiral & 0.547 & \(\pm\)0.005 \\
3 & False & on & grid & 0.530 & \(\pm\)0.003 \\
4 & False & on & spiral & 0.548 & \(\pm\)0.005 \\
5 & True & off & grid & 0.727 & \(\pm\)0.028 \\
6 & True & off & spiral & 0.770 & \(\pm\)0.020 \\
7 & True & on & grid & 0.721 & \(\pm\)0.021 \\
8 & True & on & spiral & 0.806 & \(\pm\)0.026 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Stages of the Auto-ML system  
Figure 6: Prediction of models trained with different synthetic training data on a Hessigheim3D test scene. The incorrectly labeled roof part in the stage 4 prediction leads to a poor recognition result. (**a**) Model trained with synthetic data generated with setting from stage 4. Car-IoU: 0.542; (**b**) Model trained with synthetic data generated with setting from stage 8. Car-IoU: 0.936.  
potentially affect the traceability of parameter settings that are suitable for data generation. The best solution found may then no longer be described by a single combination of parameter settings, but rather by a set of appropriate parameter settings. Table 3 provides a summary of the results from the second experiment, which compares the performance of the best model trained with synthetic data to that of the best model trained with real data. Both models underwent training with augmentation techniques, including scaling, rotation, added noise, and color variations. For each test scene, the model with the better class car IoU prediction is highlighted in green. In the conducted test the synthetically trained model performed 9.1% better than the model trained with data from the target domain.