# CarFree: Hassle-Free Object Detection Dataset Generation Using Carla Autonomous Driving Simulator
## 6 Experiments
### Evaluation
Based on this practice, besides our synthesized images, we also use the KITTI [30] dataset and the BDD100K [33] dataset, both of which contain real-world driving images. We use 500 randomly chosen KITTI images as the baseline training images. Then we add more and more images to evaluate how the added images perform to enhance the object detection accuracy. The images from the BDD100K dataset are used for evaluation purposes. The roles of these two real-world datasets were decided after carefully comparing the two datasets. The KITTI dataset only contains typical driving images under good weather conditions, whereas the BDD100K dataset provides more diverse images with various weather conditions. Thus, we concluded that the BDD100K images are better suited as the evaluation dataset because they better reflect the real world's diversity. Then our primary focus is to evaluate how our synthesized images can reinforce the baseline training images. For the object detection model, we used the YOLO v3 DNN based on the Darknet framework. The training begins with the 500 baseline images from the KITTI dataset. Then we use the following three training strategies and evaluate how well they reinforce the training:  
* **KITTI only:** In addition to the baseline images, other random KITTI images are incrementally added as well. * **KITTI+CarFree (random):** In addition to the baseline images, our synthesized images are incrementally added in random. * **KITTI+CarFree (\(\geq\)4 objects):** In addition to the baseline images, our synthesized images with at least four objects are incrementally added. The difference between **KITTI+CarFree (random)** and **KITTI+CarFree (\(\geq\)4 objects)** is that the latter is likely to provide more complexity and diversity, which are expected to have positive effects on the training performance. Figure 6(a) compares the object detection accuracy, i.e., mAP (mean average precision), of the above three training strategies with varying number of training images. The \(x\)-axis begins with 500, indicating the number of baseline images. The mAP is measured by the evaluation set with 200 random BDD100K images. As shown in the figure, as we add more and more training images, the performance generally gets better. Moreover, it shows that **KITTI + CarFree (random)** outperforms **KITTI only**, and **KITTI + CarFree (\(\geq\)4 objects)** performs even better. More specifically, with 2000 training images, **KITTI only**'s mAP is 74.51, whereas **KITTI + CarFree (random)** reaches 77.06 and **KITTI + CarFree (\(\geq\)4 objects)** reaches 79.08. The results indicate that the diversity, which is often missing in many real-world datasets, of our synthesized images significantly reinforce the training process. Figure 6(b) compares the three training strategies with another compilation of BDD100K images, which is composed of carefully chosen 200 rare images with unusual weather and lighting conditions such as rainy days, night driving, and driving in the sunset. This experiment is for estimating how the object detectors will perform in harsh driving environments. In the figure, note that with the same baseline training images, the object detection accuracy significantly deteriorates, from 62.55 to 37.77, in the harsh driving scenarios. The figure also shows that adding just a small number of synthesized images significantly improves the performance, whereas adding KITTI images hardly increases the object detection accuracy. The gap between **KITTI only** and **KITTI + CarFree (random)** is much more obvious than the results in Figure 6(a). At the 1500 training images, **KITTI only**'s mAP is just 50.7, while **KITTI + CarFree (random)** reaches 73.65 and **KITTI + CarFree (\(\geq\)4 objects)** reaches 77.51. The results indicate that when the detection model is trained only with monotonous real-world images like KITTI, it does not perform well in harsh real-world driving scenarios. In such cases, adding CarFree's synthesized images with various weather conditions significantly improves the performance.