## 5 Evaluation of Results
### Comparison between Train on Real Test on Real (TRTR) and Train on Synthetic Test on Real (TSTR)  
To evaluate the utility of the synthetic dataset, researchers [38; 39; 40; 41] trained two ML models using the real and synthetic datasets and subsequently tested the trained models with a reserved subset of the actual ballast dataset that was not utilized during training. Various performance metrics, including accuracy, precision, and F1-Score, were employed to examine the variations in model performance [42]. Metrics for models trained on the real dataset were labeled as Train on Real Test on Real (TSTR), while those for models trained on the synthetic dataset were labeled as Train on Synthetic Test on Real (TSTR). If the TSTR metrics closely align with the TSTR metrics, the synthetic ballast dataset is considered effective for training models applicable to real-world scenarios.  
Figure 8a provides a detailed illustration of the configuration of the experiment and its execution flow. Two distinct training datasets of ballast images were created to independently train a ballast segmentation model (Cascade Mask R-CNN with a Swin- Transformer as the backbone [7]) using identical network configurations. The first dataset included a subset of collected real ballast images, as described in Section 3, comprising 325 images with a total of 50,461 ballast particle instances. The second dataset consisted of a subset of synthetically generated ballast images, detailed in Section 4, containing 325 images with a total of 128,139 particles. After 36 training epochs, both segmentation models were tested using the same ballast dataset, which was randomly selected from the real-image dataset and not included in the training sets. This reserved real dataset comprised 22 images with 3505 ballast particles and was used as the testing dataset to assess and compare the performance of the two models after training.  
Additionally, among all experiments conducted, a series of traditional dataset augmentation techniques, including random flipping, random mirroring, random resizing to multiple preset aspect ratios, and random cropping, were applied to each input image while training all ballast particle segmentation models to be evaluated in this study.  
Figure 8: Comparison between TRTR and TSTR. **(a)** Experimental flowchart. **(b)** Mean average precision (\(mAP\)) comparison. **(c)** F1-Score for \(mAP\) and mean average recall (\(mAR\)) comparison [23].  
The metrics employed to evaluate the performances of TRTR and TSTR include their mean average precision (\(mAP\), Figure 8b), mean average recall (\(mAR\)), and F1-Score (Figure 8c), which are standard measures in the evaluation of image segmentation models [12; 36]. The results demonstrate that although the model trained on the real dataset (\(mAP_{max}\) = 32.1%, \(F1_{max}\) = 34.3%) exhibits superior performance compared to the model trained on the synthetic dataset (\(mAP_{max}\) = 21.2%, \(F1_{max}\) = 23.9%), the metrics remain sufficiently comparable. This suggests that synthetic images can be utilized to effectively train models for ballast particle segmentation, notwithstanding the performance gap between real and synthetic data.