# Synthetic Data Generation on Dynamic Industrial Environment for Object Detection, Tracking, and Segmentation CNNs

Danilo G. Schneider, Marcelo R. Stemmer

► To cite this version:

Danilo G. Schneider, Marcelo R. Stemmer. Synthetic Data Generation on Dynamic Industrial Environment for Object Detection, Tracking, and Segmentation CNNs. 14th Doctoral Conference on Computing, Electrical and Industrial Systems (DoCEIS), Jul 2023, Monte Da Caparica, Portugal. pp.135-146, 10.1007/978-3-031-36007-7_10. hal-05473061

HAL Id: hal-05473061

https://inria.hal.science/hal-05473061v1

Submitted on 23 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_3.png" /></figure>

# Synthetic Data Generation on Dynamic Industrial Environment for Object Detection, Tracking, and Segmentation CNNs

Danilo G. Schneider(✉) and Marcelo R. Stemmer

Departamento de Automação e Sistemas, Univesidade Federal de Santa Catarina, R. Delfino Conti, S/N, Florianópolis 88040-900, Brazil

danilo_gsch@hotmail.com, marcelo.stemmer@ufsc.br

**Abstract.** The ability of Convolutional Neural Networks (CNNs) to learn from vast amounts of data and improve accuracy over time makes them an attractive solution for many industrial problems. In the context of Future Assembly Systems such as Line-Less Mobile Assembly Systems, CNNs can be used to monitor the networked system of mobile robots, human operators, and other movable objects that assemble products in flexible environment configurations. This paper explores the use of a simulated industrial environment to autonomously generate training data for object detection, tracking, and segmentation CNNs. The goal is to adapt state-of-the-art CNN solutions to specific industry use cases, where real data annotation can be time-consuming and expensive. The developed algorithm efficiently generates new random image data, allowing accurate object detection, tracking, and segmentation in dynamic industrial scenarios. The results show the effectiveness of this approach in improving the testing of CNNs for industrial applications.

**Keywords:** Synthetic Data Generation · Convolutional Neural Networks · Object Tracking · Segmentation · Dynamic Industrial Environments

## 1 Introduction

Convolutional Neural Networks (CNNs) have proven to be a promising solution for various industrial and robotic problems [1–7]. However, training CNNs requires a large and diverse dataset of labeled images, which can be time-consuming and expensive to annotate manually. To address this issue, simulation-based data generation has emerged as a promising solution, where synthetic data can be used to train CNNs for object detection, tracking, and segmentation tasks [2–4].

In future assembly systems, flexibility is increasingly desired. Future factories may be organically configured by spatio-temporal workstations as in the Line-less Mobile Assembly System (LMAS) [8]. Although the high complexity of such system, it is a closed-world environment domain, meaning that all movable instances are known in the production process. In such dynamic environments, a surveillance camera could aid in
the monitoring of the networked system of mobile robots, human operators, and other movable objects that assemble products in flexible environment configurations.

In this paper, we investigate the following research question: Is it possible to use only synthetic labeled data to train object tracking and segmentation CNNs in a flexible industrial environment and surveillance camera domain? Our hypothesis is that adapting available robotic simulators and state-of-the-art algorithms for navigation and robotic arm manipulation, it is possible to generate large datasets on the source simulation domain similar to the real-world target domain without the real data labeling costs.

Our approach aims to adapt state-of-the-art CNN solutions to a specific industry use case, where real data annotation can be a bottleneck. The innovation achieved is the creation of an algorithm that uses a 2D Mask grid map to spawn and control the moving instances inside the camera field-of-view, assuring the long-term balance of the dataset classes. The developed algorithm efficiently generates new random image data, allowing accurate object detection and instance segmentation in dynamic industrial scenarios. Specifically, we use the generated dataset to train two networks: Fast RCNN for object detection and Mask RCNN for instance segmentation. We evaluate the performance of the trained models on a validation dataset and test the models on unlabeled images, demonstrating the effectiveness of our approach in generating synthetic labeled data in this domain.

## 2 Relation to Connected Cyber-Physical Spaces

The subject of this research is “Cooperative SLAM in dynamic industrial environments”, which we divided in three steps: First, we use a CNN to infer the segmentation of all movable instances in an image. Second, we use the segmented image to transform a depth image in an 2D grid map so the robots can use it to plan paths and navigate. Third, we deal with the data association problem on merging maps from multiple input sources. This paper covers the first step of the PhD research and is directly related to the conference theme of Technological Innovation for Connected Cyber Physical Spaces.

The development of connected cyber-physical spaces has enabled the integration of various machines, robots, and humans to work together seamlessly, increasing the efficiency and productivity of industrial processes. However, this integration requires advanced monitoring and control systems that can handle the complexity of these interconnected systems [9].

Convolutional Neural Networks (CNNs) have emerged as a powerful tool for monitoring and controlling these systems, leveraging vast amounts of data to improve their accuracy over time [4, 5]. Our proposed algorithm provides a cost-effective solution for training CNNs and overcoming the challenges associated with real data annotation. The ability to autonomously generate new random image data that accurately represents dynamic industrial scenarios is an advancement in this field, providing researchers and practitioners with a tool to train and test CNNs efficiently.

In the context of smart manufacturing, synthetic data can be used to improve the accuracy and efficiency of machine learning algorithms for a range of applications, including object detection, tracking, and segmentation in industrial environments [3, 4, 6]. By using synthetic data, researchers can simulate a variety of scenarios that might
be difficult or expensive to replicate in the real world, such as rare events, unusual environmental conditions, or equipment failure.

In conclusion, this paper represents a significant contribution to the conference theme of Technological Innovation for Connected Cyber Physical Spaces. The proposed simulation-based data generation approach addresses the challenges associated with monitoring and controlling the behavior of complex interconnected systems and leverages the power of CNNs to improve efficiency and productivity in future assembly systems. This research represents an important step towards the development of advanced monitoring and control systems that can handle the complexity of these interconnected systems.

## 3 Related Literature

Convolutional Neural Networks (CNNs) are currently the leading approach for object detection, but they often require lengthy training periods. In the context of rapid prototyping production, such as in Smart Manufacturing, the time required to train these networks can exceed the turnaround time required for production. In some cases, this can result in the robot being unable to pick up produced objects before the network has completed training, bringing the entire workflow to a halt. Although there are techniques to accelerate the training of neural networks, the acquisition and labeling of datasets can still add significant delays. Therefore, for workflows with smaller batch sizes, training a robot to detect objects may be infeasible due to these temporal constraints [5].

Object detection and segmentation are two related tasks in computer vision that involve identifying and locating objects within an image or video. Object detection typically involves identifying objects within an image and localizing them by drawing bounding boxes around them. In contrast, object segmentation goes further by accurately delineating the boundaries of objects, allowing for more precise and detailed information about their shape and position. Both tasks have numerous practical applications, including autonomous driving, medical imaging, robotics, and more. Recent advances in deep learning, particularly with Convolutional Neural Networks (CNNs), have led to significant improvements in the performance of object detection and segmentation models, making them a powerful tool for a wide range of computer vision applications. Over the years, techniques like Region Proposal Networks (RPNs) have improved the accuracy and efficiency of object detection models and resulting in popular models like Mask R-CNN [10], which is an instance segmentation CNN that adds a branch for predicting an object mask in parallel with the existing branch for bounding box recognition. For object detection it uses Faster R-CNN [11] as RPN which shares full-image convolutional features with the detection network.

More recently, other models have achieved state-of-the-art in popular benchmark datasets. For object detection, one state-of-the-art network example is YOLOv7 [12], the authors designed several trainable bag-of-freebies methods, so that real-time object detection can greatly improve the detection accuracy without increasing the inference cost and the results show that their model surpasses all known object detectors in both speed and accuracy in the range from 5 FPS to 160 FPS and has the highest accuracy 56.8% AP among all known real-time object detectors with 30 FPS or higher on GPU V100.
For semantic segmentation tasks, other currently relevant network is DeepLabv3+ [13], the authors combined methods of former networks that are able to encode multi-scale contextual information by probing the incoming features with filters or pooling operations at multiple rates and multiple effective fields-of-view, while the latter networks can capture sharper object boundaries by gradually recovering the spatial information.

While benchmark image datasets have been instrumental in advancing the field of computer vision, there are significant differences between these datasets and industrial applications that must be considered. Most benchmark datasets, such as CIFAR-10 [14], ImageNet [15], and COCO [16], contain a wide variety of images depicting everyday objects in static settings. In contrast, industrial applications involve complex environments with movable objects, dynamic lighting conditions, and a wide variety of robots and different scenarios that are difficult to capture in a single dataset.

Regarding the use of synthetic data in training, the authors of [6] developed an automated synthetic data generation pipeline to produce large-scale photo-realistic data with precise annotations, and then formulate an instance segmentation network for object recognition and a 6D pose estimation network for localization. On top of this pipeline, they presented a generic and robust sim-to-real deep-learning-based framework, namely S2R-Pick, for fast and accurate object recognition and localization in industrial robotic bin picking.

The emergence of the digital twin concept and the rise of robotics, driven by the last industrial revolutions and digital transformation, have created a need for updating virtual representations to facilitate appropriate decision-making by system operators. Multi-robot perception capabilities provide an opportunity to enhance object recognition and robot environmental understanding by combining individual observations. However, identifying the most informative camera poses, which play a crucial role in object recognition, is challenging due to the viewing angle dependency.

To address this, the authors of [7] propose a smart view selection approach that maximizes object recognition by determining the most relevant camera poses and the number of informative views. A clustering-based approach on a synthetic view dataset of traditional industrial objects is employed to maximize inter-class distance and minimize intra-class distance. The promising results, presented in terms of F1-score metric, demonstrate the effectiveness of this approach.

The authors of [4] discuss the utilization of Region-based Convolutional Neural Network (CNN) models trained on synthetic images generated from CAD models for part handling applications. The study outlines the development of an automated synthetic data generation process and the training of two CNN models aimed at detecting and distinguishing between similar-looking car components. The performance of these models was evaluated using real images, and the results demonstrate the potential for this approach to be applied to detect various parts in complex backgrounds. The use of synthetic images for CNN training also provides a streamlined process for generating a detector. The conclusion is that generating pictures and training a neural network was overall successful, still, for the high standards of industrial robotics rigorous testing has to be further performed.
Regarding autonomous vehicles, [17] address the problem of class-agnostic instance segmentation. Given the challenging task of providing enough labeled data to cover all possible semantic classes that can appear in an outdoor urban scenario. In the other hand, since we are dealing with a closed-world scenario in this paper, we expect all objects in the production system to be known.

For robotic operations such as bin picking or pick-and-place, explored by [2–4, 6], it is usual to have a training image with the objects positioned randomly, sometimes occluded or upside-down. In our case, as the goal is to track and segment mobile robots, humans and other movable objects, it is not interesting to have upside-down instances in the training data since it does not represent the normal operation target domain.

There are several robotic simulators available. [18] show an extended comparison between Unity3D, V-Rep and Gazebo, pointing that both V-Rep and Gazebo are easier to integrate with ROS. With respect to annotation camera sensors, we found three simulators that have it implemented: Gazebo (since version Fortress) [19], CARLA [20] and Nvidia Isaac Sim [21]. We decided on using Gazebo as it is fully open-source and it seems easier to integrate with ROS2.

# 4 Method

As the problem formulation, training CNNs lies in the need for large amounts of annotated images arising for each and every new part. In industrial environments there rarely are annotated images of the desired part available, however, construction data in the form of 3D CAD models is almost always available.

The objective of our algorithm is to use meshes information of industrial movable objects, robots and humans to generate large amounts of annotated images for object detection and instance, semantic or panoptic segmentation. While generating data, the goal is to simulate a real industrial navigation scenario and not just placing and dropping models on the environment.

## 4.1 Software Tools

Our algorithm relies on the Gazebo Simulator (version Garden) [19]. Gazebo is an open-source robotic simulation environment developed by the Open Source Robotics Foundation (OSRF). It provides a platform for researchers and developers to test and evaluate robotic systems in a virtual environment, allowing for faster and more cost-effective development. The simulator includes a physics engine, sensor simulation, and support for various robotic platforms and driving mechanisms, making it a popular choice for a wide range of applications, including robotics, computer vision, and artificial intelligence. The simulator has been used in various research studies, including the development of autonomous robots, navigation systems, and object recognition algorithms.

With the model meshes, it is possible to make a description file with visual, collision and inertial information and add different types of sensors and actuators to use the model in Gazebo. Since the previous version (Fortress), there is a bounding box camera sensor
and a segmentation camera sensor that can be used to save annotated images from the simulator, which fits perfectly our purpose.

Moreover, we used the Robot Operating System 2 (version Humble) [22] framework to autonomously navigate the robots and humans in the simulation environment. The Nav2 stack [23] was used for navigation and Moveit2 [24] to control 5 Degrees-of-Freedom (DoF) and 6 DoF robotic arms.

## 4.2 Data Generation Pipeline

A set of open source online available model meshes were adapted to develop and test the algorithm. Since we used a generic code to navigate and control different models, the models were adapted to have generic joints, transform frames and sensors with the same name. More specifically, 8 different human meshes with walking animations, 8 different object meshes, and 7 different robot meshes were used and are shown in Fig. 1 with the industrial background. For each model, a description file for Gazebo was created with optimized LiDAR sensor and differential or mecanum-wheeled driving systems for the robots. The created models were added to a table with specific parameters to be used in our algorithm. The parameters are shown in Table 1. Those parameters are used to correctly spawn and control the models, label them with their class id and store the number of times this model was already used by the algorithm.

<figure><img src="image_4.png" /><figcaption>**Fig. 1.** Every model used for the data generation. Robots on the left side, actors on the middle and movable objects on the right side.</figcaption></figure>

<table><caption>**Table 1.** Table with configured model parameters with one robot example</caption><thead><tr><td>Name</td><td>Type</td><td>Radius</td><td>Mode</td><td>has_arm</td><td>n_uses</td><td>class_id</td></tr></thead><tbody><tr><td>Pioneer3DX</td><td>robot</td><td>0.3</td><td>differential</td><td>False</td><td>0</td><td>10</td></tr></tbody></table>

A configuration file is also needed to setup the algorithm with parameters like maximum simulator iterations per run and maximum number of each model class to spawn. All launch configuration parameters are shown in Table 2.
<table><caption>**Table 2.** Launch parameters description table.</caption><thead><tr><td>Parameter</td><td>Description</td></tr></thead><tbody><tr><td>models DF path</td><td>Absolute path for models table</td></tr><tr><td>model resouce path</td><td>Absolute path for Gazebo model files</td></tr><tr><td>launch_config_path</td><td>Absolute path for data generation configuration files</td></tr><tr><td>instance seg</td><td>Loads instance segmentation camera sensor</td></tr><tr><td>semantic seg</td><td>Loads semantic segmentation camera sensor</td></tr><tr><td>bb2d vis</td><td>Loads 2D visible bounding boxes camera sensor</td></tr><tr><td>bb2d full</td><td>Loads 2D full bounding boxes camera sensor</td></tr><tr><td>bb3d</td><td>Loads 3D bounding boxes camera sensor</td></tr><tr><td>save_depth</td><td>Loads depth camera sensor</td></tr><tr><td>save path</td><td>Path for saving current generated data</td></tr><tr><td>camera pitch</td><td>Camera sensor pitch angle</td></tr><tr><td>n_max_r</td><td>Maximum number of robots to spawn in each iteration</td></tr><tr><td>n_max_a</td><td>Maximum number of actors to spawn in each iteration</td></tr><tr><td>n_max_o</td><td>Maximum number of objects to spawn in each iteration</td></tr><tr><td>sim iterations per run</td><td>Number of simulator iterations for each launcher run (1 iteration = 0.001 s)</td></tr></tbody></table>

Once models and parameters are configured, the full algorithm can be run in one command line in the terminal.

Simplified Pseudo-Algorithm

GenerateRandomCameraPose()
RestrictionAreaMask camera pose)
models_array[] = Select spawnModels(n_max_r,n_max_a,n_max_o)
For (model in models_array[]) do:
  GenerateRandomValidPose()
  Spawn(model)
EndFor
GenerateRandomPoseGoals()
GenerateRandomJointPositions()
For (i = 1 to sim iterations per run) do:
  Run()
  SaveData()
EndFor
Update(n uses)
The first function generates a random pose for the camera sensor using the fixed pitch angle defined in the configuration file. The *RestrictionAreaMask* function creates a binary mask based on the camera pose and its horizontal field of view to restrict the area which the models will spawn and navigate to be inside the camera reach, the mask is applied on top of the empty environment 2D occupancy grid map. An example restriction map generated is shown in Fig. 2. Note that the mask has a trapezoidal form that limits the minimum and maximum distance from camera to models by a defined constant, assuring that models will not be too close or too far from the camera during runtime.

<figure><img src="image_5.png" /><figcaption>**Fig. 2.** Example of Restriction Mask. Blue dot represents camera 2D position, red circle represents an invalid generated position for a model and green circle represents a valid generated position for the same model.</figcaption></figure>

Next, the algorithm randomly selects among the less used models, spawns each of them on valid positions using each model radius to assert that they will not spawn on top of each other by marking the occupied area on the restriction mask. Then random joint positions are generated to control the robotic arms and random goal poses for navigation. Finally, the simulator runs for the specified number of iterations saving data and when it is done the number of uses for each model is incremented in the table.

# 5 Experimental Results and Analysis

Using the proposed algorithm, two datasets were generated composed of 3 category high-level classes annotated (robot, person, object). The first dataset has 5025 images and was used to train a Faster R-CNN model for object detection whereas the second has 2613 images used on Mask R-CNN training.

For training, evaluating and testing we used Detectron2 [25], a next generation library that provides state-of-the-art detection and segmentation algorithms. It is the successor of Detectron and maskrcnn-benchmark. It supports a number of computer vision research projects and production applications in Facebook. Some of the available models are Mask R-CNN, Faster R-CNN and DeepLabv3+. The training step was done following the guidelines on Detectron2 Beginner's Tutorial, which does not use a validation set on training, thus it does not fine-tune the hyper-parameters.
## 5.1 Faster R-CNN

The first dataset was split into 90% for training and 10% for testing. Faster R-CNN model with ResNet-50 and Group Normalization backbone was used with pre-trained weights obtained with COCO 2017 dataset. The model was trained for 10000 iterations, the total_loss graph and an inference on a test image are shown in Figs. 3 and 4, respectively. Table 3 shows the evaluation using the test dataset.

<figure><img src="image_6.png" /><figcaption>Fig. 3. Graphic of total loss x iterations on Faster R-CNN training.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 4. Inference of Faster R-CNN on testing image.</figcaption></figure>

<table><caption>**Table 3.** Table with COCO evaluation metrics using the test dataset.</caption><thead><tr><td>AP</td><td>AP50</td><td>AP75</td><td>APS</td><td>APm</td><td>API</td></tr></thead><tbody><tr><td>0.521</td><td>0.735</td><td>0.529</td><td>2.649</td><td>0.628</td><td>0.526</td></tr></tbody></table>

## 5.2 Mask R-CNN

The second dataset was also split into 90% for training and 10% for testing. Previous Faster R-CNN model was used as a backbone with pre-trained weights obtained with COCO 2017 dataset. The model was trained for 10000 iterations, the total loss graph is shown in Fig. 5.

Inference on a test image and real image are shown in Figs. 6 and 7, respectively.
<figure><img src="image_8.png" /><figcaption>**Fig. 5.** Graphic of total loss x iterations on MASK R-CNN training.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>**Fig. 6.** Inference of Mask R-CNN on test image.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>**Fig. 7.** Inference of Mask R-CNN on real test image.</figcaption></figure>

## 5.3 Analysis

The total loss curve flattened in the end of the training, which means that the model has converged and is no longer improving significantly. Although a validation set was not used in the training, the preliminary results indicate the usability of the generated data. A validation dataset is recommended to evaluate the performance of the model during the training process, and to tune the hyper-parameters of the model, such as learning rate, batch size, and the number of epochs. The validation dataset is also used to prevent overfitting, which occurs when the model becomes too complex and starts to memorize the training data instead of generalizing to new data.

The detection evaluation scores on COCO evaluation metric indicate a better performance of the model when the IoU threshold is 0.5.

The inference on test data for both detection and segmentation task show a good prediction for our defined classes, so we are able now to detect and segment instances of robots, persons and objects inside the simulation environment.
The inference on the real image is promising, it detected a robot, human and object although the segmentation does not cover the objects entirely and we can note some false negatives. More similar and realistic instance models or domain adaptations could improve the prediction in the real test image.

# 6 Conclusion

In conclusion, this paper explores the use of a simulated industrial environment to autonomously generate training data for training CNNs, with the goal of adapting state-of-the-art CNN solutions to specific industry use cases where real data annotation can be time-consuming and expensive. The developed algorithm efficiently generates new random image data, allowing for accurate object detection, tracking, and segmentation in the simulated industrial scenario, but its domain is not similar enough to the real-world application yet. The results of the study demonstrate the effectiveness of this approach in improving the testing of CNNs for industrial applications and suggest that it has the potential to be a valuable tool for enhancing the performance and applicability of CNN-based solutions in the context of future assembly systems. Future research may seek to further refine the algorithm and extend its applicability to other domains. Overall, this study underscores the promise of simulated environments as a means of generating high-quality training data for machine learning applications in industrial settings.

The developed algorithm is available at [26], alongside documentation on usage and new model insertion as well as jupyter notebooks for data registration and training with Detectron2.

**Acknowledgments.** We would like to acknowledge CAPES (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior) for their financial support.

# References

1. Rai, R., Tiwari, M. K., Ivanov, D., Dolgui, A.: Machine learning in manufacturing and industry 4.0 applications. Int. J. Prod. Res. 59(16), 4773-4778 (2021). https://doi.org/10.1080/00207543.2021.1956675

2. Danielczuk, M., et al.: Segmenting unknown 3D objects from real depth images using mask R-CNN trained on synthetic data. In: International Conference on Robotics and Automation (ICRA), pp. 7283-7290. IEE/Canada (2019). https://doi.org/10.1109/ICRA.2019.8793744

3. Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., Abbeel, P.: Domain randomization for transferring deep neural networks from simulation to the real world. In: International Conference on Intelligent Robots and Systems - IROS, pp. 23-30. IEEE/RSJ (2017). https://doi.org/10.1109/IROS.2017.8202133

4. Andulkar, M., Hodapp, J., Reichling, T., Reichenbach, M., Berger, U.: Training CNNs from synthetic data for part handling in industrial environments. In: 14th International Conference on Automation Science and Engineering (CASE), Germany. IEEE (2018). https://doi.org/10.1109/COASE.2018.8560470

5. Schnieders, B., Luo, S., Palmer, G., Tuyls, K.: Fully Convolutional One-Shot Object Segmentation for Industrial Robotics. arXiv preprint arXiv:1903.00683 (2019). https://doi.org/10.48550/arXiv.1903.00683

6. Li, X., et al.: A sim-to-real object recognition and localization framework for industrial robotic bin picking. IEEE Robot. Autom. Lett. 7(2), 3961-3968 (2022). https://doi.org/10.1109/LRA.2022.3149026
7. Xu, K., Ragot, N., Dupuis, Y.: View selection for industrial object recognition. In: 48th Annual Conference of the IEEE Industrial Electronics Society – IECON, Belgium, pp. 1–6. IEEE (2022). 1109/IECON49645.2022.9969121

8. Buckhorst, A.F., do Canto, M.K.B., Schmitt, R.H.: The line-less mobile assembly system simultaneous scheduling and location problem. Procedia CIRP **106**, 203–208 (2022). https://doi.org/10.1016/j.procir.2022.02.179

9. Villalonga, A., et al.: A decision-making framework for dynamic scheduling of cyber-physical production systems based on digital twins. Annu. Rev. Control. **51**, 357–373 (2021). https://doi.org/10.1016/j.arcontrol.2021.04.008

10. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision – ICCV, pp. 2961–2969 (2017). https://doi.org/10.48550/arXiv.1703.06870

11. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems – NIPS, vol. 28 (2015). ISBN 9781510825024

12. Wang, C.Y., Bochkovskiy, A., Liao, H.Y.M.: YOLOv7: Trainable Bag-of-Freebies Sets New State-Of-The-Art for Real-Time Object Detectors. arXiv preprint arXiv:2207.02696 (2022). https://doi.org/10.48550/arXiv.2207.02696

13. Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 833–851. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_49

14. The CIFAR-10 dataset. https://www.cs.toronto.edu/~kriz/cifar.html. Accessed 30 Apr 2023

15. ImageNET. https://www.image-net.org/. Accessed 30 Apr 2023

16. COCO Dataset. https://cocodataset.org/#home. Accessed 30 Apr 2023

17. Nunes, L., et al.: Unsupervised class-agnostic instance segmentation of 3D LiDAR data for autonomous vehicles. IEEE Robot. Autom. Lett. **7**(4), 8713–8720 (2022). https://doi.org/10.1109/LRA.2022.3187872

18. De Melo, M.S.P., da Silva Neto, J.G., Da Silva, P.J.L., Teixeira, J.M.X.N., Teichrieb, V.: Analysis and comparison of robotics 3D simulators. In: 21st Symposium on Virtual and Augmented Reality – SVR, pp. 242–251, Brazil. IEEE (2019). https://doi.org/10.1109/SVR.2019.00049

19. Gazebo Simulator. https://gazebosim.org/home. Accessed 30 Apr 2023

20. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: an open urban driving simulator. In: Conference on Robot Learning – PMLR, pp. 1–16. (2017). https://proceedings.mlr.press/v78/dosovitskiy17a.html. Accessed 30 Apr 2023

21. Nvidia Isaac Sim. https://developer.nvidia.com/isaac-sim. Accessed 30 Apr 2023

22. Macenski, S., Foote, T., Gerkey, B., Lalancette, C., Woodall, W.: Robot operating system 2: design, architecture, and uses in the wild. Sci. Robot. **7**(66), eabm6074 (2022). https://www.science.org/doi/abs/10.1126/scirobotics.abm6074

23. Macenski, S., Martín, F., White, R., Ginés Clavero, J.: The marathon 2: a navigation system. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2020). https://github.com/ros-planning/navigation2, https://arxiv.org/abs/2003.00368

24. Moveit2 package. https://moveit.ros.org/. Accessed 30 Apr 2023

25. Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2 (2019). https://github.com/facebookresearch/detectron2

26. Data generation on Ignition Gazebo (Garden) for Segmentation and Object Tracking CNN's. https://github.com/danilogsch/Coop-SLAM. Accessed 30 Apr 2023