## 5 Simulation-to-Reality Transfer
\begin{table}
\begin{tabular}{c c l} \hline \hline
**Simulator** & & **Example** \\ \hline \multirow{7}{*}{Gazebo} & \(\bullet\) & Robotic Grasping using Deep Reinforcement Learning [207] \\  & \(\bullet\) & End-To-End Autonomous Exploration for Mobile Robots in Unknown Environments through Deep \\  & \(\bullet\) & Reinforcement Learning [208] \\  & \(\bullet\) & An Efficient Deep Reinforcement Learning Framework for UAVs [209] \\  & \(\bullet\) & Path Planning of Mobile Robot Using Reinforcement Learning [210] \\ \hline \multirow{7}{*}{Unity3D} & \(\bullet\) & Goal-Oriented Obstacle Avoidance by Two-Wheeled Self Balancing Robot [211] \\  & \(\bullet\) & Klcker: An Industrial Drive and Control Foosball System automated with Deep Reinforcement Learning [212] \\ \cline{1-1}  & \(\bullet\) & Research on Autonomous Navigation Control of Unmanned Ship Based on Unity3D [213] \\ \cline{1-1}  & \(\bullet\) & Crowd Navigation in an Unknown and Dynamic Environment Based on Deep Reinforcement Learning [214] \\ \cline{1-1}  & \(\bullet\) & Research on robot arm control based on Unity3D machine learning [215] \\ \hline \multirow{3}{*}{PyBullet} & \(\bullet\) & Deep Reinforcement Learning Based Trajectory Planning Under Uncertain Constraints [216] \\ \cline{1-1}  & \(\bullet\) & Robotic Lever Manipulation using Hindsight Experience Replay and Shapley Additive Explanations [217] \\ \cline{1-1}  & \(\bullet\) & Robust Quadruped Jumping via Deep Reinforcement Learning [218] \\ \hline \multirow{3}{*}{MuJoCo} & \(\bullet\) & Learning Continuous Control Actions for Robotic Grasping with Reinforcement Learning [219] \\ \cline{1-1}  & \(\bullet\) & MANGA: Method Agnostic Neural-policy Generalization and Adaptation [220] \\ \cline{1-1}  & \(\bullet\) & Diversity-Driven Exploration Strategy for Deep Reinforcement Learning [221] \\ \hline \hline \end{tabular}
\end{table} 
Table 1: DRL applications trained in different simulators. techniques have been found to be used in DRL applications in manufacturing, of which domain randomisation is the most extended. These techniques are briefly described below, along with a list of applications in which they are used both solely and in combination (see Table 2). * System identification. This is a wide field that involves creating a precise mathematical representation of the actual world in order to increase the realism of the simulation [222]. To this end, the observed data (inputs and outputs) and the knowledge of the process dynamics are used to build the model [223]. This approach has certain drawbacks, which are exacerbated when applied to industrial processes, such as the need for parameter calibration, data gathering and modelling of how external influences affect the operation of the robot (e.g., the wear-and-tear of its joints) [224]. Nonetheless, as observed in the aforementioned applications [69; 158; 159], the development of digital twin models is trending in the building of accurate mathematical models, becoming a fundamental pillar in smart manufacturing [225]. * Domain randomisation. To accommodate a wider variety of environmental setups and potential scenarios, simulation parameters are randomly generated [226]. This includes two groups of techniques that bridge the gap between the actual world and the virtual one. On the one hand, visual randomisation addresses the inclusion of randomness in the observation, such as changing the camera location, artificial light, and textures [227]. On the other hand, dynamics randomisation entails changing the simulator's physical settings, such as item sizes, friction factors, or joint motor characteristics [228]. Further, the variation of these parameters is not simply random, and its distribution affects the performance of the algorithm [229]. Indeed, many variants are proposed to obtain more robust performance, such as Active Domain Randomisation [230] and Neural Posterior Domain Randomisation (NPDR) [231]. * Domain adaptation. It aims to harmonise the spaces in order to bridge the gap between the simulated and actual environments.