# RoboTwin Metaverse Platform for Robotic Random Bin Picking
## 2 Background  
According to the literature, researchers have shown that utilizing virtual scenes can be a replacement or comparison source to their real-world counterparts. As addressed by [PERSON] et al. in [1], a lack of resources and accessibility to real-world scenes can be an issue in training robots in the digital twin. For instance, in [1], the visualization of a virtual robot that is controlled based on data captured from cameras in the physical environment was introduced to allow for the integration of knowledge and remote practice without the need for a physical robot, thereby increasing equity and inclusivity in assessment. [PERSON] et al. in [2] argued that advanced manufacturers often require dynamic changes on the factory floor to enable manufacturing. To address these requirements, they employed the ant colony optimization algorithm to program an industrial robot for obstacle avoidance and pathfinding for object picking and placement. In their work, the optimization was completed in a digital-twin environment, and after human inspection, the movements are transferred to a real robot, with all trials completed at a minimal cost. [PERSON] et al. in [3] posited that although digital-twin systems consider the manufacturing process, they often ignore the gaps and discrepancies between the simulated and real-world cases. To cope with the problems, they proposed a smart manufacturing control model that ensures cost optimization and reality awareness of resource usage and suggested that this solution could enhance a digital-twin system by connecting it to the cloud and a management execution system (MES). To summarize, the authors of [1; 2; 3] have provided valuable strategies and contributions for using virtual systems to support the digital-twin platform. However, the reported methods are not sufficiently robust to account for factory environments, primarily due to the following reasons:* The level of realism in the virtual scene plays a crucial role in its corresponding generated value. If the simulated virtual scene is not realistic enough, the performance gap can be significant. In contrast, achieving a higher level of realism is more important than designing the platform itself, as the latter is readily available in the market. * The movement of equipment is also an important factor to consider. Identifying and resolving issues in the virtual simulated scene before deployment in the real environment can greatly reduce deployment, tuning, and debugging time, as well as protect the equipment. Research on using vision-guided robots in engineering applications such as surgical operations [4], where surgeons are assisted with automated surgical subtasks, has reported several challenges due to backlash, hysteresis, and variable tensioning in cable-driven robots. To address these issues, the authors in [4] proposed to employ visual feedback, referred to as deep intermittent visual servo (IVS), to estimate the robot's next-step action instead of using information from the robot's encoders. The results showed that IVS could achieve the highest published success rates for automated surgical peg transfer and was significantly more reliable than previous techniques when instruments were changed. [PERSON] et al. in [5] stated that operating a robot in a robotic bin-picking system's environment, where workpieces are randomly stacked in the crate, could be a challenging task for the robot because the posture of each workpiece is unknown and therefore could not be preprogrammed. To solve this issue, 3D cameras were used in their research to obtain the corresponding point cloud datasets to match with the CAD model to identify the posture. However, this solution was only useful for educational settings, not for real-world implementations in factories. In industrial settings, the role of operators involves the manufacturing of workpieces rather than their design. Consequently, access to the workpiece's CAD model is often unavailable to these operators.