# RoboTwin Metaverse Platform for Robotic Random Bin Picking
## 3 The RoboTwin System and FOVision
### Physics engines:
#### Modeling and Auto AI Annotation Tool:  
It is crucial to accurately replicate the real-world environment within the simulator, particularly when it comes to the workpiece in the metaverse robotic random bin-picking system. The fidelity of the workpiece's geometry and texture directly impacts the realism of the virtual scene and can influence the gap error. For instance, if a simulator generates images without considering real-world characteristics, training an AI model with those images may result in overfitting unrealistic features and lead to failure when applied to real images. To address this challenge, we employed an RGBD camera to capture the workpiece's real-world texture and simultaneously obtained the 3D geometry point cloud for generating the workpiece's mesh. This allowed us to achieve proper UV mapping of the texture [15]. UV mapping enables the determination of the relationship between the 2D image pixel (\(u\), \(v\)) and 3D coordinates (\(x\), \(y\), \(z\)) of the mesh using the following geometric equations:  
\[u=1/2+\tan^{-1}(2(x,z))/\ 2\pi\text{ and} \tag{5}\]\[v = 1/2+\sin^{-1}(y)/\ \pi, \tag{6}\]  
where \(u\) represents the horizontal coordinate, and \(v\) represents the vertical coordinate. The resulting values of \(u\) and \(v\) fall within the range of [0, 1] and are used to determine the texture mapping for the corresponding 3D point. By combining real-world scanned point clouds with 3D-model-generated point clouds, The accuracy of the simulated workpiece as a representation of its real-world counterpart can be assured. This approach enables us to train the AI model using realistic data, thereby reducing the gap error and improving the performance of the robotic picking system. The auto AI annotation tool proposed in this research performs the following three main tasks: (1) scene generation, (2) image capturing, and (3) auto annotations. For scene generation, the tool utilizes a domain randomization algorithm to control several parameters to randomize the reaction of the camera, the lighting, the crates, and the workpieces. This process introduces variations in the scene setup, enabling a more comprehensive training dataset. Once the scene is generated, the tool captures high-quality images of the scene with the workpieces positioned within it. These images, along with the details extracted from the simulator, serve as the input for subsequent annotation processes. The tool automates the annotation process by accurately labeling the workpieces within the captured images based on the specified labeling conditions. It performs instance segmentation and identifies and categorizes objects of interest within the image using details that can only be accessed from the simulator. For this tool, a simulator with a physics engine and domain randomization was implemented, the latter of which can be defined using the following function:  
\[D(P,\ Q,\ CP,\ LP,\ LI,\ LC,\ CR), \tag{7}\]  
where each of the parameters and functions is defined as follows:  
* Domain randomization (\(D\)): A function that takes multiple randomization parameters and initiates the scene randomly based on these values. * Object posture (\(P\)): Random postures (\(x\), \(y\), \(z\), \(r_{x}\), \(r_{y}\), \(r_{z}\)) for the workpieces so each of them has a distinct posture and surface facing the camera while being randomly stacked inside a crate. Here, \(r_{x}\), \(r_{y}\), and \(r_{z}\) represent the orientation or rotation angles with respect to the \(x\)-, \(y\)-, and \(z\)-axis, respectively. * Quantity (\(Q\)): Random quantities of workpieces for each simulation scene. Depending on the quantities of workpieces, scenarios such as stacking and occlusion can have a higher probability of occurrence. * Camera posture (\(CP\)): Random position for the camera to enhance or reduce workpieces in the image. * Lighting posture (\(LP\)): Random posture for the lighting for each simulation scene so the light reflections are different.