# Seamless Human-Robot Collaborative Assembly Using Artificial Intelligence and Wearable Devices
## 3 Implementation  
### Actions Perception Module (APM)  
APM acts as the main means of capturing and interpreting the scene where the robot and operator are engaged in collaboration. It uses a convolutional neural network to detect the existence/position of parts to be assembled as well as the hands of the operator. The TensorFlow machine learning framework [18] was used, supplemented by a custom object detector, allowing to detect the parts of interest as well as the operator's hands. The real-time detection requirement was met with the use of a single shot detector (SSD) which provides satisfactory accuracy. To increase the detection accuracy of the NN, training with multiple datasets were tested:  
1. Real-life photos: short videos of objects of interest in diverse lighting conditions and background were shot. The single frames were extracted, followed by a manual annotation procedure. 2. CAD-based photos: CAD files of the objects were imported in Unity3D, textures were applied, and a script was used to automatically vary the background/viewing angles/lighting conditions of the virtual scene, capture and label photos. 3. Synthetic dataset: a combination of real-life photos and CAD-based photos. Figure 2: Flowchart of the seamless HR collaborative assembly paradigm. Augmentation of datasets using HSV filters and image reduction to 300 \(\times\) 300 pixels were applied to increase the training and execution speeds. The NN trained with the synthetic data demonstrated increased accuracy when compared to the use of real-life photos (12% increase) or CAD photos (26% increase). The overall methodology is summarized in Figure 3.  
For the APM detection algorithm to work, an input in the form of RGB photos should be provided. To avoid the installation of multiple stationary RGB cameras to achieve high coverage of the shopfloor, the necessary video stream is extracted from the built-in cameras of the AR headset that the operator is already using as a supporting device (Microsoft HoloLens 2 [27]). This allows the operator to freely move from one station to another and ensure that his/her actions are constantly tracked. The video feed of the headset is wirelessly transmitted to the APM controller allowing the utilization of low spec mobile hardware to be used, while the resource-intensive tasks are performed at a remote machine with higher processing power. To achieve low latency in HoloLens live video stream, the Mixed Reality Companion Kit library [28] was used. As soon as a part is detected, the application draws a boundary box around it, indicating its position within the frame. To identify the parts being manipulated, the percentage of overlap between the boundary boxes of the operator's hands and the objects of interest as well as the amount of time that the overlap occurs is calculated (Figure 3). A number of experiments manipulating objects of various sizes were done, allowing to set various overlap percentages and time thresholds based on the size of parts/boundary boxes size. This information is cross-checked against the task list of the specific assembly scenario, to identify the active task (including its completion status) as well as the intention of the operator on which task to carry out next. The position of the operator on the shopfloor is constantly monitored and used to supplement the scene analysis. The built-in IMU sensor of HoloLens is utilized for this reason. Prior to the beginning of the assembly scenario, the operator puts on the HoloLens headset, moves to a specific location, and initializes/calibrates his/her position inside the shopfloor. In such a way, the initial position of the operator is the same as the initial position of the virtual world (SDR). To avoid the drift phenomenon that usually occurs while calculating position using data from IMU sensors, an on-the-fly recalibration of the position of the operator is implemented. Several markers are placed in the shopfloor and at well-known predefined positions.