# Human-Robot Collaboration: An Augmented Reality Toolkit for Bi-Directional Interaction
## 5 Augmented Reality Features
### Calibration  
The pose of the cobot virtual model in AR environment is assigned through a calibration procedure. More in detail, the calibration is needed to align the virtual features with the real environment. It consists of transferring a fixed frame, whose relative position and orientation with respect to the robot base frame is known, from the real to the augmented environment. A dedicated algorithm has been developed to evaluate the calibration accuracy. It is based on a quantitative evaluation of the position and orientation errors of the virtual hologram with respect to its real twin. In Figure 4, two snapshots showing the two steps of the calibration procedure are reported: the robot hologram collocation phase (Figure 4a) and the subsequent validation phase (Figure 4b). The calibration procedure requires the positioning of two QR codes in the user's field of view in a known position and orientation with respect to the real robot base frame \(\mathcal{F}_{r}\). The first QR code allows the user to collocate the cobot hologram in the AR environment, while the second one permits the user to numerically evaluate the procedure success and compute the calibration errors in terms of position and orientation. Denote with \(\mathcal{F}_{q}=\{O_{q},x_{q}y_{q}z_{q}\}\) and \(\mathcal{F}_{p}=\{O_{p},x_{p}y_{p}z_{p}\}\) the coordinate frames attached to the top left corner of the two QR codes, respectively. Based on the previous assumption, the relative positions (\(\mathbf{p}_{q}^{r}\) and \(\mathbf{p}_{p}^{r}\)) and orientations (expressed in terms of rotation matrices [31] as \(\mathbf{R}_{q}^{r}\) and \(\mathbf{R}_{p}^{r}\)) between the QR codes' frames and the robot base frame are constant and known in the real world. The superscript \(r\) indicates that all the quantities are expressed with respect to the robot base frame. When the two QR codes are in the user's field of view, she/he presses the \"_Calibrate_\" button, shown in Figure 3, thus HoloLens begins to search for QR codes exploring the surrounding environment (Figure 4a) by exploiting the Unity QR code tracking algorithm [29]. If the search is successful, the QR tracking algorithm returns an estimation of the positions, \(\hat{\mathbf{p}}_{q}\) and \(\hat{\mathbf{p}}_{p}\), and the orientation matrices \(\hat{\mathbf{R}}_{q}\) and \(\hat{\mathbf{R}}_{v}\), of the QR frames, \(\hat{\mathcal{F}}_{q}\) and \(\hat{\mathcal{F}}_{v}\), in the HoloLens coordinate frame \(\mathcal{F}_{0}\) defined by default at each start of the application (the superscript \(0\) is omitted for notation compactness). Then, the application notifies the operator with an acoustic signal and the robot hologram can be collocated in the AR environment. Figure 3: Simulation scene control panel. Thus, the robot hologram is firstly positioned with its coordinate frame coincident with \(\hat{\mathcal{F}}_{q}\); then, it is translated of the vector \(-\mathbf{p}_{q}^{r}\) and rotated through the rotation matrix \(\mathbf{R}_{r}^{q}=\mathbf{R}_{q}^{r}\mathrm{T}\). For example, in the setup developed in this work, shown in Figure 4, the matrix \(\mathbf{R}_{r}^{q}\) is given by:  
\[\mathbf{R}_{r}^{q}= \begin{bmatrix}0&-1&0\\ 0&0&-1\\ -1&0&0\end{bmatrix} . \tag{1}\]  
In this way, the coordinate frame of the hologram is expected to be coincident with the real robot base frame. The QR code pose detection could be affected by the perspective distortion due to the relative height of the human operator point of view with respect to the QR frame during the calibration phase. Hence, if HoloLens perceives the QR plane as tilted with respect to the vertical plane, a further rotation is added in order to compensate this effect and make the QR code frame always aligned with the robot base. Once the hologram is collocated in the AR environment, the user has to verify the calibration accuracy.