# Augmented Reality-Based Interface for Bimanual Robot Teleoperation
## 3 Proposed Augmented Reality-Based User Interface
This was an iterative design process, where the main characteristics of the virtual objects (e.g., size, color, shape, etc.) and their interactions were verified and modified, connecting the workstation with the AR headset in a remote mode from the Unity editor (note that the perception of the holograms is different when showing them in a PC screen compared to when projecting them in the real world through the AR headset), until the result was satisfactory. Figure 6 shows the holograms designed for the robot references. In the case of the WR, the user can command the robot through the 3D workspace and modify the end-effector orientation. For this reason, two different holograms were designed. The translation reference hologram was modeled by a 3D orange cube; see Figure 5(a). This hologram appears when the user teleoperates the WR translation reference. To reduce the number of holograms present at any moment, this hologram disappears 3 s after the user has stopped moving the WR translation reference. The orientation reference hologram was modeled by an animated arrowed yellow circle; see Figure 5(b). This hologram appears when the user teleoperates the WR rotation reference, and disappears 3 s after the user has stopped moving the WR rotation reference. It should be noted that, in both cases, the movement of the references is relative to the position of the user, i.e., the AR headset, making their use more intuitive and natural. The STR translation reference was modeled by a yellow arrow attached to a green sphere; see Figure 5(b). Note that this hologram is constrained to the plane of the workpiece surface, allowing a 2D movement. This hologram disappears 3 s after the user has stopped moving the STR translation reference. Figure 7 shows the holograms designed for the 2D and 3D boundaries. The 3D boundary is modeled by a superellipsoid--see Figure 6(a)--which is defined as:  
\[\left|\frac{x}{W}\right|^{m}+\left|\frac{y}{H}\right|^{m}+\left|\frac{z}{M} \right|^{m}=1, \tag{1}\]  
where \(\{W,H,M\}\) are the superellipsoid axes and \(m\) represents the smoothing parameter of the superellipsoid, i.e., it is equivalent to an ellipsoid for \(m=2\), whereas it tends to a cuboid as \(m\) tends to infinity. For the bimanual robot application at hand, it has been chosen \(m=4\). The 2D boundary is modeled by a modified superellipse--see Figure 6(c)--which is defined as:  
\[\left|\frac{x}{W}\right|^{m}+\left(\frac{\max(|y|-(H-W),0)}{W}\right)^{m}=1, \tag{2}\]  
where it is implicitly assumed that the value of axis \(H\) is greater than that of axis \(W\) (the expression is easily modified for the analogous case \(H<W\)). This equation represents a rectangle with smooth corners, with \(2H\) for its long side and \(2W\) for its short side, by joining a \(2W\times 2(H-W)\) rectangle to two offset halves of an even-sided \(2W\times 2W\) superellipse. Note that if the proposed boundary holograms were permanently shown, they could occlude some real elements from the user's view, affecting the task performance. For this reason, a new material shader [80] was designed; see Figure 8. This shader computes the minimum distance between the robot end-effector and the 3D boundary, for the case of the WR, or the closest point of the robot tool to the 2D boundary, for the case of the STR. Thus, the shader only displays the affected part of the boundary hologram. That is, as the WR end-effector and/or the STR tool approach to the 3D and 2D boundaries, respectively, the part of the boundary hologram affected is progressively displayed; see Figure 7b,d. In addition to this, and according to the user requirements, two warning sounds were included in the interface: the first one to indicate that the STR tool is close to the 2D boundary; and the second one to indicate that the WR end-effector is close to the 3D boundary. Moreover, the user is able to deactivate this warning sound at any time. Once the main holograms and sound elements were implemented, some communication protocols were used and programmed.