# Human-Robot Collaboration: An Augmented Reality Toolkit for Bi-Directional Interaction
## 2 State-of-the-Art
On the other side, the operator sends inputs to the robot through hand gestures, in order to select the control strategy and the task to be performed. A polishing task is executed comparing 2D display and AR systems: the AR interface provides better feedback, allowing the user to reduce his/her effort variances during the task execution. [PERSON] et al. propose an AR system that outperforms with respect to 2D display in [20], where the AR interface reduces the task completion time (about 38%), increases the accuracy of collision prediction by an average of 15% and precision by 11%. In order to map the environment in real time and collocate the operator inside the AR reconstruction, the system set-up consists of Microsoft HoloLens headset, an inertial measurement unit, four cameras and an infrared (IR) depth sensor. The user is immersed in a virtual space, perceiving the scene through the HoloLens device and interact with it through hand gestures (through 3D game engine Unity) and voice commands. One-hand and two-hand gestures are exploited in order to specify the desired position and orientation of robot end-effector and a voice command instructs the robot to plan the trajectory, which is validated by the operator through AR. If the operator notices that an object collision could occur, she/he replans the trajectory with a stochastic planner. The same approach is proposed in [21], where an AR interface supporting robot programming is presented: pick-and-place tasks are designed using sequences of way-points. In this way, 3D trajectories are generated and eventually altered by moving, adding and deleting the way-points. Furthermore, the interface allows the users to visualize the assigned path that the robot would take from two consecutive way-points, which is planned by means of 'MoveIt\". Robot programming through AR platforms is also investigated in [22], where two trajectory generation methods are proposed: free space trajectories and surface trajectories. In the first case, the user selects the pick-and-place locations in the physical workspace through speech and gesture inputs that are recognized by HoloLens. A virtual scenario allows the user to dynamically modify the path in free space during the simulation or even during the execution. On the other hand, virtual paths can be defined on a surface by using head tracking, speech and gesture recognition. The robot end-effector can then move along the path while keeping its orientation normal to the surface, due to an impedance-controlled mode, which ensures the application of a constant normal force. Either in [21] or [22], the usability of proposed AR systems is evaluated and compared to classical approaches. In [21], volunteers have been instructed to program the robot to pick up and place a cube onto a platform, bringing it over walls of different heights. The task was completed using two different interfaces: a 2D monitor interface and the HoloLens AR application. In [22], participants were instructed to perform two tasks using the proposed AR platform and a kinesthetic teaching interface, i.e., a gamepad for storing way-points: a surface contact task, consisting in the erasure of lines drawn on a flat surface by means of a marker eraser fit in the robot end-effector, and a free space task, i.e., a pick-and-place in the presence of an obstacle. In both cases, significant reductions in the teaching time and the NASA Task Load Index [15] combined with increments of performance and usability supported the use of the AR systems for novice users. In [23], [PERSON] et al. adopt the AR to improve the navigation of a mobile robot and visualize trajectory and spatial mapping. The setup consists of a Microsoft HoloLens as the AR device, and Kuka Youbot as the mobile robot. The development platform used for HoloLens is Unity 2018, while the robot is controlled through ROS Hydro. The communication between ROS and HoloLens is ensured by the ROSBridge protocol, which follows a specific JavaScript Object Notation (JSON) notation for messages.