# Human-Robot Collaboration: An Augmented Reality Toolkit for Bi-Directional Interaction
## 2 State-of-the-Art  
Robots, sensors, devices and human resources are intelligent entities that can be integrated in a smart factory according to the paradigms of Industry 4.0 approach. Thanks to their cooperation and real-time data sharing, smart factories are able to reach high levels of flexibility and reconfigurability in order to dynamically adapt to current market changes. The aim to improve flexibility of logistic and production systems is pursued not only with technological resources, e.g., smart devices and machines, but also with a new role of human operators, as the modern concept of the so-called _Operator 4.0_[16]. In smart factories, various augmentations of the original human capabilities are adopted to empower their \"smart operators\" with new skills and gadgets to fully capitalize on the opportunities being created by Industry 4.0 technologies. For that matter, the increasingly widespread introduction of cobots in industry provides an improvement of smart operator productivity; in addition to relieving her/him from non-ergonomic and repetitive tasks, cobots and humans can interact, establishing an information stream, e.g., by means of multi-modal interfaces [16]. In Human-Robot Interaction (HRI) systems, agency cannot be exclusively attributed to humans but it is distributed among humans and non-human resources [4]; interaction is exploited by assigning operators and robots different sub-tasks leading to a common goal (_cooperation_), or a sequence of tasks accomplished together toward a shared goal (_collaboration_). In the latter case, the envisioned goal is to make both human operator and cobot proactive, providing a bi-directional stream of exchanged information. For this purpose, hand gesture recognition could help humans to convey information and express intentions in a simple way. Relevant advantages of this solution with respect to traditional human-machines interaction, for instance mouse, keyboard and 2D display requiring a fixed operating space, are as follows: high degree of differentiation, strong flexibility and high efficiency of transmission, achievable through data gloves, ultrasound, vision or wearable devices based on electromyographic signals [17]. In [18], a novel AR-based teleoperation system is proposed, where the 3D virtual robot model dynamically tracks the current real robot state. The system architecture is composed of a multi-DoF industrial robot and two RGB-D sensors, connected to each other in the ROS framework and to the AR interface, built with head and hand gesture trackers, by means of ROSBridge server. The system's maneuverability is enhanced due to this setup which guarantees interaction with the environment by hand gestures, tracked using a Leap Motion controller mounted on the HMD. An \"interaction proxy\" manages hand gestures, making task performing easier and avoiding unintentional gesture disturbances. Moreover, two control modes (coarse movement mode and fine movement mode) are proposed to prevent operational faults related to each type of movement. In [5], the adoption of the Microsoft HoloLens device allows for bi-directional communication with a cobot, both with visual and acoustic feedback. The visual ones make the operator aware of robot movements, increasing her/his trust in collaboration, whereas the acoustic feedback is useful when the 3D holograms are out of sight or in adverse lighting conditions. The development tool is Unity, together with Mixed Reality Toolkit [19], while the robot is controlled through ROS. The communication exploits the User Datagram Protocol (UDP) and it is designed aiming at information exchange reduction, also preventing communication overload. Indeed, the operator wearing the HoloLens receives information about the robot state both in terms of positions and contact forces. A light with a colour code improves the operator safety: the color is green if the robot moves away from the operator; red if the robot approaches the operator; and yellow if the robot is waiting for user input.