# Closed-Loop Robotic Arm Manipulation Based on Mixed Reality
## 4 Software Tool Implementation  
For the implementation of the proposed framework, a multi-platform, stand-alone application has been designed and developed. The application is compatible with handheld devices, such as tablets and Head-Mounted Displays (HMDs). However, it is stressed that the 3D scanning of the robotic arm surroundings is capable only for the HoloLens HMD due to the limitations of the handheld devices' hardware. Concretely, the HMD chosen is the commonly known Microsoft HoloLens MR HMD. As far as the handheld device is concerned, a common Android tablet has been chosen. Regarding the development of the framework, the software used was mainly the Unity 3D game engine, due to the wide range of MR functionalities and supporting APIs. In addition to that the Vuforia API was used for the development of the functionalities of the handheld-based MR, whereas the Mixed Reality Toolkit (MRTK) was used for the HMD based MR. The writing of the code scripts was accomplished in C# programming language, using the Microsoft Visual Studio IDE. One of the most important implementation steps is the communication of the framework with the real robot, and more specifically the robotic arm's controller. As discussed in the previous paragraphs, the main goal was set to develop a fully wireless framework. As such, for the real-time data exchange the TCP/IP protocol was implemented, which is also compatible with the UR10 interface. The communication with the robotic arm is performed in two layers. The first layer is the real time data exchange layer and its purpose is to transmit data from the robot to the backend of the developed application, so that the position and the status of the robotic arm is successfully perceived by the application and, by extension, the 3D model is updated. The second layer of communication is the remote procedure call. This method can be realized as an XML file exchange between the application and the robotic arm controller, enabling the communication of programs, i.e., motion commands and methods/function calls, from the application, i.e., the user, to the robot. The architecture of the communication interfaces, based on the UR10 implementation, is presented in Figure 2, emphasizing the steps involved (see steps 1-9), the information flow (see the data filetypes), and the communication protocols implemented in order to achieve the interface between the individual modules. Below the step sequence is presented:  
**START**  
**Step 1:** Launch application on  
Handheld device (Android)  
MR HMD (MS HoloLens)  
**Step 2:** Environmental understanding based on the device sensing system  
Get device relevant position based on image target or 3D space anchor  
3D scan environment and recognize user's hands  
**Step 3:** Control virtual robot via  
Virtual controllers, implemented on the application GUI  
Hand gestures (e.g., tap to select, tap hold to grab, drag while grabbing)  
Update visualization of the 3D robot on the real environment  
**Step 5:** Save current position  
**GoTo Step 3** for new motion OR GoTo Step 6  
**Step 6:** Upload motion list (XML file) to Cloud Database  
Digital twin of robot (ROS environment)  
Initialize communication with Cloud Database via web sockets  
Download list of motions, and robot's URDF (Unified Robot Description Format)* Calculate the kinematic values
* Check feasibility of motions list
* Setup communication framework with physical robot
* Motion execution in physical robot
* Download list of motions from digital twin
* Execute motion
* Send feedback to user for successful completion of motion  
**GoTo Step 3** until user interruption  
**END**  
Furthermore, for the facilitation of communication between the robotic arm and the standalone application, the Robot Operating System (ROS) acts as a middleware for the translation of the 3D motions into commands for the actual robot. The ROS is utilized since it provides advanced capabilities regarding the calculation of the robotic arm kinematics and reverse kinematics.