# Closed-Loop Robotic Arm Manipulation Based on Mixed Reality
## 4 Software Tool Implementation
More specifically, what is of great interest is the calculation of the robot's joints revolution angles. These values, along with the other information, such as the robot's physical properties, are saved in a separate file, also known as Unified Robot Description Format (URDF). However, in order to enable this functionality in the Unity 3D development environment, the Rosbridge API has been implemented. It is stressed that from the standalone application, every time the operator moves the 3D robotic arm, for every joint of the robot, the angular displacement is recorded in a JSON file. Concretely, for the ROS operating system, a virtual PC has been set up and the motions from the standalone application are imported in the form of a JSON file, then the ROS engine interprets the motions in robot commands with the utilization of inverse kinematics for the corresponding robotic arm which are then communicated to the robot's control box via web sockets. In an attempt to further raise the awareness of the technicians whenever they work in close collaboration with the robot, two functionalities have been developed. The first functionality is the distance indicator, which calculates the absolute distance between the user (camera position) and the base of the robot, and is expressed in meters. In the following figure (i.e., Figure 3), the distance calculation method is illustrated. The equation (Equation (2)) presented in Figure 3 is a basic vector equation for the calculation of the relative distance between two vectors. In this case, the two vectors are the relative coordinates of the device and the base of the physical robot. In order to set up a global coordinate system, the setup of a 3D (virtual) anchor is required for indicating the origin of the virtual and physical environment (0,0,0). However, as per a reviewer's comment, we have further elaborated the content of Figure 3 in order to explicitly illustrate/indicate the above-mentioned technical information. Furthermore, the equation has been introduced and discussed within the main text of the manuscript. \[d\left(\overset{\rightarrow}{r},\overset{\rightarrow}{c}\right)=\|\overset{ \rightarrow}{r}-\overset{\rightarrow}{c}\|=\sqrt{\left(x_{c}-x_{r}\right)^{2 }+\left(y_{c}-y_{r}\right)^{2}+\left(z_{c}-z_{r}\right)^{2}} \tag{2}\]  
where \(d\left(\overset{\rightarrow}{r},\overset{\rightarrow}{c}\right)\) is the absolute distance between the user (device camera) and the physical robot, in meters. \[\overset{\rightarrow}{r} = \left[ \begin{array}{c}x_{r}\\ y_{r}\\ z_{r}\end{array} \right]\text{ are the coordinates of the robot's base. }\] \[\overset{\rightarrow}{c} = \left[ \begin{array}{c}x_{c}\\ y_{c}\\ z_{c}\end{array} \right]\text{ are the coordinates of the device camera. }\]The second functionality is the display of a colored edge around the screen of the user's device. The color-coding for this functionality is based on the color coding of the  
Figure 3: Distance calculation method. Figure 2: Communication interface architecture. safety zones. In Figure 4 the three states are indicated as the user's position in relation to the robot changes. The current implementation of the proposed framework is based on the development of a mobile application which is compatible with Android-based handheld devices. The developed application currently contains all the needed functionalities for the robotic arm manipulation, which include the main functionality, the communication of the application with the control box of the robot as well as the near real-time data exchange, which is used for the visualization of the robot's current position and stance. The main HoloLens application has also been developed, however, during the development of the framework, and issues have arisen regarding the communication of the 3D map to the framework so that the dynamic safety zones can be implemented. More specifically, the issue concerns the spatial scanning of the environment of the human operator which, by extension, contains the robotic arm.