<figure><img src="image_1.png" /></figure>

HAL
open science

# Design and Manipulation in Augmented Reality of Fanuc 200iC Robot

Benaoumeur Ibari, Kamel Bouzgou, Zoubir Ahmed-Foitih, Laredj Benchikh

► To cite this version:

Benaoumeur Ibari, Kamel Bouzgou, Zoubir Ahmed-Foitih, Laredj Benchikh. Design and Manipulation in Augmented Reality of Fanuc 200iC Robot. Journal of Intelligent Computing, 2015, 6 (3), pp.84-91. hal-01275138

HAL Id: hal-01275138

https://hal.science/hal-01275138

Submitted on 9 Oct 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Design and Manipulation in Augmented Reality of Fanuc 200iC Robot

Ibari Benaoumeur¹, Bouzgou Kamel¹, Ahmed-foitih Zoubir¹, Benchikh Laredj²

¹Laboratory of Power Systems
Solar Energy and Automation L.E.P.E.S.A. USTOMB.
Oran-Algeria

¹IBISC Laboratory. University of Evry France

bouzgou_kamel@hotmail.fr, ibari_b@yahoo.fr, zfoitih@yahoo.fr, llaredj@gmail.com

**ABSTRACT:** In this paper, an AR (Augmented reality) system based on a multimodal user interface is proposed and its core techniques are described. The objective of this work is to realize certain task that operators can use the virtual robot to ameliorate the interaction with the real world using AR environment. VC++ and OpenGL are used to create this application. AR applications (ARToolKit) is used to detect the real-time position and orientation of camera and to facilitate the operator's observation and operation. Experiments prove that the system is reliable and efficient. Simulation result is obtained and it is very useful for designers to ameliorate the interaction with the real world.

**Keywords:** Augmented Reality (AR), Kinematics model, 200iC Fanuc, OpenGL, User Interface

## 1. Introduction

Augmented Reality (AR) allows the overlay of virtual images on the real world, there has been much research in the field of AR in last years, AR is used for a wide range of applications in computer vision such as computer-aided surgery, repair of complex machines, establishment modifications, interior or structural design. AR techniques can be used to help bring the right information to the right place at the right time [15]. The principle of AR is to take real images to which one adds additional information for a better perception of reality.

In [6], graphical user interface is proposed to make a task easier and mechanisms for the operator and the robot to ex- change information at different levels of detail or abstraction and to verify the robot in Virtual Reality and the wireless remote control.

In the work reported by Rastogi and Milgram [14], an operator is provided with visual feedback of a virtual wireframe robot superimposed over an actual physical robot located at its remote working environment. The virtual robot will execute a task for evaluation by the operator, and if it is satisfactory, the task will be transferred to the real robot.

The other initiative, The ARITI system [13] (Augmented Reality Interface for Teleoperation via Internet), one of the characteristics
of project ARITI is the use of the AR for the teleoperation, which is a system of teleoperation allowing the control of a robot by using the techniques of AR, it constitutes an experimental system of teleoperation via the Internet. As an application using a robot to four degrees of freedom, a new version with a robot FANUC Robotics with six degrees of freedom was designed.

In this research work, we present an idea of AR system based on a multimodal user interface, this interface allow users to interact with system using multiple different modes of communication (keyboard, Cameras and joystick), which integrates the virtual robot directly with the real environment video, without building an environment model. Video images provide rich and highly contrasted information. The ARToolkits marker detection method and OpenGL are used for rendering [7]. The virtual environment is established by VC++ and OpenGL.

## 2 Design of the AR System

The main goal of the system is to generate a path on virtual workpieces using a virtual robot file ( Virtual Reality Modeling Language) and computer generated models in AR environment.

### 2.1 Relationships between Frames

In order to calculate the position of the end-effector, ie the robot tool, the transition matrix between the camera and the robot tool (Tct, Rct) is determined using the relationships between Frames as see in figure 1. The translation Tct and the rotation Rct are given by:

<figure><img src="image_3.png" /><figcaption>Figure 1. Relationships between Frames</figcaption></figure>

$$T_{ct} = R_{cb} * T_{bt} + T_{cb} \qquad (1)$$

$$R_{ct} = R_{cb} * R_{bt} \qquad (2)$$

The transition matrix between the camera and the robot base (Tcb and Rcb) is determined using the most popular algorithm in AR applications, the ARToolKit [8].

ARToolKit includes several models of two-dimensional markers. It allows to find markers, to recognize and identify them. However, its performance in markers detection should be improved. In fact, often, it happens that markers are confused with each other or they are detected by error in foreground. ARToolKit marker is black border square surrounding a model which is compared to other pre-recorded models in ARToolKit matching template database (figure 2) [11]. ARToolKit gives the position of the marker in the camera coordinate system, and uses OpenGL matrix system for the position of the virtual object. ARToolKit defines different coordinate systems mainly used by the computer vision algorithm and the rendering.

Whereas The transition matrix between the robot base and the end-effector (Tbt and Rbt) is calculated by using the Geometric model of the FANUC 200iC robot.

### 2.2 The Direct kinematic Model

In this work we use LR Mate 200iC Robot model, an industrial 6 DOF robot from Fanuc Robotics [12]. The direct geometric model (DGM) is the set of relations which express the position of the end-effector, i.e. operational coordinates of the robot, according
to its joint coordinates. In the case of a simple open-chain, Forward kinematics model of the robot is represented by

$$X_i = f(\theta_i) \qquad (3)$$

<figure><img src="image_4.png" /><figcaption>Table 1. Modified geometric parameters D-H of the FANUC robot</figcaption></figure>

<table><caption>Table 1. Modified geometric parameters D-H of the FANUC robot</caption><thead><tr><th>i</th><th>σ<sub>i</sub></th><th>α<sub>i</sub></th><th>d<sub>i</sub></th><th>θ<sub>i</sub></th><th>r<sub>i</sub></th></tr></thead><tbody><tr><td>1</td><td>0</td><td>0</td><td>0</td><td>θ</td><td>0</td></tr><tr><td>2</td><td>0</td><td>90</td><td>75</td><td>P + θ<sub>2</sub></td><td>0</td></tr><tr><td>3</td><td>0</td><td>0</td><td>400</td><td>θ<sub>3</sub></td><td>0</td></tr><tr><td>4</td><td>0</td><td>90</td><td>75</td><td>θ<sub>4</sub></td><td>410</td></tr><tr><td>5</td><td>0</td><td>-90</td><td>0</td><td>θ<sub>5</sub></td><td>0</td></tr><tr><td>6</td><td>0</td><td>90</td><td>0</td><td>θ<sub>6</sub></td><td>0</td></tr></tbody></table>

Where σᵢ, αᵢ₋₁, dᵢ, θᵢ and rᵢ are the Denavit-Hartenberg parameters [4]. The link transformation matrix are given by the Denavit-Hartenberg and Table 1 [9]. The transition matrix of the LRMate 200iC Robot is calculated in [2], it can be represented by the transformation matrix Tₖ.₀.

$$T_{k}^{0} = k_{i=1} T_{i}^{i-1} q (i) \qquad (4)$$

Realizing the composition of transformations of equation (3) we obtain:

$$T_{6}^{0} = T_{1}^{0} T_{2}^{1} T_{3}^{2} T_{4}^{3} T_{5}^{4} T_{6}^{5} \qquad (5)$$

$$T_E^f = T_0^6 \qquad (6)$$

and

$$T_E^f = \begin{bmatrix} S_x & N_x & A_x & P_x \\ S_y & N_y & A_y & P_y \\ S_z & N_z & A_z & P_z \end{bmatrix} \qquad (7)$$

The transformation matrix of tool frame in the end-effector frame is given by:

$$E = T_E^0 \qquad (8)$$

## 2.3 Robot Fanuc Modeling

The virtual environment includes the geometrical model of robot. The virtual geometric model is created using the real dimensions of the robot. The robot is designed by the use of the CAD environment SolidWorks [16]. The SolidWorks source file is used to generate the geometrical model of robot. After the file converted into VRML file format, it can be read directly by the OpenGl Library. Then the robot model can be displayed in simulation scene. The virtual model of the FANUC 200iC robot is shown in figure 3.
<figure><img src="image_5.png" /><figcaption>Figure 3. Fanuc 200iC robot modeling in virtual environment</figcaption></figure>

## 3. Results and Analysis

In this work, we use a camera Logitech Webcam C 905, The camera specifications are:

* Resolution: 16001200.

* Up to 8-megapixel photos (enhanced from Native 2-MP HD sensor.)

* FPS at full resolution: 30

* Hi-Speed USB 2.0 certified.

The camera is calibrated with ARToolKit method "Two Step. Calibration" [1], and the intrinsic parameters are:

* Distortion factor (mm): (288,5 356,5 53,3 1,3456)

* Matrix intrinsic:

$$ \begin{pmatrix} 655,0883 & 0.00000 & 438.50000 & 0.00000 \\ 0.00000 & 759.23013 & 316.60000 & 0.00000 \\ 0.00000 & 0.00000 & 1.00000 & 0.0000 \end{pmatrix} $$

### 3.1 The User Interface

For the users, Graphic Interface is developed to interact with AR application allowing the visualization and localization of virtual objects. This interface contains 2 parts: The first one used for the application controls using the forward kinematics model by giving the robot necessary articulation to attain a desired position, The second part is to integrate the vision module to display real-time video stream from camera. It is used to show the simulation display of the virtual robot and the virtual environment.

#### 3.1.1 The Interface Control

This interface provides all the necessary tools to ensure the control of the robot and to follow the position of the end-effector relative to the camera. The robot can operate according to Joint angles where it moves following its 6 rotations ($\theta_1, \theta_2, \theta_3, \theta_4, \theta_5, \theta_6$) Figure 4. The end-effector of robot can be shown in this view. The control panel is used to control the motion of the virtual robot by pressing the button on it.

#### 3.1.2 The Vision System

The solidWorks software is used to generate the 3D model of the robot Fanuc 200iC, in this module the tracking process in
<figure><img src="image_6.png" /><figcaption>Figure 4. Fanuc 200iC robot interface control</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 5. Display real-time video stream from camera</figcaption></figure>

is used with ARToolKit and openGL, it is composed of two steps: object recognition and pose estimation. The positions and orientations of the markers relatively to the camera are calculated, and the virtual objects are rendered in video frame as see in Figure 5.

3.2 Exemplpe

A simple miniature work cell was constructed using virtual objects as shown in Figure 6. The task is to move the robot using the intrface control to the virtual objects. As the user moves the end-effector, its tip coordinates relative to the camera’s coordinate system are displayed in the interface control user for guidance.

We carried out the experiment with 20 robot positions of Tx,Ty and Tz, we computed at the same time the transforma- tion relating
the robot tool to the camera.

<figure><img src="image_8.png" /><figcaption>Figure 6. Example of the fanuc robot task</figcaption></figure>

<table><caption>Table 2. The numerical results of the calibration (Tct)</caption><thead><tr><td></td><td>Average value</td><td>Variance</td><td>Standard deviation</td></tr></thead><tbody><tr><td>Tx (mm)</td><td>181.1198</td><td>1.9960</td><td>1.4128</td></tr><tr><td>Ty (mm)</td><td>181.8311</td><td>11.0411</td><td>3.3228</td></tr><tr><td>Tz (mm)</td><td>816.0435</td><td>4.3297</td><td>2.0808</td></tr></tbody></table>

From the figure 7, we can see that the measured values are fairly constant. It is the goal for the pose estimation between the camera and the end-effector robot.

Table 2 shows that the calibration presents the best performance in terms of generalization error, the numerical results proved the effectiveness of this calibration to extend overlaying on other scene elements using a single target pose computation.
<figure><img src="image_9.png" /><figcaption>Figure 7. The calibration of the end-effector position (Tct)</figcaption></figure>

Compared to the manipulation application in a virtual reality environment based on VRML (Virtual Reality Modeling Language) in [4], the results from our AR system reveal that our new interface was successful in improving task efficiency, the AR system is added to perform a certain task using the ARToolkits algorithm
## 4. Conclusion

In this paper, we presented an AR system intended to ameliorate the interaction with the real world. For this purpose, the system vision has been developed using ARToolkits marker detection method and OpenGL library. The manipulation of 200iC FANUC Robot manipulator is solved using the kinematic model of the robot. For the use we created a Graphic User Interface to follow the evolution of virtual robot in the real environment. The analysis of results shows the performance of the system in terms of accuracy for camera localization using AR targets. To evaluate the performances of localization, we can added different pose estimation methods (Dhome et al [5], EKF [3], Hybrid EKF [10]).

## References

[1] Abdullah, Junaidi., Martinez, Kirk. (2002). Camera self-calibration for the artoolkit. In Augmented Reality Toolkit, The First IEEE International Workshop, p. 5 IEEE.

[2] Bouzgou, Kamel., Foitih, Zoubir Ahmed. (2014). Geometric modeling and singularity of 6 dof fanuc 200ic robot. In Innovative Computing Technology (INTECH), 2014 Fourth International Conference on, pages 208–214.IEEE.

[3] Chai, Lin., Hoff, William A., Vincent, Tyrone. (2002). Three-dimensional motion and structure estimation using inertial sensors and computer vision for augmented reality. Presence: Teleoperators and Virtual Environments, 11 (5) 474–492.

[4] Denavit, Jacques. (1955). A kinematic notation for lower-pair mechanisms based on matrices. *Trans. of the ASME. Journal of Applied Mechanics*, 22, 215–221.

[5] Dhome, Michel., Richetin, Marc., Lapreste, Jean-Thierry., Rives, Gerard. (1989). Determination of the attitude of 3d objects from a single perspective view., *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 11(12) 1265–1278.

[6] Ibari, Benaoumeur., Ahmed-Foitih, Zoubir., Reda, Hanifi Elhachimi Amar (2015). Remote control of mobile robot using the virtual reality. International Journal of Electrical and Computer Engineering (IJECE), 5(5).

[7] Ibari, Benaoumeur., Bouzgou, Kamel., Ahmed-Foitih, Zoubir., Benchikh, Laredj. (2015). An application of augmented reality (ar) in the manipulation of fanuc 200ic robot. *In: Innovative Computing Technology (INTECH), 2015 Fifth International Conference on*, p. 56–60. IEEE.

[8] Kato, Hirokazu., Billinghurst, Mark., Poupyrev, Ivan., Imamoto, Kenji, Tachibana, Keihachiro. (2000). Virtual object manipulation on a table-top ar environment. *In: Augmented Reality, 2000.(ISAR 2000). Proceedings. IEEE and ACM International Symposium on*, p. 111–119. IEEE.

[9] Giovanni, Legnani., Federico, Casolo., Paolo, Righettini., Bruno, Zappa. (1996). A homogeneous matrix approach to 3d kinematics and dynamics. theory. *Mechanism and Machine Theory*, 31 (5) 573–587.

[10] Maidi, Madjid., Didier, Jean-Yves., Ababsa, Fakhreddine., Mallem, Malik. (2010). A performance study for camera pose estimation using visual marker based tracking. *Machine Vision and Applications*, 21(3) 365–376.

[11] Maidi, Madjid., Mallem, Malik., Benchikh, Laredj., Otmane, Samir. (2013). An evaluation of camera pose methods for an augmented reality system: Application to teaching industrial robots. *In: Transactions on Computational Science XVII*, p. 3–30. Springer.

[12] FANUC Robot LR Mate. 200ic, fanuc robot arc mate 50ic, mechanical unit, operator's manual. Technical report, B-82584EN/ 06.

[13] Otmane, Samir., Mallem, Malik., Kheddar, A., Chavand, F. (2000). Ariti: an augmented reality interface for teleoperation on the internet. In *Advanced Simulation Technologies Conf.(ASTC2000)*, p. 254–261.

[14] Rastogi, Anu., Milgram, Paul., Grodski, Julius J. (1995). Augmented telerobotic control: a visual interface for unstructured environments. *In: Proceedings of the KBS/Robotics Conference*, p. 16–18.

[15] Regenbrecht, Holger., Baratoff, Gregory, Wilhelm Wilke. (2005). Augmented reality projects in the automotive and aerospace industries. Computer Graphics and Applications, IEEE, 25 (6) 48–56.

[16] Yuan, Feng-wei., Li, Bi-wen., He, Bin. (2007). Precise modeling of virtual device in virtual reality based on solidworks-vrml, *Mechanical & Electrical Engineering Magazine*, 10. 034.