# Automatic Mobile Robot Control and Indication Method Using Augmented Reality Technology

Koji Ohmori, Kunio Sakamoto

► To cite this version:

Koji Ohmori, Kunio Sakamoto. Automatic Mobile Robot Control and Indication Method Using Augmented Reality Technology. 9th International Conference on Entertainment Computing (ICEC), Sep 2010, Seoul, South Korea. pp.464-467, 10.1007/978-3-642-15399-0_61. hal-01055600

HAL Id: hal-01055600

https://inria.hal.science/hal-01055600

Submitted on 13 Aug 2014

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
Automatic Mobile Robot Control and Indication
Method Using Augmented Reality Technology

Koji Ohmori and Kunio Sakamoto

Department of Intelligence and Informatics, Konan University
8-9-1 Okamoto, Higashinada, Kobe 658-8501, Japan
kunio@konan-u.ac.jp

**Abstract.** A mobile robot is an automatic machine that is capable of movement in a given environment. Many techniques of automatic control are proposed. A line tracer is one of the most popular robots. The line tracer goes along a white line on the floor. The authors developed a mobile robot which moves to indicated point automatically. All you have to do is to indicate a goal point. In this paper, we propose an automatic mobile robot system controlled by a marker and remote indication using the augmented reality technology.

**Keywords:** mobile robot, automatic control, marker and remote indication

# 1 Introduction

Our developed mobile robot can move itself to indicated goal point by an operator. A trial robot system provides us two indication methods. One is how to indicate as the operator put a marker as a goal point on the floor. The other is a remote control through closed-circuit television like the operator indicates the goal point on a video monitor. The mobile robot decides a route to the goal according to a behavior decision algorithm. The automatic mobile robot moves itself so as to shorten a distance from current residence to goal in consideration of a direction to the goal.

# 2 Target Indication by Goal Marker

Fig. 1 shows a scene of automatic mobile robot operation. The authors have developed a mobile robot system using a commercial radio control car and an additional controller. This robot is made utilizing a frame of the radio control car and attached with a marker in order to measure its position on the floor. The radio controller is also reconstructed so as to operate a motion from Windows PC. The operator indicates a goal using the marker as shown in Fig. 1. To control the mobile robot automatically, a camera fixed on the ceiling shots the markers which are robot and goal positions. The direction and distance from current residence to goal are calculated after the operation system recognizes markers and measures spatial positions. To get relation between markers, we used the software library which is
called ARToolkit. ARToolkit involves a video capture, a 3D graphics generator, a spatial measuring and an overlay imaging for the creation of augmented reality applications. ARToolkit calculates three coordinate axes from a captured camera scene. The video tracking libraries calculate the real camera position and orientation relative to physical markers in real time as shown in Fig. 2.

Suppose that two 2D markers are laid on the floor. One is the marker which indicates a goal position. The rest shows the position of the mobile robot. The tracking library calculates each three coordinate axes on the two markers. Since our system program can detect a difference between two coordinate systems, we can know the direction and distance from current residence to goal. In Fig. 2, let ($X_c, Y_c, Z_c$) be three space coordinate of a camera and parameter $Z_c$ shows the depth. Moreover, let ($X_m, Y_m, Z_m$) be three space coordinate of a marker and parameter $Z_m$ shows the height. Assume that the transposed matrix [$X_c Y_c Z_c 1]^T$ means the camera coordinate and [$X_m Y_m Z_m 1]^T$ means the marker coordinate. Then the coordinate transformation between three space coordinates of a camera and a marker is given as follows by using a coordinate transformation matrix $T_{cm}$;

$$[X_c Y_c Z_c 1]^T = T_{cm} [X_m Y_m Z_m 1]^T. \quad (1)$$

Fig. 2 shows a transformation process between three space coordinates of markers on the same plane. Equation (1) shows the coordinate transformation between three space coordinate of a camera and three space coordinate of a marker. As shown in Fig. 2, the marker A and marker B are shot by the same camera. Assume that [$X_{Am} Y_{Am} Z_{Am} 1]^T$ means the coordinate of marker A and [$X_{Bm} Y_{Bm} Z_{Bm} 1]^T$ means the coordinate of marker B. From equation (1), the coordinate transformation matrixes $T_{Acm}$ and $T_{Bcm}$ are given as follows;

$$[X_c Y_c Z_c 1]^T = T_{Acm} [X_{Am} Y_{Am} Z_{Am} 1]^T, \quad [X_c Y_c Z_c 1]^T = T_{Bcm} [X_{Bm} Y_{Bm} Z_{Bm} 1]^T.$$

<figure><img src="image_2.png" /><figcaption>Fig. 1. Scene of mobile robot operation</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Fig. 2. 3D measuring for marker tracking</figcaption></figure>

The coordinate transformation matrix $T_{cm}$ shows the relation between three space coordinate of a camera and three space coordinate of a marker. When the coordinate transformation matrixes $T_{Acm}$ and $T_{Bcm}$ are determined, the camera is at the same position. Therefore the relation of the positions between the coordinate of marker A [$X_{Bm} Y_{Bm} Z_{Bm} 1]^T$ and the coordinate of marker B [$X_{Am} Y_{Am} Z_{Am} 1]^T$ is given as follows;

$$[X_{Am} Y_{Am} Z_{Am} 1]^T = T_{Acm}^{-1} T_{Bcm} [X_{Bm} Y_{Bm} Z_{Bm} 1]^T,$$

$$[X_{Bm} Y_{Bm} Z_{Bm} 1]^T = T_{Bcm}^{-1} T_{Acm} [X_{Am} Y_{Am} Z_{Am} 1]^T.$$

After a decision of the direction and distance to the goal, the automatic mobile robot judges a next action itself according to an algorithm of the motion behavior decision. Fig. 3 shows the behavior decision algorithm. Suppose a line whose direction vector is the same to a direction of movement. Firstly the mobile robot
judges whether a goal marker is in the left or right. Assume that the answer is the right. Next judgment is whether the goal exists in front or behind. This decision is to choose that the mobile robot goes straight or it makes a U-turn refers to performing a 180 degree rotation to reverse the direction of movement. If the goal exists in the same direction of movement, the robot goes straight or makes a U-turn in order to change a heading direction. In case that the goal does not lie on the line, the mobile robot moves to right in front or to left at back in order that it turns toward the goal. After slight movement the robot again judges whether a goal marker is in the left or right. To perform a series of slight movement reaches the goal. This robot can also take an action in the same way even if the goal marker exists in the left side.

### 3 Remote Control by Closed-circuit Television

Fig. 4 shows the illustration of capturing the marker on the floor using a pin-hole camera. The marker in the real world and its perspective image on the plane of a film are indicated in this figure. This marker is utilized in order to define a plane of the floor. In Fig. 4, let (X, Y, Z) be three space coordinate and parameter Z shows the depth. The S<sub>X</sub>-S<sub>Y</sub> coordinate system shows the image plane of a camera. Assume three points and a normal vector. Points (p<sub>x</sub>, p<sub>y</sub>, p<sub>z</sub>) and (t<sub>x</sub>, t<sub>y</sub>, t<sub>z</sub>) are target points which are indicated on the video screen and in real world respectively. (m<sub>x</sub>, m<sub>y</sub>, m<sub>z</sub>) is a center point of the square marker. (n<sub>x</sub>, n<sub>y</sub>, n<sub>z</sub>) is a normal vector of the floor plane and it is also z axis of the marker coordinate. If the focal length of a camera is f, p<sub>z</sub>=f. When you indicate point (p<sub>x</sub>, p<sub>y</sub>, p<sub>z</sub>) on the view screen as a target point on the floor, points (p<sub>x</sub>, p<sub>y</sub>, p<sub>z</sub>) and (t<sub>x</sub>, t<sub>y</sub>, t<sub>z</sub>) lie on the same line. This line equation is

$$[t_x, t_y, t_z]^T = t [p_x, p_y, p_z]^T \quad (t \text{ is variable}). \quad (2)$$

Meanwhile the equation of the floor plane is given as follows;

$$n_x(X - m_x) + n_y(Y - m_y) + n_z(Z - m_z) = 0. \quad (3)$$

From the equations (2) and (3), we get an intersection (t<sub>x</sub>, t<sub>y</sub>, t<sub>z</sub>) of the line and plane;

$$n_x(t p_x - m_x) + n_y(t p_y - m_y) + n_z(t p_z - m_z) = 0.$$

Then we can get the parameter t as follows and the indicated point on the real floor;

$$t = (n_x m_x + n_y m_y + n_z m_z) / (n_x p_x + n_y p_y + n_z p_z).$$

Using this parameter t, the mobile robot decides the goal point. After this calculation, the robot can move itself in the same way as the marker indication.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Behavior decision algorithm</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 4. Measurement of the 3D position</figcaption></figure>
**Acknowledgments.** This research is partially supported by “Grant-in-Aid for Young Scientists(B)” #20700112 and “Scientific Research (C) (General)” #20500481 from Ministry of Education, Culture, Sports, Science and Technology Japan(MEXT).