Visual Attention Control for a Legged Mobile Robot based on
Information Criterion
Noriaki Mitsunaga∗† and Minoru Asada∗†
mitchy@er.ams.eng.osaka-u.ac.jp asada@ams.eng.osaka-u.ac.jp*Dept. of Adaptive Machine Systems, † HANDAI Frontier Research Center,
Graduate School of Engineering, Osaka University

Abstract
Visual attention is one of the most important issues
for a vision guided mobile robot. Methods have been
proposed for visual attention control based on information criterion[3, 4]. However, the robot had tostop walking for observation and decision. This paper
presents a method which enables observation and decision more efficiently and adaptively while it is walking. The method uses the expected information gainfrom future observations for attention control and action decision. It also proposes an image compensation method to handle the image changes due to therobot motion. Both are used to estimate observation
probabilities from the observation while it is walking
and then action probabilities are estimated from a decision tree based on the information criterion. The
method is applied to a four legged robot. Discussions
on the visual attention control in the method and thefuture issues are given.

1

Introduction

Mobile robots are often equipped with visual sensors that bring a huge amount of data about the environment. For efficient decision making, attention
control which extracts necessary and sufficient information for the given task is necessary. Gaze control
is the one of most important attention control fora robot equipped with a limited view angle camera
since the decision making is highly depends on it. We
have proposed a method of efficient observation for
decision making with gaze control based on information criterion [3, 4]. However, the methods assumed
a stop-observation-act approach. To make decisions
much more efficiently, observation while it is taking
some action, for example walking, has been desired.
To use observations while it is walking, the following
issues arises, 1) there are much more disturbances tosuch observation than the stop-and-observe one, 2)
it may move too much between gaze changes to regard the observation is done at the same location or

it must consider where the observation is done, 3) itshould stop walking or slow down the motion when it
cannot make any decisions. Kosaka et al.[2] have proposed retroactive updating of position uncertainty tohandle the second problem. Moon et al.[5] have proposed a view point planning considering the second
and the third problems for navigation tasks. Burgard et al.[1] proposed a Markov localization method
which determines which direction to move for localization. However, their method have not considered
the second nor the third problem. Also main purpose of all these methods was for localization andefficiency of observation was considered from a view
point of localization only. Further, they assumed thewheeled mobile robots of which observation is rather
stable and therefore ignored the first problem.
In this paper, we propose a method of image compensation mechanism for decision making to handle thefirst and the second problems. Further, we propose
to use the expected information gain as a reliability
measure for third problem. Action decision is done
by a decision tree constructed by information criterion based on the training data.
The rest of the paper is organized as follows. First,
the method to construct action decision tree is introduced along with basic ideas related to the information criterion, the efficient observation, and the decision making. Also the expected information gain
as a reliability measure is introduced. Then, thecompensation method for image changes by walking is proposed, and the experimental results using
the RoboCup four-legged robot league platform areshown. Finally, discussion and the future issues areshown.

2

The method

2.1

Assumptions

In our experiments, the robot has to pan and tilt
its camera to acquire the necessary information foraction selection since the visual field of the camera

is limited. The environment includes several landmarks of which appearances provide the robot with
sufficient information to uniquely determine the action. Training data are given for making decisions.
We used a teaching method to collect such data. A
training datum consists of a set of the views of thelandmarks at the current position and the action toaccomplish the task. During the training period, therobot pans and tilts its camera head to observe as
many landmarks as possible. The robot stops its leg
motions while rotating the head to guarantee thatthe landmarks are observed from the same location.
We separately prepare image data for image compensation of locomotion and image shakes by walking.
2.2

Information gain by observation

Suppose we have r kinds of actions and n training
data. First, the occurrence probabilities of actions pj(j = 1, ..., r) are calculated from pj = nj /n , where
nj denotes the number of taken action j. Therefore,
the entropy H0 for the action probability is given by
H0 = −rX

pj log2 pj .

(1)

j=1

Next, the occurrence probabilities of actions after observation are calculated. After the observation, it
knows whether the landmark is inside the attention
window (θLk , θU k ] or not. We denote the number oftimes action j was taken as nIijk when the landmark
i was observed in (θLk , θU k ] and nO
ijk when not observed. Then, the occurrence probability becomes,
I,OpI,O= nI,O(2)
ikj
ikj /nik ,
Pr I
Pr O
O
where nIik =j nikj , and nik =j nikj . Next,
the entropy after the observation are calculated asfollows:

Hik

=

−

X
x={I,O}rnxik X x(p log2 pxikj ). (3)nik j=1 ikj

The information gain by this observation Iik is
H0 − Hik . The larger Ii is, the smaller the uncertainty after the observation is.
When the time for observation is constant, we canuse the information gain to make action decision tree.
However, the time for observation changes dependingon the gaze directions. Therefore, we use the information gain per time, in other words the velocity,rather than information gain itself.
We denote T as the time to get the observation after
previous one, and the information gain per time iik
is,
iik =

H0 − Hik
Iik=,
T + TC
T + TC

(4)

0.5
0.5
0.5

0.5

A
0.25

B
0.25

C
0.5

Figure 1: An example action decision tree.

where TC is a positive constant. When the directionis already observed T = 0. To keep simplicity, we
limit pan and tilt angles into several discrete values
and the robot select an observing direction nearest
to the attention window from them.
2.3

Making an action decision tree

To make an action decision tree, 1) we calculate each
information gain per time iik for action after observing a window k of landmark i, and 2) divide thedata set according to the attention window with thelargest information gain per time. We iterate theseprocesses until the information gains for all windows
become zero or the action in the subset of the training data becomes unique. In the action decision tree,
a node, an arc, and a leaf indicate the window to divide the data set, whether the landmark is observedin the window, and the action to be taken, respectively. As a result attention windows are in decreasing order of uncertainty after its observation.
2.4

Making a decision

In order to make an action decision, first, the robot
sets the observation probabilities to attention windows of the currently or previously observed directions, and 0.5 to windows of never observed directions. Then it calculates action probabilities using
the observation probabilities. An action probability
is the sum of the probability to reach leaves to takethe action in the tree. If one of the action probabilities is very high, that action would be the correctone. However, in some cases it is dangerous to use
action probabilities only. For example, in the action
decision tree of Figure 1, the action probability of C
is 0.5 before any observation is done.
Here, we propose to use an expected information gain
as a measure of reliability. We define total expected
information gain as,
−

X

nnode
)Inode {p log2 p + (1 − p) log2 (1 − p)},n
all node(5)(standing|walking|------------|---|---|---|---|---|---|-A
B
C
D
E
F
G
Figure 2: A time sequence example. At time A, it
starts walking. A, B, C,... are the beginning of awalking period.
where nnode is the number of training data that thisnode has and Inode is the expected information gain
which is calculated in building the tree, and p is
the probability that the landmark is in the attention window of the node. Since the entropy of p,p log2 p + (1 − p) log2 (1 − p) ranges from 0 to 1, andit becomes 1 when the p = 0.5 or mostly ambiguous,
we use this as the reliability measure.
When the total expected information gain is smaller
than the threshold and one of the action probability
is over the threshold, it takes that action. Then, it
observes the direction of the largest expected information gain while it is taking the action. Otherwise,
until one of them becomes high enough, it stops walking and continues to check the direction of the largest
expected information gain, updates the observation
probability, and the action probabilities.
2.5

includes the effect of locomotion. Here we define thedifference of images by,X
Dif f (i, j, 4x, 4y) ={ui (x, y)−uj (x−4x, y−4y)}2 ,(6)
where ui (x, y) denotes the value of the pixel at
the point (x, y) in image Ui . We calculate andfind the (4 x(t), 4 y(t))T which makes the smallest Dif f (t, t + 1, 4 x, 4 y) for each t.
(4 x(t), 4 y(t))T is the movement of camera in theimage coordinate. The movement of the camera in
one walking period is,t+WX

To compensate the above two, we calculate the compensation values from images taken by the walking
sequences (Figure 2). At first, the robot is standing
still, at time A it starts walking, and time B is the
beginning of the second walking period.
We denote the time of A, B, C,... as tA , tB , tC ,...,
and, the camera image at time t as Ut . At the beginning of the walking periods, for example, the difference between UtA +i and UtB +i includes the effect ofboth the shaking due to the beginning of the walking and locomotion. When it walks regularly, for example, the difference between UtF +i and UtG +i only

(7)

where, W is the walking period and t is the time in
the period of regular walking. We calculate the meanx̂L and the variance σ L of the movement. The x̂Land σ L indicates the mean and the variance of theits locomotion in the camera coordinate.
The difference of the images between in standing still
and in walking includes the effect of shaking and locomotion. So the effect of shaking at time t calculated by subtracting the effect of locomotion,t
X

Image compensation

We need some kind of compensation mechanism tomake decisions by using the action decision tree built
by statically taken images. First, it should compensate for locomotion which is common for bothwheeled and legged robots. Second, it should compensate for the shaking motion by its walking, thatcan be ignored by wheeled and slowly moving legged
robots. Compensation of camera images for walking is necessary for dynamic walking robots. While
it may possible to compensate the camera movement by active or passive control, high speed feedback or complicated mechanism are needed. Anothermethod is to compensate the image after it has beentaken. With this, we do not need precise control tocompensate its walking. We take this approach.

(4x(j), 4y(j))T ,j=t(4x(j), 4y(j))T −j=0(t − tk )x̂L
,
W

(8)

where tk is the beginning of the walking period which
includes t. We calculate the mean x̂S (θ) and thevariance σ S (θ) of this effect for each walking phaseθ.
We use x̂L , σ L , x̂S (θ), and σ S (θ) for the calculationof observation probabilities. When a landmark i isobserved in x at time t1 , the mean and variance ofthe landmark location in image at time t is,
x̄(t)
σ̄(t)

(t − t1 )x̂L,W(t − t1 )σ L
= σ S (θt1 ) +,W= x + x̂S (θt1 ) +

(9)
(10)

where θt1 is the walking phase at t1 . We use theratio of the area made by x̄(t) − σ̄(t) and x̄(t) + σ̄(t)and the area which is overlapped by the attention
window on this area as the observation probability.

3
3.1

Experiments
Task and Environment

We used a legged robot with a limited view angle for
the RoboCup SONY legged robot league (Figure 3).
We used the half of the field, there are 4 landmark
poles, a goal, and a ball. All the landmarks and theball are distinguished by their colors. The task is to25

Forward 1
Forward 2
Forward 3

20
15[pixels]

10
5
0
-5
-10

Figure 3: The SONY legged robot for the RoboCup
SONY legged robot league.

-15
0

10

20

30

40

50
60
x40[ms]

70

80

90

100

Figure 6: The 4y of forward motion.

20

xs
ys
sigma xs
sigma ys15

[pixels]

10

Figure 4: Experimental field (same as the one for
the RoboCup SONY legged robot league).

5
0
-5
-10

move to the position where the ball and the goal arein line based on the visual information. The robot
must avoid obstacle which is not distinguishable byits color.
The view angle (number of image pixels) of therobot’s camera are about 58 degrees (88 pixels) inwidth, and about 48 degrees (72 pixels) in height.
Each leg and the neck have three degrees of freedom.
The robot can rotate the pan joint from -88 to 88 degrees and the tilt joint from -80 to 43 degrees. The
maximum frame rate of the camera is 40[ms]. We
prepared three actions, move forward, left forward,
right forward. These actions are based on trot gait
and were developed without consideration for image
shaking.
As vision sensors, we used the coordinates of the image centers of the landmarks and the ball, the minimum and maximum x, y coordinates (totally four)of the goals. We used a pair of the pan (x) andthe tilt (y) angles as a sensor value, or we divided
training data set by the observation to check whether
a sensor value is in the rectangle attention window(xmin , ymin ) − (xmax , ymax ) or not.
3.2

Image compensation

Figure 5 shows the images while the robot is watching the front direction and doing forward motion.
The images are taken every 80[ms] and the walking
period is 600[ms]. Figure 6 shows the 4y of three
trials of forward motions. We can see high jumps in

-15
0

2

4

6

8

10

12

14

x40[ms]Figure 7: The x̂S , and σ S of the forward motion
watching at front direction.

every walking periods. Figures 7, 8, and 9 show thex̂S , and σ S of the forward, right forward, and left
forward actions watching at front direction.
3.3

Experimental results

We trained the robot starting from one of three positions in the middle of the field. We prepared five
directions (every 44 degrees) in the pan joint andfive directions (every 21 degrees) in the tilt joint toobserve. In order to guarantee that it can observe
same view angles by observing one direction in spiteof the shaking motions, we narrowed the angles thancamera’s field of view.
We obtained 239 training data and constructed aaction decision tree. Figure 10 shows the attention windows generated with proposed method. Figures 11,12, and 13 show the changes of expected information gain and the maximum action probability when the robot took actions beginning from thecenter of the field based on the constructed action
decision tree.
Figure 11 shows the result when action probability
threshold was 0.4 and the information gain thresholdFigure 5: The images of the camera while the robot is moving forward. The images are taken every 80[ms].

10

80

xs
ys
sigma xs
sigma ys8
6

48
Tilt angle[pixels][pixels]

4
2
0
-2
-4

16

-16

-48

-6
-8

-80
-165

-10
0

2

4

6

8

10

12

14

-99

-33
33
Pan angle[pixels]

99

165

x40[ms]Figure 8: The x̂S , and σ S of the right forward motion watching at front direction.

15

threshold was 0.4 and the information gain threshold was 0.3. It shows that by using the information
gain threshold, the frequency of standing up for observation is higher than without the threshold andlower than high action probability threshold. And
the robot reached to the goal position avoiding the
obstacle. This shows the validity of the information
gain threshold.

xsyssigma xs
sigma ys

105[pixels]

Figure 10: Created attention windows by proposed
method.

0
-5
-10
-15

4

-20
0

2

4

6
8
x40[ms]1012Discussions and conclusions

14

Figure 9: The x̂S , and σ S of the left forward motion watching at front direction.

was not used. The time the robot reached to the goal
position was shorter and the frequency of standing up
for observation was lower. However the robot ignored
the obstacle so the task has not been completed.
Figure 12 shows the result when action probability
threshold was 0.9 and the information gain threshold
was not used. We tried with the threshold 0.7 and
0.8 and it tried to avoid the obstacle but it hit tothe obstacle. With the threshold 0.9 it avoided the
obstacle and reached to the goal position. However,it frequently stand up for observation.
Figure 13 shows the result when action probability

Currently, training data should cover the case of sensor noises or of occlusions in order to handle thesecases, then we may need large training data. A
method which can find robustness of observation may
be desired. We have to determine the TC for calculation of information gain per time, and two thresholds
for maximum action probability and expected information gain for each task. These parameter should
be studied.
We proposed a visual attention control for a legged
mobile robot. It consisted of a decision tree constructed by information gain by time and the compensation mechanism for walking and locomotion.
We introduced the expected information gain as ameasure of reliability of decision. Attention control
is done by observing the direction which has largest
expected information gain calculated with the decision tree. The validity of the method was shown withreaching task by a four legged robot.

Reached
Right forward
Expected information gain
Maximum action probability
Action with maximum probability
The action of the robot

Left forward
Forward
1.8
Stay

1.5
1
0.5
0
0

100

200

300

400
x40[ms]

500

600

700

800

Figure 11: Changes of expected information gain,
the maximum action probability, actions which had
maximum action probabilities, and which was takenby the robot. The starting point of robot was the center of the field. The action probability threshold was
0.4 and the information gain threshold was not used.

probability and expected information gainprobability and expected information gain

Expected information gain
Maximum action probability
Action with maximum probability
The action of the robot

ReachedRight forward
Left forward
Forward
1.8
Stay

1.5
1
0.5
0
0

500

1000

1500
2000
x40[ms]

2500

3000

3500

Figure 12: Changes when the starting point of robot
was the center of the field. The action probability
threshold was 0.9 and the information gain threshold
was not used.

Acknowledgments
This research was supported by the Japan Scienceand Technology Corporation, in Research for the
the Core Research for the Evolutional Science andTechnology Program (CREST) titled Robot Brain
Project in the research area ”Creating a brain.”

References[2] A. Kosaka, M. Meng, and A. C. Kak. Vision-guided
mobile robot navigation using retroactive updating of
position uncertainty. In Proceedings of the 1993 IEEE
International Conference on Robotics and Automation,
volume 2, pages 1–7, 1993.
[3] N. Mitsunaga and M. Asada. Observation strategy
for decision making based on information criterion. In
Proceedings of the 2000 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1038–
1043. 2000.
[4] N. Mitsunaga and M. Asada. Sensor space segmentation for visual attention control of a mobile robot
based on information criterion. In Proceedings of the2001 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1714–1719. 2001.
[5] I. Moon, J. Miura, and Y. Shirai. Dynamic motion
planning for efficient visual navigation under uncertainty. In Y. Kakazu, M. Wada, and T. Sato, editors, In Proc. of the Intelligent Autonomous Systems
5, pages 172–179, 1998.

Expected information gain
Maximum action probability
Action with maximum probability
The action of the robot
probability and expected information gain

[1] W. Burgard, D. Fox, and S. Thrun. Active mobile
robot localization. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence(IJCAI). Morgan Kaufmann, San Mateo, CA, 1997.

ReachedRight forward
Left forward
Forward
1.8
Stay

1.5
1
0.5
0
0

200

400

600
x40[ms]

800

1000

1200

Figure 13: Changes when the starting point of robot
was the center of the field. The action probability
threshold was 0.4 and the information gain threshold
was 0.3.