SICE Annual Conference 2010
August 18-21, 2010, The Grand Hotel, Taipei, Taiwan

Robust Falling-down Avoidance Control for Acrobat
Robot Using Switching Controller
Y.Yashiroand M.Yamakita
Dept. of Mechanical and Control Eng.
Tokyo Institute of Technology
Tokyo, JAPAN
Email: yashiro@ac.ctrl.titech.ac.jp
yamakita@ctrl.titech.ac.jp

S.Hirano

ZW.LuoThe Institute of Physicaland Chemical Research(RIKEN)
Nagoya, JAPAN

Dept. of Computer and Systems Eng.
Kobe University
Kobe, JAPAN

Abstract—In this paper, we propose methods of designingsystematic controller including identiﬁcation to stabilize theacrobat robot with reinforcement learning algorithm. We model
the ﬁnal stage of the robot at falling-down as a switched systemwith few contact conditions. For this model, we can systematicallydesign a SDOF controller with for an unstable LTI system
whose parameters are calculated by a closed loop identiﬁcation.
Considering the change of the dimension of the subsystem, we
add a new subsystem with a reduced dimension to the switchedsystem. we conﬁrm the effectiveness of these methods withexperiments using a real robot system.
Index Terms—Identiﬁcation, Robot dynamics, Switched systems.

subsystems. So we consider a the new subsystem with areduced the dimension, and add to the switched system.
In this study, we conﬁrm the effectiveness of the proposed
method with experiments using a real robot system.
II. C ONTROLLED S YSTEM
In this study, we use a humanoid type acrobat robot KHR1 (produced by KONDO KAGAKU Co.) as a basic system.
It has 17 servomotors , 2-axis gyro and 3-axis acceleration
sensor on its head. It also has micro switches on the soles tosense their contact condition to the ground.

I. I NTRODUCTION
Recently variable constraint systems are focused, and as a
typical case of them, an acrobat robot with a horizontal bar
was studied. For the generation of acrobatic motion with ahorizontal bar, giant swing, taking off, somersault in the airand landing have been realized [3] [4] [5] [6].
And Falling-down avoidance control applying the reinforcement learning algorithm called Q-Learning to the robot
was also realized [7] [8]. It is hard to design systematiccontroller for landing acrobat robot, because it has manycontact conditions to the ground, and actuators of real robot
system don’t move to the same way every time. In this point,
it is effective to use reinforcement learning for acrobat robot.
But this method needs unrealistic number of learning process,which leads to the difﬁculty to realize falling-down avoidancewith the real robot system.
Therefore, we systematically design another controller toimprove the robustness and reduce the number of learning
process. The controller is designed for only the neighborhoodof ﬁnal desired posture of the robot, which has less contactconditions than those in Q-Learning. Since we have to model
the neighborhood as an unstable switched system, we use aclosed loop identiﬁcation, and we design a switched dynamic
output feedback (SDOF) controller to stabilize the system.
This method is, however, not originally considered thechange of the number of the dimension when switching theFig. 1.

KHR-1

A servo motor, KHR-786ICS is used at each joint.

- 2479 -TABLE I
S PECIFICATION OF MOTOR
Size
MassTorqueSpeed

41×35×21[mm]45[g]8.7[kg/cm]
0.17[s]/60[deg]PR0001/10/0000-2479 ¥400 © 2010 SICE

C. Subsystem with Reduced Dimension

III. C ONTROL L AW
Since the dynamics of the system change according toconditions to the ground, we model it as a switched system
whose subsystems have some different dimensions. When
we design a SDOF controller, we need subsystems for eachcontact condition, and subsystems for changing the dimension.

When switching from the subsystem with higher dimension
to that with lower dimension, we assume external force λ(k)andx(k + 1) = A(x) + Bu(k) + B w w(k) + Eλ(k)(6)

E T x(k + 1) = 0

are satisﬁed where Ex(k) = 0 stands for constraints. By
eliminating λ from two equations, we have

A. SDOF Controllerx(k + 1)

We deﬁne the switched system with N subsystemsx(k + 1) = Aα x(k) + Bα u(k) + Bαw w(k),z(k) = Cαz x(k) + Dαz u(k) + Dαzw w(k),y(k) = Cα x(k) + Dαyw w(k) (α = {1, 2, ..., N })= (I − E(E T E)−1 E T )A(x) + (I − E(E T E)−1 E T )Bu(k)(1)+ (I − E(E T E)−1 E T )B w w(k)= A (x) + B  u(k) + B w  w(k).
(7)


w
where x, z, y, u, w are state of system, controlled output, We get new subsystem (A , B , B , C), and when the conobserved output, input, and disturbance. And subsystems can straint is satisﬁed, 1 step of this subsystem is inserted between
the two subsystems.
be switched i → j(i, j ∈ α) at arbitrary steps.
Then, if the LMIIV. M ODELING WITH I DENTIFICATION⎞⎛Since we control the robot with the input to motor angle,
X+X  −Si(·)(·)(·) (·) (·)⎟ not torque, and need to get unstable model from the data,⎜ I+W −MiY +Y  −Ni
(·)
(·)
(·)
(·)
⎟
⎜
from the stabilized closed loop system, we use a closed loop⎜00γI(·) (·) (·) ⎟⎟⎜⎟ identiﬁcation method.
⎜ Ai X+Bi Li Ai +Bi Ri Ci
Biw +Bi Ri Diyw
Sj
(·)
(·)
⎟⎜
⎝
QiY Ai +Fi Ci
Fi Diyw +Y Biw
MjNj(·) ⎠Ciz X+Diz Li>0Ciz +Diz Ri Ci

D z wi +Diz Ri Diyw

00γI((i, j) ∈ α × α)(2)

has the solution(Si , Mi , Ni , X, Y, W, Li , Fi , Qi , Ri , γ), SDOF
controller is given as [1].
η(k + 1) = Aci η(k) + Bic y(k),u(k) = Cic η(k) + Dic y(k)(3)

Aci = V −1 [Qi − Y (Ai + Bi Ri Ci )X− V Bic Ci X − Y Bi Cic U ]U −1 ,cBi = V −1 (Fi − Y B i Ri ),Cic = (Li − Ri Ci X)U −1 ,Dic = Ri ,V U = W − Y X.

A. Parameter of Robot
As shown in Fig.2, we consider the robot 9 links body inthis study. The posture of the robot is deﬁned asΘ = [θ3r , θ4r , θ5r , θ6r , θ3l , θ4l , θ5l , θ6l ][deg].

The position of C.O.G of the robot in Σ0 coordinate iscalculated with the posiion of C.O.G, weight, and relative
angle of each link. The origin of Σ0 coordinate is the center ofthe box made by toes and heels. Thus the position and velocityof the C.O.G of the robot are expressed with vectors
G = [GX , GY , GZ ]T [m], V = [VX , VY , VZ ]T [m/s]. (9)Input and output for identiﬁcation and controller areu = [u1 , u2 , u3 ] = [θ3 , θ4 , θ5 ],y = [y1 , y2 ] = [GX , GZ ] − ydes ,(4)The controller designed by (3) is said to have the γperformance, and satisfy∞
k=0z(k)2 < γ(10)where ydes is the ﬁnal desired position.

B. γ-performanceγ −1(8)∞k=0w(k)2 , ∀∞w(k)2 > 0.

(5)

k=0

This condition means that the L2 gain of the closed loop
system is less than or equal to γ. Therefore, when we design
the SDOF controller, we add the condition of minimizing γ
to LMI solver.

B. Closed Loop Identiﬁcation
We use exogenous input [r1 , r2 ]T , and combined output
p = [y, u]T in the closed loop shown in Fig.3. We get theinnovation model expressed in (11) with the identiﬁcation.
x(k+1) = Ax(k) +- 2480 -p(k) =C1
C2

B1x(k) +r1 (k)B2

r2 (k)+K100

r1 (k)D21

D22

r2 (k)

K2+e1 (k)e2 (k)e1 (k)e2 (k)(11),

x(k+1)

BodyG

θ6l

Link5

θ5l

0x(k)

0

Asxs (k)y(k) =CCsB
0u(k) +0
Ks+ Du(k)+e(k).

2) SR algorithm for unstable system: SR algorithm use thegrammian of the system. When it is unstable, controllable andobservable grammians P, Q are given with Riccati equationsand Lyapnov equation as

θ3l

Link3x(k)xs (k)+If the dynamics of two subsystems coincide, the model canbe expressed in (11).

θ4l

Σ0A(13)

Link4Z=xs (k+1)

X = AT (X − XB[Im + B T XB]−1 B T X)A,Y
Link1+2Y = A(Y − Y C T [Ip + CY C T ]−1 CY )ATX

F = −(Im + B T XB)−1 B T XA,
Fig. 2.

L = −AY C T (Ip + CY C T )−1Model of the robot

(14)

(15)

P = (A + BF )P (A + BF )T + B(Im + B T XB)−1 B T ,Transforming with r1 = e2 = 0, we get the model of thesystems as−1−1C2 ]x(k) + B2 D22u(k) + K1 e1 (k),x(k + 1) = [A − B2 D22y(k) = C1 x(k) + e1 (k).
(12)
We use ORT method, which is one of the subspace identiﬁcation, to calcurate matrices of the model [2]. The dimensionsof the matrices are the sum of the dimension of the plant andthe dimension of the controller. Therefore we need to lower thedimensions using SR algorithm, including deriving grammianof the system and SVD.
e

Q = (A + LC)T Q(A + LC) + C T (Ip + CY C T )−1 C.
(16)
And calculate the SVD of the product SR, where P =S T S, Q = RT R. We compare to the size of each singularvalue of SVD, and we can deside the number to which welower the dimension of the system.
V. E XPERIMENT OF I DENTIFICATION
We model the subsystems for each contact condition withORT method. For this method, we use SDOF controller
designed with the matrices calculated MATLAB command
n4sid. We get the data for n4sid with the real robot system
under stable ﬁnal desired posture, so the system model of thesematrices is also stable.
In ORT method, we use random r1 and random integer r2 ,| r1 |≤ 0.005,

H(z)

r2
r1+ーC(z)Fig. 3.

++uP(z)++y

Closed loop for identiﬁcation

1) ORT method: ORT method gives the good result when
the data for identiﬁcation have colored noise, because thismethod divides the data into deterministic, and stochastic parts
with orthogonal decomposition.
Because of this, the model of the system is expressed in
the combination of deterministic subsystem, and stochastic
subsystems as| r2 |≤ 2(17)as exogenous input. The numbers of data for ORT method
in each subsystem are 1627 for normal contact, and 1123 for
tiptoe contact. The size of calculated matrices is 6 + 6 = 12
: the size of the stable system + the size of SDOF controller.
Considering the existence of feet bottom angle, we choose
and lower the dimension of normal contact condition to 6,
and tiptoe contact condition to 8. The eigenvalues of two
subsystems are shown in Fig.4 and Fig.5. Some eigenvalues
are out of unit circle, so the subsystems modeled with closed
loop identiﬁcaiton are on-target unstable systems. Eigenvaluesof deterministic and stochastic subsystems have some similarities, we express equations in (11).
VI. E XPERIMENT OF SDOF CONTROLLER
We apply SDOF controller designed with closed loop identiﬁcation to the real robot system.

- 2481 -e(k),

2
A1
As1
1.5

1

1

0.5

1 steptiptoe02−0.5

32 step continuous reverse−1or heel on the ground

−1.5

−2
−2−1.5

Fig. 4.

−1

−0.5

00.5

11.5

2Fig. 6.

eigenvalues of normal contact condition

Switching rule of subsystems

4A2As2•SDOF controller is low gained as

3u ← 0.14u (mode = 1)

2u ← 0.20u (mode = 2)
u ← 0.24u (mode = 3)

10because the size of the control input of the originalcontroller designed by the LMI solver which minimizes
γ is too large for the real robot system to keep links andmotors good and unbreakable. We decide the levels oflowering gain and check the stability of the controller
with a simulater, which has the same conditions of thereal system.

−1−2−3−4−2−1Fig. 5.

012345(20)6eigenvalues of tiptoe contact condition

B. Result

A. Condition
Sampling interval of robot is 75 [ms].
Subsystems of the posture we have to stabilize aremode1: normal contact condition
mode2: tiptoe contact condition
mode3: reducing dimension conditionand switching rule among subsystems is shown in Fig.6.
When mode2, we add condition with 2 step continuous
reverse of the robot because of the delay of input to servo
motor.
• We let the robot fall-down with the 2 steps,••

Θ = Θf inal + [−9, 0, 0, 0, −9, 0, 0, 0] [deg] (18)and after that, SDOF controller starts to inject the control
input to the servo motors. SDOF controller stops when
position and velocity of C.O.G reaches the neighborhoodof ydes .
• The ﬁnal desired posture is

The result is shown in Fig.7, where position and velocity ofthe C.O.G and controller mode number are shown, and Fig.8,
where input and servo motor angles are shown. They showthat the robot avoids falling-down and C.O.G and servo motor
angles nearly converge with the ﬁnal desired posture. Fig.9
shows the motion of the real robot system.
VII. E XPERIMENT WITH MOVING PATTERN
The main purpose of SDOF controller is to reduce the number of reinforcement learning process. Sufﬁcient Q-Learning
process can design the moving pattern with 4 posture frominitial posture to ﬁnal desired posture [8]. But with real robot
system, the moving pattern is not perfect because of the shortnumber of learning process, and robot fall-down.
So we use SDOF controller at the end of the moving patternwith 4 posture which cannot avoid the robot from fallingdown, and check the effectiveness the conbination of SODF
controller and Q-Learning.

A. Condition
Θf inal = [−55, 80, −28, 15, −55, 80, −28, −15][deg]Most of the experimental conditions are the same in VI.A,
but in addition to 2 step falling-down, we let the robot moveydes = [−0.003, 0.176][m](19) as- 2482 -GxGz

center of G [m]

0.2
0.1
0
−0.1
00.5

1velocity [m/s]1

1.5
time [s]

2

2.5

3Vx
Vz

0.5

1/7.5[s]3/7.5[s]5/7.5[s]7/7.5[s]9/7.5[s]11/7.5[s]13/7.5[s]15/7.5[s]0−0.5−100.5

1mode number

41.5
time [s]22.5

3controller mode
32
1000.5

Fig. 7.

1

1.5
time [s]22.5

3C.O.G and controller mode number

motor input [degree]

20
u1u2
u3

100−10−20
00.5

1

1.5
time [s]22.5

3

motor degree [degree]

100
theta3
theta4
theta550

Fig. 9.

Movie of stabilization with SDOF controller

0B. Result

−50
−100
00.5

Fig. 8.

1

1.5
time [s]22.5

3

Input and servo motor angles

The result is shown in Fig.10,and Fig.11. They show that
SDOF controller starts to inject the control input from 1.2[s],and the robot avoids falling-down and C.O.G and servo motor
angles nearly converge with the ﬁnal desired posture. Fig.12
shows the motion of the real robot system.
VIII. C ONCLUSION

Θ(k) =⎧⎪[0, 4, 7, −25, −25, 60, −60, 15] if k=1,2,3,4,5
⎪
⎪
⎪
⎪⎪⎪
⎪
⎨[−40, 60, −20, −25, −28, 60, −47, 25] if k=6,7,8⎪⎪[−37, 70, −40, −15, −37, 60, −35, 15] if k=9,10⎪⎪⎪⎪⎪⎪⎩[−50, 80, −29, −15, −50, 80, −29, 15] if k=11,12,13(21)During moving, SDOF controller starts if starting condition(22) is satisﬁed.
k ≥ 11(22)| y1 | ≥ 0.02

In this paper we have proposed methods to avoid fallingdown for an acrobat robot using SDOF controller, and identiﬁcation method to get unstable model from the data got
by stablilized systems. Then we checked the effectivenessof the proposed methods, and robustness of the combinationof reinforcement learning and systematic controller by someexperiments with the real robot system.
In this paper, we use only C.O.G for control output. However, we should use the size of control input too, and SDOF
controller will be more useful for the real systems.
The model is not accurate near the switching point between
the subsystem, so we change the switching rule. And themost effective condition to start SDOF controller is not found.
So we need to improve the accuracy and ﬁnd the proper- 2483 -GxGzcenter of G [m]

0.2
0.1
0−0.1
00.5

1

velocity [m/s]1

1.5
time [s]

2

2.5
VxVz

0.5

1/3[s]

2/3[s]

3/3[s]

4/3[s]

5/3[s]

6/3[s]

7/3[s]

8/3[s]

0−0.5−100.5

1mode number

4

1.5
time [s]22.5controller mode

3
2
100

0.5

Fig. 10.

1

1.5
time [s]2

2.5

C.O.G and controller mode number with moving

motor input [degree]

20
u1u2
u3

100−10−20
00.5

1

1.5
time [s]2

2.5

motor degree [degree]

100
theta3r
theta4r
theta5r

50

Fig. 12.

Movie of moving and stabilization with SDOF controller

0−50−100
00.5

Fig. 11.

11.5
time [s]22.5

Input and servo motor angles with moving

condition to switch subsystems, and starting condition ofSDOF controller.
And we also need to make sure the reduction of reinforcement learning process by the combination with systematiccontroller.

[5] M. Yamakita, M. Kishikawa and T. Sadahiro : Motion Control for Robust
Landing of Acrobat Robot (SMB) Proc. of the IEEE/RSJ Int. Conf. onIntellig ent Robots and Systems (IROS), pp.1141/1146 (2003)[6] T.Rokusho, M.Yamakita, S.Hirano, Z.W.Luo : Landing Control of Acrobat Robot by RHC - Experimental Evaluation -, Proc. of SICE-ICASEInternational Joint Conference 2006, pp.4808-4813, Oct. 2006.
[7] T.Kochiya, M.Yamakita : Falling Avoidance Control of Acrobat Robot byReinforcement Learning, Proc. of SICE Annual Conference 2007, pp.322326, Sept. 2007[8] H.Suzuki, M.Yamakita : Falling down Avoidance Control for Acrobat
Robot by QLearning with Function Approximatation, SICE Annual Conference 2008, pp.836-841, August. 2008.

R EFERENCES[1] J.Daafouz and J.Bernussou : Robust Dynamic Output Feedback
Control for Switched Systems IEEE Conference on Decision andControl,pp.4389-4394,December (2002)[2] T.Katayama, H.Kawauchi and G.Picci : Subspace Identiﬁcation of Deterministic Part of State Space Model Operating in Closed Loop CD-ROM
Proc. 6th European Control Conference, Porto,pp.2505-2510 (2001)[3] M. Yamakita and T. Sadahiro : Realization of Acrobatic Motions by
Super-Mechano Boy(SMB) Proc. of ICASE/SICE Joint Workshop (2002)[4] T. Sadahiro and M. Yamakita : Landing Control of Acrobat Robot (SMB)
Satisfying Various Constraints Proc. of the IEEE Int. Conf. on Robotics
and Automation (ICRA), pp.3017/3022 (2003)- 2484 -