2010 International Conference on Emerging Security Technologies

Learning Arm Motion Strategies for Balance
Recovery of Humanoid Robots
Masaki Nakada∗ , Brian Allen† , Shigeo Morishima∗ , Demetri Terzopoulos†∗ Faculty of Science and Engineering

Waseda University, Tokyo, Japan
Email: masakinakada@akane.waseda.jp† Department of Computer Science

University of California, Los Angeles, USA
Abstract— Humans are able to robustly maintain balance inthe presence of disturbances by combining a variety of control
strategies using posture adjustments and limb motions. Such
responses can be applied to balance control in two-armed bipedal
robots. We present an upper-body control strategy for improving
balance in a humanoid robot. Our method improves on lowerbody balance techniques by introducing an arm rotation strategy(ARS). The ARS uses Q-learning to map sensed state to theappropriate arm control torques. We demonstrate successfulbalance in a physically-simulated humanoid robot, in responseto perturbations that overwhelm lower-body balance strategies
alone.

neglected. The inverted pendulum with a massless leg has
been a very common model for biped research because of
the convenience (e.g., the Linear Inverted Pendulum Model).
This approach has contributed to much research in bipedal
locomotion. However, the human upper body also plays arole in locomotion and the maintenance of balance. Humanmovements such as a forward lunge and arm rotation strategies(ARS) are able to alter the angular momentum in orderto maintain balance. Humans can accomplish useful ARS
behaviors subconsciously.
The focus of this paper is the use of ARS to maintainbalance, thereby increasing the stability of bipedal robots.
Unfortunately, it is nontrivial to calculate the necessary torquebecause of the complexity of the dynamics and random effects
from the environment. The timing and strength of the required
torque are very important. Rather than attempting a physicalcomputation, we use machine learning to determine how toreact appropriately depending on the situation.
Reinforcement learning is useful when we have a specificgoal but are uncertain how to achieve it. Our problem is of
this sort, where the final goal is defined as a stable balance,
but the process to reach the goal is unclear. We employ
the Q-Learning method, which is one of the most popular
reinforcement learning algorithms. Even though this algorithm
is simple, it is powerful enough to find an optimal solution that
achieves the goal.

I. I NTRODUCTION
Balance control is an important topic for humanoid robotics
and is becoming increasingly necessary for humanoid robots
that must function within a human-centric environment. Regardless of the quality of bipedal locomotion, a humanoid
robot must still be prepared for unexpected perturbations that
could throw it off balance. These events are unpredictableand potentially unavoidable, therefore, it is necessary to haverobust controllers for balance maintenance and recovery.
When the disturbance is relatively small, it is sufficientto add torque at the ankles in order to create an angularmomentum that recovers balance. In the face of larger perturbations, however, additional recovery strategies are needed,such as bending at the hips to produce an additional restoring
momentum or taking protective steps to bring the center ofpressure back within the support region.
Balance recovery is difficult, as are other aspects of bipedal
locomotion research. Humanoid robot walking dynamics are
non-linear and high dimensional. Moreover, it is sometimes
impossible to actuate torques when the ground is too rough or
the robot stands on one leg. In the former case, it is hard tocompute when and where to step because the expected force
and torque is ambiguous and it remains a difficult problem torecover from the unstable condition. However, humans andother animals can maintain their balance in spite of these
theoretical difficulties. This implies the existence of good,biomimetic balancing controllers for humanoid robots.
While many researchers have considered the role of thelower body in balance, the upper body has been relatively
978-0-7695-4175-4/10 $26.00 © 2010 IEEE
DOI 10.1109/EST.2010.18

II. R ELATED W ORK
We discuss three areas related to our research: (1) bipedalbalancing, (2) balancing methods that consider the torso’s
angular momentum, and (3) the use of machine learning forbalancing.
Humanoid postural stability has been of research interest fora considerable length of time. The zero-moment point (ZMP)
was introduced forty years ago [1], providing an instantaneous
measure of bipedal balance. When statically stable, the ZMP
is equivalent to the measured center of pressure (CoP) [2].
Hermami and Katbab [3] presented ankle and hip strategies
with an inverted pendulum model. This enabled a push recovery for small perturbation. The stepping strategy presented by165

III. T HE A RM ROTATION S TRATEGY
Human balance strategies can be divided into three categories based on the magnitude of the balance-disturbing
perturbation [12]. When the perturbation is small, balance ismaintained by applying torque at the ankle. In a more severe
perturbation, the hip joint is also recruited. For perturbations
that cannot be balanced by hip and ankle torques, humans willtake a protective step [13].
In response to some perturbations, humans augment lowerbody balancing torques with upper-body motions. Whenpushed, our arms automatically engage in reactive behavior
so that we can maintain the stability of our body. Thisaction is usually automatically created by our neuromuscularsystem; therefore we seldom realize how we react to a certain
perturbation.
One of the reasons that we move our arms is simply tochange the position of our CoM and CoP. If we are leaning
forward on the edge of box, for example, we will try to keepbalance by stretching out our arms backward. This is very
simple way to maintain balance and the arms clearly has animportant role in controlling balance.
As a next stage, assume that we are unexpectedly pushed bysomeone. We will generally apply a balance-restoring torqueat the ankles as was done in prior research as a first attempt tomaintain balance. The ankles play an important role in balance
maintenance; however, it is not the only action that we takeas a first recovery strategy. We also use our upper body androtate our arms to maintain balance. Therefore, this also needs
to be considered in humanoid robots or physically simulatedcharacters in order to improve balance control and yield morehuman-like behavior.
The main purpose of the ARS is to create angular momentum by the rotation and to defeat the angular momentum
associated with falling. The magnitude of the momentum
depends on the speed of the rotation, weight, and length of
the arms. This momentum affects the torso and, in theory, theamount can be calculated through articulated-body dynamics.
However, it is not easy to compute in the real world because of
the high dimensionality and the unexpected nature of external
forces.
Additionally, the timing of the ARS is also very important,because the angular momentum created by a perturbation
changes consistently. Therefore, we need to apply adequate
compensatory torque with specific timing that counteracts thefalling momentum and returns the body to a stable state.
We apply a reinforcement learning algorithm for humanoids
so that they can learn how to rotate their arms depending on
their state and the strength of the perturbations.

Goddard was introduced as a third strategy to prepare for alarger perturbation. These three strategies; ankle, hip, and step,
were combined by Hofmann et al. [4] and they have been used
as a basic strategy for biped balancing.
The linear inverted pendulum model (LIPM) [5] considers abipedal robot with a COM constrained to the horizontal plane;
therefore, the dynamics can be analyzed in one-dimension.
Because of its simplicity, the model has been used in muchresearch on bipedal walking.
Fundamentally, balancing a biped requires maintaining the
CoP within a region of support. Small changes in the CoP canbe directly actuated by applying torques at the ankles. LIPM
has been extended, by considering angular momentum, which
is known as the reaction mass pendulum (RMP) model [6].
Pratt et al. [7] suggest that angular momentum can be
regulated to improve balance, proposing the linear inverted
pendulum plus flywheel model. This model adds a rotational
inertia component to the LIPM. The augmented model has
an enlarged capture region, defined as the area a potential
protective-stepping foot can be placed to keep balance.
Goswami and Kallem [8] quantify the role of angularmomentum in balance with the introduction of the groundpoint with zero rate of change of angular momentum (ZRAM).
In the field of character animation, [9] proposed a linear
and angular momentum controller for both CoM and CoP. This
work focused both on linear and angular momentum and madeit possible to control them simultaneously, leading to morenatural looking results. The arm was lifted when the bodywas pushed off balance by an external force, resulting in anatural human-like behavior.
In this paper, we consider the application of torque atthe shoulder joint of our biomechanical model to create arm
movement, which we expect to change the angular momentum,
thus restoring balance.
Various machine learning algorithms have been applied
to bipedal walking balance. The approach in [10] was todetermine the capture point; i.e. the point where the robot
should step in order to recover balance after a perturbation.
Even though, in theory, they could compute where to step,
it was difficult to realize this computation in a real-time
simulation because of the modeling assumptions and errors;even tiny errors can result in a balance failure. Therefore, they
applied a machine learning algorithm to learn how to adjust
those errors and get the appropriate capture points, leading togreater robustness.
Ito et al. [11] applied a machine learning technique tolearn the torque pattern for balancing. They could eliminate
the process of feedback from the ground reaction force. The
information is essential to maintain balance; however, it is
not easy to calculate the right magnitude. They could ignore
this information by learning to add a torque with periodic
external forces. This learning process yielded a controller foreach situation.

A. The Role of Angular Momentum
As a first step, we consider the case when the disturbanceis not very strong and the ankle strategy suffices to maintainbalance. For this strategy, we show the importance of angularvelocity by considering the total torques on the body,
TCoM =
Text + Ci + roc × Fi = 0,
(1)

166

where TCoM is an torque added to the center of mass, Text
is the sum of the torque by external force, Ci is the inertia
couple, and roc is the distance from origin to the CoM. Fi is
the inertial force.
If we restrict our consideration to two dimensions, we canderive the following formula from equation 1:
xCOP = xCOM +ḢG − yG L̇xyF Fdist+L̇y + mgL̇y + mg

Then, we can rewrite Equation 3 as,





∧
vB
mE −mroc
LG
=
HG
0
I
ωB


2
 Mlegiθ̇legi
+
Hlegi
i=1

2 

Marmi
θ̇armi
+
Harmi
i=1


Mf reeθ̇f ree+Hf ree(2)where xCOP and xCOM are the position of CoP and CoM,
respectively, ḢG is the rate of change of the angular momentum around CoM, L̇x and L̇y are the x and y components ofthe time derivative of the linear momentum, and g is gravity.
This equation shows we can change the position of the
CoP by changing ḢG . This means the time derivative of
the angular momentum, which is the rate of change of theangular momentum around the CoM, can make difference ofthe position of the CoP.
When we control the balance using an ankle strategy, thisCoP is a very important constraint. It must be inside of theregion of support. Moreover, this determines whether or notbalance can be maintained. Therefore, good control of the
CoP is necessary and Equation (2) shows angular momentum
around CoM affects the position of the CoP. Now, we knowthat the rate of the change of angular momentum is thefactor of balancing, which means we must take the angularmomentum into consideration when we investigate humanoidbalance.
Additionally, this equation shows that increasing ḢG decreases the effect of Fdist . Therefore, it is better to rotate thebody in the direction of the disturbance as fast as we can if
we want to minimize the effect of the disturbance.
When our body is in motion, the linear momentum L andangular momentum H can be written as
⎡
⎤

 
 v
∧
LG
Mθ̇ ⎣ B ⎦mE −mroc
ωB
=
(3)
HG
0
I
Hθ̇
θ̇

From this, we can derive the following equation for HG :
HG

+

2


Hlegi θ̇legi +

2


2


Harmi θ̇armi

i=1

Hf ree θ̇f ree

(7)

i=1

This equation shows that when θ̇armi changes, HG alsochanges depending on the value of θ̇armi . This means thatthe joint speed of the arms affect angular momentum. That is
to say, ARS can control the angular momentum around theCoM. As we explained in the previous section, HG is a key
element for balance; therefore, Equation (7) shows the ARS
can contribute to bipedal balancing.
Goswami and Kallem [8] considered the robot as rotationally stable when ḢG = 0, thus we use this as a control target
for the robot.
1) Application of Angular Momentum: In the previous
section, we could prove that principled changes in angularmomentum are useful for bipedal balance. In this section, weshow how the torque generated by the momentum works inbalancing. We assume that the torque at the center of mass isτ , which is generated by angular momentum with ARS. We
begin with a simple planar model. The equations of motion
are represented as follows:τmẍ = fR sin(θ) − cos(θ)
(8)
l
τ
(9)
mz̈ = −mg + fR cos(θ) + sin(θ)
lJ θ̈ = τ(10)where the fR is the reaction force from the ground, m is
the total mass of the model, θ and J is the tilting angle androtational inertia of the body, respectively, g is the gravitational
acceleration constant, x and z are the position of the CoM
coordinates and l is the half length of the body.
Even though we did not use the simplified model in our
experiments, let us consider the Linear Inverted Pendulum
Model to illustrate the theory. Using this abstracted model,
we can suppose z is a constant value, z = z0 and z” = 0,
because it does not move to z direction. Then, we can solve
for fR from equation (9) asMθ̇ = {Mleg1 , Mleg2 , Marm1 , Marm2 , Mf ree }(4)

where Mlegi and Harmi are the linear and angular inertia
matrix of leg and arm, and Mf ree and Hf ree are the matrixof the composite of the rest of body parts, respectively. We
can also divide the joint speed vector as follows:
TT
TTθ̇ = {θ̇leg1, θ̇leg2, θ̇arm1, θ̇arm2, θ̇fTree }= IωB +

i=1

where E is the 3 × 3 identity matrix, m is a total scalar massof the robot, roc is a vector representing the distance frombase to CoM and a∧ represents the skew-symmetric matrix ofvector a, I is the 3 × 3 inertia tensor with respect to the CoM,Ṁθ and Ḣθ are 3 × n (where n is the number of degree of
freedom) linear and angular inertia matrices.
If we divide each inertia matrix to each part of the body,
we can suppose the following relationship:Hθ̇ = {Hleg1 , Hleg2 , Harm1 , Harm2 , Hf ree }

(6)

fR =

(5)

167

mg
1
− tan(θ)τ
cos(θ)
l(11)

Replacing cos(θ) = zl0 and sin(θ) = xl , Equation (11) can be
written as
mg
1 x
fR =
l−
τ
z0
l z0

Q(a, s) ←

1
g
x−
τ
z0
mz0

This is called the Q value and the agent choose the action
so that it can maximize this value. The parameter α is called
the learning rate and it determines to what extent the newly acquired value affects the old value. The γ is called the discountfactor and it determines how important the future rewards are.
As this value increased, the future rewards are more carefully
considered. These values must be tuned depending on theintent of the learning. This significantly affects the learning
result. The decision-making function is called policy and Qlearning is the algorithm which can find the best policy within
the given environment. The policy is stochastic; therefore, ittakes some random action in a certain period. This prevents
the agent from sticking in a local minimum.

(12)

(13)

This equation shows that the value of τ changes the acceleration of the CoM. Therefore, we can accelerate the body bychanging the amount of torque. Also, from Equation (10) we
can derive the following equation:
θ̈ =

1
τ
J

(15)

α

Next, we represent the fR with constant values, x and τ . We
can substitute fR in the equation (8) and we obtainẍ =Q(a, s) + α [R(s)+ γ max
Q(a , s ) − Q(a, s)](14)

IV. E XPERIMENTAL M ODEL AND E NVIRONMENT

which shows that the torque can change the acceleration of therotation of the body. Equations (13) and (14) show the torquearound the CoM affects the balancing of the body. This torqueis originally from the ARS; therefore, we can determine theeffect of the ARS to the balance of the body in this way.

The Open Dynamics Engine (ODE) [16] was used tosimulate the humanoid robot. Although ideal validation would
use a physical robot, humanoid robots can be expensive anddifficult to test without risk of damage or injury. On theother hand, simulation has the benefits of easy, safe andinexpensive prototyping. Further, measurement and analysisof the resulting motion is precise.
The system is simulated with realistic values for gravity(9.8 m/s2 ) and friction. The simulated robot is of roughly
human proportions and mass and bilaterally symmetric hands,
arms, legs and feet. The specific dimensions of the model areprovided in table I.

B. Reinforcement Learning
Machine learning is one of the most popular areas becauseof its potential uses in numerous fields. Many machine learning
methods had been introduced. One way to categorize themis if they are supervised or unsupervised. Reinforcement
learning [14] is an unsupervised machine learning method.
One big advantage of reinforcement learning is that we only
need to set the final goal and the remainder of the process canbe learned by the agent itself. This is useful when the goal
and the method are defined but the process is unknown.
In this work, we have a specific goal, which is to keep
the body upright, but the best method to obtain this result is
unclear. It is not easy to compute the exact torque and thetiming of the reactive movement needed to recover balance.
Therefore, we apply the reinforcement learning algorithm toour model, allowing it to learn how to maintain balance undera given perturbation, action settings, and environment. We
expect it to learn an optimal process to reach its goal andrealize balance control.
1) Q-Learning: We have chosen a common reinforcement
learning algorithm, Q-Learning [15], based on the temporal
difference approach. Q-Learning has an agent as a main
character for the problem and an environment in which theagent can act and the state transitions is described.
When the agent performs an action in the environment,
the state is changed. Then, the environment evaluates if thechanging state was beneficial in achieving the final goal, andwill then return a reward to the agent. Considering the action,state and reward, the agent decides which action it would takeas a next step. This process is repeated until it reaches a final
goal. In the end, it learns the best way to accomplish the goal.
The way they evaluate the reward is based on the following
formula:TABLE IP HYSICAL PARAMETERS OF THE SIMULATED HUMANOID ROBOT.

Torso
Upper Arm
Forearm
Thigh
Shin
Foot

Mass (kg)
10
0.1
2
5
0.1
0.5

Length (cm)
50
0.1
50
50
0.1
20

Width (cm)
30
0.1
5
15
0.1
10

DoF
0
2
0
0
1
0

A. Learning Parameters
As described in section III-B.1, the feature states, learning
rate, discount factor, and reward function are important components, and they must be specified carefully. The results canbe totally different depending on those factors. The parametersettings in our experiments are as follows:
1) Feature States: These are the values which are evaluatedto determine whether the trial succeeded or failed. They canalso be used in the reward function to define how much reward
should be given to the agent.
We chose four parameters that we regard important as away to judge if the posture of the robot is balancing. The four
elements are the position and the velocity of the center ofmass, the leaning angle, and the angular velocity of the body.

168

We also defined the limits for each parameter so that we
can define if the trial is a success. When one of the featurestates exceeds the limits, it is considered a failed trial and thelearning process is reset. In our work, we judged the result as
successful when every value for these 4 parameters are within
the defined limit.
We also need an discretized table to store the Q values andupload them considering the future rewards. Q-Learning is
one of the derivative-based methods from Temporal Difference
approach. This approach considers a future result; therefore, itis necessary to keep the temporal result first, and then upload
it reflecting the result of the future action.
Position, velocity, and angle are the continuous values;
therefore, we need to discretize the space so that we can store
the temporal value obtained in a certain state.
2) Learning Rate: This rate specifies how quickly thesystem should incorporate new information and can range from0, implying no learning, to 1 which would cause the agent torely only on current information. We use a learning rate of
α = 0.1.
3) Discount Factor: This parameter specifies how expected
future rewards are treated, with values of 0 ignoring future
rewards and 1 foregoing immediate rewards in favor of only
considering long-term rewards. Our system specifies γ = 0.9.
4) Reward Function: The reward function has three possible values based on the instantaneous state of the system.
Failure to maintain balance, detected by feature values outside
an allowed, predefined range, results in a large negative rewardof −4.0. Feature values within the allowed range imply thatthe robot is balancing successfully, and a small positive rewardof 2.0 is returned. Finally, if the robot is balancing, and thelinear and angular velocity are also sufficiently close to zero,then the robot is in a very stable state and a more positive
reward of 3.0 is given.

Fig. 1. This is a result of trials with and without ARS when perturbation
comes only from back side of the model to forward. We changed themagnitude of the perturbation and got how extent the model can sustain its
balance. ARS enlarged the range of perturbation impulses.

tricky parts in reinforcement learning; therefore, we had totry different combinations in order to find the best learningparameters. The learning process took approximately two
minutes for each trial. This computational speed is acceptablebecause the process is off-line.
B. Quantitative Evaluation
Figure 1 shows graph of the range of the perturbation
impulses limited to 1 direction. As a first experiment, welimited the direction of the force only to the forward and triedwith different forces changing every 1 N*s(Newton*seconds).
We push our model from the back and measure how strongly
it can resist the perturbation. The graph at the left shows theresult with ARS and the one at the right shows the result
without ARS. As we can see in the graphs, the range of theperturbation impulses is enlarged when we added ARS. This
result proves the effectiveness of ARS.
As a next step, we expanded the strategy to 2D space.
We applied different perturbation forces from different
directions—every 30 degrees with different forces. Figure 2
shows the result of the simulation. As shown in graph, therange of perturbation impulses is expanded by using our
method. ◦ and  shows the maximum perturbations that the
robot could withstand without losing balance with and without
ARS, respectively. The distance from the center is equal to thestrength of the perturbation and the angle from the horizontal
axis is the direction of the perturbation. The most effective
direction of our method was when the force was from 0, 90,
180, and 270 degrees. The other angles were not very different
between the two strategies; however, it was slightly improved.
Our method was robust to perturbations between 1.01 and1.14 times the force required to disrupt the standing balance
controller. This result shows that our strategy improves bipedalbalancing.
We were able to improve the existing balancing method byusing ARS. Our new method can be embedded between theankle and hip controllers in the conventional push recovery
strategy. We can consider our new strategy as a second step,

V. R ESULTS
Performance of the algorithm is measured in simulation,
as described in section IV. Successful balance behaviors
were verified visually. Such manual verification was needed
because the simple pre-defined range of feature values used
for reward did not always imply true success, as the learning
system sometimes found unexpected solutions. For example,
the system could learn to stay upright by bracing itself with
one hand on the ground. While such states qualified as
success for the learning reward, they were not counted as
successful balancing strategies. Such outcomes were rare, but
their existence required the manual validation of balancing
behaviors.
A. Learning Results
We considered the learning of bipedal balance maintenanceusing the Q-learning algorithm. We set the environment, states,
parameters and feature values as we explained in the previous
section and executed trials until the robot kept balance orfell down. The different parameter settings and the featurestates resulted in different outcomes. This tuning is one of the169identical to the real world. The real world has random wind,energy loss, and some other unexpected factors. Therefore, thenext step should be to to validate our strategy by implementing
it in physical anthropomorphic robots.
R EFERENCES[1] M. Vukobratović, A. Frank, and D. Juricić, “On the stability of bipedlocomotion.” IEEE transactions on bio-medical engineering, vol. 17,
no. 1, p. 25, 1970.
[2] P. Sardain and G. Bessonnet, “Forces acting on a biped robot. center ofpressure-zero moment point,” IEEE Transactions on Systems, Man, andCybernetics, Part A: Systems and Humans, vol. 34, no. 5, pp. 630–637,
2004.
[3] H. Hemami and A. Katbab, “Constrained inverted pendulum modelfor evaluating upright postural stability,” Journal of Dynamic Systems,
Measurement, and Control, vol. 104, p. 343, 1982.
[4] A. Hofmann, M. Popovic, and H. Herr, “Exploiting angular momentum
to enhance bipedal center-of-mass control,” IEEE Trans. Rob. Autom,
2007.
[5] S. Kajita, F. Kanehiro, K. Kaneko, K. Yokoi, and H. Hirukawa, “The 3D
Linear Inverted Pendulum Mode: A simple modeling for a biped walking pattern generation,” in Proceedings of the IEEE/RSJ International
Conference on Intelligent Robots and Systems, 2001, pp. 239–246.
[6] S. Lee and A. Goswami, “Reaction mass pendulum (RMP): An explicit
model for centroidal angular momentum of humanoid robots,” in 2007
IEEE International Conference on Robotics and Automation, 2007, pp.
4667–4672.
[7] J. Pratt, J. Carff, S. Drakunov, and A. Goswami, “Capture point: A
step toward humanoid push recovery,” in Humanoid Robots, 2006 6thIEEE-RAS International Conference on, 2006, pp. 200–207.
[8] A. Goswami and V. Kallem, “Rate of change of angular momentum andbalance maintenance of biped robots,” in IEEE International Conferenceon Robotics and Automation, vol. 4. Citeseer, 2004, pp. 3785–3790.
[9] A. Macchietto, V. Zordan, and C. Shelton, “Momentum control forbalance,” ACM Transactions on Graphics (TOG), vol. 28, no. 3, p. 80,
2009.
[10] J. Rebula, F. Canas, J. Pratt, and A. Goswami, “Learning capture point
for improved humanoid push recovery,” in IEEE-RAS 7th International
Conference on Humanoid Robots, Pittsburgh, PA, 2007.
[11] S. Ito, K. Moriki, H. Kawasaki, and M. Sasaki, “Robot experiment of
torque learning for biped balance with respect to periodic external force,”
in Advanced Robotics, 2005. ICAR’05. Proceedings., 12th International
Conference on, 2005, pp. 418–423.
[12] A. Alexandrov, A. Frolov, and J. Massion, “Axial synergies during
human upper trunk bending,” Experimental Brain Research, vol. 118,
no. 2, pp. 210–220, 1998.
[13] B. Maki, W. McIlroy, and S. Perry, “Influence of lateral destabilizationon compensatory stepping responses,” Journal of biomechanics, vol. 29,
no. 3, pp. 343–353, 1996.
[14] R. Sutton and A. Barto, Reinforcement learning. MIT Press, 1998.
[15] C. Watkins and P. Dayan, “Q-learning,” Machine learning, vol. 8, no. 3,
pp. 279–292, 1992.
[16] R.
Smith.
Open
dynamics
engine.
[Online].
Available:
http://www.ode.org/

Fig. 2. This is a result of trials with and without ARS when perturbation
came from various direction. We changed the magnitude and direction of theperturbation and got how extent the model can keep its balance. ARS enlarged
the range of perturbation impulses.

yielding a new 3 + 1 approach for push recovery.
VI. C ONCLUSION
We have proposed a new arm rotation strategy (ARS) forrecovering balance after a perturbation, potentially leading
to more stable bipedal robots working in collaboration withtraditional lower-body strategies. While simple controllers forupper body (e.g., ankle and hip PD controller) work well, no
one has suggested simple and effective controller for lower
body. Our strategy has improved the robustness to the severityof the disturbance and the complexity of the ground. It is
more robust than traditional ones because it controls the upper
body. When the terrain is rough, it is not easy to control the
lower body. For instance, the torque at the ankle does notefficiently affect the body. Moreover, finding the capture points
for stepping turns out to be extremely difficult. By contrast, thedifference of the ground condition does not affect our strategy.
Therefore, this strategy is effective in any situation and resultsin more robust balancing than the one without our method.
In our work, the Q-learning process was off-line; therefore
we did not need to pay much attention for the computationalcost. Even though the learning process can be computationallyexpensive, once a controller is found, the on-line execution isvery fast.
We have employed simulation to test our method and havedemonstrated the utility of ARS. We have simulated the robot
and its environment realistically, but our simulator is still not170