Proceedings of the 2018 IEEE
International Conference on Robotics and Biomimetics
December 12-15, 2018, Kuala Lumpur, Malaysia

Pseudo-Locomotion Design with a Quadrotor-Assisted Biped Robot
Azumi Maekawa1 , Ryuma Niiyama2 , and Shunji Yamanaka3

Abstract— Real-world mechanical characters play an important role in our daily lives and provide rich experiences. This
study presents a novel approach to develop a biped robot thathas the ability to display walking appearance with dynamic
mobility, regardless of its environment. We utilize a quadrotor
as an overhead tether to provide mobility and stability. Using
the thrust of the rotors, we developed a biped robot forentertainment purpose. It is able to move agilely using novel
foot trajectory generation. This method focuses on the velocityof the robot’s foot that is in contact with the ground. The
control policy is designed to generate in the robot’s legs suchthat it reduces the velocity of the foot in contact with the groundto zero. The policy used for the foot trajectory generation istrained by deep reinforcement learning with a physics-based
simulator.

I. INTRODUCTION
There has been a desire to create mechanical charactersthat are capable of life-like motions and behaviors. These
characters play an important roles in animatronics in themeparks and as electro-mechanical toys, and robotic pets.
However, though the progress in modern robotics technology
has been remarkable, there are still many practical difﬁculties and problems remain with regard to creating nuanced,
high-quality character motions akin to those in the virtualenvironment.
This study focuses on a bipedal walking motion, which
is one of such difﬁculty. The ability to locomote freely is
an important element for a legged character. The bipedal
locomotion problem continues to exist even today, as seenin the DARPA Robotics Challenge[1]. Realizing movement
in a bipedal robot akin to that of a living thing is oneof the biggest challenges so far, and many studies havebeen conducted in this area. In addition to control problems,
the process of creating these robots is not only very timeconsuming, but also requires considerable resources including manpower. However, it is possible to provide the desired
visual experience while avoiding these problems by focusing
on the appearance features.
Ghassami et al. proposed a bipedal robot prototype thatused the buoyancy of a helium ﬁlled balloon[2]. This approach enabled various motions, such as walking, jumping,and dancing. It stabilized the robot intrinsically and prevented it from falling. However, the technique could not
1 Azumi Maekawa is with the Graduate School of Engineering,
the University of Tokyo, 4-6-1 Komaba, Meguro-ku, Tokyo, JAPANazumi@star.rcast.u-tokyo.ac.jp

2 Ryuma Niiyama is with the Graduate School of Information Scienceand Technology, the University of Tokyo, 7-3-1 Hongo, Bunkyo-Ku, Tokyo,
JAPAN niiyama@isi.imi.i.u-tokyo.ac.jp
3 Shunji Yamanaka is with Institute of Industrial Science, theUniversity of Tokyo, 4-6-1 Komaba, Meguro-ku, Tokyo, JAPAN

yam@iis.u-tokyo.ac.jp

978-1-7281-0377-8/18/$31.00 © 2018 IEEE

Fig. 1.

Appearance of the bipedal robot using a quadrotor.

confer dynamic motions since it was greatly inﬂuenced by
air resistance and wind.
Some approaches use mechanical structures consisting ofgears and link structures[3][4][5], cams and crank-sliders[6]to realize the appearance of walking motion in a robot
intended for entertainment. Although users can design anarbitrary motion trajectory of the robot’s end effectors byusing these methods, it is limited to a cyclic and static
motions, and stable locomotion skill is lacking in most cases.
Megaro et al. proposed an interactive design systemthat could generate robots to locomote stably with arbitrary, user-created morphologies[7]. Bharaj et al. proposed
a computational design method that allows casual users tocreate personalized walking toys using pre-deﬁned templatemechanisms[8]. Although these methods enable users todesign a variety of robotic creatures, the motion of the robot,
especially the bipedal locomotion, is relatively slow andlimited to quasi-static motion.
Our goal is to develop a robot that has the ability to display
walking appearance with dynamic mobility, regardless of
its environment. This study refers to this motion pseudolocomotion. This paper presents an approach that utilizes
a quadrotor as an overhead tether to provide mobility andstability. This approach aims to enable a rich physical expression by allowing the robot to enhance its maneuverability. To
achieve pseudo-locomotion with the biped robot, we developed a novel foot trajectory generation method using deep

2462

Authorized licensed use limited to: University of Tokyo. Downloaded on August 08,2020 at 06:25:54 UTC from IEEE Xplore. Restrictions apply.

reinforcement learning (DRL). The details of this method are
discussed in the following sections.
II. SYSTEM OVERVIEW
A. Hardware Design
The robot consists of a quadrotor and legs. This study
used DJI Phantom 3 Advanced as the quadrotor. It is oneof the a consumer unmanned aerial vehicle (UAV), and its
onboard processing unit incorporates an inertial measurement
unit (IMU) for attitude control and a distance sensor thatmeasures the current height from the ground. The weight isapproximately 1280 g, and the ﬂight time is up to about 23
min with no payload.
Figure 2 shows the overview of the leg unit. In this paper,
we designed simple hardware for simplicity with regard to
fabrication and control and to enable a convenient evaluationof our approach. The leg unit consists of 3D printed link partsand off-the-shelf servo motors. The 3D printed parts wereprinted on Aspect RaFaEl300 using ASPEX-PA material.
The material is a kind of nylon. It has excellent toughness,and it is possible to print parts with high ﬂexibility dependingon the structure. B and D in Figure 2 are 3D printed partsand have relatively high rigidity. E, which is the foot, has
a ﬂexible structure and can deform easily according to theapplied force, and D and E are fabricated as one part. Eachleg has two degrees of freedom (DOF) and employs Kondo
KRS-2572 HV, which is a position-controlled servo, one foreach DOF (A and C in Figure 2).
The rotation axes of the servo motors were designed to beparallel to each other and to allow the foot to move in thesagittal plane. Figure 2 shows the leg posture when the angle
command to the motor is 0 rad. In this paper, we deﬁned thebending direction of the knee in the direction opposite to
the heading, inspired by the appearance of a bird’s leg. The
range of motion in each servo motor was -1 rad to 1.5 rad.
The weight of the leg unit was 206 g, and the total weight of
the robot was about 1870 g, including the part that connects
leg units and quadrotor.
B. Motion Control

Fig. 2.

Components of the leg unit.

by deﬁning the robot model according to the properties of therobot’s legs. We deﬁned the robot model as shown in Figure

Fig. 3. Robot model used for policy training in the simulation environment.

We focused on the velocity of the foot in contact with theground. In biped locomotion, a gait is generated by each leg
repeating the transition between a stance phase and a swing
phase. One of the features of walking appearance is that the
speed of the grounded foot is zero.
This feature is often used as a constraint when considering
natural walking behavior in computer animation[9][10]. To
achieve this feature, it is necessary to control the swing
leg step properly while the stance leg maintains contact
with the ground as the movement of the quadrotor changes
successively. The design of the control system becomes
a complex and daunting task with the increasing of the
DOF. This study therefore proposes a gait generation method
focusing on the grounded foot state by using DRL. We
trained the control policy in the simulation environment
based on the conﬁguration of the hardware design since DRL
takes a considerable amount of time for learning. We started

3. We used MuJoCo [11], which it is widely used for DRL,
for the simulation environment. In the MuJoCo environment,
the robot conﬁguration is represented by MJCF, where themain part is an XML tree. We modeled the quadrotor asa body part assuming that the height from the ground isconstant and the rotation during movement is negligible.
The body was set at the position where the feet touched
the ground when the command to the actuators was 0 rad.
The leg unit comprised the position control actuators andthe rigid links. The length of both links of the leg is set to0.27 m based on the hardware design. The foot, which is thecontact part with the ground, was assumed to form an arc
shape of radius 0.75 × 10−3 m, which was used to simplify
the contact calculation.
In reinforcement learning problems, the robot observes thestate s and takes the action a according to a policy in order to2463
Authorized licensed use limited to: University of Tokyo. Downloaded on August 08,2020 at 06:25:54 UTC from IEEE Xplore. Restrictions apply.

Algorithm 1 Calculate rewardα, β,  > 0
if 0 ≤ vbody <  thenif left foot and right foot are in contact with the groundthenreward ← (α − (vf ootL + vf ootR ))elsereward ← −β
end ifelse
if left foot is in contact with the ground and right footis not thenreward ← (α − vf ootL )else if right foot is in contact with the ground and leftfoot is not thenreward ← (α − vf ootR )else if left foot and right foot are in contact with theground thenreward ← (α − (vf ootL + vf ootR ))elsereward ← −β
end ifend ifmaximize the sum of the reward. The state s consists of theroot body’s velocity vbody and current angles of each joint.
The action a from the policy speciﬁes the target angles ofactuators at each joint and is used to compute torques to be
applied to each joint. The policy was queried at 20 Hz, andthe observed state space was mapped to the action space.
The reward was designed based on the robot’s feet state as
shown in Algorithm 1. The reward described in Algorithm
1 indicates that the following states are desired: 1) when the
body stops, both feet are in contact with the ground, and the
speed of the each foot is 0 m/s, 2) when the body is moving,
one foot is or both feet are in contact with the ground, andthe speed of the foot in contact with the ground is 0 m/s.
A penalty is applied when both feet are not in contact withthe ground regardless of the velocity of the body. We set thehyperparameters as follows  = 1 × 10−3 , α = β = 1.

Algorithm 2 vbody in each episodeinitialize vbody
X ∼ U (a, b), Y ∼ N (μ, σ 2 )for i = 0, 1, 2, · · · , do
if i mod k thenvbody ← vbody + X
clip(vbody , 0, vmax )else
vbody ← vbody + c × Y
end ifend for

III. EXPERIMENTS
To verify the proposed method, we conducted the experiments on policy training and real robot control.
A. Policy Training
We used Proximal Policy Optimization (PPO) [12], a stateof-the-art algorithm in continuous state-action space control,
for the policy training. The policy network architecture
consisted of 2 fully connected hidden layers, and each layer
has 64 hidden units. A hyperbolic tangent was used as theactivation function of these layers. We used Adam[13] as an
optimizer, and the hyperparameters are shown in Table I. We
set an episode as 500 steps.
TABLE IH YPERPAREMETERS USED IN THE TRAINING POLICY EXPERIMENT
Hyperparameter
Learning rate
Num. epochs
Minibatch size
Discount rate
Clipping Value
3 × 10−4
10
64
0.99
0.2

The policy was trained under the following conditions.
1) The body has one degree of translation freedom (x axis
in Figure3).
2) Each leg has two degrees of rotation freedom.
3) vbody changes according to Algorithm 2.
4) The robot gets the reward according to Algorithm1.
Algorithm 2 represents a randomly changing vbody foreach step k, and X and Y represent the disturbance noise
to allow the robot to acquire a robust policy. U (a, b) andN (μ, σ 2 ) respectively represent uniform distribution in [a, b]
and a Gaussian distribution with mean μ and variance σ 2 . We
set a = −0.3, b = 0.3, vmax = 1.8, c = 0.05, μ = 0, σ 2 = 1
empirically.

Fig. 4.

Learning curve for policy training in the simulation environment.

Figure 4 shows the learning curve of the training. It showsthat the learning has converged at about one million samples.
The theoretical maximum value of the return that could beobtained in an episode was 500. Thus, the results show that

2464
Authorized licensed use limited to: University of Tokyo. Downloaded on August 08,2020 at 06:25:54 UTC from IEEE Xplore. Restrictions apply.

Fig. 5.

Foot trajectories at (a) vbody = 0.5 m/s, (b) vbody = 0.75 m/s, (c) vbody = 1.0 m/s (5.0 m ≤ x ≤ 10.0 m)

about 70 % of the theoretical value could be obtained withthe learned policy.
Figure 5 shows the trajectory of the foot when the bodywas subjected to constant velocities (0.5, 0.75, and 1.0 m/s)
with the learned policy. The x position is from 5 to 10
m, where the velocity of the body was in the steady state.
We found that the velocity of the foot in contact withthe ground was almost 0 m/s, and pseudo-locomotion was
generated. The results also show that the stride and walking
cycle were changed according to the body’s velocity, thus
mimicking nature; it is known that a positive correlation
exists between movement speed and stride in many living
creatures. Therefore, the results of this experiment indicatethat the obtained policy follows one aspect of gaits in living
things.

Fig. 6.

Control system in the real-world environment.

B. Real World Environment
The real-world environment necessitates measurement of
the motion of the quadrotor with sensors so as to estimate
the body’s velocity. In this paper, we used motion capture
cameras, OptiTrack Flex 3, to fulﬁll this requirement. The
command to provide the input to the actual servo motors wascalculated by the learned policy using the estimated velocityof the quadrotor. However, it is difﬁcult to precisely simulate
properties of an actual servo motor, such as response speed,
temperature characteristics, disturbance, etc. Therefore, even
if the same input is given, the identical result is not necessarily returned between the MuJoCo motor and the actual
motor.
In this experiment, we constructed a control system, as
shown in Figure 6, and controlled the real robot with thefollowing steps. The velocity of the quadrotor was estimatedfrom the tracking data obtained from motion capture cameras
and inputted into the learned policy. The policy returned the
optimized action, and the robot’s pose in the next step inthe simulation environment was computed using this action.
Then, angle positions of the robot model’s leg joints were
inputted to the real servo motors. By repeating this step,
the robot’s leg motion could be controlled continuously
in the real-world environment, and pseudo-locomotion was
generated according to the velocity of the quadrotor in realtime. The bipedal motion generation was executed in externalCPU, and servo motors connected in a daisy chain were
controlled with the microcontroller, Arduino MEGA, using
serial communication. In this experiment, the quadrotor wascontrolled manually using a remote. Figure 7 shows thegenerated gait of the real robot. The result of the experiment

2465
Authorized licensed use limited to: University of Tokyo. Downloaded on August 08,2020 at 06:25:54 UTC from IEEE Xplore. Restrictions apply.

Fig. 7.

Generating pseudo-locomotion experiment on hardware.

shows that pseudo-locomotion was generated even in theactual robot, as shown in Figure 7. We found that the systemenabled the user to choreograph the walking motion of therobot interactively in real time by controlling the quadrotor
movement.
IV. CONCLUSION AND FUTURE WORK
We proposed a biped robot for intended for entertainment
using a quadrotor. This robot can move agilely and generated
pseudo-locomotion in real time using the control policy,
wherein the velocity of the foot in contact with the ground iszero according to the velocity of the quadrotor. The control
policy is optimized with a deep reinforcement learning based
method. Our method allows the robot to automatically generate a foot trajectory that realized the appearance of bipedal
walking with mobility and stability. Thus, our approach has
the potential to realize a virtual reality experience in the realworld by enabling the physical movement, which has been
impossible so far due to the constraints of the mechanism
and real-world properties.
Presently, the motion is limited to velocity in one direction and the walking motion occurs in the sagittal plane.
Therefore, increasing the DOF of the body and legs will
enable various motions. It is possible to adjust the generated
trajectories slightly with the current method by tuning thehyperparameters. However, it is difﬁcult to make the movements feature the desired motions. Thus, our future work willattempt to create a control system that realizes the desired
trajectories of the biped robot using multirotor. In this study,
the leg was controlled by the off-board controller, and power
was supplied externally using wires. In order to obtain highmobility, it is necessary to use a load battery and an onboard
controller. Furthermore, in this study, we controlled the robot
in the real world using a kinematics-based approach. In the
future, We will consider the dynamics of the system andattempt to allow the robot to acquire various motions.

R EFERENCES[1] E. Krotkov, D. Hackett, L. Jackel, M. Perschbacher, J. Pippine,
J. Strauss, G. Pratt, and C. Orlowski, “The darpa robotics challenge
ﬁnals: results and perspectives,” Journal of Field Robotics, vol. 34,
no. 2, pp. 229–240, 2017.
[2] S. Ghassemi and D. Hong, “Feasibility study of a novel robotic
system ballu: Buoyancy assisted lightweight legged unit,” in Humanoid
Robots (Humanoids), 2016 IEEE-RAS 16th International Conference
on. IEEE, 2016, pp. 144–144.
[3] D. Ceylan, W. Li, N. J. Mitra, M. Agrawala, and M. Pauly, “Designingand fabricating mechanical automata from mocap sequences,” ACM
Transactions on Graphics (TOG), vol. 32, no. 6, p. 186, 2013.
[4] S. Coros, B. Thomaszewski, G. Noris, S. Sueda, M. Forberg, R. W.
Sumner, W. Matusik, and B. Bickel, “Computational design of mechanical characters,” ACM Transactions on Graphics (TOG), vol. 32,
no. 4, p. 83, 2013.
[5] B. Thomaszewski, S. Coros, D. Gauge, V. Megaro, E. Grinspun, and
M. Gross, “Computational design of linkage-based characters,” ACM
Transactions on Graphics (TOG), vol. 33, no. 4, p. 64, 2014.
[6] L. Zhu, W. Xu, J. Snyder, Y. Liu, G. Wang, and B. Guo, “Motionguided mechanical toy modeling.” ACM Trans. Graph., vol. 31, no. 6,
pp. 127–1, 2012.
[7] V. Megaro, B. Thomaszewski, M. Nitti, O. Hilliges, M. Gross, and
S. Coros, “Interactive design of 3d-printable robotic creatures,” ACM
Transactions on Graphics (TOG), vol. 34, no. 6, p. 216, 2015.
[8] G. Bharaj, S. Coros, B. Thomaszewski, J. Tompkin, B. Bickel, andH. Pﬁster, “Computational design of walking automata,” in Proceedings of the 14th ACM SIGGRAPH/Eurographics Symposium on
Computer Animation. ACM, 2015, pp. 93–100.
[9] F. Multon, L. France, M.-P. Cani-Gascuel, and G. Debunne, “Computer
animation of human walking: a survey,” The journal of visualizationand computer animation, vol. 10, no. 1, pp. 39–54, 1999.
[10] C. K. Liu and Z. Popović, “Synthesis of complex dynamic charactermotion from simple animations,” ACM Transactions on Graphics(TOG), vol. 21, no. 3, pp. 408–416, 2002.
[11] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine formodel-based control,” in Intelligent Robots and Systems (IROS), 2012
IEEE/RSJ International Conference on. IEEE, 2012, pp. 5026–5033.
[12] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov,“Proximal policy optimization algorithms,” arXiv preprint
arXiv:1707.06347, 2017.
[13] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.

ACKNOWLEDGMENT
I would like to thank M. Inami for useful advice. This
work was supported by JST ERATO Grant Number JPMJER1501, Japan.

2466
Authorized licensed use limited to: University of Tokyo. Downloaded on August 08,2020 at 06:25:54 UTC from IEEE Xplore. Restrictions apply.