2011 IEEE/RSJ International Conference onIntelligent Robots and Systems
September 25-30, 2011. San Francisco, CA, USA

VocaWatcher: Natural Singing Motion Generator
for a Humanoid Robot
Shuuji Kajita, Tomoyasu Nakano, Masataka Goto, Yosuke Matsusaka, Shin’ichiro Nakaoka, and Kazuhito Yokoi

Abstract— In this paper, we describe VocaWatcher, a novel
robot motion generator that enables a humanoid robot tosing with realistic facial expressions and naturally synthesized
singing voices. This robot singer is an important and attractive humanoid robot application for the entertainment scene;moreover, it promotes state-of-the-art integration of robot engineering, music processing, and image processing. To overcome
the difficulties of generating natural facial expressions thatare precisely synchronized with singing voices, VocaWatcher
imitates a human singer by analyzing a video clip of a humansinging, recorded by a single video camera. VocaWatcher
can control mouth, eye, and neck motions by imitating thecorresponding human movements, which are estimated without
using any markers in the video. It can also synthesize singing
voices by imitating the pitch and dynamics of the human singing
in the same video.

I. I NTRODUCTION
Since Kato’s WABOT-2 [1] played an electronic organ in
1985, music has been an important and attractive application
for humanoid robots. Humanoid robots have been developedthat play various instruments, including a flute [2] and atheremin [3]. Singing humanoid robots have also been developed with synthesized singing voices; however, such robots[4], [5] do not appear to be natural because of the limitationsof manual control. A humanoid robot produced by Murata,
Nakadai, et al.[5] sang and moved in time to musical beats,using a real-time beat tracking technique. However, noneof these robots could generate realistic facial expressions
synchronized with a naturally synthesized singing voice.
We previously demonstrated a singing humanoid robot in
2009; however, its facial expressions and singing voice wereneither realistic nor natural. This robot, Cybernetic Human
HRP-4C (Fig. 1), was designed so that its dimensions (158cm height and 43 kg weight with 42 DOF) closely resembled
those of an average young Japanese female [6]. HRP-4C had
the advantage of eight motors in its small head allowing
realistic facial expressions; however, it was not easy tocontrol those motors in a natural way. Our first demonstrationof HRP-4C singing was at the public exhibition CEATEC
JAPAN 2009 [7], which was achieved in collaboration with
the Yamaha Corporation. Facial expressions and neck motions were generated by well-known techniques, such as key
poses and semi-automatic motion generation. Singing voices
were synthesized using Vocaloid2, a commercial software
Shuuji Kajita, Tomoyasu Nakano, Masataka Goto, Yosuke Matsusaka,
Shin’ichiro Nakaoka, and Kazuhito Yokoi are with the National Instituteof Advanced Industrial Science and Technology (AIST), 1-1-1 Umezono,
Tsukuba, Ibaraki 305-8568, Japan (e-mail: s.kajita@aist.go.jp).

978-1-61284-455-8/11/$26.00 ©2011 IEEE

Fig. 1.

A humanoid robot named Cybernetic Human HRP-4C.

system developed by the Yamaha Corporation [12]. Ourdemonstration was successful; however, there was muchroom for improvement, which motivated the work presented
in this paper.
A promising solution to the problem of generating realistic
humanoid robot motions is the imitation of human motions,as shown in computer graphics. Wilbers, Ishi, and Ishiguro[8] used a motion capture technique to imitate a human facewearing 31 reflective markers, which controlled the face of
their android Repliee Q2 that wore the same number of markers. Jaeckel, Campbell, and Melhuish [9] also controlled theface of their humanoid robot using video input from a humanface; however, they did not use markers. Instead, they usedperson-specific Active Appearance Models (AAMs) fitted
to video frames when tracking facial movements. However,the use of AAMs required training data. Furthermore, audio
signal processing did not help facial control in previous
studies.
We propose a robot motion generation method, which
we refer to as VocaWatcher, using both image and musicsignal processing to produce a realistic singing performancewith the HRP-4C robot. When provided with a recorded
video clip of a singing performance by a human singer, ourrobot can sing naturally by synthesizing the singing voice
imitating the pitch and dynamics of the human singing in
the clip. The robot also generates synchronous mouth, eye,and neck motions to imitate the facial expressions of the2000VocaWatcher
Neck motion

head pose

generator
eye aperture

Iris / Blinkeye position

Face tracking
Video(faceAPI)Lip detector

SingerAudio

Singing analysis

Song lylics

Eye motion
generator

mouth apertureface center

Human

center of iris

detector

Mouth motion
generator

mora sequence and timing

Singing synthesis

Singing voice &(Vocaloid2)

breathing soundFig. 3. A recording scene with the target human singing: a female singer
on the right was recorded using a home video camera on the top left.

VocaListenerFig. 2. An overview of our singing robot system which consists of two
subsystems: VocaWatcher and VocaListener.

same human who is singing. VocaWatcher requires neither
facial markers nor multiple cameras and it can utilize audiobased timing information that is estimated by analyzing thehuman’s singing voice to help generate the robot’s motions.
II. OVERVIEW OF THE SINGING ROBOT SYSTEM
Our singing robot system uses the HRP-4C and consists of
two subsystems: VocaWatcher for generating singing motions
and VocaListener for synthesizing singing voices, as shownin Fig.2. These subsystems analyze audio signals and video
frames in a video clip of a human singer, and imitate thehuman singing. Note that analysis, generation, and synthesis
are not executed in real time; nonetheless, the robot canplay back the synthesized singing voice along with thesynchronized motions.
VocaListener [11] analyzes the audio signals of the singing
voice in the video clip with the help of the written text of
its lyrics. The system automatically identifies each musicalnote produced by the singing voice and estimates expressive
information, consisting of the pitch and dynamics, to synthesize a robot singing voice with a variety of voice timbres.
The main advantages of VocaListener are that its singing
voice appears to be highly natural without laborious manualadjustments and that the synthesized voice timbre can bechanged easily. In this paper, we extended VocaListener toimitate breathing sounds that make the robot singing more
realistic.
VocaWatcher analyzes the human face and head in thevideo clip frames, but without any markers. VocaWatcher
automatically detects the head position and rotation, theiris and eyelid, and the mouth aperture, to generate neck,eye, and mouth motions, using the vowel sequence andprecise timing information provided by VocaListener. The
main advantages of VocaWatcher are highly realistic motion
generation without laborious manual adjustment and precise
synchronization of motions with the synthesized singing
voice.
Video clips of recorded singing performances were used
to demonstrate our system (Fig.3). In the video clips, eachof three different female singers sang three different songs

having different moods. For example, one of the songs isentitled “PROLOGUE” (song number: RWC-MDB-P-2001
No.7) and it was taken from the RWC Music Database(Popular Music) [10], which is a copyright-cleared music
database utilized by researchers worldwide (more than 300
institutes) as a common resource for music research. During
recordings of singing performances, a singer stood in frontof the microphone and a fixed camera (Fig.3). The singer
was allowed to move her head and hands naturally, but she
was asked to stay within a specified area to allow the upper
half of her body to be recorded.
III. S INGING SYNTHESIS FROM HUMAN VOICE
A. VocaListener: An automatic parameter estimation system
for singing synthesis by using a singing voice and its lyrics
VocaListener [11] iteratively estimates the parameters for
pitch (F0 ) and the dynamics (power) for a singing synthesis
system (e.g., Yamaha’s Vocaloid [12]) to synthesize singing
similar to a human singer (Fig. 4). The singing voice of a human is natural, and the synthesized singing voice imitates it
to produce a human-like and natural output without any timeconsuming manual adjustments. Iterative estimation provides
robustness allowing the use of different singing synthesis
systems and different singer databases. The mean error after
iterations was much smaller than the previous method [13](see [11] for details)1 .
VocaListener also provides a highly accurate lyrics-tosinging synchronization function. Given a singing voice andthe corresponding lyrics without any score information, VocaListener automatically synchronizes them and determines
where each musical note corresponds to a mora2 in the lyrics.
We also provide an interface for easily correcting errors
simply by pointing them out.
B. Extending VocaListener to imitate breathing
A singer has to breathe while singing; thus, the mouth of
the robot made facial motions to imitate changes in a singer’s
1 Demonstration videos including examples of synthesized singing areavailable at http://staff.aist.go.jp/t.nakano/VocaListener/
2 Mora is a partly decomposed unit of syllables. Each mora of Japanese
pronunciation is mapped into a musical note, where the mora representation
can be classified into three types: “V”, “CV”, and “N”. “V” denotes vowel,“C” denotes consonant, and “N” denotes syllabic nasal.

2001

4JOHJOHWPJDFQQQ

4JOHJOHTZOUIFTJ[FS

4JOHJOHBOBMZTJT

4JOHJOHTZOUIFTJ[FS

4JOHFS%# "40
Head roll (deg))VNBO
4JOHFS-ZSJDT

4JOHFS%# #VocaListener

0
-20
-40

CCCCCCCCCCCCCCC

0

50

100

150

200

250

300

50

100

150

200

250

300

50

100

150
times (s)

200

250

300

CCCCCCCCHead pitch (deg)CCCCCCC-ZSJDTBMJHONFOU CC

20

'
QPXFS

1BSBNFUFSFTUJNBUJPO*UFSBUJWFFTUJNBUJPOVOUJMUIFTZOUIFTJ[FETJOHJOH
CFDPNFTDMPTFSUPTJOHJOHCZBIVNBOTJOHFS

UJNF

UJNF

4JOHJOHTZOUIFTJTHead yaw (deg)UJNF

Fig. 4. Overview of VocaListener, which iteratively estimates parametersof pitch and dynamics for singing synthesis from the human singing voiceand the song lyrics.

facial expressions during breathing (see IV-D). It would haveappeared unnatural if there was no breathing sound during
these mouth-opening motions. We addressed this issue byextending VocaListener to synthesize breathing sounds byimitating the breathing sounds of a human singer, because
the original VocaListener did not imitate breathing sounds.
Our automatic breathing detection method [14] estimatedthe onset time and duration of every breathing sound produced by a human singer. This method detected almost
all the breathing sounds, although non-breathing sounds
were sometimes wrongly detected as breathing sounds. We
improve breathing detection performance by developing a
new technique to recover from errors by eliminating detectedsounds that were not located immediately before a phrase,
and whose length was less than 50 ms or more than 1225 ms.
This range was identified by our previous study investigating
the length of breathing sounds [14]. Here a phrase was defined as a non-silent section in the singing voice synthesized
by VocaListener. If an error persisted after the improvement
technique, it could be manually corrected.
Each breathing sound detected in human singing was
imitated using our new breathing synthesis system, which
is based on the speech manipulation system TANDEMSTRAIGHT [15]. We required the breathing sound of thetarget singer; hence, we used the same singing synthesis system (i.e., Vocaloid) to synthesize an example of a breathingsound. The synthesized example was analyzed by TANDEMSTRAIGHT to obtain its spectral envelope. The duration andpower of the spectral envelope were then controlled to ensurethat the duration and power of the synthesized breathingsound were similar to each breathing sound detected in thehuman singing.
IV. FACE MOTION ANALYSIS
Face motion analysis used commercial face tracking software and our original image processing method for the eyelid
aperture, iris position, and mouth aperture. These results
were not directly mapped to the robot motion; instead, they
were used with a modification as explained in Section V.

40
20
0
-20
-40

0

40
20
0
-20

0

Fig. 5.

Singer head posture detected by faceAPI.

300

301
302
601

0

400

401 402
701

1
602

2

3
702

6

7

100 101 102
201 202 20345207 206 205
105

104

103

Fig. 6. Feature points (face landmarks) detected by faceAPI. They areused as reference information for our additional image processing.

A. Face tracking
We used video data with 960×540 pixels and 29.97 frames
per second (FPS). The first step was to process the video, forwhich we used commercial face tracking software, faceAPI,
produced by Seeing Machines Inc.[16]. This system tracks
a human face in the video and estimates the head position
and rotation in 3D space. Fig.5 shows the head rotation of
the performer from the start to the end of a song of 298.2 slength. The same software also detected feature points on theface (face landmarks) for every frame of the video (Fig.6).
For example, the points numbered 0, 601, 1, 602 correspondto the right eye’s area. Unfortunately, these points did notreflect the eyelid aperture and the distance between points
601 and 602 never changed, even when the right eyelid wasclosed.
This meant we had to perform further image processing
to detect the eyelid aperture, which is explained in the next
section.
B. Background of iris and eyelid detection
For many years, iris and eyelid detection has been developed for use in different applications. For example, Mat-2002

sumoto et al.[17] developed an algorithm for detecting gaze
direction and produced a gaze-controlled wheel chair. Morris
et al.[18] developed an algorithm to detect eye blinking andproduced a command interface for people with paralysis.
However, no previous research has investigated iris andeyelid detection for a singing face. Iris and eyelid detection
for singing face presents the following new problems.
A singer might maintain a narrowing of his[her] eyes
for long periods during a performance. Thus, we had tocontinuously detect the aperture ratio of the eyelid in eachframe, whereas conventional methods only detect eye blinks
as discrete values.
A singer might continuously move his[her] head during
a performance. Thus, we had to capture the whole imageof the face from a distant position, because we could notforce a singer to wear an eye camera simply to obtain ahigh-resolution image of the eye.
To deal these problems, we developed a novel iris and eyelid detection algorithm for use during singing. Our algorithm
has the following advantages:• Detect the iris, even if the eyelid is half closed.
• Detect the aperture ratio of the eyelid as a continuous
value.
• Detect the above information, even from a lowresolution image.
We explain the algorithm in the next.

where I is the input image, |∇I| is the edge image detectedby a Sobel operator, A is the accumulated voting result of
the circular Hough transform, D is the color distance, I r is
the iris color, k is an integration constant, and L is a 2D
map of the iris likelihood. In this particular experiment, weused color distance calculated from a gray scale image where
the iris was assumed to be black. We consider that the color
distance measure can be easily extended to a multi-scale fordetecting non-black irises.
Finally, the position of the iris, p, is estimated using theiris likelihood L, as follows.
p = arg max Lxy

(5)

xy

Eyelid aperture rate detection uses the subpixel algorithm: In most of source images, the height of the eye
is about 3-6 pixels. If we use a conventional pixel-basedalgorithm, the eyelid aperture ratio can only be acquired as
discrete values between 3 and 6. We needed to acquire ahigher resolution eyelid aperture ratio than 3-6, to produce asmooth and realistic facial motion with the HRP-4C robot.
To solve this problem, we developed a subpixel algorithm
for estimating the aperture ratio.

C. Algorithm for iris and eyelid detection
During iris and eyelid detection, we first focus on the eyeregion of the input image, which is detected using faceAPI(the rectangular area supported by feature points 0, 601, 1,
602 as shown in Fig.6).
Fig.7 shows an overview of the iris and eyelid detection
process.
Input image

Sobel filter

Hough transformand accumulation
Color distancecalulation

Fig. 7.

+

Integrated mapof iris liklihoodkOverview of the iris and eyelid detection process.

Our algorithm consists of following parts:
Enhanced Hough voting with color distance: The irisof the singer is occluded by a half-closed eyelid for a long
period during the performance. In order to ensure robust iris
detection, we developed an algorithm that uses a regular
Hough transform-based voting map, and integrates color
similarity of the iris in voting. The algorithm is formalized
as follows:
L=

A + kD(1)

Ax+a,y+b←

Ax+a,y+b + |∇I|(a = r sin θ, b = r cos θ)

(2)

|∇I|
Dxy

=
=

(dI/dx + dI/dy)1/2(1 − Ixy − I r )2 ,

(3)
(4)

Fig. 8.

Relation of pixel and real image.

In our subpixel algorithm, we assume that each observed
pixel is a sum of the continuous intensity of the real object
IC: x+ δx y+ δy2
2
I C dxdy
x− δx
y− δy
2
2
,
(6)
Ixy =
δxδy
where δx and δy are the height and width of the pixels,
respectively (see Fig.8).
If we assume that the intensity of the eyelid is I e and theintensity of the iris is I r , then the pixel between the boundaryof the eyelid and the iris will be observed as a weighted sumof both intensities:
 y+ δyb x+ δx x+ δx2
2
I e dxdy + y− δy x− δx2 I r dxdy
b
x− δx
2
2
2
Ixy =
δxδy
δy
e
r
(y + δy2 − b)I + (b − (y − 2 ))I
,
(7)
=
δy
where b is the boundary position of the eyelid.
When we accept this assumption, we can estimate theboundary position b given the pixel value Ixy , which hasa higher resolution than δy as follows.
δyIxy − (y + δy/2)I e + (y − δy/2)I rb =(8)
Ir − Ie

2003

a

e

=

:=




0
e−eminemax −emin

1



px
+pr


(e < emin )(emin ≤ e < emax )(e ≥ emax )(9)

Ixy , emin := 2I e pr , emax := 2I r pr ,px −prwhere p is the position of the iris (estimated in the previousstep), pr is the radius of the iris, and a is the aperture rate.
D. Mouth aperture detection
A mouth aperture is the distance between the upper andlower lips, which can be calculated from the feature points(e.g., the distance between feature points 202 and 206) asshown in Fig.6. Unfortunately, faceAPI occasionally failed
to track the singer’s lips in our experiment, because they
moved too fast. To solve this problem, we added an extra
image-processing step to detect lip movements in the video.
First, we extract a line in the image along the facecenter line3 , which was estimated by faceAPI. Lines obtained
from the video frames are combined into a 2D image that
represents the vertical motion of lips during a performance.
To estimate the center of the upper and lower lips, we useda particle filter to track lip areas using RGB color distance.
Fig.9 shows the estimated lip motions.

The bold line in Fig.10 shows the mouth aperture at
the beginning of the song. The shaded bands indicate thestarts and ends of the moras obtained using VocaListener, as
explained in Section III. The sound of each mora is indicatedat the top. Using this graph, we observed the following
features of human lip movement.
1) The mouth was open even when the singer was notpronouncing the lyrics. For example, the mouth was
open for about forty percent of the period between 19.6
and 20.2 s, which was just before the start of the song.
This indicates that the singer took a breath. Anotherbreath can be observed between 22.1 and 22.5 s. In
this period, the singer opened her mouth even wider
than during the singing period.
2) The mouth aperture did not remain constant during
each mora. Typical examples are in the /ma/ sounding
period from 21 to 21.2 s and the period from 22.5 to
22.8 s. The singer almost closed her mouth at the beginning then opened it again quickly. This is a featureof the lip movements with the bilabial consonants /m/,/p/, /b/, etc. We also saw variable fluctuations in othermora periods, which were related to consonants or theemotional expression of the singer.
1
mouse aperture c (-)

I r is given as a constant, while I e is calculated fromneighboring pixels outside of the iris.
In this specific experiment, we used a simplified formula:20

lip image (pixel)30

chi do

ma ru

to ki

ma

ta

22.5

23

0.6
0.4
0.2
0
19

yU
yL

10

ta

(mora type)

0.8

19.5

Fig. 10.

20

20.5

21
time (s)21.5

22Mouth aperture ratio and the mora.

40
50
60
70

V. ROBOT MOTION GENERATION

80

Fig.11 shows the joints in the head of HRP-4C [19].
All joints were driven by servomotors, which are positioncontrolled and accept the target position references at 200
Hz. Therefore, our goal was to calculate a reference trajectory specifying the joint angles every 5 ms. The following
subsections describe the generation process for the neck, eyeand mouth references.

90
100
46 47

48 49

Fig. 9.

50

51 52

53 54 55 56
time (s)

57

58 59

60 61

62

Lip motion estimated by particle filter.

We define the mouth aperture ratio c as follows:0(d < dmin )d−dmin(dmin ≤ d < dmax )c = dmax −dmin1(d ≥ dmax )d := |yU − yL |,A. Neck motion(10)

where yU and yL are the positions of the upper and lower
lip, which were obtained by the particle filter. The parametersdmin and dmax were determined, such that c = 0 represents
the closed mouth and c = 1 represents the mouth at the
maximum aperture.
3 We defined the face center line to pass the feature point 202 in parallelwith the vector from the point 101 to 104 in Fig.6.

The robot had three neck joints, NECK Y, NECK P, and
NECK R that rotate the head around the yaw, pitch, and rollaxes, respectively. We used the head postures obtained in
IV-A with low-pass filtering as the reference trajectories forthese joints.
B. Eye motion
The EYELID P joint drives the eyelids together on both
the right and left eyes. Likewise, the EYE Y and EYE P
joints drive both eyeballs around the yaw and pitch axes,
respectively.

2004

TABLE II
K EY POSES FOR VOWELS AND BREATH .

EYEBROW_P
EYELID_P

Vowel
MOUTH P (deg)UPPERLIP P (deg)LOWERLIP P (deg)CHEEK P (deg)Nonlinear gain s (-)
Nonlinear gain k (-)

EYE_P
CHEEK_P
NECK_R
EYE_Y

NECK_P

/a/
9
-5
5
0
0.5
0.5

/i/
0
-25
25
-2
0.3
0.7

/u/
0
-23
24
0
0.3
0.6

/e/
6
0
0
-1
0.6
0.8

ta

chi do

ma ru

to ki

21

21.5

/o/
8
-10
10
0
0.6
0.8

Breath
10
0
0
0
0.5
0.5

UPPERLIP_P
NECK_Y
30

LOWERLIP_P

ma

ta

22.5

23

ma

ta

22.5

23

Lip shape (deg)20

MOUTH_P

10
0
-10
-20

MOUTH_P
UPPERLIP_P
LOWERLIP_P-30

Fig. 11.

Face and neck joints of HRP-4C [19].

19

CHEEK_P
19.5

20

20.5

22

(a) Mora-based pattern generated by vowel type

TABLE IJ OINTS AFFECT LIP SHAPE .
purposeopen/close the jaw
lift up/down the upper lip
extract/retract the lower lip
lift up/down the mouth corners

taNonlinear gain parameters

Joint name
MOUTH P
UPPERLIP P
LOWERLIP PCHEEK P

1

movable range (deg)0 – 10-25 – 00 – 25-3.3 – 0

C. Mouth motion
Four joints were used to control the lip shape of HRP4C, as shown in Table I. Each joint had a distinct role
and a specific movable range. When we set all the joint
angles to zero, the robot closed her mouth. These joint angles
were calculated based on the mora obtained by VocaListenerin III, and the mouth aperture obtained in IV-D. Thus, allparameters were derived from actual visual and auditory
information.
There is known to be a characteristic mouth shape foreach vowel. We specified the key mouth shapes for the fivevowels used in Japanese language (a,i,u,e,o) and for a breath,as shown in Table II. The nonlinear gains s and k are extra
motion parameters, which will be explained later.
Using each mora type and its timing, we generated apattern that corresponded to the vowel sequence in the song.
During this step, all periods without moras were considered
to be breathing. Fig.12(a) shows the pattern generated using
this method. The joint trajectories were generated so theyreached the key mouth shape angle at the start time of each

ma ru

to ki

0.6
0.4
0.2

s(UPPERLIP_P)
k(UPPERLIP_P)

019

To reproduce the human singer’s expressions, we usedthe eyelid aperture calculated by (9) in IV-C. The reference
trajectory for EYELID P was set as the low-pass filteredeyelid aperture multiplied by an appropriate gain.
To operate the gaze action, we specified the EYE Y
reference using the iris position obtained by (5) in IV-C.
The EYBROW P and EYE P joints were kept at zero;
therefore, the robot maintained her eyebrows at a constantheight and exhibited a horizontal gaze motion.

chi do

0.8

19.5

20

20.5

21
time (s)21.522(b) Nonlinear gain parameters

Fig. 12.

Patterns generated from the mora.

mora. We generated a smooth path using cubic splines andspecified a transient time for each joint.
This mora-based pattern allowed our robot to achieve
nominal vowel lip shapes with correct timing nevertheless,although it did not reflect the breath timing, bilabial consonants, or emotional expression of the original singer seen in
the mouth aperture data of Fig.10.
A straightforward solution is to multiply the mora-based
pattern by the mouth aperture ratio. In other words, themora-based pattern is modulated by the mouth aperture. We
tested this and confirmed that a natural lip motion could be
generated. However, we also observed that the lip movement
for the vowels /i/, /u/, and /o/ was smaller than the specified
key mouth shape. This was because the aperture ratio was
not normalized for each key mouth shape; instead, it was
normalized for the maximum mouth opening. For example,
the human mouth aperture ratio for the /i/ sound hardly
exceeds 0.6; thus, the modulated lip motion is always lessthan sixty percent of the desired key mouth shape.
To solve this problem, we introduced a modulation using
the nonlinear gain shown in Fig.13. For a given mouth
aperture ratio, c, the parameters {s, k} determine the shapeof the nonlinear gain g(c, s, k) required for modulation, such2005

mora sequencevowel to keypose

mora-based angle

joint angle

and interpolation

qimora

qi

Human

Robot

{s, k}
1
mouth aperture ratio ckg (c, s, k)
s

0

c

1

Nonlinear gain(a) at the start of singing, voicing /ta/, time = 20.29s, frame = 608

closedFig. 13.

Mouth aperture modulation.

breath/a/ sound/i/ sound/o/ soundbreath30

Mouth joints (deg)2010(b) closing eyes, voicing /ru/ sound, time = 74.17s, frame = 2223

0
-10

MOUTH_P
UPPERLIP_P-20

LOWERLIP_P
CHEEK_P

-30
19

ta

(mora type)
19.5

20

chi do20.5

Fig. 14.

ma ru
21
time (s)

to ki
21.5

22

ma

ta

22.5

23

Generated mouth motion.

(c) voicing long /ra/ sound, time = 151.48s, frame = 4540

thatqig(c, s, k)=:=g(c, s, k)qimora(k/s)c1−k1−s (c − s) + k(11)(0 ≤ c < s)(s ≤ c ≤ 1)

where qi and qimora are the i-th mouth joint angles andthe mora-based angle, respectively.
The parameters {s, k} were determined for each vowel,
as shown in the two rows at the bottom of Table II. These
parameters were changed over time by a mora sequence andsmoothly interpolated by cubic spline (Fig.12(b)).
Fig.14 shows the pattern generated and the some examplesof lip shapes produced by HRP-4C. Our method allowed
us to produce a lip sync motion that was accurate andexpressive.

(d) interlude, no voice, watching left, time =204.77s, frame = 6137

Fig. 15. Snapshots of the face of the original human singer (on the left)
and the face of the robot singer (on the right) having the generated facial
expressions imitating the human singer.

2) In Fig.15(b), the robot eyes were not tightly closed,
unlike the human eyes. This was caused by the current
eyelid joint setting, which maintains a slight openstate to avoid problems such as over-current and motor
burning.

VI. R ESULTS
Figure 15 shows four different pairs of snapshot of theoriginal human face (left column) and the corresponding faceof the singing HRP-4C (right column). This figure showsthat we succeeded in effectively making the robot imitate
the original human singer. However, when we compared
the human and robot faces in detail we still observed thefollowing differences.
1) The mouth aperture of HRP-4C was smaller thanthat of the human singer in Fig.15(a) and (c). Thiswas caused by a technical limitation of the current
mechanism in our robot.

One of the advantages of our robot system can be seenin the interlude of the song (Fig.15(d)). In this snapshot,
the human singer voluntarily moved her head and eyes eventhough she was not singing, and these movements were wellimitated by our robot. We found that the imitation of such
subtle and unconscious motions contributed to making ourrobot performance appear much more realistic.

2006

ACKNOWLEDGEMENTS
We thank the members of the Humanoid Robotics Groupof AIST, especially Kanako Miura for her help and advice at
an early stage of the experiment and Kenta Yonekura for his
work choreographing HRP-4C’s arm motions. We also thankYoshio Matsumoto for his helpful advice. Finally, we gratefully acknowledge the support of Hirohisa Hirukawa andSatoshi Sekiguchi, the directors of the Intelligent Systems
Research Institute and the Information Technology ResearchInstitute of AIST, respectively.
R EFERENCES

Fig. 16. Our robot singer HRP-4C on the stage in CEATEC JAPAN 2010.

We demonstrated HRP-4C singing at CEATEC JAPAN2010. This is Japan’s largest consumer electronics exhibition
and it was held in Chiba city in October 2010. To make therobot performance more attractive, we choreographed arm
motions so that the arms could move naturally in time tothe music. This was achieved using our own motion creation
software, Choreonoid[20]. We did not include leg motions,because the robot was fixed on a stand to perform songs
in a standing posture. The demonstration of our singing
robot was successful and it received a lot of audienceand media coverage.4 We found that some of the audience
did not realize that the singer was a robot at first sight,
because of the naturalness of the singing voice and motions.
However, other audience members also expressed a feelingof ’creepiness’. Demonstration video clips are available on
our website (http://staff.aist.go.jp/t.nakano/VocaWatcher/).
VII. C ONCLUSIONS AND FUTURE WORK
We have described a humanoid robot singer that sings
with highly natural synthesized singing voices and that
produces highly realistic facial expressions. By integrating
visual and audio information, the robot motions synchronizedwith singing voices were generated from a human singer
performance recorded using a single camera.
We aim to improve the robot mechanism and overcome
some current limitations in the singing motions. For example,
HRP-4C cannot open her mouth sufficiently wide and shecannot close her eyes tightly. Furthermore, it takes a day
to generate robot motions from a given a video clip in thecurrent implementation, which needs to be reduced in future.
Our experiences have shown us that a realistic robot singer
attracts considerable public attention, and we hope to makeit more widely available in future.
4 Someexamplesare
http://www.diginfo.tv/
2010/10/13/10-0217-r-en.php
and
http://
blogs.wsj.com/japanrealtime/2010/10/05/
japans-next-pop-idol-is-a-robot/ .

[1] I. Kato et al., “The robot musician WABOT-2,” Robotics, vol.3,
pp.143-155, 1987.
[2] K. Chida et al. “Development of a new anthropomorphic flutist robot
WF-4,” in Proc. of ICRA2004, pp.152-157, 2004.
[3] T.Mizumoto et al. “Thereminist Robot: Development of a robot
theremin player with feedforward and feedback arm control based ona theremin’s pitch model,” in Proc. of IROS2009, 2009.
[4] Y. Kuroki, M. Fujita, T. Ishida, K. Nagasaka and J. Yamaguchi, “A
small biped entertainment robot exploring attractive applications,” inProc. of ICRA2003, pp.471-476, 2003.
[5] K. Murata, K. Nakadai, et al., “A robot singer with music recognition
based on real-time beat tracking,” in Proc. of ISMIR 2008 – Session
2b – Music Recognition and Visualization, pp.199-204, 2008.
[6] K.Kaneko, F.Kanehiro, M.Morisawa, K.Miura, S.Nakaoka andS.Kajita “Cybernetic Human HRP-4C,” in Proc. IEEE/RSJ Int. Conference on Humanoid Robots, pp.7-14, 2009.
[7] M.Tachibana, S.Nakaoka and H.Kenmochi, “A singing robot realized
by a collaboration of VOCALOID and Cybernetic Human HRP-4C,”
in Proc. of InterSinging2010, Tokyo, 2010.
[8] F. Wilbers, C. Ishi and H. Ishiguro, “A blendshape model for mapping
facial motions to an android,” in Proc. of the IROS2007, pp.542-547,
2007.
[9] P. Jaeckel, N. Campbell, C. Melhuish, “Facial behavior mapping —From video footage to a robot head, ” Robotics and Autonomous
Systems, vol.56, pp.1042-1049, 2008.
[10] M. Goto et al., “RWC music database: Music genre database andmusical instrument sound database,” in Proc. of ISMIR2003, pp.229230, 2003.
[11] T.Nakano and M.Goto, “VocaListener: A singing-to-singing synthesis
system based on iterative parameter estimation,” in Proc. of the SMC
2009, pp.343-348, 2009.
[12] H. Kenmochi and H. Ohshita, “Vocaloid – Commercial singing synthesizer based on sample concatenation,” in Proc. Interspeech 2007,
pp. 4010-4011, 2007.
[13] J. Janer, J. Bonada, and M. Blaauw, “Performance-driven control forsample-based singing voice synthesis,” in Proc. of DAFx-06, pp.41-44,
2006.
[14] T. Nakano, J. Ogata, M. Goto, and Y. Hiraga, “Analysis and automatic
detection of breath sounds in unaccompanied singing voice,” in Proc.
ICMPC 10, 2008.
[15] H. Kawahara, M. Morise, T. Takahashi, R. Nisimura, T. Irino, and
H. Banno, “Tandem-STRAIGHT: A temporally stable power spectral
representation for periodic signals and applications to interference-free
spectrum, F0, and aperiodicity estimation,” in Proc. of ICASSP 2008,
pp. 3933-3936, 2008.
[16] Seeing Machines http://www.seeingmachines.com[17] Y. Matsumoto, T. Ino and T. Ogasawara, “Development of intelligent
wheelchair system with face and gaze based interface,” in Proc. of
ROMAN 2001, pp.262-267, 2001.
[18] T. Morris, P. Blenkhorn and F. Zaidi, “Blink detection for real-time
eye tracking,” Journal of Network and Computer Applications, Volume
25, Issue 2, pp.129-143, 2002.
[19] S. Nakaoka, F. Kanehiro, K. Miura, et al. “Creating facial motionsof Cybernetic Human HRP-4C,” in Proc. of Humanoids2009, pp.561567, 2009.
[20] S. Nakaoka, S. Kajita and K. Yokoi, “Intuitive and flexible userinterface for creating whole body motions of biped humanoid robots,”
in Proc. of the IROS2010, pp.1675-1682, 2010.

2007