Expression Control
in Singing Voice
Synthesis[Features, approaches,

evaluation, and challenges][Martí Umbert, Jordi Bonada,
Masataka Goto, Tomoyasu Nakano,]In the context of singing voice synthesis, expression control manipulates a set of voice features related to a particular emotion, style, or
singer. Also known as performance modeling, it has been
approached from different perspectives and for different purposes,
and different projects have shown a wide extent of applicability. The
aim of this article is to provide an overview of approaches to expression
control in singing voice synthesis. We introduce some musical applications that use singing voice synthesis techniques to justify the need foran accurate control of expression. Then, expression is defined andrelated to speech and instrument performance modeling. Next, we present the commonly studied set of voice parameters that can change

and Johan Sundberg

Digital Object Identifier 10.1109/MSP.2015.2424572
Date of publication: 13 October 2015

image licensed by ingram publishing1053-5888/15©2015IEEE

IEEE SIGNAL PROCESSING MAGAZINE [55] november 2015

voices that are difficult to produce naturally (e.g., castrati).
More examples can be found with pedagogical purposes or as
tools to identify perceptually relevant voice properties [3].
Project
WEBSITE
These applications of the so-called music information
Cantorhttp://www.virsyn.deresearch field may have a great impact on the way we interCANTOR DIGITALIS
https://cantordigitalis.limsi.fr/
act with music [4]. Examples of research projects using singChaNTeR
https://chanter.limsi.fr
ing voice synthesis technologies are listed in Table 1.
The generic framework of these systems is represented
FLINGER
http://www.cslu.ogi.edu/tts/flinger
in­F
igure 1, based on [5]. The input may consist of the
LYRICOShttp://www.cslu.ogi.edu/tts/demos
score(e.g., the note sequence, contextual marks related to
ORPHEUS
http://www.orpheus-music.org/v3
loudness, or note transitions), lyrics, and the intention
SINSY
http://www.sinsy.jp(e.g., the style or emotion). The intention may be derived
SYMPHONIC CHOIRS
http://www.soundsonline.com/Symphonic-Choirs
from the lyrics and score content (shown by the dashed
VIRTUAL INSTRUMENTline). The input may be analyzed to get the phonetic tranVOCALISTENER
https://staff.aist.go.jp/t.nakano/VocaListener
scription, the alignment with a reference performance, or
VOCALISTENER
http://www.vocaloid.com/lineup/vocalis
contextual data. The expression control generation block(PRODUCT VERSION)
represents the implicit or explicit knowledge of the system
VOCALISTENER2
https://staff.aist.go.jp/t.nakano/VocaListener2
as a set of reference singing performances, a set of rules,
VOCALOID
http://www.vocaloid.com
or statistical models. Its output is used by the synthesizer
VOCAREFINER
https://staff.aist.go.jp/t.nakano/VocaRefinerto generate the sound, which may be used iteratively to
VOCAWATCHER
https://staff.aist.go.jp/t.nakano/VocaWatcher
improve the expression controls.
A key element of such technologies is the singer voice
model
[1],
[2], [6], although due to space constraints, it is not
perceptual aspects of synthesized voices. After that, we provide an
described here in detail. For the purpose of this article, it is more
up-to-date classification, comparison, and description of a selecinteresting to classify singing synthesis systems with respect to thetion of approaches to expression control. Then, we describe how
control parameters. As shown in Table 2, those systems are classified
these approaches are currently evaluated and discuss the benefits
into model-based and concatenative synthesizers. While, in signalof building a common evaluation framework and adopting permodels, the control parameters are mostly related to a perception
ceptually-motivated objective measures. Finally, we discuss theperspective, in physical models, these are related to physical aspects
challenges that we currently foresee.
of the vocal organs. In concatenative synthesis, a cost criterion isused to retrieve sound segments (called units) from a corpus that are
SINGING VOICE SYNTHESIS SYSTEMS
then transformed and concatenated to generate the output utterIn recent decades, several applications have shown how singing voiceance. Units may cover a fixed number of linguistic units, e.g.,synthesis technologies can be of interest for composers [1], [2]. Techdiphones that cover the transition between two phonemes or a morenologies for the manipulation of voice features have been increasflexible and wider scope. In this case, control parameters are also
ingly used to enhance tools for music creation and postprocessing,
related to perceptual aspects.
singing a live performance, to imitate a singer, and even to generate[TABLE 1] RESEARCH PROJECTS USINGSINGING VOICE SYNTHESIS TECHNOLOGIES.

Adaptation
Process

Score
Analysis
LyricsExpressionControl
Generation

Intention

Singing
Performances

Rules

Synthesizer

Models[FIG1] Generic framework blocks for expression control.

IEEE SIGNAL PROCESSING MAGAZINE [56] november 2015

Sound

Within the scope of this review, we focus on the perceptual
aspects of the control parameters, which are used to synthesize
expressive performances by taking a musical score, lyrics, or an
optional human performance as the input. However, this article
does not discuss voice conversion and morphing in which input
voice recordings are analyzed and transformed [7], [8].
EXPRESSION IN MUSICAL PERFORMANCE AND SINGING
Expression is an intuitive aspect of a music performance, but it iscomplex to define. In [5, p. 2], it is viewed as “the strategies andchanges which are not marked in a score but which performers
apply to the music.” In [9, p. 1], expression is “the added value of aperformance and is part of the reason that music is interesting tolisten to and sounds alive.” A complete definition is given in[10, p. 150], relating the liveliness of a score to “the artist’s understanding of the structure and ‘meaning’ of a piece of music, andhis/her (conscious or unconscious) expression of this understanding via expressive performance.” From a psychological perspective,
Juslin [11, p. 276] defines it as “a set of perceptual qualities thatreflect psychophysical relationships between ‘objective’ propertiesof the music, and ‘subjective’ impressions of the listener.”
Expression has a key impact on the perceived quality and naturalness. As pointed out by Ternström [13], “even a single sine wave
can be expressive to some degree if it is expertly controlled inamplitude and frequency.” Ternström says that musicians caremore about instruments being adequately expressive than sounding natural. For instance, in Clara Rockmore’s performance ofVocalise by Sergei Vasilyevich Rachmaninoff, a skillfully controlled
Theremin expresses her intentions to a high degree (all citedsounds have been collected and shown online; see [51]), despite
the limited degrees of freedom.
In the case of the singing voice, achieving a realistic sound synthesis implies controlling a wider set of parameters than just theamplitude and frequency. These parameters can be used by a singing
voice synthesizer or to transform a recording. From a psychologicalperspective, pitch contour, vibrato features, intensity contour, tremolo, phonetic timing, and others related to timbre are the main control parameters that are typically used to transmit a message with acertain mood or emotion [12] and shaped by a musical style [14].
These are described in detail in the section “Singing Voice Performance Features.”
Nominal values for certain parameters can be inferred fromthe musical score through the note’s pitch, dynamics, andduration as well as its articulation, such as staccato or legato

marks. However, these values are not intrinsically expressive per
se. In other words, expression contributes to the differences
between these values and a real performance. Different strategies
for generating expression controls are explained in the section“Expression-Control Approaches.”
It is important to note that there is more than one acceptable
expressive performance for a given song [1], [3], [15]. Such variability complicates the evaluation and comparison of different
expression-control approaches. This issue is tackled in the “Evaluation” section. Besides singing, expression has been studied inspeech and instrumental music performance.
CONNECTION TO SPEECH AND INSTRUMENTAL
MUSICAL PERFORMANCE
There are several common aspects of performing expressively
through singing voice, speech, and musical instruments. In
speech, the five acoustic attributes of prosody have been widely
studied [16], for instance, to convey emotions [17]. The most studied attribute is the fundamental frequency (F0) of the voice source
signal. Timing is the acoustic cue of rhythm, and it is a rather
complex attribute given the number of acoustic features to which
it is related [16, p. 43]. Other attributes are intensity, voice quality(related to the glottal excitation), and articulation (largely determined by the phonetic context and speech rate).
Expressive music performance with instruments has also
been widely studied. Several computational models are reviewed
in [18, p. 205], such as the KTH model, which is based “on performance rules that predict the timing, dynamics, and articulation from local musical context.” The Todd model links themusical structure to a performance with simple rules like measurements of human performances. The Mazzola model analyzes musical structure features such as tempo and melody anditeratively modifies the expressive parameters of a synthesized
performance. Finally, a machine-learning model discovers patterns within a large amount of data; it focuses, for instance, on
timing, dynamics, and more abstract structures like phrases andmanipulates them via tempo, dynamics, and articulation. In [5],30 more systems are classified into nonlearning methods, linear
regression, artificial neural networks, and rule-/case-basedlearning models, among others.
In this review, we adopt a signal processing perspective to focus
on the acoustic cues that convey a certain emotion or evoke asinging style in singing performances. As mentioned in [12,p. 799], “vocal expression is the model on which musical[TABLE 2] Singing voice synthesis systems and control parameters.
Singing synthesis systems
MODEL-BASED SYNTHESISPARAMETERS

SIGNAL MODELSPHYSICAL MODELS

F0, RESONANCES (CENTER FREQUENCYAND BANDWIDTH), SINUSOID FREQUENCY,
PHASE, AMPLITUDE, GLOTTAL PULSE
SPECTRAL SHAPE, AND PHONETIC TIMINGVOCAL APPARATUS-RELATED PARAMETERS(TONGUE, JAW, VOCAL TRACT LENGTH
AND TENSION, SUBGLOTTAL AIR PRESSURE,
AND PHONETIC TIMING)IEEE SIGNAL PROCESSING MAGAZINE [57] november 2015

CONCATENATIVE SYNTHESISFIXED LENGTHUNITSNONUNIFORMLENGTH UNITS

F0, AMPLITUDE, TIMBRE,
AND PHONETIC TIMING

expression is based,” which highlights the topic relevance for boththe speech and the music performance community. Since there isroom for improvement, the challenges that we foresee aredescribed in the “Challenges” section.
SINGING VOICE PERFORMANCE FEATURESIn the section “Expression in Musical Performance and Singing,”
we introduced a wide set of low-level parameters for singing voice
expression. In this section, we relate them to other musical elements. Then, the control parameters are described, and finally, we
illustrate them by analyzing a singing voice excerpt.
FEATURE CLASSIFICATIONAs in speech prosody, music can also be decomposed into various
musical elements. The main musical elements, such as melody,
dynamics, rhythm, and timbre, are built on low-level acoustic
features. The relationships between these elements and theacoustic features can be represented in several ways [19, p. 44].
Based on this, Table 3 relates the commonly modeled acoustic
features of the singing voice to the elements to which they
belong. Some acoustic features spread transversally over several
elements. Some features are instantaneous, such as F0 andintensity frame values, some span over a local time window, suchas articulation and attack, and others have a more global temporal scope, such as F0 and intensity contours or vibrato and tremolo features.
Next, for each of these four musical elements, we provide
introductory definitions to their acoustic features. Finally, these
are related to the analysis of a real singing voice performance.
MELODY-RELATED FEATURESThe F0 contour, or the singer’s rendition of the melody (notesequence in a score), is the sequence of F0 frame-based values [20].
F0 represents the “rate at which the vocal folds open and close
across the glottis,” and acoustically it is defined as “the lowest­periodic cycle component of the acoustic waveform” [12, p. 790].
Perceptually, it relates to the pitch, defined as “the aspect of auditory sensation whose variation is associated with musical melodies”[21, p. 2]. In the literature, however, the pitch and F0 terms areoften used indistinctly to refer to F0.
The F0 contour is affected by microprosody [22], i.e., fluctuations in pitch and dynamics due to phonetics (not attributable toexpression). While certain phonemes such as vowels may have stable contours, other phonemes such as velar consonants may fluctuate because of articulatory effects.

A skilled singer can show expressive ability through the melody rendition and modify it more expressively than unskilled
singers. Pitch deviations from the theoretical note can be intentional as an expressive resource [3]. Moreover, different articulations, i.e., the F0 contour in a transition between consecutive
notes, can be used expressively. For example, in staccato, shortpauses are introduced between notes. In the section “Transverse
Features,” the use of vibrato is detailed.
DYNAMICS-RELATED FEATURES
As summarized in [12, p. 790], the intensity (related to the perceived loudness of the voice) is a “measure of energy in the acoustic signal” usually from the waveform amplitude. It “reflects theeffort required to produce the speech” or singing voice and ismeasured by energy at a frame level. A sequence of intensity values provides the intensity contour, which is correlated to the
waveform envelope and the F0 since the energy increases with
the F0 so as to produce a similar auditory loudness [23]. Acoustically, vocal effort is primarily related to the spectrum slope of theglottal sound source rather than the overall sound level. Tremolo
may also be used, as detailed in the section “Transverse Features.”Microprosody also has an influence on intensity. The phonetic
content of speech may produce intensity increases as in plosives or
reductions like some unvoiced sounds.
RHYTHM-RELATED FEATURES
The perception of rhythm involves cognitive processes such as “movement, regularity, grouping, and yet accentuation and differentiation”[24, p. 588], where it is defined as “the grouping and strong/weak relationships” among the beats or “the sequence of equally spaced phenomenal impulses which define a tempo for the music.” The tempo
corresponds to the number of beats per minute. In real-life performances, there are timing deviations from the nominal score [12].
Similar to the role of speech rate in prosody, phoneme onsets
are also affected by singing voice rhythm. Notes and lyrics arealigned so that the first vowel onset in a syllable is synchronizedwith the note onset and any preceding phoneme in the syllable isadvanced [3], [25].
TIMBRE-RELATED FEATURES
The timbre mainly depends on the vocal tract dimensions and on
the mechanical characteristics of the vocal folds, which affect thevoice source signal [23]. Timbre is typically characterized by anamplitude spectrum representation and is often decomposed into
source and vocal tract components.

[TABLE 3] CLASSIFICATION OF SINGING VOICE EXPRESSION FEATURES.
MELODYDYNAMICSVIBRATO AND TREMOLO (DEPTH AND RATE)ATTACK AND RELEASEARTICULATION
F0 CONTOURF0 FRAME VALUE
DETUNING

INTENSITY CONTOURINTENSITY FRAME VALUERHYTHM
PAUSES
PHONEME TIME LAG
PHRASING
NOTE/PHONEME ONSET/DURATION
TIMING DEVIATIONTEMPOIEEE SIGNAL PROCESSING MAGAZINE [58] november 2015

TIMBRE
VOICE SOURCE
SINGER’S FORMANTSUBHARMONICS
FORMANT TUNINGAPERIODICITY
SPECTRUM

3 kHz, the so-called singer’s formant cluster [23]. This makes it
easier to hear the singing voice over a loud orchestra. The
affected harmonic frequencies (multiples of F0) are radiated mostefficiently toward the direction where the singer is facing, normally the audience.
Changing modal voice into other voice qualities can be used
expressively [26]. A rough voice results from a random modulationof the F0 of the source signal (jitter) or of its amplitude (shimmer).
In a growl voice, subharmonics emerge because of half-periodic
vibrations of the vocal folds, and, in breathy voices, the glottis does
not completely close, increasing the presence of aperiodic energy.

The voice source can be described in terms of its F0, amplitude, and spectrum (i.e., vocal loudness and mode of phonation). In the frequency domain, the spectrum of the voicesource is generally approximated by an average slope of −12 dB/
octave, but it typically varies with vocal loudness [23]. The voicesource is relevant for expression and is used differently among
singing styles [14].
The vocal tract filters the voice source, emphasizing certain
frequency regions or formants. Although formants are affected by
all vocal tract elements, some have a higher effect on certain formants. For instance, the first two formants are related to the produced vowel, with the first formant being primarily related to thejaw opening and the second formant to the tongue body shape.
The next three formants are related to timbre and voice identity,
with the third formant being particularly influenced by theregion under the tip of the tongue and the fourth to the vocal
tract length and dimensions of the larynx [23]. In western male
operatic voices, the third, fourth, and fifth formants typicallycluster, producing a marked spectrum envelope peak around

Cents

TRANSVERSE FEATURES
Several features from Table 3 can be considered transversal giventhat they are spread over several elements. In this section, wehighlight the most relevant ones.
Vibrato is defined [23] as a nearly sinusoidal fluctuation of F0. In
operatic singing, it is characterized by a rate that tends to range
from 5.5 to 7.5 Hz and a depth around ±0.5 or 1 semitones.

My
(a)

Heart
Cents

Un-Chain-

Chain My Heart
(b)

0.5
0
–0.5

0

0.5

1

1.5
Seconds
(c)

2

2.5

0

0.5

1

1.5
(d)

2

2.5

3

0

0.5

1

1.5
(e)

2

2.5

3

0

0.5

1

1.5
Seconds
(f)

2

2.5

3

–500
–1,000–1,500Power

Amplitude

Un–500–1,000–1,500

3

0.03
0.02
0.01
0

[FIG2] The expression analysis of a singing voice sample: (a) score, (b) modified score, (c) waveform, (d) note onsets and pitch,(e) extracted pitch and labeled notes, and (f) extracted energy.

0.4

–20
Amplitude (dB)

0.2
0
–0.2
–0.4
1.72

–40
–60
–80
–100

1.74

1.76
1.78
Time (seconds)
(a)

1.8

–120

0

500

1,000
Frequency (Hz)
(b)

1,500

[FIG3] The growl analysis of a singing voice sample: (a) waveform and (b) spectrum with harmonics (dashed lines) andsubharmonics (circle).

IEEE SIGNAL PROCESSING MAGAZINE [59] november 2015

Tremolo [23] is the vibrato counterpart observed in intensity. It iscaused by the vibrato oscillation when the harmonic with the greatest
amplitude moves in frequency, increasing and decreasing the distanceto a formant, thus making the signal amplitude vary. Vibrato may be
used for two reasons [23, p. 172]. Acoustically, it prevents harmonics
from different voices from falling into close regions and producing
beatings. Also, vibratos are difficult to produce under phonatory difficulties such as pressed phonation. Aesthetically, vibrato shows that thesinger is not running into such problems when performing a difficult
note or phrase such as a high-pitched note.
Attack is the musical term to describe the pitch and intensity
contour shapes and duration at the beginning of a musical note orphrase. Release is the counterpart of attack, referring to the pitch
and intensity contour shapes at the end of a note or phrase.
As summarized in [27], grouping is one of the mental structures that are built while listening to a piece that describes thehierarchical relationships between different units. Notes, the lowest-level units, are grouped into motifs, motifs are grouped into
phrases, and phrases are grouped into sections. The piece is thehighest-level unit. Phrasing is a transversal aspect that can be represented as an “arch-like shape” applied to both the tempo andintensity during a phrase [15, p. 149]. For example, a singer may
increase the tempo at the beginning of a phrase or decrease it at
the end for classical music.
SINGING VOICE PERFORMANCE ANALYSIS
To illustrate the contribution of the acoustic features to expression,
we analyze a short excerpt from a real singing performance (anexcerpt from the song “Unchain My Heart;” see [51]). It contains
clear expressive features such as vibrato in pitch, dynamics, timing
deviations in rhythm, and growl in timbre. The result of the analysis is shown in Figures 2 and 3. (The dashed lines indicate harmonic frequencies, and the circle is placed at the subharmonics.)
The original score and lyrics are shown in Figure 2(a), where eachsyllable corresponds to one note except for the first and last ones,

which correspond to two notes. The singer introduces somechanges, such as ornamentation and syncopation, which are represented in Figure 2(b). In (c), the note pitch is specified by the
expected frequency in cents, and the note onsets are placed at the
expected time using the note figures and a 120-beats/minutetempo. Figure 2(d) shows the extracted F0 contour in blue and thenotes in green. The microprosody effects can be observed, forexample, in a pitch valley during the attack to the word “heart.” At
the end, vibrato is observed. The pitch stays at the target pitch for ashort period of time, especially in the ornamentation notes.
In a real performance, the tempo is not generally constantthroughout a score interpretation. In general, beats are not equally
spaced through time, leading to tempo fluctuation. Consequently,
note onsets and rests are not placed where expected with respect tothe score. In Figure 2(d), time deviations can be observed between
the labeled notes and the projection colored in red from the score.
Also, the note durations differ from the score.
The recording’s waveform and energy, which are aligned to theestimated F0 contour, are shown in Figure 2(e) and (f), respectively.
The intensity contour increases/decays at the beginning/end of eachsegment or note sequence. Energy peaks are especially prominent at
the beginning of each segment since a growl voice is used, andincreased intensity is needed to initiate this effect.
We can take a closer look at the waveform and spectrum of a windowed frame, as shown in Figure 3. In the former, we can see thepattern of a modulation in the amplitude or macroperiod, whichspans over several periods. In the latter, we can see that, for the windowed frame, apart from the frequency components related to F0
around 320 Hz, five subharmonic components appear between F0
harmonics, which give the growl voice quality. Harmonics aremarked with a dashed line and subharmonics between the second
and the third harmonics with a red circle.
If this set of acoustic features is synthesized appropriately, thesame perceptual aspects can be decoded. Several approaches thatgenerate these features are presented next.

Expression-Control
Approaches

Performance
DrivenNoniterative

Iterative

Statistical Modelingof Expression
Controls

Rule
Based

Analysis by
Synthesis

Unit Selectionof Expression
Controls

CorpusDerivedCorpus-Based Methods[FIG4] Classification of expression-control methods in singing voice synthesis.

IEEE SIGNAL PROCESSING MAGAZINE [60] november 2015

[TABLE 4] A COMPARISON OF APPROACHES FOR EXPRESSION CONTROL IN SINGING VOICE SYNTHESIS.
TYPE

REFERENCECONTROL FEATURESSYNTHESIZERSTYLE OR EMOTIONINPUTLANGUAGEPERFORMANCEDRIVEN[29]

UNIT SELECTIONOPERASAMPLE BASED

GENERIC[31]

TIMING, F0, INTENSITYSAMPLE BASED[32]

TIMING, F0, INTENSITY, TIMBRESAMPLE BASEDPOPULAR MUSIC(RWC-MDB)1
MUSIC GENRE(RWC-MDB)2SCORE,SINGING VOICELYRICS, MIDI NOTES,SINGING VOICELYRICS, SINGINGVOICELYRICS, SINGINGVOICEGERMAN[30]

TIMING, F0, INTENSITY,
SINGER’S FORMANT CLUSTERTIMING, F0, INTENSITY, VIBRATO[33]

TIMING, F0, SINGER FORMANTRESYNTHESISOF SPEECHCHILDREN’SSONGSSCORE, TEMPO,SPEECHJAPANESE[3]

TIMING, CONSONANT DURATION,VOWEL ONSET, TIMBRE CHANGES,FORMANT TUNING, OVERTONESINGING, ARTICULATION SILENCETO NOTE
TIMING, MICROPAUSES, TEMPO
AND PHRASING, F0, INTENSITY,
VIBRATO AND TREMOLO,TIMBRE QUALITYTIMBRE (MANUAL), PHONETICS,
TIMING, F0,
INTENSITY, MUSICAL
ARTICULATION, SUSTAINS,VIBRATO AND TREMOLO(RATE AND DEPTH)FORMANTSYNTHESISOPERASCORE, MIDI,
OR KEYBOARD

NOT SPECIFIEDSAMPLE BASEDANGRY, SAD,HAPPYSCORE, LYRICS,
SWEDISH,TEMPO, EXPRESSIVE ENGLISH
INTENTIONSSAMPLE BASED

GENERICSCORE, LYRICS,
TEMPOJAPANESE,ENGLISH,SPANISH[25]

TIMING, F0, TIMBREHMM BASED

CHILDREN’SSONGSSCORE AND LYRICSJAPANESE[42]

TIMING, F0, VIBRATO
AND TREMOLO,TIMBRE, SOURCEBASELINE F0 (RELATIVETO NOTE), VIBRATO RATE
AND DEPTH (NOT TREMOLO),INTENSITY

HMM BASED

CHILDREN’SSONGSMUSICXML2SCORE

JAPANESE,ENGLISHSAMPLE BASED

CHILDREN’SSONGSSCORE (NOLYRICS TO
CREATE MODELS)

JAPANESESAMPLE BASEDJAZZ STANDARDSSCORELANGUAGE
INDEPENDENT

RULE
BASED

[37]

[40]

STATISTICAL
MODELING

[22]

UNIT
SELECTION
1
2

[43]

F0, VIBRATO, TREMOLO,INTENSITYSPANISH
JAPANESE
JAPANESE

Real World Computing (RWC) Music Database: https://staff.aist.go.jp/m.goto/RWC-MDB/
MUSICXML: http://www.musicxml.com

EXPRESSION-CONTROL APPROACHESIn the section “Singing Voice Performance Features,” we defined
the voice acoustic features and related them to aspects of music
perception. In this section, we focus on how different approaches
generate expression controls. First, we propose a classification of
the reviewed approaches and then we compare and describe them.
As will be seen, acoustic features generally map one to one toexpressive controls at the different temporal scopes, and the synthesizer is finally controlled by the lowest-level acoustic features(i.e., F0, intensity, and spectral envelope representation).
CLASSIFICATION OF APPROACHESTo see the big picture of the reviewed works on expression control,
we propose a classification in Figure 4. Performance-driven
approaches use real performances as the control for a synthesizer,
taking advantage of the implicit rules that the singer has applied
to interpret a score. Expression controls are estimated and applied
directly to the synthesizer. Rule-based methods derive a set ofrules that reflect the singers’ cognitive process. In analysis by synthesis, rules are evaluated by synthesizing singing voice

performances. Corpus-derived, rule-based approaches generate
expression controls from the observation of singing voice contours
and imitating their behavior. Statistical approaches generate singing voice expression features using techniques such as hidden
Markov models (HMMs). Finally, unit selection-based approaches
select, transform, and concatenate expression contours fromexcerpts of a singing voice ­database (DB). Approaches using a
training database of expressive singing have been labeled as corpus-based methods. The difficulties of the topic reviewed in thisarticle center on how to generate control parameters that are perceived as natural. The success of conveying natural expression
depends on a comprehensive control of the acoustic features
introduced in the section “Singing Voice Performance Features.”
Currently, statistical approaches are the only type of system thatjointly model all of the expression features.
COMPARISON OF APPROACHESIn this article, we review a set of works that model the featuresthat control singing voice synthesis expression. Physical modelingperspective approaches can be found, for instance, in [28].

IEEE SIGNAL PROCESSING MAGAZINE [61] november 2015

NotesPhonetic
Alignment

Lyrics

Notes and Lyrics
Extract Acoustic
Features

Timing

Mapping

Controls

Synthesizer

Sound

Recording
F0, Dynamics,
Timbre, Vibrato (Rateand Depth)

Extract Acoustic
FeaturesUpdateParameters(Only Iterative Approaches)[FIG5] The general framework for performance-driven approaches.

Within each type of approach in Figure 4, there are one or moremethods for expression control. In Table 4, we provide a set ofitems that we think can be useful for comparison. The “Type” column refers to the type of expression control from Figure 4 towhich the reference belongs. In the “Control Feature” column, welist the set of features addressed by the approach. The “Synthesizer”
column provides the type of synthesizer used to generate the singing voice, and the “Style or Emotion” column provides the emotion, style, or sound to which the expression is targeted. The“Input” column details the input to the system (e.g., the score, lyrics, tempo, and audio recording). The “Language” column lists thelanguage dependency of each method, if any.
We have collected samples in [51] as examples of the results of
the reviewed expression-control approaches. Listeners will observe
several differences among them. First, some samples consist of a cappella singing voices, and others are presented with backgroundmusic, which may mask the synthesized voice and complicate theperception of the generated expression. Second, the samples correspond to different songs, making it difficult to compare approaches.
Athough the lyrics in most cases belong to a particular language, insome samples, they are made by repeating the same syllable, such as/la/. We believe that the evaluation of a synthesized song can be

performed more effectively in a language spoken by the listener.
Finally, the quality of the synthetic voice is also affected by the type ofsynthesizer used in each sample. The difficulties in comparing themand the subsequent criticisms are discussed in the “Evaluation” and“Challenges” sections.
PERFORMANCE-DRIVEN APPROACHES
Performance-driven approaches use a real performance to control
the synthesizer. The knowledge applied by the singer, implicit in
the extracted data, can be used in two ways. In the first one, control
parameters such as F0, intensity, and timing from the reference
recording are mapped to the input controls of the synthesizer sothat the rendered performance follows the input signal expression.
Alternatively, speech audio containing the target lyrics is transformed to match the pitch and timing of the input score. Figure 5
summarizes the commonalities of these approaches on the inputs(reference audio, lyrics, and, possibly, the note sequence) and intermediate steps (phonetic alignment, acoustic feature extraction, andmapping) that generate internal data such as timing information,
acoustic features, and controls used by the synthesizer.
In Table 5, we summarize the correspondence between theextracted acoustic features and the synthesis parameters for each of[TABLE 5] MAPPING FROM ACOUSTIC FEATURES TO SYNTHESIZER CONTROLS.
MAPPED SYNTHESIS PARAMETERS
ACOUSTICFEATURES

[29]

[30]

[31]

[32]

[33]

F0

F0

SMOOTHED AND
CONTINUOUS PITCH

INCLUDED IN F0
IMPLICITLY
DYNAMICSPHONEME TIMINGVIBRATOS FROM INPUTOR FROM DB SINGER
DYNAMICSONSETS OF VOWELSOR VOICED PHONEMES

MIDI NOTE NUMBER,
PITCH BEND ANDSENSITIVITY
INCLUDED IN F0
IMPLICITLY
DYNAMICSNOTE ONSET AND
DURATIONF0

VIBRATOMIDI NOTE NUMBER,
PITCH BEND ANDSENSITIVITY
INCLUDED IN F0
IMPLICITLY
DYNAMICSNOTE ONSET AND
DURATION

SINGER’S FORMANT
CLUSTER AMPLITUDENOT USEDNOT USEDMIXING DIFFERENTVOICE QUALITY dBs

SINGER’S FORMANT CLUSTERAMPLITUDE AND Amplitude
Modulation OF THE
SYNTHESIZED SIGNAL

ENERGY
PHONETIC
ALIGNMENTTIMBRE

IEEE SIGNAL PROCESSING MAGAZINE [62] november 2015

INCLUDED IN F0
IMPLICITLY
DYNAMICSPHONEME TIMING

these works. The extracted F0 can be mapped directly into the F0control parameter, processed into a smoothed and continuous version, or split into the Musical Instrument Digital Instrument(MIDI) note, pitch bend, and its sensitivity parameters. Vibrato canbe implicitly modeled in the pitch contour, extracted from the
input, or selected from a database. Energy is generally mapped
directly into dynamics. From the phonetic alignment, note onsets
and durations are derived, mapped directly to phoneme timing, or
mapped either to onsets of vowels or voiced phonemes. Concerning
timbre, some approaches focus on the singer’s formant cluster,and, in a more complex case, the output timbre comes from a mixture of different voice quality databases.
Approaches using estimated controls achieve different levels ofrobustness depending on the singing voice synthesizers and voice
databases. In the system presented in [29], a unit selection framework is used to create a singing voice synthesizer from a particularsinger’s recording in a nearly automatic procedure. In comparison
to a sample-based system, where the design criterion is to minimize the size of the voice database with only one possible unit sample (e.g., diphones), the criterion in unit selection is related to
redundancy to allow the selection of consecutive units in the database at the expense of having a larger database. The system automatically segments the recorded voice into phonemes by aligningit to the score and feeding the derived segmentation constraints toan HMM recognition system. Units are selected to minimize a costfunction that scores the amount of time, frequency, and timbre
transformations. Finally, units are concatenated. In this approach,the main effort is put on the synthesis engine. Although it uses aunit selection-based synthesizer, the expression controls for pitch,timing, dynamics, and timbre, like the singer’s formant, areextracted from a reference singing performance of the target score.
These parameters are directly used by the synthesizer to modify theselected units with a combination of sinusoidal modeling (SM)with time domain pitch synchronous overlap add (TD-PSOLA)
called SM-PSOLA. Editing is allowed by letting the user participate
in the unit selection process, change some decisions, and modify
the unit boundaries. Unfortunately, this approach only manipulates
the singer’s formant feature of timbre so that other significant timbre-related features in the opera singing style are not handled.
In [30], the steps followed are: extraction of acoustic features
such as energy, F0, and automatic detection of vibrato sections;
mapping into synthesis parameters; and phonetic alignment. The
mapped controls and the input score are used to build an internalscore that matches the target timing, pitch, and dynamics, andminimizes the transformation cost of samples from a database.
However, this approach is limited since timbre is not handled andalso because the expression features of the synthesized performance are not compared to the input values. Since this approachlacks a direct mapping of acoustic features to control parameters,
these differences are likely to happen. On the other hand, the possibility of using a singer DB to produce vibratos other than theextracted ones from the reference recording provides a new degreeof freedom to the user.
Toward a more robust methodology to estimate the parameters,in [31], the authors study an iterative approach that takes thetarget singing performance and lyrics as input. The musical scoreor note sequence is automatically generated from the input. The
first iteration provides an initialization of the system similar to theprevious approach [30]. At this point, these controls can be manually edited by applying pitch transposition, correction, vibrato modifications, and pitch and intensity smoothing. The iterative processcontinues by analyzing the synthesized waveform and adjusting thecontrol parameters so that, in the next iteration, the results arecloser to the expected performance. In [32], the authors extend thisapproach by including timbre. Using different voice quality databases from the same singer, the corresponding versions of the target song are synthesized as in the previous approach. The system
extracts the spectral envelopes of each one to build a three-dimensional (3-D) voice timbre space. Next, a temporal trajectory in thisspace is estimated from the reference target performance to represent its spectral timbre changes. Finally, singing voice synthesis
output is generated using the estimated trajectory to imitate thetarget timbre change. Although expression control is more robust
than the previous approach, thanks to iteratively updating theparameters and by allowing a certain degree of timbre control,
these approaches also have some limitations. First, it cannot be
assured that the iterative process will converge to the optimal set ofparameter values. Second, the timbre control is limited to the variability within the set of available voice quality databases.
In [33], naturally spoken readings of the target lyrics are transformed into a singing voice by matching the target song properties
described in the musical score. Other input data are the phonetic
segmentation and the synchronization of phonemes and notes.
This approach first extracts acoustic features such as F0, spectralenvelope, and the aperiodicity index from the input speech. Then, acontinuous F0 contour is generated from discrete notes, phoneme
durations are lengthened, and the singer’s formant cluster is generated. The fundamental frequency contour takes into account fourtypes of fluctuations: 1) overshoot (F0 exceeds the target note after
a note change), 2) vibrato, 3) preparation (similar to overshootbefore the note change), and 4) fine fluctuations. The first three
types of F0 fluctuations are modeled by a single second-order transfer function that depends mainly on a damping coefficient, a gain
factor, and a natural frequency. A rule-based approach is followed
for controlling phoneme durations by splitting consonant-to-voweltransitions into three parts. First, the transition duration is not[TABLE 6] SINGING VOICE-RELATED KTH RULES’ DEPENDENCIES.
ACOUSTIC FEATURECONSONANT DURATIONVOWEL ONSETFORMANT FREQUENCIESFORMANT FREQUENCIES
SPECTRUM SLOPEVIBRATOPITCH IN COLORATURAPASSAGESPITCH PHRASE ATTACK(AND RELEASE)IEEE SIGNAL PROCESSING MAGAZINE [63] november 2015

DEPENDENCIESPREVIOUS VOWEL LENGTHSYNCHRONIZED WITH TIMINGVOICE CLASSIFICATIONPITCH, IF OTHERWISE F0 WOULD EXCEEDTHE FIRST FORMANTDECREASE WITH INCREASING INTENSITY
INCREASE DEPTH WITH INCREASING
INTENSITY
EACH NOTE REPRESENTED AS A VIBRATOCYCLEAT PITCH START (END) FROM (AT) 11
SEMITONES BELOW TARGET F0

modified for singing. Then, the consonant part is transformed
based on a comparative study of speech and singing voices. Finally,
the vowel section is modified so that the duration of the three parts
matches the note duration. With respect to timbre, the singer’s formant cluster is handled by an emphasis function in the spectral
domain centered at 3 kHz. Amplitude modulation is also applied tothe synthesized singing voice according to the generated vibrato’s
parameters. Although we have classified this approach as performance-driven since the core data are found in the input speechrecording, some aspects are modeled, such as the transfer function
for F0, rules for phonetic duration, and a filter for the singer’s formant cluster. Similarly to [29], in this approach, timbre control islimited to the singer formant, so the system cannot change othertimbre features. However, if the reference speech recording contains voice quality variations that fit the target song, this can add
some naturalness to the synthesized singing performance.
Performance-driven approaches achieve a highly expressive control since performances implicitly contain knowledge naturally
applied by the singer. These approaches are especially convenient for
creating parallel database recordings, which are used in voice conversion approaches [8]. On the other hand, the phonetic segmentation may cause timing errors if not manually corrected. The
noniterative approach lacks robustness because the differences
between the input controls and the controls extracted from the synthesized sound are not corrected. In [32], timbre control is limitedby the number of available voice qualities. We note that a humanvoice input for natural singing control is required for theseapproaches, which can be considered a limitation since it may not
be available in most cases. When such a reference is not given, otherapproaches are necessary to derive singing control parameters fromthe input musical score.
RULE-BASED APPROACHES
Rules can be derived from work with synthesizing and analyzingsung performances. Applying an analysis-by-synthesis method, an
ambitious rule-based system for western music was developed at
KTH in the 1970s and improved over the last three decades [3]. By

synthesizing sung performances, this method aims at identifying
acoustic features that are perceptually important, either individuallyor jointly [15]. The process of formulating a rule is iterative. First, atentative rule is formulated and implemented and the resulting synthesis is assessed. If its effect on the performance needs to bechanged or improved, the rule is modified and the effect of theresulting performance is again assessed. On the basis of parameterssuch as phrasing, timing, metrics, note articulation, and intonation,
the rules modify pitch, dynamics, and timing. Rules can be combined
to model emotional expressions as well as different musical styles.
Table 6 lists some of the acoustic features and their dependencies.
The rules reflect both physical and musical phenomena. Somerules are compulsory and others optional. The consonant duration
rule, which lengthens consonants following short vowels, also
applies to speech in some languages. The vowel onset rule corresponds to the general principle that the vowel onset is synchronized with the onset of the accompaniment, even though lag andlead of onset are often used for expressive purposes [34]. The spectrum slope rule is compulsory as it reflects the fact that vocal
loudness is controlled by subglottal pressure and an increase ofthis pressure leads to a less steeply sloping spectrum envelope.
The pitch in coloratura passages rule implies that the fundamental
frequency makes a rising–falling gesture around the target frequency in legato sequences of short notes [35]. The pitch phrase
attack, in lab jargon referred to as bull’s roaring onset, is an ornament used in excited moods and would be completely out of place
in a tender context. Interestingly, results close to the KTH rules
have been confirmed by machine-learning approaches [36].
A selection of the KTH rules [15] has been applied to the Vocaloid synthesizer [37]. Features are considered at the note level(start and end times), intra- and internote (within and between
note changes), and to timbre variations (not related to KTH
rules). The system implementation is detailed in [38] along withthe acoustic cues, which are relevant for conveying basic emotions such as anger, fear, happiness, sadness, and love/tenderness[12]. The rules are combined in expressive palettes indicating towhat degree the rules need to be applied to convey a target[TABLE 7] The SELECTION OF RULES FOR SINGING VOICE: LEVEL OF APPLICATION AND AFFECTED ACOUSTIC FEATURES.
LEVEL
NOTE

INTRA-/INTERNOTETIMBRERULES
DURATION CONTRAST

AFFECTED ACOUSTIC FEATURESDECREASE DURATION AND INTENSITY OF SHORT NOTES PLACED NEXT TO LONG NOTESPUNCTUATIONTEMPO
INTENSITY
TRANSITIONSPHRASING ARCH
FINAL RITARDANDOINSERT MICROPAUSES IN CERTAIN PITCH INTERVAL AND DURATION COMBINATIONSCONSTANT VALUE FOR THE NOTE SEQUENCE (MEASURED IN BEATS/min)
SMOOTH/STRONG ENERGY LEVELS, HIGH PITCH NOTES INTENSITY INCREASES 3 dB/OCTAVELEGATO, STACCATO (PAUSE IS SET TO MORE THAN 30% OF INTERONSET INTERVAL)INCREASE/DECREASE TEMPO AT PHRASE BEGINNING/END, SAME FOR ENERGYDECREASE TEMPO AT THE END OF A PIECEATTACKNOTE ARTICULATIONRELEASEPITCH SHAPE FROM STARTING PITCH UNTIL TARGET NOTE, ENERGY INCREASES SMOOTHLYPITCH SHAPE FROM THE STARTING TO THE ENDING NOTE, SMOOTH ENERGYENERGY DECREASES SMOOTHLY TO zero, DURATION IS MANUALLY EDITEDVIBRATOAND TREMOLOMANUAL CONTROL OF POSITION, DEPTH, AND RATE (COSINE FUNCTION and RANDOM
FLUCTUATIONS)BRIGHTNESSROUGHNESS
BREATHINESSINCREASE HIGH FREQUENCIES DEPENDING ON ENERGYSPECTRAL IRREGULARITIESMANUAL CONTROL OF NOISE LEVEL (NOT INCLUDED IN EMOTION PALETTES)IEEE SIGNAL PROCESSING MAGAZINE [64] november 2015

[TABLE 8] Contextual factors in HMM-based systems.
HMM-BASED APPROACHES
[25]
[42]

LEVELS
PHONEMENOTE

CONTEXTUAL FACTORS
P/C/N PHONEMES
P/C/N NOTE F0, DURATIONS, AND POSITIONS WITHIN THE MEASURE

PHONEMEMORA

FIVE PHONEMES (CENTRAL AND TWO PRECEDING AND SUCCEEDING)NUMBER OF PHONEMES IN THE P/C/N MORAPOSITION OF THE P/C/N MORA IN THE NOTEMUSICAL TONE, KEY, TEMPO, LENGTH, AND DYNAMICS OF THE P/C/N NOTEPOSITION OF THE CURRENT NOTE IN THE CURRENT MEASURE AND PHRASE
TIES AND SLURRED ARTICULATION FLAGDISTANCE BETWEEN CURRENT NOTE AND NEXT/PREVIOUS ACCENT AND STACCATO
POSITION OF THE CURRENT NOTE IN THE CURRENT CRESCENDO OR DECRESCENDONUMBER OF PHONEMES AND MORAS IN THE P/C/N PHRASENUMBER OF PHONEMES, MORAS, AND PHRASES IN THE SONGNOTEPHRASESONG[22]

NOTE REGIONNOTEMANUALLY SEGMENTED BEHAVIOR TYPES (BEGINNING, SUSTAINED, ENDING)
MIDI NOTE NUMBER AND DURATION (IN 50-millisecond UNITS)
DETUNING: MODEL F0 BY THE RELATIVE DIFFERENCE TO THE NOMINAL NOTE

P/C/N: PREVIOUS, CURRENT, AND NEXT.

emotion. The relationship between application level, rules, andacoustic features is shown in Table 7. As an example of the complexity of the rules, the punctuation rule at the note level inserts
a 20-millisecond micropause if a note is three tones lower than
the next one and its duration is 20% larger. Given that this work
uses a sample-based synthesizer, voice quality modifications areapplied to the retrieved samples. In this case, the timbre variations are limited to rules affecting brightness, roughness, andbreathiness and, therefore, do not cover the expressive possibilities of a real singer.
Apart from the KTH rules, in corpus-derived rule-based systems, heuristic rules are obtained to control singing expression
by observing recorded performances. In [6], expression controls
are generated from high-level performance scores where theuser specifies the note articulation, pitch, intensity, and vibrato
data that are used to retrieve templates from recorded samples.
This work, used in the Vocaloid synthesizer [39], models the
singer’s performance with heuristic rules [40]. The parametricmodel is based on anchor points for pitch and intensity, which
are manually derived from the observation of a small set ofrecordings. At synthesis, the control contours are obtained byinterpolating the anchor points generated by the model. The
number of points used for each note depends on its absolute
duration. The phonetics relationship with timing is handled bysynchronizing the vowel onset with the note onset. Moreover,manual editing is permitted for the degree of articulation application as well as its duration, pitch and dynamics contours, phonetic transcription, timing, vibrato and tremolo depth and rate,
and timbre characteristics.
The advantage of these approaches is that they are relatively
straightforward and completely deterministic. Random variations
can be easily introduced so that the generated contours are different for each new synthesis of the same score, resulting in distinct
interpretations. The main drawbacks are that either the models
are based on few observations that do not fully represent a given
style or they are more elaborate but become unwieldy due to thecomplexity of the rules.

STATISTICAL MODELING APPROACHES
Several approaches have been used to statistically model and­characterize expression-control parameters using HMMs. They
have a common precedent in speech synthesis [41], where theparameters such as spectrum, F0, and state duration are jointly
modeled. Compared to unit selection, HMM-based approaches tend
to produce lower speech quality, but they need a smaller data set totrain the system without needing to cover all combinations of contextual factors. Modeling a singing voice with HMMs amounts tousing similar contextual data as those used for speech synthesis,
adapted to singing voice specificities. Moreover, new voice characteristics can be easily generated by changing the HMM parameters.
These systems operate in two phases: training and synthesis.
In the training part, acoustic features are first extracted from thetraining recordings, such as F0, intensity, vibrato parameters, andMel-cepstrum coefficients. Contextual labels, i.e., the relationships of each note, phoneme, or phrase with the preceding andsucceeding values, are derived from the corresponding score andlyrics. Contextual labels vary in their scope at different levels,
such as phoneme, note, or phrase, according to the approach, assummarized in Table 8. This contextual data are used to build theHMMs that relate how these acoustic features behave according
to the clustered contexts. The phoneme timing is also modeled insome approaches. These generic steps for the training part in
HMM-based synthesis are summarized in Figure 6. The figure
shows several blocks found in the literature, which might not bepresent simultaneously in each approach. We refer to [41] for thedetailed computations that HMM training involves.
In the synthesis part, given a target score, contextual labels arederived as in the training part from the note sequence and lyrics.
Models can be used in two ways. All necessary parameters for singing
voice synthesis can be generated from them; therefore, state durations, F0, vibrato, and Mel-cepstrum observations are generated tosynthesize the singing voice. On the other hand, if another synthesizer is used, only control parameters, such as F0, vibrato depth andrate, and dynamics need to be generated, which are then used as
input of the synthesizer.

IEEE SIGNAL PROCESSING MAGAZINE [65] november 2015

ScoreContext-Dependent
Label Extraction

FeatureExtraction

Singing Voice
Recordings

Preprocessing
F0 Vib Pow

MelCepstrum

F0, MFCC
Models

HMM Training
State Duration
ModelsForced Alignment

Phoneme Timing(Time Lags)

Clustering

Time Lags
Generation
Models[FIG6] Generic blocks for the training part of HMM-based approaches.

As introduced in the section “Classification of Approaches,” statistical methods jointly model the largest set of expression features
among the reviewed approaches. This gives them a better generalization ability. As long as singing recordings for training involve different voice qualities, singing styles or emotions, and the target
language phonemes, these will be reproducible at synthesis given theappropriate context labeling. Model interpolation allows new models
to be created as a combination of existing ones. New voice qualities
can be created by modifying the timbre parameters. However, thisflexibility is possible at the expense of having enough training recordings to cover the combinations of the target singing styles and voice
qualities. In the simplest case, a training database of a set of songs
representing a single singer and style in a particular language would
be enough to synthesize it. As a drawback, training HMMs with large
databases tends to produce smoother time series than the original
training data, which may be perceived as unnatural.
In [25], a corpus-based singing voice synthesis system basedon HMMs is presented. The contexts are related to phonemes,note F0 values, and note durations and positions, as we show inTable 8 (dynamics are not included). Also, synchronization
between notes and phonemes needs to be handled adequately,
mainly because phoneme timing does not strictly follow the scoretiming, and phonemes might be advanced with respect to thenominal note onsets (negative time lag).
In this approach, the training part generates three models: 1) for
the spectrum and excitation (F0) parts extracted from the training database, 2) for the duration of context-dependent states, and 3)

to model the time lag. The second and third model note timing andphoneme durations of real performances, which are different thanwhat can be inferred from the musical score and its tempo. Timelags are obtained by forced alignment of the training data with context-dependent HMMs. Then, the computed time lags are related totheir contextual factors and clustered by a decision tree.
The singing voice is synthesized in five steps: 1) the input score(note sequence and lyrics) is analyzed to determine note durationand contextual factors, 2) a context-dependent label sequence ofcontextual factors as shown in Table 8 is generated, 3) the song
HMM is generated, 4) its state durations are jointly determinedwith the note time lags, and 5) spectral and F0 parameters are generated, which are used to synthesize the singing voice. The authors
claim that the synthesis performance achieves a natural singing
voice, which simulates expression elements of the target singer
such as voice quality and singing style (i.e., F0 and time lag).
In [25], the training database consists of 72 minutes of a malevoice singing 60 Japanese children’s songs in a single voice quality.
These are the characteristics that the system can reproduce in atarget song. The main limitation of this approach is that the contextual factors scope is designed to only cover phoneme and note
descriptors. Longer scopes than just the previous and next noteare necessary to model higher-level expressive features such asphrasing. Although we could not get samples from this work, anevolved system is presented next.
The system presented in [25] has been improved and is publicly
available as Sinsy, an online singing voice synthesizer [42]. The newIEEE SIGNAL PROCESSING MAGAZINE [66] november 2015

characteristics of the system include reading input files inMusicXML format with F0, lyrics, tempo, key, beat, and dynamics aswell as extended contextual factors used in the training part, vibrato
rate and depth modeling, and a reduction of the computational cost.
Vibrato is jointly modeled with the spectrum and F0 by including
the depth and rate in the observation vector in the training step.
The new set of contexts automatically extracted from the musicalscore and lyrics used by the Sinsy approach are also shown in Table 8.
These factors describe the context such as previous, current, and next
data at different hierarchical levels: phoneme, mora (the sound unit
containing one or two phonemes in Japanese), note, phrase, and theentire song. Some of them are strictly related to musical expression
aspects, such as musical tone, key, tempo, length and dynamics ofnotes, articulation flags, or distance to accents and staccatos.
Similar to [25], in this case, the training database consists of
70 minutes of a female voice singing 70 Japanese children’s songs
in a single voice quality. However, it is able to reproduce more
realistic expression control since vibrato parameters are also
extracted and modeled. Notes are described with a much richer
set of factors than the previous work. Another major improvement is the scope of the contextual factors shown in Table 8,
which spans from the phoneme level to the whole song and is,therefore, able to model phrasing.
In [22], a statistical method is able to model singing styles.
This approach focuses on baseline F0; vibrato features such as its
extent, rate, and evolution over time, not tremolo; and dynamics.
These parameters control the Vocaloid synthesizer, and so the timbre is not controlled by the singing style modeling system but isdependent on the database.
A preprocessing step is introduced after extracting the acoustic features such as F0 and dynamics to get rid of the microprosody effects on such parameters by interpolating F0 in unvoiced
sections and flattening F0 valleys of certain consonants. The
main assumption here is that expression is not affected byphonetics, which is reflected in erasing such dependencies in theinitial preprocessing step and also in training note HMMs insteadof phoneme HMMs. Also, manual checking is done to avoid errors
in F0 estimation and MIDI events such as note on and note off
estimated from the phonetic segmentation alignment. A novel
approach estimates the vibrato shape and rate, which at synthesis
is added to the generated baseline melody parameter. The shape isrepresented with the low-frequency bins of the Fourier transformof single vibrato cycles. In this approach, context-dependent
HMMs model the expression parameters summarized in Table 8.
Feature vectors contain melody, vibrato shape and rate, anddynamics components.
This last HMM-based work focuses on several control features
except for timbre, which is handled by the Vocaloid synthesizer.
This makes the training database much smaller in size. It consistsof 5 minutes of five Japanese children’s songs since there is noneed to cover a set of phonemes. Contextual factors are rich at
the note level since the notes are divided into three parts (begin,sustain, and end), and the detuning is also modeled relative to thenominal note. On the other hand, this system lacks the modelingof wider temporal aspects such as phrasing.
UNIT SELECTION APPROACHES
The main idea of unit selection [29] is to use a database of singing
recordings segmented into units that consist of one or more phonemes or other units such as diphones or half phones. For a target
score, a sequence of phonemes with specific features such as pitchor duration is retrieved from the database. These are generallytransformed to match the exact required characteristics.
An important step in this kind of approach is the definition of
the target and concatenation cost functions as the criteria on which
unit selection is built. The former is a distance measure of the unittransformation in terms of a certain acoustic feature such as pitch or
duration. Concatenation costs measure the perceptual consequences

Source Unit A
Source Unit A
F0 Contour

Labeled Note
F0 Source Unit A
F0 Source Unit B
Generated Target F0

Source Unit B
Source Unit B
F0 Contour

Transformed Source Unit A
Transformed Source Unit B
Target Note
SequenceTime[FIG7] The performance feature (F0) generated by unit selection.

IEEE SIGNAL PROCESSING MAGAZINE [67] november 2015

[TABLE 9] UNIT SELECTION COST FUNCTIONS.
COSTNOTE DURATIONPITCH INTERVALCONCATENATIONCONTINUITYDESCRIPTIONCOMPARE SOURCE AND TARGET UNIT NOTE DURATIONSCOMPARE SOURCE AND TARGET UNIT NOTE INTERVALSFAVOR COMPATIBLE UNITS FROM THE DB
FAVOR SELECTION OF CONSECUTIVE UNITS

of joining nonconsecutive units. These cost functions’ contributions
are weighted and summed to get the overall cost of the unit sequence.
The goal is then to select the sequence with the lowest cost.
Unit selection approaches present the disadvantages of requiring a large database, which needs to be labeled, and the subcost
weights need to be determined. On the other hand, the voice quality and naturalness are high because of the implicit rules applied by
the singer within the units.
A method to model pitch, vibrato features, and dynamics basedon selecting units from a database of performance contours has
recently been proposed [43]. We illustrate it in Figure 7 for the F0
contour showing two selected source units for a target note sequence
where units are aligned at the transition between the second andthird target notes. The target note sequence is used as input to generate the pitch and dynamics contours. A reference database is usedthat contains extracted pitch, vibrato features, and dynamics fromexpressive recordings of a single singer and style. In addition to thesefeatures, the database is labeled with the note pitches, durations, andstrength as well as the start and end times of note transitions. Thisapproach splits the task of generating the target song expression contours into first finding similar and shorter note combinations (source
units A and B in Figure 7), and then transforming and concatenating
the corresponding pitch and dynamics contours to match the target
score (the dashed line in F
­ igure 7). These shorter contexts are the socalled units, defined by three consecutive notes or silences, so thatconsecutive units overlap by two notes. The contour of dynamics isgenerated similarly from the selected units.
With regard to unit selection, the cost criterion consists of the
combination of several subcost functions, as summarized inTable 9. In this case, there are four functions and unit selection is
implemented with the Viterbi algorithm. The overall cost function
considers the amount of transformation in terms of note durations(note duration cost) and pitch interval (pitch interval cost) to preserve as much as possible the contours as originally recorded. It
also measures how appropriate it is to concatenate two units (concatenation cost) as a way of penalizing the concatenation of units
from different contexts. Finally, the overall cost function also favors
the selection of long sequences of consecutive notes (continuitycost), although the final number of consecutive selected units
depends on the resulting cost value. This last characteristic is relevant to be able to reflect the recorded phrasing at synthesis.
Once a sequence is retrieved, each unit is time scaled and pitch
shifted. The time scaling is not linear; instead, most of the transformation is applied in the sustain part and keeping the transition(attacks and releases) durations as close to the original as possible.
Vibrato is handled with a parametric model, which allows the original rate and depth contour shapes to be kept.

COMPUTATIONOCTAVE RATIO (SOURCE/TARGET UNIT NOTES)OCTAVE RATIO (SOURCE/TARGET UNIT INTERVALS)ZERO IF CONSECUTIVE UNITSPENALIZE SELECTION OF NONCONSECUTIVE UNITS

The transformed unit contours are overlapped and added after
applying a cross-fading mask, which mainly keeps the shape of
the attack to the unit central note. This is done separately for theintensity, baseline pitch and vibrato rate, and vibrato depth contours. The generated baseline pitch is then tuned to the target
note pitches to avoid strong deviations. Then, vibrato rate anddepth contours are used to compute the vibrato oscillations,which are added to the baseline pitch.
The expression database contains several combinations ofnote durations, pitch intervals, and note strength. Such a database can be created systematically [44] to cover a relevant portionof possible units. Notes are automatically detected and then manually checked. Vibrato sections are manually segmented, and thedepth and rate contours are estimated. An important characteristic of such a database is that it does not contain sung text, only
sung vowels to avoid microprosody effects when extracting pitchand dynamics.
This approach controls several expression features except for
timbre aspects of the singing voice. In our opinion, a positive characteristic is that it can generate expression features without suffering from smoothing as is the case in HMMs. The selected units
contain the implicit rules applied by the singer to perform a vibrato,
an attack, or a release. In addition, the labeling and cost functions
for unit selection are designed in a way that favors the selection oflong sequences of consecutive notes in the database to help the
implicit reproduction of high expression features such as phrasing.
Similarly to the KTH rules, this approach is independent of phonetics since this is handled separately by the controlled synthesizer,
which makes it convenient for any language. The lack of an explicit
timbre control could be addressed in the future by adding control
features such as the degree of breathiness or brightness.
WHEN TO USE EACH APPROACH
The use of each approach has several considerations: the limitationsof each one; whether singing voice recordings are available sincethese are needed in model training or unit selection; the reason forsynthesizing a song, which could be for database creation or rule
testing; or flexibility requirements such as model interpolation. In
this section, we provide brief guidelines on the suitability of eachtype of approach.
Performance-driven approaches are suitable to be applied, by
definition, when the target performance is available, since theexpression of the singer is implicit in the reference audio and it
can be used to control the synthesizer. Another example of applicability is the creation of parallel databases for different purposessuch as voice conversion [8]. An application example for the caseof speech to singing synthesis is the generation of singing

IEEE SIGNAL PROCESSING MAGAZINE [68] november 2015

performances for untrained singers, whose timbre is taken fromthe speech recording and the expression for pitch and dynamics
can be obtained from a professional singer.
Rule-based approaches are suitable to be applied to verify thedefined rules and also to see how these are combined, for example,
to convey a certain emotion. If no recordings are available, rules
can still be defined with the help of an expert so that theseapproaches are not fully dependent on singing voice databases.
Statistical modeling approaches are also flexible, given that it ispossible to interpolate models and create new voice characteristics.
They have the advantage that, in some cases, these are part of complete
singing voice synthesis systems, i.e., those that have the score as input
and that generate both the expression parameters and output voice.
Similarly to rule-based and statistical modeling approaches,
unit selection approaches do not need the target performance,
although they can benefit from it. On the other hand, unit selection approaches share a common characteristic with performancedriven approaches. The implicit knowledge of the singer iscontained in the recordings, although in unit selection it isextracted from shorter audio segments. Unlike statistical models,no training step is needed, so the expression databases can be
improved just by adding new labeled singing voice recordings.
EVALUATIONIn the beginning of this article, we explained that a score can beinterpreted in several acceptable ways, making expression a

subjective aspect to rate. However, “procedures for systematic andrigorous evaluation do not seem to exist today” [1, p. 105], especially if there is no ground truth to compare with. In this section,
we first summarize typical evaluation strategies. Then, we propose
the initial ideas to build a framework that solves some detectedissues, and finally, we discuss the need for automatic measures torate expression.
CURRENT EVALUATION STRATEGIES
Expression control can be evaluated from subjective or objective
perspectives. The former typically consists of listening tests whereparticipants perceptually evaluate some psychoacoustic characteristic such as voice quality, vibrato, and overall expressiveness ofthe generated audio files. A common scale is the mean opinion
score (MOS), with a range from one (bad) to five (good). In pairwise comparisons, using two audio files obtained with different
system configurations, preference tests rate which option
achieves a better performance. Objective evaluations help to compare how well the generated expression controls match a reference real performance by computing an error.
Within the reviewed works, subjective tests outnumber theobjective evaluations. The evaluations are summarized inTable 10. For each approach, several details are provided such as a
description of the evaluation (style, voice quality, naturalness,
expression, and singer skills), the different rated tests, and information on the subjects if available. Objective tests are done only

[TABLE 10] CONDUCTED SUBJECTIVE AND OBJECTIVE EVALUATIONS PER APPROACH.
TESTS
TYPE

APPROACH

METHODDESCRIPTIONSUBJECTSPERFORMANCEDRIVEN[29]

SUBJECTIVERATE VOICE QUALITY WITH PITCH MODIFICATION OFTEN PAIRS OF SENTENCES (SM-PSOLA versus TD-PSOLA)

10 SUBJECTS[30]SUBJECTIVE

INFORMAL LISTENING TESTNOT SPECIFIED[31]

OBJECTIVETWO TESTS: LYRICS ALIGNMENT AND MEAN ERRORVALUE OF EACH ITERATION FOR F0 AND INTENSITYCOMPARED TO TARGETNO SUBJECTS[32]OBJECTIVETWO TESTS: 3-D VOICE TIMBRE REPRESENTATIONAND EUCLIDEAN DISTANCE BETWEEN REAL AND
MEASURED TIMBRENO SUBJECTS[33]

SUBJECTIVE

PAIRED COMPARISONS OF DIFFERENT CONFIGURATIONSTO RATE NATURALNESS OF SYNTHESIS IN A SEVEN-STEPSCALE (−3 TO 3)

10 STUDENTS WITH NORMALHEARING ABILITY[3]SUBJECTIVE

LISTENING TESTS OF PARTICULAR ACOUSTIC FEATURES

15 SINGERS ORSINGING TEACHERS

[37]

NONE

NONE

NONE

[40]

SUBJECTIVE

LISTENING TESTS RATINGS (1–5)

50 SUBJECTS WITH DIFFERENTLEVELS OF MUSICAL TRAINING[25]

SUBJECTIVE

LISTENING TEST (1–5 RATINGS) OF 15 MUSICAL PHRASES.
TWO TESTS: WITH AND WITHOUT TIME-LAG MODEL

14 SUBJECTS[42]

SUBJECTIVENOT DETAILED (BASED ON [25])NOT SPECIFIED[22]

SUBJECTIVERATE STYLE AND NATURALNESS LISTENING TESTSRATINGS (1–5) OF TEN RANDOM PHRASES PER SUBJECT

10 SUBJECTS[43]

SUBJECTIVERATE EXPRESSION, NATURALNESS, AND SINGER SKILLSLISTENING TESTS RATINGS (1–5)

17 SUBJECTS WITH DIFFERENTLEVELS OF MUSICAL TRAINING

RULE
BASED

STATISTICAL
MODELING

UNIT
SELECTIONIEEE SIGNAL PROCESSING MAGAZINE [69] november 2015

for performance-driven approaches, i.e., when a ground truth isavailable. In the other approaches, no reference is directly used
for comparison, so only subjective tests are carried out. However,in the absence of a reference of the same target song, the generated performances could be compared to the recording of anothersong, as is done in the case of speech synthesis.
In our opinion, the described evaluation strategies are devisedfor evaluating a specific system and, therefore, focus on a concrete
set of characteristics that are particularly relevant for that system.
For instance, the evaluations summarized in Table 10 do notinclude comparisons to other approaches. This is due to the substantial differences between systems, which make the evaluationand comparison between them a complex task. These differences
can be noted in the audio excerpts of the accompanying Web site
to this article, which were introduced in the section “Comparisonof Approaches.” At this stage, it is difficult to decide which method
more efficiently evokes a certain emotion or style, performs better
vibratos, changes the voice quality in a better way, or has a bettertiming control. There are limitations in achieving such a comprehensive evaluation and comparing the synthesized material.
TOWARD A COMMON EVALUATION FRAMEWORKThe evaluation methodology could be improved by building thesystems under similar conditions to reduce the differences among
performances and by sharing the evaluation criteria. Building acommon framework would help to easily evaluate and compare
the singing synthesis systems.
The main blocks of the reviewed works are summarized in­Figure 8. For a given target song, the expression parameters are
generated to control the synthesis system. To share as many commonalities as possible among systems, these could be built undersimilar conditions and tested by a shared evaluation criterion.
Then, the comparison would benefit from focusing on the technological differences and not on other aspects such as the target
song and singer databases.
Concerning the conditions, several aspects could be shared
among approaches. Currently, there are differences in the target
songs synthesized by each approach, the set of controlled expression features, and the singer recordings (e.g., singer gender, style,or emotion) used to derive rules, train models, build expression
databases, and build the singer voice models.

TargetSong

SimilarConditions

Expression
Control

Synthesis
SystemSingerRecordings

SingerDB[FIG8] The proposed common evaluation framework.

A publicly available data set of songs, with scores (e.g., inMusicXML format) and reference recordings, could be helpful if
used as target songs to evaluate how expression is controlled byeach approach. In addition, deriving the expression controls andbuilding the voice models from a common set of recordings would
have a great impact on developing this evaluation framework. If all
approaches shared such a database, it would be possible to compare how each one captures expression and generates the control
parameters since the starting point would be the same for all ofthem. Additionally, both sample- and HMM-based synthesis systems
would derive from the same voice. Thus, it would be possible totest a single expression-control method with several singing voicesynthesis technologies. The main problem we envisage is thatsome approaches are initially conceived for a particular synthesis
system. This might not be a major problem for the pitch contour
control, but it would be more difficult to apply the voice timbre
modeling of HMM-based systems to sample-based systems.
The subjective evaluation process is worthy of particular note.
Listening tests are time-consuming tasks, and several aspects need
to be considered in their design. The different backgrounds related
to singing voice synthesis, speech synthesis, technical skills, andthe wide range of musical skills of the selected participants can betaken into consideration by grouping the results according tosuch expertise, and clear instructions have to be provided on whatto rate, e.g., which specific acoustic features of the singing voice tofocus on, and how to rate using pairwise comparisons or MOS.
Moreover, uncontrolled biases in the rating of stimuli due to theorder in which these are listened to can be avoided by presenting
them using pseudorandom methods such as Latin squares, andthe session duration has to be short enough so as not to decrease
the participant’s level of attention. However, often, the reviewed
evaluations are designed differently and are not directly comparable. Next, we introduce a proposal to overcome this issue.
PERCEPTUALLY MOTIVATED OBJECTIVE MEASURESThe constraints in the section “Toward a Common Evaluation Framework” make it unaffordable to extensively evaluate different configurations of systems by listening to many synthesized performances.
This can be solved if objective measures that correlate with perception
are established. Such perceptually motivated objective measures canbe computed by learning the relationship between MOS and extracted
features at a local or global scope. The measure should be ideally independent from thestyle and the singer, and it should provide
ratings for particular features such as timing,
vibrato, tuning, voice quality, or the overallSound
performance expression. These measures,
besides helping to improve the systems’ performance, would represent a standard forevaluation and allow for scalability.
Evaluation
The development of perceptually motiCriteriavated objective measures could benefit fromapproaches in the speech and audio processing fields. Psychoacoustic and cognitivemodels have been used to build objective

IEEE SIGNAL PROCESSING MAGAZINE [70] november 2015

metrics for assessing audio quality and speech intelligibility [45],and its effectiveness has been measured by its correlation to MOS
ratings. Interestingly, method-specific measures have been computed in unit selection cost functions for speech synthesis [46].
Other approaches for speech quality prediction are based on a loglikelihood measure as a distance between a synthesized utterance
and an HMM model built from features based on MFCCs and F0 of
natural recordings [47]. This gender-dependent measure is correlated to subjective ratings such as naturalness. For male data, it
can be improved by linearly combining it with parameters typicallyused in narrow-band telephony applications, such as noise or
robotization effects. For female data, it can be improved by linearly
combining it with parameters related to signal-like duration, formants, or pitch. The research on automatic evaluation of expressive performances is considered an area to exploit, although it isstill not mature enough [48]; e.g., it could be applied to develop
better models and training tools for both systems and students.
Similar to the speech and instrumental music performance communities, the progress in the singing voice community could be
incentivized through evaluation campaigns. These types of evaluations help to identify the aspects that need to be improved and canbe used to validate perceptually motivated objective measures.
Examples of past evaluation campaigns are the Synthesis Singing
Challenge [52] and the Performance Rendering Contest (Rencon)
http://renconmusic.org/ [48]. In the first competition, one of the target songs was compulsory and the same for each team. The performances were rated by 60 participants with a five-point scale
involving the quality of the voice source, quality of the articulation,
expressive quality, and the overall judgment. The organizers concluded that “the audience had a difficult task, since not all systems
produced both a baritone and a soprano version, while the quality ofthe voices used could be quite different (weaker results for thefemale voice)” [52]. Rencon’s methodology is also interesting.
Expressive performances are generated from the same Disklavier
grand piano so that the differences among approaches are only due
to the performance and are subjectively evaluated by an audienceand experts. In 2004, voice synthesizers were also invited. Favorable
reviews were received but not included in the ranking.
CHALLENGES
While expression control has advanced in recent years, there aremany open challenges. In this section, we discuss some specific challenges and consider the advantages of hybrid approaches. Next, we
discuss important challenges in approaching a more human-likenaturalness in the synthesis. Then, requirements for intuitive andflexible singing voice synthesizers’ interfaces are analyzed, along with
the importance of associating a synthetic voice with a character.
TOWARD HYBRID APPROACHES
Several challenges have been identified in the described
approaches. Only one of the performance-driven approaches deals
with timbre, and it depends on the available voice quality databases. This approach would benefit from techniques for the analysis of the target voice quality, its evolution over time, andtechniques for voice quality transformations so to be able

to synthesize any type of voice quality. The same analysis andtransformation techniques would be useful for the unit selection
approaches. Rule-based approaches would benefit from machinelearning techniques that learn rules from singing voice recordings
to characterize a particular singer and explore how these are combined. Statistical modeling approaches currently do not utilize
comprehensive databases that cover a broad range of styles, emotions, and voice qualities. If we could take databases that efficiently
cover different characteristics of a singer in such a way, it would
lead to interesting results such as model interpolation.
We consider the combination of existing approaches to havegreat potential. Rule-based techniques could be used as a preprocessing step to modify the nominal target score so that it contains
variations such as ornamentations and timing changes related to thetarget style or emotion. The resulting score could be used as thetarget score for statistical and unit selection approaches where
the expression parameters would be generated.
MORE HUMAN-LIKE SINGING SYNTHESISOne of the ultimate goals of singing synthesis technologies is tosynthesize human-like singing voices that cannot be distinguishedfrom human singing voices. Although the naturalness of synthesized singing voices has been increasing, perfect human-like naturalness has not yet been achieved. Singing synthesis technologies
will require more dynamic, complex, and expressive changes inthe voice pitch, loudness, and timbre. For example, voice quality
modifications could be related to emotions, style, or lyrics.
Moreover, automatic context-dependent control of thosechanges will also be another important challenge. The current technologies synthesize words in the lyrics without knowing their meanings. In the future, the meanings of the lyrics could be reflected insinging expressions as human singers do. Human-like singing synthesis and realistic expression control may be a very challenging
goal, given how complex this has been proven for speech.
When human-like naturalness increases, the “Uncanny Valley”
hypothesis [49] states that some people may feel a sense of creepiness. Although the Uncanny Valley is usually associated with robots
and computer graphics, it is applicable even to singing voices. In
fact, when a demonstration video by VocaListener [31] first
appeared in 2008, the Uncanny Valley was often mentioned by listeners to evaluate its synthesized voices. An exhibition of a singer
robot driven by VocaWatcher [50] in 2010 also elicited more reactions related to the Uncanny Valley. However, we believe that such a
discussion of this valley should not discourage further research.
What this discussion means is that the current technologies are ina transitional stage towards future technologies that will go beyond
the Uncanny Valley [50] and that it is important for researchers tokeep working toward such future technologies.
Note, however, that human-like naturalness is not always
demanded. As sound synthesis technologies are often used to provide
artificial sounds that cannot be performed by natural instruments,
synthesized singing voices that cannot be performed by human singers are also important and should be pursued in parallel, sometimes
even for aesthetic reasons. Some possible examples are extremely fastsinging or singing with pitch or timbre quantization.

IEEE SIGNAL PROCESSING MAGAZINE [71] november 2015

MORE FLEXIBLE INTERFACES FOR SINGING SYNTHESIS
User interfaces for singing synthesis systems will play a more
important role in the future. As various score- and performancedriven interfaces are indispensable for musicians in using generalsound synthesizers, singing synthesis interfaces have also hadvarious options such as score-driven interfaces based on the
piano-roll or score editor and performance-driven interfaces inwhich a user can just sing along with a song and a synthesis system then imitates him or her (as mentioned in the section “Performance-Driven Approaches”). More intuitive interfaces that do
not require time-consuming manual adjustment will be an
important goal for ultimate singing interfaces. So far, direct
manipulator-style interfaces, such as the aforementioned scoreor performance-driven interfaces, are used for singing synthesis
systems, but indirect producer-style interfaces, such as those thatenable users to verbally communicate with and ask a virtual
singer to sing in different ways, will also be attractive to help
users focus on how to express the user’s message or intention
through a song, although such advanced interfaces have yet to be
developed. More flexible expression control of singing synthesis inreal time is also another challenge.
MULTIMODAL ASPECTS OF SINGING SYNTHESIS
Attractive singing synthesis itself must be a necessary condition for
its popularity, but it is not a sufficient condition. The most famous
virtual singer, Hatsune Miku, has shown that having a charactercan be essential to make singing synthesis technologies popular.
Hatsune Miku is the name of the most popular singing synthesis
software package in the world. She is based on Vocaloid and has asynthesized voice in Japanese and English with an illustration of a
cartoon girl. After Hatsune Miku originally appeared in 2007, many
people started listening to a synthesized singing voice as the main
vocal of music, something rare and almost impossible before Hatsune Miku. Many amateur musicians have been inspired and motivated by her character image together with her voice and havewritten songs for her. Many people realized that having a character
facilitated writing lyrics for a synthesized singing voice and thatmultimodality is an important aspect in singing synthesis.
An important multimodal challenge, therefore, is to generate
several attributes of a singer, such as a voice, face, and body. The faceand body can be realized by computer graphics or robots. An example of simultaneous control of voice and face was shown in the combination of VocaListener [31] and VocaWatcher [50], which imitatessinging expressions of the voice and face of a human singer.
In the future, speech synthesis could also be fully integrated
with singing synthesis. It will be challenging to develop new voicesynthesis systems that could seamlessly generate any voice produced by a human or virtual singer/speaker.
ACKNOWLEDGMENTS
We would like to thank Alastair Porter for proofreading and Merlijn
Blaauw for reviewing the article. Some works presented in the article were supported in part by the Core Research for Evolutional
Science and Technology funding program provided by the Japan
Science and Technology Agency.

AUTHORS
Martí Umbert (marti.umbert@upf.edu) earned his M.S. degree
in telecommunications at the Universitat Politècnica de Catalunya, Barcelona, Spain, in 2004. In 2010, he earned his M.Sc.
degree in sound and music computing from the Universitat
Pompeu Fabra (UPF) in Barcelona, Spain. He is currently aPh.D. candidate in the Music Technology Group at UPF. He has
worked on speech technologies both at the Universitat Politècnica de Catalunya and in the private sector. His research is carried out within the audio signal processing team, and he isinterested in singing voice synthesis and expression modelingbased on unit selection and corpus generation.
Jordi Bonada (jordi.bonada@upf.edu) earned his Ph.D.
degree in computer science and digital communications fromthe Universitat Pompeu Fabra (UPF), Barcelona, Spain, in 2009.
He is currently a senior researcher with the Music Technology
Group at UPF. He is interested in singing voice modeling andsynthesis. His research trajectory has an important focus ontechnology transfer acknowledged by more than 50 patents. He
has received several awards, including the Rosina Ribalta Prizeby the Epson Foundation in 2007 and the Japan Patent
Attorneys Association Prize by the Japan Institute of Invention
and Innovation in 2013.
Masataka Goto (m.goto@aist.go.jp) received the doctor of engineering degree from Waseda University in Tokyo, Japan, 1998. He iscurrently a prime senior researcher and the leader of the MediaInteraction Group at the National Institute of Advanced Industrial
Science and Technology, Japan. Over the past 23 years, he has published more than 220 papers in refereed journals and international
conferences and has received 40 awards, including several best
paper awards, best presentation awards, the Tenth Japan Academy
Medal, and the Tenth Japan Society for the Promotion of Science
Prize. In 2011, as the research director, he began the OngaCREST
Project, a five-year Japan Science and Technology Agency-funded
research project on music technologies.
Tomoyasu Nakano (t.nakano@aist.go.jp) earned his Ph.D.
degree in informatics from the University of Tsukuba, Japan, in
2008. He is currently a senior researcher at the National Instituteof Advanced Industrial Science and Technology, Japan. His
research interests include singing information processing,human–computer interaction, and music information retrieval.
He has received several awards including the Information
Processing Society of Japan (IPSJ) Yamashita SIG ResearchAward and the Best Paper Award from the Sound and Music
Computing Conference 2013. He is a member of the IPSJ and the
Acoustical Society of Japan.
Johan Sundberg (jsu@csc.kth.se) earned his Ph.D. degree inmusicology at Uppsala University Sweden, in 1966 and is doctor
honoris causae at the University of York, United Kingdom (1996)
and at the University of Athens, Greece (2014). He had a personal
chair in music acoustics at KTH and was the founder and head ofits music acoustics research group, which he was part of until
2001. His research concerns particularly the singing voice andmusic performance. He is the author of The Science of the Singing
Voice (1987) and The Science of Musical Sounds (1991). He is aIEEE SIGNAL PROCESSING MAGAZINE [72] november 2015

member of the Royal Swedish Academy of Music, a member of theSwedish Acoustical Society (president 1976–1981), and a fellow of
the Acoustical Society of America. He was awarded the Silver Medalin Musical Acoustics in 2003.
REFERENCES[1] X. Rodet, “Synthesis and processing of the singing voice,” in Proc. 1st IEEE Benelux
Workshop on Model-Based Processing and Coding of Audio (MPCA), 2002, pp. 99–108.
[2] P. R. Cook, “Singing voice synthesis: History, current work, and future directions,” Comput. Music J., vol. 20, no. 3, pp. 38–46, 1996.
[3] J. Sundberg, “The KTH synthesis of singing,” Adv. Cognit. Psychol., vol. 2, nos.
2–3, pp. 131–143, 2006.
[4] M. Goto, “Grand challenges in music information research,” in M. Müller,
M. Goto, and M. Schedl (Eds.), Multimodal Music Processing. Dagstuhl Publishing, Saarbrücken/Wadern, Germany, 2012, vol. 3, pp. 217–225.
[5] A. Kirke and E. R. Miranda, Guide to Computing for Expressive Music Performance. New York: Springer, 2013, Ch. 1.
[6] J. Bonada and X. Serra, “Synthesis of the singing voice by performance samplingand spectral models,” IEEE Signal Processing Mag., vol. 24, no. 2, pp. 67–79, 2007.
[7] H. Kawahara, R. Nisimura, T. Irino, M. Morise, T. Takahashi, and B. Banno,“Temporally variable multi-aspect auditory morphing enabling extrapolation
without objective and perceptual breakdown,” in Proc. IEEE Int. Conf. Acoustics,
Speech and Signal Processing (ICASSP), Taipei, Taiwan, 2009, pp. 3905–3908.
[8] H. Doi, T. Toda, T. Nakano, M. Goto, and S. Nakamura, “Singing voice conversion method based on many-to-many eigenvoice conversion and training data
generation using a singing-to-singing synthesis system,” in APSIPA/ASC, Dec.
2012, pp. 1–6.
[9] S. Canazza, G. De Poli, C. Drioli, A. Roda, and A. Vidolin, “Modeling and control of expressiveness in music performance,” Proc. IEEE Special Issue Eng. Music,
vol. 92, no. 4, pp. 686–701, 2004.
[10] G. Widmer, “Using AI and machine learning to study expressive music performance: Project survey and first report,” AI Commun., vol. 14, no. 3, pp. 149–162, 2001.
[11] P. N. Juslin, “Five facets of musical expression: A psychologist’s perspective onmusic performance,” Psychol. Music, vol. 31, no. 3, pp. 273–302, 2003.
[12] P. N. Juslin and P. Laukka, “Communication of emotions in vocal expression andmusic performance: Different channels, same code,” Psychol. Bull., vol. 129, no. 5, pp.
770–814, 2003.
[13] S. Ternström. (2002, June). Session on naturalness in synthesized speech
and music. in Proc. 143rd Acoustical Society of America (ASA) Meeting. [Online].
Available: http://www.pvv.ntnu.no/~farner/sonata/ternstrom02.html[14] M. Thalén and J. Sundberg, “Describing different styles of singing: A comparison of a female singer’s voice source in ‘classical,’ ‘pop,’ ‘jazz,’ and ‘blues’,” Logoped.
Phoniatrics Vocol., vol. 26, no. 2, pp. 82–93, 2001.

[27] R. L. de Mantaras and J. Ll. Arcos, “AI and music: From composition to expressive performance,” AI Mag., vol. 23, no. 3, pp. 43–57, 2002.
[28] M. Kob, “Singing voice modelling as we know it today,” Acta Acustica, vol. 90,
no. 4, pp. 649–661, 2004.
[29] Y. Meron. (1999). High quality singing synthesis using the selection-based
synthesis scheme. Ph.D. dissertation, Univ. of Tokyo. [Online]. Available: http://
www.gavo.t.u-tokyo.ac.jp/~meron/sing.html[30] J. Janer, J. Bonada, and M. Blaauw, “Performance-driven control for samplebased singing voice synthesis,” in Proc. 9th Int. Conf. Digital Audio Effects (DAFx),
2006, vol. 6, pp. 42–44.
[31] T. Nakano and M. Goto, “VocaListener: A singing-to-singing synthesis system
based on iterative parameter estimation,” in Proc. 6th Sound and Music Computing Conf. (SMC), July 2009, pp. 343–348.
[32] T. Nakano and M. Goto, “VocaListener2: A singing synthesis system able
to mimic a user’s singing in terms of voice timbre changes as well as pitch anddynamics,” in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing(ICASSP), Prague, Czech Republic, May 2011, pp. 453–456.
[33] T. Saitou, M. Goto, M. Unoki, and M. Akagi, “Speech-to-singing synthesis: Converting speaking voices to singing voices by controlling acoustic features unique tosinging voices,” in Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz, NY, Oct. 2007, pp. 215–218.
[34] J. Sundberg and J. Bauer-Huppmann, “When does a sung tone start,” J. Voice,vol. 21, no. 3, pp. 285–293, 2007.
[35] J. Sundberg, “Synthesis of singing, in Musica e Tecnologia: Industria e Cultura per lo
Sviluppo del Mezzagiorno,” in Proc. Symp. Venice. Venedig: Unicopli, 1981, pp. 145–162.
[36] M. C. Marinescu and R. Ramirez, “A machine learning approach to expression modeling for the singing voice,” in Proc. Int. Conf. Computer and­Computer Intelligence (ICCCI). New York: ASME Press, 2011, vol. 31, no. 12,
pp. 311–316.
[37] M. Alonso. (2005). Expressive performance model for a singing voice synthesizer. Master’s thesis, Universitat Pompeu Fabra. [Online]. Available: http://mtg.
upf.edu/node/2223[38] R. Bresin and A. Friberg, “Emotional coloring of computer-controlled music
performances,” Comput. Music J., vol. 24, no. 4, pp. 44–63, Winter 2000.
[39] H. Kenmochi and H. Ohshita, “VOCALOID—Commercial singing synthesizer
based on sample concatenation,” in Proc. Interspeech, Antwerp, Belgium, Aug.
2007, pp. 4009–4010.
[40] J. Bonada. (2008). Voice processing and synthesis by performance samplingand spectral models. Ph.D. dissertation, Universitat Pompeu Fabra, Barcelona.
[Online]. Available: http://www.mtg.upf.edu/node/1231[41] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, “Simultaneous modeling of spectrum, pitch and duration in HMM-based speech synthesis,” in Proc. 6th European Conf. Speech Communication and Technology(Eurospeech), Budapest, Hungary, 1999, pp. 2347–2350.

[15] A. Friberg, R. Bresin, and J. Sundberg, “Overview of the KTH rule system formusical performance,” Adv. Cognit. Psychol., vol. 2, nos. 2–3, pp. 145–161, 2006.

[42] K. Oura, A. Mase, T. Yamada, S. Muto, Y. Nankaku, and K. Tokuda, “Recent
development of the HMM-based singing voice synthesis system—Sinsy,” in Proc.
Int. Speech Communication Association (ISCA), 7th Speech Synthesis Workshop(SSW7), Tokyo, Japan, Sept. 2010, pp. 211–216.

[16] N. Obin. (2011). MeLos: Analysis and modelling of speech prosody and speaking style. Ph.D. dissertation, Université Pierre et Marie Curie-Paris VI, France.
[Online]. Available: http://articles.ircam.fr/textes/Obin11e/index.pdf[43] M. Umbert, J. Bonada, and M. Blaauw, “Generating singing voice expression
contours based on unit selection,” in Proc. Stockholm Music Acoustics Conference(SMAC), Stockholm, Sweden, Aug. 2013, pp. 315–320.

[17] M. Schröder, “Expressive speech synthesis: Past, present, and possible futures,”in Affective Information Processing. London: Springer, May 2009, pp. 111–126.

[44] M. Umbert, J. Bonada, and M. Blaauw, “Systematic database creation for expressive singing voice synthesis control,” in Proc. Int. Speech Communication
Association (ISCA), 8th Speech Synthesis Workshop (SSW8), Barcelona, Spain,
Sept. 2013, pp. 213–216.

[18] G. Widmer and W. Goebl, “Computational models of expressive music performance: The state of the art,” J. New Music Res., vol. 33, no. 3, pp. 203–216, 2004.
[19] M. Lesaffre. (2005). Music information retrieval conceptual framework,
annotation and user behaviour. Ph.D. dissertation, Ghent University. [Online].
Available: https://biblio.ugent.be/publication/3258568[20] J. Salamon, E. Gómez, D. P. W. Ellis, and G. Richard, “Melody extraction frompolyphonic music signals: Approaches, applications and challenges,” IEEE Signal
Processing Mag., vol. 31, no. 2, pp. 118–134, Mar. 2014.
[21] C. J. Plack and A. J. Oxenham, “Overview: The present and future of pitch,” in
Pitch—Neural Coding and Perception, Springer Handbook of Auditory Research.
New York: Springer, 2005, vol. 24, Chap. 1, pp. 1–6.
[22] K. Saino, M. Tachibana, and H. Kenmochi, “A singing style modeling system
for singing voice synthesizers,” in Proc. Interspeech, Makuhari, Japan, Sept. 2010,
pp. 2894–2897.
[23] J. Sundberg, The Science of the Singing Voice. Northern Illinois Univ. Press,
DeKalb, Illinois, 1987.

[45] D. Campbell, E. Jones, and M. Glavin, “Audio quality assessment techniques—A review, and recent developments,” Signal Process., vol. 89, no. 8, pp. 1489–1500, Aug. 2009.
[46] M. Chu and H. Peng, “An objective measure for estimating MOS of synthesized
speech,” in Proc. 7th European Conf. Speech Communication and Technology(Eurospeech), Aalborg, Denmark, Sept. 2001 pp. 2087–2090.
[47] S. Möller, F. Hinterleitner, T. H. Falk, and T. Polzehl, “Comparison of approaches for instrumentally predicting the quality of text-to-speech systems,” inProc. Interspeech, Makuhari, Japan, Sept. 2010, pp. 1325–1328.
[48] H. Katayose, M. Hashida, G. De Poli, and K. Hirata, “On evaluating systems
for generating expressive music performance: The Rencon experience,” J. New Music Res., vol. 41, no. 4, pp. 299–310, 2012.
[49] M. Mori, K. F. MacDorman, and N. Kageki, “The uncanny valley [from thefield],” IEEE Robot. Automat. Mag., vol. 19, no. 2, pp. 98–100, June 2012.

[24] E. D. Scheirer, “Tempo and beat analysis of acoustic musical signals,” J. Acoust.
Soc. Amer., vol. 103, no. 1, pp. 588–601, 1998.

[50] M. Goto, T. Nakano, S. Kajita, Y. Matsusaka, S. Nakaoka, and K. Yokoi, “VocaListener and VocaWatcher: imitating a human singer by using signal processing,” in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP),
Kyoto, Japan, Mar. 2012, pp. 5393–5396.

[25] K. Saino, H. Zen, Y. Nankaku, A. Lee, and K. Tokuda, “An HMM-based singing voicesynthesis system,” in Proc. Interspeech, Pittsburgh, PA, USA, Sept. 2006, pp. 1141–1144.

[51] Music Technology Group of the Universitat Pompeu Fabra website. [Online].
­Available: www.mtg.upf.edu/publications/ExpressionControlinSingingVoiceSynthesis[26] A. Loscos and J. Bonada, “Emulating rough and growl voice in spectral domain,” in Proc. 7th Int. Conf. Digital Audio Effects (DAFx), Naples, Italy, Oct.
2004, pp. 49–52.

[52] [Online]. Available: http://www.interspeech2007.org/Technical/synthesis_
of_singing_challenge.php

IEEE SIGNAL PROCESSING MAGAZINE [73] november 2015

[SP]