This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TAFFC.2018.2813381, IEEE
Transactions on Affective Computing
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING

1

Defining laughter context for laughter synthesis
with spontaneous speech corpus
Tomohiro Nagata and Hiroki Mori
Abstract—In this paper, conversational laughter was synthesized by a statistical model-based speech synthesis framework using
spontaneous speech corpora. The phonetic transcriptions of natural laughter in these corpora were annotated, and the context
required to synthesize the laughter that accompanies speech sounds was defined from the perspective of the (1) phonetic properties of
the current segment, (2) phonetic properties of previous and succeeding segments, and (3) positional factors of the current segment orlaughter bout. Laughter was synthesized using the defined context and the framework of HMM-based speech synthesis.
To confirm the influence of the contextual factors on the naturalness of speech, a subjective evaluation was performed. As the result of
the evaluation, the naturalness of the entire utterance was improved by using the contextual factors defined in this study. This result
confirmed the importance of defining the appropriate context to synthesize natural conversational laughter.
Index Terms—Laughter, Spontaneous speech corpus, HMM-based speech synthesis

F

1

I NTRODUCTIONI

N recent years, human-human and human-machine interactions have attracted increasing interest. To ensurethat human-machine interactions resemble those between
humans, machine speech must express both linguistic information and paralinguistic information such as the speaker’s
attitude, intentions, and emotions. To express such information, a speech synthesizer must produce non-lexical sounds
in addition to speech sounds.
Laughing is a typical method of conveying non-verbal
information and has attracted research interest. Laughter
research has a long history and has been studied fromvarious perspectives, including generational, sociological,
and physiological ones [1]. In recent years, incorporating
laughter-based behavior into human-machine interaction
models has become a hot topic. For example, the ILHAIRE
project [2] researched the fundamentals and applications oflaughter, which include laughter detection, recognition andsynthesis [3]–[10].
Laughter is highly diverse. It has many forms, andappears in various situations [11], [12]. Also, it has beensuggested that there is a gradual change from smiling to“pure” laughter, i.e., speech-smile → speech-laugh → laughter, each of which has different acoustic characteristics andcommunicative functions [13]–[16]. All of these phenomena
must be taken into consideration in order to make a machine laugh, which cannot be realized merely by replaying
recorded laughter sound. To cope with such diversity, aflexible framework of laughter synthesis is essential.
However, laughter synthesis is still a developing field.
In [6] and [7], methods for synthesizing laughter vowels
using linear prediction were proposed, where the laughter
energy envelope was modeled via a mass-spring system. A
method of laughter vowel synthesis using formant synthesis•

T. Nagata and H. Mori are with Graduate School of Engineering, Utsunomiya University, Japan.
E-mail: ken1@speech-lab.org

to control the expressivity of laughter was developed in[8]. In [9] and [10], a diphone concatenation-based laughter
synthesis method was proposed. Additionally, a method oflaughter synthesis was developed using a 3D model of the
vocal tract in [10].
In more recent studies, a method of laughter synthesis
via a hidden Markov model (HMM)-based approach was
proposed in [17] and [18]. In the method, the excitation andspectral parameters of laughter are modeled via HMMs; thedifferences in terms of the acoustic features are represented
by the context label. In [19], the acoustic features of laughter
were controlled via an arousal-driven method. In relation
to these “pure” laughter synthesis, it has been proposedthat speech-laughs be synthesized by replacing vowels of
the speech-smile model with vowels of the laughter model[20], [21].
However, most corpus-based studies use induced laughter rather than that in conversational scenes, even though
the synthesized laughter can be used in human-machinecommunication. Laughter in conversation occasionally accompanies speech sounds. It has been reported that even
if the naturalness of the laughter itself is high, the overall
naturalness of an utterance is reduced when it is connectedwith speech sounds [9]. In [10], laughter synthesized by diphone concatenative synthesis was perceived as unnatural.
To properly synthesize laughter in conversational scenes,
the laughter model needs to consider the context in whichlaughter is placed.
In this study, we focus on the statistical parametric speech synthesis framework. In this framework, theprosodic and segmental features of phonemes that vary due
to various factors are expressed as the context [23]. Unlike
simple concatenative methods, this method can robustly
model the acoustic properties of segments, which can be
explained by a combination of contextual factors. Thus,
laughter that is suitable for a situation can be synthesizedusing the statistical parametric framework if we can define
a proper laughter context.

1949-3045 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TAFFC.2018.2813381, IEEE
Transactions on Affective Computing
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING

In the HMM-based laughter synthesis mentioned above[17], [18], the datasets used were not laughs from communication scenes but ones collected by emotion induction using
joke videos. The authors believe that no previous work has
attempted to synthesize laughter in conversation using a
spontaneous dialog speech corpus as a dataset.
The goal of this study is to synthesize natural laughterin conversational scenes. To achieve this, we need to collectnatural laughter in spontaneous conversations rather thanartificially induced laughter.
To properly model laughter in conversation, we need to
identify effective contextual factors. Urbain et al. investigated the effect of contextual information on the naturalnessof synthesized laughter [18]. They tried to add “an extended
context including the information available thanks to thesyllabic annotation (e.g., position of the phone in the syllable, position of the syllable in the word, etc.)” [18] to simple
phone information, but failed to prove the effectiveness ofthe extended context information. In the following papers
to [18], contextual factors are basically same [19], [22]. So
far, no previous work has further explored the contextual
factors for HMM-based laughter synthesis. Hence, the main
contribution of this study is to clarify the contextual factors
required for laughter synthesis with spontaneous dialog
speech. We define a relatively simple laughter context as
the first step.
In this paper, the natural laughter that accompanies
speech is synthesized with spontaneous speech corpora
using the HMM-based speech synthesis framework and this
newly defined context. The corpora used in this study is
described in Section 2. Section 3 shows the laughter annotation method, and some statistics of laughter annotated
for the corpora. In section 4, the context for laughter in aconversation is defined; the laughs are synthesized based on
the context using the HMM-based speech synthesis framework. In this study, the position of laughter in an utterance
that includes speech sounds (a relative position with speechsounds) was defined, as was the basic context (e.g. phonetictranscription, position of the syllable, etc. [17], [18]). In
section 5, the effectiveness of the newly defined context isinvestigated via a subjective evaluation of naturalness.

2

D IALOG SPEECH CORPUS

Laughter in dialog may play a social role; thus, it is necessary to consider what factors are different from laughter
alone. For example, laughter overlaps more frequently inconversation than with speech sounds [24]. In fact, overlapping laughter has a different acoustic form than nonoverlapping laughter does [25]. Thus, because there may be
a difference in acoustic features between laughter sounds
in conversation and induced laughter, it may not be appropriate to exclusively use induced laughter to synthesize
laughter in conversation. Therefore, we focus on laughter
included in spontaneous dialog speech corpora for laughter
synthesis. In this study, the Utsunomiya University Spoken
Dialogue Database for Paralinguistic Information Studies(UU Database) [26] and the Online Gaming Voice chat
Corpus (OGVC) [27] were used.

2

TABLE 1
The number of laughs in the UU Database.

Speaker Num Speaker Num Speaker Num
FJK
8
FKC
22
FMS
16
FMT
34
FNN
14
FSA
26
FSH
17
FTH
5
FTS
40
FTY
19
FUE
14
FYH
23
MKK
22
MKO
20
TABLE 2The number of laughs in the OGVC.

Speaker Num Speaker Num Speaker Num
01 MMK 26 03 FMA 74 05 MYH 52
01 MAD 118 03 FTY
41 05 MKK 87
02 MFM 142 04 MNN 154 06 FTY 251
02 MEM 145 04 MSY 246 06 FWA 175
2.1

UU Database

The UU Database is a speech corpus for studying linguistic
and phonetic phenomena in expressive spoken dialog. The
database consists of natural dialog spoken by seven pairs ofcollege students (12 female speakers and 2 male speakers).
The task of the dialogs is “four-frame cartoon sorting.”
Thanks to the amusing nature of the task, the databaseis characterized by a wide variety of recorded expressive
dialog speech. The total number of utterances is 4840.
The speech recorded in the UU Database is annotatedwith the labels of non-verbal sounds such as laughter,
inhalation, and coughing. The total number of laughs is 280.
The number of laughs by each speaker is shown in Tab. 1.
2.2

OGVC

The OGVC is an emotional speech corpus that can compare
spontaneous speech and acted speech. In this paper, spontaneous speech is exclusively used. The voice chat during theonline game was recorded as spontaneous speech. In total,
9114 spontaneous utterances were recorded by 13 speakers(4 female speakers and 9 male speakers).
In the OGVC, the location of the laughter is annotatedin the transcription. The total number of laughs is 1593. The
number of laughs by each speaker is shown in Tab. 2.

3

S EGMENTATION OF LAUGHTER

Laughter is difficult to identify. In particular, the distinction between pure “laughter” and “speech-laugh” [14] isimportant for corpus annotation. We define “laughters” inthis paper as laughing sounds that do not overlap speech,
as opposed to “speech-laughs”, which overlap speech. We
shall exclude speech-laughs from the scope of the current
study, though the distinction is not straightforward both intheory and practice. We shall avoid the difficulty by relying
on the laughter labels provided by the target corpora, theUU Database and the OGVC. In both corpora, the annotatedlaughters (shown in Tables 1 and 2) do not include speechlaughs, according to the user manuals.

1949-3045 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TAFFC.2018.2813381, IEEE
Transactions on Affective Computing
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING

3exhalation
respiration inhalation
consonant
vowel
call
bout
episodeFig. 1. The hierarchical structure of laughter (adapted from [28]).

The structure of laughter is hierarchical, and its hierarchy
is roughly divided into the segmental level, the syllabic level
and the phrasal level [28]. Fig. 1 shows the structure oflaughter. At the segmental level, laughter is split into consonants and vowels. Here, “vowel” and “consonant” denotesound-pause alternations; therefore, they have a different
meanings from a phonetic perspective [28]. At the syllabic
level, laughter is segmented into syllable (calls), which consist of the initial and final consonants and a vowel. At thephrasal level, laughter is segmented into phrase (bouts) consisting of some calls. Additionally, multiple phrases establish
the sentence level (i.e., the laughter episode).
In this study, each laughter episode is segmented intobouts and calls.

˳˷

Unvoiced
Nasalized

˸

Prolonged

Fig. 2. Auxiliary symbols.

3.2

Annotation of laughter transcription

Laughter episodes can be composed of several bouts separated by inhalation. In this study, a laughter episode isrepresented by a “{laugh}.” First, the laughing part ofan utterance is segmented by the laughter episode. Next,
the laughter episode is segmented by bouts and inhalation,
which are denoted as “b” and “h”, respectively.

Laughter was annotated using the description system defined in Sect. 3.1. The annotator was the first author andPraat [30] was used for the annotation. For this annotation,
the target speakers were limited to females because the
HMM synthesis requires that the training data have as
similar voice quality as possible in order to train stablemodels.
The annotation of laughter of speaker FTS in the UU
Database and speakers 03 FTY, 06 FTY and 06 FWA in
the OGVC was completed. Since the purpose of this study
is to synthesize laughter accompanied by speech sounds,
isolated laughter (i.e. there are no speech sounds just before
or just after the laugh) was excluded from the scope of theannotation.

3.1.2

3.3

3.1
3.1.1

Description of laughter in this study
Phrasal and sentence level

Syllabic level

In the AVLaughterCycle Database [29], which was built for
automatic laughter processing (i.e., HMM-based laughter
synthesis [17], [18]), a detailed description of laughter is provided, e.g., a transcription using the International Phonetic
Alphabet (IPA). Although such descriptions are considered useful for high-quality laughter synthesis, professional
knowledge and long observation times are required. This isa major barrier when transcribing large-scale speech corpus
or voice resources. Therefore, as a simplified method, calls
are described using kana (Japanese syllabic characters) inthis study. The description is transcribed once via kana, andthen it is converted to phoneme sequences. For example,
laughter “ハハハ” is described as [hahaha].
It is impossible to express differences between sounds
due to phenomena such as unvoicedness, nasalization andprolongation by using only the phoneme symbols. To enable
such descriptions, a set of auxiliary symbols, as shown in
Fig. 2, was defined.

Annotation results

An actual example of annotated laughter is shown in Fig. 3.
Laughter episodes and the transcription of speech sounds
are described as layers ①. This example is an utterance
containing a laughter episode. In layer ②, the structure of alaughter episode is described. The laughter episode consistsof a bout and an inhalation. In layer ③, the structure of a boutis described using calls. The phonetic notation is extended
by auxiliary symbols used to describe calls.
A histogram of the vowels in calls is shown in Fig. 4.
For vowels in call, /a/, /u/ and /e/ were dominant. The
vowels /i/ and /o/ were rare, and some speakers did not
produce them at all.
The most frequent vowel was /u/, and there are few
calls consisting only of vowels. This is shown in Fig. 5. Blackand gray bars indicate the number of calls containing only
vowels and the first calls containing only vowels, respectively. In contrast, the vowels /a/ and /e/ had relatively

1949-3045 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TAFFC.2018.2813381, IEEE
Transactions on Affective Computing
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING

4

①
②
③

Fig. 3. An example of laughter annotation.

unvoiced

a
FTS

30

47

19

voicedunvoiced

mixed

voiced7

Speakeri
03_FTY

42

70

u

34

single-call boute
06_FTY

25

9

11

58

12

21

31

0

9

23

91

50

100

13

68

20

40

79

o
0

06_FWA

multi-call bout

80

100

proportion to all bouts [%]

14

150

60

200

Fig. 6. The breakdown of bout’s voiced/unvoiced structure.

Frequency
06_FWA

Fig. 4. The histogram of vowels in calls.

voiced

126

89

65

06_FTY

124

Frequency

03_FTY25

vowel only (1st call)20

vowel onlyFTS
unvoiced 14 19
0

15

38

44

100

200

300

400

500

Fig. 7. The number of voiced/unvoiced call.

10

is illustrated in Fig. 7. These results confirm that the numberof unvoiced calls is small.

5

0

/a/

/i/

/u/

/e/

/o/

Vowel
Fig. 5. The number of calls in terms of vowels only.

more calls consisting only of vowels. Moreover, most of thecalls tended to appear at the initial position of the bout.
Fig. 6 illustrates the breakdown of the bout’s structure.
Bouts consisting of only one call, i.e., single-call bouts, accounted for 17% of the total, and unvoiced bouts accounted
for 35%. In contrast, there were few multi-call bouts where
all of the calls were unvoiced. This is the same trend as in[31]. Additionally, the distribution of the voicedness of calls

4

L AUGHTER SYNTHESIS MODEL

4.1

Definition of context

Various kinds of contexts (dialog acts [32], functions as
a response token [33], interpersonal stance [34], etc.) may
affect the acoustical properties of laughter sounds. In thisstudy, we defined similar contexts that are used in TTS
systems as a first step.
The context defined in this study is shown in Tab. 3.
To express the difference in sounds characterized by the
vocal tract configuration and phonation, the transcription ofcurrent calls was included. Here, the syllabic characters described in Sect. 3.1 were used. Voiced/unvoiced, oral/nasal
and prolonged sounds were clearly distinguished via auxiliary symbols.

1949-3045 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TAFFC.2018.2813381, IEEE
Transactions on Affective Computing
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING

5

}



A
B

Feature vector

Analysis

5 states left-to-right HSMM
0-39th mel-cepstral coefficient,logarithmic fundamental frequency,∆, ∆∆Speech signals were sampled at a rateof 16 kHz and windowed by a 25 ms
Hamming window with a 5 ms shift

5000

5000

Frequency [Hz]

TABLE 4
The training condition for the laughter model

Model

○

○

Frequency [Hz]cc : The transcription of current calls
cl : The transcription of preceding segment
cr : The transcription of succeeding segment
pl : The laughter position for an utterance
pc : The call position for a boutnc : The number of call for a bouthuhu

TABLE 3
Defined context for laughter

00.1983000Time [s](a) natural single-call bout

0.165

Time [s](b) synthesized single-call boutFig. 8. Example of single-call bout.

Two contexts were added as well. Context A includes
the transcription of the preceding and succeeding segments
to consider the influence of the difference between preceding and succeeding sounds. Here, “segments” are either
phonemes (for speech sound) or calls (for laughter).
Context B includes contextual factors that have moreglobal information. The laughter location in an utterance
was added to consider whether the laughter is placed at
the beginning, middle, or end of an utterance. The distribution of call duration is dependent on the call position [11].
Therefore, the call position was also added to the context.
The number of calls in a bout was included, similar to thenumber of morae in an accentual phrase, which is often usedin HMM-based speech synthesis.

4.3

Results of synthesized laughter

As an example of a single-call bout, the waveform andspectrum of laughter [hu] are shown in Fig. 8. Laughter [hu]˚˚is a fricative sound without
vocal fold vibration, as shownin Fig. 8 (a), and it is a typically a single-call bout in a daily
conversation scene such as giggles or self-derision. Fig. 8 (b)shows the waveform and spectrum of the synthesized singlecall bout, from which it can be confirmed that the synthesized[hu] is similar to the turbulence of natural laughter.
˚Similarly, the waveform and spectrum of laughter[huhuhu] are shown in Fig. 9 as an example of a multi˚ bout. From Fig. 9 (b), it can be confirmed that thecallnatural multi-call bout’s tendency in which the call amplitude
gradually decreases is reflected in the synthesized laughter.

4.2 HMM-based laughter synthesis
The laughter of speakers FTS, 06 FTY and 06 FWA wasused for model training. The laughs of speaker 03 FTY werediscarded because the amplitude levels of some recorded
sounds were too small. The total amount of training data
was 109 bouts. The HMM-based speech synthesis system(HTS, version 2.2) [35] was used for the model training
and laughter synthesis. The speech analysis conditions andtraining model are shown in Tab. 4. Here, the numberof states for HSMMs was determined by seeking fromthe range of 5–10 by assessing the quality of synthesizedlaughter; we used 5. The model was trained via the Speaker
Adaptive Training technique using a shared decision tree[36]. When constructing the tree, questions about contextual
factors in Tab. 3 were applied, and the nodes were split withthe maximum likelihood criterion.
The test laughter was synthesized with the model thatwas trained using all of the training data except the test
laughter itself (leave-one-out method). Therefore, the number of synthesized laughs is 109 bouts. Each laughter was
synthesized adaptively to fit the model of the original
speaker and of her laughter.

5

N ATURALNESS E VALUATIONIn this section, the overall naturalness of laughter accompanying the speech sounds is evaluated.
To evaluate the overall naturalness, synthesized laughter
was connected to the speech sounds. Furthermore, to confirm the effectiveness of considering the laughter context,
synthesized laughter considering the context was comparedto the laughter synthesized without context.
5.1

Stimulus

The conditions for synthesizing laughter were as follows:•••

baseline (BL)The laughter was synthesized using the context label
considering only current calls.
baseline plus context A (BL+A)
The laughter was synthesized using the context label
considering the preceding and following segments.
baseline plus context A and B (BL+AB)
The laughter was synthesized using the context thatconsiders more global contextual factors in addition
to context A.

1949-3045 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TAFFC.2018.2813381, IEEE
Transactions on Affective Computing
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING

hu

hu

○

6

hu

**
**
**
**

3.0

Frequency [Hz]

5000

2.0

Mean opinion score

4.0

5.0

**

0
0

0.39091.0Time [s](a) natural multi-call bout** : p < .01

RESYN
hu

hu

○

BL

BL+ A

BL+AB

hu

Fig. 10. The distribution of the naturalness score.

Frequency [Hz]

5000

0
0

0.48

Time [s](b) synthesized multi-call boutFig. 9. Example of multi-call bout.

•analysis-synthesis (RESYN)
The laughter was re-synthesized from natural laughter in the test utterance via the analysis-synthesis.

The speech sounds were connected to the laughter synthesized under these conditions. The speech sounds were resynthesized via analysis-synthesis, to avoid the influence ofthe quality of the speech part on the naturalness evaluationof the laughter part. The number of stimuli was 45 for eachcondition.
5.2

Experimental conditions

The stimuli for the experiments were created under four
conditions. Hence, the number of stimuli was 4 (conditions)× 45 (test utterances) = 180. The number of single-call boutand multi-call bout are 9 and 36, respectively. The number ofcall of multi-call bouts is distributed within a range of 2 to 5.

Seventeen undergraduate male students and two undergraduate female students from Utsunomiya University, allof whom were Japanese native speakers and had no special
knowledge about speech science, voluntarily participatedin the experiment. The evaluation was performed via a
web-based interface. The birthplace of the subjects was
distributed over a wide area of Japan. The subjects evaluated the naturalness of each stimulus on a 5-point scale:“1: unnatural,” “2: somewhat unnatural,” “3: neither (“どちらともいえない”),” “4: somewhat natural,” and “5: natural.”Naturalness was defined as “the degree of matching ofsegmental/prosodic features of speech and laughter parts.”
This also considers the degree of discordancy of paralinguistic information perceived from the speech or laughterparts.
Subjects were asked to listen to the stimuli using headphones in a quiet room. They were instructed to listen toeach stimulus only once.
5.3

Results

As a result of the naturalness evaluation, the distribution ofmean opinion score (MOS) is shown in Fig. 10. The boxplot
represents the mean opinion score averaged over all test
utterances. The averaged mean opinion score of BL, BL+A,
BL+AB and RESYN is 2.10, 2.32, 3.01 and 4.13, respectively.
A one-way ANOVA test revealed a significant main effect of
the synthesized conditions (F (3, 176) = 220.8, p < .01). The
result of multiple comparison by Tukey’s HSD test revealed
a significant difference between RESYN and other conditions (p < .01) and between BL+AB and other conditions(p < .01). These results confirmed that the overall naturalness was improved by defining the context that has moreglobal information, such as the laughter and call position.
Distributions of averaged MOS for each laughter structure are shown in Fig. 11. The averaged MOS of single-callbouts and multi-call bouts was 3.15 and 2.82, respectively. A

1949-3045 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TAFFC.2018.2813381, IEEE
Transactions on Affective Computing
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING

7

*

hu

hu

4.0
3.0

* : p < .05

singlemulti

Frequency [Hz]

2.0

5000

1.0

Mean opinion score

5.0

hu

Fig. 11. The distribution of the mean opinion score for each laughter
structure.

6

C ONCLUSIONSIn this paper, conversational laughter, which typically accompanies speech sounds, was synthesized via a statistical model-based speech synthesis framework using spon-

0

1.755

Times [s](a) Synthesized with BL

hu

hu

hu

Frequency [Hz]

5000

0
0

1.755

Times [s](b) Synthesized with BL+A

hu

hu

hu

5000
Frequency [Hz]t-test revealed a significant effect of the laughter structure(t(58.4) = −2.14, p < .05). This result implies the possibility that the structure of laughter affected the naturalness.
However, there is also a possibility that the different speechsounds that were connected to the synthesized laughter
affected the naturalness. In order to assess the structuraleffect on the naturalness in more detail, it will be necessary
to carefully control the sentences to be connected.
An example of naturalness improvement is shown in
Fig. 12. The figure shows the waveform and spectrum ofan utterance with laughter. The gray part of the waveform
represents the speech sounds, and the black part represents
the laughter. Fig. 12 (a) is the synthesized laughter without
considering the context. Synthesized laughter is perceived
as very unnatural because all calls have the same acoustic
features. Fig. 12 (b) is synthesized laughter that considers the preceding and succeeding segments. The acoustic
feature of the first call is different from the others, andthe naturalness is somewhat improved. Fig. 12 (c) is thesynthesized laughter that considers more global contextual
factors. Different acoustic features are generated dependingon the call position and sound like human laughter. From
this, we determined that the call position contributes greatly
to improving the naturalness.
There were also a few utterances for which the naturalness was not improved, even by using the extended context.
These are utterances where the naturalness of the laughter
itself is low. An example of poorly synthesized laughter
is shown in Fig. 13. In this example, [hu] has the sameacoustic features that are repeatedly being synthesized. Suchmonotonous repetition of the same sound causes the degradation of naturalness, known as the “machine gun effect.”A possible cause of such a phenomenon is the sparsenessof the training data. We observed that some nodes stopped
growing at an early stage of the decision tree-based context
clustering during the model construction. A potential solution is to add more fine-grained context that describes the
position of the calls or to increase the training data.

000Times [s]1.755(c) Synthesized with BL+AB
Fig. 12. Example that the naturalness of entire speech was increased.

1949-3045 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TAFFC.2018.2813381, IEEE
Transactions on Affective Computing
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING

hu
○

hu

hu

8

hu

hu

This work was supported by JSPS KAKENHI Grant
Number 26280100.

R EFERENCES[1]Frequency [Hz]

5000

0
0

Time [s]0.7332

Fig. 13. Example of low naturalness.

taneous speech corpora. To prepare the training data,
the transcription of calls was annotated for laughter withspeech sounds in two spontaneous speech corpora: the UU
Database and OGVC. In terms of contextual factors, thetranscription of the preceding and succeeding segments,
the call position, the number of calls and the bout position
were defined. Laughter was synthesized using the definedcontext and the framework of HMM-based speech synthesis. The synthesized laughter exhibited acoustic features
resembling natural laughter.
To confirm the influence of the contextual factors onnaturalness, a subjective evaluation was performed. The
naturalness was evaluated for utterances that include bothspeech sound and laughter. The naturalness of the entire
utterance was improved by using the defined contextual
factors in this study. From this, we concluded that it is necessary to use the appropriate contextual factors to synthesize
natural conversational laughter.
This study was conducted as a first step in laughter
synthesis that considers the context of natural conversation. Its effectiveness was confirmed under very limited
conditions. As future tasks, it is necessary to consider thegender difference of laughter, to improve the naturalness byincreasing the annotated training data, and to define further
contextual factors. Additionally, paralinguistic information
perceived from laughter was not evaluated in this study.
Because laughing is a medium of conveying the speaker’s
emotional and mental states, paralinguistic information perceived from laughter may play an important role in communication. Therefore, future tasks should include examining
the influence of laughter on perceived paralinguistic information and the context required to control paralinguistic
information.

ACKNOWLEDGMENTS
The authors thank Dr. Yoshiko Arimoto for her enthusiastic
discussion on corpus use and laughter annotation. The
authors also thank Dr. Takashi Nose for his technical advice
on HMM-based synthesis.

R.R. Provine, “Laughter: a scientific investigation,” Penguin
Books, 2001.
[2] S. Dupont, H. Çakmak, W. Curran, T. Dutoit, J. Hofmann, G.
MkKeown, O. Pietquin, T. Platt, W. Ruch and J. Urbain, “Laughter
research: a review of the ILHAIRE project,” Toward Robotic Socially Believable Behaving Systems - Volume I, 2016, pp. 172–181.
[3] K. P. Truong and D.A. van Leeuwen, “Automatic detection oflaughter,” Proc. Interspeech, 2005, pp. 485–488.
[4] M. T. Knox and N. Mirghafori, “Automatic laughter detection
using neural networks,” Proc. Interspeech2007, 2007, pp. 2973–
2976.
[5] T. Neuberger, A. Beke and M. Gósy, “Acoustic analysis and automatic detection of laughter in Hungarian spontaneous speech,”Proc. ISSP, 2014, pp. 281–284.
[6] S. Sundaram and S. Narayanan, “Automatic acoustic synthesis ofhuman-like laughter,” J. Acoust. Soc. America, 121, 2007, pp. 527–
535.
[7] A. T. Sathya, K.K. Sudheer and B. Yegnanarayana, “Synthesis oflaughter by modifying excitation characteristics,” J. Acoust. Soc.
America, 133, 2013, pp. 3072–3082.
[8] J. Oh and G. Wang, “Lolol: laugh out loud on laptop,” Proc.
International Conference on New Musical Instruments, 2013.
[9] J. Trouvain and M. Schroder, “How not to add laughter to synthetic speech,” Proc. Workshop Affective Dialog Systems, 2004,
pp. 229–232.
[10] E. Lasarcyk and J. Trouvain, “Imitating conversational laughterwith an articulatory speech synthesis,” Proc. Interdisciplinary Workshop Phonetics of Laughter, 2007, pp. 43–48.
[11] J. A. Bachorowski, M. J. Smoski and M. J. Owren, “The acoustic
features of human laughter,” J. Acoust. Soc. America, 110, 2001,
pp. 1581–1597.
[12] M. Schröder, “Experimental study of affect bursts,” Proc. ISCA
Workshop “Speech and Emotion”, 2000, pp. 132–137.
[13] W. Ruch and P. Ekman, “The expressive pattern of laughter,”Emotion, qualia and consciousness, 2001, pp. 426–443.
[14] J. Trouvain, “Phonetic aspects of “speech-laughs”,” Proc. the 2nd
Conference on Orality & Gestuality (ORANGE), 2001, pp. 634–639.
[15] K.J. Kohler, “’Speech-laugh’, ’speech-laugh’, ’laughter’ and theirsequencing in dialogic interaction,” Phonetica, 65, 2008, pp. 1–18.
[16] K. El Haddad, I. Torre, E. Gilmartin, H. Çakmak, S. Dupont, T.
Dutoit and N, Campbell, “Introducing AmuS: The amused speech
database,” Proc. International Conference on Statistical Language andSpeech Processing, 2017, pp. 229–240.
[17] J. Urbain, H. Cakmak and T. Dutoit, “Development of HMM-based
acoutic laughter synthesis,” Interdisciplinary Workshop Laughter andOther Non-Verbal Vocalisations in Speech, 2012.
[18] J. Urbain, H. Cakmak and T. Dutoit, “Evaluation of HMM-basedlaughter synthesis,” Proc. ICASSP 2013, 2013, pp. 7835–7839.
[19] J. Urbain, H. Çakmak, A. Charlier, M. Denti and T. Dutoit,“Arousal-driven synthesis of laughter,” IEEE Journal of Selected
Topics in Signal Processing, 8, 2014, pp. 273–284.
[20] K. El Haddad, S. Dupont, J. Urbain and T. Dutoit, “Speech-laugh:an HMM-based approach for amused speech synthesis,” Proc.
ICASP, 2015, pp. 4939–4943.
[21] K. El Haddad, H. Çakmak, S. Dupont and T. Dutoit, “Breath andrepeat: An attempt at enhancing speech-laugh synthesis quality.”
Proc. Signal Processing Conference, 2015, pp. 355–358.
[22] H. Çakmak, K. El Haddad and T. Dutoit, “Audio-visual laughter
synthesis system,” Proc. the 4th Interdisciplinary Workshop on
Laughter and Other Non-verbal Vocalisations in Speech, 2015,
pp. 11–13.
[23] T. Yoshimura, K. Tokuda, T. Kobayashi and T. Kitamura, “Simultaneous modeling of spectrum pitch and duration in HMM-based
speech synthesis,” IEICE Trans. Information & Systems, 83, 1999,
pp. 2347–2350.
[24] K. Laskowski and S. Burger, “Analysis of the occurrence of laughter in meetings,” Proc. Interspeech, 2007, pp. 1258–1261.
[25] K. P. Truong and J. Trouvain, “On the acoustics of overlapping laughter in conversational speech,” Proc. Interspeech, 2012,
pp. 851–854.

1949-3045 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TAFFC.2018.2813381, IEEE
Transactions on Affective Computing
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING

9[26] H. Mori, T. Satake, M. Nakamura and H. Kasuya, “Constructing
a spoken dialog corpus for studying paralinguistic informationin expressive conversation and analyzing its statistical/acoustic
characteristics,” Speech Communication. 53, 2011, pp. 36–50.
[27] Y. Arimoto, H. Kawatsu, S. Ohno and H. Iida, “Naturalistic
emotional speech collection paradigm with online game and its
psychological and acoustical assessment,” Acoustical Science andTechnology, 33, 2012, pp. 359–369.
[28] J. Trouvain, “Segmenting phonetic units in laughter,” Proc. 15thInt’l Congress of Phonetic Sciences (ICPhS), 2003, pp. 2793–2796.
[29] J. Urbain, T. Dutoit, A. Moinet, R. Niewiadomski, C. Pelachaud,
B. Picart, J. Tilmanne and J. Wagner, “The AVLaughterCycle
database,”, Proc. 7th Int’l Conf. Language Resources and Evaluation(LREC 2010), 2010, pp. 2996–3001.
[30] P. Boersma and D. Weenink, “Praat: doing phonetics by computer[Computer program],” http://www.praat.org/.
[31] H. Mori, “Toward morphological classification of affect bursts,”
Proc. Acoust. Soc. Japan 2015 Spling Meeting, 2015, pp.405–406 (inJapanese).
[32] H. Bunt, J. Alexandersson, J.-W. Choe, A. Fang, K. Hasida, V.
Petukhova, A Popescu-Belis and D. Traum, “ISO 24617-2: A
semantically-based standard for dialog annotation,” Proc. 8th Int’l
Conf. Language Resources and Evaluation (LREC 2012), 2012, pp. 430–
437.
[33] R. Gardner, “When listeners talk: Response tokens and listenerstance,” Amsterdam: John Benjamins, 2001.
[34] M. Mehu, “Smiling and laughter in naturally occurring dyadic
interactions: Relationship to conversation, body contacts, and displacement activities,” Human Ethology Bulletin, 26, 2011, pp. 10–28.
[35] K. Tokuda, H. Zen, J. Yamagishi, T. Masuko, S. Sako, A.W. Blackand T. Nose, “The HMM-based speech synthesis system (HTS),”
http://hts.sp.nitech.ac.jp/.
[36] J. Yamagishi, M. Tamura, T. Masuko, K. Tokuda and T. Kobayashi,“A context clustering technique for average voice models,” IEICE
Trans. Information and Systems, E86-D, 2003, pp. 534–542.

Tomohiro Nagata received B.E. and M.E. degrees from Utsunomiya University in 2011 and2013, respectively. He is currently working toward the Doctor’s degree. His research interests
include spontaneous speech synthesis, affective
computing, human-machine interaction, and social signals processing.

Hiroki Mori received B.E., M.E. and Ph.D. degrees from Tohoku University, in 1993, 1995
and 1998, respectively. He was with the Graduate School of Engineering, Tohoku University
in 1998. He is an associate professor of Utsunomiya University. His research interests include
verbal and nonverbal communication, corpus design and development for speech science andconversation analyses, analysis and modelingof prosody and paralanguage, and expressive
speech synthesis. He is a member of the IEICE,
the Acoustical Society of Japan, IPSJ, and ISCA.

1949-3045 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.