Laughter and Other Non-Verbal Vocalisations Workshop 2024
16-17 July 2024, Belfast, United Kingdom

Preliminary analysis of facial expressions and body movements of four types oflaughter
Kexin Wang1,3 , Carlos Ishi2,3 , Ryoko Hayashi3
1 ATR, 2 RIKEN, 3 Kobe University

wkx19970303@hotmail.com, carlos@atr.jp, rhayashi@kobe-u.ac.jp
total of approximately 12 hours of conversation data by fourmale and four female speakers. The annotation of laughter wasconducted at the bout level, according to the definition oflaughter bout in [7, 8].

Abstract
In the present work, we explored the facial expressions andbody movements of different laughter types. 1806 laughter
events were extracted from a multimodal dataset of three-party
conversations, categorized into four types: mirthful, boosting,smoothing, and softening. The results showed that laughter
performing different social functions is related to different
visual expression patterns. Mirthful and boosting laughter
showed a similar tendency that be accompanied by largerchanges in facial expressions and body movements thansmoothing and softening laughter, for example, higher cheeksraising, wider mouth open and apparent upper body frontward.
We also attempted to analyze the dynamic changes occurring
within a laughter event. Our findings provide hints for thedesign of human-like conversational agents.
Index Terms: laughter types, facial expression, bodymovement, spontaneous conversation

2.2. Four types of laughter
Referring to several previous works on classifying laughter
functions [9-11], we categorized laughter into four main types
based on its function in the conversation: (1) Mirthful laughter
is characterized by a sense of delight or pleasure in response tosomething funny, amusing, or enjoyable, which is typicallyspontaneous. (2) Boosting laughter is prompted by a desire orconscious effort to create a positive atmosphere, such as making
jokes or self-mockery. (3) Smoothing laughter serves tofacilitate social interactions and maintain human relationships,
often occurring in situations that are not necessarily humorous,
such as showing comprehension, expressing interest or simply
acting as a backchannel. (4) Softening laughter is used to reduceawkwardness or intrusion, for example, making negative
comments or sharing different opinions from other speakers,
and to signal that the speaker does not intend harm or offense.
The laughter function was annotated by three research
assistants, all Japanese native speakers, based on audio andvideo of the laughter events with 5-second pre- and 6-second
post-context. The post-context is set to be longer than the precontext to account for body movements after a laughter bout.
Decisions were made by the majority vote of three
annotators. 470 cases of perfect agreement (26%), 953 cases ofagreement between two annotators (53%), and 383 cases ofcomplete disagreement (21%) were obtained. The four
predominant laughter types (mirthful, boosting, smoothing andsoftening) account for 40%, 12%, 9%, and 8%, respectively.
Other laughter types, for example, self-conscious laughter,dumbfounded laughter, and bitter laughter were also observed.
As for the few appearances, they were not used in the analysis.

1. Introduction
Laughter is an important nonverbal behavior in humancommunication. It not only arises from humor and amusement,
but also occurs in a wide range of social contexts [1, 2]. Given
that different types of laughter contain diverse functions in
specific situations, what kind of cues do humans use torecognize and discern them?
Some previous works have suggested that genuine mirthfullaughter can be distinguished from voluntary polite laughter
based on acoustic features, such as duration, vowel quality, andvoice quality [3-5]. Moreover, research on laughter has also
expanded to explore its multimodal expressions, particularly inthe field of laughter synthesis in communicational agents.
However, most of them focused on symbolic facial expressions,and may ignore the variations or dynamic features in smilingfaces according to different types of laughter [6].
In this study, we used a face-to-face free-topic conversation
dataset to investigate the relationship between different laughter
types and their facial expressions and body motions.

2.3. Annotation of multimodal expression
The label sets used to annotate the visual features are shown as
follows. The degree of movement was also taken into
consideration, for example, 3 levels of cheeks raising (3 levels:0: not; 1: slightly; 2: clearly) (Fig. 1), 2 levels for mouth open,
and 3 levels for body frontward. The label “other” in the facial
expressions includes motions difficult to identify due to othermodalities like head downward or hands covering the face.

2. Analysis data
2.1. Dataset
In this study, we used a three-party conversation dataset. It
includes several sets of free-topic conversations (such as family,
recent work and past experiences), each lasting 15 to 20 minutes.
The participants were colleagues from the same workplace andall of them were Japanese speakers. Headset microphones andcameras were set up for each speaker, so that multimodal data
including audio, visual, and motion data were collected. For thepresent analysis, 1806 laughter events were extracted from a⚫
⚫
⚫
⚫
⚫

21

Cheeks: not raised, raised1, raised2, otherEyes: closed, narrowed, normal, open, other
Mouth: closed, open1, open2, other
Upper body: front1, front2, front3, back, left/right, turnback, shoulders tremble, shoulders move, other, nomotion
Head: up, down, front, tilt, other, no motion

10.21437/LW.2024-6

The annotation was conducted by the first author and threeresearch assistants. They were instructed to choose the most
predominant expressions for each modality within a laughter
event, and the perceived degree. If there are clear changes in theexpression, multiple labels are allowed. The agreement of 4annotators was measured by Cronbach’s alpha of the motion
degrees for cheeks, lip corners, eyes, mouth and Fleiss' kappa
for the label categories in upper body and head, with 400
samples (25 samples per each of the 8 speakers) annotated by
all four annotators. The values were 0.75, 0.51, 0.82, 0.68, 0.59,
and 0.53, respectively.

(“no motion”). However, mirthful and boosting laughter
involved more upper body movement than smoothing andsoftening laughter, particularly evident in large upper bodyforward (“front2” and “front3”), returning to the normal
position (“turn back”) and shoulders trembling.
Furthermore, two types of shoulder movements wereobserved in our data: rapid rhythmic trembling (“shoulderstremble”) and single movement (“shoulders move”), bothassumed to be influenced by vocalization. Rhythmic trembling
movements were relatively more common in mirthful andboosting laughter, while single movements were more common
in softening laughter. We also observed individual differencesin shoulder movements. Two participants exhibited shoulder
movements in approximately half of their laughter events, while
the other two participants only displayed shoulder movements
was less than 10% of cases.
Regarding head motions, as they are easy to be affected by
the motion of upper body, only those not synchronized withupper body motions were annotated as head motions. Whileapproximately 80% of laughter showed no head movement,
mirthful laughter showed a significantly higher proportion of
downward and frontward head movements compared to theother three types.

3. Results
3.1. Analysis of facial expressions
Fig. 2 shows the distributions of the facial expressions (cheeks,eyes and mouth) according to the different laughter types. Chisquared tests were conducted to verify if the occurrence rates of
specific labels are significantly higher (▲p < 0.05, ▲▲ p <0.01) or lower (▽ p < 0.05, ▽▽ p < 0.01) among the four typesof laughter. The meanings of symbols are the same in Figure 3.
From the results of cheek movements, it can be observedthat over 90% of laughter involved cheek raising in all fourtypes. However, mirthful laughter and boosting included ahigher occurrence of level 2 raising, which was significantly
higher than the proportions of the other two laughter types.
Conversely, level 1 raising in smoothing laughter and softening
laughter were both over 70%. Besides, we also observed minor
cases of raising degree increasing from level 1 to level 2(“raised 1+2”) within a laughter event in all laughter types.

4. Conclusions
In this work, we analyzed facial expressions and body
movements for four types of laughter. A general distinction was
found between laughter with positive emotions and attitudes(mirthful and boosting) and laughter serves to facilitate
interaction, relieve tension, and establish social harmony(smoothing and softening). The former two types tend to beaccompanied by larger changes in smiling faces and upper body
movements, which indicates that the degree of emotionalarousal may affect the behavior of visual expressions. However,mirthful laughter and boosting laughter can still be
distinguished by the distribution of head motions.

Regarding eye motions, around 80% of laughter involved
eye narrowing in all four types. The rate of normal eye openness
was higher in smoothing laughter and lower in mirthful laughter,
while the occurrence of eyes closed was more often in mirthfullaughter. Additionally, changes in the degree of eye openness(“smaller” and “bigger”) during laughter events were also
observed. In mirthful and boosting laughter, further eyenarrowing (“smaller”) was slightly more common. This
dynamic change may be influenced by the degree of mirth or
the intention to liven up the atmosphere.
Although eye narrowing is associated with cheek raising,
the proportion of normal eyes open is higher than cheeks notraised in boosting, smoothing, and softening laughter. In otherwords, cheeks raising is not necessarily accompanied by eyes
narrowing. Moreover, we examined whether the crow's feet(wrinkles around the eye corners) appeared in the smiling face.
As a result, 66% of mirthful laughter, 70% of boosting laughterand only 34% of both smoothing and softening laughter wereobserved. The results indicate that eye behaviors may be the cue
to distinguish involuntary laughter from voluntary ones.
Regarding mouth opening, over 90% of laughter was
accompanied by mouth opening. However, mirthful andboosting laughter tend to involve wider open compared tosmoothing and softening laughter. Conversely, despite thelower rates, keeping the mouth closed occurred more often insmoothing and softening laughter.

On the other hand, the patterns of smoothing laughter andsoftening laughter were very similar and hard to be
distinguished only by facial expressions and body movements.
Some works show the possibility of laughter with different
pragmatic functions containing different gaze patterns [12, 13].
In order to improve the distinction of different laughter types,
our next step is to explore the relation between laughter types
and gaze behavior, as well as the audio features and contextual
information.
The results in the present work show potential for a deeper
understanding of the nuances of laughter and its role incommunication, which can also be applied to the synthesis ofnatural and expressive laughter.

5. Acknowledgements
This study was funded by JSPS KAKENHI Grants (No.
JP22H04875 and 21H04348). We would like to thank Yuka
Nakayama, Naomi Uratani, and Yumiko Kondo for theircontributions to the data annotations and helpful discussions.

3.2. Analysis of body movements
Fig. 3 shows the distributions of the body movements(upper body and head motions) according to the different
laughter types. Overall, in around half of the cases of smoothing
and softening laughter, the upper body remains relatively still226. References[1]

R. R. Provine, Laughter: A Scientific Investigation. New York:Viking, 2000.
[2] W. Scott, Ha!: The Science of When We Laugh and Why. New
York: Basic Books, 2014.
[3] H. Tanaka, and N. Campbell, “Classification of social laughter innatural conversational speech,” Computer Speech & Language,
vol. 28, no. 1, pp. 314–325, Jan. 2014.
[4] N. Lavan, S. K. Scott, and C. McGettigan, “Laugh like you mean
it: Authenticity modulates acoustic, physiological and perceptual
properties of Laughter,” Journal of Nonverbal Behavior, vol. 40,no. 2, pp. 133–149, Jun. 2016.
[5] T. Shochi, M. Guerry, H. Suzuki, and M. Kanzaki, “Vocal aspectof social laughter during virtual interaction,” Working Papers em
Linguística, vol. 19, no. 2, pp. 54–77, Dec. 2019.
[6] C. T. Ishi, T. Minato, H. Ishiguro, “Analysis and generation oflaughter motions, and evaluation in an android robot,” APSIPA
Transactions on Signal and Information Processing, vol. 8, no. 1,pp. 1–10, Jan. 2019.
[7] M. J. Owren, R. M. Seyfarth, and D. L. Cheney, “The acoustic
features of vowel-like grunt calls in chacma baboons (Papiocyncephalus ursinus): Implications for production processes andfunctions,” the Acoustical Society of America, vol. 101, pp. 2951–2962, May. 1997.
[8] K. P. Truong, J. Trouvain, and M. P. Jansen, “Towards anAnnotation Scheme for Complex Laughter in Speech Corpora,”
in Proc. Interspeech 2019, Graz, Austria, Sep. 2019, pp. 529–533.
[9] H. Giles, and G. S. Oxford, “Towards a multidimensional theoryof laughter causation and its social implications,” Bulletin of theBritish Psychological Society, vol. 23, no. 79, pp. 97–105, Jan.
1970.
[10] C. Mazzocconi, T. Ye, and J. Ginzburg, “What's your laughter
doing there? A taxonomy of the pragmatic functions of laughter,”IEEE Transactions on Affective Computing, vol. 13, no. 3, pp.
1302–1321, 1 Jul-Sep. 2022.
[11] D. P. Szameitat, K. Alter, A. J. Szameitat, C. J. Darwin, D.
Wildgruber, S. Dietrich, and A. Sterr, “Differentiation ofemotions in laughter at the behavioral level,” Emotion, vol. 9, no.
3, pp. 397–405, Jun. 2009.
[12] E. Gironzetti, L. Pickering, M. Huang, and Y. Zhang, “Smilingsynchronicity and gaze patterns in dyadic humorousconversations,” Humor, vol. 29, no. 2, pp. 301–324, Jan. 2016.
[13] C. Mazzocconi, V. Maraev, V. Somashekarappa, and K. Howes,“Looking for Laughs: Gaze Interaction with Laughter Pragmatics
and Coordination,” in Proc. ICMI 2021, Montreal, Canada, Oct.
2021, pp. 636–644.

Figure 2: Distributions of lip corners, eyes and mouth
motion categories for four laughter types.
The symbols within the bars mean significantly higher(▲p < 0.05, ▲▲ p < 0.01), or lower occurrences(▽ p < 0.05, ▽▽ p < 0.01), after chi-squared tests.

Figure 1: Examples of 3 levels of cheeks raising.
From left to right: not raised, slightly raised, clearly raised.

Figure 3: Distributions of upper body and head motion
categories for four laughter types.

23