TAPAS
Training Adapted Personalised Affective
Social Skills with Cultural Virtual Agents

Multimodal Voice Activity Prediction:
Turn-taking Events Detection
in Expert-Novice Conversation
Kazuyo Onishi, Hiroki Tanaka, and Satoshi Nakamura
Nara Institute of Science and Technology, Japan

HAI 2023 ‒ Conference in Human-Agent Interaction '23
4 ‒ 7 December, Gothenburg, Sweden

CodeJapan Science and Technology Agency

SymbioticInteraction

ANR

AGENCENATIONALE
DE LA
RECHERCHE

2BackgroundTurn-taking is …
the process of communicating in a conversationin which the speaker and the listener take turns speaking
Turn-holdTurn-shift

Back-channel (BC)... various turn-taking

Challenges for Virtual Agents
Improving virtual agent usage satisfaction ...
need human-like turn-taking performance [Toshihiko+, 2009]→ Predictive modeling to determine turn-taking

3Related Work
Previous Models
Uni ed approach to various turn-taking eventswith a voice activity prediction (VAP) model [Skantze, 2017]
Future Voice Activity (VA)?

various turn-taking events

VAPfifi• Proposed architecture to process audio and linguistic features [Roddy+, 2018]• De ne turn-taking events that extend to and evaluate Transformer models [Ekstedt+, 2022a]• Investigate the impact of prosody on models [Ekstedt+, 2022b]

4

In This StudyResearch Question
Humans make predictions based on three features (audio, linguistic, non-verbal)→ Non-verbal features have not been veri ed in VAP model

Are non-verbal features useful in voice activity prediction models?

Experiments
Verify three conditions(1) Audio Features Only
(2) Non-verbal Features Onlyfi(3) Audio + Non-verbal Features

5

Multimodal Model
Evaluating Turn-taking Events From VAP
Future VA

Output Window
0 0Transformer11 10Label00...
Index: 0 ...

...
Index: 44... Index: 255

Mapping with zero-shot

CPC
AudioFeatures

LinearNon-verbalFeatures■ SHIFT/HOLD■ SHORT/LONG■ SHIFT-pred.

S■ BC-pred.
BCSSLH＊ Code is available

Predicted Area

＊ Using code from previous research [Ekstedt+, 2022a]

6

Input Features
Audio Features
Audio Waveform• Mixing two speakersʼ audio waveforms

Non-verbal Features
Gaze Direction• Direction vector of line of sight(x, y)

Action Units
Voice Activity Frame (VA-frame)• Eyebrows, eyelids, mouth, and facial expressions• VA expressed as active 1 and inactive 01111100000
0000001110

Voice Activity History (VA-history)• Percentage of VA in the particular sections
VA1 active time
VAhistory(section) =VA2 active time

Head Pose
headx,y,z(t) =((x, y, z)t − (x, y, z)t−1)2z

Articular Points• Capture gestures from joint coordinatesnn(Articulart − Articulart−1)

2※ Di erence from one previous frame
ffx※ Di erence from one previous frame

Articular n(t) =ffy• Head movement in 3 dimensions

p2p3
p4

p1

p5p6p77Data
Corpus
NoXi Database

ExpertNovice• A pair of people through a screen• Expert shares topics with Novice• Multilingual database (English, German, French, ...)
• Dialogue similar to an agent system• Record audio and video
Splitting of Training Data
Number ofSessions
Train
54
Validation
12
Test
18

Duration [h]
16.87
3.35
5.29

S/H

S/L

1015/11757 3382/1015
246/2447
595/246354/3659 1007/354

S-pred.

BC-pred.

1015
246
354

3375
592
1005

similar to an agent system

(1) Audio Features
Audio Features Only
Audio FeaturesAudio
VAVAWaveform frame history

Turn-taking Events (F1 Score)
S/H

S/L

S-pred.

BC-pred.

0.734
0.885
0.888

0.629
0.804
0.811

0.657
0.704
0.725

0.650
0.691
0.687

+ VA-frame• Improved scores on all turn-taking events• Facilitated speaker identi cation of mixed audio waveforms+ VA-historyfi• Improved S-pred. scores by +2.1 points (p < 0.025)• Speech ratios a ect turn shiftff✔︎✔︎✔︎✔︎✔︎✔︎8

(2) Non-verbal Features
Non-verbal Features Only
Non-verbal FeaturesGaze AU Head ArticularTurn-taking Events (F1 Score)
S/H

S/L

S-pred. BC-pred.

0.814
0.837
0.856
0.856

0.625
0.649
0.643
0.631

0.656
0.676
0.697
0.692

0.530
0.591
0.428
0.467

• Achieves 70 points for S-pred. and 59 points for BC-pred.
• Outperforms results with audio waveform-only input• Non-verbal features have a signi cant impact on turn-taking

ff• Possibility that action units in particular a ect BC-pred. scoresfi✔︎✔︎✔︎✔︎

9

Audio + Non-verbal Features
Audio Features and Non-verbal Features
Audio Features

Non-verbal Features

AudioVAVAGaze AU Head Articular
Waveform frame history

Turn-taking Events (F1 Score)
S/H

S/L

S-pred.

BC-pred.

0.889
0.897
0.892
0.891

0.810
0.812
0.814
0.816

0.738
0.744
0.739
0.735

0.688
0.707
0.693
0.691

Scores signi cantly di erent compared toall audio features only

S/H and S/L
No signi cant di erence
Due to high baseline F1 scores and a the bias in the number of tasks
S-pred.
Signi cant di erence when adding gaze direction or action units

ff

ff

ff

ff

fi

fi

fi

BC-pred.
Signi cant di erence when adding action unitsfi✔︎✔︎✔︎✔︎✔︎✔︎✔︎✔︎✔︎✔︎✔︎✔︎✔︎✔︎✔︎✔︎10Result
All Audio Features and All Audio and Non-verbal Features
Audio Features,

Audio + Non-verbal

mean of F1 score

Features,(SD)mean of F1 score

S/H

0.888 (0.006)

S/L

Cohenʼsvalue

0.890 (0.009)

0.144

0.782

0.811 (0.012)

0.810 (0.013)

0.035

0.950

S-pred.

0.725 (0.011)

0.748 (0.012)

1.985

0.000

BC-pred.

0.687 (0.013)

0.705 (0.005)

1.900

0.001

• Finally, +2.3 points improvement for S-pred. and +1.8 points for BC-pred.
• Non-verbal features are useful in VAP model

王将𝑑𝑃

11

金将

12Summary
Are non-verbal features useful in VAP models?
(1) Audio Features Only

• VA-frame is essential for speaker identi cation in mixed audio waveform• Speech ratio a ects turn-shift and improves +2.1 points(2) Non-verbal Features Only

• 70 points for turn-shift and 59 points for back-channel• Non-verbal features have a signi cant impact on turn-taking(3) Audio + Non-verbal Features• +2.3 improvement for turn-shift and +1.8 for back-channel• Action units are particularly e ective in predicting turn-shift

fififfffand back-channel

CodeAppendix