Deception Detection and Analysis inSpoken Dialogues based on fastText
Naoki Hosomi, Sakriani Sakti,
Koichiro Yoshino, Statoshi Nakamura
NAIST, Japan
PRESTO, JST, JapanRIKEN-AIP, Japan
Nara Institute of Science and Technology
Augmented Human Communication Laboratory
PRESTO, Japan Science and Technology Agency
2018/11/14 ©Koichiro Yoshino AHC-Lab.
NAIST, PRESTO JSTAPSIPA 20181

Deceptions in dialogues• An act that intentionally causes another persons to hold afalse belief
× Ignorance, mistaking, false recognition• It is hard for humans to predict deceptions (Chance-level)– Characteristic features of deception/truth are too small to becaptured by humans• Humans have biases– Truth-bias: people tend to judge
speech as valid regardless whether
it is true or not

2018/11/14 ©Koichiro Yoshino
AHC-Lab. NAIST, PRESTO JSTAPSIPA 20182

Deception detection by machines• It is expected that machines can capture small difference
between deceptive and truthful behaviors– E.g., supervised learning• It is also expected to predict deceptions from natural spoken
conversations– Without using special equipment
such as polygraph– Deception detection from natural
speech is also useful for dialogue
tactics (non-cooperative case)
Deception detection based on polygraph
From：http://www.pref.tottori.lg.jp/98373.htm
2018/11/14 ©Koichiro Yoshino
AHC-Lab. NAIST, PRESTO JSTAPSIPA 20183

Deception detection from spoken language
Using acoustic features andtheir emotion labels on SVM
0.6％ improvements fromchance-level[Amiriparian, 2016]

Using acoustic, lexical and features
depending on individuality by rules
6% improvements from chance-level[Hirschberg, 2005]

uh…, I didn’t eat that cookie.

Building Large-scale corpora
Acoustic and lexical (embedding) features for
Bidirectional LSTM
F-1: 0.64 (Precision:0.67, Recall:0.61)[Levitan, 2015], [Mendels, 2017]2018/11/14 ©Koichiro Yoshino
AHC-Lab. NAIST, PRESTO JSTAPSIPA 20184

Research purpose• Building multimodal deception detection– Hypothesis: multimodal (acoustic and linguistic) has different
contributions to deception detection– Experiments: Combine features with multi-layer perceptron (MLP)
classifier and compare with single-modal features• Comparison between deception detections by humans and bystatistical models– Hypothesis: It is hard for humans to capture deceptions– Experiments: giving the same training & test to both humanparticipants and the model; then comparing these results

2018/11/14 ©Koichiro Yoshino
AHC-Lab. NAIST, PRESTO JSTAPSIPA 20185

Acoustic features• Emotion and deception has an strong relation– 384 dimensional acoustic features used for emotion recognition(INTERSPEECH2009 competition) were extracted by OpenSMILE
toolkit– The feature is raw-level but expected to contribute for deception
detection [Mendels 2017]
SpeechFeatures
OpenSMILE

ZCRmeanRMS Energy standard deviation

2018/11/14 ©Koichiro Yoshino
AHC-Lab. NAIST, PRESTO JST

F0kurtosis, skewness

HNR

extremes: value, rel, position, range

MFCC 1-12

linear regression: offset, slope, MSE
APSIPA 20186

Linguistic features• Features extracted from fastText [Joulin 2016]– Distributed representation of sentence– The representation is optimized not only to embed similar words
into close points but also improving the labeling accuracy
𝑒1

Fully connected NN𝑒2𝑚𝑦𝑒𝑁−1Embedding dog𝑥𝑁
2018/11/14 ©Koichiro Yoshino
AHC-Lab. NAIST, PRESTO JST𝑒𝑁

fastText
APSIPA 20187Model (MLP)• Linear interpolation of MLP based on acoustic features andlinguistic features (fastText)
Acoustic

w1 = 0.6, w2 = 0.4(Decided by validation set)

𝑥1

𝑦1

𝑥𝑛

Linguistic

𝑤1 𝑦1
Label1: Truth𝑥384

Label2: Deception
fastText
dog𝑦2

𝑤2 𝑦2

𝑥𝑁

2018/11/14 ©Koichiro Yoshino
AHC-Lab. NAIST, PRESTO JSTAPSIPA 20188

Corpus (CSC deceptive speech)• Dialogue corpus including deception– Naïve English speakers (American, 16 males and 16 females)– 7 hours speech and their transcriptions– Truth/deception labels are annotated by the speakers themselves,pushing the button during the conversation
Transcription examples

labelwell, yeah, there's a chance.

Tuh actually, I did well. excellent.

D

2018/11/14 ©Koichiro Yoshino
AHC-Lab. NAIST, PRESTO JSTAPSIPA 20189

Experimental setup• Experiment 1:– Deception detection based on acoustic (single), linguistic (single)
and acoustic+linguistic (multimodal) features• Experiment 2:– Deception detection by humans (6 participants)• Non-native (average TOEIC score was 861.7)• Participants are given speeches and transcriptions to decide theirlabels (3 conditions: acoustic, linguistic and multimodal)• Participants freely confirmed utterances and their labels in thetraining data before the experiment– Compared with machine results
2018/11/14 ©Koichiro Yoshino
AHC-Lab. NAIST, PRESTO JSTAPSIPA 201810

Given tasks to human participants• Texts, speeches or texts+speeches of human utterance aregiven to participants– Randomly chosen from text
– Order of experiments are randomly selected• No dialogue contexts are given– Human participants just judged labels by using texts and speechesof utterances themselves

uh actually, I did well.

2018/11/14 ©Koichiro Yoshino
AHC-Lab. NAIST, PRESTO JST

Deceptionuh actually,
I did well.

Classification

APSIPA 201811

Number of dataset• Randomly selected utterances that has more than 5 words– It is very hard to label shorter sentences• Samples are selected to adjust the ratio of truth/deception
as 1:1– Chance rate of our setting is 50%– Human participants don’t know the ratio#Truth sentences#Deceptive sentences

Test5050

Train

1800

1800

Validation

2002002018/11/14 ©Koichiro Yoshino
AHC-Lab. NAIST, PRESTO JSTAPSIPA 201812

Results of proposed classifiers
Results of MLP classifiers
Accuracy

Precision (D)Recall (D)F-1 (D)

Acoustic

0.580

0.577

0.600

0.588

Linguistic

0.620

0.630

0.580

0.604

Acoustic+Linguistic

0.640

0.667

0.560

0.609

• The method using multimodal feature achieved the state of
the art accuracy & F-1 score for deception labels– Combining acoustic and linguistic features has an advantage thanusing single-modal features

2018/11/14 ©Koichiro Yoshino
AHC-Lab. NAIST, PRESTO JSTAPSIPA 201813

Results of human participants
Results of human participants
Accuracy

Precision (D)Recall (D)F-1 (D)

Acoustic

0.515

0.524

0.370

0.414

Linguistic

0.510

0.515

0.387

0.425

Acoustic+Linguistic

0.512

0.498

0.360

0.405

Machine (best)

0.640

0.667

0.560

0.609

• It was hard for human participants to predict deceptions– Significantly smaller than any machine learning based methods (p<.05)– Precision was chance level but recall was smaller• It indicates the truth-bias of humans(humans tend to judge that the utterance is true)
2018/11/14 ©Koichiro Yoshino
AHC-Lab. NAIST, PRESTO JSTAPSIPA 201814

Summary and future works• Summary– Multimodal features contributes to improve the deception detection
accuracy• Note that sentence that have more than 5 words are used, that is
why the linguistic feature has larger contributions– Deception detection ability of human is approximately chance-level– Human tend to predict an utterance to be true regardless of anactual label• Future works– Experiments with Native speakers– Using dialogue histories for the prediction
2018/11/14 ©Koichiro Yoshino
AHC-Lab. NAIST, PRESTO JSTAPSIPA 201815