Hierarchical Tensor Fusion Network for Deception
Handling Negotiation Dialog Model
11,2,31,3
1,3Nguyen The Tung , Koichiro Yoshino , Sakriani Sakti , and Satoshi Nakamura

1Augmented Human Communication – Nara Institute of Science and Technology
2PRESTO, JST, Japan
3Center for Advanced Intelligence Project (AIP), RIKEN, Japan

2. Problems: basic fusion methods

1. Overview
Problems: Current multimodal fusion methods
cannot take full advantage of the rich multimodal
information.
• Do not differentiate the abstraction level ofinformation• Complex and inefficient learning of featuresinteraction
Our solution: Hierarchical tensor fusion network(Hierarchical TFN)
• Combination of hierarchical fusion (Tian et.al2015) and tensor fusion (Zadeh et.al 2017)• Balance the abstraction level and learningfeatures interaction efficiently.
Results:• Proposed fusion method outperforms the othersby more than 4%.
• Achieves highest DA selection accuracy whenusing labels from Hierarchical-TFN-based
deception detector.

pitch (F0)energyHirschberg et.al (2005)

classification?
truth

3. Proposed fusion method

2. Problems: Advanced fusion methods
Hierarchical fusion:acoustic

acoustic…

…

acoustic…

visual…

…

…

…

𝑥, 𝑥 = 𝑉 × 𝐴

M

Advantages of hierarchical tensor fusion:
 Balance the abstraction level of different modalities.
 Separate learning of intra-modality and inter-modality
interactions.
 Forcing the network to learn useful intra-modality
interactions from certain modalities.
 Prevent learning of unimportant interactions, reduce
unnecessary parameters and make network structuresimpler.

…

…
Fusion subnetwork𝑣 = vector of visual embedding, 𝑣 = 𝑉
𝑎 = vector of acoustic embedding, 𝑎 =A

• Separate learning of intra-modality
interactions (embedding subnetwork)
and inter-modality interactions (fusionsubnetwork) Fusion subnetwork

Embedding subnetworks

𝑀𝑖𝑗 = 𝑣𝑗 . 𝑎𝑖

Embedding subnetworks

…

…

…

𝑀𝑖𝑗 = 𝑣𝑗 . 𝑎𝑖𝑣 = vector of visual embedding, 𝑣 = 𝑉
𝑎 = vector of acoustic embedding, 𝑎 =A…

visualM𝑥, 𝑥 = 𝑉 × 𝐴

…

…

…

…

Tensor fusion (TFN):…

…

…

visual• Entangle the learning of
intra-modality and intermodality interactions …

• Cannotbalancemodality
abstraction level 

…

…

…

…

…

4. Experiment #1
Deception detection

Hierarchical tensor fusion (Hierarchical TFN):• Can balance modality
abstraction level 

visualPérez-Rosas et.al (2016)

face AU* values
head direction* face AU: Face Action
head position
Unit (Eckman, 1978)

current…

acousticlie

Late fusion:o No
distinction
ofmodality abstractionlevel o Cannot learns intermodality interactions
in…

visualDeception detection :• Classification human’s spoken utterances into lie
or truth.
• Current state-of-the-art models use multimodal
approach

Early fusion:• No
distinction
ofmodality abstractionlevel o Entangle the learningof intra-modality andinter-modalityinteractions acoustic

Background: An effective negotiation system needs
to know whether the other party (user) is lying or
not to choose the most appropriate response.

Multimodal fusion methods used
multimodal deception detection works.

Dataset:• Real-life trial (Rosas et.al 2015):
recordings from court trials, 245(105/140; deceptive/truthful)• Simulated health consultation (Tunget.al 2018): 1021 (177/844)• Total: 1266 (282/984)
Features extraction:• Visual: Face Action Units, using(Baltrusaitis et.al 2016)• Acoustic: IS_09 emotion acousticfeatures set, (Eyben et.al 2010)
Experiment setup:• 4-fold cross-validation• Utterances from same recording
belongtosameset(train,development or test)
Lie100%

Truth

80%
60%
40%
20%
0%

Partition 1 Partition 2 Partition 3 Partition 4 Partition 5

Experimental results

4. Experiment #2: Negotiation System’s dialog management
Negotiation system’s dialog management
verbalUser

non-verbal

outputExperimental results
Previousstate

NLU*

Dialog state
tracker

Deception
detection

NLG**Currentstatea* Natural Language

PolicymanagerSystemUnderstanding** Natural LanguageGeneration

Dialog modeling:• Model the dialog management process using Partially Observable Markov Decision
Process (POMDP).
• Dialog state: s = 𝑢, 𝑑 - 𝑢: user’s dialog act, 𝑑: user’s deception.
• State transition: 𝑃 𝑢𝑡+1, 𝑑𝑡+1 𝑢𝑡 , 𝑑𝑡 , 𝑎ො 𝑡 = 𝑃 𝑢𝑡+1 𝑑𝑡+1, 𝑢𝑡 , 𝑑𝑡 , 𝑎ො 𝑡 𝑃 𝑑𝑡+1 𝑑𝑡 , 𝑎ො 𝑡𝑖𝑛𝑡𝑒𝑛𝑡𝑖𝑜𝑛 𝑚𝑜𝑑𝑒𝑙• Train the dialog management using reinforcement learning:𝑡+1 𝑡+1
𝑄 𝑠 𝑡 , 𝑎𝑡 = 1 − 𝛼 𝑄 𝑠 𝑡 , 𝑎𝑡 + 𝛼 𝑟 𝑡 + 𝛾 max
𝑄𝑠,𝑎𝑡+1𝑎𝑑𝑒𝑐𝑒𝑝𝑡𝑖𝑜𝑛 𝑚𝑜𝑑𝑒𝑙

Deception labels used fordialog management

System DA selection
accuracy

Chance rate deception

65.69%

Gold-label deception

80.31%

Single visual prediction

70.15%

Single acoustic prediction

66.22%

Multi early prediction

66.48%

Multi late prediction

68.58%

Multi hierarchy prediction

69.10%

Multi TFN prediction

69.66%

Multi Hierarchical TFN
prediction

71.20%• Human expert selects best reaction in eachdialog turn (based on annotated user’s action
and user’s deception)• Compare system’s choice with human choicefor each dialog turn.
• Highest accuracy of DA selection achieved
when using labels predicted by Hierarchical
TFN deception detection model.

Model

Accuracy Precision Recall F1-score

Singleacoustic

53.78%

0.475

0.500

0.487

Multi late49.28%
53.42%
54.68%

0.409
0.460
0.479

0.353
0.357
0.381

0.388
0.402
0.425

Multi
hierarchical

53.78%

0.473

0.471

0.472

Multi TFN

50.36%

0.421

0.353

0.384

Multi
hierarchical
TFN

58.63%

0.530

0.500

0.515

Single visual

Multi early• Precision, recall, and F1-score aremeasured for deceptive label (positive).
• Single visual model performance ismuch worse than single visual acoustic• The Hierarchical TFN outperforms allother methods significantly.

5. Discussion• Collect/augment more multimodal
deception data for evaluation on alarger scale• Applied this fusion methods for othermultimodal processing tasks: emotion
or sentiment analysis