階層的 Tensor Fusion を用いた交渉対話における嘘検出
Hierarchical tensor fusion for deception detectionof negotiation dialog
グエン テトウン 1∗ 吉野幸一郎 1 サクリアニ サクティ1 中村哲 1
Nguyen The Tung1
Koichiro Yoshino1
Sakriani Sakti1 Satoshi Nakamura1
奈良先端科学技術大学院大学
Nara Institute of Science and Technology
11

Abstract: In negotiation dialogs, it is common for negotiator to use lies for getting a more
beneﬁcial outcome. Therefore, to negotiate eﬃciently, it is important to know whether the otherparty is telling the truth or not. This deception information can be determined from various visualand acoustic clues. In this paper, we proposed a new method called hierarchical tensor fusion
network for combining visual and acoustic modalities for the task of deception detection in negotiation dialog. The hierarchical tensor fusion model has the best performance in our experiments,
outperforming the existing approaches used in previous studies (hierarchical and tensor fusion).
We inspected the performance of negotiation dialog system when using output labels from multiple
deception detection methods and saw that the dialog system achieves highest performance whenusing labels from the hierarchical TFN model.

1

Introduction

In a negotiation, interlocutors can use lies to gain
advantages and reach outcomes that are more beneﬁcial for them. In order to successfully negotiate withhuman users, dialog systems need an accurate deception detection module [7].
According to existing studies, acoustic factors (pitch,intensity, and speaking rate) and visual clues (facialexpressions) are good indicators to tell if someone isbeing deceptive [1, 2]. From these literatures, it is expected that incorporating visual features to acousticfeatures has the potential to achieve high deception
detection accuracy.
There are many diﬀerent methods to combine features from multiple modalities. A majority of multimodal processing works still use traditional methods
such as early or late fusion. On the other hand, moreadvanced combination methods like hierarchical fusion [3] or tensor fusion network [4] have been proposed and was shown to outperform the early andlate fusion in tasks such as emotion recognition orsentiment analysis.
∗ 連絡先： 奈良先端科学技術大学院大学

奈良県生駒市高山町８９１６番地の５E-mail: nguyen.tung.np5@is.naist.jp

In this study, we propose the hierarchical tensor
fusion network (Hierarchical TFN), a new type ofmethod for fusing multi-modal features eﬃciently using neural network. Our proposal can be seen as anintegration of the hierarchical and the tensor fusion
network. The proposed method balances the feature
abstraction level by a hierarchical structure and explicitly combines diﬀerent types of features by outer
product, thus eliminating the weaknesses of previousmethods.
We conducted experiment on deception detection
task and observed that the model based on the proposed hierarchical tensor fusion method achieved thebest score, outperformed the existing methods. Inaddition, we used the predicted labels from deception
detection models based on diﬀerent fusion methods as
inputs of an reinforcement-learning-based negotiation
dialog manager [5]. We observed that when using thepredicted deception labels from the hierarchical tensor
fusion based classiﬁers, the dialog manager achieved
a high score.

2

Related works

There are a lot of studies about how to combine
multi-modal features eﬃciently. Tian et.al., [3] proposed hierarchical fusion for emotion recognition. This
network has a hidden layer that connects with bothacoustic and facial input feature vectors. The architecture was constructed under the assumption that
acoustic and facial features have diﬀerent levels of abstractions. Tensor fusion network (TFN) [4] composesa tensor of the acoustic features and facial features.
This fusion method achieved high performance in thetasks of face recognition and sentiment analysis. Suchperformance can be explained by the fact that TFN
can consider the outer product of diﬀerent type features explicitly. Despite of those achievements, hierarchical and tensor fusion still contain some weaknesses. For the tensor fusion network, it assumes
that all modalities have the same level of abstraction,
thus making the network structure large and complex.
This property makes it diﬃcult to train a networkfrom small amount of data. The hierarchical fusion
uses concatenation when fusing diﬀerent modalities,
which makes learning of multi-modal fusion very complicated. From these observations, we proposed theHierarchical TFN and expected that it would able tomodel the important features contributing to the improvement of deception detection accuracy.

3…

…

…
…

visual…

M

…

…

…

𝑥, 𝑥 = 𝑉 × 𝐴
𝑀𝑖𝑗 = 𝑣𝑗 . 𝑎𝑖
Fusion subnetwork:𝑣 = vector of visual embedding, 𝑣 = 𝑉
𝑎 = vector of acoustic embedding, 𝑎 =A Learning inter-modality interactions

Embedding subnetworks:
Learning intra-modality interactions

図 1: Hierarchical tensor fusion network.

data consists of recorded videos of health consultation
dialogs [5]. In total, our deception detection dataset
used in the experiment includes 1265 utterances.
表 1: Results of deception detection.
ModelAccuracy

Precision

Recall

F1-score

single acoustic
single visual
multi early
multi latemulti hierarchicalmulti TFN
multi hierarchical TFN

53.78%
49.28%
53.42%
54.68%
53.78%
50.36%
58.63%

0.47470.40950.46030.47940.47330.42160.53040.50000.35250.3566
0.38110.47130.35250.50000.48700.38790.40180.42470.47230.38390.5148

Table 1 summarizes performance of diﬀerent models
in the deception detection task. We can see that thehierarchical TFN outperforms all the other methods.

MethodThe structure of our proposed network is shown inFigure 1. As can be seen from this ﬁgure, the hierarchical TFN structure is similar to a hierarchicalnetwork, but multi-modality fusion is performed byusing the outer product (same as TFN) instead of concatenating. Our proposed method’s advantages overa hierarchical network is similar to that of TFN overearly fusion thanks to the use of the outer product forfusion. The hierarchical TFN also resembles a TFN
structure; however, raw acoustic features are used as
inputs of tensor fusion instead of an acoustic embedding vector as we can see in TFN.

4

acoustic

表 2: Accuracy of dialog acts selection when using
diﬀerent deception labels result.
Deception labels used for dialog management

DA
accuracy

chance rate deception
gold-label deception

65.69%
80.31%

single visual prediction
single acoustic prediction
multi early prediction
multi late prediction
multi hierarchical prediction
multi TFN prediction
multi hierarchical TFN prediction

70.15%
66.22%
66.48%
68.58%
69.10%
69.66%
71.20%

Experiments

The dataset we used for deception detection includes two types of data. The ﬁrst one is created by
splitting videos from [6] into utterances. The second

In the next experiment, we used predicted labelsfrom deception detection models for a dialog manager
that decides output dialog acts (DA) on the basis of
the user’s deception information (whether the user islying or not). The dialog act labels are decided by
POMDP-based dialogue manager [5], which can incorporate the deceptive labels with the user’s dialogact labels for decide a better dialogue act of the system. It is clear that the dialog manager achieved highscore when using deception labels from hierarchical
TFN model.

5Conclusion

In this research, we proposed a new method calledhierarchical tensor fusion network for combining features from diﬀerent modalities. We applied the proposed method to build a classiﬁcation model for thetask of deception detection. Results from our experiment showed that the hierarchical tensor fusion modelhas the best performance, outperforming the existing
methods. We also conducted experiment about theDA selection accuracy of a negotiation dialog system
when using output labels from multiple deception detection models and observed that the dialog manager
achieves high performance when using labels from thehierarchical TFN fusion.

Acknowledgment
Part of this work was supported by JSPS KAKENHI Grant Number JP17H06101.

参考文献[1] Hirschberg, J., Benus, S., Brenier, J.M., Enos,F., Friedman, S., Gilman, S., Girand, C., Graciarena, M., Kathol, A., Michaelis, L.,et.al: Distinguishing deceptive from non-deceptive speech,Interspeech, pp. 1833–1836 (2005)[2] Ekman, P. and Friesen, W.V.: Detecting deception from the body or face, Journal of personality and Social Psychology,, Vol.29, No.3, pp. 288(1974)[3] Tian, L., Moore, J., and Lai, C.,: Recognizing
emotions in spoken dialogue with hierarchically
fused acoustic and lexical features, Spoken Language Technology Workshop (SLT), pp. 565–572(2016)[4] Zadeh, A., Chen, M., Poria, S., Cambria, E., andMorency, L.P.: Tensor fusion network for multimodal sentiment analysis, Proceedings of the2017 Conference on Empirical Methods in Natural Language Processing, pp. 1103–1114 (2017)[5] The Tung, N., Yoshino, K., Sakti, S., and Nakamura, S.: Impact of deception information onnegotiation dialog management, IWSDS,, (2018)[6] Pérez-Rosas, V., Abouelenien, M., Mihalcea, R.,
and Xiao, Y.: Verbal and nonverbal clues forreal-life deception detection. EMNLP, pp.2336–2346 (2015)[7] The Tung, N., Yoshino, K., Sakti, S., and Nakamura, S.: Dialog Management of Healthcare
Consulting System by Utilizing Deceptive Information JSAI, (2019)