MDPE: A Multimodal Deception Dataset with Personality and Emotional Characteristics

## Abstract

Abstract Deception detection has garnered increasing attention in recent years due to the significant growth of digital media and heightened ethical and security concerns. It has been extensively studied using multimodal methods, including video, audio, and text. In addition, individual differences in deception production and detection are believed to play a crucial role.Although some studies have utilized individual information such as personality traits to enhance the performance of deception detection, current systems remain limited, partly due to a lack of sufficient datasets for evaluating performance. To address this issue, we introduce a multimodal deception dataset MDPE 1 1 1 The dataset is available at https://github.com/cai-cong/MDPE, licensed under CC BY-NC-SA-4.0 license. . Besides deception features, this dataset also includes individual differences information in personality and emotional expression characteristics. It can explore the impact of individual differences on deception behavior. It comprises over 104 hours of deception and emotional videos from 193 subjects. Furthermore, we conducted numerous experiments to provide valuable insights for future deception detection research. MDPE not only supports deception detection, but also provides conditions for tasks such as personality recognition and emotion recognition, and can even study the relationships between them. We believe that MDPE will become a valuable resource for promoting research in the field of affective computing.

## 1 Introduction

Generally, deception refers to the act of misleading, tricking, or deceiving others []. It involves hiding the truth or presenting false information to create an impression that is not accurate. Deception can take many forms, including both verbal and nonverbal information []. And it also occurs in various contexts, such as interpersonal relationships, business, politics, and entertainment. Deception is often considered unethical and can have serious consequences for trust and relationships.

As deception has expanded to other fields such as social media, interviews, online transactions, and deception in daily life,the need arises for a reliable and efficient system to aid the task of detecting deceptive behavior. Many machine learning approaches have been proposed in order to improve the reliability of deception detection systems []. In particular, physiological, psychological, visual, linguistic, acoustic, and thermal modalities have been analyzed in order to detect discriminative features and clues to identify deceptive behavior [, , , ]. Video-based deception detection is a current priority in deception research, because behavioral cues can be extracted from videos in a cheaper, faster, and non-invasive manner [], which is preferable to invasive approaches that extract clues through devices attached to human bodies (e.g., polygraphs). Visual clues of deception include facial emotions, expression intensity, hands and body movements, and microexpressions. These features were shown to be capable of discriminating between deceptive and truthful behavior [, ]. Acoustic features took into account the pitch and speaking rate among other measurements to specify whether certain features are associated with an act of deceit []. Linguistic features were usually extracted from the language, words usage, and consistency of the statements made by a person []. Recently, multimodal analysis has gained a lot of attention due to their superior performance compared to the use of unimodal modalities. In the deception detection field, several multimodal approaches [, , , ] have been suggested to improve deception detection by integrating features from different modalities. This integration created a more reliable system that is not susceptible to factors affecting sole modalities and polygraph tests.

In addition, it is firmly believed that there are individual differences in deception production and detection [, , ].
Specifically, it includes cognitive level, personality traits, psychological characteristics, and emotional expression. Everyone has different personalities and psychological characteristics, and the expression of emotions is also various. It has been demonstrated through several studies that personality factors and emotional cues play a significant role in subjects’ ability to deceive and detect deception [, ]. Emotion is a fundamental aspect of human communication that interacts with cognition, guiding social behavior in both human-to-human interactions and human-computer interactions [, ]. Emotional characteristics are important, because deceptive behavior can trigger emotional states, leading to behavioral changes that serve as deceptive clues [, ]. However, it is difficult to directly improve the accuracy of deception detection using emotional features []. One of the reasons is that emotional expression is also a part of deception. It is usually difficult to detect whether a deceiver’s emotional expression is genuine or disguised.

To address this issue, we propose a multimodal deception dataset MDPE. It not only collects subjects’ deception information, but also personality information and emotional expression information. Each subject was required to conduct another emotional experiment in addition to engaging in deception, in order to obtain their true emotional expression. Although our research was conducted in the laboratory to provide clear and comparable conversations, we provided subjects with effective monetary incentives to detect and generate effective deceptive behavior []. To our knowledge, this is the largest multimodal deception dataset in the released dataset and the only deception detection dataset with personality and emotional characteristics.

To sum up, our contributions are threefold:

- •
We propose a novel multimodal deception dataset MDPE with personality and emotional characteristics, composed of facial video, and audio recordings and transcript. And an easily replicable experimental protocol has also been provided to researchers.
- •
We provide a benchmark for deception detection from multimodal signals, and discussed the impact of personality traits and emotional cues on deception detection.
- •
We offer new possibilities to facilitate further affective computing research, encourage the development of new methods that utilize individual differences for deception detection, as well as for tasks such as personality recognition and emotion recognition.

## 2 Related Work

Deception Dataset   Pérez-Rosas et al. [] introduced a new multi-modal deception dataset Real-life Trial having real-life videos of courtroom trials. They demonstrated the use of features from different modalities and the importance of each modality in detecting deception. They also evaluated the performance of humans in deception detection and compared it with their machine learning models. The Box-of-Lies dataset [] was released with video and audio from a game show, and presents preliminary findings using linguistic, dialog, and visual features. Multiple modalities have been introduced in the hope of enabling more robust detection. Pérez-Rosas et al. [] introduced a dataset for deception including video and thermal imaging, as well as physiological and audio recordings. Gupta et al. [] proposed Bag-of-Lies, a multimodal dataset with gaze data for detecting deception in casual settings. Speth Jeremy et al.[] proposed a multimodal deception database DDPM contains almost 13 hours of recordings of 70 subjects, as well as physiological signals such as thermal video frames and pulse oximeter data. Most studies on deception detection are designed and evaluated on private datasets, typically with relatively small sample sizes, and MDPE dataset addresses these drawbacks. Table 1 compares the sample size and length for existing datasets and MDPE.

**Table 1: Comparison of the subject count and length for several databases for deception detection**
| dataset | Subjeet Count | Length(Minutes) |
| --- | --- | --- |
| Multimodal | 30 | - |
| Real Trials | 56 | 56 |
| Box-of-Lics | 26 | 144 |
| Bag-of-Lies | 35 | <241 |
| DDPM | 70 | 776 |
| MDPE | 193 | 6209 |

Multimodal Deception Detection   Decades of research in psychology, and deception detection have documented verbal and nonverbal behavioral cues indicative of deceptive communication. Visual cues such as the frequency and duration of eye blinks [, , ], dilation of pupils [, ], and facial muscle movements [, ] have been found to distinguish between deceptive and truthful behavior. Vocal cues can be indicative of deception, with deceptive speakers tending to speak with higher and more varied pitch [, ], shorter utterances, and less fluency [, ] than truthful speakers. Deception also correlates with verbal attributes of speech, with deceivers tending to communicate with less cognitive complexity, fewer self-references, and more words indicative of negative emotions [, ].
Mohamed Abouelenien et al. [] explored a multimodal deception detection approach and integrates multiple physiological, linguistic, and thermal features. They used a decision tree model, to gain insights into the features that are most effective in detecting deceit.
Leena Mathur et al. [] analyzed the discriminative power of features from visual, vocal, and verbal modalities affect for deception detection. They experimented with unimodal Support Vector Machines (SVM) and SVM-based multimodal fusion methods to identify effective features for detecting deception.

Individual Difference Deception   Some studies confirm that some of the five NEO-FFI (Neuroticism-Extraversion-Openness Five-Factor Inventory) dimensions are related to deception [, ]. Sarah Ita Levitan et al. [] reported the role of personality factors derived from the NEO-FFI and of gender, ethnicity and confidence ratings on subjects’ ability to deceive and to detect deception. Justyna Sarzyńska et al. [] reports correlations between the ability to lie and extraversion, as well as conscientiousness. Personality characteristics are a promising set of information for deception detection, and similarly, emotional characteristics are also important. Joseph P. Gaspar et al. [] integrate prior theory and research on emotions, emotional intelligence, and deception and introduce a theoretical model. This model explores the interplay between emotional intelligence (the ability to perceive emotions, use emotions, understand emotions, and regulate emotions; and deception.
Mircea Zloteanu et al hold strong beliefs about the role of emotional cues in detecting deception, and explored how decoders’ emotion recognition ability and senders’ emotions influence veracity judgements [].
Joseph P. Gaspar et al. [] believe that emotions are both an antecedent and a consequence of deception, and they introduce the emotion deception model to represent these relationships. This model broadens their understanding of deception in negotiations and accounts for the important role of emotions in the deception decision process.
To our knowledge, MDPE is the only deception detection dataset with personality and emotional characteristics.

## 3 Dataset

### 3.1 Materials

We collect our deception dataset using: a sports camera Gopro Hero9 with a resolution of 1920x1080 and a frame rate of 60 fps. The voices of the subjects are also recorded by the built-in microphone of the camera. A Thinkpad laptop was provided to subjects for watching emotion induction videos during the emotion experiment. The experimental place is in a professional recording stdio, and during the data collection process, only the subject and interviewer stay in the room. Some materials were prepared by the Data Collection Coordinator (DCC). The experimental setup is shown in Figure 1.

Figure: Figure 1: The example of the place setup for data acquisition.
Refer to caption: extracted/5736428/site.png

### 3.2 Participants

There were 193 subjects in this study, of which 130 were female and 63 were male. They are all native Chinese speakers from different backgrounds. Their age ranged from 18 to 69 years old, and they had various professions including students, workers, teachers, retirees, etc.

Firstly, we segmented the raw video, resulting in 1808 minutes of deceptive video and 4401 minutes of emotional video, totaling 6209 minutes.
Each of the 193 subjects provided 24 responses, 9 of which were deceptive. The length of deception videos ranged from 4 minutes to 27 minutes. Each subject had 16 emotional videos, with lengths ranging from 19 minutes to 38 minutes (including the time spent watching emotional induction videos).

Before the experiment began, the subjects were informed of all experimental procedures. The subjects explicitly consented to record their conversation and publish the video data in a scientific conference or journal. And we do not publish any privacy sensitive data, and the anonymity of participants will be guaranteed. All data were collected under a protocol approved by the authors’institution’s Human Subjects Institutional Review Board.

### 3.3 Procedure

Personality Characteristics Collection: Subjects were required to fill out a Big Five personality questionnaire [], which consists of 60 questions. Each question was marked with a score indicating whether the descriptions match their own, with 1 indicating strong disagreement and 5 indicating strong agreement. Details can be found in the appendix A.

Emotional Experiment: Subjects were asked to watch 16 emotional induction videos, including two induction videos for each emotion of sadness, happiness, relaxation, surprise, fear, disgust, anger, and neutral.
There are a total of 39 induction videos, of which 17 are from the Chinese Emotional Video System (CEVS) []. Each video segment has been labeled and evaluated to ensure that it can induce corresponding emotions. Another 22 are from our online collection. Because the CEVS only include six emotions: sadness, happiness, fear, disgust, anger, and neutral, and some videos of the CEVS are outdated and cannot successfully induce corresponding emotions in our pre-experiments. Each video we collected online was annotated by 12 data annotators based on CEVS selection criteria and evaluation methods, and the results showed that each video triggered strong emotions.

Before the emotional experiment began, the DCC randomly selected 16 induction videos (ensure two videos for each emotion) for the subjects to watch. After watching each video, subjects were required to describe their feelings and then fill out an “Emotional Scale", which quantified 8 emotions. Subjects rated their 8 emotions, ranging from 1 to 5, with 1 indicating no such emotion and 5 indicating the strongest emotion. Details of the emotion scale can be found in the appendix B.

Deception Experiment: The deception data collection process follows DDPM []. The interviewer conducted an interview with the subject, asking a total of 24 questions. These 24 questions were jointly formulated by 5 psychology researchers with over 5 years of experience. Before the interview, details of the emotion scale can be found in the appendix C. The DDC randomly selected 9 questions that must lie and hand them over to the subject (the interviewer does not know which 9 questions). The first 3 questions will not be selected, which means that the first three “warm up” questions were always to be answered honestly. They allowed the subject to get settled, and gave the interviewer an idea of the subject’s demeanor when answering a question honestly.

The subject have a maximum of 15 minutes to prepare, and during the preparation process, they must remember these 9 questions and think about how to deceive in the upcoming interview process. During the interview process, when asked these 9 questions, the subject must lie, and when asked the remaining 15 questions, they must tell the truth. Subjects were motivated to deceive successfully through two levels of bonus compensation: if they were able to deceive the interviewer in five or six of the nine deceptive responses, they were given a 150 percent of a base incentive payment; the base payment was doubled if they were successfully deceptive in seven or more questions. In order to collect more indistinguishable deception answers, we encourage subjects to incorporate some truth into lies when answering these deceptive questions.

During the interview process, the interviewer asked 24 questions in random order, and provide their judgment of truthful or deceptive answers to each question. And the interviewer filled out the "Interviewer Judgment Scale", which record the trust level the interviewer thinks of each answer. The trust level was divided into 1-5 points, where 1 represents definitely true and 5 represents definitely false. After the interview, the subject also filled out the "Subject Lie Confidence Scale" and be asked to rate the answer they just lied to. The same score is 1-5, where 1 represents that I have definitely deceived successfully and 5 represents that I have definitely not deceived successfully.

Each subject was required to complete the above three experiments, so that we can obtain their personality , emotional and deceptive characteristics.

## 4 Benchmark

### 4.1 Data Preprocessing

For the visual modality, we first unify the raw video to 30 fps of the frame rate, and crop and align faces via DLib Toolkit []. Figure 2 shows some examples of real and deceptive faces. Then, we use visual encoders to extract frame-level , followed by average pooling to compress them to the video level. For the audio modality, we use FFmpeg to separate the audio from the video and unify the audio format to 16kHz and mono. For the textual modality, we first extract transcript using Paraformer [], an open-source ASR toolkit. After data preprocessing, for each sample $x_{i}$, we extract acoustic features $f_{i}^{a}\in\mathbb{R}^{d_{a}}$, textual features $f_{i}^{l}\in\mathbb{R}^{d_{l}}$, and visual features $f_{i}^{v}\in\mathbb{R}^{d_{v}}$, where $\left\{d_{m}\right\}_{m\in\left\{a,l,v\right\}}$ is the feature dimension for each modality.

### 4.2 Feature extraction

Different features lead to distinct results. To guide feature selection, we evaluate the performance of different features under the same experimental setup.

Visual Modality: Compared with handcrafted features, deep features extracted from supervised models are useful for facial expression recognition []. CLIP [] is a multimodal model based on contrastive learning, where training utilizes text and images to construct positive and negative sample pairs. Pre-training is conducted on a dataset comprising 400 million pairs, resulting in strong generalization capabilities. And Vision Transformer (ViT) [] is a transformer encoder model , pre-trained in a supervised manner on a large dataset of images. Images are presented to the model in the form of sequences of fixed-size patches (resolution of 16x16) and undergo linear embedding.

Acoustic Modality: We extracted the handcrafted feature extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) [], which contains 88 acoustic parameters designed specially for speech emotional recognition tasks, covering spectral, cepstral, and prosodic features . And Wav2vec [] demonstrates the power of learning robust representations solely from speech audio, fine-tuning on transcribed speech, surpassing the best semi-supervised methods. It masks speech inputs in the latent space and addresses a contrastive task defined on quantized latent representations. It has been widely applied to downstream speech tasks. HUBERT [] utilizes offline clustering steps to provide aligned target labels for prediction losses. A key component of this approach is applying prediction losses only in masked regions, forcing the model to learn representations that combine acoustic and language models on contiguous inputs. To better distinguish between speakers, a sentence mixture training strategy WavLM [] is proposed, allowing for the unsupervised creation and merging of additional overlapping sentences during the training process.

Text Modality: Among all language models, BERT [] and its variants are widely utilized, and it uses the masked language model and next sentence prediction objectives to learn word embeddings. Here, we extract the sbert-chinese-general-v2 features, which is based on the bert-base-chinese version of the BERT model and is trained on the million-level semantic similarity data set SimCLUE. ChatGLM [] is an open-source conversational language model supporting bilingual question answering in both Chinese and English, based on the General Language Model (GLM) architecture. It demonstrats exceptional contextual understanding and more efficient inference capabilities. Baichuan [] is an open-source large-scale model with 13 billion parameters. It features a larger size, more extensive training data, and more efficient inference capabilities.

Figure: Figure 2: Some examples of truth and deception faces.
Refer to caption: extracted/5736428/face.png

### 4.3 Model Structure

For unimodal features, we utilize the fully-connected layers to extract hidden representations and predict deception:

$$ $h_{i}^{m}=\text{ReLU}\left(f_{i}^{m}W_{m}^{h}+b_{m}^{h}\right),{m\in\left\{a,l ,v\right\}}$ (1) $$

$$ $\hat{y}_{i}=\text{softmax}\left(h_{i}^{m}W_{m}^{d}+b_{m}^{d}\right),m\in\left \{a,l,v\right\}$ (2) $$

where $h_{i}^{m}\in\mathbb{R}^{h}$ is the hidden feature for each modality, $d_{i}\in\mathbb{R}^{2}$ is the estimated deception probabilities. For multimodal features, different modalities contribute differently to deception detection. Therefore, we compute importance scores $\alpha_{i}\in\mathbb{R}^{3\times 1}$ for each modality and exploit weighted fusion to obtain multimodal features:

$$ $h_{i}=\text{Concat}\left(h_{i}^{a},h_{i}^{l},h_{i}^{v}\right)$ (3) $$

$$ $\alpha_{i}=\text{softmax}\left(h_{i}^{T}W_{\alpha}+b_{\alpha}\right)$ (4) $$

Similarly, for personality and emotional expression features, we use concatenation for feature fusion. We directly use personality scale scores as personality features. For emotional features, we train an emotion recognition model first, input all emotional expression samples into the emotion recognition model, and take the last fully connected layer features for average pooling as the emotion expression feature.

Table: Table 2: Unimodal results on MDPE. "P" denotes the addition of personality features,"E" denotes the addition of emotional features

### 4.4 Implementation Details

We select the dimension of latent representations from $\left\{64,128,256\right\}$. During training, we use the Adam [] optimizer and choose the learning rate from $\left\{10^{-3},10^{-4}\right\}$. We set the maximum number of epochs to 300 and the weight decay to $10^{-5}$. Dropout [] is also employed, and we select the rate from $\left\{0.2,0.3,0.4,0.5\right\}$ to alleviate the over-fitting problem. We randomly select 5 answers (3 truths and 2 deceptions) from 24 answers in all samples as the validation set, and the remaining 19 answers as the training set. To mitigate randomness, we run each experiment five times and report the average result. And we choose the cross-entropy as loss function and the accuracy and AUC as the evaluation metric.

### 4.5 Experiment Results

Unimodal Results In this section, we establish the unimodal benchmark for MDPE and report results in Table 2. We hope this benchmark can provide guidance for feature selection and point the way to developing powerful feature extractors.
For the visual modalities, VIT achieves better results than CLIP, possibly because VIT is trained on supervised datasets and can reveal more deceptive clues than CLIP features that use text as a supervisory signal.
For the acoustic modality, the deep features outperform handcrafted features. This may be because egemas are features related to emotions, using a emotion-related handcrafted acoustic feature may limit performance. In contrast, deep features can capture more universal acoustic representations for deception detection.
For the textual modality, we focus on textual encoders that support Chinese (large language models are generally trained on multilingual corpora containing Chinese).
And the textual modality can achieve best performance than the visual and acoustic modalities, which indicates that our dataset textual reveals more deceptive clues than other modalities.

Of course, we have also incorporated personality and emotional features into deception detection. Models trained with personality features almost often exhibit superior performance. These findings demonstrate the importance of personality in deception detection tasks. And the addition of emotional features has also improved the performance of deception detection models, but the improvement is not comparable to personality features, even some results have decreased. This may be because personality traits are directly usable features, and emotional features are extracted by emotion recognition models. The quality of features is also influenced by the emotion recognition models. In the future, better methods for using emotional expression features can be explored. By incorporating both personality and emotional features, the deception detection model achieved the highest performance in unimodal results, These results show that personality and emotional expression characteristics are indeed important for deception detection tasks, and using them can truly achieve deception detection modeling based on individual differences. It is worth mentioning that among all the unimodal results, Baichuan always achieved the best results.
It is the model with the largest number of parameters among the features we used, and it further demonstrates the potential of large language models in deception detection.

Multimodal Results In Table 3, we select several well-performing unimodal features and report their fusion results. Experimental results demonstrate that multimodal fusion consistently improves performance. The reason lies in the fact that deception cues can be conveyed through multiple modalities. The integration of multimodal information allows the model to better comprehend the video content and accurately detection deception. Firstly, almost all features have achieved performance improvements in multimodal feature fusion, but the fusion of visual and acoustic modalities has hardly improved or even declined. It indicates that the addition of text features make the performance of the model tend to stabilize. This is consistent with human judgment. When people judge whether others are lying, they tend to express the truth or falsehood of the content, because it difficult to judge from visual or acoustic features. In the future, further exploration and research should be conducted on deceptive clues in visual and acoustic features.

Similar to unimodal results, models trained with personality features typically exhibit excellent performance. The addition of emotional features makes the performance of the model unstable. Some features have been improved, while others have decreased. Finally, the best result always comes from the three modalities fusion, and the three modalities fusion with personality and emotional characteristics has achieved the best performance.

Table: Table 3: Multimodal results on MDPE. We select several well-performing unimodal features and report their fusion results. Here, “V”, “A” and “T” represent the visual, acoustic and textual modalities, respectively.

## 5 Discussion

### 5.1 Future work

Firstly, dimensional emotions are also important. In fact, we are labeling MDPE with dimensional emotions (including deception and emotion), which can not only study the impact of dimensional emotional features on lie detection at the individual level, but also explore more emotional clues in the deception process. Secondly, we extracts deep features from some pre-trained models and uses simple models for deception detection. In the future,larger models should be designed to be used for deception detection tasks. Thirdly, this article simply concatenates personality and emotional expression features to assist in deception detection tasks. In the future, more complex feature fusion or model fusion algorithms can be used for deception detection. Finally, in this paper, we only focuses on deception detection tasks, but MDPE includes individual personality, deception, and emotional expression information, which can not only support deception detection tasks, but also tasks such as personality recognition and emotion recognition. Even these individual level information can be used to assist other tasks, as shown in Figure 3. In fact, there have been many studies on the relationship between personality and emotion [, , ], but the lack of relevant datasets has led to slow research progress in this field, and we believe that MDPE can provide valuable resources for these research directions.

Figure: Figure 3: Possible future work on the MDPE
Refer to caption: extracted/5736428/future.png

### 5.2 Limitations

Firstly, although the subjects were required that they must lie about the deception questions, and verified the deceptive questions and content with the Interviewer after the deception experiment, we do not know whether the subjects have actually deceived on the deception questions. Secondly, although our induction videos have been annotated by professionals to demonstrate their reliability and validity, there are still some subject whose feelings are inconsistent with the emotions we expected to induce. This is because different people have different understandings of the video content, triggering different emotions. Thirdly, relying on self-assessment scales for data annotation is a subjective process for subjects, which may lead to bias in subsequent analysis. Different subjects may have significant differences in their perception of emotions. In addition, MDPE only collects native Chinese speakers, there may be cultural differences in deception detection. Finally, gender imbalance among subjects in MPDE is a common issue in human data collection [, ].

## 6 Conclusion

We present MDPE, a dataset for deception detection featuring three categories of data modalities, as well as personality and emotional characteristics. Firstly, diverse modalities carry complementary information that can be jointly exploited. By providing access to multiple synchronized modalities, MDPE enables cross-modal analyses that have the potential to improve the understanding of the relationships between video, audio, and text.
Secondly, it can help improve the understanding of deceive behavior, aiming to develop reliable deception detection algorithms and enhance the security issues related to deception in our society. Thirdly, MDPE provides the personality traits and emotional expression characteristics of each subject, which can help analyze the impact of personality and emotional expression on deceive behavior. In addition, MDPE not only supports deception detection models, but also provides conditions for personality recognition and emotion recognition tasks, and can even study the relationship between deception, personality, and emotion, such as using personality features to improve the performance of emotion recognition tasks. Finally, to promote reproducibility, MDPE also provided a set of benchmark experiments. Although the proposed model focuses on deception detection, the use of personality and emotional features also demonstrates the predictive potential of our dataset. They represent a good starting point for future work that researchers and developers can use as a benchmark. By openly sharing MDPE, we hope to ignite new advances in this critical area of affective computing.

## Checklist

- 1.
For all authors…
(a)
Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b)
Did you describe the limitations of your work?
[Yes] See 5.2
(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] See 5.2
(d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
- 2.
If you are including theoretical results…
(a)
Did you state the full set of assumptions of all theoretical results?
[Yes]
(b)
Did you include complete proofs of all theoretical results?
[Yes]
- 3.
If you ran experiments (e.g. for benchmarks)…
(a)
Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See 4.4
(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
(d)
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No]
- 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets…
(a)
If your work uses existing assets, did you cite the creators?
[Yes]
(b)
Did you mention the license of the assets?
[Yes]
(c)
Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d)
Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] See 3.2
(e)
Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] See 3.2
- 5.
If you used crowdsourcing or conducted research with human subjects…
(a)
Did you include the full text of instructions given to participants and screenshots, if applicable?
[Yes]
(b)
Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[Yes]
(c)
Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[Yes]

## Appendix A Big Five Personality Inventory Second Edition (BFI-2)

Below are some descriptions of personal characteristics, some may or may not apply to you. Please fill in the corresponding number on the horizontal line before each sentence below to indicate whether you agree or disagree with this description.

- 1.
Outgoing personality, enjoys socializing
- 2.
Soft hearted and compassionate
- 3.
Lack of organization
- 4.
Calm and adept at handling pressure
- 5.
Not very interested in art
- 6.
Strong and confident personality, daring to express one’s own opinions
- 7.
Humble and respectful towards others
- 8.
Relatively lazy
- 9.
Being able to maintain a positive attitude even after experiencing setbacks
- 10.
Interested in many different things
- 11.
I rarely feel excited or particularly want to do anything
- 12.
Often picking on others’ faults
- 13.
Reliable and reliable
- 14.
Irregular mood and frequent emotional fluctuations
- 15.
Skilled in creativity and able to find smart ways to do things
- 16.
Relatively quiet
- 17.
Lack of empathy towards others
- 18.
Work in a planned and organized manner
- 19.
Easy to get nervous
- 20.
Enthusiastic with art, music, or literature
- 21.
Often in a dominant position, like a leader
- 22.
Often having disagreements with others
- 23.
It’s difficult to start taking action to complete a task
- 24.
Feeling secure and satisfied with oneself
- 25.
Disliking discussions with strong knowledge or philosophy
- 26.
Not as energetic as others
- 27.
Be magnanimous and magnanimous
- 28.
Sometimes I lack a sense of responsibility
- 29.
Emotionally stable and less likely to get angry
- 30.
Almost no creativity
- 31.
Sometimes shy and introverted
- 32.
Helpful and selfless towards others
- 33.
Habit keeps things tidy and orderly
- 34.
Often worried and worried about many things
- 35.
Valuing Art and Aesthetics
- 36.
Feeling difficult to influence others
- 37.
Sometimes being rude to people
- 38.
Efficiency, starting and ending with work
- 39.
Often feeling sad
- 40.
Deep thinking
- 41.
Full of energy
- 42.
Do not trust others and doubt their intentions
- 43.
Reliable, always trustworthy to others
- 44.
Able to control one’s emotions
- 45.
Lack of imagination
- 46.
Loud and talkative
- 47.
Sometimes cold and indifferent to others
- 48.
It’s messy and doesn’t like to tidy up
- 49.
Rarely feel anxious or afraid
- 50.
Feeling bored with poetry and drama
- 51.
I prefer to have others take the lead and take responsibility
- 52.
Humility and courtesy towards others
- 53.
Have perseverance and be able to persist in completing tasks
- 54.
Often feeling depressed and unhappy
- 55.
Not very interested in abstract concepts and ideas
- 56.
Full of enthusiasm
- 57.
Think about people in the best possible way
- 58.
Sometimes they may engage in irresponsible behavior
- 59.
Emotions are variable and prone to anger
- 60.
Creative and able to come up with new ideas

## Appendix B Emotional Sacle

After watching the video, you need to rate the following emotions: sadness, relaxation, happiness, surprise, fear, anger, disgust, and neutral. Mark to what extent you feel it appropriately expresses your feelings, with intensity ranging from 1 to 5, where 1 is the least intense and 5 is the strongest.

| Video Number | Sadness | Relax | Happiness | Surprise | Fear | Angry | Disgust | Neutral |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 |  |  |  |  |  |  |  |  |
| 2 |  |  |  |  |  |  |  |  |
| 3 |  |  |  |  |  |  |  |  |
| 4 |  |  |  |  |  |  |  |  |
| 5 |  |  |  |  |  |  |  |  |
| 6 |  |  |  |  |  |  |  |  |
| 7 |  |  |  |  |  |  |  |  |
| 8 |  |  |  |  |  |  |  |  |
| 9 |  |  |  |  |  |  |  |  |
| 10 |  |  |  |  |  |  |  |  |
| 11 |  |  |  |  |  |  |  |  |
| 12 |  |  |  |  |  |  |  |  |
| 13 |  |  |  |  |  |  |  |  |
| 14 |  |  |  |  |  |  |  |  |
| 15 |  |  |  |  |  |  |  |  |
| 16 |  |  |  |  |  |  |  |  |

## Appendix C Interview Questions

- 1.
What color do you like the most? Why?
- 2.
Where is your hometown? Please briefly introduce it.
- 3.
Do you have any hobbies?
- 4.
Have you traveled in the past year?
- 5.
How do you like Beijing?
- 6.
What is your happiest experience?
- 7.
What is your favorite food?
- 8.
What is your personality like?
- 9.
What is your biggest weakness?
- 10.
What is your greatest strength?
- 11.
What do you usually do to relax?
- 12.
Which exercise or sport do you like?
- 13.
Briefly introduce your family members.
- 14.
Who is the person you have the greatest influence on you?
- 15.
Do you have any special places or tourist destinations you want to go to?
- 16.
Who is your favorite celebrity or great person?
- 17.
What is your opinion on the words "neijuan" and "tangping"?
- 18.
What is your favorite literary and artistic work?
- 19.
Have you ever received any rewards or honors in school or at work?
- 20.
What was your most unforgettable experience in the past year?
- 21.
Have you participated in any major event?
- 22.
Have you ever cheated in school or work?
- 23.
Have you concealed a fact to your family or friends in the past year?
- 24.
Have you ever lied to avoid responsibility?

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 3
presentation: 1
questions: What was the reasoning behind using a movie-based emotion induction procedure in the emotion experiment? Why is this procedure relevant in relation to the deception experiment where emotions are induced in a social situation?
soundness: 2
strengths: The new dataset can be valuable to the community if made publicly available.
The authors present extensive evaluations of existing methods/features on the dataset.
summary: The paper presents a new dataset for deception detection and emotion recognition.
The authors evaluate several approaches/features on the task of deception detection, also including emotion information.
weaknesses: The concept of emotion and emotion expression used in the paper is fuzzy.
E.g. what is a "true emotional expression" (line 074) supposed to refer to? One possibility that I might suspect is that the authors refer to whether the emotional expression is aligned with an internal state. This is a highly complex topic as emotions are not displayed directly on e.g. a person's face but are subject to regulation processes and social display rules (e.g. see Schneeberger et al., 2023; Müller et al., 2024).


The method novelty is limited, as the authors only evaluate simple combinations of existing approaches. This is not a very significant issue, as the paper presents itself as a dataset paper, where method novelty is not necessary imo. However, in my understanding it would then be important for an ICLR paper to have a more in-depth analysis of what are the implications of the new dataset for deep learning. We should be able to learn something with it. E.g., what are the remaining challenges, does it tell us something about what the current architectures cannot cover?
On what kinds of examples do the current methods fail?


References are missing for the statement "Most studies on deception detection are designed and evaluated on private datasets, typically with relatively small sample sizes" (line 104).
It would also be helpful to add information on the public availability of the existing datasets in Table 1.

In the description of the emotional experiment (lines 198-207), authors mention that they extended some existing dataset of emotional videos because it only has 6 emotions. In the end there seem to be 8. Which emotions were added and why were they considered to be relevant in the context of deception detection?

Concerning the results - the gains by including personality or emotion into the models are only marginal. Are these gain statistically significant when accounting for randomness (of the ssample/datasets, and of the methods/random number generator intialization)?

Research on deception detection raises a host of ethical issues which are not discussed in the paper.

All in all, the strength of the contribution is not too convincing to me and there are a number of open questions on the motivation/scientific background. Furthermore, the quality of the writing is not good enough at the moment, and the improvements for personality/emotion integration are only marginal.


Misc:
-----

- line 015: "role.Although"
- line 093: "introduced a new multi-modal deception dataset" - the characterisation "new" does not make sense here, as the dataset is already quite old (2015) and of course everything was new when it was introduced.
- line 102: wrongly formatted reference ("Speth Jeremy et al.Speth et al. (2021)"), this occurs several times, e.g. in 126,129 and at many other places.
- line 110: "Subjeet", missing space in "Length(Minutes)"
- line 133: non-grammatical paragraph heading "Individual Difference Deception"
- there are more formal/languageissues throughout the paper. the ones above are just some examples.
- line 376: "egemas"


References:
-----------

Schneeberger, Tanja, et al. "The deep method: Towards computational modeling of the social emotion shame driven by theory, introspection, and social signals." IEEE Transactions on Affective Computing (2023).

Müller, Philipp, et al. "Recognizing Emotion Regulation Strategies from Human Behavior with Large Language Models." arXiv preprint arXiv:2408.04420 (2024).

---

confidence: 5
contribution: 3
initial_rating: 3
presentation: 2
questions: Please see the questions raised under the 'Weaknesses' section.
soundness: 2
strengths: The authors conducted the data collection with great effort and provided comprehensive coverage regarding the data collection and label design.
summary: In this study, the authors collected data by conducting extensive interviews to create a multimodal deception dataset featuring 193 participants. This dataset also includes information about the participants' personalities and emotional states. Furthermore, the authors explored how features from different modalities and labels impact and enhance the deception detection task.
weaknesses: 1. The 'Introduction' section is somewhat verbose. While the authors did an excellent job of providing detailed background information on deception with examples from various modalities, I believe that some of this content, particularly the details in the second and third paragraphs, would be better suited for the 'Related Work' section.

2. The authors mention "effective incentives" without clarifying what these entail (whether monetary or non-monetary) or how they were distributed. Additionally, there is a typo in Table 1.

3. I suggest that the authors include a statement at the beginning of section 3.3 to clarify the order of the data collection activities. 

4. The abbreviation "DDC" at the end of page 4 is unclear—are you referring to "DCC" instead?

5. The role of the 'interviewer' is crucial in this data collection process, as they are responsible not only for asking questions but also for being deceived and labeling responses. However, the manuscript lacks details about the interviewer's background and responsibilities.

6. More specific information is needed regarding how the anonymity of participants is ensured, including what information has been removed or altered. 

7. The statement, "some videos of the CEVS are outdated and cannot successfully induce corresponding emotions in our pre-experiments," is also unclear. What does "outdated" mean in this context, and why exactly are these videos unable to evoke specific emotions?

8. Additionally, what is the source of the 22 online collected videos, and how did the 12 annotators reach a consensus on the labels?

9. In section 4.4 (feature fusion), the second paragraph is poorly explained. After reading it, I'm still unclear about the structure of the "dedicated emotion recognition model" and what specific emotional cues it utilizes. Descriptions such as "We feed a comprehensive set of emotional expression features into this model, allowing it to learn and adapt to the nuances of emotional communication" are too abstract.

10. Similarly, in the results presented in Table 2 and in the discussions regarding personality and emotional features on pages 7, 8, and 9, there is a lack of detail about what these features are and how they contribute.

11. I would also like more information about the 24 interview questions (as detailed in Appendix A.3) that were carefully designed by experienced psychology researchers. Are there any references or theories they used to create these questions?

Overall, the writing can be significantly improved, as there are many uncertainties and unclear descriptions throughout the manuscript. For example: "Before the interview, details of the emotion scale can be found in Appendix C," and "Some studies confirm that some of the five NEO-FFI (Neuroticism Extraversion-Openness Five-Factor Inventory) dimensions are related to deception." 

In conclusion, it is evident that the work feels rushed, and substantial revisions are necessary to address the missing details. As it stands, it cannot be accepted in its current form.

---

confidence: 5
contribution: 1
initial_rating: 1
presentation: 1
questions: The 24 questions used by the interviewer with a given subject were formulated specifically for this study?
soundness: 2
strengths: The topic of deception detection is a challenging and important one.
summary: The authors have introduced a multimodal deception dataset. The main novelty of this dataset is that it includes not only deception features but also individual differences in personality and emotional expression characteristics.
weaknesses: My main concerns with this paper are about the novelty, the content and the presentation style. Regarding the content, the contribution of this paper seems quite limited for a top conference on machine learning (alternative and potentially more adequate conferences could be ICMI. CSCW, ACM-MM, etc.). Regarding the presentation, there are several typos and some sections of methods (Section 4.2 for example) are very badly organized and written in a not adequate manner.

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 2
questions: 1. How do we adapt MDPE to other languages and cultures? Would personality and emotion still be as relevant in deception detection across different demographics?

2. What kinds of multimodal modules can we consider for future research? As presented in the manuscript, the authors considered several previously presented modules, but there can be additional selections for each module.

3. Although the authors consider multi-task approaches for this topic, I think that it is required to compare the results with other emotion recognition or personality-detection models. How do the authors think?
soundness: 2
strengths: S1: The introduced dataset can be one of the good contributions for deception research, providing diverse modalities with consideration of personality and emotion annotations.
S2: The dataset's size is valuable, compared to other datasets. The authors provide the comprehensive dataset of 193 subjects with more than 104 hours. It represents a variety of demographics, covering a broad range of emotions and incorporating deception and genuine responses.
S3. The authors perform extensive benchmarking with multiple features (e.g., visual, acoustic, textual) and combine them with personality and emotional data, leading to valuable findings about multimodal fusion effectiveness.
summary: The manuscript introduces the MDPE dataset, a multimodal resource for contributing to deception detection research. This dataset is organized by more than 104 hours of video, audio, and text recordings from 193 subjects, along with annotations on personality traits and emotional expressions. The authors argued that previous deception detection datasets were limited in scope, often lacking the inclusion of individual characteristics that could enhance deception prediction. The authors also demonstrated the effectiveness of MDPE via a series of experiments, indicating that multimodal and personality-aimed models perform better than unimodal models. The manuscript concludes by emphasizing the potentiality of the dataset for future research with consideration of three key tasks, deception detection, personality recognition, and emotion recognition.
weaknesses: I think that the first weakness of the manuscript is that the collected dataset includes only Chinese speakers, which are limited in its generalization. As the authors know, how do we apply this approach to future research considering other cultures and languages? Obviously, their selected text module (with Chinese-oriented LLM) is effective for their collected dataset. However, it can be applied to other datasets?

Second, although the authors indicate their approaches, I cannot find any demographic statistics in their collected dataset. Is it well-distributed with consideration of age, gender, and a number of characteristics? 

Third, the authors mentioned that emotional characteristics are included in the dataset, but the manuscript lacks a deep analysis of how these characteristics influence deception, instead focusing more on personality traits. This leads to underutilizing the dataset's emotional potential. Moreover, the authors are required to present whether the emotion recognitions are presented with multi-labels or not.

Meta-review: This paper presents a new multimodal deception detection dataset along with personality and emotional characteristics. The dataset is comprehensive and can be useful to the research community, if open sourced. 

However, the paper suffers from major methodological and conceptual issues, as pointed out by the reviewers. Unfortunately, the authors have not addressed these issues adequately and neither have they uploaded a revised version of the manuscript.

Decision: Reject