DOI: 10.5281/zenodo.17421589
title: Multimodal Deception Detection Dataset with Personality Profiling across Indonesian Ethnic Groups

description: Multimodal Deception Detection Dataset with Personality Profiling across Indonesian Ethnic Groups

This dataset presents a comprehensive multimodal approach to deception detection, integrating cultural diversity and personality profiling across Indonesian ethnic groups. The dataset comprises 196 participants stratified across six major Indonesian ethnicities (Javanese, Sundanese, Madurese, Bugis-Makassar, Batak, and Minangkabau) and four DISC personality types (Dominance, Influence, Steadiness, Conscientiousness).

Key Features



Multimodal Data: Text (Indonesian & English transcripts with linguistic features), Audio (94 MFCC and spectral features + 17 pause features), and Visual (468 facial + 10 iris + 33 pose landmarks per frame)

Cultural Diversity: Six Indonesian ethnic groups with documented metadata enabling cross-cultural behavioral analysis

Personality Profiling: DISC personality assessment for individual difference analysis

Controlled Design: Testimonial narratives about witnessed theft incidents with verified ground truth labels

High Quality: Advanced preprocessing with automated quality validation (SNR: 79.8±3.1 dB, landmark detection: 90.1-100%)


Data Collection



196 participants distributed across six ethnic groups:



Javanese (n=30)

Sundanese (n=25)

Madurese (n=43)

Bugis-Makassar (n=36)

Batak (n=31)

Minangkabau (n=31)



1,568 video recordings (8 per participant: 4 truthful, 4 deceptive)

Personality distribution: Dominance (n=28), Influence (n=32), Steadiness (n=71), Conscientiousness (n=65)

Multi-regional collection across 6 Indonesian cities (Makassar, Medan, Padang, Yogyakarta, Bangkalan/Sampang, Bandung)

Standardized CCTV footage stimuli from Indonesian news broadcasts

Professional recording equipment (iPhone 13: 4K at 60fps, TNW wireless lavalier microphone: 16kHz)


Applications



Multimodal deception detection algorithm development

Cross-cultural behavioral pattern analysis

Personality-aware detection systems

Forensic and security applications

Cultural psychology research

Fairness and bias mitigation studies


Data Format

Released Files (Total: ~18.5 GB):



Preprocessed audio: 1,568 WAV files (16 kHz mono, 16-bit PCM) - 324.2 MB compressed

Text datasets: Indonesian transcripts (16 columns) and English translations (11 columns) - 312.7 KB

Audio features: 104 acoustic features (MFCC, spectral, pause patterns) - 3.3 MB

Visual landmarks: 647,871 frames with 1,536 landmark coordinates - 18.0 GB

Metadata: Feature descriptions and documentation - 8.4 KB


Note: Raw videos are NOT publicly released to protect participant privacy. Only derived features and preprocessed audio are available.

Technical Validation



Audio quality: SNR 79.8±3.1 dB (threshold: >20 dB)

Landmark detection rates: Face 90.1%, Iris 90.1%, Pose 100.0%

Transcription reliability: Cohen's κ=0.82 [0.78, 0.86] (10% sample, two independent raters)

Class balance: Perfect 50/50 split (784 truthful, 784 deceptive)

Missing values: <0.05% across all modalities

Baseline performance: 58-69% accuracy across modalities (demonstrating data usability)


Ethics



Ethical approval: University of Surabaya Institutional Ethics Committee (Protocol No. 756/KE/XII/2025)

Informed consent: All participants provided explicit consent for public release of derived features and preprocessed audio

Data anonymization: Alphanumeric codes, no personally identifiable information

Privacy protection: Raw videos retained securely, only derived features released publicly


Citation

If you use this dataset, please cite:

Rahayu, Y. D., Fatichah, C., Yuniarti, A., Probowati, Y., & Kusnanti, E. A. (2025). Multimodal Deception Detection Dataset with Personality Profiling across Indonesian Ethnic Groups [Data set]. Zenodo. https://doi.org/10.5281/zenodo.17421590

Funding

This research was supported by the Research and Innovation Incentive Program (RIIM) from Indonesia Endowment Fund for Education (LPDP) and Indonesian National Research and Innovation Agency (BRIN), grant numbers 16/IV/KS/02/2025 and 821/PKS/ITS/2025.