A Unified Deception Corpus for Cross-Corpus Deception Detection
Year: 2023

Abstract:
Verbal deception has been studied in psychology, forensics, and computational linguistics for a variety of reasons, like understanding behaviour patterns, identifying false testimonies, and detecting deception in online communication. Varying motivations across research fields lead to differences in the domain choices to study and in the conceptualization of deception, making it hard to compare models and build robust deception detection systems for a given language. With this paper, we improve this situation by surveying available English deception datasets which include domains like social media reviews, court testimonials, opinion statements on specific topics, and deceptive dialogues from online strategy games. We consolidate these datasets into a single unified corpus. Based on this resource, we conduct a correlation analysis of linguistic cues of deception across datasets to understand the differences and perform cross-corpus modeling experiments which show that a crossdomain generalization is challenging to achieve. The unified deception corpus (UNIDECOR) can be obtained from https://www.ims. uni-stuttgart.de/data/unidecor.

heading: 1 Introduction
text: Deception detection has remained an area of vested interest in fields like psychology, forensics, law, and computational linguistics for a myriad of reasons like understanding behavioral patterns of lying (Newman et al., 2003; DePaulo and Morris, 2004), identifying fabricated information (Conroy et al., 2015), distinguishing false statements or testimonies (Şen et al., 2022) and detecting deception in online communication (Hancock, 2009). These are relevant tasks because of the truth bias, which is the inherent inclination of humans to actively believe or passively presume that a statement made by another person is true and accurate by default, without the need for evidence to substantiate this belief
(Levine, 2014). While this facilitates efficient communication, it also makes people susceptible to deception, especially in online media where digital deception (Hancock, 2009) manifests in many forms like fake news, misleading advertisements, impersonation and scams. This warrants automatic deception detection systems that can accurately distinguish between truthful and deceptive discourse solely from textual data.
The task of automatic deception detection comes with several challenges. Deception or lying is a complex human behavior and its signals are faint in text. Moreover, it is sensitive to the communication context, interlocutors, and the stake involved (Ten Brinke and Porter, 2012; Salvetti et al., 2016). Most importantly, acquiring annotated data proves to be one of the major hurdles for deception studies. Traditional data annotation methods cannot be employed because human performance is shown to be worse than machines in differentiating truths and lies (Bond Jr. and DePaulo, 2006; Vrij, 2014). One way to collect accurate data is to get the labels at source by the person producing the text. Alternatively, they can be collected using the acquired knowledge that certain types of contents are deceptive. Across the literature, different strategies like crawling fake reviews (Yao et al., 2017), collecting text from users identified as suspicious (Fornaciari et al., 2020), using non-linguistic deception cues (Fornaciari and Poesio, 2014) and soliciting through crowd-sourcing (Ott et al., 2011, 2013; Salvetti et al., 2016) have been employed to obtain reliable annotations.
The diversity in the domains of interest, the medium of deceptive communication (spoken vs. written) and dataset creation procedures make it difficult to compare cues of deception across datasets and to understand their generalizability across different domains. With this study, we aim at mitigating this situation by conducting a comparative survey of publicly available textual deception datasets.
39
We contribute (1) a consolidated corpus in a unified format and (2) conduct experiments in which we evaluate models trained on one data set on all others. Our (3) results show that cross-corpus, particularly cross-domain, generalizability is limited, which motivates future work to develop robust deception detectors. We lay the foundation for such work with (4) additional correlation analyses of the linguistic cues of deception across datasets and verify their generalizability across domains.

heading: 2 Background & Related Work
text: Deception in communication is the act of intentionally causing another person to have a false belief that the deceiver knows or believes to be false (Zuckerman et al., 1981; Mahon, 2007; Hancock, 2009). Lies, exaggerations, omissions, and distortions are all different forms of deception (Turner et al., 1975; Metts, 1989). While the definition of deception varies across literature, they concur that it is intentional or deliberate (Mahon, 2007; Gupta et al., 2013).

heading: 2.1 Domains and Ground Truth
text: Deception research is spread across disciplines which contributed to a variety of domains and consequentially to a number of data collection methods. Domains include opinions statements on a specific topic (Pérez-Rosas and Mihalcea, 2014; Capuozzo et al., 2020; Lloyd et al., 2019), open domain statements (Pérez-Rosas and Mihalcea, 2015), online reviews (Ott et al., 2011, 2013; Fornaciari and Poesio, 2014; Yao et al., 2017), deceptive dialogues in strategic games like Mafiascum1, Box of Lies and Diplomacy (de Ruiter and Kachergis, 2018; Soldner et al., 2019; Peskov et al., 2020; Skalicky et al., 2020) and court trials (Şen et al., 2022).
The ground truth generation strategies differ across datasets. While datasets of opinion statements (Pérez-Rosas and Mihalcea, 2014; Capuozzo et al., 2020; Lloyd et al., 2019), and online reviews (Ott et al., 2011, 2013; Fornaciari and Poesio, 2014; Yao et al., 2017) are collected in written form, interviews include both verbal and non-verbal content (Şen et al., 2022). Game-based corpora contain monologue (Skalicky et al., 2020) or dialogue data (Soldner et al., 2019; Peskov et al., 2020).
All of these resources contain instances that are labeled as truthful or deceptive. Only few studies employ the same procedure to generate both
1https://www.mafiascum.net/
truthful and deceptive content (Salvetti et al., 2016; Skalicky et al., 2020); most resort to separate strategies for collecting them (Ott et al., 2011, 2013; Fornaciari et al., 2020). Instances labeled as deceptive are either solicited content or collected from a source identified as deceptive. Ott et al. (2011, 2013) crawled the truthful reviews from websites of interest and the deceptive ones were crowd-sourced through AMT2, while Salvetti et al. (2016) solicited both via AMT. Yao et al. (2017) tracked fake review generation tasks from crowd-sourcing platforms to identify deceptive reviews and reviewers. For the datasets based on strategic games, the labels are assigned based on game rules. Opinion domain datasets contain stances on topics, like gay marriage and abortion, written by the same person, where the truthful labeled opinions align with the author’s true opinion and deceptive ones align with the opposite (Pérez-Rosas and Mihalcea, 2014; Capuozzo et al., 2020).

heading: 2.2 Automatic Deception Detection Methods
text: Several studies have explored the effectiveness of automatic methods to detect deception from textual data. These include feature-based classification methods with support vector machines (Ott et al., 2011; Pérez-Rosas and Mihalcea, 2014; Fornaciari and Poesio, 2014), logistic regression (de Ruiter and Kachergis, 2018), decision trees (Pérez-Rosas and Mihalcea, 2015), and random forests (Soldner et al., 2019; Pérez-Rosas and Mihalcea, 2015). Some studies also consider contextual information by using recurrent neural networks (Peskov et al., 2020) and transformer-based models (Capuozzo et al., 2020; Peskov et al., 2020; Fornaciari et al., 2021). Transformers are not always better – Peskov et al. (2020) show that BERT is en par with LSTMs while Fornaciari et al. (2021) showed that adding extra attention layers help to improve upon the previous state of the art.
Most works focused on modeling the concept of deception in one domain. An exception is Hernández-Castañeda et al. (2016) who report cross-domain classification results on OPSPAM, DEREV2014, and CROSSCULTDE, but in an allagainst-one setting, not in a one-against-one setup.

heading: 2.3 Linguistic Cues of Deception
text: To understand the phenomenon of deception better, previous studies have analyzed the linguistic
2Amazon’s Mechanical Turk, https://www.mturk.com/
cues that characterize deceptive language in written statements, spoken conversations, and online communication (Newman et al., 2003; Bond and Lee, 2005) and demonstrated that a systematic analysis of these cues can prove valuable in automated deception detection specifically in computermediated communication (Zhou et al., 2004). Newman et al. (2003) noted that the use of fewer selfreferences in deceptive statements indicate that the liars are attempting to distance themselves from the lies. The use of exclusive words (e.g., but, rather) allow deceivers to introduce communicative ambiguity into the discourse. Hancock et al. (2007) noted that these cues are broadly associated with the number of words, use of pronouns, use of emotion words, and presence of markers of cognitive complexity. They also pointed out that these cues can manifest differently based on the type and medium of discourse; real-world vs. online or monologue vs. dialogue.
While these analyses have found application in machine learning models, there are more sets of features that have been used to automatically detect deception. These include n-grams (Fornaciari and Poesio, 2014; Fornaciari et al., 2020; Ott et al., 2011), part-of-speech tags (Lloyd et al., 2019; Fornaciari et al., 2020; Pérez-Rosas and Mihalcea, 2015), lexicon-based features, including the Linguistic Inquiry and Word Count (LIWC, Pennebaker et al., 2015) psychological categories, (Pérez-Rosas and Mihalcea, 2014; Yao et al., 2017) and production rules derived from syntactic context free grammar trees (Yao et al., 2017; Pérez-Rosas and Mihalcea, 2015). Duran et al. (2010), Swol et al. (2012) and Hauch et al. (2015) conducted extensive surveys and analyses of different linguistic cues of deception.

heading: 3 Unified Deception Dataset
text: As preparation for cross-corpus analysis of the concept of deception, we consolidate publicly available textual deception datasets into a unified format.3 We now describe the included datasets. Deceptive Opinion Spam (OPSPAM). Ott et al. (2011) describes deceptive opinion spam as fraudulent reviews written to sound authentic with the goal to deceive the reader. To study the nature of such reviews, they collected truthful reviews
3We refer to our corpus as UNIDECOR: “Unified Deception Corpus”. The scripts to download and convert the dataset can be found in the following repository: https: //www.ims.uni-stuttgart.de/data/unidecor
by crawling online review platforms like TripAdvisor4 and crowd-sourced deceptive reviews via Amazon’s Mechanical Turk (AMT). The initial OPSPAM dataset published by Ott et al. (2011) contains 400 truthful and 400 deceptive reviews with positive sentiments. Ott et al. (2013) extended the dataset to include reviews with negative sentiments. The complete OPSPAM dataset contains 1600 instances labeled for veracity and sentiment. It is available publicly with a Creative Commons Attribution-NonCommercial-ShareAlike license.5 Cross-cultural Deception (CROSSCULTDE). Pérez-Rosas and Mihalcea (2014) collected the CROSSCULTDE dataset to investigate deception in a cross-cultural setting. It consists of short essays on the topics of abortion, death penalty, and feelings about a best friend, collected from the United States, India, and Mexico. We take into account the data collected from the United States and India which are in English and consist of 100 deceptive and 100 truthful essays per topic per geographical region adding up to 1200 labeled instances. The dataset is available for download without mentioning any usage restrictions.6 Deception in Reviews (DEREV2014/2018). To investigate the phenomenon of sock puppetry, Fornaciari and Poesio (2014) collected DEREV2014, containing book reviews from amazon.com that were identified as authentic or fake using predefined linguistic cues. To overcome the shortcoming that these cues cannot be used while developing a deception classifier, Fornaciari et al. (2020) released the DEREV2018 dataset, in which they collect deceptive reviews based on a priori knowledge about authors who solicited fake reviews. Additionally, the authors crowd-sourced both truthful and deceptive reviews for the same books. The DEREV2014 dataset contains 118 reviews each with a truthful label and a deceptive label, while the DEREV2018 dataset includes 1552 reviews each collected from amazon.com and through crowdsourcing with a balanced distribution of truthful and deceptive reviews. The datasets overlap by 62 reviews. Both corpora are available for download.7 Open Domain Deception (OPENDOMAIN). Pérez-Rosas and Mihalcea (2015) study deception, gender, and age detection with an open domain
4https://www.tripadvisor.com/ 5https://myleott.com/op-spam.html 6https://web.eecs.umich.edu/~mihalcea/
downloads.html 7https://fornaciari.netlify.app/
dataset acquired via AMT. Workers were asked to contribute seven true and seven plausible deceptive statements without a restriction of domain, each in a single sentence. The balanced dataset consists of 7168 annotated instances with additional demographic information. The data set is made available without specifying usage restrictions.6
Real-life Trial Data (TRIAL). To study real-life high-stake deception scenarios, Pérez-Rosas et al. (2015) collected videos of trial hearings from publicly available sources like “The Innocence Project” website8. The dataset contains multimodal information with annotations for non-verbal behavior like facial displays and gestures in addition to crowdsourced transcriptions. It contains 60 truthful and 61 deceptive reviews.This corpus is made available without specifying any usage restrictions.6
Boulder Lies and Truth Corpus (BLTC). Salvetti et al. (2016) built a balanced dataset containing reviews elicited via AMT for the domains of electronic appliances and hotels. The crowdworkers were instructed to write fake or real reviews, with positive or negative sentiment, about objects that they were familiar with or not. Unlike other datasets which limited the labeling to truthful vs. deceptive, this dataset distinguished between fake and deceptive reviews, where the former are fabricated opinions about an unknown object while the latter was a false review of a known object. The corpus contains 1492 reviews, out of which 451 are truthful and the rest is labeled as fake or deceptive. It is available through the LDC.9
8http://www.innocenceproject.org/ 9Linguistic Data Consortium, https://catalog.ldc.
Online Deceptive Reviews (ONLINEDE). To address the bottleneck that large realistic data for deception detection do not exist, Yao et al. (2017) created the ONLINEDE corpus containing manipulated reviews posted online. They employed the automatic deception detection framework outlined by Fayazi et al. (2015) to identify deceptive reviewers and reviews from social media manipulation campaigns. It contains more than 100K labeled reviews with ≈10000 deceptive instances, covering more than 30 domains. The dataset is available for research purposes from the authors. Mafiascum Dataset (MAFIASCUM). This dataset published by de Ruiter and Kachergis (2018) contains a collection of more than 700 games of Mafia, an online strategy game played on the Internet forum MAFIASCUM10. Here, players are assigned deceptive or non-deceptive roles randomly, which serve as annotations of the instances. Each of the 9000 documents contain all messages written by a single user in a specific game. The average token count in the instances (4690.69) is therefore considerably higher than in other corpora. The authors have made the dataset publicly available along with the code used for analyses.11 Miami University Deception Detection Database (MU3D). To investigate the role of gender and race in deception studies, Lloyd et al. (2019) created MU3D. It is a collection of interview videos where participants were instructed to talk truthfully or deceptively about their relationship with a person
upenn.edu/LDC2014T24 10https://www.mafiascum.net/ 11https://bitbucket.org/bopjesvla/thesis/src/ master/
whom they liked or disliked. The 80 participants, each belonging to a different gender and ethical background contributed to a positive truth, a negative truth, a positive lie and a negative lie, counting to 160 truthful and 160 deceptive interview content. The transcriptions of these videos along with demographic information, valency, and veracity annotations are made available for research purposes with a Creative Commons Attribution-NonCommercialNoDerivs license.12
Multimodal Deception in Dialogues (BOXOFLIES). To explore deception in conversational dialogue, Soldner et al. (2019) collected the BOXOFLIES dataset which is based on the “Box of Lies” game, a segment on “The Tonight Show Starring Jimmy Fallon” where two celebrity guests take turns describing the contents of a box but are allowed to lie. The opposing player must decide if they believe the description or not. The collected dataset contained 25 videos of the game, transcribed and annotated for non-verbal cues of deception and the veracity of the describer. We exported the statements containing veracity label from the dataset using ELAN13, a tool used to create and modify annotations for audio and video data. The dataset is available for download without specifying any usage restrictions.6
Diplomacy Dataset (DIPLOMACY). To study deception in a conversational context specifically in long-lasting relationships, Peskov et al. (2020) employed the negotiation-based online game DIPLOMACY. The players use deception as a strategy to convince other players to form alliances, for which they use a chat interface. Contrary to other deception datasets, DIPLOMACY contains an additional label for perceived truthfulness of an instance. The intended and perceived truthfulness of each message was annotated by the sender and the receiver respectively. Out of more than 13k messages less than 5% are labeled as intended or perceived lie, resulting in an imbalanced dataset. We use the dataset made available through ConvoKit.14
Deceptive Opinion (DECOP). To study deception in multi-domain and multi-lingual settings, Capuozzo et al. (2020), following the method described by Pérez-Rosas and Mihalcea (2014), col-
12https://sc.lib.miamioh.edu/handle/2374.MIA/ 6067
13https://tla.mpi.nl/tools/tla-tools/elan/ download/
14https://convokit.cornell.edu/documentation/ diplomacy.html
lected truthful and deceptive opinion statements on five different topics, namely abortion, cannabis legalization, euthanasia, gay marriage, and policies on migrants. The experiment was conducted for English and Italian, from which we include the English instances in UNIDECOR. They consist of 2500 opinions statements with balanced labels. This dataset can be obtained from the authors. Bluff the Listener (BLUFF). To study humorous deception with no malicious intent, Skalicky et al. (2020) compiled the BLUFF dataset. It contains data from the “Bluff the Listener” game which is part of the radio show “Wait. . . Don’t Tell Me”. It is a variation of the game “Two Truths and a Lie” in which a panelist tells three stories, two of which are true, and one of which is false. This corpus published by Skalicky et al. (2020) contains 753 humorous stories collected from 251 episodes broadcast from 2010 to 2019. The authors downloaded the transcripts from News-Bank15, a curated repository containing current and archived media. One-third of the stories are truthful while two-thirds are fabricated, counting to 251 truthful and 502 deceptive stories. The dataset is publicly available and can be downloaded via the OSF platform.16
Aggregation. We consolidate the datasets into one unified corpus in which each instance is assigned a binary label indicating if it is truthful or deceptive. We retain annotation dimensions that are available for more than one dataset (age, gender, country, and sentiment). More details on the aggregation process and a sample entry from the corpus are available in Appendix A. Table 1 provides an overview of the corpora, including size, label distribution, token and sentence counts17, along with the domain. The datasets vary greatly in its size, but the distribution of labels is mostly comparable, except for BLTC, ONLINEDE and DIPLOMACY with comparably high counts for truthful instances.

heading: 4 Similarity Analysis
text: The datasets included in UNIDECOR come from a variety of domains and differ markedly in terms of the method of collection. At the same time, datasets from the same domains also have differences (e.g., solicited reviews vs. actual reviews). To understand the differences of datasets better, we explore the similarity values between these datasets
15www.newsbank.com 16https://osf.io/download/mupd9 17Using NLTK’s wordpunct_tokenize and sent_tokenize
using the corpus-similarity measure defined by Li and Dunn (2022), which uses word unigram frequencies and character trigram frequencies of the datasets to calculate the Spearman’s ρ ∈ [−1; 1].18
Figure 1 shows a symmetrical matrix of similarity scores for dataset pairs. The correlation values could in principle be negative, but we do not observe any such values because all corpora are in the same language and have a high degree of term and character frequency overlap.
The heatmap reflects the domains of datasets. For instance, BLTC, OPSPAM, as well as DEREV2014 and DEREV2018 from the review domain have similarity scores of 0.89 and .87, respectively. The opinion statement datasets CROSSCULTDE and DECOP exhibit a high similarity score of 0.83. Similarly, MAFIASCUM and DIPLOMACY show relatively high similarity (0.77), despite differences in the game rules.
Datasets obtained under specific conditions within the same domain are assigned a lower similarity score. For instance, BOXOFLIES, which is a game that takes place in an in-person setting, differs from the online game datasets (.42 with DIPLOMACY and MAFIASCUM). We also observe similarity across domains, e.g., BLUFF is more similar to reviews than games, presumably due to its monologue setting instead of dialogue.
18We use the Python implementation https://github. com/jonathandunn/corpus_similarity

heading: 5 Linguistic Correlation Analysis
text: To understand the generalizability of linguistic cues across different dataset, we conduct a correlation analysis, similar to previous studies that focused on isolated or smaller numbers of corpora (PérezRosas and Mihalcea, 2015; Skalicky et al., 2020)

heading: 5.1 Method
text: We aim at identifying frequently used features which are general across domains. We build our analysis on the “Linguistic Inquiry and Word Count” (LIWC2219, Pennebaker et al., 2015) and Flesch-Kincaid (Kincaid et al., 1975) and Gunning Fog (Robert, 1968) readability scores as measures of complexity or sophistication of language.20
We use point-biserial correlation21 (Glass and Hopkins, 1996) to measure the relation between deception labels (discrete) and a score assigned by LIWC or readability measurement (continuous). The correlation value ranges from −1 to +1.

heading: 5.2 Results
text: Table 2 lists the features which show at least a weak correlation (> 0.15) with p ≤ 0.05 for at least three datasets. The positive and negative correlation values correspond to the strength of association with truth and deception respectively.
19https://www.liwc.app/ 20https://pypi.org/project/readability/. 21https://docs.scipy.org/doc/scipy/reference/
generated/scipy.stats.pointbiserialr.html
Deceptive language is argued to have fewer self-references (“i”) and more references to others (“shehe”, “you”), as liars attempt to distance themselves from their lies (Newman et al., 2003; DePaulo et al., 2003). Our analysis supports this hypothesis in the categories “shehe” and “you” for a substantial number of data sets. Contrary to our expectation, however, in 8 out of 14 datasets the category “i” is seen to correlate with deception and not with truth, with an exception of CROSSCULTDE (ρ = .13) and DEREV2018 (.39).
Studies have attributed less cognitive complexity in language to deceptive communication (Newman et al., 2003; DePaulo et al., 2003). Liars use fewer words related to cognitive concepts (e.g., think, believe), which should correspond to a positive correlation value for the category “Cognition” in LIWC. However, our analysis corroborates this observation only in BLUFF (ρ = .17) and DECOP (ρ = .07).
In general, we found no consistent linguistic cues across domains and datasets in our analysis. This might be because deception is highly sensitive to the goal of a lie and the stakes involved, which is not consistent across the domains under consideration.

heading: 6 Deception Detection Experiments
text: The correlation analysis in the previous section showed that deception cues do barely generalize across domains. This analysis might be limited by the choice of categories, which motivates us to conduct cross-corpus modeling experiments.

heading: 6.1 Experimental Setup
text: In the within-corpus setup, we fine-tune and evaluate RoBERTa models (Liu et al., 2019) on the same dataset via 10-fold cross-validation. In the crosscorpus setting, we train on one corpus and test on the other. To ensure comparability between these experiments, we perform 10-fold cross-validation in both settings: we also evaluate 10 times on the same corpus subsets in the cross-corpus setup. This is not strictly required but ensures comparability.
We use the English RoBERTa-base, with 12 layers, 768 hidden-states, 12 heads and 125M parameters as available in the HuggingFace implementation (Wolf et al., 2020). We finetune with default hyperparameters for 6 epochs using the Auto Model for Sequence Classification. 22
22https://huggingface.co/transformers/v3.0.2/ model_doc/auto.html

heading: 6.2 Results
text: The heatmap in Figure 2 shows the results as F1 measure for the deception label (Appendix B shows results for both labels). The diagonal corresponds to within-corpus experiments. For most datasets, the model shows better performance in the withincorpus setting than in the cross-corpus evaluation. This is not the case for MU3D, TRIAL, and OPENDOMAIN, but the difference is negligible (0.04).
Models on datasets from the same domain or which are otherwise similar (§ 4) show comparably better results in the cross-corpus setting. For instance, training on OPSPAM and testing on BLTC achieves an F1 score of 0.76 on the deception label. Training on BLTC and testing on OPSPAM is however not as good (0.66). Similar observations can be made for DEREV2014 and DEREV2018, and CROSSCULTDE and DECOP.
The heatmap shows the lowest performance for MAFIASCUM and DIPLOMACY, with an F1=0. We assume that this is a result of the imbalanced label distribution in DIPLOMACY and the long documents in MAFIASCUM (see Table 1). Similarly, the exceptionally good results on the BOXOFLIES test set are due to the bias towards the deceptive label (see appendix for F1 score on truth label).
Note that previous work reported other evaluation measures than F1, which makes this dramatically low performance difficult to compare. Our evaluation with accuracy (shown in the appendix
in Figure 4) appears to be more positive with .77 and .95.
From the sub-par results on cross-corpus experiments, we conclude that generalization across domains and dissimilar datasets is challenging, even with pre-trained language models with rich contextual information. In our future work, we plan to use this dataset to train models that can capture domain-independent cues of deception, which can presumably generalize better across datasets.

heading: 7 Conclusion & Future Work
text: Different scientific disciplines have contributed to the creation of deception datasets for textual communication in a variety of domains. In this study, we present a comprehensive survey of deception datasets in English available for research and compile them into a unified deception dataset. We are not aware of any previous work that considered a comparably large amount of corpora and evaluated models between all of them. Some of the evaluation results are encouraging, but particularly between dissimilar domains, the generalization is limited and requires future research.
The RoBERTa-based classification experiments and linguistic correlation analysis of deception cues demonstrate that it is indeed challenging to generalize the concept of deception across datasets, or domains. In the classification experiment results, the wildly diverging F1 scores can be attributed to
the complexity of the task as well as to the limitations of the approach employed. In future work, we plan to explore the reasons for this variability across datasets further.
Additionally, we acknowledge the need to address the issue of biased models, such as the ones trained on MAFIASCUM, ONLINEDE, and DIPLOMACY, which tends to favor truthful labels owing to the label imbalance in these datasets, resulting in an F1 score of 0. To overcome this challenge, we could employ techniques like oversampling to rectify the class imbalance and improve the reliability and effectiveness of our approach.
The goal of our future work is to create robust deception detection models that work reliably across corpora and domains. This includes understanding differences in the concept as it represents itself in these data and understanding differences in linguistic realization.
Our UNIDECOR dataset serves as a valuable resource for future research enabling standardized data comparison, transfer learning, and domain adaptation experiments.

heading: Acknowledgments
text: We thank Kai Sassenberg for fruitful discussions regarding the concept of deception. Roman Klinger’s research is partially funded by the German Research Council (DFG), projects KL 2869/1-2 and KL 2869/5-1.
Limitations
The goal of the current study was to unify the resources available for deception and report observations on cross-corpus and within-corpus analyses. While reporting the baseline performance using RoBERTa, we did not perform any optimization specific to the datasets. Hence, better results might be reported in the papers which handle the datasets or domains in isolation.
Ethical Considerations
The datasets used in this research are publicly available resources from previous studies. We have taken appropriate steps to ensure that we do not violate any license terms or intellectual property rights. Also, proper attribution is given to the original sources of the data. Deception is a sensitive topic, and non-anonymous data should not be used. To the best of our knowledge, all data sets that we
considered have been compiled or collected according to such standards.
The performance of deception detection systems is not perfect, making them unsuitable for examining the utterances of individuals due to the threat of incorrect predictions. Even if automatic systems might reach a close-to-perfect performance, we consider their practical application to analyze and profile people unethical. However, there might be use cases, for instance in forensics, that can be considered ethical from a utilitaristic perspective.
Given the ethical implications of employing automated deception detection systems on individual, non-anonymous statements, we propose utilizing the resources collected and models developed on anonymous data. Any data analysis that could lead back to its origin must only be conducted with the data creator’s informed consent and knowledge of potential consequences.
We consider the research in this paper to be fundamental, with the goal of better understanding human communication.

heading: A Details on the Aggregated Dataset
text: All datasets included in the unified collection contains one binary label indicating whether an instance is truthful or deceptive, the naming convention for which has been normalized retaining the original label for backward compatibility. However, some datasets like Salvetti et al. (2016) and Peskov et al. (2020) include an additional dimension for deception, where the former differentiates between lying about a known object and lying about an unknown object, and the latter contains annotations on the perceived truthfulness of the statement in addition to the actual intention. For providing a unified format, we map both these deceptive instances in Salvetti et al. (2016) to one label and since the perceived truthfulness is an independent annotation, we do not take this label into account.
In addition to truth labels, datasets also contain additional annotations like demographic information related to the author, sentiment, valency of the instance and perceived truthfulness. We retain only those annotation dimensions which are available for more than one dataset which are age, gender, country, and sentiment
The unified dataset includes corpora that are available for research purposes which are downloadable from source, made available directly by the creators, or obtained from a consortium like the Linguistic Data Consortium. We provide a script to automatically download all datasets if they are available for download, which otherwise provides instructions on how to obtain them. Once all datasets are populated in their respective folders, a second script is used to generate the unified dataset in json format. You can find the repository with instructions to obtain the aggregated UNIDECOR, Unified Deception Corpus, at https://www.ims.uni-stuttgart.de/data/unidecor.
The following entry shows an example instance from the corpus.
1 { 2 "source": "OPEN_DOMAIN", 3 "text_ID": "119_f_t_1", 4 "text": "Thad cochran has been in the us senate since before the internet
was 5 invented.", 6 "participant_ID": "NA", 7 "age": "20", 8 "sentiment": "NA", 9 "language": "EN",

heading: 10 "gender": "Female",

heading: 11 "country": "United States",

heading: 12 "original_label": "truth",

heading: 13 "truth_label": "T",

heading: 14 "topic_name": "statement",

heading: 15 "domain": "opinion",

heading: 16 "mode": "written",

heading: 17 "split": null,

heading: 18 "fold": null
text: 19 }

heading: B Additional Experimental Results

heading: Future Work
text: Some of the evaluation results are encouraging, but particularly between dissimilar domains, the generalization is limited and requires future research. The RoBERTa-based classification experiments and linguistic correlation analysis of deception cues demonstrate that it is indeed challenging to generalize the concept of deception across datasets, or domains. In the classification experiment results, the wildly diverging F1 scores can be attributed to
the complexity of the task as well as to the limitations of the approach employed. In future work, we plan to explore the reasons for this variability across datasets further. Additionally, we acknowledge the need to address the issue of biased models, such as the ones trained on MAFIASCUM, ONLINEDE, and DIPLOMACY, which tends to favor truthful labels owing to the label imbalance in these datasets, resulting in an F1 score of 0. To overcome this challenge, we could employ techniques like oversampling to rectify the class imbalance and improve the reliability and effectiveness of our approach. The goal of our future work is to create robust deception detection models that work reliably across corpora and domains. This includes understanding differences in the concept as it represents itself in these data and understanding differences in linguistic realization. Our UNIDECOR dataset serves as a valuable resource for future research enabling standardized data comparison, transfer learning, and domain adaptation experiments