MentalManip: A Dataset For Fine-grained Analysis of Mental Manipulation in Conversations (2024)

Abstract: Mental manipulation, a significant form of abuse in interpersonal conversations, presents a challenge to identify due to its contextdependent and often subtle nature. The detection of manipulative language is essential for protecting potential victims, yet the field of Natural Language Processing (NLP) currently faces a scarcity of resources and research on this topic. Our study addresses this gap by introducing a new dataset, named MENTALMANIP, which consists of 4, 000 annotated fictional dialogues. This dataset enables a comprehensive analysis of mental manipulation, pinpointing both the techniques utilized for manipulation and the vulnerabilities targeted in victims. Our research further explores the effectiveness of leading-edge models in recognizing manipulative dialogue and its components through a series of experiments with various configurations. The results demonstrate that these models inadequately identify and categorize manipulative content. Attempts to improve their performance by fine-tuning with existing datasets on mental health and toxicity have not overcome these limitations. We anticipate that MENTALMANIP will stimulate further research, leading to progress in both understanding and mitigating the impact of mental manipulation in conversations.

heading: 1 Introduction
text: Mental manipulation is a deceptive strategy aimed at controlling or altering someone’s thoughts and feelings to serve personal objectives (Barnhill, 2014). Facilitated by digital technologies, mental manipulation has gained unprecedented capability to target and influence individuals via interpersonal interactions and public dissemination of information (Ienca, 2023), causing significant mental health distress to victims (Hamel et al., 2023). Compared to overt verbal toxicity and abuse, such
as hate speech, manipulation is more insidious, deceitful, and context-dependent, posing challenges for individuals and automatic moderation tools to discern. For years, the NLP community has witnessed advancements in verbal toxicity and abuse detection, but most of those works focus on contextfree content and face challenges in identifying implicit toxicity (Wiegand et al., 2021; Mishra et al., 2020; Yin and Zubiaga, 2022; Deng et al., 2023). Existing works in dialogue systems have targeted context-aware toxicity, but are limited to direct toxicity, such as profanity, condescension and forms of hate speech (Baheti et al., 2021; Gao and Huang, 2017; Wang and Potts, 2019). We argue that existing toxicity detection resources are insufficient for developing automatic systems to detect and properly handle verbal mental manipulation. Additionally, current state-of-the-art Large Language Models (LLMs) are not well-positioned to address this issue, as demonstrated in Figure 1.
This paper introduces MENTALMANIP, a dataset with multi-level annotations for mental manipulation detection and classification. We define mental manipulation as using language to influence, alter, or control an individual’s psychological state or perception for the manipulator’s benefit.
3747
This definition aligns with one of the explanations of manipulative influence in Barnhill’s work on the philosophy of online manipulation (Barnhill, 2022). MENTALMANIP dataset contains 4, 000 multi-turn fictional dialogues between two characters extracted from online movie scripts. To enable fine-grained analysis, we proposed a labeling taxonomy covering three dimensions: presence of manipulation, manipulation technique, and targeted vulnerability, which are illustrated in Figure 2. Our taxonomy aids in the precise detection of mental manipulation and facilitates a nuanced classification of the techniques used by manipulators, as well as the vulnerabilities targeted in victims.
We conducted three classification tasks on MENTALMANIP to detect the existence of mental manipulation and its elements. Our experiments spanned state-of-the-art generative and discriminative LLMs across multiple settings. To investigate the effectiveness of existing datasets in toxic speech and mental health for our objectives, we fine-tuned LLMs with seven relevant datasets and conducted evaluations. Experimental results reveal that these LLMs are limited in understanding mental manipulation, as shown by a significant number of misclassified “manipulative” dialogues. Moreover, fine-tuning LLMs with these relevant datasets did not enhance their detection and classification capabilities on manipulative language. These findings highlight the importance of our dataset, and suggest an avenue for future studies in mental manipulation detection and analysis.
Our MENTALMANIP dataset, along with the code for statistical analysis and experiments in this paper, is available in our GitHub repository: https: //github.com/audreycs/MentalManip.

heading: 2 Related Works

heading: 2.1 Mental Health Detection
text: Leveraging NLP technologies for the early detection and intervention of mental health issues stands as a valuable endeavor. In recent decades, there has been considerable research identifying specific mental health concerns, such as stress (Guntuku et al., 2019; Nijhawan et al., 2022), depression (Eichstaedt et al., 2018; Xu et al., 2019), and suicide (De Choudhury et al., 2016; Coppersmith et al., 2018). Scalable and accessible data resources for these issues have been proposed. (Turcan and McKeown, 2019; Haque et al., 2021; Naseem et al., 2022). More recently, research has shown that LLMs, like GPT-4, exhibit promising yet limited performance on tasks related to mental health (Yang et al., 2023; Xu et al., 2023). Researchers have pointed out a lack of explainability for the detection results of LLMs, and highlighted the importance of domain-specific fine-tuning on LLMs for better performance. These findings underscore the need for data resources featuring nuanced annotations and targeting unaddressed mental health issues.

heading: 2.2 Toxic Speech Detection
text: In NLP, “toxic speech” is an umbrella term referring to language that is rude, disrespectful, or offensive, potentially harming conversation quality and negatively impacting recipients (Dixon et al., 2018). Lots of benchmark datasets have been developed to detect explicit and implicit toxic speech in online posts and comments, including those focusing on racism and sexism (Waseem, 2016; Basile et al., 2019; Hartvigsen et al., 2022; Yavnyi et al., 2023), online harassment (Hosseinmardi et al., 2015; Rosa
et al., 2018), and trolling (Miao et al., 2020). Recent works have also investigated performance of state-of-the-art LLMs on toxic speech (Wang et al., 2023). Although many mental manipulations, such as intimidation, fall under toxic speech, their subtle and complex nature create challenges beyond the capability of context-free toxicity detection methods. Existing works in dialogue systems address context-aware toxicity detection (Wang and Potts, 2019; Baheti et al., 2021; Zhang et al., 2021), but they focus on explicit toxicity and overlook implicit verbal manipulation.
Table 1 summarizes some existing datasets addressing toxicity or mental health problems.

heading: 3 Constructing MENTALMANIP

heading: 3.1 Taxonomy
text: Establishing a structured labeling taxonomy when developing a dataset is crucial. Drawing inspiration from Simon’s research on psychological manipulation (Simon and Foley, 2011), we crafted a multilevel taxonomy encompassing three dimensions:
• Presence of Manipulation: This level employs binary classification, indicating if a dialogue contains elements of mental manipulation.
• Manipulation Technique: This level identifies specific manipulation techniques used in conversation.
• Targeted Vulnerability: The last level indicates particular victim vulnerabilities exploited by the manipulator.
We present the detailed taxonomy in Figure 2, which contains 11 different techniques and 5 vulnerabilities. We provide the definition of each technique and vulnerability in Appendix A. To ensure clarity and comprehensiveness, we incorporated insights from a psychological expert and feedback from annotators.

heading: 3.2 Data Source and Preprocessing
text: We prioritize dialogues as our primary data format as they maintain original context, unlike standalone comments and posts. To guarantee a semantically rich and stylistically diverse dataset, we prioritize human-crafted content over LLM-generated material. We finally chose Cornell Movie Dialogs Corpus1 (Danescu-Niculescu-Mizil and Lee, 2011) as the data source to construct MENTALMANIP. The
1https://www.cs.cornell.edu/~cristian/Cornell_ Movie-Dialogs_Corpus.html
Cornell Movie Dialogs Corpus contains 220, 579 conversational exchanges extracted from 617 raw movie scripts spanning a wide range of genres. The overwhelming majority of dialogues occur between two characters, which we standardized for. We replaced original speakers’ names with “Person1” and “Person2” to eliminate potential biases.
Since manipulative language is relatively sparse in conversation, we need to filter the original data to get dialogues potentially containing elements of manipulation. We utilized two approaches to achieve this: 1) key phrase-based matching, and 2) BERT classification.
For key phrase-based matching, we sourced key phrases from online resources, selecting those that frequently occur in manipulative conversations, without restricting their n-gram size. After collection, we manually conducted tense conversion (converting all phrases to present tense), phrase simplification (e.g., “It’s fine, nobody cares about me anyway” to “nobody cares about me”), and merging of similar phrases. Ultimately, we obtained a list of 175 cleaned key phrases.
Appendix B presents examples of the cleaned key phrases and details of the online resources we used. The full list of cleaned key phrases is available in our GitHub repository. To screen out candidate dialogues where key phrases are present, we implemented a length-adaptive matching criterion due to the lexical diversity of language. A dialogue is considered a match if any sentence contains at least P% tokens from a key phrase. The value of P is detailed in Table 2.
For BERT classification approach, we fine-tuned a pre-trained BERT model with a sequence classification head on top. Our goal was to get a classifier to differentiate manipulative dialogues from general toxic content. To prepare the training data,
we inquired GPT-4 Turbo (Bubeck et al., 2023) by zero-shot prompting2 on whole Cornell Movie Dialogs Corpus and obtained a set of “manipulative” dialogues flagged by GPT-4. We observed that GPT-4 generated a large portion of false positives for manipulative content. We examined 1, 378 “manipulative” dialogues flagged by GPT-4, and labeled only 464 dialogues as truly manipulative, with the remaining 914 being false positives. These 1, 378 labeled dialogues were then used to train the BERT classifier. Finally, we employed BERT classifier on all identified “manipulative” dialogues to obtain highly likely manipulative dialogues.
We initially identified 1, 406 dialogues through key phrase-based matching and 3, 739 dialogues using BERT classification, totaling 5, 145 dialogues. Following this, we eliminated duplicates and lowquality dialogues, including those that were extremely short or had broken contexts. Some dialogues were also rephrased to improve readability. After these adjustments, the total number of dialogues prepared for annotation was 4, 876.

heading: 3.3 Human Annotation
text: We established our annotation platform using Label Studio3. Each dialogue represents an annotation task. We recruited 17 college students, all native or fluent English speakers, to serve as annotators. The group of annotators reflects a diverse range of characteristics including gender (14 females, 3 males), ethnicity (11 Asians, 5 Whites, 1 Latino), educational backgrounds (majors such as English, Computer Science, and Physics), and cultural backgrounds (including both US-born and non-US-born individuals). During recruitment, applicants with an educational background in psychology or linguistics were preferred. We conducted tutorial sessions, required annotators to carefully read instructions, and monitored their annotation activities. Screenshots of the annotation platform and instructions are provided in Appendix I. To ensure annotation quality, we assigned three annotators to each task. During the task assignment process, we ensured that the same pairs of annotators were not assigned to evaluate the same dialogues. This
2API calling format is presented in Appendix C. 3https://labelstud.io/
approach helped to reduce the potential for bias when assessing inter-annotator agreement.
In each task, annotators are presented a dialogue, then prompted to answer four questions:
• Q1 (binary choice): Does this dialogue contain elements of mental manipulation? (Options are “Yes” or “No”.) • Q2 (multiple choice): What techniques are used by the manipulator? (Options are techniques in Figure 2.) • Q3 (binary choice): Are there any victims resulting from manipulation in this dialogue? (Options are “Yes” or “No”.) • Q4 (multiple choice): Which vulnerabilities are targeted in the victim? (Options are vulnerabilities in Figure 2.) Q2 and Q3 are conditional upon Q1, and Q4 is conditional upon Q3. Annotators could choose at most three techniques and at most two vulnerabilities. To accommodate indecision, we included a “cannot decide” option in Q2 and Q4. Annotators were required to rate their confidence on a scale from 1 (not confident) to 5 (very confident). Furthermore, annotators could highlight sections they identified as manipulative to aid in our review. Appendix H provides an annotation example.
In total, we obtained more than 13K annotations. After quality review, the final size of well-labeled dialogues is 4, 000. Appendix D provides a detailed statistics of annotation quality, including the heat map of agreement between any two annotators, inter-gender agreement, scatter plot of agreement and confidence, and density distributions of agreement and confidence among annotators. We also calculated the inter-annotator agreement using Fleiss’ Kappa (Fleiss and Cohen, 1973) based on their answers on Q1. The score was 0.596, indicating a moderate annotator agreement. This agreement level is as per our expectation, as the judgment of manipulation is very subjective. We name this dataset MENTALMANIP, and provide samples of it as a supplementary file.

heading: 3.4 Final Label Generation
text: To prepare the dataset for experiment, final labels need to be created. As each dialogue is annotated by three annotators, we adopted two strategies for generating the final labels:
• Consensus agreement: This strategy only selects dialogues with the same annotation results from all three annotators. The accordant result becomes the final label.
• Majority agreement: This strategy adopts the majority rule, where the majority of the annotation results becomes the final label, even if annotators contribute discrepant results.
Using these strategies on annotation results of question Q1, we obtained two versions of MENTALMANIP datasets. We denote the dataset generated using consensus agreement as MENTALMANIPcon and the one using majority agreement as MENTALMANIPmaj.
We employed a specific strategy on both MENTALMANIPcon and MENTALMANIPmaj to generate the final labels for manipulative techniques and targeted vulnerabilities. If a technique
or vulnerability is annotated by at least two annotators in one task, the technique or vulnerability will be added as the answer. This resulted in some dialogues lacking technique and vulnerability labels.

heading: 3.5 Dataset Statistics
text: In this section, we delve into the statistics of our datasets, MENTALMANIPcon and MENTALMANIPmaj, as depicted in Table 3 and illustrated through Figures 3, 4, and 5. Our analysis utilizes a multi-class sentiment classification model, specifically the Distilbert-base-uncasedemotion model from Hugging Face, to determine the dominant emotion within each dialogue.
The analysis, presented in the left two panels of Figure 3, indicates a strong alignment in the distribution of manipulation techniques and vulnerabilities between MENTALMANIPcon and MENTALMANIPmaj. Additionally, the same figure’s right panel reveals that both manipulative and non-manipulative dialogues within MENTALMANIPcon exhibit similar emotional distributions, with “joy” and “anger” being the two most common emotions. Figure 4 offers a heat
map that elucidates the correlation between manipulation techniques and vulnerabilities, uncovering prevalent patterns like the association of accusations with shaming or belittling. Moreover, Figure 5’s left panel showcases a t-SNE visualization of Sentence Transformer embeddings for both manipulative and non-manipulative dialogues within MENTALMANIPcon, using the all-MiniLM-L12-v2 model from Hugging Face. This visualization underscores the difficulty of distinguishing between manipulative and non-manipulative dialogues due to their intertwined embeddings.
Furthermore, we compare MENTALMANIP with other dialogical datasets listed in Table 1, noting that MENTALMANIP encompasses a greater volume of conversational exchanges, suggesting a richer dialogue context. The Complementary Cumulative Distribution Function (CCDF) for utterance and token counts of MENTALMANIP compared to other dialogical datasets is depicted in Figure 6. The right panel of Figure 5 visualizes the distribution of these datasets in the embedding space, illustrating significant overlap among them, except for the distinct clustering pattern of Fox News comments.
In summary, our analysis highlights the challenge of differentiating between manipulative and non-manipulative dialogues, indicating that reliance on emotion classification or conventional text embeddings alone is insufficient for this purpose. Moreover, our dataset’s comparison with other datasets confirms its comprehensive distribution and diversity, aligning with the variety observed in related datasets.

heading: 4 Experiments

heading: 4.1 Experiment Setting
text: We conducted experiments of three classification tasks on MENTALMANIPcon and MENTALMANIPmaj to assess performance of stateof-art models in detecting mental manipulation: Manipulation Detection (Section 4.2), Technique Classification (Section 4.3), and Vulnerability Classification (Section 4.3). We analyzed the performance of GPT-4 Turbo (Bubeck et al., 2023), Llama-2-7B, Llama-2-13B4 (Touvron et al., 2023), and RoBERTa-base (Liu et al., 2019) across three experimental settings: zero-shot
4Both Llama-2-7B and Llama-2-13B are Chat versions.
prompting, few-shot prompting, and fine-tuning. For zero-shot prompting, we presented a dialogue to LLMs to assess if it contained elements of mental manipulation. In few-shot prompting, aside from instructions, we randomly provided one non-manipulative and two manipulative dialogues with true answers as examples. In fine-tuning, Llama-2-13B and RoBERTa-base were fine-tuned on specific datasets, with Llama-2-13B undergoing instruction tuning and RoBERTa-base receiving traditional supervised fine-tuning. Formats for zero-shot and few-shot prompting are detailed in Appendix C. For Llama’s training on different datasets, instructions were adapted to fit respective tasks. GPT-4 Turbo’s implementation followed OpenAI’s official cookbook5. Talkdown dataset was ignored due to its lengthy dialogues which far surpass the input token limit of RoBERTa-base.
For experiment data, we randomly split MENTALMANIPcon and MENTALMANIPmaj into training, validation, and test sets with a ratio of 6:2:2. We ensured proportional representation of manipulative and non-manipulative dialogues, and consistent inclusion of each technique and vulnerability across all sets. All experiments were performed on three Quadro RTX 6000 GPUs. We set the temperatures of GPT-4 Turbo and LLaMA-2 to 0.1 and 0.6, respectively. At these levels, the models exhibit more consistent and less random responses.
We seek to elucidate the following aspects: • The effectiveness of LLMs in identifying and
categorizing mental manipulation based on their inherent knowledge.
• The performance of LLMs when prompted with examples.
• The performance of LLMs post fine-tuning on relevant datasets.

heading: 4.2 Manipulation Detection
text: This task is framed as a binary classification task. In our interactions with ChatGPT and GPT4, we found it tends to mistakenly classify nonmanipulative dialogues as manipulative if they feature general toxicity, like profanity, without actual manipulative intent. Thus, we were keen to investigate the over-reactivity of LLMs when identifying mental manipulation.
Hypersensitivity of LLMs: We examined GPT4 Turbo, Llama-2-7B, and Llama-2-13B on
5https://github.com/openai/openai-cookbook/
the manipulation detection task using 899 nonmanipulative dialogues in MENTALMANIPcon. Prediction results are detailed in Table 4. GPT-4 Turbo incorrectly identified 312 dialogues as manipulative. Both Llama-2-7B and Llama-2-13B exhibited poor accuracy, mis-classifying almost all nonmanipulative dialogues, with Llama-2-13B showing slightly better performance. These results indicate Llama-2’s limited capability in accurately discerning mental manipulation.
Then, we conducted manipulation detection on the entirety of MENTALMANIPcon and MENTALMANIPmaj. Note that the distribution of manipulative and non-manipulative dialogues in both datasets is imbalanced, with manipulative dialogues being more prevalent, as detailed in Table 3. We evaluated the models based on binary Precision, binary Recall, Accuracy, micro F1, and macro F1. Because of binary classification, the accuracy has the same score as micro F1.
Experiment results are presented in Table 5 and Table 6. It is observed that MENTALMANIPmaj poses a greater challenge for prediction, as we expected. In zero-shot and few-shot prompting, Llama-2-13B classifies nearly all dialogues as manipulative, causing high recall rates. Few-shot prompting improves Accuracy and F1 scores for both GPT-4 Turbo and Llama-2-13B. For GPT4 Turbo, few-shot prompting increases its Recall, making it more likely to identify dialogues as manipulative. For Llama-2-13B, few-shot prompting makes it less sensitive and produces fewer manipulative predictions. Appendix F provides the confusion matrices for prediction results of GPT-4 Turbo and Llama-2-13B under zero-shot and fewshot prompting on MENTALMANIPcon. For finetuning, Llama-2-13B on Dreaddit gives the best performance among all finetuning results on existing datasets. Note that Dreaddit is about detecting Mental Stress. However, fine-tuning Llama-2-13B on all existing relevant datasets does not notably enhance performance beyond zero-shot or few-shot prompting outcomes. RoBERTa-base overall exhibits inferior Accuracy compared to Llama-2-13B.
1 stands for micro precision, micro recall, accuracy, micro F1 and macro F1, respectively.
Specifically, fine-tuning it on Fox News dataset results in badly degraded performance. This decline may stem from the broader semantic distribution of the Fox News dataset, as illustrated in Figure 5.

heading: 4.3 Technique and Vulnerability Classification
text: Here we examined these models on multi-label classification tasks to identify manipulative techniques and victim vulnerabilities. We present the experiment results on MENTALMANIPcon in Table 7, where we report micro Precision and micro Recall. For few-shot prompting on both classification tasks, we provided 2 randomly chosen examples. We can observe that GPT-4 Turbo performs better than Llama-2-13B under zero-shot and few-shot prompting, and few-shot prompting increases their accuracy. Fine-tuning Llama-213B on MENTALMANIPcon still gives better performance than fine-tuning RoBERTa-base. Precision and recall scores for each technique and vulnerability category are detailed in Table 8, specifically for zero-shot prompting using GPT-4 Turbo and
Llama-2-13B.

heading: 4.4 Analysis on Incorrect Predictions
text: In this section, we explore whether there are significant semantic differences between dialogue instances where models correctly predicted the label and those where they failed. We extracted dialogues from MENTALMANIPcon’s test set that were correctly and incorrectly predicted by GPT-4 Turbo. Under the zero-shot setting, GPT-4 Turbo accurately predicted 383 dialogues and inaccurately predicted 200. In the few-shot setting, the numbers were 422 correct predictions and 161 incorrect. We used t-SNE visualization of Sentence Transformer embeddings to analyze the semantic distributions of these dialogues, which are presented in Figure 7. The visualizations show that the dialogues, whether correctly or incorrectly predicted, are semantically indistinguishable, underscoring the difficulty of differentiating manipulative language based solely on lexical or semantic features.

heading: 5 Conclusion and Future Studies
text: This study introduces MENTALMANIP, a pioneering dataset aimed at identifying and classifying mental manipulation in a fine-grained level. We assessed GPT-4 Turbo, Llama-2-13B, and RoBERTabase across three classification tasks under various settings. Experiment results reveal that models’ understanding of mental manipulation do not align well with human perspectives. LLMs tend to incorrectly identify general toxicity as manipulation, a challenge particularly pronounced in smaller LLMs such as Llama-2-7B and Llama-2-13B. In future work, it would be worthwhile to expand the dataset sources beyond the Cornell Movie Dialog Corpus and incorporate real-case interpersonal interac-
tion data. We also recognize that the performance of LLMs on more complex prompting paradigms, such as chain-of-thought, can be investigated.
Detecting and finely classifying mental manipulation in conversations is a challenging task due to the implicit and context-dependent nature of the language often used. Furthermore, the subjectivity of human judgment complicates the alignment of AI models’ predictions with human choices. We have made the MENTALMANIP dataset publicly available for future studies and hope it will inspire and foster further research in various NLP tasks and applications.

heading: 6 Limitations
text: We recognize that MENTALMANIP dataset has several limitations:
Language and Format The MENTALMANIP is limited to English-language content and focuses
exclusively on dialogues between two individuals. Real-world interactions, however, are frequently more multifaceted. Consequently, this restriction may limit the dataset’s applicability to more complex scenarios. Data Source The MENTALMANIP is derived from online movie scripts, which means the speech style presented may not accurately reflect natural, reallife communication. Data Annotation The process of annotation is inherently subjective, which can introduce uncertainties in the precision of labeling. Additionally, the selection of annotators could lead to significant biases. For instance, despite the lack of notable differences in inter-annotator agreement across genders, an imbalanced gender demographic among our annotators could still influence the results. We recognize that despite our best efforts, assembling an annotator pool that perfectly mirrors the general population remains a challenging endeavor.

heading: 7 Ethics and Broad Impact
text: Before annotating, we noted that many dialogues, especially from R-rated movies, contained profanity that might upset annotators. To protect their well-being, we rephrased these instances into milder language while keeping the original context. When recruiting annotators, we emphasized ensuring a diverse team in terms of race and gender. Throughout the annotation phase, we actively encouraged annotator feedback, as summarized in Appendix G.
The MENTALMANIP dataset contains a range of uncensored sensitive materials, including hate speech, violence, threats, mental health issues, sexual content, profanity, and more. While our dataset is primarily designed for detection and classification tasks, we recognize the potential for misuse, particularly in the training of malicious generative AI systems. For example, there is a risk that the data could be used to create automated chatbot systems that employ manipulative language for unethical purposes like scams. It is crucial to address these risks to ensure responsible use.

heading: Acknowledgments
text: We thank the Label Studio Platform for providing complimentary access to the Enterprise Cloud Platform through their Academic Program. We also thank the annotators for their time and effort. This research was supported in part by grants from the
US National Library of Medicine (R01LM012837 & R01LM013833) and the US National Cancer Institute (R01CA249758).

heading: A Description of Taxonomy
text: Definitions of 11 manipulation techniques, with their abbreviations inside parentheses:
1. Denial (DEN): The manipulator either denies wrongdoing or pretends to be confused about others’ concerns. 2. Evasion (EVA): The manipulator refuses to pay attention to something or gives irrelevant or vague responses. 3. Feigning innocence (FEI): The manipulator implies that any harm caused was accidental. 4. Rationalization (RAT): The manipulator rationalizes their inappropriate behavior with excuses. 5. Playing the Victim Role (VIC): The manipulator portrays themselves as a victim to gain sympathy, attention, or divert focus from their misconduct. 6. Playing the Servant Role (SER): The manipulator disguises their self-serving motives as a contribution to a more noble cause. 7. Shaming or Belittlement (S_B): The manipulator uses sarcasm, criticism, and put-downs to make others feel inferior, unworthy, or embarrassed. 8. Intimidation (INT): The manipulator places others on the defensive by using veiled threats. 9. Brandishing Anger (B_A): The manipulator uses anger to brandish emotional intensity to shock the victim into submission. 10. Accusation (ACC): The manipulator suggests that the victim is at fault, selfish, uncaring, or leading an excessively easy life. 11. Persuasion or Seduction (P_S): The manipulator employs charm, emotional appeal, or logical reasoning to make others lower their defenses.
Definitions of 5 vulnerabilities targeted, with their abbreviations inside parentheses:
1. Naivety (NAT): The victim is easily trusting and struggles to accept that others might be malevolent. 2. Dependency (DEP): The victim has interestbased or emotional dependencies on the manipulator. 3. Over-responsibility (O_R): The victim is overly self-critical and sets high standards for themselves, often assuming undue blame and responsibility for the manipulator’s actions.
4. Over-intellectualization (O_I): The victim rationalizes the manipulator’s hurtful behavior by believing there is always a justified reason behind it. 5. Low self-esteem (L_S): The victim is selfdoubting and unconfident in pursuing their own wants and needs.

heading: B Key Phrase-based Matching
text: B.1 Key Phrases Examples • “you make me do this” • “how could you do this to me” • “know your place” • “you should not feel that way” • “what more do you want” • “i do not remember” • “i do not like drama” • “watch your step” • “you always do this” • “you are too sensitive” • “it was not intentional” • “you do not love me” • “you would do it if you love me” • “it is all in the past”
The complete set of key phrases can be found in our GitHub repository.
B.2 Online Resources We collected as many different manipulative phrases as we could from a number of websites suggested by Google by searching “phrases manipulative people use” and likewise. Some of the websites browsed are below:
• https://geediting.com/10-phrasesmanipulative-people-use-to-control-theirpartner-in-a-relationship • https://thevessel.io/phrases-manipulativepeople-use-to-play-the-victim • https://hackspirit.com/common-phrasespeople-use-to-manipulate-you • https://geediting.com/phrases-manipulativepeople-use-to-blame-others-for-their-actions
We found that there was significant overlap across phrases as we collected phrases.

heading: C Prompting Formats for GPT-4 and Llama-2
text: When designing the prompts, we aimed to craft phrases that were both straightforward and broad, reflecting inquiries that real users might pose.
C.1 Manipulation Detection
Zero-shot prompting format:
'''

heading: I will provide you with a dialogue. Please determine if it contains elements of mental manipulation. Just answer with
text: 'Yes' or 'No', and don't add anything else.
<insert dialogue> '''
Few-shot prompting format:
'''

heading: I will provide you with a dialogue. Please determine if it contains elements of mental manipulation. Just answer with
text: 'Yes' or 'No', and don't add anything else. Here are 3 examples:
Example 1: <insert example_dialogue1> <insert example_answer1>
Example 2: <insert example_dialogue2> <insert example_answer2>
Example 3: <insert example_dialogue3> <insert example_answer3>
<insert dialogue> '''
C.2 Technique and Vulnerability Classification
Zero-shot prompting format:
''' Here are the definitions of 11 different mental manipulation techniques: <insert definitions of 11 techniques>
Now, I will provide you with a dialogue that contains elements of mental manipulation. Please determine which manipulative techniques are used by the manipulator. Respond only with the names of the techniques, and do not add anything else.
<insert dialogue> '''
Few-shot prompting format:
''' Here are the definitions of 11 different mental manipulation techniques: <insert definitions of 11 techniques>
Now, I will provide you with a dialogue that contains elements of mental manipulation. Please determine which manipulative techniques are used by the manipulator. Respond only with the names of the techniques, and do not add anything else. Here are 2 examples:
Example 1: <insert example_dialogue1> <insert example_answer1>
Example 2: <insert example_dialogue2> <insert example_answer2>
<insert dialogue> '''
For vulnerability classification, the prompting formats are similar.

heading: D Analysis on Annotation Quality
text: Figure 8 is a heat map showing the inter-annotator agreement scores between any two annotators based on their answers for question Q1. For annotator ANi and annotator ANj, their agreement score is calculated as:
scoreij = ∥Annotations with same results∥
∥Common tasks of ANi and ANj∥
The last row is the average agreement score of each annotator. We can see that all annotators have a moderate to strong average agreement (≥ 0.7) with other annotators assigned with the same tasks.
Figure 8 is the scatter plot of annotators’ average confidence and inter-annotator agreement scores. 16 out of 17 annotators reported an average confidence score above 3. We calculated the Spearman’s rank correlation between annotators’ inter-annotator agreement and confidence levels. The statistic value is −0.21 and P-value is 0.41, which reveals a very slight negative correlation between inter-annotator agreement and confidence levels, which is surprising for us. This observation may be attributed to several factors. Firstly, many annotators tend to assign lower or medium confidence scores (2 or 3) when they are not entirely certain of their decisions, regardless of their comprehension of the dialogue and the available options. Conversely, some annotators habitually assign high confidence scores (4 or 5) to most of their decisions, reflecting individual differences in confidence assessment. Secondly, the assessment of mental manipulation is inherently subjective and lacks a uniform standard. Variations in what constitutes manipulation among annotators—with some setting higher thresholds than others—further diminish the reliability of inter-annotator agreement as a measure of annotation quality.
We also analyzed the inter-annotator agreement and average confidence by gender, as detailed in Table 9. On average, male annotators exhibited higher confidence scores compared to their female counterparts. Furthermore, the inter-annotator agreement was higher among male annotators than among females. These differences could be significantly affected by the number of annotators of each gender and the volume and difficulty of tasks to which they were jointly assigned.
Figure 9 illustrates the density distributions of annotators’ average agreement and confidence scores, both of which exhibit a normalized distribution.

heading: E More Statistics of MENTALMANIPmaj
text: The emotion distribution of dialogues in MENTALMANIPmaj dataset is in Figure 10. The co-occurrence details are in Figure 11.

heading: F Confusion Matrices
text: Please see Table 10, 11, 12, and 13.

heading: G Annotator Feedback
text: G.1 Prior Knowledge of Dialogue
There were several observations derived from the experiences of the annotators. Firstly, there was the incidence of prior knowledge of dialogue. The dataset was derived from dialogue in movie scripts, and as such, did include recognizable dialogue from some more well-known movie titles, such as “The Talented Mr. Ripley”. Given that annotators had more background knowledge with regards to the dialogue, and greater context, there is possibility that their annotation choices could have been influenced by their prior exposure to and knowledge of the movie dialogue.
G.2 Mutual Manipulation
Another observation from the annotation experience was that there could be mutual manipulation weaponized by both parties within a dialogue. While some dialogue clearly reflected manipulative speech by one party on the other, certain dialogues showcased manipulative tactics on both sides. Thus, it becomes difficult to differentiate between a clear perpetrator and victim, which also influences the selection of manipulation techniques during the annotation process.
G.3 Cognitive Fatigue / Overanalysis Lastly, annotators reported cognitive fatigue and over-analysis of tasks. Throughout the annotation process, which usually consisted of individual extended sessions of annotating, individuals became overly sensitive to cues and patterns. This hypersensitivity led to a heightened perception of manipulation in dialogues, such that they were unable to maintain a balanced perspective.

heading: H Annotation Example
text: Table 14 presents an annotation example.

heading: I Annotation Platform and Instruction
text: Figure 12 is the screenshot of annotation platform interface, and Figure 13 is the screenshot of instruction window.

heading: J More Dataset Statistics

heading: Future Work
text: In future work, it would be worthwhile to expand the dataset sources beyond the Cornell Movie Dialog Corpus and incorporate real-case interpersonal interac-
tion data. We also recognize that the performance of LLMs on more complex prompting paradigms, such as chain-of-thought, can be investigated. Detecting and finely classifying mental manipulation in conversations is a challenging task due to the implicit and context-dependent nature of the language often used. Furthermore, the subjectivity of human judgment complicates the alignment of AI models’ predictions with human choices. We have made the MENTALMANIP dataset publicly available for future studies and hope it will inspire and foster further research in various NLP tasks and applications