COKE: A Cognitive Knowledge Graph for Machine Theory of Mind (2024)

Abstract: Theory of mind (ToM) refers to humans’ ability to understand and infer the desires, beliefs, and intentions of others. The acquisition of ToM plays a key role in humans’ social cognition and interpersonal relations. Though indispensable for social intelligence, ToM is still lacking for modern AI and NLP systems since they cannot access the human mental state and cognitive process beneath the training corpus. To empower AI systems with the ToM ability and narrow the gap between them and humans, in this paper, we propose COKE: the first cognitive knowledge graph for machine theory of mind, formalizing cognitive processes as a chained structure. Specifically, COKE formalizes ToM as a collection of 45k+ manually verified cognitive chains that characterize human mental activities and subsequent behavioral/affective responses when facing specific social circumstances. In addition, we further generalize COKE using LLMs and build a powerful generation model COLM tailored for cognitive reasoning. Experimental results in both automatic and human evaluation demonstrate the high quality of COKE, the superior ToM ability of COLM, and its potential to significantly enhance social applications. We release our code and data at https://github.com/jincenziwu/COKE.

heading: 1 Introduction
text: In social environments, human beings must be able not only to react to what others are doing, but also to anticipate what they will do. This ability to understand and infer human goals is typically described as Theory of Mind (ToM) (Premack and Woodruff, 1978). One way of accomplishing ToM is to observe what others do in various situations, and derive a set of affective and behavioral rules.
* Work done during internship at the CoAI Group. † Equal Contribution. ‡ Corresponding author.
When the same or highly similar things arise again, we can bring out plausible predictions accordingly (Call and Tomasello, 2011). Figure 1 presents an example that someone will deliver a talk at the university tomorrow (a social circumstance), and he has substantial public speaking experience (a trigger factor). We can plausibly anticipate that he will deliver an impressive speech (a mental activity), feels joyful (an affective response), and has a restful sleep tonight (a behavioral response). Here ToM is instantiated as a chained cognitive process that derives from our knowledge, experiences, and memories (Harris et al., 1989). ToM is indispensable to humans since it allows us to leverage our own minds to simulate others’, so as to achieve efficient communication (Rabinowitz et al., 2018).
Despite its importance for social intelligence, ToM is not well internalized by modern AI and NLP systems. Shapira et al. (2023) illustrates a significant decline in performance and outright failure of Large Language Models (LLMs) in ToM tasks, particularly evident when confronted with adversarial samples. The main reason is that learning-based systems are usually trained on superficial text cor-
1 15984
pora, while lacking access to the underlying human mental state and cognitive process (Sap et al., 2022). In other words, NLP systems rely on the maximum likelihood to understand and generate texts, but do not go beneath the surface to the desires, beliefs, and intentions of humans.
In this paper, we introduce COKE: the first COgnitive KnowledgE graph formalizing cognitive process as a chained structure for machine theory of mind. Our goal is to formalize ToM and make it accessible and learnable for AI systems. In COKE, we instantiate ToM as a collection of manually verified cognitive chains that characterize humans’ mental activities in specific social circumstances along with their behavioral and affective responses (Meinhardt-Injac et al., 2018; Mehl et al., 2020). Each cognitive chain involves five types of nodes: 1) situations denote the social circumstances; 2) clues denote the trigger factors; 3) thoughts denote the mental activities; 4) actions denote the behavioral responses; 5) emotions denote the affective responses. Moreover, as shown in Figure 1, individuals react differently to the same situation due to the diversified cognitive processes. Therefore, for each situation, we derive multiple cognitive chains and further label them as positive (means optimistic) or negative (means pessimistic) to mark the chain polarity. We propose to induce the raw data from LLMs, and then recruit educated workers majoring in psychology for manual selection and revision. The resulting knowledge graph constitutes 62,328 nodes and 45,369 cognitive chains.
The construction of COKE offers the basic ToM ability to understand and infer the human goals in already collected situations (Call and Tomasello, 2011). But obviously, it is impossible to enumerate all situations in the real world. Thus we move one step further and build a cognitive language model COLM to cope with unseen situations that have not appeared in the knowledge graph. Specifically, we decompose the construction of cognitive chains into four cognitive generation tasks, then finetune LLMs using the manually collected data in COKE. By this means, we combine the commonsense knowledge embedded in LLMs and the ToM ability provided by COKE, enabling COLM to infer cognitive chains for unseen situations.
We summarize our contributions in this work as follows. 1) We propose the first cognitive knowledge graph for machine theory of mind, formalizing cognitive process as a chained structure. We instantiate human theory of mind as a collection of
45k+ manually verified cognitive chains, which provides a basic ToM ability for accessing and learning. 2) We build a powerful cognitive language model COLM by associating COKE with LLaMA-2 (Touvron et al., 2023), so as to predict cognitive chains for out-of-KG situations. 3) We conduct extensive experiments to evaluate the ToM ability of COLM and typical LLMs. The results show that COLM outperforms strong baseline models such as GPT-4 (Achiam et al., 2023) in both zero-shot and few-shot settings, proved by automatic and human evaluations in all cognitive generation tasks, which in turn demonstrates the high quality of COKE. 4) We further substantiate the potential of COKE in enhancing social applications, and prove its effectiveness on downstream emotional support conversation task.
2 COKE: Cognitive Knowledge Graph

heading: 2.1 Preliminaries
text: What is Theory of Mind? Theory of Mind refers to the ability of humans to understand and infer other people’s desires, beliefs, and intentions (Premack and Woodruff, 1978). Under specific social circumstances (situations), the core mechanism by which ToM empowers us is to ascribe others’ mental activities (thoughts), and predict their corresponding behavioral responses (actions) and affective responses (emotions) (Leslie et al., 2004; Apperly, 2010). Furthermore, the acquisition of ToM enables us to realize and appreciate that people can have different cognitive responses in the same situation due to various trigger factors (clues) like personality and experience (Meinhardt-Injac et al., 2018).
Why Do AI Systems Need ToM? ToM has been a persistent yet elusive goal of artificial intelligence for decades (Choi, 2022). AI systems need ToM to understand a user’s situations and predict subsequent reactions, so as to provide effective responses or operations that meet the user’s needs (Le et al., 2019; Dhelim et al., 2021; Langley et al., 2022). However, recent studies (Shapira et al., 2023) show that today’s LLMs still lack ToM and social intelligence. The main reason is that AI systems can only learn from text corpora in training, but cannot access the human mental state and cognitive process that determine what and why to say. Now, there are neither public resources that contain all the concepts in ToM, nor commonsense knowledge graphs that depict the structure of cognitive chains
2
in ToM. This motivates us to propose the first cognitive knowledge graph for machine Theory of Mind.
2.2 Data Structure of COKE According to the above-mentioned psychology research on theory of mind, we specify five types of nodes in COKE: situations, clues, thoughts, actions, and emotions. We here define the basic unit of COKE as the following cognitive chain: Situation⇒Clue⇒Thought⇒(Action+Emotion). This structure depicts the intact cognitive process of ToM: When a person faces a situation, some clues trigger his/her thoughts, along with his/her actions and emotions. Furthermore, to distinguish whether a cognitive chain is optimistic or pessimistic under the specific situation, we further define its polarity as positive or negative. In practice, the polarity of the cognitive chain is determined by its thought node. Notice that, in COKE, we omit the definition of edges (i.e., the connections between nodes) since they can be easily inferred when the types of nodes are already known. We then illustrate the nodes in detail. Situations in COKE denote the social circumstances in which individuals (potentially) interact with others. By referring to DailyDialog (Li et al., 2017), a widely used daily social dialogue dataset, we select the five most common social topics: School (what happened at school), Work (what happened at work), Tourism (travel and entertainment), Relationship (social activities between individuals), Ordinary Life (what happened in families). Detailed information about each topic can be found in Appendix A. Clues in COKE denote the trigger factors that direct and concretize the cognitive process. In a specific situation, humans’ mental activities are triggered and directed by relevant subjective and objective factors (Meinhardt-Injac et al., 2018). According to the taxonomy from (Baldwin, 1992), clues mainly involve the particular information about personality, knowledge, experience, education, objective facts, social rules, and so on. Thoughts in COKE denote the mental activities that act as the cognitive responses to situations. Thoughts serve as the bridge between the external environment and individual cognition, thus can be considered as the core of ToM (Westbrook et al., 2011). As mentioned before, the polarity of a cognitive chain is anchored to its thought node. In other words, an optimistic thought marks the entire cognitive chain as positive, and a pessimistic
thought marks it as negative. Actions in COKE denote the behavioral responses to situations after specific thoughts. Notice that the semantic meaning of actions may not conform to their polarity annotations. For example, in Figure 1, the action “I practice my script in front of the mirror” is a neutral sentence. However, it is the consequence of the negative thought “I may mess up the speech”, so it is still labeled as negative. Emotions in COKE denote the affective responses to situations after specific thoughts. Without loss of generality, we restrict the emotions to six basic categories (Phillip et al., 1987): Love, Surprise, Joyful, Sad, Angry, and Fearful. The first three appear in positive cognitive chains, while the last three appear in negative cognitive chains.

heading: 2.3 Data Creation and Selection
text: Recent studies have proved that LLMs trained on huge text corpora can naturally serve as a repository for data collection (West et al., 2023; Kim et al., 2023). Inspired by their successful attempts, we propose a two-step data collection approach for constructing COKE. As shown in Figure 2, 1) we first manually design suitable few-shot prompts to induce GPT-3.5 (i.e., text-davinci-002 ) Ouyang et al. (2022) to automatically generate raw data for five types of nodes in a pipeline manner. 2) We then recruit and train eight graduate students majoring in social psychology as annotators to select and revise the outputs of GPT-3.5. Next, we illustrate the details of how to prompt GPT-3.5, then introduce the data statistics after human annotation. More detailed parameters and templates used for prompting are provided in Appendix B and C.
Prompting LLMs for Situations Situations in COKE denote the social circumstances in which individuals (potentially) interact with others, which exhibit a strong correlation with social events and describe the social environment of daily lives. Unfortunately, to the best of our knowledge, there is no public dataset that contains a wide variety of social situations. Therefore, we choose the events in ATOMIC (Sap et al., 2019), a frequentlyused commonsense knowledge graph for if-then reasoning, as an alternative. However, the events from ATOMIC are still not qualified to become the situations in COKE. The reasons are two-fold, and we here take the event “PersonX misses calls from PersonY” as an example. 1) ATOMIC events replace the characters with “PersonX” and “Per-
3
sonY”, thereby losing most of the interpersonal information that is indispensable for social situations. For example, if someone misses calls from his employer, he may worry that something is wrong with his work. But if the caller changes to his girlfriend, he may sense love because she cares about him. 2) ATOMIC events omit most of the social context information, making the background environments where the events occur not available. For example, we don’t know if the calls in the above event happen at work or on vacation, so we have no idea how to reify subsequent cognitive processes.
To address the problem, we choose to prompt GPT-3.5 to rewrite ATOMIC events to qualified COKE situations. Specifically, we first manually create several demonstration (input, output) pairs such as (“PersonX misses calls from PersonY”, “I miss multiple calls from my girlfriend because I’m playing Project Zomboid”). Then we wrap the demonstrations with the task instruction (“Translate [Event] to [Situation]”) and an arbitrary event query (“PersonX delivers PersonX’s message”) to form an input template as shown in Figure 3. As a result, we collect the output situation from GPT-3.5 like “I will deliver a talk at the university tomorrow”. In practice, we manually select 400 ATOMIC events that are general and easy to adapt, then rewrite each event to 5 situations with different topics. Finally, we obtain 2,000 raw situations.
Prompting LLMs for Thoughts Thoughts in COKE denote the mental activities that act as the
cognitive responses to situations. As mentioned above, thoughts serve as the bridge between the external environment and individual cognition, and can be seen as the core of ToM. Since thoughts are triggered by certain clues, they come after clues in cognition chains. However, our pilot experiments show that directly using situations to prompt LLMs can better stimulate their generation ability and get more diverse thoughts, which is beneficial to manual selection and revision. Therefore, here we temporarily reverse the order of thoughts and clues, i.e., we first prompt LLMs using situations to generate thoughts, and then prompt LLMs using thoughts to generate clues.
Similar to the instruction prompts for collecting situations, we manually construct the demonstration as “When [Situation], I feel great/terrible since I think [Thought]” and again wrap it with the task instruction and an arbitrary situation query to form the input template. Notice that we use different sentiment terms (“great” and “terrible”) to control the polarity (positive and negative) of the generated thoughts. Since the polarity of a cognitive chain is anchored to its thought, we have also controlled the polarity of the entire cognitive chain that will be generated. After prompting, we obtain 14,400 raw thoughts, half of which are positive and the other half are negative.
Prompting LLMs for Clues, Actions, and Emotions Clues in COKE denote the trigger factors that direct and concretize the cognitive process. Thus we construct the demonstrations as “ Complete the sentence: When [Situation], I think [Thought] since [Clue]” to prompt LLMs for clues. Actions and emotions in COKE denote the behavioral and affective responses to situations after specific thoughts. We hence construct the demonstrations as “When [Situation], I think [Thought] so [Action]” and “When [Situation], I think [Thought] and I feel [Emotion]” to prompt LLMs for actions and emotions. Since emotion belongs to predefined
4
categories, we further modify its corresponding prompt to the question-answering form. Finally, we obtain 29,364 raw clues, 29,364 raw actions, and 14,400 raw emotions. Their polarities are already determined by previously generated thoughts.
Human Selection and Revision After collecting the raw data, we manually annotate a small amount of data and formulate several rules to distinguish good data from bad data. Subsequently, we use the detailed definition of nodes in COKE and the filtering rules as a tutorial to train the eight annotators. After passing an annotation qualification test, they are asked to select and revise the raw data of five types of nodes. More details about human annotation can be found in Appendix D.
As shown in Table 1, the final reserved data contains 1,200 situations, 9,788 thoughts, 21,677 clues, 19,875 actions, and 9,788 emotions, resulting in an overall retention rate of around 70%. This statistic proves that ToM ability exhibited by most powerful LLMs like GPT-3.5 is still not satisfactory enough even with delicate prompting, thus further emphasizing the necessity of our construction of COKE. After linking and ordering the obtained nodes, we instantiate ToM with a total of 45,369 cognitive chains in COKE, containing 23,252 positive chains and 22,117 negative chains in English. Example COKE data can be found in Appendix E.
3 Cognitive Language Model COLM
By consulting the constructed cognitive knowledge graph COKE, we can obtain the basic ability of theory of mind (ToM) via matching the faced situation to a similar situation in KG, and then inspecting the involved cognitive chains (a.k.a, entity linking). But obviously, COKE only collects finite situations and cannot cover the infinite and diverse situations in the real world. Inspired by the methods for automatic knowledge graph completion like COMET (Bosselut et al., 2019), we propose a cognitive language model COLM to cope with unseen situations and expand the scope of
application of COKE. COLM is built upon LLMs, aiming to integrate the commonsense knowledge inside LLMs and the ToM ability from COKE. To this end, we first decompose the cognitive process into a sequence of cognitive generation tasks, and then finetune LLaMA-2 (Touvron et al., 2023) using the collected data from COKE.

heading: 3.1 Cognitive Generation Tasks
text: In COKE, a cognitive process towards theory of mind (ToM) is instantiated as a sequenced cognitive chain containing situation, clue, thought, action, and emotion. Therefore, given a situation, we can decompose the cognitive process into four generation tasks as shown in Table 2. These four tasks work in a pipeline manner, and the complete cognitive chain can be restored by linking their generated results. 1) Clue Generation. When facing a specific situation, humans can automatically distinguish the factors that may influence beliefs and trigger thoughts. 2) Thought Generation. In a specific situation, the related clues trigger and arouse diversified human mental activities, i.e., thoughts. 3) Action Generation. Driven by specific thoughts, we humans will take corresponding actions to realize our beliefs and achieve our goals. Notice that we omit clues here since their impacts are largely covered by the triggered thoughts. 4) Emotion Generation. After forming a specific thought in a certain situation, humans will naturally generate corresponding emotions to express attitudes and views on the situation. Since emotions are limited to 6 categories, this task is a classification task.
By linking the above four tasks in a pipeline manner, we can restore the cognitive chain “Situation⇒ Clue⇒ Thought⇒ (Action+Emotion)”, thus preserving the complete cognition process of ToM. Since each cognitive chain in COKE is labeled with polarity, each cognitive generation task can be further divided into positive and negative subtasks (e.g., positive thought generation and negative thought generation).
3.2 Training COLM
After decomposing the cognitive chain into four cognitive generation tasks, we can process the data in COKE accordingly to obtain training samples. For computational efficiency, we design COLM as a multi-task controllable generation model, so that it can simultaneously accomplish four cognitive generative tasks and further control the polarity (i.e., positive or negative) of the outputs. As shown
5
in Table 2, for each task and polarity, we insert specific tokens (i.e.,[NegClue],[PosClue]) in the input X to guide the generation process of COLM. For implementation, COLM is built as a decoderonly architecture and initialized with the LLaMA-2 (Touvron et al., 2023). The model is trained with each input-output pair X-Y from any task.

heading: 4 Experiments
text: In this section, we construct a dataset from COKE to evaluate ToM ability of our cognitive language model COLM, and compare it with advanced LLMs including GPT-3.5 Turbo, GPT-4, LLaMA-2-7B (our backbone) and Mistral-7B. We first illustrate the experimental setup, then analyze the experimental results for automatic and human evaluation, and finally validate COKE’s effectiveness to empower social applications.

heading: 4.1 Experimental Setup
text: To evaluate the ToM ability of different models, we randomly split 1,200 social situations into 1,080 (90%) training and 120 (10%) validation situations. Then we can automatically split samples for different cognitive generation tasks according to the situations, and obtain training/validation splits as 19,409/2,268 (clue), 8,746/1,042 (thought), 17,982/1,893 (action), and 8,746/1,042 (emotion). Based on this setting, we ensure that the cognitive chains in the validation set all occur in UNSEEN situations, which is crucial for ToM evaluation.
For COLM, we use the Hugging Face implementation1 of LLaMA-2, and train it for 20 epochs using the AdamW optimizer (Loshchilov and Hutter, 2019) and LoRA (Hu et al., 2021) with learning rate 3e-4 and batch size 32 in Five Tesla V100 GPUs. The whole training process costs about 9 GPU hours. For baseline models, we construct manual prompts to enable them to complete the cognitive generation tasks. We evaluate 0-shot, 2- shot, and 5-shot performance of all baseline LLMs. More details can be found in Appendix F. Due to
1https://huggingface.co/meta-llama/Llama-2-7b-hf.
the data deficiency, we report the performance on the validation set.

heading: 4.2 Main Results
text: Automatic Evaluation To evaluate the clue/thought/action generation tasks, we use METEOR (Lavie and Denkowski, 2009), ROUGE (Lin, 2004), BLEU-1, BLEU-2 (Papineni et al., 2002) and BERTScore (Zhang et al., 2019) as metrics. In these three tasks, each input may be mapped to multiple ground truth outputs. Therefore, following Mostafazadeh et al. (2020), we compute the average scores between all predicted sequences and all ground truth outputs2. Moreover, to evaluate the emotion generation (classification) task, we compute the classification accuracy as the metric. We present all automatic evaluation results in Table 3.
Compared to baseline models like LLaMA-2 and Mistral, which face challenges in following task instructions, COLM shows significant performance enhancements across various cognitive generation tasks with COKE. While leveraging additional prompts boosts performance for powerful models like GPT-3.5 Turbo and GPT-4 via incontext learning in clue/thought/action generation tasks, these LLMs still struggle with complex emotional understanding (Wang et al., 2023) and fail in the emotion classification task. Compared to these powerful LLMs, COLM maintains substantial advantages across all evaluation metrics. We hereby have two observations: 1) The data we collected in COKE is of high quality and can empower the model with strong ToM ability. 2) COLM, which is designed as a controllable generative model for multiple cognitive tasks, can effectively internalize the ToM ability and cope with unseen situations.
Human Evaluation Beyond automatic evaluation, we also wonder how humans perceive the cognitive chains generated by different models since ToM is essentially a human ability. Therefore, for each model, we sample a cognitive chain for each
2Here we provide all models with the required polarity (positive or negative) via special tokens or prompts.
6
situation in the validation set (resulting in 90 samples per model) for human evaluation. We present three experts with the cognitive chains generated by COLM and top-performing LLMs in automatic evaluation, and then ask them to score on a 5-point Likert scale based on relevance, correctness, and logicality. Specifically, they were instructed to rate the chains as follows: “completely irrelevant (1 point)”, “contains irrelevant content (2 points)”, “relevant but does not follow the task definition correctly (3 points)”, “follows the task definition correctly but does not logically continue the cognitive chain (4 points)” and “continues the cognitive chain correctly and logically (5 points)”. We calculate the final score for each model by averaging these ratings. A rating of 5 indicates valid content.
The results in Table 4 illustrate that the cognitive chains generated by COLM are more acceptable to humans, which further demonstrates the superiority of its ToM ability. Even with clear task instructions and demonstrations, baseline LLMs failed to generate inferences acceptable to humans. GPT3.5-Turbo received a significant number of 3-point ratings, indicating responses that were “relevant but does not follow the task definition”, such as generating thought in clue task. Additionally, there were a few 2-point ratings, indicating “contains irrelevant content”, such as generating content unrelated to ToM definition. GPT-4 received a small number of 3-point and 4-point ratings. COLM received only a small number of 4-point ratings. As a further illustration, we present ten exemplary cogni-
tive chains generated by COLM across five diverse situations, covering various topics, in Appendix G.

heading: 4.3 Empowering Social Application
text: As COKE is a cognitive knowledge graph for machine theory of mind, we further substantiate its effectiveness in empowering social applications with ToM capabilities. Our chosen testing ground is the emotional support conversation (ESC) task (Liu et al., 2021). The prime objective of ESC is to generate empathetic and effective responses in social dialogues, with the aim of mitigating users’ emotional distress and fostering improved mental states. ESC requires ToM because it involves understanding and responding to a user’s emotional state and underlying challenges by understanding their mental processes and intentions.
7
Integration of COKE in ESC The ESC dataset has already provided the situation behind each dialogue. We then treat each user’s utterance in the dialogue as a clue, and employ COLM to infer two thoughts (1 positive and 1 negative) via thought generation. Based on the generated thoughts, we can accordingly infer actions for each thought via action generation. We simply extract verbs and nouns in thoughts or actions as the ToM knowledge keywords, and append them to the end of dialogue history, serving as an enhanced context. Based on the context, the ESC system is trained to generate the corresponding response. We here use the Blenderbot (Roller et al., 2021) as the generation model, which is the same as the original ESC paper.
ESC Performance with ToM Knowledge In our experiments, we compare three models: Vanilla ESC, ESC with COLM Actions, and ESC with COLM Thoughts. We conduct both automatic and human evaluations, following the same methodology as the original ESC paper. We sample 30 dialogues in the validation set and let two experts rate Fluency, Identification, Comfort, and Suggestions on a 3-point Likert scale. The final score is the average of these ratings. The experimental results shown in Table 5 demonstrate that, when incorporating the ToM knowledge into the response generation process, both the ESC with COLM Actions and ESC with COLM Thoughts models significantly outperform the Vanilla ESC model across all metrics. For clarity, we further present a case study and how COKE offers a more nuanced and empathetic approach to AI-driven emotional support in Appendix H .

heading: 5 Related Work
text: Cognitive knowledge is essential for modeling intuitive tasks, such as reasoning, in language models.
Previous work mainly focuses on realizing cognitive knowledge as an accessory to commonsense knowledge. They construct text-based knowledge graphs that include numerous instances of HeadRelation-Tail triplets regarding different events and objects. One of the widely-used examples of such knowledge graphs is ConceptNet (Speer et al., 2016), which provides a graph of concepts connected by relations, covering a variety of taxonomic facts. Sap et al. (2019) proposed ATOMIC, a graph of if-then inferences that models social commonsense in daily life events. Hwang et al. (2021) expanded upon ATOMIC by incorporating two additional categories of commonsense relations: physical-entity commonsense relations and event-centered commonsense relations. West et al. (2021) prompted GPT-3 to distill commonsense knowledge and create a machine-generated corpus ATOMIC10x. Kim et al. (2023) further leveraged it to generate a large-scale dataset focused on socially-grounded conversations. Furthermore, NOVATOMIC (West et al., 2023) employs natural language queries as open relations to link the commonsense knowledge, enabling its application in general reasoning tasks.
Generally, existing KGs consider eventcentered commonsense relations, such as temporal and causal relationships, represented as Event IsBefore ←−−−− Event Causes −−−−→ Event. However, they have not explicitly addressed ToM concepts and relations, which are crucial for accessing and interpreting human mental states and cognitive processes. In contrast, COKE delineates ToM concepts and structures them as a cognitive chain: Situation⇒ Clue⇒ Thought⇒(Action+ Emotion). This chained structure is designed to mirror human cognitive processes, enabling a deeper understanding of how individuals infer others’ mental states in specific social circumstances along with their behavioral and affective responses. Moreover, the ATOMIC family, which uses short phrases with generic placeholders like “PersonX”, significantly constrains the social and interpersonal information critical for complex ToM reasoning (Zalla and Korman, 2018). COKE addresses this limitation by providing a richer and more nuanced context for each cognitive concept in the chain. This approach broadens the scope of cognitive knowledge accessible to machine systems and enhances their capability in social cognition, hereby fostering downstream applications.
8

heading: 6 Conclusion
text: In this work, we present COKE, the first cognitive knowledge graph for machine theory of mind. We instantiate ToM as a series of cognitive chains to describe human mental activities and behavioral/affective responses in social situations. Through prompting GPT-3.5 and manual annotation, we collect 62k+ nodes and construct 45k+ cognitive chains. Based on COKE, we build a powerful cognitive language model COLM. COLM can handle unseen situations and predict complete cognitive chains in a pipeline manner. Automatic and human evaluations show that COLM effectively internalizes the ToM ability and outperforms strong baseline models like GPT-4.
We further demonstrate that COKE can empower LLMs with a nuanced understanding of human affective and cognitive reasoning, resulting in superior performance in ESC. By integrating COKE into LLMs, they gain the ability to comprehend and predict human cognitive processes with greater depth. We believe that COKE and COLM can facilitate social applications such as dialogue systems and autonomous agents.

heading: Acknowledgments
text: This work was supported by the National Science Foundation for Distinguished Young Scholars (No. 62125604) and the NSFC Key Project (No. 61936010). This work was also supported by Tsinghua Precision Medicine Foundation, Tsinghua University - Beijing Tsingshang Architectural Decoration Engineering Co., Ltd. Joint Institute for Smart Scene Innovation Design, and China Postdoctoral Science Foundation (No. 2023M741944).
Limitations
In this work, we introduce COKE, the first cognitive knowledge graph for machine Theory of Mind, which aims to empower AI systems with cognitive capabilities. However, we acknowledge the following omissions and inadequacies in our work.
Data Coverage We acknowledge that there is a limitation in topic coverage of proposed COKE, which only covers five social topics that are commonly discussed. Besides, COKE has a relatively small data scale in comparison with other relative knowledge graphs, due to its more specific node contents and more complicated construction process. Consequently, COKE cannot cover all situa-
tions in deployment, and the cognitive reasoning models constructed on this basis may have unreliable predictions in out-of-domain situations.
Cognitive Inference Ability As a pioneer in integrating Theory of Mind into AI systems, the proposed cognitive language model COLM still has a lot of room for improvement in inferring cognitive chains when deployed in practice. In COLM, we take LLaMA-2 as the backbone model to validate the gain of COKE on the cognitive inference ability of language models. We acknowledge that adopting larger backbone models would contribute to a more powerful inference model, which is worth exploring in more depth in the future.
Ethics Statement
We present the first cognitive knowledge graph COKE for machine theory of mind. Our graph is built based on public datasets and model generations. We strictly adhere to data source usage protocols and ensure that the proposed COKE can be released and used LEGALLY.
The construction of COKE has gone through the steps of manual selection and revision. We manually filtered the data containing potentially private information, such as phone numbers and email addresses, to protect user PRIVACY further. We also carefully delete abusive, offensive, biased, and other inappropriate content to avoid unpredictable ETHICAL hazards.
Our knowledge graph is designed to empower AI systems with the ability of cognitive inference. We are aware that this capability could be misused in malicious scenarios in the future. However, we believe the value of this work outweighs the risks, and we also call for more socially responsible research in this field.

heading: A Topic Keywords
text: We identify the top 8 keywords in five topics using TF-IDF. As indicated in Table 6, the information provided by situations in each topic is distinct and
relevant to the theme. According to the keywords “boss” or “work”, we can locate information such as “I attend my boss’s meeting” in the Work topic. The project deadline dilemma like “I bet my friend that I can finish my project before the deadline” is shared in the School topic. Besides, we can find the travel schedule like “I asked my best friend for advice on what to pack for my upcoming trip to Europe” in the Tourism topic. Date sharing like “I brought my date back to my place after the movie” can be found in the Relationship topic. Daily diary like “My friend is helping me plan my surprise party” appears in the Ordinary Life topic.

heading: B Parameters for GPT-3.5 API
text: Utilization
During the data collection process, we use the GPT3.5 API offered by OpenAI. We read the terms of service3 and follow the usage policies4 . We give the parameter details of the GPT-3.5 API utilized in data collection and the experiments in Table 7. Our data collection was finalized before the release of GPT-3.5 Turbo and GPT-4. Therefore, we opted for the powerful model text-davinci-002.

heading: C Prompts for Data Collection
text: In Table 11, we present the detailed prompts for how to extend the base event to the situation in our COKE. When we design the prompt template, we find that the base events prefixed with the token [Sentence] in the prompt lead to better generation than those with the token [Event]. Table 12 and Table 13 show the specific prompts used to collect data in the negative and positive chains for each generation task.
3https://openai.com/api/policies/service-terms/. 4https://beta.openai.com/docs/usage-policies.
11
D Instruction for Human Selection and Revision
Since our cognitive knowledge graph is closely related to psychology, we choose and train eight graduate students majoring in social psychology as annotators. They are evenly distributed by gender (four males and four females) and come from various regions. We pay the annotators approximately $12 per hour, which exceeds the local minimum wage, and the total cost for annotation is about $6,000. We illustrate the relevant background and how the data would be used clearly in the first beginning.
To construct the knowledge graph for machine Theory of Mind (ToM), we have researched relevant papers and books and discussed it with the social psychology professor several times over three months. As a result, we decompose the ToM inference into four inference tasks and clearly define the five types of nodes. In the annotation instructions, explicit definitions of nodes are provided.
In order to effectively train the annotators, we first have a testing annotation on 160 situations for each task ourselves. Afterwards, we determine what types of incorrect data annotators may encounter and utilize these annotation examples to provide specific guidance. Several common types of bad data in all tasks are 1) repetitive context, 2) unsafe words, and 3) offensive content. Therefore, we automatically filter out the repetitive context and let the annotator manually remove the unsafe terms and offensive content. Besides, the specific types of incorrect data for each task are depicted in the details instructions in Figure 5, Figure 6, Figure 7, and Figure 8. In order to ensure data diversity, we ask workers to modify repetitive data into different and reasonable data.
During training, the annotators work on ten testing situations and receive feedback after following the instructions. Then, based on their annotations of the second 10 situation examples, we evaluate if students have a solid grasp of the task. With specific guidance, students all do well in the subsequent annotation. In addition, to maintain the high quality of the data, we allow them to highlight data with which they are confused and work as the experts to make the ultimate decision. For example, in emotion data revision, the annotator assigns additional emotion labels to the illogical inference, and the experts make the final decision.
E Example Cognitive Chains in COKE
We present the cognitive chains in COKE from five topics in Figure 9 (School), Figure 10 (Work), Figure 11 (Tourism), Figure 12 (Relationship), and Figure 13 (Ordinary Life), respectively. The data in COKE is in English.

heading: F Prompts for PLMs Generation in Evaluation
text: As shown in Table 8, we present the prompts that lead the LLaMA-2, Mistral, GPT-3.5 Turbo and GPT-4 to make inferences on the validation dataset.

heading: G Case Study of Generated Cognitive Chains
text: To have a close look, in Table 9, we present ten cognitive chains generated by COLM in five situations covering different topics. It can be observed that the proposed cognitive generation model COLM can generate smooth and effective cognitive chains in unseen situations.

heading: H Empowering Emotional Support
text: Conversation with COKE
In this section, we discuss how COKE can be used to enhance social applications, specifically focusing on Emotional Support Conversation (ESC) (Liu et al., 2021). This task represents a key arena in which Theory of Mind (ToM) can play a pivotal role, given the need to understand and respond to a user’s emotional state and underlying challenges.
Emotional Support Conversation The ESC task is structured around a user in a negative emotional state, potentially due to a specific problem or challenge they are confronting. The user’s emotional state is characterized by a negative emotion label
12
(e), an intensity level of the emotion (l, on a scale of 1 to 5), and the underlying challenge causing their distress. The goal of the supporter (or AI system) is to comfort the user in a conversation, deploying support skills to reduce the intensity of the user’s negative emotions. The conversation’s effectiveness is gauged by the extent to which the user’s emotional intensity is reduced, and the ability of the supporter to accurately identify the problem, comfort the user, and offer constructive solutions or suggestions.
Experimental Results In experiments, we compare three models: Vanilla ESC, ESC w/ COLM Actions, and ESC w/ COLM Thoughts. We conduct both automatic evaluation (PPL, BLEU-2, ROUGEL, Extrema) and human evaluation (Fluency, Indentification, Comforting, Suggestion), same as the original ESC paper. The experimental results are presented in Table 5. To have a close look, we also present a case study in Table 10. The results demonstrate the potential of COKE to empower social applications, offering a more nuanced and empathetic approach to AI-driven emotional support. By integrating COKE’s cognitive chains into dialogue history, AI systems can achieve a more sophisticated understanding of human emotions and behavior, leading to more effective interactions in
emotionally charged contexts such as the ESC task. Interestingly, the ESC with COLM Thoughts model shows a slight advantage over the ESC with COLM Actions model. This could be attributed to the fact that the thoughts generated by COLM offer a more direct reflection of the user’s mental state.
13
Topic Situation Clue Thought Action Emotion
Neg. It was worth more than my car. The phone is going to be ruined. I quickly sent an email to my boss to own up to the mistake.
Fearful
Work I accidentally dropped my companies latest phone prototype. Pos. I have been working so hard on
this project. My boss will forgive me. I quickly picked it up and acted like it was no big deal. Joyful
Neg. He has been acting distant recently.
He may not show up. I asked for some friends as backup.
Sad
Relationship I also set up a romantic evening for the two of us. Pos. We have been getting along very
well lately. He is going to propose to me. I am going to take him to his favorite restaurant. Love
Neg. I have never done things differently before.
I will not be able to understand the material. I read through the material again. Fearful
School I am going to adopt a new approach to my studies. Pos. I have put in a lot of effort. It will improve my grades. I take my study time more
seriously. Joyful
Neg. I do not pack enough money. I will regret not spending my time more wisely.
I asked him to take me to as many places as he could. Sad
Tourism I will accompany my brother to Greece for the summer.
Pos. We explored different cities and towns. I will have a great time there. I pack my bags with the best clothes I have.
Joyful
Neg. He is not my real father. He will spend all my money. I find a more reliable person. Fearful Ordinary Life
My dad acts as my power of attorney . Pos. My dad is really responsible and
always helps me out. He will make good decisions on my behalf. I follow his instructions. Joyful
Table 10: Case study of emotional support conversation.
Figure 4: Empowering ESC with COLM. A detailed case study is illustrated in Table 10.
14
15
16
17
18
19
20
21
22
23
24

heading: Future Work
text: We acknowledge that adopting larger backbone models would contribute to a more powerful inference model, which is worth exploring in more depth in the future