Reasoning in Conversation: Solving Subjective Tasks through Dialogue Simulation for Large Language Models (2024)

Abstract: Large Language Models (LLMs) have achieved remarkable performance in objective tasks such as open-domain question answering and mathematical reasoning, which can often be solved through recalling learned factual knowledge or chain-of-thought style reasoning. However, we find that the performance of LLMs in subjective tasks is still unsatisfactory, such as metaphor recognition, dark humor detection, etc. Compared to objective tasks, subjective tasks focus more on interpretation or emotional response rather than a universally accepted reasoning pathway. Based on the characteristics of the tasks and the strong dialogue-generation capabilities of LLMs, we propose RiC (Reasoning in Conversation), a method that focuses on solving subjective tasks through dialogue simulation. The motivation of RiC is to mine useful contextual information by simulating dialogues instead of supplying chain-of-thought style rationales, thereby offering potential useful knowledge behind dialogues for giving the final answers. We evaluate both API-based and open-source LLMs including GPT-4, ChatGPT, and OpenChat across twelve tasks. Experimental results show that RiC can yield significant improvement compared with various baselines.

heading: 1 Introduction
text: Large language models (LLMs; OpenAI, 2022, 2023; Touvron et al., 2023; Jiang et al., 2023; Wang et al., 2024a) have made rapid advancements in recent years and have achieved excellent performance on various objective tasks, including open-domain question answering (Kamalloo et al., 2023), mathematical reasoning (Luo et al., 2023; Yang et al., 2023), and code generation (Chen et al., 2021; Roziere et al., 2023), etc.
Despite the success, research on LLMs in subjective tasks (Rottger et al., 2022; Kanclerz et al., 2023; Sun et al., 2023a) is still underexplored, as examples
*Equal contribution. BCorresponding authors.
shown in Table 1. Different from objective tasks that can often be clearly defined and solved, subjective tasks (e.g., metaphor recognition and dark humor detection) involve the capability to perceive context, language nuances, and emotions, which cannot be easily quantified or objectively measured, thereby posing challenges for current LLMs (Jentzsch and Kersting, 2023; Wachowiak and Gromann, 2023; Mao et al., 2023).
Recent methods based on chain-of-thought (CoT) style prompting have improved the reasoning abilities of LLMs, showing promising results on tasks such as commonsense and mathematical reasoning (Wei et al., 2022; Kojima et al., 2022; Zhang et al., 2023b). However, compared to these objective tasks, we found that such methods are not particularly effective on subjective tasks. As an example shown in Figure 1(b), the reasoning pathway does not comprehend the metaphorical expression in “Joseph has the heart of a lion” well,
15880
resulting in incorrect responses. Dialogue, alternatively, provides humans with a means to raise questions, convey emotions, and express opinions, which can be seen as another way to facilitate subjective reasoning (Resnick et al., 1993; Rips et al., 1999). Considering the characteristics of subjective tasks and the strong ability of dialogue generation for LLMs (Thoppilan et al., 2022; Tian et al., 2023), we propose RiC (Reasoning in Conversation), a method aiming to uncover the subjective expressions in simulated dialogues instead of objective and relatively unified reasoning pathways for better reasoning on subjective tasks. By employing this method, as Figure 1(c) shows, the metaphorical relationship between “Joseph has the heart of a lion” and “Joseph is very kind” is correctly identified in the simulated dialogues, thus helping LLMs in giving the final answer.
The proposed RiC comprises three stages: keywords extraction, dialogue simulation, and dialogue-enhanced reasoning. To enable better comprehension of the questions and dialogue generation, we first allow LLMs to extract task-relevant keywords according to the ques-
tion, which has been shown helpful for understanding the task and generating related dialogue (Zhu et al., 2023; Yu et al., 2023). Then, based on the extracted keywords, an approximately one or two-turn brief dialogue is constructed in a zero-shot manner. Finally, we enable LLMs to engage in reasoning based on both the original question and the simulated dialogue scenario.
We employ both API-based and open-source LLMs including GPT-4 (OpenAI, 2023), ChatGPT (OpenAI, 2022), and OpenChat (Wang et al., 2024a), to validate the effectiveness of our method. Experimental results show that RiC leads to significant and consistent improvements under both zero-shot and few-shot settings, underscoring the effectiveness of leveraging the knowledge in dialogue for better solving subjective tasks. The code is available at GitHub1.

heading: 2 Related Work
text: Subjective Tasks. Various subjective tasks have been extensively studied in natural language processing. We
1https://github.com/THUNLP-MT/RiC
show typical tasks in Table 1, including linguistic rhetoric, disambiguation, stance detection, and culturalrelated questions. Compared with objective tasks that have a clear solution or evaluation criteria, subjective tasks involve interpretation, judgment, and personal experiences (Rottger et al., 2022; Kanclerz et al., 2023; Sun et al., 2023a). Moreover, results in Table 1 show that the performance of LLMs on these tasks is around 30∼70 accuracy or F1 score, indicating that the tasks are indeed challenging and there is significant room for improvement even for the most advanced LLMs.
Chain-of-Thought Prompting. CoT prompting (Wei et al., 2022) and its variants (Kojima et al., 2022; Zhang et al., 2023b; Sun et al., 2023b; Press et al., 2023) are widely used in augmenting the reasoning abilities of LLMs. These methods attempt to enhance reasoning by incorporating additional rationales (Wang et al., 2022) or reasoning paths to augment contextual information, which has been shown effective for objective tasks such as commonsense reasoning (Talmor et al., 2019), open-domain question answering (Kwiatkowski et al., 2019), mathematical reasoning (Cobbe et al., 2021), and knowledge-intensive tasks which involves external knowledge (Wang et al., 2023b). Another line of work proposes reasoning through role-playing or expert modeling, aiming to answer questions or accomplish tasks through cooperation between roles or leveraging specified expert knowledge (Wang et al., 2023c; Xu et al., 2023b). However, we show in experiments that
the corresponding performance is still constrained by employing these methods in the aforementioned subjective tasks. Our main contribution is utilizing dialogue knowledge for subjective task reasoning for the first time. Prior to this, conventional reasoning methods often relied on strict reasoning pathways. Dialogue Generation and Reasoning. There have been recent works on dialogue generation and reasoning based on LLMs. Chae et al. (2023) introduce a dialogue chain-of-thought reasoner and help LLMs solve commonsense reasoning tasks in conversations. Xu et al. (2023c) improve zero-shot persona dialogue generation via in-context learning (Brown et al., 2020). Wang et al. (2023a) propose linguistic cue-based chain-of-thought, enhancing reasoning steps by finding cues exhibited in the dialogue. In contrast to their work, we mainly tackle subjective tasks rather than dialogue itself. Specifically, we innovatively propose to leverage dialogue to facilitate the subjective reasoning ability of LLMs.

heading: 3 Method
text: The overall pipeline of the proposed RiC can be divided into three steps. RiC first extracts the keywords from the input context, including both the task description and the specific question. Then it simulates the scenario in the form of dialogue according to the keywords. Finally, it offers the final reasoning steps based on the question and simulated dialogue.
Formally, given a subjective task with a description
T and a specific question Q, our goal is to let the LLM M solve the question Q. Step 1: Keywords Extraction. We first let LLM M extract the keywords K according to the description T and question Q. Specifically, we have
K = {k1, k2, ..., kn} = M(T ⊕ Q⊕ P1), (1)
where n (the number of keywords) is usually between 4∼5, ⊕ denotes concatenation operation. P1 is a prompt serving as a trigger sentence, for example, we can set P1 as “First, extract keywords from the question”. Step 2: Dialogue Simulation. Then, base on the keywords, we let LLM M construct a scenario S in the form of dialogue:
S = M(T ⊕ Q⊕K ⊕ P2), (2)
where P2 is a prompt for simulating the dialogue. For example, we can set P2 as “Then, according to the keywords, construct a scenario for the question in the form of dialogue”. For different subjective tasks, we show examples of simulated dialogues S in Figure 2. Step 3: Dialogue-Enhanced Reasoning. Finally, we take the original task description T , question Q, and the simulated dialogue S as the input, letting LLM M give the final response R:
R = M(T ⊕ Q⊕ S ⊕ P3), (3)
where P3 is the last prompt leading to the final answer which can be set as “Finally, according to the question and conversation, reason and give the final answer”.
Combine All Steps through Unified Prompting. In practice, we find that the three aforementioned steps can be combined and accomplished through a single prompt P . In this way, our method only requires inference once through the LLM to obtain the answer to the question:
P = P1 ⊕ P2 ⊕ P3, K,S,R = M(T ⊕ Q⊕ P), (4)
where an example of the unified prompt and response is shown in Table 2.

heading: 4 Experiments

heading: 4.1 Setups
text: Datasets. We evaluate the effectiveness of our method on twelve subjective reasoning datasets, which can be categorized into five types, including: Linguistic Rhetoric Tasks
• Metaphor (Mohler et al., 2016) provides a pair of sentences and aims to identify whether the metaphoric sentence is correctly interpreted.
• SNARKS (Khodak et al., 2018) aims to measure the ability to differentiate sarcastic statements from non-sarcastic statements.
• Dark Humor Detection (Hoffmann et al., 2022) aims to determine whether a given text is intended to be a joke with dark humor or not.
Disambiguation QA • Pronoun Resolution (Rudinger et al., 2018) aims
to clarify the meaning of a sentence with ambiguous pronouns to which thing refers.
• Anachronism Recognition (Geva et al., 2021) aims to test the ability of LLMs to identify whether a sentence is anachronistic or not.
Stance Detection • SEQ (Hendrycks et al., 2020) evaluates whether
LLMs are capable of identifying which simple ethical question aligns with human judgment.
• SemEval (Mohammad et al., 2016) propose a series of opinion analysis tasks. We follow Zhang et al. (2023a) and Wang et al. (2024b) to investigate LLM’s ability of stance detection for the target “Donald Trump” in tweets.
Cultural-Related Tasks • SocNorm (CH-Wang et al., 2023) is a dataset that
aims to align with social norms across American and Chinese culture.
• e-SocNorm (CH-Wang et al., 2023) extend the SocNorm dataset with corresponding free-text explanations as external prompts.
• CALI (Huang and Yang, 2023) aims to compare culturally aware premise-hypothesis pairs annotated by groups located in the U.S. and India.
Traditional Natural Language Inference
• Analytic Entailment (Srivastava et al., 2022) aims to identify whether the second sentence must be true given the meaning of the first sentence.
• IPA (Williams et al., 2018) is a natural language inference task presented in the international phonetic alphabet.
Detailed descriptions of datasets are given in appendix A. Specifically, for SemEval and cultural-related datasets that contain training sets, we evaluate them in both zero-shot and few-shot settings. For the other tasks, we use the corresponding test set from BigBench2 (Srivastava et al., 2022) in a zero-shot setting only.
Baselines. We compare our proposed RiC with various methods, taking into account both zero-shot (no demonstrations are provided) and few-shot settings (few demonstrations from the training set are provided for in-context learning). The baselines include: Zero-Shot Methods
• Direct Prompt (Brown et al., 2020) instructs LLM to answer the test question directly.
• Zero-Shot-CoT (Kojima et al., 2022) appends the prompt “Let’s think step by step” before reasoning.
• Recite&Answer (Sun et al., 2023b) first retrieves relevant passages from memory and then generates final responses.
Few-Shot Methods • In-Context Learning (ICL; Brown et al., 2020)
provides a few demonstrations including the ground-truth labels before giving the test question.
• Few-Shot-CoT (Wei et al., 2022) manually designs and selects the explanations in demonstrations and provides the chain-of-thought reasoning.
2https://github.com/google/BIG-bench/ tree/main/bigbench/benchmark_tasks/
• Auto-CoT (Zhang et al., 2023b) automatically selects demonstrations from training data based on semantic diversity for the test question.
• Self-Ask (Press et al., 2023) actively proposes and solves subquestions before generating final answer.
• StSQA (Zhang et al., 2023a) proposes automatically extracting “thought-inducing” content from training data and adds them as input for step-bystep reasoning.
• SPP (Wang et al., 2023c) proposes solo performance prompting by involving multi-turn collaboration with multi-persona during reasoning.
• ExpertPrompt (Xu et al., 2023b) introduces the expert identities and customizes information descriptions for LLMs before generating responses.
Models. For LLMs, we evaluate our method on both API-based models including GPT-4 (OpenAI, 2023) and ChatGPT (OpenAI, 2022), and open-source model OpenChat-7B (Wang et al., 2024a). In particular, we use the released API versions of gpt-4-0613 and gpt-3.5-turbo-1106 by OpenAI, and the opensource openchat-3.5 model released in Huggingface3. We set the decoding temperature as 0 to maintain the reproducibility of the responses generated by LLMs.

heading: 4.2 Zero-Shot Results
text: In Table 3, we show the main results of the baselines and our RiC method in zero-shot settings.
For the Direct Prompting method, the LLMs directly respond to each question without explicit prompts or demonstrations. On average, it gives the results of 61.29∼71.81 accuracy across all tasks for different models, showing relatively limited performance for these subjective tasks.
3https://huggingface.co/openchat/ openchat_3.5
By explicitly prompting LLMs to “Let’s think stepby-step” or “recite relevant passages then give answers” before reasoning, Zero-Shot-CoT and Recite&Answer will generate reasoning path or piece of passages according to the task and questions. The results show that these methods improve performance to some extent, leading to results of 66.04∼73.54 accuracy.
Regarding our RiC method, which involves simulated dialogues instead of reasoning paths or passages in memory, it has achieved the best results across twelve tasks. Compared to the second-best ones, our method improves absolutely by +4.51, +4.39, and +4.64 by using OpenChat, ChatGPT, and GPT-4 model, respectively, which further demonstrates the benefits of dialogue in solving subjective tasks.
Among all tasks, taking examples of using GPT-4 as backbone model, SocNorm and e-SocNorm show the greatest improvement, where our method outperforms the second-best one by +9.49 and +12.6 F1 score, respectively. These two datasets involve social norms in American and Chinese culture, which we suppose dialogue can provide relevant cultural background knowledge, thereby enhancing the performance. The improvement is also significant on analytic entailment (+10.00), anachronism recognition (+4.79) and sarcasm detection (+4.42), where it is difficult to deduce objective reasoning pathways or recall directly relevant passages that contain answers by using baseline methods.

heading: 4.3 Few-Shot Results
text: Table 4 shows the main results in few-shot settings, where we compared our method with more baselines.
In general, the vanilla ICL method gives the low-
est average results of 58.56∼63.09, which is only provided with labeled demonstrations without other contexts. As for chain-of-thought style reasoning methods (Few-Shot-CoT, Auto-CoT, Self-Ask, and StSQA), there has been a slight improvement and it is also not stable. For example, the improvement by using OpenChat and ChatGPT is generally around only 1∼3 accuracy, and StSQA even performs worse than ICL by using GPT-4. The reasoning can be that these methods are often limited to objective tasks such as mathematical and commonsense reasoning, and they have not been well validated in subjective tasks, though being provided with few demonstrations with manually-written or generated reasoning steps.
Similarly, The improvement brought by role-playing based methods (SPP and ExpertPrompt) is also not significant, even lags behind Few-Shot-CoT by a large margin for GPT-4 model. One possible reason could be that subjective tasks require an abstract and more variable range of knowledge, making it challenging to generalize and solve test questions using predefined roles in the demonstrations.
Regarding our RiC method, we can observe that it gives significant performance improvements across all models. Specifically, it outperforms the vanilla ICL method by +6.41, +8.33, and +8.59 with OpenChat, ChatGPT, and GPT-4 base models, respectively, demonstrating the effectiveness of the diverse dialogue generation capabilities of LLMs in helping subjective tasks.

heading: 5 Analyses and Discussions
text: In this section, we conduct a series of analyses to probe the reason behind the effectiveness of the RiC method. We first investigate the effectiveness of keywords and dialogue (§ 5.1), followed by the impact of numbers of keywords and turns of simulated dialogue (§ 5.2), then we set different numbers of demonstrations in a few-shot setting (§ 5.3) and compare the length of response for different methods (§ 5.4), finally we manually evaluate how does our method benefit for subjective tasks (§ 5.5).

heading: 5.1 Ablation Study
text: We first investigate the impact of keywords extraction and dialog simulation in our RiC method, results are shown in Table 5. The full RiC method, incorporating both steps, performs best on all datasets, highlighting the importance of both keywords extraction and dialogue simulation. Removing keywords extraction (RiC
w/o KE) generally leads to a performance drop by 1∼2 accuracy, showing that it is helpful for dialogue construction and subjective reasoning. Removing dialogue simulation (RiC w/o DS) further decreases the accuracy by around 1∼6 accuracy, indicating that the simulated dialogue indeed plays a crucial role in our method. Excluding both steps (RiC w/o KE&DS) leads to the worst performance, which degenerates to the vanilla reasoning way without explicit prompts. In summary, both the steps of keywords extraction and dialogue simulation are important and the best performance is achieved when both of them are utilized.

heading: 5.2 Number of Keywords and Turns of Dialogue
text: Our method does not specify the required number of keywords and turns of dialogue in P1 and P2 from Eq. 4. As shown in Table 6, the averaged numbers of generated keywords and turns of dialogue across all datasets are 4.10 and 1.24, respectively. We further specify the numbers in prompts or demonstrations in zero-shot and few-shot settings for further analysis.
Number of Keywords. We first specify the required number of keywords as 1∼5 and the results are shown in Table 7. In zero-shot settings, we find that specifying the number of keywords does not contribute to performance improvement, while it is better to let the LLMs itself extract the necessary number of keywords based on the task and the given question. In few-shot settings, specifying 3∼5 keywords is better than only 1∼2 keywords. Furthermore, setting 4∼5 different numbers of keywords instead of the fixed ones across different demonstrations can yield the best results. Turns of Dialogue. Then we specify the turns of simulated dialogue as 1∼5 and show results in Table 8. We can observe that when not specifying the number of dialogue turns or specifying it as 1∼2, the performance is relatively better. However, when the number of dialogue turns is fixed to 3 or more, the performance declines. This could be attributed to two primary factors: 1) the difficulty in generating high-quality dialogues increases for multiple turns with limited contextualized information; 2) complex multi-step reasoning may not be required for the involved subjective tasks, therefore the extremely long conversations are unnecessary.

heading: 5.3 Impact of Number of Demonstrations
text: In few-shot settings, we investigate the impact of the number of demonstrations for the baselines and our method. Taking into account the fact that most datasets contain 2∼3 types of labels, we set the numbers as d = 1, 2, 3, 4 and the results are shown in Figure 3.
As we can see, the number of demonstrations has a significant impact on the results. For example, when
d = 1 or 2, the performance is generally low with limited examples. Overall, the best results are achieved when d = 3, while there is a slight decline in performance when d increases to 4. These reflect the instability and variance of in-context learning (Zhao et al., 2021; Xu et al., 2023a).
Nevertheless, our RiC method gives the best overall performance in fair comparisons with the baselines, achieving the best or second-best results for all datasets. In practice, we found that selecting one example per label can generally achieve better and more stable performance, and there is no need to set d too large (e.g., 5 or more).

heading: 5.4 Comparison of Inference Cost
text: The length of responses also incurs certain time and monetary costs, especially for nowadays LLMs. We compare the performance and average generated tokens by GPT-4 model of baselines and our RiC method, the results are shown in Figure 4.
Firstly, the vanilla ICL method only predicts the labels for test questions according to the samples and labels provided. Although the length of the response is the shortest (less than 10 tokens on average), the general accuracy is limited. Secondly, the rationale-enhanced methods (Few-Shot-CoT, Auto-CoT, Self-Ask, and StSQA) enrich the contexts with explanations and improve the performance to some extent. Thirdly, ExpertPrompt and SPP increases the length of response
(100∼300 tokens) due to role-playing, but the performance does not improve consistently. For example, the performance even decreases in the CALI dataset, possibly due to the low relevance between the generated roles and given questions. Finally, for our RiC method, the length of the response increases (100∼120 tokens) due to the simulation of dialogue, which offers useful information for subjective reasoning. Overall, the results show that our method outperforms the above baselines substantially and consistently.

heading: 5.5 Human Evaluation
text: We attempt to analyze further how our method contributes to helping the reasoning of LLMs. In particular, we manually categorize the knowledge gained from the dialogue into different classes by voting, conducted by three individuals, as approximately represented in Figure 5. These categories were derived through the individuals’ induction and summarization.
For example, we find that the simulated dialogue can provide more context with sentiment tendencies, causal relationships of the events occurring, explanation or
viewpoints, professional knowledge through simulating relevant professional roles, or providing background knowledge about events, culture, and characters, etc., which can help solve different involved subjective tasks, as examples in Table 9 (see appendix B for more cases).

heading: 6 Conclusion
text: We introduce RiC (i.e., Reasoning in Conversation), a tuning-free method to enhance the ability of LLMs to solve subjective tasks through dialogue simulation. The core motivation of the proposed RiC is to better leverage the useful information from human conversation based on the advanced dialogue generation ability of current LLMs. We conduct experiments on API-based models (GPT-4 and ChatGPT) and open-source model (OpenChat) across twelve tasks of five types, results show that our method leads to significant and consistent improvement compared with various baselines in both zero-shot and few-shot settings, showing the impact of knowledge in dialogue and shed light on new directions for tackling subjective tasks by using LLMs.
Limitations Firstly, our proposed method focuses on improving performance on subjective tasks in zero-shot or few-shot settings, which relies on the dialogue generation and understanding capabilities of current LLMs, making it more suitable for general-purpose models. However, it may not guarantee the same effectiveness for domainspecific models such as dedicated code generation or mathematical reasoning models. Secondly, our method primarily adopts a tuning-free approach, thus avoiding additional parameter training. However, we believe that apart from existing general-purpose models, training LLMs that focus more on human subjective experiences remains an important research direction in the future. Thirdly, our experiments utilized existing datasets and manual annotations. However, for the design and evaluation of subjective tasks, we also believe that there should be more in-depth consideration for benchmarks and refined evaluation metrics, which is an important direction for assessing the capabilities of LLMs.
Ethics Statement In this paper, we utilize publicly available and widely used datasets for evaluation, including stance detection, sarcasm detection, cultural comparison, etc. We also use LLMs to generate corresponding responses. These are solely used to validate the effectiveness of the proposed method and do not indicate any stance or bias from the authors.

heading: Acknowledgments
text: This work is supported by the National Natural Science Foundation of China (No. 61925601, 62276152, 62306161).

heading: A Detailed Descriptions of Datasets
text: The resources and statistics of the datasets are shown in Table 10, and we provide detailed descriptions of each task as follows.
Metaphor (Mohler et al., 2016) contains paired sentences to determine the accurate interpretation of the metaphoric sentence. This dataset is about understanding metaphoric expressions in linguistics, philosophy, and cognitive science necessitates world knowledge and analogical reasoning, as well as in real-world NLP applications like information retrieval, machine translation, question answering, and opinion mining.
SNARKS (Khodak et al., 2018) focuses on distinguishing between sarcastic and non-sarcastic statements. Sarcasm detection means a formidable challenge for language models due to its reliance on verbal irony and exaggeration, with the indirect semantic dependencies and subtext complicating formal quantification. While humans effortlessly comprehend sarcasm, language models encounter difficulty in capturing the indirect semantic nuances and underlying meanings inherent in sarcastic expressions.
Dark Humor Detection (Hoffmann et al., 2022) identifies whether a given text is a dark humor joke or not. Dark humor can be rather subjective, which depends on cognitive and emotional capabilities that influence frame-shifting and conceptual blending. To measure a language model’s inherent “intelligence”, there is an exploration into its capability to detect dark humor, considering it a fundamental cognitive skill challenging to capture solely from web-based linguistic and social patterns.
Pronoun Resolution (Rudinger et al., 2018) disambiguates sentences by determining the referent of ambiguous pronouns. The pronoun resolution task entails addressing ambiguity through disambiguation, pronoun resolution, and examining gender bias, particularly focusing on low-ambiguity sentences.
Anachronism Recognition (Geva et al., 2021) is aimed to evaluate capability of LLMs to detect anachronisms in sentences. Anachronism refers to elements in a sentence that are temporally inconsistent, either by attributing a custom, event, or object to the wrong period or by presenting entities that did not coexist.
SEQ (Hendrycks et al., 2020) assesses the capability of LLMs to match simple ethical questions with human
judgment. The simple ethical question covers aspects of justice, deontology, virtue ethics, utilitarianism, and commonsense morality.
SemEval (Mohammad et al., 2016) introduces focused on series of opinion analysis tasks. Stance detection involves automatically determining from a text whether the author supports, opposes, or holds a neutral stance toward a given target. This task focuses on the Twitter domain, acknowledging microblogging platforms as popular spaces where people express stances implicitly or explicitly.
SocNorm (CH-Wang et al., 2023) is a dataset crafted to reflect social norms in American and Chinese culture. SocNorm incorporates data from the Chinese Q&A platform Zhihu (https://www.zhihu.com) and utilizes the existing Social Chemistry dataset (Forbes et al., 2020) as a means of comparing cultural dimensions.
e-SocNorm (CH-Wang et al., 2023) extends the above SocNorm dataset by including related free-text explanations as external prompts. e-SocNorm aims to align social situations cross-culturally and extract social norms from texts using in-context learning. Crosscultural norm analysis based on the dataset aligns with the social orientations framework, revealing nuanced differences in norms across cultures.
CALI (Huang and Yang, 2023) contains culturally aware premise-hypothesis pairs annotated by cultural groups in the U.S. and India. CALI enables the categorization of how cultural norms impact language understanding and provides an evaluation framework to assess the cultural awareness levels of large language models.
Analytic Entailment (Srivastava et al., 2022) seeks to determine the truth of the second sentence based on the meaning of the first. Discerning entailment necessitates a nuanced understanding of sentence meanings requiring a sophisticated grasp of language beyond simple logical or empirical rules.
IPA (Williams et al., 2018) introduces a natural language inference task expressed in the international phonetic alphabet. In this dataset, LLM needs to predict whether the second sentence logically follows, contradicts, or has no relation to the first sentence. Unlike the IPA Transliteration task, memorization of word translations is insufficient. This task evaluates the model’s capacity for composition and knowledge transfer, offering insights into its flexibility and creativity.

heading: B More Examples of Simulated Dialogue
text: We show more cases of different knowledge types provided by our simulated dialogue in the following Tables 11, 12, 13, 14, 15, 16, and 17.

heading: C Performance on LLaMA2 Model
text: To validate the effectiveness of our proposed method, we conduct experiments using open-source LLaMA27B-Chat. The results for the SocNorm and e-SocNorm datasets are shown in Table 18.

heading: D Compare with Dialogue Generation
text: Our method uses LLM to extract keywords, simulate dialogs, and perform dialog reasoning. Compared with other methods, the main differences are listed in Table 19. Firstly, our objective is not dialog reasoning but rather solving subjective tasks. Additionally, the objective of KPT (Zhu et al., 2023) and KRLS (Yu et al., 2023) is dialogue generation rather than solving reasoning tasks.

heading: Future Work
text: However, we believe that apart from existing general-purpose models, training LLMs that focus more on human subjective experiences remains an important research direction in the future. Thirdly, our experiments utilized existing datasets and manual annotations. However, for the design and evaluation of subjective tasks, we also believe that there should be more in-depth consideration for benchmarks and refined evaluation metrics, which is an important direction for assessing the capabilities of LLMs