# Dialog CIN: Contextual Inference Networks for Emotional Dialogue Generation
## 4 Proposed Dataset
### Data Processing  
In this study, we conducted data cleaning on the collected raw dialogue corpus. Specifically, we applied filters to remove text segments that exceeded 512 tokens in length. To eliminate dialogues containing offensive language, we employed a method that involved creating a profanity word library and combining it with regular expression matching and manual review for the purpose of cleaning. Additionally, we utilized regular expression matching to eliminate special symbols, such as \"Y,\" \"@,\" \"$,\" and others, from the dialogues. This was carried out to facilitate subsequent data analysis and processing. Furthermore, considering the research objectives and dataset characteristics, we selectively removed dialogue texts involving three or more speakers. This choice was guided by the observation that two-party dialogues are more prevalent and easier to model, while dialogues with three or more speakers may introduce additional complexities in terms of interactions and language dynamics that differ from two-party conversations. By maintaining consistency and control within the dataset, we ensured a more focused and coherent analysis. Therefore, we retained data exclusively from two-party dialogues in order to maintain the desired research scope and objectives. We conducted statistical analysis on the dialogues to gather insights into their structure and content. Specifically, we calculated the average number of utterances and tokens in the dialogues and present the detailed results in Table 2. In this context, an \"utterance\" refers to an individual linguistic unit within a dialogue, typically representing a single speaker's turn, be it a user or a system's utterance. Subsequently, we manually annotated the data based on the emotional, syntactic, and thematic features of the dialogues. To guarantee the reliability and accuracy of the annotated data, a meticulous labeling process was implemented, involving two annotators and one reviewer, all with backgrounds in natural language-processing research and equipped with professional annotation training and guidance. The annotators were entrusted with the task of labeling the data, adhering to predetermined standards and guidelines that included the use of provided feature labels. Simultaneously, the reviewer assumed the responsibility of assessing the quality of the labeled data. This labeling process followed an independent approach, aiming to minimize subjective bias and promote consistency across annotations. In instances where discrepancies emerged between the annotators, discussions were facilitated between the annotators and the reviewer, ultimately leading to a collaborative decision. By employ \begin{table}
\begin{tabular}{l c} \hline \hline
**Split** & **Quantity** \\ \hline Train & 8039 \\ Dev & 1026 \\ Test & 980 \\ Total & 10,045 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Dialogue dataset statistics. ing this rigorous annotation workflow, we ensured the high quality and credibility of our dataset, establishing a solid foundation for subsequent research and analysis in the field of natural language processing. Furthermore, we conducted an analysis of the number of utterances in each dialogue within the LifeDialog dataset, as depicted in Figure 4. It was observed that the dialogues in the dataset tend to conclude after a reasonable number of dialogue turns, with an average of approximately seven turns per dialogue. This characteristic makes the dataset well suited for training dialogue models. By examining the number of utterances in each dialogue, valuable insights can be gained regarding the length and complexity of the conversations. This information plays a vital role in enabling the model to handle interactions of varying lengths and hierarchies. Some dialogues may exhibit a relatively simplistic structure, consisting mainly of direct question-answer or simple instruction exchanges.