# Human Annotated Dialogues Dataset for Natural Conversational Agents
## 3 Human Annotated Movie Dialogues Dataset (HUMOD)  
### Dataset Design and Collection  
We developed a website for data collection and used the crowdsourcing platform Amazon Mechanical Turk (AMT) (Full anonymity of the users were maintained and no ethical concerns were raised by the host institution) for collecting human annotations on selected Cornell movie dialogues. The dataset creation is performed in the following steps: First, we used simple random sampling to select 4750 dialogue contexts from the Cornell movie dataset in which each dialogue is divided into utterances. Since the dataset contains human-human dialogues, we assigned each utterance alternatively as Speaker 1 and Speaker 2 as shown in Figure 1. The dataset consists of multi-turn dialogues ranging from two to the maximum of seven turns. Then, each dialogue history is appended with an actual reply from the movie dialogue dataset and also with a candidate negative reply which is sampled uniformly from a possible set of replies, thus making the dataset size of 9500. At the beginning, basic demographics of the users were asked including Age range, Gender, and English proficiency level. These demographic variables are collected so that the dataset can be tailored as per the needs and preferences of researchers to develop conversational agents for different demographics. In the next step, each AMT user was given a task of 20 sampled dialogue conversations and was asked to rate the last reply of the dialogue conversation from 1-5, (where 1: Irrelevant reply; 2: Weakly relevant reply; 3: Neutral; 4: Fairly relevant reply; 5: Highly relevant reply) according to the dialog context and provide their relevant reply in replacement to the last reply. The total number of completed tasks were 1425. Each dialogue-reply pair is rated by three different users, such that, for each dialog context, there are six unique responses, resulting in a total dataset size of 28,500. In order to maintain the high quality responses in the dataset, we selected responses of only those AMT users who have more than 80% hit approval rate. Each task was evaluated by a human expert and low quality tasks consisting of short or generic replies, erroneous replies consisting of gibberish text (user's reply for given dialogue context), user's replies in different languages (other than English) and the same ratings given to every dialogue-reply pairs were discarded. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & **\# of Dialogues** & **Human Annotated** & **Diverse Replies** & **Description** \\ \hline Cornell Movie-Dialogue [7] & 220K & No & No & Conversation from the movie scripts. \\ Movie-DiC [26] & 132K & No & No & American movie scripts. \\ Movie-Triples [27] & 245K & No & No & Dialogues of three turns between two interlocutors. \\ OpenSubtitles [25] & 36M & No & No & Movie subtitles which are not speaker-aligned. \\  & & & & Conversation from the movie scripts with 1 to 5 ratings and six diverse replies from humans. \\ \hline \hline \end{tabular}
\end{table} 
Table 1: A comparison of existing movie dialogue datasets with the HUMOD dataset. (*) denotes that the HUMOD dataset can be extended by replacing the diverse replies with the original reply as explained in Figure 4.  
Figures 2 and 3 present the screen shots from the website, showing how dialogue appears to the users with positive reply and candidate negative reply for the particular dialogue context. Since each task contains 20 randomly selected dialogue context-reply pairs, the chances of the same dialogue context appearing with both positive and candidate negative replies to the same user is very minimal. The six diverse replies for each dialogue context in the proposed dataset enable the extendability feature of HUMOD dataset. An example with reference to a dialogue context of how the dataset can be extended using human replies is shown in Figure 4. The seven-turn dialogue of the dataset is divided into two parts.