# Human Annotated Dialogues Dataset for Natural Conversational Agents
## 7 Conclusions and Future Work  
This paper presents the human annotated movie dialogue dataset (HUMOD) for research to develop benchmark metrics for comparison of different models on human scores and generated replies. The detailed description of the dataset construction and statistics is provided. The availability of the HUMOD dataset opens up various possibilities for research and development of complex dialogue systems for real-life applications. Different replies for the same context as well as dialogue ratings can be used to develop a metric to compare methods such as BLEU. Another interesting usage of a unique diverse reply is to train generative models that generate diverse dialogue replies which may make dialogue managers more human-like in real-life applications. We present the preliminary results to provide baselines with supervised and word-overlap metrics. HAN provides better results in comparison to word-overlap metrics since these approaches do not use any context. The BERT model outperforms the HAN model and provides good correlation on the human dialogue score; however, it is harder to train and requires fine-tuning for each different dataset.  
In future work, we will work on a metric that uses dialogue contexts and replies to produce a robust score without any training. We will also focus on generative models that can leverage all possible replies for a given context rather than a single context-reply pair. Another interesting approach is to generate diverse replies from single context-reply pairs and to use such created data as augmentation for dialogue managers. All this is relevant for the upcoming research stream of explainable AI, where NLU [39] plays a particular role, e.g., in the generation of human-understandable explanations [40; 41], where it is very useful for the development of future human-AI interfaces.  
Conceptualization, E.M. and D.S.; methodology, E.M. and D.S.; software, E.M. and D.S.; validation, E.M. and D.S.; formal analysis, E.M. and D.S.; investigation, E.M. and D.S.; resources, E.M. and D.S.; data curation, E.M. and D.S.; writing--original draft preparation, E.M. and D.S.; writing--review and editing, E.M., D.S., A.H. and M.G.; visualization, E.M. and D.S.; supervision, S.H., J.K., A.H. and M.G.; project administration, S.H. and J.K.; funding acquisition, S.H., J.K. and A.H. All authors have read and agreed to the published version of the manuscript.  
This work has been funded by the European Union Horizon2020 MSCA ITN ACROSSING project (GA no. 616757).  
The authors would like to thank the members of the ACROSSING project's consortium for their valuable inputs. We are also thankful to all the participants of the dialogue dataset survey.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Dialogue Turn** & **Correlation (_p_-Value)** \\ \hline
2-turn & 0.52 (\(<\)0.001) \\
3-turn & 0.58 (\(<\)0.001) \\
4-turn & 0.67 (\(<\)0.001) \\
5-turn & 0.61 (\(<\)0.001) \\
6-turn & 0.66 (\(<\)0.001) \\
7-turn & 0.59 (\(<\)0.001) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Correlation of BERT against different turns with human scores.