SlowGuess/ACL_2020_OCR huggingface dataset:
2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2122-2132.
Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic Turing test: Learning to evaluate dialogue responses. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1116-1126.
Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The Ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 285-294.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311-318.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 2227-2237.
Vasile Rus and Mihai Lintean. 2012. A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 157-162.
Ananya Sai, Mithun Das Gupta, Mitesh M. Khapra, and Mukundhan Srinivasan. 2019. Re-evaluating adem: A deeper look at scoring dialogue responses. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, page 6220-6227.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30 (NeurIPS), pages 5998-6008.
Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. 2017. Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 496-505.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 654-664.
A Methods to Retrieve False Candidates
To make false candidates in each pool diverse, we use two retrieval methods: lexical retrieval and embedding-based retrieval. We use Lucene for lexical retrieval, and cosine similarity of sentence vectors for embedding-based retrieval. Sentence vectors are SIF (Arora et al., 2017) weighted average of ELMo word vectors (Peters et al., 2018).
B Detailed Model Settings in the Experiments
We trained 10 different response generation systems to be ranked in the experiments. We trained them with different architectures or settings. The common settings for the model training are shown in Table 5 and the hyper-parameters of each the models are shown in Table 6.
Vocab size	16,000
Batch size	6,000 tokens
Loss	cross entropy
Learning rate	1e-4 (fixed)
Optimizer	Adam
Table 5: Common settings for the model training in the experiments.
No.	Architecture	Enc/Dec layers	Enc/Dec embed dim	Enc/Dec hidden dim
1	GRU	1 / 1	256 / 256	256 / 256
2	GRU	1 / 1	512 / 512	512 / 512
3	GRU	2 / 2	256 / 256	256 / 256
4	GRU	2 / 2	512 / 512	512 / 512
5	LSTM	1 / 1	256 / 256	256 / 256
6	LSTM	1 / 1	512 / 512	512 / 512
7	LSTM	2 / 2	512 / 512	512 / 512
No.	Architecture	Enc/Dec layers	Enc/Dec embed dim	Enc/Dec attention heads
8	Transformer	2 / 2	256 / 256	4 / 4
9	Transformer	2 / 2	512 / 512	4 / 4
10	Transformer	4 / 4	256 / 256	4 / 4
C Labels for False Candidates
As a case study, we designed a set of error labels, each of which indicates why the false candidate is false. To confirm whether we can assign the labels to the false candidates collected by our test set construction method, We assigned the labels
to 50 false candidates from our test set. We could eventually assign the labels to 22 candidates. The types of our error labels and the breakdown are listed in Table 7. The examples of false candidates (CHOSEN) corresponded to the error labels are shown in Table 4 (for labeled “Responses that have wrong subjects”), Table 8, Table 9, and Table 10.
Table 6: Hyper-parameters of each model in the experiments.
Error label	Count
Inconsistent responses with the context	8
Responses that have insufficient information	4
Responses that have wrong subjects	9
Responses with wrong tense	1
Table 7: Error labels and the breakdown of the the assigned labels.
Context:
A: 911 emergency. What is the problem?
B: I would like to report a break-in.
A: When was this break-in?
Candidates:
Ground-Truth: I believe it happened last night.
CHOSEN: I thought that would happen last night.
Table 8: Example of a false candidate labeled "Inconsistent responses with the context."
Context:
A: What's the matter with you, Paul?
B: I'm not feeling well. I think I'm having a cold.
A: Looks like it. You need to drink a lot of water and take a good rest.
Candidates:
Ground-Truth: Yeah, I will.
CHOSEN: Yeah, yeah, yeah, I...
Table 9: Example of a false candidate labeled "Responses that have insufficient information."
Context:
A: Hi, charlie, are you busy this evening?
B: Sorry, I’m afraid that I’ve got plans tonight.
A: What are you doing?
Candidates:
Ground-Truth: I’m going to my parents’house
for my father’s birthday.
CHOSEN: We were at my sister’s house for my
nephew’s birthday by 2 p.m.
Table 10: Example of a false candidate labeled "Responses with wrong tense."
<sup>7</sup>https://lucene.apache.org/