SlowGuess/NAACL_2021_OCR huggingface dataset:
Proceedings of the Second Workshop on Statistical Machine Translation, pages 228-231, Prague, Czech Republic. Association for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016b. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1192-1202, Austin, Texas. Association for Computational Linguistics.
Margaret Li, Jason Weston, and Stephen Roller. 2019. Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Shikib Mehri and Maxine Eskenazi. 2020a. Unsupervised evaluation of interactive dialog with DialogGPT. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 225-235, 1st virtual meeting. Association for Computational Linguistics.
Shikib Mehri and Maxine Eskenazi. 2020b. USR: An unsupervised and reference free evaluation metric for dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681-707, Online. Association for Computational Linguistics.
Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017. ParlAI: A dialog research software platform. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 79-84, Copenhagen, Denmark. Association for Computational Linguistics.
Myle Ott, Michael Auli, David Grangier, and Marc'Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3956-3965, Stockholm, Sweden. PMLR.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Mohammed R. H. Qwaider, Abed Alhakim Freihat, and Fausto Giunchiglia. 2017. TrentoTeam at SemEval-2017 task 3: An application of Grice maxims in ranking community question answers. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 271-274, Vancouver, Canada. Association for Computational Linguistics.
Vasile Rus and Mihai Lintean. 2012. A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 157-162, Montreal, Canada. Association for Computational Linguistics.
João Sedoc, Daphne Ippolito, Arun Kirubarajan, Jai Thirani, Lyle Ungar, and Chris Callison-Burch. 2019. ChatEval: A tool for chatbot evaluation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 60-65, Minneapolis, Minnesota. Association for Computational Linguistics.
Iulian V. Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16, page 3776-3783. AAAI Press.
Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. CoRR, abs/1706.09799.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818-2826.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierrick Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
We sweep SentencePiece (Kudo and Richardson, 2018) vocabulary size (1k,4k,8k,16k), learning rate (1e-2, 1e-3, 1e-4), dropout (0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6), attention & ReLU dropout (0.0, 0.1, 0.2, 0.3, 0.4), and label smoothing (0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8).
A.2 Standard Automatic Metrics
In § A.7 we report the full automatic evaluation results of the 14 metrics across both the single reference and multi-reference evaluation from the multi-reference automatic evaluation framework for DailyDialog released by Gupta et al. (2019),[15] which is computed using NLG-EVAL[16] (Sharma et al., 2017). This includes word-overlap metrics: BLEU (Papineni et al., 2002), METEOR (Lavie and Agarwal, 2007), and ROUGE-L (Lin, 2004) as well as embedding based metrics: SkipThought (Kiros et al., 2015), embedding average (Forgues et al., 2014), vector extrema, and Greedy Matching (Rus and Lintean, 2012). For reading ease, we report metrics scaled between 0 and 100 rather than 0 and 1. See chateval.org/RUQ for automatic metrics on the full hyper parameter sweep.
A.3 Lexical Diversity
The Gricean maxims focus on ensuring cooperation between speakers, but there is more to a conversation than cooperation—especially in an open ended conversation that might be had with a chatbot. This is where additional desiderata may come in to play, such as interestingness. One (indirect) automatic way of measuring interestingness is lexical diversity (Halliday, 1989; Laufer and Nation, 1995), by computing the n-gram type-token ratio (Li et al., 2016a). We use the same spaCy $^{17}$  tokenization used in the automatic evaluation scripts ( $\S$  A.2). $^{18}$
A.4 Pointwise Human Evaluation
We presented Amazon Mechanical Turk workers the task with 4 prompts and all system responses along with a human reference. The annotators had a maximum time allotted of 20 minutes. Our criteria
for inclusion were over 500 approved HITs, an approval rate over  $98\%$ , and location set to US. Each HIT was paid $0.15 with an overlap of 4 annotators per HIT.[19] A screenshot of the HIT is in Figure 2.
A.5 Head to Head Human Evaluation
In addition to the point-wise evaluation, we also test head-to-head pairwise performance on the evaluation set of 480 unique prompt/response pairs, as shown in § A.5. Models trained on the DailyDialog data outperform the filtered models, but there is no clear preference between base and best models.
A.6 Dialog Models
We train Transformer conditional language models in FAIRSEQ using parameters from the FLORES $^{20}$  benchmark for low-resource machine translation (Guzmán et al., 2019).
We use a 5-layer encoder and decoder, 512 dimensional embeddings, and 2 encoder and decoder attention heads. We regularize with 0.2 label smoothing, and 0.4 dropout. We optimize using Adam with a learning rate of  $10^{-3}$ . We train 100 epochs, and select the best checkpoint based on validation set perplexity. We generate with a beam size of 10, and no length penalty.
Figure 3 shows the train command.
We train and evaluate on the DailyDialog corpus (Li et al., 2017), as released by ParlAI (Miller et al., 2017).<sup>21</sup>
A.7 Full Automatic Results
Table 6 shows the hyperparameters for each system. Table 7 and Table 8 show the evaluation against the multiple references for the word based and embedding based metrics. Table 9 and Table 10 show the evaluation against the original single reference for the word based and embedding based metrics. Table 11 shows the lexical diversity, and Table 12 shows the RUQ sores.
<sup>19</sup>We aimed to compensate the crowdworkers fairly ($ 8 per hour) and did this by annotating a set of data ourselves to estimate the timing of the task
20https://github.com/   
facebookresearch/flores/tree/   
5696dd4ef07e29977d5690d2539513a4ef2fe7f0   
21https://github.com/   
facebookresearch/ParlAI/tree/   
1e905fec8ef4876a07305f19c3bbae633e8b33af
15 github.com/prakharguptaz/multirefeval  
16 github.com/Maluuba/nlg-eval  
17 spacy.io  
18 github.com/Maluuba/nlg-eval
Please Note
- You have to be an English Native Speaker.
- You have to complete the ratings for all responses. All fields are required.
This is a linguistic experiment performed at If you have any question about this study, feel free to contact Participation in this research is voluntary. You have the right to withdraw from the experiment at any time. The collected data will be used for research purposes only. Personal data will be kept confidential and will not be shared with third parties.
In this task you will read a turn of a conversation. For each conversation you will see possible responses to the last turn generated by a computer program. The programs attempt to generate responses that are relevant, while also making an interesting contribution to the conversation.
For each prompt, you will read several system possible responses, and judge each response on its appropriateness on a 1 to 5 scale from not at all appropriate to extremely appropriate. Appropriateness is defined as follows:
1. Fluency: Flows naturally and sounds like what an experienced speaker/writer of the language might say.
2. Coherence: Facts and topics are consistent, i.e. non-contradictory with previous parts of the conversation.
3. Interestingness: Contains rich, non-repetitious, and interesting information that adds to the conversation.
Example: Below we show a prompt and several possible responses, along with suggested scores for each response.
Prompt: What do you do for work?
Responses Suggested scores I doctor. Fluency: 1, Coherence: 5, Interestingness: 5. This does not sound like a native speaker's response, but there are no contradictions and it answers the question. I like pancakes. Fluency: 5, Coherence: 1, Interestingness: 3. This is grammatical but not coherent as it does not answer the question and only some information was added to the conversation. I work. Fluency: 4, Coherence: 4, Interestingness: 1. This is grammatical and mostly coherent, but no information is added to the conversation.. I work as a teacher. Fluency: 5, Coherence: 5, Interestingness: 5. An excellent on-topic response that does not contradict any previous statements.
Note: ignore things like "i ' m" this should be read as "I'm" and not docked points for grammar. There are ATTENTION CHECKS please make sure you select the number.
$\{\%$  for prompt in prompts  $\%\}$ $\{\%$  set promptloop  $=$  loop  $\%$  1. Prompt:  $\{\%$  for turn in prompt.promt  $\%\}$
{turn}
Figure 2: Instructions for AMT task.
M1	M2	M1	M2	tie
EF-BEST	DD-BEST	38.5%	42.4%	19.1%
EF-BASE	EF-BEST	34.4%	34.0%	31.7%
EF-BASE	DD-BASE	37.0%	41.6%	21.4%
DD-BASE	DD-BEST	36.3%	36.5%	27.2%
Table 5: Head to head comparison between various systems. Models trained on the DailyDialog data outperform the filtered models, but there is no clear preference between base and best models.
Figure 3: Training command.
Data	Params	bpe	lr	dropout	otherdropout	labelsmooth
DD	BASE	4	0.001	0.4	0.2	0.2
DD	BEST	4	0.001	0.0	0.1	0.4
EF	BASE	4	0.001	0.4	0.2	0.2
EF	BEST	2	0.001	0.0	0.1	0.2
Table 6: Hyperparameters for each of the four models we consider.
Data	Params	Average Max Sentence BLEU	Corpus BLEU	METEOR	ROUGE
BLEU1	BLEU2	BLEU3	BLEU4	BLEU1	BLEU2	BLEU3	BLEU4
DD	BASE	27.8	14.7	10.3	7.9	48.1	25.6	16.2	11.2	12.7	34.3
DD	BEST	33.9	21.9	17.7	15.3	53.9	36.1	28.9	25.1	17.8	39.7
EF	BASE	27.8	14.0	9.4	7.0	46.9	24.1	14.6	9.8	13.2	33.4
EF	BEST	31.7	19.1	14.9	12.7	51.0	32.8	25.5	21.8	16.9	37.2
Table 7: Word-overlap based metrics on multiple references.
Data	Params	Cosine Similarity	GreedyMatching
SkipThought	Embed. Avg.	VectorExtrema
DD	BASE	72.4	90.8	62.9	77.2
DD	BEST	73.8	92.2	65.4	79.3
EF	BASE	71.9	91.2	62.2	77.0
EF	BEST	72.8	91.6	62.7	77.9
Table 8: Embedding based metrics on multiple references.
Data	Params	Average Max Sentence BLEU	Corpus BLEU	METEOR	ROUGE
BLEU1	BLEU2	BLEU3	BLEU4	BLEU1	BLEU2	BLEU3	BLEU4
DD	BASE	15.3	7.6	5.6	4.5	12.9	6.3	4.1	3.0	6.7	20.6
DD	BEST	24.3	16.7	14.3	12.8	23.2	16.7	14.2	12.9	11.9	29.2
EF	BASE	15.9	7.4	5.2	4.1	15.8	7.5	4.7	3.3	7.2	20.4
EF	BEST	22.1	14.0	11.8	10.5	22.9	15.8	13.2	11.8	11.1	26.6
Table 9: Word-overlap based metrics on the single reference test set.
Data	Params	Cosine Similarity	GreedyMatching
SkipThought	Embed. Avg.	VectorExtrema
DD	BASE	65.3	86.3	50.6	71.3
DD	BEST	68.2	88.5	54.7	74.6
EF	BASE	64.9	86.9	50.2	71.3
EF	BEST	67.0	87.7	52.3	73.1
Table 10: Embedding based metrics on the single reference test set.
Data	Params	1-grams	2-grams	3-grams
DD	BASE	2.4	10.3	18.8
DD	BEST	3.5	18.0	35.5
EF	BASE	2.3	10.7	20.1
EF	BEST	3.8	18.3	34.6
Table 11: Type/Token ratios.
Data	Params	RUQ-train	RUQ-test
DD	BASE	28.5	12.2
DD	BEST	95.3	35.7
EF	BASE	37.9	15.5
EF	BEST	89.2	30.7
Table 12: RUQ scores on the train and test data.