SlowGuess/ACL_2020_OCR huggingface dataset:
MT + FT	All Tasks MT
PPL	BLEU	ROUGE	F1	PPL	BLEU	ROUGE	F1
4	1	2	L	4	1	2	L
ConvAI2	11.1	6.6	37.0	11.6	31.8	21.6	10.8	5.5	39.4	12.5	33.7	21.7
DailyDialog	10.4	4.0	35.6	10.0	30.8	18.2	12.0	2.9	33.9	8.7	29.2	16.2
Wiz. of Wikipedia	8.3	21.5	55.3	28.4	44.9	38.4	8.4	21.0	53.2	28.0	45.4	38.4
Empathetic Dialog	11.4	3.5	38.0	9.5	32.3	19.5	11.5	3.7	37.2	8.9	31.4	19.3
Cornell Movie	20.2	2.5	29.5	6.7	25.7	12.4	22.2	2.1	29.1	6.5	25.6	11.9
LIGHT	18.9	2.6	30.8	5.8	24.8	16.2	19.3	2.4	30.5	5.6	24.6	16.1
ELI5	21.0	3.7	38.6	7.2	22.1	23.1	24.9	3.2	35.2	6.3	20.5	21.3
Ubuntu	17.1	2.5	27.0	5.0	22.8	12.7	23.1	3.7	26.0	4.3	22.0	12.1
Twitter	30.7	3.2	16.5	3.3	14.3	9.9	38.2	2.6	19.4	3.3	16.5	9.8
pushshift.io Reddit	25.6	2.1	24.1	4.5	18.7	13.6	27.8	1.6	23.4	4.2	18.1	13.5
Image Chat	18.8	2.4	30.1	5.7	26.0	13.0	22.3	2.1	28.4	4.9	24.6	12.9
IGC	11.9	8.6	65.0	34.1	60.5	38.4	12.0	8.0	61.3	28.3	56.8	41.4
dodecaScore	17.1	5.3	35.6	11.0	29.6	19.8	19.4	4.9	34.8	10.1	29.0	19.6
Table 8: Test performance for various metrics on the dodecaDialogue tasks comparing our multi-task and multitask + fine-tuned methods.
MT + FT	All Tasks MT
PPL	BLEU	ROUGE	F1	PPL	BLEU	ROUGE	F1
4	1	2	L	4	1	2	L
ConvAI2	11.2	5.7	36.7	10.9	31.6	21.1	11.3	5.3	38.7	11.6	32.9	21.3
DailyDialog	10.2	4.4	36.8	10.7	32	18.8	11.8	3.1	34.8	9.3	30.2	17.1
Wiz. of Wikipedia	8.5	20.8	54.9	28.0	44.8	37.9	8.7	20.2	55.2	28.2	45.0	37.9
Empathetic Dialog	11.1	3.6	38.6	9.8	32.7	19.7	11.2	3.5	37.5	9.1	31.8	19.3
Cornell Movie	19.8	2.5	29.3	6.7	25.6	12.3	21.9	2.1	29.0	6.5	25.6	11.8
LIGHT	18.7	2.6	31.2	6.2	25.2	16.5	19.0	2.5	30.9	6.1	25.0	16.4
ELI5	21.1	3.7	38.7	7.3	22.1	23.2	25.0	3.2	35.3	6.3	20.6	21.2
Ubuntu	17.2	2.4	27.1	5.0	22.9	12.8	23.3	3.5	26.4	4.6	22.3	12.2
Twitter	29.8	3.2	16.7	3.5	14.5	10.1	37.0	2.6	19.7	3.6	16.8	9.9
pushshift.io Reddit	25.8	2.2	24.2	4.5	18.7	13.4	28.0	1.7	23.4	4.1	18.2	13.3
Image Chat	18.3	2.4	30.7	6.2	26.3	14.3	21.8	2.1	28.6	5.3	24.7	13.1
IGC	10.0	10.6	67.9	38.2	64.5	45.1	10.2	11.0	66.3	34.8	61.4	45.3
dodecaScore	16.8	5.3	36.1	11.4	30.1	20.4	19.1	5.1	35.5	10.8	29.5	19.9
Table 9: Validation performance for various metrics on the dodecaDialogue tasks comparing our multi-task and multi-task + fine-tuned methods.
PPL	BLEU	ROUGE	f1
4	1	2	L
ConvAI2	11.3	5.6	22.2	7.0	20.4	21.3
DailyDialog	11.8	4.8	18.9	5.6	17.6	16.6
Wiz. of Wikipedia	8.7	19.7	40.9	22.6	36.9	37.7
Empathetic Dialog	11.2	4.8	20.9	5.6	19.0	19.3
Cornell Movie	21.9	3.3	14.2	3.2	13.4	11.3
LIGHT	19.0	2.9	17.0	3.4	15.0	16.2
ELI5	25.0	1.6	14.2	2.6	9.6	16.2
Ubuntu	23.3	2.3	12.5	1.9	11.6	11.2
Twitter	37.0	2.3	9.5	1.7	8.7	8.9
pushshift.io Reddit	28.0	1.8	12.1	2.2	10.4	11.3
Image Chat (all turns)	21.8	2.1	14.7	2.5	13.6	13.1
IGC	10.2	5.5	50.7	25.3	49.1	36.0
dodecaScore	19.1	4.7	20.7	7.0	18.8	18.3
Table 10: All Tasks Multi-Tasking (MT) validation performance for various metrics on the dodecaDialogue tasks with one set of decoding parameters: a beam size of 3, minimum response length of 10, and blocking repeated tri-grams.
BLEU	ROUGE-L	F1
Score	Beam	Min L	Max L	N-gram Block	Score	Beam	Min L	Max L	N-gram Block	Score	Beam	Min L	Max L	N-gram Block
ConvAI2	5.7	10	10	128	3	31.6	10	50	128	3	21.1	3	10	128	3
DailyDialog	4.4	10	5	128	3	32.0	3	50	128	3	18.8	5	10	128	3
Wiz. of Wikipedia	20.8	10	5	128	0	44.8	10	50	128	3	37.9	10	10	128	3
Empathetic Dialog	3.6	10	5	128	3	32.7	5	50	128	3	19.7	5	10	128	3
Cornell Movie	2.5	10	5	128	3	25.6	10	50	128	3	12.3	10	20	128	3
LIGHT	2.6	3	5	128	3	25.2	5	50	128	3	16.5	5	20	128	3
ELI5	3.7	10	200	256	3	22.1	5	200	256	3	23.2	10	200	256	3
Ubuntu	2.4	10	5	128	0	22.9	10	40	128	3	12.8	2	10	128	3
Twitter	3.2	10	20	128	3	14.5	5	50	128	3	10.1	10	20	128	3
pushshift.io Reddit	2.2	10	10	128	0	18.7	5	50	128	3	13.4	5	50	128	3
Image Chat (all turns)	2.4	10	5	128	3	26.4	3	50	128	3	14.3	5	1	128	3
IGC	10.6	10	5	128	3	64.5	3	50	128	3	45.1	10	5	128	3
B Human Evaluation Further Details
We provide additional results from our human evaluations described in Section 5. In Figure 1, we compare our All Tasks MT Image+Seq2Seq model to existing baselines from both tasks; to produce those outputs, we used beam search with a beam size of 10 and tri-gram blocking. As with our experiments regarding automatic metrics, we additionally explored nucleus sampling, with parameter  $p = 0.7$ , and compared to both the baseline models as well as human outputs. In tables 12, 13, and 14, we show the full results of comparing various models both to each other and also to humans.
When collecting the model-human chats for Wizard of Wikipedia, we additionally asked the humans for a rating from 1-5 at the end of each conversation, to indicate the quality of the model's responses; we compare these Likert ratings to that of Dinan et al. (2019), which followed the same protocol, in Table 15. The findings are similar to the pairwise ACUTE-Eval results in the main paper.
Table 11: Best decoding parameters for each task, based on metric. Scores are from the best performing task-specific multi-task + fine-tuned model on validation sets. "Min L" and "Max L" refer to the minimum and maximum decoding length, where "L" is the number of tokens.
Win Percentage
(Shuster et al., 2018)	Image+Seq2Seq Nucleus	Image+Seq2Seq Beam	Human
Lose Percentage	(Shuster et al., 2018)	-	50.8	*60.7	*79.3
Image+Seq2Seq Nucleus	49.2	-	52.1	*73.8
Image+Seq2Seq Beam	*39.3	47.9	-	*79.4
Human	*20.7	*26.2	*20.6	-
Table 12: Human evaluations on Image Chat, comparing various decoding schemes for our Image+Seq2Seq model trained on all tasks MT, as well as comparisons with human outputs. Scores with * are statistically significant (binomial test,  $p < .05$ ).
Win Percentage
Lose Percentage	(Dinan et al., 2019)	-	Image+Seq2Seq Nucleus	Image+Seq2Seq Beam	Human
(Dinan et al., 2019)
Image+Seq2Seq Nucleus	40.1	-	-	70.4
Image+Seq2Seq Beam	37.9	-	-	60.0
Human	28.1	29.6	40.0	-
Table 13: Human evaluations on Wizard of Wikipedia (seen) test set, comparing various decoding schemes for our Image+Seq2Seq model trained on all tasks MT, as well as comparisons with human outputs, using ACUTE-Eval. All scores are statistically significant (binomial test,  $p < .05$ ).
Win Percentage
Lose Percentage	(Dinan et al., 2019)	-	Image+Seq2Seq Nucleus	Image+Seq2Seq Beam	Human
(Dinan et al., 2019)
Image+Seq2Seq Nucleus	37.7	-	-	75.8
Image+Seq2Seq Beam	35.9	-	-	60.5
Human	24.2	27.2	39.5	-
Table 14: Human evaluations on Wizard of Wikipedia (unseen) test set, comparing various decoding schemes for our Image+Seq2Seq model trained on all tasks MT, as well as comparisons with human outputs, using ACUTE-Eval. All scores are statistically significant (binomial test,  $p < .05$ ).
Method	Decode Method	Seen	Unseen
Human Performance	4.13 (1.08)	4.34 (0.98)
(Dinan et al., 2019)	Beam	2.92 (1.33)	2.93 (1.30)
Image+Seq2Seq (All Tasks MT)	Beam	3.42 (1.10)	3.33 (1.09)
Image+Seq2Seq (All Tasks MT)	Nucleus	2.95 (1.08)	3.43 (1.05)
Table 15: Human evaluations on Wizard of Wikipedia, comparing various decoding schemes for our Image+Seq2Seq model trained on all tasks MT, as well as comparisons with human outputs, in terms of Likert Scores. Ratings are reported as mean (stddev).
C All Tasks Multi-Task Model Examples