SlowGuess/ACL_2020_OCR huggingface dataset:
Annotation Instructions (ctd.)
Uses Knowledge
(0-1)	Given the interesting fact that the response is conditioned on, how well does the response use the fact?
·A score of 0 (no) means the response does not mention or refer to the fact at all
·A score of 1 (yes) means the response uses the fact well
Overall Quality (1-3)	Given your answers above, what is your overall impression of this utter-ance?
·A score of 1 (very bad). A completely invalid response. It would be difficult to recover the conversation after this.
·A score of 2 (bad). Valid response, but otherwise poor in quality.
·A score of 3 (neutral) means this response is neither good nor bad. This response has no negative qualities, but no positive ones either.
·A score of 4 (good) means this is a good response, but falls short of being perfect because of a key flaw.
·A score of 5 (very good) means this response is good and does not have any strong flaws.
Interesting (1-3)	Is the response dull/interesting?
·A score of 1 (dull) means that the response is generic and dull.
- thats true . i agree .
·A score of 2 (somewhat interesting) means the response is somewhat interesting and could engage you in the conversation (e.g., an opinion, thought)
- my favorite role would have to be quarterback. it is such an interesting role.
- i love tom brady. i love tom brady .
·A score of 3 (interesting) means the response is very interesting or presents an interesting fact
- i agree . did you know that space jam is the highest grossing basketball movie of all time ?
- a lot of players can make money by starring in movies . did you know space jam is the highest grossing movie of all time ?
maybe one of the broke retired players can be in the sequel !
Table 9: Annotation instructions (part 3 of 3)
Metric Name	Turn-Level Correlation	System-Level Correlation
Pearson	Spearman	Spearman	Pearson
Word-Overlap Metrics
F-1	0.1645	0.1690	0.6000	0.6120
BLEU-1	0.2728	0.2876	0.7000	0.8334
BLEU-2	0.2862	0.3012	0.9000	0.8201
BLEU-3	0.2569	0.3007	0.9000	0.9033
BLEU-4	0.2160	0.2956	0.9000	0.8740
METEOR	0.3365	0.3908	0.9000	0.9435
ROUGE-L	0.2745	0.2870	0.9000	0.8143
Embedding-Based Metrics
Greedy Matching	0.1712	0.1943	0.8000	0.5610
Embedding Average	0.1803	0.2038	0.7000	0.9166
Vector Extrema	0.2032	0.2091	0.8000	0.5838
Skip-Thought	0.1040	0.1181	0.5000	0.5142
BERTScore (base)	0.3229	0.3540	0.9000	0.9100
BERTScore (large)	0.2982	0.3252	0.9000	0.8536
Reference Free Metrics
USR-MLM	0.3086	0.3345	0.9000	0.4732
USR-DR (x = c)	0.3245	0.4068	0.7000	0.9182
USR-DR (x = f)	0.1419	0.3221	0.9000	0.8519
USR	0.4192	0.4220	1.0000	0.9276
Table 10: Correlations of all the metrics with Overall Quality ratings on Topical-Chat. All values with  $p \geq  {0.05}$  are italicized.
Metric Name	Turn-Level Correlation	System-Level Correlation
Pearson	Spearman	Spearman	Pearson
Word-Overlap Metrics
F-1	0.1422	0.1241	1.0000	0.9956
BLEU-1	0.0434	0.0469	0.6000	0.2599
BLEU-2	0.1122	0.0943	0.4000	0.6816
BLEU-3	0.1202	0.0924	0.4000	0.6668
BLEU-4	0.1353	0.0899	0.8000	0.8413
METEOR	0.2527	0.2713	0.8000	0.9065
ROUGE-L	0.0659	0.0385	0.0000	0.1710
Embedding-Based Metrics
Greedy Matching	0.0916	0.0625	0.8000	0.3808
Embedding Average	0.1182	0.1428	0.8000	0.8628
Vector Extrema	0.1570	0.1410	0.6000	0.4349
Skip-Thought	-0.0393	-0.0452	-0.2000	0.2599
BERTScore (base)	0.1690	0.1526	0.8000	0.5173
BERTScore (large)	0.1518	0.1211	0.0000	0.2410
Reference Free Metrics
USR-MLM	0.0795	0.0788	-0.4000	-0.2842
USR-DR (x = c)	0.4814	0.6087	1.0000	0.8202
USR-DR (x = f)	-0.0495	-0.0454	-0.2108	-0.0178
USR	0.4693	0.4115	1.0000	0.8084
Table 11: Correlations of all the metrics with Overall Quality ratings on PersonaChat. All values with  $p \geq 0.05$  are italicized.
Metric Name	Turn-Level Correlation	System-Level Correlation
Pearson	Spearman	Spearman	Pearson
Word-Overlap Metrics
F-1	0.0425	0.0620	0.8000	0.6481
BLEU-1	0.1794	0.1522	0.6000	0.8360
BLEU-2	0.2360	0.2081	0.7000	0.8262
BLEU-3	0.2099	0.2137	0.7000	0.9018
BLEU-4	0.2010	0.2175	0.7000	0.8663
METEOR	0.2452	0.2246	0.7000	0.9424
ROUGE-L	0.2069	0.1632	0.7000	0.8208
Embedding-Based Metrics
Greedy Matching	0.0839	0.0868	0.6000	0.5664
Embedding Average	0.0509	0.0961	0.6000	0.9204
Vector Extrema	0.1561	0.1321	0.6000	0.6113
Skip-Thought	0.0810	0.0706	0.2000	0.4725
BERTScore (base)	0.2611	0.2502	0.7000	0.9118
BERTScore (large)	0.2556	0.2263	0.7000	0.8577
Reference Free Metrics
USR-MLM	0.3264	0.3268	0.7000	0.4666
USR-DR (x = c)	0.1500	0.2213	0.9000	0.9337
USR-DR (x = f)	0.0881	0.1967	0.7000	0.8420
USR	0.2932	0.3152	0.9000	0.9329
Table 12: Correlations of all the metrics with the Understandable ratings on Topical-Chat. All values with  $p \geq 0.05$  are italicized. The USR-MLM metric has poor system-level correlations, however the USR metric leverages predictions from the other sub-metrics to improve this.
Metric Name	Turn-Level Correlation	System-Level Correlation
Pearson	Spearman	Spearman	Pearson
Word-Overlap Metrics
F-1	-0.0340	-0.0550	1.0000	0.9956
BLEU-1	0.0123	-0.0196	0.6000	0.2599
BLEU-2	0.0854	0.0221	0.4000	0.6816
BLEU-3	0.0412	0.0249	0.4000	0.6668
BLEU-4	0.0537	0.0279	0.8000	0.8413
METEOR	0.0820	0.0431	0.8000	0.9065
ROUGE-L	0.0346	0.0076	0.0000	0.1710
Embedding-Based Metrics
Greedy Matching	0.0594	0.0710	0.8000	0.3808
Embedding Average	0.0573	0.0835	0.8000	0.8628
Vector Extrema	0.1097	0.1113	0.6000	0.4349
Skip-Thought	-0.0338	-0.0297	-0.2000	0.2599
BERTScore (base)	0.0676	0.0685	0.8000	0.5173
BERTScore (large)	0.0380	0.0086	0.0000	0.2410
Reference Free Metrics
USR-MLM	0.1313	0.1186	-0.4000	-0.2842
USR-DR (x = c)	0.0728	0.1446	1.0000	0.8202
USR-DR (x = f)	-0.0390	-0.0433	-0.2108	-0.0178
USR	0.0997	0.1337	1.0000	0.8084
Table 13: Correlations of all the metrics with Understandable ratings on PersonaChat. All values with  $p \geq 0.05$  are italicized.
Metric Name	Turn-Level Correlation	System-Level Correlation
Pearson	Spearman	Spearman	Pearson
Word-Overlap Metrics
F-1	0.0301	0.0398	0.6000	0.5605
BLEU-1	0.1606	0.1334	0.7000	0.7976
BLEU-2	0.1959	0.1648	0.9000	0.7888
BLEU-3	0.1896	0.1745	0.9000	0.8979
BLEU-4	0.1799	0.1748	0.9000	0.8973
METEOR	0.2121	0.1906	0.9000	0.9297
ROUGE-L	0.1760	0.1457	0.9000	0.7902
Embedding-Based Metrics
Greedy Matching	0.0534	0.0483	0.8000	0.5271
Embedding Average	0.0477	0.0970	0.7000	0.8875
Vector Extrema	0.1009	0.0761	0.8000	0.5363
Skip-Thought	0.0959	0.0858	0.5000	0.5313
BERTScore (base)	0.2164	0.2088	0.9000	0.9024
BERTScore (large)	0.2260	0.2094	0.9000	0.8319
Reference Free Metrics
USR-MLM	0.3370	0.3254	0.9000	0.4485
USR-DR (x = c)	0.1325	0.2148	0.7000	0.9222
USR-DR (x = f)	0.0313	0.1611	0.9000	0.8808
USR	0.2763	0.3037	1.0000	0.9220
Table 14: Correlations of all the metrics with the Natural ratings on Topical-Chat. All values with  $p \geq 0.05$  are italicized. The USR-MLM metric has poor system-level correlations, however the USR metric leverages predictions from the other sub-metrics to improve this.
Metric Name	Turn-Level Correlation	System-Level Correlation
Pearson	Spearman	Spearman	Pearson
Word-Overlap Metrics
F-1	0.0815	0.0717	1.0000	0.9956
BLEU-1	-0.0072	-0.0216	0.6000	0.2599
BLEU-2	0.0838	0.0344	0.4000	0.6816
BLEU-3	0.0823	0.0457	0.4000	0.6668
BLEU-4	0.1081	0.0499	0.8000	0.8413
METEOR	0.0989	0.0950	0.8000	0.9065
ROUGE-L	0.0096	-0.0087	0.0000	0.1710
Embedding-Based Metrics
Greedy Matching	0.1029	0.0665	0.8000	0.3808
Embedding Average	0.1413	0.1152	0.8000	0.8628
Vector Extrema	0.1458	0.1375	0.6000	0.4349
Skip-Thought	-0.0355	-0.0365	-0.2000	0.2599
BERTScore (base)	0.0606	0.0585	0.8000	0.5173
BERTScore (large)	0.0494	0.0477	0.0000	0.2410
Reference Free Metrics
USR-MLM	0.0999	0.1119	-0.4000	-0.2842
USR-DR (x = c)	0.1733	0.2291	1.0000	0.8202
USR-DR (x = f)	-0.0033	0.0642	-0.2108	-0.0178
USR	0.1862	0.2430	1.0000	0.8084
Table 15: Correlations of all the metrics with the Natural ratings on PersonaChat. All values with  $p \geq 0.05$  are italicized.
Metric Name	Turn-Level Correlation	System-Level Correlation
Pearson	Spearman	Spearman	Pearson
Word-Overlap Metrics
F-1	0.1290	0.1199	0.6000	0.6483
BLEU-1	0.2097	0.2228	1.0000	0.8754
BLEU-2	0.2087	0.2353	0.9000	0.8555
BLEU-3	0.1736	0.2377	0.9000	0.9090
BLEU-4	0.1307	0.2345	0.5000	0.8464
METEOR	0.2495	0.3018	0.9000	0.9573
ROUGE-L	0.1928	0.2031	0.9000	0.8410
Embedding-Based Metrics
Greedy Matching	0.1036	0.1249	0.8000	0.6078
Embedding Average	0.1197	0.1511	1.0000	0.9460
Vector Extrema	0.1839	0.1840	0.8000	0.6275
Skip-Thought	0.0326	0.0568	0.6000	0.5237
BERTScore (base)	0.2432	0.2642	0.9000	0.9160
BERTScore (large)	0.2140	0.2328	0.9000	0.8779
Reference Free Metrics
USR-MLM	0.3099	0.3243	0.9000	0.5190
USR-DR (x = c)	0.3391	0.3650	0.3000	0.8899
USR-DR (x = f)	0.0594	0.1836	0.5000	0.8188
USR	0.4160	0.3769	0.7000	0.9270
Table 16: Correlations of all the metrics with the Maintains Context ratings on Topical-Chat. All values with  $p \geq 0.05$  are italicized. Several referenced metrics perform strongly on the system-level correlations, however USR strongly outperforms all other metrics on the turn-level correlations.
Metric Name	Turn-Level Correlation	System-Level Correlation
Pearson	Spearman	Spearman	Pearson
Word-Overlap Metrics
F-1	0.1073	0.0747	1.0000	0.9956
BLEU-1	0.0713	0.0799	0.6000	0.2599
BLEU-2	0.0949	0.1372	0.4000	0.6816
BLEU-3	0.1270	0.1461	0.4000	0.6668
BLEU-4	0.1467	0.1508	0.8000	0.8413
METEOR	0.2500	0.2564	0.8000	0.9065
ROUGE-L	0.1135	0.0910	0.0000	0.1710
Embedding-Based Metrics
Greedy Matching	0.1503	0.1631	0.8000	0.3808
Embedding Average	0.1010	0.1660	0.8000	0.8628
Vector Extrema	0.2288	0.2631	0.6000	0.4349
Skip-Thought	0.0243	0.0139	-0.2000	0.2599
BERTScore (base)	0.1770	0.1686	0.8000	0.5173
BERTScore (large)	0.1877	0.1569	0.0000	0.2410
Reference Free Metrics
USR-MLM	0.1805	0.2067	-0.4000	-0.2842
USR-DR (x = c)	0.6021	0.5625	1.0000	0.8202
USR-DR (x = f)	-0.0198	-0.0164	-0.2108	-0.0178
USR	0.6065	0.5280	1.0000	0.8084
Table 17: Correlations of all the metrics with the Maintains Context ratings on PersonaChat. All values with  $p \geq 0.05$  are italicized. Several referenced metrics perform strongly on the system-level correlations, however USR strongly outperforms all other metrics on the turn-level correlations.
Metric Name	Turn-Level Correlation	System-Level Correlation
Pearson	Spearman	Spearman	Pearson
Word-Overlap Metrics
F-1	0.2523	0.2565	0.6000	0.5944
BLEU-1	0.3144	0.3343	0.7000	0.8197
BLEU-2	0.3184	0.3323	0.9000	0.8099
BLEU-3	0.2782	0.3247	0.9000	0.9047
BLEU-4	0.2322	0.3156	0.9000	0.8883
METEOR	0.3668	0.4391	0.9000	0.9398
ROUGE-L	0.2946	0.2995	0.9000	0.8084
Embedding-Based Metrics
Greedy Matching	0.1989	0.2111	0.8000	0.5512
Embedding Average	0.1940	0.2161	0.7000	0.9056
Vector Extrema	0.2101	0.2050	0.8000	0.5694
Skip-Thought	0.1139	0.1356	0.5000	0.5187
BERTScore (base)	0.3512	0.3725	0.9000	0.9108
BERTScore (large)	0.3167	0.3349	0.9000	0.8480
Reference Free Metrics
USR-MLM	0.3189	0.3337	0.9000	0.4663
USR-DR (x = c)	0.3533	0.4877	0.7000	0.9233
USR-DR (x = f)	0.2006	0.4110	0.9000	0.8685
USR	0.4555	0.4645	1.0000	0.9297
Table 18: Correlations of all the metrics with the Interesting ratings on Topical-Chat. All values with  $p \geq 0.05$  are italicized.
Metric Name	Turn-Level Correlation	System-Level Correlation
Pearson	Spearman	Spearman	Pearson
Word-Overlap Metrics
F-1	0.0473	0.0132	1.0000	0.9956
BLEU-1	-0.1081	-0.0922	0.6000	0.2599
BLEU-2	-0.1048	-0.1010	0.4000	0.6816
BLEU-3	-0.1247	-0.1151	0.4000	0.6668
BLEU-4	-0.1359	-0.1242	0.8000	0.8413
METEOR	-0.0458	0.0116	0.8000	0.9065
ROUGE-L	-0.1456	-0.1354	0.0000	0.1710
Embedding-Based Metrics
Greedy Matching	-0.1778	-0.2080	0.8000	0.3808
Embedding Average	-0.0141	-0.0177	0.8000	0.8628
Vector Extrema	-0.1883	-0.1746	0.6000	0.4349
Skip-Thought	-0.0882	-0.0916	-0.2000	0.2599
BERTScore (base)	0.0325	0.0491	0.8000	0.5173
BERTScore (large)	-0.0418	-0.0245	0.0000	0.2410
Reference Free Metrics
USR-MLM	-0.1045	-0.1007	-0.4000	-0.2842
USR-DR (x = c)	0.0606	0.2634	1.0000	0.8202
USR-DR (x = f)	-0.0022	-0.0039	-0.2108	-0.0178
USR	0.0315	0.0171	1.0000	0.8084
Table 19: Correlations of all the metrics with the Interesting ratings on PersonaChat. All values with  $p \geq 0.05$  are italicized.
Metric Name	Turn-Level Correlation	System-Level Correlation
Pearson	Spearman	Spearman	Pearson
Word-Overlap Metrics
F-1	0.1495	0.1485	0.6000	0.5970
BLEU-1	0.2888	0.3033	0.7000	0.8357
BLEU-2	0.2819	0.3066	0.9000	0.8309
BLEU-3	0.2442	0.3106	0.9000	0.9259
BLEU-4	0.2126	0.3096	0.9000	0.9084
METEOR	0.3328	0.3909	0.9000	0.9534
ROUGE-L	0.3099	0.3273	0.9000	0.8333
Embedding-Based Metrics
Greedy Matching	0.2327	0.2306	0.8000	0.5874
Embedding Average	0.1812	0.1827	0.7000	0.9151
Vector Extrema	0.2294	0.2111	0.8000	0.5917
Skip-Thought	0.0986	0.1145	0.5000	0.5397
BERTScore (base)	0.2847	0.2947	0.9000	0.9308
BERTScore (large)	0.2909	0.3167	0.9000	0.8703
Reference Free Metrics
USR-MLM	0.2195	0.2261	0.9000	0.5070
USR-DR (x = c)	0.2285	0.4179	0.7000	0.9155
USR-DR (x = f)	0.2220	0.4468	0.9000	0.8884
USR	0.3175	0.3353	1.0000	0.9469
Table 20: Correlations of all the metrics with the Uses Knowledge ratings on Topical-Chat. All values with  $p \geq 0.05$  are italicized.
Metric Name	Turn-Level Correlation	System-Level Correlation
Pearson	Spearman	Spearman	Pearson
Word-Overlap Metrics
F-1	0.0869	0.1056	1.0000	0.9956
BLEU-1	0.0737	0.0729	0.6000	0.2599
BLEU-2	0.1083	0.0722	0.4000	0.6816
BLEU-3	0.0999	0.0594	0.4000	0.6668
BLEU-4	0.0698	0.0528	0.8000	0.8413
METEOR	0.1678	0.1719	0.8000	0.9065
ROUGE-L	0.0710	0.0632	0.0000	0.1710
Embedding-Based Metrics
Greedy Matching	0.0382	0.0057	0.8000	0.3808
Embedding Average	0.0402	0.0618	0.8000	0.8628
Vector Extrema	0.0564	-0.0008	0.6000	0.4349
Skip-Thought	-0.0686	-0.0609	-0.2000	0.2599
BERTScore (base)	0.0719	0.0465	0.8000	0.5173
BERTScore (large)	0.0271	0.0094	0.0000	0.2410
Reference Free Metrics
USR-MLM	-0.0782	-0.0756	-0.4000	-0.2842
USR-DR (x = c)	0.4508	0.6309	1.0000	0.8202
USR-DR (x = f)	-0.0927	-0.0903	-0.2108	-0.0178
USR	0.4027	0.3177	1.0000	0.8084
Table 21: Correlations of all the metrics with the Uses Knowledge ratings on PersonaChat. All values with  $p \geq 0.05$  are italicized.