SlowGuess/NAACL_2022_OCR huggingface dataset:
We take a two-step finetuning to make full use of our MTG benchmark. We first use the large rough parallel training data to train our models on the downstream tasks for 20 epochs, and then finetune the models on the small annotated training data to further improve the generation performance for 10 epochs. We evaluate the model for every 2000 steps and use the loss on development to choose the best model. The batch size is 32. The learning rate and optimizer parameters are set to the default parameters for each model. All models are trained with 32GB Tesla-V100.
Threshold	QG	TG	SG	Summ
0	82306	393792	88161	287083
0.3	80836	355034	88158	243698
0.4	79390	333461	88077	217777
0.5	71819	280376	87003	164355
0.6	32261	144109	75892	58060
Table 8: The data sizes of datasets filtered by back translation with respect to different thresholds.
Models	Arch	#langs	#vocab	#layers	#params
M-BERT	enc	104	110k	12	172M
XLM	enc	17	200k	16	570M
mBART	enc-dec	25	250k	12	680M
mT5	enc-dec	101	250k	24	13B
C Significant Test Results
The average ensemble metric scores for stage1 and stage2 in four tasks and the corresponding signifi
Table 9: The overall statistics for multilingual pretrained models. Arch means the architectures of models. # vocab means the vocabulary sizes of models. # langs, # layers and # params mean the number of languages, layers and parameters respectively.
Tasks	Mono	Multi	Cross	Zero
SG	stage1	0.268	0.270	0.258	0.125
stage2	0.284	0.284	0.289	0.167
p-value	0.000	0.000	0.000	0.000
QG	stage1	0.286	0.287	0.279	0.235
stage2	0.301	0.300	0.295	0.258
p-value	0.000	0.000	0.000	0.000
TG	stage1	0.257	0.270	0.268	0.223
stage2	0.288	0.291	0.289	0.232
p-value	0.000	0.000	0.000	0.003
Summ	stage1	0.186	0.187	0.181	0.161
stage2	0.193	0.208	0.207	0.168
p-value	0.001	0.000	0.000	0.004
Table 10: The average ensemble metric scores for XLM for stage1 and stage2 in four tasks in four settings and the corresponding t test p-values. Here stage1 represents models trained only on rouge training data while stage2 represents models further trained on human-annotated training data based on models in stage1. The bold cell means the significantly higher score between stage1 and stage2 scores.
cant test p-values are displayed in Table 10. As it shows, adding human-annotated training data can always improve the model performance under different settings. The improvements are significant in all settings.
We present detailed experimental results of our four baseline models under four different evaluation settings here.
Table 11: The whole results under the monolingual evaluation scenarios.
Table 12: The whole results under the multilingual evaluation scenarios.
Task	Model	Language	N-gram-based	Embedding-based	Diversity	Ours
BLEU	ROUGE-L	METEOR	BERTScore	Distinct-1	Distinct-2	Ensemble
SG	M-BERT	en->de	2.28	13.6	0.127	0.721	0.953	0.995	34.529
en->fr	1.43	14.1	0.182	0.72	0.938	0.995	30.507
en->es	1.57	13	0.111	0.72	0.943	0.993	30.368
en->zh	4.72	25.4	-	0.664	0.997	0.302	26.423
XLM	en->de	3.2	25.8	0.142	0.73	0.964	0.995	29.683
en->fr	4.23	26.4	0.198	0.744	0.951	0.989	27.735
en->es	3.38	24.8	0.135	0.737	0.959	0.991	28.353
en->zh	5.79	26.8	-	0.67	0.994	0.338	26.674
mBART	en->de	1.81	11.9	0.117	0.723	0.983	0.999	34.089
en->fr	1.35	12.7	0.133	0.728	0.969	0.989	29.805
en->es	1.22	11.2	0.098	0.722	0.928	0.978	30.153
en->zh	2.59	19.1	-	0.599	0.998	0.663	21.407
mT5	en->de	3.33	15	0.141	0.731	0.973	0.992	34.569
en->fr	3.2	15.9	0.203	0.746	0.954	0.981	31.988
en->es	3.2	14.9	0.131	0.743	0.965	0.988	31.759
en->zh	5.91	27.3	-	0.675	0.997	0.278	26.873
OG	M-BERT	en->de	5.53	22	0.198	0.738	0.892	0.983	30.585
en->fr	4.15	19.7	0.255	0.741	0.901	0.991	29.265
en->es	5.43	21.4	0.292	0.79	0.903	0.987	31.241
en->zh	8.57	30.3	-	0.689	0.994	0.934	26.358
XLM	en->de	10.41	37.2	0.254	0.78	0.973	0.996	29.273
en->fr	14.7	39.6	0.353	0.799	0.949	0.992	29.786
en->es	16.93	46.8	0.373	0.831	0.952	0.996	29.082
en->zh	16.07	37.9	-	0.733	0.997	1	28.626
mBART	en->de	2.25	15.9	0.14	0.701	0.929	0.996	28.605
en->fr	3.1	18.4	0.178	0.739	0.975	0.997	28.656
en->es	3.8	17.1	0.246	0.797	0.983	0.997	31.535
en->zh	13.1	35.4	-	0.722	0.998	0.998	27.967
mT5	en->de	9.84	28.5	0.246	0.782	0.984	0.998	33.221
en->fr	13.62	32.8	0.346	0.804	0.967	0.997	32.444
en->es	15.17	35.2	0.381	0.839	0.962	0.997	33.736
en->zh	15.69	38.4	-	0.736	0.997	0.999	28.698
TG	M-BERT	en->de	7.21	18	0.181	0.719	0.942	0.995	32.703
en->fr	6.78	21.9	0.221	0.734	0.928	0.994	28.731
en->es	9.32	26.2	0.278	0.756	0.92	0.996	28.972
en->zh	10.94	28.5	-	0.676	0.993	0.452	26.029
XLM	en->de	9.15	24.3	0.216	0.727	0.947	0.987	30.613
en->fr	11.54	29	0.301	0.753	0.914	0.987	28.465
en->es	12.45	31.1	0.31	0.763	0.91	0.993	29.464
en->zh	15.44	29.9	-	0.692	0.981	0.576	27.139
mBART	en->de	1.01	6.8	0.069	0.612	0.55	0.73	21.811
en->fr	2.35	14	0.095	0.63	0.526	0.741	15.719
en->es	3.84	16.2	0.13	0.626	0.513	0.747	13.509
en->zh	16.39	34	-	0.705	0.989	0.6	27.556
mT5	en->de	9.34	21.8	0.221	0.733	0.96	0.994	33.14
en->fr	12.26	28.2	0.302	0.76	0.929	0.989	29.326
en->es	14.14	31.1	0.324	0.772	0.917	0.993	30.702
en->zh	15.31	31.1	-	0.694	0.98	0.609	27.661
Summ	M-BERT	en->de	7.69	25.2	0.199	0.718	0.823	0.986	25.97
en->fr	8.94	29.2	0.284	0.733	0.769	0.982	22.519
en->es	10.16	32.5	0.276	0.745	0.757	0.979	20.582
en->zh	13.08	37.7	-	0.702	0.859	0.92	19.381
XLM	en->de	8.36	31.1	0.199	0.721	0.807	0.965	21.47
en->fr	11.79	35.5	0.289	0.75	0.748	0.94	20.301
en->es	11.93	36.6	0.285	0.75	0.737	0.953	20.678
en->zh	14.58	38.7	-	0.706	0.841	0.937	19.607
mBART	en->de	0.61	6.7	0.084	0.533	0.24	0.409	6.789
en->fr	6.23	23.7	0.204	0.679	0.608	0.857	16.018
en->es	1.45	16.5	0.093	0.581	0.36	0.615	6.274
en->zh	15.95	39.7	-	0.709	0.841	0.972	19.802
mT5	en->de	3.92	22.5	0.135	0.702	0.847	0.955	23.368
en->fr	3.99	25.7	0.179	0.727	0.837	0.959	21.158
en->es	5.55	28.9	0.217	0.733	0.799	0.956	19.613
en->zh	8.43	32.5	-	0.685	0.897	0.965	20.679
Table 13: The whole results under the English centric cross-lingual evaluation scenarios.
Task	Model	Language	N-gram-based	Embedding-based	Diversity	Ours
BLEU	ROUGE-L	METEOR	BERTScore	Distinct-1	Distinct-2	Ensemble
SG	M-BERT	de->en	2.59	18.5	0.1	0.893	0.918	0.988	30.794
de->fr	1.29	13.5	0.19	0.717	0.927	0.995	31.616
de->es	1.58	12.6	0.11	0.716	0.929	0.991	30.216
de->zh	4.57	24.1	-	0.656	0.989	0.518	26.703
XLM	de->en	3.18	19.7	0.104	0.894	0.961	0.991	31.967
de->fr	3.85	26	0.189	0.744	0.949	0.986	27.691
de->es	3.42	25	0.131	0.737	0.961	0.989	28.057
de->zh	5.66	26.5	-	0.669	0.991	0.437	26.488
mBART	de->en	1.99	18.4	0.093	0.887	0.94	0.988	32.055
de->fr	1.39	12.8	0.163	0.733	0.982	0.997	32.795
de->es	2.99	14.1	0.122	0.741	0.99	1	32.247
de->zh	3.29	22.4	-	0.62	0.994	0.317	22.664
mT5	de->en	3.37	20.1	0.105	0.901	0.968	0.995	32.24
de->fr	2.89	15.5	0.196	0.744	0.95	0.98	32.463
de->es	2.93	14.1	0.124	0.74	0.965	0.988	31.902
de->zh	5.39	26.9	-	0.673	0.996	0.275	26.654
OG	M-BERT	de->en	8.46	33.8	0.16	0.889	0.888	0.985	30.344
de->fr	3.65	18.3	0.237	0.739	0.901	0.993	29.692
de->es	4.56	20.6	0.276	0.786	0.914	0.988	30.602
de->zh	8.17	29.8	-	0.688	0.993	0.927	27.83
XLM	de->en	13.39	39.3	0.192	0.907	0.952	0.994	31.989
de->fr	12.11	36.7	0.319	0.79	0.949	0.991	29.54
de->es	14.3	44	0.34	0.822	0.95	0.996	28.685
de->zh	14.32	35.6	-	0.722	0.997	1	28.406
mBART	de->en	8.39	35.1	0.151	0.899	0.978	0.999	30.625
de->fr	3.36	18.8	0.2	0.75	0.988	0.998	30.037
de->es	4.03	18	0.252	0.799	0.985	0.997	32.577
de->zh	8.65	31	-	0.691	0.996	0.989	26.313
mT5	de->en	12.34	39.5	0.187	0.909	0.967	0.998	31.723
de->fr	10.51	28.4	0.306	0.787	0.966	0.996	31.792
de->es	11.61	31.2	0.332	0.827	0.962	0.997	33.198
de->zh	12.39	34.9	-	0.72	0.998	0.999	27.706
TG	M-BERT	de->en	9.2	30.6	0.158	0.879	0.94	0.99	29.629
de->fr	5.78	20.1	0.201	0.727	0.923	0.991	28.339
de->es	6.49	22.5	0.233	0.741	0.917	0.99	27.685
de->zh	9.85	26.7	-	0.669	0.99	0.498	26.422
XLM	de->en	11.84	30.6	0.183	0.883	0.963	0.994	30.658
de->fr	10.2	27	0.276	0.744	0.906	0.982	27.617
de->es	10.82	28.7	0.278	0.755	0.906	0.991	28.232
de->zh	13.54	27.8	-	0.68	0.98	0.559	28.158
mBART	de->en	2.72	19.1	0.083	0.833	0.739	0.881	22.604
de->fr	5.3	19.5	0.148	0.689	0.773	0.883	22.339
de->es	1.76	14.9	0.132	0.662	0.662	0.838	19.864
de->zh	14.6	31.1	-	0.694	0.988	0.591	28.533
mT5	de->en	12.53	34	0.19	0.892	0.976	0.996	31.707
de->fr	10.32	25.9	0.276	0.75	0.924	0.985	30.037
de->es	11.64	28.4	0.291	0.761	0.912	0.991	29.469
de->zh	13.22	28.5	-	0.682	0.979	0.582	27.805
Summ	M-BERT	de->en	10.84	36.2	0.154	0.869	0.822	0.985	20.938
de->fr	8.08	28.3	0.258	0.729	0.775	0.98	22.388
de->es	8.78	31.6	0.26	0.742	0.757	0.977	21.011
de->zh	11.9	36.9	-	0.7	0.865	0.89	19.931
XLM	de->en	11.69	35.5	0.17	0.873	0.753	0.943	22.014
de->fr	10.44	34.4	0.275	0.745	0.738	0.93	21.136
de->es	10.32	35.1	0.268	0.745	0.734	0.945	19.705
de->zh	13.25	37.3	-	0.7	0.837	0.921	19.377
mBART	de->en	12.86	37.2	0.17	0.871	0.833	0.991	21.008
de->fr	10.53	31.9	0.284	0.743	0.788	0.987	22.358
de->es	11.52	33.6	0.283	0.745	0.752	0.985	20.568
de->zh	9.8	33	-	0.679	0.86	0.972	20.075
mT5	de->en	6.04	31.4	0.118	0.872	0.844	0.961	23.391
de->fr	3.51	24.7	0.176	0.723	0.819	0.944	20.324
de->es	5.01	28.1	0.207	0.728	0.779	0.94	20.423
de->zh	7.63	31.1	-	0.679	0.887	0.964	20.901
Table 14: The whole results under the German centric cross-lingual evaluation scenarios.
Table 15: The whole results under the French centric cross-lingual evaluation scenarios.
Task	Model	Language	N-gram-based	Embedding-based	Diversity	Ours
BLEU	ROUGE-L	METEOR	BERTScore	Distinct-1	Distinct-2	Ensemble
SG	M-BERT	es->en	2.12	18.4	0.1	0.892	0.909	0.985	31.191
es->de	2.21	13.7	0.134	0.718	0.93	0.993	34.32
es->fr	1.42	13.4	0.194	0.72	0.931	0.993	31.622
es->zh	4.51	24.2	-	0.658	0.995	0.337	26.406
XLM	es->en	3.12	19.6	0.103	0.895	0.966	0.996	31.257
es->de	2.83	25	0.139	0.727	0.962	0.994	30.004
es->fr	3.94	26.3	0.192	0.743	0.948	0.986	27.971
es->zh	5.35	25.9	-	0.668	0.99	0.483	26.153
mBART	es->en	0.9	15.2	0.065	0.881	0.925	0.995	33.321
es->de	0.39	5.9	0.063	0.67	0.963	0.997	30.696
es->fr	0.81	11.2	0.11	0.699	0.97	0.998	28.099
es->zh	5.08	26.1	-	0.668	0.999	0.305	26.645
mT5	es->en	3.4	20.2	0.106	0.901	0.966	0.994	31.215
es->de	3.17	15	0.141	0.732	0.971	0.992	34.703
es->fr	3.17	16.3	0.201	0.746	0.955	0.982	32.183
es->zh	5.61	26.8	-	0.673	0.996	0.286	27.04
OG	M-BERT	es->en	8.89	34.5	0.164	0.888	0.87	0.982	30.078
es->de	5.18	20.9	0.185	0.735	0.919	0.989	31.312
es->fr	3.99	18.2	0.254	0.735	0.906	0.993	29.173
es->zh	7.82	29.1	-	0.683	0.994	0.929	26.005
XLM	es->en	13.48	39.6	0.198	0.907	0.958	0.997	32.042
es->de	9.61	36.3	0.239	0.774	0.972	0.996	29.52
es->fr	13.9	38.6	0.343	0.796	0.948	0.991	28.99
es->zh	14.78	36.7	-	0.727	0.997	1	28.566
mBART	es->en	4.96	30.8	0.124	0.89	0.98	0.998	30.121
es->de	2.35	17.6	0.139	0.726	0.966	0.997	32.497
es->fr	3.29	18.6	0.193	0.745	0.983	0.998	29.198
es->zh	12.07	34.5	-	0.719	0.999	0.998	28.107
mT5	es->en	13.34	40.9	0.198	0.911	0.971	0.998	31.936
es->de	8.53	26.3	0.227	0.774	0.984	0.998	32.628
es->fr	12.89	31.1	0.333	0.797	0.967	0.996	32.371
es->zh	13.37	36	-	0.726	0.997	0.998	28.779
TG	M-BERT	es->en	9.94	31.8	0.167	0.884	0.955	0.995	29.842
es->de	5.81	16.5	0.165	0.714	0.941	0.993	30.79
es->fr	6.13	21.2	0.237	0.734	0.932	0.996	28.129
es->zh	9.73	26.9	-	0.669	0.992	0.418	27.009
XLM	es->en	12.25	31.7	0.188	0.886	0.966	0.994	30.547
es->de	8.01	22.5	0.206	0.721	0.947	0.991	29.972
es->fr	10.99	28.2	0.288	0.748	0.906	0.981	28.193
es->zh	14.36	28.3	-	0.685	0.978	0.612	27.391
mBART	es->en	16.65	39.1	0.21	0.901	0.984	0.999	32.993
es->de	0.95	6.4	0.085	0.605	0.492	0.672	23.184
es->fr	7.21	22.9	0.187	0.712	0.818	0.901	24.795
es->zh	15.27	32.3	-	0.697	0.988	0.577	27.849
mT5	es->en	13	34.5	0.196	0.893	0.977	0.996	31.113
es->de	8.27	20	0.201	0.726	0.959	0.994	32.597
es->fr	10.83	26.6	0.285	0.753	0.928	0.988	29.288
es->zh	13.67	29.2	-	0.685	0.979	0.602	27.73
Summ	M-BERT	es->en	11.41	36.6	0.16	0.869	0.821	0.985	20.613
es->de	7.22	24.9	0.197	0.716	0.808	0.983	25.962
es->fr	8.61	28.7	0.28	0.732	0.772	0.983	22.311
es->zh	12.34	37.3	-	0.7	0.854	0.904	19.625
XLM	es->en	11.18	35.2	0.173	0.872	0.749	0.946	21.567
es->de	7.01	29.6	0.184	0.715	0.8	0.96	21.158
es->fr	10.48	34	0.276	0.742	0.728	0.924	20.617
es->zh	11.01	35.1	-	0.69	0.834	0.929	20.141
mBART	es->en	13.28	37.8	0.172	0.872	0.833	0.992	20.997
es->de	1.21	7.2	0.089	0.663	0.675	0.964	20.631
es->fr	10.82	31.8	0.29	0.742	0.789	0.987	22.652
es->zh	10.38	33.5	-	0.683	0.87	0.966	19.889
mT5	es->en	6.66	32.6	0.122	0.875	0.861	0.975	23.134
es->de	3.88	22.4	0.134	0.703	0.849	0.96	23.164
es->fr	4	25.7	0.18	0.728	0.847	0.968	20.854
es->zh	2.97	19.8	-	0.612	0.758	0.814	18.781
Table 16: The whole results under the Spanish centric cross-lingual evaluation scenarios.
Table 17: The whole results under the Chinese centric cross-lingual evaluation scenarios.
Task	Model	Language	N-gram-based	Embedding-based	Diversity	Ours
BLEU	ROUGE-L	METEOR	BERTScore	Distinct-1	Distinct-2	Ensemble
SG	M-BERT	en->de	0.06	3.2	0.053	0.694	0.92	0.991	34.856
en->fr	0.04	3.9	0.041	0.705	0.921	0.991	32.033
en->es	0.06	3.5	0.04	0.707	0.918	0.991	31.123
en->zh	0	0.2	-	0.542	0.919	0.991	20.574
XLM	en->de	0.02	7.2	0.05	0.634	0.469	0.5	18.939
en->fr	0.02	5.9	0.037	0.626	0.38	0.412	14.273
en->es	0.09	8.5	0.035	0.646	0.516	0.547	16.903
en->zh	0	0	-	0.45	0.609	0.574	16.581
mBART	en->de	0.1	3.8	0.056	0.705	0.976	0.999	32.962
en->fr	0.07	4.6	0.048	0.715	0.977	0.999	31.386
en->es	0.08	4.1	0.046	0.719	0.976	0.999	32.222
en->zh	0	0.2	-	0.55	0.976	0.999	21.89
mT5	en->de	0.06	2.3	0.046	0.699	0.983	0.997	33.87
en->fr	0.03	3.1	0.039	0.709	0.983	0.997	31.869
en->es	0.05	2.7	0.033	0.712	0.983	0.997	32.738
en->zh	0	0.1	-	0.542	0.983	0.998	20.781
OG	M-BERT	en->de	0.62	4.3	0.071	0.724	0.908	0.987	32.122
en->fr	0.58	3.3	0.06	0.731	0.911	0.989	29.54
en->es	0.49	2.4	0.057	0.747	0.907	0.988	29.76
en->zh	0.04	0.5	-	0.551	0.906	0.987	22.383
XLM	en->de	1.96	17.6	0.097	0.726	0.938	0.984	29.795
en->fr	2.16	16.2	0.081	0.742	0.943	0.99	28.605
en->es	7.46	23.9	0.177	0.757	0.941	0.985	28.485
en->zh	0	0	-	0.438	0.099	0.075	16.413
mBART	en->de	1.37	6.2	0.08	0.738	0.982	1	31.099
en->fr	1.15	6.7	0.061	0.745	0.982	1	28.971
en->es	0.93	3.4	0.072	0.761	0.982	1	29.98
en->zh	0.19	8.5	-	0.557	0.982	1	17.838
mT5	en->de	1.43	5.4	0.082	0.737	0.97	0.998	31.781
en->fr	1.17	4.9	0.064	0.744	0.971	0.998	29.666
en->es	1.07	3.8	0.072	0.76	0.971	0.998	29.345
en->zh	0.22	0.8	-	0.556	0.971	0.998	22.886
TG	M-BERT	en->de	5.14	11.8	0.136	0.697	0.928	0.988	31.819
en->fr	3.6	10.8	0.117	0.702	0.936	0.989	28.926
en->es	3.76	10.4	0.142	0.711	0.928	0.988	28.513
en->zh	0.56	2.1	-	0.557	0.928	0.988	21.746
XLM	en->de	2.58	12.5	0.12	0.665	0.83	0.877	26.312
en->fr	3.26	12.7	0.12	0.685	0.887	0.933	25.239
en->es	4.9	18.5	0.169	0.705	0.875	0.936	24.466
en->zh	0.01	0	-	0.446	0.372	0.224	16.684
mBART	en->de	5.49	14.4	0.142	0.706	0.983	0.999	32.232
en->fr	3.79	12.6	0.118	0.713	0.988	1	28.936
en->es	3.89	12.3	0.149	0.721	0.983	0.999	29.65
en->zh	0.71	2.9	-	0.56	0.983	0.999	22.608
mT5	en->de	5.41	12.9	0.133	0.696	0.974	0.996	31.885
en->fr	4.04	11.5	0.112	0.703	0.979	0.997	29.098
en->es	4.09	11.3	0.138	0.712	0.975	0.996	29.73
en->zh	0.79	2.4	-	0.553	0.973	0.995	21.886
Summ	M-BERT	en->de	2.24	9.8	0.078	0.675	0.816	0.983	23.995
en->fr	1.49	9.4	0.075	0.698	0.821	0.986	23.347
en->es	1.38	8.9	0.074	0.701	0.815	0.983	22.207
en->zh	0.04	0.5	-	0.521	0.815	0.983	13.572
XLM	en->de	1.85	14.5	0.08	0.652	0.61	0.777	18.486
en->fr	1.29	13	0.066	0.678	0.639	0.816	18.639
en->es	4.18	20.3	0.141	0.686	0.643	0.818	17.303
en->zh	0	0	-	0.424	0.682	0.274	12.837
mBART	en->de	2.2	10.3	0.07	0.675	0.852	0.992	23.317
en->fr	1.46	9.9	0.07	0.695	0.853	0.992	22.576
en->es	1.35	9.3	0.074	0.699	0.852	0.992	21.736
en->zh	0.06	0.4	-	0.521	0.853	0.992	13.065
mT5	en->de	1.11	8	0.05	0.654	0.842	0.936	19.314
en->fr	0.63	7.4	0.047	0.672	0.845	0.939	17.417
en->es	0.59	7.1	0.055	0.675	0.842	0.937	17.085
en->zh	0.01	0.3	-	0.514	0.842	0.937	13.39
Table 18: The whole results under the zero-shot evaluation scenarios.