SlowGuess/ACL_2020_OCR huggingface dataset:
(UFAL), Faculty of Mathematics and Physics, Charles University.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Philipp Probst, Anne-Laure Boulesteix, and Bernd Bischl. 2019. Tunability: Importance of hyperparameters of machine learning algorithms. Journal of Machine Learning Research, 20(53):1-32.
Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In *Meeting of the North American Chapter of the Association for Computational Linguistics* (NAACL), New Orleans, USA.
Brian Richards. 1987. Type-token ratios: What do they really tell us? Journal of child language, 14(2):201-209.
Shruti Rijhwani, Jiateng Xie, Graham Neubig, and Jaime Carbonell. 2019. Zero-shot neural transfer for cross-lingual entity linking. In Thirty-Third AAAI Conference on Artificial Intelligence (AAAI), Honolulu, Hawaii.
Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. 2020. A constructive prediction of the generalization error across scales. In International Conference on Learning Representations.
Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2019. Wikimatrix: Mining  $135\mathrm{m}$  parallel sentences in 1620 language pairs from wikipedia. arXiv preprint arXiv:1907.05791.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, Florence, Italy. Association for Computational Linguistics.
Christopher KI Williams and Carl Edward Rasmussen. 1996. Gaussian processes for regression. In Advances in neural information processing systems, pages 514-520.
Ruochen Xu, Yiming Yang, Naoki Otani, and Yuexin Wu. 2018. Unsupervised cross-lingual transfer of word embedding spaces. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2465–2474.
Pengcheng Yang, Fuli Luo, Peng Chen, Tianyu Liu, and Xu Sun. 2019. Maam: A morphology-aware alignment model for unsupervised bilingual lexicon
induction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3190-3196.
Daniel Zeman, Jan Hajic, Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018a. CoNLL 2018 shared task: Multilingual parsing from raw text to universal dependencies. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1-21, Brussels, Belgium. Association for Computational Linguistics.
Daniel Zeman, Jan Hajic, Martin Popel, Martin Pothast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018b. Conll 2018 shared task: multilingual parsing from raw text to universal dependencies. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1-21.
Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Earth mover's distance minimization for unsupervised bilingual lexicon induction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1934-1945.
8636
A Questionnaire
An example of the first questionnaire from our user case study is shown below. The second sheet also included the results in 44 more language pairs. We provide an answer key after the second sheet.
Please provide your prediction of the BLEU score based on the language pair and dataset features (the domain of the training and test sets is TED talks). After you finish, please go to sheet v2.
idx	Source Language	Target Language	Parallel Sentences (k)	Source vocab size (k)	Source subword vocab size (k)	Target vocab size (k)	Target subword vocab size( k)	BLEU
1	Basque (eus)	English	5	20	8	9	6
2	Slovak (slk)	English	61	134	8	36	8
3	Burmese (mya)	English	21	101	8	21	8
4	Korean (kor)	English	206	386	9	67	8
5	Lithuanian (lit)	English	42	108	8	29	8
6	Arabic (ara)	English	214	308	8	69	8
7	Czech (ces)	English	103	181	8	47	8
8	Esperanto (epo)	English	7	21	8	10	6
9	Finnish (fin)	English	24	77	8	22	8
10	Albanian (sqi)	English	45	93	8	30	8
11	Vietnamese (vie)	English	172	66	8	61	8
8637
Please provide your prediction of the BLEU score in the yellow area given all the information in this sheet. Note that all experiments are trained with the same model.
idx	Source Language	Target Lang.	Parallel Sentences (k)	Source vocab size (k)	Source subword vocab size (k)	Target vocab size (k)	Target subword vocab size( k)	BLEU
1	Basque (eus)	English	5	20	8	9	6
2	Slovak (slk)	English	61	134	8	36	8
3	Burmese (mya)	English	21	101	8	21	8
4	Korean (kor)	English	206	386	9	67	8
5	Lithuanian (lit)	English	42	108	8	29	8
6	Arabic (ara)	English	214	308	8	69	8
7	Czech (ces)	English	103	181	8	47	8
8	Esperanto (epo)	English	7	21	8	10	6
9	Finnish (fin)	English	24	77	8	22	8
10	Albanian (sqi)	English	45	93	8	30	8
11	Vietnamese (vie)	English	172	66	8	61	8
12	French (fra)	English	192	158	8	65	8	37.74
13	Estonian (est)	English	11	39	8	14	7	9.9
14	Macedonian (mkd)	English	25	61	8	23	8	21.75
15	Bosnian (bos)	English	6	23	8	9	6	32.42
16	Swedish (swe)	English	57	84	8	34	8	33.92
17	Polish (pol)	English	176	267	8	63	8	21.51
18	Persian (fas)	English	151	148	8	57	8	24.5
19	Kurdish (kur)	English	10	39	8	14	7	6.86
20	Hungarian (hun)	English	147	305	8	56	8	22.67
21	Slovenian (slv)	English	20	58	8	20	8	14.18
22	Romanian (ron)	English	181	205	8	63	8	32.42
23	Russian (rus)	English	208	291	8	68	8	22.6
24	Serbian (srp)	English	137	239	8	54	8	30.41
25	Tamil (tam)	English	6	27	8	10	6	1.82
26	Kazakh (kaz)	English	3	15	8	7	5	2.05
27	Marathi (mar)	English	10	29	8	13	7	3.68
28	Ukrainian (ukr)	English	108	191	8	48	8	24.09
29	Thai (tha)	English	98	323	8	45	8	20.34
30	Belarusian (bel)	English	5	20	8	8	5	2.85
31	Turkish (tur)	English	182	304	8	63	8	22.52
32	Azerbaijani (aze)	English	6	23	8	9	6	3.1
33	German (deu)	English	168	194	8	61	8	33.15
34	Bulgarian (bul)	English	174	216	8	62	8	35.78
35	Norwegian (nob)	English	16	36	8	17	7	29.63
36	Georgian (kat)	English	13	44	8	15	7	4.94
37	Danish (dan)	English	45	72	8	31	8	37.73
38	Armenian (hye)	English	21	56	8	20	8	13.97
39	Mandarin (cmn)	English	200	481	9	67	8	17.0
idx	Source Language	Target Language	Parallel Sentences	Source vocab	Source subword	Target vocab	Target subword	BLEU
40	Indonesian (ind)	English	87	76	8	43	8	27.27
41	Galician (glg)	English	10	28	8	13	7	16.84
42	Portuguese (por)	English	185	165	8	64	8	41.67
43	Urdu (urd)	English	6	13	6	10	6	3.38
44	Italian (ita)	English	205	195	8	67	8	35.67
45	Spanish (spa)	English	196	179	8	66	8	39.48
46	Greek (ell)	English	134	171	8	54	8	34.94
47	Bengali (ben)	English	5	18	8	9	6	2.79
48	Japanese (jpn)	English	204	584	9	67	8	11.42
49	Malay (msa)	English	5	13	7	9	6	3.68
50	Dutch (nld)	English	184	172	8	63	8	34.27
51	Croatian (hrv)	English	122	191	8	52	8	31.84
52	Hebrew (heb)	English	212	276	8	68	8	33.89
53	Mongolian (mon)	English	8	21	8	11	6	2.96
54	Hindi (hin)	English	19	31	8	19	7	14.25
8638
29+2:31A 962:1bs6LεI:u
8C:od a'LO S:CsO 8E8E E I H
:K  $\varepsilon 6\varepsilon :\forall \wedge \omega$  9εζ:RIs Lε:snae
8639
B Representative datasets
In this section, we show the searching results of most/least representative subsets for the rest of the five tasks.
Most representative Least representative Random Search
Figure 4: Beam search results (beam size=100) for up to the 5 most (and least) representative datasets for the remaining NLP tasks. We also show random search results of corresponding sizes.
C New Model
In this section, we show the extrapolation performance for new models on BLI, MA and the remaining systems of UD.
Figure 5: RMSE scores of BLI task from dataset-wise mean value predictor (the dashed black line in each graph) and predictors trained with experimental records of other models and 0-5 records from a new model (as indicated by the title of each graph).
Figure 6: RMSE scores of MA task from dataset-wise mean value predictor (the dashed black line in each graph) and predictors trained with experimental records of other models and 0-5 records from a new model (as indicated by the title of each graph)
8641
Figure 7: RMSE scores of UD task from dataset-wise mean value predictor (the dashed black line in each graph) and predictors trained with experimental records of other models and 0-5 records from a new model (as indicated by the title of each graph).
8642
D Feature importance
In this section, we show the plots of feature importance for all the tasks.
8643
8644
BLI: Feature Importance
Features
8645
8646