SlowGuess/ACL_2020_OCR huggingface dataset:
In this section, we describe the corpora on which we trained our model. Due to copyright constraints we were not able to use the National Corpus of Polish (NKJP), the most commonly known Polish corpus. Instead, we combined several other publicly available corpora and created a larger, but less representative corpus.
Corpus	Tokens	Texts	Avg len
NKJP	1357M	3.9M	348
OSCAR	6710M	145M	46
Open Subtitles	1084M	1.1M	985
Wikipedia	260M	1.5M	190
Wolne Lektury	41M	5.5k	7450
Allegro Articles	18M	33k	552
Total	8113M	150M	54
Table 2: Overview of corpora used to train HerBERT compared to the NKJP. Avg len is the average number of tokens per document in each corpus.
Wikipedia Polish version is among the top 10 largest Wikipedia versions. However, it is still relatively small compared to the English one (260M vs 3700M words). To extract a clean corpus from the raw Wikipedia dump, we used the tools provided by XLM.5 However, we did not lowercase the text and did not remove diacritics.
Wolne Lektury (eng. Free Readings) is an online repository of over 5k books, written by Polish authors or translated into Polish. Although the majority of the books in the dataset were written in the 19th or 20th century and they might not be fully representative of the contemporary Polish, they are free to download and can be used as a text corpus.
Open Subtitles is a multilingual parallel corpus based on movie and TV subtitles (Lison and Tiedemann, 2016) from the opensubtitles.org website. As a result, it contains very specific, mostly conversational text consisting of short sentences. Since the translations are community-sourced, they may be of substandard quality. The Polish part of the dataset is relatively large compared to the other corpora (see Table 2).
OSCAR is a multilingual corpus created by Ortiz Suárez et al. (2019) based on Common Crawl<sup>6</sup>. The original dataset lacks information about the language used in particular documents. Categorization to specific languages was automated by a classifier, splitting whole Common Crawl into many monolingual corpora.Duplicates were removed from the dataset to increase its quality. We
only use the Polish part of the corpus and use texts longer than 100 words.
Allegro Articles Additionally, we obtained over 30k articles from Allegro.pl - a popular e-commerce marketplace. They contain product reviews, shopping guides and other texts from the e-commerce domain. It is the smallest corpus we've used, but it contains high-quality documents from the domain of our interest.
4.3.2 Model
Architecture HerBERT is a multi-layer bidirectional Transformer. We use  $\mathrm{BERT}_{\mathrm{BASE}}$  architecture configuration with 12 layers, 12 attention heads and hidden dimension of 768.
Loss We train HerBERT with a MLM objective. According to the updated version of BERT, we always mask all tokens corresponding to the randomly picked word. Whole word masking objective is more difficult to learn than predicting subword tokens (Joshi et al., 2019; Martin et al., 2019).
In the original BERT training setup tokens are masked statically during the text preprocessing phase. In HerBERT, we chose to use dynamic token masking, which follows the training setup of the RoBERTa model (Liu et al., 2019).
We decided not to use the NSP objective. Previous studies by Yang et al. (2019) and Liu et al. (2019) showed that this objective is too easy and does not improve performance on downstream tasks.
Data preprocessing We tokenize corpus data into subword tokens using BPE. We learn BPE splits on Wolne Lektury and a publicly available subset of National Corpus of Polish. We choose these two datasets because of their higher quality compared to the rest of our corpus. We limit the vocabulary size to 50k tokens.
Our datasets contain a lot of small fragments of coherent text that should be treated as separate documents. We remove degenerated documents that consist of less than 20 tokens from available corpora. Maximal segment length is 512 as it was originally proposed in BERT. We do not accumulate short examples into full 512 token segments because such sequences would be incoherent with frequent topic changes. The only exception to this rule is the Open Subtitles dataset, where subsequent parts of dialogues were connected to form larger documents. The aforementioned training
$^{5}$ https://github.com/facebookresearch/XLM
http://commoncrawl.org/
Model	AVG	NKJP-NER	CDSC-E	CDSC-R	CBD	PolEmo2.0-IN	PolEmo2.0-OUT	Czy wiesz?	PSC	AR
Random	28.3	20.7	59.2	0.9	11.2	27.8	28.5	18.9	30.4	56.9
LSTM	63.0	45.0	87.5	84.7	20.7	79.6	60.7	22.3	84.4	81.8
LSTM + fastText	67.7	67.3	87.8	81.6	32.8	83.2	61.1	27.5	86.5	81.4
LSTM + ELMo	76.6	93.0	88.9	90.4	50.2	88.5	72.1	28.8	92.7	85.1
LSTM + ELMo + fine-tune	76.7	93.4	89.3	91.1	47.9	87.4	70.6	30.9	93.7	86.2
LSTM + ELMo + attention	75.8	93.0	90.0	90.3	46.8	88.8	70.2	26.0	92.1	85.4
Multi-BERT	79.5	91.4	93.8	92.9	40.0	85.0	66.6	64.2	97.9	83.3
Slavic-BERT	79.8	93.3	93.7	93.3	43.1	87.1	67.6	57.4	98.3	84.3
XLM-17	80.2	91.9	93.7	92.0	44.8	86.3	70.6	61.8	96.3	84.5
HerBERT	80.5	92.7	92.5	91.9	50.3	89.2	76.3	52.1	95.3	84.5
Table 3: Baseline evaluation on KLEJ benchmark.  $AVG$  is the average score across all tasks.
setup gives us a slightly better performance on downstream tasks than simply selecting all available data.
Hyperparameters We train HerBERT using Adam optimizer (Kingma and Ba, 2014) with parameters:  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ ,  $\epsilon = 10^{-8}$ . We use learning rate burn-in over the first 500 steps, reaching a peak value of  $10^{-4}$ ; the learning rate is then linearly decayed for the rest of the training. We train the model with a batch size of 570. Her-BERT was trained for 180k steps, without showing signs of saturation.
We first compare models based on their average performance. Even though it is not the definite metric to compare models, especially as not all tasks are equally difficult, it gives a general notion of the model performance across all tasks.
In comparison to baselines based on the LSTM architecture, Transformer-based models clearly show superior performance. The only exception is ELMo, which achieves competitive results on many tasks. On two of them, the fine-tuned ELMo model achieves the best score. In general, the evaluation shows major shortcomings of multilingual pretrained BERT models for Polish, and possibly other low resource languages. Overall, every LSTM-
based baseline is still on average worse than any of the tested Transformer-based models.
The KLEJ benchmark was designed to require additional knowledge to promote general language understanding models. As expected, we observe significant increases of models quality when using pretrained word embeddings. The vanilla LSTM model achieves the average score of 63.0 while supplying it with the fastText embeddings boosts performance to 67.7. Usage of more recent, contextualized embeddings (ELMo) increases the score to 76.6.
Focusing on fewer languages seems to result in a better model. The Slavic-BERT has higher scores than Multi-BERT on seven out of nine tasks. However, without a detailed ablation study, it is difficult to infer the main reason resulting in better performance. It can also be attributed to a better tokenizer, additional Russian News corpus or longer training (the Slavic-BERT was initialized with Multi-BERT weights).
The training corpus seems to play an important role in the performance of a downstream task. Both HerBERT and ELMo models were trained mainly on web crawled texts and they excel at tasks from an online domain (CBD, PolEmo-IN, PolEmo-OUT, and AR). On the other hand, the other Transformer-based models are superior on the Czy wiesz? task. It can be related to the fact that it is a Wikipedia
based question-answering task and the aforementioned models were trained mainly on Wikipedia corpus. Interestingly, the Slavic-BERT, which was additionally trained on Russian News corpus, has a lower score on the Czy wiesz? task than MultiBERT and XLM-17.
HerBERT achieves highly competitive results compared to the other Transformer-based models. It has the best performance on average and achieves state-of-the-art results on three tasks, PolEmo-IN, PolEmo-OUT and CBD. Moreover, HerBERT has the smallest performance gap between PolEmo-IN and PolEmo-OUT, which suggests better generalization across domains. Compared to the other Transformer-based models it performs poorly on Czy wiesz? and PSC tasks.
The KLEJ benchmark proved to be challenging and diverse. There is no clear winner among evaluated models; different models perform better at different tasks. It suggests that the KLEJ benchmark is far from being solved, and it can be used to evaluate and compare future models.
We introduce the KLEJ benchmark, a comprehensive set of evaluation tasks for the Polish language understanding. Its goal is to drive the development of better NLU models, so careful selection of tasks was crucial. We mainly focused on a variety of text genres, objectives, text lengths, and difficulties, which allows us to assess the models across different axes. As a result, KLEJ benchmark proves to be both challenging and diverse, as there is no single model that outperforms others on all tasks.
We find it equally important to provide a common evaluation interface for all the tasks. For that purpose, many existing resources had to be adapted, either automatically (NKJP-NER, PSC) or manually (Czy wiesz?). to make it easier to use.
It's worth mentioning that the main weakness of creating such benchmarks is focusing only on the model performance and not the model efficiency, e.g. in terms of training data, speed or a number of parameters. It seems reasonable to derive additional benchmarks by requiring a given level of efficiency from participating models. We leave it as future work.
We also present HerBERT, a Transformer-based model trained specifically for Polish and compare it with other LSTM- and Transformer-based models. We find that it is the best on average and achieves
highest scores on three tasks. We plan to continue the work on HerBERT and use the KLEJ benchmark to guide its development.
Mikhail Arkhipov, Maria Trofimova, Yuri Kuratov, and Alexey Sorokin. 2019. Tuning multilingual transformers for language-specific named entity recognition. In Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing, pages 89-93, Florence, Italy. Association for Computational Linguistics.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019. On the cross-lingual transferability of monolingual representations.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606.
Alexis Conneau and Douwe Kiela. 2018. Senteval: An evaluation toolkit for universal sentence representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018).
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670-680, Copenhagen, Denmark. Association for Computational Linguistics.
Alexis Conneau, Rudy Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating crosslingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Sławomir Dadas, Michal Perelkiewicz, and Rafal Poświata. 2019. Evaluation of sentence representations in polish.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018).
S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339.
Arkadiusz Janz. 2019. ELMo embeddings for polish. CLARIN-PL digital repository.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. Cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
Łukasz Kobyński and Maciej Ogrodniczuk. 2017. Results of the poleval 2017 competition: Part-of-speech tagging shared task. In Proceedings of the 8th Language and Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics, pages 362-366.
Jan Kocón, Piotr Młkowski, and Monika Zasko-Zielińska. 2019. Multi-level sentiment analysis of PolEmo 2.0: Extended corpus of multi-domain consumer reviews. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 980-991, Hong Kong, China. Association for Computational Linguistics.
Katarzyna Krasnowska-Kieraś and Alina Wróblewska. 2019. Empirical linguistic study of sentence embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5729-5739.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. arXiv preprint arXiv:1901.07291.
Pierre Lison and Jörg Tiedemann. 2016. Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France. European Language Resources Association (ELRA).
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Michal Marcinczuk, Marcin Ptak, Adam Radziszewski, and Maciej Piasecki. 2013. Open dataset for development of polish question answering systems.
In Proceedings of the 6th Language & Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics, Wydawnictwo Poznanskie, Fundacja Uniwersytetu im. Adama Mickiewicza.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. The sick (sentences involving compositional knowledge) dataset for relatedness and entailment.
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villelumont de la Clergerie, Djamé Seddah, and Benoit Sagot. 2019. CamemBERT: a Tasty French Language Model. arXiv e-prints, page arXiv:1911.03894.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730.
Maciej Ogrodniczuk and Łukasz Kobyński, editors. 2018. Proceedings of the PolEval 2018 Workshop. Institute of Computer Science, Polish Academy of Sciences, Warsaw, Poland.
Maciej Ogrodniczuk and Łukasz Kobyński, editors. 2019. Proceedings of the PolEval 2019 Workshop. Institute of Computer Science, Polish Academy of Sciences, Warsaw, Poland.
Maciej Ogrodniczuk and Mateusz Kopec. 2014. The Polish Summaries Corpus. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014.
Pedro Javier Ortiz Suárez, Benoit Sagot, and Laurent Romary. 2019. Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7), Cardiff, United Kingdom. Leibniz-Institut für Deutsche Sprache.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL.
Adam Przepiórkowski. 2012. Narodowy korpus jezyka polskiego. Naukowe PWN.
Michal Ptaszynski, Agata Pieciukiewicz, and Paweł Dybała. 2019. Results of the poleval 2019 shared task 6: First dataset and open shared task for automatic cyberbullying detection in polish twitter. Proceedings of the PolEval 2019 Workshop, page 89.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad:  $100,000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392.
Barry Haddow Rico Sennrich and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Association for Computational Linguistics (ACL), pages 1715-1725.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In the Proceedings of ICLR.
Alex Wang, Ian F. Tenney, Yada Pruksachatkun, Katherin Yu, Jan Hula, Patrick Xia, Raghu Pappagari, Shuning Jin, R. Thomas McCoy, Roma Patel, Yinghui Huang, Jason Phang, Edouard Grave, Haokun Liu, Najoung Kim, Phu Mon Htut, Thibault F'evry, Berlin Chen, Nikita Nangia, Anhad Mohananey, Katharina Kann, Shikha Bordia, Nicolas Patry, David Benton, Ellie Pavlick, and Samuel R. Bowman. 2019b. jiant 1.2: A software toolkit for research on general-purpose text understanding models. http://jiant.info/.
Aleksander Wawer and Maciej Ogrodniczuk. 2017. Results of the poleval 2017 competition: Sentiment analysis shared task. In 8th Language and Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Alina Wróblewska and Katarzyna Krasnowska-Kieras. 2017. Polish evaluation dataset for compositional distributional semantics models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 784-792.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine
translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.0823.