SlowGuess/ACL_2020_OCR huggingface dataset:
Piotr Rybak<sup>1</sup>, Robert Mroczkowski<sup>1</sup>, Janusz Tracz<sup>1</sup>, and Ireneusz Gawlik<sup>1,2</sup>
1 ML Research at Allegro.pl, ul. Grunwaldzka 182, 60-166 Poznan, Poland {firstname_lastname}@allegro.pl
$^{2}$  AGH University of Science and Technology  
Faculty of Computer Science, Electronics and Telecommunications, Department of Computer Science  
al. A. Mickiewicza 30, 30-059 Kraków, Poland
In recent years, a series of Transformer-based models unlocked major improvements in general natural language understanding (NLU) tasks. Such a fast pace of research would not be possible without general NLU benchmarks, which allow for a fair comparison of the proposed methods. However, such benchmarks are available only for a handful of languages. To alleviate this issue, we introduce a comprehensive multi-task benchmark for the Polish language understanding, accompanied by an online leaderboard. It consists of a diverse set of tasks, adopted from existing datasets for named entity recognition, question-answering, textual entailment, and others. We also introduce a new sentiment analysis task for the e-commerce domain, named Allegro Reviews (AR). To ensure a common evaluation scheme and promote models that generalize to different NLU tasks, the benchmark includes datasets from varying domains and applications. Additionally, we release HerBERT, a Transformer-based model trained specifically for the Polish language, which has the best average performance and obtains the best results for three out of nine tasks. Finally, we provide an extensive evaluation, including several standard baselines and recently proposed, multilingual Transformer-based models.
The field of natural language understanding (NLU) experienced a major shift towards knowledge reusability and transfer learning, a phenomenon well established in the field of computer vision. Such a shift was enabled by recent introduction of robust, general-purpose models suitable for fine-tuning, like ELMo (Peters et al., 2018), ULMFiT (Howard and Ruder, 2018) and BERT (Devlin et al., 2019).
These models significantly improved the state-of-the-art on numerous language understanding tasks. Since then, the progress accelerated significantly and the new Transformer-based (Vaswani et al., 2017) models are being published every month to claim the latest state-of-the-art performance.
Such a pace of development would not be possible without standardized and publicly available NLU evaluation benchmarks. Among the most popular ones is the recently introduced GLUE (Wang et al., 2019a) consisting of a collection of tasks such as question answering, sentiment analysis, and textual entailment with texts coming from a diverse set of domains. Some tasks come with numerous training examples, while others have limited training data. On top of that, for some tasks, the training set represents a different domain than the test set. This promotes models that learn general language representations and are effective at transferring knowledge across various tasks and domains. The GLUE benchmark is constructed based on existing datasets, and its main contribution is the careful choice of tasks together with an online evaluation platform and a leaderboard.
Unfortunately, most of the progress in NLU is happening for English and Chinese. Other languages lack both pretrained models and evaluation benchmarks. In this paper, we introduce the comprehensive multi-task benchmark for the Polish language understanding - KLEJ (eng. GLUE, also abbreviation for Kompleksowa Lista Ewaluacje Jezykowych, eng. Comprehensive List of Language Evaluations). KLEJ consists of nine tasks and, similarly to GLUE, is constructed mostly out of existing datasets. The tasks were carefully selected to cover a wide range of genres and different aspects of language understanding. Following GLUE, to
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1191-1201 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
simplify a model evaluation procedure, we adjusted the tasks to fit into a unified scoring scheme (either text classification or regression). Alongside the benchmark, we introduce HerBERT, a Transformer-based model trained on several Polish text corpora. We compare HerBERT with a set of both standard and recently introduced NLU baselines.
1. KLEJ: A set of nine tasks constructed from both existing and newly introduced datasets used for the Polish language understanding evaluation,
2. An online platform<sup>1</sup> to evaluate and present the model results in the form of a leaderboard,
3. HerBERT: Transformer-based model for the Polish language understanding,
4. Allegro Reviews: A new sentiment analysis task for the e-commerce domain,
5. Evaluation of several LSTM-based baselines, multilingual Transformer-based models and HerBERT.
The rest of the paper is organized as follows. In Section 2, we provide an overview of related work. In Section 3, we describe the tasks that make up the KLEJ benchmark. In Section 4, we give an overview of the selected baseline methods and introduce the new Transformer-based model for Polish. In Section 5, we evaluate all models using KLEJ benchmark. Finally, we conclude our work in Section 6.
The evaluation of NLU models was always an integral part of their development. Even though there are many established tasks on which to evaluate newly proposed models, there is no strict standard specifying which one to choose. The difficulty of a fair comparison between models eventually led to the introduction of multi-task benchmarks that unify the evaluation.
One such benchmark is SentEval (Conneau and Kiela, 2018). It consists of seventeen established tasks used to evaluate the quality of sentence embeddings. Additionally, ten probing tasks are provided to detect what linguistic properties are retained in sentence embeddings. In all tasks, models
take either a single sentence embedding or a pair of sentence embeddings as the input and solve a classification (or a regression) problem. The authors released a toolkit for model evaluation. However, they do not provide a public leaderboard to compare the results of different models.
Another benchmark for evaluating models is decaNLP (McCann et al., 2018), which consists of ten pre-existing tasks. In contrast to SentEval, choice of tasks is much more diverse, ranging from machine translation, semantic parsing to summarization. All tasks have been automatically converted to a question answering format.
Finally, the GLUE benchmark (Wang et al., 2019a) proposes a set of nine tasks. All of them are constructed from existing, well-established datasets. Authors selected tasks that are more diverse and more difficult than SentEval. Otherwise, the design of the benchmark is similar to SentEval.
The aforementioned benchmarks are limited to the English language. Noteworthy attempts at providing multi-language benchmarks include XNLI dataset (Conneau et al., 2018), with the MNLI (Williams et al., 2018) dataset translated by professional translators into 14 languages. A similar effort is XQuAD (Artetxe et al., 2019) which is a translation of the SQuAD dataset (Rajpurkar et al., 2016) into 10 languages.
None of these efforts includes Polish. Other resources to evaluate the Polish language understanding models are scarce. Recently, Krasnowska-Kieras and Wróblewska (2019) prepared their version of the SentEval probing tasks for the Polish language. However, it is more suited for analyzing the sentence embeddings linguistic properties than assessing their quality.
The PolEval (Wawer and Ogrodniczuk, 2017; Kobyliński and Ogrodniczuk, 2017; Ogrodniczuk and Kobyliński, 2018, 2019) platform organizes an annual competition in natural language processing for the Polish language. During the first three editions, it assembled 11 diverse tasks and attracted over 40 teams. It could serve as the natural benchmark for the Polish language understanding, but it lacks the common interface for all tasks, making it difficult and time-consuming to use. We include one of the PolEval tasks into the KLEJ Benchmark.
Recently Dadas et al. (2019) introduced a benchmark similar to the KLEJ benchmark proposed in
$^{2}$ https://github.com/facebookresearch/ SentEval
3http://poleval.pl
<https://klejbenchmark.com
Name	Train	Dev	Test	Domain	Metrics	Objective
Single-Sentence Tasks
NKJP-NER	16k	2k	2k	Balanced corpus	Accuracy	NER classification
CDSC-R	8k	1k	1k	Image captions	Spearman corr.	Semantic relatedness
CDSC-E	8k	1k	1k	Image captions	Accuracy	Textual entailment
Multi-Sentence Tasks
CBD	10k	-	1k	Social Media	F1-Score	Cyberbullying detection
PolEmo2.0-IN	6k	0.7k	0.7k	Online reviews	Accuracy	Sentiment analysis
PolEmo2.0-OUT	6k	0.5k	0.5k	Online reviews	Accuracy	Sentiment analysis
Czy wiesz?	5k	-	1k	Wikipedia	F1-Score	Question answering
PSC	4k	-	1k	News articles	F1-Score	Paraphrase
AR	10k	1k	1k	Online reviews	1 - wMAE	Sentiment analysis
Table 1: The overview of tasks in the KLEJ benchmark. It consists almost exclusively of classification tasks, except for CDSC-R and AR which are regression.
this paper. It contains two sentiment analysis tasks, topic classification and a Polish translation of the SICK dataset (Marelli et al., 2014). Similarly to their work, we use the same sentiment analysis dataset, but transform it into a more difficult task. We also use the analogous dataset to SICK but created from scratch for the Polish language. Finally, we considered the topic classification task to be too easy to include into the proposed benchmark. Overall, KLEJ benchmark consists of nine tasks. They are more diverse, cover a wider range of objectives and evaluate not only single sentences but also whole paragraphs.
KLEJ consists of nine Polish language understanding tasks. Similarly to GLUE, we choose tasks from different domains and with different objectives. In contrast to previous benchmarks, we include several tasks that take multiple sentences as input. We decided to focus on tasks which have relatively small datasets – most of them have less than 10k training examples. Moreover, some tasks require extensive external knowledge to solve them. Such a setup promotes knowledge transfer techniques like transfer learning, instead of training separate models for each task from scratch. In effect, KLEJ supports the goal of creating a general model for the Polish language understanding. We present all tasks in the following sections and summarize them in Table 1.
3.1 NKJP-NER
We use the human-annotated part of the NKJP (Narodowy Korpus Jezyka Polskiego, eng. National Corpus of Polish) (Przepiórkowski, 2012) to create the named entity classification task.
The original dataset consists of 85k sentences, randomly selected from a much larger, balanced and representative corpus of contemporary Polish. We use existing human-annotations of named entities to convert the dataset into a named entity classification task. First, we filter out all sentences with entities of more than one type. Then, we randomly assigned sentences into training, development and test sets in such a way, that each named entity appears only in one of the three splits. We decided to split the sentences based on named entities to make the task more difficult. To increase class balance, we undersample the persName class and merge date and time classes. Finally, we sample sentences without any named entity to represent the noEntity class.
The final dataset consists of 20k sentences and six classes. The task is to predict the presence and type of a named entity. Although the named entity classification task differs from traditional NER task, it has a comparable difficulty and evaluates similar aspects of language understanding. At the same time, it follows the common technical interface as other KLEJ tasks, which makes it easy to use. We use accuracy for evaluation.
3.2 CDSC
The Compositional Distributional Semantics Corpus (Wróblewska and Krasnowska-Kieras, 2017) consists of pairs of sentences which are human-annotated for semantic relatedness and entailment. Although the main design of the dataset is inspired by SICK, it differs in details. As in SICK, the sentences come from image captions, but the set of chosen images is much more diverse as they come from 46 thematic groups. We prepared two KLEJ tasks based on the CDS Corpus.
3.2.1 CDSC-R
The first task is to predict relatedness between a pair of sentences, ranging from 0 (not related) to 5 (very related). The score is the average of scores assigned by three human annotators. We use the Spearman correlation to measure the performance of the model.
3.2.2 CDSC-E
The second task uses the textual entailment annotations to predict if the premise entails the hypothesis (entailment), negates the hypothesis (contradiction), or is unrelated (neutral). Even though there is an imbalanced label distribution (most of them are neutral) we follow Krasnowska-Kieras and Wróblewska (2019) and use accuracy as an evaluation metric.
3.3 CBD
The Cyberbullying Detection task (Ptaszynski et al., 2019) was a part of the 2019 edition of the PolEval competition<sup>4</sup>. The goal is to predict whether a given Twitter message is a case of cyberbullying. We use the dataset as-is and use F1-Score to measure the performance of a given model, following the original design of the task.
3.4 PolEmo2.0
The PolEmo2.0 (Kocon et al., 2019) is a dataset of online consumer reviews from four different domains, namely: medicine, hotels, products and university. It is human-annotated on a level of full reviews, as well as individual sentences. It consists of over 8000 reviews, about  $85\%$  of which are from the medicine and hotel domains. Each review is annotated with one of four labels: positive, negative, neutral or ambiguous. The task is to predict the correct label.
We use the PolEmo2.0 dataset to form two tasks. Both of them use the same training dataset, i.e. reviews from medicine and hotel domains, but are evaluated on a different test set.
3.4.1 In-Domain
In the first task, we use accuracy to evaluate model performance within the in-domain context, i.e. on a test set of reviews from medicine and hotels domains.
3.4.2 Out-of-Domain
In the second task, we test the model on out-of-domain reviews, i.e. from product and university domains. Since the original test sets for those domains are scarce (50 reviews each) we decided to use the original out-of-domain training set of 900 reviews for testing purposes and create the new split of development and test sets. As a result, the task consists of 1000 reviews, which is comparable in size to the in-domain test dataset of 1400 reviews.
3.5 Czy wiesz?
The Czy wiesz? (eng. Did you know?) dataset (Marcinczuk et al., 2013) consists of almost 5k question-answer pairs obtained from Czy wiesz... section of Polish Wikipedia. Each question is written by a Wikipedia collaborator and is answered with a link to a relevant Wikipedia article.
The authors of the dataset used it to build a Question Answering system. Then, they evaluated the system using 1.5k questions. For each question, they took the top 10 system responses and manually annotated if the answer was correct. Positive responses to 250 questions were further processed and only relevant continuous parts of responses were selected by human annotators. Following this procedure, we have manually extracted shorter responses from the remaining positive examples. Finally, we used these annotations to create positive question-answer pairs.
To select the most difficult negative answers, we used the byte-pair encoding (BPE) (Rico Sennrich and Birch., 2016) token overlap between a question and a possible answer. For each question, we took only four most similar negatives and removed ones with a similarity metric score below the threshold  $\tau = 0.3$ . On average, the negative answers were much longer than the positive ones. Since it could be potentially exploited by the model, we decided to balance the length of the positive and negative
4http://2019.poleval.pl/index.php/ tasks/task6
answers. To sample the most relevant part of a negative example, we used BPE based metric with an additional penalty for the number of sentences:
$$
\widehat {\mathrm {s i m}} _ {\mathrm {B P E}} = \frac {\mathrm {s i m} _ {\mathrm {B P E}}}{1 . 2 ^ {\# \mathrm {s e n t s}}}
$$
The task is to predict if the answer to the given question is correct or not. Since the dataset is highly imbalanced, we chose F1-score metric.
3.6 PSC
The Polish Summaries Corpus (PSC) (Ogrodniczuk and Kopec, 2014) is a dataset of summaries for 569 news articles. For each article, the human annotators created five extractive summaries by choosing approximately  $5\%$  of the original text. Each summary was created by a different annotator. The subset of 154 articles was also supplemented with additional five abstractive summaries each, i.e. not created from the fragments of the original article.
Based on PSC we formulate a text-similarity task. We generate the positive pairs (i.e. referring to the same article) using only those news articles which have both extractive and abstractive summaries. We match each extractive summary with two least similar abstractive ones of the same article. We use the same similarity metric as in the preparation of the Czy wiesz? dataset, calculating the BPE token overlap between the extractive and abstractive summary.
To create negative pairs, we follow a similar procedure. For each extractive summary, we find two most similar abstractive summaries, but from different articles. We remove examples with similarity below the threshold  $\tau = 0.15$ . To increase the difficulty and diversity of the task, we filter out multiple abstracts from the same article. As a result, there is at most one negative pair created from each pair of articles.
In total, we obtain around 4k examples. We randomly split the dataset into train and test based on the articles of the extracts to further increase the task's difficulty. For evaluation, we use F1-score.
3.7 AR
We introduce a new sentiment analysis dataset, named Allegro Reviews (AR), extracting 12k product reviews from Allegro.pl - a popular e-commerce marketplace. Each review is at least 50 words long and has a rating on a scale from one
(negative review) to five (positive review). The task is to predict the rating of a given review.
To counter slight class imbalance in the dataset, we propose to evaluate models using  $wMAE$ , i.e. macro-average of the mean absolute error per class. Additionally, we transform the rating to be between zero and one and report  $1 - wMAE$  to ensure consistent metric interpretation between tasks.
In this section, we present an overview of several baseline models, which we evaluated using the KLEJ benchmark. We divide these models into three main groups: (1) the LSTM-based (Hochreiter and Schmidhuber, 1997) models using pretrained word embeddings, (2) models based on Transformer architecture and (3) BERT model trained on Polish corpora. We also include the simple baseline by sampling targets from a training set.
4.1 LSTM-based models
We chose the standard Bidirectional LSTM text encoder as the base architecture. Following the GLUE experiments setup (Wang et al., 2019a) we trained it jointly as the multi-task learner on all KLEJ tasks.
The architecture consists of two parts: a shared sentence encoder and a task specific classifier. The sentence representation model is a two layer BiLSTM with 1024 hidden units, 300 dimensional word embeddings and max pooling. The classifier is an MLP with 512 dimensional hidden layer.
We perform training in two stages. First, we pretrain the whole model in a multi-task scheme. In the second stage, we freeze the sentence encoder and fine-tune the classifiers separately for each task. The initial learning rate in both phases was set to  $10^{-4}$  with linear decay down to  $10^{-5}$ . Pretraining progress is measured by the macro average of all task metrics. We train models with a batch size of 128, except for the ELMo version, which is trained with a batch size of 64. For tasks without development set, we use  $10\%$  of training examples as validation data.
We used jiant (Wang et al., 2019b) library to train the LSTM-based models and report the median performance of 5 runs.
4.1.1 Vanilla BiLSTM
The simplest version of the LSTM-based models is a BiLSTM sentence encoder with an MLP classifier
trained from scratch without any form of transfer learning, i.e. without the usage of pretrained word embeddings.
4.1.2 fastText
Before contextual word embeddings became widely adopted, models were enhanced with pretrained word vectors. To evaluate their impact on KLEJ tasks, we initialize word embeddings with fastText (Bojanowski et al., 2016) trained on Common Crawl and Wikipedia for Polish language (Grave et al., 2018).
4.1.3 ELMo
ELMo (Peters et al., 2018) is a bidirectional language model using character-level convolutions. In contrast to fastText, ELMo's embeddings capture word-level semantics in a context of the whole sentence.
We conducted more thorough experiments with ELMo embeddings. During the fine-tuning stage in training on a downstream KLEJ task, we modified the sentence encoder parameters and trained the entire architecture with only a word embedding's weights unmodified. Additionally, we experimented with the attention mechanism (Conneau et al., 2017) between all words in tasks with a pair of sentences.
We use publicly available pretrained ELMo weights for Polish language (Janz, 2019).
4.2 Transformer-based models
Recently, the best results on the GLUE benchmark were obtained by Transformer-based models inspired by the Bidirectional Encoder Representations (BERT) model. All of them are pretrained on large text corpora using some variant of Masked Language Model (MLM) objective. In this section, we describe three such models: Multilingual BERT, XLM (Lample and Conneau, 2019) and Slavic-BERT (Arkhipov et al., 2019). At the time of writing this paper, these are the only available Transformer-based models that were trained with Polish text.
To evaluate these models we fine-tune them on each task separately. For training we used the transformers (Wolf et al., 2019) library. All models were trained for 4 epochs with a batch size of 32 and using a linearly decaying learning rate scheme starting at  $2 \times 10^{-5}$  with a 100 iteration warm-up. We use Adam optimizer with parameters:
$\beta_{1} = 0.9, \beta_{2} = 0.999, \epsilon = 10^{-8}$ . We report the median performance of 5 runs.
4.2.1 Multilingual BERT
The BERT is a popular model based on the Transformer architecture trained using MLM and Next Sentence Prediction (NSP) objectives. We use the Multilingual Cased BERT model, which was trained on 104 languages (including Polish), selecting ones with the largest among all Wikipedia corpora. It uses the shared WordPiece (Wu et al., 2016) tokenizer with the vocabulary size of  $110\mathrm{k}$ .
4.2.2 XLM
The Cross-lingual Language Model (XLM) is based on BERT. It differs from BERT in that it does not use NSP objective, has more layers (16 vs 12), more attention heads (16 vs 12), larger hidden layers size (1280 vs 768) and a larger vocabulary (200k vs 110k). Moreover, the vocabulary is learned on a corpus for which the most popular languages were undersampled to balance the number of tokens between high- and low-resource languages. We use the XLM-17 model, which was trained on Wikipedia for 17 languages (including Polish).
4.2.3 Slavic-BERT
The Slavic-BERT is a BERT model trained on four Slavic languages (Polish, Czech, Russian, and Bulgarian). Contrary to previous models, Arkhipov et al. (2019) used not only Wikipedia but also the Russian News corpus. To avoid costly pretraining, the model was initialized with Multilingual BERT.
4.3 HerBERT
None of the above models was optimized for Polish and all of them were trained on Wikipedia only. We decided to combine several publicly available corpora and use them to train a Transformer-based model specifically for the Polish language.
4.3.1 Corpora