SlowGuess/ACL_2020_OCR huggingface dataset:
Gözde Gül Şahin†, Yova Kementchedjhievaα, Phillip Rust†, Iryna Gurevych†
†Ubiquitous Knowledge Processing Lab (UKP),
Department of Computer Science, Technical University of Darmstadt
$^{\alpha}$ Department of Computer Science, University of Copenhagen
$\dagger_{\mathrm{www.ukp.tu - darmstadt.de}}$
Deep neural models have repeatedly proved excellent at memorizing surface patterns from large datasets for various ML and NLP benchmarks. They struggle to achieve human-like thinking, however, because they lack the skill of iterative reasoning upon knowledge. To expose this problem in a new light, we introduce a challenge on learning from small data, PuzzLing Machines, which consists of Rosetta Stone puzzles from Linguistic Olympiads for high school students. These puzzles are carefully designed to contain only the minimal amount of parallel text necessary to deduce the form of unseen expressions. Solving them does not require external information (e.g., knowledge bases, visual signals) or linguistic expertise, but meta-linguistic awareness and deductive skills. Our challenge contains around 100 puzzles covering a wide range of linguistic phenomena from 81 languages. We show that both simple statistical algorithms and state-of-the-art deep neural models perform inadequately on this challenge, as expected. We hope that this benchmark, available at https://ukplab.github.io/PuzzLing-Machines/, inspires further efforts towards a new paradigm in NLP—one that is grounded in human-like reasoning and understanding.
Kahneman (2011) discusses the two modes of human thinking which perfectly encapsulate the current (so called System1) and the desired state (System1+System2) of the deep learning field. System1 handles tasks that humans consider fast, intuitive and automatic, such as object detection and document classification. Recent deep learning (DL) models have shown great promise at this type of tasks—thanks to large training datasets. Yet, it is through slow, rational and sequential mechanisms that human-like abstract reasoning happens,
Chikasaw	English
1. Ofi'at kowī'ā lhiyohli.	The dog chases the cat.
2. Kowī'at ofi'ā lhiyohli.	The cat chases the dog.
3. Ofi'at shoha.	The dog stinks.
4. Ihooat hattakā hollo.	The woman loves the man.
5. Lhiyohlili.	I chase her/him.
6. Salhiyohli.	She/he chases me.
7. Hilha.	She/he dances.
Now you can translate the following into Chickasaw:
The man loves the woman.
The cat stinks.
I love her/him.
Translate the following into English:
Ihooat sahollo.
Ofi'at hilha.
Kowī'ā lhiyohlili.
Table 1: The "Chickasaw" puzzle (Payne, 2005)
to enable learning from just a few examples. This System2-style modeling is still in its early stages in DL, but is recognized as a much needed next step in the field (McClelland et al., 2019; Marcus, 2020; LeCun, 2020; Bengio, 2020). To foster research in this promising direction, we propose a unique challenge on "learning from small data": PuzzLing Machines, based on the Linguistic Olympiads—one of the 13 recognized International Science Olympiads targeted at high-school students.
The PuzzLing Machines challenge is based on one of the most common puzzle types in the Linguistic Olympiads: the Rosetta Stone puzzles (Bozhanov and Derzhanski, 2013), a.k.a. translation puzzles. An example is given in Table 1. Although these puzzles take the form of a traditional "machine translation" task, they are different in many ways: Rosetta Stone puzzles contain a minimal, carefully designed set of parallel expressions (words, phrases or sentences) in a for
$^{1}$ Copyright University of Oregon, Department of Linguistics.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1241-1254
eign and in a familiar language (e.g., Chickasaw-English). This minimal set is just enough to deduce the underlying translation model, which typically involves deriving mini-grammar rules, extracting a lexicon, and discovering morphological and phonological rules. The actual task then is to translate new expressions—generally in both directions—using the model deduced from the parallel data. The assignments are carefully designed so that the expressions cannot be generated through simple analogy, but rather through the application of the discovered rules. These properties distinguish the PuzzLing Machines challenge from the modern MT task, as it relies on deductive reasoning with linguistic concepts that are central to System2, rather than exploiting statistical properties from large datasets as in System1.
The lack of reasoning skills of statistical systems has recently gained a lot of attention. Various datasets that require a wide range of background knowledge and different types of reasoning abilities have been introduced, such as ARC (Clark et al., 2018), GQA (Hudson and Manning, 2019), GLUE benchmarks (Wang et al., 2018) and SWAG (Zellers et al., 2018). Our challenge distinguishes from previous benchmarks with some key properties. First, most of these reasoning tasks require external scientific or visual knowledge, which makes it hard to measure the actual reasoning performance. On the other hand, our challenge does not rely on any external, multimodal or expert-level information. Second, and more importantly, PuzzLing challenge consists of a minimal set of examples required for solution. That means, there exists no extra training data, ensuring that exploiting surface patterns would not be possible unlike in some of existing benchmarks (Gururan-gan et al., 2018).
In summary, this paper introduces a unique challenge, PuzzLing Machines, made up of  $\sim 100$  Rosetta Stone, a.k.a translation puzzles covering 81 languages from 39 different language families based on the Linguistic Olympiads. The challenge requires System2 skills—sequential reasoning and abstraction of linguistic concepts, discussed in detail in §2. We discuss the dataset and the linguistic phenomena in the resulting dataset supported with statistics and examples in §3. In §4, we present the results of intuitive baseline methods and strong MT baselines such as Transformers encoder-decoder (Vaswani et al., 2017) with integrated pretrained
language models as applied to these puzzles. We show that, unsurprisingly, the puzzles cannot be easily or robustly solved by currently existing methods. We hope that this benchmark is going to evoke development of new deep MT/NLP models that operate in a human-like manner and reason upon linguistic knowledge, providing a new future research direction for NLP.
2 Meta-linguistics
Meta-linguistics is defined by Chomsky (1976) as "the knowledge of the characteristics and structures of language" as realised on the level of phonology, morphology, syntax and semantics. Any English speaker would likely have the linguistic capacity to produce the word undo when asked "What is the opposite of do?" Only a speaker with some level of meta-linguistic awareness, however, would further be able to reflect on the structure of the word they have produced: to identify un- as a unit that serves to negate words, to spot its similarity in function to other units like dis- and de-. He/she would also be aware that un- is not interchangeable with dis- and de-, since it attaches to the front of verbs and adjectives but not to nouns.
Meta-linguistic awareness is especially useful (and often improved) in the process of learning a new language, as it allows the learner to compare and contrast the structure and characteristics of the new language to those that he/she is already familiar with. It is desirable that systems for natural language processing possess meta-linguistic awareness, too, as that could hugely improve their cross-lingual generalizability, a problem that remains open after being approached from various engineering perspectives, often with little recourse to linguistics. However, measuring the meta-linguistic awareness of a system is not trivial. Existing probing techniques are mostly designed to measure how well neural models capture specific linguistic phenomena, e.g., whether a specific layer of an English language model can capture that undo is negative, instead of testing for meta-linguistic awareness. Our challenge takes a step further and tests whether the model can apply the underlying morphological processes, e.g. of verbal negation through prefixing. In addition, our challenge spans a wide-range of language families and covers a variety of linguistic phenomena (see §3.1), that qualifies it as a favorable testbed for measuring meta-linguistic awareness.
Let us demonstrate how meta-linguistic reasoning skills are used to solve the "Chickasaw puzzle" given in Table 1. The translation model is iteratively deduced as follows: (1) the word order in Chickasaw is Subject-Object-Verb (SOV), unlike the English SVO word order; (2) nouns take different suffixes when in a subject or object position  $(at$  and  $\tilde{a}$ , respectively); (3) verbs take a suffix for 1st person singular pronomial subject or object  $(li$  and  $sa$ , respectively). Notice that, crucially, it is not possible to learn the function of the prefix  $sa$ , which corresponds to  $me$  in English, without deducing that  $lhiyohli$  corresponds to the verb  $chases$  and that third person agency in Chickasaw is not explicitly expressed. As demonstrated, inferring a translation model requires iterative reasoning on the level of words, morphemes and syntactic abstractions (classes), or, to put things differently, it requires meta-linguistic awareness.
3 The Dataset
The puzzles from Linguistic Olympiads cover many aspects of language such as phonetics, morphology, syntax and semantics. They are carefully designed by experts according to several key criteria: (1) The puzzles should be self-contained and unambiguous, meaning that no prior knowledge in the foreign language is required, just the command of one's own native language and some level of meta-linguistic awareness and that a solution is guaranteed; (2) They should require no specialized external knowledge or formal linguistic knowledge, i.e. linguistic terms are either excluded from the instructions that accompany a puzzle or they are explicitly defined; (3) The foreign language used in a puzzle should be from a truly lesser known language family (e.g. Chickasaw, Lakhota, Khmer, Ngoni), such that there is no unfair advantage to participants whose native language is related.
We based our data collection efforts on a rich and publicly available database of language puzzles maintained by the organizers of NACLO. This resource contains puzzles from IOL and a wide range of local competitions. We only included puzzles written in English (or translated to English) to ensure a quality transcription and to enable error
analysis. Expert solutions are available for most puzzles; we excluded the rest. In addition to the translation puzzle type shown in Table 1, we also collected 'matching' puzzles. These are two-step puzzles, in which the participants first align a shuffled set of sentences to obtain parallel data, and then translate a set of unseen sentences. We converted these puzzles to the translation puzzle format by referring to the solution files to align the training sentence pairs. Appendix A.1 describes how we selected the puzzles and how we transcribed them into a machine-readable format.
The final dataset contains 96 unique puzzles from 81 languages that span 39 different language families from all over the world, as well as two creoles and two artificial languages (see Appendix A.6 for the full list). Some of the large language families have multiple representatives, e.g. there are 13 Indo-European languages, seven Austronesian and six from the Niger-Congo family. But the majority of languages are single representatives of their respective family. This genealogical diversity leads to a great diversity in the linguistic phenomena attested in the data. Some puzzles are designed to explore a specific aspect of the unknown language in isolation, e.g. case markers on demonstrative pronouns in Hungarian (Pudeyev, 2009). In general, however, the correct solution of a puzzle involves processing on the level of syntax, morphology, phonology, and semantics all at once.
3.1 Linguistic Phenomena
The foreign languages used in linguistic puzzles are purposefully chosen to demonstrate some interesting linguistic phenomena, not found in English (or in the respective source language of the puzzle) (Bozhanov and Derzhanski, 2013), resulting in a challenging, non-trivial translation process between these diverse languages and English. In this section, we outline some key linguistic properties of the languages found in the dataset, but the list is by no means exhaustive.
Syntax: Three common configurations for the order between subject (S), verb (V) and object (O) in a sentence are exemplified in the dataset: SVO, SOV and VSO. In addition to these three, our dataset covers the rather rare OSV word order: see the example in Table 5 from the Australian language Dyirbal (Semenuks, 2012).
Morphology: We see examples of highly analytic languages (e.g. Yoruba from West Africa)
<sup>2</sup>http://tangra.cs.yale.edu/naclobase/
$^{3}$ NACLO (North America), OzCLO (Australia), UKLO (UK), Olimpiáda Brasileira (Brazil), OLE (Spain), Panini (India), Russian LO, Russian Little Bear, Swedish LO, Polish LO, Estonian LO, Slovenian LO, Bulgarian LO, Netherlands LO and more.
Language	Source sentence	Target sentence	Other accepted forms
1.	Chickasaw	Hilha.	(She/He) dances.	She dances.
He dances.
2a.	Blackfoot	Nitoki'kaahpinnaan.	We.PL2- camped.	We camped.
2b.	Blackfoot	Oki'kaao'pa.	We.PL2 camped.	We camped.
3.	Wambaya	Bardbi ga bungmanya.	The old woman ran [away].	The old woman ran away.
4.	Euskara	Umea etorri da.	The child has (come/arrived).	The child has come.
The child has arrived.
Table 2: Examples of special transcription notation.
Form	nyuk	duk	nuk	buk	guk	uk
After	vowel	n	r	m	ng	other
Table 3: Variants of a possessive suffix in Wembawemba and their phonological distribution.
as well as highly polysynthetic ones (e.g. Inuktitut from Canada). Within the synthetic type, we see both agglutinative languages (e.g. Turkish) and inflectional ones (e.g. Polish). Some specific morphological properties explored in the puzzles are verbal inflection with its many categories concerning tense, aspect and mood, nominal declension and noun class systems. The aforementioned "Dyirbal" puzzle also exemplifies an interesting classification of nouns, wherein women and dangerous animals and objects are treated as one class, men and other animals constitute another class and a third class captures all remaining nouns. The choice of the articles balan and bagu in Table 5, for example, is guided by this classification.
Phonology: A wide range of phonological assimilation processes interplay with the morphological processes described above and obfuscate morpheme boundaries. These can concern voicing, nasality and vowel quality, among other features. As an example of morphological and phonological processes working together, consider the realization of pronomial possession in Australian language Wembawembda (Laughren, 2009). Unlike English, which expresses this feature with pronouns his/her/its, Wembawemba expresses it with a suffix on the noun it modifies, e.g. wutyupuk ('his/her/its) stomach'. The form of the suffix, however, depends on the ending of the noun it attaches to and can vary greatly as shown in Table 3.
Semantics: Semantics come into play when we consider the compositionality of language and figurative speech: the phrase "falepak hawei" in the
Figure 1: Box-plots for Left: Word# per language and split, Right: Sentence# per split.
Indonesian language Abui, for example, literally translates into "pistol's ear", but a more fitting translation would be "trigger" (Peguševs, 2017).
As a side note, it is important to note that while here we use extensive linguistic terminology to discuss the properties of the languages in our dataset, the high-school students who participate in Linguistic Olympiads need not and may not be familiar with any of the terminology. Their good performance depends on a well-developed metalinguistic awareness, not on formal linguistic training.
3.2 Dataset statistics
In total, 2311 parallel instances are transcribed—1559 training and 752 test.  $63\%$  of the test pairs are in the English  $\rightarrow$  foreign direction, while the rest are in the foreign  $\rightarrow$  English direction.
Statistics concerning the number of words per sentence<sup>4</sup> are shown on the left of Figure 1. The majority of both training and test pairs are fairly short, but length varies considerably. This is due to the fact that some puzzles in the dataset concern the
4We naively tokenize on space.
translation of individual words, some take scope over noun-modifier phrases and some, over entire sentences. English sentences are generally longer (median 4) than their translations (median 2). This is rather intuitive considering the synthetic nature of many of the foreign languages in the dataset, wherein a single long word in the foreign language may translate into 4-5 words on the English side, as in this translation from tʌckotoyatih in the Mexican language Zoque to the English only for the tooth.
Sentence statistics about the length of the train and test split per problem are shown on the right of Figure 1. Intuitively, train splits are bigger than test splits. However, the number of training instances varies greatly between the puzzles, which is related to a number of factors such as the difficulty and type of the task, as well as the linguistic properties of the foreign language.
3.3 Train versus Test Splits
One property of the data splits in linguistic puzzles, which diverges from the standard paradigm in machine learning, is that the input test data should not be considered "held out". On the contrary, in some cases, vocabulary items attested in the input of foreign  $\rightarrow$  English test instances may be crucial to the translation of English  $\rightarrow$  foreign test instances, and vice versa. So it is only the targets of test instances that should be truly held out. This specificity is not ubiquitous across the puzzles, but it should be accounted for by any approach to their solution, for example by building the system vocabulary over the union of the train and input test data.
We attempt to solve these puzzles with models of varying complexity, i.e. from random guessing to state-of-the-art neural machine translation systems.
Random Words (RW): Since the vocabularies of source and target languages are quite small, we test what random word picking can accomplish. We simply tokenize the training sentence pairs and then randomly choose a word from the target language's vocabulary for each token in the source sentence.<sup>5</sup>
FastAlign (FA): We use the translation alignment tool FastAlign (Dyer et al., 2013), to test
whether the puzzles can be solved by early lexical translation models (Brown et al., 1993). Since FA produces alignments for each training pair, we post-process the output to create a translation dictionary separately for each direction. We then randomly choose from the translation entries for each token in source test sentence.<sup>6</sup>
Phrase Based Statistical Machine Translation (PBSMT) Since Koehn and Knowles (2017) report that PBSMT models outperform vanilla NMT models in case of small parallel training data, we use PBSMT as one of the baselines. For the foreign  $\rightarrow$  English direction, we implement two models—one using no external mono-lingual English data and one otherwise.
We implement three different models based on Transformers (Vaswani et al., 2017) using the implementation of Ott et al. (2019). In the first scenario, we train an off-the-shelf Transformer encoder-decoder model for each direction, referred to as Transformer. Second, we use a strong pretrained English language model, RoBERTa (Liu et al., 2019), to initialize the encoder of the NMT model for English to foreign translation. Finally, for foreign to English translation, we concatenate the translation features extracted from the last Transformer decoder layer, with the language modeling features extracted from RoBERTa (Liu et al., 2019), before mapping the vectors to the output vocabulary. These models are denoted as Transformer+RoBERTa.
We first compile a subset from the puzzles that are diverse by means of languages and contain translation questions in both directions. During tuning, we use the test sentences on these puzzles to validate our models. Since our foreign languages are morphologically rich, we use BPE (Sennrich et al., 2016) to segment words into subwords. For the sentences in the foreign language, we learn the BPE from the training data, while for English sentences we use the already available GPT2-BPE dictionary to exploit English language prior. For convenience,
<sup>5</sup>We don't use frequency of the words, i.e., pick words that occur more often, since they are not that meaningful due to the tininess of the data.
<sup>6</sup>We add all aligned target phrases of the source token to the dictionary. Hence, when one target phrase is seen multiple times, it is more likely to be chosen during inference.
before we train the models, we lowercase the sentences, remove certain punctuations, remove pronoun tags and brackets, and augment training data with multiple reference translations.
PBSMT: We use Moses (Koehn et al., 2007) with default settings. We employ wikitext-103 corpus to train a 5-gram English LM for the model with access to external data. The other model only uses training sentences for the LM.
NMT: Following the suggestions for low-resource NMT systems by Sennrich and Zhang (2019), we use small and few layers and high dropout rates. Similarly we use the smallest available language model (RoBERTa Base) and freeze its parameters during training to reduce the number of trainable parameters. We tune the following hyper-parameters: BPE merge parameter, learning rate and number of epochs.
The submissions to Linguistic Olympiads are manually graded by experts. For a full mark, an exact solution has to be provided, as well as a correct and detailed discussion of the underlying processes that led to this solution, e.g., concerning findings about word-order, the function of individual morphemes, etc. Participants are also given partial marks in case of partial solutions or valid discussions. Since we don't have access to expert evaluation, we use readily available automatic machine translation measures. We also note grading of system interpretations or its solution steps as an interesting future research direction.
The first is the BLEU (Papineni et al., 2002) score since it is still the standard metric in MT. We use BLEU-2 to match the lower median of sentence lengths we observe across the English and the foreign data (see Fig 1). BLEU matches whole words rather than word pieces, which prevents us from assigning partial credit to subword matches, which could be especially relevant for foreign target languages with rich morphology. We therefore use three additional metrics that operate on the level of word pieces: CharacTER (Wang et al., 2016),  $\mathrm{ChrF}$  (Popovic, 2016) and  $\mathrm{ChrF} + +$  (Popovic, 2017). CharacTER is a measure derived from TER (Translation Edit Rate), where edit rate is calculated on character level, whereas shift rate is measured on the word level. It calculates the minimum number of character edits required to adjust a hypothesis, until the reference is matched, normalized by the
length of the hypothesis sentence. For easier comparison, we report  $1.0 - \text{charACTER}$  scores. ChrF is a simple F-measure reflecting precision and recall of the matching character n-grams.  $\mathrm{ChrF}++$  adds word unigrams and bi-grams to the standard ChrF for a higher human correlation score. We experiment with different combinations of character n-grams ( $n = 3,5$  as suggested in Popovic (2016)) and word n-grams ( $n = 0,1,2$  as suggested in Popovic (2017)).