SlowGuess/ACL_2020_OCR huggingface dataset:
Finally, we also measure the average exact match of the puzzles, which is calculated as 1 if the prediction and reference sentences match and 0 otherwise. As it is not feasible to report and compare results on all of these metrics (nine in total), we compute the pair-wise Pearson correlation coefficient between them, and average over all pairs to arrive at the following four metrics that show the least correlation with each other: BLEU-2, CharacTER,  $\mathrm{ChrF}-3$  and exact match. We note, however, that of these four, exact match is really the most meaningful metric. Since the sentences in the dataset are rather short and the puzzles are designed to be solvable and unambiguous, an exact match should be attainable. Moreover, as the puzzles in the dataset are of varying difficulty, the average exact match score can be seen as a continuous metric.
We report the results for the best models in Fig. 2. The hyperparameter configuration and the development set results are given in Appendix A.4. The maximum exact match score among all results is only  $3.4\%$ ; and the highest scores are consistently achieved by PBSMT models on both directions and dataset splits.
The overall results for foreign  $\rightarrow$  English are generally higher than English  $\rightarrow$  foreign. This may be due to (a) having longer sentences for English; (b) the scores (except from EM) being more suitable for English (even the character-based ones) or (c) the more challenging nature of translation into foreign languages, which needs another dedicated study.
English  $\rightarrow$  Foreign: Initializing the NMT encoder with RoBERTa has severely worsened the results, compared to standard Transformer model. We believe the main reason is the imbalance between encoder (huge encoder) and the decoder (tiny decoder), that makes training very challenging. The gap between the simplest baselines (RW,
Figure 2: Main results (best viewed with color). Left: English  $\rightarrow$  foreign Right: foreign  $\rightarrow$  English.
FA) and more sophisticated models (Transformers, PBSMT) is also considerably small; FA even surpassing Transformers's CTER and ChrF performance. For most of the foreign languages, even when two words are semantically distant, there may still be significant morpheme overlap. These suggest that simple lexical alignment models (including random assignment) can achieve higher partial matching scores that hints at the unreliability of CTER and ChrF measures for the puzzles.
Foreign  $\rightarrow$  English: We observe that the gap between the simple and more sophisticated baselines are higher in this direction by means of all measures, as we would expect. Using RoBERTa features in the decoder does not hurt the performance while providing a small increase in EM score compared to standard Transformers. It should be noted that the decoder is still tiny and LM features are only incorporated via a separate linear layer at a very late stage, which prevents the imbalance problem we saw in English  $\rightarrow$  foreign.
We see similar results for the validation data with the exception that Transformer-based models achieve either higher or the same EM scores than PBSMT while surpassing PBSMT's BLEU-2 scores in foreign  $\rightarrow$  English. It supports the findings of Sennrich and Zhang (2019), drawing attention to the importance of hyper-parameter tuning for low-resource NMT models.
We perform manual error analysis on the predictions of our top two models for the Chickasaw puzzle presented in Table 1. The predicted translations are shown in Table 4. We also provide the predictions of the simple baselines in Appendix A.5 for
comparison. Although the PBSMT model is best on average, we find that for this particular puzzle, the Transformer model did much better. PBSMT had very few hits overall: it correctly chose to include the lexical items hattak and hollo in (1), but the position and inflection of the former is incorrect. In (5) and (6) there are indications of correct lexicon induction, but the overall quality of the translations is very poor both in terms of accuracy and fluency. The Transformer model, on the other hand, predicts fluent translations in both directions. In the direction from English to Chickasaw, we see that the model correctly acquired the relevant morphological patterns: subjects take suffix at, objects take suffix  $\tilde{a}$ , and, importantly, that first person agency is expressed through suffix  $li$ . The translations are still not perfect, though, due to lexical confusion: the words for cat and dog have been swapped in both (1) and (2), as well as the words for love and chase in (3). In the direction from Chickasaw to English, the Transformer's predictions remain fluent, but they hardly relate to the input. Contrary to the overall results, for this puzzle translation to English appears to be more challenging for the model.
Recently, reasoning tasks and datasets that require natural language processing have been introduced, such as common-sense reasoning in the form of pronoun resolution e.g., WSC (Levesque, 2011), multiple-choice question answering e.g., SWAG (Zellers et al., 2018) and ARC (Clark et al., 2018); inference tasks in the form of binary or multi-label classification problems e.g., the GLUE benchmarks (Wang et al., 2018); and visual reasoning in the form of question answering (Zellers et al.,
Chikasaw	English	PBSMT	Transformer
Now you can translate the following into Chickasaw:
(1)	Hattakat ihooã hollo.	The man loves the woman.	the the woman hattakã hollo	ihooat hattakã hollo
(2)	Kow’at shoha.	The cat stinks.	the lhiyohl stinks	ofi’at shoha
(3)	Holloli.	I love her/him.	i love him	lhiyohlili
Translate the following into English:
(4)	Ihooat sahollo.	The woman loves me.	ihoothe sahollo	the woman loves the man
(5)	Ofi’at hilha.	The dog dances.	the(he/she) dances	the cat chases the dog
(6)	Kow’ã lhiyohlili.	I chase the cat.	cat ch thei chase (him/her)	the dog stinks
Table 4: Predictions for the "Chickasaw" puzzle. Gold-standard target sentences are shown in yellow.
2019) e.g., GQA (Hudson and Manning, 2019). In these tasks, the required level of semantics is mostly limited to single sentences rather than a collection; almost all tasks target English; data is derived from running text and is mostly close-domain. In addition, some require external knowledge bases or high-level knowledge on physical models or experiments as in ARC classified by Boratko et al. (2018), which leaves room for accumulating errors from external parts and complicates the analysis of individual parts like reasoning.
Another body of early work on symbolic AI provides a different set of tools to model reasoning such as rule-engines, rule-induction algorithms, logic programs and case-based reasoning models (Kolodner, 1992). However, it is not trivial to represent and model our task in these frameworks, since they mostly require defining primitives, expressions, discrete features and cases. Furthermore, the strength of statistical/neural models has been repeatedly shown to surpass rule-based models. Our goal is to encourage researchers to incorporate reasoning into statistical models, rather than replacing them with symbolic models.
The field of NLP has developed deep neural models that can exploit large amounts of data to achieve high scores on downstream tasks. Still, the field lacks models that can perform human-like reasoning and generalization. To mitigate this gap, we draw inspiration from the Linguistic Olympiads that challenge the meta-linguistic and reasoning abilities of high-school students. We create a new benchmark dataset from available Linguistic Puzzles that spans over 81 languages from 39 language families, which is released at https://ukplab.github.io/PuzzLing-Machines/. We implement and evaluate simple baselines such as alignment, and state-of-the-art machine translation
models with integrated a pretrained English language model. We show that none of the models can perform well on the puzzles, suggesting that we are still far from having systems with meta-linguistic awareness and reasoning capabilities.
This work was supported by the German Research Foundation through the German-Israeli Project Cooperation (DIP, grant DA 1600/1-1 and grant GU 798/17-1). We would like to thank Liane Vogel, Marc Simon Uecker and Siddharth Singh Parihar for their great help during the project. We are grateful to Dragomir Radev for his feedback and continuous help with encoding problems encountered during puzzle transcription. We thank Adam Lopez and Ilia Kuznetsov for providing feedback on early drafts of the paper. We thank the area chairs and the senior area chair, whose comments helped us improve the paper.
Yoshua Bengio. 2020. Deep Learning for AI. Invited talk at AAAI.
Michael Boratko, Harshit Padigela, Divyendra Mikkileni, Pritish Yuvraj, Rajarshi Das, Andrew McCallum, Maria Chang, Achille Fokoue-Nkoutche, Pavan Kapanipathi, Nicholas Mattei, Ryan Musa, Kartik Talamadupula, and Michael Witbrock. 2018. A systematic classification of knowledge, reasoning, and context within the ARC dataset. In Proceedings of the Workshop on Machine Reading for Question Answering@ACL 2018, Melbourne, Australia, July 19, 2018, pages 60-70.
Bozhidar Bozhanov and Ivan Derzhanski. 2013. Rosetta stone linguistic problems. In Proceedings of the Fourth Workshop on Teaching NLP and CL, pages 1-8.
Peter F. Brown, Stephen Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathemat
ics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263-311.
Noam A Chomsky. 1976. Reflections on language. New York: Pantheon.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457.
Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-648, Atlanta, Georgia. Association for Computational Linguistics.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), pages 107-112.
Harald Hammarström, Robert Forkel, and Martin Haspelmath. 2019. Glottolog 4.1. Max Planck Institute for the Science of Human History.
Drew A. Hudson and Christopher D. Manning. 2019. GQA: A new dataset for real-world visual reasoning and compositional question answering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 6700-6709.
Daniel Kahneman. 2011. Thinking, fast and slow. Macmillan.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, ACL '07, pages 177-180, Stroudsburg, PA, USA. Association for Computational Linguistics.
Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28-39, Vancouver. Association for Computational Linguistics.
Janet L. Kolodner. 1992. An introduction to case-based reasoning. Artif. Intell. Rev., 6(1):3-34.
Mary Laughren. 2009. Wembawemba expressing possession. http://cxielamiko.narod.ru/zadachi/ozclo-09-state.pdf, OzCLO.
Yann LeCun. 2020. Self-Supervised Learning. Invited talk at AAAI.
Hector J. Levesque. 2011. The winograd schema challenge. In Logical Formalizations of Commonsense Reasoning, Papers from the 2011 AAAI Spring Symposium, Technical Report SS-11-06, Stanford, California, USA, March 21-23, 2011.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Gary Marcus. 2020. The next decade in AI: four steps towards robust artificial intelligence. CoRR, abs/2002.06177.
James L. McClelland, Felix Hill, Maja Rudolph, Jason Baldridge, and Hinrich Schütze. 2019. Extending machine language models toward human-level language understanding. CoRR, abs/1912.05877.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Tom Payne. 2005. Chickasaw. http://lingclub.mycpanel.princeton.edu/challenge/chickasaw.php, Linguistics Society of America.
Aleksejs Peguševs. 2017. Abui. https://ioling.org/booklets/iol-2017-indiv-prob.en.pdf, IOL.
Maja Popovic. 2016. chrf deconstructed: beta parameters and n-gram weights. In Proceedings of the First Conference on Machine Translation, WMT 2016, colocated with ACL 2016, August 11-12, Berlin, Germany, pages 499-504.
Maja Popovic. 2017. chrf++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, WMT 2017, Copenhagen, Denmark, September 7-8, 2017, pages 612-618.
Victor Pudeyev. 2009. Hungarian. http://tangra.cs.yale.edu/naclobase/get.cgi?pid=198&version=1, NACLO.
Arthur Semenuks. 2012. Dyirbal. https://ioling.org/booklets/iod-2012-indiv-prob.en.pdf, International Linguistic Olympiad.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725.
Rico Sennrich and Biao Zhang. 2019. Revisiting low-resource neural machine translation: A case study. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 211-221.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998-6008.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the Workshop: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2018, Brussels, Belgium, November 1, 2018, pages 353-355.
Weiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. 2016. Character: Translation edit rate on character level. In Proceedings of the First Conference on Machine Translation, WMT 2016, colocated with ACL 2016, August 11-12, Berlin, Germany, pages 505-510.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 6720-6731.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93-104, Brussels, Belgium. Association for Computational Linguistics.
A.1 Transcription of Puzzles
The puzzles are generally provided as pdf files. Many languages in the dataset use the Latin script, optionally with some diacritics. Some which use a non-Latin script (or have no writing system at all), are transcribed with IPA or transliterated into the Latin script. Only one language, Georgian, uses a non-Latin script, namely the Mkhedruli script. As there are various types of puzzles presented at the Olympiads, we identified the ones relevant to our task through automatic filtering for the keywords "translation" or "matching", and manually verified the results.
To represent linguistic puzzles in a unified, machine-readable format, we defined a JSON format shown in Appendix A.2. The relevant data was manually extracted from the PDF files and mapped to this format in a semi-automated fashion. We faced encoding issues with many of the puzzles. For some of these, the database owner kindly provided us with the source files of the pdf documents, which enabled us to generate UTF-8 encoding of the data; others we fixed manually. Some puzzles, which use pictorial scripts or are otherwise UTF-8 incompatible, were discarded.
During the transcription we came across various formats of linguistic annotation in the puzzles. This kind of information was not consistently provided across puzzles, but we included it where available, as it can be both helpful and crucial to a correct solution. In the next paragraphs, we provide details on the different types of annotated information and the standardized format we used to encode those.
Gender distinction in pronouns: When the foreign language does not mark gender on pronouns (or omits pronouns altogether), singular pronouns in the English translations are provided consistently as (he/she) and (him/her), or (he/she/it) and (his/her/its), as in Ex. 1 in Table 2. During evaluation, instances of this notation are accepted, as well as instances of the individual alternatives.
Number marking on pronouns: When the foreign language marks two levels of plurality for the second person pronoun you, they are marked accordingly as you.SG and you.PL. Some languages make a distinction between plural forms concerning two objects and plural forms concerning three or more objects. In this case, we mark pronouns
(not just you, but also we and they) with the notation .PL2 and .PL3, respectively. Some languages also make a distinction between an inclusive we 'you and me' and an exclusive we 'me and someone else'. We reserve we.PL2 for the inclusive sense, and mark the exclusive sense with we.PL2-. See examples 2a and 2b in Table 2. The notation presented here holds for both personal pronouns, e.g. you, and possessive pronouns, e.g. your. During evaluation, we disregard this notation on the side of the target language.
Zero-to-one matching: Words that are semantically implied or required by English grammar, but not directly expressed on the side of the foreign language are shown in square brackets in some of the puzzles, as in Table 2-Ex. 3. This bracketing exists only to aid the learning of a translation model. During evaluation, we remove these brackets from the target test sentences.
Notice that number marking and special notation for zero-to-one matching is not ubiquitous across the puzzles. We included it only when it was provided in the original puzzle.
Multiple reference translations: Occasionally, several possible translations are listed in a puzzle for a given word, phrase or sentence—see Table 2-Ex. 4. We represent these options inside parenthesis separated with a slash (/), e.g., (alternative1/../alternative N). Since the alternatives are of different granularity, nested bracketing may sometimes occur. During evaluation, we calculate the scores between the prediction and all possible references, and take the maximum.
Additional information Roughly half of the puzzles contain remarks on individual characters and diacritics in the inventory of the foreign language, e.g. "In the following orthography a colon (:) marks a long vowel, and the ? symbol marks a glottal stop (like the sound in the middle of uh-oh)". In many cases, the instructions state that these are pronunciation notes, i.e. they are made available only to allow the participants to vocalize the words they see on the page. On some occasions, however, they might introduce a character that is not present in the training data, but is relevant to the translation of the test sentences, e.g. the voiceless counterpart of a voiced consonant in a language with voice assimilation. As this kind of information cannot be mapped to the parallel data format, we include it in a separate field in the JSON files, directly as it
Source	balan	waymin	bambun	baNgu	jugaNgu	jamiman.
Gloss	OBJ	mother-in-law	healthy	SUBJ	sugar-SUBJ	fat-MAKE.
Target	Sugar makes the healthy mother-in-law fat.
Table 5: Example sentence in Dyibral.
appeared in the puzzles. 7
With the aforementioned guidelines, each puzzle was transcribed by one transcriber and verified by at least one other transcriber. For the test pairs, the direction of translation is stored as well, since a possible and singular solution is only guaranteed in the direction as given in the puzzle.
A.2 JSON File Format
Each puzzle is represented with a JSON file containing the following fields: SOURCE(Language, TARGET(Language, META, TRAIN and TEST. Each field is explained in Table 6.
A.3 Development Results
The results on the validation set are given in Fig. 3.
A.4 Hyperparameter Settings
The best hyperparameters found for each NMT model is given as following.  $FA$ : word to word alignments; PBSMT for English  $\rightarrow$  Foreign: word alignment with external English LM; PBSMT for Foreign  $\rightarrow$  English: BPE with 30 merge operations. For both Transformers-based models in Foreign  $\rightarrow$  English direction, we used BPE with 10 merge operations, learning rate of 0.001 and 500 epochs; while for the standard Transformer in English  $\rightarrow$  Foreign direction, BPE with 30 merge operations have been used. For all models except from Transformers with RoBERTa encoder, both the encoder and decoder had 1 layers, and all hidden dimesions were set to 128, dropout was set to 0.3, and the models were trained with Adam optimizer. For Transformer with RoBERTA LM Encoder for English  $\rightarrow$  Foreign, we have used 0.0001 learning rate with reduction on plateau, batches of size 2, dropout of 0.1, 1 layer, 64 embedding units, 128 hidden units, and BPE with 5 merge operations.
A.5 Chickasaw Additional Predictions
In Table 7, the predictions of RW and FA are shown for comparison.
A.6 List of Languages and Families
The full list for the languages and the families they belong to, as classified in WALS (Dryer and Haspelmath, 2013) and, where WALS lacks an entry, Glottolog (Hammarström et al., 2019), are given in Table 8.
<sup>7</sup>We believe that even if all instances of such remarks are ignored, the puzzles should remain mostly solvable, but we note that without this information, the ceiling for human performance would not be quite 100 percent.
Field	Definition	Example
SOURCELANG	Name of the source language	Foreign language e.g., Kiswahili, Pali
TARGETLANG	Name of the target language	English
META	Additional information about the foreign language provided in the original puzzle (as free text)	“The sound represented as û is a ‘nasalized’ vowel. It is pronounced like the ’a’ in ‘father’, but with the air passing through the nose, as in the French word ‘ban’.”
TRAIN	Parallel training sentences given as a list of lists	[["Bonjour”, “Good morning”], ["chat”, “cat”]], where the source and the target language is French and English respectively.
TEST	Parallel test sentences with direction information	[["Bonjour”, “Good morning”, >], ["chat”, “cat”, <]]. “>” implies that the translation is required from source to target language, vice versa for “<”
Figure 3: Development set results. Left: English  $\rightarrow$  foreign Right: foreign  $\rightarrow$  English