The Past, Present, and Future of Typological Databases in NLP
Authors: Emi Baylor
Esther Ploeger
Johannes Bjerva
Venue: EMNLP 2023 Findings (2023)
Keywords: typology
typological feature prediction
large language models
TL;DR: We investigate mismatches and errors in typological databases, related this to the current state of NLP, and argue for awareness of this issue.

Abstract: Typological information has the potential to be beneficial in the development of NLP models, particularly for low-resource languages. 
Unfortunately, current large-scale typological databases, notably WALS and Grambank, are inconsistent both with each other and with other sources of typological information, such as linguistic grammars. 
Some of these inconsistencies stem from coding errors or linguistic variation, but many of the disagreements are due to the discrete categorical nature of these databases. 
We shed light on this issue by systematically exploring disagreements across typological databases and resources, and their uses in NLP, covering the past and present.
We next investigate the future of such work, offering an argument that a continuous view of typological features is clearly beneficial, echoing recommendations from linguistics.
We propose that such a view of typology has significant potential in the future, including in language modeling in low-resource scenarios.

The Past, Present, and Future of Typological Databases in NLP
Emi Baylor*
Esther Ploeger*
Johannes Bjerva"
McGill University
Dept. of Computer Science
Dept. of Computer Science
Mila Quebec AI Institute
Aalborg University
Aalborg University jbjervašcs. aau.dk
emily.baylor@maiie bayloor@mail.meiil...
espl@cs .au.dk
Abstract
Typological information is meticulously organ- ised in resources such as the World Atlas of Lan- guage Structures (WALS) and Grambank. The in- formation in these databases is typically gathered from other resources, e.g., provided by field lin- guists in the form of grammars. However, these databases are severely limited by two factors: (i) they contain significant disagreements with one other, implying classification errors, and denoting a hurdle to investigation of language variation across languages and continents; and (ii) they explicitly enforce a categorical view language, where a continuous view may be more appropriate.
Typological information has the potential to be beneficial in the development of NLP mod- els, particularly for low-resource languages. Unfortunately, current large-scale typological databases, notably WALS and Grambank, are inconsistent both with each other and with other sources of typological information, such as linguistic grammars. Some of these inconsis- tencies stem from coding errors linguistic variation, but many of the disagreements are due the discrete categorical nature of these databases. We shed light on this issue by sys- lematically exploring disagreements across ty- pological databases and resources, and their uses in NLP, covering the past and present, We next investigate the future of such work, offer- ing argument that continuous view of ty- pological features is clearly beneficial, echoing recommendations from linguistics. We propose that such view of typology has significant potential in the future, including in language modeling in low-resource scenarios.
In this paper, we take long-term view ty- pology in the context of computational linguis- tics, NLP, and large language models. We outline past work the intersection of linguistic typology and NLP, lay out the current state of typological databases, and provide empirical evidence of their inconsistencies. Finally, we take look at the poten- tial future of such work, highlighting cases where continuous view of typological features is clearly appropriate. The core of our work is argue for the awareness of these limitations the field, and to encourage researchers to focus on the continuous scale view of typology.
1 Introduction
Linguistic typology is concerned with investigating languages based on their properties, including sim- ilarities and differences in grammar and meaning (Croft, 2002). This type of systematic investigation language allows for comparing and contrasting languages, providing insights into language, human cognition, and allowing for computational linguis- tic investigations into language structures. Further- more, typological properties offer promising av- enue enabling truly low-resource language tech- nology - efficiently using what we know abour languages may the key to effective transfer learn- ing, allowing language models to be as impactful for smaller communities as they are for the global elite.
2
Typological Resources
2.1 WALS
The World Atlas of Language Structures (WALS) is a large knowledge base of typological proper- ties at the lexical, phonological, syntactic and se- mantic level (Dryer and Haspelmath, 2013). The data in WALS is based on descriptions of linguistic structure from wide variety of academic works, ranging from field linguisties to grammars describ- ing the nuances of individual grammatical uses. Typically, WALS categorisations of features are ex- pressed as absolute features, disallowing features with conflicting values. For instance, languages
All authors contributed equally to this work.

---

are described as being strictly SVO word order, or strictly having certain number of vowels. We argue that this view is limited.
WALS
Grambank
Exact Match
VSO
verb-initial only
Soft Match
VSO
verb-initial and verb-final
No Match
VSO
verb-medial and verb-final
2.2 Grambank
Unlike WALS, Grambank focuses on what is possi- ble in a language, instead of what is most common. For example, WALS features 81 A and 81B present the one or two most dominant subject, object, and verb word orders (Dryer, 2013). Grambank, on the other hand, uses features 131, 132, and 133 to describe the possible word orders, with each fea- ture using binary values to indicate whether or not their given word order is present in the language (Skirgård al., 2023).
Table 1: Examples of cases of exact match, soft match, and no match.
subset of features from each database to one an- other. This is straightforward in many cases, as some features have exact counterparts in the other database, and can therefore be easily compared. Other features do not have exact counterparts.
As an example, feature 81A in WALS uses seven different labels (SOV, SVO, VSO, VOS, OVS, OSV, and 'No dominant order') to represent the most dominant word order in given language. Feature 8IB is similar, but uses five different labels ("SOV or SVO', 'VSO or VOS', 'SVO or VSO", 'SVO or VOS', and 'SOV or OVS') to represent the two most dominant word orders for languages with two dominant word orders. Comparatively, the word order features in Grambank (131, 132, and 133) have binary labels which represent whether or not verb-initial, verb-medial, or verb-final word orders are present in the language. Given these differ- ences, we use two methods to compare the word orders indicated in each database. In the first, we check whether not the labels in WALS exactly match the labels Grambank (exact match). In the second, we take into account the fact that WALS represents only the dominant word order(s) in a language, while Grambank represents all word or- ders that occur in that language (soft match). To do SO, we count as matches the languages where the WALS labels are subset of the Grambank labels. Examples of exact match, soft match, and no match cases can be found in Table 1.
2.3 Wikipedia as a Typological Resource
Wikipedia has potential as a typological resource due to its broad and in-depth linguistic coverage. There are thousands of articles detailing varying as- pects of different languages, with the articles them- selves written in variety of different languages. In this paper, we focus solely Wikipedia articles written in English.
Most of the other resources discussed here are created by trained linguists. While the requirement formal qualifications can increase the reliability of the information presented, it does limit the num- ber of people who can contribute to the resources. Wikipedia, which has a much lower barrier to con- tribution, allows much broader group of people share their linguistic knowledge. As such, it has the potential to contain relevant user-centred information that other typological resources lack,
3 Quantifying the Mismatches in Typological Databases
It is well esllbestabl that resources such WALS contain errors. For instance, Plank (2009) carry out an in-depth investigation of the features described for German, noting that 'a non-negligible propor- tion of the values assigned is found to be problem- atic, in the sense of being arbitrary or uncertain in view of analytic alternatives, unappreciative of di- alectal variation, unclear as to what has been coded, factually erroncous' (Plank, 2009). We begin by building upon this work. attempting to identify further errors and problems in WALS, by contrast- ing with another existing database, Grambank. This allows us to use a semi-automatic approach to quantify errors in typological databases. We match
The results of this initial experiment, displayed in Table 2, show that agreement between these two established typological databases is often quite low. For example, the agreement noun- adjective ordering is low as 72.63% in the Furasia Macroarea, 57. 14% in the North Amer- ica Macroarea, and 80.13% on average across all languages studied.
The main inconsistencies and errors in WALS and Grambank fall into one of three categories: fac- tual errors, differences due to differing language varicties, or simplifications introduced by the dis- cretization of data to fit the database formats.

---

Macroarea
W-Order
W-Order (soft)
Noun/Adj. Order
Num./Noun Order
Assoc. Piural
Redupt.
Mean
Africa
78.38%
95.95%
96.61%
98.11%
100.0%
0.00%
78.17%
Australia
0.00%
100.0%
100.0%
0.00%
50.0%
Eurasia
61.43%
98.10%
72.63%
95.05%
66.67%
35.48%
71.56%
North America
80,30%
100,00%
57.14%
71.43%
66.67%
23.53%
66.51%
Papunesia
72.60%
95.89%
92.98%
87.93%
0.00%
0.00%
58.23%
South America
7.84%
100.0%
66.67%%
97.62%
50.00%
64.43%
Average
58.12%
98.04%
80.13%
93.31%
64.62%
20.0%
69.04%
Table 2: Agreement between a subset WALS and Grambank feature encodings, aggregated by language macroarea.
3.1 Factual Errors
Which flights leave Chicago April twelfth and arrive in Indianapolis in the morning?
Factual errors can be either due to an error in the source grammar,
error made when encoding the data from the source into the database. Errors originating in the source grammar can be difficult identify and fix, because outside, independent information is needed. Errors made in the encoding process are casier to locate and rectify, by compar- ing the database to the source material. That being said, this process can still be time-consuming.
2. All flights from Washington DC San Fran- cisco after 5 pm on November twelfth econ- omy class.
3. Round trip air fares from Pittsburgh Philadelphia less than 1000 dollars.
4. What is the ground transportation available in Fort Worth Texas?
3.2 Linguistic Variations
Example sentences with Noun-Ad Noun-Adjective or- derings taken from the Universal Dependencies English Atis treebank.
Figure I:
Inconsistencies between data sources are not neces- sarily factual errors. Language changes over time, and varies across speakers and domains. Inconsis- tencies can occur when one source describes lan- guage from time, domain, and speaker group, while another describes language from different time, domain, or speaker group. In these cases, the sources are accurately describing the language, but are creating inconsistencies by unde--speiifying the context of the language they are describing.
English treebanks. This is explained by the fact that many these Noun-Adjective orderings are dates, examples of which can be seen in sentences and of Figure 1. The rest are either sentences written in the style of news headlines, as in sen- tence 3 in Figure 1, or sentences containing the word available adjective, as in sentence in Figure
We investigate this variation in language use across the Universal Dependencies treebanks for a small set of languages: Dutch, English, French, and Japanese. These languages represent the two dominant Noun-Adjective feature orderings, with French notably having a large degree of variation depending on the lexical items involved. Table 3 shows the preference of Ndje-vdee ordering, based on count of dependency links. Concretely, we investigate the head of any adjective's amod de- pendency, and investigate whether this before or after the adjective itself. Interestingly, French has a substantial differencc across linguistic variations, with spoken language having a slight preference for A-N ordering, as compared to the medical domain containing mostly N-A ordering.
3.3 Discretization of Data
The current formats of WALS and Grambank neces- sitate the division of linguistic features into fairly discrete categories. Even when some variation is included, is discretized. For example, WALS fea- ture 81A has label "No dominant order", which allows for the capture of languages that do not have one main way of ordering subjects, objects, and verbs. This puts a language that alternates between SVO, SOV, and VSO only in the same category language that has completely free word order, cras- ing any information the fundamental ways that these languages differ. Language does not fit into discrete categories, and foreing to do so in these databases removes crucial information (Levshina et al., 2023). This is well-established in linguistics,
Conspicuously, the first English treebank in our sample contains about 31% Noun adje--dcctive order- ings, while this is only 1% or 2% for the other

---

Language
Domain
N-A
A-N
tion. We then fine-tune pretrained LLMs, with an added classification head, predict a language's
Dutch
news
99%
wiki
99%
WALS word order feature, given that language's Wikipedia snippet as input.
English
news, nonfiction
31%
69%
news, wiki
99%
As hypothesised based on the above data issues, the fine-tuned models generally do not perform well. More often than not, the models output the same predictions baseline model that predicts
blog. social
99%
blog, email, reviews, social, weh
2%
98%
legal, news, wiki
2%
98%
spoken
99%
French
news, nonfiction
67%
33%
news, wiki
61%
39%
the majority class for the relevant WALS feature, implying the models are able to extract little none of the input typological information. These
spoken
44%
56%
legal, news, wiki
67%
33%
spoken
34%
66%
medical, news, nonfiction, wiki
72%
28%
results can be found in Table
blog, news, reviews, wiki
66%
34%
Japanese
blog. fiction, news, nonfiction, wiki
13%
87%
Model
Accuracy
blog, news
5%
95%
news, wiki
0%
100%
Baseline (majority class)
52. 16%
bert-base-uncased
59.05%
Table 3: Variation in noun-adjective order in UD tree- banks across domains.
roberta-base
52.16%
xlm (xlm-mll--nn-2048)
56.11%
Table 4: Large language model results when fine-tuned classify English Wikipedia articles about languages into their WALS word order feature class (feature 81A).
and has also been pointed out potential future avenue for research in computational linguistics and NLP (Bjerva and Augenstein, 2018; Bjerva al., 2019a). This further corroborated by our results in Table 3, where even languages with strict A-N ordering such as Dutch, English, and Japanese show some variety.
5
The Past, Present, and Future of Typology NLP
Discretization in databases often leads to incon- sistencies between these categorical databases and written sources that can be more nuanced. For example, WALS classifies Hungarian having "No dominant order" of subjects, objects, and verbs. The Wikipedia article about Hungarian, however, states "The neutral word order is sub- ject-verb-object (SVO). However, Hungarian is a topic-prominent language, and so has a word order that depends not only on syntax but also on the structure of the sentence".
In spite of the inconsistencies and errors we remark upon in typological databases such as WALS, there a body of research NL.P which largely appears to be unaware of these issues.
Past Considering the past years of research, typo- logical databases have mainly been used in the con- text fcature predictions. Methodologically speak- ing, features are typically predicted in the context of other features, and other languages (Daumé III and Campbell, 2007; Teh et al., 2009; Berzak et al., 2014; Malaviya et al., 2018; Bjerva et al., 2019c.a, 2020, 2019b; Vastl et al., 2020; Jäger, 2020: Choud-
4 Typological Databases and Large Language Models
hary, 2020; Gutkin and Sproat, 2020; Kumar al., 2020). That is to say, given a language where L is the set of all languages contained in a specific database, and the features of that language Fú, the setup is typically to attempt to predict some subset of features f C Fi, based on the remaining features Fj \ f. Typically, this language may be (partially) held out during training, such that a ty- pological feature prediction model is fine-tuned on L L'A. L, before being evaluated on language Varia- tions of this setup exist, with attempts to control for language relatedness in training/test sets, using ge- nealogical, areal, or structural similarities (Bjerva
L,
These data issues make it difficult to successfully incorporate typological features into language mod- cls. As simplified example, we attempt to fine- tune large language models (LLMs) to predict va- riety of languages' WALS features from Wikipedia articles about these languages. To do so, we extract the sections in these Wikipedia articles related to word ordering, using string matching to verify that they contain sufficient word order-related informa-
'https://ee http:://eewwwwwideeeeeee wikipedia. wtt//wkii/uugaa... language, accessed 16 June 2023.

---

et al., 2020; Östling and Kurfalı, 2023).
While unlikely, it is entirely possible that the re- maining features in the databases match. Further- more, the dependency count experiments for Noun- Adjective ordering assume that annotations in UD are correct.
Present more recent trend is using typolog- ical feature prediction to interpret what linguis- tic information is captured in language represen- tations of, for example, a neural language model (Malaviya et al., 2017; Choenni and Shutova, 2022; Stanczak al., 2022: Östling and Kurfah, 2023; Bjerva, 2023). Such work leans on the intuition that, if a model encapsulates typological properties. it should be possible to accurately predict these based on the model's internal representations.
Ethics Statement
No human data was gathered in this work. As this paper focuses on recommendation for future work in NLP to be aware of a core limitation in typolog- ical databases, we do not foresee any significant ethical issues stemming from this line of research.
Given the inconsistencies and errors in typologi- cal databases, it is difficult to firmly establish what may be learned about either language about NLP models by this line of research.
Acknowledgements
This work was supported by Semper Ardens: Accelerate research grant (CF:1-0454) from the Carlsberg Foundation. EB was further supported by the McGill University Graduate Mobility Award to travel to AAU to carry out this work.
Future Recent work in linguistiess has argued for a gradient approach to word order, or in other words, a continuous scale view on this type of ty- pological features (Levshina et al., 2023). We echo this recommendation, corroborated by the fact that languages typically lie on continuum in terms of word orders. Furthermore, lack of this view is the root cause of some of the disagreements between established linguistic re- sources. We provide the added perspective that taking this continuous view will also potentially lead to improvements in incorporating typological features in NLP models. Based on this line of ar- gumentation, it is perhaps not so surprising that to date, work on incorporating typological features in NLP models has only led to limited effects on performance (Ponti al., 2019; Üstün et al., 2022).
References
argue that the
Yevgeni Berzak, Roi Reichart, and Boris Katz, 2014. Reconstructing native language typology from for- eign language usage. In Proceedings the Eigh- teenth Conference on Computational Natural Lan- guage Learning, pages 21-29, Ann Arbor, Michigan. Association for Computational Linguisties.
Johannes Bjerva. 2023. The role of typological feature prediction in NLP and linguistics. Computational Linguistics.
Johannes Bjerva and Isabelle Augenstein. 2018. Track- ing typological traits of Uralic languages in dis- tributed language representations. In Proceedings of the Fourth International Workshop on Computa- tional Linguistics of Uralic Languages, pages 76- 86, Helsinki, Finland. Association for Computational Linguistics.
6 Conclusion
Typological information has the potential to im- prove downstream NLP tasks, particularly for low- resource languages. However, this can only happen if the typological data we have access to is reliable. We argue that a continuous view of typological fea- tures is necessary in order for this to be the case, and for typology to truly have an impact in low- resource NLP and language modelling. Until this is amended, we argue that it is important that the community aware of these inherent limitations when using typology for interpretability of models.
Johannes Bjerva, Yova Kementchedjhtivaa, Ryan Cot- terell, and Isabelle Augenstein. 2019a. probabilis- generative model of linguistic typology. Pro- ceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 1529-1540, Min- ncapolis, Minnesota. Association for Computational Linguistics.
Johannes Bjerva, Yova Kementchedjhieva, Ryan Cot- terell, and Isabelle Augenstein. 2019b. Uncovering probabilistic implications in typological knowledge bases. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5924-3930 Florence, Italy. Association for Compu- tational Linguistics.
Limitations
In our investigation into mismatches between ty- pological resources, we only investigate a subset.

---

Johannes Bjerva, Robert Östling, Maria Han Veiga, Jörg Tiedemann, and Isabelle Augenstein. 2019c. What do language representations really represent? Com- putational Linguistics, 45(2):381-389,
Kidd, Zoey Liu, Chiara Naccarato, Rachel Nordlinger. Anastasia Panova, and Natalia Stoynova. 2023. Why we need a gradient approach to word order. Linguistics, 61(4):825-883.
Johannes Bjerva, Elizabeth Salesky, Sabrina J. Mielke, Aditi Chaudhary, Giuseppe G. A. Celano, Rdoardo Maria Ponti, Ekaterina Vylomova, Ryan Cotterell, and Isabelle Augenstein. 2020. SIGTYP 2020 shared task: Prediction of typological features. In Proceedings of the Second Workshop on Computa- tional Research Linguistic Typology, pages 1-11, Online. Association for Computational Linguistics.
Chaitanya Malaviya, Matthew R. Gormley, and Graham Neubig. 2018. Neural factor graph models for cross- lingual morphological tagging. In Proceedings of the 56th Annual Meeting of the Association for Compu- lational Linguistics (Volume I: Long Papers), pages 2653-2666 2663, Melbourne, Australia. Association for Computational Linguistics.
Chaitanya Malaviya, Graham Neubig, and Patrick Lit- tell. 2017. Learning language representations for typology prediction. In Proceedings of the 2017 Con- ference on Empirical Methods in Natural Language Processing, pages 2529-2535, Copenhagen, Den- mark. Association for Computational Linguistics.
Rochelle Choenni and Ekaterina Shutova. 2022. Inves- tigating language relationships in multilingual sen- tence encoders through the lens of linguistic typology. Computational Linguistics, 48(3)6635-67)
Chinmay Choudhary. 2020. NUIG: Multitasking Self- attention based approach to Syp 2020 Shared Task. In Proceedings of the Second Workshop on Computa- tional Research in Linguistic Typology. Association for Computational Linguistics.
Frans Plank. 2009. WALS values evaluated.
Edoardo Maria Ponti, Helen O'horan, Yevgeni Berzak, Ivan Vulié. Roi Reichart, Thierry Poibcau, Ekate- rina Shutova, and Anna Korhonen. 2019. Modeling language variation and universals: A survey ly- pological linguistics for natural language processing. Computational Linguistics, 45(3):559 -01.
William Croft. 2002. Typology and Universals. Cam- bridge University Press.
Hal Daumé III and Lyle Campbell. 2007. A bayesian
model for discovering typological implications. In Proceedings of the 45th Annual Meeting of the Asso- ciation of Computational Linguistics. pages 65-72.
Hedvig Skirgård, Hannah J. Haynie, Damián E. Blasi, Harald Hammarström, Jeremy Collins, Jay J. Latarche, Jakob Lesage, Tobias Weber, Alena Witzlack-Makarviich,, Sam Passmore, Angela Chira, Luke Maurits, Russell Dinnage, Michael Dunn, Ger Reesink, Ruth Singer. Claire Bowern, Patience Epps, Jane Hill, Outi Vesakoski, Martine Robbeets, Noor Karolin Abbas, Daniel Auer, Nancy A. Bakker, Giulia Barbos, Robert D. Borges, Swintha Danielsen, Luise Dorenbusch, Ella Dorn, John Elliott, Gi- ada Falcone, Jana Fischer, Yustinus Ghanggo Ate, Hannah Gibson, Hans- Hans-Philipp Göbel, Jemima A. Goodall, Victoria Gruner, Andrew Harvey, Re- bekah Hayes. Leonard Heer, Roberto E. Herrera Miranda, Nataliia Hübler, Biu Huntington-Rainee Jessica K. Ivani, Marilen Johns, Erika Just, Fri Kashima, Carolina Kipf, Janina V. Klingenberg, Nikita König, Aikaterina Koti, Richard G. A. Kowa- lik, Olga Krasnoukhova, Nora L. M. Lindvall, Mandy Lorenzen, Hannah Lutzenberger. Tânia R. A. Mar- tins, Celia Mata German, Suzanne van der Meer. Jaime Montoya Samamé, Michael Müller, Saliha Muradoglu, Kelsey Necly, Johanna Nickel, Miin Norvik, Cheryl Akinyi Oluoch, Jesse Peacock, In- dia O. C. Pearey. Naomi Peck, Stephanie Petit, Sören Pieper, Mariana Poblete, Daniel Prestipino, Linda Raabe, Amna Raja, Janis Reimringer. Syd- ney C. Rey, Julia Rizaew, Eloisa Ruppert, Kim K. Salmon, Jill Sammet, Rhiannon Schembri, Lars Schlabbach, Frederick W. P. Schmidt, Amalia Skil- ton, Wikaliler Daniel Smith, Hilário de Sousa, Kristin Sverredal, Daniel Valle, Javier Vera, Judith VoB, Tim Witte, Henry Wu, Stephanie Yam, Jingting Ye, Maisie Yong. Tessa Yuditha, Roberto Zariquiey, Robert Forkel, Nicholas Evans, Stephen C. Levinson, Martin
Matthew S. Dryer. 2013. Order of subject, object and verb (v2020.3). In Matthew S. Dryer and Martin Haspelmath, editors, The World Atlas of Language Structures Online. Zenodo.
Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig.
Alexander Gutkin and Richard Sproat. 2020. NEMO:
Frequentist Inference Approach to Constrained Lin- guistic Typology Feature Prediction in SIGTYP 2020 Shared Task. In Proceedings of the Second Workshop on Computational Research in Linguistic Typology. Association for Computational Linguistics.
Gerhard Jäger. 2020. Imputing typological values via phylogenetic inference. In Proceedings of the Second Workshop on Computational Research in Linguistic Typology. Association for Computational Linguistics.
Ritesh Kumar, Deepak Alok, Akanksha Bansal, Bornini Lahiri. and Atul Kr. Ojha. 2020.
IITKGP at SIGTYP2020: Exploring rules and hybrid systems for automatic prediction of typological fea- tures. In Proceedings of the Second Workshop on Compustatonal Research in Linguistic Typology. As- sociation for Computational Linguistics.
Natalia Levshina, Savithry Namboodiripad, Mare Allassonnière-Taae Tang, Mathew Kramer, Luigi Talamo, Annemarie Verkerk, Sasha Wilmoth, Gabriela Gar- rido Rodriguez, Timothy Michael Gupton, Evan

---

Haspelmath, Simon Greenhill, Quentin Atkin- son, and Russell D. Gray. 2023. Grambank reveals the importance of genealogical constraints on linguis- tic diversity and highlights the impact of language loss. Science Advances, 916)xeadg6175.
Karolina Stanczak, Edoardo Ponti, Lucas Torroba Hen- nigen, Ryan Cotterell, and Isabelle Augenstein. 2022. Same neurons, different languages: Probing mor- phosyntax multilingual pre-trained models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1589-1598, Seattle, United States. Association for Computational Linguistics.
YW Teh, H Daumé III, and Roy. 2009. Bayesian agglomerative clustering with coalescents. In Ad- vances in Neural Information Processing Systems 20-Preccsss of the 2007 Conference.
Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gert- jan van Noord. 2022. Udapter: Typology-based lan- guage adapters for multilingual dependency parsing and sequence labeling. Computational Linguisties, 48(3):555-592.
Martin Vastl, Daniel Zeman, and Rudolf Rosa. 2020. Predicting Typological Features in WALS using Lan- guage Embeddings and Conditional Probabilities: ÜFAL Submission the SIGTYP 2020 Shared Task. Proceedings of the Second Workshop on Computa- tional Research in Linguistic Typology. Association for Computational Linguistics.
Robert Östling and Murathan Kurfahı 2023. Language Embeddings Sometimes Contain Typological Gener- alizations. Computational Linguistics, pages 1-49.


Reviews:
[{"review_id": "bRyzyFiRdp", "reviewer": "Reviewer_PNkm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing your feedback on our paper. We appreciate that you value the contributions of our paper and found our findings useful. Below, we address your main concern.\n\n> Experiments with other language models\n\nWe considered attempting the training with other types of deep learning models, however it became clear when analyzing the models’ errors that the source and target data were too inconsistent to be learned without memorization. We had previously trained on simpler models, such as basic bag-of-word models, but, as would be expected in light of the data issues, these performed poorly. \n\n> Reproducibility score\n\nTo address reproducibility concerns, we will now include detailed information to better describe our LLM training process. We will also be releasing the training data used in these experiments, as well as a repository of code needed to reproduce the experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper brings into light the typological inconsistencies and the errors that exist in the various large scale databases such as WALS and Grambank. They compare and contrast these two databases and quantify the errors that exist in them. This is essential because ensuring the accuracy and the reliability of data in these datasets is crucial for conducting credible research especially when cross-linguistic studies is done. \nThis paper also encourages future research to adopt a more continuous scale of typology than the existing enforced category nature of it. This could potentially solve some of the existing discrepancies in these large-scale typological databases. \nThe paper is also well structured with very clear and easy to understand language.", "reasons_to_reject": "Although the paper has good motivation, the experimental section falls short by a little bit. The authors report that the transformer based models couldn't perform well in the classification task on WALS features of a set of languages. Did the authors consider using any other class of language models like generative models with prompt engineering in their approach ? Also, addressing limitations mentioned in the paper itself could make for a stronger research", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper the authors study large scale typological databases such as WALS and Grambank and address the issue of inconsistent typological information among each other. The authors argue that the disagreements between the databases arise due to the categorical nature of the databases Furthermore, they point that this enforced categorical view of the language proves to be a major limitation for these databases. The authors suggest future work to focus on a continuous scale view of typology. \n\nThe main contributions in this paper are: \nThe authors explore Wikipedia as a potential resource for typological information. \nThey contrast the two databases WALS and Grambank, to identify the errors in the dataset. For this they consider the subset of the features from one database to the other and find that the main inconsistencies in WALS and Grambank fall into three categories a) factual errors, b) differing language variety and c) discretization of data and provide examples to support their claims. \nFinally the authors experiment with a set of fine tuned BERT based models to predict WALS features of languages from Wikipedia articles and conclude that these large language models perform poorly, matching the Accuracy of their simple baseline", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "I wouldn't say missing but these two are an interesting and a related read \n\nSurvey on the Use of Typological Information in Natural Language Processing Helen O'Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Anna Korhonen COLING 2016\n\nLinguistically Na ̈ıve != Language Independent: Why NLP Needs Linguistic Typology Emily M. Bender EACL 2009 Workshop on the Interaction between Linguistics and Computational Linguistics", "Typos_Grammar_Style_and_Presentation_Improvements": "Line number 25 reads  \"Linguistic typology is concerned with investigating languages languages based on their properties...\"\nLine number 152-154: \"we count as matches the languages where... \" The sentence construction seems a bit confusing. Please ignore if it was intended to be written that way."}}, {"review_id": "13AaTR4IBq", "reviewer": "Reviewer_mEdM", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reading our paper and providing your assessment of our submission. We appreciate that you find our problem formulation, empirical evidence and categorization of causes valuable contributions. Below, we address each of your concerns (reasons to reject):\n\n> Lack of proposed solutions and novelty (reason 1 and 6)\n\nWhile we agree that constructing concrete solutions to the problems we highlight is a promising future research direction, this is beyond the goal of our paper. We argue that our paper contains valuable contributions, as the mentioned issues have not been considered specifically from the perspective of NLP before. While our findings are not novel, the paper is novel because we investigate the claims with new data (Grambank, Wikipedia) and provide a classification of the issues. Our paper therefore provides a starting point for future work in solving these issues, which we will emphasize in the camera-ready version.\n\n> Limited scope (reason 2 and 5)\n\nWhile we acknowledge that experimental results with more languages, domains and features (we are unclear what the reviewer refers to with ‘contexts’) would be interesting, we argue that our experiments suffice to highlight the central issues on this topic. Additionally, adding more features would introduce several new issues: features with lower coverage may be skewed towards certain (Indo-European) languages, while it is important for our conclusions that we take a broad range of languages into account. We will add this note into our paper.\n\n> Lack of detailed comparison (reason 3)\n\nIt is unclear to us what the reviewer means with ‘detailed comparative analysis’. The purpose of our experiments is to support the claims of our position paper, and that our experiments provide sufficient details for that (i.e. results listed per feature, soft vs. exact word order matching, results split per macroarea in Table 2). We are curious what other details the reviewer deems relevant to add.\n\n> Implications on downstream tasks (reason 4)\n\nThe prediction task in this paper tests, in a straightforward way, whether or not the typological information at hand can be learned from language descriptions by deep learning models. If it had worked, we would then move on towards incorporating the information contained in the Wikipedia articles into more directly useful downstream tasks. For example, we would potentially attempt to use the information to improve low-resource language models, testing whether or not the direct linguistic information could make up for some of the lack of raw training data. That being said, the straightforward prediction task failed, indicating that the information (in conjunction with typological databases) may not be reliable enough to improve these other downstream tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses a relevant and important topic by discussing the potential impact of linguistic typology on NLP and large language models. \n\n2. The paper clearly articulates the problem it aims to address – the inconsistencies and errors in typological databases and their impact on NLP tasks. \n\n3. The paper presents empirical evidence of the inconsistencies in typological databases by comparing data from different sources. \n\n4. The authors categorize the sources of errors and limitations, classifying them into factual errors, linguistic variations, and data discretization issues. \n", "reasons_to_reject": "1. While the paper identifies the limitations and challenges of using typological databases, it provides relatively limited discussion about potential solutions to address these issues. While the authors suggest a shift towards a continuous view of typological features, they could have explored more concrete strategies or approaches that researchers could adopt to mitigate the inconsistencies and errors in the databases.\n\n2. The paper acknowledges that it only investigates a subset of typological features for certain languages, which raises questions about the generalizability of the findings. \n\n3. The paper could benefit from a more detailed comparative analysis of the specific discrepancies between different databases. \n\n4. The paper mentions that fine-tuning large language models to predict typological features from Wikipedia articles yields limited results. However, it lacks a detailed discussion of the potential implications of these results on the actual performance of NLP models in downstream tasks. \n\n5. The discussion of linguistic variations in the paper is limited to a few languages and domains. A more comprehensive analysis across a broader range of languages and contexts would provide a more nuanced understanding of how linguistic variations contribute to the inconsistencies in typological databases.\n\n6. The conclusions might not be entirely novel for experts in the field. The paper could have gone further in proposing innovative approaches or avenues for future research beyond advocating for a continuous view of typological features.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The article discusses the role of linguistic typology in investigating languages based on their properties and the potential implications for NLP and LLMs. It emphasizes the need to understand and address the limitations and inconsistencies in typological databases such as the World Atlas of Language Structures or Grambank. These databases suffer from classification errors, disagreements, and a categorical view of language features. The article advocates for a continuous view of typological features, considering the nuances and variations in language structures.\n\nThe authors present empirical evidence of the inconsistencies in typological databases by comparing data from WALS and Grambank, and they categorize the errors into factual errors, linguistic variations, and simplifications due to data discretization. They show that these issues hinder the successful incorporation of typological features into language models.\n\nThe article also highlights the past, present, and future trends in incorporating typological features in NLP, discussing research that uses these features for prediction and interpretation in language models. ", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos and other minor issues:\n\n– 025: languages languages > languages\n\n– 093: has potential > has the potential \n\n– Table 1 caption: Examples cases > Examples ofcases?\n\n– font in Table 2 is really small, there must be some way to reformet it (e.g. by adding a second row to the header)\n\n– 245: where a language model was provided the Wikipedia > where a language model was provided with the Wikipedia\n\n– 288: leans on the the intuition > leans on the intuition \n\n– 422: 'Why we need a gradient approach...' already has a proper citation (e.g. with page numbers) so please correct.\n\nIn the bibliography: please replace arXiv entries with references to published papers (e.g.: Ostling and Kurfalı's \"Language embeddings sometimes contain typological generalizations\" is now available at https://direct.mit.edu/coli/article/doi/10.1162/coli_a_00491/116637)"}}, {"review_id": "pX7UDI7cWR", "reviewer": "Reviewer_wc1P", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reading our paper and providing your assessment of our submission and. Below, we address each of your concerns:\n\n\n> Lack of novelty\n\nWhile we indeed explore areas similar to those in previous works, and indeed draw conclusions that mirror conclusions that other authors have drawn, we argue that our contribution is valuable regardless, as the mentioned issues have not been considered specifically from the perspective of NLP before. While our findings are not novel, the paper is novel because we investigate the claims with new data (Grambank, Wikipedia) and provide a classification of the issues. We argue that it is worth presenting this new supporting evidence, even if the conclusions drawn from it are not wholly novel. We will emphasize this point in the camera-ready version.\n\n> Triviality of metrics\n\nWhile the metrics used to account for matches and mismatches are indeed trivial, a more complex methodology was not required for this quantification, and was therefore not used. We refer the reviewer to the review policy, which states that ‘The goal is to solve the problem, not to solve it in a complex way. Simpler solutions are in fact preferable, as they are less brittle and easier to deploy in real-world settings.’\nOur contribution serves to empirically establish mismatches in these databases, and opens up for exploration into more complex methodology down the line, when appropriate - e.g.  if necessary for other typological features. \n\n> Single pair of databases\n\nWhile it is true that we only tested for a single pair of databases, these are two of the most frequently used typological databases in NLP. While they do not necessarily represent the broad range of all typological databases, their degree of mismatch implies that much of typology-based NLP research is based on data that is, at the very least, inconsistent. We will make this point more explicit in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written and the topic it addresses is interesting. It is worth discussing how to integrate existing linguistic resources with LLMs.\n- In general, the paper has no specific critical issues worth reporting.", "reasons_to_reject": "- Despite lacking any specific issues, the paper is found to be lacking in originality. The topics related to typological databases discussed within the paper, namely, inconsistency across databases and the inflexibility to represent gradients of frequencies of phenomena across time and domain, have already been explored in previous works. Some of these relevant works are cited in the bibliography of this paper.\n- The metrics proposed to account for matches and mismatches are rather trivial and were only tested for a single pair of databases.\n- Section 4 presents the experiments with a degree of vagueness, and there is no in-depth discussion about the results.\n- The paper's ultimate goal, which is to shed light on the limitations of typological databases when interpreting NLP models, is not fully achieved, as the reported evidence does not introduce any novel insights.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper focuses on typological databases. It discusses, in particular, their limits in terms of coverage and flexibility for representing typological properties of languages. To this aim, the authors compare WALS and Grambank to demonstrate that these resources frequently offer contrasting evidence. Furthermore, the authors show how Wikipedia can be used to acquire fine-grained information about linguistic variation within a language and briefly show an experiment where this information is used to fine-tune LLMs. ", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers of this paper found the topic to be relevant and clear, and especially found that the problems pointed out in past approaches were valid. Generally, the reviewers didn’t seem to point to many specific methodological issues that should result in lower soundness scores but found the lack of discussion in many areas to leave the paper feeling a bit shallow, noting that it could benefit from another round of revision to expand these areas. It was also noted that the scope was narrow in terms of languages and features studied, but given this is a short paper submission, I don’t think it is a major problem given that this did not seem to invalidate any of the findings in the eyes of the reviewers. Reviewers would have liked to have seen more proposed solutions to the issues raised, and in general, were not overly excited about the paper.
Decision: Accept-Findings