dataesr/scientific-paragraphs-categorization huggingface dataset:
Such techniques can lead to significant improvements in performance and parameter efficiency over monolingual baselines  (Pappas and Popescu-Belis 2017). Another, highly promising source of information for modeling cross-lingual variation can be found in the field of Linguistic Typology. This discipline aims to systematically compare and document the world's languages based on the empirical observation of their variation with respect to cross-lingual benchmarks  (Comrie 1989; Croft 2003). Research efforts in this field have resulted in large typological databasesfor example, most prominently the World Atlas of Language Structures (WALS)  (Dryer and Haspelmath 2013). Such databases can serve as a source of guidance for feature choice, algorithm design, and data selection or generation in multilingual NLP.
Previous surveys on this topic have covered earlier research integrating typological knowledge into NLP  (O'Horan et al. 2016; Bender 2016) , however, there is still no consensus on the general effectiveness of this approach. For instance,  Sproat (2016)  has argued that data-driven machine learning should not need to commit to any assumptions about categorical and manually defined language types as defined in typological databases. In this article, we provide an extensive survey of typologically informed NLP methods to date, including the more recent neural approaches not previously surveyed in this area. We consider the impact of typological (including both structural and semantic) information on system performance and discuss the optimal sources for such information. Traditionally, typological information has been obtained from hand-crafted databases and, therefore, it tends to be coarse-grained and incomplete.
Recent research has focused on inferring typological information automatically from multilingual data  (Asgari and Schütze 2017, inter alia) , with the specific purpose of obtaining a more complete and finer-grained set of feature values. We survey these techniques and discuss ways to integrate their predictions into the current NLP algorithms. To the best of our knowledge, this has not yet been covered in the existing literature. In short, the key questions our paper addresses can be summarized as follows: (i) Which NLP tasks and applications can benefit from typology? (ii) What are the advantages and limitations of currently available typological databases? Can data-driven inference of typological features offer an alternative source of information? (iii) Which methods have been proposed to incorporate typological information in NLP systems, and how should such information be encoded? (iv)  To what extent does the performance of typology-savvy methods surpass typology-agnostic baselines? How does typology compare with other criteria of language classification, such as genealogy? (v) How can typology be harnessed for data selection, rule-based systems, and model interpretation? We start this survey with a brief overview of Linguistic Typology ( § 2) and multilingual NLP ( § 3).
After these introductory sections we proceed to examine the development of typological information for NLP, including that in hand-crafted typological databases and that derived through automatic inference from linguistic data ( § 4). In the same section, we also describe typological features commonly selected for application in NLP. In § 5 we discuss ways in which typological information has been integrated into NLP algorithms, identifying the main trends and comparing the performance of a range of methods. Finally, in § 6 we discuss the current limitations in the use of typology in NLP and propose novel research directions inspired by our findings.
Documentation is empirical in nature and involves collecting texts or speech excerpts, and assessing the features of a language based on their analysis. The resulting information is stored in large databases (see § 4. 1) of attribute-values (this pair is henceforth referred to as typological feature), where usually each attribute corresponds to a construction and each value to the most widespread strategy in a specific language. Analysis of cross-lingual patterns reveals that cross-lingual variation is bounded and far from random  (Greenberg 1966b). Indeed, typological features can be interdependent: The presence of one feature may implicate another (in one direction or both). This interdependence is called restricted universal, as opposed to unrestricted universals, which specify properties shared unconditionally by all languages.
In particular, they argue that such architectures are sensitive to both local (contextual) information and general patterns, as well as to their frequency of use, similarly to natural languages. Typological documentation is limited by the fact that the evidence available for each language is highly unbalanced and many languages are not even recorded in a written form. 3 However, large typological databases such as WALS  (Dryer and Haspelmath 2013)  nevertheless have an impressive coverage (syntactic features for up to 1,519 languages). Where such information can be usefully integrated in machine learning, it can provide an alternative form of guidance to manual construction of resources that are now largely lacking for low resource languages. We discuss the existing typological databases and the integration of their features into NLP models in sections 4 and 5.
Overview of Multilingual NLP The scarcity of data and resources in many languages represents a major challenge for multilingual NLP. Many state-of-the-art methods rely on supervised learning, hence their performance depends on the availability of manually crafted data sets annotated with linguistic information (e. g. , treebanks, parallel corpora) and/or lexical databases (e. g. , terminology databases, dictionaries). Although similar resources are available for key tasks in a few well-researched languages, the majority of the world's languages lack them almost entirely. This gap cannot be easily bridged: The creation of linguistic resources is a time-consuming process and requires skilled labor. Furthermore, the immense range of possible tasks and languages makes the aim of a complete coverage unrealistic.
2017). Ammar et al. (2016)  instead used language vectors as a prior for language identity or typological features. In § 5. 2, we discuss ways in which typological knowledge is used to balance private and shared neural network components and provide informative input language vectors. In § 6. 3, we argue that language vectors do not need to be limited to features extracted from typological databases, but also include automatically induced typological information  (Malaviya, Neubig, and Littell 2017, see § 4. 3).
Hand-Crafted Documentation in Typological Databases Typological databases are created manually by linguists. They contain taxonomies of typological features, their possible values, as well as the documentation of feature values for the world's languages. Major typological databases, listed in Table  1 , typically organize linguistic information in terms of universal features and language-specific values. For example, Figure  3  presents language-specific values for the feature number of grammatical genders for nouns on a world map. Note that each language is color-coded.
In this resource, which has 142 typological features in total, features 1-19 deal with phonology, 20-29 with morphology, 30-57 with nominal categories, 58-64 with nominal syntax, 65-80 with verbal categories, 81-97 and 143-144 with word order, 98-121 with simple clauses, 122-128 with complex sentences, 129-138 with the lexicon, and 139-142 with other properties. Other databases only cover features at a specific level of linguistic description. For example, both Syntactic Structures of the World's Languages (SSWL)  (Collins and Kayne 2009)  and AUTOTYP  (Bickel et al. 2017 ) focus on syntax. SSWL features are manually crafted, whereas AUTOTYP features are derived automatically from primary linguistic data using scripts. The Valency Patterns Leipzig (ValPaL) (Hartmann, Haspelmath, and Taylor 2013) provides verbs as attributes and predicate-argument structures as their values (including both valency and morphosyntactic constraints).
For example, in both Mandinka and Sliammon, the verb to laugh has a valency of 1; in other words, it requires only one mandatory argument, the subject. In Mandinka the subject precedes the verb, but there is no agreement requirement; in Sliammon, on the other hand, the word order does not matter, but the verb is required to morphologically agree with the subject. For phonology, the Phonetics Information Base and Lexicon (PHOIBLE)  (Moran, McCloy, and Wright 2014)  collates information on segments (binary phonetic features). In the Lyon-Albuquerque Phonological Systems Database (LAPSyD)  (Maddieson et al. 2013) , attributes are articulatory traits, syllabic structures, or tonal systems. Finally, StressTyp2 (Goedemans, Heinz, and der Hulst 2014) deals with stress and accent patterns. For instance, in KoromfÃl' each word's first syllable has to be stressed, but not in Cubeo.
Other databases document various aspects of semantics. The World Loanword Database (WOLD)  (Haspelmath and Tadmor 2009)  documents loanwords by identifying the donor languages and the source words. The Automated Similarity Judgment Program (ASJP)  (Wichmann, Holman, and Brown 2016)  and the Intercontinental Dictionary Series (IDS) (Key and Comrie 2015) indicate how a meaning is lexicalized across languages: For example, the concept of WORLD is expressed as mir in Russian, and as ārkiśos. i in Tocharian A. Although typological databases store abundant information on many languages, they suffer from shortcomings that limit their usefulness. Perhaps the most significant shortcoming of such resources is their limited coverage. In fact, feature values are missing for most languages in most databases.
Other shortcomings are related to feature granularity. In particular, most databases fail to account for feature value variation within each language: They report only majority value rather than the full range of possible values and their corresponding frequencies. For example, the dominant Adjectivenoun word order in Italian is adjective before noun; however, the opposite order is also attested. The latter information is often missing from typological databases. Further challenges are posed by restricted feature applicability and feature hierarchies. Firstly, some features apply, by definition, only to subsets of languages that share another feature value. For instance, WALS feature 113A documents "Symmetric and Asymmetric Standard Negation," whereas WALS feature 114A "Subtypes of Asymmetric Standard Negation.
Feature Selection from Typological Databases The databases presented above can serve as a rich source of typological information for NLP. In this section, we survey the feature sets that have been extracted from these databases in typologically informed NLP studies. In § 5. 4, we review in which ways and to what degree of success these features have been integrated in machine learning algorithms. Most NLP work incorporated a subset of word order features from WALS  (Dryer and Haspelmath 2013) , mostly for the task of syntactic dependency parsing, where word order provides crucial guidance  (Naseem, Barzilay, and Globerson 2012). The feature subsets used in different studies are shown in Figure  4. As depicted in the figure, these studies utilized quite similar word order features. The feature set first established by  Naseem, Barzilay, and Globerson (2012)  served as inspiration for subsequent works.
Schone and Jurafsky (2001)  did not resort to basic typological features, but rather to "several hundred [implicational universals] applicable to syntax" drawn from the Universal Archive  (Plank and Filiminova 1996). Typological attributes that are extracted from typological databases are typically represented as feature vectors in which each dimension encodes a feature value. This feature representation is often binarized  (Georgi, Xia, and Lewis 2010) : For each possible value v of each database attribute a, a new feature is created with value 1 if it corresponds to the actual value for a specific language and 0 otherwise. Note that this increases the number of features by a factor of To what extent do the limitations of typological databases mentioned in § 4. 1 affect the feature sets surveyed in this section? The coverage is generally broad for the languages used in these experiments, as they tend to be well-documented.
On the other hand, 5(a) shows clusters differing from language genealogy: for instance, English and Czech are merged although they belong to different genera (Germanic and Slavic). However, this feature set fails to account for fine-grained differences among related languages: For instance, French, Spanish, and Italian receive the same encoding. 7 o sum up, this section's survey on typological feature sets reveals that most experiments have taken into account a small number of databases and features therein. However, several studies did utilize a larger set of coherent features or full databases. Although well-documented languages do not suffer much from coverage issues, we showed how difficult it is to select typological features that are non-redundant with genealogy, fully discriminative, and informative.
Automatic Prediction of Typological Features The partial coverage and coarse granularity of existing typological resources sparked a line of research on automatic acquisition of typological information. Missing feature values can be predicted based on: i) heuristics from morphosyntactic annotation that pre-exists, such as treebanks, or is transferred from aligned texts ( § 4. 3. 1); ii) unsupervised propagation from other values in a database based on clustering or language similarity metrics ( § 4. 3. 2); iii) supervised learning with Bayesian models or artificial neural networks ( § 4. 3. 3); or iv) heuristics based on co-occurrence metrics, typically applied to multi-parallel texts ( § 4. 3. 4). These strategies are summarized in Table  2. With the exception of  Naseem, Barzilay, and Globerson (2012) , who treated typological information as a latent variable, automatically acquired typological features have not been integrated into algorithms for NLP applications to date.
However, they have several advantages over manually crafted features. Unsupervised propagation and supervised learning fill in missing values in databases, thereby extending their coverage. Moreover, heuristics based on morphosyntactic annotation and co-occurrence metrics extract additional information that is not recorded in typological databases. Further, they can account for the distribution of feature values within single languages, rather than just the majority value. Finally, they do not make use of discrete cross-lingual categories to compare languages; rather, language properties are reflected in continuous representations, which is in line with their gradient nature (see § 2) 4. 3. 1 Heuristics Based on Morphosyntactic Annotation. Morphosyntactic feature values can be extracted via heuristics from morphologically and syntactically annotated texts.
Cotterell and Eisner (2017, 2018)  develop a point-process generative model of vowel inventories (represented as either IPA symbols or acoustic formants) based on some universal cognitive principles: dispersion (phonemes are as spread out as possible in the acoustic space) and focalization (some positions in the acoustic space are preferred due to the similarity of the main formants). An alternative approach to supervised prediction of typology is based on learning implicational universals of the kind pioneered by  Greenberg (1963) , with probabilistic models from existing typological databases. Using such universals, features can be deduced by modus ponens. For instance, once it has been established that the presence of "High consonant/vowel ratio" and "No front-rounded vowels" implies "No tones," the latter feature value can be deduced from the premises if those are known.
For instance, Coke, King, and Radev (2016) outperform  Wang and Eisner (2017)  for object-verb order (83A) but are inferior to it for adposition-noun (85A). In fact, each strategy is suited for different features, and requires different resources. Based on Figure  9 , the extraction of information from morphosyntactic annotation is well suited for word order features, whereas distributional heuristics from multi-parallel texts are more informative about lexicalization patterns. On the other hand, unsupervised propagation and supervised learning are general-purpose strategies. Moreover, the first two presuppose some annotated and/or parallel texts, whereas the second two need pre-existing database documentation. Strategies may be preferred according to which resources are available for a specific language.
Finally, a potential direction for future investigation emerges from this section's survey. In addition to missing value completion, automatic prediction often also accounts for the variation internal to each language. However, some strategies go even further, and "open the way for a typology where generalizations can be made without there being any need to reduce the attested diversity of categorization patterns to discrete types"  (Wälchli and Cysouw 2012). In fact, language vectors (Malaviya, Neubig, and Littell 2017; Bjerva and Augenstein 2018) and distributional information from multiparallel texts (Asgari and Schütze 2017) are promising insofar they capture latent properties of languages in a bottom-up fashion, preserving their gradient nature. This offers an alternative to hand-crafted database features: In § 6.
Moreover, they introduce (languagespecific) features for the directionality of dependents, based on combinations of the PoS tags of the head and modifiers with corresponding WALS values. This approach was further extended to tensor-based models by  Zhang and Barzilay (2015) , in order to avoid the shortcomings of manual feature selection. They induce a compact hidden representation of features and languages by factorizing a tensor constructed from their combination. The prior knowledge from the typological database enables the model to forbid the invalid interactions, by generating intermediate feature embeddings in a hierarchical structure. In particular, given n words and l dependency relations, each arc h → m is encoded as the tensor product of three feature vectors for heads Φ h ∈ R n , modifiers Φ m ∈ R n , and the arcs Φ h→m ∈ R l.
This transition-based parser selects the next action z (e. g. , SHIFT) from a pool of possible actions given its current state p t , as defined in Equation (3): P(z|p t ) = σ(g z max(0, W s t ⊕ b t ⊕ a t ⊕ l it + b) + q z ) (3) P(z|p t ) is defined in terms of a set of iteratively manipulated, densely represented data structures: a buffer b t , a stack s t , and an action history a t. The hidden representation of these modules are the output of stack-LSTMs, that are in turn fed with input word feature representations (stack and buffer) and action representations (history). The shared parameters are biased toward a particular language through language embeddings l it. The language embeddings consist of (a non-linear transformation of) either a mere one-hot identity vector or a vector of typological properties taken from WALS.
Data Selection, Synthesis, and Preprocessing Another way in which typological features are used in NLP is to guide data selection. This procedure is crucial for (1) language transfer methods, as it guides the choice of the most suitable source languages and examples; and (2) multilingual joint models, in order to weigh the contribution of each language and example. The selection is typically carried out through general language similarity metrics. For instance,  Deri and Knight (2016)  base their selection on the URIEL language typology database, considering information about genealogical, geographic, syntactic, and phonetic properties. This facilitates language transfer of grapheme-to-phoneme models, by guiding the choice of source languages and aligning phoneme inventories. Metrics for source selection can also be extracted in a data-driven fashion, without explicit reference to structured taxonomies.
Comparison In light of the performance of the described methods, to what extent can typological features benefit downstream NLP tasks and applications? To answer this key question, consider the performance scores of each model reported in Figure  10. Each model has been evaluated in the original paper in one (or more) of the four main settings, with otherwise identical architecture and hyper-parameters: with gold database features (Typology), with latently inferred typological features (Data-driven), or without both (Baseline), and with otherwise identical architecture and hyper-parameters. It is evident that typology-enriched models consistently outperform baselines across several NLP tasks. Indeed, the scores are higher for metrics that increase (Unlabeled Attachment Score, F1 Score, and BLEU) and lower for metrics that decrease (Word Error Rate, Mean Average Error, and Perplexity) with better predictions.
Extending the Usage to New Tasks and Features The trends observed in § 5 reveal that typology is integrated into NLP models mostly in the context of morphosyntactic tasks, and particularly syntactic parsing. Some exceptions include other levels of linguistic structure, such as phonology  (Tsvetkov et al. 2016; Deri and Knight 2016)  and semantics  (Bender 2016; Ponti et al. 2018a ). As a consequence, the set of selected typological features is mostly limited to a handful of word-order features from a single database, WALS. Nonetheless, the array of tasks that pertain to polyglot NLP is broad, and other typological data sets that have thus far been neglected may be relevant for them. For example, typological frame semantics might benefit semantic role labeling, as it specifies the valency patterns of predicates across languages, including the number of arguments, their morphological markers, and their order.
This information can be cast in the form of priors for unsupervised syntax-based Bayesian models  (Titov and Klementiev 2012) , guidance for alignments in annotation projection  (Padó and Lapata 2009; Van der Plas, Merlo, and Henderson 2011) , or regularizers for model transfer in order to tailor the source model to the grammar of the target language  (Kozhevnikov and Titov 2013). Cross-lingual information about frame semantics can be extracted, for example, from the Valency Patterns Leipzig database  (ValPaL). Typological information regarding lexical semantics patterns can further assist various NLP tasks by providing information about translationally equivalent words across languages. Such information is provided in databases such as the World Loanword Database (WOLD), the Intercontinental Dictionary Series (IDS), and the Automated Similarity Judgment Program (ASJP).
For instance, the parsing experiments grounded on selective sharing ( § 5. 2) could also take into consideration WALS features about Nominal Categories, Nominal Syntax, Verbal Categories, Simple Clauses, and Complex Sentences, as well as features from other databases such as SSWL, APiCS, and AUTOTYP. Likewise, models for phonological tasks  (Tsvetkov et al. 2016; Deri and Knight 2016)  could also extract features from typological databases such as LAPSyD and StressTyp2.