dataesr/scientific-paragraphs-categorization huggingface dataset:
This index finds its origin in psycholinguistic research in the middle of the twentieth century, and is meant to be "a measure of vocabulary 'flexibility' or variability"  (Johnson 1944: 1). Defined as the ratio of the number of distinct wordforms over the text length (total number of word tokens), it is influenced by a given language's propensity to shape words through morphological processes such as inflection, derivation, reduplication or compounding. Its conceptual and practical simplicity led to its dissemination and adoption as a typological index of morphological complexity  (Kettunen 2014)  either in its original formulation or more elaborated ones such as MATTR  (Covington & McFall 2010)  or MTLD  (McCarthy & Jarvis 2010). The word entropy (denoted H) is a somewhat similar index of morphological complexity stemming from the information-theoretical framework.
It is influenced by both the number of different wordforms in a text and the distribution of their relative frequency (see Section 4. 2 for details). TTR, MATTR, MTLD, and H are known as distribution-based indices, following the terminology proposed by  Bentz et al. (2016). These authors also introduced a translation-based index conceptually similar to the comparison of the relative lengths of the same text translated in several languages. Being intrinsically comparative, this kind of approaches provide an interesting way to compare how different languages distribute an identical semantic content over its words. Finally, other approaches have been elegantly derived from the concept of Kolmogorov complexity  (Juola 1998). By comparing the mathematical compressibility of different distorted variants of the same text, these distortion/compression methods aim to quantify morphological and syntactic complexities in terms of within-word and inter-word regularities respectively  (Ehret & Szmrecsanyi 2016; Juola 1998; Moscoso del Prado 2011).
Corpus description and subsampling strategy We used a subset of the Parallel Bible Corpus  (Mayer & Cysouw 2014)  which was selected and preprocessed by the organizers of the Interactive Workshop on Measuring Language Complexity (IWMLC2019), with additional preprocessing by the authors for Burmese, 2 Egyptian Arabic, and Persian. 3 The dataset 4 contains 1,150 verses which are fully parallelized with maximum overlaps across 47 typologically and geographically diverse languages (Figure  1a ). No linguistic annotation is included. However, since it is fully parallelized, we assume that the semantic content S of each verse v in a language L is equivalent for all languages. By equivalent, we mean that each verse translation is likely to convey the We have selected this corpus because it offers a sound balance between its coverage in terms of linguistic diversity (both typological and areal, see Figure  1 ) and its size, in coherence with the linguistically-informed and quantitative approach adopted here.
Our rationale is that this dataset size offers a testbed that is large enough to reveal or confirm cross-language trends while allowing to individually tag all languages in each figure. While it is also easily extendable to other languages available in the Parallel Bible Corpus, we also decided to conform to the selection made for the IWMLC2019 workshop to guarantee the coherence with other studies initiated in this framework (cf. Bentz et al. 2022 , Çöltekin & Rama 2022; Gutierrez-Vasques & Mijangos 2020). As explained in the introduction, such a parallel corpus is far from being ideal for comparing spontaneous language complexities, but it is well-adapted to our purpose of assessing the robustness of complexity indices by the subsampling approach we propose in this paper. The first aim of our study is indeed to assess whether several corpus-based metrics (described in Section 4) are robust complexity indices.
Throughout the study, we thus implement an iterated subsampling strategy by comparing, for each metric, the values estimated from the whole corpus to repeated values estimated by splitting the corpus into several smaller subsets. This way, we can both evaluate each index sensitivity to subset length and, for a given length, its sensitivity to differences in the texts themselves. More specifically, we proceed with an increasing number of subsets extracted from the Parallel Bible Corpus. For a given subset number and size, each subset contains the same number of verses across languages. The indices' robustness is thus assessed by comparing between six different configurations with respect to the total number of subsets: whole (1,150 verses), 5 (230 verses in each dataset), 10 (115 verses), 20 (57 verses), 40 (28 verses), and 60 subsets (19 verses).
With such a procedure, a complexity index is considered robust if it is characterized by a stability of the language-specific values across the six configurations, along with a limited standard deviation within each configuration. Additionally, one also expects that a robust index would keep the language ranking constant across the configurations, even if the index values are prone to a limited fluctuation. Two metrics of morphological complexity are acquired and computed from a top-down typological approach by means of two linguistic databases: Grammarbased Morphological Complexity derived from WALS (GMC_W) and Grammarbased Morphological Complexity derived from AUTOTYP (GMC_A), and a bottom-up corpus-based approach is additionally applied to the Parallel Bible Corpus to compute four indices: Word Information Density (WID), Type-Token Ratio (TTR), Measure of Textual Lexical Diversity (MTLD), word-level Entropy (H).
The score is obtained by dividing the overall sum by the total number of available linguistic features in each language. 5  Following  Kettunen (2014)  and  Sinnemäki & Di Garbo (2018)  (see also  Nichols & Bentz 2019) , another Grammar-based Morphological Complexity metric is derived from AUTOTYP (GMC_A). More specifically, we integrate the degree of inflectional synthesis of verbs. One of the linguistic features aforementioned and derived from WALS (Inflectional synthesis of the verb, 22A) was originally based on this dataset of AUTOTYP  (Bickel & Nichols 2013)  and classified languages into seven categories according to the number of grammatical categories which can be expressed by a maximally inflected verb form. Since such classification results in a loss of information as described by  Sinnemäki & Di Garbo (2018) , we decided to use the actual maximum number of inflectional categories per word.
For Malagasy and Wichí, two languages with missing information, we used a score obtained from their variant language present in AUTOTYP with a distinct Glottocode (Malagasy (mala1537) and Mataco (wich1263), respec-5. On average, there are 27 linguistic features available (out of 29) per language, with a minimum number of 23 features for Mountain Arapesh and all 29 features available for 16 languages. tively). To summarize, the dataset is studied in the light of both a comprehensive morphological complexity index (GMC_W) and a more specific one (GMC_A), in order to provide two levels of granularity in expert-based indices.
Crosslinguistic overview On a WALS-based MC scale (GMC_W index, Figure  2 ), languages distribute from -0. 92 (Vietnamese) to -0. 34 (Turkish), arguably revealing three groups according to their dominant strategy: languages with MC values higher than -½ favor inflectional coding strategies (hence MORPH strategy). Conversely below -⅔, we see that languages favoring lexical coding strategies (hence LEXICAL) span over a larger range of values. A third group consists of languages for which a more balanced strategy is observed (hence BALANCED). Please note that this trichotomy is indicative and does not pretend to identify clear-cut boundaries along the GMC_W scale but rather a hint on each language's strategic orientation. Additionally, one can expect that adding more languages to the dataset would both fill the gaps within and extend beyond the current distribution.
87) and Greenlandic (West) (GMC_W = -0. 37) sharing the same GMC_A value (5) with ten other languages, despite their differences in terms of traditional morphological typology. This first impression is confirmed by a Bayesian correlation analysis. 6 The Bayes factor (BF = 13. 19) indicates strong evidence toward a relatively low positive correlation found between the two indices (r median = 0. 37, 95% CI [0. 12, 0. 58]) as illustrated by the scatter plot in Figure  4. While a few languages oriented towards 6. All correlations reported in this paper are estimated in a Bayesian framework with the BayesFactor and bayestestR packages in R. We used the default prior options of the following two functions (correlationBF and describe_posterior) for the correlation analysis (cf. see Supplementary Information § 4.
Therefore, WID is estimated by a pairwise ratio between the number of words of verse v in English. At a global level, for each language L, WID L is obtained by averaging over the V verses in the dataset: (2) In contrast, the following three corpus-based metrics are directly calculated at the level of the entire text, without averaging nor normalizing with reference to English. Type-Token Ratio (TTR) measures the degree of lexical diversity and morphological productivity by means of vocabulary size (number of unique word types) and text length (total number of word tokens). One should note that it is known to be influenced by the text length  (Covington & McFall 2010). Following other similar studies using parallel corpus  (Bentz et al. 2016 ; Gutierrez-Vasques & Mijangos 2020, inter alia), we do not try to match text lengths across languages because they are mainly influenced by each language's morphology and wordhood, the semantic content being kept equivalent.
TTR is calculated for language Towards robust complexity indices in linguistic typology [13] L as the ratio of the number of unique word types T L over the word token count N L in the dataset: (3) A higher score of TTR denotes there are more word types generated by a language, which approximates its morphological productivity and complexity. However, as the text length gets longer, the likelihood of encountering a new word type in the text decreases. Another measure of lexical diversity, Measure of Textual Lexical Diversity (MTLD) has been proposed by  McCarthy & Jarvis (2010)  as a solution to the length-sensitivity problem of TTR. MTLD is defined as the average number of words required within a text to reach the same TTR value of 0. 72 (the threshold point of stabilization established in McCarthy & Jarvis 2010).
MTLD is calculated by using a code released by John Frens. 7 A higher score of MTLD means the text requires more words to reach the point of stabilization and therefore, it can be considered more complex in terms of morphological productivity. The advantage of MTLD is that it was designed to be less prone than TTR to variations in text length even on small dataset ranging from 100 to 2k words, thanks to its intrinsic averaging nature  (McCarthy & Jarvis 2010). We thus hypothesize that our study will confirm this enhanced robustness. Word-level (unigram) Entropy (H) is defined as the average amount of information (or unpredictability) of words and it can be estimated by Equation (  4 ), where Language L consists of a finite set of T L unique word types {W 1 , …, W T L } and P w i is the probability of ith word type, estimated from the corpus.
(4) It is another measure of morphological complexity which is also sensitive to the corpus size, as the accuracy of estimating the distribution of word probabilities depends on the corpus size. On the one hand, in  Oh (2015) , syllable unigram entropy has been shown to be more robust as the corpus size grows, with a range of convergence threshold between 50k and 70k word tokens for 4 languages. On the other hand, in  Bentz & Alikaniotis (2016) , convergence points for the word unigram block entropy of 21 languages have been established with text sizes between 20k and 60k word tokens, with an average of 38k. Since the average 7. Downloaded from https://github. com/jennafrens/lexical_diversity/, last access 2 December 2022. [14] Yoon Mi Oh and François Pellegrino number of tokens per language in our parallel corpus is 25.
A more important consequence is that the relative ranking of the languages is highly dependent on the corpus itself, as illustrated at the end of the present section in Figure  9  (leftmost panel). Each dot represents the rank (from 1 for the highest TTR to 47 for the lowest, y-axis) obtained by one language in one subset configuration on average (Whole dataset, 5, 10, 20, 40, and 60 datasets, x-axis), and for each language, an edge connects its ranks across configurations if they are not identical. Conversely, perfectly consistent rankings are displayed as gray dots and labels on the figure, without any edge. The desirable property of a consistent index, immune to fluctuation, would be to maintain rank estimations similar (leading to almost horizontal segments for small variations or no segment at all for identical ranks), which is clearly not the case with TTR, where non-marginal changes are visible for the majority of the sample.
Languages are displayed in gray when rank is preserved throughout all configurations and in orange when changes occur, with orange edges underlying the changes r ρ = 0. 362 with H), demonstrating that these corpus-based indices can be informative about a language's morphological strategy. This potential is confirmed in our sample when for each language, the whole dataset is used without subsampling to evaluate the indices (see Supplementary Information § 4. 3. 1 for a global overview of the correlations observed across the indices over the Whole set sampling configuration and § 4. 3. 2 for Bayesian correlation analyses). A median Bayesian posterior estimate r median = 0. 55, 95% CI [0. 29, 0. 72], BF = 3514, (resp. r median = 0. 58, 95% CI [0. 32, 0. 73], BF = 7720) -higher than the one reported in  Bentz et al.
5. 2 Results  Koplenig et al. (2017)  have nicely shown at a large scale (more than 1,000 languages) that there exists a statistical trade-off in how information is split between across-word and within-word regularities (Spearman correlation of at least r ρ = -0. 71 depending on the dataset considered). Projecting our language sample in a two-dimension space defined by Inter-Word Information (IWI) and WID offers a similar balance, with a correlation of r median = -0. 79, 95% CI [-0. 88, -0. 67], BF = 6. 24e+09. Since both dimensions are normalized with regard to English (at coordinates [1,1] on Figure  10 ), one can formulate a few additional comments. First, there exists a large range of variation on both axes: Greenlandic (West) conveys less than half of English-equivalent information on the across-word axis and more than twice on the within-word axis, while at the other end on the plot, Sanumá exhibits the opposite proportions.
By definition a non-parallel dataset induces the existence of crosslinguistic differences in the amount of semantic information encoded. The challenge in extending the WID approach to a nonparallel dataset is thus to neutralize those differences and, in this section, we propose a proof of concept based on the utilization of a Transformer language model to quantify this amount of semantic information (see below). Figure  12  displays the implemented procedure. Three configurations are presented in the left (Full Parallel or FP), central (Pairwise Parallel or PP), and right (Non-Parallel or NP) panels, respectively. In each panel, the datasets are schematized by a box outlined in green, English (the reference language) is denoted in a shaded box and the other languages in boxes outlined in gray.
The modifications in WID calculation induced by each configuration are shown in the formulas at the bottom of the panels. In the Full Parallel configuration (left panel), the dataset is unique and shared by all languages, including English. This configuration is the one presented in the previous sections: a common semantic information S is shared by all languages. As explained in Equation (  2 ), WID then simplifies to the ratio between the number of words in English and in any other language, and estimating S itself is unnecessary. In the following, this WID index resulting from this configuration is referred to as WID_FP in order to distinguish it from the other configurations introduced hereafter. The equation shown at the bottom of the panel summarizes WID_FP calculation and is identical to Equation (2).
In the Pairwise Parallel configuration (central panel), a different subset k is analyzed for each language k and matched to its translation in English. The semantic information is no longer the same across the languages, but for each given language k, it is shared with its English counterpart on subset k. WID_PP can be conveniently computed without explicit estimation of S k (the semantic information in subset k) since it reduces as previously to the ratio of the number of words in subset k in English and in language k. One can nevertheless expect that differences across datasets will result in fluctuations in word information density estimations. The equation shown at the bottom of the panel summarizes WID_PP calculation and its simplification. In the Non-Parallel configuration (right panel), each language, including English, is analyzed on a different subset.
9. More specifically, we estimated surprisal at the verse level with the lm-scorer package downloaded from https://github. com/simonepri/lm-scorer, last access 2 December 2022, using the GPT-2 model  (Radford et al. 2019 ). More powerful language models have been released since then, but performance optimization is not the goal of this exploratory study. This result is corroborated by the scatterplot of the 20 subsets per language in the two-dimension space defined by WID_PP and WID_NP (Figure  14 , background panel). The consistent ranking of the languages translates in a very high global correlation which is higher than one would get if semantic information was not taken into consideration, as revealed by comparing the actual correlation (red vertical line on the superimposed panel in Figure  14 ) to a distribution of simulated subsets with a 1000-fold shuffling of the semantic content associated with each subset (histogram in yellow on the same panel).
This high positive correlation is partially due to differences among languages in terms of basic properties such as the average number of words per verse for instance and can thus hide a more complex reality within each language's dataset (a phenomenon known as Simpson's paradox,  Blyth 1972). A visual inspection nevertheless reveals that a within-language positive correlation between WID_PP and WID_NP is present in 40/46 languages while a majority of their randomized counterparts logically exhibit a negative or no correlation at all (see Supplementary Information § 6. 3. 3 and § 6. 3. 4). In other words, for most languages, taking the semantic information approximated by a language model into account brings substantial information on the WID that would be calculated from the parallel version, both ruling out a pure "Simpsonian" correlation and supporting the initial assumption that surprisals calculated on English versions are informative on the semantic information present in their translated counterparts.
General discussion In this paper, we first presented our main objective consisting in evaluating to which extent language comparisons based on raw complexity indices are reliable and robust against corpus content variation. Once this methodological goal set, we briefly presented several indices that have been used as yardsticks for measuring complexity in a typological perspective. It led us to select several indices computable from raw texts and that globally encompass morphological complexity rather than just inflectional morphology (Section 2). In Section 4, we started by projecting the languages from our corpus (introduced in Section 3) into a bidimensional space defined by two complexity indices derived from WALS and AUTOTYP (GMC_W and GMC_A respectively). The moderate positive correlation present between these grammar-based indices calls for caution in their interpretation, since they only provide partial views on the overall morphological complexity.
They nevertheless offer a rich and insightful background for systematically assessing the sensitivity of four indices of morphological complexity (TTR, H, MTLD, and WID) against variation in the dataset on which they are estimated. Results show that while the indices positively correlate, partially encoding similar information, MTLD and WID are much more consistent in language complexity ranking than TTR or H across different dataset sizes and contents (results summarized in Figure  9 ). As a consequence, we insist on the fact that the interpretation of crosslinguistic differences in TTR and H should be cautious, and instead, we would suggest adopting MTLD or WID as more robust comparison metrics. Since this stress test was only performed on variable subsampled subsets of the Parallel Bible Corpus, we consider that replicating this observation on more diverse corpora will be a welcome improvement in the future.
MTLD (introduced in McCarthy & Jarvis 2010) is not conceptually different from TTR and H since it is derived from the type and token frequencies estimated in the dataset. 10 In contrast, WID (introduced in  Pellegrino et al. 2011)  10. MTLD is also conceptually similar to another variation of TTR designed to deal with disparities across corpora, namely MATTR  (Covington & McFall 2010) , briefly mentioned in Section 2. However, MATTR is based on a sliding analysis window whose length should be manually set, with a recommended minimum length of a few hundred words, which can be [30] Yoon Mi Oh and François Pellegrino relies on a normalization with regard to a reference language (set to English in our study), and thus falls into the category of translation-based indices. It is robust, even on small subsets, and natively interpretable in a crosslinguistic framework, being comparative by design.
The implemented approach nevertheless suffers from several limitations. First, by subsampling our corpus, we introduced differences in the texts used for each language, but the general style and topic remain similar (excerpts of the King James Version of the bible). It means that the intrinsic coherence existing across the verses in terms of narration and vocabulary is partially preserved, leading to a high comparability of the semantic content across the datasets. There is no problematic for the very small subsets we used here. In addition,  Wu et al. (2019)  showed that MTLD is more sensitive than MATTR in assessing meaningful differences in the context of language acquisition. Towards robust complexity indices in linguistic typology  [31]  doubt that selecting more diverse language-specific datasets would increase variation in the informational content.
Future studies will show whether the approach proposed here is viable in this context, but several encouraging observations can be drawn: (i) WID is based on densities defined as the ratio of a text informational content divided by its length. Adopting a diversified dataset will influence both the numerator and the denominator, and one can expect that language-specific regularities will preserve some invariance in WID; (ii) in the present study, each subset is very small (24 verses per language, because we needed distinct subsets for the 47 languages). Larger and more diverse corpora (both across-languages and within-language) will likely yield more robust and less text-specific estimations. (iii) Semantic embeddings  (Mikolov et al. 2013 ) are methods developed to represent texts in a vector space sensitive to syntactic and semantic relationships.