Year: 2021

Abstract:
We perform systematically and fairly controlled experiments with the 6-layer Transformer to investigate the hardness in conditional-language-modeling languages which have been traditionally considered morphologically rich (AR and RU) and poor (ZH). We evaluate through statistical comparisons across 30 possible language directions from the 6 languages of the United Nations Parallel Corpus across 5 data sizes on 3 representation levels — character, byte, and word. Results show that performance is relative to the representation granularity of each of the languages, not to the language as a whole. On the character and byte levels, we are able to eliminate statistically significant performance disparity, hence demonstrating that a language cannot be intrinsically hard. The disparity that mirrors the morphological complexity hierarchy is shown to be a byproduct of word segmentation. Evidence from data statistics, along with the fact that word segmentation is qualitatively indeterminate, renders a decades-long debate on morphological complexity (unless it is being intentionally modeled in a word-based, meaning-driven context) irrelevant in the context of computing. The intent of our work is to help effect more objectivity and adequacy in evaluation as well as fairness and inclusivity in experimental setup in the area of language and computing so to uphold diversity in Machine Learning and Artificial Intelligence research. Multilinguality is real and relevant in computing not due to canonical, structural linguistic concepts such as morphology or “words” in our minds, but rather standards related to internationalization and localization, such as character encoding — something which has thus far been sorely overlooked in our discourse and curricula.

heading: 1 INTRODUCTION

heading: 1.1 BACKGROUND AND MOTIVATION
text: Most current work on fairness in Machine Learning (ML) and Natural Language Processing (NLP) focuses on the societal biases encoded in natural language data that are propagated and amplified when they are used at scale for/as Artificial Intelligence (AI) solutions1. But little has been said or questioned about the bias, as in, the favoring of certain outcomes, implicit in our theoretical/scientific assumptions that results in the varying performance of different languages in computing.
Disparity in machine translation results For instance, results reported in Junczys-Dowmunt et al. (2016) for Phrase-Based Statistical Machine Translation (PBSMT) (Koehn et al., 2003) and Neural MT (Bahdanau et al., 2014) on the 6 official languages2 of the United Nations (UN) Parallel Corpus (Ziemski et al., 2016) indicate a disparity between EN/ES/FR and AR/RU/ZH in BLEU (Papineni et al., 2002) — translation performance in the latter group is generally worse, regardless of the MT algorithm used. AR and RU are traditionally considered morphologically complex (see e.g. Minkov et al. (2007), Seddah et al. (2010) and proceedings of related workshops in subsequent years for Statistical Parsing of Morphologically Rich Languages), and ZH morphologically frugal
1see e.g. work from conference (https://facctconference.org) and workshops in previous years on “Fairness, Accountability, and Transparency” (FAccT)
2Arabic (AR), English (EN), Spanish (ES), French (FR), Russian (RU), and Chinese (ZH)
(for its lacking determiners and plural or tense markers) (Koehn, 2005). While Koehn (2005) found translating into EN to be easier than into morphologically rich languages based on BLEU scores of word tokens from PBSMT systems of 110 language directions from the 11 Europarl languages then, Bugliarello et al. (2020) found it is easier to translate out of EN than into it based on 21 Europarl languages in BPEs (Byte Pair Encodings) (Sennrich et al., 2016) with the Transformer (Vaswani et al., 2017) in a new metric, cross-mutual information.
Disparity in language modeling results Disparate performances across different languages seem to have been implicitly accepted in that it is often believed that some languages are harder to model than others. Bender (2009) advocated the relevance of linguistic typology for the design of languageuniversal NLP systems due to differences based on crosslinguistic structural notions, such as parts of speech and morphological complexity. Cotterell et al. (2018) studied (monolingual) language models (LMs) on the 21 Europarl languages using a word-level 7-gram standard Kneser & Ney (1995) model and LSTM-LMs (Sundermeyer et al., 2012) with characters and lemmatized forms in informationtheoretic terms, and found morphological complexity to be the primary culprit for the differences in performance. Mielke et al. (2019) supplemented with the multilingual Bible corpus (Mayer & Cysouw, 2014) to extend the coverage to 69 languages, tested on RNN-LMs (an implementation of LSTM (Hochreiter & Schmidhuber, 1997)) with characters and BPEs, but concluded that basic data statistics in vocabulary size (|V |) and sequence length were the most predictive features of performance.
We noticed, however, a discrepancy in the results from Mielke et al. (2019) for ZH — it came out as the least difficult for the character model, but it is the 6th most difficult language for the BPE model. As different input representations have been tested with different architectures with divergent results in different metrics in previous studies, each of them only testing with one data size, we decided to investigate the matter more systematically once again with statistical comparisons of score distributions between languages.

heading: 1.2 RESEARCH QUESTIONS AND CONTRIBUTIONS
text: Research questions Are there any statistically significant differences in hardness when it comes to Conditional-Language-Modeling (CLMing) languages which have been traditionally considered morphologically rich (AR and RU) and poor (ZH) with the 6-layer Transformer? Is morphological complexity inherent in language? When is the notion of morphological complexity relevant in computing?
Summary of findings and insights Based on our bilingual CLMing setup with the UN Parallel Corpus in the data size range from 102 to 106 lines on the character, byte, and word levels, we find:
1. Language has more finer-grained dimensions with different representations and learning patterns. Hardness in modeling is relative to its representation granularity (representation relativity). 2. There is neutralization of source language instances, i.e. there are no statistically significant differences between source language pairs. Only pairs of target languages differ significantly. 3. On the character and byte levels, hardness is correlated with statistical properties concerning sequence length and |V | of a language, regardless of its morphological profile. As it is possible to eliminate performance disparity by decomposing sequences into finer-grained units in characters and bytes, we show that morphological complexity is not an intrinsic property of language. Unless word-based methods are used, or unless we implement/model it explicitly, the notion of morphological complexity is irrelevant in computing. 4. On the word level, hardness is correlated with |V |, and a complexity hierarchy arises through the manual preprocessing step of word tokenization. This complexity/disparity effected by word segmentation can be improved by subword tokenization but cannot be eliminated due to the fundamental qualitative differences in the definition of a “word” being one that neither holds universally nor is suitable/consistent for fair crosslinguistic comparisons. 5. Representational units of finer granularity can help close the gap in performance disparity.
Orthogonal to our main research questions, we also observed 2 types of sample-wise non-monotonicity — Double Descent (Belkin et al., 2019; Nakkiran et al., 2020) and erraticity. For reasons due to length and scope for this paper, we will defer discussions and analyses of these beyond what is addressed in § 5 to future work.
Outline of the paper In § 2, we define our method and experimental setup. We present our results and analysis on the primary representations in § 3 and those from secondary set of controls in § 4 in a progressive manner to ease understanding. Meta analysis on performance disparity and other discussions are in § 5.

heading: 2 METHOD AND DEFINITIONS
text: Conditional language modeling CLMing is the modeling of the probability of the next token, given the history of the preceding tokens and conditioning context. In our case, such conditioning context is a line from the source language. To explicitly focus on modeling the complexities that may or may not be intrinsic to the languages, we study the more fundamental process of CLMing without performing any translation. This allows us to eliminate confounds associated with generation and other evaluation metrics. One could think of our setup as estimating conditional probabilities with the Transformer, with a bilingual (one-to-one) setup where the perplexity (PP) of one target language (ltrg) is estimated given the parallel data in one source language (lsrc), where lsrc ̸= ltrg. We focus on the very basics and examine the first step in our pipeline — input representations, holding everything else constant. Instead of measuring absolute cross-entropy scores at one data size, we evaluate the relative differences between development (dev) set score distributions between languages.
Controlled experiments as basic research for scientific understanding of language data Using the UN Parallel Corpus, the data from which the MT results in Junczys-Dowmunt et al. (2016) stem, we perform a series of controlled experiments with the Transformer, holding the hyperparameter settings for all 30 one-to-one language directions from the 6 languages constant. We control for size (from 102 to 106 lines) and language with respect to representational granularity. We examine 3 primary representation types/levels — character, byte (UTF-8), and word, and upon encountering some unusual phenomena, we perform a secondary set of controls with 5 alternate representations — on the character level: Pinyin and Wubi (ASCII representations for ZH phones and character strokes, respectively), on the byte level: code page 1256 (for AR) and code page 1251 (for RU), and on the word level: BPE. These symbolic variants allow us to manipulate the statistical properties of the representations, while staying as “faithful” to the language as possible. We adopt this symbolic datacentric3 approach because we would like to more directly interpret the confounds, if any, that make language data different from other data types. We operate on a smaller data size range as this is more common in traditional language sciences and one of our higher goals is to bridge an understanding between language sciences and engineering (the latter being the dominant focus in NLP), and between traditional symbolic sciences and ML. We run statistical tests to identify the strongest correlates of performance and to assess whether the differences between the mean performance of different groups are indeed significant. We are concerned not with the absolute scores, but with the differences between score distributions from different languages.
Fair evaluation with multitexts Multitexts are multiway parallel corpora. The UN Parallel Corpus is a 6-way parallel corpus consisting of manually translated UN documents from the 25-year period between 1990 and 2014. We use the UN Parallel Corpus because it contains languages conventionally regarded as morphologically rich and poor, has quality and size sufficient for evaluation, and more importantly, it comes as raw texts (untokenized), unlike both of the corpora that Mielke et al. (2019) used. Detokenization (esp. the evaluation thereof) is not a trivial task.
Fair information-theoretic evaluation metric Most sequence-to-sequence models are optimized using a cross-entropy loss, defined as:
H(t, s) = − N∑ i=1 log2 p(ti | t<i, s) (1)
where t is the sequence of tokens to be predicted, ti refers to the ith token in that sequence, s is the sequence of tokens conditioned on, and N = |t|. It is customary to report scores as PP, which is 2 1 N H(t,s), i.e. 2 to the power of the cross-entropy averaged by the number of tokens in the dev
3Two testing/evaluation approaches — data-centric: hold the algorithm constant and tweak data, vs. algorithmcentric: hold data constant and tweak the algorithm.
data. Cotterell et al. (2018) proposed to use “renormalized” PP to evaluate LMs tokenwise fairly by dividing the overall bits per utterance/sequence by one constant token count in any one arbitrary language (e.g. so to arrive at “bits per character” in one language to evaluate all languages). But we find that it is not necessary to assign a perspective that is centered on any one particular language, when we can evaluate simply by the total number of bits for a larger portion of texts/sequences. This can be a fairer, more general and flexible way of evaluating data that has not been or cannot be perfectly segmented or aligned line by line. We hence used instead unnormalized PP, i.e. the total number of bits needed to encode the dev set (3,077 lines per language, after length filtering, in our case). As the implementation we used only reports PP, we transformed it back to entropy as defined above via H(t, s) = log2 PP (t|s)×N .
Disparity/Inequality In the context of our CLMing experiments, we consider there to be “disparity” or “inequality” between languages l1 and l2 if there are significant differences between the performance distributions of these two languages with respect to each representation. Here, by performance we mean the number of bits required to encode the held-out data using a trained CLM. With 30 directions, there are 15 pairs of source languages (lsrc1, lsrc2) and 15 pairs of target languages (ltrg1, ltrg2) possible. We compare the source languages among each other, and the target languages among each other. Each lsrc or each ltrg consists of scores from all models trained across various sizes and directions. To assess whether the differences are significant, we perform unpaired two-sided significance tests with the null hypothesis that the score distributions for the two languages are not different. Upon testing for normality with the Shapiro-Wilk test (Shapiro & Wilk, 1965; Royston, 1995), we use the parametric unpaired two-sample Welch’s t-test (Welch, 1947) (when normal) or the non-parametric unpaired Wilcoxon test (Wilcoxon, 1945) (when not normal) for the comparisons. We use the implementation in R (R Core Team, 2014) for these 3 tests. To account for the multiple comparisons we are performing, we correct all p-values using Bonferroni’s correction (Benjamini & Heller, 2008; Dror et al., 2017) and follow Holm’s procedure4 (Holm, 1979; Dror et al., 2017) to identify the pairs of l1 and l2 with significant differences after correction. We report all 3 levels of significance (α ≤ 0.05, 0.01, 0.001) for a more comprehensive overview. In contrast to Dror et al. (2017), which aimed to compare the performance of different algorithms, we compare languages (in the context of computing).
Experimental setup The systematic, identical treatment we give to our data is described as follows with further preprocessing and hyperparameter details in Appendices A and B, respectively.
After filtering length to 300 characters maximum per line in parallel for the 6 languages, we made 3 subsets of the data with 1 million lines each — one having lines in the order of the original corpus (dataset A) and two other randomly sampled (without replacement) from the full corpus (datasets B & C). Lines in all datasets are extracted in parallel and remain fully aligned for the 6 languages. For each run and each representation, there are 30 pairwise directions (i.e. one lsrc to one ltrg) that result from the 6 languages. We trained all 150 (for 5 sizes) 6-layer Transformer models for each run using the SOCKEYE Toolkit (Hieber et al., 2018). We optimize using PP and use early stopping if no PP improvement occurs after 3 checkpoints up to 50 epochs maximum, taking the best checkpoint. Characters and bytes are supposed to mitigate the out-of-vocabulary (OOV) problem on the word level. In order to assess the effect of modeling with finer granularity more precisely, all vocabulary items appearing once in the train set are accounted for (i.e. full vocabulary on train, as in Gerz et al. (2018a;b)). But we allow our system to categorize all unknown items in the dev set to be unknown (UNK) so to measure OOVs (open vocabulary on dev (Jurafsky & Martin, 2009)). To identify correlates of performance, we perform Spearman’s correlation (Spearman, 1904) with some basic statistical properties of the data (e.g. length, |V |, type-token-ratio, OOV rate) as metrics — a complete list is provided in App. C. See App. D for sample construction for statistical comparisons.

heading: 3 EXPERIMENTAL RESULTS OF PRIMARY REPRESENTATIONS
text: Subfigures 1a, 1b, and 1c present the mean results across 12 runs of the 3 primary representations — character, byte, and word, respectively. The x-axis represents data size in number of lines and y-axis the total conditional cross-entropy, measured in bits (Eq. 1). Each line connects 5 data points corresponding to the number of bits the CLMs (trained with training data of 102, 103, 104, 105, and
4using implementation from https://github.com/rtmdrr/replicability-analysis-NLP
106 lines) needed to encode the target language dev set given the corresponding text in the source language. These are the same data in the same 30 language directions and 5 sizes with the same training regime, just preprocessed/segmented differently. This confirms representation relativity — hardness in modeling is relative to its representation granularity. Languages (or any objects being modeled) need to be evaluated relative to their representation. “One size does not fit all” (Durrani et al., 2019). Our conventional way of referring to “language” (as a socio-cultural product or with traditional word-based approaches, or even for most multilingual tasks and competitions) is too coarse-grained for computing (see also Fisch et al. (2019) and Ponti et al. (2020)).
Subfigures 1d, 1e, and 1f display the corresponding information sorted into facets by target language, source languages represented as line types. Through these we see more clearly that results can be grouped rather neatly by target language — as implicit in the Transformer’s architecture, the decoder is unaware of the source language in the encoder. As shown in Table 1 in § 5 summarizing the number of source and target language pairs with significant differences, there are no significant differences across any source language pairs. The Transformer neutralizes source language instances. This could explain why transfer learning or multilingual/zero-shot translation (Johnson et al., 2017) is possible at all on a conceptual level.
In general, for character and byte models, most language directions do seem to converge at 104 lines to similar values across all target languages, with few notable exceptions. There are some fluctuations past 104, indicating further tuning of hyperparameters would be beneficial due to our present setting possibly working most favorably at 104. On the character level, target language ZH (ZHtrg) shows a different learning pattern throughout. And on the byte level, ARtrg and RUtrg display highly unstable behavior, which we refer to as erratic. Word models exhibit Double Descent across the board (note the spike at 103), but overall, difficult/easy languages stay consistent, with AR and RU being the hardest, followed by ES and FR, then EN and ZH. A practical takeaway from this set of experiments: in order to obtain more robust training results, use bytes for ZH (as suggested in Li et al. (2019a)) and characters for AR and RU (e.g. Lee et al. (2017)) — also if one wanted to avoid any “class” problems in performance disparity with words. Performance disparity for these representations is reported in Table 1 under “CHAR”, “BYTE”, and “WORD”. Do note, however, that the intrinsic performance of ZH with word segmentation is not particularly subpar. But this often does not correlate with its poorer downstream tasks results (recall results from Junczys-Dowmunt et al. (2016)). Since the notion of word in ZH is highly contested and ambiguous — i) it is often aimed to align with that in other languages so to accommodate academic theories and manual feature engineering5, ii) there is great variation among different conventions, and iii) native ZH speakers identify characters as words — there are reasons to rethink this procedure now that fairer and language-independent processing in finer granularity is possible. Li et al. (2019b) questioned the necessity of CWS in Deep Learning (DL)-based ZH NLP and presented evidence in favor of character-based processing, including results from downstream NLP tasks. In Linguistics, Duanmu (2017) presented a summary on the contested nature of wordhood in (Mandarin) ZH in relation to EN. A more native account of ZH, however, despite a couple of dialects/varieties of it being considered a high-resource language, has not yet been fully recognized and accepted in NLP.

heading: 4 UNDERSTANDING THE PHENOMENA WITH ALTERNATE REPRESENTATIONS
text: To understand why some languages show different results than others, we carried out a secondary set of controlled experiments with representations targeting the problematic statistical properties of the corresponding target languages.
5It is a “legacy interpretation” stemmed from a practical compromise from the early days in ZH NLP when the goal was to align with EN words for MT. Chinese word segmentation (CWS) has been a decades-long issue in text processing. But even in EN, for computing, the variability in “word” counts (from the trivial convention of whitespace tokenization) results in different bit counts, affecting file sizes. In NLP, such method of “word” counting brings about a high |V |, hence different tokenization schemes have been designed to mitigate this problem. For humans, there is no consensus about the definition of “words”. Even for an academic account external of CWS, it is held to be indeterminate (see Haspelmath (2011) and references therein from the past century). Kilgarriff (1997; 2014) pointed out that “words” and “word senses” and the number thereof, in terms of lexical entries for dictionaries, are contextual and arbitrary.
500000
1000000
1500000
2000000
2500000
1e+03 1e+05 number of lines
nu m
be r
of b
its
TRG
AR EN ES FR RU ZH SRC
AR EN ES FR RU ZH
(a) Code page 1256 & 1251
AR EN ES
FR RU ZH
1000000
1500000
2000000
1000000
1500000
2000000
1e+03 1e+05 1e+03 1e+05 1e+03 1e+05 number of lines
nu m
be r
of b
its
TRG
AR EN ES FR RU ZH SRC
AR EN ES FR RU ZH
(b) Code page by target
500000
1000000
1500000
2000000
2500000
1e+03 1e+05 number of lines
nu m
be r
of b
its
TRG
AR EN ES FR RU ZH SRC
AR EN ES FR RU ZH
(c) BPE
AR EN ES
FR RU ZH
900000
1200000
1500000
900000
1200000
1500000
1e+03 1e+05 1e+03 1e+05 1e+03 1e+05 number of lines
nu m
be r
of b
its
TRG
AR EN ES FR RU ZH SRC
AR EN ES FR RU ZH
(d) BPE by target
Figure 3: Byte-level (Subfigures 3a & 3b) remedies with code page 1256 for target AR and 1251 for target RU, and word-level (Subfigures 3c & 3d) remedy with BPE for all languages.
Character level We reduced the high |V | in ZH with representations in ASCII characters — Pinyin and Wubi. We replaced the ZH data in these formats only on the target side and reran the experiments involving ZHtrg on the character level. Results in Figure 2 and Table 1 show that the elimination of disparity on character level is possible if ZH is represented through Pinyin (transliteration), as in Subfigure 2c, though at the cost of the native script information. Models with ZH logographic scripts display a behaviorial tendency unlike those with other (phonetic) alphabetic scripts6 (Subfigure 2a), suggesting that this script pattern decomposes differently.
Byte level Length is the most salient statistical attribute that makes AR and RU outliers. To shorten their sequence lengths, we tested with alternate encodings on ARtrg and RUtrg — code page 1256 and 1251, which provide 1-byte encodings specific to AR and RU, respectively. Results are shown in Subfigures 3a and 3b. Not only is erraticity resolved, the number of 15 possible target language pairs with significant differences reduces from 8 with the UTF-8 byte representation to 0 (Table 1 under “ARRUt”), indicating that we eliminated disparity with this optimization heuristic. Since our heuristic is a lossless and reversible transform, it shows that a complexity that is intrinsic and necessary in language7 does not exist in computing, however diverse they may be, as our 6 are, from the conventional linguistic typological, phylogenetic, historical, or geographical perspectives.
Word level The main difference between word and character/byte models is length not being a top contributing factor correlating with performance, but instead |V | is. This is understandable as word segmentation neutralizes sequence lengths. To remedy the OOV problem, we use BPE, which learns a fixed vocabulary of variable-length character sequences (on word level, as it presupposes word segmentation) from the training data. It is more fine-grained than word segmentation and is known to better model subword units for morphologically complex languages. We use the same vocabulary of 30,000 as specified in Junczys-Dowmunt et al. (2016). This reduced our averaged OOV token rate by 89-100% across the 5 sizes. The number of language pairs with significant differences reduced to 7 from 8 for word models. While BPEs are still not as effective as our character/byte variants, their results show how more fine-grained modeling contributes positively to closing the disparity gap.

heading: 5 META RESULTS, ANALYSES, AND DISCUSSION
text: Performance disparity Table 1 lists the number of language pairs with significant differences under the representations studied. Since it is possible for our character and byte models to effect no performance disparity for the same languages on the same data, a complexity intrinsic to language does not exist. In fact, the customary expectation that languages ought to perform differently is created through our word segmentation practice. Furthermore, the order of AR/RU > ES/FR > EN/ZH
6which are sequences with an implicit/explicit pattern made up of consonants and vowels 7aside from its statistical properties related to length and vocabulary. To show something is not necessarily
true, only 1 counter observation is needed.
(Figure 1c) resembles the idea of morphological complexity. Considering there are character-internal meaningful units in languages with logographic script such as ZH (cf. Zhang & Komachi (2018)) that are rarely captured, studied, or referred to as “morphemes”, this goes to show that linguistic morphology, along with its complexity, as it is practiced today8 and that which has occurred in the NLP discourse thus far, has only been relevant on the “word” level, conceptually constrained by unstandardizable units such as “words” (and “sentences”). The definition of word, however, has been recognized as problematic for a very long time in the language sciences (cf. Footnote 5).
While the lack of significant differences between pairs of source languages would signify neutralization of source language instances, it does not mean that source languages have no effect on target. For our byte solutions with code pages, we experimented also with source side optimization in the directions that involve AR/RU as source. This affected the distribution of the disparity results for that representation — with 2 pairs being significantly different (see Table 1 under “ARRUs,t”). We defer further investigation on the nature of source language neutralization to future work.
Target language pairs with significant differences are summarized in Table 2. We show that morphological complexity can be empirically eliminated in this one-setting-for-all configuration with a 6-layer network, no hyperparameter tuning, and a maximum line length of 300 characters (and its corresponding equivalence in other representations) as constrained by our hardware and compute time listed in App. A and current data availability. A more analytical solution can be obtained through data statistics (see App. E). A conceptual solution lies in the definition of “words” and morphology.
Sample-wise Double Descent (DD) Sample-wise non-monotonicity/DD (Nakkiran et al., 2020) denotes a degradation followed by an improvement in performance with increasing data size. We
8But there are no reasons why we cannot adopt a statistical science of language in finer granularities beyond/without “words”, with standardized units (characters/bytes) and/or continuous representations. Resources, e.g. quality parallel data or contrast sets, can serve both data science and ML interpretation and evaluation well.
notice word models and character models with ZHtrg, i.e. models with high target |V |, are prone to exhibit a spike at 103. A common pattern for these is the ratio of target training token count to number of parameters falls into O(10−4) for 102 lines, O(10−3) at 103, O(10−2) at 104, and O(10−1) for 105 lines and so on. But for more atomic units such as alphabetic (not logographic) characters (may it be Latin, Cyrillic, or Abjad) and for bytes, this progression instead begins at O(10−3) at 102 lines. Instead of thinking this spike of 103 as irregular, we may instead want to think of this learning curve as shifted by 1 order of magnitude to the right for characters and bytes and/or the performance at 102 lines for words and ZH-characters due to being overparameterized and hence abnormal. This would also fit in with the findings by Belkin et al. (2019) and Nakkiran et al. (2020) attributing DD to overparameterization. While almost all work attribute DD to algorithmic reasons, findings from Chen et al. (2020) corroborate our observation and confirms that DD arises due to “the interaction between the properties of the data and the inductive biases of learning algorithms”. Other related work on the DD phenomenon and its development can also be found in their work.
Erraticity We observe another type of sample-wise non-monotonicity, one that signals irregular and unstable performance across data sizes and runs. Within one run, erraticity can be observed directly as changes in direction on the y-axis. Across runs, large variance can be observed, even with the same dataset. Erraticity can also be observed indirectly through a negative correlation between data size and performance. Many work on length bias in NMT have focused on solutions related to search, e.g. Murray & Chiang (2018). Our experiments show that a kind of length bias can surface already with CLMing, without generation taking place.
Additional related work That basic data statistics are the driver of success in performance in multilingual modeling has so far only been explicitly argued for in Mielke et al. (2019). We go beyond their work in monolingual LMs to study CLMs and evaluate also in relation to data size, representational granularity, and quantitative and qualitative fairness. To the best of our knowledge, there has been no prior work on demonstrating the neutralization of source language instances through statistical comparisons, a numerical analysis on DD for sequence-to-sequence models, the meta phenomenon of a sample-wise non-monotonicity (erraticity) being related to length.

heading: 6 CONCLUSION
text: Summary We investigate whether the performance disparity between languages which have been traditionally considered morphologically rich (AR and RU) and poor (ZH) in the 6-layer Transformer CLM is justified due to morphological complexity and find that it is not. Performance disparity can be explained by data statistics and in the context of computing, it can be eliminated by optimization on length and |V | through character/byte representations. In fact, morphological complexity is not a necessary concept in computing because “word” is not a necessary concept in computing, unless we make it so through word segmentation. A morphological complexity hierarchy can result simply through word segmentation. Furthermore, there are many possible interpretations to “words” for humans and since morphology is defined with the concept of “word”, there is no stable ground for assessing this complexity. Representational units of finer granularity were shown to help eliminate performance disparity though at the cost of longer sequence length, which can have a negative impact on robustness. In addition, we found all word models and character models with ZHtrg to behave similarly in their being prone to exhibit a peak (as sample-wise DD) around 103 lines in our setting.
Outlook ML has enabled greater diversity in NLP (Joshi et al., 2020). Fairness, in the elimination of disparity, does not require big data. This paper made a pioneering attempt to bridge research in NNs/DL, language sciences, and language engineering through a data-centric perspective. Multilinguality is real and relevant in computing not due to canonical, structural linguistic concepts such as morphology or “words” in our minds, but rather standards related to internationalization and localization, such as character encoding — something which has thus far been sorely overlooked in our discourse and curricula. We also believe that a more fine-grained statistical data science can well complement algorithmic analyses with a view that is more empirically robust (i.e. experimentally verifiable) and more relevant to machine processing, contributing to a more generalizable and interpretable pool of knowledge for ML/NNs/DL. A more comprehensive study can lead us not only to new scientific frontiers, but also to better designs and evaluation, benefitting the development of a more general, diverse and inclusive AI.
Ethics Statement: Fairness concerns
CLEARER NOMENCLATURE If/When the intent is not to explicitly model linguistic morphology in computing, one can simply describe languages and their statistical profiles with respect to their representational granularity in characters or bytes (which are and can be exhaustively standardized in computing), or refer to sequences as longer/shorter or having a higher/lower vocabulary size when comparing them with each other, rather than “richer”/“poorer” based on concepts (e.g. “words”, “sentences”) that can be ambiguous, contested, and inaccessible to many.
ACCESSIBILITY Language communities who are unfamiliar with languages similar to dominant languages or those who are reluctant to conform to one structurally similar interpretation should not have to feel inadequate in processing their own language “from scratch”, if they so choose to. As technologists, we can help take equitable measures to make fairer data representations and infrastructure available. A “word”-free view of language preempts linguistic/cultural hegemony and such an interpretation would also help make the analyses of language data more objective and clearer.
SCARCITY OF QUALITY AND/OR MULTIWAY DATA FOR SCIENCE, EVALUATION, AND CULTURE With the rise of multilingual models comes an alleged decrease in reliance on parallel corpora for MT. But parallel data, esp. high-quality, multiway and/or multi-modal parallel data, are important for scientific research and as a form of cultural and language documentation. And as this paper shows, they can also serve as evaluation data for ML models for better understanding and interpretation. As challenge sets, data for machine processing would need to be statistically diverse and challenging as well. Parallel data from previous years have often come in the form of bitexts (2-way parallel text data), usually “word”-tokenized where real length information has been compromised. (In fact, the Bible data from Mayer & Cysouw (2014) came with another confound — the ZH numeral ‘一’ (“one”) is recognized as a dash (punctuation) and hence tokenized with surrounding whitespaces.) At the time of our present study (2019-present), the UN Parallel Corpus was the only unperturbed, fully multiway parallel data sufficient for reliable evaluation for our size range. Data for science, evaluation, and culture require a long-term, stable platform and support. There are many forms of data science. But in terms of having a sustainable practice to collect and curate good data, and exercising enough of a science with that data so to improve collective intelligence and mutual understanding, instead of looking at data from an utilitarian point of view only for consumer application purposes, there seems to be room for improvement still. We hope our work could help effect a positive change in this direction.

heading: A DATA SELECTION AND PREPROCESSING DETAILS 16

heading: B HYPERPARAMETER SETTING 17

heading: C CORRELATION STATISTICS 18

heading: D STATISTICAL COMPARISONS 19

heading: E DATA STATISTICS 20
text: F ENLARGED FIGURES FOR ALL 30 LANGUAGE DIRECTIONS (AGGREGATE RESULTS FROM ALL RUNS) 24

heading: A DATA SELECTION AND PREPROCESSING DETAILS
text: The UN Parallel Corpus v1.0 (Ziemski et al., 2016) consists of manually translated UN documents from 1990 to 2014 in the 6 official UN languages. Therein is a subcorpus that is fully aligned by line, comprising the 6-way parallel corpus we use. We tried to have as little preprocessing or filtering as necessary to eliminate possible confounds. But as the initial runs of our experiment failed due to insufficient memory on a single GPU with 12 GB VRAM9, we filtered out lines with more than 300 characters in any language in lockstep with one another for all the 6 languages such that the subcorpora would remain parallel, thereby keeping the material of each language semantically equivalent to one another. 8,944,859 lines for each language were retained as our training data which cover up to the 75th percentile in line length for all 6 languages. In order to monitor the effect of data size, we made subcorpora of each language in 5 sizes by heading the first 102, 103, 104, 105, 106 lines10. We refer to this as dataset A. In addition, to better understand and verify the consistency of the phenomena observed, we made 2 supplemental datasets by shuffling the 8,944,859 lines two different times randomly and heading the number of lines in our 5 sizes for each language, again in lockstep with one another (datasets B and C).
The systematic training regime that we gave to our language directions is identical for all and we controlled also for seeds. For each of the 3 primary representations — character, byte, and word, we performed:
• 5 runs in 5 sizes (102 − 106 lines): A0 (seed=13), B0 (13), C0 (9948), A1 (9948), A2 (265), and • 7 more runs in 4 sizes (102 − 105 lines): A3 (777), A4 (42), A5 (340589), A6 (1000), A7
(83146), B1 (9948), & C1 (13).
Figure 1 shows results from all 12 runs in all sizes for the primary representations.
For the alternate/secondary representations, we performed 3 runs each in 5 sizes (102-106 lines) (A0, B0, & C0). Due to limitations in computing resources, we were not able to perform as many runs as the primary representations. But important for our statistical comparisons is that we evaluate based on an equal number of runs and on the same data for all candidates. Tables 1 & 2 are the results.
For each run and each size, there are 30 pairwise directions (i.e. 1 source language to 1 target language, e.g. AR-EN for Arabic to English) that result from the 6 languages. We trained all 150 jobs (30 directions x 5 sizes) for each run and representation using the Transformer model (Vaswani et al., 2017) as supported by the SOCKEYE Toolkit (Hieber et al., 2018) (version 1.18.85), based on MXNet (Chen et al., 2015). A detailed description of the architecture of the Transformer can be found in (Vaswani et al., 2017). The same set of hyperparameters applies to all and its values are listed in Appendix B.
For character modeling, we used a dummy symbol to denote each whitespace. For byte, we turned each UTF-8-encoded character into a byte string in decimal value, such that each token is a number between 0 and 255, inclusive. For word, we followed (Junczys-Dowmunt et al., 2016) and used the Moses tokenizer (Koehn et al., 2007) as is standard in NMT practice when word tokenization is applied and Jieba11 for segmentation in ZH.
Pinyin is a romanization of ZH characters based on their pronunciations and Wubi an input algorithm that decomposes character-internal information into stroke shape and ordering and matches these to 5 classes of radicals (Lunde, 2008). For Pinyin, we used the implementation from https: //github.com/lxyu/pinyin in the numerical format such that each character/syllable is
9GPUs used for experiments in this paper range from a NVIDIA TITAN RTX (24 GB), NVIDIA GeForce RTX 2080 Ti (11 GB), a GTX Titan X (12 GB), to a GTX 1080 (8 GB). All jobs were run on a single GPU setting. Some word-level experiments involving ARtrg or RUtrg at 106 had to be run on a CPU as 24 GB VRAM were not sufficient. Models with higher maximum sequence lengths (e.g. byte models) were trained with 24 GB VRAM. Difference in equipment does not necessarily lead to degradation/improvement in scores.
10The terms “line” and “sentence” have been used interchangeably in the NLP literature. We use “line” to denote a sequence that ends with a newline character and “sentence” as one with an ending punctuation. Most parallel corpora, such as ours, are aligned by line, as a line may be part of a sentence or without an ending punctuation (e.g. a header/title). Using a standardized unit such as “line” would also be a fairer measure to linguae/scriptiones continuae (languages/scripts with no explicit punctuation).
11https://github.com/fxsjy/jieba
followed by a single digit indicating its lexical tone in Mandarin. For Wubi, we used the dictionary from the implementation from https://github.com/arcsecw/wubi.
We have implemented all representations such that they would be reversible even when the sequence contains code-mixing.
We used the official dev set as provided in (Ziemski et al., 2016), 3,077 lines per language remained from 4,000 after filtering line length to 300 characters. Data statistics is provided in Appendix E for reference.
Notes on training time Each run of 30 directions in 5 sizes took approximately 8-12 days for character and byte models. Byte models generally took longer — hence training time is positively correlated with length (concurring with observations by Cherry et al. (2018) as they compared character with BPE models). A maximum length of 300 characters entails a maximum length of at least 300 bytes in UTF-8. Each run of word models (30 directions, 5 sizes) took about 6 days (excluding the training of some 7-9 directions out of 30 per run involving ARtrg or RUtrg at 106 on word level which took about 12-18 hours each direction to train on a CPU as these required more space and would run out of memory (OOM) on our GPUs otherwise). These figures do not include the additional probing experiments described in § 4.
B HYPERPARAMETER SETTING
• encoder transformer; • decoder transformer; • num-layers 6:6; • num-embed 512:512; • transformer-model-size 512; • transformer-attention-heads 8; • transformer-feed-forward-num-hidden 2048; • transformer-activation-type relu; • transformer-positional-embedding-type fixed; • transformer-preprocess d; transformer-postprocess drn; • transformer-dropout-attention 0.1; • transformer-dropout-act 0.1; • transformer-dropout-prepost 0.1; • batch-size 15; • batch-type sentence; • max-num-checkpoint-not-improved 3; • max-num-epochs 50; • optimizer adam; • optimized-metric perplexity; • optimizer-params epsilon: 0.000000001, beta1: 0.9, beta2: 0.98; • label-smoothing 0.0; • learning-rate-reduce-num-not-improved 4; • learning-rate-reduce-factor 0.001; • loss-normalization-type valid; • max-seq-len 300 for character, word, and BPE, 672 for all bytes, 688 for Wubi, 680 for Pinyin; • checkpoint-frequency/interval 4000.
(For smaller datasets, the end of 50 epochs is often reached before the first checkpoint. Since SOCKEYE only outputs scores at checkpoints, we adjusted the checkpoint frequency as follows to get a score outputted by the end of 50 epochs: 1000 for 100 lines for all character & byte instances, 400 for 100 lines for word and 500 for 100 lines BPE, 3450 for 1000 lines for word & BPE. For the very few cases that this default does not suffice due to bucketing of similar length sequences, we manually set the checkpoint frequency to the last batch.)

heading: C CORRELATION STATISTICS
text: Best correlating metrics, i.e. the union of top 3 metrics for all representations. For each representation, the top 3 metrics are boldfaced. All correlations are highly significant (p < 10−30), except for min source length for WORD (p ≈ 0.0001) and min target length for WORD (p ≈ 0.3861).
Metric CHAR Pinyin Wubi BYTE ARRUt ARRUs,t WORD BPE
minimum length (target) 0.84 0.85 0.86 0.60 0.84 0.84 −0.02 0.65 minimum length (source) 0.82 0.84 0.85 0.57 0.84 0.84 0.10 0.64
number of tokens (source) −0.78 −0.81 −0.82 −0.60 −0.81 −0.81 −0.59 −0.83 TTR (target) 0.83 0.83 0.84 0.48 0.81 0.81 0.61 0.83
|V | (source) −0.54 −0.51 −0.51 −0.50 −0.67 −0.68 −0.63 −0.86 data size in lines −0.80 −0.83 −0.83 −0.59 −0.81 −0.81 −0.62 −0.86 OOV token rate (target) 0.69 0.66 0.66 0.47 0.67 0.68 0.66 0.62
OOV type rate (target) 0.70 0.71 0.72 0.47 0.69 0.70 0.65 0.62
TTR (source) 0.67 0.71 0.71 0.60 0.81 0.81 0.56 0.82
The full list of metrics used for the correlation analysis is:
1. minimum length (source), 2. minimum length (target), 3. maximum length (source), 4. maximum length (target), 5. median length (source), 6. median length (target), 7. mean length (source), 8. mean length (target), 9. length std (source),
10. length std (target), 11. data size in lines, 12. number of parameters, 13. number of types (|V |) (source), 14. number of types (|V |) (target), 15. number of tokens (source), 16. number of tokens (target), 17. type-token-ratio (TTR) (source), 18. type-token-ratio (TTR) (target), 19. OOV type rate (source), 20. OOV type rate (target), 21. OOV token rate (source), 22. OOV token rate (target), 23. token ratio, 24. target type-to-parameter ratio, 25. target token-to-parameter ratio, 26. distance between the TTRs of source and target = (1 - TTRsrc/TTRtrg )2, 27. token-to-parameter ratio (i) = (median length source * median length target * num_lines) / num_parameters, 28. token-to-parameter ratio (ii) = (num_source_tokens * num_target_tokens) / num_parameters.

heading: D STATISTICAL COMPARISONS
text: Recall the definition and method for our Disparity/Inequality assessment from § 2:
In the context of our CLMing experiments, we consider there to be “disparity” or “inequality” between languages l1 and l2 if there are significant differences between the performance distributions of these two languages with respect to each representation. Here, by performance we mean the number of bits required to encode the held-out data using a trained CLM. With 30 directions, there are 15 pairs of source languages (lsrc1, lsrc2) and 15 pairs of target languages (ltrg1, ltrg2) possible. We compare the source languages among each other, and the target languages among each other. Each lsrc or each ltrg consists of scores from all models trained across various sizes and directions. To assess whether the differences are significant, we perform unpaired two-sided significance tests with the null hypothesis that the score distributions for the two languages are not different. Upon testing for normality with the Shapiro-Wilk test (Shapiro & Wilk, 1965; Royston, 1995), we use the parametric unpaired two-sample Welch’s t-test (Welch, 1947) (when normal) or the non-parametric unpaired Wilcoxon test (Wilcoxon, 1945) (when not normal) for the comparisons. We use the implementation in R (R Core Team, 2014) for these 3 tests. To account for the multiple comparisons we are performing, we correct all p-values using Bonferroni’s correction (Benjamini & Heller, 2008; Dror et al., 2017) and follow Holm’s procedure12 (Holm, 1979; Dror et al., 2017) to identify the pairs of l1 and l2 with significant differences after correction. We report all 3 levels of significance (α ≤ 0.05, 0.01, 0.001) for a more comprehensive overview. In contrast to Dror et al. (2017), which aimed to compare the performance of different algorithms, we compare languages (in the context of computing).

heading: To get samples for the statistical comparison results for the Disparity Table (Table 1):
text: For each representation, we used 3 runs (A0, B0, C0) in 5 sizes (102-106 lines) for each lsrc and each ltrg. There are:
6 lsrc (ARsrc, ENsrc, ESsrc, FRsrc, RUsrc, ZHsrc) and 6 ltrg (ARtrg, ENtrg, EStrg, FRtrg, RUtrg, ZHtrg).
We compare pairwise among the lsrc. The 15 pairs are:
ARsrc-ENsrc, ARsrc-ESsrc, ARsrc-FRsrc, ARsrc-RUsrc, ARsrc-ZHsrc, ENsrc-ESsrc, ENsrc-FRsrc, ENsrc-RUsrc, ENsrc-ZHsrc, ESsrc-FRsrc, ESsrc-RUsrc, ESsrc-ZHsrc, FRsrc-RUsrc, FRsrc-ZHsrc, RUsrc-ZHsrc.
Likewise 15 pairs among the ltrg.
For example, for the character (primary) representation to compare between ARsrc and ENsrc, we construct the sample for ARsrc (sampleARsrc) and the sample for ENsrc (sampleENsrc) as follows:
Out of the 30 CHAR directions, there are 5 directions involving ARsrc trained for each run and data size (i.e. the directions: AR-EN, AR-ES, AR-FR, AR-RU, AR-ZH).
For each direction, there are 15 models trained (3 runs x 5 sizes). We take all 75 CHAR models (15 models x 5 directions) involving ARsrc as sampleARsrc. That’s a sample of size 75.
Likewise, for ENsrc (5 directions: EN-AR, EN-ES, EN-FR, EN-RU, EN-ZH), we also have 75 data points for sampleENsrc. (Likewise also for all 6 lsrc and all 6 ltrg.)
For the comparisons, we compare pairwise, i.e. with two samples each time, but with unpaired two-sample Welch’s t-test (when normal) or the non-parametric unpaired Wilcoxon test (when not normal) because sampleARsrc and sampleENsrc have one direction that is not paired: AR-EN and EN-AR. Other directions can be seen as paired, e.g. AR-ES and EN-ES as both having the same ltrg.
12using implementation from https://github.com/rtmdrr/replicability-analysis-NLP
E D
A TA
S TA
T IS
T IC S • N um be ro ft yp es ,i .e .v oc ab
ul ar
y si
ze (| V |)
.N ot
e th
at So
ck ey
e ad
ds fo
ri ts
ca lc
ul at
io n 4
ad di
tio na
lt yp
es :<
pa d>
,< s>
,< /s
>, <u
nk >.
• N
um be
ro ft
ok en
s. T
hi s
ex cl
ud es
th e 1
E O
S/ B
O S
(e nd
-/ be
gi nn
in g-
of -s
en te
nc e)
m ar
ke ra
dd ed
by So
ck ey
e to
ea ch
lin e.
• O
ut -o
fvo
ca bu
la ry
(O O
V )t
yp e
ra te
(i n
% ),
i.e .t
he fr
ac tio
n of
th e
ty pe
s in
th e
de v
da ta
th at
is no
tc ov
er ed
by th
e ty
pe s
in th
e tr
ai ni
ng da
ta .
• O
O V
to ke
n ra
te (i
n %
), i.e
.t he
fr ac
tio n
of to
ke ns
in th
e de
v da
ta th
at is
tr ea
te d
as U
N K
no w
ns .
• Ty
pe -t
ok en
-r at
io (i
n %
), i.e
.t he
ra tio
be tw
ee n
th e
nu m
be ro
ft yp
es an
d to
ke ns
in th
e da
ta .T
hi s
is a
ro ug
h pr
ox y
fo rl
ex ic
al di
ve rs
ity in
th at
a va
lu e
of 1
w ou
ld in
di ca
te th
at no
ty pe
is ev
er se
en tw
ic e,
an d
a va
lu e
ve ry
cl os
e to
0 w
ou ld
in di
ca te
th at
ve ry
fe w
di st
in ct
ty pe
s ac
co un
tf or
al m
os ta
ll of
th e
da ta . • L in e le ng th (e xc l. E O S/ B O S m ar ke r) :m ea n±
st an
da rd
de vi
at io
n, an
d th
e 0/
25 /5
0/ 75
/1 00
-t h
pe rc
en til
e. S ta
tis tic
s fo
rd at
as et A R ep re se nt at io n C H A R B Y T
E W
O R
D B PE N um be ro fl in es 10 0 1, 00 0 1 0, 00 0 1 00 ,0 0 0 1 ,0 00 ,0 0 0 10 0 1 ,0 00 1 0, 00 0 1 00 ,0 00 1 ,0 00 ,0 00 1 00 1 ,0 00 10 ,0 00 1 0 0, 00 0 1 ,0 0 0, 0 00 1 00 1 ,0 0 0 10 ,0 00
10 0, 00 0
1 ,0 00 ,0 00
N um
be r
of T
Y PE S A R 82
10 7
1 4 9
1 76
2 39
73 95
1 30
1 45
1 62
8 66
5 ,8 65
26 ,1 05
9 7, 99 4
30 4, 9 75
7 0 0
4, 17 1
18 ,2 02
29 ,7 06
29 ,9 61
E N
75 8 1
1 22
16 7
24 4
76 82
12 7
15 1
16 6
67 9
3 ,2 89
1 2, 0 1 1
37 ,2 61
1 22 ,4 5 4
64 5
2, 85 6
10 ,2 25
26 ,1 84
29 ,0 68
E S
84 9 0
1 14
17 3
25 4
85 91
11 8
15 1
16 6
70 7
3 ,8 36
1 5, 1 1 3
49 ,4 96
1 42 ,8 0 6
66 1
3, 23 2
12 ,0 83
27 ,9 08
29 ,4 45
FR 86
9 4
1 30
17 5
25 6
87 96
13 0
15 3
17 2
7 42
3 ,8 78
1 3, 9 9 5
42 ,8 67
1 18 ,2 0 1
72 3
3, 19 3
11 ,7 31
27 ,3 93
29 ,3 33
R U
9 5
1 0 8
1 60
18 7
26 0
97 11 0
14 8
15 7
16 6
8 83
5 ,4 30
2 3, 40 0
8 1, 3 39
2 3 6, 08 5
8 42
4, 3 1 9
17 ,9 9 5
29 ,2 39
29 ,8 07
Z H
6 02
1, 2 36
2, 2 67
3, 4 00
4, 8 85
13 7
14 3
15 3
16 4
19 3
62 7
3, 17 8
1 3, 1 3 4
46 ,9 38
1 34 ,4 8 9
80 0
2, 8 49
1 0, 65 6
27 ,2 84
29 ,1 48
Z H
_p in
yi n
79 90
1 0 4
1 5 4
5 64
Z H
_w ub
i 10 0
10 6
1 2 3
1 70
5 42
A R
_c p1
25 6
8 2
10 7
14 9
17 6
23 9
R U
_c p1
25 1
9 5
10 8
16 0
18 7
26 0
N um
be r
of TO
K E
N S
A R
9 ,0 7 9
1 23 ,8 32
1 ,0 83 ,5 1 7
1 0 ,6 2 5 ,0 47
1 0 2 ,0 6 4 ,2 30
16 ,6 55
22 7, 1 63
1 ,9 85 ,0 1 4
19 ,4 8 7, 68 9
18 6 ,1 7 1, 18 0
1 ,7 76
2 3, 4 60
2 06 ,5 49
2 ,0 35 ,1 9 0
19 ,4 1 0, 5 02
2 ,9 07
2 8, 22 7
2 23 ,5 10
2 ,2 18 ,0 44
21 ,3 44 ,9 00
E N
1 1 ,7 30
15 9, 44 4
1 ,3 44 ,0 0 1
1 3 ,1 3 2 ,8 62
1 2 3 ,4 9 1 ,8 71
11 ,7 31
15 9, 4 49
1 ,3 45 ,7 7 1
13 ,1 5 8, 94 8
12 3 ,7 0 5, 12 8
2 ,0 71
2 7, 39 8
23 6, 5 6 9
2 ,3 3 9, 1 09
2 1 ,9 4 3 ,1 3 9
2 ,9 8 0
29 ,7 78
24 3, 62 8
2 ,3 69 ,7 23
22 ,3 51 ,0 59
E S
1 2 ,3 7 4
1 71 ,1 04
1 ,4 84 ,8 0 4
14 ,5 49 ,7 0 3
13 8 ,5 96 ,0 3 6
12 ,6 29
17 5, 28 6
1 ,5 13 ,7 82
1 4 ,8 21 ,4 95
14 1 ,2 76 ,7 66
2 ,2 3 2
29 ,4 61
2 63 ,0 2 4
2 ,5 88 ,7 9 1
2 4 ,6 54 ,4 4 9
3 ,1 80
3 2, 3 2 3
2 71 ,9 9 5
2 ,6 44 ,0 55
25 ,2 64 ,4 39
FR 1 2 ,4 56
1 7 9, 04 8
1 ,4 9 0 ,9 8 3
1 4 ,5 2 8 ,5 93
1 3 8 ,0 4 9, 1 89
12 ,8 75
1 85 ,2 27
1 ,5 42 ,1 05
15 ,0 55 ,6 57
1 43
,4 95 ,6 67
2 ,2 98
3 2, 0 11
2 73 ,1 95
2 ,6 8 4, 98 2
25 ,5 9 5, 4 87
3 ,2 4 5
3 4, 80 1
2 8 0, 9 28
2 ,7 27 ,3 86
26 ,0 77 ,5 53
R U
1 1 ,9 80
16 8, 15 6
1 ,4 3 6 ,0 7 8
1 4 ,1 5 1 ,7 28
1 3 4 ,7 0 6, 1 20
2 1 ,7 5 1
3 09 ,2 79
2 ,6 36 ,5 91
25 ,9 90 ,2 63
2 47
,0 98 ,7 58
1 ,8 54
2 4, 74 6
21 6, 63 8
2 ,1 5 0 ,7 4 6
2 0 ,4 2 1 ,9 6 5
3 ,1 1 0
29 ,1 06
23 1, 29 3
2 ,3 07 ,6 56
22 ,0 66 ,9 45
Z H
3, 3 18
4 2, 5 7 2
37 2, 0 03
3 ,6 59 ,6 1 7
34 ,6 72 ,6 1 2
8 ,5 59
1 16 ,6 67
1 ,0 19 ,9 6 9
9 ,9 9 0, 04 6
94 ,2 68 ,8 4 0
1 ,7 5 1
23 ,5 68
2 07 ,7 14
2 ,0 38 ,6 3 9
1 9 ,3 61 ,1 0 1
2 ,2 23
2 5, 45 9
2 15 ,1 79
2 ,0 83 ,6 32
19 ,8 85 ,7 89
Z H
_p in
yi n
10 ,8 11
1 48 ,0 8 3
1 ,2 7 9, 58 6
1 2 ,5 3 0 ,5 67
1 18 ,4 71 ,2 2 9
Z H
_w ub
i 9, 2 45
12 7, 19 0
1 ,1 0 4 ,5 2 9
1 0 ,8 8 5 ,5 08
1 0 2 ,9 0 8, 1 55
A R
_c p1
25 6
9, 07 9
12 3, 83 2
1 ,0 83 ,5 17
10 ,6 25 ,0 47
1 02
,0 64 ,2 30
R U
_c p1
25 1
11 ,9 80
16 8, 15 6
1 ,4 36 ,0 78
14 ,1 51 ,7 28
13 4 ,7 06 ,1 20
O O
V ty
pe ra
te (% ) A R 40 .7 7
23 .0 8
2 .3 1
0 .0 0
0 .0 0
40 .6 5
2 2 .7 6
2 .4 4
0 .0 0
0 .0 0
95 .8 0
79 .8 0
4 8 .0 2
2 4 .1 9
8 .1 4
1 0 .1 7
1 .3 9
0 .0 9
0 .0 0
0 .0 0
E N
2 3. 71
2 0 .6 2
11 .3 4
4 .1 2
2 .0 6
2 6 .4 7
23 .5 3
6 .8 6
0 .9 8
0 .0 0
92 .0 7
6 9 .0 8
33 .5 6
1 5 .2 7
7 .3 8
4 .7 3
1 .0 1
0 .1 8
0 .0 5
0 .0 1
E S
2 5. 23
20 .7 2
1 6 .2 2
8 .1 1
7 .2 1
2 3 .6 4
19 .0 9
11 .8 2
4 .5 5
0 .9 1
9 3 .3 2
71 .9 0
3 4 .7 2
13 .2 4
6 .1 2
5 .4 4
1 .0 6
0 .2 7
0 .1 0
0 .0 8
FR 25 .6 6
19 .4 7
1 1 .5 0
4 .4 2
2 .6 5
27 .1 2
20 .3 4
7 .6 3
2 .5 4
0 .0 0
9 2 .5 0
70 .2 5
3 2 .1 6
1 2 .3 1
5 .6 2
5 .1 1
0 .9 9
0 .2 6
0 .0 5
0 .0 2
R U
34 .9 3
2 6 .7 1
4 .1 1
2 .0 5
0 .0 0
32 .6 4
24 .3 1
3 .4 7
1 .3 9
0 .0 0
94 .5 2
7 5 .8 4
39 .1 4
1 4 .5 4
5 .6 9
6 .3 8
1 .2 7
0 .0 6
0 .0 2
0 .0 0
Z H
6 9 .7 9
3 9 .7 3
10 .5 8
1 .8 7
0 .5 6
10 .4 6
7 .1 9
3 .2 7
1 .3 1
0 .0 0
93 .1 9
7 3 .2 9
38 .5 9
1 5 .6 3
5 .6 5
76 .0 9
40 .9 5
8 .5 2
0 .8 9
0 .2 1
Z H
_p in
yi n
2 0 .4 1
1 5 .3 1
8 .1 6
4 .0 8
0 .0 0
Z H
_w ub
i 17 .5 0
1 3 .3 3
5 .8 3
3 .3 3
0 .0 0
A R
_c p1
25 6
4 0 .7 7
2 3 .0 8
2 .3 1
0 .0 0
0 .0 0
R U
_c p1
25 1
34 .9 3
26 .7 1
4 .1 1
2 .0 5
0 .0 0
O O
V to
ke n
ra te
(% )
A R
0 .7 5
0 .3 5
0 .0 0
0 .0 0
0 .0 0
0 .4 1
0 .1 9
0 .0 0
0 .0 0
0 .0 0
60 .8 6
40 .2 8
17 .9 4
8 .6 2
2 .1 0
2 .5 7
1 .9 9
0 .1 1
0 .0 0
0 .0 0
E N
0 .0 5
0 .0 2
0 .0 0
0 .0 0
0 .0 0
0 .0 6
0 .0 2
0 .0 0
0 .0 0
0 .0 0
3 9 .0 4
18 .5 7
6 .1 6
2 .7 0
1 .0 1
0 .3 5
0 .0 8
0 .0 2
0 .0 1
0 .0 0
E S
0 .0 7
0 .0 1
0 .0 1
0 .0 1
0 .0 1
0 .0 7
0 .0 2
0 .0 1
0 .0 1
0 .0 0
36 .2 1
1 8 .2 8
6 .2 8
2 .3 4
0 .8 6
0 .2 8
0 .1 3
0 .0 5
0 .0 3
0 .0 3
FR 0 .1 9
0 .1 0
0 .0 7
0 .0 0
0 .0 0
0 .2 1
0 .1 1
0 .0 7
0 .0 0
0 .0 0
35 .4 4
17 .4 7
6 .0 4
2 .1 3
0 .7 6
0 .6 4
0 .5 2
0 .3 6
0 .0 1
0 .0 0
R U
0 .3 0
0 .2 4
0 .1 6
0 .0 0
0 .0 0
0 .1 6
0 .1 3
0 .0 8
0 .0 0
0 .0 0
53 .2 2
30 .9 4
1 2 .5 9
4 .1 7
1 .2 8
0 .8 8
1 .0 7
0 .8 4
0 .0 1
0 .0 0
Z H
1 7. 83
3 .6 4
0 .3 8
0 .0 4
0 .0 1
0 .0 5
0 .0 2
0 .0 0
0 .0 0
0 .0 0
4 5 .4 8
22 .0 8
8 .3 4
3 .2 0
0 .8 3
31 .6 7
11 .7 6
1 .6 6
0 .1 4
0 .0 3
Z H
_p in
yi n
0 .0 6
0 .0 3
0 .0 0
0 .0 0
0 .0 0
Z H
_w ub
i 0 .0 7
0 .0 4
0 .0 0
0 .0 0
0 .0 0
A R
_c p1
25 6
0. 7 5
0 .3 5
0 .0 0
0 .0 0
0 .0 0
R U
_c p1
25 1
0. 3 0
0 .2 4
0 .1 6
0 .0 0
0 .0 0
T T
R (% ) A
R 0 .9 0
0 .0 9
0 .0 1
0 .0 0
0 .0 0
0 .4 4
0 .0 4
0 .0 1
0 .0 0
0 .0 0
48 .7 6
25 .0 0
12 .6 4
4 .8 1
1 .5 7
2 4 .0 8
1 4 .7 8
8 .1 4
1 .3 4
0 .1 4
E N
0 .6 4
0 .0 5
0 .0 1
0 .0 0
0 .0 0
0 .6 5
0 .0 5
0 .0 1
0 .0 0
0 .0 0
3 2 .7 9
12 .0 0
5 .0 8
1 .5 9
0 .5 6
2 1 .6 4
9 .5 9
4 .2 0
1 .1 0
0 .1 3
E S
0 .6 8
0 .0 5
0 .0 1
0 .0 0
0 .0 0
0 .6 7
0 .0 5
0 .0 1
0 .0 0
0 .0 0
31 .6 8
1 3 .0 2
5 .7 5
1 .9 1
0 .5 8
20 .7 9
10 .0 0
4 .4 4
1 .0 6
0 .1 2
FR 0 .6 9
0 .0 5
0 .0 1
0 .0 0
0 .0 0
0 .6 8
0 .0 5
0 .0 1
0 .0 0
0 .0 0
32 .2 9
12 .1 1
5 .1 2
1 .6 0
0 .4 6
2 2 .2 8
9 .1 8
4 .1 8
1 .0 0
0 .1 1
R U
0. 79
0 .0 6
0 .0 1
0 .0 0
0 .0 0
0 .4 5
0 .0 4
0 .0 1
0 .0 0
0 .0 0
47 .6 3
21 .9 4
1 0 .8 0
3 .7 8
1 .1 6
27 .0 7
14 .8 4
7 .7 8
1 .2 7
0 .1 4
Z H
18 .1 4
2 .9 0
0 .6 1
0 .0 9
0 .0 1
1 .6 0
0 .1 2
0 .0 2
0 .0 0
0 .0 0
3 5 .8 1
13 .4 8
6 .3 2
2 .3 0
0 .6 9
3 5 .9 9
1 1 .1 9
4 .9 5
1 .3 1
0 .1 5
Z H
_p in
yi n
0. 7 3
0 .0 6
0 .0 1
0 .0 0
0 .0 0
Z H
_w ub
i 1. 08
0 .0 8
0 .0 1
0 .0 0
0 .0 0
A R
_c p1
25 6
0. 90
0 .0 9
0 .0 1
0 .0 0
0 .0 0
R U
_c p1
25 1
0. 79
0 .0 6
0 .0 1
0 .0 0
0 .0 0
M ea
n lin
e le
ng th ±
st d
0/ 25
/5 0/
75 /1
00 -t
h
A R
90 .7 9 ± 5 4 .4 5
12 3 .8 3 ± 47 .0 9
1 0 8 .3 5 ± 5 3 .7 9
1 06
.2 5 ± 5 4 .8 8
1 02
.0 6 ± 5 7 .2 5
16 6 .5 5 ± 99 .8 8
22 7 .1 6 ± 86 .0 9
19 8 .5 0 ± 99 .0 2
19 4 .8 8 ± 1 01 .0 3
18 6 .1 7 ± 10 4 .8 6
17 .7 6 ± 1 0 .7 8
23 .4 6 ± 9. 12
2 0 .6 5 ± 10 .2 4
2 0 .3 5 ± 1 0 .4 6
19 .4 1 ± 10 .8 3
2 9 .0 7 ± 1 7 .4 8
2 8 .2 3 ± 1 1 .3 3
22 .3 5 ± 11 .3 0
22 .1 8 ± 11 .6 6
21 .3 4 ± 12 .1 3
8/ 33 / 91 / 13 6 /2 2 3
5 /9 4 /1 2 6/ 15 5 / 26 3
1/ 69 /1 09 / 14 7 /2 99
1/ 64 / 10 7 /1 4 7/ 29 9
1 /5 4 /1 0 2/ 1 4 5/ 30 0
16 /6 2 /1 66 / 2 47 /4 04
9/ 17 2/ 23 0/ 28 5/ 47 9
1/ 12 6/ 20 0/ 26 9/ 55 4
1 /1 17 /1 96 / 27 0 /5 61
1/ 9 8/ 18 7/ 26 5/ 56 5
1 /7 / 17 / 27 / 48
1/ 1 8 /2 4 /2 9 /5 6
1 / 13 / 2 0/ 28 /1 0 4
1/ 1 3/ 20 / 2 8/ 10 7
1/ 11 / 1 9/ 27 / 1 3 7
3/ 1 4/ 27 / 4 3/ 75
1/ 2 1 /2 8 /3 5 /7 1
1/ 14 / 2 2/ 3 0/ 1 1 3
1/ 14 / 22 / 30 / 12 2
1/ 12 / 21 / 30 / 17 3
E N
11 7 .3 0 ± 72 .5 1
1 5 9 .4 4 ± 5 8 .8 8
1 34
.4 0 ± 66 .7 9
13 1 .3 3 ± 68 .2 5
1 2 3 .4 9 ± 69 .5 3
1 1 7 .3 1 ± 7 2 .5 0
1 59
.4 5 ± 5 8. 88
13 4 .5 8 ± 6 6 .8 0
13 1 .5 9 ± 68 .3 4
12 3 .7 1 ± 69 .6 4
2 0 .7 1 ± 12 .1 6
27 .4 0 ± 10 .3 5
23 .6 6 ± 1 1 .6 8
2 3 .3 9 ± 1 2 .0 2
2 1 .9 4 ± 1 2 .2 9
29 .8 0 ± 1 7 .1 8
29 .7 8 ± 11 .3 2
2 4 .3 6 ± 1 2 .0 9
23 .7 0 ± 12 .2 0
22 .3 5 ± 12 .5 3
8 /4 9 /1 12 /1 7 0 / 26 5
8/ 11 7 /1 6 3/ 2 03 /2 8 5
1 /8 6 /1 37 /1 8 5 / 3 0 0
1/ 7 9/ 1 34 /1 83 /3 0 0
1 /6 5 /1 25 / 17 7 /3 00
8/ 4 9/ 1 12 /1 70 /2 6 5
8 /1 17 / 1 63 /2 03 / 28 5
1/ 87 / 13 7/ 18 5/ 30 0
1 / 80 / 13 4/ 18 4/ 30 0
1 /6 5 /1 25 / 1 77 /3 11
1 /9 / 19 / 30 / 46
1/ 2 0 /2 8 /3 5 /6 0
1 /1 5 /2 4 /3 2 /9 3
1 / 1 5/ 24 / 3 2/ 10 6
1/ 12 /2 2 /3 1 /1 3 6
3 /1 6 /2 7 /4 2 /7 0
1 /2 2 /3 0 /3 7 /6 8
1 / 16 / 2 4/ 33 / 1 01
1/ 15 /2 4 /3 2 /1 06
1 /1 2 /2 2 /3 2 /1 38
E S
12 3 .7 4 ± 7 8 .9 1
17 1 .1 0 ± 64 .0 3
1 4 8 .4 8 ± 73 .3 8
1 45
.5 0 ± 7 5 .3 2
1 38
.6 0 ± 7 7 .5 6
12 6 .2 9 ± 8 0. 2 1
17 5 .2 9 ± 65 .4 4
15 1 .3 8 ± 74 .7 0
14 8 .2 1 ± 76 .6 7
14 1 .2 8 ± 78 .9 6
22 .3 2 ± 13 .9 7
29 .4 6 ± 11 .3 5
2 6 .3 0 ± 1 3. 03
25 .8 9 ± 13 .3 5
2 4 .6 5 ± 1 3. 7 8
3 1 .8 0 ± 19 .3 9
32 .3 2 ± 1 2 .6 0
2 7 .2 0 ± 13 .5 7
26 .4 4 ± 13 .6 7
25 .2 6 ± 14 .1 4
6/ 4 2/ 1 19 /1 8 6 /2 96
6/ 12 7 /1 7 5/ 21 7 / 29 7
1/ 9 5/ 1 5 0/ 20 5 /3 0 0
1 /8 8 /1 48 / 20 4 /3 0 0
1/ 7 3/ 1 40 /2 0 0 /3 00
7 /4 4 /1 21 / 19 1 /3 0 2
7 /1 30 /1 79 / 2 22 /3 06
1 /9 7 /1 53 / 20 9/ 30 7
1 /8 9 /1 50 / 20 8/ 31 2
1 /7 4 /1 43 /2 03 / 31 8
1/ 7 /2 2 /3 2 /5 2
1/ 2 2/ 2 9/ 3 7/ 63
1 /1 7 /2 6 /3 6 /1 04
1 /1 6 /2 6 /3 6 /1 07
1 /1 3 /2 5 /3 5 /1 3 9
4 /1 6 /3 1 /4 3 /7 9
1/ 24 / 3 2/ 41 / 7 2
1 /1 8 /2 7 /3 7 /1 06
1 /1 6 /2 7 /3 7 /1 07
1 /1 4 /2 5 /3 6 /1 40
FR 12 4 .5 6 ± 79 .3 8
1 79
.0 5 ± 6 6 .4 4
14 9 .1 0 ± 74 .1 9
1 4 5 .2 9 ± 75 .7 5
1 3 8 .0 5 ± 7 7 .9 0
1 28
.7 5 ± 82 .1 6
1 85
.2 3 ± 68 .5 2
1 54
.2 1 ± 7 6. 66
15 0 .5 6 ± 78 .4 2
14 3 .5 0 ± 80 .9 9
22 .9 8 ± 1 4 .1 6
3 2 .0 1 ± 11 .9 9
2 7 .3 2 ± 13 .5 3
26 .8 5 ± 1 3 .8 9
25 .6 0 ± 14 .4 5
3 2 .4 5 ± 19 .5 2
34 .8 0 ± 1 3 .0 9
28 .0 9 ± 13 .9 6
27 .2 7 ± 14 .1 3
26 .0 8 ± 14 .7 1
5/ 4 5 /1 16 / 1 9 0/ 28 7
5/ 1 33 / 1 83 /2 2 8/ 30 0
1 /9 3 /1 5 2/ 2 0 6 /3 0 0
1 /8 7 /1 4 8/ 20 4 / 30 0
1/ 7 2 /1 40 / 19 9 /3 00
5/ 4 7 /1 2 1/ 19 6 / 29 7
5/ 1 38 /1 90 /2 36 / 3 12
1 /9 6 /1 57 /2 13 / 32 2
1 /9 0 /1 53 /2 12 / 32 2
1/ 75 / 14 5/ 20 7/ 33 1
1 /9 /2 2 /3 4 /5 0
1/ 2 4 /3 2 /4 1 /7 0
1 /1 7 /2 8 /3 7 /1 0 7
1 /1 6 /2 7 /3 7 /1 10
1/ 1 4/ 26 /3 7 /1 3 8
2 /1 6 /3 0 /4 6 /7 6
2 / 2 7/ 3 5/ 4 4/ 7 7
1 /1 8 /2 8 /3 8 /1 0 7
1 /1 7 /2 8 /3 8 /1 10
1 /1 4 /2 6 /3 7 /1 38
R U
11 9 .8 0 ± 7 2 .0 9
16 8 .1 6 ± 62 .1 1
1 43
.6 1 ± 7 1 .5 5
14 1 .5 2 ± 7 3. 9 5
1 34
.7 1 ± 76 .2 0
21 7 .5 1 ± 13 4 .5 1
30 9 .2 8 ± 11 4. 9 8
26 3 .6 6 ± 13 2. 7 7
25 9 .9 0 ± 13 7. 10
2 47
.1 0 ± 1 41 .1 4
1 8 .5 4 ± 10 .6 5
24 .7 5 ± 9. 53
2 1 .6 6 ± 10 .8 1
2 1 .5 1 ± 1 1 .1 2
20 .4 2 ± 1 1 .4 4
31 .1 0 ± 1 7. 42
2 9 .1 1 ± 11 .5 1
2 3 .1 3 ± 1 1 .7 1
23 .0 8 ± 12 .1 7
22 .0 7 ± 12 .5 7
1 0/ 4 4 /1 19 /1 83 / 2 6 0
5/ 12 6 / 17 4 /2 13 /2 9 8
1 /9 1 /1 46 / 19 7 /3 0 0
1 /8 5 /1 44 /1 9 9/ 3 00
1/ 70 /1 3 6/ 1 9 4/ 30 0
20 /7 7 /2 15 / 3 32 /4 89
10 /2 3 1/ 31 8 /3 9 2/ 55 4
1 /1 67 / 2 68 /3 63 /5 62
1 /1 55 /2 64 / 36 5/ 56 6
1 /1 27 /2 49 / 35 7 /5 67
1 /8 / 18 / 27 / 39
1/ 19 /2 5 /3 1 /5 8
1 / 14 /2 2 /2 9 /1 04
1/ 1 3/ 22 / 3 0/ 10 7
1/ 1 1/ 2 0/ 2 9/ 1 36
4/ 1 8/ 2 9/ 4 5/ 69
1/ 21 /2 9 /3 7 /6 8
1 /1 5 /2 3 /3 1 /1 0 4
1/ 14 / 23 / 32 /1 10
1/ 12 / 22 / 31 / 16 6
Z H
33 .1 8 ± 18 .3 8
4 2 .5 7 ± 1 6 .9 1
37 .2 0 ± 18 .4 6
3 6 .6 0 ± 19 .0 5
3 4 .6 7 ± 19 .2 9
85 .5 9 ± 51 .1 8
1 16
.6 7 ± 43 .8 2
1 02
.0 0 ± 50 .6 1
9 9 .9 0 ± 51 .4 9
94 .2 7 ± 52 .8 0
17 .5 1 ± 1 0 .6 8
2 3 .5 7 ± 9. 52
2 0 .7 7 ± 10 .4 8
2 0 .3 9 ± 1 0 .6 0
19 .3 6 ± 10 .8 7
2 2 .2 3 ± 1 2 .3 3
2 5 .4 6 ± 1 0. 29
21 .5 2 ± 10 .9 4
20 .8 4 ± 10 .9 2
19 .8 9 ± 11 .2 3
3 /1 9 /3 0 /4 5 /8 8
2 /3 2 /4 2 /5 3 /1 27
1 /2 4 /3 7 /4 9 /1 6 5
1 /2 3 /3 6 /4 9 /2 4 1
1 / 19 / 34 / 48 / 28 1
7/ 3 5 /8 1 /1 27 / 19 2
6/ 88 /1 17 / 14 8/ 24 3
1/ 66 / 10 3 /1 3 9/ 29 7
1 / 61 / 10 1 /1 38 /3 37
1 /5 1 /9 5 /1 34 / 59 4
2 /7 / 17 / 25 / 46
1/ 1 8 /2 3 /2 9 /7 6
1 / 13 / 2 0/ 28 / 1 08
1/ 1 3/ 2 0/ 2 8/ 11 1
1/ 1 1 /1 9 /2 7 /1 67
2 /1 3 /2 0 /3 1 /5 8
1 /1 9 /2 5 /3 2 /7 9
1/ 1 4/ 2 1/ 2 9/ 1 0 8
1/ 13 /2 1 /2 8 /1 11
1 /1 1 /2 0 /2 8 /1 67
Z H
_p in
yi n
1 08 .1 1 ± 65 .5 1
1 48
.0 8 ± 5 5 .5 7
12 7 .9 6 ± 63 .9 6
12 5 .3 1 ± 65 .2 5
1 1 8 .4 7 ± 6 6 .8 8
7 /4 3 /9 8 /1 61 /2 5 5
7 / 1 0 9 /1 4 9 /1 88 / 31 2
1/ 82 / 12 9 /1 7 5 /3 6 9
1 /7 6 /1 2 6/ 17 4 / 40 7
1/ 6 3/ 1 1 9/ 1 69 /6 4 5
Z H
_w ub
i 9 2. 4 5 ± 55 .2 6
1 2 7 .1 9 ± 4 8 .0 0
11 0 .4 5 ± 5 4. 7 1
10 8 .8 6 ± 56 .0 4
1 0 2 .9 1 ± 57 .4 1
8/ 3 9 /8 6 /1 37 / 21 8
7/ 94 /1 2 9 / 1 5 8/ 26 6
1/ 71 / 11 1 /1 5 0 /3 2 7
1 /6 7 /1 1 0/ 1 50 /3 6 4
1/ 5 5/ 1 0 3/ 1 46 / 62 7
A R
_c p1
25 6
90 .7 9 ± 54 .4 5
1 23
.8 3 ± 47 .0 9
10 8 .3 5 ± 53 .7 9
10 6 .2 5 ± 54 .8 8
10 2 .0 6 ± 57 .2 5
8 /3 3 /9 1 /1 3 6/ 22 3
5 /9 4 /1 26 /1 5 5/ 26 3
1/ 69 /1 09 / 14 7 /2 99
1 /6 4 /1 07 / 1 47 /2 99
1 /5 4 /1 0 2/ 14 5/ 30 0
R U
_c p1
25 1
1 1 9. 8 0 ± 72 .0 9
16 8 .1 6 ± 62 .1 1
1 43
.6 1 ± 71 .5 5
14 1 .5 2 ± 7 3 .9 5
13 4 .7 1 ± 76 .2 0
10 /4 4 /1 19 /1 8 3 / 26 0
5/ 1 26 / 1 74 /2 13 /2 98
1 / 91 / 14 6 /1 97 /3 00
1 /8 5 /1 44 / 19 9/ 30 0
1 /7 0 /1 36 / 19 4 /3 00
St at
is tic
s fo
rd at
as et
B
R ep
re se
nt at
io n
C H
A R
B Y
T E
W O
R D
B PE
N um
be ro
fl in
es 1 00
1, 0 0 0
10 ,0 00
10 0, 0 00
1 ,0 00 ,0 0 0
1 00
1 ,0 00
1 0, 00 0
1 00 ,0 00
1 ,0 00 ,0 00
1 00
1 ,0 0 0
1 0, 00 0
1 0 0, 0 00
1 ,0 00 ,0 0 0
10 0
1 ,0 0 0
10 ,0 00
10 0, 00 0
1 ,0 00 ,0 00
N um
be r
of T
Y PE S A R 8 7
1 34
1 68
25 1
43 1
88 12 4
14 7
15 9
18 1
1 ,1 2 9
6, 71 9
28 ,2 22
9 7, 46 7
31 1, 35 5
8 10
4, 40 8
18 ,8 00
29 ,6 80
29 ,9 59
E N
7 6
9 9
1 3 1
1 8 9
3 43
7 9
10 3
13 5
16 3
18 1
8 76
4 ,1 48
13 ,6 60
4 2, 84 4
14 0, 91 2
7 84
3, 40 8
10 ,7 56
26 ,8 20
29 ,1 78
E S
86 1 0 4
1 34
1 89
36 6
89 10 8
13 3
16 1
18 5
97 0
4 ,7 75
16 ,7 35
5 1, 36 6
15 4, 48 2
8 14
3, 70 5
12 ,5 29
28 ,0 68
29 ,4 71
FR 8 9
11 2
1 4 7
2 02
35 2
91 12 0
14 2
16 2
18 5
95 5
4 ,6 36
1 5, 6 39
4 5, 34 8
13 2, 88 1
8 12
3, 67 0
12 ,2 58
27 ,7 44
29 ,3 70
R U
1 0 9
1 4 2
1 73
2 04
32 0
11 2
14 1
15 9
16 1
18 0
1 ,1 42
6, 5 20
2 5, 9 59
8 3, 93 7
25 1, 3 55
99 3
4, 82 9
18 ,8 37
29 ,2 61
29 ,8 11
Z H
7 61
1, 5 75
2, 4 02
3, 4 65
4, 7 05
1 23
15 3
16 5
17 5
18 8
89 7
4, 16 2
14 ,7 14
4 6, 7 8 8
14 0, 18 8
1, 06 8
3, 7 17
1 1, 06 9
27 ,4 09
29 ,1 07
Z H
_p in
yi n
7 1
1 04
14 0
24 1
4 3 0
Z H
_w ub
i 8 7
1 24
15 4
24 6
4 2 1
A R
_c p1
25 6
8 7
13 4
16 8
25 1
43 1
R U
_c p1
25 1
1 09
1 42
1 73
2 04
3 20
N um
be r
of TO
K E
N S
A R
9 ,7 9 8
1 00 ,5 99
1 ,0 1 9, 69 6
10 ,2 48 ,9 7 6
10 2 ,4 81 ,8 1 6
1 7 ,6 5 6
18 1, 8 66
1 ,8 42 ,1 38
1 8 ,5 1 7, 66 9
18 5 ,1 72 ,6 1 8
1 ,8 2 5
18 ,7 6 0
19 0, 46 5
1 ,9 1 5 ,7 4 6
1 9 ,1 38 ,7 2 4
3 ,3 8 7
2 5, 2 34
21 0, 62 2
2 ,0 90 ,4 03
20 ,9 86 ,1 16
E N
1 1, 81 6
1 1 9, 7 9 6
1 ,2 0 1, 8 39
1 2 ,0 8 7, 92 2
1 20
,9 6 9, 16 3
11 ,8 20
11 9, 83 9
1 ,2 02 ,2 17
12 ,0 91 ,6 02
12 1 ,0 06 ,8 03
2 ,0 47
21 ,0 58
2 11 ,0 29
2 ,1 22 ,3 6 5
21 ,2 3 4, 4 37
3 ,2 62
2 4, 63 4
2 20 ,3 90
2 ,1 59 ,5 66
21 ,6 69 ,7 69
E S
13 ,5 4 3
13 5, 1 06
1 ,3 63 ,4 5 4
13 ,7 09 ,4 9 3
13 7 ,1 80 ,5 4 2
13 ,7 9 0
13 7, 6 13
1 ,3 89 ,3 15
1 3 ,9 67 ,0 72
1 39
,7 53 ,2 63
2 ,3 90
23 ,8 58
24 1, 78 9
2 ,4 3 1 ,5 7 9
24 ,3 26 ,0 5 5
3 ,7 0 2
2 8, 1 7 9
25 3, 07 5
2 ,4 85 ,2 92
24 ,9 33 ,9 87
FR 1 3, 17 3
1 34 ,7 02
1 ,3 5 8 ,5 6 7
13 ,6 62 ,0 2 4
13 6 ,8 11 ,8 7 0
13 ,6 17
1 39 ,2 50
1 ,4 04 ,0 01
14 ,1 21 ,6 29
1 41
,4 03 ,2 57
2 ,3 62
24 ,4 42
24 7, 02 7
2 ,4 8 5 ,2 0 3
24 ,8 74 ,3 5 5
3 ,6 6 4
2 8, 5 4 1
25 7, 30 5
2 ,5 27 ,8 89
25 ,3 74 ,6 29
R U
13 ,1 06
1 3 0, 28 9
1 ,3 2 5 ,7 9 3
1 3 ,3 5 6 ,7 17
1 3 3 ,6 9 9 ,4 73
24 ,0 59
23 9, 14 5
2 ,4 33 ,1 54
24 ,5 12 ,9 4 9
24 5 ,3 96 ,3 32
1 ,9 60
19 ,6 42
2 0 0, 1 8 8
2 ,0 1 4, 8 89
2 0 ,1 53 ,1 2 4
3 ,7 6 0
26 ,1 36
21 8, 3 2 5
2 ,1 70 ,4 10
21 ,7 92 ,2 16
Z H
3 ,3 29
3 3, 76 8
3 40 ,6 9 2
3 ,4 19 ,8 0 3
34 ,2 06 ,1 0 6
9 ,1 2 2
92 ,6 89
9 35 ,1 72
9 ,4 02 ,7 43
94 ,0 64 ,3 53
1 ,8 66
18 ,6 97
1 8 9, 7 7 3
1 ,9 0 6, 7 04
1 9 ,0 73 ,1 3 3
2 ,5 6 3
21 ,6 30
19 9, 0 2 8
1 ,9 46 ,5 43
19 ,5 74 ,5 72
Z H
_p in
yi n
1 1, 39 1
1 15 ,5 77
1 ,1 6 6, 2 27
1 1 ,7 32 ,2 6 4
1 17
,3 6 7, 34 9
Z H
_w ub
i 10 ,0 35
10 1, 04 9
1 ,0 1 9 ,1 1 3
1 0 ,2 4 2, 5 40
1 0 2 ,4 6 0 ,5 73
A R
_c p1
25 6
9, 79 8
10 0, 59 9
1 ,0 19 ,6 96
10 ,2 48 ,9 76
10 2 ,4 81 ,8 16
R U
_c p1
25 1
13 ,1 0 6
13 0, 2 89
1 ,3 25 ,7 93
13 ,3 56 ,7 17
13 3 ,6 99 ,4 7 3
O O
V ty
pe ra
te (% ) A R 35 .3 8
5 .3 8
0 .7 7
0 .0 0
0 .0 0
28 .4 6
4 .8 8
0 .8 1
0 .0 0
0 .0 0
93 .6 1
72 .0 7
35 .5 1
1 3 .0 7
4 .8 2
7 .8 6
0 .3 0
0 .0 2
0 .0 0
0 .0 0
E N
2 5 .7 7
1 1 .3 4
8 .2 5
2 .0 6
0 .0 0
26 .4 7
11 .7 6
5 .8 8
0 .0 0
0 .0 0
89 .5 8
61 .0 4
2 6 .2 8
10 .2 5
4 .1 8
4 .0 0
0 .4 4
0 .1 5
0 .0 4
0 .0 0
E S
2 6 .1 3
12 .6 1
1 1 .7 1
3 .6 0
0 .0 0
24 .5 5
1 0 .9 1
8 .1 8
0 .9 1
0 .0 0
90 .5 9
64 .2 7
2 7 .7 3
9 .6 1
3 .9 5
4 .0 8
0 .4 4
0 .1 7
0 .0 6
0 .0 0
FR 2 3 .0 1
1 1 .5 0
7 .9 6
2 .6 5
0 .0 0
24 .5 8
7 .6 3
5 .0 8
0 .8 5
0 .0 0
9 0 .2 3
62 .6 1
2 5 .7 9
8 .7 1
3 .3 6
3 .8 7
0 .4 7
0 .1 2
0 .0 5
0 .0 0
R U
26 .7 1
6 .1 6
2 .7 4
0 .0 0
0 .0 0
23 .6 1
4 .8 6
2 .0 8
0 .0 0
0 .0 0
93 .2 0
69 .7 5
32 .2 0
1 1 .1 0
3 .9 1
4 .3 9
0 .2 3
0 .0 3
0 .0 1
0 .0 0
Z H
61 .8 4
2 6 .9 7
6 .6 8
1 .0 1
0 .1 5
20 .2 6
2 .6 1
0 .6 5
0 .0 0
0 .0 0
8 9 .7 3
64 .1 0
2 9 .5 4
1 0 .4 4
3 .8 4
6 7 .8 9
27 .0 0
5 .2 7
0 .6 7
0 .0 4
Z H
_p in
yi n
27 .5 5
8 .1 6
2 .0 4
0 .0 0
0 .0 0
Z H
_w ub
i 2 8 .3 3
4 .1 7
0 .8 3
0 .0 0
0 .0 0
A R
_c p1
25 6
35 .3 8
5 .3 8
0 .7 7
0 .0 0
0 .0 0
R U
_c p1
25 1
2 6 .7 1
6 .1 6
2 .7 4
0 .0 0
0 .0 0
O O
V to
ke n
ra te
(% )
A R
0. 31
0 .0 1
0 .0 0
0 .0 0
0 .0 0
0 .1 1
0 .0 0
0 .0 0
0 .0 0
0 .0 0
53 .9 4
25 .8 9
9 .2 6
3 .1 4
1 .1 5
0 .8 2
0 .0 8
0 .0 0
0 .0 0
0 .0 0
E N
0. 05
0 .0 0
0 .0 0
0 .0 0
0 .0 0
0 .0 5
0 .0 0
0 .0 0
0 .0 0
0 .0 0
34 .1 1
12 .1 8
3 .5 0
1 .2 2
0 .5 0
0 .3 0
0 .0 2
0 .0 1
0 .0 1
0 .0 0
E S
0. 04
0 .0 1
0 .0 1
0 .0 0
0 .0 0
0 .0 4
0 .0 1
0 .0 1
0 .0 0
0 .0 0
3 1 .5 2
12 .5 7
3 .7 2
1 .1 5
0 .4 6
0 .1 9
0 .0 3
0 .0 4
0 .0 2
0 .0 0
FR 0. 07
0 .0 1
0 .0 0
0 .0 0
0 .0 0
0 .0 8
0 .0 0
0 .0 0
0 .0 0
0 .0 0
30 .9 6
11 .9 1
3 .3 8
0 .9 9
0 .3 9
0 .2 2
0 .0 5
0 .0 1
0 .0 1
0 .0 0
R U
0. 22
0 .0 0
0 .0 0
0 .0 0
0 .0 0
0 .1 1
0 .0 0
0 .0 0
0 .0 0
0 .0 0
4 9 .0 5
22 .7 5
7 .5 5
2 .3 6
0 .8 1
0 .7 1
0 .0 2
0 .0 0
0 .0 0
0 .0 0
Z H
10 .4 4
1 .2 1
0 .1 6
0 .0 2
0 .0 0
0 .1 0
0 .0 0
0 .0 0
0 .0 0
0 .0 0
3 7 .8 1
13 .7 4
4 .4 2
1 .4 2
0 .5 3
22 .0 2
4 .9 9
0 .7 9
0 .1 0
0 .0 0
Z H
_p in
yi n
0. 0 3
0 .0 0
0 .0 0
0 .0 0
0 .0 0
Z H
_w ub
i 0. 0 9
0 .0 0
0 .0 0
0 .0 0
0 .0 0
A R
_c p1
25 6
0. 31
0 .0 1
0 .0 0
0 .0 0
0 .0 0
R U
_c p1
25 1
0. 22
0 .0 0
0 .0 0
0 .0 0
0 .0 0
T T
R (% ) A
R 0. 89
0 .1 3
0 .0 2
0 .0 0
0 .0 0
0 .5 0
0 .0 7
0 .0 1
0 .0 0
0 .0 0
61 .8 6
35 .8 2
14 .8 2
5 .0 9
1 .6 3
2 3 .9 1
1 7 .4 7
8 .9 3
1 .4 2
0 .1 4
E N
0. 6 4
0 .0 8
0 .0 1
0 .0 0
0 .0 0
0 .6 7
0 .0 9
0 .0 1
0 .0 0
0 .0 0
4 2 .7 9
19 .7 0
6 .4 7
2 .0 2
0 .6 6
24 .0 3
1 3 .8 3
4 .8 8
1 .2 4
0 .1 3
E S
0. 6 4
0 .0 8
0 .0 1
0 .0 0
0 .0 0
0 .6 5
0 .0 8
0 .0 1
0 .0 0
0 .0 0
40 .5 9
20 .0 1
6 .9 2
2 .1 1
0 .6 4
2 1 .9 9
1 3 .1 5
4 .9 5
1 .1 3
0 .1 2
FR 0. 68
0 .0 8
0 .0 1
0 .0 0
0 .0 0
0 .6 7
0 .0 9
0 .0 1
0 .0 0
0 .0 0
40 .4 3
1 8 .9 7
6 .3 3
1 .8 2
0 .5 3
22 .1 6
12 .8 6
4 .7 6
1 .1 0
0 .1 2
R U
0. 8 3
0 .1 1
0 .0 1
0 .0 0
0 .0 0
0 .4 7
0 .0 6
0 .0 1
0 .0 0
0 .0 0
58 .2 7
33 .1 9
12 .9 7
4 .1 7
1 .2 5
2 6 .4 1
1 8 .4 8
8 .6 3
1 .3 5
0 .1 4
Z H
22 .8 6
4 .6 6
0 .7 1
0 .1 0
0 .0 1
1 .3 5
0 .1 7
0 .0 2
0 .0 0
0 .0 0
48 .0 7
22 .2 6
7 .7 5
2 .4 5
0 .7 4
4 1 .6 7
1 7 .1 8
5 .5 6
1 .4 1
0 .1 5
Z H
_p in
yi n
0. 62
0 .0 9
0 .0 1
0 .0 0
0 .0 0
Z H
_w ub
i 0. 87
0 .1 2
0 .0 2
0 .0 0
0 .0 0
A R
_c p1
25 6
0. 8 9
0 .1 3
0 .0 2
0 .0 0
0 .0 0
R U
_c p1
25 1
0. 83
0 .1 1
0 .0 1
0 .0 0
0 .0 0
M ea
n lin
e le
ng th ±
st d
0/ 25
/5 0/
75 /1
00 -t
h
A R
9 7 .9 8 ± 55 .4 7
1 00
.6 0 ± 5 9 .6 0
10 1 .9 7 ± 59 .1 0
10 2 .4 9 ± 58 .3 6
1 0 2 .4 8 ± 5 8. 39
1 76
.5 6 ± 10 1 .9 7
18 1 .8 7 ± 10 8. 68
1 84
.2 1 ± 10 7. 4 9
18 5 .1 8 ± 1 06 .2 2
18 5 .1 7 ± 10 6 .2 9
18 .2 5 ± 10 .1 2
18 .7 6 ± 11 .1 0
19 .0 5 ± 10 .9 5
1 9 .1 6 ± 1 0. 83
19 .1 4 ± 10 .8 2
33 .8 7 ± 1 8 .3 8
2 5 .2 3 ± 15 .2 6
2 1 .0 6 ± 1 2. 27
20 .9 0 ± 12 .0 2
20 .9 9 ± 12 .0 8
9/ 5 3/ 9 4/ 14 7 /2 34
2 /4 9 /1 0 1/ 1 47 /2 7 4
1/ 5 1/ 1 0 1/ 1 47 / 30 0
1/ 5 3 /1 03 / 14 7 /3 00
1 /5 3 /1 02 /1 4 7/ 3 00
1 4/ 98 / 1 70 /2 6 6/ 4 32
2 / 88 / 18 2/ 26 6/ 50 1
1 /9 0 /1 84 / 2 67 /5 51
1 /9 5 /1 85 / 26 6 /5 52
1 /9 5 /1 85 / 2 66 /5 61
2/ 10 / 17 / 27 / 43
1 /9 / 18 / 27 / 73
1/ 1 0/ 19 / 2 7/ 84
1/ 1 0 /1 9 /2 7 /1 4 3
1 /1 0 /1 9 /2 7 /1 2 9
2 /2 0 /3 2 /5 0 /7 3
1 / 13 / 2 4/ 36 /9 6
1 /1 1 /2 1 /3 0 /9 5
1 /1 1 /2 1 /2 9 /1 46
1 /1 1 /2 1 /2 9 /1 65
E N
1 18 .1 6 ± 6 8. 05
1 19
.8 0 ± 7 1. 1 5
12 0 .1 8 ± 70 .0 0
12 0 .8 8 ± 69 .4 3
1 2 0 .9 7 ± 6 9 .5 4
11 8 .2 0 ± 68 .0 5
11 9 .8 4 ± 71 .1 7
12 0 .2 2 ± 70 .0 2
12 0 .9 2 ± 69 .4 4
12 1 .0 1 ± 69 .5 5
20 .4 7 ± 11 .0 2
2 1 .0 6 ± 12 .6 0
2 1 .1 0 ± 12 .2 5
2 1 .2 2 ± 1 2. 14
2 1 .2 3 ± 12 .1 5
32 .6 2 ± 1 7. 93
2 4 .6 3 ± 1 4 .9 0
2 2 .0 4 ± 12 .7 8
21 .6 0 ± 12 .3 4
21 .6 7 ± 12 .4 0
7/ 6 4/ 1 14 /1 76 /2 56
2 /5 8 /1 2 2/ 17 4 / 28 8
1/ 6 0/ 1 2 1/ 1 76 /2 9 9
1 /6 2 /1 22 /1 7 5/ 3 0 0
1/ 6 2/ 1 22 /1 7 5 /3 00
7 / 6 4/ 11 4/ 17 6 /2 5 6
2 /5 8 /1 22 / 17 5 /2 88
1 /6 0 /1 21 / 17 6/ 29 9
1 /6 2 /1 22 /1 75 / 30 0
1 /6 2 /1 22 /1 75 / 3 03
1/ 12 /1 9 /3 1 /4 2
1 /1 1 /2 1 /3 1 /7 4
1 /1 1 /2 1 /3 0 /8 6
1 /1 1 /2 1 /3 0 /1 17
1 /1 1 /2 1 /3 0 /1 2 7
2 /1 9 /3 1 /4 7 /8 1
1/ 1 3/ 2 4/ 3 5/ 8 5
1/ 1 1/ 2 2/ 3 1 /8 9
1 /1 2 /2 2 /3 1 /1 17
1 /1 2 /2 2 /3 1 /1 29
E S
13 5 .4 3 ± 7 7 .5 4
1 35
.1 1 ± 7 9. 66
1 36
.3 5 ± 7 9 .0 0
13 7 .0 9 ± 78 .2 2
13 7 .1 8 ± 78 .4 5
1 37
.9 0 ± 78 .9 3
13 7 .6 1 ± 81 .0 3
1 38
.9 3 ± 8 0. 43
13 9 .6 7 ± 79 .6 1
13 9 .7 5 ± 79 .8 4
23 .9 0 ± 1 3 .2 9
23 .8 6 ± 14 .1 2
2 4 .1 8 ± 1 3 .9 4
24 .3 2 ± 1 3 .8 3
2 4 .3 3 ± 13 .8 6
3 7 .0 2 ± 20 .6 8
28 .1 8 ± 1 6 .9 6
25 .3 1 ± 14 .6 0
24 .8 5 ± 14 .1 3
24 .9 3 ± 14 .2 1
1 1 /8 0 /1 33 / 20 0 /2 86
2/ 65 / 13 7 /2 00 /3 0 0
1 /6 7 /1 3 8/ 2 00 /3 0 0
1 /7 0 /1 39 /1 9 9/ 3 0 0
1/ 70 /1 3 9/ 2 0 0/ 30 0
11 /8 1 /1 34 / 2 02 /2 88
2 /6 5 /1 40 /2 05 / 30 5
1 /6 9 /1 40 /2 04 / 30 9
1 /7 1 /1 41 /2 03 / 31 1
1/ 71 / 14 1 /2 03 /3 18
1 /1 4 /2 4 /3 4 /4 9
1/ 12 / 2 4/ 3 5/ 80
1/ 1 2 /2 4 /3 5 /8 7
1 /1 3 /2 4 /3 5 /1 1 9
1/ 13 / 2 4/ 35 /1 3 0
2 /2 2 /3 6 /5 4 /8 1
1 /1 4 /2 7 /4 1 /9 3
1 / 1 3/ 25 / 3 7/ 91
1 /1 3 /2 5 /3 6 /1 19
1 /1 3 /2 5 /3 6 /1 33
FR 13 1 .7 3 ± 7 5 .9 5
1 34
.7 0 ± 8 0 .4 8
1 35
.8 6 ± 7 9 .5 1
13 6 .6 2 ± 78 .7 1
13 6 .8 1 ± 78 .9 0
1 36
.1 7 ± 78 .5 0
13 9 .2 5 ± 83 .0 9
14 0 .4 0 ± 82 .1 8
14 1 .2 2 ± 8 1 .3 5
1 41
.4 0 ± 81 .5 5
23 .6 2 ± 13 .4 0
24 .4 4 ± 14 .6 9
2 4 .7 0 ± 1 4 .4 1
2 4 .8 5 ± 1 4 .2 7
2 4 .8 7 ± 14 .3 0
3 6 .6 4 ± 21 .1 8
28 .5 4 ± 17 .3 1
2 5 .7 3 ± 1 4 .9 6
25 .2 8 ± 14 .4 9
25 .3 7 ± 14 .5 7
8/ 7 4 /1 28 / 19 6 /2 8 9
2/ 6 4/ 1 35 /2 0 2 /3 00
1 /6 5 /1 3 7 / 1 9 9/ 30 0
1/ 69 / 13 8 /1 99 / 3 00
1/ 6 9 /1 38 / 2 00 /3 0 0
8 /7 7 /1 32 / 20 3 /2 96
2 /6 5 /1 40 /2 07 / 31 3
1 /6 8 /1 42 /2 06 / 3 15
1/ 7 1/ 1 43 /2 06 /3 22
1 / 7 1/ 1 43 /2 06 /3 25
1 /1 3 /2 3 /3 4 /5 3
1 / 12 / 24 / 36 / 7 8
1 /1 2 /2 5 /3 6 /9 2
1/ 1 3/ 25 / 3 6/ 11 8
1/ 1 3/ 2 5/ 36 / 1 32
2/ 2 1/ 35 / 5 4/ 83
1/ 15 /2 7 /4 2 /9 4
1 /1 3 /2 6 /3 7 /9 7
1 / 13 / 25 / 36 / 11 8
1/ 13 / 26 / 36 / 13 4
R U
1 31 .0 6 ± 7 6 .8 9
1 30
.2 9 ± 7 7 .8 0
13 2 .5 8 ± 77 .8 0
13 3 .5 7 ± 77 .0 9
1 3 3 .7 0 ± 7 7. 19
2 40
.5 9 ± 14 4 .2 5
2 39
.1 5 ± 1 44 .3 5
24 3 .3 2 ± 14 4 .1 4
24 5 .1 3 ± 14 2 .8 9
24 5 .4 0 ± 14 3. 0 8
1 9 .6 0 ± 1 0. 5 8
1 9 .6 4 ± 1 1 .5 8
20 .0 2 ± 11 .6 3
2 0 .1 5 ± 1 1 .5 0
20 .1 5 ± 1 1 .5 1
37 .6 0 ± 2 1 .3 0
2 6 .1 4 ± 15 .9 5
2 1 .8 3 ± 1 2 .7 9
21 .7 0 ± 12 .5 5
21 .7 9 ± 12 .6 3
10 /7 1 /1 2 3 /1 9 5/ 2 71
2/ 62 /1 3 0/ 1 9 3 /2 9 9
1 /6 5 /1 3 3/ 19 5 / 30 0
1/ 6 7/ 1 3 4/ 19 4 /3 0 0
1 /6 7 /1 34 / 19 5 /3 00
1 2 /1 2 5/ 22 9 /3 6 3 /5 05
2 /1 13 /2 38 / 3 52 /5 58
1 /1 17 /2 45 / 35 8 /5 65
1 /1 2 2/ 24 6/ 35 7/ 56 6
1/ 12 2/ 24 7 / 35 8/ 56 9
1 /1 2 /2 0 /2 8 /4 3
1 / 10 / 20 / 29 / 7 2
1 /1 0 /2 0 /2 9 /8 5
1/ 1 1/ 20 / 2 9/ 11 7
1/ 1 1/ 2 0/ 29 / 1 28
2/ 2 1/ 33 / 5 3/ 91
1/ 13 /2 5 /3 8 /8 9
1 /1 1 /2 2 /3 1 /8 7
1 / 11 / 22 / 31 / 12 0
1/ 11 /2 2 /3 1 /1 62
Z H
33 .2 9 ± 1 7 .4 1
33 .7 7 ± 19 .7 7
3 4 .0 7 ± 19 .5 0
3 4 .2 0 ± 1 9 .2 1
34 .2 1 ± 1 9 .2 6
9 1 .2 2 ± 50 .8 8
92 .6 9 ± 55 .2 8
93 .5 2 ± 54 .2 9
94 .0 3 ± 53 .6 7
9 4 .0 6 ± 53 .7 3
18 .6 6 ± 10 .2 8
1 8 .7 0 ± 11 .1 8
1 8 .9 8 ± 11 .0 5
1 9 .0 7 ± 1 0. 88
1 9 .0 7 ± 10 .9 0
2 5 .6 3 ± 1 3 .6 5
2 1 .6 3 ± 1 3. 19
19 .9 0 ± 11 .6 1
19 .4 7 ± 11 .1 3
19 .5 7 ± 11 .2 4
4 /1 9 /3 2 /4 7 /7 6
2 / 18 / 33 / 48 / 14 8
1 /1 8 /3 4 /4 8 /1 5 2
1/ 1 9/ 34 / 48 / 20 1
1 /1 9 /3 4 /4 8 /2 89
8 / 5 3/ 90 / 13 4 /2 0 9
2 /4 5 /9 2 /1 33 / 29 0
1/ 4 8 /9 3 /1 35 /2 83
1 /4 9 /9 4 /1 3 5/ 36 9
1 /4 9 /9 4 /1 35 /6 67
2 / 11 / 18 / 26 / 48
1/ 10 /1 8 /2 7 /6 8
1/ 1 0/ 1 9/ 2 7/ 9 9
1/ 10 /1 9 /2 7 /1 1 6
1 /1 0 /1 9 /2 7 /1 67
2 /1 6 /2 4 /3 4 /6 3
1/ 11 / 2 1/ 3 1 /8 6
1/ 1 0 /2 0 /2 8 /1 0 1
1 /1 1 /1 9 /2 7 /1 16
1 /1 1 /1 9 /2 8 /1 67
Z H
_p in
yi n
1 13 .9 1 ± 6 4 .4 6
1 15
.5 8 ± 6 9. 1 8
11 6 .6 2 ± 68 .1 3
11 7 .3 2 ± 67 .4 1
1 1 7 .3 7 ± 6 7 .5 0
8/ 6 6/ 1 09 /1 7 4 /2 65
2 /5 5 /1 1 5 /1 67 / 3 25
1/ 5 8 /1 1 7/ 1 69 /3 5 7
1 /6 0 /1 18 /1 6 9 / 3 9 4
1/ 6 0/ 1 18 /1 6 9 /6 80
Z H
_w ub
i 10 0 .3 5 ± 5 5. 7 0
10 1 .0 5 ± 60 .1 3
1 0 1 .9 1 ± 5 8 .9 4
1 0 2 .4 3 ± 5 8 .2 1
1 02
.4 6 ± 5 8. 2 5
8/ 59 / 9 9/ 15 0 /2 26
2 /4 8 /1 0 0/ 14 6 / 32 5
1/ 5 2/ 10 2/ 1 47 /3 1 8
1 /5 3 /1 02 / 1 47 /3 8 1
1/ 5 4/ 1 03 /1 4 6 /6 88
A R
_c p1
25 6
9 7 .9 8 ± 5 5 .4 7
10 0 .6 0 ± 59 .6 0
10 1 .9 7 ± 59 .1 0
10 2 .4 9 ± 58 .3 6
1 02
.4 8 ± 5 8 .3 9
9/ 5 3/ 94 / 14 7 /2 34
2/ 49 /1 01 / 1 47 /2 74
1 /5 1 /1 0 1/ 14 7/ 30 0
1 /5 3 /1 03 / 14 7/ 30 0
1 /5 3 /1 02 /1 47 / 3 00
R U
_c p1
25 1
13 1 .0 6 ± 7 6 .8 9
13 0 .2 9 ± 77 .8 0
13 2 .5 8 ± 77 .8 0
13 3 .5 7 ± 77 .0 9
13 3 .7 0 ± 77 .1 9
10 / 7 1/ 12 3 /1 95 /2 71
2 / 62 / 13 0/ 19 3/ 29 9
1 /6 5 /1 33 / 19 5 /3 00
1 /6 7 /1 34 / 1 94 /3 00
1 /6 7 /1 3 4/ 19 5/ 30 0
St at
is tic
s fo
rd at
as et
C
R ep
re se
nt at
io n
C H
A R
B Y
T E
W O
R D
B PE
N um
be ro
fl in
es 1 00
1, 0 0 0
10 ,0 00
10 0, 0 00
1 ,0 00 ,0 0 0
10 0
1 ,0 00
10 ,0 00
10 0, 00 0
1 ,0 0 0, 00 0
10 0
1 ,0 00
1 0, 00 0
1 00 ,0 00
1 ,0 00 ,0 00
10 0
1 ,0 0 0
10 ,0 00
10 0, 00 0
1 ,0 00 ,0 00
N um
be r
of T
Y PE S A R 97
13 6
16 2
2 6 1
4 27
92 12 3
14 4
16 0
17 6
1 ,1 6 5
6, 82 3
2 8, 2 9 6
97 ,3 80
3 1 0, 6 32
86 4
4, 5 30
1 8, 7 3 5
29 ,6 73
29 ,9 54
E N
78 98
12 6
18 6
3 1 0
81 10 3
13 1
15 9
17 8
90 6
4 ,1 73
1 3, 6 5 8
42 ,6 77
1 4 1, 1 63
80 2
3, 4 84
1 0, 8 0 0
26 ,7 66
29 ,2 15
E S
84 10 8
13 3
1 8 6
3 06
85 11 2
13 3
16 0
17 5
97 0
4 ,7 34
1 6, 8 5 0
51 ,2 81
1 5 4, 6 49
80 3
3, 6 86
1 2, 4 7 0
28 ,1 34
29 ,4 70
FR 87
11 3
13 9
1 9 5
3 30
91 11 7
13 4
15 9
17 9
99 1
4 ,6 03
1 5, 5 8 6
45 ,2 35
1 3 2, 4 93
85 3
3, 6 93
1 2, 1 7 0
27 ,7 31
29 ,3 55
R U
9 8
1 41
16 7
21 1
2 8 7
1 00
14 0
15 4
16 6
17 7
1 ,1 65
6, 49 0
2 6, 1 1 2
84 ,1 37
2 5 0, 7 78
1, 0 02
4, 8 58
1 9, 0 2 9
29 ,3 22
29 ,8 14
Z H
78 3
1, 55 0
2, 44 3
3, 47 7
4, 69 8
12 8
15 2
16 2
17 7
18 9
88 2
4, 16 0
1 4, 7 3 0
46 ,6 67
1 3 9, 8 95
1, 0 84
3, 7 16
1 1, 2 1 4
27 ,4 43
29 ,1 10
Z H
_p in
yi n
7 4
1 07
13 9
24 0
4 1 3
Z H
_w ub
i 9 5
1 29
15 4
24 2
4 0 8
A R
_c p1
25 6
97 13 6
16 2
26 1
42 7
R U
_c p1
25 1
98 14 1
16 7
21 1
28 7
N um
be r
of TO
K E
N S
A R
1 0, 21 2
1 04 ,1 44
1 ,0 2 1, 13 6
10 ,2 31 ,0 8 1
10 2 ,4 48 ,3 3 9
1 8 ,4 48
18 8, 50 7
1 ,8 4 5, 13 6
18 ,4 8 2, 68 9
18 5 ,1 05 ,4 9 8
1 ,8 8 9
1 9, 28 1
1 90 ,7 38
1 ,9 12 ,6 6 9
1 9 ,1 3 7, 2 40
3 ,5 18
2 5, 60 1
2 10 ,7 3 8
2 ,0 87 ,4 31
20 ,9 81 ,7 60
E N
12 ,3 00
1 2 3, 6 4 6
1 ,2 0 8, 8 27
1 2 ,0 8 9, 4 49
1 20
,9 3 3, 96 2
12 ,3 05
12 3, 68 1
1 ,2 09 ,1 87
12 ,0 93 ,1 17
12 0 ,9 71 ,4 60
2 ,1 16
21 ,5 45
2 1 2, 0 3 4
2 ,1 2 3, 1 53
2 1 ,2 31 ,4 6 5
3 ,3 6 9
25 ,1 11
22 1, 26 4
2 ,1 60 ,1 62
21 ,6 66 ,4 82
E S
1 4, 1 25
13 9, 56 3
1 ,3 6 9 ,7 7 9
1 3 ,7 0 1 ,3 8 7
1 3 7 ,1 3 3 ,4 40
14 ,3 87
1 42 ,2 37
1 ,3 95 ,3 0 5
13 ,9 58 ,2 21
13 9 ,7 0 2, 85 3
2 ,4 72
24 ,6 00
24 2, 67 2
2 ,4 3 0 ,5 8 6
24 ,3 1 9, 76 0
3 ,7 84
2 8, 7 2 6
2 54 ,0 92
2 ,4 83 ,9 95
24 ,9 25 ,2 09
FR 1 3, 79 7
1 38 ,5 28
1 ,3 6 5, 36 0
13 ,6 61 ,1 5 0
13 6 ,7 54 ,3 6 0
1 4 ,2 4 3
14 3, 1 39
1 ,4 11 ,4 51
14 ,1 19 ,8 15
14 1 ,3 43 ,6 7 5
2 ,4 53
24 ,9 5 0
2 48 ,4 88
2 ,4 85 ,1 18
2 4 ,8 6 4, 4 49
3 ,7 5 5
2 8, 90 7
2 5 8, 55 0
2 ,5 27 ,9 57
25 ,3 62 ,7 24
R U
1 3, 76 3
1 36 ,6 7 4
1 ,3 3 9, 2 86
1 3 ,3 6 9, 69 0
1 33
,6 7 1, 32 2
25 ,3 45
25 1, 23 1
2 ,4 5 7, 9 38
24 ,5 35 ,7 71
24 5 ,3 38 ,1 39
2 ,0 24
2 0, 47 4
20 1, 6 1 8
2 ,0 1 6, 4 5 1
2 0 ,1 56 ,4 6 7
3 ,8 1 3
26 ,7 72
21 9, 79 4
2 ,1 72 ,6 16
21 ,7 87 ,5 39
Z H
3 ,3 71
3 4, 42 9
3 41 ,5 36
3 ,4 19 ,6 4 8
34 ,2 00 ,5 4 3
9 ,2 35
95 ,1 00
93 8, 8 05
9 ,4 04 ,3 86
94 ,0 52 ,8 81
1 ,8 24
19 ,1 20
19 0, 48 5
1 ,9 07 ,6 4 3
19 ,0 7 3, 4 63
2 ,5 59
2 2, 06 7
1 9 9, 7 06
1 ,9 47 ,3 00
19 ,5 73 ,3 72
Z H
_p in
yi n
11 ,6 0 5
11 9, 2 58
1 ,1 71 ,6 0 5
1 1 ,7 32 ,0 10
1 1 7 ,3 5 6 ,8 8 0
Z H
_w ub
i 10 ,0 7 1
10 3, 9 44
1 ,0 22 ,8 7 0
1 0 ,2 45 ,7 16
1 0 2 ,4 5 0 ,5 8 1
A R
_c p1
25 6
10 ,2 12
1 04 ,1 44
1 ,0 21 ,1 36
10 ,2 31 ,0 81
1 02
,4 48 ,3 39
R U
_c p1
25 1
13 ,7 63
13 6, 67 4
1 ,3 39 ,2 86
13 ,3 69 ,6 9 0
13 3 ,6 71 ,3 22
O O
V ty
pe ra
te (% ) A R 28 .4 6
5 .3 8
0 .0 0
0 .0 0
0 .0 0
26 .0 2
4 .8 8
0 .0 0
0 .0 0
0 .0 0
93 .7 2
7 1 .1 3
3 5 .7 0
1 3 .4 4
5 .0 2
6 .0 6
0 .3 4
0 .0 0
0 .0 0
0 .0 0
E N
21 .6 5
1 2 .3 7
8 .2 5
2 .0 6
0 .0 0
22 .5 5
11 .7 6
4 .9 0
0 .9 8
0 .0 0
89 .4 7
60 .5 6
26 .2 7
10 .4 3
4 .3 2
3 .5 2
0 .4 9
0 .1 8
0 .0 4
0 .0 0
E S
2 5 .2 3
1 4 .4 1
1 0 .8 1
5 .4 1
0 .9 0
2 3 .6 4
1 0 .9 1
6 .3 6
0 .9 1
0 .0 0
90 .7 1
63 .9 6
2 7 .1 8
9 .7 1
3 .8 8
4 .4 4
0 .5 3
0 .1 9
0 .0 8
0 .0 0
FR 23 .0 1
9 .7 3
7 .0 8
1 .7 7
0 .0 0
22 .8 8
9 .3 2
5 .0 8
0 .8 5
0 .0 0
89 .9 2
62 .2 7
2 5 .9 6
8 .8 9
3 .4 8
3 .9 6
0 .4 9
0 .1 5
0 .0 2
0 .0 0
R U
3 3 .5 6
7 .5 3
2 .0 5
0 .0 0
0 .0 0
3 1 .2 5
5 .5 6
1 .3 9
0 .0 0
0 .0 0
9 3 .1 1
69 .6 6
32 .1 5
1 0 .9 1
3 .8 1
4 .8 2
0 .2 9
0 .0 2
0 .0 1
0 .0 0
Z H
6 1 .2 3
26 .8 7
6 .7 3
1 .4 2
0 .3 5
1 6 .9 9
3 .9 2
0 .6 5
0 .0 0
0 .0 0
90 .3 4
6 3 .4 8
29 .8 9
10 .3 1
4 .0 9
6 7 .9 8
2 6 .2 2
5 .0 3
0 .5 5
0 .1 0
Z H
_p in
yi n
27 .5 5
8 .1 6
1 .0 2
0 .0 0
0 .0 0
Z H
_w ub
i 2 3 .3 3
4 .1 7
0 .8 3
0 .0 0
0 .0 0
A R
_c p1
25 6
28 .4 6
5 .3 8
0 .0 0
0 .0 0
0 .0 0
R U
_c p1
25 1
33 .5 6
7 .5 3
2 .0 5
0 .0 0
0 .0 0
O O
V to
ke n
ra te
(% )
A R
0. 1 3
0 .0 1
0 .0 0
0 .0 0
0 .0 0
0 .0 5
0 .0 0
0 .0 0
0 .0 0
0 .0 0
53 .4 0
2 5 .2 4
9 .3 1
3 .2 5
1 .1 8
0 .7 3
0 .1 2
0 .0 0
0 .0 0
0 .0 0
E N
0. 03
0 .0 0
0 .0 0
0 .0 0
0 .0 0
0 .0 3
0 .0 0
0 .0 0
0 .0 0
0 .0 0
34 .2 6
11 .6 3
3 .4 1
1 .2 6
0 .5 1
0 .2 0
0 .0 3
0 .0 2
0 .0 0
0 .0 0
E S
0. 0 4
0 .0 1
0 .0 1
0 .0 0
0 .0 0
0 .0 5
0 .0 1
0 .0 1
0 .0 0
0 .0 0
32 .1 7
12 .0 5
3 .5 6
1 .1 6
0 .4 6
0 .2 9
0 .0 9
0 .0 4
0 .0 2
0 .0 0
FR 0. 14
0 .0 0
0 .0 0
0 .0 0
0 .0 0
0 .1 2
0 .0 0
0 .0 0
0 .0 0
0 .0 0
31 .3 9
11 .6 6
3 .3 0
1 .0 3
0 .4 1
0 .5 1
0 .0 4
0 .0 2
0 .0 0
0 .0 0
R U
0. 15
0 .0 0
0 .0 0
0 .0 0
0 .0 0
0 .0 8
0 .0 0
0 .0 0
0 .0 0
0 .0 0
4 9 .4 8
22 .4 4
7 .5 3
2 .3 3
0 .7 9
0 .3 8
0 .0 4
0 .0 0
0 .0 0
0 .0 0
Z H
11 .1 5
1 .2 5
0 .1 8
0 .0 4
0 .0 1
0 .1 0
0 .0 0
0 .0 0
0 .0 0
0 .0 0
38 .6 5
13 .9 6
4 .5 2
1 .4 2
0 .5 6
22 .0 1
4 .7 5
0 .8 1
0 .0 9
0 .0 1
Z H
_p in
yi n
0. 0 7
0 .0 0
0 .0 0
0 .0 0
0 .0 0
Z H
_w ub
i 0 .0 9
0 .0 0
0 .0 0
0 .0 0
0 .0 0
A R
_c p1
25 6
0. 13
0 .0 1
0 .0 0
0 .0 0
0 .0 0
R U
_c p1
25 1
0. 15
0 .0 0
0 .0 0
0 .0 0
0 .0 0
T T
R (% ) A
R 0. 9 5
0 .1 3
0 .0 2
0 .0 0
0 .0 0
0 .5 0
0 .0 7
0 .0 1
0 .0 0
0 .0 0
61 .6 7
3 5 .3 9
1 4 .8 4
5 .0 9
1 .6 2
24 .5 6
1 7 .6 9
8 .8 9
1 .4 2
0 .1 4
E N
0. 63
0 .0 8
0 .0 1
0 .0 0
0 .0 0
0 .6 6
0 .0 8
0 .0 1
0 .0 0
0 .0 0
42 .8 2
19 .3 7
6 .4 4
2 .0 1
0 .6 6
2 3 .8 1
1 3 .8 7
4 .8 8
1 .2 4
0 .1 3
E S
0. 5 9
0 .0 8
0 .0 1
0 .0 0
0 .0 0
0 .5 9
0 .0 8
0 .0 1
0 .0 0
0 .0 0
39 .2 4
19 .2 4
6 .9 4
2 .1 1
0 .6 4
21 .2 2
12 .8 3
4 .9 1
1 .1 3
0 .1 2
FR 0. 63
0 .0 8
0 .0 1
0 .0 0
0 .0 0
0 .6 4
0 .0 8
0 .0 1
0 .0 0
0 .0 0
40 .4 0
18 .4 5
6 .2 7
1 .8 2
0 .5 3
2 2 .7 2
1 2 .7 8
4 .7 1
1 .1 0
0 .1 2
R U
0. 71
0 .1 0
0 .0 1
0 .0 0
0 .0 0
0 .3 9
0 .0 6
0 .0 1
0 .0 0
0 .0 0
5 7 .5 6
31 .7 0
12 .9 5
4 .1 7
1 .2 4
2 6 .2 8
1 8 .1 5
8 .6 6
1 .3 5
0 .1 4
Z H
23 .2 3
4 .5 0
0 .7 2
0 .1 0
0 .0 1
1 .3 9
0 .1 6
0 .0 2
0 .0 0
0 .0 0
48 .3 6
21 .7 6
7 .7 3
2 .4 5
0 .7 3
42 .3 6
16 .8 4
5 .6 2
1 .4 1
0 .1 5
Z H
_p in
yi n
0 .6 4
0 .0 9
0 .0 1
0 .0 0
0 .0 0
Z H
_w ub
i 0 .9 4
0 .1 2
0 .0 2
0 .0 0
0 .0 0
A R
_c p1
25 6
0. 95
0 .1 3
0 .0 2
0 .0 0
0 .0 0
R U
_c p1
25 1
0. 71
0 .1 0
0 .0 1
0 .0 0
0 .0 0
M ea
n lin
e le
ng th ±
st d
0/ 25
/5 0/
75 /1
00 -t
h
A R
10 2 .1 2 ± 57 .4 0
1 0 4 .1 4 ± 5 7 .8 3
10 2 .1 1 ± 58 .0 6
10 2 .3 1 ± 58 .3 3
1 0 2 .4 5 ± 58 .4 6
1 8 4 .4 8 ± 10 5. 32
18 8 .5 1 ± 10 5. 63
18 4 .5 1 ± 10 5. 7 8
18 4 .8 3 ± 10 6. 15
1 85
.1 1 ± 10 6 .4 0
18 .8 9 ± 10 .2 7
19 .2 8 ± 10 .5 9
19 .0 7 ± 10 .7 2
1 9 .1 3 ± 1 0. 82
19 .1 4 ± 10 .8 4
3 5 .1 8 ± 1 8 .9 2
2 5 .6 0 ± 1 4. 4 2
2 1 .0 7 ± 12 .0 1
20 .8 7 ± 12 .0 0
20 .9 8 ± 12 .1 0
5/ 5 7/ 9 6/ 15 2 /2 1 3
4 /5 4 /1 0 6 / 14 9 /2 64
3/ 53 / 10 2 /1 4 7/ 29 3
1 /5 3 /1 0 2/ 1 4 7 /2 9 8
1 /5 3 /1 0 2 /1 4 7 / 3 0 0
9/ 1 01 / 17 3 /2 80 /3 8 3
6 / 9 8/ 1 91 /2 70 /4 87
3 /9 4 /1 84 / 26 6 /5 33
1 /9 5 /1 84 / 2 66 /5 51
1 /9 5 /1 8 5/ 26 6 /5 59
2/ 11 / 18 / 27 / 4 3
1 /1 0 /2 0 /2 7 /5 0
1 / 10 / 1 9/ 27 / 8 2
1 /1 0 /1 9 /2 7 /1 20
1 /1 0 /1 9 /2 7 /1 43
4 /1 8 /3 4 /5 0 /7 3
1/ 1 4/ 26 / 3 6/ 79
1/ 11 /2 1 /3 0 /8 6
1 /1 1 /2 1 /2 9 /1 23
1/ 11 / 21 / 29 /1 47
E N
12 3 .0 0 ± 72 .3 0
1 2 3 .6 5 ± 7 1. 1 2
12 0 .8 8 ± 69 .6 2
12 0 .8 9 ± 69 .5 8
1 2 0 .9 3 ± 6 9 .6 0
12 3 .0 5 ± 72 .3 5
12 3 .6 8 ± 71 .1 5
12 0 .9 2 ± 69 .6 4
12 0 .9 3 ± 69 .6 0
12 0 .9 7 ± 69 .6 2
21 .1 6 ± 1 2 .2 4
2 1 .5 5 ± 12 .3 1
2 1 .2 0 ± 1 2 .1 5
21 .2 3 ± 12 .1 7
2 1 .2 3 ± 1 2 .1 7
3 3 .6 9 ± 19 .3 2
25 .1 1 ± 1 4 .3 2
2 2 .1 3 ± 1 2 .6 7
21 .6 0 ± 12 .3 6
21 .6 7 ± 12 .4 2
9/ 6 3/ 1 16 /1 87 /2 6 8
3 /6 0 /1 28 /1 7 8 / 3 0 0
1/ 6 1/ 1 22 /1 7 5 /3 00
1 /6 2 /1 22 /1 75 / 3 00
1/ 61 / 12 2 /1 7 5/ 30 0
9 / 63 / 1 16 /1 87 /2 68
3 /6 0 /1 2 8/ 17 8/ 30 0
1 /6 1 /1 22 / 1 75 /3 00
1 /6 2 /1 2 2/ 17 5/ 30 0
1 /6 1 /1 22 /1 75 / 30 4
2/ 10 /2 1 /3 1 /4 8
1 /1 1 /2 2 /3 1 /5 5
1 /1 1 /2 1 /3 0 /8 3
1 /1 1 /2 1 /3 0 /1 18
1 /1 1 /2 1 /3 0 /1 36
3 /1 5 /3 3 /4 9 /7 8
1/ 1 3/ 26 / 3 6/ 86
1/ 12 /2 2 /3 2 /8 8
1/ 12 / 22 / 31 / 12 0
1/ 12 / 22 / 31 / 13 8
E S
1 41 .2 5 ± 8 2 .8 3
13 9 .5 6 ± 79 .5 9
13 6 .9 8 ± 78 .3 5
13 7 .0 1 ± 78 .4 5
1 3 7 .1 3 ± 7 8 .4 7
1 43
.8 7 ± 84 .3 7
14 2 .2 4 ± 81 .0 5
1 39
.5 3 ± 7 9. 72
13 9 .5 8 ± 7 9 .8 4
1 39
.7 0 ± 79 .8 7
24 .7 2 ± 14 .3 9
24 .6 0 ± 13 .9 1
24 .2 7 ± 1 3 .8 0
2 4 .3 1 ± 1 3 .8 7
2 4 .3 2 ± 1 3 .8 7
37 .8 4 ± 2 1. 74
2 8 .7 3 ± 1 6 .2 7
2 5 .4 1 ± 14 .4 6
24 .8 4 ± 14 .1 7
24 .9 3 ± 14 .2 2
11 /5 9 /1 3 4 /2 1 5/ 2 92
3/ 67 / 1 45 /2 0 1 /3 00
1 /7 0 /1 3 8 / 2 0 0/ 30 0
1/ 70 / 13 8 /1 99 / 30 0
1/ 7 0 /1 39 / 1 99 /3 0 0
1 1 /6 0 /1 36 /2 1 8/ 29 8
3 /6 9 /1 49 /2 0 6/ 30 6
1 /7 1 /1 40 /2 04 / 30 8
1/ 7 1/ 14 1/ 20 3/ 31 2
1 / 7 1/ 1 41 /2 03 /3 16
2 /1 2 /2 4 /3 7 /5 6
1 / 12 / 25 / 36 / 60
1/ 13 /2 4 /3 5 /8 4
1 / 13 / 2 4/ 35 / 1 19
1/ 1 3/ 24 / 3 5/ 13 9
4/ 17 / 3 5/ 56 / 8 6
1 /1 4 /3 0 /4 1 /9 0
1 / 1 3/ 25 / 3 6 /9 0
1/ 13 /2 5 /3 6 /1 21
1 /1 3 /2 5 /3 6 /1 40
FR 13 7 .9 7 ± 80 .1 2
13 8 .5 3 ± 79 .4 9
13 6 .5 4 ± 78 .8 0
13 6 .6 1 ± 7 8 .9 2
1 36
.7 5 ± 7 8. 96
1 42
.4 3 ± 8 2 .9 7
14 3 .1 4 ± 82 .1 8
14 1 .1 5 ± 81 .4 7
14 1 .2 0 ± 81 .5 6
14 1 .3 4 ± 81 .6 1
24 .5 3 ± 13 .7 9
24 .9 5 ± 14 .1 9
2 4 .8 5 ± 1 4. 28
2 4 .8 5 ± 14 .3 1
2 4 .8 6 ± 1 4. 31
37 .5 5 ± 20 .9 7
28 .9 1 ± 16 .4 1
25 .8 6 ± 1 4 .8 5
25 .2 8 ± 14 .5 3
25 .3 6 ± 14 .5 8
6 /6 2 /1 33 / 21 5 /2 9 3
3 /6 8 /1 4 1/ 2 0 3 /3 0 0
1 /6 8 /1 3 7/ 20 0 / 30 0
1/ 6 9/ 1 3 8/ 19 9 /3 0 0
1 /6 9 /1 38 /1 9 9/ 3 00
6 /6 4 /1 37 /2 19 / 30 3
3/ 70 / 14 5 /2 09 /3 15
1 / 71 / 14 2/ 20 6/ 31 9
1 /7 1 /1 4 2/ 20 6/ 32 3
1 /7 1 /1 43 / 20 6/ 33 0
2/ 13 / 24 / 37 / 50
1/ 1 2 /2 6 /3 6 /6 2
1/ 1 3 /2 5 /3 6 /8 4
1/ 13 /2 5 /3 6 /1 1 8
1 /1 3 /2 5 /3 6 /1 38
3 /2 0 /3 7 /5 6 /8 0
1 /1 4 /3 0 /4 2 /8 6
1 /1 3 /2 6 /3 7 /8 8
1 /1 3 /2 5 /3 6 /1 20
1 /1 3 /2 6 /3 6 /1 38
R U
13 7 .6 3 ± 8 0. 8 0
13 6 .6 7 ± 78 .7 7
1 3 3 .9 3 ± 7 7 .2 5
1 33
.7 0 ± 7 7 .2 6
13 3 .6 7 ± 77 .2 5
25 3 .4 5 ± 15 0 .9 0
2 51
.2 3 ± 14 6. 2 1
24 5 .7 9 ± 1 43 .3 1
2 45
.3 6 ± 14 3 .1 8
24 5 .3 4 ± 1 43 .1 7
20 .2 4 ± 11 .2 7
20 .4 7 ± 11 .6 1
20 .1 6 ± 1 1 .4 4
2 0 .1 6 ± 1 1. 53
20 .1 6 ± 1 1 .5 3
3 8 .1 3 ± 2 1 .1 0
2 6 .7 7 ± 15 .2 6
21 .9 8 ± 1 2 .6 1
21 .7 3 ± 12 .5 9
21 .7 9 ± 12 .6 4
11 /7 1 /1 3 0 / 21 2 /2 96
3 /6 3 /1 3 9 /2 0 0 / 3 00
1/ 67 / 13 5 /1 9 6/ 30 0
1 /6 7 /1 3 4/ 19 5 / 30 0
1/ 6 7/ 1 3 4/ 1 95 /3 0 0
21 / 1 2 6/ 24 2 /3 92 / 55 0
4/ 11 8 /2 58 /3 66 / 56 1
1/ 12 1/ 2 47 /3 60 / 5 62
1/ 12 2/ 24 7/ 35 8/ 56 6
1 /1 22 / 2 47 /3 58 /5 69
2 /1 1 /2 0 /2 9 /4 7
1 /1 0 /2 1 /2 9 /5 5
1 /1 1 /2 0 /2 9 /8 3
1 /1 1 /2 0 /2 9 /1 17
1 /1 1 /2 0 /2 9 /1 36
3 /1 8 /3 8 /5 6 /8 1
1/ 14 / 2 7/ 39 / 7 8
1 /1 2 /2 2 /3 1 /8 7
1/ 11 / 22 / 31 / 12 1
1/ 11 / 22 / 31 / 13 8
Z H
3 3 .7 1 ± 1 8. 0 3
34 .4 3 ± 19 .4 0
3 4 .1 5 ± 19 .0 6
3 4 .2 0 ± 1 9 .2 7
34 .2 0 ± 1 9 .2 8
92 .3 5 ± 52 .4 2
95 .1 0 ± 54 .2 9
93 .8 8 ± 53 .4 2
94 .0 4 ± 5 3. 75
94 .0 5 ± 53 .7 9
18 .2 4 ± 9. 97
19 .1 2 ± 10 .8 7
19 .0 5 ± 1 0 .8 6
1 9 .0 8 ± 1 0 .9 1
1 9 .0 7 ± 1 0 .9 2
25 .5 9 ± 1 3. 84
2 2 .0 7 ± 12 .6 8
19 .9 7 ± 11 .4 3
19 .4 7 ± 11 .1 7
19 .5 7 ± 11 .2 6
4/ 19 / 33 / 48 / 7 0
2/ 18 /3 5 /4 9 /1 2 7
1 / 19 / 34 / 48 / 1 33
1/ 19 /3 4 /4 8 /2 2 0
1 / 19 / 34 / 48 /2 7 2
1 2 /4 7 /9 3 /1 3 2/ 1 98
3/ 50 /9 6 /1 38 /2 48
3/ 4 9/ 94 / 13 5/ 29 1
1 / 49 / 94 / 13 5 /3 78
1/ 48 / 94 / 13 5/ 59 4
2 /1 0 /1 7 /2 6 /3 9
1 /1 0 /2 0 /2 7 /5 5
1/ 1 0 /1 9 /2 7 /8 7
1 /1 0 /1 9 /2 7 /1 1 3
1 /1 0 /1 9 /2 7 /2 4 8
2 /1 4 /2 6 /3 7 /5 2
1/ 12 / 2 2/ 31 / 9 0
1 /1 1 /2 0 /2 8 /9 2
1/ 11 / 19 / 27 / 11 5
1/ 11 / 19 / 28 / 24 9
Z H
_p in
yi n
1 16 .0 5 ± 6 7 .5 4
11 9 .2 6 ± 68 .5 4
11 7 .1 6 ± 67 .2 5
1 1 7 .3 2 ± 6 7. 49
1 1 7 .3 6 ± 6 7 .5 5
1 1 /5 8 /1 17 /1 67 / 26 3
3/ 61 / 12 2 /1 73 /3 2 0
3 /6 1 /1 1 7/ 1 69 /3 3 5
1/ 6 0/ 1 1 8/ 16 9 /3 76
1 /6 0 /1 18 / 16 9 /5 84
Z H
_w ub
i 10 0 .7 1 ± 57 .0 5
10 3 .9 4 ± 59 .4 8
10 2 .2 9 ± 5 7 .9 2
1 02
.4 6 ± 5 8. 29
1 02
.4 5 ± 5 8 .3 1
13 /5 4 /1 0 1/ 1 39 /2 1 7
3/ 5 4 /1 0 5/ 15 0 / 29 6
3/ 5 4 /1 02 / 14 7 /2 97
1 /5 3 /1 02 /1 46 / 3 98
1/ 53 / 10 3 /1 4 6/ 62 7
A R
_c p1
25 6
1 02 .1 2 ± 57 .4 0
10 4 .1 4 ± 57 .8 3
10 2 .1 1 ± 58 .0 6
10 2 .3 1 ± 58 .3 3
1 02
.4 5 ± 5 8. 4 6
5/ 57 /9 6 /1 52 / 21 3
4/ 54 /1 06 / 14 9 /2 64
3 /5 3 /1 0 2/ 14 7/ 29 3
1 /5 3 /1 02 /1 47 / 29 8
1 /5 3 /1 02 /1 47 / 3 00
R U
_c p1
25 1
13 7 .6 3 ± 8 0. 8 0
13 6 .6 7 ± 78 .7 7
13 3 .9 3 ± 77 .2 5
13 3 .7 0 ± 77 .2 6
13 3 .6 7 ± 77 .2 5
11 / 7 1/ 13 0 /2 12 /2 96
3 /6 3 /1 3 9/ 20 0 /3 00
1 /6 7 /1 35 / 19 6/ 30 0
1 /6 7 /1 34 / 19 5 /3 00
1 /6 7 /1 34 /1 95 / 3 00
St at
is tic
s fo
rd ev
el op
m en
t( de
v) se
t
A s
a di
ff er
en ts
et of
vo ca
bu la
ry is
le ar
ne d
fr om
ea ch
tr ai
ni ng
da ta
se ta
nd da
ta si
ze ,B
PE ha
s a
di st
in ct
de v
se tf
or ea
ch .
R ep
re se
nt at
io n
C H
A R
B Y
T E
W O
R D
B PE
_A B
PE _B
B PE _C N um be ro fl in es in tr ai n se t 1 00 1, 0 0 0 10 ,0 0 0 1 00 ,0 00 1 ,0 0 0, 00 0 10 0 1 ,0 00 10 ,0 00 10 0, 00 0 1 ,0 0 0 ,0 0 0 10 0 1 ,0 00 10 ,0 00
10 0, 00 0
1 ,0 00 ,0 0 0
N um
be r
of T
Y PE S A R 13 0
12 3
1 3, 83 6
7 0 8
3, 23 2
8, 9 94
11 ,4 7 3
12 ,4 30
8 6 5
3, 9 9 1
1 0, 7 3 5
12 ,9 54
1 3, 0 90
9 0 8
4, 1 1 9
1 0, 6 8 1
12 ,9 34
1 3, 09 3
E N
9 7
1 0 2
7, 19 9
6 5 6
2, 5 6 7
5, 9 9 8
7, 4 61
7, 5 56
8 01
3, 1 49
6, 5 1 8
7, 5 57
7, 5 7 8
82 3
3, 24 5
6, 5 51
7, 52 4
7, 5 73
E S
1 11
11 0
8, 55 1
6 8 0
2, 8 32
6, 9 1 9
8, 75 2
8, 8 7 1
8 3 3
3, 4 2 6
7, 52 8
8, 8 6 9
8, 9 5 6
8 3 4
3, 4 2 4
7, 52 7
8, 8 7 1
8, 93 9
FR 1 1 3
1 1 8
8, 31 2
7 4 4
2, 8 21
6, 9 2 5
8, 5 26
8, 66 6
82 7
3, 40 7
7, 4 89
8, 66 5
8, 7 0 5
88 3
3, 46 3
7, 4 31
8, 67 9
8, 7 08
R U
1 46
14 4
1 2, 81 9
8 78
3, 7 6 9
10 ,0 8 5
1 2, 5 05
1 2, 7 88
1, 02 4
4, 4 38
11 ,0 5 2
12 ,8 83
1 2, 9 5 4
1, 03 8
4, 5 09
11 ,0 12
1 2, 8 82
1 2, 9 5 8
Z H
1 ,9 7 6
15 3
7, 4 13
3, 22 4
4, 2 15
6, 3 8 6
7, 5 41
7, 65 4
3, 2 6 1
4, 48 1
6, 7 35
7, 71 9
7, 7 0 2
3, 2 6 0
4, 52 4
6, 7 85
7, 65 4
7, 7 21
Z H
_p in
yi n 98 Z H _w ub i 1 20
A R
_c p1
25 6
13 0
R U
_c p1
25 1
14 6
N um
be r
of TO
K E
N S
A R
3 34 ,3 5 8
60 5, 5 1 6
61 ,3 71
1 67 ,5 74
1 15 ,6 9 3
83 ,0 01
76 ,2 8 4
7 0, 5 2 7
1 4 9, 6 89
9 7, 8 4 3
73 ,3 1 4
6 8, 5 3 8
68 ,2 7 0
1 4 9, 62 3
9 7, 2 3 1
7 3, 54 1
6 8, 5 7 9
68 ,2 78
E N
39 1 ,2 2 2
39 1, 2 60
67 ,6 2 9
1 5 5, 82 6
1 0 1, 7 82
7 7, 0 8 9
71 ,2 76
7 0, 3 3 9
1 4 0, 2 56
9 0, 8 7 1
73 ,5 3 1
6 9, 6 3 3
69 ,3 4 8
1 4 0, 3 77
9 0, 3 6 0
73 ,4 6 8
6 9, 6 3 3
69 ,3 41
E S
4 4 3, 9 58
4 5 2, 19 0
7 8, 0 8 7
1 70 ,4 33
1 13 ,9 8 3
88 ,6 34
8 2, 28 1
81 ,3 41
1 55 ,6 8 7
1 03 ,7 48
8 4, 91 4
80 ,5 69
8 0, 3 37
1 54 ,7 4 6
1 03 ,1 72
8 4, 8 7 0
80 ,5 79
8 0, 37 1
FR 4 3 8, 0 83
4 5 2, 55 6
7 8, 7 4 5
1 66 ,2 80
1 14 ,6 9 4
88 ,7 26
8 2, 44 8
81 ,5 59
1 56 ,2 5 6
1 04 ,0 71
8 5, 11 5
80 ,8 44
80 ,6 71
1 53 ,7 4 5
1 04 ,0 91
8 5, 12 5
80 ,9 12
8 0, 65 4
R U
4 31 ,5 3 8
7 93 ,2 1 4
6 4, 1 80
17 7, 8 18
11 3, 6 2 8
79 ,7 6 3
7 2, 4 8 0
7 1, 08 1
16 3, 3 1 9
10 0, 29 4
7 5, 0 51
7 0, 20 0
6 9, 9 9 1
16 3, 8 0 6
9 9, 97 8
7 5, 0 96
7 0, 19 6
6 9, 9 8 2
Z H
1 07 ,9 90
3 01 ,0 8 5
6 0, 0 13
9 6, 74 5
80 ,2 31
6 8, 12 9
6 3, 6 1 4
6 2, 81 0
93 ,7 7 5
7 5, 63 6
6 5, 2 12
6 1, 86 7
6 1, 83 2
94 ,1 2 7
7 5, 71 8
6 5, 2 03
6 1, 91 6
6 1, 8 2 3
Z H
_p in
yi n
37 6 ,9 7 9
Z H
_w ub
i 33 0 ,7 3 4
A R
_c p1
25 6
3 3 4, 3 58
R U
_c p1
25 1
43 1 ,5 38
T T
R (% ) A
R 0. 04
0 .0 2
22 .5 4
0 .4 2
2 .7 9
10 .8 4
1 5 .0 4
17 .6 2
0 .5 8
4 .0 8
1 4 .6 4
1 8 .9 0
1 9 .1 7
0 .6 1
4 .2 4
1 4 .5 2
1 8 .8 6
19 .1 8
E N
0. 02
0 .0 3
10 .6 4
0 .4 2
2 .5 2
7 .7 8
1 0 .4 7
1 0 .7 4
0 .5 7
3 .4 7
8 .8 6
1 0 .8 5
1 0 .9 3
0 .5 9
3 .5 9
8 .9 2
10 .8 1
1 0 .9 2
E S
0. 0 3
0 .0 2
10 .9 5
0 .4 0
2 .4 8
7 .8 1
10 .6 4
1 0 .9 1
0 .5 4
3 .3 0
8 .8 7
1 1 .0 1
11 .1 5
0 .5 4
3 .3 2
8 .8 7
1 1 .0 1
1 1 .1 2
FR 0. 0 3
0 .0 3
10 .5 6
0 .4 5
2 .4 6
7 .8 0
1 0 .3 4
10 .6 3
0 .5 3
3 .2 7
8 .8 0
10 .7 2
1 0 .7 9
0 .5 7
3 .3 3
8 .7 3
1 0 .7 3
10 .8 0
R U
0. 0 3
0 .0 2
1 9 .9 7
0 .4 9
3 .3 2
1 2 .6 4
1 7 .2 5
1 7 .9 9
0 .6 3
4 .4 2
14 .7 3
18 .3 5
1 8 .5 1
0 .6 3
4 .5 1
14 .6 6
1 8 .3 5
18 .5 2
Z H
1. 83
0 .0 5
1 2 .3 5
3 .3 3
5 .2 5
9 .3 7
1 1 .8 5
1 2 .1 9
3 .4 8
5 .9 2
1 0 .3 3
12 .4 8
1 2 .4 6
3 .4 6
5 .9 7
1 0 .4 1
12 .3 6
1 2 .4 9
Z H
_p in
yi n
0. 03
Z H
_w ub
i 0. 0 4
A R
_c p1
25 6
0. 0 4
R U
_c p1
25 1
0. 0 3
M ea
n lin
e le
ng th ±
st d
0/ 25
/5 0/
75 /1
00 -t
h
A R
1 08 .6 6 ± 5 8 .0 1
19 6 .7 9 ± 1 05 .8 5
19 .9 5 ± 1 0 .5 8
54 .4 6 ± 2 9 .5 1
37 .6 0 ± 20 .5 1
26 .9 7 ± 1 4 .8 0
2 4 .7 9 ± 1 3 .5 9
22 .9 2 ± 12 .5 6
4 8 .6 5 ± 26 .3 5
3 1 .8 0 ± 17 .7 6
2 3 .8 3 ± 1 3 .1 5
22 .2 7 ± 12 .1 2
22 .1 9 ± 1 2 .0 7
4 8 .6 3 ± 26 .2 1
3 1 .6 0 ± 1 7 .5 3
23 .9 0 ± 1 3 .2 0
2 2 .2 9 ± 1 2 .1 4
2 2 .1 9 ± 12 .0 6
3/ 6 0 /1 1 0/ 1 5 3/ 27 7
6 /1 07 /1 99 / 27 7 /5 0 3
1 /1 2 /2 0 /2 7 /5 8
1/ 30 / 5 4/ 7 6/ 1 52
1/ 22 / 37 / 5 2/ 1 2 5
1 /1 6 /2 7 /3 7 /9 5
1 /1 4 /2 4 /3 4 /8 6
1 /1 4 /2 2 /3 2 /7 5
1 /2 7 /4 9 /6 8 /1 5 6
1 /1 8 /3 2 /4 4 /1 10
1/ 14 / 2 4/ 3 3/ 8 0
1/ 1 3/ 22 / 3 1/ 71
1/ 1 3 /2 2 /3 1 /7 1
1 /2 8 /4 9 /6 7 /1 45
1 /1 8 /3 1 /4 4 /1 10
1/ 14 / 2 4/ 33 / 8 7
1/ 1 3/ 22 / 3 1/ 73
1/ 1 3 /2 2 /3 1 /7 2
E N
1 2 7. 1 4 ± 6 8 .6 4
1 27
.1 6 ± 6 8 .6 5
2 1 .9 8 ± 11 .8 1
50 .6 4 ± 27 .4 4
33 .0 8 ± 1 8 .0 7
25 .0 5 ± 1 3 .5 6
2 3 .1 6 ± 1 2. 4 2
2 2 .8 6 ± 1 2 .2 7
4 5 .5 8 ± 24 .8 6
2 9 .5 3 ± 1 6 .3 0
23 .9 0 ± 1 2 .9 5
2 2 .6 3 ± 1 2 .1 3
22 .5 4 ± 1 2. 0 6
4 5 .6 2 ± 24 .6 7
2 9 .3 7 ± 1 6 .1 3
23 .8 8 ± 1 2 .9 1
2 2 .6 3 ± 1 2 .1 2
22 .5 4 ± 12 .0 5
6/ 6 8 /1 3 0/ 1 8 1/ 29 9
6/ 68 /1 30 / 1 81 /2 9 9
1 / 1 3/ 2 2/ 31 / 6 1
1/ 2 7 /5 1 /7 2 /1 3 6
1 /1 9 /3 3 /4 6 /1 10
1/ 1 5/ 2 5 /3 5 /7 8
1/ 1 4/ 23 / 3 2/ 71
1 /1 3 /2 3 /3 2 /7 1
1/ 2 6/ 4 6/ 6 5/ 1 2 9
1/ 1 7 /2 9 /4 1 /9 4
1 /1 4 /2 4 /3 3 /7 4
1 /1 3 /2 3 /3 1 /6 5
1 /1 3 /2 3 /3 1 /6 5
1/ 2 6/ 46 / 6 4/ 12 2
1/ 17 /2 9 /4 1 /9 2
1 /1 4 /2 4 /3 3 /7 3
1 /1 3 /2 3 /3 1 /6 5
1 /1 3 /2 3 /3 1 /6 6
E S
1 4 4. 2 8 ± 77 .7 8
1 4 6 .9 6 ± 7 9 .2 1
25 .3 8 ± 13 .5 6
5 5 .3 9 ± 3 0 .1 3
3 7 .0 4 ± 20 .1 3
2 8 .8 1 ± 1 5. 53
2 6 .7 4 ± 1 4 .2 9
2 6 .4 4 ± 1 4. 14
50 .6 0 ± 27 .6 6
3 3 .7 2 ± 1 8 .4 8
27 .6 0 ± 14 .8 8
2 6 .1 8 ± 1 3 .9 8
2 6 .1 1 ± 1 3. 94
50 .2 9 ± 27 .2 7
3 3 .5 3 ± 1 8. 38
27 .5 8 ± 14 .8 6
2 6 .1 9 ± 1 3 .9 9
2 6 .1 2 ± 13 .9 4
5 /7 7 /1 4 5/ 2 0 7/ 3 00
5/ 78 /1 4 8/ 2 1 1 /3 0 7
1/ 1 5 /2 5 /3 6 /6 3
1 /3 0 /5 6 /7 9 /1 41
1 /2 1 /3 7 /5 2 /1 11
1/ 17 / 29 / 4 1/ 8 5
1/ 1 5 /2 7 /3 8 /7 6
1 / 1 5/ 2 6/ 3 7/ 73
1/ 2 7 /5 1 /7 2 /1 3 3
1 /1 9 /3 4 /4 7 /1 03
1/ 1 5/ 28 / 3 9/ 75
1 /1 5 /2 6 /3 7 /7 1
1 / 15 / 2 6/ 3 7/ 7 0
1/ 28 /5 1 /7 1 /1 3 3
1 /1 9 /3 4 /4 7 /9 5
1/ 1 5/ 28 /3 9 /7 5
1 /1 5 /2 6 /3 7 /7 1
1 / 1 5/ 26 / 3 7/ 71
FR 1 42 .3 7 ± 7 7 .8 1
1 4 7 .0 8 ± 8 0. 30
2 5 .5 9 ± 13 .8 6
5 4 .0 4 ± 2 9. 5 5
37 .2 7 ± 2 0 .6 5
2 8 .8 4 ± 1 5 .7 6
26 .7 9 ± 14 .4 7
2 6 .5 1 ± 1 4 .3 5
50 .7 8 ± 2 7. 79
33 .8 2 ± 18 .7 4
2 7 .6 6 ± 15 .1 1
2 6 .2 7 ± 1 4. 20
2 6 .2 2 ± 1 4 .1 6
49 .9 7 ± 2 7. 2 5
33 .8 3 ± 1 8 .8 0
27 .6 6 ± 15 .0 9
2 6 .3 0 ± 1 4 .2 1
2 6 .2 1 ± 1 4 .1 6
4/ 74 / 14 5/ 20 5 /3 0 0
4/ 7 7/ 1 50 /2 1 2 / 3 1 0
1/ 14 /2 6 /3 6 /6 6
1/ 2 9/ 5 5/ 76 / 13 9
1/ 2 0/ 3 8/ 5 2/ 11 9
1/ 1 6 /2 9 /4 0 /8 9
1 /1 5 /2 7 /3 7 /7 4
1 /1 5 /2 7 /3 7 /7 0
1 /2 7 /5 2 /7 2 /1 3 5
1 /1 8 /3 4 /4 7 /1 0 8
1/ 1 5/ 28 / 3 9/ 8 0
1/ 14 / 2 7/ 37 / 6 9
1 /1 4 /2 7 /3 7 /6 9
1 /2 7 /5 1 /7 0 /1 3 1
1 /1 8 /3 4 /4 7 /1 0 7
1/ 1 5/ 28 / 3 9/ 84
1/ 14 / 2 7/ 37 / 6 9
1 /1 4 /2 7 /3 7 /6 9
R U
1 40 .2 5 ± 76 .2 1
2 57
.7 9 ± 1 41 .5 3
20 .8 6 ± 1 1 .2 5
5 7 .7 9 ± 32 .1 2
36 .9 3 ± 20 .7 3
25 .9 2 ± 1 4 .4 8
23 .5 6 ± 12 .9 5
2 3 .1 0 ± 1 2 .7 1
5 3 .0 8 ± 29 .0 0
3 2 .5 9 ± 1 8 .4 2
24 .3 9 ± 13 .5 9
2 2 .8 1 ± 1 2 .5 0
2 2 .7 5 ± 1 2. 46
5 3 .2 4 ± 29 .1 8
3 2 .4 9 ± 1 8 .4 9
24 .4 1 ± 13 .5 7
2 2 .8 1 ± 1 2 .5 0
2 2 .7 4 ± 12 .4 5
5/ 7 5 /1 4 1/ 20 0 /3 0 0
7/ 1 36 /2 59 /3 70 / 56 9
1/ 1 2/ 21 / 29 / 69
1/ 3 1 /5 8 /8 2 /1 85
1 /2 1 /3 6 /5 2 /1 85
1 /1 5 /2 6 /3 6 /1 28
1 /1 4 /2 3 /3 3 /1 01
1 /1 3 /2 3 /3 2 /9 5
1/ 2 9/ 53 / 7 5/ 18 5
1/ 1 8/ 3 2/ 46 / 1 66
1/ 1 4 /2 4 /3 4 /1 1 2
1 /1 3 /2 3 /3 2 /8 5
1 /1 3 /2 3 /3 2 /9 3
2/ 3 0/ 53 / 7 6/ 18 5
1/ 1 8/ 3 2/ 4 5/ 1 61
1/ 1 4 /2 4 /3 4 /1 0 5
1 /1 3 /2 3 /3 2 /9 0
1 / 1 3/ 2 3 /3 2 /9 2
Z H
3 5. 10
± 1 8 .4 8
9 7 .8 5 ± 52 .1 0
1 9 .5 0 ± 10 .4 2
3 1 .4 4 ± 1 6. 96
26 .0 7 ± 1 4. 1 9
22 .1 4 ± 1 2 .0 2
20 .6 7 ± 11 .1 6
2 0 .4 1 ± 1 1. 05
30 .4 8 ± 1 6 .4 7
24 .5 8 ± 1 3. 42
2 1 .1 9 ± 11 .4 9
2 0 .1 1 ± 1 0 .7 9
2 0 .0 9 ± 1 0 .7 9
3 0 .5 9 ± 1 6. 4 5
24 .6 1 ± 1 3. 49
2 1 .1 9 ± 11 .4 9
2 0 .1 2 ± 1 0. 81
2 0 .0 9 ± 10 .7 9
2 /2 1 /3 5 /4 9 /1 2 5
4/ 5 3/ 9 9/ 1 38 /2 68
1 /1 1 /2 0 /2 7 /6 4
1 /1 8 /3 1 /4 4 /1 0 9
1/ 1 5/ 2 6/ 36 / 93
1/ 13 /2 2 /3 1 /8 0
1 /1 2 /2 1 /2 9 /7 4
1 / 12 / 2 0/ 28 / 7 4
1/ 17 /3 0 /4 3 /1 0 4
1/ 1 4/ 24 /3 4 /9 2
1/ 1 2/ 21 / 2 9/ 71
1 /1 2 /2 0 /2 8 /6 4
1/ 12 / 2 0/ 2 8/ 66
1/ 1 8 /3 0 /4 3 /1 0 1
1 /1 4 /2 4 /3 4 /9 3
1/ 1 2/ 21 / 2 9 /7 6
1 /1 2 /2 0 /2 8 /6 4
1/ 1 2/ 20 / 2 8/ 65
Z H
_p in
yi n
1 2 2. 52
± 65 .5 9 6/ 67 /1 2 5 / 17 3 /3 53
Z H
_w ub
i 1 0 7. 4 9 ± 5 6 .7 3 4/ 6 0/ 1 08 /1 5 1 /2 94
A R
_c p1
25 6
1 08 .6 6 ± 5 8. 01 3/ 6 0/ 1 1 0/ 1 53 / 27 7
R U
_c p1
25 1
1 40 .2 5 ± 7 6. 21 5 /7 5 /1 4 1 /2 0 0 / 3 00

heading: F ENLARGED FIGURES FOR ALL 30 LANGUAGE DIRECTIONS (AGGREGATE RESULTS
text: FROM ALL RUNS)
Fairness in Representation for Multilingual NLP: Insights from Controlled Experiments on Conditional Language Modeling
Version 0.3.2

Review summaries:
This paper explores the performance of conditional language models for different languages with the aim of establishing "fairness" in NLP systems across languages. In particular, the paper focuses on perceived difficulty of "morphologically rich" languages for NLP.  The paper presents a series of experiments with 6 languages in the UN parallel corpus, using different units (words, sub-words, characters, bytes) for training the conditional language models. The paper concludes with the general observation that the use of characters or bytes as units instead of words removes (or reduces) the performance disparity between different languages. 

At its core, this paper focuses on testing whether languages with different morphological complexity equally challenging to language-model in a bilingual / MT-like setting using a 6 layer Transformer architecture.  Specifically, it tests the assumption whether the previous observations about languages with higher morphological complexity are difficult to language-model, using a controlled experiment with a n-way parallel corpus across 6 languages of interest. The key finding of this paper is a simple one : that the choice of input representation is highly important, given everything else is kept constant, and that differences in language-model perplexities can be normalized and made to disappear by choosing the same representation (word/character/byte)

This paper asks: Are languages which have been traditionally considered morphologically rich (Arabic and Russian) and poor (Chinese) equally hard to learn by transformer-based conditional language model (CLM).  To quantify how hard it is for CLM to model a language $l_1$, this paper proposes to train a transformer-based encoder-decoder model to translate $l_1$ to another language $l_2$, and uses the \"unnormalized perplexity score\" (the number of bits needed to encode the same development set for the target language) as an indicator for the \"hardness\" for CLM to learn the $l_2$. For two languages $l_1$ and $l_2$, there exists **disparity** if their perplexity distributions are significantly different. This paper includes extensive experiments across 30 language pairs using three different representation level (including character, byte, and word) for different dataset sizes.  They have the following findings:  1.  They find that the performance disparity between morphologically     rich and poor languages cannot be justified due to morphological     complexity. Representation level matters more.  2.  They find that the hardness for CLM to model a language is related     to the level of representation used to encode the language, the     vocabulary size, and/or the sequence length.  3.  They find that it is equally hard to learn a certain target language     (that are tested in their experiment) using transformer-based     conditional language models, independent of the source language.

In this paper, authors aim to untangle the disparities in performance between morphologically rich languages (e.g. RU) and morphologically poor languages (e.g. ZH). In order to untangle such disparities, the authors propose a highly controlled experiment setup, where they control for a number of potentially confounding factors (e.g. language mappings, data size, text encoding, etc.). After the controlled experiments, the main take-away of the paper is that vocab size seems to be the biggest indicator of how well or poorly a CLM will perform — since there can be several different granularities for the concept of a “word” for morphologically rich languages, and this can explain why modelling morphologcially rich languages can be more difficult.  They perform additional experiments with different representations of ZH, and were able to mimic the struggles of morphologically rich languages with ZH, and use this as further evidence to support the claim that the debate within NLP about the best segmentation is irrelevant.  Overall, this work is exciting, has has a lot of potential to help inform future language modelling for morphologically rich languages.  The very controlled nature of the experiment in the paper is what makes it most convincing, and this is the paper’s largest contribution.