Representation and Bias in Multilingual NLP: Insights from Controlled Experiments on Conditional Language Modeling
Authors: Ada Wan
Venue:  (2021)
Keywords: multilinguality
science for NLP
fundamental science in the era of AI/DL
representation learning for language
conditional language modeling
Transformer
Double Descent
non-monotonicity
fairness
meta evaluation
visualization or interpretation of learned representations

Abstract: Inspired by the phenomenon of performance disparity between languages in machine translation, we investigate whether and to what extent languages are equally hard to "conditional-language-model". Our goal is to improve our understanding and expectation of the relationship between language, data representation, size, and performance. We study one-to-one, bilingual conditional language modeling through a series of systematically controlled experiments with the Transformer and the 6 languages from the United Nations Parallel Corpus. We examine character, byte, and word models in 30 language directions and 5 data sizes, and observe indications suggesting a script bias on the character level, a length bias on the byte level, and a word bias that gives rise to a hierarchy in performance across languages. We also identify two types of sample-wise non-monotonicity --- while word-based representations are prone to exhibit Double Descent, length can induce unstable performance across the size range studied in a novel meta phenomenon which we term "erraticity". By eliminating statistically significant performance disparity on the character and byte levels by normalizing length and vocabulary in the data, we show that, in the context of computing with the Transformer, there is no complexity intrinsic to languages other than that related to their statistical attributes and that performance disparity is not a necessary condition but a byproduct of word segmentation. Our application of statistical comparisons as a fairness measure also serves as a novel rigorous method for the intrinsic evaluation of languages, resolving a decades-long debate on language complexity. While these quantitative biases leading to disparity are mitigable through a shallower network, we find room for a human bias to be reflected upon. We hope our work helps open up new directions in the area of language and computing that would be fairer and more flexible and foster a new transdisciplinary perspective for DL-inspired scientific progress.

Under review as a conference paper at ICLR 2021
REPRESENTATION AND BIAS IN MULTILINGUAL NLP:
INSIGHTS FROM CONTROLLED EXPERIMENTS ON CONDITIONAL LANGUAGE MODELING
Anonymous authors authors Paper under double-blind review
ABSTRACT
Inspired by the phenomenon performance disparity between languages in ma- chine translation, we investigate whether and to what extent languages are equally hard to "conditional- laaanaallnnngge-mooeeeee and expectation of the relationship between language, data representation, size, and performance. We study one-to-one, bilingual conditional language modeling through series of systematically controlled experiments with the Transformer and the 6 languages from the United Nations Parallel Corpus. We examine character, byte, and word models in 30 language directions and data sizes, and observe indications suggesting a script bias on the character level, a length bias on the byte level, and word bias that gives rise to hierarchy in performance across languages. while word-based
Our goal is to improve our understanding
We also identify two types of sample-wise non-monotonicity representations are prone to exhibit Double Descent, length can induce unstable performance across the size range studied in novel meta phenomenon which we term erraticity. By eliminating statistically significant performance disparity on the character and byte levels by normalizing length and vocabulary in the data, we show that, in the context of computing with the Transformer, there is no complexity intrinsic to languages other than that related to their statistical attributes and that performance disparity not a necessary condition but a byproduct word segmen- faimess measure also serves
tation. Our application of statistical comparisons as a novel rigorous method for the intrinsic evaluation of languages, resolving a decades-long debate on language complexity. While all these quantitative biases leading to disparity are mitigable through shallower network, we find room for a human bias to be reflected upon. We hope our work helps open up new directions in the area of language and computing that would be fairer and more flexible and foster a new transdisciplinary perspective for DL-inspired scientific progress.
INTRODUCTION
With transdisciplinary approach to explore a space at the intersection of Deep Learning (DL) / Neural Networks (NNs), language sciences, and language engineering, we report our undertaking in use-inspired basic research with an application-related phenomenon inspiration, we seek fundamental scientific understanding through empirical experimentation. This is not an application machine translation (MT) paper, but one that strives to evaluate and seek new insights language in the context of DL with a consideration to contribute to our evaluation, segmentation, and model interpretation practice in multilingual Natural Language Processing (NLP).
Our inspiration: performance disparity in MT The use case that inspired our investigation is the disparity of MT results reported in Juncyys Baanezyy-Dowmumn Dowmunt et al. (2016). Of the 6 official languages of the United Nations (UN) Arabic (AR), English (EN), Spanish (ES), French (FR), Russian (RU), and Chinese (ZH), results with target languages AR, RU, and ZH seem to be worse than those with EN/ES/FR, regardless of the algorithm, may it be from phrased-based Statistical MT (SMT/Moses

---

Under review as a conference paper at ICLR 2021
(Koehn et al. 2007)) Neural MT (NMT The languages have the same amount of line-alignede high-quality parallel data available for training, evaluation, and testing. This prompts the question: are some languages indeed harder to translate from or to?
Problem statement: are all languages equally hard to Conditional- Languooaa-Laggugg-MMooddde (CLM)?
A similar question concerning (monolingual) language modeling (LMing) was posed in Cotterell et al. (2018) and Mielke al. (219)) along with the introduction of a method to evaluate LMs with multiway parallei corpora (multitexts) in iiformation-theoreeee terms. To explicitly focus on modeling the complexities that may or may not be intrinsic to the languages, we study the more fundamental process of CLMing without performing any translation. This allows us to eliminate confounds associated with generation and other evaluation metrics. One could think of our effort as estimating conditional probabilities with the Transformer, with bilingual setup where perplexity of one target language (Zug) ) is estimated given the parallel data in one source language (ssc), where lac F lug. We focus on the very basics and examine the first step in our pipeline input representation, holding everything else constant. Instead of measuring absolute cross-entropy scores, we evaluate the relative differences between languages from across 5 magnitudes of data sizes in 3 different representation types/levels. We consider bias to be present when performance disparity in our Transformer models is statistically significant.
1.1 SUMMARY OF FINDINGS AND CONTRIBUTIONS
In investigating performance disparity as a function of size and data with respect to language and representation on the Transformer in the context of CLMing, we find:
1. in a bilingual (one-to-one) CLMing setup. there is neutralization of source language instances, i.e. there are no statistically significant differences between source language pairs. Only pairs of target languages differ significantly (see Table
2. We identify 2 types of sample-wise non-monotonicity on each of the primary representation levels we studied:
(a) Double Descent (Belkin et al. 2019: Nakkiran et al., 2020): on the word level, for all languages, performance at 10a lines is typically better than 10 before it improves again at 10 and beyond. This phenomenon can also be observed in character models with ZH target language well on the word level with non-neural n-gram LMs;
(b) erraticit: performance is irregular and exhibits great variance across runs. We find sequence length to be predictive of this phenomenon. We show that this be rectified by data transformation or hyperparameter tuning. In our study, erraticity affects AR and RU on the byte level where the sequences are too long with UTF-8 encoding and ZH when decomposed into strokes on the character level.
3. In eliminating performance disparity through lossless data transformation on the character and byte levels, we resolve language complexity ($ and App. D]. We show that, in the context of computing with the Transformer, unless word-based methods are used, there is no linguitic/morpphlooicca complexity applicable or necessary. There is no complexity that is intrinsic to a language aside from its statistical properties. Hardness in modeling is relative to and bounded by its representation level (representation relativity). On the character and byte levels, hardness is correlated with statistical properties concerning sequence length and vocabulary of language, irrespective of its linguistic typological, phylogenetic, historical, or geographical profile, and can be eliminated. On the word level, hardness is correlated with vocabulary, and complexity hierarchy arises through the manual preprocessing step of word tokenization. This complexity/disparity effected by word segmentation cannot be eliminated due the fundamental qualitative differences in the definition of "word" being one that neither holds universally nor is suitable/consistent for fair crosslinguistic comparisons. We find clarification of this expectation of disparity necessary because more diligent error analyses need to be afforded instead of simply accepting massively disparate results or inappropriately attributing penfor-pefoormacc to linguistic reasons.
4. Representational units of finer granularity can help close the gap in performance disparity.
5. Bigger/overprrameterizee models can magnify/exacerrate the effects of differences in data statistics. Quantitative biases that lead to disparity are mitigable through numerical methods.
We provide re-visualization of these grouped in 6 facets by target language in Figure 4 in Appendix A
2

---

Under review as a conference paper at ICLR 2021
Outline of the paper In define method and experimental setup. We present our results and analyses on the primary representations in $ and those from secondary set of controls in $ $ in progressive manner to ease understanding, Meta analyses on faimess evaluation, non-monotonic behavior, and discussion on biases are in $5 Additional related work is in $ We refer our readers to the Appendices for more detailed descriptiins/diicussion and reports on supplementary experiments.
2 METHOD AND DEFINITIONS
Controlled experiments basic research for scientific understanding Using the United Na- tions Parallel Corpus (Ziemski et al. 2016), the data from which the MT results in In/Jnccyy-Dowmmn
al. (016)) stem, we perform a series of controlled experiments on the Transformer, holding the hyperparameter settings for all 30 one-to-one language directions from the 6 languages constant. We control for size (from 102 to 10 lines) and language with respect to representational granularity. We examine 3 primary representation types- character, byte (UTF-8), and word, and upon encountering some unusual phenomena, we perform a secondary set of controls with alternate representations - on the character level: Pinyin and Wubi (ASCII representations for ZH phones and character strokes, respectively), ) on the byte level: code page 1256 (for AR) and code page 1251 (for RU), and on the word level: Byte Pair Encoding (BPE) (Sennrich al. 2016), adapted compression algorithm from Gage (1994). These symbolic variants allow us to manipulate the statistical properties of the representations, while staying "faithful" to the language as possible. We adopt this symbolic data-centric approach because we would like to more directly interpret the confounds, if any, that make language data different from other data types. We operate on smaller data size range as this is more common in traditional domain sciences and one of our higher goals is to bridge an understanding between language sciences and engineering (the latter being the dominant focus in NLP). We run statistical tests to identify the strongest correlates of performance and to assess whether the differences between the mean performance of different groups are indeed significant. We are concerned not with the absolute scores, but with the relations between scores from different languages and the generalizations derived therefrom.
Information- fair evaluation with multitexts Most sequence-to-sequenee models are optimized using cross-entropy loss (see Appendix [B for definition). [Cotterell et al. (2018) propose to use "renormalized" perplexity (PP) to evaluate LMs fairly using the total number of bits divided by some constant. In our case, we choose instead a simpler method using an "unnormalized" PP, directly using the total number of bits needed to encode the development (dev) set, which has constant size of 3,077 lines per language.
Disparity/Inequality In the context of our CLMing experiments, we consider there to be "dis- parity" or "inequality" between languages I and l2 if there are significant differences between the performance distributions of these two languages with respect to each representation. Here, by performance we mean the number of bits required to encode the held-out data using a trained CLM. With 30 directions, there are 15 pairs of source languages (lur11, 4are2) and 15 pairs of target languages (ugg, 4eg2) possible. To assess whether the differences are significant, we perform unpaired two-sided significance tests with the null hypothesis that the score distributions for the two languages not different. Upon testing for normality with the Shapiro- Wilk test (Shapiro & Wilk, 1965 Royston 1995). we use the parametric unpaired two-sample Welch's t-test (Welch) 1947) (when normal) the non-parametric unpaired Wilcoxon test (Wilcoxon] 1945) (when not normal) for the comparisons. We use the implementation in R (R Core Team 2014) for these 3 tests. To account for the multiple comparisons we are performing, we correct all pl p-values using Bonferroni's 's correction (Benjamini & Heller 2008 Dror al. 2017) and follow Holm's 's procedure (Holm 1979 Dror et al. 2017) to identify the pairs of 1 and l2 with significant differences after correction. We report all 3 levels of significance (a < 0.05, 0.01, 0.001) for a more comprehensive evaluation.
Experimental setup with further preprocessing and hyperparameter details in Appendices B and C respectively. The distinctive point of our experiment is that the training regime is the same for all (intuition in App. 0.1
The systematic, identical treatment we give to our data is described as follows
using implementation fttptt/p///ggthhoooooo///rrrrrrp
NI.
3

---

Under review as a conference paper at ICLR 2021
After filtering length to 300 characters maximum per line in parallel for the 6 languages, made 3 subsets of the data with million lines each- - one having lines in the order of the original corpus (dataset A) and two other randomly sampled (without replacement) from the full corpus (datasets B & C). Lines in all datasets are extracted in parallel and remain fully aligned for the 6 languages. For each run and each representation, there are 30 pairwise directions (i.e. one 4 to one lug) that result from the 6 languages. We trained all 150 (for 5 sizes) 6-layer Transformer models for each run using the SOCKEYE Toolkit (Hieber al. 2018). We optimize using PP and use early stopping no PP improvement occurs after 3 checkpoints up to 50 epochs maximum, taking the best checkpoint. Characters and bytes are supposed to mitigate the out-of-vocabulary (OOV) problem on the word level. In order to assess the effect of modeling with finer granularity more precisely, all vocabulary items appearing once in the train set are accounted for (i.e. full vocabulary on train, as Gerz et al. (2018a b)). But we allow our system to categorize all unknown items in the dev set to be unknown (UNK) so to measure OOVs (open vocabulary on dev (Jurafsky & Martin 2009)) ). To identify correlates of performance, we perform Spearman's ' correlation (Spearman 1904) with some basic statistical properties of the data (e.g. length, vocabulary size (V), type-token-rant rate) as metrics- a complete list thereof is provided in Appendix [F] For each of the 3 primary representations. - character, byte, and word, we performed 5 runs total in 5 sizes (10--106 lines) (runs AO, BO, CO, AI, & A2) and 7 more runs in 4 sizes (10--10s lines) (A3-7, B1, & CI), also controlling for seeds. For the alternate/secondaryy representations, we ran 3 runs each in 5 sizes (102-108 lines) (AO, BO, & CO).
OOV
EXPERIMENTAL RESULTS OF PRIMARY REPRESENTATIONS
Subfigures la and present the mean results across 12 runs of the 3 primary representations character, byte, and word, respectively. The x-axis represents data size in number of lines and y-axis the total conditional cross-entropy, measured in bits (Eq. [ in Appendix (B). Each line connects data points corresponding to the number of bits the CLMs (traied with training data of 10², 10 10*, 10², and 10 lines) need to encode the target language dev set given the corresponding text in the source language. These the same data in the same 30 language directions and 5 sizes with the same training regime, just preprocessed/segmeett languages (or any objects being modeled) need to be evaluated relative to their representation. "One size does not fit all" (Durrani et al. 2019), our conventional way of referring to "language" (as a socio-cultural product or with traditional wose-based approaches, or even for most multilingual tasks and competitions) is too coarse-grained grained (see also Fisch et al. (2019) and Ponti et al. (2020)).
differently. This confirms representation relativity
Subfigures ld Te and Tf display the corresponding information sorted into facets by target language, source languages represented as line types. Through these we see more clearly that results can be grouped rather neatly by target language (cf. figures sorted by source language in Appendix (H] as implicit in the Transformer's architecture, the decoder is unaware of the source language in the
encoder. As shown in Table T in $ summarizing the number of source and target language pairs with significant differences, there are no significant differences across any source language pairs. The Transformer neutralizes source language instances. This could explain why transfer learning or muliilingual/zero-hhhot translation (Johnson et al. 2017) is possible all on a conceptual level.
In general, for character and byte models, most language directions do seem to converge at 10 lines to similar values across all target languages, with few notable exceptions. There some fluctuations past 10a, indicating further tuning of hyperparameters would be beneficial due to our present setting possibly working most favorably at 10 On the character level, target language ZH (ZHerg) shows a different learning pattern throughout. And on the byte level, ARtry and RU display non-monotonic and unstable behavior. which we refer to as erratic. Word models exhibit Double Descent across the board (note the spike at 10²), but overall, difficult/easy languages stay consistent, with AR and RU being the hardest, followed by ES and FR, then EN and ZH. practical takeaway from this set of experiments: in order to obtain more robust training results, use bytes for ZH (as suggested in Li et al. (2019h)) and characters for AR and RU (e.g. Lee et al. (2017)) - also if one wanted to avoid any "class" problems in performance disparity with words. Performance disparity for these representations is reported in Table I under "CHAR", "BYTE", and "WORD". Do note, however, that the intrinsic performance of ZH with word segmentation is not particularly subpar. But this often does not correlate with its poorer downstream tasks results (recall results from Junezys-Dowmunt et al. (2016)). Since the notion of word in ZH is highly contested and
4

---

Under review as a conference paper at ICLR 2021
aa) CHAR
(b) BYTE
WORD
SRC
TRG
AR
EN
EN
ES
ES
FR
FR
ZH
(h)
(d) CHAR by target
(e) BYTE by target
(I) WORD by target
Figure 1: Number of hits (the lower the better) as function of data size plotted for all 30 directions. Subfigures ld Tc and TT depict the corresponding information 12 runs), respectively, but sorted in facets by target language and with error hars. Legend in Subfigure [g shows the correspondence between colors and source languages, in Subfigure Ih between line types and target languages. (These figures are also shown enlarged in Appendix
1a Tb and Tc (showing mean
ambiguous - 1) it is often aimed to align with that in other languages so to accommodate manual feature engineering and academic theories, 2) there is great variation among different conventions, 3) native ZH speakers identify characters as words - there are reasons to rethink this procedure now processing finer granularity is possible (cf. Li al, (2019b)
that fairer and manguage-indeeendeee as well as Duanmu (2017) for summary on the contested nature of wordhood in ZH). A more native analysis of ZH, despite being considered a high-resource language, has not yet been recognized in NLP.
UNDERSTANDING THE PHENOMENA WITH ALTERNATE REPRESENTATIONS
To understand why some languages show different results than others, we carried out secondary set of control experiments with representations targeting the problematic statistical properties of the corresponding target languages. (An extended version of this section is provided in Appendix
Character level We reduced the high | ZH with representations in ASCII characters - Pinyin and Wubi. The former is romanization of ZH characters based their pronunciations and the latter an input algorithm that decomposes character internal information into stroke shape and ordering and matches these to classes of radicals (Lunde 2008). We replaced the ZH data in these formats only the target side and reran the experiments involving ZH.rg on the character level. Results Figure 2 and Table [ show that the elimination of disparity character level is possible if ZH is represented
5

---

Under review as a conference paper at ICLR 2021
(a) Wubi
(b) Wubi by target
(c) Pinyin
(d) Pinyin by target
Figure 2: Character-level remedies for ZH: Wubi vs. Pinyin.
(a) Code page 1256 & 1251 (b) Code page by target
(c) BPE
(d) BPE by target
Figure 3: Byte-level (Subfigures 3a 3b) remedies with code page 1256 for target AR and 1251 for
target RU, and word-level (Subfigure) Sc & 3d) remedy with BPE for all languages.
through Pinyin (transliteration), as in Subfigure 2c] But models with ZH logographic scripts display
behaviorial tendency unlike those with other (phonetic) alphabetic scripts (Subfigure 22). Work published thus far using Wubi with the Transformer seems to have needed some form of architectura
modification (Gao al., 2020) different architecture altogether (Nikolov al. 2018 Zhang et al. 2019), suggesting a possible script bias (to be further discussed in Undder "Basis for blases")
Byte level Length the most salient statistical attribute that makes AR and RU outliers. To shorten their sequence lengths, tested with alternate encodings ARtry and RUtry code page 1256 and 1251, which provide 1-byte encodings specific to AR and RU, respectively. Results are shown in Subfigures 3a Not only erraticity resolved, the number of 15 possible target language pairs with significant differences reduces from 8 with the UTF-8 byte representation to 0 (Table under "ARRU,"), ") indicating that we eliminated disparity with this optimization heuristic. Since our heuristic is a lossless and reversible transform, it shows that a complexity that is intrinsic and necessary in language") does not exist in computing, however diverse they may be, as our are, from the conventional linguistic typological, phylogenetic, historical, geographical perspectives. Please refer Appendix D for our discussion on language complexity.
Word level The main difference between word and character/byte models is length being a top contributing factor correlating with performance, but instead [V] is. This is understandable as word segmentation neutralizes sequence lengths. To remedy the OOV problem, a fixed vocabulary of variable-length character sequences (on word level, as it presupposes word
BPE, which leams
aside from its statistical properties related to length and vocabulary. "Language" here refers to language represented through all representations.

---

Under review as a conference paper at ICLR 2021
Table I: Number of language pairs out 15 with significant differences, with respective p-values. ARRU, refers to AR & RU being optimized only on the target side; whereas ARRU.,, denotes optimization on both source and target sides (relevant for directions AR RU and RU-AR).
CHAR
Pinyin
Wubi
BYTE
ARRU,
ARRU,
WORD
BPE
p-value
SIC
tg
SEC
Lgg
SEC
ug
SEC
trg
SIC
ug
SIC
ugg
STC
trg
SIC
trg
0.05
0
7
0
4
0
8
0
9
0
4
0
4
0
11
0
10
0.01
0
5
0
0
6
0
8
0
3
0
4
8
0
8
0.001
0
3
0
0
5
0
8
0
0
0
2
0
8
0
7
segmentation) from the training data. more fine-grained than word segmentation and is known for its capability to model subword units for morphologically complex languages (e.g. AR and RU). We use the same vocabulary of 30,000 as specified in Innunzyys-ovwe Dowmunt al. (2016). This reduced our averaged OOV token rate by 89-100% across the 5 sizes. The number of language pairs with significant differences reduced to 7 from for word models, showing how finer-grained modeling has positive effect on closing the disparity gap.
META-RESULTS, ANALYSIS, AND DISCUSSION
Performance disparity Table [ lists the number of language pairs with significant differences under the representations studied. Considering how it is possible for our character and byte models to effect no performance disparity for the same languages on the same data, this indicates that disparity is not necessary condition. In fact, the customary expectation that languages ought to perform differently stems from our word segmentation practice. Furthermore, the order of AR/RU ES/FR EN/ZH (Figure [Tc) resembles the idea of morphological complexity. Considering there are Character-interaa meaningful units in languages with logographic script such as ZH (cf. Zhang Komachi (2018) that are rarely captured, studied, or referred as "morphemes", this goes to show that linguistic morphology, along with its complexity, as is practiced today" and that which has occurred in the NLP discourse thus far, has only been relevant on and bounded the "word" level. The definition of word, however, has been recognized as problematic for very long time in the language sciences (see Haspelmath (201T) and references therein from the past century). Since the conventional notion of word, which has been centered on English and languages with alphabetic scripts, has a negative impact on languages both morphologically rich (see Minkov et al. (2007). Seddah et al. (2010), inter alia), AR and RU in case, as well morphologically "frugal" " (Kochn) 2005), as in ZH, finer- finer-grained grained modeling with characters and bytes (or n-gram variants/pieces thereof) indeed more sensible option and enables greater variety of languages be handled with more simplicity, fairness, independence, and flexibility.
While the lack of significant differences between pairs of source languages would signify neutraliza- tion of source language instances, it does not mean that source languages have no effect on target. For our byte solutions with code pages, we experimented also with source side optimization in the directions that involve AR/RU as source. This affected the distribution of the disparity results for that representation with pairs being significantly different (see Table T Junder "ARRU,,,"" ""). We defer further investigation on the nature of source language neutralization to future work.
Sample-wise Double Descent (DD) Sample-wise non-monotonicity/DDD (Nakkiran et al. 2020 denotes a degradation followed by an improvement in performance with increasing data size. We notice word models and character models with ZHarg, i.c. models with high target [V], are prone to exhibit spike at 10?. A common pattern for these is the ratio of target training token count to number of parameters falls into 0(10-4) for 102 lines, O(10-3 3) at 103, O(10-3 3) at 10*, and 0(10-1) for 10S lines and so on. But for more atomic units such as alphabetic (not logographic) characters (may be Latin, Cyrillic, or Abjad) and for bytes, this progression instead begins at 0(10-3) at _ lines. Instead of thinking this spike of 10's as irregular, may instead want to
But there are no reasons why linguistics or linguistic typology cannot encompass a statistical science of language beyond/withowd "words", or with continuous representations of characters and hytes. In fact, that could complement the needs of language engineering and the NNs/DL/MM. communities hetter.
7

---

Under review as a conference paper at ICLR 2021
think of this learning curve as shifted by 1 order of magnitude to the right for characters and bytes and/or the performance at 102 lines for words and ZH- 3H-characters due to being overparameterized and hence abnormal. This would also fit in with the findings by [Belkin et al. (2019) and Nakkiran et al. (2020) attributing DD to overparameterization. If we could use this ratio and logic of higher [V to automatically detect "non-atomic" units, ones that can be further decomposed, this observation could potentially be beneficial for advancing other sciences, c.g. biology. From cognitive modeling perspective, the similarity in behavior of ZH characters and words of other languages can affirm the interpretation of wordhood for those ZH speakers who identify ZH characters as words (see also last paragraph in $ and Appendix D.. While almost all work attribute DD to algorithmic reasons, concurrent work by Chen et al. (2000) corroborates our observation and confirms that DD arises due to "the interaction between the properties of the data and the inductive biases of learning algorithms". Other related work on DD and its more recent development can also be found in their work.
We performed additional experiments testing our setting on the datasets used by the Nakkiran et al.] (2020) and testing our data non-neural LMM. Results support our findings and are provided in Appendix K Number of model parameters can be found in Appendix
Erraticity We observe another type of sample-wise non-monotonicity, one that signals irregular and unstable performance across data sizes and runs. Within one run, erraticity can be observed directly as changes in direction on the y-axis. Across runs, large variance can be observed, even with the same dataset (see Figure 18 in Appendix MM. Erraticity through a negative correlation between data size and performance. Many work on length bias in NMT have focused on solutions related to search, e.g. Muuray & Chiang (2018). Our experiments show that a kind of length bias can surface already with CLMing, without generation taking place. If the connection between erraticity and length bias can indeed be drawn, it could strengthen the case for global conditioning (Sountsov & Sarawagi 2016). (See Appendix M for more discussion and results.)
also be observed indirectly
Script bias, erraticity, word bias are these necessary conditions? To assess whether the observed phenomena are particular to this one setting, we performed one run with dataset A in 4 sizes with the primary representations on 1-layer Transformers (see Appendix N We observed no significant disparity across the board. It shows that larger/overparameteiiee models can magnify/exaccrbate the differences the data statistics. That hyperparameter tuning - this case, through the reduction of the number of layers can mitigate effects from data statistics is, to the best of our knowledge, a novel insight, suggesting also that a general expectation of monotonic development as data size increases can indeed be held. Our other findings remain consistent (representational relativity, source language neutralization, and DD on word level).
Bases for biases Recall in $ [ we "consider bias to be present when performance disparity in our Transformer models is statistically significant". As shown in our data statistics and analysis (Appendices) DJand rrespeccieey) script bias, length bias wrt erraticity in CLMing, and word bias all evident the vocabulary and length information in the data statistics. Hence these disparities in performance are really result of the Transformer being able to model these differences in data at such a magnitude that the differences are statistically significant. The meta phenomenon of erraticity, however, warrants an additional consideration indicative of the empirical limits of our compute (cf. Xu et 2020)) even when the non- monotonicity is not observed during the training of each model.
In eliminating performance disparity in character and byte models by normalizing vocabulary and length statistics in the data, we demonstrated that performance disparity as expected from the morphological complexity hierarchy is due to word tokenization, not intrinsic or necessary in language. This is the word bias. Qualitative issues the concept of word will persist and make crosslinguistic comparison involving "words" unfair even if one were to be able to find quantitative solution to mitigate the OOV issue, the bottleneck in word-based processing. We humans have a choice in how see/process languages. That some might still prefer to continue with a crosslinguistic comparison with "words" and exert the superiority of "word" tokenization speaks for a view that is centered on "privileged" languages in that case, word bias is human bias.
And, in eliminating performance disparity across the board with our one-layer models, we show that all quantitative differences in data statisties between languages can also be modeled in "zoomed-
8

---

Under review as a conference paper at ICLR 2021
out"/"desensitizee" mode, suggesting that while languages can be perceived as being fundamentally different in different ways in different granularities, they can also be viewed as fundamentally similar.
6 ADDITIONAL RELATED WORK
Similar to our work in testing for hardness are Cotterell et al. (2018), Mielke et al. (2019), and Bugliarello et al. (2020). The first two studied (monolingual) LMs the former tested on the Europarl languages (Koehn 2005) with n-gram and character models and concluded that morphological complexity was the culprit to hardness, the latter studied 62 languages of the Bible corpus (Mayer & Cysouw 2014) in addition and refuted the relevance of linguistic features in hardness based on character and BPE models on both corpora in word-tokenized form. [Bgliarello et al. (2020) compared translation results of the Europarl languages with BPEs one data size and concluded that it is easier to translate out of EN than into it, statistical significance was, however, not assessed. In contrast, we ablated away the confound of generation and studied CLMing with controls with a broader range of languages with more diverse statistical profiles in 3 granularities and up to 5 orders of magnitude in data size. That basic data statistics are the driver of success in performance in multilingual modeling has so far only been explicitly argued for in Mielke et al. (2019). We go beyond their work in monolingual LMs to study CLMs and evaluate also in relation to data size, representational granularity, and quantitative and qualitative fairness.
Bender (2009) advocated the relevance of linguistic typology for the design of language-independnnt NLP systems based on crosslinguistic differences in word-based structural notions, such as parts of speech. Ponti et al. (2019) found typological information to be beneficial in the few-shot setting on the character level for 77 languages with Latin scripts. But no multilingual work has thus far explicitly examined the relation between linguistic typology and the statistical properties of the data, involving languages with diverse statistical profiles in different granularities.
As obtaining training data is often the most difficult part of an NLP or Machine Learning (ML) project, Johnson et al. (2018) introduced extrapolation methodology to directly model the relation between data size and performance. Our work can he viewed as one preliminary step towards this goal. To the best of our knowledge, there has been no prior work on demonstrating the neutralization of source language instances through statistical comparisons, a numerical analysis on DD for sequence-to- sequence models, the meta phenomenon of sample-wise wise monooooonnyyyty (erraticity) being related to length, or the connection between effects of data statistics and modification in architectural depth.
CONCLUSION
Summary We performed novel, rigorous relational assessment of performance disparity across different languages, representations, and data sizes in CLMing with the Transformer. Different dispar- ity patterns were observed on different representation types (character, byte, and word), which can be traced back to the data statistics. The disparity pattern reflected on the word level corresponds to the morphological complexity hierarchy, reminding us that the definition of morphology is predicated on the notion of word and indicating how morphological complexity can be modeled by the Transformer simply through word segmentation. As we were able to eliminate disparity on the same data on the character and byte levels by normalizing length and vocabulary, we showed that morphological complexity is not a necessary concept but one that results from word segmentation and is bounded to the word level, orthogonal to the performance of character or byte models. Representational units of finer granularity were shown to help eliminate performance disparity though at the cost of longer sequence length, which can have negative impact on robustness. In addition, we found all word models and character models with ZHegg to behave similarly in their being prone to exhibit a peak (as sample-wise DD) around 1 lines in our setting. While biggerooerparammttriied models can magnify the effect of data statistics, exacerbating the disparity, we found decrease in model depth can eliminate these quantitative biases, leaving only the qualitative aspect of "word" and the necessity of word segmentation in question.
Outlook Machine learning has enabled greater diversity in NLP (Joshi et al. 2020). Fairness, in the elimination of disparity, does not require big data. This paper made a pioneering attempt to bridge research in DI/NNs, language sciences, and language engineering through data-centric perspective.

---

Under review as a conference paper at ICLR 2021
We believe statistical science for NLP as data science well complement algorithmic analyses with an empirical view contributing to a more generalizable pool of knowledge for NNs/DL/ML. A more comprehensive study not only can lead to scientific frontiers, but also better design and evaluation, benefitting the development of a more general, diverse and inclusive Artificial Intelligence.
REFERENCES
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32): 1584))):519911111
2019. ISSN 8027-8424. doi:
1903070116 URL
https: www. pwas. org/ wwwtentttnnttteeeeeeeeeeeett
E. M. Bender. Linguistic Fundamentals for Natural Language Processing: 100 Essentials from Morphology and Syntax. 2013.
Emily M. Bender. Linguistically naive != language independent: Why NLP needs linguistic ty- pology. In Proceedings of the EACL 2009 Workshop the Interaction between Linguistics and Computational Linguistics: Virtuous, Vicious or Vacuous?, pp. 26-32, Athens, Greece, March 2009, Association for Computational Linguistics. URLhttse www. aclweb.org.
ant anthology/W009- www.ooyyyW00---0 0106
Yoav Benjamini and Ruth Heller. Screening for partial conjunction hypothess.. Biometrics, 64(4): 1215-1222, 2008. ISSN 0006341X, 15410420. URL Inttp://wwy jst.rrsor/sttteeoe..e.. 25502204
Christian Bentz, Tatyana Ruzsics, Alexander Koplenig, and Tanja Samardžić. A comparison between morphological complexity measures: Typological data vs. language corpora. In Proceedings of the Workshop Computational Linguistics for Linguistic Complexity (CL4LC), pp. 142- 153, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee. URL https
Emanuele Bugliarello, Sabrina J. Mielke, Antonios Anastasopoulos, Ryan Cotterell, and Naoaki Okazaki. It's s easier to translate out of English than into it: Measuring neural translation difficulty by cross-mutual information. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. (640-1649, Online, July 2020. Association for Computa- tional Linguistics. doi: 10.18653/v1/2020..e manee.mminn.ni... 149, URL Ctthttps: /wwww aww.wb..wbbb.... org/
anthology/2020.0 acl main. 149
Mauro Cettolo, Christian Girardi, and Marcello Federico. Wit Web inventory of transcribed and translated talks. In Proceedings of the 16th Conference of the European Association for Machine Translation (EAMT), pp. 261-268 Trento, Italy, May 2012.
Lin Chen, Yifei Min, Mikhail Belkin, and Amin Karbasi. Multiple descent: Design your own generalization curve, 2020.
Stanley F. Chen and Joshua Goodman. An empirical study of smoothing techniques for language modeling. Computer Speech Language, 13(4):359 394, 1999. ISSN 0885-2308. doi: https:
1tbbi.og/10..0006/ssaa..lo wwwp:////csaaa..99999.9 1999.0128. URL http:////// / http:///wwwww... http://wwwwinccccc rect. www/sccinneeeccccoo
articl /
Tianqi Chen, Mu Li, Yurian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv:1512.01271 2015.
Colin Cherry, George Foster, Ankur Bapna, Orhan Firat, and Wolfgang Macherey. Revisiting cearacter-based neural machine translation with capacity and compression. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4295-4305. Association for Computational Linguistics, 2018. URL (tLhtttp. awlweb. org/ anthology/
D1 1461
10

---

Under review as a conference paper at ICLR 2021
Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: recoder-decodee approaches. In Proceedings of SSST-8, Eighth Workshop Syntax, Semantics and Structure in Statistical Translation, pp. 103-111, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.31 14--4012 URL
https: www. ac web www..or..eaae....e.. www.oorgggntoooggggeooooooo 4044
Ryan Cotterell, Sabrina J. Mielke, Jason Eisner, and Brian Roark. Are all languages equally hard to language-model? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Short Papers), pp.
New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 11111865/11/118-0555 URL http/hhtss://tc N1 2085
wwwweb..w........... org/ anthology/
Rotem Dror, Gili Baumer. Marina Bogomolov, and Roi Reichart. Replicability analysis for natural language processing: Testing significance with multiple datasets. Transactions of the Association for Computational Linguistics, 5:471 486, 2017.
//cc wwtp:/ewwboooo..eeoooooo.oo wrt.ooogg/googo/o/.oooooooooo
Q17-1033
San Duanmu. Word and wordhood, modern. In Rint Sybesma (ed.), Encyclopedia of Chinese Language and Linguistics, pp. 543-549. Brill, 2017.
Nadir Durrani, Fahim Dalvi, Hassan Sajjad, Yonatan Belinkov, and Preslav Nakov. One size does not fit all: Comparing NMT representations of different granularities. In Proceedings of the 2019 Conference of the North Americun Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1504-1516, Minncapolis, Minnesota. June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19- 1154.
URL https: www www.wwwwwwbb........ wwt.://wwww..re/atttttooooo wwt../aaggnnoog//.....oooo. 1154
Adam Fisch, Jiang Guo, and Regina Barzilay. Working hard or hardly working: Challenges of
integrating typology into neural dependency parsers. In Proceedings of the 2019 Conference on
Empirical Methods in Natural Language Processing and the 9th International Joint Conference
Natural Language Processing DEMNLP-IJCNLAN pp. 5713-5719, Hong Kong, China, November
2019, Association for Computational Linguistics. doi: 10. 1/149-55/1 URL https
Philip Gage. new algorithm for data compression. C Users J. 12(2):23-38, February 1994. ISSN 0898-9788. URL wwwp.aaawwbb.ooe//aatttologoo//o
acm wtt.aaaww.....iiaa.....oo.......
Yingqiang Gao. Nikola I. Nikolov, Yuhuang Hu, and Richard H.R. Hahnloser. Character-level translation with self-attention. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1591-1604, Online, July 2020. Association for Computa- 145,
tional Linguistics. doi: 10. 10.18653/w1/2020.ac anthelogy/2020..m acl main. matainn11 145
org
Martin Gellerstam. Translationese in Swedish novels translated from English. In Lars Wollin and Hans Lindquist (eds.), Translation Studies in Scandinavia, pp. 88-99 38-95. CWK Gleerup, 1986.
Daniela Gerz, Ivan Vulić, Edoardo Ponti, Jason Naradowsky, Roi Reichart, and Anna Korhonen. Language modeling for morphologically rich languages: Character-aware modeling for word-level prediction. Transactions of the Association for Computational Linguistics, 6:451-465, 2018a. doi: 10.1 162/tac// a_00032. URL https:
www. aclweb. org/ wnwhalnthooooeeeeeeooo 1032
Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen. On the relation between linguistic typology and (limitations of) multilingual language modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 316-327, Brussels, Belgium, October-Novembbrr 20183 Association for Computational Linguistics. URL
ht http://wwwwww.oooooooootraaoaata
Martin Haspelmath. The indeterminacy of word segmentation and the nature of morphology and syntax. Folia Linguistica, 201
11

---

Under review as a conference paper at ICLR 2021
Kenneth Heafield. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pp. 187-197, Edinburgh. Scotland, July 2011. As- sociation for Computational Linguistics. URL https: / //ww://wwwww.w/ aclweb org/ anthology. w11-2123
Kenneth Heafield, Ivan Pouzyrevsky. Jonathan H. Clark, and Philipp Kochn. Scalable modified Kneser-Ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 690-696, Sofia, Bulgaria, Co
August 2013. Association for Computational Linguistics. URL http://wwwwwwwww.boooo
anthology//P1--1122
Felix Hieber, Tobias Domhan, Michael Denkowski, David Vilar, Artem Sokolov, Ann Clifton, and Matt Post. The Sockeye neural machine translation toolkit at AMTA 2018. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume Research Papers), pp. 200-207 Association for Machine Translation in the Americas, 2018. URL
web org/ antho homttoooooyy.. oogy / 1-180
Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65-70, 1979. ISSN 03036898, 14679469. URL http::/wwxe http://wwww.tt..ooooooo.otoo.... www.:rsttoooooooooo org/
stable/4615733
Mark Johnson, Peter Anderson, Mark Dras, and Mark Steedman. Predicting accuracy on large datasets from smaller pilot data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguisticy (Volume Short Papers), pp. 450-455, Association for Computational Linguisties, 2018. httLttttp//aalle www.wwwwwwwww.w..eeeooeoeeeeeeee http:///aaaoooooo///-----00000
Melvin Johnson, Mike Schuster, Quoe V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot transla- tion. Transactions of the Association for Computational Linguistics, 5:339-351, 2017. doi:
1116/taal_a_00666 URL //ww: aclweb. org wnwaanonoooooaaa Q17 1024
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6282-6293, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/0020.acl mcl-mann URL
https: http://wwwwwwwwbbcce org/ wtt.oo/aaaaaooog///22ee2020
main. 560,
Marcin Junczys-Dowmun Tomasz Dwojak, and Hicu Hoang. Is neural machine translation ready for deployment? case study on 30 translation directions. In /WSLT 2016, Seattle, October 2016. URL https: www htttp::/wwwwmicrroofftt com coaa://nuu/rree http:///wwsrrrreerrc///////u
neural machine translation hewpiioo--raddddddddeddeeee deploymentll-tt-eeooo case study- 30-\
trans Translation. oicciiioonn/rriiisssss/
Daniel Jurafsky and James H. Martin, Speech and Language Processing: An Introduction to Natural Language Processing. Computational Linguistics, and Speech Recognition. Pearson Prentice Hall, second edition, 2009.
Reinhard Kneser and Hermann Ney. Improved backing-off for m-gram language modeling. In 1995 International Conference on Acoustics, Speech, and Signal Processing, volume 1, pp. 181-184. IEEE, 1995.
Philipp Koehn. Europarl: A Parallel Corpus for Statistical Machine Translation. In Conference Proceedings: the tenth Machine Translation Summit, pp. 86. Phuket, Thailand, 2005. AAMT, AAMT. URL http: /t archive. info/MTS 2005 -oeeoooonnn..e .pdf
Philipp Koehn and Rebecca Knowles. Six challenges for neural machine translation. In Proceedings of the First Workshop Neural Machine Translation, pp. 28 39.39.60 Association for Computational Linguistics, 2017. doi: 10. (0.18653/11/117-32000 URL http: //w/wwbw.bbbb////// org/ hnthology/ W1 3-044404
12

---

Under review as a conference paper at ICLR 2021
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Kallison-Burchh Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pp- 177-180. Association for Computational Linguistics, 2007. BtLhhtp:://g http:///wwweebb..ga/eecec
P07- 2045
Jason Lee, Kyunghyun Cho, and Thomas Hofmann. Fully character-level neural machine translation without explicit segmentation. Transactions of the Association for Computaaional Linguisties, 365-378, December 2017. doi: 10.1162/tacl_ a a_00667. URL BtL.https:/// wwwp://wwwwww//.. www.bb.rrwwwbrrgogo
anthology/Q17- 1026
Bo Li, Yu Zhang, Tara Sainath, Yonghui Wu, and William Chan. Bytes are all you need: End- to-end multilingual speech recognition and synthesis with bytes. In ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5621-5625. IEEE, 2019a.
Xiaoya Li, Yuxian Meng, Xiaofei Sun, Qinghong Han, Arianna Yuan, and Jiwei Li. Is word segmentation necessary for deep learning of Chinese representations? In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 3242-3252, Florence. Italy, July 2019b. Association for Computational Linguistics. doi: 10.18653/11/119-13144 URL
https: / www. www.:w/wwww................ watprr/aaattttttttooaeeo P19-11331111111111
Ken Lunde. CJKV Information Processing. O' O'Reilly Media, Inc., 2nd edition, 2008. ISBN 0596514476, 9780596514471.
Thomas Mayer and Michael Cysouw. Creating massively parallel Bible corpus. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014), pp. 3158 3163, Reykjavik, Iceland, May 2014. European Languages Resources Association (ELRA).
Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. What kind of language is hard to language-model? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4975-4989, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18883/11/19-111
URL https: www. lelweb
org/ htt/aaattoooyyyl///aa/
Einat Minkov, Kristina Toutanova, and Hisami Suzuki. Generating complex morphology for machine
translation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp. 128-135, Prague, Czech Republic, June 2007. Association for Computational Linguisties. URL https:: www. wwwwwb.ww.............. wttp.//aaaoooloooooooooooo/////. 1017
Kenton Murray and David Chiang. Correcting length bias in neural machine translation. In Proceed- ings of the Third Conference Machine Translation: Research Papers, pp. 212-223, Belgium, Brussels, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-66 URL https: www. a web. org www.eb/rgggaaatoolo/oooooo
6322
Preetum Nakkiran. More data can hurt for linear regression: Sample-wise double descent, 2019.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever, Deep double descent: Where bigger models and more data hurt. In International Conference on Learning Representations, 2020. URLhtttpp
http:///pppeeeiieeeee URL//tttss://ppenrvviee nnt:/
Nikola Nikolov, Yuhuang Hu, Mi Xue Tan, and Richard H.R. Hahnloser. Character-level Chinese- English translation through ASCII encoding. In Proceedings of the Third Conference on Machine Translation: Reseurch Pupers, pp. 10-16, Belgium, Brussels, October 2018. Association for
Computational Linguistics.. URL hhttps: /ww. aclweb. org/ http.//////nntoooo//ll---------/ 6302
Franz Josef Och and Hermann Ney. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):1955) 2003. doi: 10. 10.1162/89110000221337777 URL
fittps://www. ac web. www.:wwwwww...oooooooooooooooeo.
13

---

Under review as a conference paper at ICLR 2021
M Opper, W Kinzel, J Kleinz, and R Nehl. On the ability of the optimal perceptron gen- eralise. Journal of Physics A: Mathematical and General, 23(11): 1 -586, jun 1990. doi: 10. 10.10888/030544402233111
URL
2F2342F1182FD11
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACI-HLT 2019: Demonstrations, 2019.
Edoardo Maria Ponti, Ivan Vulić, Ryan Cotterell, Roi Reichart, and Anna Korhonen. Towards zero-shot language modeling. In Proceedings the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language
Processing (EMNLP-JJCNL)), pp. 2900-2910, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/11/D19/ 1288. URL https: //w::////wwwww.. www www.waccllgg aclweb
arg/aanthology/D11 9-
Edoardo Maria Ponti, Helen O' O'Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. Modeling language variation and universals: A survey on typological linguistics for natural language processing. 2020.
Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2014. URL http::tt:////t /wwtt. E-projecct org/
Patrick Royston. Remark as r94: A remark on algorithm as 181: The w-test for normality. Journal of the Royal Statistical Society. Series C (Applied Statisticcs) 4445)5547-551, 1995. ISSN 00359254,
14679876. URL/htpp: www. jstor. www.::/sstoooooosoooo.o wttps/taaleee/...1..2..
Djame Seddah, Sandra Koebler, and Reut Tsarfaty (eds.). Proceedings of the NAACL HLT 2010 First Workshop on Statistical Parsing of Morphologically-PE Languages, Los Angeles, CA, USA,
June 2010. Association for Computational Linguistics. URL/ttps::
Mothooogy/W11--- 1400
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 118655/11//16-1166 URL httpt:://///om www. ac webb orgg/anthology/P16-
S. S. Shapiro and M. B. Wilk. An analysis of variance test for normality (complete samples) 111, 12 1965. ISSN 0006-3444. doi: 10. 1..009/biomee/5..3455591.. URL
Biometrika, /ww/oog.oooooo..oooooooooo.....o
https: /ooi:
3-4.59) 591
Pavel Sountsov and Sunita Sarawagi. Length bias in encoder decoder models and a case for global conditioning. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1516-1525, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16. 1158.
C. Spearman. The proof and measurement of association between two things. The American Journal of Psychology. 15(1):72-101 101, 1904. ISSN 00029556. URL pattp://vee Jstor orq/stable stt/se/lbee/11115
Felix Stahlberg and Bill Byrne. On NMT search errors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 0366,,362,, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/D19--13 URL WtRL:htts::///tt www. a web web. http://wwwwwwwbb.bbrrrt.tttttaa/
1331
Reut Tsarfaty, Djamé Seddah, Yoav Goldberg, Sandra Kuebler, Yannick Versley, Marie Candito, Jennifer Foster, Ines Rehbein, and Lamia Tounsi. Statistical parsing of morphologically rich languages (SPMRL) what, how and whither. In Proceedings of the NAACL HLT 2010 First Workshop Statistical Parsing of Morphologically- USA, June 2010. Association for Computational Linguistics. URL https: www. aclweb org/antho wwaetthooogyyyy/m ogy www.///l00 0-
Languages, pp. 1-12, Los Angeles, CA.
14

---

Under review as a conference paper at ICLR 2021
Reut Tsarfaty, Djamé Seddah, Sandra Kübler, and Joakim Nivre. Parsing morphologically rich languages: Introduction to the special issue. Computational Linguistics, 39(1): 22. 2013. doi: 10.1162/COLI 00133. URL https::/www.w.... www.wwawww......m...
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett (eds.). Advances in Neural Information Processing Systems 30, pp. 5998 6008. Curran Associates, Inc., 2017. URL http: //papers... n ps. wttppppppppppppppccc//ccccc -tt.e//
you need. .
L. Welch. The Generalization of Student's' SStuent''s Problem when Several Different Population Variances are Involved. Biomettrika, (40- 35, 01 1947. ISSN 0006-3444. doi: 10.10933/iomeet34.122
28. /MM
Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80-83, 1945. ISSN 00994987. /tt. http://ttr....ttaaae//a////e/
Yilun Xu, Shengjia Zhao, Jiaming Song. Russell Stewart, and Stefano Ermon. theory of usable infor- mation under computational constraints. In International Conference on Learning Representations, 2020.
Longtu Zhang and Mamoru Komachi. Neural machine translation of logographic language using sub-character level information. In Proceedings of the Third Conference Machine Translation: Research Papers. pp. 17-25, Belgium, Brussels, October 2018. Association for Computational Linguistics. URL https:://// /www. aclweb oc..ebb.rgg//nthoooogy.. /W8 6303
Wei Zhang, Feifei Lin, Xiaodong Wang, Zhenshuang Liang, and Zhen Huang. Subcharacter Chinese- English neural machine translation with Wubi encoding, 2019.
Michat Ziemski, Marcin Junczys-Dowmann and Bruno Pouliquen. The United Nations Parallel Corpus v1.0. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France, may 2016. European Language Resources Association (ELRA). ISBN 778-2-9517408-9.1
15

---

Under review as a conference paper at ICLR 2021
APPENDICES
RE-VISUALIZATION OF FIGURE I JUNCZYS-DOWMUNT ET AL.
(016) IN
FACETS BY TARGET LANGUAGE
17
B DATA SELECTION AND PREPROCESSING DETAILS
17
C HYPERPARAMETER SETTING
SETTING
19
D DATA STATISTICS
20
E SCORE TABLES
24
F CORRELATION STATISTICS
25
G ENLARGED FIGURES FOR ALL 30 LANGUAGE DIRECTIONS (AGGREGATE RESULTS
FROM ALL RUNS)
26
H SAMPLE FIGURES FROM RUN AO, ALSO SORTED SOURCE LANGUAGE FOR CON-
TRAST
42
LANGUAGE PAIRS WITH SIGNIFICANT DIFFERENCES
43
LANGUAGE COMPLEXITY
44
SAMPLE-WISE DOUBLE DESCENT (DD)
46
OUR EXPERIMENTAL FRAMEWORK ON DD DATASETS FROM (NAKKIRAN
ET AL. 2002]
46
TOKEN- TO PARA METER RATIO FOR NON-NOONEE NEURAL PONOLINGUAA LMS
47
L NUMBER
PARAMETERS
50
M ERRATICITY
53
M. ERRATICITY AS LARGE VARIANCE: EVIDENCE FROM DIFFERENT RUNS OF THE
SAME DATA
53
M.2 ADDITIONAL EXPERIMENT WITH LENGTH FILTERING 300 BYTES
53
N EXPERIMENTS WITH ONE-LAYER TRANSFORMER
56
PAQS (PREVIOUSLY ASKED QUESTIONS)
57
ONE SETTING FOR ALL
57
0.2 TRANSI ATIONESE WORD ORDER
57
P UNDERSTANDING THE PHENOMENA WITH ALTERNATE REPRESENTATIONS (EX-
TENDED VERSION)
59
16

---

Under review as a conference paper at ICLR 2021
A RE-VISUALIZATION OF FIGURE IN JUNCZYS-DOWMUNT ET AL. (2016) IN FACETS BY TARGET LANGUAGE
AR
EN ES
SRC
AR
EN
BLEU
ZH
FR
RU
ZH
20
0-
KMT 1.2M Pb-SNT
WMTT 1.2M F.b. SNTT NNT Pb-SAT SYSTEM
Figure 4: Results of the Moses bascline systems (right group in each facet) and neural models (left) with 1.2 million iterations (I iteration corresponds to I mini-batch) for the 30 directions of the 6-way UN corpus, tokenized (ZH segmented), lowercased, and length filtered to 100 BPE tokens.
B DATA SELECTION AND PREPROCESSING DETAILS
The UN Parallel Corpus v1. ZZemmsk et al. 2016) consists of manually translated UN documents from 1990 to 2014 in the official UN languages. Therein is a subcorpus that is fully aligned by line, comprising the 6- way parallel corpus we use. We tried to have as little preprocessing or filtering as necessary to eliminate possible confounds. But as the initial runs of our experiment failed due to insufficient memory than 300 characters in any language in lockstep with one another for all the 6 languages such that the subcorpora would remain parallel, thereby keeping the material of each language semantically equivalent to one another. 8,944.859 lines for each language were retained as our training data which cover up the 75th percentile in line length for all 6 languages. In order monitor the effect of data size, we made subcorpora of each language in 5 sizes by heading the first 102, 10 10 10", 106 line We refer this as dataset A. In addition, to better understand and verify the consistency the phenomena observed, we made supplemental datasets by shuffling the 8,944,859 lines two different times randomly and heading the number of lines in our 5 sizes for each language, again in lockstep with one another (datasets B and C).
single GPU with 12 GB VRAM we filtered out lines with more
*GPUs used for experiments in this paper range from a NVIDIA TITAN RTX (24 GB), NVIDIA GeForce RTX 2080 Ti (1I GB), GTX Titan X (12 GB), GTX 1080 GB). All jobs were run single GPU setting. Some word-level experiments involving ARtry or RUtry at 10 had to be run on a CPU as 24 GB VRAM were not sufficient. Models with higher maximum sequence lengths (e.g. byte models) were trained with 24 GB VRAM. Difference in equipment does not necessarily lead to degradation/mmproeement in scores.
'The terms "line" and "sentence" have denote a sequence that ends with a newline character and "sentence" as one with an ending punctuation. Most parallel corpora, such as ours. are aligned by line, as a line may be part of a sentence or without an ending punctuation (e.g. a header/title). Using standardized unit such as "line" would also be fairer measure to linguac/scriptiones continuae (languages/seripts with no explicit punctuation).
used interchangeably in the NLP literature. We use "line" to
17

---

Under review as a conference paper at ICLR 2021
For character modeling, used dummy symbol denote each whitespace. For byte, turned each UTF----ecodeee character into a byte string in decimal value, such that each token is a number between O and 255, inclusive. For word, the Moses tokenizer (Koehn et al. 2007) as is standard in NMT practice when word tokenization is applied and Jieb:[ segmentation in ZH.
followed (Junzyy-Dowmunt et al. 2016) and used
For Pinyin, we used the implementation from numerical format such that each character/syllable is followed by a single digit indicating its lexical tone in Mandarin. For Wubi, we used the dictionary from the implementation from https:/ github. ccm/ www.aeaeswwaaa wubi
in the
We have implemented all representations such that they would be reversible even when the sequence contains code-mixing.
We used the official dev set as provided in (Ziemski et al. 2016), 3,077 lines per language remained from 4,000 after filtering line length to 300 characters. Data statistics is provided in Appendix Doo reference.
The systematic training regime that we give to our language directions are identical for all. For each primary representation type (character, byte, and word), we performed:
5 runs in 5 sizes (10 - 100): AO (seed=13), BO (13), CO (9948), AI (9948), A2 (265), and 7 more runs in 4 sizes (10" - 109): ) A3 (777), A4 (42), AS (340589), A6 (1000), A7 (83146), B1 (9948), & C1 (13).
For each run and each size, there are 30 pairwise directions (i.e. 1 source language target language, e.g. AR-EN for Arabic to English) that result from the 6 languages. We trained all 150 jobs for each run and representation using the Transformer model (Vaswani al., 2017) as supported by the SOCKEYE Toolkit (Hieber et al., 2018) (version 1.18.85) based on MXNet (Chen et al. 2015). A detailed description of the architecture of the Transformer can be found in (Vaswani et al. 2017]. The same set of hyperparameters applies to all and its values are listed in Appendix
Notes on training time Each run of 30 directions in sizes took approximately 8-12 days for character and byte models. Byte models generally took longer hence training time is positively correlated with length (concurring with observations by Cherry et al.] (2018) as they compared character with BPE models). A maximum length of 300 characters entails maximum length of at least 300 bytes in UTF-8. Each run of word models (30 directions, 5 sizes) took about 6 days (excluding the training of some 7-9 directions out of 30 per run involving ARtrg RUtrg at 10 on word level which took about 12-18 hours each direction to train on a CPU as these required more space and would run out of memory (OOM) on our GPUs otherwise). These figures do not include the additional probing experiments described in $
Evaluation metric defined as:
Most peuuence-to-sequenee models are optimized using a cross-entropy loss,
Htt,s)= - Σ logz p(te ter, s)
where t is the sequence of tokens to be predicted, 4 refers to the ith token in that sequence, is the sequence of tokens conditioned on, and = [t). is customary to report scores as PP, which is M(t,s,, i.e. 2 to the power of the cross-entropy averaged by the number of tokens (based on whichever granularity of unit is used for training) in the data. Cotterell et al. (2018) propose to use "renormalized" PP to evaluate LMs fairly through the division of an arbitrary constant. In our case, we choose instead a simpler method of using an "unnormalized" PP, i.c. the total number of bits needed to encode the development (dev) set, which has a constant size of 3,077 lines per language (after length filtering of the same dev set used in sizes. As the implementation we used (SOCKEYE (Hieber et al. 2018) ) only reports PP, we transform it back to entropy as defined above by noting that Ht, s) log, PP(t(s) N.
Dowmunt et al. (2016) for all various training
https:://i com wwtpc/////////ie//..m
18

---

Under review as a conference paper at ICLR 2021
C HYPERPARAMETER SETTING
encoder transformer,
decoder transformer;
num-layers 6:6;
num-embed 512:512:
tranformor-moodel-sizz 512:
transfommor-attention-nee 8;
farrsormmrr-foo--forrd--------- 2048;
transformerraciivttioo--yyy relu;
huunsdoome--ppaiiiinammmeemmdeie fixed;
transformer-prepprcees pranseosmerrssssppoeesss drn;
drrppoot attention 0.1;
-orpooomre-droouutt act 0.1:
transfomer-dropour
proprrrtt-roooppoosssst prepost 0.1:
batch-size 15;
batch-type sentence;
maax--um-chhcckpiinnnn------
3;
nux--ohs 50;
optimizer adam;
optimized-min metrei perplexity;
optimizer-parann params epsilon: 0.000000001, betal: 0.9, beta2: 0.98;
label-smoothing 0.0;
learning rate-reducc-numee caatllccc-um--ae not-improved 4;
coaminn-rate-reduee-faaee 0.001;
los--normalizaioonyyy valid;
max-seq-len 300 for character, word, and BPE, 672 for all bytes, 688 for Wubi, 680 for Pinyin;
theckpoint-frequeneeye frequency/interval 4000.
(For smaller datasets, the end of 50 epochs is often reached before the first checkpoint. Since SOCKEYE only outputs scores checkpoints, we adjusted checkpoint frequency as follows to get a score outputted by the end of epochs: 1000 100 lines for character byte instances, 400 100 lines for word and 500 for 100 lines BPF, 3450 for 1000 lines for word & BPE. For the very few cases that this default does not suffice due to bucketing of similar length sequences, manually set the checkpoint frequency to the last batch.)
19

---

Under review as a conference paper at ICLR 2021
as Dype solen rato in St, Le the rain between the number of syes and talens in the dai. This is mugh prosy for loxical divenayy in that value of wouldd indic that no type is over soce te9co, and value vay close to soull indicate that very fow
2.00
Statistics for dataset A
658888
Number tspes, Le vochhllyy size (VV) Not tha Sockeye 2015 for II allcuttion additonal oyes: quido cos clo, cunlo.
Number of tokess, This exclude the BOSSBO (end lend-begi weg.eegiiiii weg.iiing--e of scnennne marker added by Sockeye to each Line.
0 22
211
01200005 1000 percentc
daaa
diainct pes acCOLIE for almest of the ditt.
98),
length EOSBOOS mikk''s
D DATA STATISTICS
123
ou of wwcabulary (OOV)
DOV token mie (in 90.1c
20

---

Under review as a conference paper at ICLR 2021
205859
683
88588
5
200500
18
23
Statistics for dataset B
0
B ES
285884
1001 2000
40 38
10100
285289 8.
14
10
50 200200
20
20
21

---

Under review as a conference paper at ICLR 2021
FAR
025553
23
88550
302550
205540
855689
Statistics for dataset C
58005
BREAGR
58
088880
888880
635-07
020003
1126
1200
22043
22

---

Under review as a conference paper at ICLR 2021
As difiooo sut vocalalay is lamned from cach tskkiigg densect and data size. BIEE has distanet day sct Cor cach
1.0
2.0
Statistics for development (dev) set
2100
22
23

---

Under review as a conference paper at ICLR 2021
10.00
26
2010 00 33
500
cums for all sizce monkinng altenstt reprocctaioon (BPE, Plrpia, Wobi, up 1256 and op 1251).
Number of bits to cuoolic the dos dato for cast of the bangongs directions. Shown monrrtts over:
K00
ris for CHAR, BYTE, and WORD from 100 100,000
24
1
350
42
505
SCORE TABLES
20 200
2U
02 50
DRAZ
DVK
ESIC
DVR
E
24

---

Under review as a conference paper at ICLR 2021
F CORRELATION STATISTICS
Best correlating metrics, i.c. the union of top metrics for all representations. For each representation, the top 3 metries boldfaced.
All correlations are highly significant (p 10 10"),,, except for min source length for WORD (p R 0.0001) and min target length for worn (p 0.3861).
Metric
CHAR
Pinyin
Wubi
BYTE
ARRU,
ARRU..
WORD
BPE
minimum length (target)
0.84
0.85
0.86
0.60
0.84
0.84
0.02
0.65
minimum length (source)
0.82
0.84
0.85
0.57
0.84
0.84
0.10
0.64
number tokens (source)
0.78
0.81
0.82
0.60
00.1
-8.0
0.59
0.83
TTR (target)
0.83
0.83
0.84
0.48
0.81
0..81
0.61
0.83
[V] (source)
0.54
0.51
0.51
0..50
0.67
-0.68
0.63
0.86
data size lines
0.80
0.83
0.83
0.59
0.81
-0.81
0.62
0.86
OOV token rate (target)
0.69
0.66
0.66
0.47
0.67
0.68
0.66
0.62
OOV type rate (target)
0.71
0.72
0.47
0.69
0.70
0.65
0.62
TTR (source)
0.67
0.71
0.71
0.60
0.81
0.81
0.56
0.82
The full list of metrics used for the correlation analysis is:
1. minimum length (source), minimum length (target),
3. maximum length (source),
4. maximum length (target),
5. median length (source),
6. median length (target),
7. mean length (source),
8. mean length (target).
9. length std (source),
10. length std (target),
11. data size lines,
12. number of parameters,
13. number of types (V)) (source),
14. number of types (V) (larget),
15. number of tokens (source),
number tokens (larget).
17. type-token-ratie (TTR) (source),
18. pee-token-ratiin (TTR) (target).
19. OOV type rate (source).
OOV type rate (target),
OOV token rate (source),
22. OOV token rate (target),
23. token ratio,
24. target sype-to-parameter ratio.
target token-to-pann tor-ttt--praeterrrrr ratio,
26. distance between the TTRs of source and target (1- TI-TIRRec/UBBB-0
27. token-to-parce parameter ratio (i) = (median length source median longth target * num_lines) nam__arameters.
28. token-to- parameter
toke--to-parameter ratio (ii) (num_sont caati__souee__toee
cain__targt_tokeee num_parameters.
25

---

Under review as a conference paper at ICLR 2021
G
ENLARGED FIGURES FOR ALL 30 LANGUAGE DIRECTIONS (AGGREGATE RESULTS FROM ALL RUNS)
1800000
1500000
TRG
-- ES
number of bits
ZH
SRC
1200000
ES
FR
ZH
900000
1e+03
1e+05
number lines
Figure 5: CHAR: character models
26

---

Under review as a conference paper at ICLR 2021
AR
ES
1500000
1000000
AR
EN
FR
number bits
RU
ZH
SRC
AR
EN
RU
1500000
1000000
1e+03
1e+05
1e+03 number of lines
1e+05
1e+03
1e+05
Figure 5: CHAR: character models (target language facet)
27

---

Under review as a conference paper at ICLR 2021
1750000
TRG
1500000
EN
ES
number of bits
ZH
SRC
1250000
ES
FR
1000000
750000
1e+03
1e+05
number of lines
Figure 6: CHAR with Pinyin for ZH Htng
28

---

Under review as a conference paper at ICLR 2021
AR
ES
2000000
1500000
1000000
AR
EN
FR
number bits
RU
ZH
SRC
2000000
AR
EN
RU
1500000
1000000
1e+03
1e+05
1e+03
1e+05
1e+03
1e+05
number of lines
Figure 6: CHAR with Pinyin for ZHerg (target language facet)
29

---

Under review as a conference paper at ICLR 2021
1750000
TRG
1500000
EN
ES
number of bits
ZH
SRC
11
1250000
ES
FR
1000000
750000
1e+03
1e+05
number of lines
Figure 7: CHAR with Wubi for ZH
30

---

Under review as a conference paper at ICLR 2021
AR
ES
2000000
1500000
1000000
AR
EN
FR
number bits
RU
ZH
SRC
2000000
AR
EN
RU
1500000
1000000
1e+03
1e+05
1e+03
1e+05
1e+03
1e+05
number of lines
Figure 7: CHAR with Wubi for ZHtny (target language facet)
31

---

Under review as a conference paper at ICLR 2021
2500000
2000000
TRG
EN
ES
number of bits
ZH
SRC
1500000
ES
FR
1000000
1e+03
1e+05
number of lines
Figure 8: BYTE models with UTF-8 encoding
32

---

Under review as a conference paper at ICLR 2021
AR
ES
2500000
2000000
1500000
AR
EN
1000000
FR
number bits
RU
ZH
SRC
AR
EN
2500000
RU
2000000
1500000
1000000
1e+03
1e+05
1e+03
1e+05
1e+03
1e+05
number of lines
Figure 8: BYTE models with UTF-8 encoding (target language facet)
33

---

Under review as a conference paper at ICLR 2021
2000000
TRG
EN
1500000
ES
number of bits
ZH
SRC
ES
FR
ZH
1000000
1e+03
1e+05
number of lines
Figure 9: BYTE with ARtng & RUth, optimized with code pages 1256 & 1251 (ARRUtrg)
34

---

Under review as a conference paper at ICLR 2021
ES
2000000
1500000
TRG
1000000
EN
number of bits
RU
2H
FR
RU
ZH
SRC
EN
2000000
ES
ZH
1500000
1000000
1e+03
1e+05
1e+03 number lines
1e+05
1e+03
1e+05
Figure 9: BYTE with ARtrg RUtrg optimized with code pages 1256 1251 (target language as facet)
35

---

Under review as a conference paper at ICLR 2021
1800000
1500000
TRG
121 1)
ES
FR
number of bits
SRC
1200000
AR
FR
RU
900000
1e+03
1e+05
number lines
Figure 10: BYTE with directions AR-RU & RU-AR optimized on both source and target sides (ARRU
36

---

Under review as a conference paper at ICLR 2021
ES
1500000
1000000
TRG
EN
number of bits
RU
2H
FR
RU
ZH
SRC
EN
ES
ZH
1500000
1000000
1e+03
1e+05
1e+03 number lines
1e+05
1e+03
1e+05
Figure 10: BYTE with directions AR-RU & RU-AR optimized on both source and target sides (target language as facet)
37

---

Under review as a conference paper at ICLR 2021
1200000
TRG
1000000
EN
ES
number of bits
ZH
SRC
ES
FR
800000
1e+03
le+05
number of lines
Figure 11: WORD models
38

---

Under review as a conference paper at ICLR 2021
AR
ES
1200000
1000000
800000
AR
EN
FR
number bits
RU
ZH
SRC
AR
3200000
EN
RU
1000000
800000
1e+03
1e+05
1e+03 number of lines
1e+05
1e+03
1e+05
Figure 11: WORD models (target language facet)
39

---

Under review as a conference paper at ICLR 2021
1600000
1400000
TRG
EN
ES
1200000
number of bits
ZH
SRC
ES
FR
1000000
800000
1e+03
1e+05
number of lines
Figure 12: BPE models
40

---

Under review as a conference paper at ICLR 2021
AR
ES
1500000
1200000
900000
AR
EN
FR
number bits
RU
ZH
SRC
AR
EN
1500000
RU
1200000
900000
1e+03
1e+05
1e+03
1e+05
1e+03
1e+05
number of lines
Figure 12: BPE models (target language facet)
41

---

Under review as a conference paper at ICLR 2021
H SAMPLE FIGURES FROM RUN AO, ALSO SORTED SOURCE LANGUAGE FOR CONTRAST
(a) CHAR
(b) CHAR by target
(c) CHAR by source
Figure 13: CHAR: character models from run A0
(a) BYTE
(b) BYTE by target
(c) BYTE by source
Figure 14: BYTE: byte models from A0
(a) WORD
WORD by target
WORD by source
Figure 15: WORD: word models from run A0
42

---

Under review as a conference paper at ICLR 2021
I LANGUAGE PAIRS WITH SIGNIFICANT DIFFERENCES
15 (non-directional language pairs total possible from 30 language directions, p=0.001.
LANG PAIR
CHAR
Pinyin
Wubi
BYTE
ARRU,
ARRU
WORD
BPE
AR EN
X
X
X
AR-ES ES
EN ES
X
AR-FR
X
EN-FR
X
ES-FR
AR-RU
EN-RU
X
X
X
X
ES-RU
FR-RU
X
AR-2H
X
X
X
X
X
EN-ZH
X
ES-ZH
X
X
X
FR-2H
X
X
X
RU-2H
X
X
X
X
Language pairs with significant differences indicate that the 2 languages are not equally/similarly good or equally/similarly bad.
Character models with ZH behave differently but the disparity can climinated with Pinyin.
Byte models with AR and RU exhibit unstable performance due to length but this can be rectified with compression the target side only (ARRU,).
Word-based models, including BPE, however, consistently favor and ZH (though more of "mis-segmentatioon for the latter, see $ and Appendix [ and disfavor AR and RU (as morphologically complex languages with higher OOV rates).
43

---

Under review as a conference paper at ICLR 2021
J LANGUAGE COMPLEXITY
In the words of Bentz et al. (0166:
Languages often compared with regard to their complexity from compu- tational, theoretical and learning perspective. In computational linguistics, it is generally known that methods mainly developed for the English language do not necessarily transfer well to other languages. The cross-linguistic variation in the amount of information encoded at the level of a word is, for instance, recognized as one of the main challenges for multilingual syntactic parsing (formulated as The Architectural Challenge (Tsarfaty et al. 2013). Complexity of this kind is also found to influence machine translation: translating from morphologically rich languages into English is easier than the other way around (Koehn 2005).
Morphology is "the study of the formation and internal structure of words". Morphemes are "the smallest meaningful units of language". (Bender 2013)
****
AR and RU are traditionally considered morphologically complex (see e.g. Minkov et al. (2007). Seddah et al. (2010) and proceedings of related workshops in subsequent years), and ZH lacking morphological richness (Koehn, 2005). But this definition of morphology is predicated on the notion of word, defined primarily from an alphabetic perspective. As pointed out by [Zhagg & Komachil (2018). "the important differences between logographic and alphabetic writing systems have long been overlooked". In logographic languages (i.e. languages with logographic scripts), there can be units within a character that carry semantic and phonetic information that have never been accounted for in the traditional practice of morphology or in the computation of morphological complexity. For example, in the comparison of different morphological complexity measures by [Bentz et al. (2016), all measures studied are defined with the notion of word (8 Yet, there no universally valid definition of "word" - the form/idea (as in, the philosophical concept) of a "word" may be there for most languagcs/cultures (though that is certainly also debatable), but its instantiations are different in different languages/cultures as well as in different genres/settings within one language. The variability in the definition of word is evident in the variation in language-specific word tokenization algorithms, along with the "indeterminacy of word segmentation" definition of "word" advocated by matpelmath esp. for logographic languages such as ZH (see Duanmu (017 and et al. (2019b) for how some ZH speakers do indeed consider a ZH character to be a word or how "word", as conventionally used in NLP, is not a native term or does not correspond with speakers' judgement).
work-in-progrent status for the (201)) as well as the contested nature of wordhood,
Our results with the Transformer indicate that a notion of morphological complexity can be modeled given our word tokenization scheme, confirming that morphological complexity is only predicated on the notion of word and bounded within the word level, and orthogonal to the performance of character or byte models. That is, unless word-based segmentation has been applied, there is no reason to attribute crosslinguistic performance disparity to differences in morphological complexity. In fact, on the character and byte level, we were able to achieve performance without disparity. Hence disparity is not a necessary condition but expectation that has been in mutual reinforcement with our practice of word segmentation, while the definitions of "morphological complexity" and "word" are in circular dependency with each other.
In this paper, we resolve language complexity, more specifically that of morphological complexity, in the context of computing through CLMing with the Transformer, in that we explain away the representation granularities and criteria relevant for such calculation.
TLDR: Up to the point of our taking up the subject of language complexity in this paper, there has been not a rigorous definition of "language complexity". Conventionally. "language complexity" is synonymous to "linguistic complexity" (with the tradition of "linguistics" being primarily word-
$ 'n exception could be that of the type/token ratio (TTR). One could imagine applying TTR on the character level for ZHH, and that would be indicative of its morphological richness on the character level. However, that has thus far never been practiced or recognized in NI.P.
44

---

Under review as a conference paper at ICLR 2021
based), and people just assume linguistic complexity, e.g. morphologicallsyntatii complexity. to be intrinsic and necessary in languages (across representation levels). Our findings show that linguistic complexity is relative to the representation granularity, i.c since morphology is based on words, it is bounded to the word level.
An alternative perspective, with finer prints:
We have also developed more rigorous interpretation. We take on the definition of "language complexity" as one that is related to the statistical attributes of languages. We assume and define solving as the elimination of statistically significant performance disparity.
In larger (6-layer) models, and according to the conventional definition of "language" - i.e language whole, solved language complexity with compression of AR and RU in byte representations. In smaller (1-layer) models, one can think of the situation as: i) no complexity has been modeled by the Transformer hence there is nothing to solve, or ii) there is no complexity between these languages to begin with, or iii) the Transformer solved the complexity.
With respect to each representation level/granularity the larger models:
one can think of as having solved complexity with byte representations with I-layer models - for these 6 languages empirically. Theoretically, there could be languages with longer sequence lengths than RU and AR, in those cases, don't claim have solved the matter empirically but only resolved it conceptually. But this is the most that anyone could do at the moment, there is no relevant parallel data available.
BYTE:
one can think of us as having solved it via bytes or I-layer models. Whether we can be considered to have solved via Pinyin for ZH depends on whether the evaluator accepts decomposition into a phonetic representation only qualifies solution for the ZH language.
CHARACTER:
WORD; one can think of us as having solved it via bytes or 1-layer models. It not possible to solve it strictly within the word level without creating word segmentation criteria that would be unrelatable to native speakers. And since "word" is exclusively a human concept, we must either claim that a universal solution is undefined or undefinable for computing, or retreat to a unit that is the greatest common factor crosslinguistically. Since some ZH speakers consider ZH characters as words, return to the character-level solution.
It is beyond the scope of our paper to solve the qualitative disparity on the word level. However, we do advocate a more inclusive evaluation and critical reflection on the possibility of discontinuing the usage of "word"" as such a non-technical term biases against both "morphologically complex" and "morphologically simple" languages. The world of languages in written form can be divided into those with logographic scripts and those with (phonetic) alphabetic ones, with the unit of character being the greatest common factor of them all, from the human perspective. For technical processing, esp. for fair multilingual measures that are more standardized, such as those based on bytes or characters. There is room for improvement in the design of character encoding that complements the statistical profiles, e.g. with relative rank in sequence length, of different languages. We believe there is crosslinguistic systematicity on the character level to be leveraged.
to-eenueee- modeling with the Transformer, we recommend
One's 's readiness to accept this as solution to language complexity can be subjective matter. One may insist that language complexity be solved exclusively with monolingual LMing (which lies outside the scope of the present work), instead of being confounded with the logic of one language being conditional on another. One may also object to the idea of (re-)solving morphological complexity being equivalent could also be e.g. syntactic complexity (although as substantial "information concerning syntactic units and relations is expressed at word level" in morphologically rich languages Tarrfaty et al.) 2010), the boundary between morphology and syntax is less distinct for some languages than others (Haspelmath]] 2).I)). If, however, our results could be extended, we wonder if syntactic complexity could be due to our sentence segmentation or a combination of word and sentence segmentation practice. That we leave for future work for those who are interested in the topic.
leading to solving language complexity as a whole, for there
45

---

Under review as a conference paper at ICLR 2021
K SAMPLE-WISE DOUBLE DESCENT (DD)
K.1 OUR EXPERIMENTAL FRAMEWORK ON DD DATASETS FROM (NAKKIRAN ET AL.. 2020
Text experiments from previous work reporting sample-wise DD involved words (Belkin al. 2019) and BPEs (Nakkiran et al. 2020).
applied our experimental framework - by testing data points with 10 lines - on the datasets reported in (Nakkiran et al., 2020) to exhibit DD. WMT E4'EN-FR was reported to demonstrate model-wise DD and IWSLT (Cettolo et al. 2012) DE-EN model-wise and sample-wise DD. We downloaded and prepared the data with scripts from the FAIRSEQ Toolkit (Ott et al. 2019 The WMT data was preprocessed with 40,000 BPE operations and IWSLT 10,000. Our focus is on sample wise DD and hence our goal was would apply also to these datasets. We used the same training regime with the Transformer and Adam SOCKEYE as before and tested both language directions on the entirety of both datasets, with no subsampling. For the IWSLT dataset, we tested data sizes with 102 10t lines, then at 160, 239 that the total number of lines available. For the WMT dataset, we tested from 10 to 10", then at 35, 762, 532.
if the spike 10 we observed with the UN data
(a) WMT 14
(b) WMT' 14 by target
(c) IWSLT' 14
(d) IWSLT' 14 by target
Figure 16: WMT 14 FR and FR EN and IWSLT' 14 DE-EN and EN-DE: sample wise DD shown at 10
Table 2: Target-Traie Train-Token--ee
ratio (TTT2P ratio) for WMT 14 EN-FR and FR-EN
Number lines
100
1,000
10,000
100,000
1,000,000
10,000,000
35,762,532
oum train tokens
3,248
33,768
313,154
3,123.129
30,852,455
308.640,462
1.174,344,513
FR: num train tokens
3,548
36,507
339,803
3,414,959
13,865,679
343,344,536
1,327,817,765
EN-FR num params
45,609.474
51,039.363
62,871,584
75.630.304
85,210,087
108.226,335
131,417,633
TTT2P ratio
0.000078
0.000715
0.005405
0.045153
0.397438
3,172468
11.917483
params
45,540.219
50,692,575
83,936,258
107,378.859
TPT2P ratio
0.000071
0.000666
0.005058
0.041894
0,367570
2,874313
10.541771
This shows that the effect we reported in $ 5 also holds on these datasets: "the ratio of target training token count to number of parameters falls into O(10-1) for 102 lines, 0(10~3) 103. 0(10-2) at 10 and 0(10-1) for 10 lines and so on".
prepare-wmt.len2f--eh and http://aaaaaaaaaaaaaa////o//oooa
exampl exmmp]ess//rranslleiiioorppspss iwwpearre-wsis11....
mmax-seq-len 300; beockkooitt-requeeee 4000 except for cases where 50 epochs would be reached before
the first checkpoint: 400 for 102 lines and 3450 for 10 lines.
46

---

Under review as a conference paper at ICLR 2021
Table 3: Target-Traie Train-Token- Liren-tt-PPrrameee ratio (TTT2P ratio) for IWSLT 14 DE-EN and EN-DE
Number of lines
100
1,000
10,000
100,000
160,239
DE: num train tokens
2,874
27,675
253,757
2,519,534
4,035,591
EN: num train tokens
2,739
26,416
245,659
2,461,879
3,949.114
DE-EN num params
45,297,348
49,410,683
53,639,825
55,189,376
55,428,584
TTT2P ratio
0.000060
0.000535
0.004580
0.044608
0.071247
EN-DE num params
45,405,078
49,809,797
54,300,056
56,245,643
56,564,366
TTT2P ratio
0.000063
0.000556
0.004673
0.044795
0.071345
K.2 TOKEN-TOU -EEAN-TO-AAA TO-PPARAMETER RATIO FOR NON-NEURAL MONOLINGUAL LMs
We experimented also on KenIM (Heafield) 2011 Heafield et al., 2013), non-neural LM with modified Kneser-Ney smoothing (Kneser & Ney 1995] Chen & Goodman 1999), and found that on the word level, such spike (or a hump) is common across all languages, see Figure 17 The sargettookn--e to-paareeterr ratio is under for most of these smaller data sizes. This seems related to the analytical findings in Opper et al. (990) where the pseudo-invers solution to a simple learning problem shown exhibit non-monotonicity. with the peak exactly the ratio of data to parameters (a) approaches 1.
dataset
1500000
125090
C- DIDER
20000
MORD
1052000
19X000
72200
18ta size
Figure 17: Kneser-Ney (monolingual) n-gram LMs on the same data (A) used our neural CLMs
The number of parameters a k-gram model is the number unique n n-grams, 1 < n S Table shows the ratios for trigram model (all n-gram models of higher order exhibit the same effect).
47

---

Under review as a conference paper at ICLR 2021
On word level, where the function of number of bits to data size is not always monotonic, we observe less of a monotonic development whenever the token-to- secen-to-prameter ratio is smaller than 1. This is more notably shown in the first 4 sizes in AR with hump-like curve before the performance improves at 10 This different from the sharper descent for ES and FR, where only the first two data sizes have non-monotonic relationship and ratio as a rough proxy for over- (< 1) and under- pateer--rramttiizttionn (>> 1), this can be seen an instance of non-monotonictty with respect data size in the "critical regime", i.e. when the model transitions from being (heavily) over- to under inder-parameteriza (Belkin et al. 2019 Nakkiran 2019).
ratio less than 1. TTkiing the
A remark modeling with finer granularity Our KenLM results show the performance of bytes and characters is not on par with that words with non non-neural algorithms. NNs/DL has enabled much progress in this regard.
48

---

Under review as a conference paper at ICLR 2021
Table 4: Token-to- -arrammetrr ratios on non non-neural monolingual trigram LMs
lang numlines
mum tokets
Inigrams
higrams]
[trignams]
num paraes
tokens/praam
CHAR
AR. 100
9077
85
025
2854
3904
2.925563525
AR. 1000
123832
110
1577
8592
10279
12.04708629
AR 10000
1083517
152
3216
21479
24847
43 6075626
100000
10625147
179
5114
44251
49544
4567859
AR_ 10060000
102064250
242
8317
99112
1029
EN_IOU
11730
78
800
2532
3416
EN_1000
159644
84
1215
5808
7107
22.43478261
EN_ 10000
1344001
125
2532
17181
19638
67,7488
EN 100000
13132863
EN.
125491871
7126
70406
77779
1587
ES_ 120
12574
87
781
2308
3266
3.788752504
ES_ 1900
171104
95
1210
5045
0345
ES_ ES_10000
1484834
117
2334
18113
81.97499366
BS_
14549700
37991
ES. 1000000
33596036
257
7217
67280
74724
I854 02816
FR. 100
12456
89
836
2610
3585
1.522620014
FR. 1000
179144
97
1259
5711
7067
25. 33578005
HR_100I0
1490993
2007
19022
7H. WW.S820111
F00100000
14528593
178
4240
35081
29079
FR_ 1000000
138019189
259
7353
69522
1789 722012
RU. 100
11980
96
952
3051
1011
2.921 2381228722
RU 1000
10000
1436078
168156
111
163
1415
7106
24147
1693
19 19088557
59 4725154
TOBIOO
14151726
190
5737
44071
285.0455818
1000000000
134706120
263
10186
96975
105824
1277.75572
ZIL 100
3315
605
2036
2634
5275
0.6290097393
2IL 1000
42572
1239
13266
26811
39316
1.0828 16
7H 10000
572001
175730
246
RO0D
ZH, 10H0000
7HL 100000
34672612
4855
241215
611213
3977112
968852
1212470
5993215
3.015528100
7.54866 1906
BYTE
AR_ 100
16655
76
220
1163
1559
10AAEE 1302
AR_ 1000
227163
539
2070
2707
83 83.91688216
133
1616
125255
AR 100000
19487689
148
2844
17266 7
1128 674215
EN_
EN 10U
AR. 1000000
166711880
159449
11731
165
79
1219
5219
807
40507
2533
7110 2116
141 $419
49801
22 1074
3.431 1212
40%% 2447
EN 100000
13158948
154
130
3971
2527
1971
34985
19796
O111
338, A590335
EN 1000000
125706128
169
6422
66006
1600 030009
FS. 100
12629
88
766
2414
1384
1884443084
175286
1146
8148
28. S4RK90N
BS_ 1SKOO
1513753
12)
2409
14894
17424
NG. S791
100000
16821895
154
3925
31905
411.89125 2512
BS. 1000000
141276766
169
6138
62199
0960
2056, 250779
FR. 100
12875
90
830
2560
27. AA7447AT
8.69971 1264
1000
185227
1227
FR.
PR, TOHNOO
15055637
156
133
2402
4014
33105
15615
18240
17773
B4. 54575666
4US 00176525
FR_ 1000000
143495667
175
6423
69044
2031. BOS 103
RU_ 100
21751
100
475
1365
11 21 185567
309279
RUI 100000
RU 10000
259902633
2636801
160
151
3364
1898
18321
8430
21844
10477
13322
1189 757077
25L 607
2H_
KU. 1000000
247098758
9559
140
169
1524
6224
45935
4722 11 113553
Z3L 10000
7H 100000
9900046
1019959
116667
146
167
5396
9770
11000
36176
12857
111000
24, 30667669
109. 905357
7.426761722
541 SA
WORD
7H. 1000000
TOO
94268840
1776
860
196
1534
1600
AR_ 10000
AR. 100000
AR_
2005190
206569
23450
97997
26108
19864
11811
10005300
1383009
168002
20003
22555
20000
09014230
125778
AR 1000000
19410502
2071
304978
682
1567
1999
27596
EN 10000
236569
12014
R3397
155493
2500N
EN 100000
2359100
37264
425349
1117800
EN. 1000000
21943139
123457
1818166
6505850
2232
1605
1974
ES. 100000
ES. 10000
BS_ 1000
2588791
263024
29461
49499
15116
499564
5000000
09900
13199
1166
116177
160078
20534
ES 1000000
24654409
142809
2206
745
1721
32011
3881
14550
23608
FR. 100000
PR_ 10000
FR. 1000000
25595487
268-9822
273195
11824
12870
1998
1150000
1150965
170729
0000
3,20122
1.6551 115155902
0801211800 906
1854
1589
1080
24746
5433
15S11
20015
0.609870225
RU, 10000
216638
23405
106516
162401
0.7300677SM
RÚ. 100000
RU_ 1000000
20421965
2150746
236088
81342
3299028
670657
1917195
1306351
LU44478655
L681053852
2IL_100
1751
1434
3678
0,4750739513
19532
731 1000
23568
3181
13698
1934
36520
0.645450011
7H. 10000
207714
13157
96829
169642
2705008
0,2675826267
ZHL 100000
2038639
46941
554739
1278188
1879868
1.08445559
10L00000000
19361101
2527710
8401311
11065513
1.399995774
49

---

Under review as a conference paper at ICLR 2021
1,000000
714
50
110 00
1.0
17.000
26.11
60
62 (5 10
11 000
DIS MELIN
1.00
81 1212-
215
05 317
200
315
800
12.
48
3.000 000
6TN-EE 105 7IN (90
000 611-993
110 0010
OU OE MATHR THE CENN
WORDD
10.00
09 354
71 030 4TC
00 201
Number model parameters for dataset A
DI IE
01 085 61
1.000
65 223
05
06
INNI TO
TH NE
OI
OF
15.30 72
45,1 1A1 30
EAOSA MEI
20
1.00
(5.15 15
1,000 100
014
01.007000 00L
24.02:20 22
24.20. SIT
ON
18.80 26
11:011 111
15.055 20
16.2
11.20
04-
11.50
44
231 000
062
46 082
646
035
01
377 530
2O EED
INTE 05-440
314
06.20 200
39
500
66,
5,003
200
11-00
311
14.508 dir
125
11,200 455
$1.00 000
11.200 130
201
14.500
$1.000
14:244
11.501
1.00 000
03 300
09 228
L NUMBER OF MODEL PARAMETERS
08
01
221 012
377
AI OO THE CO ENN
47.
110
13 313
185 AND
CHAI
11 000
41 383 080
ANS
1.000
$1.500 350
14-20
44
04. 220 808
911
212 531
350
26, 522 200
350
CL. 701 250
11 21 AOR
M. 200 AND
SEN THD
AND
MLTTE TTD
1E STD
TND
OND
180
44
Nunber of FAARAN
Ieperrodes Natho Lano
28 wats
AX TR
ES US
IS-AR
65 EN
23
TH AR
22H0
50

---

Under review as a conference paper at ICLR 2021
110,00 0n 1,000000
413
50 30.00.000 500
DEMIN 50
10.50 B5S
de
12.00 3G
18- 0U
1T 30
010
10
62 575
81 381 30
IN ON
300
320
225
EEN
340
48 32-
05
510 100-
DEL A00UM
160
201 A01
30) OND 209
1100
06 300 550
20
H80009
65 15-
Number of model parameters for dataset B
25 114 221
55 701
10 422
04 300
OA
CA. T3
15.30 220
45.00 205
E5 1N-T11 S17
26
15.00 9T7
14.200000 UTT
300
11.3 11.25.00 VTT
30
11 201
24.200
04.000
11:
41.
24.
060
232
311
043
13.244 221
400
00
355
001
11
14
01
44.
202 305
225
10.000
271
312
310
16. 300
90
40
01. 73
3,009
11.35-000
11.712200
10.00,0000 013
31.307 39
11
11.20 E3
300
16-200 511
300
41.00 60
14.204.000
44.07%
06
735 1T1
1.000
600
15
45
392
26 411 821
SH
1000
421
450
120
(3 DN
11.07 023
0003
00.00.440
105
100
TOU
204
44:20
$1.00
45,
301
414
04 31.000
7T8
OL 531
218
AND
THE
AOD
SOL AVD
Number PRAMAMS
29 Seeth
M6.33
A8 M0
AV-21
AM
25
201
BT AR
TM
2H EN
225
1S

---

Under review as a conference paper at ICLR 2021
110, 00 1,000000
318
20-000
10 000000 538
18.25 10 25
10 00L0N420
71
11
BACOUA 460
00
05
10.00
20
25
HI- 750
453
00 200
80 381
222
d5 3T7 2M
011
DEL (AN 501
4N0 109 013
DE O1
1100 3,000
200 0200000
30) 300 TH0
OA 20 TEE
26 800 100
65 99. RED
M. 304
100
H80000
24
Number of model parameters for dataset C
2%
196
25 217 823
55 KTV
10 415
353
10 403
04
(5.62 803
BLLUA EU
15.90 365
45.08 301
777
BANGN IN
RLOO IN
15-70
18 MAS
15.0H 312
44-300 d1T
44 20.00 DH
11,30 DE
72
1,000,
00.00
44 04.00
21.22.
44
41
060
710
550
204
353
020
11
44
44
315 753
217 525
520 505
31.22 422
10.000
312
10
46,
46
3,000
28.20 33
22)
01
44.05.10
281
64,000
000
44.000 060
11.300 200
210
GTT
16.00000 200
41.50 200
14.300
14.507
1810800000 164
10.000 576
TM4 LTS
1.000
15
1000
06, 022
EN
271
0003
110
200
007
44:30
$1.00
45,
510 3M4
222 644
04 017 201
01. SE0 313
316
331-31
SUR BND
STR
THE
THE
Number PRAMAMS
POCTIL palle
M8.33
A8 M0
AW-21
AM
25
211
BT AR
TM
28 EN
225
29
52

---

Under review as a conference paper at ICLR 2021
M ERRATICITY
Length has been an issue since the dawn of the encoder-decoder approach for NMT (Cho et al. 2014). Most work length bias, except for that by e.g. [Sounnsoo & Sarawagi (20T6); seems to have focused on the evaluation of generated translation ouiput and monitored performance degradation with respect to sequence length, often arguing that beam size plays role (Koehn & Knowles 2017 Murray & Chiang 2018). (Related work in Stahlberg & Bymee 2019) provides good summary on this issue.) While there could also be confounds in search, our experiments show that kind of length bias can surface already with CLMing, without generation taking place. To our knowledge, length bias has not been expressed as a sample-wise non-monotonicity across a large data size range as ours. While the connection between erraticity in CLMs and length bias in NMT models remains to be verified on a case-by-case basis, the knowledge of length also contributing to robustness (not just consistently poor/poorer performance) could support further experimentatiio//epiicatio of any study. Failed attempts to reproduce results may be explainable by erraticity.
One may argue that erraticity may not be relevant when each model is more optimally trained (as opposed to being treated with our one-setting setting-for-all regime). But we do want to stress that this very stark contrast between erratic and non-erratic behavior is possible, prompting a question on fairness: is there a one-for-all setting under which the languages with non-erratic behavior shown in our study would demonstrate erraticity and vice versa?
To the best of our knowledge, the meta phenomenon of erraticity, as a sample-wise non-monotonicity measured intrinsically with cross-entropy and contributing large variance across runs, a novel and original discovery and contribution to research in robustness. We hope our work would inspire further evaluation on other models/architectture, reflection and theories on our assumption of unbounded computation (e.g. Xu et al. ()020)) as well as new understanding and solutions that take data statistics and realistic computational aspects into account. We defer a more comprehensive analysis of erraticity with further experiments to future work.
M.1 ERRATICITY AS LARGE VARIANCE: EVIDENCE FROM DIFFERENT RUNS OF THE SAME DATA
To confirm that erraticity is not due to data-specific reasons, c.g. when certain data segments might be "easier" to model than others, we show figures from 2 runs (Figs. [8a and 18b) on the same dataset of wildly differing performance that only differ in seed. Note that changes in the y-direction can vary much, indicating large variance across runs.
By establishing that high variance holds across sample sizes, we showcased how it'd be possible to just test on 2 or 3 data points of smaller sizes get gauge the robustness in higher order. It serves as a signal of when the system is being "stress-tested" and hyperparameters need re-tuning. Spot-testing on couple of smaller data sizes can indeed save much time and energy. Take our run BO byte models as an example: the training of the - model for EN-RU took 15 minutes, 10x 40 minutes, 1 hour 50 minutes, and 10 3 hours 36 minutes. One can imagine how these would just be a fraction of training time for bigger models. (Likewise, for our ratio of target training token count to number of parameters knowing when a representation might be prone DD within a data size range could help prevent practitioners from prematurely declaring experimental results as negative or from unnecessarily rerunning an experiment because bigger data did not lead to better results.)
M.2 ADDITIONAL EXPERIMENT WITH LENGTH FILTERING TO 300 BYTES
Figure 19a and 19h show results of additional experiment with subset of data in byte (UTF-8) representation length- filtered to 300, including dev data:
Erraticity remains for AR and RU. Scores are lower, though they cannot be compared with the experiments in the main paper due to difference in dev data size (3,077 lines vs. 1,804 lines here). Number of total lines for train is 5,533.672 lines for each language, from which we took the initial 102-106. As in our main experiments, tails of longer lines. 300 bytes aren't 't long sequences, but without data transform or hyperparameter tuning, things can look unfair. The EN translation of the longest RU line in this dataset is: "47. It is
filtered out only whole lines, ie. not by discarding the
53

---

Under review as a conference paper at ICLR 2021
740075
Tumber cfiica
Harbbr of D9a
(a) run AO
(b) run AI
Figure 18: Same data with differing seeds
noted that there is a lack of information provided by the Government of Trinidad and Tobago with regard to the legal status of the Convention the domestic legislation."
54

---

Under review as a conference paper at ICLR 2021
Figure 19: Additional experiment with maximum length of 300 bytes (with no hyperparamter tuning, in our blind one-setting-for -or- evaluation). Considering there are languages with much higher character sequence length than RU, there is food for thought for the design of next-generation Multilingual Plane.
55

---

Under review as a conference paper at ICLR 2021
N EXPERIMENTS WITH ONE-LAYER TRANSFORMER
We performed run with dataset in 4 sizes (10--10a lines, seed=13) with the primary representations of characters, bytes, and words, 1-layer Transformers (num-layers 1:1, all other hyperparameters remain the same as for our main experiments). We compared this against run AO in sizes with the same seed. (Based on how our null hypothesis is set up, the higher the number of runs, the more likely it is for there to be disparity. Important is that we evaluate based on an equal number of runs and on the same data for all candidates.) Results are shown in Table | with no statistically significant disparity observed on the models trained with layer across the board.
Many are under the impression that big data is the cause to the neutralization of language instances in DL/NNs. But, as this set of experiments shows, it is possible for there to be no statistically significant differences between them, with as little as our smallest data size of 100 lines.
Table 5: Number of language pairs out of 15 with significant differences, with respective p values. BYTEstayers the representation with erratic ARerg and RUtr--
BYTEgtagera Clyyrr
WORDslayera
CHAR Jayer
BYTE
WORD
WORDe blayer
p-value
STC
trg
STC
ug
SIC
trg
STC
teg
SIC
tg
STC
trg
0.05
0
0
0
0
5
0
0
0
0
0
0.01
0
0
0
0
0
0.001
0
0
0
0
0
(a) CHaRRtayer
(b) TTTeyeee
(c) WORD Llager
(d) CHARIayer Hayer by target
(e) BYTEnteyer by target
WORD WORDDaager by target
Figure 20: One-layer Transformer models
56

---

Under review as a conference paper at ICLR 2021
O PAQS (PREVIOUSLY ASKED QUESTIONS)
0.1 ONE SETTING FOR ALL
Q: Normally, one trains a model with the objective of optimizing based on the training and evaluation data with hyperparameter tuning. The experiments here used one setting for all. Some model configurations might train better and converge close to their optima while other configurations might not reach their full potential. Can this not create distortion in the results?
A: For conventional engineering practice, we agree that hyperparameter tuning would be a sine qua non. However, the evaluation objective is the relational distance between languages, hence need to see a different light. Here a loose analogy:
Assume objects in 3 different locations in space.
Relative evaluation from one setting allows one to capture the distance between these objects. It does not matter whether these three objects are in their "best" states.
For example, if one were to use camera to capture these 3 objects and one does not adjust the setting (using just one random aperture, shutter speed, and focus), i.e. no tuning to capture any of these specifically, nor does try model these to their individual bests separately, what would result could be a picture that captures one of these objects more favorably than the others, or it could be that all of these would be blurred. But either way, there is degree of blurriness to be measured, giving us an idea of the relative distance between the objects. Such relative measurement is the evaluation strategy that our paper adopts.
Now, to add to the camera analogy, say one of the objects is running water, which was extra blurry [erraticity]: we suggest freezing the water, so even from the one arbitrary angle, it could be captured better. And it worked.
Also, while one might generally like to have "pretty" photo, one that is e.g. taken with sub-optimal lighting, say, overexposure, can have telling effect as it can bring out details in something dark, like black box.
Alternatively, one can tune hyperparameters for each model individually such that each model would be a more optimized one and then compare these models. In that case, one would be interpreting the differences between language in terms of hyperparameters, and the paper would be one that is algorithm-centric. That is of course also possibility. Our approach, however, is data-centric one. We would, first of all, like to understand the nature of language data, i.e. what it is about language, if there is anything at all, that makes it a different data type than other data, and what kind of structural constraints, if any, that we need to take into consideration. Then with findings from this data perspective, we try to relate back to the algorithm and make connections so to create a more holistic picture.
O.2 TRANSLATIONESE / WORD ORDER
Q: Multitexts are parallel texts or translations with the same meaning. There is little to no variation in word order, hence they are just "Translationese" (Gellerstam 1986). That is why they turn out to be the same, with no performance disparity.
A: Our findings do show that when the semantics is properly controlled, such as in multitexts, the factors influencing performance are statistical properties related to sequence length and vocabulary, e.g. |V| TTR. and the languages tested can be different. Semantic equivalence is also not reason why we should expect neutralization of source language instances, as that would mean we should expect equal results across target languages.
We agree that faithfulness is often a priority in producing good translations. Whether the translations produced by humans machines, only a single best translation surface the translation of choice. There may be many other competing hypotheses, but regardless of whether it is done through an automatic ranking algorithm by a machine or through human expert, the purpose of
57

---

Under review as a conference paper at ICLR 2021
translation is the same. However, styles and preferences in translations vary. While faithfulness is generally preferred in the translations legal texts, more freedom with skillful rearrangement of and play words (or rather, character sub-character sequences) sounds being criterion for literary texts could be appreciated by certain readers. We agree that could be very interesting and necessary model these variations, and understand that languages surface in many multimodal forms beyond the confines of texts as well. But with a data-driven perspective, to model this broader variation language, need corresponding datasets the difference in e.g. sequential order is explicit. And for evaluation, we would require an even more systematic meta evaluation, one that spans different datasets.
suggest contrast where
But the argument that language or data could be different beyond how it appears in one dataset is irrelevant the evaluation of experiments involving said dataset.
58

---

Under review as a conference paper at ICLR 2021
P UNDERSTANDING THE PHENOMENA WITH ALTERNATE REPRESENTATIONS (EXTENDED VERSION)
[Appendix P an extended version of $
To understand why some languages show different results than others, we carried out a secondary set of control experiments with representations targeting the problematic statistical properties of the corresponding target languages.
Character level On the character level, it is well known that ZH differs from the other languages in its high [V], in this study it has an averaged ttd of 2550 1449 across all 5 data sizes from all 3 datasets compared to 1 170++87 from all other 5 languages combined, may these be in Latin or Cyrillic alphabet or the Abjad script. But what is often not known is that the character sequence length of logographic languages such as ZH is typically short (think and compare the sequence length of the Ancient Egyptian hieroglyphs or the Demotic script with that of the Greek script on the Rosetta Stone). Here our case, the averaged mean sequence length in characters for ZH is 19, compared to 119= from the other 5 languages. Heuristics to mitigate high [V] often involve decomposition, which automatically resolve the problem of short sequence length. We tried methods to lower character [V| with representations in ASCII characters - Pinyin and Wubi. The former is a romanization of ZH characters based on their pronunciations and the latter is an input algorithm that decomposes character-intenat internal information into stroke shape and ordering and matches these to 5 classes of radicals (Lunde 2008). We replaced the ZH data with these formats only on the target side and reran the experiments involving ZH target language (ZHirg) the character level.
Results in Figure 2 and Table show that the elimination of disparity on character level is possible if ZH is represented through Pinyin (transliteration). as in Subfigure 2c But Wubi exhibits erraticity (Subfigure 2a). Wubi in our data has a maximum sequence length of 688 characters. As we shall also show in our byte-level analysis below, there are reasons to attribute length as cause to erraticity.
Decomposition into strokes may seem like natural remedy analogous to decomposing an EN word into character sequences, but one needs to be mindful of not exceeding an optimal length given finite computation. Considering the ZH in the UN data is represented in simplified characters, decomposing traditional characters would surely complicate the problem. As there are also sub-character semantic and phonetic units (Zhang & Komachi 2018 that can be exploited for information and aligned with character sequences of other alphabets, qualitative advances in this area can indeed be new state of the art.
Byte level On the byte level, we observe irregularity for AR and RU We find minimum sequence length of the target language be one of the highest metrics correlating positively with the total number of bits (p 0.60) [3] Our data is based on 300 characters as maximum length per line. While we wanted to retain at Teast 75% of the UN data after length filtering, this length still renders maximum sequence length that exceeds 100 words (the default maximum length for the word alignment model, GIZA++ (Och & 2003), in the traditional SMT pipeline). Translated into bytes with UTF-8 8 encoding, data with 300 characters maximum gives us, e.g. for the 10%- datasets, an averaged std of 185+ 106 in length for AR and 246 for RU, considerably larger than that for ZH (94+:3) and for EN/ES/FR (1175.41+77). With UTF-8 encoding, each character in AR, RU, and ZH contains 2 or more bytes. ZH typically has shorter line length in characters, compensating for the total byte sequence in length, even when most ZH characters are 3 bytes each. However, AR and RU generally have long line length in characters. so when converted to bytes, the sequence length remains long even when most of the characters might be just bytes each. Results from our pairwise comparisons indicate 8 mon-directional) language pairs to be significantly different (see Table I Junder "BY "BYTE"): ES-RU. EN-RU, FR-RU, RU-ZH, AR-RU, AR-EN, AR-ZH, and AR-FR all involving AR or RU. (Appendix Ilitts also the language pairs with significant differences for other representations.)
12 Figures are rounded to whole number. Complete tables of data statistics are provided in Appendix D Top-3 correlates for each representation he found in Appendix F
59

---

Under review as a conference paper at ICLR 2021
Leveraging prgguage-spccifie code pages be useful practical trick, reminder that there alternatives to UTF-8 for analyses and back-end processing data clean and homogeneous and success of larger-scale prediction is not concern. But more sustainable alternative is to design more adaptive and flexible character encoding scheme in general, taking into account the statistical profiles such as length (wit characters and bytes) and sub-character (aomm//elementay//oooondd) information of all (or as many as possible) of the world's languages.
Word level The main difference between word and character/byte models the absence of length as top contributing factor correlating with performance. Instead, what matters more are metrics concerning word vocabulary, with top correlate being OOV token rate in the target language (p 0.66). This is understandable lengths in phonetic alphabetic scripts are shortened through multiple-characee the shorter lengths in logographic scripts (ef. difference in length for the 3 scripts on the Rosetta Stone, logographic scripts are typically shorter than phonetic ones) are lengthened by the insertion of whitespaces. To remedy the OOV problem, we use BPE, which learns fixed vocabulary of variable-length character sequences (on word level, as it presupposes word segmentation) from the training data. It is more fne-grained than word segmentation and is known for its capability model subword units for morphologically complex languages (c.g. AR and RU). We use the same vocabulary of 30,000 as specified in Junczys-Dowmin by 89-100% across the 5 sizes. The number of language pairs with significant differences (p s 0.001) reduced to from for word models, showing how finer- grained modeling has positive effect on closing the disparity gap.
word segmentation neutralizes sequence lengths the longer
groupings, while
et al. (2016). This reduced our averaged OOV token rate
60

---

Under review as a conference paper at ICLR 2021
Version 1.1 (graphs to be updated, score tables added)
61


Reviews:
[{"review_id": "zh4Xv5Ynj_D", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors attempt to investigate to what extent languages are hard to conditionally language-model. They do this by using some information theoretic measures. Claims:\n- There are no statistically significant differences between source language representations, but there are significant difference between pairs of target language representations.\n- There is no complexity that intrinsic to a language except its statistical properties concerning sequence length and vocabulary (unless word-based methods are used).\n- They also observe phenomena such as Double Descent and erraticity.\n\n----\nStrengths:\n- The Experiments are extensive.\n- The relative similarity of source language representations is interesting and worth exploring further.\n\nWeaknesses: \n- The diagrams are difficult to read\n- The paper is hard to follow and would benefit from a clearer focus rather than the broad range of topics covered here. For example:\n    - It is difficult to understand what the methods/terms (the information theoretic measure used, double descent) are - little time is spent explaining these.\n    - Double descent is discussed in the paper but it is still made not clear why this is relevant in the paper.\n    - Several portions of text are repeated - with some editing, space can be made to discuss concepts important to the paper\n- The authors make recommendations for modeling (Eg. using char level or byte level models for certain models - which have been extensively studied for this): this is not followed up with any concrete results on translation/downstream tasks or pointing out relevant work.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZTDrA_qwKLm", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper provides an empirical investigation of an important problem: the transferability of language modeling signals across languages in the transformer model. This is an important question because it can teach us both on the relations between languages and the properties of the transformer model (although it is not easy to tease the two effects apart).\n\nThis is a thorough paper with a large number of experiments and with interesting conclusions that nicely generalize the low level patterns observed in the experiments. These conclusions are likely to be useful for the research community as part of its on going investigation of language transfer and the transformer model.\n\nI have several comments though:\n\n1. The language of the paper is often very complicated. Just as a couple of examples: It was very hard for me to follow the abstract, the first paragraph, the (very long) sentence that start with \"in order to eliminate\" (1.1), item 3 in the list of contributions and this is just a partial list.  I ask that if the paper is accepted the authors will try to improve this aspect.\n\n2. The writing is often over pedagogical and I often got the feeling that the authors try to educate their readers (but not in the positive sense of the word). I would try to avoid this style. \n\n3. This work seems highly relevant to the following paper:\n\n\"Towards Zero-shot Language Modeling.\" Edoardo Maria Ponti, Ivan Vulic,Ryan Cotterell, Roi Reichart and Anna Korhonen . EMNLP 2019\n\nI think the discussion parts can gain form comparing the conclusions of the two papers, when relevant.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "XFM-CP46DgZ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is trying to answer an important question: How does representation play a role in carrying meanings? In doing so, the authors experimented with 6 languages in  3 + 5 kinds of representations. The authors concluded that the different performances among language pairs maybe be the result of word segmentation in different ways.\n\nThis is an interesting step towards understanding meaning representations, especially in languages that do not have an alphabet. However,  as much as I agree with some of the final conclusions, the soundness of the experiments appears to be in question.\n\nMy main concerns are about the additional experiments with Chinese.\n\nThe authors claimed that \"On the character level, target language ZH (ZHtrg) shows a different learning pattern throughout.\" There are two types of character-level representations used: Wubi and Pinyin. Wubi was originally invented for professional typesetters so that they can type fast. The segmentation may not be correlated with the meaning of the word at all, as claimed by the papers cited by the authors. Pinyin, on the other hand, is highly ambiguous. One pinyin may representation dozens of words and the authors did not take tones into considerations at all. \n\nThe author also mentioned that \"After filtering length to 300 characters maximum per line in parallel for the 6 languages, we made 3 subsets of the data with 1 million lines each\". Each language carries meaning differently and the information density is drastically different. 300 characters in Chinese carry much more information than 300 characters in English. This is an unfair comparison.\n\nIt would make a lot more sense if the authors treat each language differently because of their orthographic differences.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6vp_hHxTzOu", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper investigates whether languages are equally hard to Conditional-Language-Model (CLM). To do this, the authors perform controlled experiments by modeling text from parallel data from 6 typologically diverse languages. They pair the languages and perform experiments in 30 directions with Transformers, and compare 3 different unit representations: characters, bytes, and word-level (BPE). \n\nI appreciate the authors' effort for their systematic controlled experiments. However, I'm leaning towards rejecting this paper since I think some of the claims made in the paper are too strong and not really backed up by their experiments. \n\nSome comments:\n* The term \"Conditional-Language-Model\" can be misleading, since this paper model a target language conditioned on a source language, so more like in a machine translation setting rather than standard language modeling setting where you can also condition on the previous history.\n* I'm also not sure if comparing perplexity by conditioning on another different language (source language) is correct. The experiments would be clearer if done with standard LM with Transformers encoder model like BERT for example.\n* At the end of Section 2, the authors mention about \"generalizations\", but I couldn't really find any discussion about this in the paper. Maybe this can be clarified?\n* I found that claiming script bias in character models is too strong, if the experiment only shows bias in ZH (and this is expected since its character-level has different notion with languages with Latin script).\n* Byte-level: I found that there is still some \"erraticity\" in Figure 3(b) especially when the data size increases (which is more practical in real world application), so this is not entirely resolved?\n* I also think the summary in Section 1.2 stating that linguistic typological information is not necessary given \"statistical properties concerning sequence length and vocabulary\" is not necessarily valid since these two properties are the results from linguistic typology information.\n\n\nMissing references:\n1. From characters to words to in between: Do we capture morphology? Clara Vania and Adam Lopez. ACL 2017.\n2. Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss. Barbara Plank, Anders Søgaard and Yoav Goldberg. ACL 2016. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper study to what extent languages are hard to model by a conditional language-model based on information-theoretic measurements. 

Overall, the reviewers value the systematic and extensive controlled experiments present in the paper. However, the presentation of the paper makes it very hard to follow and reviewers all still complain that it is hard to understand the take-home message of the paper.  

Despite the reviewers also appreciate the authors' effort in improving the paper, submitting the revision, responding to the feedback, they still conclude that significant reorganizing and revising of the paper is needed before it can be published. 

In particular, the paper may be able to improve by backing up the empirical study with some linguistic phenomena or by a more careful rewriting in explaining and discussing the empirical results. 

Some other strong arguments such as "Our application of statistical comparisons as a fairness measure also serves as a novel rigorous method for the intrinsic evaluation of languages, resolving a decades-long debate on language complexity." may need to be carefully revised. In this particular example, it is unclear how this paper "resolve" the debate on language complexity by demonstrating a few experiments.  Several sentences like this one should be revised.  


Decision: Reject