Fairness in Representation for Multilingual NLP: Insights from Controlled Experiments on Conditional Language Modeling
Authors: Ada Wan
Venue: ICLR 2022 Spotlight (2022)
Keywords: fairness
evaluation
multilingual NLP / multilinguality
representation learning for language data
statistical comparisons
Double Descent
conditional language modeling
data-centric approach
diversity in AI
morphology
Transformer
meta evaluation
visualization or interpretation of learned representations
character encoding
internationalization and localization
robustness
statistical science for NLP
science in the era of AI/DL (AIxScience)
transdisciplinarity

Abstract: We perform systematically and fairly controlled experiments with the 6-layer Transformer to investigate  the hardness in conditional-language-modeling languages which have been traditionally considered morphologically rich (AR and RU) and poor (ZH). We evaluate through statistical comparisons across 30 possible language directions from the 6 languages of the United Nations Parallel Corpus across 5 data sizes on 3 representation levels --- character, byte, and word. Results show that performance is relative to the representation granularity of each of the languages, not to the language as a whole. On the character and byte levels, we are able to eliminate statistically significant performance disparity, hence demonstrating that a language cannot be intrinsically hard. The disparity that mirrors the morphological complexity hierarchy is shown to be a byproduct of word segmentation. Evidence from data statistics, along with the fact that word segmentation is qualitatively indeterminate, renders a decades-long debate on morphological complexity (unless it is being intentionally modeled in a word-based, meaning-driven context) irrelevant in the context of computing. The intent of our work is to help effect more objectivity and adequacy in evaluation as well as fairness and inclusivity in experimental setup in the area of language and computing so to uphold diversity in Machine Learning and Artificial Intelligence research. Multilinguality is real and relevant in computing not due to canonical, structural linguistic concepts such as morphology or "words" in our minds, but rather standards related to internationalization and localization, such as character encoding --- something which has thus far been sorely overlooked in our discourse and curricula. 

Published as a conference paper at ICLR 2022
FAIRNESS IN REPRESENTATION FOR MULTILINGUAL NLP: INSIGHTS FROM CONTROLLED EXPERIMENTS ON CONDITIONAL LANGUAGE MODELING
Ada Wan University Zurich ada. wan@uzh. ch
ABSTRACT
We perform systematically and fairly controlled experiments with the 6-layer Trans- former to investigate the hardness in conditional-languann language-modeling languages which have been traditionally considered morphologically rich (AR and RU) and poor (ZH). We evaluate through statistical comparisons across 30 possible language directions from the languages of the United Nations Parallel Corpus across 5 data sizes on 3 representation levels character, byte, and word. Results show that performance is relative to the representational granularity of each of the languages, not the language eliminate statistically significant performance disparity, hence demonstrating that a language cannot be intrinsically hard. The disparity that mirrors the morphological complexity hierarchy is shown to be a byproduct of word segmentation. Evidence from data statistics, along with the fact that word segmentation qualitatively inde- terminate, renders decades-long debate on morphological complexity (unless it is being intentionally modeled a word-based, meaning-driven context) irrelevant the context of computing. The intent of our work is to help effect more objectivity evaluation as well as fairness and inclusivity in experimental
whole. On the character and byte levels, are able
and adequacy setup in the area of language and computing Learning and Artificial Intelligence research. Multilinguality is real and relevant computing not due to canonical, structural linguistic concepts such morphology or ""ordd"" in our minds, but rather standards related to internationalization and localization, such character encoding something which has thus far been sorely overlooked in our discourse and curricula.
to uphold diversity in Machine
INTRODUCTION
1.1
BACKGROUND AND MOTIVATION
Most current work on fairness in Machine Learning (ML) and Natural Language Processing (NLP) focuses the societal biases encoded in natural language data that are propagated and amplified when they are used at scale for/as Artificial Intelligence (AJ) solution But little has been said or questioned about the bias, in, the favoring of certain outcomes, implicit in our theoretical/scientific assumptions that results in the varying performance of different languages in computing.
Disparity in machine translation results For instance, results reported in Junczys- Dowmunt
et al. (2016) for Phrase-Basee Statistical Machine Translation (PBSMT) (Koehn et al. (203)) and
Neural MT (Bahdanau al. 2014) on the 6 official languages" of the United Nations (UN) Parallel Corpus (Ziemski et al. (016) indicate a disparity between EN/ES/FR and AR/RU/ZH in BLEU translation performance in the latter group is generally worse, regardless of the MT algorithm used. AR and RU are traditionally considered morphologically complex (see e.g. Minkov et al. (0077) Seddah et al. (2010) and proceedings of related workshops in subsequent
(Papineni et al. aal]2200)
see e.g. work from conference (https: /faacttc/ ference. and workshops in previous years on "Faimess, Accountability, and Transparency" (FAccTT)
Arabic (AR), English (EN), Spanish (ES), French (FR), Russian (RU), and Chinese (7H)

---

Published as a conference paper at ICLR 2022
years for Statistical Parsing of Morphologically Rich Languages), and ZH morphologically frugal (for its lacking determiners and plural or tense markers) [Koehn] 2005). While Koehn (2005) found translating into EN to be easier than into morphologically rich languages based on word-level BLEU scores from PBSMT systems of 10 language directions from the 11 Europarl languages then, Bugliarello al. (220)) found it is easier to translate out of EN than into it based on 21 Europarl languages in BPEs (Byte Pair Encodings) (Sennrich et al. 2016) with the Transformer (Vaswani et al. 2017) in a new metric, cross-mutual information.
Disparity in language modeling results Disparate performances across different languages seem to have been implicitly accepted in that it is often believed that some languages are harder to model than others. [Bender (2009) advocated the relevance of linguistic typology for the design of language- universal NLP systems due to differences based on crosslinguistic structural notions, such as parts of speech and morphological complexity. Cotterell al. (2018) studied (monolingual) language models (LMs) on the 21 Europarl languages using a word-level 7-gram standard Kneser & Ney (1995) model and LSTM-LMs (Sundermeyer al. 2012) with characters and lemmatized forms in informatiin-thhorree the differences in performance. MMilkee al. (2019) extended the coverage to 69 languages with the multilingual Bible corpus (Mayer & Cysouw| 2014), tested on RNN-LMs (an implementation of LSTM (Hochreiter & Schmidhuber] 1997)) with characters and BPEs, but concluded that basic data statistics in vocabulary size (VV) and sequence length were the most predictive performance features.
terms, and found morphological complexity to be the primary culprit for
We noticed, however, a discrepancy in the results from Mielke et al. (2119) for ZH- it came out as the least difficult for the character model, but it is the 6th most difficult language for the BPE model. As different input representations have been tested with different architectures with divergent results in different metries in previous studies, each of them only testing with one data size, decided to investigate the matter more systematically once again with statistical comparisons of score distributions between languages.
1.2 RESEARCH QUESTIONS AND CONTRIBUTIONS
Research questions Are there any statistically significant differences in hardness when it comes to Langdttiooa-Langgggg---ooo
(CLMing) languages which have been traditionally considered morphologically rich (AR and RU) and poor (ZH) with the 6-layer Transformer? Is morphological complexity inherent in language? When is the notion of morphological complexity relevant in computing?
Summary of findings and insights Based on our bilingual CLMing setup with the UN Parallel Corpus in the data size range from 102 to 10 lines on the character, byte, and word levels, we find:
1. Language has many finer-grained dimensions with different representations and learning patterns. Hardness in modeling is relative to its representational granularity (representation relativity).
2. There is neutralization of source language instances, i.c. there are statistically significant differences between source language pairs. Only pairs of target languages differ significantly.
3. On the character and byte levels, hardness is correlated with statistical properties concerning sequence length and [V[ of a language, regardless of its morphological profile. As it is possible to climinate performance disparity by decomposing sequences into finer-grained units in characters and bytes, we show that morphological complexity is not an intrinsic property of language. Unless word-based methods are used, unless we implement/model it explicitly, the notion of morphological complexity is irrelevant in computing.
4. On the word level, hardness correlated with [V. and a complexity hierarchy arises through the manual preprocessing step of word tokenization. This complexity/disprriy effected by word segmentation can be improved by subword tokenization but cannot be eliminated due to the fundamental qualitative differences in the definition of a "word" being one that neither holds universally nor is suitable/consistent for fair crosslinguistic comparisons.
5. Representational units of finer granularity can help close the gap in performance disparity.
Orthogonal to our main research questions, we also observed types of sample- sample-wise non- monoooniiitt Double Descent (Belkin et al. Nakkiaaa et al.) (0200) and erraticity. For reasons due length and scope for this paper, we will defer discussions and analyses of these beyond what is addressed in to future work.
2

---

Published as a conference paper at ICLR 2022
Outline of the paper In we define method and experimental setup. We present our results and analysis on the primary representations in $ and those from the secondary set of controls in $ in progressive manner to case understanding. Meta analysis on performance disparity and other discussions are in $ 5
2 METHOD AND DEFINITIONS
Conditional language modeling CLMing is the modeling of the probability of the next token, given the history of the preceding tokens and conditioning context. In our case, such conditioning context a line from the source language. To explicitly focus on modeling the complexities that may or may not be intrinsic the languages, we study the more fundamental process of CLMing without performing any translation. This allows us to eliminate confounds associated with generation and other evaluation metrics. One could think of our setup as estimating conditional probabilities with the Transformer, with a bilingual (one-to-one) setup where the perplexity (PP) of one target language (Irg) is estimated given the parallel data one source language (lsr), where laxe * lug. We focus on the very basics and examine the first step in our pipeline - input representations, holding everything else constant. Instead of measuring absolute cross-entropy scores at one data size, we evaluate the relative differences between development (dev) set score distributions between languages.
Controlled experiments as basic research for scientific understanding of language data Using the UN Parallel Corpus, the data from which the MT results in JunczyysDowmunt et al. (2016) stem, we perform a series of controlled experiments with the Transformer, holding the hyperparameter settings for all 30 one-to-one language directions from the languages constant. We control for size (from 10a to 10t lines) and language with respect to representational granularity. We examine 3 character, byte (UTF-8), and word, and upon encountering some unusual phenomena, we perform a secondary set of controls with 5 alternate representations on the character level: Pinyin and Wubi (ASCII representations for ZH phones and character strokes, respectively), on the byte level: code page 1256 (for AR) and code page 1251 (for RU), and on the word level: BPE, These symbolic variants allow us to manipulate the statistical properties of the representations, while staying "faithful" to the language possible. We adopt this symbolic data- centric Sentriddapproact approach because we would like to more directly interpret the confounds, if any, that make language data different from other data types. We operate on a smaller data size range this is more common in traditional language sciences and one of our higher goals is to bridge an understanding between language sciences and engineering (the latter being the dominant focus in NLP), and between traditional symbolic sciences and ML. We run statistical tests to identify the strongest correlates of performance and assess whether the differences between the mean performance of different groups are indeed significant. We are concerned not with the absolute scores, but with the differences between score distributions from different languages.
primary representation types/levels
Fair evaluation with multitexts Multitexts are multiway parallel corpora. The UN Parallel Corpus a 6-way parallel corpus consisting of manually translated UN documents from the 25-year period between 1990 and 2014. We use the UN Parallel Corpus because it contains languages conventionally regarded as morphologically rich and poor, has quality and size sufficient for evaluation, and more importantly, it comes raw texts (untokenized), unlike both of the corpora that Mielke et al. (019) used. Detokenization (esp. the evaluation thereof) is not a trivial task.
Fair information thhorreti evaluation metric using a cross-entropy loss, defined as:
Most sequeecce-t-sequeene models are optimized
H(t,s)= -- Σ
logz p(t: () 1=1
(1)
where t is the sequence of tokens to be predicted, t; refers to the 5th token in that sequence, is the sequence of tokens conditioned on, and N = f.. is customary to report scores as PP, which is 2 (tt,s), i.e. 2 to the power of the cross-entropy averaged by the number of tokens in the dev
Two testinglevaluation approaches data-centric: hold the algorithm constant and tweak data, vs. algorithm- centrie: hold data constant and tweak the algorithm.
3

---

Published as a conference paper at ICLR 2022
data. Cotterell et al. (2018) proposed to use "renormalized" PP to evaluate LMs tokenwise fairly by dividing the overall bits per utterance/sequence by one constant token count in any one arbitrary language (e.g. so arrive at "bits per character" in one language to evaluate all languages). But find that it is not necessary to assign a perspective that is centered on any one particular language, when
evaluate simply by the total number of bits for a larger portion of texts/sequences. This can be a fairer, more general and flexible way of evaluating data that has not been or cannot be perfectly segmented or aligned line by line. We hence used instead unnormalized PP, i.e. the total number of bits needed to encode the dev set (3,077 lines per language, after length filterin,, in our case). As the implementation we used only reports PP, we transformed it back to entropy as defined above via (t, 8) loga PP((() X N.
Disparity/Inequaltyy the context of our CLMing experiments, we consider there to be "dis- parity" or "inequality" between languages I and l2 if there are significant differences between the performance distributions of these two languages with respect to each representation. Here, by performance mean the number of bits required to encode the held-out data using a trained CLM. With 30 directions, there are 15 pairs of source languages (lac1, late2) and 15 pairs of target languages (rg1, possible. We compare the source languages among each other, and the target languages among each other. Each Ln or each luy consists of scores from all models trained across various sizes and directions. To assess whether the differences are significant, we perform unpaired two-sided significance tests with the null hypothesis that the score distributions for the two languages are not different. Upon testing for normality with the Shapiro- Wilk test (Shapiro & Wilk, 1965) Royston 1995), we use the parametric unpaired two-sample Welch's t-test (Welch) 1947) (when normal) or the non- non-parametric unpaired Wilcoxon test (Wilcoxon] 1945) (when not normal) for the comparisons. We use the implementation in R (R Core | 1)) for these 3 tests. To account for the multiple comparisons we are performing, we correct all p-values using Bonferroni correction (Benjamini & Heller 2008 Dror et al., 2017) and follow Holm's 's procedure (Holm 1979. Dror et al.) 2017) to identify the pairs of l and 12 with significant differences after correction. We report all levels of significance (a < 0.05, 0.01, 0.001) for more comprehensive overview. In contrast to Dror et al. (2017), which aimed to compare the performance of different algorithms, we compare languages (in the context of computing).
Experimental setup The systematic, identical treatment we give to our data is described as follows with further preprocessing and hyperparameter details in Appendices A and B respectively.
After filtering length to 300 characters maximum per line in parallel for the 6 languages, we made 3 subsets of the data with million lines each one having lines in the order of the original corpus (dataset A) and two other randomly sampled (without replacement) from the full corpus (datasets & C). Lines in all datasets are extracted in parallel and remain fully aligned for the languages. For each run and each representation, there are 30 pairwise directions (i.e. one lave to one lig) that result from the 6 languages. We trained all 150 (for 5 sizes) 6-layer Transformer models for each run using the SOCKEYE Toolkit (Hieber et al. 2018). We optimize using PP and use early stopping if no PP improvement occurs after 3 checkpoints up to 50 epochs maximum, taking the best checkpoint. Characters and bytes are supposed to mitigate the
(OOV) problem the word level. In order to assess the effect of modeling with finer granularity more precisely, all vocabulary items appearing once in the train set accounted for (i.e. full vocabulary on train, as in Gerz et al. (2018alb) b)). But we allow our system to categorize all unknown items in the dev set to be unknown (UNK) so to measure OOVs (open vocabulary on dev (Jurafsky & Martin 2009)). To identify correlates of performance, we compute Spearman's correlation (Spearman) 19044) with some basic statistical properties of the data (e.g. length, [V]. type-token-rattec OOV rate) as metrics complete list is provided in App. C See App. for sample construction for statistical comparisons.
3 EXPERIMENTAL RESULTS OF PRIMARY REPRESENTATIONS
Subfigures Ta 1b and 1c show the mean results across 12 runs of the 3 primary representations character, byte, and word, respectively. The x-axis represents data size in number of lines and the y-axis the total conditional cross-entropy, measured in bits (Eq. I]. Each line connects 5 data points corresponding to the number of bits the CLMs (trained with training data of 10a, 10 10
using implementation from http://p////.ibb..ooo////arrr//r
wiaobiiityaaaa iity-ana ysi NI.P
4

---

Published as a conference paper at ICLR 2022
MISSET MAMM
(a) CHAR
(b) BYTE
WORD
SAC
AR
EN
EN
ES
ES
FR
FR
RU
RU
ZH
(h)
(d) CHAR by target
(e) BYTE by target
(f) WORD by target
Figure 1: Number of bits (the lower the better) across data size from 102 to 10 lines plotted for all 30 directions. Suhfigures Ta Tb and show mean scores information respectively sorted shows the correspondence between colors and source languages, in Subfigure between line types and target languages. (These figures are also shown enlarged in Appendix F Please note that results pertinent to our first research question of this paper conceming statistically significant differences
12 Suhfigures le and deppicct the corresponding
facets by target language and with error bars. Legend in Subfigure [g
summarized in Table figures are visual aid only. We are concerned with the absolute scores but the distances between scores, 1.c. spaces between the sets of lines by lag. The point here is to show the differences in Transformer's overall learning patterns relative to the representational granularity.)
10 and 8 lines) needed to encode the target language dev set given the corresponding text in the source language. These are the same data in the same 30 language directions and 5 sizes with the same training regime, just pepprocessed/eegmeene relativity - hardness in modeling is relative to its representational granularity. Languages (or any objects being modeled) need be evaluated relative their representation. "One size does not fit all" Durrani et al.) 2019). Our conventional way of referring to "language" (as socio-cultural product or with traditional word-based approaches, or even for most multilingual tasks and competitions) too coarse-graine graiied for computing (see also Fisch et al. (2019) and Ponti et al. (2020)).
differently. This confirms representation
Subfigures Id Te and If display the corresponding information sorted into facets by target language, source languages represented as line types. Through these we see more clearly that results can be grouped rather neatly by target language - as implicit in the Transformer's architecture, the decoder unaware of the source language in the encoder. As shown in Table I in summarizinn the number of source and target language pairs with significant differences. there are no significant differences across any source language pairs. The Transformer neutralizes source language instances. This could explain why transfer learning or multilingual/zero-ee translation (Johnson et al.) 2017) is possible at all conceptual level.
In general, for character and byte models, most language directions do seem to converge at 10 lines similar values across all target languages, with few notable exceptions. There some fluctuations
5

---

Published as a conference paper at ICLR 2022
past 10*, indicating further tuning of hyperparameters would be beneficial due to our present setting possibly working most favorably at 10* On the character level, target language ZH (ZHug) shows a different learning pattern throughout. And on the byte level, ARug and RU display highly unstable behavior, which we refer to as erratic. Word models exhibit Double Descent across the board (note the spike 109), but overall, difficult/easy languages stay consistent, with AR and RU being the hardest, followed by ES and FR, then EN and ZH. A practical takeaway from this set of experiments: in order obtain more robust training results, use bytes for ZH (as suggested in Li al. (2019a) and characters for AR and RU (e.g. Lee et al. (2017) also if one wanted to avoid any "class" problems performance disparity with words. Performance disparity for these representations is reported in Table Junder "CHAR". "BYTE", and "WORD". Do note, however, that the intrinsic performance of ZH with word segmentation not particularly subpar. But this often does not correlate with its poorer downstream tasks results (recall results from Junczys- Seeryys-Doomunn Dowmunt et al. (2016)) Since the
notion of word in ZH is highly contested and ambiguous - i) it is often aimed to align with that in accommodate academic theories and manual feature engineering ii) there is great variation among different conventions, and iii) native ZH speakers identify characters as words that fairer and manguage-independeee processing in finer granularity is possible. Li et al. (2019b) questioned the necessity of CWS in Deep Learning (DL)-based ZH NLP and presented evidence in favor of character-based processing, including results from downstream NLP tasks. In Linguistics, Duanmu (2017) presented summary on the contested nature of wordhood in (Mandarin) ZH in relation to EN. more native account of ZH, however, despite a couple of stalects/varietic of it being considered high high-resource language, has not yet been fully recognized and accepted in NLP.
other languages
there reasons to rethink this procedure
UNDERSTANDING THE PHENOMENA WITH ALTERNATE REPRESENTATION
To understand why some languages show different results than others, we carried out secondary set of controlled experiments with representations targeting the problematic statistical properties of the corresponding target languages.
(a) Wubi
(b) Wubi by target
(c) Pinyin
(d) Pinyin by target
Figure 2: Character-level remedies for ZH: Wubi Pinyin.
Character level We reduced the high VV in ZH with representations in ASCII characters - Pinyin and Wubi. We replaced the ZH data these formats only on the target side and reran the experiments involving ZHug on the character level. Results in Figure and Table [ show that the elimination of
'It is "legacy interpretation" which stemmed from practical compromise from the carly days ZH NLP when the goal was to align with EN words for MT. Chinese word segmentation (CWS) has been a decades-lone -oon issue in text processing. But even in EN, for computing, the variability in "word" counts (from the trivial convention of whitespace tokenization) results in different bit counts, affecting file sizes. In NLP, such method of "word" counting brings about a high [V], hence different tokenization schemes have been designed to mitigate this problem. For humans, there is no consensus about the definition of wwords". Even for purely academic account, it is held to be indeterminate (see Haspelmath (2011) and references therein from the past century). Kilgarriff (1997 2014) pointed out that "words" and "word senses" and the number thereof, in terms of lexical entries for dictionaries, contextual and arbitrary.

---

Published as a conference paper at ICLR 2022
(a) Code page 1256 & 1251 (b) Code page by target
(c) BPE
(d) BPE by target
3b remedies with code page 1256 for target AR and 1251 for target RU, and word-level (Subfigures 3c & 3d) remedy with BPE for all languages.
Figure 3: Byte-level (Subfigures 3a
disparity on the character level possible if ZH represented through Pinyin (transliteration). as in Subfigure[2c] though at the cost of the native script information. Models represented through Wubi, an input algorithm that decomposes and matches these to 5 classes of radicals (Lunde 2008). display a behavioral tendency unlike those with other (phonetic) alphabetic script (Subligure 2ap, suggesting that this script/stroke pattern decomposes differently. But ZH the language is not outlier all around.
cetracteerinntera information into stroke shape and ordering
Byte level Length is the most salient statistical attribute that makes AR and RU outliers. To shorten their sequence lengths, tested with alternate encodings on ARgg and RUng - code page 1256 and 1251, which provide 1-byte encodings specific to AR and RU, respectively. Results are shown in Subfigures and Not only is erraticity resolved, the number of possible target language pairs with significant differences reduces from 8 with the UTF-8 byte representation to 0 (Table under "ARRU,"), indicating that we eliminated disparity with this optimization heuristic. Since our heuristic is lossless and reversible transform, it shows that a complexity that is intrinsic and necessary in language" does not exist computing, however diverse they may be, as our are, from the conventional linguistic typological, phylogenetic, historical, geographical perspectives.
Word level The main difference between word and character/byte models is length not being a top contributing factor correlating with performance, but instead [V] is. This is understandable word segmentation neutralizes sequence lengths. To remedy the OOV problem, we use BPE, which learss fixed vocabulary of variable-length character sequences (on word level, it presupposes word segmentation) from the training data. It is more mneggraieed than word segmentation and is known to better model subword units for morphologically complex languages. We use the same vocabulary of 30,000 specified in Junczys- Dowmunt et al. (2016). This reduced averaged OOV token rate by 90-100% across the sizes. The number of language pairs with significant differences reduced to from for word models. While BPEs are still not effective results show how finer-grained modeling contributes positively to closing the disparity gap.
character/byte variants, their
5 META RESULTS, ANALYSES, AND DISCUSSION
Performance disparity Table lists the number of language pairs with significant differences under the representations studied. Since is possible for character and byte models effect performance disparity for the same languages on the same data, a complexity intrinsic to language does not exist. fact, the customary expectation that languages ought perform differently created through our word segmentation practice. Furthermore, the order of AR/RU > ES/FR EN/ZH (Figure Ic resembles the idea of morphological complexity. Considering there are character-internal
6 whicc are sequences with an implicit/explici pattern made up of consonants and vowels "sside from its statistical properties related to length and vocabulary. To show something is not necessarily true, only 1 counter observation is needed.
7

---

Published as a conference paper at ICLR 2022
Table 1: Disparity Table Number language pairs out 15 with significant differences. with respective p-values. ARRU, refers to AR & RU being optimized only on the target side; whereas ,RRU,, denotes optimization on both source and target sides (relevant for directions AR- AR-RU and RU-AR).
CHAR
Pinyin
Wubi
BYTE
ARRU,
ARRU,
WORD
BPE
p-value
sre
trg
STC
trg
STC
trg
STC
ug
STC
tg
SrC
ug
STC
trg
STC
trg
0.05
0
7
0
4
0
8
0
9
0
4
0
4
0
11
0
10
0.01
0
5
0
2
0
6
0
8
0
3
0
4
0
8
0
8
0.001
3
0
0
0
8
0
2
0
0
7
Table 2: Target language pairs with significant differences indicate that the 2 languages are not equally/similarly good or equally/similarly bad. 15 (non-directional) language pairs total possible from 30 language directions,
p=0.001.
LANGesa PAIR
CHAR
Pinyin
Wubi
BYTE
ARRU,
ARRU,
WORD
BPE
AR EN
X
X
X
AR-RS
FN-BS
AR FR
X
EN FR
X
X
ES-FR
AR-RU
X
EN-RU
X
X
X
X
ES-RU
X
FR-RU
X
AR 2H
X
X
X
EN 2H
X
X
ES 2H
X
X
X
FR-20 2H
X X
X
X
RU-2H
X X
X X
X
meaningful units in languages with logographic script such as ZH (cf. Zhang & Komachi (2018) that are rarely captured, studied, or referred as "morphemes". this goes to show that linguistic morphology, along with its complexity, as it is practiced today"Jad that which has occurred in the NLP discourse thus far, has only been relevant on the "word" level, conceptually constrained by unstandardizable units such "words" (and "sentences"). The definition of word, however, has been recognized as problematic for a very long time in the language sciences (cf. Footnote 55.
While the lack of significant differences between pairs of source languages would signify neutraliza- tion of source language instances, it does not mean that source languages have no effect on the target. For our byte solutions with code pages, experimented also with source side optimization in the directions that involve AR/RU as source. This affected the distribution of the disparity results for that representation with 2 pairs being significantly different (see Table under "ARRU,,")) "). We defer further investigation on the nature of source language neutralization to future work.
Target language pairs with significant differences are summarized in Table 2 We show that mor- phological complexity can be empirically eliminated in this one-setting-for-lll configuration with 6-layer network, no hyperparameter tuning, and a maximum line length of 300 characters (and its corresponding equivalence in other representations) as constrained by our hardware and compute time listed App.[A]aam current data availability. A more analytical solution can be obtained through data statistics (see App. Eb. A conceptual solution lies in the definition of "words"" and morphology.
Sample-wise Double Descent (DD) Sample-wise non- oon-monotoniciyyDDD NNakkiran et al. 2020 denotes a degradation followed by an improvement in performance with increasing data size. We notice word models and character models with ZH i.e. models with high target [V], are prone
But there are no reasons why we cannot adopt statistical science of language in finer granularities heyond/withouts "wwords", with standardized units (characters/hytes) and/or continuous representations. Resources, both data science and MI. interpretation and evaluation well.
e.g. quality parallel data or contrast sets,
8

---

Published as a conference paper at ICLR 2022
to exhibit spike at 10 A common pattern for these is the ratio of target training token count to number of parameters falls into O(10 4) for 102 lines, O(10 at 10 0(10 at 104, and O(10-1) for 10th lines and on. But for more atomic units such as alphabetic (not logographic) characters (may it be Latin, Cyrillie, or Abjad) and for bytes, this progression instead begins at 0(10~9) at 102 lines. Instead of considering this spike of 10 irregular, we may instead want to think of this learning curve as shifted by 1 order of magnitude to the right for characters and bytes and/or the performance at 10a lines for words and ZH-characters due being overparameterized and hence abnormal. This would also fit in with the findings by Belkin et al. (2019) and Nakkiran et al. (2020) attributing DD to overparameterization. While almost all work attribute DD algorithmic reasons, findings from Chen et al. (2020) corroborate our observation and confirms that DD arises due to "the interaction between the properties of the data and the inductive biases of learning algorithms". Other related work on the DD phenomenon and its development can also be found in their work.
Erraticity We observe another type of sumple-wise non-monotonicity one that signals irregular and unstable performance across data sizes and runs. Within one run, erraticity can be observed directly as changes direction on the y-axis. Across runs, large variance can be observed, even with the same dataset. Erraticity can also be observed indirectly through negative correlation between data size and performance. Much work on length bias in NMT have focused on solutions related to search, e.g. Murray & Chiang (2018). Our experiments show that a kind of length bias can surface already with CLMing, without generation taking place.
Additional related work That basic data statistics are the driver of success in performance in multilingual modeling has so far only been explicitly argued for in MMelke et al. (2019). We go beyond their work in monolingual LMs to study CLMs and evaluate also in relation to data size, representational granularity, and quantitative and qualitative fairness. To the best of our knowledge, there has been no prior work on demonstrating the neutralization of source language instances through models, the meta
statistical comparisons, a numerical analysis on DD for sequence-to-sequeee phenomenon of a sample-wise conmooottniyctty (erraticity) being related to length.
6 CONCLUSION
Summary We investigate whether the performance disparity between languages which have been traditionally considered morphologically rich (AR and RU) and poor (ZH) in the 6-layer Transformer CLM due to morphological complexity is justified and find that it is not. Performance disparity can be explained by data statistics and in the context of computing, it can be eliminated by optimization on length and [V] through character/byte representations. In fact, morphological complexity is not a necessary concept in computing because "word" is not necessary concept in computing, unless we make so through word segmentation. A morphological complexity hierarchy can result simply through word segmentation. Furthermore, there are many possible interpretations to "words" for humans and since morphology is defined with the concept of "word", there is no stable ground for assessing this complexity. Representational units of finer granularity were shown to help eliminate performance disparity though at the cost of longer sequence length, which can have a negative impact on robustness. In addition, we found all word models and character models with ZHag to behave similarly in their being prone to exhibit peak (as sample-wise DD) around 10s lines in our setting.
Outlook ML has enabled greater diversity in NLP (Joshi al. 2020 Fairness, in the elimination of disparity, does not require big data. This paper made a pioneering attempt to bridge research in NNs/DL, language sciences, and language engineering through data-centric perspective. Multilin- guality is real and relevant in computing not due to canonical, structural linguistic concepts such as morphology or "words" in our minds, but rather standards related to internationalization and localization, such as character encoding our discourse and curricula. We also believe that a more fine-grained statistical data science can well complement algorithmic analyses with a view that is more empirically robust (i.e. experimentally verifiable) and more relevant to machine processing, contributing to more generalizable and inter- pretable pool of knowledge for ML/NNs/DL. A more comprehensive study can lead us not only to new scientific frontiers, but also to better designs and evaluation, benefitting the development of a more general, diverse and inclusive AI.
something which has thus far been sorely overlooked in

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT: FAIRNESS CONCERNS FOR MULTILINGUALITY
Clearer nomenclature If/When the intent is not explicitly model linguistic morphology in computing, one can simply describe languages and their statistical profiles with respect to their representational granularity in characters or bytes (which are and/or can be exhaustively standardized in computing, or refer to sequences as longer/Shorter or having a higher/lower vocabulary size when comparing them with each other, rather than ""cooer""pporer" based on concepts (e.g. "words", "sentences") ") that can be ambiguous, contested, and inaccessible to many.
Accessibility Language communities who are unfamiliar with languages similar to dominant languages or those who are reluctant to conform to one structurally similar interpretation should not have to feel inadequate in processing their own language "from scratch", if they choose. As technologists, we can help take equitable mcasures to make fairer data representations and infrastructure available. A "word"-free view of language preempts linguistic/cultural hegemony and such an interpretation would also help make the analyses language data more objective and clearer.
Scarcity of quality and/or multiway data for science, evaluation, and documentation With the rise of multilingual models comes an alleged decrease in reliance on parallel corpora for MT. But data, esp. high-fidelity/quaaitt textual and multimodal multiway parallel data, play not only an important role in scientific research, but also one for historical/cultural documentation. And as this paper shows, they can also serve as evaluation data for MI. models for better understanding and interpretation. challenge sets, data for machine processing would need to be statistically diverse and challenging. Parallel data from previous years have often come in the form of bitexts (2-way parallel text data), usually "word" -tokenized where real length information has been compromised. (The Bible data from Mayer & Cysouw (2014) came with another confound the ZH numeral ("one") is recognized as a dash (punctuation) and hence tokenized with surrounding whitespaces.) At the UN Parallel Corpus was the only unperturbed, fully
the time of our present study multiway parallel data sufficient for reliable evaluation for our size range. Data for science, evaluation, and documentation require long-tenu stable platform(s) and support. There are many forms of data science. But in terms of having a sustainable practice to collect and curate good data, and exercising enough of science with that data so to improve collective intelligence and mutual understanding. instead of looking at data from an utilitarian point of view only for consumer application purposes, there seems be room for improvement still. We hope our work could help effect a positive change in this direction 10
ACKNOWLEDGMENTS
The author thanks all anonymous reviewers and PCs well the following individuals for their informative comments on previous version(s) of this paper (names in alphabetic order by last name): Benjamin Börschinger, Jordan Boyd-Graber, Christian Buck, Kyunghyun Cho, Kenneth Church, Miryam de Lhoneux, Rotem Dror, San Duanmu, Chris Dyer, Roman Flury, John Goldsmith, Yannic Kilcher, Sandra Kübler, Thomas McColgan, Jason Naradowsky, Ryokan Ri, Anna Rogers, Mark Rowan, Nikunj Saunshi, Rico Sennrich, Ekaterina Vylomova, Jason Weston, and Kie Zuraw.
We also thank Timothy Baldwin, Kevin Duh, Daniela Gerz, Kenneth Heafield, Felix Hieber, Marcin Junezys-Dowmunt, Gal Kaplun, Dan Klein, Sabrina Mielke, Mathias Müller, Nakkiran Preetum, and Annette Rios for their kind responses in correspondence concerning related work and software. We thank Ismail Moukadiri for his help with the Arabic language.
Some initial experiments were supported by the computing resources at the Department of Informatics and Computational Linguistics at UZH and the Institute of Neuroinformatics at UZH and ETH Zurich (for which we especially thank Richard H.R. Hahnloser, Nikola I. Nikolov, Yuhuang Hu, and Pawel Pyk).
9 "ooe that "high-quality" does not nccessarily mean "clean(ed)" data - it would depend on the situation/task.
Pleaas further discussion in Wan 2022
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Dzmitry Bahdanau. Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. arXiv e-prints,
September 2014. URL https
7/arxiv. org/ aataabs11455 1409. 0473
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32): (16632)::589-110 1589991-4449944444,, 2019. ISSN 0027-8424. doi: 00.1073/paas.19000700 URL
https: www phas org/ wwwt//ttttnnccccccecc 92/25444
15849
Emily M. Bender. Linguistically naïve := language independent: Why NLP needs linguistic ty- pology. In Proceedings of the EACL 2009 Workshop on the Interaction between Linguistics and Computational Linguistics: Virtuous, Vicious or Vacuous?, pp. 26-32, Athens, Greece, March 2009, Association for Computational Linguistics. URL https:: www www.wa..wwwbrrrgg..oo
anthologyy/W99-0105
Yoav Benjamini and Ruth Heller. Screening for partial conjunction hypotheses. Biometrics, 64()): jst.o.org/sstt.o.eoe.oe..
2215-1222. 2008. ISSN 0006341X, 15410420. [RLhttp:: 25502204
Emanuele Bugliarello, Sabrina J. Mielke, Antonios Anastasopoulos, Ryan Cotterell, and Naoaki Okazaki. It's s easier to translate out of English than into it: Measuring neural translation difficulty by cross-mutual information. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 1640-1644 Online, July 2020. Association for Computa- tional Linguistics. doi: 10. 10.18655//1/2020.all main. 149. URL http:tts::/// www.wwwwwwwww......wwwwww.w org/
anthology/ watpoll0yyy2222.cccooo -aa.aaaclmaaa... m.
Lin Chen, Yifei Min, Mikhail Belkin, and Amin Karbasi. Multiple descent: Design your own generalization curve, 2020.
Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang. Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv: 1 2015.
Colin Cherry, George Foster, Ankur Bapna, Orhan Firat, and Wolfgang Macherey. Revisiting caaracter-based neural machine translation with capacity and compression. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4295-430s Association for Computational Linguistics, 2018. URL http: http:///awwwb...... org/ anthologh/
D18- 1461
Ryan Cotterell, Sabrina J. Mielke, Jason Fisner, and Brian Roark. Are all languages equally hard to language-model? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Short Papers), pp. 536-541, New Orleans, Louisiana, June 2018 Association for Computational Linguistics. doi: 10. 10.18653/11/N18-220 N18-2085)
URL https: /www www.wb.ww..ccc.... org/ http///tthhhyg/y////oo
Rotem Dror, Gili Baumer, Marina Bogomolov, and Roi Reichart. Replicability analysis for natural
language processing: Testing significance with multiple datasets. Transactions of the Association for Computational Linguistics, 5:471 2017. htLt:tt:/ /ttp://acwwwe. org/ anthology
017-1033
San Duanmu. Word and wordhood, modern. In Rint Sybesma (ed.), Encyclopedia of Chinese Language and Linguistics. pp. 543 549. Brill, 2017.
Nadir Durrani, Fahim Dalvi, Hassan Sajjad, Yonatan Belinkov, and Preslav Nakov. One size does not fit all: Comparing NMT representations of different granularities. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 1504-1516, Minneapolis, Minnesota. June 2019. Association for Computational Linguistics. doi: 10.18653/v 10.18653/v1N19. 1154.
URL https::///........... www..www.mmmmmmmm...o..... a c web. www.org//attoolooooooooooooooooo ogy / 9-1154

---

Published as a conference paper at ICLR 2022
Adam Fisch, Jiang Guo, and Regina Barzilay. Working hard or hardly working: Challenges of integrating typology into neural dependency parsers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLL pp. 5713-5719, Hong Kong, China, November
2019. Association for Computational Linguistics. doi: 10. 10.18653/11/D11-1557 URL https
org/ wwtpg//ggatggg/g/.n.aeeo
Daniela Gerz, Ivan Vulié, Edoardo Ponti, Jason Naradowsky, Roi Reichart, and Anna Korhonen.
Language modeling for morphologically rich languages: Character-awarc modeling for word-level prediction. Transactions of the Association for Computational Linguistics, 6:451 2018a. doi: 10.1 12.116//acc_a, __00222..00 URLbttps: www. beclweb..o org/ wttp:////aa..ooo//ooooo/ooe
Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen. On the relation between linguistic typology and (limitations of) multilingual language modeling. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 316-327, Brussels, Belgium, October-November 2018b. Association for Computational Linguistics. URL
Linguitstss
https: wwwwwwlwwwbb... wwt.://wwww..rr/aaooooooo.o..//a
Martin Haspelmath. The indeterminacy of word segmentation and the nature of morphology and syntax. Folia Linguistica, 2011.
Felix Hieber, Tobias Domhan, Michael Denkowski, David Vilar, Artem Sokolov, Ann Clifton, and Matt Post. The Sockeye neural machine translation toolkit at AMTA 2018. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers), pp. 200-207, Association for Machine Translation in the Americas, 2018, URL http:///clweet org/ antaa logy 18800
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 988: httptt::
1780, November 1997. ISSN 0899-7667. doi: www..//..............
wrg.oo./0..... 11 62/ www.:/ec.... 1997. 1735]
Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statisties, 6(2):65 70, 1979. ISSN 03036898, 14679469. URL htttp://wwl. jstor /rg/
stable/461573
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's 's multilingual neural machine translation system: Enabling zero-shot transla- tion. Transactions of the Association for Computational Linguistics, 5:339-351, 2017. doi:
10. 10.1162/tac_ a 00065.005 BRLhttpss:/
acwweb. org/ /tt.o/ocy/11111122224
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp- 6282-6293, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/20200 http://222.2ccll//// main.m....560. URL https: wwwp///wwwwwww wwwwwb.ww.b......... org/ anthology/ 200022200..0 acl main. 560.
Marcin Junczys-Dowmunt, Tomasz Dwojak, and Hieu Hoang. Is neural machine translation ready for deployment? A case study on 30 translation directions. In /WSLT 2016, Seatlle October 2016. URL httptt::
neuraal- mi crosoft. com /en////nss///e//ee http-/uu//eeeercc///uuuluoe cat on/ 30-1
connslationn-ireattiii
Daniel Jurafsky and James H. Martin. Speech and Language Processing: An Introduction Natural Language Processing, Computational Linguistics, and Speech Recognition. Pearson Prentice Hall, second edition, 2009.
Adam Kilgarriff. "I don't believe in word senses". CoRR, cmp-lg/9712006, 1997. URL http:
7/arxiv or.oor//aaa wtgg/ab///cttt http://rrr/aba/mmmggg//9/2200000
Adam Kilgarifff. "How many words are there?". 2014. URL http:://wwwww..tttchnnnnnnn.
ht//ppcconnten//pllolaeto Many Words_ 2014.pdff
12

---

Published as a conference paper at ICLR 2022
Reinhard Kneser and Hermann Ney. Improved backing-off off for m-gram language modeling. In 1995 International Conference on Acoustics, Speech, and Signal Processing, volume 1, pp. 181-184. IEEE, 1995.
Philipp Koehn. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pp. 79-86, Phuket, Thailand, September 13-15 2005. URL
/ettps:/aaclane
org. 2005 mt summi t papers .1
Philipp Koehn, Franz J. Och, and Daniel Marcu. Statistical thrase-based translation. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pp. 127-133, 2003. URL URL http:::t/////t /wwwn aclweb
org/antho Logy /N0y//N3-110 1017
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison- Burch,, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pp- 177-180. Association for Computational Linguistics, 2007. URL (tLLhtt::// /ac 1 web wwwpee..rgg//tthhoooooooooooooo.
P P07-2045
Jason Lee, Kyunghyun Cho, and Thomas Hofmann, Fully character-level neural machine translation without explicit segmentation. Transactions of the Association for Computational Linguistics, 365-378, December 2017. doi: 10. 1a11662/aal_a_000666
www. ac www.ww.........................w org/
anthology/Q17-1022
Bo Li, Yu Zhang, Tara Sainath, Yonghui Wu, and William Chan. Bytes are all you need: End- to-end multilingual speech recognition and synthesis with bytes. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5621-5625. IEEE, 2019a.
Xiaoya Li, Yuxian Meng, Xiaofei Sun, Qinghong Han, Arianna Yuan, and Jiwei Li. Is word segmentation necessary for deep learning of Chinese representations? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3242-3252, Florence, Italy, July 2019b. Association for Computational Linguistics. doi: 10. 1111866/1//119-1114 URL
https: www. , acl www.wweebbcoo org/ anthology loyy 1atp:/19111111
Ken Lunde. CJKV Information Processing. O'Reilly Media, Inc., 2nd edition, 2008 ISBN 0596514476, 9780596514477
Thomas Mayer and Michael Cysouw. Creating a massively parallel Bible corpus. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014), pp. Reykjavik, Iceland, May 2014. European Languages Resources Association (ELRA).
Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. What kind of language is hard to language-model? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4975-4989, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/P19- 14911 URL https: / www. ac web
http/anthoologg/PP131133
Einat Minkov, Kristina Toutanova, and Hisami Suzuki. Generating complex morphology for machine translation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp. 128-135, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL lnttt::::/// http::::/wwww... www.ww.wllwe.. acwweb................. htt.:eeww....aaa.eeeeeeee wwwpo//ag.tgog/o/ooooooooooooe/e
1017
Kenton Murray and David Chiang. Correcting length bias in neural machine translation. In Proceed- ings of the Third Conference Machine Translation: Research Papers, pp. 212-212222 212-223, Belgium, Brussels, October 2018. Association for Computational Linguistics. doi: 10.18653/w1/W188 6322.. URL http::tttt//wtt www......./............/oo/////.
C. web. orgg www.oooooyyoooooo / 8- 6322
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang. Boaz Barak, and llya Sutskever. Deep double descent: Where bigger models and more data hurt. International Conference on Learning Representations, 2020. httphhtppa//opeeeviieeeeee
13

---

Published as a conference paper at ICLR 2022
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: method for automatic the evaluation of machine translation. In Proceedings of the 40th Anmual Meeting the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/1073083.10733 1073135. URL https: www. ac web
org/antho Sannthooooy/PO02-100
Edoardo Maria Ponti, Helen O'Horan Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. Modeling language variation and universals: A survey on typological linguistics for natural language processing, 2020.
Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2014. URL http: www. R projectt - org/
Patrick Royston. Remark as r94: A remark on algorithm as 181: The w-test for normality. Journal of the Royal Statistical Society. Series C (Applied Statisttics) 4444)5547-, 1995. ISSN 00359254, 14679876. URL BRLLhtpp:/// //ww: wwww. jstor.
Djame Seddah, Sandra Kuebler, and Reut Tsarfaty (eds.). Proceedings of the NAACL HLT 2010 First Workshop on Statistical Parsing of Morphologically Rich Languages, Los Angeles, CA, USA, June 2010. Association for Computational Linguistics.
anthology/Wii WW1 1400
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers). pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguisties. doi: 10.18653/v1/P16-111 1162. URL https: /www aclweb
org/antho 1 Logy Pttp/P11611666 1162
S. Shapiro and M. B. Wilk, An analysis of variance test for normality (complete samples) Biometrika, www.:oo..rrg.... https://dot:
12 1965. ISSN 0006-3444. doi: 10. 10.1093/biomee/52.33.55050 URL
C. Spearman. The proof and measurement of association between two things. The American Journal of Psychology. 15(1):72-101, 1904. ISSN 00029556. URL http:: //wp:.//www... /ttp://twwwtrorroo////s//.//.... oxc/stable/1412155
Martin Sundermeyer. Ralf Schlüter, and Hermann Ney. Lstm neural networks for language modeling. In INTERSPEECH, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5998 6008. Curran Associates, Inc., 2017. URL http: 7/papers www..ipppccc.mc... wttpssppppppppppeeeeeteeeeeec.e http:/pppwe..aaaaaaataatattaaoa. cettttiionnnslllle- you- need. pdf
Ada Wan. A statistical typology of language in finer granularity with parallel data. 2022.
B. L. Welch. The Generalization of 'Student's Problem when Several Different Population Variances are Involved. Biometrika, 34(1 (2:11122:228838) 35, 01 1947. ISSN 3444. doi: 10. 10.1099/biooeer34.112 28. httpp::/t///di..... http://ii.orgg//// 1093/b99iiiic omet 34 1-2.28
Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80-83, 1945.
ISSN 00994987.
Longtu Zhang and Mamoru Komachi. Neural machine translation of logographic language using sub-character level information. In Proceedings of the Third Conference on Machine Translation: Research Papers. pp. 17-25, Belgium. Brussels, October 2018. Association for Computational wtwp:e/...ggaaaa.ooooooooooooooo
Linguistics. URL lht www wwwwwb.ww.......... org/ ant holggy /wll W18 6303
Michal Ziemski, Marcin
and Bruno Pouliquen. The United Nations Parallel Corpus v1.0. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Tenth International Conference Language Resources and Evaluation (LREC 2016), Paris, France, may 2016. European Language Resources Association (ELRA). ISBN 978-2-9517408-9.1
14

---

Published as a conference paper at ICLR 2022
APPENDICES
DATA SELECTION AND PREPROCESSING DETAILS
B HYPERPARAMETER N
17
C CORRELATION STATISTICS
18
STATISTICAL COMPARISONS
19
E DATA STATISTICS
20
F ENLARGED
30 LANGUAGE DIRECTIONS (AGGREGATE
FROM ALL RUNS)
24
15

---

Published as a conference paper at ICLR 2022
A DATA SELECTION AND PREPROCESSING DETAILS
The UN Parallel Corpus v1.0 ZZiemski et al. 2016) consists of manually translated UN documents from 1990 to 2014 in the 6 official UN languages. Therein is a subcorpus that is fully aligned by line, comprising the 6- 6-way parallel corpus use. We tried to have as little preprocessing or filtering as necessary to eliminate possible confounds. But as the initial runs of our experiment failed due to insufficient memory on single GPU with 12 GB VRAM we filtered out lines with more than 300 characters in any language in lockstep with one another for all the 6 languages such that the subcorpora would remain parallel, thereby keeping the material of each language semantically equivalent to one another. 8,944,859 lines for each language were retained as our training data which cover up to the 75th percentile in line length for all 6 languages. In order to monitor the effect of data size, we made subcorpora of each language in 5 sizes by heading the first 10², 102, 10*, 10 10 [2 We refer to this as dataset A. In addition, to better understand and verify the consistency of the phenomena observed, we made 2 supplemental datasets by shuffling the 8,944,859 lines two different times randomly and a the number of lines in our sizes for each language, again in lockstep with one another (datasets B and C).
[
The systematic training regime that we gave to our language directions is identical for all and we controlled also for seeds. For each of the 3 primary representations character, byte, and word, we performed:
5 runs in sizes (102 - -10 lines): AO (seed=13), BO (13), CO (9948), A1 (9948), A2 (265), and 7 more runs in 4 sizes (102 10 lines): A3 (777), A4 (42), A5 (340589), A6 (1000), A7 (83146), B1 (9948), & C1 (13).
Figure I shows results from all 12 runs in all sizes for the primary representations.
For the alternate/secondary representations,
performed 3 runs each in 5 sizes (102-106 lines) (A0, BO, & CO). Due to limitations in computing resources, we were not able to perform as many runs as the primary representations. But important for our statistical comparisons that we evaluate based on an equal number of runs and on the same data for all candidates. Tables T & 2 are the results.
For each run and each size, there are 30 pairwise directions (i.e. 1 source language target language. e.g. AR-EN for Arabic to English) that result from the 6 languages. We trained all 150 jobs (30 directions x sizes) for each run and representation using the Transformer model (Vaswani et al.. 2017) as supported by the SOCKEYE Toolkit (Hieber et al. 2018) (version 1.18.85), based on MXNet
(Chen et al. 20155. A detailed description of the architecture of the Transformer can be found in
VVaswani et al. 2017). The same set of hyperparameters applies to all and its values are listed in
Appendix|B B
For character modeling, we used dummy symbol to denote each whitespace. For byte, turned each UTF-S-encoded character into a byte string in decimal value, such that each token is a number between 0 and 255, inclusive. For word, we followed (Jnczys-Dowmung et al. 2016) and used the Moses tokenizer (Koehn et al. (007) as is standard in NMT practice when word tokenization is applied and Jieba for segmentation in ZH.
Pinyin is a romanization of ZH characters based on their pronunciations and Wubi an input algorithm that decomposes character-interall information into stroke shape and ordering and matches these to 5 classes of radicals (Lunde) 2008). For Pinyin, we used the implementation from https / //github. com/ http//yyu/ppnnnnn in the numerical format such that each character/syllabe is
"'GUUs used for experiments in this paper range from a NVIDIA TITAN RTX (24 GB), NVIDIA GeForce RTX 2080 Ti (11 GB), a GTX Titan X (12 GB). to a GTX 1080 (8 GB). All jobs were run on a single GPU setting. Some word-level experiments involving ARag RUng at 10 had to be run on CPU as 24 GB VRAM were not sufficient. Models with higher maximum sequence lengths (e.g. byte models) were trained with 24 GB VRAM. Difference in equipment does not necessarily lead to dbgradation/mmpooeeenn
scores.
The terms "line" and "sentence" have been used interchangeably in the NLP literature. We use "line" to denote sequence that ends with a newline character and "sentence" as with ending punctuation. Most parallel corpora, such ours, are aligned by line, as line may be part of a sentence without an ending punctuation (e.g. a header/title). Using a standardized unit such as "line" would also be a fairer measure to linguac/scriptiones continuae (languages/seripts with no explicit punctuation). pottps://ght thub. com/ fxs jibab
16

---

Published as a conference paper at ICLR 2022
followed by single digit indicating its lexical tone in Mandarin. For Wubi, used the dictionary from the implementation from bttpps://ggthtie com/ wrtp/cc/esecwubtccccc
We have implemented all representations such that they would be reversible when the se- quence contains code-mixing. Additional code will be made available at attttts::/github. com/ dadasci
We used the official dev set as provided in (Ziemski et al. 2016), 3,077 lines per language remained from 4,000 after filtering line length to 300 characters. Data statistics is provided in Appendix for reference.
Notes on training time Each run of 30 directions sizes took approximately 8- days for character and byte models. Byte models generally took longer- hence training time is positively correlated with length (concurring with observations by Cherry et al.] (2018) as they compared character with BPE models). A maximum length of 300 characters entails maximum length of al least 300 bytes in UTF-8. Each run word models (30 directions, sizes) took about days (excluding the training of some 7-9 directions out of 30 per run involving ARug RUng at 10 on word level which took about 12-18 hours each direction to train on a CPU as these required more space and would run out of memory (OOM) the additional probing experiments described GPUs otherwise). These figures do not include
B HYPERPARAMETER SETTING
encoder transformer:
decoder transformer;
num-layers 6:6;
num-emhed 512:512;
transfomerr-mnddl-iiiee 512;
transfommer-aatcnnioon-hcaak 8;
2048;
transformer-aciiiation- -yppe relu:
transformer-posiitinaal postrrrrrppssttooommmmbeddiiniei type fixed;
transfomer-preppee prepooccss cunsformer-postprossse drn;
transformer-dropouut attention 0.1;
cranformer-dropoutt act 0.1;
transforner-drroouu--rree 0.1;
batch-size 15;
batch-type sentence;
max-num-chie checkpoint-not-inee
3;
max max-num-eee num-epochs 50;
optimizer adam:
optimized- metric perplexity;
optimizer-parana parizer epsilon: 0.000000001, betal: 0.9, beta2: 0.98;
lahel-smoothing 0.0;
learning- raterredue----eeeeeeeee wawtluccecuumo------ee num--ot-improved44
learning- hateingg-rt--eeeu---aetttteeeeee 0.001;
loss-normalization-yyy valid:
max-seq-len 300 for character, word, and BPE, 672 for all bytes, 688 for Wubi, 680 for Pinyin;
theckpoint-frequeneeel frequency/interval 4000.
(For smaller datasets, the end of 50 epochs is often reached before the first checkpoint. Since SOCKEYE only outputs scores checkpoints, we adjusted the checkpoint frequency as follows to get a score outputted by the end of epochs: 1000 100 lines for character byte instances, 400 100 lines for word and 500 for 100 lines BPF, 3450 for 1000 lines for word & BPE, For the very few cases that this default does not suffice due to bucketing of similar length sequences, manually set the checkpoint frequency to the last batch.)
17

---

Published as a conference paper at ICLR 2022
C CORRELATION STATISTICS
Best correlating metrics, i.c. the union of top metrics for all representations. For each representation, the top 3 metries boldfaced.
All correlations are highly significant (p < 10 10""),, except for min source length for WORD (p 23 0.0001) and min target length for woRn (p 0.3861).
Metric
CHAR
Pinyin
Wubi
BYTE
ARRU,
ARRU..
WORD
BPE
minimum length (target)
0.84
0.85
0.86
0.60
0.84
0.84
00.02
0.65
minimum length (source)
0.82
0.84
0.85
0.57
0.84
0.84
0.10
0.64
number tokens (source)
0.78
0.81
0.82
0.60
0.81
-8.0
0.59
0.83
TTR (target)
0.83
0.83
0.84
0.48
0.81
0.81
0.61
0.83
[V] (source)
0.54
0.51
0.51
-..50
0.67
-0.68
0.63
0.86
data size lines
0.80
0.83
- 0.83
0.59
0.81
-0.81
0.62
0.86
OOV token rate (target)
0.69
0.66
0.66
0.47
0.67
0.68
0.66
0.62
OOV type rate (target)
0.71
0.72
0.47
0.69
0.70
0.65
0.62
TTR (source)
0.67
0.71
0.71
0.60
0.81
0.81
0.56
0.82
The full list of metrics used for the correlation analysis is:
1. minimum length (source),
2. minimum length (target),
3. maximum length (source),
4. maximum length (target),
5. median length (source),
6. median length (target).
7. mean length (source),
8. mean length (target),
9. length std (source),
10. length std (target),
data size in lines,
12. number of parameters,
13. number of types ((V) (source),
14. number of types (V)) (target),
number of tokens (souree),
number tokens (larget),
17. type-token-ratio (TTR) (source),
18. type-token-ratio (TTR) (target),
19. OOV type rate (source).
OOV type rate (target),
21. OOV token rate (source),
22. OOV token rate (target),
23. token ratio,
24. target type-to-parametee ratio,
target token-to-pa to-parameter ratio,
26. distance between the TTRs of source and target (T,
27. token-to-parameet parameter ratio (i) (median length source * median length target $ num lines) / num parameters,
28. token-to- token-to-paramett parameter ratio (ii) (num_soue caaum_souree_tkeeee num_target__toeeee num_pparmmttrss
18

---

Published as a conference paper at ICLR 2022
D STATISTICAL COMPARISONS
Recall the definition and method for our Disparity/nequality assessment from $
In the context of our CLMing experiments, we consider there to be "disparity" "inequality" between languages 4 and l2 if there significant differences between the performance distributions of these two languages with respect to each representation. Here, by performance encode the held-out data using a trained CLM. With 30 directions, there are 15 pairs of source languages ; and 15 pairs of target languages (hrg1, luya) possible. We compare the source languages among each other, and the target languages among each other. Each fare or each lug consists of scores from all models trained across various sizes and directions. To assess whether the differences are significant, we perform unpaired two-sided significance tests with the null hypothesis that the score distributions for the two languages are not different. Upon testing for normality with the Shapiro- Shapiro-Wilk Wilk test (Shapiro Wilk] 1965 Rooston 1995), we use the parametric unpaired two-sample Welch's t-test (Welch) 1947)
mean the number of bits required
(when normal) or the non-parametric unpaired Wilcoxon test (Wilcoxon, 1945
(when not normal) for the comparisons. We the implementation R (R Core
Team 2014) for these 3 tests. To account for the multiple comparisons we are performing, correct all p-values using Bonferroni's 's correction (Benjamini &
Heller 2008 Dror et al. 2017) and follow Holm's procedure (Holm 1979 Dror et al.
2017) identify the pairs of 1 and l2 with significant differences after correction. We report all 3 levels of significance (ar < 0.05, 0.01, 0.001) for a more comprehensive overview. In contrast to Dror et al. (2017), which aimed to compare the performance of different algorithmss, we compare languages (in the context of computing).
To get samples for the statistical comparison results for the Disparity Table (Table
For each representation, we used runs (AO, BO, CO) in 5 sizes (102-106 lines) for each 4se and each lug. There are:
6 Iae (ARares EN ES FRASe RUsee g and
6 lug (ARug. ENug. ESug. FRug. RU lg. ZHng).
We compare pairwise among the Isr.. The 15 pairs are:
AR - AR -FRee ARace-UUght ARge- EN ES a.,, C.Nre-FRRan ENac--RUse ENge-ZHare ESS.. -R - -USS. -
FR FR
RU.ne-ZHH
Likewise 15 pairs among the lug-
For example, for the character (primary) representation to compare between AR and EN
construct the sample for ARar (sampleaRare) and the sample for ENsre (samplezwere) as follows:
Out of the 30 CHAR directions, there are 5 directions involving ARe trained for each run and data size (i.e. the directions: AR-EN, AR-ES, AR-FR, AR-RU, AR-ZH).
For each direction, there are 15 models trained (3 runs 5 sizes). We take all 75 CHAR models (15 models x 5 directions) involving AR as sampl lensore. That's 's sample of size 75.
Likewise, for EN (5 directions: EN-AR, EN-ES, EN-FR, EN-RU, EN-ZH), also have 75 data points for samplegsure- (Likewise also for all 6 la and all 6 (mg.)
For the comparisons, we compare pairwise, i.e. with two-sample Welch's 's t-test (when normal) or the -on-arrmettrc impaired Wilcoxon test (when not normal) because samplexnsee and s samplerxae Lexarr have one direction that is not paired: AR-EN and EN-AR. Other directions can be seen as paired, c.g. AR-ES and EN-ES as both having the same lag.
samples each time, but with unpaired
14 using implementation from htt. www.ccaaillltteoo iityttaaaa ysi NI.P
19

---

Published as a conference paper at ICLR 2022
Dype colen cato lin %), ie the abo nto herveen the nunter rypes and tokens in the dua This is muuhh pooyy for loxical diversity in that value of would indicate this no type is over sood teico, and valuo vary close to O wooL indicato that vary few
15 10
5.
Statistics for dataset A
005888
Number of types, le vocshulryy size (IVD) Note tha Sockeys sods 1o cale diioo additioal open: quis. eo dio sunks.
training
Number of tokeess This exclude the BOSSBOS (end weweeeeiinggg regenning-e of sentence) marker udded by Sockeye to each line.
covered
211
standaa devistion, 1005 percentle
as UNICoownS
data
9), the Inclion of
of tokens
disinct pes accOLIE for almost all of the ditt.
Line length EOSIBOS markerss
DATA STATISTICS
Out secabalary (OOV)
mi (in %0.i.
1,3
TITEE
E
20

---

Published as a conference paper at ICLR 2022
025550
035385
200200
170
2580
1.28
055525
Statistics for dataset B
RRARKR
B508
28588
25
2.0 18.208
15000
28
1213
3352
50 200200
22
20
133T
21

---

Published as a conference paper at ICLR 2022
013085
58558
1.74
25
005540
22.020
7.20 5g
836688
011
Statistics for dataset C
BREEGE
088888
288888
635-
685558
3863
51
23
1.00
14
22
22

---

Published as a conference paper at ICLR 2022
10.00 1/00/
As dififrree of vocabulay is leamed from each coiiiigg dotsect and data size. BIPE has distacet day sct for cach
13.00
Statistics for development (dev) set
210
6722
22
23

---

Published as a conference paper at ICLR 2022
F ENLARGED FIGURES FOR ALL 30 LANGUAGE DIRECTIONS (AGGREGATE RESULTS FROM ALL RUNS)
1800000
1500000
TRG
ES
number of bits
ZH
SRC
1200000
ES
FR
900000
1e+03
1e+05
number lines
Figure 4: CHAR: character models
24

---

Published as a conference paper at ICLR 2022
AR
ES
1500000
1000000
AR
EN
FR
number bits
RU
ZH
SRC
AR
EN
RU
1500000
1000000
1e+03
1e+05
1e+03 number of lines
1e+05
1e+03
1e+05
Figure 4: CHAR: character models (target language facet)
25

---

Published as a conference paper at ICLR 2022
1750000
TRG
1500000
ES
number of bits
ZH
SRC
1250000
ES
FR
...
1000000
750000
1e+03
1e+05
number of lines
Figure 5: CHAR with Pinyin for ZH ZHtrg
26

---

Published as a conference paper at ICLR 2022
AR
ES
2000000
1500000
1000000
AR
EN
FR
number bits
RU
ZH
SRC
2000000
AR
EN
RU
1500000
1000000
1e+03
1e+05
1e+03 number of lines
1e+05
1e+03
1e+05
Figure 5: CHAR with Pinyin for ZHerg (target language facet)
27

---

Published as a conference paper at ICLR 2022
1750000
TRG
1500000
EN
ES
number of bits
ZH
SRC
1250000
ES
FR
1000000
750000
1e+03
1e+05
number of lines
Figure 6: CHAR with Wubi for ZH
28

---

Published as a conference paper at ICLR 2022
AR
ES
2000000
1500000
1000000
AR
EN
FR
number bits
RU
ZH
SRC
2000000
AR
EN
RU
1500000
1000000
1e+03
1e+05
1e+03 number of lines
1e+05
1e+03
1e+05
Figure 6: CHAR with Wubi for ZHtry (target language facet)
29

---

Published as a conference paper at ICLR 2022
2500000
2000000
TRG
ES
number of bits
ZH
SRC
1500000
ES
FR
1000000
1e+03
1e+05
number of lines
Figure 7: BYTE models with UTF-8 encoding
30

---

Published as a conference paper at ICLR 2022
AR
ES
2500000
2000000
1500000
AR
EN
1000000
FR
number bits
RU
ZH
SRC
AR
EN
2500000
RU
2000000
1500000
1000000
1e+03
1e+05
1e+03 number of lines
1e+05
1e+03
1e+05
Figure 7: BYTE models with UTF-8 encoding (target language facet)
31

---

Published as a conference paper at ICLR 2022
2000000
TRG
1500000
ES
number of bits
ZH
SRC
ES
FR
1000000
1e+03
1e+05
number of lines
Figure 8: BYTE with ARtng & RUtry optimized with code pages 1256 & 1251 (ARRUtrg)
32

---

Published as a conference paper at ICLR 2022
ES
2000000
1500000
TRG
1000000
EN
number of bits
RU
2H
FR
ZH
SRC
EN
2000000
ES
ZH
1500000
1000000
1e+03
1e+05
1e+03 number lines
1e+05
1e+03
1e+05
Figure 8: BYTE with ARtrg RUtry optimized with code pages 1256 1251 (target language as facet)
33

---

Published as a conference paper at ICLR 2022
1800000
1500000
TRG
ES
FR
number of bits
SRC
1200000
AR
FR
900000
1e+03
1e+05
number lines
Figure BYTE with directions AR-RU & RU-AR optimized on both source and target sides (ARRU
34

---

Published as a conference paper at ICLR 2022
1500000
1000000
TRG
EN
number of bits
RU
2H
FR
RU
2H
SRC
EN
ES
ZH
1500000
1000000
1e+03
1e+05
1e+03 number lines
le+05
1e+03
1e+05
Figure 9; BYTE with directions AR-RU & RU-AR optimized both source and target sides (target language as facet)
35

---

Published as a conference paper at ICLR 2022
1200000
TRG
1000000
ES
number of bits
ZH
SRC
ES
FR
800000
1e+03
le+05
number of lines
Figure 10: WORD models
36

---

Published as a conference paper at ICLR 2022
AR
ES
1200000
1000000
800000
AR
EN
FR
number bits
RU
ZH
SRC
AR
1200000
EN
RU
1000000
600000
1e+03
1e+05
1e+03 number of lines
1e+05
1e+03
1e+05
Figure 10: WORD models (target language facet)
37

---

Published as a conference paper at ICLR 2022
1600000
1400000
TRG
ES
1200000
number of bits
ZH
SRC
10 31
14
3.17
12
43
ES
10
FR
1000000
800000
1e+03
1e+05
number of lines
Figure 11: BPE models
38

---

Published as a conference paper at ICLR 2022
AR
ES
1500000
1200000
900000
AR
EN
FR
number bits
RU
ZH
SRC
AR
EN
1500000
RU
22
1200000
900000
1e+03
1e+05
1e+03 number of lines
1e+05
1e+03
1e+05
Figure 11: BPE models (target language facet)
39

---

Published as a conference paper at ICLR 2022
Faimess in Representation for Multilingual NLP: Insights from Controlled Experiments Condi- tional Language Modeling
Version 1.1
ICLR 2022 camera-ready copy (20220510)
40


Reviews:
[{"review_id": "s38gVK8oHWq", "reviewer": "Reviewer_UwwP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Existence of \"inherent\" difference in difficulty of NLP for different\nlanguages is interesting for theoretical reasons as well as the\napplied NLP. The paper also presents a large number of experiments in\na controlled way for this purpose.\n\nAlthough I agree that the topic is important, and the study covers a\nfair amount of experiments with sound methodology, I do not think we\nlearn a lot from the present paper: (1) the effectiveness of sub-word\nunits for morphologically rich languages is expected, and (2) the\ndomain of application (conditional LMs) is rather narrow. Given the\nissue is studies using (non-conditional) LMs earlier (as the paper\ncites), the benefits of narrowing it further down to conditional LMs\nis not clear to me (after reading the paper). The issue with word\nsegmentation in Chinese is also studied before, and I do not see any\nadditional information the present paper provides.\nBesides the main above point, which is the basis of my main evaluation \nof the paper, there are a few minor issues I list below.\n\n- The paper is written in a mainly error-free English, but it is\n  rather difficult to read due to the organization and the flow of the\n  concepts/text. Although I do not have many concrete suggestions, the\n  this prevented me from understanding some simpler issues at first\n  reading. Examples include:\n    - There are some unclear statements like \"To which extent is\n      morphological complexity necessary in computing?\" (p2): this\n      probably means \"handling/modeling morphological complexity\" but\n      literally, \"need for complexity\" is not something anyone would\n      claim to be necessary.\n    - There are divergences from the main topic of discussion, for \n      example \"Our conventional way of referring to “language” (as a\n      socio-cultural product or with traditional word-based\n      approaches, or even for most multilingual tasks and\n      competitions) is too coarse-grained ...\" (p5) is just to general\n      to substantiate here while defining the training set size in the\n      previous context. The same goes for the discussion of ZH as\n      \"high-resource\" language on page 6: it is not clear to me why\n      this is relevant here, and what is the relation to the overall\n      aim of the paper.\n\n- Overall, the paper feels too dense. In my opinion, paper could\n  benefit from presenting less, explaining the main matters more\n  clearly. For example, I do not fully see the utility of the\n  byte-based models, especially that the paper goes into resolving\n  practical issues due to code-set differences among languages. The\n  additional data may be of interest for a small subset of the\n  audience, but for the most, this is not relevant for the main\n  question tackled. Leaving some of these data/exposition to appendix\n  may allow allow clearer exposition of the main points.\n\n- The figures, even the enlarged ones in the appendix, are rather\n  difficult to read: too many lines/patterns and too small fonts. No\n  concrete suggestions, but a clearer display of the results would\n  help the reader a lot.\n\n- \"Summary of findings and insights\" is too detailed at this point. A\n  more concise, general version would be much better. Many of the\n  references/notation here is not possible digest before reading the\n  rest of paper.\n\n- Although the data used is a standard data set, it would be useful\n  for most readers to have more detailed description of the data\n  (e.g., the type of documents, size of the overall data that the\n  parts used are sampled, ...). It would also be good to indicate what\n  a \"line\" corresponds to (is it a sentence, a paragraph, or some\n  other unit?).\n\n- Footnote marks should be place after punctuation.\n\n", "summary_of_the_paper": "This paper explores the performance of conditional language models for\ndifferent languages with the aim of establishing \"fairness\" in NLP\nsystems across languages. In particular, the paper focuses on\nperceived difficulty of \"morphologically rich\" languages for NLP.  The\npaper presents a series of experiments with 6 languages in the UN\nparallel corpus, using different units (words, sub-words, characters,\nbytes) for training the conditional language models. The paper\nconcludes with the general observation that the use of characters or\nbytes as units instead of words removes (or reduces) the performance\ndisparity between different languages.\n", "summary_of_the_review": "Although I agree that the topic is important, and the study covers a\nfair amount of experiments with sound methodology, I do not think we\nlearn a lot from the present paper: (1) the effectiveness of sub-word\nunits for morphologically rich languages is expected, and (2) the\ndomain of application (conditional LMs) is rather narrow. Given the\nissue is studies using (non-conditional) LMs earlier (as the paper\ncites), the benefits of narrowing it further down to conditional LMs\nis not clear to me (after reading the paper). The issue with word\nsegmentation in Chinese is also studied before, and I do not see any\nadditional information the present paper provides.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hnMoO8gGQCH", "reviewer": "Reviewer_UWb7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I want to thank the authors for the interesting paper. Overall, the\npaper contains a substantial amount of experiment results with some\ninteresting findings, but the paper is not easy to read and not easy to\nfollow. I am also not sure if this paper reveals substantially new\nthings to the readers.\n\nThe strength of the paper is as follows:\n\n-   This paper proposes to use conditional language modeling (machine\n    translation) to evaluate how hard it is for transformer models to\n    model a language. This is a clever setting that allows one to fix a\n    source language and compare the difficulty of modeling different\n    target languages.\n\n-   The paper uses statistical tests to verify if there exists\n    performance disparity between languages of different morphological\n    complexity.\n\n-   This paper shows that transformer conditional language models are\n    equally capable of modeling the six languages evaluated in the paper\n    when the suitable representation level is chosen. This indicates\n    that transformer models are **fair** and are not biased toward any\n    the language they tested, and morphological complexity is not indicative\n    of how hard it is for transformer-based conditional language models\n    to learn.\n\nThe weakness of this paper and questions about the paper is listed as\nfollows:\n\n### Not enough novelty compared to prior works\n\nAn important prior work of this paper is Mielke et al. (2019). I find\nthis paper highly resembles the work of Mielke’s from some aspects. I\nwill summarize the work of Mielke et al. (2019) as follows first:\n\n1.  Research Question: Do LSTM-based language models serve all\n    languages? What are the characteristics of languages that are easier\n    for LSTM-based language models to model?\n\n2.  Experimental Setup: Use parallel corpora containing 21 languages.\n    Train a mono-lingual character/BPE LSTM language model on each\n    language, and evaluate the development set perplexity for each\n    language.\n\n3.  Results and Findings: Language difficulties are not significantly\n    predicted by linguistic factors (including morphological\n    complexity), no matter the representation is character or BPE. For\n    LSTM language model with character-based representation, the\n    difficulty for LSTM to model correlates with the raw length in\n    characters. For LSTM language model with BPE-based representation,\n    the difficulty for LSTM to model correlates with the vocabulary\n    size.\n\nThe main difference between this work and Mielke et al. (2019) are as\nfollows:\n1.  The models used in this work are transformer-based conditional\n    language models, while Mielke et al. (2019) used RNN-based\n    (unconditioned) language models.\n\n2.  The indicator of performance in this paper is unnormalized\n    perplexity score, and Mielke et al. (2019) used the normalized\n    version.\n\n3.  This paper uses statistical tests to compare if there exists\n    performance *disparity* between language pairs, which was not\n    presented in Mielke et al. (2019) (since Mielke et al. (2019) did\n    not use conditional language modeling).\n\nThe main findings of Mielke et al. (2019) that I listed above are also\nobserved in this work. And this work also has the following new findings\nthat are not presented Mielke et al. (2019):\n\n1.  They find that for transformer conditional language models, the\n    difficulty of modeling a certain target language is independent of\n    the source language.\n\nWhile the experiment settings are slightly different, from the aspects\nof the model architecture and evaluation metrics, the core of the experiment\nand the main takeaway does not make me learn substantially new things. I\ndo recognize that it is important to empirically verify the results of\nMielke et al. (2019) on transformer-based conditional language models,\nbut I have reservations about whether the contribution of this work is\nample enough.\n\nAdditionally, I think this paper will definitely benefit from a more\nthorough discussion of their difference with the previous works.\n\n### Experiment settings are not clear enough to follow\n\n-   In the \"Disparity/Inequality\" part in Section 2, the paper states\n    \"Each $l_{src}$ or each $l_{tgt}$\n    consists of scores from all models trained across various sizes and\n    directions.\" I have difficulty understanding how the comparison\n    tests can be done. In my opinion, this is the most novel setting in\n    this work, and I would like to see more elaboration! Since for each\n    source/target language, there are scores for different directions,\n    dataset sizes, random seeds, and datasets, how is the comparison\n    between the \"distribution of the scores\" done? As far as I\n    understand, the \"distribution of the scores\" of a language will\n    consist of scores obtained under different settings, and I am not\n    sure if comparing the distribution as a whole is rational. Since the\n    score distribution under certain factor fixed (for example, when the\n    dataset size is fixed) may be statistically significantly different,\n    but when all the factors change together, we cannot tell the\n    influence of each factor.\n\n-   Is there any reason why not running the $10^6$ for setting\n    A3-7, B1 & C1.\n\n-   I don’t understand how the random seeds are controlled in their\n    experiments. Is the random seed only related to how the model is\n    initialized? Or the random seed will also affect how dataset B and C\n    are sampled? Why is there not a C(0) model (in appendix A). I think\n    for different data sizes and datasets, they should have a common\n    random seed, and the experiments should be run using multiple\n    different common random seeds.\n\n-   I have difficulty understanding what \"transformer-preprocess d;\n    transformer-prostprocess drn\" in appendix B is. I have never seen\n    this parameter in the context of transformer-based language models.\n\n### Some words or statements are ambiguous\n\n-   In Section 1.1, \"favoring of certain outcomes\". I do not understand\n    what \"certain outcomes\" are in this paper after I read the whole\n    paper. Do the authors mean \"better at modeling certain languages\"?\n\n-   In Section 1.2, \"bounded by representation level\". I am not sure\n    what \"bounded by\" means here , and I am skeptical about the phrase\n    \"bounded by\". Since the experiments only use a set of\n    hyperparameter, the claim that \"representation bounds hardness\" may\n    be too strong. May be the hyperparameter just don’t work for certain\n    representation.\n\n-   Section 2, in **Conditional language modeling**: \"without performing\n    any downstream translation\". What is \"downstream translation\"? This\n    seems not to be a common phrase in the context of machine\n    translation, and may need some explanation. Do the authors try to\n    refer to something like beam search?\n\n-   Section 2, in **Conditional language modeling**: \"To explicitly\n    focus on modeling the complexities that may or may not be intrinsic\n    to the languages\". I don’t understand why this paper is able to\n    explicitly focus on \"modeling the complexities\" under their setting.\n    I also don’t quite get what is intrinsic to the languages and what\n    is not.\n\n-   Section 2, in **Controlled experiments as basic research for\n    scientific understandings**: \"that make language data different from\n    other data types\". I fail to understand what \"other data types\" this\n    paper is trying to refer to. I also don’t see the connection between\n    the relationship between the quoted line above and \"data-centric\n    approach\".\n\n-   Section 5, \"disparity is not a necessary condition\". What is the\n    necessary condition for?\n\n### Some claims in the paper are not well supported by the experiments results\n\nFor example, the paper writes in page 6 that \\\"on the byte level,\n$AR_{trg}$ and $RU_{trg}$ display non-monotonic and unstable\nbehaviors\\\". I am not sure what the \\\"non-monotonic behaviors\\\" the\npaper is referring to, but I think in Figure 1(e), not only AR and RU\nare non-monotonic, all results in Figure 1(e) are not monotonic. All of\nthem decay from $1e-02$, reach the lowest perplexity at $1e-04$, and\nrise afterward.\n\n### Miscellaneous concerns\n\n-   I am concern about the title \\\"Fairness in Representation\\\". First,\n    since the word fairness is mostly used to refer to societal biases\n    in NLP community, I am not sure if the word 'fairness' may be\n    misleading at the first sight. Maybe statistical bias will be a more\n    better choice. Second, I am not sure what \\\"Fairness in\n    Representation\\\" means; it seems to refer to the performance\n    disparity between different representation levels. But since the core\n    problem in the paper is the disparity between different languages of\n    different morphological complexity, I guess the whole paper is more\n    like \\\"Transformer's Fairness for Different Languages\\\" instead of\n    \\\"Fairness in Representation\\\".\n\n-   Some passages seem to be not highly related to the experiment\n    results. For example, in page 6, I don't see what the passages \\\"But\n    this is often \\... and accepted in NLP.\\\" has to do with the\n    experiment result in this section.\n\n-   I am concerned about using Pinyin in Chinese. Will single Pinyin\n    correspond to multiple characters in Chinese? If so, then changing\n    the target from character to Pinyin does not merely change the\n    vocabulary size and makes the whole comparison unclear. Whether\n    Pinyin can be an alternative for character needs to be carefully\n    discussed.\n\n-   I am concerned about the paper's only using a set of hyperparameter. I\n    understand that using the same set of hyperparamater is for\n    \\\"controlled experiments as basic research for scientific\n    understanding\\\", but sometimes the paper attributes the performance\n    fluctuation to the fact that they did not search for better\n    hyperparameter. This seems contradicting, and one can question that\n    the results in the paper will not hold if we had searched the\n    optimal parameters for different language pairs, dataset, data\n    sizes.\n\n-   If the goal is to compare morphologically rich languages (AR, RU)\n    and frugal language (ZH), then why does the paper also compare the\n    other three languages in United Nations Parallel Corpus? What do\n    their results tell us? What is the morphological complexity of the\n    remaining three languages?\n\n-   The languages used in this paper is not too many. The conclusion of\n    Mielke et al. (2019) are based on 21 languages, I am not sure\n    whether the conclusion of this paper will still hold if more\n    languages are included.\n\n-   If in 2(b,d), only ZH is changed, I think it is better to only show\n    the result of ZH as a target since all remaining 5 subsubfigures are\n    the same as in 1(d).\n\n### Presentation problems\n\nThe figures are not easy to read. Figure 1, 2, 3 are too small and are\nunreadable on A4 paper. The problem of the figure hinders my\nunderstanding to some extent. For example, in the second last line in\npage 7, it is said that \\\"AR/RU \\> ES/FR \\> EN/ZH\\\", I cannot arrive at\nthe conclusion since Figure 1(c) is too small to read. While the enlarged\nversions of figures are provided in the appendices, I still find them not\nso useful for some cases; for example, the standard error bars in Figure\n4 EN are overlapped with the lines.", "summary_of_the_paper": "This paper asks: Are languages which have been traditionally considered\nmorphologically rich (Arabic and Russian) and poor (Chinese) equally\nhard to learn by transformer-based conditional language model (CLM).\n\nTo quantify how hard it is for CLM to model a language $l_1$, this paper\nproposes to train a transformer-based encoder-decoder model to translate\n$l_1$ to another language $l_2$, and uses the \\\"unnormalized perplexity\nscore\\\" (the number of bits needed to encode the same development set\nfor the target language) as an indicator for the \\\"hardness\\\" for CLM to\nlearn the $l_2$. For two languages $l_1$ and $l_2$, there exists\n**disparity** if their perplexity distributions are significantly\ndifferent. This paper includes extensive experiments across 30 language\npairs using three different representation level (including character,\nbyte, and word) for different dataset sizes.\n\nThey have the following findings:\n\n1.  They find that the performance disparity between morphologically\n    rich and poor languages cannot be justified due to morphological\n    complexity. Representation level matters more.\n\n2.  They find that the hardness for CLM to model a language is related\n    to the level of representation used to encode the language, the\n    vocabulary size, and/or the sequence length.\n\n3.  They find that it is equally hard to learn a certain target language\n    (that are tested in their experiment) using transformer-based\n    conditional language models, independent of the source language.", "summary_of_the_review": "This paper studies the fairness of transformer models to model languages\nof different morphological complexity. This work extends the previous\nobservation from Mielke et al. (2019) and show that similar results hold\nfor transformer-based conditional language models: morphological\ncomplexity is not indicative of the difficulty for transformer to conditional\nlanguage model, and representation level matters more. This indicates\nthat transformers are **fair** towards the languages tested in this\nwork, in terms of conditional language modeling. They propose to use\nconditional language models to make it easier to compare the difficulty\nof modeling a specific target language, and the usage of statistical\ntests for testing performance disparity among languages is a novel act.\nHowever, experiment settings and findings/results resemble the work of\nMielke et al. (2019) to some degree. The part on statistical testings is\nsomewhat baffling and some of the statements are ambiguous. The figures\nfor the main experiments are not very easy to read. The title is also\nsomewhat not quite indicative of the content (while it does seem\nrelevant), mainly because of the term \\\"fairness in representation\\\".\n\n## Update on Nov 22\nAfter a thorough and in-depth discussion with the authors, I upvote the score from 3 (reject, not good enough) to 5 (marginally below the acceptance threshold), meaning that I am still inclined to reject.\nI change my score mainly because the authors have addressed many of my initial concerns, including the experiment settings, some odd wordings or claims, and the presentation problems.\nBut I still do not find the paper is good enough to be accepted. \nI am still not convinced by the authors' claim of their novelty; the novelties listed by the authors seem to be more of \"something that was not being done in prior works\", but I am not sure how those \"something\" can much advance our understanding on this topic.\nOverall, I think this paper presents some nice and well-controlled experiments, but the interpretation of the results tends to be too high-level and might be overstating; I often have difficulty linking the quantitative results and qualitative interpretations.\nI am not sure what I have learned from this paper.\n\n\n## Edit again on Nov 22\nI will upvote my score to 6. \nAfter going over the paper several times, I think there indeed are certain empirical contributions of the paper. \nI think varying the data size is an important experiment that has yet to be presented in any prior works and is valuable to be empirically verified, I do agree with the authors on this.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LMTYc71MEYU", "reviewer": "Reviewer_y4Vs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Very nice writeup and structure, the paper is for the most part a very enjoyable read.\n- Bold in tackling important questions from a fresh angle.\n\nWeaknesses (may be obsolete, see discussion):\n- The contributions are overstated in light of the experiment setup.\n- Only six resource-rich languages are used and only one respective dataset, the UN parallel corpus.", "summary_of_the_paper": "The paper asks an interesting question, one that appeared eternally relevant for NLP: Are languages harder or easier to contextually language-model by Transformer in light of their varying morphological complexity? The claimed main finding is provocative, indicating that the debate morphological variation is not relevant in the context of the given computational machinery.", "summary_of_the_review": "I recommend that the paper be rejected so that it may further develop its empirical setup & verify (or reject) the findings on a larger dataset with more substantial linguistic depth (more resource variation) and breadth (more languages). In its current form, the paper's larger-than-life claims are simply not sufficiently supported with the choice of one dataset and six resource-rich languages, even if the hypothesis and findings are indeed interesting.\n\nEdit after first round of author discussion, 2021-11-16:\n\nI am concerned that there is a disconnect between the stated contributions of the paper (which are expressed as rather broad and general in its narrative) on one side, and the setup of the empirical investigation on the other side. While I may view the insight as to the (ease of) availability of broad-coverage linguistic resources sympathetically, the fact still remains that the findings are closely tied with the UN parallel corpus and its six languages. This in itself would not disqualify the paper at face value. However, text e.g. under \"Summary of findings\" 1-5 and later in \"Conclusion\" comes across as masking this fact and painting a much broader picture.\n\nI would be much more inclined to accept this (undoubtedly insightful) paper should it insert a clear Limitations section which would reflect on the preliminary nature of these findings given the dataset constraints.\n\nAs for the language resources, I would very much love to hear which of those e.g. in OPUS (https://opus.nlpl.eu/index.php) were not fitted to the experiment while only the UN corpus passed the filter. This would be an insightful addition to the limitations section, as the author response now seems to indicate massive limitations in all the other available datasets. Do share with the community!\n\nEdit after 2021-11-18:\n\nUpvoting the overall score after insight re: multi-parallel datasets available (see discussion).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FtRjPQ5Qh6C", "reviewer": "Reviewer_86m6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper contains a series of sound, controlled experiments to study the role of input representations in language-modeling. I think the experiments are well designed, and with a clear focus to answer the research questions laid out in 1.2. Overall, I do think that this paper will benefit the multilingual NLP community in thinking carefully about the choice of input representations, rather than advocating for a one-choice fits all solution. However, I do have a few concerns that prevent me from giving a higher score. \n\n1. The claim that this paper \"renders a decade-long debate on morphology irrelevant\" is made multiple times in this paper, and in my opinion, is too strong a claim to be made on the basis of experiments on a single task (conditional LM), on a handful of languages (however typologically diverse they may be), and using a single architecture (6-layer Transformer). I would recommend scoping this claim down to something that can be clearly supported from the results of this paper.\n\n2. Second, how does the practical recommendation in Section 3 (\"A practical takeaway from this set of experiments: in order to obtain more robust training results, use bytes for ZH and characters for AR and RU\") square with the multilingual nature of recent models which typically have a shared input representation for many languages? I think more generally this point is one of several that seems slightly disconnected from downstream applications (the claim about morphological complexity being irrelevant would also fall in this bucket.).\n\n3. There are a few citations from prior work in NLP that are worth contextualizing here. Specifically, the work of Gillick at al. on byte level processing for sequence tagging [1], that of Wang et al. on byte-level Neural MT, and more recently, the work by Clark et al. on building a tokenization-free encoder seem particularly relevant [3]. In fact, the whole body of literature on character-level neural models are particularly relevant and the omission of these is critical, especially given the stance of the paper in advocating moving beyond \"word\" level models (e.g. Footnote 8)\n\n[1] - https://arxiv.org/pdf/1512.00103\n[2] - https://arxiv.org/abs/1909.03341\n[3] - https://arxiv.org/abs/2103.06874", "summary_of_the_paper": "At its core, this paper focuses on testing whether languages with different morphological complexity equally challenging to language-model in a bilingual / MT-like setting using a 6 layer Transformer architecture.  Specifically, it tests the assumption whether the previous observations about languages with higher morphological complexity are difficult to language-model, using a controlled experiment with a n-way parallel corpus across 6 languages of interest. The key finding of this paper is a simple one : that the choice of input representation is highly important, given everything else is kept constant, and that differences in language-model perplexities can be normalized and made to disappear by choosing the same representation (word/character/byte)", "summary_of_the_review": "This paper contains sound and controlled experiments that test the role of input representation in bringing disparity to language modeling results, which I believe will be useful for the larger community. At the same time, several claims in the paper are too strong or not grounded in practical use-cases, and some relevant literature is not referred to, leading me to be marginally inclined to accept this paper.\n\n### Edit - 22 November, 2021\nI spent some time reading the discussion between the authors and the other reviewers, as well as revisiting v0.2 of the paper. In light of these discussions, I think I am fairly comfortable accepting this paper as my main concerns have been addressed by the review and the comments of the authors. Hence, I revised my overall recommendation to 8, while adjusting a few other scoring parameters too.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7AhgHQeAYmv", "reviewer": "Reviewer_kpoN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n— Very interesting and engaging work, with \n--- Experiments are very controlled, from data, number of lines, language mappings, and segmentations. \n— Experiments seem to be reproducible just from the description in the Experimental Setup. Very clear. \n— Thoughtful connections from human language processing and linguistics to try to explain the Char/Byte/Word performance disparities. \n--- Really interesting follow up experiments on ZH!\n\nCons: \n— The “A0, B0, C0, A1 & A2, A3-7”, etc. nomenclature for experiments is unhelpful to readers. Can the experiment names be changed to be something more intuitive, so that readers don’t need to memorize “oh, A0 is one of the controlled experiments for data size”?\n-- Figures in this paper are largely unreadable and do not help as visual aids. I have to zoom in to 310% (at minimum!!!), and even then my eyes struggle to distinguish the colours and line styles from each other. Also, the subfigures within subfigures are not readable. This is a big problem, because the figures are used to support claims that the authors are trying to make, and I think many readers would be unable to draw the same conclusions from these figures themselves. \n \n\nFormatting, grammar, etc.:\n1. Abstract: “Performance disparity is not a necessary condition” is worded strangely to me. \n2. Introduction: Formatting at bottom of 1.1 is weird. Use of bullet points makes no sense here, they just split up the sentence. \n3. S2, “Experimental setup“: There’s a double parenthesis )) with the (Jurafsky & Martin, 2009) citation\n4. S5, your tables do not follow the same typesetting conventions as your Figures (i.e., the description is on top)\n", "summary_of_the_paper": "In this paper, authors aim to untangle the disparities in performance between morphologically rich languages (e.g. RU) and morphologically poor languages (e.g. ZH). In order to untangle such disparities, the authors propose a highly controlled experiment setup, where they control for a number of potentially confounding factors (e.g. language mappings, data size, text encoding, etc.). After the controlled experiments, the main take-away of the paper is that vocab size seems to be the biggest indicator of how well or poorly a CLM will perform — since there can be several different granularities for the concept of a “word” for morphologically rich languages, and this can explain why modelling morphologcially rich languages can be more difficult.  They perform additional experiments with different representations of ZH, and were able to mimic the struggles of morphologically rich languages with ZH, and use this as further evidence to support the claim that the debate within NLP about the best segmentation is irrelevant.\n\nOverall, this work is exciting, has has a lot of potential to help inform future language modelling for morphologically rich languages.  The very controlled nature of the experiment in the paper is what makes it most convincing, and this is the paper’s largest contribution. ", "summary_of_the_review": "Reasons for score:  \nFrankly, if the figures and tables in this paper were significantly improved, this would be an 8 or higher. The main motivation of the paper is exciting, the methods are well controlled, and the findings are interesting. The core of the problem (as I discuss in \"Cons\") is that the claims made in the paper rely on the figures/tables for illustration as evidence, and the figures/tables are entirely unhelpful. In otherwords, the tables and figures detract from your arguments because they are unreadable. Once this can be fixed, I think its great work, and it would be interesting and directly useful to many people in NLP. \n\nEDIT - Wednesday, 17 November, 2021: In light of the authors rebuttal to my review, and their changes to v2 of the document, I am bumping my score up from 5 (marginally below threshold) to 8 (good paper, accept). My biggest problem with the paper before, were the figures, but I think that the authors addressed my comments to the best of their abilities by adding a lot more clarity to the textual descriptions of the images, in the paper and in the captions. The figures are still not great, but with their new textual descriptions, you can look at the general shapes of things, and visually understand the point, and before v2 I did not have this same level of visual understanding.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors address a very important question pertaining to the relevance of morphological complexity in the ability of transformer based conditional language models. Through extensive (controlled) experiments using 6 languages they answer as well as raise very interesting questions about the role of morphology/segmentation/vocab size which mat spawn more work in this area.

All the reviewers were positive about the paper and agreed that the paper made significant contributions which would be useful to the community. More importantly, the authors and reviewers engaged in meaningful and insightful discussions through the discussion phase. The authors did a thorough job of addressing all reviewer concerns and changing the draft of the paper accordingly. 

I have no hesitation in recommending that this paper should be accepted.
Decision: Accept (Spotlight)