Representation and Bias in Multilingual NLP: Insights from Controlled Experiments on Conditional Language Modeling

Inspired by the phenomenon of performance disparity between languages in machine translation , we investigate whether and to what extent languages are equally hard to “ conditional-language-model ” . Our goal is to improve our understanding and expectation of the relationship between language , data representation , size , and performance . We study one-to-one , bilingual conditional language modeling through a series of systematically controlled experiments with the Transformer and the 6 languages from the United Nations Parallel Corpus . We examine character , byte , and word models in 30 language directions and 5 data sizes , and observe indications suggesting a script bias on the character level , a length bias on the byte level , and a word bias that gives rise to a hierarchy in performance across languages . We also identify two types of sample-wise non-monotonicity — while word-based representations are prone to exhibit Double Descent , length can induce unstable performance across the size range studied in a novel meta phenomenon which we term erraticity . By eliminating statistically significant performance disparity on the character and byte levels by normalizing length and vocabulary in the data , we show that , in the context of computing with the Transformer , there is no complexity intrinsic to languages other than that related to their statistical attributes and that performance disparity is not a necessary condition but a byproduct of word segmentation . Our application of statistical comparisons as a fairness measure also serves as a novel rigorous method for the intrinsic evaluation of languages , resolving a decades-long debate on language complexity . While all these quantitative biases leading to disparity are mitigable through a shallower network , we find room for a human bias to be reflected upon . We hope our work helps open up new directions in the area of language and computing that would be fairer and more flexible and foster a new transdisciplinary perspective for DL-inspired scientific progress . 1 INTRODUCTION . With a transdisciplinary approach to explore a space at the intersection of Deep Learning ( DL ) / Neural Networks ( NNs ) , language sciences , and language engineering , we report our undertaking in use-inspired basic research — with an application-related phenomenon as inspiration , we seek fundamental scientific understanding through empirical experimentation . This is not an application or machine translation ( MT ) paper , but one that strives to evaluate and seek new insights on language in the context of DL with a consideration to contribute to our evaluation , segmentation , and model interpretation practice in multilingual Natural Language Processing ( NLP ) . Our inspiration : performance disparity in MT The use case that inspired our investigation is the disparity of MT results reported in Junczys-Dowmunt et al . ( 2016 ) . Of the 6 official languages of the United Nations ( UN ) — Arabic ( AR ) , English ( EN ) , Spanish ( ES ) , French ( FR ) , Russian ( RU ) , and Chinese ( ZH ) , results with target languages AR , RU , and ZH seem to be worse than those with EN/ES/FR , regardless of the algorithm , may it be from phrased-based Statistical MT ( SMT/Moses ( Koehn et al. , 2007 ) ) or Neural MT ( NMT ) .1 The languages have the same amount of line-aligned , high-quality parallel data available for training , evaluation , and testing . This prompts the question : are some languages indeed harder to translate from or to ? Problem statement : are all languages equally hard to Conditional-Language-Model ( CLM ) ? A similar question concerning ( monolingual ) language modeling ( LMing ) was posed in Cotterell et al . ( 2018 ) and Mielke et al . ( 2019 ) along with the introduction of a method to evaluate LMs with multiway parallel corpora ( multitexts ) in information-theoretic terms . To explicitly focus on modeling the complexities that may or may not be intrinsic to the languages , we study the more fundamental process of CLMing without performing any translation . This allows us to eliminate confounds associated with generation and other evaluation metrics . One could think of our effort as estimating conditional probabilities with the Transformer , with a bilingual setup where perplexity of one target language ( ltrg ) is estimated given the parallel data in one source language ( lsrc ) , where lsrc 6= ltrg . We focus on the very basics and examine the first step in our pipeline — input representation , holding everything else constant . Instead of measuring absolute cross-entropy scores , we evaluate the relative differences between languages from across 5 magnitudes of data sizes in 3 different representation types/levels . We consider bias to be present when performance disparity in our Transformer models is statistically significant . 1.1 SUMMARY OF FINDINGS AND CONTRIBUTIONS . In investigating performance disparity as a function of size and data with respect to language and representation on the Transformer in the context of CLMing , we find : 1. in a bilingual ( one-to-one ) CLMing setup , there is neutralization of source language instances , i.e . there are no statistically significant differences between source language pairs . Only pairs of target languages differ significantly ( see Table 1 ) . 2 . We identify 2 types of sample-wise non-monotonicity on each of the primary representation levels we studied : ( a ) Double Descent ( Belkin et al. , 2019 ; Nakkiran et al. , 2020 ) : on the word level , for all languages , performance at 102 lines is typically better than at 103 before it improves again at 104 and beyond . This phenomenon can also be observed in character models with ZH as a target language as well as on the word level with non-neural n-gram LMs ; ( b ) erraticity : performance is irregular and exhibits great variance across runs . We find sequence length to be predictive of this phenomenon . We show that this can be rectified by data transformation or hyperparameter tuning . In our study , erraticity affects AR and RU on the byte level where the sequences are too long with UTF-8 encoding and ZH when decomposed into strokes on the character level . 3 . In eliminating performance disparity through lossless data transformation on the character and byte levels , we resolve language complexity ( § 4 and App . J ) . We show that , in the context of computing with the Transformer , unless word-based methods are used , there is no linguistic/morphological complexity applicable or necessary . There is no complexity that is intrinsic to a language aside from its statistical properties . Hardness in modeling is relative to and bounded by its representation level ( representation relativity ) . On the character and byte levels , hardness is correlated with statistical properties concerning sequence length and vocabulary of a language , irrespective of its linguistic typological , phylogenetic , historical , or geographical profile , and can be eliminated . On the word level , hardness is correlated with vocabulary , and a complexity hierarchy arises through the manual preprocessing step of word tokenization . This complexity/disparity effected by word segmentation can not be eliminated due to the fundamental qualitative differences in the definition of a “ word ” being one that neither holds universally nor is suitable/consistent for fair crosslinguistic comparisons . We find clarification of this expectation of disparity necessary because more diligent error analyses need to be afforded instead of simply accepting massively disparate results or inappropriately attributing under-performance to linguistic reasons . 4 . Representational units of finer granularity can help close the gap in performance disparity . 5 . Bigger/overparameterized models can magnify/exacerbate the effects of differences in data statistics . Quantitative biases that lead to disparity are mitigable through numerical methods . 1We provide a re-visualization of these grouped in 6 facets by target language in Figure 4 in Appendix A . Outline of the paper In § 2 , we define our method and experimental setup . We present our results and analyses on the primary representations in § 3 and those from secondary set of controls in § 4 in a progressive manner to ease understanding . Meta analyses on fairness evaluation , non-monotonic behavior , and discussion on biases are in § 5 . Additional related work is in § 6 . We refer our readers to the Appendices for more detailed descriptions/discussions and reports on supplementary experiments . 2 METHOD AND DEFINITIONS . Controlled experiments as basic research for scientific understanding Using the United Nations Parallel Corpus ( Ziemski et al. , 2016 ) , the data from which the MT results in Junczys-Dowmunt et al . ( 2016 ) stem , we perform a series of controlled experiments on the Transformer , holding the hyperparameter settings for all 30 one-to-one language directions from the 6 languages constant . We control for size ( from 102 to 106 lines ) and language with respect to representational granularity . We examine 3 primary representation types — character , byte ( UTF-8 ) , and word , and upon encountering some unusual phenomena , we perform a secondary set of controls with 5 alternate representations — on the character level : Pinyin and Wubi ( ASCII representations for ZH phones and character strokes , respectively ) , on the byte level : code page 1256 ( for AR ) and code page 1251 ( for RU ) , and on the word level : Byte Pair Encoding ( BPE ) ( Sennrich et al. , 2016 ) , an adapted compression algorithm from Gage ( 1994 ) . These symbolic variants allow us to manipulate the statistical properties of the representations , while staying as “ faithful ” to the language as possible . We adopt this symbolic data-centric approach because we would like to more directly interpret the confounds , if any , that make language data different from other data types . We operate on a smaller data size range as this is more common in traditional domain sciences and one of our higher goals is to bridge an understanding between language sciences and engineering ( the latter being the dominant focus in NLP ) . We run statistical tests to identify the strongest correlates of performance and to assess whether the differences between the mean performance of different groups are indeed significant . We are concerned not with the absolute scores , but with the relations between scores from different languages and the generalizations derived therefrom . Information-theoretic , fair evaluation with multitexts Most sequence-to-sequence models are optimized using a cross-entropy loss ( see Appendix B for definition ) . Cotterell et al . ( 2018 ) propose to use “ renormalized ” perplexity ( PP ) to evaluate LMs fairly using the total number of bits divided by some constant . In our case , we choose instead a simpler method of using an “ unnormalized ” PP , directly using the total number of bits needed to encode the development ( dev ) set , which has a constant size of 3,077 lines per language . Disparity/Inequality In the context of our CLMing experiments , we consider there to be “ disparity ” or “ inequality ” between languages l1 and l2 if there are significant differences between the performance distributions of these two languages with respect to each representation . Here , by performance we mean the number of bits required to encode the held-out data using a trained CLM . With 30 directions , there are 15 pairs of source languages ( lsrc1 , lsrc2 ) and 15 pairs of target languages ( ltrg1 , ltrg2 ) possible . To assess whether the differences are significant , we perform unpaired two-sided significance tests with the null hypothesis that the score distributions for the two languages are not different . Upon testing for normality with the Shapiro-Wilk test ( Shapiro & Wilk , 1965 ; Royston , 1995 ) , we use the parametric unpaired two-sample Welch ’ s t-test ( Welch , 1947 ) ( when normal ) or the non-parametric unpaired Wilcoxon test ( Wilcoxon , 1945 ) ( when not normal ) for the comparisons . We use the implementation in R ( R Core Team , 2014 ) for these 3 tests . To account for the multiple comparisons we are performing , we correct all p-values using Bonferroni ’ s correction ( Benjamini & Heller , 2008 ; Dror et al. , 2017 ) and follow Holm ’ s procedure2 ( Holm , 1979 ; Dror et al. , 2017 ) to identify the pairs of l1 and l2 with significant differences after correction . We report all 3 levels of significance ( α ≤ 0.05 , 0.01 , 0.001 ) for a more comprehensive evaluation . Experimental setup The systematic , identical treatment we give to our data is described as follows with further preprocessing and hyperparameter details in Appendices B and C , respectively . The distinctive point of our experiment is that the training regime is the same for all ( intuition in App . O.1 ) . 2using implementation from https : //github.com/rtmdrr/replicability-analysis-NLP After filtering length to 300 characters maximum per line in parallel for the 6 languages , we made 3 subsets of the data with 1 million lines each — one having lines in the order of the original corpus ( dataset A ) and two other randomly sampled ( without replacement ) from the full corpus ( datasets B & C ) . Lines in all datasets are extracted in parallel and remain fully aligned for the 6 languages . For each run and each representation , there are 30 pairwise directions ( i.e . one lsrc to one ltrg ) that result from the 6 languages . We trained all 150 ( for 5 sizes ) 6-layer Transformer models for each run using the SOCKEYE Toolkit ( Hieber et al. , 2018 ) . We optimize using PP and use early stopping if no PP improvement occurs after 3 checkpoints up to 50 epochs maximum , taking the best checkpoint . Characters and bytes are supposed to mitigate the out-of-vocabulary ( OOV ) problem on the word level . In order to assess the effect of modeling with finer granularity more precisely , all vocabulary items appearing once in the train set are accounted for ( i.e . full vocabulary on train , as in Gerz et al . ( 2018a ; b ) ) . But we allow our system to categorize all unknown items in the dev set to be unknown ( UNK ) so to measure OOVs ( open vocabulary on dev ( Jurafsky & Martin , 2009 ) ) . To identify correlates of performance , we perform Spearman ’ s correlation ( Spearman , 1904 ) with some basic statistical properties of the data ( e.g . length , vocabulary size ( |V | ) , type-token-ratio , OOV rate ) as metrics — a complete list thereof is provided in Appendix F. For each of the 3 primary representations — character , byte , and word , we performed 5 runs total in 5 sizes ( 102-106 lines ) ( runs A0 , B0 , C0 , A1 , & A2 ) and 7 more runs in 4 sizes ( 102-105 lines ) ( A3-7 , B1 , & C1 ) , also controlling for seeds . For the alternate/secondary representations , we ran 3 runs each in 5 sizes ( 102-106 lines ) ( A0 , B0 , & C0 ) .

Summary: The paper investigates whether languages are equally hard to Conditional-Language-Model (CLM). To do this, the authors perform controlled experiments by modeling text from parallel data from 6 typologically diverse languages. They pair the languages and perform experiments in 30 directions with Transformers, and compare 3 different unit representations: characters, bytes, and word-level (BPE).