Fairness in Representation for Multilingual NLP: Insights from Controlled Experiments on Conditional Language Modeling

1 INTRODUCTION . 1.1 BACKGROUND AND MOTIVATION . Most current work on fairness in Machine Learning ( ML ) and Natural Language Processing ( NLP ) focuses on the societal biases encoded in natural language data that are propagated and amplified when they are used at scale for/as Artificial Intelligence ( AI ) solutions1 . But little has been said or questioned about the bias , as in , the favoring of certain outcomes , implicit in our theoretical/scientific assumptions that results in the varying performance of different languages in computing . Disparity in machine translation results For instance , results reported in Junczys-Dowmunt et al . ( 2016 ) for Phrase-Based Statistical Machine Translation ( PBSMT ) ( Koehn et al. , 2003 ) and Neural MT ( Bahdanau et al. , 2014 ) on the 6 official languages2 of the United Nations ( UN ) Parallel Corpus ( Ziemski et al. , 2016 ) indicate a disparity between EN/ES/FR and AR/RU/ZH in BLEU ( Papineni et al. , 2002 ) — translation performance in the latter group is generally worse , regardless of the MT algorithm used . AR and RU are traditionally considered morphologically complex ( see e.g . Minkov et al . ( 2007 ) , Seddah et al . ( 2010 ) and proceedings of related workshops in subsequent years for Statistical Parsing of Morphologically Rich Languages ) , and ZH morphologically frugal 1see e.g . work from conference ( https : //facctconference.org ) and workshops in previous years on “ Fairness , Accountability , and Transparency ” ( FAccT ) 2Arabic ( AR ) , English ( EN ) , Spanish ( ES ) , French ( FR ) , Russian ( RU ) , and Chinese ( ZH ) ( for its lacking determiners and plural or tense markers ) ( Koehn , 2005 ) . While Koehn ( 2005 ) found translating into EN to be easier than into morphologically rich languages based on BLEU scores of word tokens from PBSMT systems of 110 language directions from the 11 Europarl languages then , Bugliarello et al . ( 2020 ) found it is easier to translate out of EN than into it based on 21 Europarl languages in BPEs ( Byte Pair Encodings ) ( Sennrich et al. , 2016 ) with the Transformer ( Vaswani et al. , 2017 ) in a new metric , cross-mutual information . Disparity in language modeling results Disparate performances across different languages seem to have been implicitly accepted in that it is often believed that some languages are harder to model than others . Bender ( 2009 ) advocated the relevance of linguistic typology for the design of languageuniversal NLP systems due to differences based on crosslinguistic structural notions , such as parts of speech and morphological complexity . Cotterell et al . ( 2018 ) studied ( monolingual ) language models ( LMs ) on the 21 Europarl languages using a word-level 7-gram standard Kneser & Ney ( 1995 ) model and LSTM-LMs ( Sundermeyer et al. , 2012 ) with characters and lemmatized forms in informationtheoretic terms , and found morphological complexity to be the primary culprit for the differences in performance . Mielke et al . ( 2019 ) supplemented with the multilingual Bible corpus ( Mayer & Cysouw , 2014 ) to extend the coverage to 69 languages , tested on RNN-LMs ( an implementation of LSTM ( Hochreiter & Schmidhuber , 1997 ) ) with characters and BPEs , but concluded that basic data statistics in vocabulary size ( |V | ) and sequence length were the most predictive features of performance . We noticed , however , a discrepancy in the results from Mielke et al . ( 2019 ) for ZH — it came out as the least difficult for the character model , but it is the 6th most difficult language for the BPE model . As different input representations have been tested with different architectures with divergent results in different metrics in previous studies , each of them only testing with one data size , we decided to investigate the matter more systematically once again with statistical comparisons of score distributions between languages . 1.2 RESEARCH QUESTIONS AND CONTRIBUTIONS . Research questions Are there any statistically significant differences in hardness when it comes to Conditional-Language-Modeling ( CLMing ) languages which have been traditionally considered morphologically rich ( AR and RU ) and poor ( ZH ) with the 6-layer Transformer ? Is morphological complexity inherent in language ? When is the notion of morphological complexity relevant in computing ? Summary of findings and insights Based on our bilingual CLMing setup with the UN Parallel Corpus in the data size range from 102 to 106 lines on the character , byte , and word levels , we find : 1 . Language has more finer-grained dimensions with different representations and learning patterns . Hardness in modeling is relative to its representation granularity ( representation relativity ) . 2 . There is neutralization of source language instances , i.e . there are no statistically significant differences between source language pairs . Only pairs of target languages differ significantly . 3 . On the character and byte levels , hardness is correlated with statistical properties concerning sequence length and |V | of a language , regardless of its morphological profile . As it is possible to eliminate performance disparity by decomposing sequences into finer-grained units in characters and bytes , we show that morphological complexity is not an intrinsic property of language . Unless word-based methods are used , or unless we implement/model it explicitly , the notion of morphological complexity is irrelevant in computing . 4 . On the word level , hardness is correlated with |V | , and a complexity hierarchy arises through the manual preprocessing step of word tokenization . This complexity/disparity effected by word segmentation can be improved by subword tokenization but can not be eliminated due to the fundamental qualitative differences in the definition of a “ word ” being one that neither holds universally nor is suitable/consistent for fair crosslinguistic comparisons . 5 . Representational units of finer granularity can help close the gap in performance disparity . Orthogonal to our main research questions , we also observed 2 types of sample-wise non-monotonicity — Double Descent ( Belkin et al. , 2019 ; Nakkiran et al. , 2020 ) and erraticity . For reasons due to length and scope for this paper , we will defer discussions and analyses of these beyond what is addressed in § 5 to future work . Outline of the paper In § 2 , we define our method and experimental setup . We present our results and analysis on the primary representations in § 3 and those from secondary set of controls in § 4 in a progressive manner to ease understanding . Meta analysis on performance disparity and other discussions are in § 5 . 2 METHOD AND DEFINITIONS . Conditional language modeling CLMing is the modeling of the probability of the next token , given the history of the preceding tokens and conditioning context . In our case , such conditioning context is a line from the source language . To explicitly focus on modeling the complexities that may or may not be intrinsic to the languages , we study the more fundamental process of CLMing without performing any translation . This allows us to eliminate confounds associated with generation and other evaluation metrics . One could think of our setup as estimating conditional probabilities with the Transformer , with a bilingual ( one-to-one ) setup where the perplexity ( PP ) of one target language ( ltrg ) is estimated given the parallel data in one source language ( lsrc ) , where lsrc ̸= ltrg . We focus on the very basics and examine the first step in our pipeline — input representations , holding everything else constant . Instead of measuring absolute cross-entropy scores at one data size , we evaluate the relative differences between development ( dev ) set score distributions between languages . Controlled experiments as basic research for scientific understanding of language data Using the UN Parallel Corpus , the data from which the MT results in Junczys-Dowmunt et al . ( 2016 ) stem , we perform a series of controlled experiments with the Transformer , holding the hyperparameter settings for all 30 one-to-one language directions from the 6 languages constant . We control for size ( from 102 to 106 lines ) and language with respect to representational granularity . We examine 3 primary representation types/levels — character , byte ( UTF-8 ) , and word , and upon encountering some unusual phenomena , we perform a secondary set of controls with 5 alternate representations — on the character level : Pinyin and Wubi ( ASCII representations for ZH phones and character strokes , respectively ) , on the byte level : code page 1256 ( for AR ) and code page 1251 ( for RU ) , and on the word level : BPE . These symbolic variants allow us to manipulate the statistical properties of the representations , while staying as “ faithful ” to the language as possible . We adopt this symbolic datacentric3 approach because we would like to more directly interpret the confounds , if any , that make language data different from other data types . We operate on a smaller data size range as this is more common in traditional language sciences and one of our higher goals is to bridge an understanding between language sciences and engineering ( the latter being the dominant focus in NLP ) , and between traditional symbolic sciences and ML . We run statistical tests to identify the strongest correlates of performance and to assess whether the differences between the mean performance of different groups are indeed significant . We are concerned not with the absolute scores , but with the differences between score distributions from different languages . Fair evaluation with multitexts Multitexts are multiway parallel corpora . The UN Parallel Corpus is a 6-way parallel corpus consisting of manually translated UN documents from the 25-year period between 1990 and 2014 . We use the UN Parallel Corpus because it contains languages conventionally regarded as morphologically rich and poor , has quality and size sufficient for evaluation , and more importantly , it comes as raw texts ( untokenized ) , unlike both of the corpora that Mielke et al . ( 2019 ) used . Detokenization ( esp . the evaluation thereof ) is not a trivial task . Fair information-theoretic evaluation metric Most sequence-to-sequence models are optimized using a cross-entropy loss , defined as : H ( t , s ) = − N∑ i=1 log2 p ( ti | t < i , s ) ( 1 ) where t is the sequence of tokens to be predicted , ti refers to the ith token in that sequence , s is the sequence of tokens conditioned on , and N = |t| . It is customary to report scores as PP , which is 2 1 N H ( t , s ) , i.e . 2 to the power of the cross-entropy averaged by the number of tokens in the dev 3Two testing/evaluation approaches — data-centric : hold the algorithm constant and tweak data , vs. algorithmcentric : hold data constant and tweak the algorithm . data . Cotterell et al . ( 2018 ) proposed to use “ renormalized ” PP to evaluate LMs tokenwise fairly by dividing the overall bits per utterance/sequence by one constant token count in any one arbitrary language ( e.g . so to arrive at “ bits per character ” in one language to evaluate all languages ) . But we find that it is not necessary to assign a perspective that is centered on any one particular language , when we can evaluate simply by the total number of bits for a larger portion of texts/sequences . This can be a fairer , more general and flexible way of evaluating data that has not been or can not be perfectly segmented or aligned line by line . We hence used instead unnormalized PP , i.e . the total number of bits needed to encode the dev set ( 3,077 lines per language , after length filtering , in our case ) . As the implementation we used only reports PP , we transformed it back to entropy as defined above via H ( t , s ) = log2 PP ( t|s ) ×N . Disparity/Inequality In the context of our CLMing experiments , we consider there to be “ disparity ” or “ inequality ” between languages l1 and l2 if there are significant differences between the performance distributions of these two languages with respect to each representation . Here , by performance we mean the number of bits required to encode the held-out data using a trained CLM . With 30 directions , there are 15 pairs of source languages ( lsrc1 , lsrc2 ) and 15 pairs of target languages ( ltrg1 , ltrg2 ) possible . We compare the source languages among each other , and the target languages among each other . Each lsrc or each ltrg consists of scores from all models trained across various sizes and directions . To assess whether the differences are significant , we perform unpaired two-sided significance tests with the null hypothesis that the score distributions for the two languages are not different . Upon testing for normality with the Shapiro-Wilk test ( Shapiro & Wilk , 1965 ; Royston , 1995 ) , we use the parametric unpaired two-sample Welch ’ s t-test ( Welch , 1947 ) ( when normal ) or the non-parametric unpaired Wilcoxon test ( Wilcoxon , 1945 ) ( when not normal ) for the comparisons . We use the implementation in R ( R Core Team , 2014 ) for these 3 tests . To account for the multiple comparisons we are performing , we correct all p-values using Bonferroni ’ s correction ( Benjamini & Heller , 2008 ; Dror et al. , 2017 ) and follow Holm ’ s procedure4 ( Holm , 1979 ; Dror et al. , 2017 ) to identify the pairs of l1 and l2 with significant differences after correction . We report all 3 levels of significance ( α ≤ 0.05 , 0.01 , 0.001 ) for a more comprehensive overview . In contrast to Dror et al . ( 2017 ) , which aimed to compare the performance of different algorithms , we compare languages ( in the context of computing ) . Experimental setup The systematic , identical treatment we give to our data is described as follows with further preprocessing and hyperparameter details in Appendices A and B , respectively . After filtering length to 300 characters maximum per line in parallel for the 6 languages , we made 3 subsets of the data with 1 million lines each — one having lines in the order of the original corpus ( dataset A ) and two other randomly sampled ( without replacement ) from the full corpus ( datasets B & C ) . Lines in all datasets are extracted in parallel and remain fully aligned for the 6 languages . For each run and each representation , there are 30 pairwise directions ( i.e . one lsrc to one ltrg ) that result from the 6 languages . We trained all 150 ( for 5 sizes ) 6-layer Transformer models for each run using the SOCKEYE Toolkit ( Hieber et al. , 2018 ) . We optimize using PP and use early stopping if no PP improvement occurs after 3 checkpoints up to 50 epochs maximum , taking the best checkpoint . Characters and bytes are supposed to mitigate the out-of-vocabulary ( OOV ) problem on the word level . In order to assess the effect of modeling with finer granularity more precisely , all vocabulary items appearing once in the train set are accounted for ( i.e . full vocabulary on train , as in Gerz et al . ( 2018a ; b ) ) . But we allow our system to categorize all unknown items in the dev set to be unknown ( UNK ) so to measure OOVs ( open vocabulary on dev ( Jurafsky & Martin , 2009 ) ) . To identify correlates of performance , we perform Spearman ’ s correlation ( Spearman , 1904 ) with some basic statistical properties of the data ( e.g . length , |V | , type-token-ratio , OOV rate ) as metrics — a complete list is provided in App . C. See App . D for sample construction for statistical comparisons .

Summary: This paper asks: Are languages which have been traditionally considered morphologically rich (Arabic and Russian) and poor (Chinese) equally hard to learn by transformer-based conditional language model (CLM).  To quantify how hard it is for CLM to model a language $l_1$, this paper proposes to train a transformer-based encoder-decoder model to translate $l_1$ to another language $l_2$, and uses the \"unnormalized perplexity score\" (the number of bits needed to encode the same development set for the target language) as an indicator for the \"hardness\" for CLM to learn the $l_2$. For two languages $l_1$ and $l_2$, there exists **disparity** if their perplexity distributions are significantly different. This paper includes extensive experiments across 30 language pairs using three different representation level (including character, byte, and word) for different dataset sizes.  They have the following findings:  1.  They find that the performance disparity between morphologically     rich and poor languages cannot be justified due to morphological     complexity. Representation level matters more.  2.  They find that the hardness for CLM to model a language is related     to the level of representation used to encode the language, the     vocabulary size, and/or the sequence length.  3.  They find that it is equally hard to learn a certain target language     (that are tested in their experiment) using transformer-based     conditional language models, independent of the source language.