A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Models
Authors: Yi Zhou
Jose Camacho-Collados
Danushka Bollegala
Venue: EMNLP 2023 Main (2023)
Keywords: mask language models
social bias
task performance
model size
model type
training corpora
tokenisation
language

Abstract: Various types of social biases have been reported with pretrained Masked Language Models (MLMs) in prior work. However, multiple underlying factors are associated with an MLM such as its model size, size of the training data, training objectives, the domain from which pretraining data is sampled, tokenization, and languages present in the pretrained corpora, to name a few. It remains unclear as to which of those factors influence social biases that are learned by MLMs. To study the relationship between model factors and the social biases learned by an MLM, as well as the downstream task performance of the model, we conduct a comprehensive study over 39 pretrained MLMs covering different model sizes, training objectives, tokenization methods, training data domains and languages. Our results shed light on important factors often neglected in prior literature, such as tokenization or model objectives.

A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Models
Yi Zhou
Jose Camacho-Collados Danushka Bollegala
Cardiff University" University of Liverpool Amazon
{Zhouy131 CamachooCllaadoss @cardiifeefafffffaeeaa ff .ww.ifffccc...c. ac. uk danushka@liverrpool uk
Abstract
Nevertheless, it remains unclear as how these factors influence social biases in MLMs and their downstream task performance.
Various types of social biases have been re- ported with pretrained Masked Language Mod- els (MLMs) in prior work. However, multi- ple underlying factors are associated with an MLM such as its model size, size of the train- ing data, training objectíves, the domain from which pretraining data is sampled, tokenization, and languages present in the pretrained corpora, to name few. It remains unclear as to which of those factors influence social biases that learned by MLMs. To study the relationship between model factors and social biases learned by an MLM, well the downstream task performance of the model, we conduct a comprehensive study over 39 pretrained MLMs covering different model sizes, training objec- tives, tokenization methods, training data do- mains and languages. Our results shed light on important factors often neglected in prior litera- ture, such as tokenization or model objectives.
Evaluating the impact of these factors chal- lenging due to three main reasons: (a) The factors that we consider within a model are not indepen- dent, rather, they exhibit complicated interdepen- dence and affect the performance of models simul- tancously. (b) MLMs are diverse with different architectures, configurations and parameters. The diversity across models requires the need for gen- eralisation and abstraction when considering the values of factors. (c) Many recent works proposed debiasing methods to mitigate social biases MLMs (Webster et al., 2020; Lauscher et al., 2021; Schick et al., 2021; Guo et al., 2022). However, most debiasing methods tend to worsen the per- formance of MLMS in downstream tasks (Meade et al., 2022). Therefore, it is crucial to consider the trade-off between social bias and downstream task performance when comparing MLMs.
1 Introduction
Masked Language Models (MLMs) have achieved promising performance in many NLP tasks (Devlin et al., 2019; Liu et al., 2019; Liang et al., 2023). However, MLMs trained on massive amounts of textual training data have also been found to encode concerning levels of social biases such as gender and racial biases (Kaneko and Bollegala, 2019; May et al., 2019; Dev al., 2020; Silva et al., 2021; Kaneko et al., 2022). spite of the overall success MLMs across NLP tasks, such biases within MLMs raise ethical considerations and underscore the need for debiasing methods to ensure fair and unbiased MLMs.
To address the non-independent issue factors, we propose a method using Gradient Boosting (Fre- und and Schapire, 1997) to consider dependencies among factors. Moreover, we use the coefficient of determination (R"; Nagelkerke et al., 1991) measure to analyse the importance of factors. Regarding the diversity of MLMs, converge broad set of MLMS across multiple languages and 4 domains, resulting in 39 MLMs for evalua- tion in total. Moreover, we incorporate TweetEval and GLUE to evaluate the downstream task perfor- mance of MLMs, meanwhile, evaluating their so- cial biases intrinsically using All Unmasked Likeli- hood with Attention weights (AULA; Kaneko and Bollegala, 2022) two benchmarks StereoSet (SS; Nadeem al., 2021) and crowdsourced stereotype pairs benchmark (CP: Nangia al., 2020a). Note that we are not proposing novel bias evaluation measures this paper. Instead, we use existing metrics such as AULA to evaluate social biases.
On the other hand, MLMs are trained by con- sidering and optimising various underlying factors that contribute to their performance on downstream tasks. These factors include but are limited to parameter size, tokenization methods, training ob- jectives and training corpora. The performance of MLMs is affected by the interplay of such factors.

---

Our experimental results indicate that model size, training objectives and tokenization are the three most important categories of factors that affect the social bias and downstream task performance of MLMs. Interestingly, we observe that models using Byte-Pair Encoding (BPE; Sennrich et al., 2016) include lower level of social biases, while achiev- ing the best downstream performance compared to models using other tokenization methods. Overall, multilingual models tend to have less biases than their monolingual counterparts.
in transfer learning. Choenni et al. (2021) inves- tigated the types of stereotypical information that are captured by pretrained language models and showed the variability of attitudes towards various social groups among models and the rapid changes in emotions and stereotypes that occur during the fine-tuning phase.
Existing bias evaluation methods use different strategies such as pseudo likelihood (Kaneko and Bollegala, 2022), cosine similarity (Caliskan et al., 2017: May et al., 2019), inner-product (Ethayarajh et al., 2019), to name few. Independently of any downstream tasks, intrinsic bias evaluation mea- sures (Nangia al., 2020b; Nadeem et al., 2021; Kaneko and Bollegala, 2022) evaluate social bi- ases in MLMs stand alone. However, given that MLMs are used to represent input texts in vari- ety of downstream tasks, multiple previous works have proposed that social biases should be evalu- ated with respect to those tasks (De-Artcaga et al., 2019; Webster et al., 2020). Kaneko and Bolle- gala (2021) showed that there exists only weak correlation between intrinsic and extrinsic social bias evaluation measures. In this paper, we use an intrinsic bias evaluation measure, namely AULA, to evaluate social biases in MLMs. AULA has been shown to be the most reliable bias evaluation mea- sure (Kaneko and Bollegala, 2022), hence we use it as our bias evaluation measure.
2 Related Work
As MLMs have been successfully applied to di- verse NLP tasks, it is important to study the factors that determine their social biases. Rogers et al. (2020) reviewed the current state of knowledge regarding how BERT works, what kind of informa- tion it learns and how it is encoded, typically al- terations its training objectives and architecture, the overparameterizztion issue and approaches to compression. Xia et al. (2020) studied contextu- alised encoders in various aspects and discussed the trade-off between task performance and the potential harms contained in the pretraining data. Later, Pérez-Mayos et al. (2021) investigated the effect of pretraining data size on the syntactic capa- bilities of RoBERTa and they showed that models pretrained with more data tend to encode better syntactic information and provide more syntactic generalisation across different syntactic structures. However, these studies focus on MLMs in down- stream tasks, while none consider the social biases in MLMs.
Although we specifically focus on MLMs in this paper. evaluating the performance predictors for Neural Architecture Search (NAS) (White et al., 2021; Elsken et al., 2019) has been conducted for designing high performance neural networks. Gen- cralisability of the identified factors is an important in NAS because the selected neural architecture would be trained on different datasets to obtain different models. Although it would be ideal to perform a controlled training of MLMs where we experimentally fix all other factors except for a sin- gle target factor and then analyse the trained MLM, this is an expensive undertaking given the compu- tational cost of training MLMs on large datasets. On the other hand, we consider already pre-trained MLMs that are publicly made available and do not train any MLMs during this work.
On the other hand, models trained for different downstream tasks have been found to exhibit social biases. Kiritchenko and Mohammad (2018) evalu- ated gender and racial biases across 219 automatic sentiment analysis systems and discovered statisti- cally significant biases occurring in several systems. Díaz al. (2018) studied age-related biases in sen- timent classification and discovered that significant age bias is encoded in the output of many sentiment analysis systems as well as word embeddings.
Zhao et al. (2020) focused on gender bias in multilingual embeddings and its influence the process of transfer learning for NLP applications. They showed that the level of bias in multilingual representations varies depending on how the em- beddings are aligned to different target spaces, and that the alignment direction can also affect the bias
3 Analysis of Factors
In order to study the impact of different factors in MLMs, we consider 30 factors and split them into categories. The details of each individual factor

---

are provided in Appendix C.
better word-level comprehension and contextual understanding for MLMs (Cui et al., 2021). Apart from these two masking techniques, we consider three other training objectives: (a) next sentence prediction, (b) sentence ordering prediction, and (c) mention reference prediction. We consider the training objectives to be binary and assign if each of them is used and 0 otherwise.
3.1 Model Size
Models with smaller sizes are generally more lightweight and require less computational re- sources, making them suitable for deployment on resource-constrainee devices or in environments with limited processing power. However, larger models tend to have better performance on down- stream tasks but demand more memory, computa- tional power, and longer inference times. On the other hand, the different architectures of models have various numbers of layers as well as training parameters. Recently, MLMs have achieved im- pressive results on downstream tasks by scaling model size or training with larger datasets (Con- neau et al., 2020; Goyal et al., 2021; Liang et al.. 2023). To investigate the impact of model size, we consider 3 factors: (I) parameter size, (2) number of layers and (3) vocabulary size.
Model distillation is a training technique aim- ing to train small student model to transfer the knowledge and capabilities from a larger teacher model (Hinton et al., 2015). This technique has been shown to effectively compact large language model, while retaining comparable performance compared to the original model (Sanh et al., 2019; Wu et al., 2022). Model distillation is regarded as a binary factor, which is assigned 1 if an MLM uses model distillation, otherwise, it returns 0.
3.3 Training Corpora
The parameter size is considered as a categorical feature, which we divide the parameter size of an MLM into 3 categories according to pre-defined ranges. Specifically, we assign S if the parameter size of an MLM is less than 100M, M if the size is within 100M-300M, and L if the size is greater than 300M. Similarly, we convert the vocabulary size into the same three categories for the models with vocabulary sizes less than 50K, within 50K- 100K and greater than 100K, respectively. For the number of layers, we use the number as a feature: 6, 12 and 24 layers.
Training corpora are the collections of texts or data used to train MLMs. According to Kalyan et al. (2021), training corpora can be classified into four types: (1) general, (2) social media, (3) language- specific and (4) domain -ppeciic. In order to con- duet a comprehensive study of MLMs trained us- ing different types of training corpora, we cover all four types of training corpora, resulting in four different domains (including general domain): (1) General Domain: BookCorpus (Books), Wikipedia (Wiki), Common Crawl-News (CCNews), Open- WebText (OWT), and Stories; (2) Social Media: Tweets and the Reddit Abusive Language English dataset (RALE); (3) Legal Domain: Patent Liti- gations, Caselaw Access Project (Caselaw) and Google Patents Public Data (GooglePatents); (4) Biomedical Domain: Full-text biomedical papers from the Semantic Scholar Open Research Corpus (S2ORC), PubMed Abstracts (PMed), PMC Full- text articles and the Medical Information Mart for Intensive Care III (MIMIC3); Finally, we also con- sider multilingual corpus: CommonCrawl Corpus in 100 languages (CC100). Each of the training corpora considered as an individual binary factor.
3.2 Training Methods and Objectives
In this category, we consider the methods used dur- ing model pretraining as well as the training objec- tives of MLMs. First, we take into account different masking techniques, starting with the masking tech- nique initially proposed for training BERT (Devlin et al.. 2019). Masked language modelling is an objective used during pretraining to improve the model's understanding, in which a certain percent- age of words in the input text are randomly selected and replaced with a special [MASK] token, then the model is trained to predict the original word based on its context. Later, they further proposed whole word masking, which aims to improve the handling of individual words with a context. Rather than randomly selecting WordPiece (Wu et al., 2016) produced subtokens to mask, whole word mask- ing always masks the entire words at once, which has been shown to reduce ambiguity and enable
Owing to the domain of an MLM being asso- ciated with the training corpora sampled in that certain domain, as a separate factor. This domain factor is included as a categorical factor, with 4 different domains as categories: general domain, social media, legal domain and biomedical domain. Finally, we take
additionally consider domain

---

into account continuous training as a binary factor which takes 1 if an MLM is continuously trained on training corpora from different domains and if the model is trained from scratch.
Model Type
Models
General Domain
mherta-huse, robertalargge, hert-hnse-cased, hert-
large-cased, hert- crrt-hoe--uncsees bert-lageeunsaseed
srge--cred-whalee.. www.e-wwod--aaakeeeeee. masking, alhert-hase v2, alhert-large-
large x2, albert-xlarge
slarge v2,
(Monalingual)
alhert axlarge v2, distilbert base cased, distilbert- uncased, distilraberaa
large uncased-
3.4 Tokenization
whole word masking. nielsrfeoreff roberta large, nielsolcoref' roberta-baar niehsr/coref bert bert-lage
nielst/coref bert base,
Tokenization is essential process of breaking down a sequence of text into smaller units, which is able to convert unstructured text data into for- mat that can be processed by ML.ss Prior works study different tokenization methods on MLMs in different languages (Park et al., 2020; Rust et al., 2021; Toraman et al.. 2023), however, the impact of tokenization methods on social bias in MLMs and different downstream tasks remains unrevealed. Therefore, we consider the three most commonly used tokenization methods categorical factors: BPE, WordPiece and SentencePiece.
Domain-specific
candiffolltwitter roberta hase. candiffnlptwwiter-- scratch roberta hase,
vinai/bertweet hase, GroNLPhtateEET: al-
nual/bertweet-larg
lenai/biomed roberta base. sabbelegal--obertobeess emilyyalsenteer/Biocc ClinicaaCCC...BE..TT
(Monolingual)
bionlphluuter..pumne himoommemmmmicmmmmeeeeee -11
768_A-12, 12, hionlyybluchert, puchmeeerttahhmddd.o rubmod__imicc_mmm
24_ H. 1024_A-16,, paaiffinpphwiiiee--mhrrttt
2022. 154m,
cardifhhlptwwitee--bboe roherta- hasettaaaaaeeee 2022
154m
ber--bse--mutilinguu-aaes bert-bne-matrllnganll (Multilingua))
General Domain
uncased, distilbert-hase aanse-mutilingull-cees xlm-moberta-hase hasee sim-roherta-lng wim-v- tim-v-hase
Domain-specife (Muliiligual)
cardifinlptwitter xlm roberta base
Table I: Summary of the MLMs in the analysis.
3.5 Language
In this category. we consider both monolingual and multilingual MLMs. Specifically, we regard lan- guage categorical factor and use English and Mulrilingual to represent if an MLM is monolin- gual or multilingual, respectively. addition, the number of languages is also considered as a sepa- rate factor, which categorical and takes the actual number of languages an MLM trained on.
Monolingual and domain-specifie models We consider the MLMs either directly or continuously trained in domain-specific English corpora with different settings, consisting of RoBERTa in social media domain (Barbieri et al., 2020), BERT in so- cial media domain (Nguyen et al., 2020; Caselli et al., 2021), RoBERTa in legal domain (Geng et al., 2021), RoBERTa in biomedical domain (Gu- rurangan et al., 2020), and BERT in biomedical domain (Alsentzer al., 2019; Lee et al., 2020; Peng al., 2019).
4 Masked Language Models
To conduct a comprehensive study of different fac- tors of MLMs affecting social bias and task perfor- mance, we evaluate 39 pretrained MLMs, which we divide into four categories follows. Table 1 summarizes all models.
Multilingual and general domain models We take into account the multilingual MLMs with different settings pretrained the general do- main, containing different settings of multilingual BERT (Devlin et al., 2019), DistilIBERT (Sanh et al., 2019) and XLM-R (Conneau et al., 2020).
Monolingual and general domain models We take the monolingual MLMs with different set- tings pretrained in general domain, including RoBERTa (Liu et al., 2019), BERT (Devlin et al., 2019), ALBERT (Lan et al., 2019), Dis- tilBERT (Sanh et al., 2019), CoreRRoBERTa and CorefBERT (Ye et al., 2020).
Multilingual and domain-specific models We select the multilingual MLMs pretrained in domain- specific corpora, containing XLM-T (Barbieri al., 2022).
5 MLM Evaluation Metries and Tasks
'In our initial selection of models, nielsr/coref- ceaee-rrbbbraareee vinai/bertweet-lne larree, xlm-role -omrraaeta- base and facebook/xlm- -laae- attained an unusual poor perfor- mance on the Corpus of Linguistic Acceptability (CoLA) (i.c., a subtask of GLUE) and TveetEval. This is likely caused by an implementation issue that would need some modification. Therefore, to avoid potentially false outliers, we omitted these five models when evaluating TwcetEval and GLUE.
Our goal in this paper is study the relationship between model factors and social bias as well as downstream task performance in pretrained MLMs. For this purpose, we conduct our evaluations using three different types of tasks.

---

5.1 Social Bias
profession, while CP is crowdsourced and anno- tated by workers in the United States, consisting of nine types of social biases: race, gender, sex- ual orientation, religion, age, nationality, disabil- ity, physical appearance, and socioeconomic sta- tus/occupation. We follow the work from Kancko and Bollegala (2022)² and use the default setting for evaluation. We denote the AULA computed on the CP and SS datasets by A-CP and A-SS, respec- tively, in the rest of the paper.
For an MLM under evaluation, we compare the pseudo-likelihodd scores returned by the model for stereotypical and anti-stereotypical sentences using AULA (Kaneko and Bollegala, 2022). AULA eval- uates social biases by taking into account the MLM attention weights the importance of tokens. This approach shown be robust against frequency biases in words and offers more reliable estimates compared to alternative metrics used to evaluate social biases in MLMs.
5.2 Downstream Performance
Following the standard evaluation protocol, we provide AULA the complete sentence S .1..... 4,. which contains a length |S| sequence of tokens f;, to MLM with pretrained parame- ters B. We compute the Pseudo Log-Likelihood, denoted by PSL(S), to predict all tokens S ex- cluding begin and end of sentence tokens. The PLL(S) score of sentence S given by (1) can be used to evaluate the preference expressed by an MLM for S:
To further investigate the impact factors of an MLM in terms of its downstream tasks per- formance, we evaluate MLMs on two additional benchmark datasets: The General Language Un- derstanding Evaluation (GLUE) benchmark (Wang et al., 2018)² and social media TweetEvvl bench- mark (Barbieri et al., 2020)."
GLUE GLUE is comprised of tasks for evalu- ating natural language understanding systems. The tasks in GLUE are Corpus of Linguistic Accept- ability (CoLA; Warstadt et al., 2019), Stanford Sentiment Treebank (SST-2; Socher et al., 2013), Microsoft Research Paraphrase Corpus (MRPC; Dolan and Brockett, 2005). Semantic Textual Sim- ilarity Benchmark (STS-B; Cer et al., 2017). Quora Question Pairs (QQP; Iyer al., 2017). Multi- Genre NLI (MNLI; Williams et al., 2018), Question NLI (QNLI; Rajpurkar et al., 2016), Recognizing Textual Entailment (RTE; Dagan et al., 2005; Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009) and Winograd NLI (WNLI; Levesque et al., 2012). These tasks are framed classifi- cation tasks for either single sentences pairs of sentences. We follow the finetuning procedures from prior work (Devlin et al., 2019; Liu et al., 2019) and report results on the development sets.
S
PLL(S) = S
log P(t|S:0)
(1)
Here O; is the average of all multi-head atten- tion weights associated with 4, while Stt, 0) is the probability assigned by the MLM to token t; conditioned on S.
Given a sentence pair, the percentage of stereo- typical (5*t) sentence preferred by the MLM over anti aatiistereotypical ( one is considered as the AULA bias score of the MLM and is given by (2):
100 N
Σ I(PLL(S") PLL((*)))
AULA
(2)
(8tt 89L)
Here, N is the total number of text instances and I is the indicator function, which returns 1 if its argument is True and ( otherwise. AULA score given by (2) falls within the range [o, 100] and an unbiased model would return bias scores close to 50, whereas bias scores less or greater than 50 indicate bias directions towards the anti- stereotypical or stereotypical group, respectively.
TweetEval TweetEval is a unified Twitter bench- mark composed of seven heterogeneous tweet classification tasks. The tasks in TweetEval are emoji prediction (Barbieri et al., 2018), emo- tion recognition (Mohammad al., 2018), hate speech detection (Basile et al., 2019), irony de- tection (Van Hee et al., 2018), offensive language identification (Zampieri et al., 2019), sentiment
Social Bias Benchmarks We conduct experi- ments the two most commonly used social bias evaluation datasets for MLMs: StereoSet (SS) and Crowdsoureed Stereotype Pairs benchmark (CP). SS contains associative contexts, which cover four types of social biases: race, gender, religion, and
evaluate.bias_Ine com/Kanneooasahhrr/ hhttsss//itthiioo/aaaaeeeeeaenas
'nttss://gluebenchhertt
thub. httpt//ccw.ccccrr.fa/////tttttte

---

analysis (Rosenthal al., 2017) and stance detec- tion (Mohammad al., 2016). We use the default setting as the TweetEval original baselines to fine- tune pretrained MLMs on the corresponding train- ing set and report average results on the test set after 3 runs.
squared error (RMSE) for each prediction task. We use the regression models implemented in sklearn with the default parameters and take the averages over three independent runs.
Regression model comparison The comparison of different regression models trained using the features from the monolingual MLMs in the gen- eral domain is shown in Table while the per- formance of regression models trained using the features from all the MLMs is shown in A.I. From Table 4, we observe that gradient boosting obtains the best performance in terms of both It and RMSE on both A-CP and TweetEval, while the decision tree obtains the best performance on A- ASS. Almost all of the regression models return negative R² scores for GLUE, which proves hard to predict from the analysed factors. An error anal- ysis of each GLUE subtask on gradient boosting in terms of RD scores is shown in Appendix B. contrast, the three remaining social-related evalu- ations, including TweetEval, can be predicted to reasonable extent. The linear regression model obtains the lowest R²² scores for both A-CP and GLUE, which indicates that linear model may not be suitable for predicting the performance of MLMs. Given the results, we decided use gradi- ent boosting for the rest the experiments in this paper.
5.3 Correlation between tasks
Table 2 shows the Pearson and Spearman's correla- tion coefficients between each pair of task perfor- mances the 39 MLMs. We observe moderate correlation between A-SS and GLUE, which indi- cates that better performance in GLUE entails a higher stereotypical bias in SS (this also true for CP but to a much lesser extent). In contrast, there is no significant correlation observed between the models' performance on downstream tasks, i.e., TweetEval vs. GLUE.
Task pair
Pearson
Spearman
A-CP vs. A-SS
0.607*
0.623
A-CP Vs. TweetEval
0.193
0.214
A-CP GLUE
0.244
0.286
A-SS vs. TweetEval
0.338
0,304
A-SS vs. GLUE
0.382
0.487'
TweetEval vs. GLUE
0.229
0.309
Table 2: Pearson and Spearman correlations between the models' performance pairs of tasks, where t denotes statistically significant (p < 0.05) correlations.
Feature importance In addition, to investigate the influence of the factors the performance of MLMs using R² and RMSE, compute the importance score of each factor after training the regression model using the Gini importance imple- mented in sklearn. The Gini importance com- puted as the normalized total reduction of the cri- terion brought by that feature. It calculates the global contribution of each feature by aggregat- ing the losses incurred in each split made by that feature (Delgado- Panadero et al., 2022).
6 Regression Analysis
To investigate the importance of the different fac- tors on social bias and the task performance of MLMs, we train regression model.
6.1 Experimental setting
We generate the features for each MLM according its factors as deseribed in $3. An example of the features of an individual model is given in Table 3. These features are fed into a regression model as input for training, using both social bias and task performance as output.
DDelgado-Panadero
6.2 Results
Feature importance Table shows the Gini im- portance scores of each factor. Due to space con- straints in this table, we have omitted the factors that received score of 0 importance (the full table is shown in $ A.2). The parameter size obtains the largest importance score for A-CP, while it is the second most important factor for TweetEval and GLUE. Tokenization is the most important factor for A-SS and the second most for A-CP.
In order to select the best regression model for this purpose, we compare the performance of 6 dif- ferent regression models on each prediction task, namely gradient boosting, support vector machine, gaussian process regression, decision tree, random forest and linear regression. We compute the perfor- mance of the regression models by means of their coefficient of determination (R²) and root mean

---

Model
Features
bert-large-cased
Parameter Size: L. Distill: Train-Books: 1, Train Wiki: 1, Train-CChewe CCnews: Train-OWT:
0. Train-Stories: 0. Train-Tweets O. Train-RAL:F: O. Train-Patenn O, Train-Casclaww 0. Train-
GooglePatents: Train-S2ORC: 0, Train- MTMin- 0, Train-PMess O Train PMC: 0. Train-CC100:
0, Cont-train: 0. Uncased: 0. Domain: general, Language: en, Number of Languages: Vocabulary size: S, Tokenization: WordPiece, Number of Layers: 24, Masked Language Modelling: 1, Whole Word Masking: 0, Next Sentence Prediction: 1, Sentence Order Prediction: 0, Mention Reference
Prediction: 0
Table 3: Example of features MLMs given to a regression model for training according to considered factors.
Models
A-CP
A-SS
TweetEval
GLUE
Factors
A-CP
A-SS
TweetEval
GLUE
Gradient Boosting
0.594
0.652
0,481
-0.201
Without Removing
0.594
0.652
0.481
40.201
SVM
0.033
0.164
0.135
-0.053
Parameter Size
0,381
0,567
0.121
-0.288
Guussian Process
0.172
0.129
0.323
-0.077
Distill
0.608
0.682
0.468
40.254
Decision Tree
0.229
0.776
0.072
-1.399
Train-CCNews
0.594
0.641
0.488
40.222
Random Forest
0.434
0.554
-0.193
-0.033
Train-OWT
0,594
0.639
0,464
-0.218
Linear Regression
0.016
0.741
0.353
8466
Train-Stories
0.598
0.640
0.499
-0.228
Gradient Boosting
2.661
2.385
1.459
1 1883
Cont-train
0.579
0.634
0.401
-0.174
RMSE
SVM
4.107
3.696
1,886
1.764
Uncased
0,590
0.278
0.373
-0.420
Gaussian Process
3.801
3.772
L,668
1.784
Vocabulary Size
0.579
0.640
0.465
-0.179
Decision Tree
3.667
1.913
1.847
2.663
Tokenization
0.544
0.480
0.496
-0.750
Random Forest
3,143
2,701
2,214
1.798
Linear Regression
4.144
2.056
1,630
4.157
Layers
0.508
0.596
0,440
0.056
MLM
0.601
0.645
0.483
-0,239
WWM
0.604
0.642
0.382
+0.134
Table 4:
Comparison of different regression models using the features from the monolingual MLMs in the general domain. The highest Ri2 (on the top) and the lowest RMSE (on the bottom) are shown in bold.
NSP
0.594
0.765
0.732
40.612
SOP
0.604
0.567
0.083
-0.641
MRP
0.602
0.639
0.349
-0.146
Table 6: n1² scores after removing individual factors.
The most important factors on each predicted task (i.e, those factors that lead larger R² drop) are shown in bold, and the second-best ones are underlined. MLM. WWM, NSP, SOP and MRP represent masked language modeling, whole word masking, next sentence predic- tion, sentence ordering prediction and mention reference prediction, respectively.
Factors
A-CP
A-SS
TveetEval
GLUE
Parameter Size
0.297
0.134
0.244
0.200
Distill
0.007
0.001
0.005
0.002
Train-CCCewws
0.005
0.001
0.014
0.001
Train-OWT
0,004
0,002
0,015
0.000
Train-Stories
0.004
0.001
0.026
0.003
Cont-train
0.037
0.001
0.072
0.007
Uncased
0.071
0.080
0.007
0.035
Vocabulary Size
0,005
0.000
0.016
0.001
portant factors (i.e., the ones that obtain non-zero importance scores) at time and track the R² score returned by the gradient boosting models trained on different tasks. Table 6 shows the result. In this table, the lower R² score indicates the more impor- tant the factor is. Consistent with the result shown in Table 5, sentence order prediction and param- eter size are the most and second most important factors for TweetEval, respectively, and parame- ter size is the most important factor for A-CP, For A-SS, uncased and tokenization are the most and second most important factors, respectively. In ad- dition, show the corresponding important scores for each factor for training decision tree models in A.3 and observe largely similar conclusions to the result presented Table 5.
Tokenization
0.191
0.356
0.019
0.059
# Layers
0,056
0.047
0.093
0.520
MLM
0.136
0.115
0.024
0.008
WWM
0.162
0.169
0.015
0.013
NSP
0.001
0,041
0.125
0.047
SOP
0.002
0.052
0.256
0.100
MRP
0.023
0.001
0.071
0.005
Table 5: The important scores for each of the factors for training the gradient boosting regression model. The highest importance scores shown in bold, and the second-hight highest ones are underlined. cont-train, MLM, WWM, NSP, SOP and MRP represent continual training. masked language modeling, whole word masking, next sentence prediction, sentence ordering prediction and mention reference prediction, respectively.
Factor analysis To further study the effects of factors on MLMs, we eliminate each of the im-
Types of features To further explore the impact of a group of factors, we consider the 5 categories

---

in $3. To investigate the effect of certain group, we conduct ablations by removing a group of fac- tors as well as retaining only one group at time. Tables 7 and 8 show the corresponding results. The training objectives group of factors is regarded as paramount for social bias evaluation (i.c., A-CP and A-SS) in Table 7, and their removal leads to substantial decrease in the R² scores.
MLMs when considering a specific feature asso- ciated with a factor for each task. To capture the overall trend, we extend our analysis to include not only monolingual MLMs in the general domain but also domain-specific and multilingual Ms.s.
A-CP
A-SS
Tweetbval
GLUE
Parameter Size S (xr < 100M) (100M s 300M)
Meanwhile, Table 8 shows much lower R² scores across all the cases. This is because of the discrepancy between removing group of factors and retaining only one category, as the latter entails a reduced utilization of features during the train- ing process, Despite this limitation, we observe that by keeping tokenization only, the regression model can still make relatively accurate predictions for social bias evaluation, indicating that tokeniza- tion is important for social bias evaluation. In con- trast, training corpora and training objectives are the most important ones on the downstream tasks.
54.866
57.730
59.903
78.827
$7.388
64.472
(z > 300M)
64.040
81.449
Number Layers
layers
53,383
55.097
60.116
76.288
12 layers
57.546
60.398
64.472
81.225
24 layers
59.336
60.104
64.714
82.163
Vocabulary Size
S < 50K) M (50K s a s 100K)
58.196
60.674
62.101
82,413
58.422
60.322
65.676
81.472
(zz > 109K)
52.386
53,602
59.714
74.832
Tokenization
BPE WordPiece
58.422
60.322
65.676
81.472
57.838
60.086
62. 101
81.287
SentencePiece
54,800
56,382
60.470
77.210
Training Objectives
MLM
57,904
60.294
65.676
81.028
Categories
A-CP
A-SS
TweetFval
GLUE
NSP
56.142
611,593
MLM SOP
56,995
59.185
57.314
84,926
Without Removing
0.594
0.652
0.481
-0,201
MLM MRP
52,140
56.230
60.012
79.872
Model Size
0,587
0.670
0,304
-0,290
WWM + NSP
59.715
60.805
62. (2.225
80.420
Training Corpora
0.631
0.607
0.322
-0.217
Training Objectives
-1,010
-1.300
0.415
-0.222
Table 9: Comparison of social bias and task perfor-
Tokenization
0.563
0.100
0.428
-1.210
mance of the top 5 performing MLMs on different tasks, according different features associated with the im- portant factors. MLM, NSP, WWM, SOP and MRP represent masked language modeling, next sentence prediction, whole word masking, sentence ordering pre- diction and mention reference prediction, respectively.
Language
0.589
0.646
0.450
-0.173
Table 7: R² scores removing features from a single category. The most important categories on each task (i.e., those causing larger R² drop) are shown in bold.
Table 9 shows the average scores for the top-5 performing MLMs for each category of the im- portant factors. Note that the number of models associated with each category within a specific fac- tor is different. For example. there are 21 MLMs in the medium and 12 in the large categories, whereas there are only 6 in the small category for the pa- rameter size factor. Because some outlier cases are affecting the overall averages, in Table 9 we com- pute the top-5 performing models in each category, whereas the averages over all models are shown A.44.
Categories
A-CP
A-SS
TweetFval
GLUR
ALL
0.594
0.652
0.481
-0.201
Model Size
0.096
-0.420
0.191
-0.800
Training Corpora
-0.044
0.028
0.340
-0.225
Training Objectives
-0.180
0.319
-0.944
0.008
Tokenization
0.240
0.358
0.251
-0.024
Language
0.000
-0.004
-0.488
-0.005
Table 8: R² scores keeping features from one category only. The most important factors on each predicted task are shown in bold.
7 Qualitative Analysis
For the model size, we observe that the MLMs trained with a parameter size greater than 100M tend to have better downstream task performance, while reporting higher levels of social biases com- pared the ones with smaller parameter size. The models trained with 6 layers demonstrate the least amount of social biases (i.c., A-CP and A-SS close to 50), however, the ones trained with 24 lay-
With the knowledge that model size, tokenization, and training objectives are the three primary cat- egories of factors influencing social bias and task performance in MLMs from $6.2, we further inves- tigate the factors within each category to discern their individual contributions in MLMs. For this purpose, we calculate the average performance of

---

ers obtain better downstream task performance. As for the vocabulary size, the models trained with small- and medium-sized vocabulary obtain the best performance on GLUE and TweetEval, respec- tively, whereas the ones with a large vocabulary size contain less degree of social biases. Regarding the tokenization methods, we see that the mod- els using SentencePiece contain the lowest level of social biases, while the models using BPE ob- tain better downstream task performance. With respect to training objectives, the models trained with both masked language modelling and men- tion reference prediction contain the fewest de- grees of social biases, On the other hand, models trained with masked language modelling only and models with both masked language modelling and sentence ordering prediction return the best down- stream task performance on TweetEvall and GLUE, respectively.
Models
A-CPP
A-58
TweetEval
GLUE
Monolingual Multilingual
54.157
56,062
60.953
79.648
50.342
52.092
71.723
General Domain
54.157
56,062
60.953
79.648
Domain-speciffe
54,042
56,962
61.00 062
75.547
Social Media Domain
53.325
57.438
64,339
75.713
The performance of monolingual vs. multi- lingual models and general domain vs. domain-specific models on social bias and downstream tasks.
Table 10:
of languages, thus incorporating a greater spectrum of cultural diversity (Liang et al., 2020; Ahn and Oh, 2021). Consequently, the presence of these di- vergent viewpoints within the model can potentially mitigate social biases. Conversely, monolingual models obtain better performance on TweetEval and GLUE, which are limited to English.
General vs. Domain-specific. Table 10 bottom shows the performance of models from different domains. Recall from $ that we include domain- specific models for social media, legal and biomedi- cal domains. As TweetEval is social media bench- mark, we additionally include the performance of models in the social media domain. Models in the social media domain contain the least bias accord- ing to A-CP and achieve the best performance on TweetEval. Conversely, the performance of models in the general domain is better than domain-specific ones GLUE and obtains the lowest bias score for A-SS. This result is surprising given the focus on improving tasks of the given domain for domain-specifie models, rather than improve in general tasks.
8 Discussion
Model size. Larger MLMs have a higher capabil- ity to achieve better performance downstream tasks, while smaller models computationally efficient and require fewer hardware resources. From Table 9, we see that models with a larger number of parameters and more layers report better performance in GLUE and TweetEval compared to the smaller and shallower models, while at the same time demonstrating lesser social biases in terms of A-CP and A-SS scores. Because the learn- ing capacity of MLMs increases with the number of parameters and the depth (i.e. number of layers), it is not surprising that larger models outperform the smaller ones on downstream tasks. However, it is interesting to observe that social biases do not necessarily increase with this extra capacity of the MLMs. In the case of gender-related biases, Tal et al. (2022) showed that even if the gender bias scores measured on Winogender (Rudinger et al., 2018) smaller for the larger MIMs, they make more stereotypical errors with respect to gender. However, whether this observation generalises to all types of social biases remains an open question.
Conclusion
Despite the extensive prior work evaluating social bias in MLMs, the relationship between social bi- and the factors associated with MLMs remains unclear. To address this gap, we conducted the first-ever comprehensive study of this type through predictive factor analysis to investigate the impact of different factors on social bias and task perfor- mance in pretrained MLMs, considering 39 models with 30 factors. We found that model size, tok- enization and training objectives are the three most important factors across tasks. In terms of social biases, domain-specific models do not appear to be more or less biased than general domains. while multilingual models, which are trained on corpora covering different languages and cultures, appear to be less socially biased than the monolingual ones.
Mono- vs. Multi-lingual. Table 10 top compares the performance of monolingual multilingual MLMs. We see that multilingual models demon- strate lower levels of social biases compared to their monolingual counterparts. This aligns with prior works that conjecture multilingual language model to benefit from training over a larger number

---

Acknowledgements
work and used multiple corpora and benchmark datasets that have been collected, annotated and repeatedly used for evaluations in prior works. To the best of our knowledge, no ethical issues have been reported concerning these datasets.
Jose Camacho-Collados and Yi Zhou are supported by UKRI future leaders fellowship. Danushka Bollegala holds concurrent appointments as Pro- fessor at University of Liverpool and an Amazon Scholar. This paper describes work performed at the University of Liverpool and is not associated with Amazon.
The gender biases considered in the bias evalu- ation datasets in this paper cover only binary gen- der (Dev al., 2021). However, non-binary gen- ders are severely underrepresented in textual data used to train MLMs. Morcover, non-binary gen- ders are often associated with derogatory adjectives. Evaluating social bias by considering non-binary gender is important.
Limitations
This paper studies the impact of underlying factors of MLMs on social bias and downstream task per- formance. In this section, we highlight some of the important limitations this work. We hope this will be useful when extending our work in the future by addressing these limitations.
Furthermore, biases are not limited word rep- resentations but also appear sense representa- tions (Zhou et al., 2022). However, our analysis did not include any sense embedding models.
As deseribed in $ 6, the regression models take into account in this paper are not able to be properly trained based on the features that we con- sidered. Extending the factors and further explor- ing the reason for the poor performance regres- sion models on GLUE is one future direction.
References
Jaimeen Ahn and Alice Oh. 2021. Mitigating language- dependent ethnic bias in BERT. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 533-549, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
We limit our work in this paper to focusing on evaluating intrinsic social bias captured by MLMs. However, there are numerous extrinsic bias evalua- tion datasets existing such as BiasBios ADeAAat et al.. 2019), STS-bias (Webster et al., 2020), NLI- bias (Dev et al., 2020). Extending our work to evaluate the extrinsic biases in MLMs will be a natural line of future work.
Emily Alsentzer. John Murphy, William Boag, Wei- Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clin- ical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop. pages 72-78, Minncapolis, Minnesota, USA. Associ- ation Computational Linguistics.
Furthermore, analysis focuses MLMs and not considering generative language models such GPT-2 (Radford al.), Transformer-Xi (Dai et al., 2019), XLNet (Yang et al., 2019) and GPT- 3 (Brown et al., 2020). Extending our work to in- vestigate the relationship between models' factors and social bias as well as downstream performance is deferred to future work.
Francesco Barbieri, Luis Espinosa Anke, and Jose Sumacho-Collados 2022. Xlm-t: Multilingual lan- guage models in twitter for sentiment analysis and beyond. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 258- 266.
Francesco Barbieri, Jose Camacho-Collados, Luis Es- pinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644-1650, Online. Association for Computational Linguisties.
Finally, although we tried to collect many MLMs as possible, the final number may be in some cases insufficient draw conclusive numeri- cal conclusions the regression analysis.
Ethical Considerations
Francesco Barbieri, Jose Mamacho-Colladee Francesco Ronzano, Luis Espinosa-Anin Anke, Miguel Ballesteros, Valerio Basile, Viviana Patti, and Horacio Saggion. 2018. SemEval 2018 task 2: Multilingual emoji prediction. In Proceedings of the 12th International Workshop Semantic Evaluation, pages 24-33, New Orleans. Louisiana, Association for Computational Linguistics.
In this paper, aim to investigate which factors affect the social bias captured by MLMs. Although we used existing datasets that are annotated for social biases, we did not annotate nor release new datasets as part of this research. In particular, we did not annotate any datasets by ourselves in this

---

Valerio Basile, Cristina Bosco, Elisabetta Fersini. Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 54-63, Min- neapolis, Minnesota, USA. Association for Compu- tational Linguistics.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. In Proceedings of the First international con- ference on Machine Learning Challenges: evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, pages 177-190.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Car- bonell, Quoe Le, and Ruslan Salakhutdinov. 2019. Transformer- XL: Attentive language models beyond fixed exed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1978-2988, Florence, Italy, Asso- ciation for Computational Linguistics.
Luisa Bentivogli, Ido Kalman Dagan, Dang Hoa, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth pascal recognizing textual entailment challenge. In TAC 2009 Workshop. no publisher.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Maria De-Arteaga, Alexey Romanov. Hanna Wal- lach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik. Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In proceedings of the Conference on Fairness, Accountability, and Transparency, pages 120- 128.
Aylin Caliskan, Joanna J Bryson, and Arvind Narayanam. 2017. Semanties derived automatically from lan- guage corpora contain human-like biases. Science, 356(6334): 183-186.
Ángel Delgado-Panadees Beatriz Hernández-Lore María Teresa Gareía- Ordás, and José Alberto Benítez- Andrades. 2022. Implementing local- explanxxaain in gradient boosting trees: Feature contribution. In- formation Sciences, 589: 199-212.
Tommaso Caselli, Valerio Basile, Jelena Mitrović, and Michael Granitzer. 2021. HateBERT: Retraining BERT for abusive language detection in English. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), pages 17-25, Online. As- sociation for Computational Linguistics.
Sunipa Dev, Tao Li, Jeff M Phillips, and Vivek Sriku- mar. 2020, On measuring and mitigating biased in- ferences of word embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 7659-7666.
Daniel Cer, Mona Diab, Eneko E Agirre, lñigo Lopez- Gazpio, and Lucia Specia. 2017. Semeval-2017 task I: Semantic textual similarity multilingual and cross- lingual focused evaluation. In The 11th International Workshop on Semantic Evaluation (SemExal-2017), pages 1-14.
Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Ar- jun Subramonian, Jeff Phillips, and Kai-Wei Chang. 2021. Harms of gender exclusivity and challenges in non non-binary representation in language technologies. In Proceedings of the 2021 Conference Empiri- cal Methods in Natural Language Processing, pages
Rochelle Choenni, Ekaterina Shutova, and Robert van Rooij. 2021. Stepmothers are mean and academics are pretentious: What do pretrained language models learn about you? In Proceedings of the 202/ Con- ference on Empirical Methods in Natural Language Processing, pages 1477-1491.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4188 Minneapolis. Minnesota. Association for Computational Linguistic..
Alexis Conncau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451.
Mark Díaz, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Darren Gergle. 2018. Addressing age- related bias in sentiment analysis. In Proceedings of the 2018 chi conference on human factors in comput- ing systems, pages 1-14.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. Pre-training with whole word masking for chinese bert. IEEE/ACM Transac- tions on Audio, Speech, and Language Processing, 29:3504-3514.
William B Dolan and Chris Brockett. 2005. Automati- cally constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).

---

Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2019. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21.
Masahiro Kaneko and Danushka Bollegala. 2019. Sender--reserving debiasing for pre-trained word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guisties, pages 1641-1650, Florence, Italy. Associa- tion for Computational Linguistics.
Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. 2019. Understanding undesirable word embedding associations. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 1696-1705.
Masahiro Kaneko and Danushka Bollegala. 2021. De- biasing pre-trained contextualised embeddings. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guisties: Main Volume, pages 1256 Online. Association for Computational Linguisties.
Yoav Freund and Robert F. Schapire. 1997. A decision- theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1): 139.
Saibo Geng, Rémi Lebret, and Karl Aberer. 2021, Le- gal transformer models may not always help. arXiv preprint (wpXi::21090000
Masahiro Kaneko and Danushka Bollegala. 2022. Un- masking the mask-evaluating social biases in masked language models. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 36, pages 11954-19922
Danilo Giampiccolo. Bernardo Magnini, Ido Dagan, and William B Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1-9.
Masahiro Kaneko, Danushka Bollegala, and Naoaki Okazaki. 2022. Debiasing isn't enough!
the effectiveness of debiasing MLMs and their social biases in downstream tasks. In Proceedings of the 29th International Conference on Computational Lin- guistics, pages 1299-1310, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Naman Goyal, Jingfei Du. Myle Ott, Giri Ananthara- man, and Alexis Conneau. 2021. Larger-scale trans- formers for multilingual masked language modeling. In Proceedings of the 6th Workshop on Represen- lation Learning for NLP (RepLANIP--002),, pages 29-33.
Svetlana Kiritchenko and Saif Mohammad. 2018. Ex- amining gender and race bias in two hundred senti- ment analysis systems. In Proceedings of the Seventh Joint Conference on Lexical and Computational Se- mantics, pages 43-53.
Yue Guo, Yi Yang, and Ahmed Abbasi. 2022. Auto- debias: Debiasing masked language models with automated biased prompts. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 1012-1023, Dublin, Ireland. Association for Compu- tational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learn- ing of language representations. arXiv preprint arXiv:19009 1142..
Suchin Gururangan, Ana Marasovie, Swabha Swayamdipta, Kyle Lo. Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Anne Lauscher, Tobias Lueken, and Goran Glavaš. 2021. Sustainable modular debiasing of language models. In Findings of the Association for Computational Linguistics: EMNLP 2021. pages 4782-4797, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiceolo, Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising textual entail- ment challenge. In Proceedings of the Second PAS- CAL Challenges Workshop on Recognising Textual Entailment, volume 7.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1200
Geoffrey Hinton.
Oriol Vinyals, and Jeff Dean, 2015, Distilling the knowledge in a neural network. arXiv preprint arxiv: 02533.....
Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thir- teenth international conference on the principles of knowledge representation and reasoning.
Shankar Iyer, Nikhil Dandekar, and Kornl Csernai. 2017. First quora dataset release: Question pairs.
Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Na- man Goyal, Marjan Ghazvininejad, Luke Zettle- moyer, and Madian Khabsa. 2023. Xlm-v: Overcom- ing the vocabulary bottleneck in multilingual masked language models. arXiv preprint arXiv:2301. 10472.
Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, and Sivanesan Sangeetha. 2021. Ammus: A survey of mansformer--ased pretrained models natural lan- guage processing. arXiv preprint N0X.:2210.005555

---

Sheng Liang, Philipp Dufter, and Hinrich Schütze. 2020. Monolingual and multilingual reduction of gender bias in contextualized representations. In Proceed- ings of the 28th International Conference on Com- pulational Linguistics, pages 5082-5093, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020b. CrowS CrowS-pairs: A chal- lenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLPP), pages 1953-1967, Online. As- sociation for Computational Linguisties.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Dangi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta:
Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. weRTweet: A pre-trained language model for English tweets. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9-14, On- line. Association for Computational Linguistics.
robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:1907.1 7et7. 16692.
Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 622-628, Minneapolis, Min- nesota. Association for Computational Linguistics.
Kyubyong Park, Joohong Lee, Seongbo Jang, and Da- woon Jung. 2020. An empirical study of tokenization strategies for various korean nlp tasks, In Proceed- ings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 133-142.
Nicholas Meade, Elinor Poole-Dayan, and Siva Reddy 2022. An empirical survey of the effectiveness of debiasing techniques for pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers). pages 1878 1198..
Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Trans- fer learning in biomedical natural language process- ing: An evaluation of BERT and ELMo on ten bench- marking datasets. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 38-65, Florence. Italy. Association for Computational Linguisties.
Laura Pérez-Mayos, Miguel Ballesteros, and Leo Wan- ner. 2021. How much pretraining data language models need to learn syntax? In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 1571-1582.
Saif Mohammad, Felipe Bravo- Marquez Mohammad Salameh, and Svetlana Kiritchenko. 2018. SemEval- 2018 task I: Affect in tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 1-17, New Orleans, Louisiana. Association for Computational Linguistics.
Alee Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodci, Ilya Sutskever, et al. Language mod- els are unsupervised multitask learners.
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sob- hani, Xiaodan Zhu, and Colin Cherry. 2016. SemEval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-201e pages 31- 41, San Diego California. Association for Computa- tional Linguisties.
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392
Moin Nadeem, Anna Bethke, and Siva Reddy, 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pages 5356 5371, Online. Association for Computational Linguistics.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8:842-866 866.
Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2011 task 4: Sentiment analysis in Twitter. In Proceedings of the 1 11th International Workshop on Semantic Evaluation (SemEval-20177 pages 502- 518, Vancouver, Canada. Association for Computa- tional Linguistics.
Nico JD Nagelkerke et al. 1991. A note on gen- eral definition of the coefficient of determination. biometrika, 78(3)691-692
Rachel Rudinger, Jason Naradowsky. Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8-14. Association for Computational Linguis- ties.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020a. CrowS-pairs. A chal- lenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Con- ference Empirical Methods in Natural Language Processing (EMNLP), pages 1953-1967, Online. As- sociation for Computational Linguistics.

---

Phillip Rust, Jonas Pfeiffer, Ivan Vulié, Sebastian Ruder. and Iryna Gurevych. 2021. How good is your tok- enizer? the monolingual performance of multilin- gual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 I1th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 3118-3135.
Glue: multi-task benchmark and analysis platform for natural language understanding. EMNLP 2018. page 353.
Alex Warstadt, Amanpreet Singh, and Samuel Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-664 1-1--
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1rxiv:1910.0110
Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, Ed Chi, and Slav Petrov. 2020. Measuring and reducing gendered correlations in pre-trained models. arXiv preprint arXiv:2010000032.
Timo Schick. Sahana Udupa, and Hinrich Schütze. 2021. Self-diagnosis and self-debiasing: A proposal for re- ducing corpus-based bias in nlp. Transactions of the Association for Computational Linguisties, 9:1408-
Colin White, Arber Zela, Binxin Ru, Yang Liu, and Frank Hutter. 2021. How powerful are performance predictors in neural architecture search? In Advances in Neural Information P Processing Systems.
1424.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 17255
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad- coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume (Long Papers), pages 112-1122.
Andrew Silva, Pradyumna Tambwekar, and Matthew Gombolay. 2021. Towards a comprehensive under- standing and accurate evaluation of societal biases in pre-trained transformers. In Proceedings of the 202] Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2383-2389, Online. Association for Computational Linguistics.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine trans- lation, arXiv preprint orXiv:/669/ 1009000144...
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D Manning. Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empiri- cal methods in natural language processing, pages 1661-.
Zhengxuan Wu, Atticus Geiger, Joshua Rozner, Elisa Kreiss, Hanson Lu, Thomas Icard, Christopher Potts, and Noah Goodman. 2022. Causal distillation for language models. In Proceedings of the 2022 Con- ference of the North American Chupter of the Asso- ciation for Computational Linguisties: Human Lan- guage Technologies, pages 4288 4295.
Yarden Tal, Inbal Magar, and Roy Schwartz. 2022. Fewer errors, but more stereotypes? the effect of model size on gender bias. In Proceedings of the 4th Workshop Gender Bias in Natural Language Pro- cessing (GeBNLP), pages 112-120, Seattle, Wash- ington. Association for Computational Linguistics.
Patrick Xia, Shijie Wu, and Benjamin Van Durme. 2020. Which" bert? a survey organizing contextualized encoders. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7516-7533.
Cagri Toraman, Eyup Halit Yilmaz, Furkan Sahinuç. and Oguzhan Ozcelik. 2023. Impact of tokenization on language models: An analysis for turkish. ACM Transactions on Asian and Rew-Resorree Language Information Processing, 12
Zhilin Yang. Zihang Dai, Yiming Yang. Jaime Car- bonell, Russ Salakhutdinov, and Quoe V Le. 2019. Xinet: Generalized autoregressive pretraining for lan- guage understanding. Advances in neural informa- tion processing systems, 32.
Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. 20mEvvl-2018 task 3: Irony detection in En- glish tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 39-50, New Orleans, Louisiana. Association for Computational Linguistics.
Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020. Corefer- ential reasoning learning for language representation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP). pages 71 7170-7186.
Alex Wang. Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar.

---

2019. SemEval 2019 task 6: Identifying and cat- egorizing offensive language in social media (Of- fensEval). In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 75-86, Min- neapolis, Minnesota, USA. Association for Compu- tational Linguistics.
Factors
A-CP
A-SS
TweetFval
GLUR
Parameter Size
0.297
0.134
0.244
0.200
Distill
0.007
0.001
0.005
0.002
Train-Booke
0.040
0.003
0.006
0.003
train- Wiki
0.000
0.000
0.000
0.000
Crain
0.005
0.001
0.014
0.001
Train-OWT
0.004
0.002
0.015
0.000
Jieyu Zhao, Subhabrata Mukherjee, Saghar Hosseini, Kai-Wei Chang, and Ahmed Hassan Awadallah. 2020. Gender bias in multilingual embeddings and cross- lingual transfer. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 2896-2907.
Train-Stories
0.004
0.001
0.026
0.003
truin- Tweets
0.000
0.000
0.000
0.000
train-RALE
0.000
0.000
0.000
0.000
Priin-Paeent
0.000
0.000
0.000
0.000
Cascl-C
0.000
0.000
0.000
0.000
truin- PogglePaagsss
0.000
0.000
0.000
0.000
Zhou, Masahiro Kaneko, and Danushka Bollegala. 2022. Sense embeddings are also biased-ealluating social biases in static and contextualised sense beddings. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pages 1924 1935.
train-S2ORC
0.000
0.000
0.000
0.000
train-MIMIC3
0.000
0.000
0.000
0.000
Train-PMed
0.000
0.000
0.000
0.000
train-PMe (MC
0.000
0.000
0.000
0.000
Train-CC100
0.000
0.000
0.000
0.000
Cont-train
0.037
0.001
0.072
0.007
Uncased
0.071
0.080
0.007
0.035
A Performance on All MLMs
Domain
0.000
0.000
0.000
0.000
Language
0.000
0.000
0.000
0.000
A.1 Performance of Regression Models over All MLMS
Languages
0.000
0.000
0.000
0.000
Vocabulary Size
0.005
0,000
0.016
0.001
Tokenization
0.191
0.356
0.019
0.059
The comparison of different regression models us- ing the features from all 39 MLMs is shown in Table 11. The R² scores obtained by all the re- gression models are not as high as the ones when only taking into account the monolingual MLMs in the general domain. On the other hand, when considering all the MLMs, the decision tree model obtains higher R² scores than gradient boosting.
Layers
0.056
0.047
0.093
0.520
MLM
0.136
0.115
0.024
0.008
WWM
0.162
0.169
0,015
0.013
NSP
0.001
0.041
0.125
0,047
SOP
0.002
0.052
0.256
0.100
MRP
0.023
0.001
0.071
0.005
Table 12:
The important scores for each of the factors for training the gradient boosting regression model over all the ML.s. The highest importance scores are shown in hold, and the second highest ones are underlined. Due the page limitation, we do not show the factors that ob- tain O important scores for all the tasks. MLM, WWM, NSP, SOP, and MRP represent masked language mod- elling, whole word masking, next sentence prediction, sentence ordering prediction, and mention reference pre- diction, respectively.
Models
A-CP
A-SS
TweetEval
GLUE
Gradient Bousting
0.298
0,231
0.299
-0.991
SVM
-0.741
0,066
0.185
0.022
Gaussian Process
-0,548
-0.039
0.574
-0.032
Decision Tree
0.476
0.436
0.340
-0.999
Random Forest
-0.328
0.314
0.401
0.027
Lincar Regression
-1.793
-1.761
-0.247
-5.184
Gradient Bonsting
2.963
3.071
2.726
5.862
SVM
4.667
3.385
2.939
4,333
RMSE
Gaussian Process
4.401
3.570
2.125
4,449
Decision Tree
2,560
2.631
2.645
6.194
Random Forest
4.076
2.901
2.520
4.321
Linear Regression
5.911
5.820
3.635
10.893
A.3 Important Scores of Factors for Training Decision Tree Models
Table II:
Comparison of different regression models using the features from all the MLMs. The highest R²
(on the top) and the lowest RMSE (on the bottom) are shown in bold.
Table 13 shows the full table of important scores for each of the factors used for training decision tree models over all 39 MLMS. We derive similar conclusions regarding the most predictive factors. Specifically, parameter size is the most predictive factor on A-CP and the second most predictive factor on TweetEval. Tokenization and number of layers are the most predictive factors on A-SS and GLUE, respectively.
A.2 Important Scores of Factors for Training Gradient Boosting Models
Table 12 shows the full table of important scores for each of the factors used for training gradient boosting regression models over all 39 MLMs with- out removing the factors that obtain the important score of O.

---

Factors
A-CP
A-SS
TweetFval
GLUR
A-CP
A-SS
Tweetbval
GLUE
Parameter Size
0.308
0.110
0.230
0.137
mance of MLMs on different tasks, according to dif- ferent features associated with the important factors. Parameter Size
Distill
0.000
0.000
0.000
0.004
S (ar < 100M)
54,365
56.718
59.903
78.827
M (100M s 300M)
53.329
55.558
60.547
76.760
Train-Books
0.000
0.000
0.000
0.000
L (e > 300M))
53.598
55.437
61,895
78.377
train- Wiki
0.000
0.000
0.000
0.000
0.000
0.000
0.172
0.000
Number of Layers
Train-OWT
0,054
0.000
0,000
0.000
6 layers
53,383
55,097
60.116
76.288
12 layers
53,368
55,560
60.280
76.019
Train-Stories
0.000
0.000
0.000
0.000
layers
53.975
59.994
75.564
train- Tweets
0.000
0.000
0.000
0.000
Vocabulary Size
train-RALE
0,000
0.000
0.000
0.000
< 50K)
53.921
55,4 55,488
59.55
train-Pateens Patent
train-
0,000
0.000
0,000
0.000
(50K a 1008)
54.388
57,801
63.36
78.18
truin-Casee Cain-Casllaw
0,000
0.000
0.000
0.000
L 100K)
51.107
52.370
59.71
74.83
Paoglet
0.000
0.000
0.000
0.000
Tokenization
train-S2ORC
0.000
0.000
0.000
0.000
BPE:
53,457
56.419
62.247
77.674
truin- train-MIMIC3
0,000
0.000
0,000
0.000
WordPiece
53.559
55.053
59.685
77.392
train-PMed
0.000
0.000
0.000
0.000
SentencePiece
53.912
55.612
60.470
77.210
train-PMC
0.000
0.000
0.000
0.000
Training Objectives
train- train-CC CC00
0.000
0.000
0.000
0.000
MLM
53.714
56.673
63.174
77.115
Cont-train
0.000
0.000
0.000
0.000
MLM NSP
52.599
53.579
59.171
76.498
Uncased
0.096
0.085
0.013
0.045
MLM SOP
56.995
59.185
57.314
84.926
Domain
0.000
0.000
0.000
0.000
MLM + MRP
52.140
56,230
60.012
79.872
Language
0.000
0.000
0.000
0.000
WWM
59.715
62.225
80.420
# Languages
0.000
0.000
0.000
0.000
Vocabulary Size
0,000
0.000
0,000
0.000
Table 14: Comparison of social bias and task perfor-
Tokenization
0.211
0.390
0.000
0.004
# Layers
0.004
0.083
0.160
0.518
MLM
0,000
0.000
0.000
0.000
MLM, NSP, WWM, SOP and MRP represent masked language modeling, whole word masking, next sentence prediction, sentence ordering prediction and mention reference prediction, respectively
WWM
0.289
0.278
0,015
0,072
NNSP
0,000
0.000
0.000
0,000
SOP
0.000
0.000
0.308
0.172
MRP
0.038
0.054
0,103
0.047
Table 13: The important scores for each of the factors for triningg the decision tree model over all the MLMs. The highest importance scores are shown in bold, and the second highest ones are underlined. Due to the page limitation, we do not show the factors that obtain important scores for all the tasks. MLM, WWM, NSP, SOP, and MRP represent masked language modelling, whole word masking, next sentence prediction, sentence ordering prediction, and mention reference prediction, respectively.
than 100K) obtain the best performance on both TweetEval and GLUE, while the ones with a large vocabulary size (i.e., greater than 100K) contain lesser degrees of social biases. Regarding the to- kenization methods, BPE contain the lowest level of social bias (i.c., A-CP close to 50) and the best performance on both TweetEval and GLUE, while WordPiece is better for A-SS. Models using SentencePiece to- kenization perform consistently worse across the board, but we should also note the limited number of language models analysed with this tokenization method.
see that the models using
A.4 Comparison of All MLMs in Different Categories
Table 14 shows the comparison of social bias and task performance of MLMS on different tasks, ac- cording to different features associated with the important factors. For model size, we observe that the MLMs trained with a parameter size greater than 100M tend to have less social bias compared with the ones trained with smaller parameter size while reporting competitive performance in both TweetEval and GLUE. The models trained with 6 and 12 layers contain less social bias with gener- ally high downstream task performance. As for vocabulary size, the models trained with a medium- size vocabulary (i.e., greater than 50K and less
B Error Analysis on GLUE
From Table 15, we observe that CoLA, MRPC and RTE obtain negative R² scores. This indi- cates gradient boosting is not able to be properly trained to predict the performance of models on CoLA, MRPC and RTE given the considered fea- tures. However, the GLUE score is computed by taking the average of the performance over the 9 subtasks. The non-predictive subtasks especially on RTE task result in a negative H² value when training gradient boosting to predict the perfor-

---

CoLA
SST-2
MRPC
STS-B
QQP
MNLI
QNLI
RTE
WNLI
-0.070
0.294
-1.339
0.122
0.125
0.648
0.032
-11.793
0.182
Table 15: R² scores returned by gradient boosting on each GLUE sutasks.
mance of the MLM GLUE.
C Details of Factors
Table 16 shows the description of each factor in- cluding the feature types as well as the potential value for each factor. As shown in the table, most of the factors are binary, while factors such param- eter size, domain, language, number of languages, vocabulary size, tokenization and number of layers are categorical. Table 17 provides the details of factors corresponding to each language model.

---

Factors
Description
Parameter Size
categorical, the parameter size is divided into three categories: S (ar 100.V),
M (100M & S 300M), L (x 300M)
Distill
binary, if model uses model distillation, otherwise 0
Train-Booke
binary, if BookCorpus used as training corpus, otherwise
Train- Wiki
binary. if Wikipedia is used as training corpus, otherwise 0
Ceenss
binary, if CC-News is used as training corpus, otherwise 0
Train-OWT
binary, if OpenWebText is used as training corpus, otherwise ()
Train-Stories
binary, if Stories is used as a training corpus, otherwise 0
Train-Tweets
binary, if Tweets is used as a training corpus, otherwise O
Train-RALE
binary, if RALE used training corpus, otherwise
Train-Patent
binary, if Patent Litigations is used training corpus, otherwise 0
Train-Caselaw
binary, if Caselaw Access Project is used a training corpus, otherwise O
Train-Googlepe GooglepPatents
binary, if Google Patents Public Data is used as training corpus, otherwise O
TroRn
binary, if S2ORC is used as a training corpus, otherwise 0
TIMI3
binary, if MIMIC3 used as training corpus, otherwise
Train-PMed
binary, if PubMed Abstracts used as training corpus, otherwise 0
Train-PMC
binary, if PMC Full-text articles are used training corpus, otherwise 0 binary, if CommonCrawl data in 100 language (CC100) is used as training corpus, otherwise
Train-CC100
Cont-train
binary, if model continued trained based on a pretrained MLMs, other- wise 0
Uncased
binary, if the model is uncased, otherwise 0
Domain
categorical, domain of the models is based on the domain of training corpus that the model used during pretraining. The 4 domains are considered: general domain, social media, legal domain, biomedical domain
Language
categorical, English only or multilingual
Number of Languages
categorical, take the real number of the languages that trained on categorical, the vocabulary size is divided into three categories: S (xr 50K),
Vocabulary size
M (50K = r = 100K), L (x > 100K)
Tokenization
categorical, three categories: BPE, WordPiece, SentencePiece
Number of Layers
categorical, three categories: 6 layers, 12 layers, 24 layers binary, if the model uses the masking technique, otherwise 0
Masked Language Modelling
Whole Word Masking
binary, if the model uses the whole word masking technique, otherwise 0
Next Sentence Prediction
binary, if the model uses next sentence prediction objective, otherwise 0
Sentence Order Prediction
binary, if the model uses sentence order prediction objective, otherwise 0 binary, if model uses mention reference prediction objective, otherwise
Mention Reference Prediction
Table 16: The description of the features each factor that consider in this paper.

---

Models
Pura
Desil
Traiting Coppri
Cost
Usc
Donai
Lang
n
Long
We
Tokeniostion
Layers
Objectiivss
niborte-loee
125M
0
Bovkk bepos, Fingish Witrpeda
(17Nem Apenhchiieei Stonee
General
Iagisn
WP:
12
MI
nobom large
155
0
BookClorjss Einglish Witipedia
CCNeva OyeaWeMMestt Stotics
O
9
Genera
Eaglish
50K
BPC
24
MLM
1003
Englida Wikipoii
0
Gasenal
Eaglids
50k
WondPicee
12
MLM, NSP
lage saied
340
English Wilipedis
Gescral
Eaglish
30X
WondPiese
MLM. NSP
bast-laac unssed
110M
Higgish wilepadis
Gcazul
Eaglisi
30K
WondFisss
MLM, NSP
ben large uncesed
Trgliih Wilopedis
Eaglish
WordPlece
lett-arge-car cacc)
336M
BookCoppus Englihh Wiliped
Geacra
English
30X
WondPlese
wbileranet-mmsting
battlarg--atcnes)
1MM
HoookCorprs, Peglish Wietpedka
Gearr
Iaglich
10K
Wordrirze
WWM. NSP
alber v2
17
Exslidh Wikepolia
BootClopus.
0
Genera
Eiglish
30X
WondPlece
MLM. NSP
ahons-arg--1t
17
0
Enelidh Wikepaii
Rovee serper.
0
0
Genena
Eaglesh
S0K
WordPieee
24
MLM, NSP
Betibber--ee hoeebeer-eeeeweddd
6MM
0
English Wikkpolia
linost berpen.
0
0
Gearrl
Eaglish
10K
WerdiWeze
MI MIM, NSP
65M
Englidi Wikgolia
HoncCleyys,
0
1
Gascra
Englids
30K
WandPiczz
MLM, NSP
distibobera base
82M
Wovilt Figlish Wikipealo
CCNNews OpenWebllee. Stories
Genera
English
50K
BPE
Musked
sllee-sslage-22
123M
Euglish Wilipelia
Exglisi
30X
ScatessePiove
MLM.
sthen-targe-x2
Ilgglihh Wilepadis
Eaglish
nchetts-lnge
BookCieyss, English Willpedli,
CCNoc Oyen/WcbTTaa, Stoties
English
50X
BPC
128M
BodkCleppa, Enplali Williealis
C1.Nemc Upbksockeeenhbblinnn Stones
(maml
Iaglich
50K
MI
nelectooet ber base
110
Eiglish Wikpolia
Boottboper
Gearal
Eaglich
10K
WondPieze
MI MRP
melspcore Sont larg
1403
0
Enslish Wikpolia
0
Goncral
Eaglish
30k
WordiPece
24
MLM. MRP
centifi phiiiinphittees
Hovil beyes, Fingish Witppedo
iiberta
125M
CCNews Open
0
Social Media
English
50K
BPE
12
MLM
hace
Sarric, Englisa beezls
BookCheess, Englih Willyedlas
125
Sozial Modia
Engliss
SOK
12
MLLM
Swetee, SBM rerets
3455 Englih from 01/2012 08/2019 5M Tweets ralasad
100
Sociá Molia
Eaglish
64X
BPE
845M English Tweets sreamed fiom
340M
5M Tweet relael
08/2119
Social Melia
Englisi
64X
BPE
Bh CDVill11 parolanic
GroNt Jobaeetttee
100
Hoviet bepas, Figgish Witipodo
RALE Redlin
Sociel Media
Paglich
14
WondfWizee
M, NSP
BooRCoejis, English Witipedia
oobentallagg-
353M
CCNewa WpasWib
0
Social Melia
Exgiish
50k
BPE
MLM
2012-1 54m
Samnos Heglcc tarzis
BookCrejus, Eiglish Witipedia
ndoctta beac- 3022 154m
1253
CCNewa OpenWabtees.
1
0
Social Modia
Englida
S0K
BPE
32
MLM
Suries Exglica tweets
BonkClejas, Englah Wrlppealas,
cchollegst nibeita
128
Putent Liigatons, Cusclox
(375mm OpewWerleer Stores.
Lzzal
Eaglich
50K
Phojec
Wilipedia
Eaglish
papers
Clii:IBERT
110M
BookCiejus, Englih Willyedia
MIMIC
Biunedical
English
30X
WondPiese
MLM.
BookChepps, Englath Wiliyeala,
hoso-cousk +1.2
10M
Pabittd Aberracs
Hommedical
Paglich
10K
WondMiize
M, NSP
PMC Fill test ortcles
Nonpprieber guhnee,
BooRt oejes, English Wilipedia
mink unssed L.
110M
0
MIMICS and
I
Biuaslical
Eagishh
31K
WundPisse
12
MLM, NSP
12 12,8--688_ A-12
Puckied abrdnads
Bookk Ioree, Finglish Witipolo
mimiccmccerd
140M
0
MIMIUSS and PUAMed ubstact
Nommaiccal
Eaglish
1IK
Werdifizze
34
MI MIM, NSP
24H HDLA 16
1773
0
Wilipolia
0
Gescul
Multiiagul
104
1238
BPE
12
MLM, NSP
mahiinggaleenst
mabrinngsll encosed
base
177M
Wilipedio
Geaent
Maininggol
104
120K
BPC
MLM. NSP
Wrillien baz
1543
Wikipesiin
Maltiliged
104
120K
Mattlingo
sineclstts-lagee
550M
Crand (CC)0)
Gencal
Multilaguu
100
2508
ScatessxPPoss
1913
2.33 ef lilkned Cimmon
Crpail dus (DC100)
Genenl
Mulillngaa
100
IM
SeatenezPioss
castifin pratiion
sirs wiberta bose
370M
1983 mnltiingaa tweets
Social Media
Malrilaguu
100
1.734M
ScateneePlsee
MLM
Table 17: The details of factors corresponding to each model. Cont and UnC represent continual training and uncased, respectively. MLM, NSP, WWM, SOP, and MRP represent masked language modelling, next sentence prediction, whole word masking, sentence ordering prediction, and mention reference prediction, respectively.


Reviews:
[{"review_id": "Io5foeAZps", "reviewer": "Reviewer_8eDx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**RR1:** Possible overfit of nonlinear regression model.\n\n**Response:** Response: Our goal in this work is to find factors that best explain the performance and social biases (if any) present in MLMs. In particular, we do not propose to predict such targets for unseen models (i.e. generalisation of the prediction model is not within the scope of this paper). On the other hand, if the factors we use can explain well (overfit to) the observed targets then that would be considered as more suitable.\n\n**RR2:** It is relatively hard to interpret the result from regression/averaging.  Prior work usually uses side-by-side comparison.\n\n**Response:** Please note that we use 39 models in our analysis, which makes it difficult to perform 39 x 38 / 2 = 741 side-by-side comparisons, which would also be difficult to interpret individually and when aggregated. For this reason, we group models by higher-level attributes (e.g. pre-train corpora, model size, tokeniser, etc.) and analyse trends that can be observed over those attributes via averaging. \n\nOn the other hand, as we already mentioned in the introduction section of the paper, one of the challenges in our analysis is that the underlying factors are not independent. Therefore, we use non-linear regression methods such as gradient boosted tree regression to consider the dependencies among those different factors. \n\n**Missing Reference:** Related work on NAS\n\n**Response:** Thank you for pointing out this work on NAS. Although it is not focused on social biases in MLMs, which is the main focus of our work in this paper, we agree that it would be useful to cite this work for its relevance to the factor analysis we perform. Therefore, we will include this reference in the final version of the paper if accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- the paper is well-written and easy to follow\n\n- comprehensive regression analysis that covers a wide range of existing pretrained MLMs", "reasons_to_reject": "- Possible overfit of nonlinear regression model: they use 39 data points (i.e., models) to fit 30 categorical features with gradient boosting. It does not evaluate the generalization ability (e.g., cross-validation).\n\n- It is relatively hard to interpret the result from regression/averaging. Prior work usually uses side-by-side comparison, accompanied by statistical tests (e.g., bert vs. distill-bert, roberta vs. distill-roberta, etc), when analyzing categorical factors.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The goal is to investigate the relationship between pretraining factors of masked language models and social bias & downstream performance.\n- To this end, they first take 39 off-the-shelf models and evaluate them on Crowdsourced Stereotype Pairs (CP), SteroSet (SS), TweetEval, and GLUE benchmark.\n- Then they perform regression analysis: 39 data points (i.e., models), 4 aforementioned metrics as response variables, and 30 categorical features are pretraining factors that cover five groups (model size, training methods, training corpora, tokenization, language).\n  - They compare various regression models: Gradient Boosting, SVM, Gaussian Process Regression, Decision Tree, Random Forest, and Logistic Regression. The performance is measured by the coefficient of determination (R^2) and RMSE. They find that GLUE cannot be reliably predicted by all regression models, and that gradient boosting is the best overall fit for the other three metrics.\n  - With gradient boosting, they compare the feature importance. The analysis highlights the importance of model size, tokenization, and training objectives.\n  (1) element-wise analysis: gini importance score for each feature, RMSE drop by removing a single feature.\n  (2) group-wise analysis: RMSE drop by removing a single group (out of five) of features, RMSE score by only retaining a single group.\n- They also present an empirical comparison within some of the important features by direct averaging: while tokenization is important, they find that BPE is generally better than WordPiece or SentencePiece; multilingual models have lower social bias while monolingual models have better performance on English-only downstream tasks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 468: nd RMSE -> and RMSE\n\n- For comprehensiveness, it would be helpful if you can list 30 features (similar to Table 3) along with the performance for all 39 models in the appendix."}, "extra_text": {"Missing_References": "Related work w.r.t. methodology: The framework bears some similarity with work on prediction-based neural architecture search (NAS). In particular, [1] uses gradient boosting based on binary features to predict task accuracy (section 3). The main difference is that [1] solely uses architecture-related features and uses prediction as an intermediate step within search iterations; this work uses mostly pretraining-related features along with some architecture-related features and performs extensive analysis.\n\n[1] https://arxiv.org/pdf/2007.04785.pdf"}}, {"review_id": "o03wX6HXNA", "reviewer": "Reviewer_QwCm", "summary": "", "questions": "A. For this regression analysis, is there theory/significance testing around this? Is the R^2 value good even though it is negative for GLUE? Even if this is a reliable model/significant R^2, some discussion confirming this would help the reader in contextualizing the results and their confidence in the strength of the findings.\n\nB. Do the other models with lower R^2 (e.g. SVM, decision tree, RFs) also point to the same conclusions regarding the most predictive factors? If so, that lends a little more credibility to the conclusions of the work.\n\nC. In Table 9, why does the small model do so well on GLUE? Is this due to small sample sizes or is there something anomalous (or absent) in the “S” split which is causing this aberrant behavior?\n\nD. Are the bold numbers in Table 9/10 significant? Which ones, if any, are?\n\nE. How does this analysis be related to SHAP value analysis (and why wasn’t SHAP value analysis considered/discussed?)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**RR1:** The regression analysis is difficult to interpret (e.g. R^2 for GLUE is always negative) -> As one example, I would imagine distillation should have some consistently correlated impact on GLUE. \n\n**Response:** As already stated above in the response to Reviewer 1, we use a regression model to analyse the impact of factors because those factors are not independent. By using a non-linear regression model (such as the gradient boosted tree regression we use in the paper) we are able to take into account the dependencies among the underlying factors. \n\nAs mentioned in Lines 470-472 (and also in the Limitations section) R^2 for GLUE is always negative, which indicates that a regression model cannot be properly trained based on the features that we consider in this paper for predicting GLUE scores. Moreover, the important scores in Table 5 show the distribution of the important factors (i.e. the sum of the important score for each column is 1). We will clarify this to better explain the results.\n\nDistillation indeed has non-zero important scores, indicating that distillation is an important factor for predicting GLUE score. However, distillation is not as predictive as other factors such as parameter size, tokeniser etc. \n\n**RR2:** A description of how the authors categorise models and the number of models for each factor.\n\n**Response:** Thank you for the suggestion. We will add this information in the Appendix as a table, showing which models belong to which category.\n\n**Q1:** Is the R^2 value good even though it is negative for GLUE? Even if this is a reliable model/significant R^2, some discussion confirming this would help the reader in contextualizing the results and their confidence in the strength of the findings.\n\n**Response:** Thank you for the helpful suggestion. Indeed, negative R^2 scores for GLUE indicate that a regression model cannot be properly trained based on the features that we consider for predicting GLUE scores. This is not the case for the social bias benchmarks, for example, in which the R^2 score is substantial in most cases and the importance of relative features can be properly assessed. We will make this clear in the final version and contextualize based on the R^2 values.\n\n**Q2:** Do the other models with lower R^2 also point to the same conclusions regarding the most predictive factors?\n\n**Response:**  The conclusions are largely similar. In the case of Random Forest, the most predictive factor on A-CP and TweetEval is the parameter size, while the most predictive factors on A-SS and GLUE are tokenization and the number of layers, respectively. These results agree with what is shown in Table 5, except that the parameter size is the second best on TweetEval in Table 5. \n\nSimilarly, in the case of decision trees, parameter size is the most predictive factor on A-CP, and the second most predictive factor on TweetEval. Tokenization and number of layers are the most predictive factors on A-SS and GLUE, respectively. \n\n**Q3:** In Table 9, why does the small model do so well on GLUE?\n\n**Response:** Please note that the number of models in each category is different and this is the reason behind this result. Specifically, there are 21 MLMs in the Medium category and 12 in the Large category, whereas there are only 6 in the Small category. Because some outlier cases are affecting the overall averages in each class, we compute the top-5 performing models in each class. For example, for S (Small) the top-5 GLUE scores are 85.348, 79.924, 78.487, 75.699, 74.676, while those in L (Large) are 83.221, 82.862, 81.546, 80.323, 79.293. The averages of these top-performing models show that L (avg 81.449) models have better GLUE scores than S (avg 78.827). We will include this information in the final version with the extra page allowed.\n\n**Q4:** Are the bold numbers in Table 9/10 significant? Which ones, if any, are?\n\n**Response:** We conducted Clopper-Pearson exact tests to check whether the best performances (indicated by bold values)  for A-CP and A-SS are statistically significant. According to the Clopper-Pearson exact test, A-SS scores obtained by large models in the vocabulary size category are statistically significant in Table 9. Moreover, A-SS scores obtained by the models using MLM + NSP training objectives are also statistically significant. Furthermore, in Table 10, all the bold values for A-SS are also statistically significant. \n\n**Q5:** Why not use SHAP?\n\n**Response:** Please note that we have 30 factors in our analysis, which results in 2^29 - 1 = 536,870,911 coalitions (subsets). Although there are efficient approximations for SHAP calculations, we did not explore this path due to computational issues, and instead used R^2 and Gini importance in this work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The posed question of determining the factors that affect social biases and subsequent methodology is an interesting one to explore, well-motivated as a natural “scaled up” version of similar prior work.\n* The conclusions drawn from the paper are insightful directions for future work and model development in the pretrained MLM space.", "reasons_to_reject": "* The regression analysis, which is one of the core components of the paper, is difficult to interpret (e.g. R^2 for GLUE is always negative), leading to unclearness around whether the results/analysis of the factors are significant or meaningful; this is also mentioned in the Limitations. \n  *  As one example, I would imagine distillation should have some consistently correlated impact on GLUE, but its importance is only 0.002?\n* While the appendix includes a description of all the factors and what each value means, it should also include all the models, how the authors categorized them, and the number of models with each value, for each factor. In particular, there is no mention of significance in any of the results, and maybe some categories are much smaller than others, which may lead to skewed results.", "questions_to_authors": "A. For this regression analysis, is there theory/significance testing around this? Is the R^2 value good even though it is negative for GLUE? Even if this is a reliable model/significant R^2, some discussion confirming this would help the reader in contextualizing the results and their confidence in the strength of the findings.\n\nB. Do the other models with lower R^2 (e.g. SVM, decision tree, RFs) also point to the same conclusions regarding the most predictive factors? If so, that lends a little more credibility to the conclusions of the work.\n\nC. In Table 9, why does the small model do so well on GLUE? Is this due to small sample sizes or is there something anomalous (or absent) in the “S” split which is causing this aberrant behavior?\n\nD. Are the bold numbers in Table 9/10 significant? Which ones, if any, are?\n\nE. How does this analysis be related to SHAP value analysis (and why wasn’t SHAP value analysis considered/discussed?)", "paper_topic_and_main_contributions": "The paper studies 39 pretrained Masked Language Models (MLMs) by identifying 30 factors and modeling the correlation between these factors and several benchmarks ranging from social bias, GLUE, and TweetEval. After determining the best predictive model from those ~30 factors to score, they identify which factors (features) are most important. They conclude that model parameters size and tokenization are typically the most important factors, followed by training objectives. In subsequent analysis, they compare all models of each class to, e.g., demonstrate that larger models score better on the benchmarks.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "L468: “nd” -> “and”\n\nL588-590 is unclear: models with 6 layers show a comparable capability to what? Reduced social bias or better performance on downstream tasks?\n\nWhen there is more space, or in the appendix, please include the sizes of the datasets used.\n\nAs mentioned, please include how what values each model's factors were labeled in the appendix, and ideally (commit to) releasing code too."}}, {"review_id": "nRaCPdj4OU", "reviewer": "Reviewer_Swdc", "summary": "", "questions": "(1) Why was the research design observational rather than experimental, and how does it limit the ability to establish causal relationships between model attributes and social biases in MLMs?\n\n(2) Can the consideration of multiple factors simultaneously (e.g., model size, training objectives, corpora, tokenization techniques, and language) lead to complex interactions and potential confounding effects?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**RR1/Q1:** Why was the research design observational rather than experimental, and how does it limit the ability to establish causal relationships between model attributes and social biases in MLMs?\n\n**Response:** As correctly pointed out an experimental setting where we experimentally change a single independent variable, keeping all other variables fixed (i.e. treating them as controlled variables) would be the ideal scenario for making straightforward conclusions. Unfortunately, however, retraining large MLMs is expensive, which forces us to resort to the observational analysis of existing pretrained MLMs, as conducted in this paper. However, one of the main methodological contributions of this paper is to demonstrate how such an observational analysis could be successfully conducted using existing pretrained MLMs. Given that many MLMs are trained and publicly released by different researchers in NLP/ML, we consider the ability to perform such an observational analysis in a cost-effective manner will be of interest to the wider EMNLP audience.\n\n**RR2/Q2:** Can the consideration of multiple factors simultaneously lead to complex interactions and potential confounding effects?\n\n**Response:** Indeed as we explicitly mentioned under challenge (a) in the introduction (Lines 46-50), we recognise this problem and propose to use gradient boosting regression, which considers the non-linear interactions among the underlying factors when predicting the given target variable (e.g. social bias scores, GLUE scores, etc.). Moreover, in order to further limit this potential issue on similar features, we cluster the features into high-level categories.\n\n**RR3:** The authors excluded de-biased models (CDA, Dropout, Context-Debias, Auto-Debias) as well as a prevalent bias benchmark (SEAT) within the experimental setup.\n\n**Response:** We agree that analysing the effect of debiasing methods is an interesting research direction. However, our goal in the paper is to investigate the underlying factors of pretrained MLMs. Therefore, we did not consider any debiased models as not to interfere with the main analysis. For evaluation, we decided to use two recent and commonly used benchmark datasets, CrowS-Pairs [Nangia+ 2020] and StereoSet [Nadeem+ 2021], because, unlike SEAT which is extended on word association datasets, these are explicitly designed to measure social bias on pretrained masked language models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main strengths of this paper pertains to the diversity of model types (monolingual/general, monolingual/domain-specific, and multi-lingual/general MLMs) and exhaustiveness of model factors (size, training data, tokenization methodology, pre-training objective) within the experimental setup. The authors also utilize two widely known bias benchmarks (StereoSet and Crow-S-Pair) while adapting their standard evaluation protocols to use a more representative metric (AULA), leveraging attention weights to encode token importance (robust against frequency biases).", "reasons_to_reject": "The main weaknesses of this paper pertains to the research design (observational rather than experimental), which limits the ability to establish causal relationships between model attributes and the prevalence of social biases in MLMs. Without experimental manipulation of variables, it is challenging to determine whether the identified correlations are genuinely indicative of cause-and-effect relationships. Additionally, the study considers multiple factors simultaneously, such as model size, training objectives, corpora, tokenization techniques, and language, which can lead to complex interactions and potential confounding effects. The inability of controlling all potential confounding variables is understandable due to training cost constraints, however, the authors should be cautious when interpreting the results and consider these limitations when drawing conclusions (challenging to establish causation). Finally, the authors excluded de-biased models (CDA, Dropout, Context-Debias, Auto-Debias) as well as a prevalent bias benchmark (SEAT) within the experimental setup.", "questions_to_authors": "(1) Why was the research design observational rather than experimental, and how does it limit the ability to establish causal relationships between model attributes and social biases in MLMs?\n\n(2) Can the consideration of multiple factors simultaneously (e.g., model size, training objectives, corpora, tokenization techniques, and language) lead to complex interactions and potential confounding effects?", "paper_topic_and_main_contributions": "This work aims to measure the impact of model attributes (size, training data, tokenization methodology, pre-training objective) on the prevalence of social biases in MLMs. To do so, the authors propose the use of predictive factor analysis, considering 39 models with 30 factors. The models used for analysis are monolingual/general, monolingual/domain-specific, and multi-lingual/general MLMs. The factors include model size (varying from less than 100M params to greater than 300M params), training objectives (whole word masking, subword masking, sentence ordering prediction, mention reference prediction), training corpora (general domain, social media, legal domain, academia domain, biomedical domain, multilingual), tokenization techniques (BPE, WordPiece, SentencePiece), and language (monolingual, multi-lingual). The bias scores for each model were computed using two prevalent bias benchmark (StereoSet and Crow-S-Pairs). General downstream performance was computed using GLUE and TweetEval. In order to conduct the analysis, the factors were fed into a regression model as input for training, using both bias score and task performance as output. The results indicate that model size, tokenization and training objectives are the three most consequential factors and multilingual models appeared to be less socially-biased than the monolingual ones.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Textural -> textual (Introduction)"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper systematically studies the impact of a large number of diverse MLM properties on task performance and social biases, applying a factor analysis/regression framework. Its main contribution is the large-scale analysis (many factors, many model) enabled by the adopted analysis methodology.

Overall, this is an innovative paper, which suggests a promising methodology for the ever important task of understanding the impact of MLM design decisions on their performance and biases.

The main weakness is the poor fit of the regression model for the GLUE benchmark, which calls into question the validity of all following analyses on GLUE -- to the extent that I would suggest to remove the GLUE results from the paper. At the very leas, the paper needs to add clear warning messaging around the reliability of the GLUE results. I also suggest to check whether specific GLUE task(s) are responsible for the low R2 value. 

I concur with all other concerns raised by the reviewers (and discussed subsequently) and take these into account.
Decision: Accept-Main