Understanding the Effect of Model Compression on Social Bias in Large Language Models
Authors: Gustavo Gonçalves
Emma Strubell
Venue: EMNLP 2023 Main (2023)
Keywords: social bias
large language models
model compression
quantization
knowledge distillation

Abstract: Large Language Models (LLMs) trained with self-supervision on vast corpora of web text fit to the social biases of that text. Without intervention, these social biases persist in the model's predictions in downstream tasks, leading to representational harm.
Many strategies have been proposed to mitigate the effects of inappropriate social biases learned during pretraining. 
Simultaneously, methods for model compression have become increasingly popular to reduce the computational burden of LLMs. Despite the popularity and need for both approaches, little work has been done to explore the interplay between these two.
We perform a carefully controlled study of the impact of model compression via quantization and knowledge distillation on measures of social bias in LLMs.
Longer pretraining and larger models led to higher social bias, and quantization showed a regularizer effect with its best trade-off around 20\% of the original pretraining time.

Understanding the Effect of Model Compression on Social Bias in Large Language Models
Gustavo Gonçalves 12 and Emma Strubell
Languugge Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, USA NOVA LINCS, Universidade NOVA de Lisboa, Lisbon, Portugal Allen Institute for Artificial Intelligence, Seattle, WA, USA iggoncalv, estrubell } @cs. cmu cdu
Abstract
prohibitively expensive both financially and envi- ronmentally (Hessenthaler et al., 2022). At the same time, the compression of LLMs has been intensely studied. Pruning, quantization, and dis- tillation are among the most common strategies to compress LLMs. Pruning reduces the parameters of trained model by removing redundant con- nections while preserving equivalent performance to their original counterparts (Liebenwein et al., 2021; Ahia al., 2021). Quantization reduces the precision of model weights and activations to improve efficiency while preserving performance (Ahmadian et al., 2023). Finally, knowledge distil- lation (Hinton et al., 2015) trains smaller more efficient model based on a larger pre-trained model.
Large Language Models (LLMs) trained with self-supervision on vast corpora of web text fit to the social biases of that text. Without intervention, these social biases persist in the model's predictions in downstream tasks, lead- ing representational harm. Many strategies have been proposed to mitigate the effects of inappropriate social biases learned during pre- training. Simultaneously, methods for model compression have become increasingly pop- ular reduce the computational burden of LLMs. Despite the popularity and need for both approaches, little work has heen done to explore the interplay between these two. We perform carefully controlled study of the im- pact of model compression via quantization and knowledge distillation measures of social bias in LLMs. Longer pretraining and larger models led higher social bias, and quantiza- tion showed regularizer effect with its best trade-off around 20% of the original pretraining time."
While much research has been done on mea- suring and mitigating social bias in LLMs, and making LLMs smaller and more efficient, by using one or combination of many compression meth- ods (Xu et al., 2021), little research has been done regarding the interplay between social biases and LLM compression. Existing work has shown that pruning disproportionately impacts classification accuracy on wwwfrreuencc categories computer vision models (Hooker et al., 2021), but that prun- ing transformer models can have beneficial effect with respect to bias when modeling multilingual text (Hooker et al., 2020; Ogueji et al., 2022). Fur- ther, Xu and Hu (2022) have shown that compress- ing pretrained models improves model fairness by working as regularizer against toxicity.
1 Introduction
Large Language Models (LLMs) are trained on large corpora using self-supervision, which allows models consider vast amounts of unlabelled data, and learn language patterns through mask- ing tasks (Devlin et al., 2019; Radford et al., 2019). However, sule-supe allows LLMs to pick up social biases contained in the training data. Which is amplified by larger models, more data, and longer training (Kaneko et al., 2022; Kaneko and Bollegala, 2022; Kurita al., 2019; Delobelle and Berendt, 2022).
Unlike previous work, our work focuses on the impacts of widely used quantization and distilla- tion on the social biases exhibited by a variety of both encoder- and decoder-only LLMs. We fo- cus on the effects of social bias over BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019) and Pythia LLMs (Biderman et al., 2023). We evaluate these models against Bias Bench (Meade et al., 2022), compilation of three social bias datasets.
Social biases in LLMs are an ongoing prob- lem that is propagated from pretraining to finetun- ing (Ladhak et al., 2023; Gira et al., 2022). Biased pretrained models are hard to fix, as retraining is
'https: / htt////tttt thub. com/ /gtp://ggnccass/////.. FMNT.P2023 Im m___omprees ress oow_annd_eeeee soc ialiall bias
In our experimental results we demonstrate a cor-

---

relation between longer pretraining, larger models, and increased social bias, and show that quantiza- tion and distillation can reduce bias, demonstrating the potential for compression as a pragmatic ap- proach for reducing social bias in LLMs.
model will pick an equal ratio of both stereotypi- cal and anti-strrotyppcal choices, thus an optimal score for this dataset is ratio of 50%.
StereoSet is composed of crowdsourced samples. Each sample is composed of masked context sentence, and set of three candidate answers: 1) stereotypical, 2) anti-stereotypical,, and 3) un- related. Under the SS formulation, an unbiased model would give balanced number of classifica- tions of types 1) and 2), thus the optimal score is also 50%. The SS datasct also mcasures if we are changing the language modeling properties of our model. That is, if our model picks a high percent- age of unrelated choices 3) it can be interpreted as losing its language capabilities. This is defined the Language Model (LM) Score.
2 Methodology
We were interested in understanding how dynamic Post-Training Quantization (PTQ) and distillation influence social bias contained in LLMs of different sizes, and along their pretraining. In dynamic PTQ. full-precision floating point model weights are stat- ically mapped to lower precisions after training, with activations dynamically mapped from high to low precision during inference. To this end, in Section 2.1 we present the datasets of the Bias Bench benchmark (Meade et al., 2022) that enable evaluate three different language modeling tasks across the three social bias categories. In Section 2.2 we lay out the models we studied. We expand on the Bias Bench original evaluation by looking at the Large versions of the BERT and RoBERTa models, and the Pythia family of au- toregressive models. The chosen models cover different language modeling tasks and span across wide range of parameter sizes, thus providing a comprehensive view of the variations of social bias.
SEAT evaluates biases in sentences. SEAT task is defined by two sets of attribute sentences, and two other sets of target sentences. The objec- tive of the task is to measure the distance of the sentence embeddings between the attribute and tar- get sets to assess preference between attributes and targets (bias). We provide more detail of this formulation in Appendix A.
2.2 Models
In this work, we focus on two popular methods for model compression: knowledge distillation and quantization. We choose these two methods given their competitive performance, wide deployment given the availability of distributions under the Hug- gingFFace and Pytorch libraries, and the lack of understanding of the impact of these methods on social biases. We leave the study of more elaborate methods for improving model efficiency such as pruning (Chen et al., 2020), mixtures of experts (Kudugunta et al., 2021). I), and adaptive computation (Elbayad et al., 2020) to future work.
2.1 Measuring Bias
We use the Bias Bench benchmark for evaluating markers of social bias in LLMs. Bias Bench com- piles three datasets, CrowS-Pairs (Nangia et al., 2020). StereoSet (SS) (Nadeem et al., 2021), and SEAT (Kaneko and Bollegala, 2021), for measur- ing intrinsic bias across three different identity cate- gories: GENDER, RACE, and RELIGION. While the set of identities covered by this dataset is far from complete, it serves useful indicator as these models are encoding common social biases; how- ever, the lack of bias indicated by this benchmark does not imply an overall lack of inappropriate bias in the model, for example with respect to other groups. We briefly describe each dataset below; refer to the original works for more detail.
Since model compression affects model size, we are particularly interested in understanding how pretrained model size impacts measures of social bias, and how that changes as a function of how well the model fits the data. We are also inter- ested in investigating how the number of tokens observed during training impacts all of the above. We experiment with three different base LLMs: BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and Pythia (Biderman et al., 2023), with uncompressed model sizes ranging from 70M pa- rameters to 6.9B parameters. BERT and RoBERTa
CrowS-Pairs is composed of pairs of minimally distant sentences that have been crowdsourced. A minimally distant sentence is defined as a small number of token swaps in a sentence, that carry different social bias interpretations. An unbiased

---

Model
Params
Size (MB)
GENDER
RACE
RELIGION
BERT Base
110M
438
57.25
62.33
62.86
+ DYNAMIC PTQ int8
110M
181
57.25
10.19 62.14
19.53 46.67
+ CDA (Webster et al., 2020)
110M
01.14
56.11
45.63 56.70
42.86 60.00
DROPOUT (Webster et al., 2020)
110M
55.34
13.30 59.03
47.62 55.24
INLP (Ravfogel et al., 2020)
110M
46.10 51.15
15.63 67.96
41.91 60.95
SELF-DEBIAS (Schick al., 2021)
110M
44.6 52.29
15.63 56.70
16667 56.19
SENTDEBIAS (Liang al., 2020)
110M
44.96 52,29
10.39 62.72
10.95 63.81
BERT Large
345M
1341
$1.52 55.73
11.94 60.39
14.76 67.6
+ DYNAMIC PTQ int8
345M
432
46.87
50.38
10.78 63.11
47.62 55.24
DistiBBRRT
66M
268
51.15 16.10
19.32
46.99
44.76 58.10
RoBERTa Base
123M
498
60.15
63.57
60.95
DYNAMIC PTQ int8
123M
242
46.51
53.64
45.04 58.53
410.47 49.52
CDA (Webster et al., 2020)
110M
56.32 33.83
to.19 63.76
10.95 59.05
+ DROPOUT (Wehster et al., 2020)
110M
30.76 59.39
01.17 62.40
42.86 57.14
+ INLP (Ravfogel et al., 2020)
110M
55.17 $4.98
41.75 61.82
11.91 62.86
+ SELF-DEBIAS (Schick et al., 2021)
110M
57.09 $3.06
LL.17 62.40
09.52 51.43
+ SENTDEBLAS (Liang et al., 2020)
110M
52.11 18.04
+1.55 65.12
41.9 40.95
RoBERTa Large
354M
1422
60.15
10.58 64.15
10.95 61.90
+ DYNAMIC PTQ int8
354M
513
$2.68
57.47
40.20
63.37
40:95 60.00
DistilRoBERTa
82M
329
52.87
60.08
12.86 63.81
Table 1: CrowS-Pair stereotype scores for GENDER, RACE, and RELIGION for BERT and RoBERTa models. Stereotype scores closer to 50% indicate less biased model hehavior. Bold values indicate best method per bias category. Results on the other datasets displayed similar trends and were included in Appendix B for space.
represent two similar sets of widely used and stud- ied pretrained architectures, trained on different data with small overlap. RoBERTa pretraining was done over 161 GB of text, which contained the 16GB used to train BERT, approximately ten-fold increase. RoBERTa also trained for longer, with larger batch sizes which have shown decrease the perplexity of the LLM (Liu et al., 2019).
the teacher model (soft-targets) and the true labels (hard-targets) to better generalize to unseen data.
Quantization compresses models by reducing the precision of their weights and activations during inference. We use the standard PyTorch implemen- tation' to apply dynamic PTQ over the linear layers of the transformer stack, from fp32 full-precision to quantized int8 precision. This work analyzes quantized BERT, RoBERTa, and Pythia models of comprehensive range of sizes.
The set of checkpoints released for the Pythia model family allows us to assess an even wider variety of model sizes and number of training to- kens, including intermediate checkpoints saved dur- ing pretraining, that we can observe how bias varies throughout pretraining. We used the mod- els pretrained on the deduplicated version of The Pile (Gao al., 2021) containing 768GB of text.
Results
Dynamic PTQ and distillation lower social bias. In Table analyze the effects of dynamic PTQ and distillation in the CrowS dataset, where BERT Base and RoBERTa Base are our baselines. To compare quantization and distillation, we add three debiasing baselines also referenced by Meade et al. (2022) that are competitive strategies to reduce bias. The INLP (Ravfogel et al., 2020) baseline consists of a linear classifier that learns to predict the target bias group given set of context words, such
Knowledge distillation (Hinton et al., 2015) is popular technique for compressing the knowledge encoded in larger teacher model into smaller student model. In this work, we analyze Distil- BERT (Sanh al.. 2019) and DistilRoBERTa* distilled LMs. During training the student model minimizes the loss according to the predictions of
"https:/ /ptcccch..cccccoocoooooocccccc ott/:utoriii..oooooo wet.ooiiilllleeeeooo wttps//seeeeeeeeeeeeeeee/eeeeeee itepppppeeeeceppss dynamic_ ionion htal.

---

600
LM
Gencirr bias
Race bias
Religion bias
Figure 1: LM score GENDER, RACE, and RELIGION bias on the SS dataset across all Pythia models. Darker data points show later pretraining steps, and more transparent points to earlier steps. The included table shows the Kendall Tau C, for the correlation across "All" model sizes, frel-prenision "Original". and "int8" model sizes.
Model Size
Best
Step
Bias
Model Size
Best
Step Nr.
Bias
LM Score
Nr.
G. / RA. RE.
LM Score
G. / RA. / RE.
70M
89.2
21K
59.8 59.845844/ 4 58.6
70M
87.7
29K
57.5 54.8 / 58.0
160M
90.2
36K
61.4 57.6/ 59.4
160M
89.0
21K
61.1 56.3 57.7
57.7
410M
91.6
114K
65.2766.77. 64.5
410M
90.5
50K
/5.2/88.4// 63.6 64.2
1.4B
92.6
129K
66.2
1.4B
91.4
29K
66.1/59.7 63.3
2.8B
92.9
114K
67.1 63.77 66.8
2.8B
91.6
50K
64.1/ 61.9
64.1
6.9B
92.7
129K
69.0 64.0 68.4
6.9B
91.4
21K
67.3 / 60.1 /
Table 2: Bias measured using SS for the full-precision Pythia models having the best LM score per model size.
Table 3: Bias measured using SS for int8 quantized Pythia models having the best LM score model size.
'helshe', The Self-Debias baseline was proposed by Schick et al. (2021), and uses prompts to encourage models to generate toxic text and learns to give less weight to the generate toxic tokens. Self-Debias does not change the model's internal representation, thus it cannot be evaluated on the SEAT dataset.
function of the pretraining of the Pythia models in Figures 2 to 7, 9, 10 and The BERT/RoBERTa Base and Large versions are roughly comparable with the 160M and 410M Pythia models. For the SS dataset, the 160M model is consistently less biased than the 410M model. However, this is not the case for the other two datasets where the 160M struggles in the RACF category while assess- ing the distance of sentence embeddings (SEAT); and in the RELIGION category while swapping min- imally distant pairs (CrowS). This illustrates the difficulty of distinguishing between semantically close words, and shows the need for larger models pretrained for longer and on more data.
Notable trends in Table are the reduction of social biases when applying dynamic PTQ and dis- tillation, which can compete average with the specifically designed debias methods. Additional results in Appendix also display similar trends. On the SS dataset in Table we are also able to observe that the application of distillation provides remarkable decreases in social biases, at the great expense of LM score. However, dynamic PTQ shows better trade-off in providing social bias reductions, while preserving LM score.
Longer pretraining and larger models lead more socially biased models. We study the ef- fects longer pretraining and larger models on social bias, by establishing the correlation of these variables in Figure 1. Here we can observe that as the model size increases so does the LM model score and social bias across the SS dataset. More- over, later stages of pretraining have higher LM model score, where the social bias score tends be high. The application of dynamic PTQ shows a regularizer effect on all models. The Kendall Tau across the models and categories shows a strong
One model size does not fit all social biases. In Table and the equivalent Tables in Appendix B we can see that social bias categories respond dif- ferently to model size, across the different datasets. While BERT Base/Large outperforms RoBERTa in GENDER, the best model for RACE and RELIGION varies across datasets. This can be explained by the different dataset tasks and the pretraining.
In Appendix we show the social bias scores as

---

correlation between LM score and social bias. Sta- tistical significant tests were performed using a one-sided t-test to evaluate the positive correlation. Tables 2 and 3 show at what step, out of the 21 we tested, the best LM scores occur on the SS dataset. In Table 2 the best LM score increases monotonically with model size and so do the social biases. Interestingly, as the model size increases the best LM score appears after around 80% of the pretraining. In opposition, in Table 3, with dynamic PTQ the best LM score occurs around 20% of the pretraining and maintains the trend of higher LM score and social bias, albeit at lower scores than the original models. This shows an interesting possibility of early stopping depending on the deployment task of the LLM.
days. A CPU implementation was used given the quantization backends available in PyTorch. Exper- iments that did not require quantization ran using an NVIDIA A100 40GB GPU and took approxi- mately 5 hours to run.
Ethics Statement
We reiterate that this work provides limited West- view of Social bias focusing only on three main categories: GENDER, RACE, and RELIGION. Our work is further limited to a binary definition of GENDER, which we acknowledge that does not re- flect the current society's ' needs. Moreover, we must also reiterate that these models need to be fur- ther studied and are not ready for production. The effects of quantization along pretraining should be considered as preliminary results.
4 Limitations
While this work provides three different datasets, which have different views on social bias and allow for an indicative view of LLMs, they share some limitations that should be considered. The datasets SS and CrowS define an unbiased model as one that makes an equal amount of stereotypical and anti-stereee strreoypttt choices. While agree that this makes a good definition of impartial model it is a limited definition of an unbiased model. This has also been noted by Blodgett et al. (2021), showing that CrowS is slightly more robust than SS by tak- ing "extra steps to control for varying base rates be- tween groups." (Blodgett et al., 2021). We should consider that these datasets depict mostly Western biases, and the dataset construction since it is based assessors it is dependent on the assessor's views. Moreover, Blodgett al. (2021) has also noted the existence of unbalanced stereotype pairs in SS and CrowS, and the fact that some samples in the dataset are not consensual stereotypes.
Acknowledgments
This work has been partially funded by the FCT project NOVA LINCS Ref. UIDP/04516/2220, by the Amazon Science TaskBot Prize Chal- lenge and the CMUIPortugal projects iFetch Ref. GoLocal Ref. CMUP-ER/TICCW06620014 and by the FCT Ph.D. scholarship grant Ref. WTRH//BD/44022//200
and
We would like ac- knowledge the NOVASearch group for providing compute resources for this work. Any opinions, findings, and conclusions in this paper are the au- thors' and do not necessarily reflect those of the sponsors.
References
Orevaoghene Ahia, Julia Kreutzer, and Sara Hooker. 2021. The Low-Resource Double Bind: An Empir- ical Study of Pruning for now-Resource Machine Translation. In EMNLP (Findings), pages 33 3316- 3333. Association for Computational Linguisties.
All datasets only explore three groups of biases: GENDER, RACE, and RELIGION, which are not by any means exhaustive representations of social bias. The experiments in this paper should be considered indicative of social bias and need to be further stud- ied. Additionally, the GENDER category is defined as binary, which we acknowledge that does not reflect the timely social needs of LLMs, but can be extended to include non-binary examples by improving existing datasets.
Arash Ahmadian, Saurabh Dash, Hongyu Chen, Bharat Venkitesh. Stephen Gou, Phil Blunsom, Ahmet Üstün, and Sara Hooker. 2023. Intriguing Properties of Quantization at Scale. CoRR, abs/2305. 110.2220...966.....
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. CoRR, abs/2304.01373..
We benefited from access to a cluster with two AMD EPYC 7 662 64-Core Processors, where the quantized experiments ran for approximately

---

Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu. Robert Sim, and Hanna M. Wallach. 2021. Stereo- typing Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets. In ACI/IJCNPP (1), pages 1004-1015, Association for Computational Linguistics.
Masahiro Kaneko and Danushka Bollegala. 2022. Un- masking the Mask Evaluating Social Biases in Masked Language Models. In AAAI, pages 11954- 11962. AAAI Press.
Masahiro Kaneko, Danushka Bollegala, and Naoaki Okazaki. 2022. Debiasing Isn't Enough! the Effectiveness of Debiasing MLMs and Their Social Biases in Downstream Tasks, In COLING, pages 1299-1310. International Committee Computa- tional Linguistics.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang. Zhangyang Wang, and Michael Carbin. 2020. The Lottery Ticket Hypothesis for Pre-trained BERT Networks. In NeurlPS.
Pieter Delobelle and Bettina Berendt. 2022. FairDistil- lation: Mitigating Stereotyping in Language Models. In ECML/PKDD (2), volume 13714 14 of Lecture Notes in Computer Science, pages 638-654. Springer.
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Lu- ong, and Orhan Firat. 2021. Beyond Distillation: Task-level Mixture-of--Expert for Efffcientt Inference. In EMNLP (Findings), pages 3577-3599. Associa- tion for Computational Linguistics.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLT 2019, Minneupolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pages 1171-4186. Association for Computational Linguisties.
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W. Black, and Yulia Tsvetkov. 2019. Measuring Bias in Contextualized Word Representations. CoRR. abs/1906.07337.
Faisal Ladhak, Esin Durmus, Mirac Suzgun, Tianyi Zhang, Dan Jurafsky, Kathleen R. McKeown, and Tatsunori Hashimoto. 2023. When Do Pre-Training Biases Propagate to Downstream Tasks? A Case Study in Text Summarization. EACL, pages 3198- 3211. I. Association for Computational Linguistics.
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. 2020. Depth-Adaptive Transformer. In ICLR. OpenReview.net.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing. Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2021. The Pile: An 800GB Dataset of Diverse Text for Language Model- ing. CoRR, 0bs/2101.00027.
Paul Pu Liang, Irene Mengze Li, Emily Zheng- Yao Chong Lim, Ruslan Salakhutdinov, and Louis- Philippe Morency. 2020. Towards Debiasing Sen- tence Representations. In ACL, pages 5502-5515. Association for Computational Linguistics.
Lucas Liebenwein, Cenk Baykal, Brandon Carter, David Gifford, and Daniela Rus. 2021. Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy. In ML.Sss misys.org.
Michael Gira, Ruisu Zhang, and Kangwook Lee. 2022, Debiasing Pre-Trained Language Models via Effi- cient Fine-Tuning. In LT-EDI, pages 59-69. Associa- tion for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretrain- ing Approach. CoRR, abs/1907.11692.
Marius Hessenthaler, Emma Strubell, Dirk Hovy, and Anne Lauscher. 2022. Bridging Fairness and Envi- ronmental Sustainability in Natural I Language Pro- cessing. EMNLP, pages 7817-7836 Association for Computational Linguistics.
Nicholas Meade, Elinor Poolc-Dayam, and Siva Reddy, 2022. An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Mod- els. In ACL (l), pages 1878-1899 Association for Computational Linguistics.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in neural network. In NIPS Workshop on Deep Learning.
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. 2021. What Do Com- pressed Deep Neural Networks Forget?
Moin Nadeem, Anna Bethke, and Siva Reddy, 2021. StereoSet: Measuring stereotypical bias in pretrained language models. ACL/JCCNLP (1), pages 5356- 5371. Association for Computational Linguisties.
Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. 2020. Characterising Bias in Compressed Models. CoRR, abs/2010.03058.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman, 2020. CrowS-Pairs: A Chal- lenge Dataset for Measuring Social Biases in Masked Language Models. In EMNLP (I), pages 1953-1967. Association for Computational Linguistics.
Masahiro Kaneko and Danushka Bollegala. 2021. De- biasing Pre-trained Contextualised Embeddings. In EACL, pages 1256-1266. Association for Computa- tional Linguistics.

---

Kelechi Ogueji. Orevaoghene Ahia, Gbemileke Onilude, Sebastian Gehrmann, Sara Hooker, and Julia Kreutzer. 2022. Intriguing Properties of Compres- sion on Multilingual Models. In EMNLP, pages 9092-9110. Association for Computational Linguis- tics.
Where A and B represent the attribute sets, and X and Y are the target tion Equation (1) denotes mean cosine similarity between the target word embeddings and the at- tribute word embeddings:
words. The func-
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
sww, A, B)
=
1
Σ
cos( w. a)
Σ BEB
cos w.
b).
|A|
B
aEA
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection. ACL. pages 7237-7255 Association for Computational Linguistics.
(2)
The reported score of the benchmark (effect size) is given by:
d M(ts(e, A, A) ((, B))
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, distilled version of BERT: Smaller, faster, cheaper and lighter. CoRR, abs/1910.01108.
o((s(t,X,
(3)
Where μ and are the mean and standard de- viation respectively. Equation (3) is designed that scores closer zero indicate the smallest pos- sible degree of bias. SEAT extends the previous formulation by considering the distance sentence embeddings instead of word embeddings.
Timo Schick, Sahana Udupa, and Hinrich Schüitze. 2021. Self-Diagnosis and self-Dehiasing: A Proposal for Reducing Corpus-Basee Based Bias in NLP. Trans. Assoc. Comput. Linguistics, 9:1408-1422
Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beu- tel, Emily Pitler, Ellie Pavlick, Jilin Chen, and Slav Petrov. 2020. Measuring and Reducing Gen- dered Correlations in Pre-trained Models. CoRR, abs/2010.06032.
B Additional Plots and Tables
Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Ju- lian J. McAuley, and Furu Wei. 2021. Beyond Pre- served Accuracy:
Evaluating Loyalty and Robustness BERT Compression. EMNLP (1), pages 10653-
10659, Association for Computational Linguistics.
Guangxuan Xu and Qingyuan Hu. 2022. Can Model Compression Improve NLP Fairness. CoRR, abs/2201.0p 008542.
A Details of Metric Calculation
A.1 SEAT
The SEAT task shares the same task as WEAT task, which is defined by four word sets, two attribute sets, and two target sets. For example, to decide the presence of gender bias the two attribute sets are disjoint sets given by: 1) masculine set of words, such ( ('man', 'boy', 'he", ...), and 2) set of feminine words ( /'woman', 'girl', 'her', ...). The target sets will characterize concepts such as 'sports' and 'culinary'.
WEAT evaluates how close are the attribute sets from the target sets to determine the existence of bias. Mathematically this is given by:
s(A,
Σ s(x, A.
Σ yEY
syy, A, B)
ZEX
(1)

---

Original Models
Dynamic Quantized Models int8
70-
85-
50
70M
K10M
698 18
1.49
7DM
210V
144
6.88
160M
2.88
150M
15
229
york
Zickk
Training steps (thousands)
Figure 2: Crows GENDER bias with Quantized Results
Original Models
Dynamic Quantized Models int8
85-
60-
70M 160M
410M
690 18
1.48
70M
410V
1.48
6.58
2.85
180M
18
2.88
40
3016
Training steps (thousands)
Figure 3: Crows RACE with Quantized Results
Original Models
Dynamic Quantized Models int8
75-
Bias
ane
lbsk
Tose
109
2034
Training steps (thousands)
Figure 4: Crows RELIGION bias with Quantized Results

---

Original Models
Dynamic Quantized Models int8
70-
10
70M 160M
K10M
1.49
6.8
7JH
410V
144
6.88
18
2.88
150M
18
228
20rk
1.3%
Zickk
Training steps (thousands)
Figure 5: Stereoset GENDER bias with Quantized Results
Original Models
Dynamic Quantized Models int8
70
85-
score
70M 160M
410M
6.6 18
1.48
70N
410V
1.48
6.58
2.85
160M
18
2,88
45
3076
123
Training steps (thousands)
Figure 6: Stereoset RACE bias with Quantized Results
Original Models
Dynamic Quantized Models int8
35
Bias score
50-
lbsk
10W
2034
Training steps (thousands)
Figure 7: Stereoset RELIGION bias with Quantized Results

---

Original Models
Dynamic Quantized Models int8
56-
34-
70M 160M
410M
1.43
6.98
TOM
410M
1445
6.00
18
Training steps (thousands)
Figure 8: Stereoset LM Score with Quantized Results
Table 4: SS stereotype Stereotype scores closer to 50% indicate less biased model behavior. Bold values indicate the best method per bias and LM Score. Results are on the SS test set. random model (which chooses the stereotypical candidate and the anti-stercotypical candidate for each example with equal probability) obtains stereotype score of 50% in expectation.
and language modeling scores (LM Score) for BERT, and RoBERTa models.
Model
GENDER bias
RACE bias
RELIGION bias
LM Score
BERT Base
60.28
57.03
59.70
84.17
+ DYNAMIC PTQ int8
3.29 56.99
12336 54.67
4287 56.83
2294 81.23
+ CDA (Webster et al., 2020)
J0.67 59.61
10.30
56.73
11.3 58.37
1.09 83.08
DROPOUT (Webster et al., 2020)
50.38 60.66
t0.44 57.07
00.57 59.13
1.14 83.04
INLP (Ravfogel et al., 2020)
33.03 57.25
57.29
4244 57.26
80.63
SELF-DERIAS (Schick al., 2021)
J0.94 59.34
54.30
U244 57.26
84.09
+ DEBTENCEDEBIIS (Liang al., 2020)
J0.91 59.37
10.75
57.78
00.97 58.73
00.0 84.20
BERT Large
22.6 63.24
10.04 57.07
10.24 59.94
60.24 84.41
+ DYNAMIC PTQ int8
J082 59.46
11.86 55.17
43.74 55.96
33.12 81.05
Distil BERT Base
8873 51.55
16.40
50.63
09:57 49.87
L30.30 53.87
RoBERTa Base
66.32
61.67
64.28
88.95
+
DYNAMIC PTQ int8
62.40
43.15
58.52
J0.03 64.25
83.20
+
CDA (Webster et al., 2020)
64.43
60.95
J0.23 64.51
83.83
DROPOUT (Webster et al., 2020)
J0.06 66.26
60.41
22.0 62.08
@0.11 88.81
INLP (Ravfogel et al., 2020)
99.06 60.82
43.41 58.26
33.4 60.34
00.70 88
+ SELF-DEBIAS (Schick et al., 2021)
$1.28 65,04
42.89 58.78
31.44 62.84
J0667 88.26
+ LeliIEncDDBbIAS (Liang et al., 2020)
3355 62.77
11.05 62.72
00.37 63.91
00.0] 88.94
RoBERTa Large
*0 66.83
11.37 60.30
10.21 64.49
"0.14 89.09
+ DYNAMIC PTQ int8
.272 63.60
12.10 59.57
00.40 63.88
J0.68 88.27
Distil RoBERTa Base
2204 64.28
10.36 61.31
61.16 65.44
60.24 89.19

---

Table 5: LM Scores vs. Biases on the SS dataset of the int8 models, the same steps with the best LM Score for the original (full-precision) models (Table 2)
Table LM Scores vs. Biases on the SS dataset of the original (full-precision) models, at the same steps with the best LM Score for the int8 models (Table 3)
Model Size
Step
Bias
Model Size
Step Nr.
Bias
LM Score
Nr.
G. / RA. RE.
LM Score
G. / RA. RE.
70M
87.7
21K
55.4/ / 56.8 58.8
70M
88.4
29K
58.9 / 53.4 58.0
160M
88.3
36K
59.4/ 54.7 57.3
160M
89.8
21K
62.7 57.7 57.0
410M
88.7
114K
/3.8 60.9
410M
91.5
50K
67.2
60.5 63.3
1.4B
90.1
129K
65.5 / 60.0 62.5
1.4B
91.8
29K
65.9 61.2 64.9
2.8B
90.5
114K
64.3 / 58.3 62.0
2.8B
92.4
50K
65.3 / 63.5 63.8
6.9B
90.5
129K
66.6 62.2 64.7
6.9B
92.2
21K
67.0 / 61.0 64.9
Original Models
Dynamic Quantized Models int8
02-
7M 100M
410M
1445
8.95
700
430M
1.40
6.88
2.00
162M
10
2.00
107E
some
Zoog
Training steps (thousands)
Figure 9: Scat GENDER bias with Quantized Results
Original Models
Dynamic Quantized Models int8
0.8
3.6
Bias
410M
100
430M
1.40
6.08
100M
ware
Training steps (thousands)
Figure 10: Seat RACE bias with Quantized Results

---

Original Models
Dynamic Quantized Models int8
64-
3.2
VOM
410M
1.45
8.95
700
430M
1.40
6.08
1004
2.00
162M
10
2.00
Lane
Training steps (thousands)
Figure 11: Seat RELIGION bias with Quantized Results
Table 7: GENDER bias on SEAT dataset. Effect sizes closer to 0 are indicative of less biased model representations. Bold values indicate the best method per test. Statistically significant effect sizes at p < 0.01 denoted by *, The final column reports the average absolute effect size across all six gender SEAT tests for each model.
Model
weat6
weat6b
weat7
weat7b
weat8
weat8b
Avg. Effect
BERT Base
0.931
0.090
-0.124
0.937
0.783
0.858
0.620
DYNAMIC PTQ int8
0.614
0.000
-0.496
0.711
0.401
0.549
J0.18 0.462
CDA
0.846
0.186
-0.278
1.342
0.831
0.849
to.12
0.722
DROPOUT
1.136
0.317
0.138
1.179
0.879
0.939
10.14 0.765
INLP
0.317
-0.354
-0.258
0.105
0.187
-0.004
(0.466 0.204
SENTENCEDEBIAS
0.350
-0.298
-0.626
0.458
0.413
0.462
J0.186 0.434
BERT Large
0.370
-0.015
0.418
0.221
-0.259
0.710
J0.288
0.332
DYNAMIC PTQ int8
0.905
0.273
1.097
0.894
0.728
1.180
10,226
0,846
Distil BERT
0.061
-0,222
0.093
-0.120
0.222
0.112
10,482
0.138
RoBERTa Base
0.922
0.208
0.979
1.460
0.810
1.261
0.940
DYNAMIC PTQ int8
0.350
0.177
0.389
1.038*
0.349
0.897
10.40
0.533
CDA
0.976
0.013
0.848
1.288
0.994
1.160
J0.660
0.880
DROPOUT
1,134
0.209
1.161
1.482
1.136
1.321
10.134
1.074
INLP
0.812 *
0.059
0.604
1.407
0.812 *
1.246
J0.117
0.823
SENTENCEDEBIAS
0.755
0.068
0.869
1.372
0.774
1.239
10.094
0.846
RoBERTa large
0.849
0.170
-0,237
0.900
0.510
1.102
JO.322 0.628
DYNAMIC PTQ int8
0.446
0.218
-0.368
0.423
-0.040
0.303
00640 0.300
Distil RoBERTa
1.229
0.192
0.859
1.504
0.748
1.462
70,099 0.999

---

Table RACE bias SEAT dataset. ABWS: angry-black sngry-black-womm--ttrrrooooo woman-stereotype.. Effect sizes closer indicative of less biased model representations. Bold values indicate the best method per test. Statistically significant effect sizes at < 0.01 are denoted by The final column reports the average absolute effect size across all seven race SEAT tests for each model.
Avg. Effect
Model
ABWS
ABWS-b
wcal3
weat3b
wcal4
weat5
weatSb
BERT Base
-0.079
0.690
0.778
0.469
0.901
0.887
0.539 *
0.620
DYN. PTQ int8
0.772
0.425
0.835
0.548
0.970
1.076
0.517
10.15
0.735
CDA
0.231
0.619
0.824
0.510
0.896
0.418
0.486
10.051
0.569
DROPOUT
0.415
0.690
0.698
0.476
0.683
0.417
0.495
40.067
0.554
INLP
0.295
0.565
0.799
0.370
0.976
1.039
0.432
0.639
SENTDEBIAS
-0.067
0.684
0.776
0.451
0.902
0.891
0.513
0.612
BERT Large
-0.219
0.953
0,420
-0.375
0.415
0.890
-0.345
40.10
0.517
+ DYN. PTQ int8
0.660
-0.118
-0.173
0.093
-0.318
0.337
0.364
10.308
0.295
Distil BERT
1.081
-0.927
0,441
0.202
0.358
0.726
-0.076
10.076
0.544
RoBERTa Base
0.395
0.159
-0.114
-0.003
-0.315
0.780
0.386
0.307
+ DYN. PTQ in18
0.660
-0.118
-0.173
0.093
-0.318
0.337
0.364
10012
0.295
+ CDA
0.455
0.300
-0.080
0.024
-0.308
0.716
0.371 *
10.015
0.322
+ DROPOUT
0.499
0.392
-0.162
0.044
-0.367
0.841
0.379
10.0%6 0.383
+ INLP
0.222
0,445
0,354 *
0.130
0.125
0.636
0.301
10.009 0.316
+ SENTDEBIAS
0.407
0.084
-0.103
0.015
-0.300
0.728
0.274
10.034 0.273
RoBERTa Large
-0.090
0.274
0.869
-0.021
0.943
0.767
0.061
10.125
0.432
DYN. PTQ in18
-0.065
-0.014
0.587
-0.190
0.572
0.580
-0.173
10.004
0.312
Distil RoBERTa 0.774
0.112
-0.062
-0.012
-0.410
0.843
0.456
0.381
Table RELIGION bias on SEAT dataset. Effect sizes closer to are indicative of less biased model representations. Bold values indicate the best method per test. Statistically significant effect sizes at < 0.01 denoted by *, The final column reports the average absolute effect size across all four religion SEAT tests for each model.
Model
religionl
religionlb
religion2
religion2b
Avg. Abs. Effect.
BERT Base
0.744
-0.067
1.009
-0.147
0.492
+ DYNAMIC PTQ int8
0.524
-0.171
0.689
-0.205
20.095 0.397
+ CDA
0.355
-0.104
0.424
-0.474
30.152 0.339
DROPOUT
0.535
0.109
0.436
-0.428
J0.115 0.377
INLP
0.473
-0.301
0.787
-0.280
00.031 0.460
SENTENCEDEBIAS
0.728
0.003
0.985
0.038
$0.033 0.439
BERT Large
0.011
0.144
-0.160
-0.426
$0.306 1.686
+ DYNAMIC PTQ int8
0.524
-0.171
0.689
-0.205
00.095 0.397
Distil BERT
0.172
0.529 *
0.318
0.076
30.218 0.274
RoBERTa Base
0.132
0.018
-0.191
-0166
0.127
+ DYNAMIC PTQ int8
0.527
0.567
0.079
0.020
00.12 0.298
CDA
0.341
0.148
-0.222
-0.269
60.119 0.245
DROPOUT
0.243
0.152
-0.115
-0.159
10.041 0.167
INLP
-0.309
-0.347
-0.191
-0.135
19.199 0.246
+ SENTENCEDEBIAS
0.002
-0.088
-0.516
-0.477
60.14 0.271
RoBERTa Large
-0.163
-0.685
-0.158
-0.542
602260 0..87
+ DYNAMIC PTQ int8
0.117
-0.292
0.293
0.015
00.05 0.1
Distil RoBERTa
0.490
0.019
0.291
-0.131
10.10 0.232


Reviews:
[{"review_id": "Ai7xCgavqi", "reviewer": "Reviewer_az8p", "summary": "", "questions": "- A. Table 2-3 in the main text are captioned as SteroSet dataset with Pythia and Pythia-int8. Table 5-6 in the appendix have seemingly the same captions but with different values. Please clarify this.\n\n- B. Question on DistillBERT bias score (Table 1: CrowS data): 51.15, 46.99, 58.10 (gender, race, religion). The score is notably different from that of a recent paper [2]: 56.87, 60.97, 66.67 (see their Table 2). Could you explain the possible reasons for the discrepancy?\n\n[2] https://aclanthology.org/2023.acl-long.878.pdf (I understand the paper was published shortly before the EMNLP deadline and thus there is no novelty concern.)", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to carefully review our paper. We appreciate the opportunity to address your concerns and present our case for the acceptance of this paper.\n\nWe are pleased that you found merit in our paper and its contributions to the field, namely: valuable insights into the utilization of out-of-the-box compression methods as potential solutions for mitigating social bias.  The effects of compression on LLMs' social bias are still underexplored, and we focused particularly on studying the social biases of these models while in their pre-trained state.\n\nWe acknowledge that the results from the SEAT dataset show a mixed narrative, though we disagree that mixed results should be a reason for rejection. With an extra page of content, we can add more discussion of these results into the paper. Notably, RoBERTa Base/Large achieves a competitive lower average bias that the distilled version struggles to enhance (Tables 7 to 9). However, the trend that the distilled version is better holds true for BERT. It's important to note that RoBERTa is a better-trained version of BERT, with a longer training duration over a larger dataset. We hypothesize that the SEAT task is more complex and benefits from models with more representativeness, thus making it more challenging to improve on social bias with out-of-the-box compression methods, which can crudely reduce the model's expressiveness. \n\nThe ambivalence of results in applying the studied out-of-the-box compression algorithms on the SEAT dataset is something to be noted. We argue that even though these straightforward methods may not enhance every social bias or perform consistently across all models, our work demonstrates their potential as effective strategies for swiftly reducing social bias. We do not anticipate these methods to consistently outperform debiasing approaches, but we believe that achieving similar performance to debiasing methods will inspire the community to expand on this line of research, for example by exploring methodology in the intersection of compression and debiasing.\n\nWe noticed that we got an average reproducibility score, but could not find improvement points with respect to reproducibility. We will be happy to revise this either on the main paper, or appendix, and we will release the modifications that we made to the BiasBench codebase.\n\nFinally, we want to thank you for your technical writing improvements, which we will incorporate into the final version of our paper.\n\n**QA Section**\n\nQ1: \"Table 2-3 in the main text are captioned as SteroSet dataset with Pythia and Pythia-int8. Table 5-6 in the appendix have seemingly the same captions but with different values. Please clarify this.\"\n\nA1: Thank you for raising this point, as it shows that a rephrasing of the captions is required. In Tables 2 and 3 we analyze the three types of bias at the point where the LM score reaches its peak, comparing the unquantized and quantized models.\nMoving on to Tables 5 and 6, we utilize the frozen step numbers determined in Tables 2 and 3 to examine both the LM score performance and social bias performance in the opposite model. In other words, we evaluate the performance of the original/quantized model at the point where the previous one had shown its peak. For instance, referring to Table 2, we note that for the unquantized model with 70M parameters, the highest LM score is 89.2 at step 21K, accompanied by its respective biases. Now, turning to Table 5, we observe that at the same step, 21K, which yielded the best result for the unquantized model, the LM score for the quantized model is 87.7.\n\nQ2: \"Question on DistillBERT bias score (Table 1: CrowS data): 51.15, 46.99, 58.10 (gender, race, religion). The score is notably different from that of a recent paper [2]: 56.87, 60.97, 66.67 (see their Table 2). Could you explain the possible reasons for the discrepancy?\n[2] https://aclanthology.org/2023.acl-long.878.pdf (I understand the paper was published shortly before the EMNLP deadline and thus there is no novelty concern.)\"\n\nA2: Thank you for pointing out this very interesting work. Their DistilBERT results are indeed very different from ours, unfortunately, it does not seem that they open-sourced their codebase, and included implementation details such as the used seed to generate such results. The utilization of different seeds has a big performance impact as shown by Sellam et al. (Figure 4) [3]. We think this could be one of the primary reasons for the discrepancy of our results with [2]. We utilize the BiasBench codebase to reproduce results and perform our compression experiments, without further details from [2] we cannot say with certainty why the results differ.\n[3] https://arxiv.org/pdf/2106.16163.pdf", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- With the help of recent Pythia model checkpoints, the paper investigates bias progression along with various model sizes and training steps.\n\n- The experiment validates the positive effect of PTQ on CrowS and SteroSet. (1) It consistently reduces bias for BERT-base/large and RoBERTa-base/large for CrowS and SteroSet (Table 1, Appendix Table 4). (2) For SteroSet, when selecting models based on the best LM score, it reduces bias of Pythia (Table 2, 3). It additionally shows best LM score of PTQ occurs early in the training progress. (3) When comparing models with the same training steps, PTQ usually reduces biases (Appendix Figure 2-7).", "reasons_to_reject": "- The effect of knowledge distillation is not sufficiently studied. (1) The conclusion is based only on the comparison between BERT-base vs. DistilBERT and RoBERTa-base vs. DistilRoBERTa. (2) The conclusion that KD reduces bias is mostly true for CrowS (Table 1) and  SteroSet (Appendix Table 4) datasets (although the religion bias sometimes increases). However, the SEAT results have a mixed trend (Appendix Table 7-9): DistillBERT reduces bias while DistilRoBERTa increases bias, in terms of average effect. \n\n- As acknowledged above, PTQ has positive results on CrowS and SteroSet. However, the result on SEAT is somewhat negative.\n(1) PTQ effect of BERT and RoBERTa is ambivalent (Appendix Table 7-9): BERT-base (gender -0.158, race +0.115, religion -0.095), bert-large (+0.226, -0.305, -0.095), roberta-base (-0.406, -0.012, +0.172), roberta-large (-0.64, +0.004, +0.052).\n(2) For PTQ of Pythia (Appendix Figure 9-11), bias often increases for moderate-size models (i.e., >=410M), when comparing models with the same training steps. That is, when overlapping the left and right curves, the PTQ curve often occurs above the original curve.\n\n- In summary, both methods reduce bias in CrowS and SteroSet datasets. However, the appendix shows a different trend with the SEAT dataset. Considering CrowS and SteroSet datasets are under question [1], the inconsistent result from SEAT raises more concerns.\n\n[1] https://aclanthology.org/2021.acl-long.81.pdf", "questions_to_authors": "- A. Table 2-3 in the main text are captioned as SteroSet dataset with Pythia and Pythia-int8. Table 5-6 in the appendix have seemingly the same captions but with different values. Please clarify this.\n\n- B. Question on DistillBERT bias score (Table 1: CrowS data): 51.15, 46.99, 58.10 (gender, race, religion). The score is notably different from that of a recent paper [2]: 56.87, 60.97, 66.67 (see their Table 2). Could you explain the possible reasons for the discrepancy?\n\n[2] https://aclanthology.org/2023.acl-long.878.pdf (I understand the paper was published shortly before the EMNLP deadline and thus there is no novelty concern.)", "paper_topic_and_main_contributions": "The paper empirically analyzes the effect of model compression on the social bias of pretrained language models: knowledge distillation and dynamic post-training quantization (PTQ). In the experiment, social bias is evaluated with three intrinsic bias benchmarks (CrowS-Pairs, StereoSet, SEAT) that cover three types of bias (gender, race, religion). The experimented models include BERT-base/large, RoBERTa-base/large, and Pythia models (decoder-only) with various sizes.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Table 1 & 4. The side numbers (relative change) and boldness are sometimes misleading. Since the optimal score is 50 for CrowS &  SteroSet, the improvement should be based on absolute deviation. For instance, in Table 1 religion column, roberta-base+ptq should have side number: abs(49.52-50)-abs(60.95-50)=-10.47, and this cell should be bolded since this is the least biased one (closest to 50). This is the practice used in the prior work [3]; see their Table 4 GPT2+SentenceDebias.\n\n- Table 5 caption: steps of \"int8\" model, LM score of \"original\" model. Table 6 caption: steps of \"original\" model LM score of \"int8\" model. Possibly a mix-up.\n\n[3] https://aclanthology.org/2022.acl-long.132.pdf"}}, {"review_id": "RvjfCLXa04", "reviewer": "Reviewer_nH78", "summary": "", "questions": "1. Given that CrowS-Pairs and StereoSet contain 9 and 4 types of social bias, respectively, I wonder why you only consider the three identity categories rather than all bias types that are included in the CrowS-Pairs and StereoSet?\n\n2. There are other debiasing methods widely used for debiasing LMs, such as Counterfactual Data Augmentation debiasing (CDA; Webster et al., 2020), dropout debiasing (DO; Webster et al., 2020) and Token-level debiasing (AT), did you compare the results of using compression methods with them?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to carefully review our paper. We appreciate the opportunity to address your concerns and present our case for the acceptance of this paper.\n\nWe are pleased that you found merit in our paper and its contributions to the field. We acknowledge that there is more analysis to be done on the impacts of distillation on social biases. However, to achieve this analysis we would need to engage in the task of creating our own version of the distilled BERT and RoBERTa models, which escapes our out-of-the-box premise. The disclosure of intermediary checkpoints for the distilled student versions would enable a swift study of bias evolution for distilled models. Nevertheless, our results allow us to understand that the pre-trained distilled models tend to significantly decrease bias. We agree that a logical next step would be to analyze the effects of bias during the distillation of the pre-training process.\nWe agree that the study of re-learned biases during finetuning is an interesting and complementary experiment to this work. However, for this short paper, we focus the scope of our work on the pre-trained models, before finetuning. \nWith the simplicity of the techniques used in this work, we hope to inspire future research into the study of more elaborate compression methods across a variety of LLMs, while also examining social bias.\n\nTables 1,4,7,8, and 9 explicitly compare intra-model differences i.e. Base, Large, and Distilled through the absolute difference alongside the table values. We can also compare the inter-model performance across those model sizes by performing a rank analysis. This involves determining the ranking (1st, 2nd, or 3rd) of models with lower biases. We can observe an average rank order, according to the table below: Distil < Base < Large.\n\nModel   |  Avg. Rank\n\nDistill    |     1.56\n\nBase    |     2.17\n\nLarge   |     2.28\n\nWe observed in our experiments that some biases will not benefit from the regularization effects of compression methods. However, the overall trend is that larger models and longer training can lead to higher biases.\nOur choice of model parameters across the Pythia family was also to observe the effects of similar-sized models to Large (Pythia-410M), Base (Pythia-160M), and Distilled (Pythia-70M). This exercise needs to be done with caution, as we are comparing encoder models with decoder-only, where encoders benefit from bidirectionality while decoders do not. We can observe a consistent pattern in the line plots where the prevailing order from more biased models to less biased models is 410M (Green) > 160M (Yellow) > 70M (Blue). Moreover, PTQ tends to lower the bias during some pretraining parts across these 3 model sizes.\n\nFinally, we want to thank you for your technical writing improvements, which we will incorporate into the final version of our paper.\n\n**QA Section**\n\n\n\nQ1: \"Given that CrowS-Pairs and StereoSet contain 9 and 4 types of social bias, respectively, I wonder why you only consider the three identity categories rather than all bias types that are included in the CrowS-Pairs and StereoSet?\"\n\nA1: We limited our analysis to 3 types of social bias, influenced by building on top of the BiasBench benchmark [1]. This benchmark focuses on the 3 types of social biases that are common across datasets and allow for a high-level comparison. We acknowledge in the Limitations section that this is a limited representation of social bias that needs to be expanded in future work.\n\nQ2: \"There are other debiasing methods widely used for debiasing LMs, such as Counterfactual Data Augmentation debiasing (CDA; Webster et al., 2020), dropout debiasing (DO; Webster et al., 2020) and Token-level debiasing (AT), did you compare the results of using compression methods with them?\"\n\nA2: We indirectly compared our results with CDA (Webster et al., 2020), and dropout debiasing (DO; Webster et al., 2020). These two methods are baselines of the BiasBench (Meade et al. 2022) paper, which forms the base for our evaluation. Due to space constraints, we report INLP, Self-Debias, and SentenceDebias baselines as they are, on average, more performant than CDA and DO when applied to BERT and RoBERTa. These can be included in the final version of our paper for clarity, as these results were already reported in the BiasBench paper. With respect to the Token-level debiasing (AT), we did not compare against this debiasing technique as it happens at fine-tuning time and our work focuses on the effects of social-bias using out-of-the-box pre-trained models.\n\n[1] - Nicholas Meade, Elinor Poole-Dayan, Siva Reddy\nAn Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models. ACL (1) 2022: 1878-1898", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a comprehensive investigation into the impact of model compression techniques, specifically quantization and knowledge distillation, on social bias measures in large language models (LLMs). The study provides substantial evidence supporting the observation that longer pretraining durations and larger model sizes are associated with higher levels of social bias in LLMs. This work sheds light on the possibility of mitigating social bias in LLMs through compression techniques, even without explicitly engaging in the process of debasing. The paper is easy to follow.", "reasons_to_reject": "Further investigation and analysis are necessary especially to ascertain the presence and persistence of social biases in the student model after the training process. Kaneko et al. (2022) showed that LMs debiased using different methods still re-learn social bias during finetuning on downstream tasks. It is not clear whether the social biases present in DistillBERT are re-learned during the training of the student model. \n\nFurthermore, it should be noted that the statement made in lines 258-259, claiming that \"longer pretraining and larger models lead to more socially biased models\", is drawn from observations solely concerning Pythia models (results shown in Figure 1). However, this statement does not hold true when comparing BERT base and BERT large models, as demonstrated in Table 1. Specifically, the analysis reveals that BERT large achieves lower bias scores on both gender and race biases.", "questions_to_authors": "1. Given that CrowS-Pairs and StereoSet contain 9 and 4 types of social bias, respectively, I wonder why you only consider the three identity categories rather than all bias types that are included in the CrowS-Pairs and StereoSet?\n\n2. There are other debiasing methods widely used for debiasing LMs, such as Counterfactual Data Augmentation debiasing (CDA; Webster et al., 2020), dropout debiasing (DO; Webster et al., 2020) and Token-level debiasing (AT), did you compare the results of using compression methods with them?", "paper_topic_and_main_contributions": "In this paper, the authors study the effects of model compression approaches (i.e., quantization and knowledge distillation) on social bias measures in large language models (LLMs). Their findings show that longer pretraining and larger models tend to exhibit higher levels of social bias. Moreover, the study demonstrates that compression approaches, such as quantization and knowledge distillation, have the potential to reduce social bias in LLMs.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It would be better to mention the full name of \"PTQ\" (in line 85) when first mentioned to ensure clarity and proper identification."}, "extra_text": {"Missing_References": "1. Webster, K., Wang, X., Tenney, I., Beutel, A., Pitler, E., Pavlick, E., Chen, J., Chi, E. and Petrov, S., 2020. Measuring and reducing gendered correlations in pre-trained models. arXiv preprint arXiv:2010.06032.\n2. Kaneko, M. and Bollegala, D., 2022, June. Unmasking the mask–evaluating social biases in masked language models. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 11954-11962."}}, {"review_id": "dEu9uVvTeA", "reviewer": "Reviewer_KtY1", "summary": "", "questions": "1. In Table 1, the Stereotype scores closer to 50% indicate a less biased model. For the RELIGION column, why you highlight the 40.95 which is not the result that is the closest to 50%? This makes me quite confused.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to carefully review our paper. We appreciate the opportunity to address your concerns and present our case for the acceptance of this paper.\n\nWe are pleased that you found merit in our paper and its contributions to the field. Our work aims to explore the potential of using out-of-the-box compression methods, which are easily accessible and are the first line of techniques for deployment in production environments. Therefore, we focused on simple compression techniques, whose effects on social bias are underexamined.\nWe recognize the simplicity of the techniques used in this work, which we believe to be a strength of the paper, demonstrating that out-of-the-box compression methods can already help mitigate social bias. We also acknowledge that this work examines a subset of all social biases, as stated in the Limitations section of our paper. Yet, we feel that we experimented with a wide enough variety of biases, techniques and models to demonstrate the promise of methods combining compression and social bias mitigation sufficient to demonstrate a clear trend and inspire future research.\n\nWe want to highlight that our experiments were designed to include both encoder and decoder-only variations of LLMs. We chose the Pythia model family to represent state-of-the-art decoder-only models since it uses an open-source architecture based on GPT-NeoX [1].\nWhile we are also curious about the performance of other state-of-the-art decoder-only LLMs such as OpenAI’s GPT models, we note that it is currently impossible to perform this study on those models due to the model parameters not being shared, hence we used a close open-source alternative., We believe there is significant value in embracing open-source initiatives that allow us to peek into the performance during pretraining, which is a perspective currently beyond reach with OpenAI’s GPT models. \n  \nWe noticed that we got an average reproducibility score, but could not find improvement points with respect to reproducibility. We will be happy to revise this either on the main paper, or appendix, and we will release the modifications that we made to the BiasBench codebase.\n\nFinally, we want to thank you for your technical writing improvements, which we will incorporate into the final version of our paper.\n\n[1] Black, Biderman, Hallahan, Anthony, Gao, Golding, He, Leahy, McDonell, Phang, Pieler, Prashanth, Purohit, Reynolds, Tow, Wang, and Weinbach. \"GPT-NeoX-20B: An Open-Source Autoregressive Language Model.\" In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models. 2022.\n\n\n**QA Section**\n\nQ1: \"In Table 1, the Stereotype scores closer to 50% indicate a less biased model. For the RELIGION column, why you highlight the 40.95 which is not the result that is the closest to 50%? This makes me quite confused.\"\n\nA1: Thank you for pointing this out. You are correct, and we apologize for the confusion. The intended emphasis was on the Dynamic PTQ int8 cell with the value of 49.52, which is indeed the value closest to 50%. We will amend this for the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper explores an underexamined interplay between model compression and social bias reduction in LLMs. This is an interesting new direction for bias analysis.\n\n2. The experiments systematically evaluate various compression methods on multiple model architectures. The results clearly demonstrate compression can reduce biases to a certain degree.\n\n3. The writing clearly explains the background, experiments, results, and implications. The visualizations effectively summarize key trends.\n\nOverall, this paper makes a solid contribution to an important emerging topic with thoughtful experiments and analysis. The presentation and writing quality are publication-worthy.", "reasons_to_reject": "1. Limited technical contributions. The compression techniques evaluated are standard existing methods like quantization and distillation. The debiasing baselines are also from prior work. There is little technical innovation.\n\n2. Limited datasets and models. The bias benchmarks only assess gender, race, and religion. Other important biases and datasets are not measured. Also missing are assessments on state-of-the-art generative models like GPT.\n\n3. Writing logic needs improvement. Some parts, like introducing debiasing baselines in the results, make the flow confusing.", "questions_to_authors": "1. In Table 1, the Stereotype scores closer to 50% indicate a less biased model. For the RELIGION column, why you highlight the 40.95 which is not the result that is the closest to 50%? This makes me quite confused.", "paper_topic_and_main_contributions": "This paper examines the effect of model compression techniques like quantization and knowledge distillation on social biases in large language models (LLMs). \n\n**Key points:**\n\n- This paper evaluates compressed versions of BERT, RoBERTa, and Pythia LLMs on benchmark datasets measuring gender, race, and religion biases.\n\n- The results show that compression via quantization and distillation tends to reduce social bias while maintaining language modeling performance. The reductions are comparable to specifically designed debiasing techniques.\n\n- Longer pretraining and larger uncompressed models lead to higher social bias.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Explain what is PTQ when it first appears."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies how the compression of LLMs affects their social biases. This kind of study is very timely as efficient versions of LLMs are becoming more widespread and may affect users in unpredictable ways. An interesting aspect of this study is that it ablates its results over model size and training steps, by comparing different model checkpoints. The findings are non-trivial, as the Authors find that quantisation and distillation can reduce bias. However, the results on the SEAT dataset remain mixed. Moreover, some conclusions regarding the effect of model size based on Pythia seem to be contradicted by BERT-style models. Both of these points warrant further discussion on possible explanations, but overall I recommend to accept this paper.
Decision: Accept-Main