An Empirical Study on the Characteristics of Bias upon Context Length Variation for Bangla (2024)

Abstract: Pretrained language models inherently exhibit various social biases, prompting a crucial examination of their social impact across various linguistic contexts due to their widespread usage. Previous studies have provided numerous methods for intrinsic bias measurements, predominantly focused on high-resource languages. In this work, we aim to extend these investigations to Bangla, a low-resource language. Specifically, in this study, we (1) create a dataset for intrinsic gender bias measurement in Bangla, (2) discuss necessary adaptations to apply existing bias measurement methods for Bangla, and (3) examine the impact of context length variation on bias measurement, a factor that has been overlooked in previous studies. Through our experiments, we demonstrate a clear dependency of bias metrics on context length, highlighting the need for nuanced considerations in Bangla bias analysis. We consider our work as a stepping stone for bias measurement in the Bangla Language and make all of our resources publicly available to support future research.1

heading: 1 Introduction
text: Language models, encompassing both contextfree and contextualized variants, have increasingly demonstrated human-like biases (e.g., Bolukbasi et al., 2016; Caliskan et al., 2017). With the introduction of newer concepts, such as more sophisticated language models, correspondingly nuanced strategies for bias detection have become necessary (May et al., 2019; Kurita et al., 2019; Guo and Caliskan, 2021). As a consequence, sentence-level bias detection strategies have emerged. However, bias detection strategies primarily concentrate on English, with limited research in other languages. Recent efforts target bias detection in Dutch, Arabic, and Chinese languages (Chávez Mulsa and
*Both authors contributed equally 1https://github.com/csebuetnlp/BanglaContextualBias
Spanakis, 2020; Lauscher et al., 2020; Liang et al., 2020). In the case of Indic languages, Pujari et al. (2020) conducted a comprehensive analysis of bias linked to binary gender associations in the Hindi language. Moreover, Malik et al. (2022) underscore the vital role of cultural awareness in examining bias measurement by conducting socially aware experiments on the Hindi language. Despite these valuable contributions, Bangla, the sixth most spoken language in the world with over 230 million native speakers comprising 3% of the world’s total population2, has received scant attention in bias analysis and remains an underrepresented language in the NLP literature due to a lack of quality datasets (Joshi et al., 2020). This gap in research significantly limits our understanding of the bias characteristics present in existing language models under various linguistic contexts for this widely spoken language.
Addressing this limitation, this work endeavours to introduce Bangla, a low-resource language into the realm of bias analysis, through a study focused on gender bias. We also posit the question: does the amount of contextual information provided to a model influence the application of bias measurement methods in contextual settings? To answer this query, in this study, we present (1) an empirical investigation comprising the creation of a dataset tailored for intrinsic gender bias measurement in Bangla, (2) discussions on necessary adaptations to apply existing bias measurement methods for Bangla, and (3) an examination of the impact of varying context lengths on bias measurement methodologies within a Bangla-based framework. Our findings reveal notable dependencies of bias metrics on context length, shedding light on nuanced considerations for bias analysis in language models.
2https://w.wiki/Psq
1501

heading: 2 Linguistic Characteristics of Bangla: Gender Perspectives
text: Bangla as a language has some inherently different characteristics in representing gender as opposed to English. Bangla lacks gender-specific pronouns unlike English (he, she) and uses a common pronoun for both genders. But it represents Boy-Girl, ManWoman etc word pairs (common nouns) similarly like English. Because of common nouns being gendered in Bangla, we use common nouns instead of pronouns for experiments where masking the gendered word of a sentence is necessary.

heading: 3 Methodology
text: Our research focuses on bias measurement in contextual settings. We provide two intrinsic bias measurement methodologies for comparison. We choose these methods because they represent two distinct approaches to bias measurement: embedding extraction and mask prediction.

heading: 3.1 Baseline: WEAT and SEAT
text: As our initial baselines, we utilize WEAT (Caliskan et al., 2017) and SEAT (May et al., 2019), two well-established methods based on the embedding extraction approach for measuring bias. WEAT is designed as a statistical measure for the association strength between a pair of word vectors. To conduct this experiment, we curate a dataset specifically for Bangla by adapting the original dataset to fit into the Bangla context. We use distinct sets of Target vs Attribute word categories as shown in Table 1. To extract the corresponding embedding vectors, we train static word embedding models (word2vec and GloVe) on Bangla2B+ (Bhattacharjee et al., 2022). Subsequently, we compute effect sizes (measuring the size of bias) and corresponding p-values to assess statistical significance. The SEAT experiment extends WEAT to be applicable for sentence embeddings allowing assessment of modern contextual embedding systems for bias. For the SEAT experiment, we use template sentences for each category having Target vs Attribute words from Table 1. Methodological details are further provided in appendix B.

heading: 3.2 Contextualized Embedding Association Test (CEAT)
text: To quantify the inherent biases in Contextual Word Embeddings (CWE) produced by pre-trained language models, we employ CEAT (Guo and
Caliskan, 2021)—an extension of WEAT. As opposed to WEAT, CEAT accounts for variations in calculated effect sizes based on changes in its input context, generating a representation of random effects in the effect size distribution (Hedges, 1983). Specifically, it utilizes a random-effects model to compute the weighted mean of the effect sizes and the corresponding statistical significances as a measure of bias. The mathematical foundations of this approach are elaborated in Appendix D. In addition to reporting effect sizes, we aim to demonstrate how the effect size is influenced by variations in input context length as an extended study.
For a particular segment length l, we generate ns CWE from ns extracted sentences for each stimulus s. We do this for selected lengths of sentences (l = 9, 25, 75, >75) which we refer to as segments3. For each segment length l, we randomly sample for each stimulus N times. If the stimulus appears in less than N sentences, we sample with replacement to ensure that the distribution is preserved. We provide the analysis and results for N = 5000 (Table 2) and N = 1000 (appendix D.4).

heading: 3.3 Log Probability Bias Score Test
text: We explore the mask prediction based approach by adopting the framework introduced by Kurita et al. (2019). This method assesses bias in contextual models that are trained using a masked languagemodelling (MLM) objective. Given BERT’s training objective to predict [MASK] tokens, we design distinct template sentences for individual categories of Target vs Attribute pairs (Table 1). Using the predicted values of corresponding mask tokens, we report the effect size of each category.
We use generalized template sentences suitable for any contrasting Target vs Attribute word pairs (Figure 5c). We compute the bias by calculating ptgt and pprior where
1. ptgt = P([MASK] = [TARGET] | sentence) (We replace only [TARGET] with [MASK]).
2. pprior = P([MASK]=[TARGET] | sentence) (We replace both [TARGET] and [ATTRIBUTE] with [MASK]).
Finally, we compute the association between Target and Attribute using log ptgtpprior , which is our
3We refer to segment length as the total number of words in a sentence that we are feeding into the model to extract embedding. It is ensured that a word from the stimulus whose embedding is extracted exists in the sentence.
measure of bias. For notation purpose, we refer to ptgt as Fill Bias Score, pprior as Prior Bias Score and log ptgtpprior as the Prior Corrected Score or Log Probability Bias Score. Additionally, we study different sentence structures with varying amounts of context to examine how the variations influence the bias scores.

heading: 4 Data Preparation
text: We adopt the data preparation procedures based on the specific requirements of each experiment4.

heading: 4.1 Stimuli
text: In our Gender Bias experiments, we utilize categories from the original WEAT (Caliskan et al., 2017), we translate some words directly and culturally adapt others. For example, we include local floral species under the “Flowers” category and use common regional male and female names instead of English counterparts. Table 1 presents these categories, with examples provided in appendix A.1.

heading: 4.2 Contextualized Word Embedding
text: We generate the embeddings for stimuli from commonly used language models supporting Bangla (details in D.2). For extracting context-rich sentences we utilize the Bangla2B+ dataset (Bhattacharjee et al., 2022). Subsequently, we use the list of Target versus Attribute words to extract sentences containing these words by pattern matching method from unorganized raw data. Furthermore, to ensure effective data aggregation, we supplement the words having low sentence count with additional sentences to reach a minimum threshold.
Given the complexity of Bangla word suffixes, merely matching root words is ineffective and results in significant data loss. Bangla word
4We preprocess our sentences using the bangla text normalization proposed by Hasan et al. (2020)
suffixes often carry semantic values that resolve co-references, ensure subject-verb agreement etc. Even suffixes at times create entirely new words, altering the sentence’s semantics. To address this issue, we curate distinct suffix groups corresponding to the most commonly associated suffixes for each word in our designated word list. By associating each word with its respective set of suffixes, we construct different variations of a root word and extract sentences containing each variation. To extract the corresponding embeddings, we feed these sentences into a language model and take the target word embedding from the final layer.
We use around 250+ words across all categories and around 3 million sentences altogether in order to extract word embeddings and conduct CEAT experiment. During the embedding extraction process, we try to retain the entire word embedding, including its suffixes, to ensure the preservation of semantic nuances. This approach enables us to better capture the semantic representation of a word within a specific context. To achieve this, we perform mean pooling on the logits of all fragments of the target word after it is tokenized by the model. We provide a more comprehensive analysis of the dataset creation procedure along with examples in appendix C.

heading: 4.3 Context Aware Templates
text: We follow the context-based templating of Kurita et al. (2019) in order to carry out experiments for Bangla to calculate log-probability bias. For this, we hand-engineer five different types of context aware sentence structures with placeholders for Target words (Male terms vs Female terms) and Attribute words (Positive qualities vs Negative qualities) (examples in Appendix A.4). These range from simple sentences with no context (S1) to sentences with significant context drawn from the Bangla2B+ dataset (S5). Our objective was to introduce variability in both subject and object positions within sentences while minimizing the number of structures employed, thereby also incorporating variations in context length.
To construct our experimental dataset, we incorporate 110 positive and 70 negative attribute words. This process yields a diverse array of sentences capturing various linguistic contexts. We also use 4 different male and female terms (common noun) each. We report the bias on an aggregation of all these male and female terms due to the absence of gender specific pronouns in Bangla. In total, we
generate 3600 sentences, collectively representing the spectrum of contexts under scrutiny.

heading: 5 Results and Evaluation

heading: 5.1 Case Study: Effect of Context Variation on CEAT
text: We employ CEAT to assess the impact of contextual variance on bias, as depicted in Table 2. The choice of sample size N = 5000 is validated from the results of Guo and Caliskan (2021) as they have shown there is no significant difference between samples of N = 1000 and N = 10000. Our study focuses on elucidating how the length of contextual input influences effect size.
Effect size demonstrates the variability of observed bias based on segment length, stabilizing with increased contextual information. Figure 1 illustrates the dynamic changes in effect size between two models as context length varies. We observe that a moderate context length (around 20 words) is the optimum point for consistent results. We employ both fixed and random sets to sample
combinations for each CEAT experiment, where fixed sets allow for cross-model comparisons and random sets assess the impact of context variation on effect size for a certain segment length. Experiments with 5000 and 1000 samples do not exhibit a significant change in effect size, but decreased number of cases yielding statistically significant values.
Our results indicate statistically significant bias, varying across models, with some instances showing bias in the opposite direction. Notably, the MuRIL model demonstrates heightened context sensitivity for fixed samples. Table 2 demonstrates that statistically insignificant results mostly spawn in lower segment lengths, an observation that is consistent in detailed result tables provided in appendix D.3.
Key Take-away: Effect sizes converge to a definite value after a moderate amount of context length, but the differences in value are not drastic. Additionally, more context length ensures more statistically significant results.

heading: 5.2 Case Study: Effect of Context Variation on Log Probability Bias Scores
text: The template-based methodology, as introduced by Kurita et al. (2019), offers a direct approach for querying models based on modeling objectives, demonstrating enhanced consistency in human bias evaluation. The Fill Bias Score provides a direct insight into model biases and comprises of two components: the inherent language bias, quantified as the prior bias score, and the bias introduced by the presence of attributes, which is the actual bias measure referred to as the Prior Corrected Score or Log Probability Bias Score. In practical scenarios, models engage with naturally occurring sentences.
In Figure 2, our focus is directed solely towards the examination of negative traits within the context of BanglaBERT Generator (additional results in Figure 8). A consistent distribution of corrected bias scores across all sentence structures imply that the disparity in prior bias distribution is due to inherent language bias.
For sentence structures S1 to S3, the prior bias score exhibits increased inherent language bias with the introduction of additional words, leading to an expanded range. An opposite trend is observed for S4 to S5, where values tend to cluster around a neutral point. This observed trend from S1 to S3 indicates a shift in the model’s behavior as the attribute adopts a more context-rich setup, highlighting the model’s distinct preferences. Moreover,
certain corrected bias scores shift from negative to positive values with increased context, consistent with the observations in 5.1.
Sentence structures S4 and S5 emulate a more natural linguistic setting. Excessive context opens the model to assign higher probabilities to nontarget words, leading to a shift in focus and a decrease in the difference between probabilities for male and female target words. This phenomenon is evident from the plots in Figure 2. The plots reveal values tightly clustered around the neutral point for both corrected bias scores and prior bias scores (more examples in appendix 8).
Key Take-away: Providing excessive context and complicated structure shifts focus of the model, allowing inherent language bias to become the primary influence on the bias score.

heading: 6 Conclusion
text: In this research, we aim to examine bias in Bangla language models through creating a curated dataset and assert that the bias result outcome is influenced by the amount of context used in templates. Further exploration can be conducted on other lowresource languages. In future, we plan to investigate the effects of bias in downstream applications of Bangla language models, with the goal of developing language-specific debiasing methods to mitigate harmful bias in Bangla embeddings and extend these efforts to generative models as well.

heading: Limitations
text: Although our work is a stepping stone for introducing bias analysis in Bangla, there are limitations that highlight opportunities for future research. To maintain compliance with standard bias measurement methods, most of our datasets are adapted from existing datasets and therefore synthetic in nature. Moreover, our investigations predominantly focus on gender bias. Our motivation to only work with gender bias in this particular work stems from two reasons. Firstly, gender bias is universally prevalent. Secondly, compared to the others, gender bias exhibits significantly more nuanced variations, making it a rich area of exploration for preliminary work. Several flaws in the intrinsic bias assessment techniques that we applied have already been noted (Blodgett et al., 2020). Our goal was to lay the groundwork for future studies on Bangla bias instead of focusing on the flaws of the alreadyestablished methods. Further experimentation with other forms of biases such as social, religious, political etc., along with corresponding debiasing methods can be explored in future extensions.
Another limitation of our study is the reliance on controlled templates for bias analysis, without considering downstream applications. It would be interesting to extend this work by studying the prevalence of bias in real-world applications such as personalized dialogue generation (Zhang et al., 2018), summarization (Hasan et al., 2021, Bhattacharjee et al., 2023a), and paraphrasing (Akil et al., 2022). Finally, our study does not cover generative language models, which have seen significant advancements recently. Ensuring fairness in these models is crucial, and therefore, studying bias properties in both Bangla-specific (Bhattacharjee et al., 2023b) and multilingual (Touvron et al., 2023) generative models is also a promising direction for future research.

heading: Ethical Considerations
text: Since our work focuses on gender bias and datasets related to this social prejudice, it can be potentially triggering to people. However, it is necessary to conduct this research in order to ensure fairness in the field of natural language models. We also acknowledge the fact that our work focuses on gender as a binary entity, non-binary entities can be a space for further investigation.

heading: Appendix

heading: A Sample words and sentences for each experiment

heading: A.1 Categories
text: Figure 4 contains examples of words in each category. In the WEAT experiments, we use these words in each category and extract their embeddings using models. We then perform bias detection calculations. For the other experiments, we use this group of words in different sentences having varying context.

heading: A.2 SEAT sentence examples
text: In order to construct sentences for SEAT experiment, we use template sentences and insert words from Figure 4 in each of these templates. We use the translated versions of the template sentences from the original SEAT experiment. Figure 5b contains examples of sentences related to a Flower word and a Male term.

heading: A.3 Log Probability Bias examples
text: In Figure 5c, we present example sentences for the log probability bias experiment. In each sentence, the Target word and the Attribute word is highlighted, these words are systematically masked in order to calculate the probability bias score and effect size. For better clarity, we highlight the target words as red and the attribute words as blue. By following the templating algorithm, we calculate the fill bias scores, corrected bias scores and the logarithmic differences between these probabilities.

heading: A.4 Context Aware Sentence Structures
text: In Figure 6b, we provide examples of each structure of sentences from S1 to S5, the context increases from S1 to S5. Furthermore, between S2 and S3, the main difference is the variation of the subject-object placement. Finally in S5, a sentence is picked out from real life examples (newspapers/articles) on the internet.
In Table 3, we provide the organizational structure of the different category of sentences that we used for bias measurement in mask language modelling technique.
In Figure 3, we provide the list of words that we use in order to calculate the aggregate values of fill bias score vs corrected bias score graphs. Since Bangla does not contain gendered pronouns,
we use a number of gendered nouns to replace the usual strategy of using gendered pronouns for experimentation. We calculate the aggregation of these groups of gendered nouns in order to include a wider range of gendered words.

heading: A.5 CEAT sentence examples
text: We provide an example of the types of sentences that were used for the CEAT experiment in Figure 5a, it can be noticed that these long sentences contain much more context than the other experiments. The reason is that these sentences are scraped from actual human texts, newspapers, articles, books etc. The goal was to represent regularly used human language for bias measurement.

heading: B Details of Benchmarking methods
text: We describe the methodological details of the benchmarking experiments we conduct for Bangla.

heading: B.1 Word Embedding Association Test (WEAT)
text: In order to quantify bias in English language embeddings, Caliskan et al. (2017) proposed the Word Embedding Association Test (WEAT), they compared two sets of target words with two sets of attribute words. They quantified their comparison by calculating the effect size(d) and statistical significance (p-value).
To execute the WEAT experiment, we use distinct sets of Target vs Attribute words categorized as shown in Table 1. For the extraction of embedding vectors corresponding to each word, we use two distinct Bangla pretrained word embedding models, specifically the word2vec and the GloVe embedding models. Subsequently, we compute effect sizes and corresponding p-values to assess statistical significance, with a significance threshold set at p < 0.05.
To calculate the effect size -
ES = meanx∈Xs (x,A,B)− meany∈Y s (y,A,B)
std_devw∈X∪Y s (w,A,B)
In this context, “effect size” refers to the size of the bias as determined by the WEAT metric. The effect size is expressed as Cohen’s d, a common measure of the difference between two means that has been standardized by the standard deviation of the data. A larger Cohen’s d value denotes a stronger bias between the targets. According to Cohen’s effect size metric, d > |0.5| and d > |0.8| are medium and large effect sizes, respectively. (Rice and Harris, 2005)

heading: B.2 Sentence Encoder Association Test (SEAT)
text: To perform the SEAT experiment for Bangla language, following the approach of May et al. (2019), we curate a comprehensive list of sentence templates. Each target word from the WEAT target list is incorporated into the SEAT template sentences. We use the Bangla translated version of the semantically bleached templates in May et al. (2019). We use the final layer of BanglaBERT (Bhattacharjee et al., 2022) to extract embeddings for each sentence. We then use these embeddings to calculate the effect size of the curated list of sentences based on the mentioned categories.

heading: C CEAT Data Extraction
text: Data extraction for CEAT experiment is carried out using the Bangla2B+ dataset. Examples of naturally occurring sentences from the dataset is provided in Figure 5a. As mentioned in section 4.2, we utilize pattern-matching to extract the suitable sentences. We create unique suffix groups for each word from our categories.

heading: C.1 Relevance of Suffix
text: Unlike English, Bangla words typically contain suffixes when used in a sentence. Due to this characteristic, it was necessary for our methodology to consider the presence of suffixes before applying pattern matching to extract the groups of sentences for each word in the category. Figure 7a depicts the presence of suffix for a root word in naturally occurring sentences. If the presence of suffixes is not accounted for, the number of extracted sentences for a specific root word is significantly reduced.

heading: C.2 Significance of Unique Suffix Groups
text: Our investigation further reveales that relying on a common group of suffixes across all words in our Target vs Attribute categories is inadequate and error-prone. Since in Bangla, each word has its own unique set of suffixes that are appended in sentences. In Figure 7b, we provide examples of correct vs wrong sets of suffixes for a specific root word. It is evident that if wrong set of suffixes are applied to a word, it would result in erroneous extraction of sentences.
To tackle this characteristic, we create a list of 21 distinct suffix groups and link each word to its corresponding group. Each suffix group contains 2 to 15 suffixes based on the type of word the group will be assigned to. This process enables accurate sentence extraction based on the specific word-suffix combination within the dataset.

heading: D CEAT

heading: D.1 Random Effects Model
text: Random Effects Model5 (also known as Variance Components Model) is a statistical model where the model parameters are random variables. The model assumes that the data being analysed are drawn from a hierarchy of different populations whose differences relate to that hierarchy. Our
5"Random effects model", Wikipedia, last modified 8 December, 2023, https://en.wikipedia.org/wiki/ Random_effects_model
calculation of CEAT assumes that the differences between effect size due to contextualized variation for two sets of target words in terms of their relative similarity to two sets of attribute words is accounted for some random variable uncorrelated with independent variables.
Components of the same category have constant heterogeneity over time. These differences are caused by random contextual factors, represented by a random variable that does not directly influence the independent variables in the model.
The effect size (Cohen’s d) for ith sample is calculated by
ESi = meanx∈Xs(x,A,B)−meany∈Y s(y,A,B)
std_devw∈X∪Y s(w,A,B)
The in-sample variance estimation (denoted by Vi), is the square of std_devw∈X∪Y s(w,A,B). The between-sample variance, σ2between is estimated using the same principle of ANOVA and calculated using the formula
σ2between =
{ Q−(N−1)
c if Q ≥ N − 1 0 if Q < N − 1
where Wi = 1
vi ,
c = ∑ Wi − ∑
W 2i∑ Wi ,
and
Q = ∑
WiES 2 i −
∑ (WiESi) 2
∑ Wi
The weight vi is the assigned weight to each effect size in measuring the combined effect size (CES). The parameter is determined by calculating the inverse of the sum of estimated in-sample variance Vi and estimated between-sample variance in the distribution of random effects σ2between.
vi = 1
Vi + σ2between
CES is the sum of the weighted effect sizes divided by the sum of all weights,
CES(X,Y,A,B) = ∑N i=1 viESi∑N
i=1 vi
Hypothesis Test: The standard error (SE) of CES is calculated to derive the hypothesis test. SE is calculated with the formula below
SE(CES) =
√ 1
∑N i=1 vi
Based on the central limit theorem, the limiting form of the distribution of CESSE(CES) is the standard normal distribution (Montgomery and Runger, 2010). We noticed that some of the CES values are negative, so we use a two tailed p− value which can test the bias significance in two directions. The hypothesis for which the two-tailed p− value provides significance test is that there is no difference between all the contextualized variations of the two sets of target words in terms of their relative similarity to two sets of attribute words is given by the following formula
Pcombined(X,Y,A,B) = 2×[1−ϕ(| CES
SE(CES) |)]
where ϕ stands for for the standard cumulative distribution function and SE stands for the standard error.

heading: D.2 Language Models Used for CEAT Embedding Extraction
text: For extracting the embeddings necessary for CEAT experimentation, we use the output of the following models:
• BanglaBERT Large (Bhattacharjee et al., 2022) was trained using ELECTRA methodology (Clark et al., 2020). It contains 24 hidden layers. We use the outputs of the final layers as our word embeddings. We use both the generator (52M parameters) and the discriminator (339M parameters) versions of the model separately as they are trained on Masked Language Modelling (MLM) and Replaced Token Detection (RTD) objectives respectively.
• MuRIL (Khanuja et al., 2021) is a BERT (Devlin et al., 2019) model trained on two different language modelling objectives, Masked Language Modelling (MLM) and Translation Language Modelling (TLM). We use the MuRIL-large-cased version with 24 layers and 506M parameters. We extract the hidden unit values of the top layer as its CWE of 1024 dimensions. The base model has 238M parameters.
• XLM-RoBERTa (Conneau et al., 2020) is a transformer-based model (Vaswani et al., 2017) designed for multilingual natural language processing tasks. It was trained with multilingual MLM objective. We use the large version with 24 hidden layers and 560M parameters. The embeddings are taken from topmost layer with 1024 dimensions. The large model comprises of 560M parameters.
D.3 Results for sample size, N = 5000
In the main section, we mentioned a short result for CEAT analysis in Table 2. We present that in
broader form with more segment length variations in table 5. We include two more segment lengths for each model here.
D.4 Results for sample size, N = 1000 In this section, we are focusing on a smaller sample size, specifically N = 1000. We present our results in Table 6. One noticeable change is the reduction in cells demonstrating statistically significant values. However, for the most part, individual cells show only minor changes.
The key characteristics of the model, as highlighted for the N = 5000 sample in Table 5, are still quite evident in Table 6. This suggests that achieving similar results is possible even with a reduced sample size, especially when faced with resource constraints. Nevertheless, it is crucial to recognize that while the overall trends in the model’s behavior remain consistent, there are nuanced alterations in the statistical significance of certain cells.
Given these observations, we propose that optimizing the sample size could be a promising avenue for further investigation. Determining the optimal sample size, one that ensures reliable results without sacrificing statistical significance, presents an interesting area for future research and in-depth exploration.
−0. 020 −0. 015 −0. 010 −0. 005 0.0 00 0.0 05 0.0 10
Prior Bias Distribution
−0.006
−0.004
−0.002
0.000
0.002
C or
re ct
ed B
ia s
D is
tri bu
tio n
Prior Bias vs Corrected Bias: Positive Traits S1
−0. 01 0.0 0 0.0 1 0.0 2 0.0 3
Prior Bias Distribution
−0.004
−0.002
0.000
0.002
0.004
0.006
0.008
C or
re ct
ed B
ia s
D is
tri bu
tio n
Prior Bias vs Corrected Bias: Positive Traits S2
−0. 02 0.0 0 0.0 2 0.0 4 0.0 6 0.0 8 0.1 0 0.1 2
Prior Bias Distribution
0.00
0.01
0.02
0.03
C or
re ct
ed B
ia s
D is
tri bu
tio n
Prior Bias vs Corrected Bias: Positive Traits S3
−0. 08 −0. 06 −0. 04 −0. 02 0.0 0 0.0 2
Prior Bias Distribution
−0.025
−0.020
−0.015
−0.010
−0.005
0.000
0.005
0.010
C or
re ct
ed B
ia s
D is
tri bu
tio n
Prior Bias vs Corrected Bias: Positive Traits S4
−0. 075 −0. 050 −0. 025 0.0 00 0.0 25 0.0 50 0.0 75
Prior Bias Distribution
−0.02
−0.01
0.00
0.01
0.02
C or
re ct
ed B
ia s
D is
tri bu
tio n
Prior Bias vs Corrected Bias: Positive Traits S5
(a) Prior Bias Score vs Corrected Bias Score plots for positive traits in BanglaBERT - Large Generator.
−6 −4 −2 0 2 Prior Bias Distribution 1e−6
−3.0
−2.5
−2.0
−1.5
−1.0
−0.5
C or
re ct
ed B
ia s
D is
tri bu
tio n
1e−6
Prior Bias vs Corrected Bias: Positive Traits S1
1e−6
1e−6
−1.0 −0.8 −0.6 −0.4 −0.2 0.0 0.2 Prior Bias Distribution 1e−5
−3
−2
−1
0
1
C or
re ct
ed B
ia s
D is
tri bu
tio n
1e−6
Prior Bias vs Corrected Bias: Positive Traits S2
1e−5
1e−6
−4 −2 0 2 Prior Bias Distribution 1e−6
−1.5
−1.0
−0.5
0.0
0.5
C or
re ct
ed B
ia s
D is
tri bu
tio n
1e−6
Prior Bias vs Corrected Bias: Positive Traits S3
1e−6
1e−6
−6 −4 −2 0 2 Prior Bias Distribution 1e−6
−2.0
−1.5
−1.0
−0.5
0.0
0.5
1.0
1.5
C or
re ct
ed B
ia s
D is
tri bu
tio n
1e−6
Prior Bias vs Corrected Bias: Positive Traits S4
1e−6
1e−6
−6 −4 −2 0 2 4 Prior Bias Distribution 1e−6
−2
−1
0
1
2
3
4
C or
re ct
ed B
ia s
D is
tri bu
tio n
1e−6
Prior Bias vs Corrected Bias: Positive Traits S5
1e−6
1e−6
(b) Prior Bias Score vs Corrected Bias Score plots for positive traits in MuRIL - Large (cased).

heading: Future Work
text: Determining the optimal sample size, one that ensures reliable results without sacrificing statistical significance, presents an interesting area for future research and in-depth exploration. −0. 020 −0. 015 −0. 010 −0. 005 0. 0 00 0. 0 05 0. 0 10
Prior Bias Distribution
−0. 006
−0. 004
−0. 002
0. 000
0. 002
C or
re ct
ed B
ia s
D is
tri bu
tio n
Prior Bias vs Corrected Bias: Positive Traits S1
−0. 01 0. 0 0 0. 0 1 0. 0 2 0. 0 3
Prior Bias Distribution
−0. 004
−0. 002
0. 000
0. 002
0. 004
0. 006
0. 008
C or
re ct
ed B
ia s
D is
tri bu
tio n
Prior Bias vs Corrected Bias: Positive Traits S2
−0. 02 0. 0 0 0. 0 2 0. 0 4 0. 0 6 0. 0 8 0. 1 0 0. 1 2
Prior Bias Distribution
0. 00
0. 01
0. 02
0. 03
C or
re ct
ed B
ia s
D is
tri bu
tio n
Prior Bias vs Corrected Bias: Positive Traits S3
−0. 08 −0. 06 −0. 04 −0. 02 0. 0 0 0. 0 2
Prior Bias Distribution
−0. 025
−0. 020
−0. 015
−0. 010
−0. 005
0. 000
0. 005
0. 010
C or
re ct
ed B
ia s
D is
tri bu
tio n
Prior Bias vs Corrected Bias: Positive Traits S4
−0. 075 −0. 050 −0. 025 0. 0 00 0. 0 25 0. 0 50 0. 0 75
Prior Bias Distribution
−0. 02
−0. 01
0. 00
0. 01
0. 02
C or
re ct
ed B
ia s
D is
tri bu
tio n
Prior Bias vs Corrected Bias: Positive Traits S5
(a) Prior Bias Score vs Corrected Bias Score plots for positive traits in BanglaBERT - Large Generator. −6 −4 −2 0 2 Prior Bias Distribution 1e−6
−3. 0
−2. 5
−2. 0
−1. 5
−1. 0
−0. 5
C or
re ct
ed B
ia s
D is
tri bu
tio n
1e−6
Prior Bias vs Corrected Bias: Positive Traits S1
1e−6
1e−6
−1. 0 −0. 8 −0. 6 −0. 4 −0. 2 0. 0 0. 2 Prior Bias Distribution 1e−5
−3
−2
−1
0
1
C or
re ct
ed B
ia s
D is
tri bu
tio n
1e−6
Prior Bias vs Corrected Bias: Positive Traits S2
1e−5
1e−6
−4 −2 0 2 Prior Bias Distribution 1e−6
−1. 5
−1. 0
−0. 5
0. 0
0. 5
C or
re ct
ed B
ia s
D is
tri bu
tio n
1e−6
Prior Bias vs Corrected Bias: Positive Traits S3
1e−6
1e−6
−6 −4 −2 0 2 Prior Bias Distribution 1e−6
−2. 0
−1. 5
−1. 0
−0. 5
0. 0
0. 5
1. 0
1. 5
C or
re ct
ed B
ia s
D is
tri bu
tio n
1e−6
Prior Bias vs Corrected Bias: Positive Traits S4
1e−6
1e−6
−6 −4 −2 0 2 4 Prior Bias Distribution 1e−6
−2
−1
0
1
2
3
4
C or
re ct
ed B
ia s
D is
tri bu
tio n
1e−6
Prior Bias vs Corrected Bias: Positive Traits S5
1e−6
1e−6
(b) Prior Bias Score vs Corrected Bias Score plots for positive traits in MuRIL - Large (cased)