Selection Collider Bias in Large Language Models
Authors: Emily McMilin
Venue: Submitted to ICLR 2023 (2023)
Keywords: large language models
causal inference
selection bias

Abstract: In this paper we motivate the causal mechanisms behind sample selection induced collider bias (selection collider bias) that can cause Large Language Mod- els (LLMs) to learn unconditional dependence between entities that are unconditionally independent in the real world. We show that selection collider bias can become amplified in underspecified learning tasks, and although difficult to overcome, we describe a method to exploit the resulting spurious correlations for determination of when a model may be uncertain about its prediction. We demonstrate an uncertainty metric that matches human uncertainty in tasks with gender pronoun underspecification on an extended version of the Winogender Schemas evaluation set, and we provide online demos where users can evaluate spurious correlations and apply our uncertainty metric to their own texts and models. Finally, we generalize our approach to address a wider range of prediction tasks.

Under review as a conference paper at ICLR 2023
SELECTION INDUCED COLLIDER BIAS IN LLMs: A GENDER PRONOUN UNCERTAINTY CASE STUDY
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we cast the problem of task underspecification in causal terms, and develop a method for empirical measurement of spurious associations between gender and gender-neutral entities for unmodified LLMs, detecting previously un- reported spurious correlations. We then describe lightweight method exploit the resulting spurious associations for prediction task uncertainty classification, achieving over 90% accuracy on Winogender Schemas challenge set. Finally, we generalize our approach to address wider range of prediction tasks and pro- vide open-source demos for each method described here.
INTRODUCTION AND RELATED WORK
This paper investigates models trained to estimate the conditional distribution: P(Y| [X, S), where the cause of sample selection bias in training dataset. Selection bias is not an uncommon problem, as most datasets are subsampled representations of larger population, yet few are sampled with randomization (Heckman 1979).
1.1 CAUSAL DAGS AND BIASES
Sample selection bias occurs when some mechanism, observed or not, causes preferential inclusion of samples into the dataset (Bareinboim and Pearl, 2012). Employing the language of causal infer- ence, selection bias distinct from both confounder and collider bias. Confounder bias can occur when two variables have common cause, whereas collider bias can occur when two variables have a common effect. Correcting for confounding bias requires that one condition upon the common cause variable; conversely correcting for collider bias requires that one does not condition upon the common effect (Pearl 2009).
The type of selection bias that interests us here is that which involves more than one variable (observed or not), whose common effect results in selection bias. Such assumed relationships can be compactly and transparently represented causal data-generating process (DGP) in the form of a directed acyclic graph (DAG), for example illustrated in Figure [ The absence of arrows connecting nodes in causal DAGS encodes assumptions, for example that W and G in Figure ((a) are stochastically independent of one another. The direction of the arrowhead en- codes our assumptions about the direction of cau- sation. For example, the two arrows departing from W and G toward S Sencode the assumption that Sis common effect of W and G.
(a) Well-specified: Symbolic entity G. common cause of both dataset features and labels.
(b) Underspecifted: G unobserved by fea- tures, thus features contain no causes of the labels.
Figure 1: Data generating process for high dimensional data, such as in NLP, where X and Y represent high dimensional text fea- tures: the dataset features and labels, while W. G, and S represent low dimensional sym- bolic entities that may cause the text.
In Figure the twice-encireled node, S, symbol- izes some mechanism that can cause samples to be selected into the dataset. To capture the statistical process of sampling for dataset formation, one must condition on S, thus inducing the collider bias relationship between W and G into the DGP.

---

Under review as a conference paper at ICLR 2023
We will use the term selection induced collider bias to refer to circumstances such this one, when the selection bias mechanism induces a collider bias relationship in the dataset that would not have been there otherwisd" Selection induced collider bias has been covered in medical and epidemiological literature (Gritlith et al. 2020) (Munafò et al. 2018) (Cole et al.] 2009) and re- ceived extensive theoretical treatment from Pearl and Bareinhoim in (Bareinboim and Pearl 2012), [Bareinboim et al. (2014), (Bareinboim and Tian 2015) and (Bareinboim and Pearl 2016). yet has received very little attention deep learning literature.
1.2 UNDERSPECIFICATION AND SPURIOUS ASSOCIATIONS
We define a learning task as underspecified when none of the features available to the model (at training or inference time) are causes of the label. Figure ( encodes this relationship with the absence of an arrow between features, X, and labels, With no causal features available, models must resort to learning any spurious associations that will reduce predictive risk, regardless of how tenuous the association may be. We refer to these as otherwise non-interacting spurious associations.
We would like to draw a distinction between the type of spurious association induced by underspec- ification, and the spurious associations most often addressed in today's literature. For example, the task of predicting cow vs camel (perhaps based on spurious grassy vs sandy background pixel fea- tures), would not be considered an underspecified task, due to the availability of the causal cow vs camel pixel features in the foreground. From causal perspective, the symbolic background entity is common cause of both the pixel features and the labels, inducing confounder bias and thus the learning of spurious associations along a secondary path (Arjovsky et al. 2019), in addition to the primury direct causal path from feature to label.
natural question to ask is, how does spurious association flow from X Y, if not through some confounding variable like background, nor though direct causal path. As demonstrated in (D' Amoorr et al. 2020), weakly-interactiin to changes in the random seed initialization. In this work, by focusing variables engaged in relationship of selection induced collider bias, we are able to open up tertiary path between and Y: the path along X < W + S G + Y in Figure [(b] In distinction (D' Amour al. 2020), we argue this causal perspective facilitates the identification of otherwise non-interacting (and pre- viously unreported) spurious associations, and importantly enables the injection of these 'benign' spurious tokens into text at inference time, to achieve an uncertainty measurement.
prediction tasks display significant variance, even due
CONTRIBUTIONS
In this paper make the following contributions:
We cast the problem of task underspecification causal terms and apply causal inference methods to hypothesize the effects of selection induced collider bias underspecified tasks.
We test these hypotheses on unmodified and widely used pre-trained LLMs via a case study of gender pronoun resolution, resulting in two findings:
A method for empirical measurement of spurious correlations between gender and gender-
neutral entities for unmodified LLMs which permits measurement of previously unreported spurious correlations between gender vs location and time.
method for quantifying inference-time task uncertainty with an accuracy of over 90% when testing RoBERTa-lann with the Winogender Schema challenge test set.
To demonstrate that both above methods are reproducible, lightweight (dozens of lines of code), time-efficient (takes seconds), and plug-n-play compatible with almost any BERT-like LLM,, we provide open-source and running demos:
-Spurious Correlations: https: //hugg ing aace.ac.... cocee.coo. co/
Uncertainty: https: hugggi ng ace. co/ spaces / wapprapprrr5p (11//////ncrrtr Int
We generalize our approach to address a wider range of prediction tasks and provide results on generic DGP that are consistent with our empirically measured results on LLMs.
Although conflated, collider hias occur independent of selection bias and vice versa Hermán 2017).
2

---

Under review as a conference paper at ICLR 2023
3 PROBLEM SETTING: GENDER PRONOUN RESOLUTION TASKS
An example of underspecification can be found in the gender pronoun resolution task in Figure 2(b
Gender pronoun resolution will serve as case study for the rest of this paper, for largely two reasons: 1) it is well-studied problem with many recent advances (Cao and Daumé III 2020), (Lu et al., 2018), (Webster et al. 2020), (Zhao al. 2018) and yet remains a challenging problem for modemn LLMs (Deng et al. (022) and 2) it provides many human-relatabe relatable underspecified task
scenarios.
To apply causal infer- ence methods to gender pronoun resolution, we
2010: The payenologist quealionnire [MASK] be quicky assessed.
"In 2010: The paychologiat told the man questionaire (MASK] could make prelininary assessment"
must make some initial as- sumptions about plau- sible DGP for the rele- vant features and labels, that we can later support with empirical measure- ment. These assumptions have been transparently represented in Figure and Figure 2 Note the heterogenous nature of the DAG variables, in which X and Y represent high dimensional features like text in the dataset, while W, G, and S represent low dimensional symbolic entities that may cause the text.
(a) Well-specified (X + y): Masked (b) Underspecified (X + Y): Masked pronoun coreferent with psychologist.
pronoum coreferent with the man.
Figure 2: Gender pronoun resolution task with modified Winogender Schema (Rudinger et al. 2018) test sentences.
Specifically in Figure 2 the symbol X represents the text sentences in our dataset, and Y represents the label: a gender pronoun. The arrow pointing from X to Y encodes our assumption that X is more likely to cause Y, rather than vice versa. The remaining symbols in Figure 2 are not recorded in the dataset. The symbol G represents gender and in well-specified gender resolution tasks, G causes both X and Y. W represents gender-neutral entities that are not the cause of Y, but still of interest because they cause X. Additionally, in order to measure the effects of selection induced collider bias, we must find entities for W that are also the cause of S: selection into the dataset.
The W + + G relationship can represent any selection bias mechanism that has a gender depen- dency upon otherwise gender-neutral entities. For example, in data sources like Wikipedia written about people, is plausible that access (c.g. access to resources) has become increasingly less gen- der dependent sources like Reddit written by people, W + in the case of subreddits about gender-neutral hobbies, the style of the moderation and community may result in gender-disparate access to a given subreddit. In both cases, the disparity in access can result in preferential inclusion of samples into the dataset, on the biases of gender. In this paper, we use W to represent date and place, for our LLM measurements.
the date approaches more modern times, but evenly in every place. In data G can plausibly capture the scenario that even
Figure l(b) and Figure 2(b) are the underspecified counterparts to the well-specified prediction tasks in Figure (a) and Figure 2(a) As defined above, to achieve the underspecification of interest here, we must obscure any causal features from X. In the case of gender pronoun resolution, this is achieved by removing the path between G' and X. Further, because W is also gender-neutre once we have removed any gender-identifying features from X, we should additionally remove the path between X and Y, as there is no longer any feature in X causing Y.
With the causal path from X + Y removed, we would then expect X and Y to be unconditionally independent in the real world (RW). However, in the scenario depicted in Figure [[bb and Fig- ure 26b selection induced collider bias has opened the backdoor path, X + W W G + Y, Thus, the learned model (LM) is encouraged learn any spurious association along this path that reduces predictive risk, resulting in X and Y becoming unconditionally dependent.
Formally, selection induced collider bias causes the transformation: (Y 11. 11. X)R S (Y J. X)MM
due to the spurious path through S: X + W + S + G - Y. Whereas spurious associations learned in the well-specified model would more likely travel a path directly through confounding shortcut feature, such as a sandy or grassy background, was discussed previously in the camel vs cow example.
A single prediction task may be partitioned into well-specified and underspecified tub'tasks. For example, Figure 2 may he well-specified for part-of-speech tagging.
3

---

Under review as a conference paper at ICLR 2023
4 METHOD: SPURIOUSNESS DUE TO SELECTION INDUCED COLLIDER BIAS
In addition to transparently stating our assumptions, the causal DAGs in Figure I and Figure entail specific conditional probabilities that should be empirically measurable, under the condition of two additional assumptions: For a given causal graph, Gr, and a measured probability distribution Pr, we are assuming 1) the Markov assumption: (X (ILL(ZZ) = (X ILLI (2) Pr and 2) faithfulness: (XLLY(Z)) = (XLLY (Peters et al. 2017). where Z is the set (W, S.
To capture the statistical process of dataset formation, we implicitly condition on S=1 1 for all the samples in the dataset. Conditioning on S induces collider bias between G' and W in the form of S's structural equation (Pearl 2009): S f& (WW G, U,) (where U, is the exogenous noise of the variable), entailing a conditional dependency between otherwise non- non-interacting entities: G, C.
However, at inference- -imee we only have access to X, Y. Here we will show that for underspecified tasks, we would expect P(Y|X) IX) to be distributed similarly to P(G]W) Revisiting the underspec- ified DAGs in Figure ((b) and Figure 2(b and applying the Markov and faithfulness assumptions, we can estimate the conditional probability of a gender pronoun, Y, given gender-neutral text, X.
Equation T shows a mapping from the target unbiased quan- tity to the measured selection biased data, defined in (Bareinboim and Pearl 2012). Equation 2 assumes very high correlation between the textual form of gender in Y (as gen- der pronoun), with the symbolic variable for gender, G. Equa- tion replaces S with the variables in its structural equation,
P(Y[X) = S-V]X,S-1)
(1)
P(GIX,5-1)
(2)
~ P)
(3)
P(GIW)
(4)
S = (W. G. U.). which entails the conditional dependence P(G * P(G), and thus we add W behind G's conditioning bar. Finally, Equation (4) assumes we have an underspecified (gender-neutral) text, X, so P(G = P(G), and thus we remove X from behind the conditioning bar.
Equation Equation (4) show that P(Y ~ P(GIW) in underspecified tasks, providing information about the otherwise inaccessible latent representations for G and W, and specifying a measurable relationship between X and Y, that we can validate in the next section.
4.1 EXPERIMENTAL SETUP: MASKED GENDER TASK (MGT) CHALLENGE SET
To validate spurious associations between otherwise non-interacting entities, such as time and gender, desire a challenge test set of gender-neutral text. However, the text from the example in Figure [ does not satisfy this requirement due to associations between gender and occupation. We also desire a test set compatible with the Masked Language Modeling (MLM) objective used in the pretraining of L.L for the greatest applicabil- ity of the results. Table [] shows the heuristic used to create the MGT challenge set composed of (2 Python f-string templates) x (5 tenses of verb to-be) x (6 wllfe.sstages x (30 values for Was date) x (20 values for W as place) 36000 gender- neutral test sentences.
In Section 4.2 each plotted dot is the softmax probability (averaged over 60 nender-neutral texts) for predicted gender pronouns. These are plotted against year (or country), where the year (or country) along the x-axis matches the nender-neutral W value injected into the gender-neutral tex
As the final layer in the pre-trained L.MM is a softmax over the entire tokenizer's s vocabulary, we sum (without normalization) the sender-identifer portion (as listed in Table 3) of the probability mass from the top five predicted word
4.2 RESULTS: PRE-TRAINED LLMS GENDER- HENDER-DATE AND GENDER-PLACE DEPENDENCIES
Figure shows pre-trained BERT (Devlin al. 2018 and RoBERTa (Liu al. 2019), base and large, results following the experimental setup described above.
Tessts compatible with the MLM objective can avoid fine-tuning weights during measurement procedures. *For example, the purple and green dots the x-axis position of 1938 are the female and male pronoun softmax probabilities for the masked word in input texts like 'In 1938, [MASK] will became a teenager'. 'e pick the number & = 5 for the 'top_k' predicted words, because 5 is the default value for the 'top_k' argument in the Hugging Face "fillmask()" function used for inference.
4

---

Under review as a conference paper at ICLR 2023
Table 1: Heuristic for creating gender-neutral input texts for the MGT challenge set, and example rendered text. For verb we used past, present and future tenses of the verb to be: ['was', 'became", 'is', 'will be', "becomes'), J. and for life we used proper and colloquial terms for range of life stages: ['a child', 'a kid', 'an adolescent', "a teenager', "an adult', 'all grown up'] excluding life stages past adulthood due to age gender-inequities. For the text versions of w, we used range of 30 dutes and 20 places, defined in Appendix C
W Category
Python E-string, templates
Example text
[XASK] IIin (w), [MASK] (verb) (Lite.stage)."r
{verks) (life stage} fw)
[[MASK] was teenager. 1953.' "In Mali. [MASK] will be adult.'
Date Place
Pronoun vs Year
Predicted Pronoun Country Mai rratiiiol SEHI
Mole:
BERT base
Toll thee Coeficet box Pree
Aoerage sutteee partabal
Aud ropdpp sultr as probuliiie
Dabo Corfterr to Couttyy
Cender froeeie
logo
lage
sere lang
Protratoo Robonto large
Agg
27
233
(b) Difference, between the female and male pro- noun, linear fit slope and coefficients
(a) Spurious correlation between gen- der and time plotted averaged soft- max probabilities for predicted gender pronouns vs a range dates.
(c) Spurious correlation between gender and place list of countries, ordered by their Global Gender Gap rank (see Appendix C.T
Pearson's
for Figure 3(a) (top) and Figure 3(e) (hottom).
Figure 3: Softmax probabilities and linear fit coefficients for LLM predictions displaying spurious correlation between gender, G, and gender-neutral entities, W, injected into gender-neutral input texts described in T
The shaded regions in Figure 3(a) and Figure 3(c) show the 95% confidence interval for the linear fit, and Figure 3(b) shows the slope and Pearson's r correlation coefficient (following (Rudinger) jet al. 2018)) of the y-axis value against the index of the x-axis, for these plots. It is noteworthy that the magnitude of the spurious correlations appears similar across model size, but from this data alone, it is unclear the role of model overparameterization in the learning of spurious associations due selection induced collider bias.
Overall, in Figure 3 we can see that the injection of different W values into X results in a changed prediction for Y consistent with the P(G(W) conditional dependency entailed by collider bias at S. These spurious associations of gender vs time and place have not been reported before this work. Further, with the derivation that P(Y|X) ~ P(G)W) underspecified tasks, additional such spurious associations can be more easily identifiable and measurable using the inference-time method introduced here.
METHOD: EXPLOITING FOR UNCERTAINTY MEASUREMENT
In (Bareinboim al. 2014) it is proven that one can recover the unbiased conditional distribution 1), if and only if the selection
P(Y[X) from a causal DAG, Gs. with selection bias:
mechanism can become conditionally independent the effect, given the cause: (Y IL S|X)Gs- However, in Figure ((b) we can see (Y W S|X) ) trivially, due to the lack of a direct path between X and Y. Thus, the conditional distribution learned on an underspecified task with selection induced collider bias will not converge toward the unbiased distribution without additional data assump- tions (Bareinboim et al. 2014). However, in this section we will exploit the outcomes of selection bias our benefit, in method we call W-injection and describe below.
5

---

Under review as a conference paper at ICLR 2023
Table Extended Winogender evaluation sentences and uncertainty results for occupation as "Doc- tor'. Only sentence IDs 5 & 6 are well-specified for gender pronoun prediction, and appropriately all models show the lowest uncertainty for these sentences. For example, cutoff threshold of 1.0 for our uncertainty metric would correctly classify each sentence as well or underspecified.
Uncertainty
ID
Sentence with Participant and Coreferent Highlighted
BERT
RoBERTa
base
large
base
large
In DATE: The doctor told the man that [MASK] would be on vacation next week.
1.7
1.8
15.0
14.0
DATE:
The doctor
woman [MASK] would on vacation next week.
4.3
27.3
In DATE:
The doctor told someone that [MASK] would be vacation next week
10.6
8.0
13.3
20.2
In DATE: The doctor told the patient that [MASK] would be on vacation next week.
1.9
6.6
14.7
16.6
In DATE: The doctor told the man that [MASK] would
without vaccination.
0.0
0.0
In DATE: The doctor told the woman that [MASK] would he risk without the vaccination.
0.3
0.1
0.7
0.5
In DATE: The doctor told someone that [MASK] would be at risk without the vaccination.
11.3
10.5
41,3
16.4
DATE: The doctor told patient [MASK] would at without the vaccination.
6.1
12.3
5.1 EXPERIMENTAL SETUP: EXTENDING WINOGENDER TEXTS AND USING SPURIOUSNESS
We test W-injection using the Winogender Schema evaluation set (Rudinger et al. 2018 , composed
of 120 sentence templates, hand-written in the style of the Winograd Schemas (Levesque et al.,
2012), and already seen in Figure 2
The 'Sentence" column in Table 2 shows ex- ample texts from our extended version of the Winogender evaluation set, where the occupa- tion is 'doctor'. Each sentence the evalu- ation set contains: 1) professional, referred to by their profession, such as 'doctor' 2) context appropriate participant, referred by one of: { I'man', 'woman', 'someone', other) where other is replaced by context specific term like 'patient', and 3) single pronoun that is ei- ther coreferent with (1) the professional or (2)
Softmax Probsbility Famale Pronouns Cate Dacto Sentence
3) Partopent: "mant Contleent:: Jcttor
sochmue Probubiiity over Genderrdd Pratettora
21 Partdpant: 'wmann" Conetrent: Doctor
1) Partipant:: "somerre" Coeelerer: Dachor
1) Partripatt: "ther!! Coreleeet Tactor
3) Partcipett: "man' Contereet:: man'
3) Partdiant: 'wman' Coveferent: 'weman!
7) Partdpent: 'smmeoee" Coreferenc: 's0 meone'
31 Partcpantt "oher" Coreleiere litter
the participant in the sentence (Rudinger et al. 2018). As was the case the MGT challenge set, this pronoun is replaced with [MASK] for prediction.
1930
1940
1463
1902
7000
2020
Our extensions to the evaluation set are two- fold: 1) we add { ('man', "woman') to the list of words used to describe the particiiaat in order to add well-specified tasks to the existing Wino- gender set, which all underspecifted, and 2) we perform W-injection by prepending each sentence with the phrase 'In DATE where 'DATE' is replaced by range of years from 1901 to 2016" similar to what was done in Fig- ure 3(a)
Figure 4: Averaged softmax percentages from RoBERTa large for predicted female gender pro- nouns (normalized over all gendered predictions) vs a range of dates (injected into the text), for the 'Doctor' Winogender texts listed in Table 2
In Sentence IDs 4 of Table 2 the masked pronoun is coreferent with the professional, who is always referred to as the "doctor".. Whereas in Sentence IDs 5 - 8, the masked pronoun is coreferent with the participant, who referred as { ('man', 'woman', 'someone', and 'patient'), ). respectively. Thus, of the eight sentences, six are underspecified for the pronoun prediction task, with only IDs 5 & as well-specified. An uncertainty metric should only show low uncertainty for IDs
SSiiilar results can be obtained with the W-injection of countries, as was done in Figure We picked a slightly narrower and more modern date range compared to that of Figure 3 for semantic consistency with some of the more modern accupations in the Winogender evaluation
3(

---

Under review as a conference paper at ICLR 2023
RoBERTa large Uncertainty Pronoun Prediction for the Professional Winogender-like Texts
Uncertainty Motric
Participant idenlified as 'man'
Participant identified as 'woman'
Underspaciied
Participant identified as 'someone'
20
Participant identified 'other'
plumber
firelightrr
palmtrr
chel
archiltec
afficar
peramedio
janllr
lawver
chemist
advisor
physician
surgeon
dockor
worker
technical
specialis
scientist
son
appraiser
broker
dispalcher
pharmacis
sucftor
barfendor
reterination
Instructor
courselor
peychologist
educator
cashie
rrdiltion
Inerapis
planner
lirrrran
per peralogal
nurse
receptiorist eesser
nutriteriitt
dielllan
secreiary
hyglenist
pathologiis
programing
invest
(a) Masked pronoun is coreferent with the professional (similar to IDs 1-4 in Table are underspecified.
2
so all these sentences
RoBERTa large Uncertainty in Pronoun Prediction for the Parlicipant Winogender-lke Texts
60
Uncertainty Metric
Participant identified
Wallspacifed
High Unnertaennty
40
Participant identified 'woman'
Uncertsinty
Participant identilied as 'someons' Uncerspootted
20
Participant Identifled as 'other.
plumber
machamic
trefightte
painter
inspaate
mechinie
seginent
programmee
Chet
archite
richllec
office
paramend
innilal
lawyer
chemiet
advisor
physican
surgeod
doctor
wotk
technicion
specialis
spientin
investinns
salesper perseesoo
apprassed
adminittra intrataoo
broker
disattcer
sidior
barfender
baker
Instructor
deik counseior
psychologlal
edutator
cashier
rraditioner
therapes
planner
llorarian
per aleggll
receptionial
dielilian
Tygienist pallfologist
super
veler
heir
(b) Masked pronoun coreferent with the participant (similar to IDs 5-8 in Table 'man' and 'woman' are well-specified, and the rest are underspecified.
2 thus sentences containing
Figure 5: RoBERTa-large uncertainty metric on all Winogender Schema occupations. Setting threshold at 1.0 for the metric. produces greater than 90% outcomes for both the arue-positive clas- sification of the underspecified tasks and true-negative classification of well-specified tasks.
5.2 RESULTS AND DESIGN OF SIMPLE UNCERTAINTY METRIC
Figure shows the predicted softmax probability for female pronouns for the masked words in the Table 2 sentences, normalized the gendered predictions of the top five predicted words from pre-trained RoBERTa large. Similar to the findings in (Rudinger et al. 2018). at any given W value, the softmax probabilities for female pronouns are higher for masked pronouns coreferent with the patient opposed to the doctor, indicating the now well-known learned spurious association between gender vs occupation.
Our contribution here is that we see no gender-time spurious associations in the well-specified sen- tence IDs 5 & 6. Further, in Figure we can see that the spurious associations due to the W-injection of unrelated spurious association (time vs gender) appears additive with the existing spurious sociation between occupation and gender.
As simple and naive single-value uncertainty metric, we can measure the absolute difference be- tween the averaged softmax probabilities for the first and last several dates along the x-axis in Fig- ure For this uncertainty metric, we would expect larger values for underspecified prediction tasks, in which W-injection has a larger influence on the prediction. For the predictions in Figure 4 this metric is shown in the "Uncertainty" columns in Table for all four LLMSs studied in this paper. Here we see uncertainty values closest to 0 for well-specified sentence IDs 5 & 6, with cutoff threshold of 1.0 producing 100% outcomes for both the true-positive classification (the model should be uncer- tain) of the underspecified tasks, and true-icgative classification (the model should not be uncertain) of well- rell-specified tasks, for the sentences in Table 2]
Our extended version of the Winogender Schema contains (60 professional occupations ) x (4 partic- ipant texts) ) (30 values for DATE ) (2 sentence templates J. This totals 14,400 test sentences, which we provide as input text to the 4 pre-trained models thus far investigated in this paper: BERT base and large, and RRBE base and large.
One template with the masked pronoun coreferent with the professional and the other with the participant.
7

---

Under review as a conference paper at ICLR 2023
We calculate the above-described uncertainty metric for all 60 occupations in the Winogender eval- uation set and show the results from RoBERTa large in 1) Figure 5(a) with input sentences like IDs 1 4 where the masked pronoun is coreferent with the professional, and 2) Figure [5b] with input sentences like IDs 5 8 where the masked pronoun is coreferent with the participant. In these plots the x-axis is ordered from lower higher female representation, according to Bureau of Labor Statistics 2015/16 statistics provided by (Rudinger et al. 2018), and the y-axis is the uncertainty metric defined in the proceeding paragraphs.
Similar to the 'doctor' example, in Figure 5 we again see high uncertainty for all six of the un- derspecified tasks, and low uncertainty for the two well-specified tasks, for almost all Winogender occupations. For each occupation, we should and largely do see high uncertainty for all four partici- pants (including sentences containing 'man' and 'woman") ') in Figure 5(a) and only high uncertainty for two participants (excluding sentences contain 'man' and 'woman' in Figure 5(b)
Concretely, even if we focus exclusively on the most challenging Winogender sentence-typer -tppes in which gender-identifingg text: 'man" or 'woman' is co-occurring but not coreferent with the masked-out pronoun, with a thresholding value of 1.0 we find the uncertainty metric has true- positive classification rate of 90.0% for the underspecified sentences, and true-negative classifica- tion rate of 91.7% for the well-specified sentences.
Overall, see this simple metric can provide accurate task uncertainty classification for unmodified pre-trained LLMs, with only several additional inference runs. is noteworthy that the metric underperforms roles most traditionally male, e.g. plumber to electrician, perhaps due to the strong gender-occupation association overpowering our weaker gender-time association. Finally, show similar plots for BERT and RoBERTa base and BERT large in Appendix D but note the uncertainty metric appears more accurate for more overparameterized models.
6 EXTENDING TO MORE GENERAL SETTING
We now explore more general problem space where the symbols in Figure take the follow- ing meanings: G is the causal parent of Y, and W is the non-causal parent of Y, yet nonetheless included because W is a cause of both X and S, where S has the same meaning as before. We can thus partition any feature space into G. and candidates for W. A candidate can be validated as a suitable W feature by checking for the conditional dependencies which we plot below. To make this hypothetical example slightly more concrete, we parameterize the DAGs in Figure [ with the structural causal model (SCM) shown on the right.
Equation 5 and Equation (6 define W and G as independent exogenous O-mean Gaussian noise, for which we set = 10 so that we can more easily trace the amplified noise through the DACd Equation ? defines S as an un- weighted combination
G = a 10,
(5)
W = a 1)(
(6)
2
S = (W- GG+N(O,1))) > 2a
(7)
W, G and exogenous noise, with the selection mechanism setting all values above 2a to 1, and to otherwise, reduc-
X : W +10..1
(8)
Y = +
ing the dataset about 5% of its original size. Equation ( and Equation (9) we set y to 0 for the underspecified task and to 1 for the well-specified
task, consistent with a path weight for the grayed out arrows G + X and X + Y in Figure [(b)
and full path weight for those same arrows in Figure ((a)
Figure plots the statistical relationships entailed by the SCM above. Columns (i) plots X vs Y for the unsampled, and columns (ii) plots X vs Y for the S 1 sampled distributions. In both Figure and Figure 6(b) we can see that selection induced collider bias has little effect on the distributions (i) vs (ii) for the well-specified SCM, but causes the underspecified SCM's Pearson's 's r coefficient 0.7. The latter is consistent with Section 4.2 in which entities such as gender and date (as well as gender and place) that are uncorrelated in the unsampled population, become spuriously correlated in LLMs trained on subsampled populations, where we claim the subsampling caused selection induced collider bias.
to go from about
'We set different noise weights G and W by arbitrarily dividing by 2 in Equation 6 to reduce the likelihond of unintentionally constructing an unfaithful graph.
8

---

Under review as a conference paper at ICLR 2023
(00
(1)
%
(a) (W + (0, 1)) > 2a sampled.
(b) (W + GG + -V 1)) > 2x sampled.
Figure 6: Statistical relationships induced by the SCM defined in Equation 5 to Equation ( with Equation ( separately defined above, with the well-specified (top row blue) and underspecified (bottom row in green) DGP in Figure]
Columns (iii) and (iv) in Figure 6 plot the unsampled and the S 1 sampled distributions for W vs G. Comparing Figure [((]] and Figure [6b]]
TRANO
see the direction of the correlation coefficient has been flipped for both the well and underspecified SCMs. However, the direction of the correlation coefficient flips in the X vs Y distributions of column (ii) only for the underspecified SCM. This is consistent with our derivation of P(YIX) ~ P(GIW) for underspecified tasks in Section
Sumpee Deteree K
In Figure 7 we replace Equation with X == 3W GG + +O,, 1). where 3 takes increasingly larger values: 1) 8 0.01 2) B 0.1 and 3) 3 = (thus identical to Equation (8). ). as a toy demonstration of W-injection. Similar the uncertainty results in Section 5 we see underspecified tasks are more sensitive to W-injection, while well-specified are not.
A 0.01
p-1
Figure 7: W-injection implemented via increasingly larger values of B as the W X path weight.
7 DISCUSSION AND REPRODUCIBILITY
As underscored in DD D'Ammuur et al. 2020), the prevalence of underspecified tasks in machine learn- ing requires the development of nuanced stress tests. this work we strived to make our methods accessible and extensible other tasks in machine learning. Please see more details in Appendix
We have argued that underspecified prediction tasks leave models vulnerable to selection induced collider bias which can result in the learning of otherwise non-interacting spurious associations, such as previously unreported associations between gender vs time and gender vs place, which we demonstrate on unmodified pre-trained BERT-like LLMs. We have introduced a lightweight inference-time technique for injecting spurious signals into prediction tasks to determine if the task well-specified or underspecified, and demonstrated this in the form of an uncertainty metric on an established evaluation set. While not a universal solution, we hope our work can lead to the development of targeted heuristies, using our method to determine when the model is uncertain about prediction task, and thus an alternate value should be returned (for example 'they' in the case of gender pronoun resolution).
For further research, we believe our work complements that of (Vig al., 2020), which uses causal mediation analysis to intervene on LLMs at the individual attention head and neuron level to provide insights into the model's internal causal mechanisms mediating gender bias. Despite the more lim- ited nature of our non-invasive investigation, our surprising empirical findings about the additivity of unrelated spurious associations is consistent with their surprising finding that gender bias appears decomposable between the elements of the direct and indirect effect within the model. We hope to be able to complement more of their internal model findings via external methods.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Guirajani. and David Lopez-Paz. Invariant risk minimization, 2019.
arxiv. org/ abs/////////// / /1907990000 02893
Elias Bareinboim and Judea Pearl. Controlling selection bias in causal inference. In Neil D.
Lawrence and Mark Girolami, editors, Proceedings of the Fifteenth International Conference Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Re- search, pages 100-108, La Palma, Canary Islands. 21-23 Apr 2012. PMLR. URL https:
//proceedings mlr prwss/eeeeeeeeee wttss///////aaeeeeeeeeeee
.html
Elias Bareinboim and Judea Pearl. Causal inference and the data-fusion problem. Proceedings of the National Academy of Sciences, 113(2))7345-73552 2016. doi: 10. 1tt.3//0s...110071111
URL https:/ / www. pnass http://s.....t///a///////....... 107./////00003//aaaaa. 1510507113
Elias Bareinboim and Jin Tian. Recovering causal effects from selection bias. Proceedings of the
AAAI Conference Artificial Intelligence, 29(1), Mar. 2015. doi: 10. http://aaii.iii....... 22911.9679, URL
Elias Bareinboim, Jin Tian, and Judea Pearl. Recovering from selection bias in causal and statistical
inference. Proceedings of the AAAI Conference on Artificial Intelligence, 28(1), Jun. 2014. URL
https://ojs aaai. org/ 1e../inen index. wwwp..ppppe........... pttp://AAA/atttilo///vvle///////
Yang Trista Cao and Hal Daumé III. Toward gender-inclusive coreference resoluttto In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4568-4595, Online. July 2020. Association for Computational Linguisties. doi: 10. 10.18653//1/2020.aac main. 418. URL htttts::/aalaatttoottt http:/oooyorr//220..0
418
Stephen R Cole, Robert W Platt, Enrique Schisterman, Haitao Chu, Daniel Westreich, David Richardson, and Charles Poole. Illustrating bias due conditioning collider. International Journal of Epidemiology, 39())447-229 11 2009. ISSN 0300-5771. doi: 10. 00.10934je/dyy3344.
URL
2yy333
Alexander D' Amour, Katherine A. Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beu- tel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D. Hoffman, Farhad Hormoz- diari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingam, Mario Lucic, Yi-An Ma, Cory Y. McLean, Diana Mineu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natarajan, Christopher Nielson, Thomas F. Osborne, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin Senevirane, Shannon Sequeira, Harini Suresh, Victor Veitch, Max Vla- dymyrov, Xuezhi Wang, Kellie Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, and D. Sculley. Underspecification presents challenges for credibility in modern machine learning. wtggabs////oooooooooo//goo//..br abs 2011 03395
CoRR, abs/2011.03399 2020. http::t::://// //axxii..////x.....//./iv org/
Yuntian Deng, Volodymyr Kuleshov, and Alexander M. Rush. Model criticism for long-form text generation, 2022. /ttLttttt:///aattt/ http://wwwwvr..g...a..oooo abs abs//210.....a...p..oppaato 2111221111.8444.4
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep
bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL
http:://arxiv.oz. http:/bbs//1100/ 04805
Gareth Griffith, Tim T. Morris, Matthew J. Tudball, Annie Herbert, Giulia Mancano, Lind- sey Pike, Gemma C. Sharp, Jonathan Sterne, Tom M. Palmer, George Davey Smith, Kate Tilling, Luisa Zuccolo, Neil M. Davies, and Gibran Hemani. Collider bias undermines our understanding of COVID-19 disease risk and severity. Nature Communications, 11(1), November 2020. doi: 00.1038/444144--20---44444 ss1467-020-19478-
URL https: htttts://acc..og/11...
James J. Heckman. Sample selection bias as a specification error. Econometrica, 47(1):13-1611 1979. ISSN 00129682, 14680262. URL http: wttp::/wssooooooo/...oooie
Miguel A. Hernán. Invited Commentary: Selection Bias Without Colliders. American Journal of Epidemiology, 185( 1050, 05 2017. ISSN 0002-9204 22229262. doi: 10. 10.10977/j///wx0077 URL
https://do1 www.ooi.ogg/10.0 000///00..101/.80 kwx077
10

---

Under review as a conference paper at ICLR 2023
Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Pro- ceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR 12, page 552-561. AAAI Press, 2012. ISBN 9781577355601.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.1 1692, 2019. URL htt.ttttt/a//....tt/ee
907 11692)
Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. Gender bias in org/
neural natural language processing, 2018. httt:htts//aaaeee wtt.v/orggbbbb///..oooo
Marcus R Munafö, Kate Tilling, Amy E Taylor, David M Evans, and George Davey Smith. Collider scope: when selection bias can substantially influence observed associations. Int. Epidemiol., 47(1):226-235, February 2018.
Judea Pearl. Causality. Cambridge University Press, Cambridge, UK, edition, 2009. ISBN 978- 0-521-89560-6. doi: 10. 10.101//B09978011 1803161.
Jonas Peters, Dominik Janzing, and Bernhard Schlkopf. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, 2017. ISBN 0262037319.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. CoRR, abs/1804.09331, 2018. URL http:://axiv..rrr
1804.09300
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang. Yaron Singer, and Stuart Shieber. Causal mediation analysis for interpreting neural nlp: The case of gender bias, 2020.
abbs////////// /2004. 12265
Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, Ed Chi, and Slav Petrov. Measuring and reducing gendered correlations in pre-trained models, 2020. URL httpptt::///rxttt wrw.wvv.rrrrreoo. org/ /2000. 06032
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wer Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Short Papers), pages 15-20, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/1//118-2003. URL https://aclaathorr antholothhoooy...oomooooo www/oogg//-1// aanthelooy..rg//118 2003
logy.
A REPRODUCIBILITY
Please see the following sources to reproduce the methods and measurements in this paper:
Spurious Correlations Open Source Hugging Face Space (screen shot at Figure 10
httpss:/hueqinggaan wtt.co//ppaaae ppo/spacss/// aapaces/paperstii httppppppp11111./ / iour
Uncertainty Measurement Open Source Hugging Face Space (screen shot Figure
https: /ttee////hgggnnng ctticee.cccaaee
More General Setting Toy SCM htttps://tinypuu htte:/yyrr..../////aae/eem com/2ub4xy19
Github repo to replicate all the plots in this paper:
https: http:://gitttb..ot/
anon-anon- anwny/sicssss..
B GENDER-IDENTIFYNN WORDS
See Table 3 for the list of gender-identifying words that would contribute to total softmax proba- bilities masses accumulated for female and male genders for given prediction in Figure For example, if an LLM predicted 'her' in addition to 'she', we would sum their two softmax probabili- ties together for the final total softmax prohability assigned to 'female..
11

---

Under review as a conference paper at ICLR 2023
Table A list of explicitly gendered words described in
BIS
HER
HIMSELS
HERSELI
MEN
WOMEN
HUSBAND
WIFE
BOYFRIEND
GIRLFRIEND
BROTHER
SESTER
AUTOO
ACTIUSS
C VARIABLE X-AXIS VALUES
For {w} require list of values that are gender-neutral in the real world, yet due selection induced collider bias are hypothesized to be a spectrum of gender-inequitable values in the dataset. time women have become more likely to be recorded
For W as date, we just time itself, into historical documents reflected in Wikipedia, so we pick years ranging from 1801 2001. For W as place, use the bottom and top 10 World Economic Forum Global Gender Gap ranked countries (see details in Appendix C.I.
C.I PLACE VALUES
Ordered list of bottom 10 and top World Economic Forum Global Gender Gap ranked countries used for the x-axis in Figure Bic) that were taken directly without modification from https:
7/www3, worummm org/ docs WEF GGGR 2021.pdf
'Afghanistan', 'Yemen', 'Iraq', 'Pakistan', 'Syria', 'Democratic Republic of Congo', 'Iran', 'Mali', 'Chad', 'Saudi Arabia', 'Switzerland', 'Ireland', 'Lithuania', 'Rwanda', 'Namibia', 'Sweden', 'New Zealand', 'Norway', 'Finland', 'Iceland'
D EXTENDED WINOGENDER UNCERTAINTY RESULTS ON MORE LLMS
Figure g shows uncertainty results for all Winogender occupations where the masked pronoun is coreferent with the professional. Because the injected text (one of: ('man', 'woman', 'somcone', 'other') is referring to the participant and not the professional, all these sentences underspec- ified. The plots show that like RoBERTa large in Figure ((a) RoBERTa base tends to report un- certainty results above for most occupations, regardless of the word injected into the evaluation text for the participant, thus the model does not become erroneously certain about gender when the words 'man' and 'woman' are injected into the text. However, note that it is difficult to such a trend in BERT base and large.
Figure shows uncertainty results for all Winogender occupations where the masked pronoun is coreferent with the participant, unlike Figure 8 where the pronoun is coreferent with the profes- sional. Because the injected text (again one of: "man', 'woman', 'someone', 'other') } is referring to the participant, the sentences containing 'man' and 'woman' are well well-specified, while the rest are underspecified. We see uncertainty results closer to 0 for most occupations when 'man" or 'woman' has been injected into the evaluation text for the parlicipani, and generally above 0 otherwise, in particular for more highly overparameterized models like BERT large and RoBERTa base & large in Figure 5(b)
It is more difficult to this trend in BERT base
E MODEL UNCERTAINTY DEMO
See Figure [I for open-source freely available demonstration where users choose their own input text and select almost any BERT-like model hosted on Hugging Face to test for model uncertainty using selection induced collider bias induced spurious correlations.
12

---

Under review as a conference paper at ICLR 2023
BERT base Uncertainty Pronoun Prediction for the Professional Winogender-like Texts
60
Uncortainty Motric
licipard idenlified
Participant identified 'woman'
Participant identified as 'someone'
Participant identified as 'other'
plumber
glodetc
frreiihte
pai painter
inspectoor
mechinet
englieed
programmel
Chel
archilted
cificol
paramidd
laseyo
chemnied
shyekcon
singee
Bocton
workal
suserviis auserviis
bechiniion len
specialist
scientist
scleesee
agorratenn
adminitiraton
dispalcher
pharmsdds
barfender
zoursetoo
beychologist
mhrrapise
plannee
librarian
pars agal
receppiionall
nutiitiiiis
dielllan
hyg
patholage
(a) BERT basc
BERT large Uncertainty Pronoun Prediction for the Professional Winogender-llke Texts
60
Uncortainty Motric
licipan
Participant identified 'oman'
Participant identified as 'someone'
20
Participant identified as lother!
plumber
mechamc
fireightt
nointee
respectoo
machhnet
snglieee
programmes
Chill
archiltsc
cificor
paramed
jenitor
lswayr
cheiiel
advigor
phycican
singeen
Bocdal
workel
maragee
superviso auservisor
specialisk
scientist
selleseer
opporatonn
adminietraton
brokee
dis despatcher
barfender
zoursetoo
peychologist
oduraaor
cashier
morapiist
llbrarian
agal
nurse
receppiianial
dielilian
Iygeenta sentel
(b) BERT large
RoBERTa base Uncertainly in Pronoun Prediction for the Professional Winogender-ikke Texts
60
Uncortainty Motric
licipaann
Participant identified 'aman'
Participant identified as 'someone'
20
Participant Identified as lother"
plumber
elocttoc
firel frreighte ghier
pai painter
irepertor
rischinet
englinee)
programing mrogammeer
Chi
archilesc
cificoal
paramed
jamifor
lawayr
cheimitt
adveco
phyclcee
sligeen
Bocton
watker
marsnee
superviss auserviior
led peddini lun
specialiil
scientist
apporatenn
#dminitratoo
broker
di paptcche
barferder
Instnctoor
zaursetoo
edurator
cashier
horrspiit
Henner
lbrrtiann
receppianial
esser
dielllian
Iygeenis gentel
pathologgit
investt
pars
(c) RoBERTa base
Figure 8: Uncertainty results for all Winogender occupations where the masked pronoun coref- erent with the gender-unidentified professional, thus all sentences underspecified. These plots show that like RoBERTa large in Figure 5(a) RoBERTa base tends to report uncertainty results above 0 for most occupations, regardless of the word injected into the evaluation text for the par- ticipant, thus the model does not become erroneously certain about gender when the words 'man' and 'woman' are injected into the text. However, note that it more difficult see such a trend in BERT base and large.
13

---

Under review as a conference paper at ICLR 2023
BERT base Unceriainly in Pronoun Prediction for the Participanl in Winogender like Texts
60
Uinceriiinty Metric
Par soipent ideniifiec as 'man'
Pariicipatt identfiie 'woman'
Pariccppatt identifee as 'someone'
20
Maracipant idonchec 'other
machh mett
chell
ponket
botlorr
dispatchee
pharmanee
accountorn
velatiin intetinnl lan
bakoe
indiructon
esamlind cours olee derk
peych peych holchhlogee
educaats
cashier
Pramitoon
thorapie
plrreer
fétratann
peralegee
natse
halrdredsel
Ruitioonts cann
pelioologee!
programme
admi Mest
pas
(a) BERT base
BERT Unceriainly Pronoun Prediction for the Participant Winegender-Bke Texts
Uncertainty Metric
Participant lnentified as 'man'
40
Participant icenáiicdd
Perticipent loentified as 'someone'
20
Participant icentfied as 'ather'
plumton
Imspectorr
machini
programmna
chdl
archtlett
difics Tim
physttin
cocter
wotker
man negit
technkkan
irvedlgstor
salesperson
adminicarate edminicarator
brokes
Sepaliche
phit Piarmocto fel
desour foott
peleiii relestratian lan
baker
Inderiitto
asammiie
COU Secatc
clerk
pagra pegrshokong hodiol
edticaator
teachee
therappll
planner
llbratiinn
paralegal
rurse
recoppiantt
dielltinn
progra
para
(b) BERT large
Uncerlainly
Participant in Winegender-iike Texts
Uncertainly Metric
Participant loentifled as 'man'
40
Participant iscntificd as 'woman'
Porlicipent
Parlicipant icentfidd as 'ather'
plumbre
mochante
imepection
programima progranmma
Chall
archilect
office
para medi
teaeye
dhemme
advesoor
plycition
cooter
wotker
nas sagen
sptertss
livestigloo
calceperson
adminisa initraatrrr
brokerr
dipatiche her
phiil mico lot
pelleri interiiiattn
baker Instructor
escim lnneeer
Cou roalor
derk
educator
anciitionns
therppplet mpppet
plarmer
librrian
paralegel
nurso
resossiontst
dilliiann
parhelogie
(c) RoBERTa base
Figure 9: Uncertainty results for all Winogender occupations where the masked pronoun is corefer- ent with the participant, thus the sentences containing 'man' and 'woman' are well-specified, while the rest are underspecified. We see uncertainty results closer to for most occupations when 'man' or 'woman' has been injected into the evaluation text for the participant, and generally above otherwise, in particular for more highly overparameterized models like BERT large and RoBERTa base & large Figure 5(b) is more difficult see this trend BERT base.
14

---

Under review as a conference paper at ICLR 2023
Clicklr
Clickir
Clicker
Adde
diz
vida rouard
top 20 Gateel Gonder Gea wheld
Subrediit
vell sell-deet fomals ridautto loe bhudee
mocil
Uiee boy other coumple, cur taki whii load model1)
Input fields
Pick spertuen renver sxported ark anc on
Go separmad vakor reci ar
clabalc vdee por ainatarmee all saors, les peound, togpaec eegline cvenaosh go si-villl future auruleg casss wilgs gring cataddies ta aecicludo sdutions in won Jaine atackeigg deog hutter jul palaman inteeriinp phrc
scincs, shelem, movis off, time Santiorm pakemangs muss Showerticgte deSchealCan englann feo tndayber parggenne
select aadd-aa-d-aatt auttaeed then moree of Hlughng model sapports fltemads turtonn right yoire uray timete bodd)
Ciiywe
medieeeddi then
Roseour roed
retectir
retert-legs
Fick vent to the pred reatee pandered onmmantyy
Alis will sho denorwoor spectal sakon you sulli oe ing and dharyes scua ao reglaned speenors of yoryea Booetbbeee
Outputsi
to gow.eeeeeeegeeeooooooooooooo
Semple meddl
hobet Bry prodect oy lenale poream.
prodeting poreers.
Figure 10: Demo where users can choose their input text and select almost any BERT-like model hosted on Hugging Face to test for selection induced collider bias induced spurious correlations. https://huggingiee
spaces s papprr18p. spwces/paper51ee
wwt.5////////isss// spurious!
15

---

Under review as a conference paper at ICLR 2023
youselectad
tr hggiggge oxtmacennn pnracenpppeeiii pig.
robertt lang
longe uncosed
roberta base
bett base uncised
adl-aerock
Recosoorpptiiotee Wingeendee
enchtect
baker
batander
broker
coaleer
englneer
Seeighter
Pagerie
Iopeddrr
Inttuctr
Inendigttar
jartor
Rayer
Ebestan
mactible
more
mechani
rura
nuntionks
ofter
paieer
parileaal
paramedt
pertologli
piveraccee
plyiddnn
plamber
prechisser
psychologle
moxplioa
acleddi
speciaie
sspervitor
eaggoo
piccyuuronn
selected biddyyur-nnn abous, your own tads nan- dellmtted sentances here. aue include aulinga Milkied pranoun. suss required format, sikck occuzation abovs bnoked, see sern eann tost for this
isput taxts
Nunit
evilaatio numbee a nomppands namberr andic like
24 the curtomer salved ve with the menager beceuse MMS could be othle to the img error.
21 Someo ne had to spea with the marager recause IUSK would be able Be tho Bilng avor.
24 man uslad speak with the manage because MASK would ba alse to Ei Biing erno.
en The womann asked to coeck with the monogeer because brAk wasll ald abrl to the billling moaa.
fo cuntanee axtied speat with the manaage bee W sar io di tilling never quilliyy
surooo to speuk with thereonager secauso HASK vorsec fa the bi ling quickt..
the asked speck with the muregee tecans tonteclu to is the ta
quickly.
The weman apeak with the manager because MAck wa thethe be quiddty.
unterttaattyynnn results!
by
top
gendered.
large gender pronoun uncertainty 'manager' sentences
16
20
Serdince number
Demo where users can choose their own input text and select almost any BERT-like model hosted on Hugging Face to test for model uncertainty using selection induced collider bias induced spurious correlations.
Figure 11:
16


Reviews:
[{"review_id": "s0OnrS1EbX", "reviewer": "Reviewer_ukSe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper \"Selection Collider Bias in Large Language Models\" explores a type of collider bias in which the shared effect variable is the selection process by which training data is generated. They show that the effect of this selection process can be highly pronounced in cases of textual ambiguity, leading to variables that are causally disconnected in the real word (e.g. gender vs year) exhibiting strong predicted correlations. They use this spurious correlation to define a novel uncertainty metric that can help indicate when a model is conditioning on non-causal variables. They also demonstrate the validity of their model formulation on a more general toy example. The authors also open-sourced their code with an easy-to-use UI so that others can easily experiment with it.", "summary_of_the_review": "Overall the paper is clear, well-written, and well-organized. However the primary contribution of the paper is a novel mathematical formulation of the already well-established phenomenon that if a model cannot predict a label by conditioning on causally-related words, then the model will try to predict the label using whatever non-causal word correlations are present, and these may be sampling artifacts. The authors' new proposed formalism for describing this phenomenon does not lead to a new method for diagnosing it a priori, nor does it lead to a new approach to alleviating it. Therefore this reviewer does not feel that this paper meets the bar for acceptance.", "strength_and_weaknesses": "Strengths:\n\n- The paper is well-organized and clearly written.\n\n- The paper combines techniques from the usually-disjoint domains of causal modeling and LLMs.\n\n- The paper takes pains to motivate the approach throughout its first several sections, both conceptually and mathematically.\n\nWeaknesses:\n\n- Referring to S as \"access\" is confusing, as this implies that it is an indicator of whether a given gender had access to the process that produced the dataset. However it's possible that a gender did have access, and yet the data was nonetheless highly gender biased, for example a corpus of harlequin novels written by female authors. It would therefore makes more sense to call this variable something like \"representation\", as what it actually expresses is the presence or absence of certain conditionally gendered data in the dataset.\n\n- Need more information about how the dataset used to perform the fine-tuning in Figure 2 was constructed. Was there a heuristic used to identify sentences in an existing corpus that have a particular gender-ambiguous structure? Or was the heuristic used to construct the sentences directly? If the latter, then how were the \"true\" labels known? All of this information should be described in more detail in Section 3.1.\n\n- The selection criteria of $(W ± G + N(0,1)) > 2\\alpha$ used as a cutoff in the toy example of Section 8 seems like it would be throwing away a considerable majority of the W, G values. The authors should state the fraction of sampled values that satisfy the selection cutoff, preferably with an associated figure.\n\n- The selection collider bias effect is presented as a general method, but (aside from the toy example) is presented solely in the context of gender biasing, to the point that one of the nodes is explicitly stated to be gender (G). The benefits of this approach would be clearer is it were demonstrated for other types of biases, for example bad weather being textually correlated with negative outcomes of all kinds, for causally unrelated reasons. For example \"it was a [dark and stormy/sunny and bright] morning when the S&P 500 [crashed/rallied]\".\n\n- Part of the stated contribution of the paper is the introduction of a new technique for modeling uncertainty, however this measure is not investigated in much detail. Some natural additional steps would have been to (1) compute its correlation with a measure of intra-profession gender representation over time, (2) compare it to the standard deviation over softmax probabilities rather than taking their most extreme differences, (3) use it to identify specific scenarios in which a human observer might think that the LLM is conditioning on causally meaningful information, but really is conditioning on acausal correlations.\n\n- Although the paper centers around a novel data biasing mechanism, it does not make any attempt to describe how one might go about alleviating this biasing, or detecting it a priori. (The proposed uncertainly measure is only useful if you already know where to look.)\n\nTypos:\n\n- Bottom of page 3: \"we could the use\"\n\n- Top of page 5: \"supports our supports our\"\n\n- Top of page 9: \"only holds underspecified models\"\n\n- Middle of page 11: opening-quote marks in list of countries are backwards\n\n- Figure 3: Pearson's r should be lower-case in the legend", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Yq3npUcgi1", "reviewer": "Reviewer_LK4n", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores the spurious correlation problem, a common problem in current LMs that learns dependency relations between two unconditional entities. The authors take \"date\", \"place\", and \"gender\" as examples and analyze the spurious correlation between gender and date (or place). The authors also provide an uncertainty metric to evaluate and identify spurious correlations. ", "summary_of_the_review": "The paper focuses on an interesting problem that explores spurious correlation from a causal mechanism perspective. However, due to missing necessary explanations, it is hard for me to understand the details of the paper. ", "strength_and_weaknesses": "Strength:\n\nIt is interesting to see how LMs learn spurious bias from causal mechanisms.  \n\n\n\n\n\nWeaknesses:\n\n[1]  There are important concepts lacking illustrative explanations. It makes the whole paper hard to read. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BC_BOMoY_E8", "reviewer": "Reviewer_qdVm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on what it refers to as \"selection collider bias\", using a causal inference framework to discuss spurious correlations in language data. The paper seems largely focused on formalizing notions of selection bias and spurious correlations through the language of causal inference, and presenting reasoning about factors involved in these phenomena. Concretely the paper offers a couple of experiments: the first involves fine-tuning LMs by \"masking common gender-identifying words for prediction\", and showing that both these fine-tuned models and normal pre-trained LMs show an interaction between gender pronoun preference and mentions of years or of countries. The second experiment involves augmenting the WinoGender dataset with prefixes mentioning a date, as well as explicit mentions of gender for one of the antecedent entities, and showing that there are correlations between the date and gender preferences for the pronoun, but only when the pronoun's antecedent is not explicitly specified. The third experiment runs a toy simulation to further demonstrate the relationship between data, label, and causal variables depending on underspecification and sampling. ", "summary_of_the_review": "The paper focuses on an important overall problem, uses an interesting theoretical framing, and has at least one seemingly interesting empirical finding. However, it does not attempt to contextualize with respect to related literature, does not provide a clear statement (or demonstration) of what contribution it is making, and has additional issues with methodological and theoretical clarity.", "strength_and_weaknesses": "Strengths: The paper involves the important topic of spurious correlations, and it frames that topic within an interesting causal inference framework. The paper identifies spurious correlations between model gender preference and gender-irrelevant cues like date and country.\n\nWeaknesses: The paper makes no effort to contextualize its contribution with respect to related work. The paper cites a bit of work while explaining and motivating the relevant concepts and formalisms that it uses, but there is no Related Work section to lay out how the paper's contribution relates to and is novel with respect to all of the other literature in this area.\n\nIt is not only the contextualization of the contribution that the paper doesn't make clear -- the paper doesn't really make a clear argument for what contribution it is making in the first place. The first symptom of this is that the introduction spends a lot of time defining \"selection collider bias\" and briefly arguing for its relevance, but makes no mention at all of what contribution the paper plans to make with respect to this selection collider bias concept. The paper then proceeds to present a number of theoretical discussions and empirical demonstrations, so in theory there could be good contributions that should simply be laid out more explicitly (in the introduction and elsewhere) -- but at the moment I don't think the paper is making any clear statement for what exactly it's trying to accomplish, and as I elaborate in the next paragraph, this contribution isn't clear from the content of the experiments/theoretical discussions either.\n\nExamining the experiments themselves to extract a potential contribution, it seems to me that the most concrete finding is that pre-trained LMs' preference for different pronoun genders is influenced by irrelevant cues like year and country mentions, but more so/exclusively so when actual relevant gender cues are not available/sufficient to inform the prediction. This characterization seems reasonably clear from the second experiment -- but the first experiment is harder to interpret, in large part because the specific details of the data and task used for fine-tuning are not actually provided, so it is difficult to know exactly what to make of the stronger correlations for the fine-tuned models. But the fact that the original LMs also show some spurious correction between year/country and gender preference is somewhat interesting. However, there are numerous issues surrounding this potentially interesting finding: 1) it's not clear whether it's a novel finding, since there is no comparison to related literature, 2) the paper does not provide any particular motivation for having chosen date and country as possible spurious correlating variables with gender, apart from speculating that they could show spurious correlations (wrt this, see also below), and 3) the paper puts no particular focus on this finding and makes no argument about its significance.\n\nIt seems likely that the authors want to focus more on a theoretical contribution that emerges from conceptualizing spurious correlations within the causal framework that they use. While I'm happy to be convinced otherwise on this, I'm also just not seeing any clear, novel insight that emerges from the theoretical framing or reasoning here -- mainly it seems just to be restating and reframing the existence of spurious correlations. Additionally, some of the theoretical framing strikes me as unclear and potentially problematic: the specific example I have in mind is the conceptualization of variable G in Section 3, which is described as corresponding to \"gender\", but it is not clear whether this refers to gender of the writer of the text or gender of the entity being referred to by a pronoun or pronouns in the text. Assumption of a causal relationship between G and Y seems to suggest G refers to gender of entities referred to by pronouns, but the discussion of a relationship between G and dataset access seems to suggest G refers to gender of writers of text. Obviously these two variables cannot be conflated, so the authors should clarify their thinking here.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject