Isolated Causal Effects of Natural Language
Authors: Victoria Lin
Louis-Philippe Morency
Eli Ben-Michael
Venue: ICML 2025 poster (2025)
Keywords: causal inference
natural language processing
omitted variable bias
Primary area: general_machine_learning->causality
TL;DR: We propose a framework for estimating the causal effects of interventions encoded in language.

Abstract: As language technologies become widespread, it is important to understand how changes in language affect reader perceptions and behaviors. These relationships may be formalized as the *isolated causal effect* of some *focal* language-encoded intervention (e.g., factual inaccuracies) on an external outcome (e.g., readers' beliefs). In this paper, we introduce a formal estimation framework for isolated causal effects of language. We show that a core challenge of estimating isolated effects is the need to approximate all *non-focal* language outside of the intervention. Drawing on the principle of *omitted variable bias*, we provide measures for evaluating the quality of both non-focal language approximations and isolated effect estimates themselves. We find that poor approximation of non-focal language can lead to bias in the corresponding isolated effect estimates due to omission of relevant variables, and we show how to assess the sensitivity of effect estimates to such bias along the two key axes of *fidelity* and *overlap*. In experiments on semi-synthetic and real-world data, we validate the ability of our framework to correctly recover isolated effects and demonstrate the utility of our proposed measures.

Isolated Causal Effects of Natural Language
Victoria Lin' Louis-Philippe Morency
Eli Ben-Michael
Abstract
Does the amount netspeak
review make more
helpful
to readers?
local languaga (intervention)
outcooe
As language technologies become widespread, it is important to understand how changes in lan- guage affect reader perceptions and behaviors. These relationships may be formalized as the isolated causal effect of some focal language- encoded intervention (e.g., factual inaccuracies) on an external outcome (e.g., readers' beliefs). In this paper, we introduce a formal estimation framework for isolated causal effects of language. We show that a core challenge of estimating iso- lated effects is the need approximate all non- focal language outside of the intervention. Draw- ing the principle of omitted variable bias, we provide measures for evaluating the quality of both non-focal language approximations and iso- lated effect estimates themselves. We find that poor approximation of non-focal language can lead bias in the corresponding isolated effect estimates due to omission of relevant variables, and we show how to assess the sensitivity of ef- fect estimates to such bias along the two key axes of fidelity and overlap. In experiments on semi- synthetic and real-world data, we validate the abil- ity of our framework to correctly recover isolated effects and demonstrate the utility of our proposed measures.
Ralphal
nelspeek
a(X)
isclated
nSE helptal
decompose
effect
helpful
helpful
non-focal language ("everything else")
ins helptil
How best
helptul
appessiinato?
a aX()
(a) The isolated causal effect of a language attribute like netspeak can be learned by decomposing a text X into focal language al () and non non-focal language (X).
xX
non Pocal language opprosimations
isolated effect estimate
corpus
loxicon
postfiss health
helpful
appre cration
red ceerieg Iaa Rdellty
2g (X)
Isolated
02p12e mution
transformor
dimeraiion dimarntiio
helpful
low ceetiop
5XX)
prompt engineoring
lpolated
estiaato
++ helpful
Iow Sdbiliyy
(X)
(b) Non-focal language cannot he measured directly and instead must be approximated. Due to potential omitted variable bias, which approximation we choose can significantly affect the iso- lated effect estimate.
Figure I. Isolated causal effects allow understand how changes in language affect reader perceptions and behaviors.
1. Introduction
(e.g., profanity) cause readers to perceive a passage of text as hateful? Does use of rapport-building language by thera- pists help to improve patients' mental health outcomes? Do factual inaccuracies propagated in have negative impacts on readers' beliefs or behaviors?
The widespread use of language technologies has given rise to an ever-expanding amount of human- and machine- generated text data. From this vast body of data emerges the opportunity to understand how information contained in language relates to real-world outcomes. Elucidating these relationships can help answer scientifically interesting ques- tions and provide interpretability texts and the models that generate them. For instance, what language attributes
texts
The way in which language choices affect reader perceptions can be formalized as the causal effect of some language- encoded intervention-oficn a linguistic attribute-on on an external outcome (Lin et al., 2023). However, because lan- guage is highly aliased (i.e., correlated with itself), the effect of such an intervention may be influenced by the surround- ing linguistic context (Fong & Grimmer, 2023). For in- stance, machine -eeerraee texts with factual inaccuracies may also contain other undesirable attributes likely to in-
CCrneeiic Mellon University, Pittsburgh, PA, USA. Correspon- dence to: Victoria Lin <vlin2 www.eeeeedddwccmccduu.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Isolated Causal Effects of Natural Language
fluence readers' reactions, such inflammatory language. If the causal effect of the factual inaccuracies is estimated without accounting for aliased attributes, the resulting esti- mate may contain the collective effect of both the factual inaccuracies and some portion of the related inflammatory Lunguage-makinn it difficult to determine whether action should be taken to address factual inaccuracies only, inflam- matory language only, or both.
2. Problem Setting
Consider a text dataset D = ((),,))),... Y....... (X,, Y.) where texts X; X are drawn ii.d. from a distribu- tion P. and individuals with potential outcome functions Y((-) X' + R are drawn i..d. from the population g, where (1())) denotes the potential outcome of individual i if they were to read text æ (Neyman, 1923 [1990]; Rubin, 1974). We study a setting in which all confounding between the intervention and the outcome is captured in the text. Such settings are an important and common case in natural language processing (NLP) due to the widespread prac- tice of labeling text data using external annotators who are randomly assigned to texts. This assignment mechanism in effect creates a randomized text experiment, eliminating con- founding external to the text (Lin et al., 2024). Prominent NLP benchmark datasets such as SST (Socher et al., 2013), SQuAD (Rajpurkar et al., 2018), and MNLI (Williams et al., 20, for instance, all fall under this category.
Motivated by this limitation, we propose new target of inference: the isolated causal effect for natural language (or isolated effect for brevity). We define the isolated effect as the causal effect of only the part of the language contained in the intervention, or the average causal effect of the focal text intervention over all possible variations of the rest of the text (Figure la).
In practice, estimating such effect poses several major challenges. (1) We must be able to formally define and approximate not only the focal intervention but also the non- focal language of a text: that is, all parts of the text external to the focal intervention. (2) Incorrect modeling of the non- focal language can lead to a biased or invalid estimate of the isolated effect due to omission of key language context-a form of omitted variable bias (OVB) (Figure 1b). In other words, valid isolated causal effects can only be measured if the non-focal language is well-approximmted. Therefore, it is important to be able to assess the robustness of the isolated effect estimate to errors or omissions in the non- focal language approximations.
Now let X be parameterized where a(-) X + {0, 1} is the intervention the focal language attribute for which we learn a causal effect- and (.) : X - Rd is the non-focal portion of the text. In this setting, we consider a(-) to be a known mapping from the text to the intervention of interest, a(-) is also known as a "codebook function"' (Egami et al., 2022)
X {a(X), o (X)),
In naturally occurring text, (X) is almost certainly dis- tributed differently when a(X) = 1 compared to when a(X) = O. For instance. suppose aXX) is humor, and X is from a corpus of movie scripts. The non-focal (i.e., non-h parts of the scripts may include properties like positive emotion or optimism. As these properties are posi- tively correlated with humor, it much more likely when a(X) 1 that they also equal I and when a(X)= 0 that they also equal 0.
To address these challenges and to provide a practical path toward estimating isolated effects, this paper introduces a formal estimation framework for isolated effects of lan- guage. Within this framework, we explore how the way we approximate non-focal language impacts isolated effect estimates due to omission of key variables, and we draw on OVB principles to define measures that assess the sensitivity of effect estimates to bias along the two key axes of fidelity and overlap.
To isolate the effect of only the focal language attribute, we must learn that effect over all possible variations of the non-focal text to be sure that no influence comes from the non-focal text. Formally, this is akin to learning the effect while enforcing the same non-focal text distribution Ps for both conditions of (X).
Our experiments ! demonstrate the validity of our frame- work on both semi-synthetic and real-world data. Using evaluation settings where the ground truth is known, we observe that our estimation framework is able to recover the true isolated effect across multiple interventions. We further show that our measures of overall robustness OVB cor- respond closely to how well an estimator is able to recover the true effect, while fidelity and overlap provide additional insight into why an estimate is or is not correct. We suggest that these measures may be particularly useful for analy- sis in world settings where the true isolated effect is unknown.
Definition 2.1 (Isolated causal effect). Let p* be some target distribution over the non-focal language. Then the isolated causal effect of a(X) on Y is given by:
T" = Ey P = (x))(
-
We distinguish this from the natural causal effect defined by Lin et al. (2023), where (X) follows its natural dis- tribution for both conditions of a(X). The natural causal effect is the collective effect of the focal language attribute a(X) and the parts of the non-focal language with which it
Our data and code are publicly available at https:// github. www.hwwww.ceeeeemeeemeeeeeeeeeee
teft-effettt.
2

---

Isolated Causal Effects of Natural Language
is naturally correlated.
First, let us define the transporting importance weight y as:
We generalize three common assumptions for valid causal inference to the language setting:
(2d' 1)P* (x)) P
1. Consistency. Y = (())= = ( The
Using the importance weight, we can identify the estimand r" from the observed data D (X, Y,,Y) where X ~ P and Y follows the resulting induced distribution on the observed responses Pg:
observed outcome Y for an individual corresponds to the potential outcome ((a(X), (X))) associated with the text they actually receive.
2. No unmeasured confounding. (x) TT a (X))) for all x E X. All confounding factors between the intervention and the outcome captured by the non- focal portion of the text.
x" (o)
((X))=(d=())) y
ED P((x
(1 ((xx ED P(ax(x))P(((x)==0pp))))))))))))) = (X)* Y
3. Overlap. 0 < P(a(X)= (X)) < 1. The interven- tion a(X) has a non-zero probability of taking either value, regardless of the non-focal portion of the text. Note that this excludes the possibility that a(X) is a deterministic function of (X).
This identifies the isolated effect as a difference in wereghhee averages of the outcome between texts with and without the focal language attribute a(X).
As we mention earlier, the assumption of no unmeasured confounding is commonly fulfilled for NLP datasets due to text-annotator assignment protocols that eliminate external confounding. In practice, it is also reasonable to believe that the overlap assumption is fulfilled since any representation of the non-focal language a (XX is non-exhaustive, and so a(X) cannot be determined solely from the representation of (X). The most difficult of the three assumptions to fulfill then is consistency, i.c., that observed outcomes cor- respond to potential outcomes. If the approximation of the non-focal language a (X) is missing important information, then consistency may not hold. Part of the technical con-
If we use only the importance weights, however, we run the risk that errors or misspecifications in the weights will lead to errors in the estimated isolated effect. Therefore, building on causal inference principles in non-language settings, we can also incorporate an outcome model g, defined as:
g(a', a (X)) (M
Assuming we have access to texts X* ~ Pa (or have access to the data-generating process of P*), we can identify 7" using the following doubly robust construction:
tribution of this paper is to characterize the implications of this assumption failing to hold.
T" = Ex + Exhla((),, (((()))))((((((()()()))))
(TDR)
3. Isolated Causal Effects of Language
In this section, we describe how to identify, estimate, and evaluate the quality of isolated effects of language. We de- fine estimands for isolated effects, present doubly robust estimators, and discuss how approximating non-focal lan- guage during estimation can give rise to omitted variable bias. Derivations and technical results are in Appendix A.
This construction-commeeee commonly used in causal inference to identify and estimate unbiased effects and increasingly used in machine learning contexts as well-confers robustness to misspecification mis-estimation in either the IPW term or the outcome modeling term (Robins et al., 1994; Byrd & Lipton, 2019; Kallus et al., 2022).
While Pa can be any distribution over (X) (discussed further in Appendix A.1.3), in practice it can be unclear how to define and estimate P* (X)( explicitly, as this requires characterizing full distribution over the non-focal text probabilities. Therefore, we introduce two important realistic choices of Pa that make the problem tractable.
3.1. Identification
The definition of the isolated causal effect requires that a (X) follow the same target distribution la when a(X) = 1 and when a(X)= = O, even if it does not do so naturally. To induce (X) ) to follow the same specific target distribution under both conditions of a(X), we draw on importance weighting (IPW) principles to transport (X) from its natural distribution P to the target distribution P", then supplement this with an outcome model.
First, we can set (' = PXa)(X),, where again P is the distribution of the observed X. We refer to the isolated effect in this case as the Isolated Average Treatment Effect (IATE) the corpus from which the texts originate.
3

---

are the out- Isolated Causal Effects of Natural Language
The corresponding importance weight becomes:
or the outcome model y are correct-i.e., 7 = or g g-then TDR is an unbiased estimator for 7" Moreover, the estimator is asymptotically normal with a closed-form variance, allowing for estimation of trustworthy confidence intervals. See Kennedy (2024) for a review on these types of estimators.
2a -
7(a'aa P(a(X)==a) =a (X))
Second, we can set Pa (X) to equal (()(())((X) 1), the distribution of non-focal language among the treated (i... where (XX) = 1) in the corpus where the texts orig- inate. We refer to this as the Isolated Average Treatment effect on the Treated (IATT). Estimating the IATT instead of the IATE can be beneficial in settings with potential over- lap violations; we elaborate on this in Section 3.3. The corresponding importance weight becomes:
3.3. Sensitivity to Omitted Variable Bias
Omitted variable bias. When representing natural lan- guage, including all "variables" in modeling is not feasible, as a full representation of language is nearly infinitely high- dimensional (e.g.. one-hot encoding of the entire English vocabulary). Instead, the non-focal language is more re- alistically approximated the "short," lower-dimensional representation (x (e.g., language model embedding), However, representations of language necessarily omit infor- mation relative to the original text. In this section, we link the notion of information loss from language representation to omitted variable bias. We use recent work on establishing OVB bounds for non-parametric models (Chernozhukov et al., 2024) and adapt it to natural language setting to study the impact of omitted information in approximations of non-focal language and isolated effect estimates.
a(a',a = P(a(X)=1) =)
(1-d') =1=0)))(())110((
P(a(X)= = ([(((())(((((x=
3.2. Estimation
Having written r" in terms of the observable data, we de- scribe how to estimate it in practice.
Nuisance parameters. To estimate TDR, several nuisance parameters need to first be estimated: the outcome model g and the importance weight y. This requires approximat- ing the non-focal language a (X) with a language repre- sentation. We refer to the approximation of the non-focal language using the notation a (X), where the subscript indicates that this is mapping of the high-dimensional non-focal language a (X) to a lower-dimenssonal "short" representation space Rd (following the terminology used to denote a smaller feature set in Chernozhukov et al. (2024)).
We begin by defining the fidelity metric o² and the overlap metric U²:
o² = EE(((Y o(a((x)
v2 (xx
where g(a(X), and r(a(X), ax())) a(a()),55(X))
come model and importance weight that use the short non- focal language representation (x))). We call these the short outcome model and short importance weight. The fidelity metric indicates how close the short outcome model is to the true outcome model g(((),, (X)), while the overlap metric indicates how well the overlap assumption for valid causal inference is fulfilled by the short importance weights. For both metrics, smaller value is better.
With this mapping. classifier can be trained on a separate sample to predict a(X) given af(()) as input. Such a clas- sifier outputs predicted probabilities P(a(X) = a' (x)) which can be used to estimate 9. Using the approximation
(X), an outcome model O(a(X), (X)) can also be esti- mated on an separate sample where both texts and outcomes are available.
Then the OVB of TDR. that is, TDR using the short out- come model and short importance weight- is bounded:
Estimator. Consider data D = (X,, X; ~ P and Y, ~ py: and X, ~ pa (i E [m].j E (m)). Then the estimator for 7" is given by
TDR. - - OVB
1
FDR =
001, -000
where Cy and Cn are user-set sensitivity parameters for the explanatory power of omitted variables toward the outcome model and importance weight. The OVB bounds allow us to define lower and upper bounds on the isolated effect:
1
-
where the estimated 7 uses the appropriate probability esti- mates from the IATE and IATT definitions above.
CCr Cp): (o):Tbn(G (C),CD) TDR, - VoRv2CyCp
Like other doubly robust estimators, TDR has a number of desirable properties. First, as long as either the weights y
The fidelity-overlap tradeoff. A tradcoff between fidelity and overlap emerges when choosing how to approximate
4

---

We estimate ,2, D2, and the robustness value from the data using debiased estimators (Appendix A.3). We note that under this type of estimation, it is possible for the estimated Isolated Causal Effects of Natural Language
the non-focal language af (X) as (X). If (X) is high- dimensional dense representation like a language model embedding, then model fidelity is likely to be good, as the short outcome model g(a(XX), (X))) has plenty of infor- mation with which to make predictions. However, repre- sentations with good fidelity are also more prone to overlap violations. While we assume in Section 2 that strict over- lap is fulfilled, ((a(X) a' (X)) that are very close to 0 and 1 ("near overlap violations") can still skew the importance weights to extreme values, heavily impact- ing effect estimates. These near overlap violations occur more often if ((a(X) a' (X)) is computed using high- dimensional dense representations for df(X), as the greater number of dimensions makes more likely that certain values of a (X) are almost exclusively seen with either a(X)= = or (XX)= 0.2
4.1. Datasets
Amazon (partially controlled setting). The Amazon dataset (McAuley & Leskovec, 2013) consists of reviews from the Amazon e-commerce site, each with number of "helpful" votes. To reduce unmeasured factors in the data, we generate a new semi-synthetic outcome Y by predicting the number of helpful votes as a linear function of a(X), (X), then adding noise. Here, ((X), (X) encode the 10 categories from the lexicon LIWC (see Section 4.2.1) that are most predictive of vote count. These categories are binarized to take the value 1 if the category is present in the text and 0 otherwise. We note that while the semi-synthetic construction allows us to control the outcome Y. we do not have influence over the joint distribution P(a(X), (X))-
In this partially controlled data setting, we know both (1) the true isolated effect of each of the 10 lexical categories and (2) that the outcome model g can be fully learned from the text. Combined, these allow us to evaluate whether our esti- mator F is able to recover the true isolated effect under best-case conditions, To allow for controlled evaluation un- der more challenging conditions, we also generate a second semi-synthetic Y where helpful votes are predicted as a non- linear function of a(X), (x)):: this setting is discussed in Section B.2.
Importantly, the oivelat tradeoff can be balanced by considering the overall robustness value of the isolated ef- fect that uses the non-focal language representation (X):
RV= = TDR.
σv
Intuitively, the robustness value is a measure of the effect es- timate's ' trustworthiness: it indicates how robust the estimate is to OVB. The robustness value can be seen the amount of explanatory power that can be lost from approximating (X) as (X) before the isolated effect is no longer cor- rect in sign (positive or negative). A larger robustness value corresponds a higher tolerance to OVB.
Semaglutide vs. Tirzepatide (SvT) (real-world setting). Here, we consider a slightly different setting in which the intervention and outcome are both encoded in text (in con- trast to the setting where the intervention is encoded in the text and the outcome is a numerical value external to the text). The SvT dataset (Dhawan et al., 2024) consists of posts from weight-loss communities on the social media site Reddit that mention one of two weight-loss medications: semaglutide or tirzepatide. From these posts, Dhawan et al. extracted the language-encoded binary intervention a(X) (which weight-loss medication the user took) and binary outcome Y (whether the user lost more than 5 percent of their starting body weight). The dataset further includes "ground truth" causal effect from clinical trial on the effects of semaglutide versus tirzepatide at various doses (Frías et al., 2021). Dhawan et al. used weight loss under 5 mg tirzepatide versus 1 mg semaglutide as the true effect. We compute confidence intervals for this true effect using information available in the clinical trial.
22 to be negative; this indicates that something may have gone wrong with importance weight estimation (potentially a severe overlap violation).
Finally, we emphasize that while OVB may correspond to how close an effect estimate is to the ground truth, it is a complementary measure. By assessing effect estimates through the lens of each vetric-fidellin overlap, and ro- bustness value-we gain greater insight into how and why different non-focal language approximations can influence isolated effect estimation.
4. Experiments
This dataset allows us both to evaluate the validity of iso- lated effect estimates in realistic setting and to assess how different approximations of af (X) can impact our estimates.
To assess the validity of isolated effects estimated using our framework, we examine how well isolated effect 7" with our estimator PDR We evaluate on two natural language datasets-on one semi-synthetic and one real-world-in which true isolated effects are known.
recover the true
4.2. Implementation
4 1. APPROXIMATING NON-FOCAL. LANGUAGE
"The IATT is less susceptible to overlap violations than the IATE, as the IATT requires only that P(a(X)= = 1 d{(X)) < 1.
To construct a non-focal language approximation (X)).. we explore number of language representations varying in
5

---

Isolated Causal Effects of Natural Language
complexity. In this section, we describe each representation and discuss how it might fare in the fidelity-overlap tradeoff. Implementation details can be found in Appendix C.2.
the information contained in a paragraph, the age of the writer if they have any health conditions. Using this form of prompting on GPT-3.5, Dhawan et al. (2024) extract set of 10 health-related attributes from Reddit posts in the SvT dataset. Of these, we exclude 3 attributes from which weight loss can be directly computed. We treat the remain- ing discrete variables as a type of language representation and therefore an approximation of the non-focal language.
Lexicon. One simple language representation is a vector of interpretable categories encoded by a lexicon, which maps words in its vocabulary to those categories. These categories are usually relatively few in number, so the dimensionality of the category vector is fairly low. However, lexicons are limited by their vocabulary and are also unable to capture context or sentence-level meaning. Therefore, we expect that lexicon-derived non-focal language representations may have good overlap but poor model fidelity. We use two well-known lexicons in our experiments: the human expert- designed LIWC (Pennebaker et al., 2015) and the semi- automatically generated Empath (Fast et al., 2016).
4.2.2. MODELING AND ESTIMATION
For each non-focal language representation af (X). ), we use 5-fold gross-fitting to train an outcome model 9 to predict Y given (X) and a classifier to predict a(X) given a We use P(a(X) timate 7. Within the training folds, we conduct f-fold cross-validation to select model hyperparameters. For the linear-outcome case of the Amazon dataset, we use a lo- gistic regression classifier and linear regression outcome model (and neural networks for the nonlinear case). For the SvT dataset, we use gradient boosting models for both our classifier and outcome model. Additional model details, including libraries and hyperparameters, are available in Appendix C.3.
a' dx)a5(X)) from this classifier to es-
Language model embedding. Sentence embeddings from transformer-based language models are among the most commonly used language representations for machine learn- ing. These embeddings are information-ric in content and syntax and perform excellently on a wide variety of tasks. However, language model embeddings tend to be relatively high-dimensional compared to lexicons, and as result, embedding-derriee non-focal representations may achieve good model fidelity but suffer from overlap violations. In our experiments, we use embeddings extracted from the pre- trained transformers BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), MPNet (Song et al., 2020), and MiniLM (Wang et al., 2020). For RoBERTa and MPNet, we also use singular value decomposition (SVD) to create a lower- dimensional version of each embedding. We refer these smaller 200-dimension representations as RoBERTa+SVD and MPNet+SVD.
With I and 9 estimated, we are able to compute ēDR on the estimation folds, which we call Destimate- When estimating the IATE, we directly use Destimate as the source of texts X P for the outcome modeling term, the target distribution is equal to the observed data distribution. When estimating the IATT, we draw X from the subset of D Destimate where a(X) = 1. We estimate the IATE for the Amazon dataset and the LATT for the SvT dataset to maintain better overlap. SenteCon. SenteCon is a language representation in which a lexicon-based layer is constructed over language model embeddings (Lin& & Morency. 2023). Like lexicon represen- tations, a SenteCon representation consists of vector of interpretable categories where each category is associated with numerical weight. Because the categories are derived from an existing lexicon, the dimensionality of the Sente- Con category vector should also be low. SenteCon does not rely exclusively on pre-defined vocabulary and is able to capture sentence context. Consequently, we expect that a SenteCon-derived non-focal language representation will have reasonable model fidelity while also not being signif- icantly affected by overlap violations. In our experiments, we use two different base lexicons for SenteCon (LIWC or Empath). We refer to these variants as SenteCon-LIWC and SenteCon-Empath.
We compare our isolated effect estimates against naive estimator that does not isolate the focal attribute from the non-focal portion of the text (i.e., an estimator of the natural effect). In general, we expect this not to correctly recover the
true isolated effect since it has no adjustment for isolation.
5. Results and Discussion
In this section, we evaluate how well our method is able to recover true isolated causal effects in the Amazon and SvT datasets. Following this evaluation, we more closely examine the relationship between isolated effect estimation and omitted variable bias.
5.1. Amazon Dataset
In the Amazon dataset, we examine the isolated effects of the 10 predictive LIWC categories used to construct the semi-synthetic outcome. This section discusses the linear- function outcome, but the same trends appear in the nonlin- ear case (results in Section B.2).
LLM prompting. As large language model (LLM) capabil- ities continue to expand, it has become possible to extract attributes from text passage simply by prompting an LLM. For instance, we might ask an LLM to tell us, based on
6

---

Isolated Causal Effects of Natural Language
true naive ours
overapp
true naive ours
overlpp 1
6.
LLM prompting
6.4
62
LIWC
Edeily
# ddieensions
Empath
fidelity
SenfeCon-LIWW
0.1
0.15
SonteCon Empath
robusthess value?
0.4
MiniLM
0.3
0/2
2
BERT
rabusthess value? 1
3.0
3.5
4.0
5.0
RoBERTa
0.15
isolated ellect (Intervention home]
# dimensions
RoB
0.1
(a) Isolated effect of home.
MPNet
0.35
true naive ours
overaap
MPN
prassing
0.30
Gasadbee doreciiv flryas
-0.2-0.1 0.0 0.1 0.2 0.3 isolated effect
Edeily
ddmersions
Figure 3. Isolated causal effect of weight-loss medication in the SvT dataset. Error bars denote 95% confidence intervals. The blue dotted lines surrounding the true effect mark its 95% confidence interval. Representations (X) are ordered loosely by complexity, with less complex representations appearing closer the top
robusthess value?
2
isolated effect (intervention = netspeak)
# cimensions
(b) Isolated effect of netspeak.
D2, particularly in Figure 2a), it seems that outcome model performance gains are more significant in this case.
Figure 2. Isolated causal effects of linguistic attributes on help- fulness in the Amazon dataset. Error bars correspond to 95% confidence intervals.
5.2. Semaglutide vs. Tirzepatide Dataset
In the SvT dataset, we use the intervention and outcome from Dhawan et al. (2024). We treat the Reddit post text as the non-focal language a (X), and we explore how each of the non-focal language representations in Section 4.2.1 impacts effect estimation.
Across the 10 categories, we iteratively set each category to be a(X) and estimate its isolated effect while using the remaining lexical categories at (X). To explore the fidelity-overlap tradeoff under controlled conditions, we evaluate our isolated effect estimate and our three OVB- derived metrics- g², 1.? and robustness value- -under dif- ferent choices of (X). For each intervention, we restrict the number of remaining lexical categories used as df(X), beginning with categories and ending with 9.
We first observe that all of our isolated effect estimates have wide 95% confidence intervals that include both the true isolated effect and 0 (Figure 3). Looking solely at the point estimates, we see that almost all of the representations yield positive isolated effect estimates that are consistent with the ground truth. Of these, SenteCon-Empath comes closest to recovering the true isolated effect, with MiniLM close second-but but a large amount of uncertainty remains. As result, we may not be able to use the point estimates alone to determine which representation best approximates non-focal language.
Over multiple interventions (Figures 2a, 2b; additional re- sults in Appendix 1.,,, we observe that as the dimensional- ity (number of categories) of (X( increases, so does the proximity of the isolated effect estimate to the ground truth. Moreover, the behavior of the fidelity and overlap metrics 8² and D22 is also consistent with expectations. As dimen- sionality increases, 6² decreases, indicating that outcome model fidelity is improving. At the same time, 2 increases, consistent with worsening overlap.
We look instead to fidelity and overlap, where we observe interesting behavior. The high-dimensional MPNet embed- ding has a much larger 2 than any other representation, suggesting a near overlap violation. Interestingly. BERT and RoBERTa-which have the same dimensionality as MPNet-exhibit much better overlap than MPNet, possibly due to the additional optimization of MPNet for sentence- level tasks in the library used to extract its embedding. We
We further see that robustness values increase with ax) dimensionality, suggesting that gains in model fidelity out- weigh losses in overlap. This may not be the case for all datasets. As this dataset does not experience significant problems with overlap (as seen from the limited range of
7

---

Isolated Causal Effects of Natural Language
also see that several representations, such as the lexicon Empath and the diimensionality-reducee RoBERTa+SVD, have negative S Because doubly robust estimators like the one we use for D2 do not necessarily satisfy criteria like being non-negative in noisy settings, we hypothesize that these negative values (which are small in magnitude) may be due to noise in estimation. Fidelity, on the other hand, is much more consistent across all representations. In general, fidelity is expected to improve (i.e., o² should decrease) with the dimensionality of the representation, as higher- dimensional representations are likely to contain more in- formation; however, this may be negated by strong regular- ization in the outcome model. These results suggest that the fidelity-overlan tradeoff depends on some notion of repre- sentation complexity that may go beyond the dimensionality of the representation alone.
-00000
0.015
0.4
025
0.3
movement
0.030
8
masked
0.2
science 960
0.045
0.1-
exercise healing
unadjusted
0.0
A
0.0
0.1
0.2
0.3
0.4
cy
Figure 4. OVB lower bound of CenteCon-Emppit isolated effect estimate in the SyT dataset. "Unadjusted" marks the point estimate lower bound without OVB.
Finally, we find that the two representations with the high- est robustness values are LLM prompting, which produces a positive but conservative effect estimate, and SenteCon- Empath, which produces an estimate very close to the true effect. MiniL.M, which like -ompaah has a point estimate close the clinical benchmark, has only mid- dling robustness value due poor overlap. The robustness of LLM prompting is not unexpected: Dhawan et al. (2024) carefully designed their prompting procedure to extract dis- crete variables for the specific task of estimating the effect of tirzepatide versus semaglutide on weight loss, so we ex- pect this representation to yield good results. Importantly, however, the SaateCoon-Empatee representation which is not specifically designed for this task-has a similarly high robustness value, suggesting that equally effective repre- sentations can be found without requiring extensive human design effort.
One way of understanding the scale of a robustness value is calibrate using the explanatory power lost when intentionally omitting variables known to have influence on the effect estimate. We focus on one representation-- SenteCon- SenteCon-Empathh in the SvT dataset- -ndseeett-nddddde the lower OVB bound of its associated isolated effect estimate. We recall that this bound corresponds to the least possible value of the effect point estimate at a given level of OVB.
In Figure 4, we plot this bound against the sensitivity pa- rameters Cy and CD. which denote the explanatory power of omitted variables toward the outcome model y and the importance weight y, respectively. This contour plot shows how the lower OVB bound of the effect estimate changes as the hypothetical explanatory power of the variables omitted from the SenteCon- Empahh representation increases. We color the 0 contour red to highlight the significance of the lower OVB bound "crossing" from positive to negative as Cy and CD increase. Once the bound is negative, we can no longer be certain that the point estimate of our isolated effect is positive.
Our results also illustrate the potential of dimensionality re- duction methods like SVD in non-focal language approxima- tion. Both RoBERTa and MPNet benefit from singular value thresholding across all OVB metries: overlap improves, fi- delity remains similar, and robustness value increases. More- over, after SVD is applied, both representations' effect point estimates move from outside the true effect confidence inter- val to inside the true effect confidence interval. These results suggest that dimensionality reduction can significantly im- prove the utility of high-dimensional non-focal language representations. Given the low computational and human overhead of these unsupervised post- processoonsing techniques, they may often be worth trying.
We then plot four red triangles to mark the explanatory power lost by explicitly omitting the category with that name (movement, science, exercise, or healing) from the CenteCon-Empath representation." We choose categories we believe to be relevant to the intervention and outcome. For each category omitted, we see the loss of explanatory power brings the point estimate closer to the O contour but is not nearly enough to cross it. Turning then to the masked marker, we look at the explanatory power lost by omitting key information from the non-focal text (X) itself. We create version of each Reddit post where we mask medi- cation type, body weight, and terms like "gain" and "loss."
A closer look at OVB and robustness. While robust- ness values can be compared among non-focal language representations providdin some sense of how relatively robust each corresponding effect estimate is bias-it it is not immediately clear whether even the representation with the best robustness value is robust in absolute terms.
"hhese values can be computed explicitly by re-fitting the out- come models and importance weights (Appendix C.4).
8

---

Isolated Causal Effects of Natural Language
The resulting SenteCon-Empatt representations experience much larger drop in explanatory power, but the lower bound of the estimate still remains positive. These results suggest that the isolated effect estimate is robust (though noisy, as the wide confidence intervals may indicate), as the lower bound on the point estimate remains positive even un- der levels of OVB comparable to removing relevant lexical categories or masking key information from the text.
2024a). While these methods are not developed specifically for text, the parallels between general covariate representa- tion and language representation are evident.
7. Conclusion
In this paper, we propose a framework for estimating the isolated causal effect of a focal language-based interven- tion. Estimating isolated effects is challenging because it requires to model not only the focal intervention but also the non-focal language of the text. We introduce measures for assessing the sensitivity of isolated effect estimates to omitted variable bias in their non-focal language approxima- tions along the axes of fulelity and overlap. We demonstrate the ability of our framework to correctly recover isolated effects across multiple language-encoded interventions, and explore how the way approximate non-focal language impacts fidelity, overlap, and the robustness of the effect estimates themselves.
6. Related Work
Causal effects of text. Our work on isolated causal effects is situated within a recent literature on estimating text- based causal effects. Egami et al. (2022) describe a conceptual codebook framework for causal inference using text. Build- ing on this, Fong& & Grimmer (2023) conduct randomized text experiments where texts are programmatically gener- ated from pre-specified attributes. Lin et al. (2023) further propose a method for transporting natural (i.e., non-isolated) causal effects from randomized text experiments to poten- tially non-randomized target distributions.
Our results point to several avenues for future research. This paper studies a setting in which confounding is contained fully in the text. Though NLP datasets are not often released with external confounding data like information about an- notators, it may still be interesting to study the case where external confounding is present and measured. Additionally, encoding function a(-) as an accurate parameterization of the intervention of interest, but if a-) does need to be estimated, then estima- tion error can lead additional bias. Characterizing and counteracting this is important future work. Finally, our findings on OVB and robustness suggest a compelling line of research on learning representaaiions-pernee not only of language that optimize the oidelity-overiap tradeoff to min- imize omitted variable bias, making them explicitly suitable for the task of causal inference.
Most directly related to our work are several methods for estimating isolated effects of natural language using specific language representation pipelines. Though these works do not explicitly distinguish isolated and natural effects, their estimands are defined such that they are isolated, and so we view these methods as complementary to ours. Pryzant et al. (2021) estimate the effect of a proxy linguistic attribute from observational data, where all other language information is represented by an embedding from a transformer trained to capture confounding. Dhawan et al. (2024) estimate effects from observational data using the LLM prompting approach we describe earlier. Finally, a recent paper by Imai & Nakamura (2024) estimates text effects via a randomized experiment in which LLM-generated texts are shown to hu- man respondents; text representations can then be extracted directly from the generating LLM.
in this paper we treat the focal language- language-encoding
Impact Statement
Omitted variable bias. The diversity of language repre- sentations that can be used in text-based causal inference highlights the strong need for way to understand the qual- ity of representations and effect estimates. Our based metrics provide a way to do this flexibly across any data setting, which is important for real-world data where the ground truth is not known. Our metrics draw directly on the Chernozhukov et al. (2024) operationalization of OVB, which in turn builds on history of foundational work on OVB (Goldberger, 1991; Frank, 2000; Angrist & Pischke, 2009; Oster, 2019; Cinelli & Hazlett, 2019).
Broader impact. Recent advances in NLP have dramati- cally increased the availability of language data and models for common users. The resulting proliferation of texts and models has raised potential ethical concerns around factual inaccuracies (Monteith et al., 2024; Zhou et al., 2023), bias (Wan et al., 2023; Ferrara, 2024), and the black-box internals of models (Guidotti et al., 2018: McDermid al., 2021). These concerns emphasize the growing need to understand the impacts of texts and language models on the readers that consume them. Our work on isolated causal effects builds toward this goal.
Noting the importance of covariate representation in causal inference, Clivio et al. (2024b) have proposed learning rep- resentations that minimize information loss when balancing covariates, which can help to improve overlap (Clivio et al.,
Ethical considerations. The empirical analysis contained in this work relies partially on representations from pre- trained large language models, which may encode biases from their training data. Interpretations of causal effects that

---

Isolated Causal Effects of Natural Language
rely on such representations should consider these biases. We additionally acknowledge the environmental impact of training the language models used in this work.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pp. 4171-41 Min- neapolis, Minnesota, June 2019. Association for Compu- tational Linguistics. doi: 10.18653//1/N119-1443 URL https: fttpss:/aacaartholoooo httpa//llaaattoolooooooo////---- org/N19-14234
Acknowledgements
This material based upon work partially supported by the National Institutes of Health (awards RO1MH125740, ROIMH132225, R21MHI 3076116777 and U01MH136533 Vic-
toria Lin is supported by Meta Research PhD Fellowship. Any opinions, findings, conclusions, recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors, and no official endorsement should be inferred.
Dhawan, N., Cotta, L., Ullrich, K., Krishnan, R., and Mad- dison, C. J. End-to-end causal effect estimation from unstructured natural language data. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: Forum?id=gzzQARC) /tt.:p::///peeeevieeee wpwpp/eeeneeeeeeeeeeeeeeeeee.eem net/
References
Angrist, J. D. and Pischke, J.-S. Mostly harmless econo- metrics: An empiricist's companion. Princeton university press, 2009.
Egami, N., Fong, C. J., Grimmer, J., Roberts, M. E., and Stewart, B. M. How to make causal inferences using texts. Science Advances, 8(4))xaag22622, 2022. doi: 10. 1266sciadv.abg26522 URL https: http:::::///www.innccee.e science. o awg//oi//e / /10.0000010 1126 /t1///////////// swt.dv.iiiiv....... abg2652.
Byrd, J. and Lipton, Z. What is the effect of im- portance weighting in deep learning? In Chaud- huri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learn- ing, volume 97 of Proceedings of Machine Learn- ing Research, pp. 872-881. PMLR, 09-15 Jun 2019. BtRhttps:: /ttp///peceennsssee/e/// mlr / byrd19a. html.
Fast, E., Chen, B., and Bernstein, M. S. Empath: Un- derstanding topic signals in large-scale text. In Pro- ceedings of the 2016 CHI Conference Human Fac- tors in Computing Systems, CHI '16, pp. 4647-4657, New York, NY, USA, 2016. Association for Comput- ing Machinery. ISBN 9781450333627. doi: 10.1145/ 8858036.2858533. URL https://doo org/10. 011222888836.888888
Chernozhukov, V., Cinelli, C., Newey, W., Sharma, A., and Syrgkanis, V. Long story short: Omitted variable bias in causal machine learning, 2024. URL https://arrat org/abs/ /2t...22..... 13398.
Ferrara, E. Fairness and bias in artificial intelligence: A brief survey of sources, impacts, and mitigation strate- gies. Sei, 6(1), 2024. ISSN 2413-4155. doi: 10. 3390/sci6010003, URL https:/ /ww.. mdpi com/ 2413- 4155/6/1//5/
Cinelli, C. and Hazlett, C. Making Sense of Sensitivity: Extending Omitted Variable Bias. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82 (1):39-67, 12 2019. ISSN 1369-7412. doi: 10.1111/rssb. 12348. URL httpps://don http://ooo...g///////// htttp:://dol..cr/00..11111 12348.
Fong, C. and Grimmer, J. Causal inference with latent treatments. American Journal of Political Science, 67 (2):374-389, 2023. doi: https://to..rr///11111/aaaa 12649. URL https: htttpp/a///llneelbbrrrryy.////// wilee.ill.. com/ /ww./aa///////ooo/////..//aabbb/ /1abs/110.. 10. 1111/ajpss.... 11t.1///..p..1999.....
Clivio, O., Bruns-Smith, D., Feller, A., and Holmes, C. Towards principled representation learning to improve overlap in treatment effect estimation. In 9th Causal Inference Workshop al UAI 2024, 2024a. URL https: // //openreview..o.o net umrumm um I70iiii...
Frank, K. A. Impact of a confounding variable
on a regression coefficient. Sociological Methods & Research, 29(2):147-194, 2000. doi: 10.1177/
Clivio, O., Feller, A., and Holmes, C. Towards represen- tation learning for weighting problems in design-based causal inference. In Kiyavash, N. and Mooij, J. M. (eds.), Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence, volume 244 of Proceedings of Machine Learning Research, pp. 156-880. PMLR, 15- 19 Jul 2024b. URL https: /pppoceeeinngss mlr. press/v244 ccivioi o2 html.
00491221100229000001, URL https: pettps//ddo..org 10. 1w11110009921000090000000
Frías, J. P., Davies, J., Rosenstock, J., Manghi, F. C. P., Landó, L. F., Bergman, B. K., Liu, B., Cui, X., and Brown, K. Tirzepatide versus semaglutide once weekly in patients with type 2 diabetes. New England Jour- nal of Medicine, 503 2021. doi: 10.1056/
10

---

Isolated Causal Effects of Natural Language
NEJMoa2107519. URL https:///ww. newp:/wwwweeeeneee.eeeee........ nemmorrg//// doi/ful1/10. 105110566/EE /tt./o11100..105////.22222111111 NEJMoa2107511
Liu, Y., Oit, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy,
O., Lewis, M., Zettlemoyer, L.., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach, 2019, URL https: / wrwpvv.vivv.////.///...//..aaav. orq/ /as/////////
Goldberger, A. S. A course in econometrics. Harvard University Press, 1991.
McAuley, J. and Leskovec, Hidden factors and hid- den topics: Understanding rating dimensions with re- view text. In Proceedings of the 7th ACM Conference on Recommender Systems, RecSys '13, Pp. 165-172, New York, NY, USA, 2013. Association for Comput- ing Machinery. ISBN 9781450324090. doi: 10.1145/ 2507157.2507163. URL https: org/10. 1145/2507193 25071633.
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Gian- notti, F., and Pedreschi, D. A survey of methods for explaining black box models. ACM Comput. Surv., 51()), August 2018. ISSN 0360-0300. doi: 10. 145/33233336 URL httpt: //oii org/oogg//ooo..oooo 10. 100...11453336000
Imai, K. and Nakamura, K. Causal representation learn- ing with generative artificial intelligence: Application to texts treatments, 2024. URL https://aaxx: org/abs/////ooooooor..goo.c.. /tabs//2100 2410.2100.00 00903.
McDermid, J. A., Jia, Y., Porter, Z., and Habli, I. Artificial intelligence explainability: the technical and ethical dimensions. Philosophical Transactions of the Royal Soci- ery A: Mathematicaal, Physical and Engineering Sciences, 379(2207):20200363, 2021. doi: 10. (0.10///stt..22200.0666 ttttpa://rooalloociiiccooo
Kallus, N., Mao, X., Wang. K., and Zhou, 7. Dou- bly robust distributionally robust off-policy evaluation and learning. In Chaudhuri, K., Jegelka, S., Song, L.,, Szepesvari, C., Niu, G., and Sabato, S. (eds.), Pro- ceedings of the 39th International Conference on Ma- chine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 10598-106331 PMLR, 17-23 Jul 2022. URL.tttps // //tpceeee/eeeeee/ooooooooocccoe ngs. ml presss/ v1 v162 htt//aallss22a.nam
URL brg/doi/abs///eeeb ooto/ 0363.
Monteith, S., Glenn, T., Geddes, R., Whybrow, P. C., Achtyes, E., and Bauer, M. Artificial intelligence and increasing misinformation. The British Journal of Psy- chiatry, 224(2):33-35, 2024. doi: 10. 2e..1992/bpp2222311.
Kennedy, E. H. Semiparametric doubly robust targeted dou- ble machine learning: a review. Handbook of Statistical Methods for Precision Medicine, pp. 207-236, 2024.
Neyman, J. On the application of probability theory to agricultural experiments. essay on principles. section 9, Statistical Science, 5(4):465-472, 1923 [1990].
Lin, V. and Morency, L.-P. SenteCon: Leveraging lexicons to learn humen-lnterpretabble language rep- resentations. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 4312-4331, Toronto, Canada, July 2023. Association for Computa- tional Linguistics. doi: 12116663///2022..mmmnnne... 264. URL https: ////aalanntt /acaaaaaaannooooo.oaaaaa hwwphoooogyoooogg///ooooo wrg.//gg//302 2023. findings- 264.
Oster, E. Unobservable selection and coefficient stabil- ity: Theory and evidence. Journal of Business & Eco- nomic Statistics, 37(2):187-204 2019. doi: 10.1080/ 07350001 URL https: http:/oo.oorr///// 10.1080/0735002 wtt.:/350111..1.....1 22..2.111..221
Pennebaker, J. W., Boyd, R. L., Jordan, K., and Blackburn, K. The development and psychometric properties of liwc2015. Technical report, 2015. URL http: /wwwc. net
Lin, V., Morency, L.-P., and Ben-Michael, Text-transport: Toward learning causal effects of natural language. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1288-1304, Singapore, De- cember 2023. Association for Computational Linguistics. doi: 10. 1t2.18663./1122222.mmmllppppppp m2. URL ht tps: //aclanthologgeeeeooee wwwp:/thhoooooooooo..e.e 2tapg///222222 emn wmw.lmmmaal. 82.
pdf.
Pryzant, R., Card, D., Jurafsky, D., Veitch, V., and Sridhar, D. Causal effects of línguistic properties. In Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D.. Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y. (eds.), Proceedings of the 2021 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pp. 4095-4109, Online, June 2021. Association for Computational Linguistics. doi: 10. 16.188/111/2221. main URL https: http:///caaathhooooooo org 2rg/2021- www.aaaallaaaaa -aa.mmaarrr..... 323.
Lin, V., Ben-Michael, E., and Morency, L.-P. Optimizing language models for human preferences causal infer- ence problem. In Kiyavash, N. and Mooij, J. M. (eds.), Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence, volume 244 of Proceedings of Machine Learning Research, pp. 2250-2270. PMLR, 15-
Rajpurkar, P., Jia, R., and Liang, P. Know what you don't know: Unanswerable questions for SQuAD. In Gurevych, I. and Miyao, Y. (eds.), Proceedings of the
19 Jul 2024. URL https: htttt::::/ppccceeee /ttp///eceeengs...//o/p mlr. press/v22 /esss//2441110 html.

---

Isolated Causal Effects of Natural Language
56th Annual Meeling of the Association for Compula- fional Linguistics (Volume 2: Short Papers), pp. 784-789, Melbourne, Australia, July 2018. Association for Compu- tational Linguisties. doi: 10.18653/v1/P11-2111 https:/ / ac1 antaaantthooy.. org/ -
pdf.
Williams, A., Nangia, N., and Bowman, S. A broad- coverage challenge corpus for sentence understanding through inference. In Walker, M.. Ji, H., and Stent, A. (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 1112-11222 1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. (0.18653/1/N18-1101 / //aclanthology org/ 1tfg/N18-110011
URL
Robins, J. M., Rotnitzky, A., and Ping Zhao, I.. Estimation of Regression Coefficients When Some Regressors are not Always Observed. Journal of the American Statistical Association, 89(427):846-866 1994.
Rubin, D. B. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of edu- cational Psychology, 66(5):688, 1974.
URL https:
Zhou, J., Zhang, Y., Luo, Q.. Parker, A. G., and De Choud- hury, M. Synthetic lies: Understanding ai- ti-generated misinformation and evaluating algorithmic and human solutions. Proceedings of the 2023 CHI Confer- ence on Human Factors in Computing Systems, CHI '23, New York, NY, USA, 2023, Association for Com- puting Machinery. ISBN 9781450394215. doi: 10. 115/3544548.35813111 URL https: httpp:://doi.oggg/ 10.1145/3544 35881333331111
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning,
C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over sentiment treebank. In Yarowsky, D., Baldwin, T., Korhonen, A., Livescu, K., and Bethard, (eds.), Proceedings of the 2013 Confer- ence on Empirical Methods in Natural Language Process- ing. pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguisties. URL https: //acl antho 1 logy org/ wwwg/11----1 -
Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.-Y. Mpnet: Masked and permuted pre-training for lan- guage understanding. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 16857-16867, Curran Associates, Inc., 2020. URL https: neurips..............m.......... cc
(aa600ee93a6622ee22eccbbbb333333 Pape. pdf.
Wan, Y., Pu, G.. Sun, J., Garimella, A., Chang, K.-W., and Peng, N. "kelly is warm person, joseph is role model": Gender biases in LLM-generated ref- erence letters. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computa- tional Linguistics: EMNLP 2023, pp. 3730-3748, Singa- pore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/112002..fddinggg
243. URL httph:tp//// http://////claaaoooyo.oooo Findings-emiip.24
Wang, W. Wei, F. Dong, L., Bao, H., Yang, N., and Zhou, M. Minilm: Deep self-attention distil- lation for task-agnostic compression of pre-trained transformers. In Larochelle, H., Ranzato, M., Had- sell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, vol- ume 33, pp. 5776-5788. Curran Associates, Inc., 2020. URL https://proceedii hwweiinnnnn.nnnnnnneeeeeeeee ca/ /ttppppr_flllesppppp////////////
12

---

Isolated Causal Effects of Natural Language
A. Derivations and Proofs
A.1. ATE and ATT
A.1.I. IDENTIFYING THE ESTIMAND
Proof. Equivalence of TDR and r".
We have X P.Y ~ py. = (X,Y), (.Y)amm and X* where (x(*)) ~ P Because we set the value of a(x*), we use the notation X p and (X*) p interchangeably. In principle, text comes from finite sample space, so we use summations and probability mass functions to describe it.
We want to show the following:
TDR = Ep
a(0,aa(X'))
Cyvng [Bo (M(a(X) = 1xx)(x)) - -,
First, notice
(2a' - 1)P*
(a),a((X))
P(a(X)= = =]([(()))))(((
(2a' - (2d=1I)((((x)))1())===
P(a(X)= = (100))/0000)))((())PP)00))00000
- (2)'
((ax(X))a(X) =d'))(((X))=)))) = d')
(2a' (X))
P(as()),al) aXX) = a')
(x))
ifa =1 if
('))
Now consider the first term of TDR:
ED ((x( - ((x))) ( -((())
(x()((X))
= =Er(ro Ex ((x)) = (Y - ([](((
(X()(())
P(as(X), a(x) = 0) (Y g(0, (X)))()0( = 03
(x))(())
= =Ey(ng
Ex P(as(X), a(x) = (V g(1 (X))() = 1)
(X))
-Br()~G
Ex
a(X)) = =0) (Y g(0,0 (X)) = 00}
13

---

Isolated Causal Effects of Natural Language
Now can rewrite
(X))
|
Ex
Σ 
=Evr~o
Ex
- (Y(d
LEE
= d') (Y(d
Σ (z))
-Errne
)()
- (x)))
[fa(z)= = d' d') = (x). - -(x))
P*(d(())
= =Erise
Σ
((a)(t) (odd - - olo
LEN
= =Ev(~~
E(((()))) g(d', (x)))
LEX
EYcin [Ex: ((d''a g(d,, (X*))]
Ex [Ey() ( By Brine Y (X)))]
Then consider the second term of TDR:
Ex--p-(((,,d((''))) - g(0, (x--=(0(16(())))00000000011 = Ex-
(x) -(0,
Then putting everything together,
TDR = 0- 000 + Fx (V(1
(V, - v(0) = +
A.1.2. y FOR TWO SPECIAL CASES
(1) IATE: With Pt (x)) = (Xa)()), we can rewrite 7:
(2a' ()
rad. = P(ax()))(((())=dd = =d(a{(()))
(2a' (2a'-1)P2 (1))_))()((X)
((ax()))((a(x)= = (X)
2a' 11
P(a(() = a (X(())
14

---

Isolated Causal Effects of Natural Language
(2) IATT: Likewise, with (X)) ((aa(X))a(X) 1), we can rewrite 7y
(x)) (l, = (
P(a((X)a((x)=1
P(ax(
P (((x))))
= P(a(X) -(a(()) = (x))
P(a(X)= =1
Pd (X))
700,d
((ax 0a
P(a{())a(())) =1
((((X)) (X)(
P(a(X) ((
P(())))) ((ax))=((((((
= -10(x)))(o{(())
d' (lax') = (xa)())) 7ld' (x()) = P(a(X) = P(a(X) (())))) =
A. 1.3. A GENERAL P
Rather than setting a specific (X)), we can identify the isolated effect for any target distribution P for which we have corpus T. Consider corpus T where (X) follows target distribution P and a corpus S where (X) follows the initial distribution P. Let C' be random variable that indicates which corpus text comes from.
Then we notice that:
(X)) can be equivalently written P(a T).
(X) can be equivalently written as (Xa 5).
Then we have
(X)) ((x)))
(x))
P(a(())(C=T)
= =Xx(())
P (IC==T((X))P(XXX)))CC
P(C ((oP)))))(( (
P(C TT) X P(C = =T(=[())
P(C= = S)
1
P(C ((D{(X))
P(a(X) ())
All quantities are easily estimated: P(C S), P(C = T) from sample proportions: P(C = (X)) and P(C =
(X)) from a classifier trained on both corpora that predicts given (X) as features; and P(a(X) = d' (X) S) from classifier trained on corpus that predicts a(X) given (X) as features.
15

---

Isolated Causal Effects of Natural Language
A. 1.4. UNBIASEDNESS OF FDR GIVEN ONE CORRECT MODEL
Proof. (Ex,x = when either 7('', (x)))) - r(a'' (X)) or (((dd, - g(d''aa (x)))
Consider data D = (X,, Y), X and Y Py; and X, P (i E [n].j E [m]).
First, we rewrite:
Ey~P, [x [DDll = Ey~P, Ex.x
+ Σ
-(xxx)), - (((())),
1 m
By E (1,, - -(0,)))))0000x3
171
+
By-p,
- -(a()))) (X,,))
1
m
Ex. -(L
m
+
= By-p, -
11
+ = = 0)7 - ox
(x))(
Ex- 0x,
Ex. 1{a(())= 1} (((x)
((X.) =1) (V, (Vi- 3(1 ((x,))
m
Ex (Xx3))
m
Ex- 1(a(X.)= = 0} P(as(X.). = (Y: -000 ((0,0{(X.)) o5(xa)) |])
(x))(XX))
BynP,
Case 1: Fa'', (x()) = r(d', xx() (i.e., (=()(()) (x(()()))
((a5(X))(())=o)) P(o=(X),a(x)=a))
First, we consider the IPW term:
n
(x)) -Y---(1 g(d' -
Ey ~ Pu
Ex [[((x,)= = d'} ((X)) ())
77
a')
1=1
Ex (a((x)) d') (x())
=(xx) (Y, ā(d, (((.)))
EymP,
71
(()
Ex -Y()~
((a(()),((z)=d = a²) (((dda(a))
FEX
- - afd'. = a' (') = a = a(z))
= =Er()~ [x (Y (x')) H(d', (x*)) ))]
(following TDR identification)
=Er(>~ [xx |Y(a', a (X')]] By Ex- [(xd,(((X")]]
Ex(rn [Ex- (a', a (X*))] Ex- Iā(a', (x()))]
16

---

Isolated Causal Effects of Natural Language
Next, we can rewrite the outcome modeling term:
1
T
Ex
Mid ((x])] = Ex
Σ
D -
m
==1
L
Σ d(dd, (x(a = (x))
Σ O(d a P'))<()
Ex
a (X"))
So now we have
Bymp, [xxx [Fnl]] = Ex (1 + +yy (Ex. (VI,a - Ex (xx))
- (Bxx [000, [xx Ry [Fx- (x*)) - Ex (700 (x()))))
Er(rn [xx- (L, (X - Ex()_ [xx-
Brcing [Br -(0,xx))
Case 2: g(d', (x()) = g(a', (X)).
Again, we consider the IPW term:
[a((x)) = (X,))(X)))
Px Px
Ex
Plag(X)), (X),) = d)) (Y, o(d (x(,))
1=1
(Xi)) (Y: - g(a', (X.)))
BymP,
Ex
[(a(Xi) = a'} a(xx)))),((X.) = =a))
i=1
p'(a((x))
= =Er(~= Ex
Σ 1[a(x)= = ='} ((a{(1))((z)) = a²) (Y(a',
ZEE
g(a',a (()))((( a(() = a(x))
Σ (())(x))
= =Erc>~
P(o5(t),((z==d)) = ((z))
LEE
(( 1fa(z) =
= (())
E((e(o))(o)) = d)) (((((())
P(a)(=).a(z) a') { (
Ercya
LEX
= Σ P(a)()) = d)) p*(a(()))
o(o)()).o(x)==o)) P By ((a'aa - a(a''a
ZEX
= - P(a)))),a()) = dd) (x((x)) Ercy~o (z) - Byo-a
p(()()).(())=0) =a)
ZEX
P*(a((z)
> P(a{(z),((z) = dd) P(ag(o). a(x) =a)) 0
ZEX
0
17

---

Isolated Causal Effects of Natural Language
Now looking the outcome modeling term,
1 Σ Bx- lg(dl (Xx]) E ([(]),
Ex (())((((( = a<(x))
Leex
Σ (x)) (1fa((x*)) - {x))))]
ZER
Σ (z))g(a')) (x)))
ZEN
) lo (x*))
= Exx -r (Y (X
So now we have
Ey~P, (x.xx-xxFRi
+
=
A. 1.5. CONFIDENCE INTERVALS FOR FDR
Consider data X₂)) (X...... X Xi....... Xx),, iY...., (Y. =0 -Y,,.,,,,, O,....,,,,,.......
0,..,, 0). Then following
standard procedures for doubly robust estimators (Kennedy, 2024), the estimator for the closed-form variance of fDR is derived from the influence function as follows.
n+m
Var(FDR)
1[k - m m
TI m
k=1
m 1fk (5) (a(())) o (Xx)))
7DR
n
For the IATE case, we set P* (X)) P(a (X)), meaning there no external X Instead, the outcome modeling term also computed over i [n], giving the variance:
2x(X,) - (Y, (x))) .DR 1 Σ g(1, g(0,) -(1,,((x))))((006(xxx)) P(a((x))a=(x.)) n2
Var(FDR)
For the IATT case, we set (x = P(o(X))((X) = 1), so that there is again no external X*. Instead, the outcome modeling term is computed over the subset of E [n] where a(X,)
1a((,)) = 1}
Var(FDR)
Σ
P(a(X,) = (71, - -70 a(x))
im1
a(Xi)
(1 (lox 1(a{(x)))
P = Ola (X.) - (( -DR
P(a(X,)== 1)
18

---

Isolated Causal Effects of Natural Language
Asymptotic normality is established using the CLT (Kennedy, 2024):
TDR T + 1),
Var(FDR
which gives us the following confidence intervals:
$DR Za/2 Var(Fon), F + 2a/2V Var(FoR)
A.2. OVB Metries
Following Chernozhukov et al. (2024), we can define "short" version of our isolated effect estimand as the difference between (), g(O.-) where we use the "short" representation of the non-focal language, (x(X). in place of the true representation a)(x)):
= = -00
Using the proof in Appendix A.I.I, we also have that
- = TDR, = -x (otp::/oeo..eeoo)oo00000oo4444// -00-00)000000HHkkkkkd/d-oo-oo-d-
This allows us to align our estimand with Chernozhukov et al. (2024), where 9s the short outcome model (g(a', (X))) in our setting) and as are the short Riesz representer weights (7(a', (X))) in our setting). Then it follows directly that:
a² = =Ep((Y 9. = Epp((Y - gla(x),
22 = Erla?]
Eph((a()),
-2p=1h(1,d(((")) 9
A.3. OVB Estimators
Following the procedure in Chernozhukov et al. (2024), we construct debiased estimators for r² and v
n
Σ (Y, d(()))
2
Σ
((())))
Then the OVB bounds (DRR tR) on the isolated effect estimate are:
CT,(CCC = Vo2D2CyCD
B. Additional Results
B.I. Additional Interventions (Linear Amazon Outcome)
In this section, we provide and discuss isolated effect estimates for additional interventions on the Amazon dataset (Figure 5). As is the case for the results in the main paper, we see here that as the dimensionality of a (X) increases, fidelity improves
19

---

Isolated Causal Effects of Natural Language
Irue naive
overaap
true naive
overisp
35
30
Idmonsions
7
ideilty!
dmeensions
lidelity
6
8
6
5
robustieess value 1
robustness value?
0.8
0.9
3
0.8
0.7
0.5
0.6
0.5
0.5
3
6
2
isolated effect (Intervention temale)
dimenalons
isolated effect (intervention filler)
/dmeessoos
(a) Isolated effect of female.
(b) Isolated effect of filler.
true naive curs
overiap
9
8
ddmnnsone
7
fideliy.
6
#
robustnoss value?
12
3
2
0.8
4.5
5.0
5.5
6.0
6.5
7.0
6
isolated effect (Intervention sexual)
4 dimensions
(c) Isolated effect of sexual.
Figure 5. Isolated causal effects of linguistic attributes on helpfulness in the Amazon dataset (linear semi-synthetic outcome). Error bars correspond to 95% confidence intervals.
(i.e., the fidelity metric decreases) while overlap becomes worse (i.e., the overlap metric increases). The robustness value suggests overall improvement with increasing dimensionality, though we again note that this may not be the case for some datasets where overlap violations outweigh fidelity improvements.
Interestingly, for all three interventions, we observe that as the number of dimensions increases from 2 to around 5, the isolated effect estimates do not move closer to the ground truth (the point estimates actually move farther, but their confidence intervals suggest that this is not statistically significant). Only after dimensions does the proximity of the estimates to the ground truth increases with dimensionality, We observe this to be the case for the netspeak intervention shown in the main paper well. We speculate that this may be due the way in which the n-dimensional non-focal language representations are constructed. The n-dimensiinal (x(X) representation is always the same lexical categories rather than a random sample of n out of the 9 categories. Therefore, it is possible that the specific additional categories included in the 3- to 5-dimensional representations do not provide much additional information about the outcome, explaining the behavior of the estimates.
B.2. Nonlinear Amazon Outcome
In this section, we discuss results of isolated effect estimation on a more complex version of the Amazon dataset where the semi-synthetic outcome is a nonlinear function of a (X), (X). The outcome in this setting differs from the one described in Section 4.1 only in that a nonlinear gradient boosting model is used to predict helpful vote count from a(X), (X)) instead of linear regression model; this prediction is then noised to obtain the semi-synthetic outcome.
Using this new outcome, we follow the protocol described in Section 5.1 to obtain effect estimates for the two attributes featured in Figure 2: home and netspeak. During estimation, we use simple feedforward neural networks with no more than
"The last several dimensions for female are exception this, where overlap slightly improves-thin this may be by chance due better regularization in the classifier models.
20

---

Isolated Causal Effects of Natural Language
Irue naive
overlap
true naive
overiap
4003
0.0
3000
2002
5.0
8
ddmnnsoons
7
60
lidelity 1
dmmnnsoon
fideiiy
6
55
00
6
robustress value
robustiees value
0.30
0.23
0.25
0.15
0.05
0.00
isolated effect (intervention home)
dimensions
Isolated effect (intervention netspeak)
dimensions
(a) Isolated effect of home.
(b) Isolated effect of netspeak.
Figure 6. Isolated causal effects of linguistic attributes helpfulness in the Amazon datasct (nonlinear semi-synthetic outcome). Error bars correspond to 95% confidence intervals.
3 layers to fit our importance weight and outcome models.
The results of these additional experiments are consistent with those included in the main paper. For both interventions, we observe that the point estimate of the isolated effect generally grows closer to the ground truth as the number of dimensions increases (i.e., as the number of omitted variables decreases). Likewise, the behavior of the fidelity and overlap metrics 72 and 222 remains consistent with expectations: as dimensionality increases, does z2 while 6² decreases. Occasionally slight variability in these trends appears,
would expect from the noisiness of estimation with more complex models.
Turning to robustness, we see that for home, robustness value generally increases with the number of dimensions, suggesting that gains in model fidelity outweigh losses in overlap. For netspeak, robustness value remains about the same up until 6 features, then decreases sharply. This coincides with the effect point estimate starting to move away from the ground truth, though the estimate's 's confidence intervals do still contain the true effect. This suggests that the worsening overlap outweighs gains in model fidelity and that for this effect, the optimal non-focal language representation a (X) may contain only features.
Finally, we note that once the final feature is added for netspeak, D2 sharply increases, signaling much worse overlap. This is an interesting illustration of how soft overlap violations can occur once sufficient information is contained in af(X) such that the model can fully predict a(X).
C. Experiments
C.1. Data
Table 1. Composition of data splits and licensing information.
Samples per fold
# folds
License
Amazon
1,000
Unknown
SvT
1,012
5
Unknown
C.2. Language Representation Implementation
To implement lexicons, use the third-party 1iwe Python library and the empath library released by its ereators. SenteCon-LIWC and SenteCon-Empatt representations are obtained using the sentecon library released by its creators. BERT and RoBERTa embeddings are obtained via the HuggingFace transformers library using the pre-trained models
and roberta-base, respectively. MPNet and MiniLM embeddings are obtained via the HuggingFace sentence-trann formers library using the pre-trained models all www.mmeetaaeeeeeeeeeeeeeeeaeaeee all-Min -2-22, respectively. Finally, LLM (GPT-3.5) prompting covariates are taken directly from the SvT dataset
and
21

---

Isolated Causal Effects of Natural Language
Table 2. Technical details for language representation implementations.
Language
Library
Version
Model
LIWC
Python
liwc
0.5.0
Empath
Python
empath
0.89
SenteCon
Python
sentecon
0.1.9
BERT embedding
Python
transformers
4.32.1
bert-base-uncasel
RoBERTa embedding
Python
transformers
4.32.1
zoberta-base
MiniLM embedding
Python
sentence-traasformeer
2.2.2
all-WirilMM-16-22
MPNet embedding
Python
sentence-traasfoonee
2.2.2
all-mpnet-base-v
GPT-3.5 prompting
9pt-3.5-turbo
released by Dhawan al. (2024). Additional technical details are provided in Table 2.
C.3. Model Details and Hyperparameters
All outcome models and (XX) classifiers are implemented using the scikit-leara Python library (version 1.3.0). Gradient boosting models base learners. Neural networks used for outcome models in the nonlinear Amazon setting are implemented with the MLPRegressor class and tuned over the following possible layer counts and sizes: (128.), (128, 128), (128, 256, 128).
subsample proportion of 0.7, i.e., 70% of training samples used to fit the individual
Logistic and linear regression models are optimized for L ratio over the range [0.0, 0.1, 0.5, 0.7, 0.9, 0.95, 0.99, 1.0], where 1.0 corresponds to L1 penalty only and 0.0 corresponds to L2 penalty only. Logistic regression models are further tuned for C' (inverse regularization strength) over the following search space: [0.001, 0.01, 0.1, 1.0, 10, 100). For all interventions, optimal hyperparameters are linear regression L1 ratio of 0.5, logistic regression L1 ratio 0.0, and C' of 0.001.
Additionally, the naive estimator is computed formally as follows:
n Σ (a(x,)) ((13)))--11((x (Xx))))))
naive
12
C.4. OVB Lower Bound Analysis (Computing Cy and Cp)
Here, we describe our method for computing the explanatory power lost by omitting information from our SenteCon-Empath non-focal language representation.
First, let ({(() (X)sE denote the "full" CanteCon-Empattee representation (i.e., containing all lexical categories and representing the unmasked text). Now let a{(X) denote a SenteCon-Empatt representation with omitted information.
Then following Chernozhukov et al. (2024), the explanatory power lost from this omitted information can be computed explicitly Cy and Cp:
Cy
Ep (Y g(((()),_(X))s8-))
Cp Ep(y(a(X), ( (X)
We construct representations d5(X)sE- in which each of the labeled lexical categories (movement. science, exercise, and healing) is omitted, as well as a representation of the masked text. We then fit outcome models and a(X) classifiers using each d{(X)sE-s following the same model fitting methodology described in the main paper, and obtain g()(X), ag (X)EE))) and (x)))), (X)ssE=) These are used to compute Cy and CD over D.
22

---

Isolated Causal Effects of Natural Language
C.5. Computing Resources
All experiments were conducted on consumer-level machines. Experiments involving language models, such as those with MPNet and SenteCon embeddings, were conducted using consumer-level NVIDIA GPUs.
23


Reviews:
[{"review_id": "DkuGL5z6tS", "reviewer": "Reviewer_gbt6", "summary": "This paper, based on the principle of omitted variable bias, proposes a framework to estimate the sensitivity of bias in evaluating the non-focal language outside of the intervention and the quality of isolated effect estimation along the two key dimensions of  fidelity and overlap.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful questions and feedback! We address your comments and clarify points below.\n\n**[Technical language]** We will revise the paper to more clearly introduce and contextualize technical terms.\n\n**[Motivation for isolated effects]** Due to character limits, please see our response to reviewer ZnfT titled [Why isolated effects vs. natural effects?].\n\n**[Methodological contributions]** Likewise, please see our response to reviewer ZnfT titled [Novelty of proposed framework].\n\n**[Derivation of formulas]** The importance weight in line 144 and estimand $\\tau^*$ in line 158 draw from the transportability literature and doubly robust estimation literature, respectively. In Appendix A.1.1, we provide the full derivation of the importance weight and show the equivalence of $\\tau^*$ in line 158 to $\\tau^*$ in Definition 2.1. We initially left these derivations in the appendix due to space constraints. We agree some technical details may have felt abrupt as a result and will move key derivations to the main paper.\n\n**[Mediation analysis]** We appreciate this insightful question. Though there are parallels in intuition between isolated effects and direct effects in mediation analysis, they are different technical problems. Our setting is *not* a mediation setting:\n- In mediation analysis, such as in the papers you mention, interventions (e.g., math operations) cause mediators (e.g., the text “surface”), and mediators cause outcomes. Only the intervention can be controlled directly in the study, while the mediator cannot.\n- In our setting, the entire text is a high-dimensional treatment where both the focal language $a(X)$ and the non-focal language $a^c(X)$ can be directly intervened on by randomizing the text. $a^c(X)$ is not caused by $a(X)$ and *can* be directly controlled, so it is not a mediator.\n\n**[Nonlinear $Y$, $a(X)$, $a^c(X)$ relationship]** $Y$ and $a(X)$, $a^c(X)$ have a linear relationship only in the Amazon data setting. Our SvT dataset results demonstrate that the true effect can still be recovered even when the relationship between the text and outcome is nonlinear and complex.\n\nTo further explore this, we conduct additional experiments ([anonymous link](https://naturl.link/nonlinear) to results) on a new version of the Amazon dataset where $Y$ and $a(X)$, $a^c(X)$ have a nonlinear relationship. We fit a nonlinear gradient boosting model over $a(X)$, $a^c(X)$ on the true helpful vote count, then use the predicted vote count (+noise) as our new semi-synthetic $Y$. Using MLPs with <4 layers to fit importance weights and outcome models, we follow the protocol in Sections 4.2.2 and 5.1 to estimate effects for the attributes featured in Fig. 2.\n\nThe results of these additional experiments are consistent with the previous Amazon results:\n- For both *home* and *netspeak*, as # dimensions increases (i.e., as # omitted variables decreases), the effect point estimate grows closer to the ground truth, and overlap becomes worse while fidelity improves. There is slight variability in these trends, as we would expect from the extra noise from more complex models.\n- For *home*, robustness value increases with dimensionality, suggesting that gains in fidelity outweigh losses in overlap. For *netspeak*, robustness value decreases sharply after 6 features, suggesting that worse overlap outweighs improved fidelity. This is especially true on the 9th feature, where the overlap sharply worsens, indicating that $a(X)$ can be almost fully predicted from $a^c(X)$.\n\nWe thank you for this question and look forward to including these results in the paper.\n\n**[Complex $a(X)$]** If $a(X)$ is continuous rather than binary, this introduces a new causal problem where it is common to estimate a dose-response curve or incremental effect instead of an average treatment effect. As the study of continuous treatment effects is its own active area of research in causal inference (e.g., Kennedy et al. 2017; Brown et al. 2021), we believe it is beyond the scope of this paper.\n\n**[SvT confidence intervals (CIs)]** Wide CIs suggest that the estimates from the data are noisy (i.e., imprecise) but not necessarily that they are not robust. Robustness in causal inference refers to the stability of the effect estimate under potential errors or omissions in modeling, assumptions, and/or data. Our OVB analysis in Fig. 4 suggests that the SenteCon-Empath estimate (which does have wide CIs) *is* robust in this sense since the point estimate remains correctly positive even when we compromise the estimator by removing important features.\n\nWe hope our response addresses any potential concerns and that you will consider revising your score. Thank you again!\n\n---\nBrown et al. \"Propensity score stratification methods for continuous treatments.\" Stat. Med. 40.5 (2021): 1189-1203\n\nKennedy et al. \"Non-parametric methods for doubly robust estimation of continuous treatment effects.\" J. R. Stat. Soc. Ser. B Methodol. 79.4 (2017): 1229-1245", "claims_and_evidence": "The writing of this work is unclear and contains a large number of technical terms, such as  language attributes and  focal language, as well as specialized terminology in causal estimation, making it challenging for readers outside the field to clearly understand the paper. Additionally, the motivation of the study is not well-articulated—why is it necessary to study the isolated causal effect? Could the authors further elaborate on this?", "methods_and_evaluation_criteria": "1. Could the authors further clarify their methodological contributions? From the paper, both the  doubly robust construction and  effect estimation seem to be a combination of existing techniques (please correct me if I’m wrong). It appears that the authors have merely applied existing methods to a new setting.\n\n2. Could the authors clarify the differences and connections between the method proposed in this paper and approaches that use mediation analysis to analyze specific effects? For example, in [1,2], mediation analysis is used to study the effect of LLMs. Similarly, one could consider treating focal language as the variable of interest and non-focal language as a mediator, as they together constitute the input text, while the goal is to isolate the effect of focal language on the output. This would require a strict disentanglement of the mediating effect introduced by non-focal language. Could the authors further elaborate on how their method differs from the mediation-based approaches mentioned above?\n\n[1] Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schölkopf, and Mrinmaya Sachan. A causal framework to quantify the robustness of mathematical reasoning with language models. arXiv preprint arXiv:2210.12023, 2022.\n\n[2] Han Y, Xu L, Chen S, et al. Beyond Surface Structure: A Causal Assessment of LLMs' Comprehension Ability[J]. arXiv preprint arXiv:2411.19456, 2024.", "theoretical_claims": "This paper does not include additional theoretical explanations for the proposed method. Could the authors further clarify certain formulas, such as the importance weight in Line 144 and the estimand $\\tau^*$ in Line 15? How are these formulas derived—are they based on existing methods or newly developed? Since the reviewer is not familiar with this field, the appearance of these formulas feels quite abrupt. If these formulas are based on existing methods, it would be helpful to indicate that.", "experimental_designs_or_analyses": "How does the accuracy of isolated effects estimated, using the proposed metrics such as confidence intervals, fidelity, and overlap, perform in more complex situations, such as when Y and $a(X)$, $a^c(X)$ exhibit a nonlinear relationship or when $a(X)$ is not a simple binary variable? Additionally, I noticed that in Figure 3, the confidence intervals of the proposed method on the SvT dataset are quite large, with only the SenteCon-Empath case coming close to the true isolated effect. Does this suggest that the method may not be robust enough across different tasks?", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "The reviewer is not familiar with the relevant work in this field and therefore cannot provide further comments.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AcHFo2807f", "reviewer": "Reviewer_sf2t", "summary": "The paper introduces a framework for estimating isolated causal effects of language, which focuses on how specific linguistic attributes influence external outcomes while controlling for non-focal language to mitigate OVB. It uses doubly robust estimators to ensure unbiased estimations. The authors define three metrics (fidelity, overlap, and robustness value) to assess estimation quality. Experiments on a semi-synthetic dataset and a real-world dataset show that SenteCon-based representations yield the most reliable estimates, LLM-Prompting is effective but costly, and MPNet embeddings suffer from overlap violations. The study highlights the fidelity-overlap tradeoff to show the importance of balancing representations for robust causal inference for texts.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! We appreciate your positive remarks describing our work as theoretically and empirically well-supported, experimentally sound, and enhancing the rigor of text-based causal inference. We address further comments and questions below.\n\n**[Experiments with additional LM embeddings and dimensionality reduction]** We include additional results ([anonymous link](https://naturl.link/extra-embeds)) on the SvT dataset using embeddings from 3 common LMs: BERT, RoBERTa, and MiniLM. We find that the 3 new LM embeddings improve upon the previous MPNet over all metrics but still do not match SenteCon-Empath.\n- The MiniLM effect point estimate is almost as close to the true effect as SenteCon-Empath, but its robustness value remains middling due to poorer overlap. The BERT and RoBERTa point estimates are only slightly better than MPNet, and their robustness values are lower than MiniLM.\n- BERT and RoBERTa have much better overlap than either MPNet or MiniLM, but worse fidelity. Since BERT and RoBERTa are higher-dimensional than MiniLM, this is surprising but may be due to optimization of MiniLM for sentence-level tasks in the sentence-transformers library. Interestingly, this result suggests the fidelity-overlap tradeoff involves more than just the dimensionality of the representation.\n\nWe also include the results of dimensionality reduction using SVD on the MPNet and RoBERTa embeddings, which are the two most complex representations. \n- For both representations, dimensionality reduction improves both the robustness value and the point estimate, highlighting the potential of data-based post-processing of representations. \n- In fact, both point estimates move from outside the true effect confidence interval to inside the true effect confidence interval after SVD.\n\nWe thank you for suggesting these experiments and look forward to including them in the paper.\n\n**[Overlap assumption]** It is true that overlap violations can be a concern in language settings (D’Amour et al. 2017). As we mention in our paper, high-dimensional representations with good fidelity are prone to overlap violations, and so we rely on our OVB metrics, overlap and robustness value, to warn us when such violations may be occurring. In this paper, we assume that strict overlap holds (i.e., that there is *some* non-zero chance of each $a(X)$ condition given $a^c(X)$). In our experiments, we have one result that suggests a “soft” overlap violation (i.e., the probability of one $a(X)$ condition is very small but *not* non-zero given $a^c(X)$): the MPNet result on SvT, where the overlap metric is fairly large. We also have a negative overlap value for Empath on the same dataset; however, for this metric we use a doubly robust estimator, and doubly robust estimates do not necessarily satisfy criteria like being non-negative in noisy settings. Therefore, this negative overlap value (which is small in magnitude) may not necessarily be due to an overlap violation.\n\n**[Representations for balancing fidelity-overlap tradeoff]** We agree that the fidelity-overlap tradeoff naturally suggests learning representations that optimize the tradeoff. Experiments that achieve this require further theoretical analysis and empirical work and are outside the scope of this paper—this is in fact the basis of ongoing research—but we are happy to expand on this topic when discussing future work in our conclusion.\n\n**[Exact outcome model]** For Amazon, the outcome model is a linear regression model of the semi-synthetic $Y$ fit over the LIWC categories that form $a(X)$, $a^c(X)$. For SvT, the outcome model is a gradient boosting classifier fit over $a(X)$ and each named non-focal language representation.\n\n**[Non-synthetic Amazon experiments]** We did not use the original Amazon data because the true causal effects of the text interventions are unknown, so there is unfortunately no way to validate effect estimates on this data. The SvT dataset allowed us to validate our methods in a natural setting in which the true effect *was* known through an accompanying external clinical trial. Since such trials are highly resource-intensive, for Amazon we rely on the more common practice of obtaining true effects by generating a semi-synthetic dataset (Veitch et al. 2020; Pryzant et al. 2021). If you are interested in experimental results from a more complex semi-synthetic Amazon dataset, please see our response to reviewer gbt6 titled [Nonlinear $Y$, $a(X)$, $a^c(X)$ relationship].\n\nWe hope that our response addresses any concerns you may have and that you will consider revising your score. Thank you again!\n\n---\nD’Amour et al. \"Overlap in observational studies with high-dimensional covariates.\" J. Econom. 221.2 (2021): 644-654\n\nPryzant et al. \"Causal Effects of Linguistic Properties.\" NAACL 2021\n\nVeitch et al. \"Adapting text embeddings for causal inference.\" UAI 2020", "claims_and_evidence": "I think most claims are well-supported by theoretical and empirical evidence.  This tradeoff concept is valid. But this phenomenon may be general in standard causal inference and classical machine learning where more features can hamper overlap. They do put it into a text-specific lens.", "methods_and_evaluation_criteria": "I think the doubly robust is a good choice for this task. Generally, the methods and evaluation make sense for the problem.", "theoretical_claims": "I checked equations in 3.1 and 3.2 and briefly looked at Appendix A. I am not aware of any issues.", "experimental_designs_or_analyses": "The experiment design is sound. It is expected that using SenteCon yields the most balanced results as it is a mixture of discrete and continuous embeddings. Adding more experiments using textual embeddings from common LMs might enhance persuasiveness.", "relation_to_broader_scientific_literature": "This paper builds on some recent research like codebook functions (Egami et al., 2022), natural text experiments (Fong & Grimmer, 2023), and methods that integrate textual features with causal inference (Pryzant et al., 2021). Unlike prior studies that estimate the effect of a focal text feature while allowing correlated text attributes to influence outcomes, this work aims to isolate a binary linguistic attribute by controlling for the nonfocal text distribution. It also extends OVB analysis in high-dimensional settings by introducing fidelity and overlap metrics. These contributions enhance the rigor of text-based causal inference by providing new evaluation metrics and sensitivity checks.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "This is not a big issue, but I have some concerns about the overlap assumption mentioned in the problem setting. For simple tasks, such as binary intervention and low-dimensional focal representation, positivity is easier to achieve. However, this may not hold for more complex tasks, and it might be related to the scale of the data. The subsequent experiments also show that overlap is violated under certain conditions.", "other_comments_or_suggestions": "- Based on the framework of this paper, researchers may expect deeper discussion or more varied experiments of how some representation learning methods can be specifically designed for text tasks to balance the tradeoff (for future work). \n- The authors could try some dimension reduction methods for high-dimensional language embeddings, as it is a common approach for handling text embeddings. Reducing dimensionality within a dataset might capture unique non-focal features specific to the data.", "questions_for_authors": "- What is the exact outcome model?\n- Did you try non-synthetic Amazon experiments? Is the original data too random?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UgR4I6YIrS", "reviewer": "Reviewer_ZnfT", "summary": "Effects of the attribute of a text on an outcome can be influenced by the surrounding linguistic context around this attribute. This motivates defining an \"isolated causal effect of this attribute (the \"focal\" text), where the surrounding context is marginalized (the \"non-focal\" text). A framework for estimation of this treatment effect is introduced, with assumptions, identification, a doubly-robust estimator and a sensitivity analysis. Notably, confidence intervals for the estimated isolated effect typically include the ground-truth isolated effect, in stark contrast to the vanilla natural effects from the literature.\n\n(EDIT : updated score based on authors' rebuttal)", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive remarks describing our work as interesting, theoretically correct, and largely well-justified. We address questions and concerns below.\n\n**[Why isolated effects vs. natural effects?]** To illustrate why isolated causal effects are important for language, consider the effect of misinformation in online posts on readers’ voting decisions. Texts containing this type of misinformation also often contain other attributes likely to influence voting—e.g., politically inflammatory messaging. To determine whether action needs to be taken against misinformation, we would need to isolate its effect from the effects of other correlated attributes.\n\nSimilarly, in scientific applications, researchers are also frequently concerned with the isolated effect and conduct randomized clinical trials to eliminate effects of potential confounders. In our experiments on the SvT dataset to estimate effects of weight loss medication, we use text data from social media posts, which often encode additional information related to weight loss like users’ exercise habits and diet. We find—as you highlight—that the isolated causal effect estimate from SenteCon-Empath corresponds to the ground truth from real-world clinical trials, while the natural effect estimate does not.\n\nThis problem, also known as *aliased treatments*, is an important problem for social scientists studying the effects of language, and prior to our work the proposed solution was to carefully design a text experiment that constructs artificial texts that are not aliased (Fong & Grimmer, 2023). Our work expands the scope of possible research by allowing scientists to use naturally occuring text to study these kinds of effects instead of relying on resource-intensive text experiments with artificial data.\n\nWe will include this additional motivation in the introduction and in the discussion of the SvT results.\n\n**[Novelty of proposed framework]** Treatment effect estimation with doubly robust estimators is of course well established in the causal inference literature. A main contribution of our paper is to conceive of, formalize, and parameterize natural language—a complex, unstructured, and high-dimensional form of data—in a way that allows us to use classical causal frameworks like doubly robust effect estimation, which are designed for much simpler forms of data. Indeed, leveraging classical causal estimators to solve new problems in new settings is an active area of research (e.g., Dudik et al. 2011; Schnabel et al. 2016; Azizzadenesheli et al. 2019; Byrd & Lipton 2019; Kallus et al. 2022). Our work introduces the concepts of isolated causal effects and non-focal language, and we define a new problem formulation, estimand, and practical estimation framework for these causal effects. Once language has been formalized in this way, we see the ability to draw on well-studied tools that already exist for estimating robust causal effects as a strength.\n\nLikewise, once language has been parameterized as focal and non-focal components, our OVB estimators indeed naturally follow the work on bounding OVB of Chernozhukov et al. (2024), which presents modern tools for this classical problem. Our main contribution here is to draw the connection between language representation and OVB, which to our knowledge has not previously been articulated. We explore the idea that language representation is lossy specifically in a way that is detrimental to isolated causal effect estimation, and we link that information loss to OVB. We use this connection to introduce OVB-based metrics for evaluating the sensitivity of isolated effect estimates to language representation that are distinct from traditional NLP methods for measuring information loss.\n\n**[Clivio et al. papers]** We will refine the language in our discussion of these references.\n\n**[$C_Y$ and $C_D$]** We originally left the mathematical definitions of $C_Y$ and $C_D$ in Appendix C.4 due to space constraints but will move them to the main paper.\n\n**[Robustness value]** Robustness in causal inference refers to the stability of the effect estimate under potential errors or omissions in modeling, assumptions, and/or data. Our robustness value measures how much OVB can be present before the point estimate of the causal effect changes from the correct to the incorrect sign. A larger robustness value indicates that the effect estimate better tolerates omission of key variables, suggesting that it is stable and therefore robust. We will emphasize this intuition when we define the robustness value in Section 3.3.\n\nWe hope that our response addresses any concerns you may have and that you will consider revising your score. Thank you again!\n\n---\nAzizzadenesheli et al. \"Regularized Learning for Domain Adaptation under Label Shifts.\" ICLR 2019\n\nDudík et al. \"Doubly robust policy evaluation and learning.\" ICML 2011\n\nSchnabel et al. \"Recommendations as treatments: Debiasing learning and evaluation.\" ICML 2016", "claims_and_evidence": "I find that while the framework is interesting, there lacks justification for *why* these isolated effects should be considered, in contrast to the natural effect. In the Amazon dataset, the estimator recovers the true isolated effect than the naive method that estimates the natural effect : sure, but the isolated effect was defined as such in the semi-synthetic dataset and the first estimator was inherently designed to estimate it. However, it is interesting to see that the true causal effect on the SvT dataset is the result of an RCT and the estimator developed by authors recovers it unlike the naive method. This suggests that the RCT-given causal effect is the isolated effect and not the natural effect. Thus, if I did not miss anything, I find that authors should expand on this and include more justification (which could be simple citation of previous literature) on why the isolated effect matches such a \"ground truth\" effect, unlike the natural effect.\n\nOther than that, all other claims seem well justified.", "methods_and_evaluation_criteria": "Yes, they seem natural.", "theoretical_claims": "I checked the proofs and they look correct.", "experimental_designs_or_analyses": "The analysis in 5.2 seems to match the practice of the Chernozhukov et al. (2024) reference in the paper.", "relation_to_broader_scientific_literature": "I am not familiar with causal inference for text part, but to the best of my knowledge, all the relevant literature on the treatment effect estimation and OVB part is cited.\n\nOne caveat however: the information loss and the confounding error from the two Clivio et al. (2024) references seem to refer to the same quantity.", "essential_references_not_discussed": "I'm not aware of such missing references", "other_strengths_and_weaknesses": "Strengths :\n\n- The paper is generally clear.\n- The perspective is interesting.\n\nWeaknesses :\n\n- There might be a lack of novelty in the treatment effect estimation part, as I do not see how the proposed framework differs from previous work in treatment effect estimation if one posits $T' = a(X)$, $X' = a^c(X)$, $Y'(t) = Y(T'=t, X')$ and performs classical treatment effect estimation, including OVB from Chernozhukov et al. (2024) using $X'$ as covariates, $T'$ as treatment, $Y'(t)$ as potential outcomes. Notably, if I understood correctly, $a(X)$ and $a^c(X)$ are not estimated but given, although $a^c(X)$ is too high-dimensional for direct usage in estimation, thus \"covariates\" $X'$ and \"treatments\" $T'$ are given as in any vanilla treatment effect estimation task.", "other_comments_or_suggestions": "- IMO $C_Y$ and $C_D$ should be defined in the main text.\n\n- Authors might include more justification on why these robustness values actually measure... robustness.", "questions_for_authors": "I refer to my two main concerns:\n\n(1) Can you include a justification for why the proposed isolated causal effects are more appropriate than natural effects, as outlined in the \"Claims and Evidence\" section?\n\n(2) Can you explain how your framework differs from vanilla treatment effect estimation, as outlined in the \"Other Strengths And Weaknesses\" section?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a framework for estimating isolated causal effects of language—effects attributable to specific, focal linguistic interventions while controlling for other, non-focal textual content. Grounded in the principle of omitted variable bias (OVB), the paper formalizes the problem setup, applies doubly robust estimation techniques, and proposes diagnostic measures (fidelity, overlap, and robustness value) to assess the sensitivity of effect estimates to imperfect representation of non-focal language. Through experiments on both semi-synthetic Amazon data and a real-world dataset aligned with clinical trial outcomes, the authors demonstrate that their framework can recover isolated effects more faithfully than standard natural effect estimators. 

While the technical novelty may not be particularly strong, the framework effectively bridges language modeling, dimensionality reduction, and OVB-based sensitivity analysis, and reviewers appreciated the comprehensive evaluation and strengthened empirical results provided during the rebuttal. Overall, the paper presents a well-motivated and thoughtfully constructed framework that is likely to inspire further work at the intersection of language and causal inference. All reviewers were satisfied with the authors' clarifications and raised their scores.
Decision: Accept (poster)