ICML 2025

Isolated Causal Effects of Natural Language
Victoria Lin 1 Louis-Philippe Morency 1 Eli Ben-Michael 1
Abstract
As language technologies become widespread, it
is important to understand how changes in lan-
guage affect reader perceptions and behaviors.
These relationships may be formalized as the
isolated causal effect of some focal language-
encoded intervention (e.g., factual inaccuracies)
on an external outcome (e.g., readers’ beliefs).
In this paper, we introduce a formal estimation
framework for isolated causal effects of language.
We show that a core challenge of estimating iso-
lated effects is the need to approximate all non-
focal language outside of the intervention. Draw-
ing on the principle of omitted variable bias, we
provide measures for evaluating the quality of
both non-focal language approximations and iso-
lated effect estimates themselves. We find that
poor approximation of non-focal language can
lead to bias in the corresponding isolated effect
estimates due to omission of relevant variables,
and we show how to assess the sensitivity of ef-
fect estimates to such bias along the two key axes
of fidelity and overlap. In experiments on semi-
synthetic and real-world data, we validate the abil-
ity of our framework to correctly recover isolated
effects and demonstrate the utility of our proposed
measures.
1. Introduction
The widespread use of language technologies has given
rise to an ever-expanding amount of human- and machine-
generated text data. From this vast body of data emerges
the opportunity to understand how information contained in
language relates to real-world outcomes. Elucidating these
relationships can help answer scientifically interesting ques-
tions and provide interpretability to texts and the models
that generate them. For instance, what language attributes
1Carnegie Mellon University, Pittsburgh, PA, USA. Correspon-
dence to: Victoria Lin <vlin2@andrew.cmu.edu>.
Proceedings of the 42 nd International Conference on Machine
Learni...