Text-Transport: Toward Learning Causal Effects of Natural Language
Authors: Victoria Lin
Louis-Philippe Morency
Eli Ben-Michael
Venue: EMNLP 2023 Main (2023)
Keywords: causal inference
causal effects
distribution shift
domain adaptation
transportability

Abstract: As language technologies gain prominence in real-world settings, it is important to understand *how* changes to language affect reader perceptions. This can be formalized as the *causal effect* of varying a linguistic attribute (e.g., sentiment) on a reader’s response to the text. In this paper, we introduce Text-Transport, a method for estimation of causal effects from natural language under any text distribution. Current approaches for valid causal effect estimation require strong assumptions about the data, meaning the data from which one *can* estimate valid causal effects often is not representative of the actual target domain of interest. To address this issue, we leverage the notion of distribution shift to describe an estimator that *transports* causal effects between domains, bypassing the need for strong assumptions in the target domain. We derive statistical guarantees on the uncertainty of this estimator, and we report empirical results and analyses that support the validity of Text-Transport across data settings. Finally, we use Text-Transport to study a realistic setting—hate speech on social media—in which causal effects do shift significantly between text domains, demonstrating the necessity of transport when conducting causal inference on natural language.

TEXI-TRANSPORT: Toward Learning Causal Effects of Natural Language
Victoria Lin
Louis-Philippe Morency
Carnegie Mellon University
Carnegie Mellon University
vlin2@andrew. www.2anndee..mu..coo cmu.edu
morencyêcs cmu edu
Eli Ben-Michael
Carnegie Mellon University
entemmichael@deee edu
Abstract
linguistic attributes or topics make text soem hatefal
response
As language technologies gain prominence in real-wor woaldwrldd settings, is important under- stand how changes to language affect reader perceptions. This can formalized as causal effect of varying linguistic attribute (e.g., sentiment) on a reader's response to the text. In this paper, we introduce TEXT- TRANSPORT, method for estimation of causal effects from natural language under any text dis- tribution. Current approaches for valid causal effect estimation require strong assumptions about the data, meaning the data from which estimate valid causal effects often is not representative of the actual target domain of interest. To address this issue, leverage the notion of distribution shift to describe estimator that transports causal effects between domains, bypassing the need strong assump- tions in the target domain. We derive statistical guarantees on the uncertainty of this estimator, and report empirical results and analyses that support the validity of TEXT- TRANSPORT across data settings. Finally, we use TEXT- TRANSPORT to study realistic setting hate speech on social media-in which causal ef- fects do shift significantly between text mains, demonstrating the necessity of transport when conducting causal inference on natural language.
Is
profanity
netspeak
(lack insight
familyy
attiibute
Causal
profonity
+
profanity
Text-Transport
fnmilyy
family
hateful
heleful
heteful
hateful
source (Reddit)
target (Gab)
Figure TEXT- TRANSPORT facilitates estimation of text causal effects on any target domain by transporting causal effects from a source domain.
understand how changes to language affect reader perceptions--formaiizee in statistical terms the causal effect of varying a linguistic attribute reader's response the text (Figure 1).
A core technical challenge for causal inference is that valid causal effects can only be estimated on data in which certain assumptions are upheld: namely, data where confounding is either fully mea- sured absent entirely (Rosenbaum and Rubin, 1983). Since confounding-tte the presence of factors that affect both the reader's choice of texts to read and the reader's response (e.g.. political affiliation. age, mood) extremely difficult to measure fully in text settings, estimation of causal effects from natural language remains open problem. One resource-intence approach is to run a randomized experiment, which climinates confounding by suring that respondents are randomly assigned texts to read (Holland, 1986: Fong and Grimmer, 2021). However, effects estimated from randomized exper- iments may not generalize outside of the specific data on which they were conducted (Tipton, 2014;
1 Introduction
What makes comment on social media site seem toxic or hateful (Mathew et al., 2021; Guest et al., 2021>? ? Could it be the use of profanity, or a lack of insight? What makes a product review more or less helpful to readers (Mudambi and Schuff, 2010; Pan and Zhang, 2011)? Is it the certainty of the review, perhaps the presence of justifica- tions? As language technologies are increasingly deployed in real-world settings, interpretability and explainability natural language processing have become paramount (Rudin, 2019; Barredo Arrieta et al., 2020). Particularly desirable is the ability to

---

Bareinboim and Pearl, 2021). Therefore, learning causal effects for new target domain can require a new randomized experiment each time.
bution P is given by:
N
((P)
Ex~P|X(())
N 1=1
In this paper, propose to bypass the need for strong data assumptions in the target domain by framing causal effect estimation as a distribution shift problem. We introduce TEXT-RRANPPORT, a method for learning text causal effects in any tar- get domain, including those that do not necessarily fulfill the assumptions required for valid causal in- ference. Leveraging the notion of distribution shift, we define causal estimator that transports causal effect from causally valid source domain (e.g.. randomized experiment) to the target domain, and we derive statistical guarantees for our proposed estimator that show that has desirable properties that can be used to quantify its uncertainty.
(1)
Σ 1 TEN
Σ (())
Now let X be parameterized as X aX)),, where a(X) is the text attribute of interest and a (X() denotes all other attributes of the text X. Note that for a text causal effect be meaningful, a (X) must be interpretable. This may be achieved by having a human code a(X) or using a lexicon or other automatic coding method. Again for simplicity, we assume a(X) E {0, 13.
Definition 1 (Natural causal effect of an attribute). Let Pi (X) be distribution such that a( aXX) = and (X) PX))(X)a((X) = 1), and let PX be a distribution such that a(X) and ot (X) (X)) (X) = 0).
We evaluate TEXT-TRANSEE TRANSPORT empirically us- ing a benchmarking strategy that includes knowl- edge about causal effects in both the source do- main and the target domain. We find that across three data settings of increasing complexity, TEXT- TRANSPORT estimates are consistent with effects directly estimated on the target domain, suggesting successful transport of effects. We further study realistic setting user responses hateful content on social media sites-in in which causal effects do change significamtly between text domains, demon- strating the utility of transport when estimating causal effects from natural language. Finally, we conduct analyses that examine the intuition behind why TEXT-TRAN TRATTRRNNSORTT is an effective estimator of text causal effects.
Then the causal effect of a(X) on Y is given by:
- =(()) ((Po)
(2)
Here, T" is the natural effect of a(X). Lin- guistic attributes are subject to aliasing (Fong and Grimmer, 2021), in which some other linguistic attributes (e.g., the k-th linguistic attribute a (X)))) may be correlated with both the linguistic attribute of interest a(X) and the response Y. such that ((aa(X)xlad
1) + (()((a)))((X) For example, optimism may naturally co-occur with positive emotion, meaning that the natural effect of optimism also contains in part the effect of posi- tive emotion. In contrast, the isolated effect would contain only the effect of optimism. In this pa- per, we choose to focus natural effects due to the way linguistic attributes manifest in real-world language. That is, since optimism nearly always co-occurs with positive emotion, may be difficult to interpret the effect of optimism when positive emotion is removed (the isolated effect), so we instead focus on their collective natural effect.
0).
2 Problem Setting
Consider collection of texts (e.g., documents, sentences, utterances) X, where person (i 1,...,N) is shown text X; from X. Using the potential outcomes framework (Neyman, 1923 [1990]; Rubin, 1974), let (x) denote the potential response of respondent i they were to read text x, where Y; : X' + R. Without loss of generality, assume that each respondent in reality reads only one text X,, so their observed response is (x.))..
Our goal is then to learn TT, the natural causal effect of the attribute a( X) on response Y in the text domain of interest PT, We consider use cases where it is not possible to directly estimate the effect from target data X 'PTT, either because PT does not fulfill the assumptions required for valid causal inference or simply because the response is not measured the domain of interest.
Then the average response (PP) across the N respondents when texts X are drawn from distri-

---

3 TEXI-TRANSPORT
3.2 Importance weight estimation
To estimate causal effects under target text distribution withwthhut computing effects on p' directly we propose TEXI-TRANSPORT, a method for transporting a causal effect from a source text distribution pR that does fulfill the as- sumptions required for valid causal inference and with respect to which pT is absolutely continuous, Our approach can help generalize the causal find- ings of PR, which are specific to the data domain of pR, to any text distribution of interest p" For mathematical convenience, we consider the source distribution pi to be a randomized experiment. We note that any crowdsourced dataset in which samples are randomly assigned to crowdworkers can be considered randomized experiment.
Estimating the transported response fu( ((P)) first requires computing either the derivative diZ or the individual probabilities PR (X()), PT) (X). While there are many potential ways estimate this quantity, we propose the classification ap- proach TEXT-TRANSPORTaR and the language model approach TEXT-TRANSPORTA (Figure 2). The classification ap- proach for estimating Je X) relies on the notion that the density ratio can be rewritten in a way that makes estimation more tractable. Let C' denote the distribution (or corpus) from which a text is drawn, where C = T denotes that it is drawn from p" and C R denotes that it is drawn from PR. Then de
de (X)
TEXT- TRANSPORRT
d (X) can be rewritten as follows:
3.1 Transporting effects
dPT
P(C R) P(C==) P(C = R|X) (5)
We characterize this problem as an instance of dis- tribution shift, allowing to define causal effect estimator fJ that uses the density ratio between distributions an importance weight trans- port the effect from pR pJ, Given X; pR. d' and letting dew (I)
dPR (X)
P(C = T and P(C = R|X) can be esti- mated by training binary classifier Ma X -> [0, 1 to predict if text X came from T or R. P(C R) and P(C T) are defined by their sample proportions (i.e., by their proportion of the total text after combining the two corpora).
(xr be the density ratio! (*())
between pf and PR.
i(P")
JPR (X))
Because language models are capable of learning text distributions, are able to take alternative estimation ap- proach that does not require learning language model trained on samples from pll or pT, for instance, can compute the probability of texts under the learned distributions.
TEXT-TRANSPORTE
nt
which gives us the effect estimate under pr;
T -PP) -P())
(4)
dPT de (X). A
Intuitively, as all observed texts are drawn from PR, the role of the importance weight is to increase the contribution texts that are similar to texts from pf, while reducing the contribution of texts that are representative of p That is, if (X;) is high and p"(x)) is low, then X, will have greater contribution to ((PT). To highlight this transport of pM PT, in the remainder of this paper we will refer ji(PT) as R-I.
Pre-trained large language models (LLMs) are particularly useful for estimating f and pr since their training corpora are large enough to approxi- mate the distribution of the English language, and their training data is likely to have included many plt or p'l of interest. Following recent advances in LLMs, one way of obtaining (R(X) and fT (X) from an LLM is to prompt the LLM in a way that induces it to focus on På or pT Once the LLM has been prompted toward either plt or PT sen- tence probabilities from the LLM can be treated as reflections of p*(X) or p"(X)) respectively. We provide examples of such prompts in Figure 2, and explore this approach in our experiments.
strength of this estimator is that we are able to quantify statistical uncertainty around the causal ef- fect. We demonstrate (with derivations and proofs in Appendix B) that A R-T has a number of desir- able properties that allow us to compute statistically valid confidence intervals: (I) it is an unbiased es- timator of p(P"")). (2) it is asymptotically normal, and (3) it has closed-form variance and unbi- ased, casy-to-compute variance estimator.
PIn practice, rather than use the ratios d= deR (X) )) (X)),)
pR(X(
Moore generally, this is the Radon-Nikodym derivative, formally defined in Appendix A.
directly, we use stahilized (í.c., normalized) versions that can- cel out the sample proportions. See Appendix C.

---

TEXT-TRANSPORTet
TEXT- EAX-TTRNSPORTL
of your when we give thanks for the wiemth and love thai suroonnds
It was lic, but we noeded the information.
This yeur we give thinks for the warmch and love that surrundds
It was bi, but we noeded the information.
p
pz
pR
P7
surplo
sptement with
dajerrent wilf
x Kg) pa
Xx, Xg
LLM
LLM
Classifler
promps
promett
trah
Tain
-
(xxx (M)
(
P T " PIC PIC RX
PCC - R)
di
Figure Two proposed approaches for estimating importance weights.
4 Experimental Setup
pit
pft
We conduct empirical evaluations to assess the validity of causal effects obtained through TEXT- TRANSPORT in three different data settings of in- creasing complexity (Table 1).
Amazon
Sound quality is likewise decent...
This printer being an all one, serves several functions...
EmoBank
have become more open- minded, more responsible...
I've even tried rewriting the cor- rupted sections...
4.1 Evaluation methodology
To assess the validity of TEXT- TRANSPRR,,,, we conduct experiments comparing the transported average response R- with jt the average re- sponse under PR, and ji the average response under PT_ A valid transported response R-T will be similar to ji" and significantly different from f.. To quantify these differences, we compare es- timated averages and 95% confidence intervals, as well as normalized RMSE (RMSE divided by the standard deviation of the target response) between R-T and A.=
Hate Speech
Your reply is complete non-sequitur
You really are trained little monkey and don't even know it.
Table 1: Source (PR) and target (P²) distributions for each evaluation dataset.
We choose three crowdsourced datasets, parti- tion each into pR and PT and compute i. fi" and "t''"" We estimate confidence intervals for each average response through bootstrap resam- pling with 100 iterations.
As we mention previously, validating transported causal effects requires an evaluation strategy in which causal effects under pR and pr are both known. Therefore, each of our evaluation datasets consists of a single crowdsourced dataset that can be divided into parts (e.g., сorpus of movie re- views can be split by genre), such that they possess the following properties. First, allow R and ATT to be computed directly, both pR and pT are randomized experiments (we reiterate that pl is a randomized experiment only for the purposes of validation; we would not expect actual p' to be randomized). Second, the response Y is measured for both P and PT. Third, p and pT are dis- tinct in way where we would expect the average response differ between the two. This allows us evaluate whether transport has successfully oc- curred (if the average response the same between plt and pf, transport will not do anything).
Baselines. While a small number of prior stud- ies have proposed estimators of some type of text causal effect from observational (i.e., non- randomized) data, effects obtained from these meth- ods are not directly comparable to those obtained using TEXT-TRANSPOR (further discussion of these methods can be found in Section 6). How- ever, rather than using the density ratio to transport effects, other transformations of the source distri- bution to the target distribution possible. One intuitive baseline train predictive model the source distribution, which is then used to gener- ate pseudo-labels on the target distribution. These pseudo-labels can be averaged to produce a naive estimate (the naive baseline).

---

4.2 Datasets
4.3 Implementation
Linguistic attributes. To automatically obtain lin- guistic attributes a( from the text, we use the 2015 version of the lexicon Linguistic Inquiry and Word Count (LIWC) to encode the text as lexical categories (Pennebaker et al., 2015). LIWC is a human expert-constructed lexicon- generally viewed with a vocabulary of 6,548 words that belong one or more of its 85 categories, most of which are related to psychology and social interaction. We bi- narize the category encodings to take the value I if the category is present in the text and 0 otherwise. TEXT-TRANSPORTAR
Amazon (controlled setting). The Amazon dataset (McAuley and Leskovec, 2013) consists of text reviews from the Amazon e-commerce website, where the reader response is the number of "helpful" votes the review has received. We choose reviews of musical instruments as pf and reviews of office supplies as pT.
gold standard for lexicons-
To construct a best-case scenario in which there are no unmeasured factors in the data, we gener- ate a new semi-synthetic response Y by predict- ing the number of helpful votes al aXX), (X). We use a noisy version of this predic- tion as our new Y. This ensures that all predictable variability the response Y is captured the text. Furthermore, we sample reviews into pR and pr according to their predicted likelihood of being in P or pr when accounting only for a a(X), a This provides controlled data setting in which we know that model is capable of distinguishing between pi and pf, such that can evaluate TEXT-TRANSPORT under best-case conditions.
function of
We use the follow- ing procedure to implement TEXT-TRANSHE TRANSPORTTs First, consider data Di from pit and data Dr from PT We take 10% of DR and 10% of Dy our classifier training set Dirain. Next, we train a classifier Mg on Ptrain to distinguish between pit and p" For our classifier, we use embeddings from pre-trained MPNet (Song et al., 2020), a well- performing sentence transformer architecture, inputs to logistic regression.
From Mg. can obtain P(C = TIX) and P(C X) for all texts X in the remaining 90% of DR. We compute P(C R) and P(C T) as
EmoBank (partially controlled setting). The EmoBank dataset (Buechel and Hahn, 2017) con- sists of sentences that have been rated by crowd- workers for their perceived valence Y (i.c., the positivity or negativity of the text). To construct pR, we sample sentences such that texts with high writer-intended valence occur with high probabil- ity, and construct pT, we sample sentences such that texts with low writer-intended valence occur with high probability. This partially controls the data setting by guaranteeing that the source and target domains differ on single atribbue--riter- intended calence-thal is known to us (but hidden from the models).
respectively. Then we have and DrI
D
( dP)
P(C= T[X) |Prl P(C R| |Dal
(6)
In the case of the Amazon dataset, we note that although we can estimate the classification prob- abilitics as P(C = T|X), PCC = R|X), the truc probabilities are already known, as we use them to separate texts into p and pT Therefore-te evaluate the effectiveness of TEXT-TRANSPORT under conditions where we know the classifier to be correct-we use the known probabilities P(C TIX), R|X) in our evaluations on the Amazon dataset only.
Hate Speech (natural setting). The Hate Speech dataset (Qian al., 2019) consists of comments from the social media sites Reddit and Gab. These comments have been annotated by crowdworkers as hate speech or not hate speech. The Reddit comments are chosen from subreddits where hate speech is more common, and Gab is a platform where users sometimes migrate after being blocked from other social media sites. To represent real- istic data setting in which we have no control over the construction of the source and target distribu- tions, we treat the corpus of Reddit comments as pit and the corpus of Gab comments as pt
TEXI-TRANPPORTLA This approach can be implemented without any training data, leaving the full body of text available for estimation. In our experiments, we estimate p*(X) and pXX through prompting. For each dataset, we provide pre-trained GPT-3 with prompt that describes På or PT, then follow the prompt with the full text from each sample X pP On the EmoBank dataset, for instance, we provide GPT-3 with the prompts "You are writing a positive statement" (for plt the high-valence distribution) and "You are

---

source transported - target
Amazon
naive
LM
clf
1.1
1.2
1.3
1.4 EmoBank
1.5
1.6
1.7
naive
LM
clf
2.8
3.0
3.2
Hate Speech
naive
LM
clf
0.15
0.20 response
0.25
Figure 3: Validation of transported responses R- against known target responses ji and source responses jt
Amazon
EmoBank
Hate Speech
Naive baseline
0.073
0.903
0.351
TRANSORR
0.019
0.832
0.257
TEXT-TRANSPORTIM
0.035
0.378
0.943
Table 2: Normalized RMSE of transported responses R-T against known target responses A²"
writing negative statemen"" (for pT, the low- valence distribution). A full list of prompts can be found in Appendix D.3.
sults, our transported response RR is statisti- cally significantly different from jt while being statistically indistinguishable from fiP In contrast, the naive baseline produces an estimate with confi- dence intervals that overlap both f and jt" and its RMSE is higher than both TRANSPAAT esti- TEANSOOA ai in this
After prompting the model with either pR or PT, we compute the token probabilities over each X, then compute sentence probabilities as the prod- uct of the token probabilities. If text has multiple sentences, treat the average of the sentence prob- abilities the overall text probability. Finally, (X) importance weight. compute the ratio ()(X)
mates. The success setting suggests that if the classifier is known to be good estimator of the probabilities P(C X) and P(C Rt. X). the transported estimates will be correct. The success of TEXT-TRANSPORTLM in this setting, on the other hand, suggests that prompting GPT-3 can in fact be an effective way of estimating p px( and pT (X) and that the ratio between the two can also be used to produce valid transported estimates.
5 Results and Discussion
5.1 Validity of TEXT-TRANSPORT
We evaluate the validity of our TEXI-TRANSPORY responses on the Amazon, EmoBank, and Hate Speech data settings (Figure 3, Table 2).
We further find that as the data setting becomes less controlled (i.e., EmoBank and Hate Speech), our transported responses continue to show en- couraging trends-that is, the transported effect
We observe that on the Amazon dataset, both TEXT- TRANSPRRRN and TEXI-TRANSPORTLM are well-validated. For both sets of Amazon re-

---

indeed moves the responses away from fi and toward ji', while transported responses from the naive baseline exhibit little to no movement toward the target. When evaluating TEXT- LRANSPORTTM on EmoBank, ft >?" and j have no statistically significant difference. However, in evaluations of TEXT-TRANSPORTAlr we find that RR though transported in the right direction- retains statistically significant difference from ji"; and when evaluating SEXT-TRANSPORTTL on Hate Speech, we observe wide confidence intervals for A R-T that cover both ji" and j.,, though the point estimates of j R-T and j are very close.
5.2 A realistic use case: What makes a comment hateful?
source
transported
target
profanity
netspeak
insight
function
family
Finally, note that TEXT- TEXI-TRANSPORTLY is less stable than TEXT-TRANSPORT.l with respect the width of its confidence intervals, although the transported point estimates are better. This is particularly highlighted by the higher RMSE of TEXT-TRANSER TNASPORTI. compared to the naive baseline on the Hate Speech dataset, in spite of TEXI-TRANSPORTIM much better point es- timate." 3 We posit that the instability of TEXT-TRANSPORTLX due to the very small probability of any particular text occurring under a given probability distribution, as well as potential lack of precision introduced when using prompting to target an LLM to a specific distribution. We observe that both (X) and (X) are typically both very small, and any difference between them-while minute in absolute terms -i amplified when taking their ratio. As a result, [T(X) under
-0.2
0.0
0.2
0.4
effect
Figure 4: Source and transported natural causal effects from Reddit Gab.
In this section, we highlight a realistic use case of TEXT-TRANSPORT. Taking our Hate Speech dataset, we examine the source, transported, and target effects of five linguistic attributes, where Reddit is again the source distribution and Gab is the target distribution (Figure 4). Transported effects are estimated using TEXI-TRANSPORIE
We find that the causal effects of these five lin- guistic attributes estimated dircetly on Reddit differ significantly from their counterparts that have been transported to Gab. Though we would not have access to effects estimated directly on the target distribution in a real-world setting, we are able to validate here that the effect shifts are consistent with causal effects directly estimated from Gab.
the range of importance weights p*(X) TEXI-TRANSPORTIM is much larger than the range of deT (X) under TEXI-TRANSPORTaR introducing large amount of variability when estimating jt
Negative causal effects. After transport to Gab, the linguistic attributes netspeak, insight, and fam- ily are all shown to have significant negative effects on whether a comment is perceived as hate speech, while they found to have significant effect in the original Reddit data. In other words, when Gab users use netspeak, make insights, talk about family, these conversational choices cause readers to perceive the overall comment as less hateful, while the same does not hold true for Reddit users.
Often, however, TEXT-TRANSPORTIE can still produce reasonable confidence intervals (as is the case for the Amazon and EmoBank datasets), and it illustrates the efficacy of the TEXT- TRANSPORT method under one of the simplest implementations (since no additional model training or even access to target data is required).
Positive causal effects. In contrast, after trans- port to Gab, the linguistic attribute function is shown to have a significant positive effect on whether comment is perceived as hate speech, while it was found to have no significant effect in the original Reddit data. Function words com- prise articles, pronouns, conjunctions, negations,
'In this sense, metrics like RMSE can be somewhat reduc- tive, as they penalize the larger confidence interval but fail to capture the fact that the "transported" effect under the naive estimator has moved very little toward the target distribution, while the tran transported effect under the LM approach has cor- rectly made a much larger shift toward the target distribution.

---

Classification approach
Language modeling approach
They have the financial backing of powerful special interests and the determination to ignore the will of the peuple of Alaska and the American public.
Pushed as an effort to promote "Sonoma County" " wines and consumer education effort, the new law instead forces vintners to needlessly sulty their package and undermines their own marketing efforts.
The efforts to save the mills failed party because the interna- tional market for steel had largely dried up.
Yes, Self. om also bothered that this observation ignores half-eaten cheese sundwiches, incomplete insect collections, and locks of infants' hair, forgotten in closets, basements, and warehouses.
Opera Cancelled Over a Depiction of Mahammad
Where the authority?
Die As US Helicopter Crashes in Iraq
Suddenly, Amy screamed.
Table 3: Texts from the EmoBank source distribution P& (texts with high writer-intended valence) with the highest TEXT-TRANSPORT importance weights. Target distribution pd comprises texts with low writer-intended valence.
and other words that serve primarily grammatical purposes, and prior work has found that they can be highly suggestive of person's psychological state (Groom and Pennebaker, 2002; Chung and Pennebaker, 2007; Pennebaker, 2011).
and low PX),), meaning they should be similar to texts from p" despite actually coming from p&, We observe that texts from pit (i.e., texts with greater probability of high writer-intended valence) with high weights are in fact qualitatively similar to texts from pT (i.e., texts with greater probability of low writer-intended valence). That is, although texts from PR should be generally positive, the texts with the highest weights are markedly neg- ative in tone, making them much more similar texts from PT These observations support the in- tuition that the change of measure transports causal effects to the target domain by looking primarily at the responses to texts in the source domain that are most similar to texts in the target domain.
Though the difference between the original and transported effect not statistically significant, pro- fanity is also found to have a more positive effect on whether comment is perceived as hate speech after transport to Gab compared to Reddit. This indicates that Gab users' use of profanity in their comments causes readers to perceive the overall comment as more hateful than if a Reddit user were to make profane remarks. This effect shift may be explained by the specific nature of the profanity used on Gab, which is qualitatively harsher and more offensive than the profanity used on Reddit.
Related Work
The differences in these transported and origi- nal causal effects emphasize the importance of our method. An automatic content moderation algo- rithm, for instance, would likely need to consider different linguistic factors when deciding which comments to flag on each site.
FEXT-TRANSPORT draws on a diverse body of prior work from machine learning and causal infer- ence, including the literature on domain adaptation, distribution shift, and generalizability and trans- portability. We build on methods from these fields to define a novel, flexible estimator of causal effects in natural language.
5.3 An intuition for effect transport
Text causal effects. A number of approaches have been proposed for estimating causal effects from natural language. Egami et al. (2018) con- struct conceptual framework for causal inference with text that relies on specific data splitting strate- gies, while Pryzant et al. (2018) describe proce- dure for learning words that are most predictive of desired response. However, the interpretability of the learned effects from these works is limited. a subsequent paper, Pryzant et al. (2021) introduce a method for estimating effects of linguistic proper-
Previously in Section 3.1, we stated that the intu- dP" ition behind the change of measure deæ (X) as an importance weight in the estimator á R-T was to increase the contribution of texts that are similar to PT, while reducing the weight of texts that are most representative of PR. To explore whether this is indeed the case, identify the texts from EmoBank with the highest importance weights for each of our estimation approaches (Table 3). Texts with large importance weights have high p" (X)

---

ties from observational data. While this approach targets isolated effects of linguistic properties, requires responses to be measured on the target domain, and accounts only for the portion of confounding that is contained with the text. Finally, in recent paper, Fong and Grimmer (2021) conduct randomized experiments over text attributes to determine their effects. While allowing for valid causal inference, the resulting constructed texts are artificial in nature and constitute a clear use case for TEXT- TRANSPORT, which can trans- port effects from the less-realistic experimental text domain to more naturalistic target domains.
due to the highly confounded nature of natural language. To address this challenge, we propose TEXT-TRANSPORT, a novel method for estimat- ing text causal effects from distributions that may not fulfill the assumptions required for valid causal inference. We conduct empirical evaluations that support the validity of causal effects estimated with TEXI-TRANSPORT, and examine realistic data setting-hatte speech and toxicity on social me- dia sites- in which TEXT- TR--TRAA TRANSPORT identifies significant shifts causal effects between text do- mains. Our results reinforce the need to account for distribution shift when estimating text-based causal effects and suggest that TEXT-TRANSPORT is a compelling approach for doing so. These promis- ing initial findings open the door to future explo- ration of causal effects from complex unstructured data like language, images, and multimodal data.
Domain adaptation, distribution shift, and transportability. Importance weighting has been widely used in the domain adaptation literature to help models learn under distribution shift (Byrd and Lipton, 2019). Models are trained with importance- weighted loss functions to account for covariate and label shift (Shimodaira, 2000; Lipton et al., 2018; Azizzadenesheli et al., 2019), correct for se- lection bias (Wang et al., 2016; Schnabel et al., 2016; Joachims al., 2017), and facilitate off- policy reinforcement learning (Mahmood et al., 2014: Swaminathan and Joachims, 2015).
Acknowledgements
This material is based upon work partially sup- ported by the National Science Foundation (awards 1722822 and 1750439) and the National Institutes of Health (awards R01MH125740, ROIMH132225, R0IMH096951, and R21 R2IMH130767), Victoria Lin is partially supported by Meta Research PhD Fel- lowship. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors, and no official endorse- ment should be inferred.
In parallel, line of work studying the external validity of estimated causal effects has emerged within statistical causal inference (Egami and Hart- man, 2022; Pearl and Bareinboim, 2022). These works aim to understand the conditions under which causal effects estimated on specific data can generalize or be transported to broader set- tings (Tipton, 2014; Bareinboim and Pearl, 2021). Prior work has also used density ratio-style impor- tance weights to estimate average causal effects with digeesiiienii interventions (de la Cuesta et al., 2022; Papadogeorgou et al., 2022).
Limitations
Although TEXT-TRANSPORT is effective ac- counting for distribution shift to estimate effects of linguistic attributes in target domains without data assumptions, the method relies on the existence of source domain that satisfies the data assumptions required for valid causal inference. Such source domain-even small or limited one-may not always be available. We plan address this lim- itation future work, where transport from any source domain is possible.
We emphasize that TEXN-PRNNPORTT is con- ceptually novel and methodologically distinct from these prior works. In our work, we explore an open problem-cam causal effect estimation from define new framework for learning causal effects from natural language. We propose novel solu- tion that uses tools from the domain adaptation and transportability literature, and we introduce novel methods for estimating natural effects practice in language settings.
Additionally. TEXT-TRANSPORT proposes a framework for estimating natural causal effects from text. However, we discuss above, in some cases it may also be of interest to estimate isolated causal effects from text. In future work, we will extend TEXI-TRANSPORT to include an estimator for isolated causal effects.
7 Conclusion
this paper, we study the problem of causal ef- fect estimation from text, which is challenging
Finally, a requirement of the target distribution

---

pT is that it is absolutely continuous with respect the source distribution PR. The absolute con- tinuity assumption is violated if text that would never occur in pf could possibly occur in pr Therefore, this assumption may not be satisfied the source and target distributions are completely unrelated and non-overlapping, even in latent space. Practically speaking, this means that it may not be possible to transport effects between distributions that are extremely different: for instance, from a corpus of technical manuals to a corpus of Shake- spearean poetry.
ings the AAAI Conference on Artificial Intelligence. 26(1):698
Alejandro Barredo Arricta, Natalia Díaz-Rodriguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Al- berto Barbado, Salvador Garcia, Sergio Gil-Lopez. Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. 2020. Explainable artificial intel- ligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion, 58:82-115.
Sven Buechel and Udo Hahn. 2017. EmoBank: Study- ing the impact of annotation perspective and repre- sentation format on dimensional emotion analysis. In Proceedings of the 15th Conference of the Turo- pean Chapter of the Association for Computational Linguisties: Volume 2, Short Papers, pages 578-585, Valencia, Spain. Association for Computational Lin- guistics.
10 Ethies Statement
Broader impact. Language technologies are as- suming an increasingly prominent role in real- world settings, seeing use in applications like healthcare (Wen et al., 2019; Zhou et al., 2022; Reeves et al., 2021), content moderation (Pavlopou- los et al., 2017; Gorwa et al., 2020), and marketing (Kang et al., 2020). As these black-box systems become more widespread, interpretability and ex- plainability in NLP are of ever greater importance. TEXI-TRANSPORT builds toward this goal by providing a framework for estimating the causal ef- fects of linguistic attributes on readers' responses to the text. These causal effects provide clear insight into how changes to language affect the perceptions of readers- important factor when considering the texts that NLP systems consume or produce.
Jonathon Byrd and Zachary Lipton. 2019. What is the effect of importance weighting in deep learning? Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 872-881. PMLR.
Cindy Chung and James W Pennebaker. 2007. The psychological functions of function words. Social communicution, 1:343-359.
Brandon de Cuesta, Naoki Egami, and Kosuke Imai. 2022. Improving the external validity of conjoint analysis: The essential role of profile distribution. Political Analysis, 30(1):19-45.
Naoki Egami, Christian J. Fong. Justin Grimmer, Mar- garet E. Roberts, and Brandon M. Stewart. 2018. How to make causal inferences using texts.
Ethical considerations. TEXT-TRANSPORR re- lies on pre-trained large language models to com- pute text probabilities. Consequently, it is possi- ble that these text probabilities
Naoki Egaai and Erin Hartman. 2022. Elements of external validity: Framework, design, and analysis. American Political Science Review, page 1-19.
are used to transport causal effect estimates- may encode some of the biases contained in large pre-trained models and their training data. Interpretations of causal effects produced by TEXT-TRANSE should take these biases into consideration.
Christian Fong and Justin Grimmer. 2021. Causal in- ference with latent treatments. American Journal of Political Science.
Robert Gorwa, Reuben Binns, and Christian Katzen- bach. 2020. Algorithmic content moderation: Tech- nical and political challenges in the automation of platform governance. Big Data & Society. 7():2053951719897945
Additionally, we acknowledge the environmen- tal impact of large language models, which are used in this work.
Carla J Groom and James W Pennebaker. 2002. Words. Journal of Research in Personality, 36(6):655 621.
References
Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar. 2019. Regularized learn- ing for domain adaptation under label shifts. In Inter- national Conference on Learning Representations.
Ella Guest, Bertie Vidgen, Alexandros Mittos, Nishanth Sastry, Gareth Tyson, and Helen Margetts. 2021. An expert annotated dataset for the detection of online misogyny. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: Main Volume, pages 1336-1350. Online. Association for Computational Linguistics.
Elias Bareinboim and Judea Pearl. 2021. Transportabil- ity of causal effects: Completeness results. Proceed-

---

Jaroslav Hájek. 1971. Comment on " "An essay on the logical foundations of survey sampling, part one". : In V.P. Godambe and D.A. Sprou, editors, Foundations of Staristical Inference. Holt, Rinehart and Winston, Toronto.
Georgia Papadogeorgou, Kosuke Imai, Jason Lyall, and Fan Li. 2022. Causal inference with spatio-temporal data: Estimating the effects of airstrikes insurgent violence in Iraq. Journal of the Royal Statistical So- ciety. Series B: Statistical Methodology, 84(5):1969- 1999.
Paul W Holland. 1986. Statistics and causal infer- ence. Journal of the American Statistical Association, 81 (39)):995-990
John Pavlopoulos, Prodromos Malakasiotis, and lon Androutsopoulos. 2017. Deeper attention to abusive user content moderation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Lan- guage Processing, pages 122--1135, Copenhagen, Denmark. Association for Computational Linguis- tics.
Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017. Unbiased learning-to-rank with bi- ased feedback. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, WSDM 17. page 781 1-89,, New York, NY, USA. Association for Computing Machinery.
Judea Pearl and Elias Bareinboim. 2022. External Va- lidity: From Do-Calculus Transportability Across Populations. 1 edition, page 451 451-482. Association for Computing Machinery, New York, NY, USA.
Yue Kang, Zhao Cai, Chee-Wee Tan, Qian Huang, and Hefu Liu. 2020. Natural language processing (nlp) in management research; A literature review. Journal of Management Analyties, 1(2):139-172
James W. Pennebaker. 2011, The secret life of pronouns. New Scientist, 21 2 (2828):42-45.
Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. 2018. Detecting and correcting for label shift with black box predictors. Proceedings of the 35th International Conference on Machine Learning, vol- ume 80 of Proceedings of Machine Learning Re- search, pages 3122-3130. PMLR.
James W Pennebaker, Ryan I. Boyd, Kayla Jordan, and Kate Blackburn. 2015. The development and psycho- metric properties of liwe2015. Technical report.
Reid Pryzant, Dallas Card. Dan Jurafsky, Victor Veitch, and Dhanya Sridhar. 2021. Causal effects of linguis- tic properties. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4095-4109) Online. Association for Computational Linguistics.
Rupam Mahmood, Hado P van Hasselt, and Richard S Sutton. 2014. Weighted importance sam- pling for off-policy learning with linear function ap- proximation. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.
Reid Pryzant, Kelly Shen, Dan Jurafsky, and Stefan Wagner. 2018 Deconfounded lexicon induction for interpretable social science. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long Pa- pers), pages 1615-1625, New Orleans, Louisiana. Association for Computational Linguisties.
Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukher- jee. 2021. Hatexplain: A benchmark dataset for explainable hate speech detection. Proceedings of the AAAI Conference on Artificial Intelligence. 35(17):14867-1433
Jing Qian. Anna Bethke, Yinyin Liu, Elizabeth Belding, and William Yang Wang. 2019. A benchmark dataset for learning to intervene in online hate speech. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing SEMNLP-IJCNLP. pages 4755- 4764, Hong Kong, China. Association for Computa- tional Linguisties.
Julian McAuley and Jure Leskovec. 2013. Hidden fac- tors and hidden topics: Understanding rating dimen- sions with review text. In Proceedings of the 7th ACM Conference on Recommender Systems, RecSys '13, page 165-172, New York, NY, USA. Association for Computing Machinery.
Susan M. Mudambi and David Schuff. 2010. Research note: What makes helpful online review? study of customer reviews on amazon.com. MIS Quarterly, 34(1):185-200,
Ruth M. Reeves, Lee Christensen, Jeremiah R. Brown, Michael Conway, Maxwell Levis, Glenn T. Gobbel, Rashmee U. Shah, Christine Goodrich, Tben Ricket, Freneka Minter, Andrew Bohm, Bruce E. Bray. Michael E. Matheny, and Wendy Chapman. 2021. Adaptation of an nip system to a new healthcare en- viroument to identify social determinants of health. Journal of Biomedical Informatics, 120: 103851.
Jerzy Neyman. 1923 [1990]. On the application of prohability theory to agricultural experiments. essay on principles. section 9. Statistical Science, 5(4):655- 472.
Yue Pan and Jason Q. Zhang. 2011. Born unequal: A study of the helpfulness of user-generate product reviews. Journal of Retailing, 8744)559--61.
Paul R Rosenbaum and Donald B Rubin. 1983. The
Central Role of the Propensity Score Observational Studies for Causal Effects. Biometrika, 70(1 1-555..

---

Donald B Rubin. 1974.
Estimating causal effects of treatments randomized and nonrandomized studies. Journal of educational Psychology, 66(5):688.
Cynthia Rudin. 2019. Stop explaining black box chine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, (5):206-225.
Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims, 2016. Rec- ommendations as treatments: Debiasing learning and evaluation. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Pro- ceedings of Machine Learning Research, pages 1670- 1679, New York, New York, USA. PMLR.
Hidetoshi Shimodaira. 2000. Improving predictive in- ference under covariate shift by weighting the log- likelihood function. Journal of Statistical Planning and Inference. 90(2):227-24
Kaitao Song. Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu, 2020. Mpnet: Masked and permuted pre- training for language understanding. In Advances in Neural Information Processing Systems, volume 33, pages 16857-16867. Curran Associates, Inc.
Adith Swaminathan
Thorsten Joachims. 2015. Counterfactual risk minimization: Learning from logged bandit feedback. In Proceedings of the 32nd International Conference on Machine Learning, vol- ume 37 of Proceedings of Machine Learning Re- search, pages 314-823, Lille, France. PMLR.
Elizaheth Tipton. 2014. How generalizable is your periment? an index for comparing experimental sam- ples and populations. Journal of Educational and Behavioral Statistics, 39(6):478-50)
Xuanhui Wang, Michael Bendersky, Donald Metzler, and Mare Najork. 2016. Learning to rank with se- lection bias in personal search. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 16, page 115-124, New York, NY, USA. As- sociation for Computing Machinery.
Andrew Wen, Sunyang Fu. Sungrim Moon, Mohamed EI Wazir, Andrew Rosenbaum, Vinod Kaggal, Sijia Liu, Sunghwan Sohn, Hongfang Liu, and Jungwei Fan. 2019. Desiderata for delivering nlp accelerate healthcare ai advancement and a mayo clinic nlp- as-a-service implementation. NPJ digital medicine, 131)
Binggui Zhou, Guanghua Yang. Zheng Shi, and Shao- dan Ma. 2022. Natural language processing for smart healthcare. IEEE Reviews in Biomedical Engineer- ing, pages 1-17.

---

A Change of measure with Radon-Nikodym derivatives
The Radon-Nikodym derivative can be used to express probability density function in terms of another probability density function, when the densities are related by change of measure. Specifically, if we have two probability measures defined on the same sample space, with one measure IP absolutely continuous with respect to another measure Q then there exists Radon-Nikodym derivative Z such that:
PP(A) = / ZdQ
for any event A in the sample space. Intuitively, this means that we can define the probability of any event under the measure P in terms of the probability of the same event under the measure Q, by weighting the probabilities by factor given by the Radon-Nikodym derivative.
Now, suppose we have two probability density functions p(x) and g(z) defined on some real-valued random variable X, with (x) > O for all æ. We want to express p(x) in terms of g(z) by a change of measure. To do this, we can define new probability measure IP as:
P(A) / p(x) g(x)
g(x)dx = p(x)do
for any event A in the sample space. IP is absolutely continuous with respect to the measure defined by q(x),, then there exists Radon-Nikodym derivative ((x) such that:
dIP
p(x))
dQ ao (x) (x)
((x)
((x)
This means that we can express p(z) in terms of q(x) and the Radon-Nikodym derivative (xx) as:
p(x) = Z(x)q(z)
B Statistical properties of ft vr
As we mention in the main paper, the transport estimator R-I has a number of desirable statistical properties that allow us to quantify its uncertainty, included unbiasedness, asymptotic normality, and closed-form variance. In this section, we provide proofs and derivations for these properties.
B.1 Unbiasedness
We can show that R-T is an unbiased estimator for u(PT):
R-T Exapxa
R-T
TPT
ExxupR
dPR (X))
i=1
n E Ex,xPR PR
ddpT dpR (Xx)
(7)
Ex,-Pr (X,(x))
U(PT)

---

B.2 Closed-form variance and confidence intervals
Let X be the space of all texts, and consider the finite-sample setting where Y2 is fixed (i.c., non-random) for all E [n]. Then the variance of the estimator is given by:
dPP
Varla
>T
Y1 = Var
(x.) dlp
1
dPP'
Var
dIPR (x.)):(Xi)
n
1
dPp (X,,), काम (x))
dP
Cov
2
dum
dPF
dpF
Cov
dPR x ())){())({;; x),
dpR (2)) (x)'' -
LE
F'EX'
dPPT
dPT
dPR (z)
(z)) Cov 2)
dPR
ZEN T'E
dPT
dPT dip (IIXX = E'H
dPR (z)
n
-I ZEX' x'cx
JPT
JP
बIR (1T -P(())B((())
बाम (τ)
i=1 xxx d'cx''
If I =x'
dpr2
Var
R-T
Yi]
1
(x) (x)) ((a)
n2
dPR
dpT2
1
(x)
(x) (z) p2(x))
n
1=1
TE
I 2',
dpR (z) dP
dP
Varlp R-7
Yi
n
i=1 EEN ZeN
t t'+r
dPT
dPT
dpa (z)
x'

---

Putting the two cases together.
diip
1
VarliR--TIy)
[i]
dpp (2)
n
i=1
T
dRR (x) dF
dPP
dPR (YM
xE x'EX
x f'
dPT2
dpp
dPR (x) (x)
Σ ZEN dPR
(x)))
TEN
dPT
dFT
Σ
dP (r)
dPR (x'
#EX x'EX
x' x'Fr
dP
dPT2 dpn
Σ
(z)))
(x)
ddp
1=1
TE
TE
dpr A= B= R
apt (x))
dP'
dIp (o))(())
(R(()) -
T
dPT
dPP (W"
Σ TEX
dFR (o)Y((x) - (x) + Σ
n2
dPR
ZEA
P2pn(0 (x) !0?
ZEX
dPT
dpT
diw (x))((2) AV)) (R(x)) + :/:
2 Σ
(28))
n
x
SEX
P (x) (c))-i)) ))
14
DEN
n
dFR (x))(()) -ia) 2 p²((x) + fs (2
dPT
Σ DEN
2
i=1
dP'T
dpu (T)Y((+)) - (2.) (R(()
n
i=1
ZEX
Then finally, letting ja BmupT
Di Yx(x)]
Bnmpa 1 Et
dPT
(x)) we have
aPR
n
Var(AR = Do-Varrii (P)[Vi]
2 (R(())
dPT
-Ey
den dلा (1))((a) -Pa)
n
TEX
(8)
(o) NN) m)] p(((x)
dPT Σ Ey
Σ
dPR
n?
1=1 ZEN

---

With the central limit theorem (CLT), we establish asymptotic normality:
RR-T m(PT) - (0,
(9)
Varlf S
which we can use to estimate confidence intervals using the following unbiased variance estimate:
dPT (X,)),(X,) - f
Varli- R-
Σ iE(n)
(10)
n
dPR
Hajek estimators
In practice, to maintain the stability of the importance weights (which can be very small), the Hajek (1971) estimator is often used in place of the instead of the standard Horvitz- Horvitz-Thompson estimator. With the Hajek estimator, the importance weights are normalized by the average importance weight. Then letting the importance weight be denoted by y, have the estimator
a R vT
(X,))
For TEXT- TEXT-TRANSPORT.IS
dPP 8
n2 Σ dPR (X,)
disr
dPR (Xi)
and for TEXT- TRX--TRRASPORILL
ps (x,) pR((X))
1 n2
(x,)
n Σ
::
pR(X))
D Experimental Details
D.I Data
DR
Dr
D train
natal
License
Amazon 2,561
5,000
889
7,561
Unknown
EmoBank
3,350
1,320
529
4,670
CC-BY-SA 4.0
Hate Speech 22,250
33,776
5,415
56,026
CC BY-NC 4.0
Table 4: Composition of data splits. For each dataset, the number of samples in DR. Dr, and Devosa is given, along with total samples each dataset. Licensing information is also provided.
Details of our datasets are provided in Table 4, including dataset composition and licensing information. All three datasets are publicly available, and all are in English.
D.2 Model details
We used HuggingFace's implementation of MPNet in its sentence-transformee library (version 2.2.2), using the pre-trained model all all-mpnet-base-v2 Embeddings from MPNet are 768 dimensions. Our logistic regression classifier was implemented in scikit-learn (version 1.0.2). All hyperparameters were to their default values.
TEXI-TRANSPORTAr
TEXT-TRANSPORTLE We used text-davinci-003 from the OpenAI API (version 0.27.4) as our pre-trained GPT-3. We prompted GPT-3 and computed sentence probabilities through the API. We set temperature to 0 and the maximum number of generated tokens to since we wanted the model to echo our text input rather than generate new texts.

---

D.3 Prompts
pR prompt
pr prompt
Amazon
You are writing review for your purchase of musical instrument on Amazon, Consider the fol- lowing sentence.
You are writing a review for your purchase of an office product on Amazon. Consider the following sentence.
EmoBank
You are writing positive statement. Consider the following sentence.
You are writing a negative statement. Consider the following sentence.
Hate Speech
are writing comment on toxic subreddit of the social media site Reddit, Consider the following sentence.
You are writing a comment on the alt-right social media site Gah. Consider the following sentence.
Table 5: The full list of prompts provided to GPT-3 to induce them focus on p and pf for each of the evaluation datasets.
25% Quantile
Median
75% Quantile
Amazon
8.429
997.189
3,013 x 100
EmoBank
0.223
8.780
225.346
Hate Speech 0.120
1.515
23.268
Table 6: Probability ratios between GPT-3 language models that have been given a P prompt and models that have been given pT prompt. median ratio greater than suggests that prompting has been successful.
The prompts we used induce GPT-3 toward the source distribution PR and the target distribution pT are provided in Table confirm that these prompts indeed induce GPT-3 move toward PR or pr we conducted the following empirical validation. Given text X ~ p", we computed the ratio between P(X")) for GPT-3 that had been given a plt prompt and P(X for GPT-3 that had been given a pll' prompt-in other words, the probability ratio Peria p (X
Parts pr (X
Since the texts X are drawn from PR, then if the prompts indeed direct GPT-3 toward the intended distribution, we would expect this ratio to have a median value greater than I, as p"(X") should be larger than p" (X""). We report medians and quantiles across the three evaluation datasets in Table
We observe that across all three datasets, the median ratio is in fact greater than 1, indicating that our prompting strategy is successfully targeting GPT-3 to PR or PT, The median ratio for the Hate Speech dataset-whin while still greater than 1-is much closer to 1 than the Amazon EmoBank datasets, which is consistent with the intuition that targeting very specific distributions like Reddit and Gab with prompting can be more challenging.
D.4 Computing resources
All experiments were conducted on machines with consumer-level NVIDIA graphics cards. We estimate the number GPU hours used in our experiments to fewer than 10.


Reviews:
[{"review_id": "ugoddhZFYV", "reviewer": "Reviewer_VqXU", "summary": "", "questions": "A. Why was there no comparison to other methods? Please clarify if this is the first estimation protocol for causal effects when the target distribution is not a randomized experiment. \nBesides using the density ratio as the importance weight (eqn 3), could other transformations be tested quickly? You could motivate better why this mapping was selected.    \n\nB. It is assumed that any crowdsourced dataset represents a randomized experiment and therefore does not have any confounding between the choice to read a text and the response to the text. Is it possible that other conditions of the crowdsourced experiment or motivations for participating in the experiment are confounding the response?\n\nC. In the benchmark portion of the evaluation, the first (controlled setting) source and target datasets are obtained by splitting a single crowdsourced dataset into two parts which are expected to be distinct, and the average response is expected to differ between the two. How did you choose the partitions/categories and was there any pre-evaluation to check that the response was distinct? \n\nMinor clarification questions:\n\nD. An assumption of the target distribution $P^T$ is that it is absolutely continuous with respect to the source distribution $P^R$. Can you expand on any practical limitations this introduces? When would this not be satisfied? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback and for pointing out many strengths of our approach! We address your questions below.\n\n[**Question A, comparison to other methods**] As we mention in our Related Works section, a small number of prior studies have proposed estimators of some type of text causal effect from observational (i.e., non-randomized) data (Pryzant et al. 2018, Pryzant et al. 2021). However, the effects obtained from these methods are not directly comparable to those obtained using Text-Transport. The 2018 Pryzant et al. paper estimates word-level effects rather than linguistic attribute-level effects, limiting their interpretability. The 2021 Pryzant et al. paper estimates isolated effects rather than natural effects; but isolated effects are difficult to interpret in a natural language setting, as we highlight in lines 132-143 of our paper. Moreover, for both existing methods, the estimates of the causal effects are only valid if all potential confounders are measured in the data. This requirement is seldom fulfilled for text data, as (1) reader attributes are rarely recorded for the text datasets used in modern NLP, and (2) there are many varied factors that can influence both a person’s decision to read a text and the person’s response to the text (e.g., political affiliation, age, mood).\n\n[**Question A, motivation of density ratio**] Other transformations of the source distribution to the target distribution are indeed possible—for instance, the naive baseline described by reviewer aAKu, in which a predictive model trained on the source distribution is used to produce pseudo-labels on the target distribution, which are then averaged. The results of this baseline are shown below:\n\n| Dataset \t| Method | Distribution | Response | Lower CI | Upper CI |\n|-------------|:------:|:------------:|:--------:|:--------:|:--------:|\n| Amazon  \t|\t-   |\tsource\t| 1.564\t| 1.438\t| 1.684\t|\n| Amazon  \t|\t-   |\ttarget\t| 1.191\t| 1.115\t| 1.265\t|\n| Amazon  \t|   clf  |  transported | 1.243\t| 1.149\t| 1.322\t|\n| Amazon  \t|   LM   |  transported | 1.214\t| 1.066\t| 1.371\t|\n| Amazon  \t|  naive |  transported | 1.353\t| 1.141\t| 1.594\t|\n| EmoBank \t|\t-   |\tsource\t| 3.229\t| 3.217\t| 3.242\t|\n| EmoBank \t|\t-   |\ttarget\t| 2.689\t| 2.677\t| 2.702\t|\n| EmoBank \t|   clf  |  transported | 2.844\t| 2.821\t| 2.868\t|\n| EmoBank \t|   LM   |  transported | 2.677\t| 2.470\t| 2.88 \t|\n| EmoBank \t|  naive |  transported | 3.135\t| 3.11 \t| 3.161\t|\n| Hate Speech |\t-   |\tsource\t| 0.135\t| 0.117\t| 0.152\t|\n| Hate Speech |\t-   |\ttarget\t| 0.263\t| 0.244\t| 0.283\t|\n| Hate Speech |   clf  |  transported | 0.179\t| 0.134\t| 0.227\t|\n| Hate Speech |   LM   |  transported | 0.246\t| 0.000\t| 1.000\t|\n| Hate Speech |  naive |  transported | 0.138\t| 0.080\t| 0.200\t|\n\nWe can see that this type of transformation is empirically less effective than the density ratio transformation used in Text-Transport, as the naive estimator produces effect estimates that are farther from the true target effects. In fact, for the EmoBank and Hate Speech datasets, there is essentially no movement of the transported estimate toward the target distribution, while for the Amazon dataset, there is less movement compared to either implementation of the Text-Transport approach. Density ratios have been shown in the domain adaptation literature to be effective in helping models learn under distribution shift (see citations in our Related Works, lines 560-568). We will motivate the use of the density ratio in the context of the domain adaptation literature earlier in our paper, when we first introduce the method; and we will include the results of the naive baseline.\n\n[**Question B**] Yes, it is possible in the sense that certain types of people may be attracted to crowdworking platforms like Mechanical Turk or Prolific. However, this type of confounding (selection bias) can be present in any type of randomized experiment—that is, respondents in traditional randomized experiments or trials may also have specific motivations (often financial) for participating that make them different from the overall population. Tackling this issue has been the focus of a large literature, using techniques related to the method we develop here. For example, the work of Egami and Hartman (2021), Pearl and Bareinboim (2022), and Tipton (2014) that we discuss in the Related Work section approach this issue as either a covariate- or label-shift problem. In principle, it is possible to combine our method with the techniques in this literature to generalize to a target domain specified by both the text corpus and the distribution of respondents. We hope to examine this in future work.\n\n[**Question C**] For the Amazon and Hate Speech datasets, we chose partitions that already existed in the datasets—for instance, the Amazon dataset is divided into product categories, and the Hate Speech dataset contains posts from both Reddit and Gab, making for an intuitive split. For the EmoBank dataset, as we mention in Section 4.2, we chose to split along writer-intended valence to create a partially controlled data setting in which the source and target differ on a single attribute that is known to us but hidden from the models. We did check that the average responses were distinct in the source and target distributions; the separation of responses across source and target can be seen in Figure 3 of our paper.\n\n[**Question D**] The absolute continuity assumption is violated if a text that would *never* occur in the source distribution could possibly occur in the target distribution. Therefore, this assumption may not be satisfied if the source and target distributions are completely unrelated and non-overlapping, even in latent space. Practically speaking, this means that it may not be possible to transport effects between distributions that are extremely different: for instance, from a corpus of technical manuals to a corpus of Shakespearean poetry.\n\n[**Minor suggestions**] Thank you for these helpful suggestions! We will implement them in the next version of the paper. Following suggestion B, we will include citations for this statement (much of the literature on transportability is premised on this notion, e.g. the transportability papers cited in our Related Works: Tipton 2014, Barenboim and Pearl 2021). Regarding suggestion D, $\\tau^*$ is conventional notation for a general causal effect. We agree that superscripting $T$ may be clearer notationally and will make this change.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The results are convincing and sound; in three scenarios where the causal effect differs significantly between the source and target distributions, TEXT-TRANSPORT accurately estimates the effect in the target distribution indirectly through the source estimate.\n\n* The authors do a great job motivating the problem, both conceptually and notationally. The Problem Setting is easy to read; equations 3 and 4 are intuitively explained. Overall the paper is a pleasure to read. ", "reasons_to_reject": "* There is no comparison with competing methods (if they exist)", "questions_to_authors": "A. Why was there no comparison to other methods? Please clarify if this is the first estimation protocol for causal effects when the target distribution is not a randomized experiment. \nBesides using the density ratio as the importance weight (eqn 3), could other transformations be tested quickly? You could motivate better why this mapping was selected.    \n\nB. It is assumed that any crowdsourced dataset represents a randomized experiment and therefore does not have any confounding between the choice to read a text and the response to the text. Is it possible that other conditions of the crowdsourced experiment or motivations for participating in the experiment are confounding the response?\n\nC. In the benchmark portion of the evaluation, the first (controlled setting) source and target datasets are obtained by splitting a single crowdsourced dataset into two parts which are expected to be distinct, and the average response is expected to differ between the two. How did you choose the partitions/categories and was there any pre-evaluation to check that the response was distinct? \n\nMinor clarification questions:\n\nD. An assumption of the target distribution $P^T$ is that it is absolutely continuous with respect to the source distribution $P^R$. Can you expand on any practical limitations this introduces? When would this not be satisfied? ", "paper_topic_and_main_contributions": "This paper is concerned with determining the causal effects of language properties on the response of the reader. For example, how the presence of profanity may affect the reader's perception of the text as hate speech. Estimating causal effects directly given a dataset requires that the data was obtained by a randomized experiment, an assumption likely to go unmet in most cases. In this paper, the authors propose a method to transform a causal estimate from a source dataset (meeting this assumption) to a target dataset. In this way, we can obtain causal estimates in related domains without the additional overhead of collecting new data.  ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Some minor suggestions:\n\nA. Intro lines 051-053: give an example scenario of confounding \n\nB. At line 061 it is stated \"effects estimated from randomized experiments may not generalize outside of the specific data on which they were conducted\". Although this seems like an intuitive statement, it would be helpful to see it formalized if possible. If citations or a small sketch in the appendix could be added, it would help interested readers understand the problem better from the offset.\n\nC. Organization: Section 5.3 would have been easier to read closer to sections 4.1-4.2, on the topic of the dataset distributions. \n\nD. Notation: is $\\tau^*$ (eqn 2) conventional notation for the natural effect? Should the $T$ at line 144 (and after) be a superscript?"}}, {"review_id": "Mcq18fvhNN", "reviewer": "Reviewer_aAKu", "summary": "", "questions": "1. The probability produced by LLM is known to be poorly calibrated. How do you think will it affect the validity of your approach? Any possible workarounds? \n\n2. How would you extend your framework to a more complicated setting where you have m known domains and n unknown domains to be transferred? (In your paper you assume that m=n=1)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We appreciate your positive feedback and constructive questions.\n\n[**Comparison to naive estimator**] Thank you for this suggestion! We report the results of this naive baseline below, in which a predictive model trained on the source distribution is used to produce pseudo-labels on the target distribution, which are then averaged. We will include these results in the next version of our paper. Following the reviewer’s suggestion of additional evaluation metrics, we also report the normalized RMSE (RMSE divided by the standard deviation of the target response) of our transported response $\\hat{\\mu}_{R \\rightarrow T}$ compared to the known target $\\hat{\\mu}_T$ across 100 bootstrap iterations.\n\n| Dataset \t| Method | Distribution | Response | Lower CI | Upper CI |  RMSE |\n|-------------|:------:|:------------:|:--------:|:--------:|:--------:|:-----:|\n| Amazon  \t|\t-   |\tsource\t| 1.564\t| 1.438\t| 1.684\t|   -   |\n| Amazon  \t|\t-   |\ttarget\t| 1.191\t| 1.115\t| 1.265\t|   -   |\n| Amazon  \t|   clf  |  transported | 1.243\t| 1.149\t| 1.322\t| 0.019 |\n| Amazon  \t|   LM   |  transported | 1.214\t| 1.066\t| 1.371\t| 0.035 |\n| Amazon  \t|  naive |  transported | 1.353\t| 1.141\t| 1.594\t| 0.073 |\n| EmoBank \t|\t-   |\tsource\t| 3.229\t| 3.217\t| 3.242\t| - \t|\n| EmoBank \t|\t-   |\ttarget\t| 2.689\t| 2.677\t| 2.702\t| - \t|\n| EmoBank \t|   clf  |  transported | 2.844\t| 2.821\t| 2.868\t| 0.832 |\n| EmoBank \t|   LM   |  transported | 2.677\t| 2.470\t| 2.88 \t| 0.378 |\n| EmoBank \t|  naive |  transported | 3.135\t| 3.11 \t| 3.161\t| 0.903 |\n| Hate Speech |\t-   |\tsource\t| 0.135\t| 0.117\t| 0.152\t| - \t|\n| Hate Speech |\t-   |\ttarget\t| 0.263\t| 0.244\t| 0.283\t| - \t|\n| Hate Speech |   clf  |  transported | 0.179\t| 0.134\t| 0.227\t| 0.257 |\n| Hate Speech |   LM   |  transported | 0.246\t| 0.000\t| 1.000\t| 0.943 |\n| Hate Speech |  naive |  transported | 0.138| 0.080\t| 0.200\t| 0.351 |\n\nWe can see that in contrast to our method, Text-Transport, the naive estimator produces effect estimates that are farther from the true target effects. In fact, for the EmoBank and Hate Speech datasets, there is essentially no movement of the transported estimate toward the target distribution, while for the Amazon dataset, there is less movement compared to either implementation of the Text-Transport approach. Interestingly, the RMSE highlights one particular result: the lower RMSE of the naive estimator on the Hate Speech dataset compared to the Text-Transport LM approach, which is due to the larger confidence interval of the LM approach. This type of metric penalizes the larger confidence interval but fails to capture the fact that the \"transported\" effect under the naive estimator has moved very little toward the target distribution, while the transported effect under the LM approach has correctly made a much larger shift toward the target distribution. In this sense, metrics like RMSE can be somewhat reductive.\n\nAdditionally, we note that in future work, the type of predictive modeling used in the naive estimator can be incorporated into the Text-Transport method via the *doubly robust estimator*, defined as follows:\n\n$$\\hat{\\mu}(P^T)_{DR}=\\frac{1}{n}\\sum\\_{i=1}^n \\frac{\\hat{d\\mathbb{P}^T}}{d\\mathbb{P}^R}(X_i)(Y_i(X_i)-\\hat{m}(X_i)) +\\frac{1}{n}\\sum\\_{i=1}^n\\hat{m}(X_i)$$\n\nThe doubly robust estimator is commonly used in statistical causal inference to reduce the variance of estimates. By using a learned outcome model $\\hat{m}(X)$, the doubly robust estimator ensures that the variance of the estimate is a function of the difference between the ground truth outcome $Y(X)$ and the predicted outcome $\\hat{m}(X)$, rather than a function of $Y(X)$ itself.\n\n[**Instance-level estimation error**] Under this framework, instance-level estimation error cannot be computed because there is no 1:1 mapping between points in the source distribution and points in the target distribution. However, we hope that our inclusion of RMSE above—and the results of the naive estimator—will provide further evidence for the validity of Text-Transport.\n\n[**Question 1**] While some prior research has suggested that probabilities from LLMs are poorly calibrated, our empirical results indicate that these probabilities still contain sufficient signal to successfully transport effect estimates across text distributions. Improving the probability calibration of LLMs is also an active area of research, and very recently, works have shown that the newest LLMs (e.g., GPT-4 in Nori et al. 2023) display better probability calibration compared to previous LLMs. We note that the implementation of the LLM approach described in this paper is the first step in a longer research agenda on estimating text causal effects, and ongoing improvements in LLM probability calibration also pose interesting future directions for our work.\n\nHarsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan & Eric Horvitz. (2023) Capabilities of GPT-4 on Medical Challenge Problems. \n\n[**Question 2**] Introducing additional source and target domains could be as straightforward as training a multi-class classifier over multiple text domains (for the classification approach) or prompting the LLM to target multiple text domains. We chose $m=n=1$ for evaluation purposes only; the Text-Transport approach is not inherently limited to a single source and target domain. The $m > 1$ case does offer several interesting questions about aggregating across estimates. For example, if $m >1$ and $n = 1$, directly using Text-Transport would give $m$ different estimates of the effect in the target domain. While it is valid to simply average these different estimates together, alternatives such as precision weighting may perform better. Future work can help adjudicate what methods of aggregation work best.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A novel framework to facilitate causal effect estimation from text under different domains, which could be potentially useful for computational social science and causal NLP research community.\n\n2. Empirical results on three datasets validate the effectiveness of their proposed method.", "reasons_to_reject": "1. Lack of comparison to some naive estimator baselines to further validate the effectiveness of the proposed framework. The paper only shows results on an estimator derived from eq 3. A direct baseline would be: applying a trained classifier (e.g. helpfulness prediction classifier for Amazon dataset) on the source domain to produce pseudo labels on the target domain, then using the pseudo labels to conduct randomized experiments to compute the causal effect. \n\n2. Evaluation metrics of the validity could be more comprehensive. Besides confidence interval, I imagine that computing instance-level estimation (ITE) error could also help to show the accuracy of the transport estimator. ", "questions_to_authors": "1. The probability produced by LLM is known to be poorly calibrated. How do you think will it affect the validity of your approach? Any possible workarounds? \n\n2. How would you extend your framework to a more complicated setting where you have m known domains and n unknown domains to be transferred? (In your paper you assume that m=n=1)", "paper_topic_and_main_contributions": "This paper tackles the problem of domain shift under causal estimation from text. The proposed framework, TEXT-TRANSPORT, calculates the density ratio between source and target domains via a trained classifier or a prompted LLM to distinguish where the given text comes from. The authors validate their framework empirically on three datasets and further demonstrate a use case of their proposed framework on hate speech detection.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LMdmm7A1vk", "reviewer": "Reviewer_g3EB", "summary": "", "questions": "* For the LLM-based method, did you assess if prompted output is indeed more similar to texts from the target vs. source corpora either qualitatively or quantitatively?\n* Not a question, but please feel free to engage with other points above and/or note if I've misunderstood/misrepresented anything from your paper in this review.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and positive comments!\n\n[**LLM prompting**] Thank you for this point! We chose to use Text-Transport with an LLM prompting-based approach to illustrate the efficacy of the method with one of the \"simplest\" implementations (since no additional model training or even access to target data is required). It is true that using prompting to target an LLM to a specific distribution may not always be precise—as may be the case for the Hate Speech dataset, where the transported point estimate is very close to the target point estimate (suggesting some targeting signal) but the confidence intervals are wide (suggesting lack of precision). However, we believe that the success of the LLM approach in two additional data settings demonstrates that there is merit to the approach, and we see the question of how LLM targeting (or the use of LLMs in this context more generally) can be refined as an opportunity for future work.\n\n[**Question 1, assessing prompted output**] Qualitatively, we did observe that LLMs that had been prompted toward the EmoBank target distribution (low valence) found texts with low valence to be more probable (see Table 2 of our paper). To support these observations, we report additional quantitative evidence below. Let $X^R$ denote a text drawn from $P^R$. We report the ratio of $P(X^R)$ for an LLM that has been prompted toward $P^R$ versus an LLM that has been prompted toward $P^T$—in other words, the quantity $\\frac{P\\_{LLM\\_{P^R}}(X^R)}{P\\_{LLM\\_{P^T}}(X^R)}$. \n\nIf prompting indeed directs the LLM toward the intended distribution, we would expect this ratio to have a median value greater than 1. We report medians and quantiles across the three evaluation datasets:\n\n| Dataset \t| 25% Quantile |  Median | 75% Quantile |\n|-------------|:------------:|:-------:|:------------:|\n| Amazon  \t| \t8.429\t| 997.189 | 3.013*10^5   |\n| EmoBank \t| \t0.223\t|  8.780  | 225.346  \t|\n| Hate Speech | \t0.120\t|  1.515  | 23.268   \t|\n\nWe observe that across all three datasets, the median ratio is greater than 1; and for the Amazon and EmoBank datasets, the ratio is much greater than 1. The median ratio for the Hate Speech dataset—while still greater than 1—is indeed closer to 1 than the others, which is consistent with the reviewer’s intuition that targeting very specific distributions like Reddit and Gab with prompting can be more challenging.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The method for transporting causal effects and derivation seem to be sound and seem, as far as I am aware, to be novel. (Note that I have some familiarity but am not an expert on causal methods.)                                                                                                                                           \n                                                                                                                                                                        \n* The experimental results generally seem to support the claims made in the paper (though see comments on the LLM-based estimator below).                               \n                                                                                                                                                                        \n* The paper is clearly written, organized well, and situated in existing literature on measuring causal effects in text as well as domain adaptation / distribution shift.", "reasons_to_reject": "The intuition behind a classification-based estimator makes sense to me; I'm a bit more hesitant about the LLM-based one, both in principle and results. In principle, the paper states that LLM training corpora are \"large enough to approximate the distribution of the English language\" (line 230), and for certain domains (like hate speech) I'm not sure that's the case, given possible filtering of training data etc. It's not obvious that prompting to generate hate speech on Reddit or Gab really generates distributions close to a per-platform basis vs. just hate speech more generally, and the results on hate speech seem to support that (the confidence interval for the LLM-based estimator spans both domains).                                                                                                                             \n\n_edit, after author response_: the response (and answer to my question below) has helped clarify both the purpose of the LLM-based estimator and its performance on the hate speech data.", "questions_to_authors": "* For the LLM-based method, did you assess if prompted output is indeed more similar to texts from the target vs. source corpora either qualitatively or quantitatively?\n* Not a question, but please feel free to engage with other points above and/or note if I've misunderstood/misrepresented anything from your paper in this review.", "paper_topic_and_main_contributions": "The paper proposes a method to estimate causal effects of linguistic attributes in a target text distribution given measured causal effects from a source text distribution (Text-Transport): reweight observations from the source distribution more highly when similar to the target texts and vice versa (density ratio). (Derivations for the causal estimator are provided in the appendix.) To estimate this, two methods are proposed: classification (clf), based on a binary classifier that identifies if a text came from source or target, and LM, based on prompting a LLM and using the output probabilities. Through experiments on a range of controlled (Amazon reviews, tweet valence) and natural (hate speech) settings, the results (a) show that their method for transporting causal effects in controlled settings generally align with the actual effects; (b) discuss LIWC attributes with causal impact for hate speech detection on Reddit vs. Gab.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes text-transport, a method to transform a causal estimate from a source to a target dataset. With this method, one can obtain causal estimates in related domains without the additional overhead of collecting new data. Two methods are proposed: one based on a binary classifier to distinguish between source and target domains and one based on prompting an LLM. The authors empirically show their framework's validity on three datasets and provide additional insights for the use case of hate speech detection. Comprehensive experiments and a thorough analysis of results support the paper's claims. The paper is well-written and exciting to read. 

The reviewers have raised concerns about comparing with baseline methods and the rationale behind the proposed method. These concerns have been addressed by the authors in their rebuttals, which need to be incorporated into the final version of the paper.
Decision: Accept-Main