Venue: NAACL 2024

# Causal Inference for Human-Language Model Collaboration

 Bohan Zhang

University of Michigan

Ann Arbor, MI, USA zbohan@umich.edu Yixin Wang

University of Michigan

Ann Arbor, MI, USA yixinw@umich.edu Paramveer S. Dhillon

University of Michigan

Ann Arbor, MI, USA dhillong@umich.edu

###### Abstract

In this paper, we examine the collaborative dynamics between humans and language models (LMs), where the interactions typically involve LMs proposing text segments and humans editing or responding to these proposals. Productive engagement with LMs in such scenarios necessitates that humans discern effective text-based interaction strategies, such as editing and response styles, from historical human-LM interactions. This objective is inherently causal, driven by the counterfactual 'what-if' question: how would the outcome of collaboration change if humans employed a different text editing/refinement strategy? A key challenge in answering this causal inference question is formulating an appropriate causal estimand: the conventional average treatment effect (ATE) estimand is inapplicable to text-based treatments due to their high dimensionality. To address this concern, we introduce a new causal estimand--_Incremental__Sylistic Effect (ISE)_, which characterizes the average impact of infinitesimally shifting a text towards a specific style, such as increasing formality. We establish the conditions for the non-parametric identification of ISE. Building on this, we develop _CausalCollab_, an algorithm designed to estimate the ISE of various interaction strategies in dynamic human-LM collaborations. Our empirical investigations across three distinct human-LM collaboration scenarios reveal that _CausalCollab_ effectively reduces confounding and significantly improves counterfactual estimation over a set of competitive baselines.

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1630–1647

June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

Dialog agents like ChatGPT and Claude, built on Pretrained Language Models (LMs), have significantly transformed the landscape of text generation, showcasing extraordinary performance improvements across a wide range of tasks Brown et al. (2020); OpenAI (2023); Chowdhery et al. (2023). This shift has heralded a new era in interaction design, particularly in dialog systems. As illustrated in Figure 1, these systems are designed for interactive human collaboration, involving sequential actions such as editing, rephrasing, or revising text Lee et al. (2022). This innovative design has empowered users to augment their own capabilities in various fields, including data analysis, customer support, and social media strategy formulation Brynjolfsson et al. (2023); Noy and Zhang (2023); Epstein et al. (2023). Consequently, it has become crucial for users to learn how to adeptly collaborate with these LMs to fully harness their potential. _This paper investigates how users can optimize their collaboration with LMs by drawing on insights from historical human-LM interactions._ We primarily seek to identify effective collaboration strategies from past dialogues, with the goal of enhancing the synergy between human intuition and machine intelligence in these sophisticated dialog systems.

Improving human-LM collaboration by having humans learn from past human-LM interactions is an inherently causal problem. Applying editing strategies from past successful collaborations may not always be effective, since the success of these strategies could be confounded by specific prompt setups. For instance, editing strategies such as _"increasing the level of politeness in the generated text"_ may prove beneficial when collaborating on customer support responses, as it helps to build rapport and maintain a positive relationship with the customer. However, the same strategy may not be as effective when working on a scientific research paper, where a neutral and objective tone is often preferred to convey the findings accurately. Similarly, _"adjusting the text to use more confident language"_ may be advantageous when crafting persuasive arguments in an editorial piece, but it could be less appropriate when generating content for a balanced news article, where impartiality is key.

These examples illustrate how editing strategies

(increasing politeness and adjusting language confidence) can have varying levels of effectiveness depending on the specific collaboration context (customer support, scientific writing, editorials, and news articles). They underscore the importance of understanding the causal impact of such strategies across different situations to determine their overall usefulness in enhancing human-LM collaboration and bring us to the key counterfactual question:
**How would the collaboration outcome change if we implemented an alternative text editing strategy?1** Answering such questions will provide insights into editing strategies that reliably improve human-LM collaboration versus those that only work in certain situational contexts.

Footnote 1: We use the terms “editing” and “refinement” interchangeably, and they mean the same in our context.

This causal inference problem, however, is challenging since it is unclear what constitutes a suitable causal estimand for human-LM collaboration. The traditional causal estimand, Average Treatment Effect (ATE) [20], faces limitations in this context due to the high-dimensional nature of text-based treatments, which violate the "positivity" assumption. Positivity requires that all possible word sequences occur with non-zero probability in all editing strategies. This condition is readily met in binary treatment situations, where treatments are categorized as either 'treated' or 'control.' However, it becomes problematic for text-based treatments. Numerous word sequences either fail to form coherent sentences or are implausible as human edits, resulting in some configurations having a zero probability of occurring. Hence, the question arises: **What is an appropriate causal estimand for human-LM collaboration?** This causal estimand must account for the distinctive characteristics of text as a treatment, particularly its high-dimensional and non-binary nature.

In this paper, we propose a novel causal estimand for human-LM collaboration--Incremental Stylistic Effect (ISE). ISE is based on the key insight that instead of focusing on the effect of specific edits (e.g., insertion/deletion/rephrasing of words) as effective collaboration (text refinement) strategies, we should instead focus on the implications of these edits on text style. For instance, consider a financial analyst collaborating with an LM to write a quarterly report for clients. Multiple edits could make the writing more formal, e.g., removing contractions ('can't' \(\rightarrow\) 'cannot') or replacing phrasal verbs with more precise terms. In this case, the ISE of "formalizing" text would measure the cumulative incremental effect of systematically enhancing formality through any such edits. If the analyst finds that increasing formality improves client evaluations (outcome), they learn that a more formal writing style is effective and can implement appropriate edits in future reports to increase formality (the exact wording changes don't matter). ISE is also more practical to communicate and is _actionable_. Instead of recommending specific edits, which may not always be applicable, it advises users on broader stylistic changes that are likely to enhance collaboration outcomes. Finally, the universal applicability of these stylistic changes meets the positivity condition in causal inference, as there is always the possibility to make any text more formal (in our stylized example).

Next, we present an algorithm _CausalCollab_, to evaluate the effectiveness of human-LM collaboration strategies over time, using the Incremental Stylistic Effect (ISE) as its guiding estimand. This algorithm operates by identifying and analyzing prevalent stylistic changes in past human-LM interactions. It then evaluates the impact of these changes in various dynamic human-LM collaboration contexts. Our empirical studies, encompassing three distinct scenarios of human-LM collaboration, demonstrate that _CausalCollab_ is effective in mitigating confounding factors and enhancing counterfactual estimation and provides valuable in

Figure 1: An interactive view of human-LM Collaborative Writing. The writer iteratively selected and rewrote suggestions from the LM to make the article have a better outcome.

 

sights for humans to improve their collaboration strategies with LMs.

**Contributions:** This paper makes the following contributions:

* Formalizing the problem of dynamic human-LM interaction as a causal inference problem.
* Introducing a novel causal estimand for human-LM collaboration--Incremental Stylistic Effect (ISE) which addresses the issues of high-dimensionality for text-based treatments.
* Providing theory establishing identification conditions for ISE.
* Proposing a new algorithm called _CausalCollab_ that employs ISE to effectively extract key editing strategies for human-LM collaborations.
* Thorough, empirical validation of _CausalCollab_ on three datasets establishing its superior ability for counterfactual prediction.

## 2 Related Work:

Our work is related to two strands of prior work.

_Human-LM Collaboration._ Building on the foundational research in human-LM collaboration across text generation Chakrabarty et al. (2022); Goldfarb-Tarrant et al. (2019), dialogue Gabriel et al. (2019); Bonaldi et al. (2022), and summarization Avinesh et al. (2018); Shapira et al. (2021), our work delves into optimizing human interaction strategies with LMs. We focus on understanding the causal impacts of various editing strategies, such as the editing strategies employed by systems like R3 Du et al. (2022) and CoAuthor Lee et al. (2022). Our key contribution is the development of the Incremental Stylistic Effect (ISE) estimand (and the associated _CausalCollab_ algorithm), which assesses the cumulative incremental effect of style-based edits on collaboration outcomes. This novel approach guides how humans can adapt their collaboration methods with LMs more effectively, enhancing the synergy between human intuition and machine intelligence in complex tasks like those explored in TaleBrush Chung et al. (2022) and Dramatron Mirowski et al. (2023). By offering a framework for analyzing and optimizing human-LM interactions, we aim to improve both the practical application and theoretical understanding of these human-LM collaboration dynamics.

_Causal Inference for Text:_ Our research is also situated within a rapidly evolving landscape of studies applying causal methods to language tasks, particularly in the context of human-LM collaborations. Influential works such as Veitch et al. (2020) have pioneered the use of text embeddings from LMs and topic modeling to address textual confounding, improving treatment effect estimation. This approach is further expanded by a series of studies Egami et al. (2022); Roberts et al. (2020); Sridhar and Getoor (2019); Pryzant et al. (2021), which propose learning latent representations of high-dimensional texts as confounders or treatments, utilizing topic modeling and Variational Autoencoders (VAEs). These studies highlight the importance of distilling low-dimensional latent representations for accurate treatment effect estimation from complex, high-dimensional data such as text Louizos et al. (2017); Kim et al. (2021).

Our work distinguishes from this line of work in addressing the dynamic nature of human-LM collaborations and proposing a novel causal estimand--Incremental Stylistic Effect (ISE) for this scenario. Further, unlike prior work, we employ a novel combination of G-estimation Taubman et al. (2009); Naimi et al. (2014); Van der Laan et al. (2011); Petersen et al. (2012) and Conditional Variational Autoencoders (CVAE) for learning low-dimensional latent representations in a setup with time-varying textual treatments.

## 3 Causal Inference for Human-LM Collaboration

We begin with framing human-LM collaboration as a causal inference problem.

### A causal perspective on Human-LM collaboration

We conceptualize the interaction between a human and a language model (LM) as a sequential series of human actions, each focused on optimizing task outcomes. In these iterative collaborations, humans consistently adjust the responses generated by LMs to attain the desired results. This approach is supported by various studies, including Xu et al. (2023); Bonaldi et al. (2022); Du et al. (2022); Lee et al. (2023), which explore the nuances of human-LM interactions in achieving specific goals.

To establish our notation, we represent the refinement action (such as the post-refinement text) performed by user \(i\) at time \(t\) as \(A_{it}\), with \(t~{}=~{}1~{}\dots~{}T\). Further, let's denote \(Y_{i}(a_{1},\ldots,a_{T})\) as the outcome rating (like a quality score) of the text, assuming user \(i\)_had executed_ the refinement \(a_{1}\) at time \(t=1\), \(a_{2}\) at time \(t=2\), ..., and \(a_{T}\) at time \(t=T\) dur 

ing their interactions with the LM. This outcome \(Y_{i}(a_{1},\ldots,a_{T})\) is conceptualized as a _potential outcome_ or _counterfactual outcome_(Imbens and Rubin, 2015; Pearl, 2009; Hernan and Robins, 2010). In this context, the sequence of user refinements is regarded as the "treatment," and the outcome rating of the text is the "outcome." The potential outcome \(Y_{i}(a_{1},\ldots,a_{T})\) is observable only when user \(i\) actually performs these refinements, that is, when \(A_{it}=a_{t}\) for all \(t=1,\ldots,T\). All other potential outcomes \(Y_{i}(a^{\prime}_{1},\ldots,a^{\prime}_{T})\), where \(a^{\prime}_{t}\neq A_{it}\), are unobserved.

At each time step \(t\), the refinement action \(A_{it}\) by user \(i\) is typically in response to the output of the LM, denoted as \(L_{it}\). For instance, \(L_{it}\) could be the LM's suggestions for completing an unfinished essay at time \(t\) during the interaction with user \(i\), while \(A_{it}\) might involve selective editing, refinement, and rewriting by the human participant. This action \(A_{it}\) then acts as a prompt for the LM's subsequent response \(L_{i,t+1}\) at the following time step. The two-step process of human-LM collaboration can be visualized as a directed acyclic graph (DAG), as shown in Figure 2, with the user subscript \(i\) omitted for simplicity; this DAG can be readily extended to more than two steps. (A real-world example of our human-LM collaborative writing setup is shown in Figure 1.)

Our goal is to evaluate the impact of various user refinements on the final text outcome by examining a dataset comprising historical human-LM interactions. This dataset typically includes: (1) the sequence of actions performed by previous users, denoted as \(\{\{A_{it}\}_{i=1}^{I}\}_{t=1}^{T}\), (2) the LM's responses at each time step, represented by \(\{\{L_{it}\}_{i=1}^{I}\}_{t=1}^{T}\), and (3) the observed outcomes, such as the quality scores of the texts, labeled as \(\{Y_{i}\}_{i=1}^{I}\), for all the interactions \(i~{}=~{}1~{}\dots~{}I\). These observed outcomes are the potential outcomes for the actual action taken by the user, expressed as \(Y_{i}=Y_{i}(A_{i1},\ldots,A_{iT})\).

To evaluate the efficacy of user refinements, we are often interested in the causal estimand known as the average treatment effect (ATE) of a given action sequence \((a_{1},\ldots,a_{T})\). This is mathematically represented as \(\mathbb{E}\left[Y_{i}(a_{1},\ldots,a_{T})-Y_{i}(a_{1}^{\varnothing},\ldots,a_ {T}^{\varnothing})\right]\), where \(a_{1},\ldots,a_{T}\) denotes the sequence of user actions. This expectation is calculated across users. The sequence \(a_{1}^{\varnothing},\ldots,a_{T}^{\varnothing}\) represents a baseline or 'null' action sequence, wherein the user does not perform any refinements.

### The challenge of textual treatments in causal human-LM collaboration

The causal estimand of ATE can be identified using standard tools in dynamic treatment regimes when the treatment is binary. For example, when each user can only choose between two actions to execute at each time step, we can identify the ATE using the \(g\)-formula,

\[\mathbb{E}\left[Y_{i}(a_{1},\ldots,a_{T})\right]=\int\mathbb{ E}\left[Y_{i}|\bar{A}_{iT}=\bar{a}_{T},\bar{L}_{iT}=\bar{l}_{T}\right]\] \[\times\prod_{t=1}^{T}P(L_{it}=l_{t}|\bar{A}_{i,t-1}=\bar{a}_{t-1} ,\bar{L}_{i,t-1}=\bar{l}_{t-1})\mathrm{d}\bar{l}_{T},\]

where \(\bar{A}_{it}=A_{i,1:t},\bar{a}_{t}=a_{1:t}\), \(\bar{L}_{it}=L_{i,1:t}\), and \(\bar{l}_{t}=l_{1:t}\)(Hernan and Robins, 2010).

This \(g\)-formula for ATE requires two conditions: the sequential exchangeability condition and the positivity (a.k.a. overlap) condition. The sequential exchangeability condition roughly requires that \(\bar{L}_{it},\bar{A}_{i,t-1}\) capture all confounders (variables that affect both \(A_{it}\) and \(Y_{i}\)) at time \(t\). This requirement is often satisfied in human-LM collaboration: the LM's text response \(\bar{L}_{it}\) and the previous actions \(\bar{A}_{i,t-1}\) are the only factors that affect both the outcome and the human's actions (cf. Fig. 2).

The second positivity condition, however, is often violated in handling textual treatments in human-LM collaboration. The positivity condition loosely requires that all possible values of the actions \(a_{t}\) shall occur with nonzero probability given any confounder values \(\bar{l}_{t}\). This condition is challenging to satisfy even when \(T=1\); the reason is that each \(a_{t}\) is a textual treatment--composed of a sequence of words--hence high-dimensional. It often violates the positivity condition in that many values of the treatment are not possible: a sequence of randomly chosen words likely occurs with zero probability as a human refinement strategy. This violation of positivity renders the ATE hard to identify in human-LM collaboration.

Moreover, the causal estimand involving the average treatment effect (ATE), represented as \(\mathbb{E}\left[Y_{i}(a_{1},\ldots,a_{T})-Y_{i}(a_{1}^{\varnothing},\ldots,a_ {T}^{\varnothing})\right]\), holds lim

Figure 2: The causal graph of human-LM collaboration (\(T\)=2).

 

ited practical significance for enhancing future human-LM interactions. Specifically, the treatment \(a_{t}\) usually refers to the post-refinement human text; for example, setting the text to "the prince proposed to the princess." The ATE in this context quantifies the effect of this specific post-refinement text on the outcome, regardless of the context (like the LM's preceding prompts or human responses). However, this ATE might not be practically useful since such text refinement cannot be universally applied across different human-LM collaboration scenarios. For instance, applying this edit is illogical when the initial LM prompt pertains to the animal world, rather than a story about a Prince and princess. Consequently, the traditional causal estimand of ATE is unsuitable for causal inference in scenarios involving textual treatments.

### Incremental Stylistic Effect (ISE): A causal estimand for textual treatments

Considering the limitations of the average treatment effect (ATE) in the context of textual treatments for human-LM collaboration, it becomes essential to ask: what is an appropriate causal estimand for these types of collaborations? We propose the causal estimand--_incremental stylistic effect_ (ISE), which focuses on the impact of _style change_ in human refinements, a concept we will define more formally later. Focusing on stylistic modifications helps overcome the challenges typically encountered with raw textual treatments. _Style Change_ as a treatment (e.g., editing the text to be more formal) can often meet the positivity condition; that is, all variations of the 'style change' treatment have a nonzero probability of occurrence since any style modification can be applied regardless of the confounder values. Moreover, it is also more practically relevant than ATE: regardless of the context, a user can always apply the "style change" treatment to the text, e.g., rewrite to be more formal.

More formally, we define the "style change" treatment as an intervention on some dimension-reducing function \(f_{t}(\cdot)\) of the post-refinement text and the LM's previous response \(f_{t}(A_{it},L_{it};\bar{A}_{i,t-1},\bar{L}_{i,t-1})\) (abbreviated as \(f_{t}(\bar{A}_{it},\bar{L}_{it})\)) at each time step; it captures how \(\bar{A}_{it}\) is different from \(L_{it}\) given previous histories \(\bar{A}_{i,t-1},\bar{L}_{i,t-1}\), hence revealing the style change performed by user \(i\) at time \(t\). Thus we define _incremental stylistic effect_ (ISE) as

\[\mathrm{ISE}=\lim_{\delta\to 0}[\mathbb{E}\left[Y_{i}(\{ f_{t}(\bar{a}_{t},\bar{L}_{it})+\delta\}^{T}_{t=1})\right]\\ -\mathbb{E}\left[Y_{i}(\{f_{t}(\bar{a}_{t},\bar{L}_{it})\}^{T}_{ t=1})\right]]/\delta, \tag{1}\]

where \(Y_{i}(\{f_{t}(\bar{a}_{t},\bar{L}_{it})\}^{T}_{t=1})\) denotes the potential outcome of executing the style change sequence \((f_{1},\ldots,f_{T})\); \(f_{t}(\bar{a}_{t},\bar{L}_{it})\) extracts the style feature of interest on text \(a_{t},L_{it}\) given history up to \(t-1\); it can characterize how \(a_{t}\) differs from \(L_{it}\) in terms of politieness (or formality), for instance. At a high level, ISE characterizes the impact of an infinitesimal style change sequence on the final outcome; we focus on infinitesimal changes due to the hardness to quantify style change.

We next define the conditional expected potential outcome over style change as follows,

\[\mathbb{E}\left[Y_{i}(\{f_{t}(\bar{a}_{t},\bar{L}_{it})\}^{T}_{t= 1})\mid\bar{L}_{iT}\right]\\ \triangleq\int\{\mathbb{E}\left[Y_{i}(\bar{a}^{\prime}_{T})\mid \bar{L}_{iT}\right]\\ \times\prod_{t=1}^{T}P(A_{it}=\bar{a}^{\prime}_{t}\mid\bar{L}_{it },\bar{A}_{i,t-1},\\ f_{t}(\bar{A}_{it},\bar{L}_{it})=f_{t}(\bar{a}_{t},\bar{L}_{it}))\} \mathrm{d}\bar{a}^{\prime}_{T}. \tag{2}\]

It describes the conditional potential outcome of the style change sequence \(f_{1:T}\) as the average outcome over all refinements \(\bar{a}^{\prime}_{T}\) that correspond to the same style change \(f_{1:T}(\bar{a}_{t},\bar{L}_{it})\) given contexts \(\bar{L}_{it}\)'s. This definition aligns with functional interventions (Puli et al., 2020; Correa and Bareinboim, 2020; Pearl, 2009; Wang and Jordan, 2021), where interventions are performed on some deterministic functions of high-dimensional treatments. It reflects the goal of assessing the impact of style change, regardless of context or specific text edit.

To estimate ISE from observational human-LM interaction data, we establish nonparametric identification conditions for the causal estimand ISE.

**Theorem 1** (Non-parametric identification of ISE).: _Under (1) the positivity condition for \(\{f_{t}(\cdot,\cdot)\}^{T}_{t=1}\), and (2) the sequential exchangeability condition, the ISE of the style change sequence \(f_{1:T}\) can be non-parametrically identified by plugging Eq. (3) into Eq. (1),_

\[\mathbb{E}\left[Y_{i}(\{f_{t}(\bar{a}_{t},\bar{L}_{it})\}^{T}_{t= 1})\right]\\ =\int\mathbb{E}|Y_{i}|\{f_{s}(\bar{A}_{is},\bar{L}_{is})=f_{s}( \bar{a}_{s},\bar{L}_{is})\}^{T}_{1},\bar{L}_{iT}=\bar{l}_{T}\\ \times\prod_{t=1}^{T}P(\bar{L}_{it}=\bar{l}_{t}\mid\{f_{s}(\bar{A }_{is},\bar{L}_{is})=f_{s}(\bar{a}_{s},\bar{L}_{is})\}^{t-1}_{s=1},\\ \bar{L}_{i,t-1}=\bar{l}_{t-1})\,\mathrm{d}\bar{l}_{T}.\] (3) 

The proof of Thm. 1 is in Appendix A. Thm. 1 generalizes the classical \(g\)-formula for ATE: when \(f_{t}(\cdot)\)'s are identity functions as opposed to dimensionality-reducing functions, Eq. (3) recovers the classical \(g\)-formula Hernan and Robins (2010). The sequential exchangeability condition requires \(Y_{i}(\bar{a}_{t})\perp A_{it}|\bar{A}_{i,t-1},\bar{L}_{t},\forall t=1,\ldots,T\)Hernan and Robins (2010). The positivity condition requires that \(P(f_{t}(A_{it},L_{it})\in\mathcal{V}|\bar{L}_{i,t-1})>0\) for any set \(\mathcal{V}\) that satisfies \(P(f_{t}(A_{it},L_{it})\in\mathcal{V})>0\)Imbens and Rubin (2015). Thm. 1 shows that, if we are interested in some style changes \(f_{t}\) that are common in historical human-LM interactions, one can employ Eqs. (1) and (3) to estimate their ISE. Such common style changes are more likely to satisfy the positivity condition; loosely, these style changes can be applied to any text, regardless of context.

### CausalCollab: An algorithm for dynamic human-LM Collaboration

We next operationalize Thm. 1 to perform causal inference for human-LM collaboration.

The first step is to extract common style changes \(f_{1:T}\) in historical human-LM collaboration dataset. These are the style changes often adopted by human users; their ISE can be calculated using Thm. 1. We perform this extraction by fitting a conditional variational autoencoder (CVAE) Zhao et al. (2017); Lopez-Martin et al. (2017); Mishra et al. (2018); Kim et al. (2021b) to all historical human-LM interactions \(\{\bar{A}_{iT},\bar{L}_{iT}\}^{I}_{i=1}\):

\[z_{it} \sim\mathcal{N}(0,\sigma^{2}I_{K}), \tag{4}\] \[A_{it}~{}|~{}\bar{A}_{i,t-1},z_{it},\bar{L}_{it} \sim\mathcal{N}(h(\bar{A}_{i,t-1},z_{it},\bar{L}_{it}),\sigma^{2}I _{d}),\]

where \(K\ll d\). We use variational inference Blei et al. (2017) to infer the latent variables \(z_{it}\), employing a variational approximating family of \(z_{it}\sim\mathcal{N}(\hat{f}_{t}(\bar{A}_{it},\bar{L}_{it}),\sigma^{2})\). Specifically, we maximize the evidence lower bound (ELBO) of the CVAE over \(h\) and \(\hat{f}\)Kingma Welling (2014). The resulting posterior mean of the latents \(z_{it}\) reveals the common style changes in the dataset. One may choose the style changes of interest as the posterior mean of \(z_{it}\) in the CVAE fit, i.e. setting \(f_{t}=\hat{f}_{t}\).2

Footnote 2: This procedure recovers conditional probabilistic PCA, if \(h(\cdot)\) is constrained to be linear Tipping and Bishop (1999).

Given the chosen style changes, we use Thm. 1 to estimate their ISEs. To employ Eq. (3), we fit an outcome model of \(Y_{i}\) against \(\{\hat{f}_{s}(A_{is},L_{is}),\bar{L}_{is},\bar{A}_{i,s-1}\}\). One example is generalized additive models3:\(Y_{i}=b_{1}(\hat{f}_{s}(\bar{A}_{is},\bar{L}_{is}))+b_{2}(\bar{L}_{is})+b_{3}( \bar{A}_{i,s-1})+\epsilon_{i}\), where both \(b_{k}(\cdot),k=1,2,3\) are neural networks and \(\epsilon_{i}\sim\mathcal{N}(0,\sigma^{2})\). We then employ Monte Carlo estimation Shapiro (2003) for the integrals in Eqs. (1) and (3). We leave the details of these steps to Appendix B.

Footnote 3: While any choice of outcome model is valid, we find that generalized additive models can often help reduce variance in the later Monte Carlo estimation Kroese et al. (2013).

These two steps constitute _CausalCollab_ Algorithm 1), an algorithm that performs causal inference for human-LM collaboration.

## 4 Empirical Studies

### Experiment setup and evaluation metrics

**Evaluation metrics of _CausalCollab_:** The quality of ISE estimates from _CausalCollab_ can be evaluated by evaluating the quality of the intermediate potential outcome estimates for \(\mathbb{E}\left[\{Y_{i}(\{f_{t}(a_{t},L_{it})\}_{i=1}^{T}]\}\right)\). It is because ISE is the limiting difference between two such potential outcome estimates. The potential outcome estimates can only be evaluated using semi-synthetic studies since potential outcomes are in general not observable in real data Imbens and Rubin (2015). Thus, we simulate semi-synthetic data from real human-LM interactions and assess the closeness between the actual potential outcome \(Y_{i}(\{f_{t}(a_{t},L_{it})\}_{i=1}^{T})\) and the _CausalCollab_ estimates of \(\mathbb{E}\left[Y_{i}(\{f_{t}(a_{t},L_{it})\}_{i=1}^{T})\right]\), uniformly averaging over all possible values of \(a_{t}\).

**Confounder, Outcome, Observational and Counterfactual Data:** To effectively test causal methods, both observational and counterfactual data are necessary Pearl (2009); Imbens and Rubin (2015). Constructing counterfactual language datasets presents a significant challenge because 

generating plausible "what if" scenarios requires a deep understanding of text and is often subjective as the possible counterfactual space of language is vast. Hence, we use ChatGPT to generate the counterfactual data thanks to their ability to generate a large volume of coherent texts given counterfactual instructions Li et al. (2023); Fryer et al. (2022). Thus, if for an observation, we ask ChatGPT to rewrite \(\bar{a}_{2}\) to counterfactual \(\bar{a}^{\prime}_{2}\) so that \(Y(\bar{a}^{\prime}_{2})=0\) and vice versa for \(Y(\bar{a}_{2})=0\).

We then identify a potential confounding signal \(X\) embedded within the high dimensional \(L_{i}\) that will decide both the outcome \(Y\) and the treatment \(\bar{A}_{2}\). For example, if the outcome is the formality of articles in the human-LM collaborative writing task, then the type of articles (e.g., argumentative or creative) can confound the impact of human refinement on the outcome. For instance, if, in the observational dataset, argumentative articles tend to be more formal and creative ones less formal, then for effectively testing our proposed ISE estimand, it's essential to see a non-existent or reversed correlation in the counterfactual data. In other words, we would need the creative articles to be more formal and the argumentative articles to be less formal in the counterfactual scenario.

After identifying the confounding signal, we establish an \(\alpha\sim\) split in the dataset to generate the observational and counterfactual data quantitatively. In the observational data, the probability \(P(Y=0|X=1)\) is set to \(\alpha\), and \(P(Y=0|X=0)\) to \(1-\alpha\). Conversely, in the counterfactual data, these probabilities are reversed: \(P(Y=0|X=1)\) becomes \(1-\alpha\) and \(P(Y=0|X=0)\) is \(\alpha\). For instance, with \(\alpha=0.2\), in observational data, an argumentative article (\(X=1\)) has a 80% (0.8) chance of being formal (\(Y=1\)), but in counterfactual data, this likelihood drops to 20%. Here, \(\alpha\) indicates the strength of the confounding correlation, with a lower \(\alpha\) (for \(\alpha<0.5\)) suggesting a stronger correlation. Our data generation approach remains the same for different \(\alpha\).

Labeling subjective textual outcomes (Y), like formality, is challenging. LMs trained on diverse textual datasets, often provide more consistent and accurate annotations for subjective tasks (such as political affiliation labeling, relevance assessment, and stance detection) than humans. Here again we employ ChatGPT for labeling task-specific outcomes. Detailed information about the prompts used for counterfactual generation and outcome labeling across different datasets is in Appendix D.

We employ three human-LM dynamic interaction datasets for our empirical study, each focusing on distinct tasks: CoAuthor Lee et al. (2022), Baize Xu et al. (2023), and DIALOCNAN Bonaldi et al. (2022). CoAuthor features 1445 human-GPT-3 collaborative writing tasks, focusing on the dynamic between human edits and language model suggestions in both creative and argumentative writing. Baize, created through self-chat with ChatGPT, contains over 200k dialogues from sources like Quora and StackOverflow, focusing on the helpfulness of revised answers in chats. DIALOCONAN addresses multi-turn counter-narrative generation against hate speech using a hybrid human-machine approach, editing dialoges generated by DialoGPT Zhang et al. (2020) and T5 Raffel et al. (2020). The dataset details are in Appendix C and are summarized in Table 1. More implementation details of CVAE and the Monte Carlo estimations for _CausalCollab_ are provided in Appendix F. For comparison, other than CVAE, we also implement a PCA to extract style changes adopted by humans. We refer to CVAE and PCA as _treatment embeddings_ in the results discussion as they both attempt to learn low-dimensional embeddings for human treatments.

### Results

**Quantitative analysis:** The results of using each method for predicting the outcome in each of the three datasets (averaged over three random seeds) are shown in Table 2. The performance is measured using Mean Squared Error (MSE) so a lower number indicates better performance. Across all three datasets, G-estimation + CVAE or PCA (Row 5 and 6) intervention significantly narrows the gap between observational and counterfactual MSE. Our approach improves counterfactual performance significantly while maintaining competitive or better observational performance in all three datasets compared to methods without adjustments.

The large performance gap between the counterfactual and observational performance when there is no confounder adjustment shows that baseline models heavily rely on confounders for prediction. However, using G-estimation alone without treatment embeddings for adjustment is not effective in closing the gap for the Baize and DIALOCONAN datasets and it also performs worse than the one with treatment embeddings in the CoAuthor dataset. Using treatment embeddings alone also 

shows significantly worse performance compared to when it is combined with G-estimation across all datasets. This suggests that integrating both G-estimation and treatment embeddings are crucial for causal observational and counterfactual predictions. CVAE consistently matches the performance of PCA, which implies that the human strategies underlying \(A_{i}\) may be invariant. If a human rewrites the LM output to be more formal, they may be rewriting phrases from the informal subspaces to formal subspaces, which can be linear and independent of the words themselves.

**Qualitative Analysis: What does \(z\) capture?** Our quantitative results show that the learned treatment embeddings successfully narrow the performance gap between the counterfactual and observational data. But what does \(z\) capture that can help predict the outcomes? We identify words in the human refinement \(A_{i}\) that are semantically closest to their learned treatment embeddings \(z_{i}\) from CVAE. As the outcome of the CoAuthor dataset, _formality_, is more intuitive for analysis than the other two, our following investigation is based on the CoAuthor dataset and specifically on the second step of the collaboration (\(A_{2}\)).

To embed words in \(A_{2}\), we trained a 50-dimensional Word2Vec model Mikolov et al. (2013) for consistency with \(z_{2}\)'s dimension. Cosine distances between Word2Vec embeddings of each word in \(A_{2}\) and the corresponding \(z_{2}\) were calculated. These distances, standardized and represented by color intensity (darker hues indicating closer proximity to \(z_{2}\) and stop words in black), are visualized for three different lengths of \(A_{2}\)s in Figure 3. The top 3 words that were closest to \(z_{2}\) are: 'Appreciate', 'purses', and 'individual' for Figure 3a; 'apprise', 'consequential', and 'wellness' for Figure 3b; 'jurisdiction', 'individuals', 'may' for Figure 3c. Conversely, words deemed informal or neutral, such as 'someone', 'people', and 'better', were observed to be more distant from \(z_{2}\). The qualitative findings suggest that the CVAE model is capable of learning explainable human strategies according to the outcomes of the task.

**Robustness to 1) \(\alpha\sim\text{split}\), 2) noise, and 3) dimension of \(z_{i}\):** We assess the robustness of our methods to \(\alpha\sim\text{split}\), noise, and the varying dimensions of the CVAE's latent space by conducting em

Figure 3: Three examples of \(A_{2}\) with different lengths. The color of words is decided by their cosine distance to the treatment \(z_{2}\) learned by the CVAE. The darker the color, the closer a word is to \(z_{i}\). Stopwords are black.

 

pirical studies on the CoAuthor dataset. These tests were performed with and without G-estimation, incorporating \(\alpha\in\{0,2,5,10,20,30,50\}\), noise levels \(\sigma\in\{0,0.5,1.0,1.5,2.0\}\) and dimensions for \(z_{i}\) set at \(\{2,20,50,100,200\}\).

As shown in Figure 4, our adjustment keeps the counterfactual and observational performances close consistently regardless of the intensity of the confounding correlations (Note the confounding correlation is stronger as for smaller \(\alpha\), i.e., \(\alpha<0.5\).) When \(\alpha\) is \(0.5\), the confounder \(L_{i}\) is independent of the independent variable \(A_{i}\), so the performances on both data are expected to be close. For noise and dimension of \(z_{i}\), detailed results are provided in the Appendix G; we observed that our approach significantly outperformed others in scenarios with higher noise levels. Additionally, it demonstrated minimal performance variation across the different dimensions of \(z_{i}\).

## 5 Conclusion

In conclusion, our work introduces the Incremental Stylistic Effect (ISE) as a novel causal estimand for human-LM collaboration, addressing the challenges posed by high-dimensional text-based treatments in causal inference. We establish the conditions for the non-parametric identification of ISE, providing a solid theoretical foundation for our approach. Building upon this, we propose the _CausalCollab_ algorithm, which leverages G-estimation for time-varying text variables and Conditional Variational Autoencoders (CVAE) to extract human strategies, enabling the estimation of ISE in practice.

Our research makes significant contributions to the field of human-LM collaboration by offering a fresh perspective on understanding the dynamics of these interactions through a causal lens. By focusing on the stylistic implications of text edits rather than specific wording changes, ISE provides a more practical and actionable approach to identifying effective collaboration strategies. The universal applicability of stylistic changes also ensures that the positivity condition in causal inference is met, overcoming the limitations of traditional causal estimands like Average Treatment Effect (ATE).

We demonstrate the effectiveness of our approach through extensive empirical studies on three diverse human-LM collaboration datasets. The results validate that _CausalCollab_ significantly narrows the gap between observational and counterfactual evaluations, highlighting its ability to accurately estimate the causal impact of interaction strategies. This has important implications for enhancing human-LM collaboration, as it enables users to make informed decisions about their editing strategies based on a deeper understanding of their causal effects.

In summary, our work introduces a novel causal estimand, establishes its theoretical properties, proposes a practical algorithm for its estimation, and validates its effectiveness through empirical studies. We believe that our contributions provide a valuable foundation for further research in this area and have the potential to significantly improve the dynamics of human-LM collaboration. As language models continue to advance and become more integrated into various domains, understanding and optimizing the interaction between humans and these models will be crucial. Our work takes an important step in this direction, and we hope it inspires further exploration of causal approaches to human-LM collaboration.

**Data & Code for this paper is available at: [https://github.com/XMUBQ/drr-text](https://github.com/XMUBQ/drr-text).**

Figure 4: Performances of our methods under different \(\alpha\sim\) split, levels of noise \(\sigma\), and dimensions of \(z_{i}\) on the coauthor dataset. As \(\alpha\) increases, the confounding correlation weakens, and our adjustment maintains similar counterfactual and observational performances. The method is robust to varying noise levels \(\sigma\), keeping both counterfactual and observational performance low. The choice of \(z_{i}\) dimension has minimal impact on performance, indicating that predictive treatments can be effectively represented in a low-dimensional space.

 



### Limitations

A limitation of our work is that it purely estimates the effectiveness of existing human-LM collaboration strategies, as opposed to directly finding the optimal collaboration strategies, which could be an interesting avenue for future work. Giving optimal strategies requires more complex methods and models like Q-learning with deep generative models to guide human behaviors.

## Ethical Considerations

Our use of OpenAI API as well as CoAuthor, DIALOCONAN, and BAIZE datasets follows their corresponding licenses. The API and datasets can be used for research purposes. As we rebuild the datasets to fit our task, the rebuilt datasets will also only be used for research purposes and cannot be redistributed. No information enables unique identifications of individual people. The DIALOCOMAN dataset collects offensive languages online. The original authors (Bonaldi et al., 2022) mentioned several guidelines to avoid the certain side effects of creating such a dataset. Our change to the dataset is more stylistic and we don't notice any change that violates their guidelines.

## Acknowledgments

This work was supported in part by the Office of Naval Research under grant number N00014-23-1-2590 and the National Science Foundation under Grant No. 2231174 and No. 2310831.

## References

* Avinesh et al. [2018] PVS Avinesh, Carsten Binnig, Benjamin Hattasch, Christian M. Meyer, and Orkan Ozyurt. 2018. Sherlock: A system for interactive summarization of large text collections. _Proc. VLDB Endow._, 11(12):1902-1905.
* Blei et al. [2017] David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. 2017. Variational inference: A review for statisticians. _Journal of the American Statistical Association_, 112(518):859-877.
* Bonaldi et al. [2022] Helena Bonaldi, Sara Dellantonio, Serra Sinem Tekiroglu, and Marco Guerini. 2022. Human-machine collaboration approaches to build a dialogue dataset for hate speech countering. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 8031-8049, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _Advances in Neural Information Processing Systems_, 33:1877-1901.
* Brynjolfsson et al. [2023] Erik Brynjolfsson, Danielle Li, and Lindsey R. Raymond. 2023. Generative AI at work. Technical report, National Bureau of Economic Research.
* Chakrabarty et al. [2022] Tuhin Chakrabarty, Vishakh Padmakumar, and He He. 2022. Help me write a poem: Instruction tuning as a vehicle for collaborative poetry writing. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 6848-6863, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Chowdhery et al. [2023] Akanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. PALM: Scaling language modeling with pathways. _Journal of Machine Learning Research_, 24(240):1-113.
* Chung et al. [2022] John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Huaran Lee, Eytan Adar, and Minsuk Chang. 2022. Talebrush: Sketching stories with generative pretrained language models. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_, CHI '22, pages 1-19, New York, NY, USA. Association for Computing Machinery.
* Cohen [1960] Jacob Cohen. 1960. A coefficient of agreement for nominal scales. _Educational and Psychological Measurement_, 20(1):37-46.
* Correa and Bareinboim [2020] Juan Correa and Elias Bareinboim. 2020. A calculus for stochastic interventions: Causal effect identification and surrogate experiments. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 34, pages 10093-10100. AAAI Press.
* Du et al. [2022a] Wanyu Du, Zae Myung Kim, Vipul Raheja, Dhruv Kumar, and Dongyeop Kang. 2022a. Read, revise, repeat: A system demonstration for human-in-the-loop iterative text revision. _Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022)_, pages 96-108.
* Du et al. [2022b] Wanyu Du, Vipul Raheja, Dhruv Kumar, Zae Myung Kim, Melissa Lopez, and Dongyeop Kang. 2022b. Understanding iterative revision from human-written text. _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_, pages 3573-3590.
* Egami et al. [2022] Naoki Egami, Christian J. Fong, Justin Grimmer, Margaret E. Roberts, and Brandon M. Stewart. 2022. How to make causal inferences using texts. _Science Advances_, 8(42):eabg2652.
* Epstein et al. [2016] Ziv Epstein, Aaron Hertzmann, Investigators of Human Creativity, Memo Akten, Hang Farid, Jessica

Fjeld, Morgan R. Frank, Matthew Groh, Laura Herman, Neil Leach, et al. 2023. Art and the science of generative AI. _Science_, 380(6650):1110-1111.
* Fryer et al. [2022] Zee Fryer, Vera Axelrod, Ben Packer, Alex Beutel, Jilin Chen, and Kellie Webster. 2022. Flexible text generation for counterfactual fairness probing. In _Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH)_, pages 209-229, Seattle, Washington (Hybrid). Association for Computational Linguistics.
* Gabriel et al. [2019] Raefer Gabriel, Yang Liu, Anna Gottardi, Mihail Eric, Anju Khatri, Anjal Chadha, Qinlang Chen, Behnam Hedayatnia, Pankaj Rajan, Ali Binici, et al. 2019. Further advances in open domain dialog systems in the third alexa prize socialbot grand challenge. _Amazon Science_.
* Goldfarb-Tarrant et al. [2019] Seraphina Goldfarb-Tarrant, Haining Feng, and Nanyun Peng. 2019. Plan, write, and revise: An interactive system for open-domain story generation. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_, pages 89-97, Minneapolis, Minnesota. Association for Computational Linguistics.
* Hernan and Robins [2010] Miguel A. Hernan and James M. Robins. 2010. Causal Inference.
* Higgins et al. [2017] Irina Higgins, Loic Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-VAE: Learning basic visual concepts with a constrained variational framework. _ICLR (Poster)_, 3.
* Imbens and Rubin [2015] Guido W. Imbens and Donald B. Rubin. 2015. _Causal Inference in Statistics, Social, and Biomedical Sciences_. Cambridge University Press.
* Kim et al. [2021a] Hyemi Kim, Seungjae Shin, JoonHo Jang, Kyungwoo Song, Weonyoung Joo, Wanmo Kang, and Il-Chul Moon. 2021a. Counterfactual fairness with disentangled causal effect variational autoencoder. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 35, pages 8128-8136.
* Kim et al. [2021b] Jaehyeon Kim, Jungil Kong, and Juhee Son. 2021b. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In _International Conference on Machine Learning_, pages 5530-5540. PMLR.
* Kingma and Welling [2014] Diederik P. Kingma and Max Welling. 2014. Autoencoding variational bayes. In _2nd International Conference on Learning Representations, ICLR 2014, Banffi, AB, Canada, April 14-16, 2014, Conference Track Proceedings_.
* Kroese et al. [2013] Dirk P. Kroese, Thomas Taimre, and Zdravko I. Botev. 2013. _Handbook of Monte Carlo Methods_. John Wiley & Sons, Hoboken, NJ.
* Lee et al. [2022] Mina Lee, Percy Liang, and Qian Yang. 2022. Coauthor: Designing a human-AI collaborative writing dataset for exploring language model capabilities. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_, CHI '22, New York, NY, USA. Association for Computing Machinery.
* Lee et al. [2023] Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard-Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, Rose E Wang, Minae Kwon, Joon Sung Park, Hancheng Cao, Tony Lee, Rishi Bommasani, Michael S. Bernstein, and Percy Liang. 2023. Evaluating human-language model interaction. _Transactions on Machine Learning Research_.
* Li et al. [2023] Yongqi Li, Mayi Xu, Xin Miao, Shen Zhou, and Tieyun Qian. 2023. Large language models as counterfactual generator: Strengths and weaknesses. _arXiv preprint arXiv:2305.14791_.
* Lopez-Martin et al. [2017] Manuel Lopez-Martin, Belen Carro, Antonio Sanchez-Esguevillas, and Jaime Lloret. 2017. Conditional variational autoencoder for prediction and feature recovery applied to intrusion detection in IoT. _Sensors_, 17(9):1967.
* Louizos et al. [2017] Christos Louizos, Uri Shalit, Joris M. Mooij, David Sontag, Richard Zemel, and Max Welling. 2017. Causal effect inference with deep latent-variable models. _Advances in Neural Information Processing Systems_, 30.
* Maas et al. [2013] Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In _Proceedings of the International Conference on Machine Learning_, volume 30, apage 3, Atlanta, Georgia.
* Mikolov et al. [2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. _arXiv preprint arXiv:1301.3781_.
* Mirowski et al. [2023] Piotr Mirowski, Cory W. Mathewson, Jaylen Pittman, and Richard Evans. 2023. Co-Writing screenplays and theatre scripts with language models: Evaluation by industry professionals. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_, pages 1-34.
* Mishra et al. [2018] Ashish Mishra, Shiva Krishna Reddy, Anurag Mittal, and Hema A. Murthy. 2018. A generative model for zero shot learning using conditional variational autoencoders. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_, pages 2188-2196.
* Naimi et al. [2014] Ashley I. Naimi, Erica E.M. Moodie, Nathalie Auger, and Jay S. Kaufman. 2014. Constructing inverse probability weights for continuous exposures: A comparison of methods. _Epidemiology_, pages 292-299.
* Noy and Zhang [2023]Shaked Noy and Whitney Zhang. 2023. Experimental evidence on the productivity effects of generative artificial intelligence. _Science_, 381(6654):187-192.
* OpenAI [2023] OpenAI. 2023. GPT-4 technical report 

Judea Pearl. 2009. _Causality_. Cambridge University Press.
* Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in python. _Journal of Machine Learning Research_, 12:2825-2830.
* Petersen et al. (2012) Maya L. Petersen, Kristin E. Porter, Susan Gruber, Yue Wang, and Mark J. Van Der Laan. 2012. Diagnosing and responding to violations in the positivity assumption. _Statistical Methods in Medical Research_, 21(1):31-54.
* Pryzant et al. (2021) Reid Przyant, Dallas Card, Dan Jurafsky, Victor Veitch, and Dhanya Sridhar. 2021. Causal effects of linguistic properties. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4095-4109, Online. Association for Computational Linguistics.
* Poli et al. (2020) Aahlad Poli, Adler Perotte, and Rajesh Ranganath. 2020. Causal estimation with functional confounders. _Advances in Neural Information Processing Systems_, 33:5115-5125.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _The Journal of Machine Learning Research_, 21(1):5485-5551.
* Roberts et al. (2020) Margaret E. Roberts, Brandon M. Stewart, and Richard A. Nielsen. 2020. Adjusting for confounding with text matching. _American Journal of Political Science_, 64(4):887-903.
* Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. _arXiv preprint arXiv:1910.01108_.
* Shapira et al. (2021) Ori Shapira, Ramakanth Pasunuru, Hadar Ronen, Mohit Bansal, Yael Amsterdam, and Ido Dagan. 2021. Extending multi-document summarization evaluation to the interactive setting. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 657-677, Online. Association for Computational Linguistics.
* Shapiro (2003) Alexander Shapiro. 2003. Monte Carlo sampling methods. _Handbooks in Operations Research and Management Science_, 10:353-425.
* Sridhar and Getoor (2019) Dhanya Sridhar and Lise Getoor. 2019. Estimating causal effects of tone in online debates. In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_, pages 1872-1878. International Joint Conferences on Artificial Intelligence Organization.
* Taubman et al. (2009) Sarah L. Taubman, James M. Robins, Murray A. Mittleman, and Miguel A. Hernan. 2009. Intervening on risk factors for coronary heart disease: an application of the parametric g-formula. _International journal of epidemiology_, 38(6):1599-1611.
* Tipping and Bishop (1999) Michael E. Tipping and Christopher M. Bishop. 1999. Probabilistic principal component analysis. _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_, 61(3):611-622.
* Van der Laan et al. (2011) Mark J. Van der Laan, Sherri Rose, et al. 2011. _Targeted Learning: Causal Inference for Observational and Experimental Data_, volume 4. Springer.
* Veitch et al. (2020) Victor Veitch, Dhanya Sridhar, and David Blei. 2020. Adapting text embeddings for causal inference. In _Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI)_, volume 124 of _Proceedings of Machine Learning Research_, pages 919-928. PMLR.
* Wang and Jordan (2021) Yixin Wang and Michael I. Jordan. 2021. Desiderata for representation learning: A causal perspective. _arXiv preprint arXiv:2109.03795_.
* Xu et al. (2023) Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2023. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 6268-6278, Singapore. Association for Computational Linguistics.
* Zhang et al. (2020) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT: Large-scale generative pre-training for conversational response generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 270-278, Online. Association for Computational Linguistics.
* Zhao et al. (2017) Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. _arXiv preprint arXiv:1703.10960_.

 

Proof of Thm. 1

We first prove Eq.3. Thm.1 is then an immediate consequence of Eqs.1 and 3,

Begin with the definition of expected potential outcome over style change in Eq.2

\[{\mathbb{E}}\left[\{Y_{i}(\{f_{t}(\bar{a}_{t},\bar{L}_{it})\}^{T}_{ t=1})\mid\bar{L}_{iT}\right]\] \[\triangleq\int\{{\mathbb{E}}\left[Y_{i}(\bar{a}^{\prime}_{T}) \mid\bar{L}_{iT}\right]\] \[~{}~{}~{}~{}\times\prod_{t=1}^{T}P(A_{it}=\bar{a}^{\prime}_{t} \mid\bar{L}_{it},\bar{A}_{i,t-1},\] \[~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}f_{t}(\bar{A}_{it},\bar{L}_{it}) =f_{t}(\bar{a}_{t},\bar{L}_{it}))\}{\rm d}\bar{a}^{\prime}_{T} \tag{5}\] \[=\int\{{\mathbb{E}}\left[Y_{i}\mid\bar{A}_{iT}=\bar{a}^{\prime}_ {T},\bar{L}_{iT}\right]\] \[~{}~{}~{}~{}~{}\times\prod_{t=1}^{T}P(A_{it}=\bar{a}^{\prime}_{t} \mid\bar{L}_{it},\bar{A}_{i,t-1},\] \[~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}f_{t}(\bar{A}_{it},\bar{L}_{ it})=f_{t}(\bar{a}_{t},\bar{L}_{it}))\}{\rm d}\bar{a}^{\prime}_{T}\] (6) \[=\int\{{\mathbb{E}}\left[Y_{i}\mid f_{T}(\bar{A}_{iT},\bar{L}_{ iT})=f_{T}(\bar{a}_{T},\bar{L}_{iT}),\right.\] \[~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\bar{a}^{\prime}_{T-1}, \bar{L}_{i,T}\Big{]}\] \[~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }\times\prod_{t=1}^{T-1}P(A_{it}=\bar{a}^{\prime}_{t}\mid\bar{L}_{it},\bar{ A}_{i,t-1},\] \[~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}f_ {t}(\bar{A}_{it},\bar{L}_{it})=f_{t}(\bar{a}_{t},\bar{L}_{it}))\}{\rm d}\bar{ a}^{\prime}_{T-1}\] (7) \[={\mathbb{E}}\left[Y_{i}\mid\{f_{t}(\bar{A}_{it},\bar{L}_{it})=f_ {t}(\bar{a}_{t},\bar{L}_{it})\}^{T}_{t=1},\bar{L}_{iT}\right] \tag{8}\]

The second equality is due to the sequential exchangeability condition \(Y_{i}(\bar{a}_{t})\perp A_{it}\mid\bar{A}_{i,t-1},\bar{L}_{t}\) for all \(t=1,\ldots,T\). The third equality integrates out \(a^{\prime}_{T}\) while keeping \(\bar{a}^{\prime}_{T-1}\). The fourth equality repeats the integration for all \(a^{\prime}_{1:T}\).

Therefore, we have

\[{\mathbb{E}}\left[\{Y_{i}(\{f_{t}(\bar{a}_{t},\bar{L}_{it})\}^{ T}_{t=1})\}\right] \tag{9}\] \[={\mathbb{E}}\left[{\mathbb{E}}\left[\{Y_{i}(\{f_{t}(\bar{a}_{t}, \bar{L}_{it})\}^{T}_{t=1})\mid\bar{L}_{iT}\}\right]\right]\] (10) \[=\int{\mathbb{E}}\left[Y_{i}\mid\{f_{t}(\bar{A}_{it},\bar{L}_{it} )=f_{t}(\bar{a}_{t},\bar{L}_{it})\}^{T}_{t=1},\right.\] \[~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }\bar{L}_{i,T}=\bar{l}_{T}\big{]}\] \[\times\prod_{t=1}^{T}P(\bar{L}_{it}=\bar{l}_{t}\mid\{f_{s}(\bar{ A}_{is},\bar{L}_{is})=f_{s}(\bar{a}_{s},\bar{L}_{is})\}^{t-1}_{1},\] \[~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }\bar{L}_{i,t-1}=\bar{l}_{t-1})\,{\rm d}\bar{l}. \tag{11}\]

The first equality is due to the tower property. The second equality is due to a similar calculation as in the classical \(g\)-formula for binary treatments, following the causal graph in Fig.2 (Hernan and Robins, 2010). Eq.11 is the same as Eq.3; thus, Thm.1 is proved.

## Appendix B Details of _CausalCollab_

### Monte Carlo for G-estimation

As G-estimation is traditionally used for binary time-varying tasks, we aim to expand it to language tasks where the treatment and confounder texts are high-dimensional. In a two-step example, the primary objective of G-estimation is to estimate the conditional expectation

\[{\mathbb{E}}(Y|\bar{A}_{2}=\bar{a}_{2})={\mathbb{E}}(Y|A_{1}=a_{1},A_{2}=a_{2})\]

By the law of total expectation, it equals

\[\sum_{l_{1},l_{2}}{\mathbb{E}}\left(Y\middle|\bar{A}_{2}=\bar{a}_{2 },\bar{L}_{2}=\bar{l}_{2}\right)\] \[~{}~{}~{}~{}~{}~{}~{}~{}~{}\times P(L_{2}=l_{2}|A_{1}=a_{1},L_{1}= l_{1})\] \[~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\times P(L_{1}=l_{1}) \tag{12}\]

To estimate this, we first train a logistic regression to obtain \(P(Y=1|\bar{A}_{2},\bar{L}_{2})\) which equals to the expectation \({\mathbb{E}}\left(Y\middle|\bar{A}_{2}=\bar{a}_{2},\bar{L}_{2}=\bar{l}_{2}\right)\) when the outcome is binary. Then, apply a Monte Carlo approach in the following steps:

* Assuming \(L_{2}\) follows a Gaussian distribution \(\mathcal{N}(\mu(A_{1},L_{1}),I_{d})\) given previous steps where \(\mu\) can be approximated by an MLP trained on the observational data. For simplicity, the variance is set to \(I_{d}\) where \(d\) is the initial dimension of the text variables.
* Sampling \(L_{1}\): For each observation of \(a_{1},a_{2}\), first draw \(n_{1}\) samples of \(l_{1}\) from the entire observational data.
* Sampling \(L_{2}\): Under the sampled \(l_{1}\), draw \(n_{2}\) samples of \(l_{2}\) from the Gaussian distribution \(\mathcal{N}(\mu(l_{1},a_{1}),\sigma^{2})\) where \(\mu\) is the trained MLP mentioned above.
* With all sampled variables, plug them into the trained logistic regression model to get \(P(Y=1|\bar{A}_{2}=\bar{a}_{2},\bar{L}_{2}=\bar{l}_{2})\), which is equivalently the expectation when \(Y\) is binary. Then take the average of the expectations over all samples to approximate equation12 to estimate the causal effect \({\mathbb{E}}[Y|\bar{A}_{2}=\bar{a}_{2}]\).
* To study the effect of style changes adopted by humans, replace \(\bar{A}_{2}\) with the latent variables \(\bar{z}_{2}\) learned from CVAEs at each time step during the sampling and compute the target expectation 

Dataset Details

### CoAuthor

Lee et al. (2022) presents the design and creation of a novel dataset aimed at studying human-LM collaboration in writing tasks. It is designed to have a better understanding of language model capabilities and limitations and prompt the development of more effective human-LM collaborative writing systems. The dataset includes 1445 collaborative writing tasks where 830 are creative story writings and 615 are argumentative essay writing. On average, each article has 11.8 suggestion queries. Human participants receive suggestions from GPT-3 based on the current text and then decide to select, reject, or revise the suggestions. The acceptance rate of LM suggestions by humans is 72.3%. However, in all texts, 72.6% are still completed by the human writer. This means that LMs can provide good intuition, but humans still have more complex strategies (revising) to have better outcomes for the article.

Here, \(L_{i}\) represents the accumulated texts at time step \(i\) plus the LM's suggestions. \(A_{i}\) represents selected and revised suggestions from human. If the LM's suggestions are rejected at step \(i\), \(A_{i}\) is set to none. The outcome is the _formality_ of the article. The confounding signal is the type of articles, either argumentative or creative as given by the original dataset. When the confounding correlation \(\alpha=0.2\), in the observational data, there are 800 articles with formal outcomes and 582 with informal outcomes; in the corresponding counterfactual data, there are 667 formal articles and 715 informal articles. Quality control of outcome labels was performed on a small scale (n = 100) for ChatGPT-generated examples using a human annotator. The labels generated by the human and AI annotators yielded a significant positive correlation Cohen's \(\kappa\) = 0.87 Cohen (1960)).

### Baize

Xu et al. (2023) propose an innovative chat data collection procedure using the ChatGPT in a self-chat setup for efficiently fine-tuning large language models. ChatGPT alternated between the roles of a human and an LM assistant to complete a chat. A question sourced from popular online platforms like Quora and StackOverflow was considered as the initial dialogue prompt. The LM will answer the question and the human will rewrite the answer to make it more helpful. This iterative process continues throughout the chat. Baize consists of more than 200k dialogues across multiple domains in total. In our settings, we randomly select the 1260 dialogue samples whose topics cover a wide range of programming languages like C++ and Python sourced from StackOverflow. The average interaction turn in this part of the dataset is 3.81. Unlike the coauthor dataset where the confounding signal is naturally defined in the original dataset, Baize does not have tentative answers as the data is built to provide a high-quality chat corpus. Thus, we again ask ChatGPT to rewrite half of the original \(A_{i}\) to be more tentative and the other half to be more confident without losing their original information. The prompts used for confidence rewriting are listed in Appendix D.3. The process of building counterfactual data, outcome labeling, and split of data remains the same as described in SS4.1.

\(L_{i}\) here represents all finished turns of the chat plus the LM's original answer in time step \(i\). \(A_{i}\) represents the rewritten answer by the human. The outcome \(Y\) is the helpfulness of the adjusted answers to the questions. The confounding signal is the confidence of the answer. Tentative answers with less confidence may be associated with less helpful answers. When the confounding correlation \(\alpha=0.2\), in the observational data, there are 668 dialogues with helpful outcomes and 582 with unhelpful outcomes; in the corresponding counterfactual data, there are 590 helpful dialogues and 660 unhelpful dialogues.

### DialOCONAN

DIALOCONAN Bonaldi et al. (2022) contributes the first large-scale dialogue dataset for training multi-turn counter-narrative (CN) generation models against hate speech (HS). The authors use a hybrid human-machine approach to generate the dialogues: concatenating and paraphrasing existing HS/CN pairs and using generative language models like DialoGPT Zhang et al. (2020) and T5 Raffel et al. (2020) to generate completely new dialogues. The generated dialogues are edited and validated by trained human annotators.

Similar to Baize, \(L_{i}\) represents the HS plus the LM's original CN. \(A_{i}\) represents the rewritten CN by the human. The outcome \(Y\) is the effectiveness of the adjusted CN in refuting the HS. The confounding signal is the formality of the CN as a more formal response may be more effective in addressing the HS. We randomly select 1200 dia 

logues from the original dataset and the average interaction turn in this part is 2.71. Again due to the lack of confounding signals in the original dataset, we add the confounding signal by randomly rewriting half of the original \(A_{i}\) to be more informal and the other half to be more formal similar to what we did for Baize. More examples of the synthetic generation for the three datasets are shown in Appendix E. When the confounding correlation \(\alpha=0.2\), in the observational data, there are 615 dialogues with effective outcomes and 585 with ineffective outcomes; in the corresponding counterfactual data, there are 569 effective dialogues and 631 ineffective dialogues.

## Appendix D Prompts

For all prompts mentioned below, we are using gpt-3.5-turbo with them in the OpenAI API.4

Footnote 4: [https://platform.openai.com/docs/models/gpt-3-5](https://platform.openai.com/docs/models/gpt-3-5)

### For Counterfactual Generation

#### d.1.1 CoAuthor

For the coauthor dataset, the prompt for counterfactual generation is: "Paraphrase the following text using more [formality] language: [TEXT]" where [formality] can be either 'casual' or 'formal' and [TEXT] is the observational text.

#### d.1.2 Baire

For Baize dataset, as the outcome is more complex, the LMs need some contexts and examples to generate quality output. The prompt is:

"Given the following incomplete conversation between the Inquirer and the Responder, please complete the Responder's answers by replacing '[ANSWER]'. The inquirer's questions are already given to you. The responder's answers should NOT be helpful at all to the Inquirer's questions but remain in the main topic of the conversation. You should give the complete conversation including ALL inquirer's questions and responder's answers. I will give you an example first. Suppose you get an incomplete conversation like this:

Inquirer: How do I make a flat list out of a list of lists in python?

Responder: [ANSWER]

Inquirer: This is hard for me to understand. Could you explain more about it?

Responder: [ANSWER]

A sample completion of the above conversation:

Inquirer: How do I make a flat list out of a list of lists in python?

Responder: It's similar to theoretical physics' parallel universes, where individual universes (lists) are unified in a multiverse system, presenting a compelling analogy.

Inquirer: This is hard for me to understand. Could you explain more about it?

Responder: Certainly! Have you ever pondered the intricacies of the quantum yield in photosynthesis? It's a fascinating phenomenon that involves the conversion of light energy into chemical energy.

Now here's the conversation you should complete: [CONV]" where [CONV] is the conversation to be completed.

#### d.1.3 DialOCONAN

Similar to Baize, We give examples and contexts to ChatGPT in the prompt. We also ask it to generate the reason why ChatGPT believes its outputs are more effective in order to get more reasonable responses. The prompt for DIALCONAN dataset is:

"Given the following incomplete conversation between two persons, please complete the conversation by replacing '[REPLY]'. The countermeasures (CN) are trying to refute the hate speech (HS). However, the reply you generate should NOT be effective in refuting the HS at all. The reply should not be too long. You should give the complete conversation including ALL previous HS and CN. Also, give an explanation of why you think the reply is not effective following the conversation starting with 'Explanation:'. I will give you an example first. Suppose you get an incomplete conversation like this:

HS: There's a rumour that Terry Gilliam's new production has been cancelled because the theatre is concerned about his anti-trans statements. More cancel culture madness!

CN: [REPLY]

HS: Its dangerous censorship and an assault on free speech.

CN: [REPLY]

A sample output should be like:

HS: There's a rumour that Terry Gilliam's new production has been cancelled because the theatre is concerned about his anti-trans statements. More cancel culture madness!

CN: I think Terry Gilliam has made some great movies in the past.

HS: Its dangerous censorship and an assault on 

free speech.

CN: Some of his films really have unique storylines.

Explanation: The replies are not effective in refuting the hate speech because they merely comment on Terry Gilliam's past work and don't address the issues of cancel culture, alleged anti-trans statements, or concerns about censorship and free speech. The replies are vaguely relevant in that they pertain to Terry Gilliam, but they don't engage with the main points being made by the hate speech.

Here's the conversation we request you to complete: [CONV]" where [CONV] is the conversation to be completed.

### For Outcome Labeling

#### d.2.1 CoAuthor

The ChatGPT will receive a system prompt first: "You are able to decide the formality of the given text.". Then the prompt for rating the formality is: "[TEXT] Decide the formality of the above text. Reply in one word with either formal or informal only." where [TEXT] is the text to be rated.

#### d.2.2 Baize

We ask CHATGPT to also generate reasons for its ratings of helpfulness based on several metrics to ensure the reliability of the outputs: "We request you to evaluate the helpfulness of the Responder in response to the Inquirer's questions from the following conversation. Please indicate whether the Responder's answers are 'Helpful' or 'Not Helpful' to the inquirer's questions in the first line of your output. Then starting from the second line, please provide a comprehensive explanation of your evaluation from perspectives of clarity, factualness, relevance and comprehensiveness, ensuring objectivity and avoiding any potential bias. However, you can be and tough grader and your explanation shouldn't be too long. Here's the conversation: [CONV]" where [CONV] is the conversation to be rated.

#### d.2.3 Dialoconal

As we already asked ChatGPT to generate reasons for counterfactual rewriting, here we didn't rate its effectiveness again. We just treated the counterfactual rewriting as "ineffective" and the original as "effective" due to the nature of the data collection procedure of the original dataset.

### For Confounding Signal

#### d.3.1 Baize

To rewrite the text with more confidence: "We request you to paraphrase a piece of text using language with more confidence. The text is an answer to a question raised by a human. You don't need to know the exact question but you can paraphrase under an imagination of a conversation between two humans. One person tries to give an answer with more confidence. Please output your paraphrase directly. Here's the text: [TEXT] Here's your output:" where [TEXT] is the text to be rewritten.

To rewrite the text with less confidence: "We request you to paraphrase a piece of text using language with more uncertainty. The text is an answer to a question raised by a human. You don't need to know the exact question but you can paraphrase under an imagination of a conversation between two humans. One person tries to give an answer but may lack confidence. Please output your paraphrase directly. Here's the text: [TEXT] Here's your output:"

#### d.3.2 Dialoconan

To rewrite the text to be more formal: "We request you to paraphrase a piece of text using more formal language. The text is a counter-narrative to heate speech. You don't need to know the exact context but you can paraphrase under an imagination of a conversation between two humans. One person tries to give a formal counter-narrative. Please output your paraphrase directly without any prefixes. Here's the text: [TEXT] Here's your output:" where [TEXT] is the text to be rewritten.

To rewrite the text to be more casual: "We request you to paraphrase a piece of text using more casual language. The text is a counter-narrative to heate speech. You don't need to know the exact context but you can paraphrase under an imagination of a conversation between two humans. One person tries to give a casual counter-narrative. Please output your paraphrase directly without any prefixes. Here's the text: [TEXT] Here's your output:"

## Appendix E Synthetic Generation Examples

### Coauthor

A creative story but was rewritten to be more formal as highlighted in yellow:

"Humans once wielded formidable magical power. But with over 7 billion of us on the planet now, Mana has spread far too thinly to have any 

effect. When hostile aliens reduce humanity to a mere fraction, the survivors discover an old power has begun to reawaken once again. The initial manifestation of the intriguing novel capability resides in the aptitude to decelerate the passage of time for approximately 6 seconds. This phenomenon holds practical applicability across various scenarios. One instance pertains to a situation where a man faced a potential car collision with a tree, but due to his heightened state of panic, he was able to manipulate time and decelerate its flow. Consequently, he successfully accessed a means of exit-disembarked from the vehicle, and landed onto some vegetation, uninjured, while the automobile combusted into fragments that did not include any components of his physical being!"

### Baize

An example of helpful with less confidence responses:

Inquirer: I'm having an issue with Flot's Pie Chart not showing up in IE7 when using Javascript.

Responder: Hmm, I think I might have some additional information on that matter.

Inquirer: Sure, the chart appears fine in other browsers, but in IE7 it doesn't show up at all.

Responder: I'm not entirely sure, but I think it might be worth considering if the version of Flot you're currently using is compatible with IE7.

### Dialoconan

An example of an ineffective and casual countermarrative to hate speech.

HS: Gays are sick.

CN: Oh, dude, I know this awesome guy who's gay and seriously the nicest and most caring person you'll ever meet.

HS: Are you sure? Gays are poisoning our children with their ricky homosexuality.

CN: Hey, did you know about this one person I know who's gay? They're actually super successful in their career. It just goes to show that being gay doesn't define someone's abilities or achievements.

## Appendix F Details of _CausalCollab_ Implementation in Empirical Studies

Treatment Embeddings.For CVAE, both the encoder (mean and variance network) and decoder are two-layer MLP to avoid overfitting on relatively small datasets. The initial dimension for the latent variable \(z_{i}\) is 50 and we will present the results for different dimensions in SS G.2. CVAE models are trained for 500 epochs and the learning rate is \(1e^{-4}\). The loss is a weighted sum of the MSE decoding loss and the KL divergence loss following beta-VAE Higgins et al. (2017). For PCA, it decomposes \(A_{i}\) to the same dimension as CVAE and plug it into the G-estimation.

Monte Carlo Estimation for Eqs. (1) and (3).The MLP for sampling \(L_{2}\) has three layers. The hidden dimension between the layers is 128. Batch normalization and LeakyReLU activation Maas et al. (2013) are deployed between the layers. The model is trained for 1000 epochs and the learning rate is \(1e^{-5}\).

We use the scikit-learn Pedregosa et al. (2011) implementation for logistic regression as the outcome model. The solver is 'liblinear' and the regularizer is 1. The number of samples for \(L_{1}\) and \(L_{2}\) are both 50 so for each Monte Carlo estimation we will have \(n_{1}\times n_{2}=2500\) samples.

Text Variable Representations.We use uncased base distilbertSanh et al. (2019) to get embedding representations for all text variables so the dimension of the representation is 768 which highlights the challenge of the task in the causal context. We add a Gaussian noise \(\mathcal{N}(0,\sigma^{2})\) to the representation of \(A_{i}\) to further test the robustness of our methods. The initial \(\sigma\) is 1 and the influence of different levels of noise is shown in SSG.1.

For all three datasets, we do a 5-fold cross-validation and report the average performance.

## Appendix G Details of Robustness Analysis

### Noise

We evaluated our methods' robustness to noise by testing our model with and without G-estimation in the CoAuthor dataset with noise levels \(\sigma\in\{0,0.5,1.0,1.5,2.0\}\). The treatment embedding is learned by CVAE. As expected, results shown in Figure 4 indicated a performance decline with higher noise but our methods are robust to different levels of noise by keeping both counterfactual and observational performances low and comparable. Our methods manifest more advantages when the noise is larger.

### Dimension of \(z_{i}\)

We examined the effect of \(z_{i}\)'s dimensionality on our method's performance, testing our model with and without G-estimation in the CoAuthor dataset 

with dimensions of \(\{2,20,50,100,200\}\). The treatment embedding is learned by CVAE. As shown in Figure 4, the performances exhibit a minimal variation (\(3\%\)) for different choices of dimensions. This suggests that the predictive treatments encoded by \(z_{i}\) can be effectively represented within a low-dimensional space as we proposed.