SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lai Wei
Life Sciences Institute
weilatim@gmail.com
Muhammad Qasim Elahi
elahi0@purdue.edu
Mahsa Ghasemi
mahsa@purdue.edu
Murat Kocaoglu
mkocaglu@purdue.edu
Structural causal bandit provides a framework for decision-making problems when causal information is available. It models the stochastic environment with a structural causal model (SCM) that governs the causal relations between random variables. In each round, an agent applies an intervention (or no intervention) by setting certain variables to some constants, and receives a stochastic reward from a non-manipulable variable. Though the causal structure is given, the observational and interventional distributions of these random variables are unknown beforehand, and they can only be learned through interactions with the environment. Therefore, to maximize the expected cumulative reward, it is critical to balance the exploration-versus-exploitation tradeoff. We consider discrete random variables with a finite domain and a semi-Markovian setting, where random variables are affected by unobserved confounders. Using the canonical SCM formulation to discretize the domains of unobserved variables, we efficiently integrate samples to reduce model uncertainty, gaining an advantage over those in a classical multi-armed bandit setup. We provide a logarithmic asymptotic regret lower bound for the structural causal bandit problem. Inspired by the lower bound, we design an algorithm that can utilize the causal structure to accelerate the learning process and take informative and rewarding interventions. We establish that our algorithm achieves a logarithmic regret and demonstrate that it outperforms the existing methods via simulations.
Sequential decision-making in uncertain environments is one of the most fundamental problems across scientific disciplines, including robotics, economics, social science, clinical trials, and agriculture [32, 26, 36, 7, 3]. In almost all these applications, it is required to implement interventions to control some aspects of the system to optimize an outcome of interest. In economics and political science, the government could use fiscal policy including changing the interest rate and taxation to achieve regulation and control of a country's economy. In this paper, we deal with designing an adaptive policy in an uncertain stochastic environment to learn and apply optimal interventions.
In sequential decision-making problems, it is critical to balance the explore-versus-exploit tradeoff, which deals with choosing between informative options and empirically more rewarding alternative options. The multi-armed bandit (MAB) [33, 30, 18] is a classical problem formulation for achieving such a tradeoff. In MAB, a decision-maker sequentially chooses an action corresponding to pulling
an arm from a set of arms, each generating stochastic rewards with probability distribution unknown to the decision-maker. The objective is to achieve the maximum cumulative reward over a time horizon, or equivalently to minimize the regret, which measures suboptimality in cumulative rewards against selecting the best action all the time. The classic stochastic bandit formulation only considers the connection between an action and its expected reward. However, in real-world applications, an action can affect the entire system in a more complex way. As the agricultural example given in [19], changing the moisture level could affect the temperature and nutrient absorption, which are also crucial to crop yields. The relations between such factors cannot be characterized by the MAB.
Causal knowledge enables us to model the causal relations between multiple factors in the environment. In machine learning research, it has been noted that causal models could help improve the generalization performance when data distribution shifts [11], and address unstable translation of successful reinforcement learning methods in simulation to real-world problems [12] such as self-driving cars and recommendation systems. The casual bandit models the stochastic environment with a causal graph that connects a set of random variables of interest, and actions are modeled by interventions on different subsets of nodes. It provides a critical interpretation of causal influences among random variables [28] that allows us to reason about what will happen to the environment after certain interventions are made to the data-generating process. Compared to the classic MAB with only reward feedback, the casual bandit represents a more detailed model of real-world decision-making problems, including richer feedback from multiple random variables and the causal relations among them. By applying causal reasoning, the learning process can be accelerated and more rewarding interventions can be selected. In applications where the space of intervention is large and explicit experiments are costly, such as social science or economics, it has special value.
Due to the reasons mentioned above, there has been a recent surge of interest in the research community on causal decision-making. Some early attempts include [6], in which the authors took a causal approach to study how to make final decisions based on natural predilections, and proposed an algorithm by modifying Thompson sampling [33]. In [19], the investigators started from a parallel causal structure and extended the algorithm to the general causal bandit problem. Subsequent work [37] took preprocessing steps to get rough estimates of interventional distributions and use them to design efficient experiments to discover the best intervention. In [1], the Bayesian optimization technique is employed to solve the same problem.
The most closely related works focusing on regret minimization in the structural causal bandit with confounders are [22, 23]. In both works, the authors took a graphical characterization using do-calculus [28] to reduce the large intervention set to the possibly optimal minimal intervention set and run a KL-UCB algorithm [14] within the reduced action space. The later work is an extension of the previous one that includes non-manipulable variables. Other lines of causal bandit research normally assume additional prior information, such as infinite observational data [39] or marginal interventional distributions [24, 8]. A general linear model is adopted in [13], which is different from the non-parametric setup in this paper.
In this paper, we focus on the MAB sequential decision-making problem where the arms correspond to interventions on an arbitrary causal graph, including unobserved confounders affecting multiple pairs of variables (also known in causal terminology as the semi-Markovian setting). We recognize the structural causal bandit falls into the category of structured bandit [10, 35]. It has been noted the classical upper confidence bound (UCB) algorithm and Thompson sampling may not fully leverage structural information [31, 20]. It is worth considering the potential limitations of algorithms based on these ideas, such as CRM-AL [25] and generalized Thompson sampling [27]. To address this issue and make systematic use of a known causal structure, we design an algorithm that can meticulously utilize causal structural information. In particular, we make the following contributions:
- We provide a logarithmic asymptotic regret lower bound for the structural causal bandit problem with latent confounders by leveraging the canonical SCM formulation [40] to create a space of all possible interventional distributions given a causal graph.
- Inspired by the lower bound, we extend [22] and apply approximate allocation matching to design an algorithm that can effectively utilize causal information provided by the causal graph to accelerate the learning process and take informative and rewarding interventions.
- We analyze the algorithm to provide a problem-dependent logarithmic upper bound on expected regret and compare it with [22] to show it has a better theoretical guarantee.
- We complement the theoretical results by evaluating the proposed approach in a variety of experimental settings featuring different causal structures. We show that our algorithm outperforms the existing baselines in terms of empirical regret.
2 Problem Statement and Background
In this section, we present the formulation of the structural causal bandit problem with latent confounders. We follow the convention to represent a random variable and its value in a capital letter and a lowercase letter respectively. A multivariate random variable is represented in a bold letter.
2.1 Structural Causal Model
Our approach adopts the structural causal model (SCM) [29] to provide a causal perspective on the data-generating process. This allows us to express the relationship between variables and capture important causal concepts, including unobserved variables, observational distributions, and interventional distributions. For a random variable  $\mathbf{X}$ , let  $\Omega(\mathbf{X})$  denote its domain.
Definition 1 (SCM). A structural causal model is a 4-tuple  $\langle \mathbf{U},\mathbf{V},\mathbb{F},P(\mathbf{U})\rangle$  where:
1.  $\mathbf{U}$  is a set of unobserved independent background variables (also called exogenous), that determine the randomness of the model.
2.  $\mathbf{V} := \{V_i \mid i = 1, \dots, n\}$  is the set of observable variables (also called endogenous).
3.  $\mathbb{F}$  is a set of structural equations  $\{f_1, f_2, \dots, f_n\}$  such that for each  $V_i \in \mathbf{V}$ ,  $f_i : \Omega(\mathbf{U}_i) \times \Omega(\mathsf{P}a_i) \to \Omega(V_i)$  defines a mapping, where  $\mathbf{U}_i \subseteq \mathbf{U}$  and  $\mathsf{P}a_i \subseteq \mathbf{V}$  is the parent set of  $V_i$ . So  $\mathbb{F}$  as a whole determines a mapping relationship from  $\Omega(\mathbf{U})$  to  $\Omega(\mathbf{V})$ .
4.  $P(\mathbf{U})$  is a joint probability distribution over all the exogenous variables.
Each structural causal model is associated with a causal graph  $\mathcal{G} = \langle \mathbf{V},\mathcal{E},\mathcal{B}\rangle$ , where the node set  $\mathbf{V}$  corresponds to the set of observable variables,  $\mathcal{E}$  is the set of directed edges and  $\mathcal{B}$  is the set of bidirected edges. For each  $V_{j}\in \mathsf{P}\mathsf{a}_{i}$ , there exists a directed edge in  $\mathcal{E}$  such that  $V_{j}\to V_{i}$  which indicates functional dependency from  $V_{j}$  to  $V_{i}$ . Without showing unobserved exogenous variable  $\mathbf{U}$  explicitly, the confounding effects of  $\mathbf{U}$  are represented using bidirected edges in  $\mathcal{B}$ , which connect multiple pairs of observable variables in  $\mathbf{V}$ . The presence of  $V_{i}\leftrightarrow V_{j}$  in  $\mathcal{G}$  represents unmeasured factors (or confounders) that may influence both  $V_{i}$  and  $V_{j}$ . In other words,  $f_{i}$  and  $f_{j}$  share common exogenous variables  $\mathbf{U}_i\cap \mathbf{U}_j\neq \emptyset$  as input. Let  $|\Omega (\mathbf{V})|$  be the cardinality of  $\Omega (\mathbf{V})$ .
Assumption 1. We assume each  $V \in \mathbf{V}$  can only take a finite number of values, i.e.,  $|\Omega(\mathbf{V})|$  is finite.
Within the causal graph, an intervention on a subset of random variables  $\mathbf{S} \subseteq \mathbf{V}$  denoted by the do-operator  $do(\mathbf{S} = \mathbf{s})$  sets the structural equation for each  $S_{j} \in \mathbf{S}$  to be  $S_{j} = s_{j}$ . We also refer to  $\mathbf{s}$  as intervention for brevity. If a node  $V_{i} \notin \mathbf{S}$ , its structural equation remains to be  $V_{i} = f_{i}(\mathbf{U}_{i}, \mathsf{P}\mathsf{a}_{i})$ . The empty intervention denoted by  $do(\emptyset)$  does not change the structural equation of any random variable, and the distribution of  $\mathbf{V}$  is also called observational distribution. Let  $\mathbb{1}$  denote the indicator function. An SCM induces interventional distributions: if  $\mathbf{v}$  is consistent with intervention  $\mathbf{s}$ ,
$$
P (\mathbf {v} \mid d o (\mathbf {S} = \mathbf {s})) = \sum_ {\mathbf {u}} P (\mathbf {u}) \prod_ {V _ {i} \notin \mathbf {S}} \mathbb {1} \left\{v _ {i} = f _ {i} \left(\mathrm {p a} _ {i}, \mathbf {u} _ {i}\right) \right\} := P _ {\mathbf {s}} (\mathbf {v}). \tag {1}
$$
Otherwise,  $P(\mathbf{v} \mid d o(\mathbf{S} = \mathbf{s})) = 0$ . For a more detailed discussion on SCMs, we refer to [29].
2.2 Causal Bandit Problem with Confounders
The causal bandit problem with confounders studies the sequential decision-making problem with causal information provided by a causal graph  $\mathcal{G} = \langle \mathbf{V},\mathcal{E},\mathcal{B}\rangle$  with confounding effects represented by bidirected edge set  $\mathcal{B}$ . The causal graph  $\mathcal{G}$  is given but the exact interventional and observational distributions are unknown. Without loss of generality, we assume the reward is collected from node  $V_{n}$ , which is bounded in  $[0,1]$  and can not be intervened on. Let  $\mathcal{I}\subseteq \{\mathbf{s}\in \Omega (\mathbf{S})\mid \mathbf{S}\subseteq \mathbf{V}\backslash V_n\}$  be the space of allowed interventions. The decision-maker is required to decide which intervention to apply
at each time step. The expected reward of intervention  $\mathbf{s} \in \mathcal{I}$  is  $\mu_{\mathbf{s}} = \mathbb{E}_{P_{\mathbf{s}}}[V_n] = \sum_{\mathbf{v} \in \Omega(\mathbf{V})} v_n P_{\mathbf{s}}(\mathbf{v})$  where  $P_{\mathbf{s}}$  is defined in (1) and  $\mathbb{E}_P$  means the expectation is computed with probability distribution  $P$ .
At each time  $t \in \{1, \dots, T\}$ , the agent follows an adaptive allocation policy  $\pi$  to select an intervention  $\mathbf{s}_t \in \mathcal{I}$  and observe the causal bandit feedback  $\mathbf{V}_t \sim P_{\mathbf{s}_t}(\mathbf{V})$ , which is a realization of all observable variables after intervention. In fact, policy  $\pi$  is a sequence  $\{\pi_t\}_{t \in \mathbb{N}}$ , where each  $\pi_t$  determines the probability distribution of taking intervention  $\mathbf{S}_t \in \mathcal{I}$  given intervention and observation history  $\pi_t(\mathbf{S}_t | \mathbf{s}_1, \mathbf{v}_1, \dots, \mathbf{s}_{t-1}, \mathbf{v}_{t-1})$ . Let  $\mu_* := \max_{\mathbf{s} \in \mathcal{I}} \mu_{\mathbf{s}}$  and  $\Delta_{\mathbf{s}} = \mu_* - \mu_{\mathbf{s}}$  denote the optimal mean reward and the expected optimality gap of intervention  $do(\mathbf{S} = \mathbf{s})$  respectively. Given a causal graph  $\mathcal{G}$ , the objective of the causal bandit problem is to design a policy  $\pi$  to maximize the expected cumulative reward, or equivalently, to minimize the expected regret:
$$
R _ {T} ^ {\pi} := \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(\mu_ {*} - V _ {n, t}\right) \right] = \sum_ {\mathbf {s} \in \mathcal {I}} \mathbb {E} \left[ N _ {T} (\mathbf {s}) \right] \Delta_ {\mathbf {s}}, \tag {2}
$$
where  $N_{T}(\mathbf{s})$  is the total number of times intervention  $do(\mathbf{S} = \mathbf{s})$  is taken by policy  $\pi$  until time horizon till  $T$ , and the expectation is computed over different realizations of  $\{\mathbf{S}_t, \mathbf{V}_t\}_{t=1}^T$  from the interactions between the random policy  $\pi$  and the causal bandit model  $\langle P_{\mathbf{s}} \rangle_{\mathbf{s} \in \mathcal{I}}$ . Thus,  $R_{T}^{\pi}$  is the gap between the expected cumulative rewards of selecting the best intervention all the time and that by the policy  $\pi$ . The regret decomposition in (2) is from [18], and it expresses the expected regret with products of the expected number of suboptimal action selections and expected reward gaps.
The large action space in causal bandits poses many challenges to the learning problem. It is desirable to reduce the action space without excluding all optimal interventions. The do-calculus introduced in [28] provides a set of guidelines for evaluating invariances across interventions. In this context, our specific focus is on Rule 3, which provides the conditions under which a series of interventions have no impact on the outcome variable. For example, Rule 3 implies that  $P_{\mathbf{W},\mathbf{Z}}(y) = P_{\mathbf{Z}}(y)$  if we have  $(\mathbf{Y} \perp \mathbf{W}|\mathbf{Z})_{\mathcal{G}_{\overline{\mathbf{W}} \cup \mathbf{Z}}}$ , where  $\mathcal{G}_{\overline{\mathbf{W}} \cup \mathbf{Z}}$  is the subgraph of DAG  $\mathcal{G}$  with incoming edges to the set  $\mathbf{W} \cup \mathbf{Z}$  removed. This concept leads to the notion of a minimal intervention set (MIS), which is a subset of variables  $\mathbf{S} \subseteq \mathbf{V} \setminus \mathbf{V}_{\mathbf{n}}$  such that there is no  $\mathbf{S}' \subset \mathbf{S}$  for which  $\mu_{\mathbf{s}[\mathbf{S}']} = \mu_{\mathbf{s}}$  holds for every Structural Causal Model (SCM) with a causal graph  $\mathcal{G}$  [22]. Here,  $\mathbf{s}[\mathbf{S}']$  denotes an intervention on  $\mathbf{S}' \cap \mathbf{S}$  with values consistent with  $\mathbf{s}$ . An MIS  $\mathbf{S}$  is considered a possibly optimal MIS (POMIS) if some intervention  $\mathbf{s} \in \Omega(\mathbf{S})$  can achieve the optimal mean reward in an SCM with a causal graph  $\mathcal{G}$ . We define  $\mathcal{I} = \{\mathbf{s} \in \Omega(\mathbf{S}) | \mathbf{S} \text{ is a POMIS}\}$ .
At an intuitive level, it may seem logical that the most effective course of action would be to intervene on the immediate causes (parents) of the reward variable  $V_{n}$ . This approach would provide a higher level of control over  $V_{n}$  within the system. If the reward variable  $V_{n}$  is not confounded with any of its ancestors, its parent set  $\mathsf{P}\mathsf{a}_n$  is the only POMIS. In more general causal graphs where  $V_{n}$  is confounded with any of its ancestors, the paper [22] proves that multiple POMISs exist, which can include variables that are not parents of  $V_{n}$ . The paper also provides graphical criteria and an efficient algorithm for constructing a set of all POMISs for a given causal graph. For an effective method to search for  $\mathcal{I}$ , we refer to [22] and rely on their graphical criteria to construct a set of POMISs. This set of POMISs is used to form the set of possible optimal actions  $\mathcal{I}$  for use in the causal bandit algorithm. We make the following assumption, which we expect to hold in a general scenario [38], except for certain specifically designed SCMs.
Assumption 2. Within interventions on POMISs, there exists a unique optimal intervention  $\mathbf{s}_{*}$ .
3 Response Variables and Space of Interventional Distribution Tuples
The unspecified domain of  $\mathbf{U}$  makes it inconvenient to be directly applied to the learning problem. However, it is noted in [29, Ch. 8] that if each variable in  $\mathbf{V}$  takes finite states,  $\Omega(\mathbf{U})$  can be partitioned and  $\mathbf{U}$  can be projected to a collection of finite-state response variables. The resulting model is equivalent to the original one with respect to all observational and interventional distributions. Such a technique was used to bound the causal and counterfactual effects with observational distribution [29, Ch. 8]. We take it to define a parameterized space of interventional distribution tuples.
(a) Causal graph 1
(b) Structural equivalent of (a)
(c) Bow graph
Figure 1: Causal graphs and their structural equivalents
3.1 Response Variables and Canonical Structural Causal Model
In Definition 1, each structural equation  $V_{i} = f_{i}(\mathbf{U}_{i},\mathsf{P}\mathsf{a}_{i})$  connects random variable  $V_{i}$  to its parents  $\mathsf{P}\mathsf{a}_i$ . As  $\mathbf{U}_i$  varies along its domain, regardless of how complex the variation is, the only effect it can have is to switch the relationship between  $\mathsf{P}\mathsf{a}_i$  and  $V_{i}$ . Since there are at most  $|\Omega (V_i)|^{|\Omega (\mathsf{P}\mathsf{a}_i)|}$  mapping relationships from  $\mathsf{P}\mathsf{a}_i$  to  $V_{i}$ , we can decompose  $f_{i}$  as follows,
$$
V _ {i} = f _ {i} \left(\mathbf {U} _ {i}, \mathrm {P a} _ {i}\right) = \underline {{f}} _ {i} \left(M _ {i}, \mathrm {P a} _ {i}\right), M _ {i} = \bar {f} _ {i} \left(\mathbf {U} _ {i}\right), \tag {3}
$$
where  $M_{i} \in \{0, \dots, |\Omega(V_{i})|^{|\Omega(\mathsf{Pa}_{i})|} - 1\}$  is a random variable corresponding to a mapping from  $\Omega(\mathsf{Pa}_{i})$  to  $\Omega(V_{i})$ . For an observable variable without parents, we simply have  $M_{i} \in \{0, \dots, |\Omega(V_{i})| - 1\}$ . Such a random variable  $M_{i} \in \mathbf{M}$  attached to each  $V_{i} \in \mathbf{V}$  is called response variable in [5]. Treating response variables as exogenous variables, the resulting SCM  $\langle \mathbf{M}, \mathbf{V}, \underline{\mathbb{F}}, P(\mathbf{M}) \rangle$  is called canonical SCM in [40], where  $\underline{\mathbb{F}} = \{\underline{f}_1, \underline{f}_2, \dots, \underline{f}_n\}$ .
The decomposition of structural equations in equation (3) can be represented graphically by including response variables in  $\mathbf{M}$ . Since  $V_{i} = \underline{f}_{i}(M_{i},\mathsf{P}\mathsf{a}_{i})$ , for each  $V_{i}$  there is a directed edge  $M_{i}\to V_{i}$ . We also know that if there exists  $V_{i}\leftrightarrow V_{j}$  in  $\mathcal{G}$ , both  $V_{i}$  and  $V_{j}$  are affected by common exogenous variables  $\mathbf{U}_i\cap \mathbf{U}_j\neq \emptyset$ . As a result,  $M_{i}$  and  $M_j$  are correlated and they are also connected by a bidirected edge. For example, the causal graph in Fig. 1(a) has a structural equivalent in Fig. 1(b).
Given intervention  $\mathbf{s} \in \mathcal{I}$ , there exists a deterministic relationship from  $\mathbf{M}$  to  $\mathbf{V}$ . To further explain the idea, consider the bow graph example in Fig. 1(c), where  $V_{1}, V_{2}$  are binary variables taking value 0 or 1. As illustrated above, the response variables are  $M_{1} \in \{0,1\}$  and  $M_{2} \in \{0,1,2,3\}$ , and the structural equations for  $V_{1}$  and  $V_{2}$  are as follows:
$$
V _ {1} = \underline {{f}} _ {1} (M _ {1}) = M _ {1},   V _ {2} = \underline {{f}} _ {2} (M _ {2}, V _ {1}) = \left\{ \begin{array}{l l} 0 & M _ {2} = 0 \\ V _ {1} & M _ {2} = 1 \\ 1 - V _ {1} & M _ {2} = 2 \\ 1 & M _ {2} = 3 \end{array} \right..
$$
If  $do(\emptyset)$  is applied,  $(V_1, V_2) = (0, 1)$  when  $(M_1, M_2) = (0, 2)$  or  $(0, 3)$ . If  $do(V_1 = 1)$  is applied,  $(V_1, V_2) = (1, 0)$  when  $M_1 = 0$  or 1 and  $M_2 = 0$  or 2. It shows that for a given intervention  $do(\emptyset)$  or  $do(V_1 = 1)$ , there exists a deterministic mapping from  $\mathbf{M}$  to  $\mathbf{V}$ . Besides, the true value of  $\mathbf{M}$  can only be inferred from  $\mathbf{V}$  since  $\mathbf{M}$  is unobservable. Since  $M_1$  and  $M_2$  are correlated, we define their joint distribution as  $P(M_1 = i, M_2 = j) = p_{ij}$ . We can further express the observational and interventional probabilities of  $\mathbf{V}$  with the probability of  $\mathbf{M}$  as follows,
$$
P (V _ {1} = 0, V _ {2} = 0) = p _ {0 0} + p _ {0 1}, \quad P (V _ {1} = 0, V _ {2} = 1) = p _ {0 2} + p _ {0 3},
$$
$$
P \left(V _ {1} = 1, V _ {2} = 0\right) = p _ {1 0} + p _ {1 2}, \quad P \left(V _ {1} = 1, V _ {2} = 1\right) = p _ {1 1} + p _ {1 3}, \tag {4}
$$
$$
P _ {d o (V _ {1} = 0)} (V _ {2} = 0) = p _ {0 0} + p _ {0 1} + p _ {1 0} + p _ {1 1}, P _ {d o (V _ {1} = 0)} (V _ {2} = 1) = p _ {0 2} + p _ {0 3} + p _ {1 2} + p _ {1 3},
$$
$$
P _ {d o (V _ {1} = 1)} (V _ {2} = 0) = p _ {0 0} + p _ {0 2} + p _ {1 0} + p _ {1 2}, P _ {d o (V _ {1} = 1)} (V _ {2} = 1) = p _ {0 1} + p _ {0 3} + p _ {1 1} + p _ {1 3}.
$$
It can be seen these parameters are sufficient for specifying the model of Fig. 1(c). Such a parameterization technique can be extended to general causal graphs with latent confounders [40].
3.2 Space of Interventional Distribution Tuples