SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In a causal graph, a confounded component (c-component) is a maximal set of vertices connected with bidirected edges [34]. Note that a singleton node without bidirected edges is also a c-component. For example, there are two c-components in Fig. 1(a):  $\{V_1, V_3, V_4\}$  and  $\{V_2\}$ . In fact for any causal graph  $\mathcal{G}$ , its observable variables  $\mathbf{V}$  can be uniquely partitioned into c-components  $\{\mathbf{V}^1, \ldots, \mathbf{V}^{n_{\mathrm{c}}(\mathcal{G})}\}$ , where  $n_{\mathrm{c}}(\mathcal{G})$  is the total number of c-components in  $\mathcal{G}$ . Then  $\mathbf{M}$  can also be partitioned into  $\{\mathbf{M}^1, \ldots, \mathbf{M}^{n_{\mathrm{c}}(\mathcal{G})}\}$ , where each  $\mathbf{M}^j$  contains response variables adjacent to  $\mathbf{V}^j$ . For example in Fig. 1(b),  $\mathbf{M}$  is partitioned into  $\{M_1, M_3, M_4\}$  and  $\{M_2\}$ . Within each  $\mathbf{M}^j$ , the response variables are correlated since they are connected by bidirected edges. Besides,  $\mathbf{M}^1, \ldots, \mathbf{M}^{n_{\mathrm{c}}(\mathcal{G})}$  are mutually independent since  $\mathbf{M}^i$  and  $\mathbf{M}^j$  are not connected with bidirected edges for any  $i \neq j$ . As a result,  $P(\mathbf{m}) = \prod_{j=1}^{n_{\mathrm{c}}(\mathcal{G})} P(\mathbf{m}^j)$ . By concatenating  $P(\mathbf{m}^j)$  for each  $\mathbf{m}^j \in \Omega(\mathbf{M}^j)$ , we construct a vector  $\mathbf{p}_j \in \Delta(|\Omega(\mathbf{M}^j)|)$ , where  $\Delta(|\Omega(\mathbf{M}^j)|) := \left\{\mathbf{p}_j' \in \mathbb{R}_{\geq 0}^{|\Omega(\mathbf{M}^j)|} \mid \mathbf{1}^\top \mathbf{p}_j' = 1\right\}$  and  $\mathbf{1}^\top$  is the transpose of the all-ones vector.
Let the parent set of a c-component  $\mathbf{V}^j$  be  $\mathsf{P}\mathsf{a}_{\mathbf{V}^j} := (\cup_{i: V_i \in \mathbf{V}^j} \mathsf{P}\mathsf{a}_i) \setminus \mathbf{V}^j$ . When taking intervention  $do(\mathbf{S} = \mathbf{s})$ , the values of  $\mathbf{V}^j \cap \mathbf{S}$  is set to  $\mathbf{s}[\mathbf{V}^j]$ , which denotes the values of  $\mathbf{V}^j \cap \mathbf{S}$  that is consistent with  $\mathbf{s}$ . Notice that  $\mathbf{M}^j$  picks the mapping functions from  $\mathsf{P}\mathsf{a}_i$  to  $V_i$  for all  $V_i \in \mathbf{V}^j$ . Seeing values  $\mathbf{v}^j, \mathsf{p}\mathsf{a}_{\mathbf{V}^j}$  and  $\mathbf{s}[\mathbf{V}^j]$ , there exists a set of configurations of  $\mathbf{M}^j$ , denoted by  $B_{\mathcal{G},\mathbf{s}[\mathbf{V}^j]}\left(\mathbf{v}^j, \mathsf{p}\mathsf{a}_{\mathbf{V}^j}\right) \subseteq \Omega(\mathbf{M}^j)$ , that can make this happen. By marking configurations in  $B_{\mathcal{G},\mathbf{s}[\mathbf{V}^j]}\left(\mathbf{v}^j, \mathsf{p}\mathsf{a}_{\mathbf{V}^j}\right)$  with 1 and others to be 0, we construct a vector  $b_{\mathcal{G},\mathbf{s}[\mathbf{V}^j]}\left(\mathbf{v}^j, \mathsf{p}\mathsf{a}_{\mathbf{V}^j}\right) \in \{0,1\}^{\left|\Omega(\mathbf{M}^j)\right|}$ . With  $\mathbf{M}^1, \ldots, \mathbf{M}^{n_c(\mathcal{G})}$  being mutually independent, the interventional distribution can be factorized as
$$
P _ {\mathbf {s}} (\mathbf {v}) = \prod_ {j = 1} ^ {n _ {\mathrm {c}} (\mathcal {G})} P \left(\mathbf {M} ^ {j} \in B _ {\mathcal {G}, \mathbf {s} [ \mathbf {V} ^ {j} ]} \left(\mathbf {v} ^ {j}, \mathsf {p a} _ {\mathbf {V} ^ {j}}\right)\right) = \prod_ {j = 1} ^ {n _ {\mathrm {c}} (\mathcal {G})} b _ {\mathcal {G}, \mathbf {s} [ \mathbf {V} ^ {j} ]} ^ {\top} \left(\mathbf {v} ^ {j}, \mathsf {p a} _ {\mathbf {V} ^ {j}}\right) \mathbf {p} _ {j}. \tag {5}
$$
In the bow graph example in Fig. 1(c), there is one c-component, and vector  $b_{\mathcal{G}, \mathbf{s}[\mathbf{V}^j]}^\top(\mathbf{v}^j, \mathsf{pa}_{\mathbf{V}^j})$  can be constructed by referring to (4), where  $\mathbf{p}_1$  is a concatenation of  $p_{i,j}$ . The result in (5) generalizes the parameterization for the bow graph to a general causal graph. Based on it, given a causal graph  $\mathcal{G}$ , we define the space of interventional distribution tuples as the following.
Definition 2 (Space of Interventional Distribution Tuples). Given a causal graph  $\mathcal{G} = \langle \mathbf{V},\mathcal{E},\mathcal{B}\rangle$  the space of interventional (and observational) distribution tuples  $\mathcal{P} = \langle P_{\mathbf{s}}\rangle_{\mathbf{s}\in \mathcal{I}}$  is
$$
\mathcal {M} _ {\mathcal {G}} := \left\{\mathcal {P} ^ {\prime} \Big | \forall \mathbf {s} \in \mathcal {I}, \mathbf {v} \in \Omega (\mathbf {V}): P _ {\mathbf {s}} ^ {\prime} (\mathbf {v}) = \prod_ {j = 1} ^ {n _ {\mathrm {c}} (\mathcal {G})} b _ {\mathcal {G}, \mathbf {s} [ \mathbf {V} ^ {j} ]} ^ {\top} (\mathbf {v} ^ {j}, \mathsf {p a} _ {\mathbf {V} ^ {j}}) \mathbf {p} _ {j}, \mathbf {p} _ {j} \in \Delta (| \Omega (\mathbf {M} ^ {j}) |) \right\}.
$$
Space  $\mathcal{M}_{\mathcal{G}}$  contains all interventional distribution tuples associated with canonical SCMs  $\langle \mathbf{M}, \mathbf{V}, \underline{\mathbb{F}}, P(\mathbf{M}) \rangle$  with arbitrary  $P(\mathbf{M})$ . Since the space of  $\{\mathbf{p}_1, \dots, \mathbf{p}_{n_c(\mathcal{G})}\}$  is compact and a continuous image of a compact space is compact,  $\mathcal{M}_{\mathcal{G}}$  is also a compact space.
4 An Asymptotic Regret Lower Bound
In this section, we present an asymptotic information-theoretic regret lower bound for the causal bandit problem with confounders. It quantifies the optimal asymptotic performance of a uniformly good causal bandit policy defined below.
Definition 3 (Uniformly Good Policy). Given a causal graph  $\mathcal{G} = \langle \mathbf{V},\mathcal{E},\mathcal{B}\rangle$ , a causal bandit policy  $\pi$  is uniformly good if for any  $\alpha >0$ , the expected regret of  $\pi$  for any interventional distribution tuple setup  $\mathcal{P}\in \mathcal{M}_{\mathcal{G}}$ , denoted by  $R_T^\pi (\mathcal{P})$ , satisfies
$$
\lim  _ {T \to \infty} R _ {T} ^ {\pi} (\mathcal {P}) / T ^ {\alpha} = 0.
$$
A policy  $\pi$  being uniformly good indicates that it can achieve subpolynomial regret for any  $\mathcal{P} \in \mathcal{M}_{\mathcal{G}}$ . The information-theoretic regret lower bound sets the limit for the asymptotic regret of all such policies. Let  $D(P \parallel Q)$  denote the Kullback-Leibler (KL) divergence of two probability distributions  $P$  and  $Q$  and let  $\mathcal{I}_{-\mathbf{s}} = \mathcal{I} \setminus \{\mathbf{s}\}$ .
Theorem 1. Given a causal graph  $\mathcal{G} = \langle \mathbf{V},\mathcal{E},\mathcal{B}\rangle$ , let  $\mathcal{P} = \langle P_{\mathbf{s}}\rangle_{\mathbf{s}\in \mathcal{I}}\in \mathcal{M}_{\mathcal{G}}$  be the true interventional distribution tuple with a unique optimal intervention  $\mathbf{s}_*\in \mathcal{I}$  with mean reward  $\mu_{*}$ . The expected regret for any uniformly good causal bandit policy  $\pi$  satisfies
$$
\liminf_{T\to \infty}R_{T}^{\pi}\bigl(\mathcal{P}\bigr) / \ln T\geq C(\mathcal{P},\mathcal{G}),
$$
where  $C(\mathcal{P},\mathcal{G})$  is the value of the optimization problem given below,
$$
\begin{array}{l} \mathbb {O} (\mathcal {P}, \mathcal {G}): \underset {\eta_ {\mathbf {s}} \geq 0, \forall \mathbf {s} \in \mathcal {I} _ {- \mathbf {s} *}} {\text {m i n i m i z e}} \sum_ {\mathbf {s} \in \mathcal {I} _ {- \mathbf {s} *}} \eta_ {\mathbf {s}} \Delta_ {\mathbf {s}}, (6) \\ \text {s . t .} \max  _ {\mathbf {s} \in \mathcal {I} _ {- \mathbf {s} _ {*}}} \mu_ {\mathbf {s}} ^ {\prime} \leq \mu_ {*}, \forall \mathcal {P} ^ {\prime} \in \left\{\mathcal {P} ^ {\prime \prime} \in \mathcal {M} _ {\mathcal {G}} \left| \sum_ {\mathbf {s} \in \mathcal {I} _ {- \mathbf {s} _ {*}}} \eta_ {\mathbf {s}} D (P _ {\mathbf {s}} \| P _ {\mathbf {s}} ^ {\prime \prime}) <   1, P _ {\mathbf {s} _ {*}} ^ {\prime \prime} = P _ {\mathbf {s} _ {*}} \right. \right\}, (7) \\ \end{array}
$$
where  $\mu_{\mathbf{s}}^{\prime}$  is the expected reward of intervention s according to probability distribution  $P_{\mathbf{s}}^{\prime}$ .
Theorem 1 can be viewed as an extension of [10, Th. 1], and we defer its proof to appendices. The optimization problem  $\mathbb{O}(P,\mathcal{M}_{\mathcal{G}})$  is a semi-infinite program since there are infinite  $\mathcal{P}'$  in (7). To interpret the lower bound, (6) is a minimization of the regret, and its solution indicates an optimal allocation of  $O(\ln T)$  number of explorations, which is  $\eta_{\mathrm{s}}\ln T$  for each  $\mathbf{s}\in \mathcal{I}_{-\mathbf{s}_*}$  as  $T\to \infty$ . In (7),  $\sum_{\mathbf{s}\in \mathcal{I}}\eta_{\mathbf{s}}D(P_{\mathbf{s}}\parallel P_{\mathbf{s}}^{\prime \prime})$  can be viewed as the distance generated between true interventional distribution tuple  $\mathcal{P}$  and an alternative  $\mathcal{P}''\in \mathcal{M}_{\mathcal{G}}$  by an exploration allocation strategy  $\langle \eta_{\mathrm{s}}\rangle_{\mathrm{s}\in \mathcal{I}_{-\mathrm{s}_*}}$ . For any  $\mathcal{P}'$  in (7),  $\mu_{\mathbf{s}_*}' = \mu_*$  since  $P_{\mathbf{s}_*}' = P_{\mathbf{s}_*}$ , and as a result,  $\max_{\mathbf{s}\in \mathcal{I}_{-\mathbf{s}_*}}\mu_{\mathbf{s}}'\leq \mu_*$  indicates intervention  $\mathbf{s}_*$  is also optimal in  $\mathcal{P}'$ . So (7) sets a constraint for  $\langle \eta_{\mathrm{s}}\rangle_{\mathrm{s}\in \mathcal{I}_{-\mathrm{s}_*}}$ : the models allowed to have distance  $< 1$  from the true model must also take  $\mathbf{s}_*$  to be optimal. Or conversely,  $\langle \eta_{\mathrm{s}}\rangle_{\mathrm{s}\in \mathcal{I}_{-\mathrm{s}_*}}$  must generate distance  $\geq 1$  for any  $\mathcal{P}'\in \mathcal{M}_{\mathcal{G}}$  with  $P_{\mathbf{s}_*}' = P_{\mathbf{s}_*}$  not taking intervention  $\mathbf{s}_*$  to be optimal.
Remark 1. The  $O(\ln T)$  regret lower bound holds only if there exists  $\mathcal{P}' \in \mathcal{M}_{\mathcal{G}}$  such that  $P_{\mathbf{s}_*}' = P_{\mathbf{s}_*}$ , which is a condition meaning  $\mathcal{P}$  can not be distinguished from all other interventional distribution tuples in  $\mathcal{M}_{\mathcal{G}}$  by only taking intervention  $\mathbf{s}_*$ . This condition holds for almost all causal bandit problems. Otherwise, constraint (7) is relaxed, and the value of the optimization problem  $C(\mathcal{P}, \mathcal{M}_{\mathcal{G}}) = 0$ , which indicates sub-logarithmic regret can be achieved. Following the same line of proof in [17, Th. 3.9], it can be shown simple UCB can achieve this sub-logarithmic, in fact constant, expected regret.
5 SCM-based Causal Bandit Algorithm
The regret lower bound in Theorem 1 suggests a potentially optimal exploration strategy, i.e. to take each intervention  $\mathbf{s} \in \mathcal{I}_{-\mathbf{s}_*}$  up to  $\eta_{\mathrm{s}} \ln T$  times. However, it requires the ground truth  $\mathcal{P}$  to solve the optimization problem  $\mathbb{O}(\mathcal{P}, \mathcal{G})$ . The allocation matching principle [10] essentially replaces the true model with an estimated one to solve the optimization problem, and controls explorations to match with the solution. It is expected that the estimated model converges to the ground truth at a fast enough rate so that the actual explorations match with the optimal exploration strategy. We follow this idea to design a causal bandit policy, whose pseudo-code is shown in Algorithm 1. We call this SCM-based Approximate Allocation Matching (SCM-AAM).
Similar to [22], SCM-AAM only intervenes on POMISs for the purpose of reducing the action space. The algorithm is anytime without requiring the knowledge of horizon length. It takes two tuning parameters  $\epsilon = (0,1 / |\mathcal{I}|)$  and  $\gamma > 0$  as inputs that control the exploration rate, which affects finite-time performance. At each time  $t$ , let  $N_{t}(\mathbf{s})$  be the number of times the intervention  $do(\mathbf{S} = \mathbf{s})$  is taken so far, and let  $N_{t}(\mathbf{v},\mathbf{s})$  be the number of times we observe  $\mathbf{v}$  with intervention  $do(\mathbf{S} = \mathbf{s})$ . The algorithm maintains a set of empirical interventional distributions  $\langle \bar{P}_{\mathbf{s},t}\rangle_{\mathbf{s} \in \mathcal{I}}$  and empirical mean rewards  $\langle \bar{\mu}_{\mathbf{s},t}\rangle_{\mathbf{s} \in \mathcal{I}}$ , where  $\bar{P}_{\mathbf{s},t}(\mathbf{v}) = N_{t}(\mathbf{v},\mathbf{s}) / N_{t}(\mathbf{s})$  for each  $\mathbf{s} \in \mathcal{I}$ .
The main body of the algorithm is composed of two components: exploitation and exploration. At each round  $t$ , it first attempts to evaluate if enough information has been collected to determine the optimal action. Inspired by the lower bound, enough distance is generated by the sampling history between  $\langle \bar{P}_{\mathbf{s},t} \rangle_{\mathbf{s} \in \mathcal{I}}$  and  $\mathcal{P}' = \langle P_{\mathbf{s}}' \rangle_{\mathbf{s} \in \mathcal{I}}$  if  $\sum_{\mathbf{s} \in \mathcal{I}} N_t(\mathbf{s}) D(\bar{P}_{\mathbf{s},t} \| P_{\mathbf{s}}') > (1 + \gamma) \ln t$ . So we can construct a confidence set  $\mathcal{C}_t$  with high probability to contain the true model  $\mathcal{P}$ , which is defined as
$$
\mathcal {C} _ {t} := \left\{\mathcal {P} ^ {\prime} \in \mathcal {M} _ {\mathcal {G}} \mid \sum_ {\mathbf {s} \in \mathcal {I}} N _ {t} (\mathbf {s}) D \left(\bar {P} _ {\mathbf {s}, t} \| P _ {\mathbf {s}} ^ {\prime}\right) \leq (1 + \gamma) \ln t \right\}. \tag {8}
$$
Algorithm 1: SCM-based Approximate Allocation Matching (SCM-AAM)  
Input :Causal graph  $\mathcal{G} = \langle \mathbf{V},\mathcal{E},\mathcal{B}\rangle$ $\mathcal{I} = \{\mathbf{s}\in \Omega (\mathbf{S})\mid \mathbf{S}$  is POMIS  
Initialization:  $N_{t}^{\mathrm{e}} = 0$ $\forall \mathbf{s}\in \mathcal{I}:N_{t}(\mathbf{s}) = 0$   
Set :  $\epsilon \in (0,1 / |\mathcal{I}|)$  and  $\gamma >0$   
Output : Sequence of interventions  
Select each intervention  $\mathbf{s}\in \mathcal{I}$  once  
for  $t = |\mathcal{I}| + 1$  to  $T$  do  
    Compute empirical distributions and mean rewards  $\langle \bar{P}_{\mathbf{s},t}\rangle_{\mathbf{s}\in \mathcal{I}}$  and  $\langle \bar{\mu}_{\mathbf{s},t}\rangle_{\mathbf{s}\in \mathcal{I}}$   
With  $\mathcal{C}_t$  defined in (8), compute UCB  $\tilde{\mu}_{\mathbf{s}}(\mathcal{C}_t) = \sum_{\mathbf{v}\in \Omega (\mathbf{V})}v_n\max_{\mathcal{P}'\in \mathcal{C}_t}P_{\mathbf{s}}'(\mathbf{v}),\forall \mathbf{s}\in \mathcal{I}$   
if  $\exists s_{*,t}:\bar{\mu}_{s_{*,t}} > \tilde{\mu}_{s}(\mathcal{C}_{t}),\forall s\in \mathcal{I}_{-s_{*,t}}$  then  
    Select  $\mathbf{S}_t\gets \mathbf{s}_{*,t}$  and  $N_{t}(\mathbf{S}_{t})\leftarrow N_{t}(\mathbf{S}_{t}) + 1$  \% exploitation  
else if  $\min_{\mathbf{s}\in \mathcal{I}}N_t(\mathbf{s})\leq \epsilon N_t^{\mathrm{e}}$  then  
    Select  $\underline{\mathbf{S}}_t\gets$  arg  $\min_{\mathbf{s}\in \mathcal{I}}N_t(\mathbf{s})$  \% forced exploration  
    Set  $N_{t}^{\mathrm{e}}\gets N_{t}^{\mathrm{e}} + 1$  and  $N_{t}(\underline{\mathbf{S}}_{t})\gets N_{t}(\underline{\mathbf{S}}_{t}) + 1$   
else  
    Select  $\overline{\mathbf{S}}_t\gets$  arg max  $\mathbf{s}\in \mathcal{I}\tilde{\mu}_{\mathbf{s}}(\mathcal{C}_{t})$  \% exploration with greedy approximation  
    Set  $N_{t}^{\mathrm{e}}\gets N_{t}^{\mathrm{e}} + 1$  and  $N_{t}(\overline{\mathbf{S}}_{t})\gets N_{t}(\overline{\mathbf{S}}_{t}) + 1$
With  $V_{n}$  being the reward node, for each  $\mathbf{s} \in \mathcal{I}$ , the algorithm computes the UCB of the mean reward of taking intervention  $\mathbf{s}$  as  $\tilde{\mu}_{\mathbf{s}}(\mathcal{C}_t) = \sum_{\mathbf{v} \in \Omega(\mathbf{V})} v_n \max_{\mathcal{P}' \in \mathcal{C}_t} P_{\mathbf{s}}'(\mathbf{v})$ . If there exists an intervention  $\mathbf{s}_{*,t}$  with empirical mean reward  $\bar{\mu}_{\mathbf{s}_{*,t}} > \tilde{\mu}_{\mathbf{s}}(\mathcal{C}_t)$ ,  $\forall \mathbf{s} \in \mathcal{I}_{-\mathbf{s}_{*,t}}$ , we are confident about  $\mathbf{s}_{*,t}$  to be the optimal intervention. So the algorithm enters the exploitation phase and selects  $\mathbf{S}_t = \mathbf{s}_{*,t}$ .
If no such  $\mathbf{s}_{*,t}$  exists, the algorithm needs to explore. According to the allocation principle, one is inclined to solve  $\mathbb{O}(\langle \bar{P}_{\mathbf{s},t}\rangle_{\mathbf{s}\in \mathcal{I}},\mathcal{G})$  defined in Theorem 1 and make  $N_{t}(\mathbf{s})\approx \eta_{\mathbf{s}}\ln t$  for each  $\mathbf{s}\in \mathcal{I}$ . Nevertheless, there is no guarantee that this semi-infinite program can be solved efficiently. Therefore, we take a greedy approximation approach. Ideally, to get out of exploration, the algorithm needs to reduce  $\max_{\mathbf{s}\in \mathcal{I}_{-s_*}}\tilde{\mu}_{\mathbf{s}}(\mathcal{C}_t)$  to a value below  $\mu_{*}$ , thus it selects  $\overline{\mathbf{S}}_t = \arg \max_{\mathbf{s}\in \mathcal{I}}\tilde{\mu}_{\mathbf{s}}(\mathcal{C}_t)$ . The algorithm maintains a counter of exploration steps  $N_t^e$ . If  $\min_{\mathbf{s}\in \mathcal{I}}N_t(\mathbf{s})\leq \epsilon N_t^e$ , it generates forced exploration by selecting the least selected intervention  $\underline{\mathbf{S}}_t$ . Such a mechanism ensures every action is persistently taken in exploration phases so that  $\langle \bar{P}_{\mathbf{s},t}\rangle_{\mathbf{s}\in \mathcal{I}}$  converges to the ground truth eventually.
Remark 2. With factorization in (5),  $\mathcal{C}_t$  corresponds to setting constraint for  $\{\mathbf{p}_1, \ldots, \mathbf{p}_{n_c(\mathcal{G})}\}$  as
$$
\sum_ {\mathbf {s} \in \mathcal {I}} \sum_ {\mathbf {v} \in \Omega (\mathbf {V})} N _ {t} (\mathbf {s}, \mathbf {v}) \ln \frac {\bar {P} _ {\mathbf {s} , t} (\mathbf {v})}{\prod_ {j = 1} ^ {n _ {c} (\mathcal {G})} b _ {\mathcal {G} , \mathbf {s} [ \mathbf {V} ^ {j} ]} ^ {\top} \left(\mathbf {v} ^ {j} , \mathfrak {p a} _ {\mathbf {V} ^ {j}}\right) \mathbf {p} _ {j}} \leq (1 + \gamma) \ln t, \tag {9}
$$
where the expression on the left of inequality is a convex function of  $\mathbf{p}_1, \ldots, \mathbf{p}_{n_{\mathrm{c}}(\mathcal{G})}$ . Define convex set  $\mathcal{C}_t^{\mathfrak{p}} = \left\{\left\{\mathbf{p}_1, \ldots, \mathbf{p}_{n_{\mathrm{c}}(\mathcal{G})}\right\} \mid \mathbf{p}_j \in \Delta(|\Omega(\mathbf{M}^j)|), \forall j \in \{1, \ldots, n_{\mathrm{c}}(\mathcal{G})\}\right.$  and (9) is true. Then the UCB can be derived by maximizing a set of concave functions over  $\mathcal{C}_t^{\mathfrak{p}}$  as follows,
$$
\tilde {\mu} _ {\mathbf {s}} (\mathcal {C} _ {t}) = \sum_ {\mathbf {v} \in \Omega (\mathbf {V})} v _ {n} \max _ {\mathcal {P} ^ {\prime} \in \mathcal {C} _ {t}} P _ {\mathbf {s}} ^ {\prime} (\mathbf {v}) = \sum_ {\mathbf {v} \in \Omega (\mathbf {V})} v _ {n} \exp \Big (\max _ {\{\mathbf {p} _ {1}, \ldots , \mathbf {p} _ {n _ {c} (\mathcal {G})} \} \in \mathcal {C} _ {t} ^ {\mathfrak {p}}} \sum_ {j = 1} ^ {n _ {c} (\mathcal {G})} \ln \left(b _ {\mathcal {G}, \mathbf {s} [ \mathbf {V} ^ {j} ]} ^ {\top} (\mathbf {v} ^ {j}, \mathsf {p a} _ {\mathbf {V} ^ {j}}) \mathbf {p} _ {j}\right) \Big).
$$
We use  $\tilde{\mu}_{\mathbf{s}}(\mathcal{C}_t)$  instead of  $\tilde{\mu}_{\mathbf{s}}^{\prime}(\mathcal{C}_t) = \max_{\mathcal{P}^{\prime}\in \mathcal{C}_t}\sum_{\mathbf{v}\in \Omega (\mathbf{V})}v_nP_{\mathbf{s}}^{\prime}(\mathbf{v})$  as UCB since  $\sum_{\mathbf{v}\in \Omega (\mathbf{V})}v_nP_{\mathbf{s}}^{\prime}(\mathbf{v})$  in general is a non-concave function of  $\mathbf{p}_1,\dots ,\mathbf{p}_{n_c(\mathcal{G})}$ . Also, notice that  $\forall \mathbf{s}\in \mathcal{I}:\tilde{\mu}_{\mathbf{s}}(\mathcal{C}_t)\geq \tilde{\mu}_{\mathbf{s}}^{\prime}(\mathcal{C}_t)$ .
In a pure topology-based approach [22], each intervention on POMISs is treated independently during the algorithm execution. Whereas SCM-AAM can leverage the canonical SCM to incorporate the sampling results from different interventions together. After each new sample  $\mathbf{v}_t$  with intervention  $\mathbf{s}_t$  is observed, it is used to update the parametric space for interventional distribution tuples  $\mathcal{C}_t$ .
6 Finite Time Regret Analysis
We present a finite time problem-dependent regret upper bound for SCM-AAM in Theorem 2. The complete proof is provided in the appendices. We also give an interpretation of Theorem 2, and compare the regret upper bound with [22] in the setting where the reward node  $V_{n}$  is binary,
Theorem 2. For the causal bandit problem with unobserved confounders, suppose the causal graph  $\mathcal{G} = \langle \mathbf{V},\mathcal{E},\mathcal{B}\rangle$  is given and Assumptions 1 and 2 are true. For any interventional distribution tuple  $\mathcal{P}\in \mathcal{M}_{\mathcal{G}}$ , any  $\kappa >0$  and horizon of length  $T$ , the expected regret for SCM-AAM with parameters  $\epsilon \in (0,1 / |\mathcal{I}|)$  and  $\gamma >0$  satisfies
$$
R_{T}^{\mathrm{SCM - AAM}}(\mathcal{P})\leq (1 + \kappa)(1 + \gamma)\sum_{\mathbf{s}\in \mathcal{I}_{- \mathbf{s}_{*}}}\zeta_{\mathbf{s}}(\mathcal{P})\Bigl(\Delta_{\mathbf{s}} + \frac{\epsilon|\mathcal{I}|}{1 - \epsilon|\mathcal{I}|}\Bigr)\ln T + c,
$$
where  $c$  is a suitably large universal constant depending on  $\mathcal{P}$ ,  $\kappa$  and tuning parameters  $\epsilon$  and  $\gamma$ , and
$$
\forall \mathbf {s} \in \mathcal {I} _ {- \mathbf {s} _ {*}}: \zeta_ {\mathbf {s}} (\mathcal {P}) = \left\{ \begin{array}{l l} 0, & \tilde {\mu} _ {\mathbf {s}} (\mathcal {C} _ {\mathbf {s}} (0, \mathcal {P})) \leq \mu_ {*}, \\ \max  _ {\eta_ {\mathbf {s}} \geq 0} \eta_ {\mathbf {s}} \text {s . t .} \tilde {\mu} _ {\mathbf {s}} (\mathcal {C} _ {\mathbf {s}} (\eta_ {\mathbf {s}}, \mathcal {P})) \geq \mu_ {*}, & o t h e r w i s e, \end{array} \right.
$$
in which  $\mathcal{C}_{\mathbf{s}}(\eta_{\mathbf{s}},\mathcal{P})\coloneqq \bigl \{\mathcal{P}^{\prime}\in \mathcal{M}_{\mathcal{G}}\mid \eta_{\mathbf{s}}D(P_{\mathbf{s}}\parallel P_{\mathbf{s}}^{\prime}) + \epsilon \eta_{\mathbf{s}} / 2\sum_{\mathbf{x}\in \mathcal{I}_{-s}}D(P_{\mathbf{x}}\parallel P_{\mathbf{x}}^{\prime})\leq 1\bigr \} .$
Theorem 2 indicates the expected number of samples for each  $\mathbf{s} \in \mathcal{I}_{-\mathbf{s}_*}$  can be approximately bounded by  $\zeta_{\mathbf{s}}(\mathcal{P}) \ln T$ . It can be interpreted as follows: when  $\mathbf{s} \in \mathcal{I}_{-\mathbf{s}_*}$  is selected approximately up to  $\zeta_{\mathbf{s}}(\mathcal{P}) \ln T$  times, its UCB can be pushed below  $\mu_*$ . The presence of terms containing  $\epsilon$  is due to forced exploration. In [22], KL-UCB and Thompson sampling are employed to apply interventions on POMISs. If the reward node  $V_n \in \{0,1\}$ , they enjoys expected regret  $\sum_{\mathbf{s} \in \mathcal{I}_{-\mathbf{s}_*}} (\Delta_{\mathbf{s}} \ln T) / D(P_{\mathbf{s}}(V_n) \| P_{\mathbf{s}_*}(V_n)) + c'$  for some insignificant  $c'$  [14, 2]. In this situation, Theorem 2 shows that the SCM-AAM enjoys smaller expected regret when we disregard the small constants  $\kappa, \gamma$  and  $\epsilon$ . To see it, we express the leading constant for KL-UCB and Thompson as
$$
1 / D \left(P _ {\mathbf {s}} \left(V _ {n}\right) \| P _ {\mathbf {s} _ {*}} \left(V _ {n}\right)\right) = \max  _ {\eta_ {\mathbf {s}} \geq 0} \eta_ {\mathbf {s}} \text {s . t .} \tilde {\mu} _ {\mathbf {s}} \left(\mathcal {C} _ {\mathbf {s}} ^ {\prime} \left(\eta_ {\mathbf {s}}, P \left(V _ {n}\right)\right)\right) \geq \mu_ {*},
$$
where  $\mathcal{C}_{\mathbf{s}}^{\prime}(\eta_{\mathbf{s}},P(V_n)) = \{P_{\mathbf{s}}^{\prime}(V_n)\in [0,1]\mid \eta_{\mathbf{s}}D(P_{\mathbf{s}}(V_n)\parallel P_{\mathbf{s}}^{\prime}(V_n))\leq 1\}$ . Note that  $\mathcal{C}_{\mathbf{s}}^{\prime}(\eta_{\mathbf{s}},P(V_n))$  only sets constraint on  $P_{\mathbf{s}}^{\prime}(V_n)$ , while  $\mathcal{C}_{\mathbf{s}}(\eta_{\mathbf{s}},\mathcal{P})$  sets constraints on  $P_{\mathbf{s}}^{\prime}(\mathbf{V})$ . As a result,  $\mathcal{C}_{\mathbf{s}}(\eta_{\mathbf{s}},\mathcal{P})\subseteq \mathcal{C}_{\mathbf{s}}^{\prime}(\eta_{\mathbf{s}},P(V_n))$ , so that  $\zeta_{\mathbf{s}}(\mathcal{P})\leq 1 / D(P_{\mathbf{s}}(V_n)\parallel P_{\mathbf{s}_*}(V_n))$ .
7 Experimental Results