SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We compare the empirical performance of SCM-AAM with existing algorithms. The first baseline algorithm employs the simple UCB algorithm [4] on a reduced action set that includes interventions on POMISs. The other two baselines as introduced in [22], utilize the KL-UCB algorithm and Thomposn sampling (TS) to intervene on POMISs. We compare the performance of all the algorithms on three different causal bandit instances shown in Fig. 2. We choose the input parameters of the SCM-AAM algorithm to be  $\gamma = 0.1$  and  $\epsilon = 1 / |5\mathcal{I}|$  in the simulations. We set the horizon to 800 for all three tasks and repeat every simulation 100 times. Additionally, we include a confidence interval around the mean regret, with the width of the interval set to twice the standard deviation. The structural equations for the three causal bandit instances, as well as experimental details, can be found in the appendices. We plot the mean regret against time, where time in this context corresponds to the number of actions. All the nodes in Fig. 2 take binary values, either 0 or 1.
For task 1, we consider the causal graph shown in Fig. 2(a) with POMISs  $\{V_1\}$  and  $\{V_2\}$ . For task 2 and task 3, we consider more complex causal graphs, as displayed in Fig. 2(b) and (c). The POMISs for task 2 include  $\{V_1\}$ ,  $\{V_3\}$ , and  $\{V_3, V_4\}$ , while for task 3, they are  $\{V_1, V_4, V_6\}$  and  $\{V_5, V_6\}$ . The UCB algorithm has inferior performance compared to other baseline algorithms, especially in the more complicated tasks 2 and 3. The TS algorithm outperforms both UCB and KL-UCB in all three settings; however, it still incurs more regret than our proposed algorithm. The experiments demonstrate that our proposed SCM-AAM algorithm consistently outperforms other baseline algorithms by incurring lower empirical regret across all three tasks. Notably, the performance advantage of SCM-AAM is more significant for tasks 2 and 3, which have a more intricate causal graph structure with a higher number of nodes and edges. The results indicate that
(a) Task 1
(b) Task 2
(c) Task 3
Figure 2: Causal graphs used in the experiments
(b) Task 2
(c) Task 3
Figure 3: Regret versus time for the instances of structural causal bandits shown in Figure 2
incorporating causal structural information through the SCM-AAM algorithm can significantly reduce expected regret, especially for more complex causal graphs.
The simulations were conducted on a Windows desktop computer featuring a 12th generation Intel Core i7 processor operating at  $3.1\mathrm{GHz}$  and 32 GB of RAM. No GPUs were utilized in the simulations. The runtime for each iteration of the SCM-MAB, involving 800 arm plays, was approximately 2 minutes for both Task 1. However, for Task 2 and Task 3, which encompass larger underlying causal graphs, the runtime for every iteration increased to around 20 minutes. The algorithm code is provided at https://github.com/CausalML-Lab/SCM-AAM.
In this paper, we studied the causal bandit problem with latent confounders. With causal information provided by a causal graph, we present a problem-dependent information-theoretic regret lower bound. Leveraging the canonical SCM, we take an approximate allocation matching strategy to design the SCM-AAM algorithm. By analyzing SCM-AAM, we show it has a problem-dependent logarithmic regret upper bound. The analytic result is complemented with numerical illustrations featuring a variety of causal structures. It is shown that SCM-AAM exhibits promising performance in comparison with classic baseline algorithms. Since SCM has a natural application in counterfactual reasoning, extending the proposed algorithm and theoretical results to a counterfactual decision-making setup is an interesting future direction.
This research has been supported in part by NSF CAREER 2239375. Most of the work was completed when Lai Wei was a postdoctoral researcher at Purdue University.
[21] T. Lattimore and C. Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
[22] S. Lee and E. Bareinboim. Structural causal bandits: where to intervene? Advances in Neural Information Processing Systems, 31, 2018.
[23] S. Lee and E. Bareinboim. Structural causal bandits with non-manipulable variables. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4164-4172, 2019.
[24] Y. Lu, A. Meisami, A. Tewari, and W. Yan. Regret analysis of bandit problems with causal background knowledge. In Conference on Uncertainty in Artificial Intelligence, pages 141-150. PMLR, 2020.
[25] A. Maiti, V. Nair, and G. Sinha. A causal bandit approach to learning good atomic interventions in presence of unobserved confounders. In The 38th Conference on Uncertainty in Artificial Intelligence, 2022.
[26] J. S. McMullen. Entrepreneurial judgment as empathic accuracy: A sequential decision-making approach to entrepreneurial action. Journal of Institutional Economics, 11(3):651-681, 2015.
[27] P. A. Ortega and D. A. Braun. Generalized thompson sampling for sequential decision-making and causal inference. Complex Adaptive Systems Modeling, 2(1):1-23, 2014.
[28] J. Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669-688, 1995.
[29] J. Pearl. Causality. Cambridge university press, 2009.
[30] H. Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527-535, 1952.
[31] D. Russo and B. Van Roy. Learning to optimize via information-directed sampling. Operations Research, 66(1):230-252, 2018.
[32] V. Srivastava, P. Reverdy, and N. E. Leonard. Surveillance in an abruptly changing world via multiarmed bandits. In IEEE Conf. on Decision and Control, pages 692-697, Los Angeles, CA, Dec. 2014.
[33] W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285-294, 1933.
[34] J. Tian and J. Pearl. A general identification condition for causal effects. In Eighteenth national conference on Artificial intelligence, pages 567-573, 2002.
[35] B. P. Van Parys and N. Golrezaei. Optimal learning for structured bandits. arXiv preprint arXiv:2007.07302, 2020.
[36] C. H. Weiss, M. J. Bucuvalas, and M. Bucuvalas. Social science research and decision-making. Columbia University Press, 1980.
[37] A. Yabe, D. Hatano, H. Sumita, S. Ito, N. Kakimura, T. Fukunaga, and K.-i. Kawarabayashi. Causal bandits with propagating inference. In International Conference on Machine Learning, pages 5512-5520. PMLR, 2018.
[38] P. B. Yale. Geometry and symmetry. Courier Corporation, 2014.
[39] J. Zhang and E. Bareinboim. Transfer learning in multi-armed bandit: a causal approach. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, pages 1778–1780, 2017.
[40] J. Zhang, J. Tian, and E. Bareinboim. Partial counterfactual identification from observational and experimental data. In International Conference on Machine Learning, pages 26548-26558. PMLR, 2022.
A Proof of Asymptotic Regret Lower Bound in Theorem 1
The lower bound is derived following the same strategy in [21] by applying divergence decomposition and Bretagnolle-Huber inequality. For completeness, we reproduce both proofs in this section. Readers familiar with these results can skip them. Besides, we note that a more general result on controlled Markov chains is proven in [15].
Recall that a policy  $\pi$  is composed of a sequence  $\{\pi_t\}_{t\in \mathbb{N}_{>0}}$ , where at each time  $t\in \{1,\dots ,T\}$ ,  $\pi_{t}$  determines the probability distribution of taking intervention  $\mathbf{S}_t\in \mathcal{I}$  given intervention and observation history  $\pi_t(\mathbf{S}_t\mid \mathbf{s}_1,\mathbf{v}_1,\ldots ,\mathbf{s}_{t - 1},\mathbf{v}_{t - 1})$ . So the intervention and observation sequence  $\{\mathbf{S}_t,\mathbf{V}_t\}_{t\in \mathbb{N}_{>0}}$  is a production of the interactions between the interventional distribution tuple  $\langle P_{\mathbf{s}}\rangle_{\mathbf{s}\in \mathcal{I}}$  and policy  $\pi$ . Let  $T$  be the horizon length, we define a probability measure  $\mathbb{P}$  on the sequence of outcomes induced by  $\langle P_{\mathbf{s}}\rangle_{\mathbf{s}\in \mathcal{I}}$  and  $\pi$  such that
$$
\mathbb {P} \left(\mathbf {s} _ {1}, \mathbf {v} _ {1}, \dots , \mathbf {s} _ {T}, \mathbf {v} _ {T}\right) = \prod_ {t = 1} ^ {T} \pi_ {t} \left(\mathbf {s} _ {t} \mid \mathbf {s} _ {1}, \mathbf {v} _ {1}, \dots , \mathbf {s} _ {t - 1}, \mathbf {v} _ {t - 1}\right) P _ {\mathbf {s} _ {t}} \left(\mathbf {v} _ {t}\right). \tag {10}
$$
For a fixed policy  $\pi$ , let  $\mathbb{P}$  and  $\mathbb{P}'$  be the probability measures on the  $T$ -round plays of two different causal bandit instances with different interventional distribution tuples  $\mathcal{P} = \langle P_{\mathbf{s}} \rangle_{\mathbf{s} \in \mathcal{I}}$  and  $\mathcal{P}' = \langle P_{\mathbf{s}}' \rangle_{\mathbf{s} \in \mathcal{I}}$  in space  $\mathcal{M}_{\mathcal{G}}$ , which is defined in Definition 2. From (10), we get the distinction between  $\mathbb{P}$  and  $\mathbb{P}'$  is exclusively due to the separations of  $\mathcal{P}$  and  $\mathcal{P}'$ . As a result,  $D(\mathbb{P} \parallel \mathbb{P}')$  has the following decomposition, which is a standard result in MAB problems [21, Th. 15.1].
Lemma 3 (Divergence Decomposition). Given a causal graph  $\mathcal{G} = \langle \mathbf{V},\mathcal{E},\mathcal{B}\rangle$ , let  $\mathcal{P} = \langle P_{\mathbf{s}}\rangle_{\mathbf{s}\in \mathcal{I}}$  be the true interventional distribution tuple and let  $\mathcal{P}' = \langle P_{\mathbf{s}}'\rangle_{\mathbf{s}\in \mathcal{I}}$  be different from  $\mathcal{P}$ . For some fixed policy  $\pi$ , let  $\mathbb{P}$  and  $\mathbb{P}'$  be the probability measures on the  $T$  rounds of causal bandit plays in  $\mathcal{P}$  and  $\mathcal{P}'$  respectively. Then,
$$
D (\mathbb {P} \parallel \mathbb {P} ^ {\prime}) = \sum_ {\mathbf {s} \in \mathcal {I}} \mathbb {E} [ N _ {T} (\mathbf {s}) ] D (P _ {\mathbf {s}} \parallel P _ {\mathbf {s}} ^ {\prime}),
$$
where the expectation is computed with probability measure  $\mathbb{P}$ .
Proof. For a fixed policy  $\pi$ , from (10), we get
$$
\mathbb {P} (\mathbf {s} _ {1}, \mathbf {v} _ {1}, \dots , \mathbf {s} _ {T}, \mathbf {v} _ {T}) = \prod_ {t = 1} ^ {T} \pi_ {t} (\mathbf {s} _ {t} \mid \mathbf {s} _ {1}, \mathbf {v} _ {1}, \dots , \mathbf {s} _ {t - 1}, \mathbf {v} _ {t - 1}) P _ {\mathbf {s} _ {t}} (\mathbf {v} _ {t}).
$$
As a result,  $\pi_t$  is reduced and we get
$$
\log \frac {\mathbb {P} \left(\mathbf {s} _ {1} , \mathbf {v} _ {1} , \dots , \mathbf {s} _ {T} , \mathbf {v} _ {T}\right)}{\mathbb {P} ^ {\prime} \left(\mathbf {s} _ {1} , \mathbf {v} _ {t} , \dots , \mathbf {s} _ {T} , \mathbf {v} _ {T}\right)} = \sum_ {t = 1} ^ {T} \log \frac {P _ {\mathbf {s} _ {t}} (\mathbf {v} _ {t})}{P _ {\mathbf {s} _ {t}} ^ {\prime} (\mathbf {v} _ {t})}, \tag {11}
$$
which shows the distinction between  $\mathbb{P}$  and  $\mathbb{P}'$  is exclusively due to the separations of  $P_{\mathbf{s}}$  and  $P_{\mathbf{s}}'$  for each  $\mathbf{s} \in \mathcal{I}$ . Then we can decompose  $D(\mathbb{P} \parallel \mathbb{P}')$  with (11) as the following,
$$
\begin{array}{l} D (\mathbb {P} \parallel \mathbb {P} ^ {\prime}) = \mathbb {E} \left[ \log \frac {\mathbb {P} (\mathbf {S} _ {1} , \mathbf {V} _ {1} , \ldots , \mathbf {S} _ {T} , \mathbf {V} _ {T})}{\mathbb {P} ^ {\prime} (\mathbf {S} _ {1} , \mathbf {V} _ {t} , \ldots , \mathbf {S} _ {T} , \mathbf {V} _ {T})} \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \log \frac {P _ {\mathbf {S} _ {t}} (\mathbf {V} _ {t})}{P _ {\mathbf {S} _ {t}} ^ {\prime} (\mathbf {V} _ {t})} \right] = \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \log \frac {P _ {\mathbf {S} _ {t}} (\mathbf {V} _ {t})}{P _ {\mathbf {S} _ {t}} ^ {\prime} (\mathbf {V} _ {t})} \right] \\ = \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \mathbb {E} \left[ \log \frac {P _ {\mathbf {S} _ {t}} (\mathbf {V} _ {t})}{P _ {\mathbf {S} _ {t}} ^ {\prime} (\mathbf {V} _ {t})} \mid \mathbf {S} _ {t} \right] \right] = \sum_ {t = 1} ^ {T} \mathbb {E} \left[ D \left(P _ {\mathbf {S} _ {t}} \| P _ {\mathbf {S} _ {t}} ^ {\prime}\right) \right] \\ = \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \sum_ {\mathbf {s} \in \mathcal {I}} \mathbb {1} \left\{\mathbf {S} _ {t} = \mathbf {s} \right\} D \left(P _ {\mathbf {s}} \| P _ {\mathbf {s}} ^ {\prime}\right) \right] = \sum_ {\mathbf {s} \in \mathcal {I}} D \left(P _ {\mathbf {s}} \| P _ {\mathbf {s}} ^ {\prime}\right) \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {1} \left\{\mathbf {S} _ {t} = \mathbf {s} \right\} \right] \\ = \sum_ {a \in \mathcal {I}} \mathbb {E} [ N _ {T} (\mathbf {s}) ] D \left(P _ {\mathbf {s}} \parallel P _ {\mathbf {s}} ^ {\prime}\right), \\ \end{array}
$$
The other tool to prove the regret lower bound is the Bretagnolle-Huber Inequality.
Lemma 4 (Bretagnolle-Huber Inequality [21, Th. 14.2]). Let  $P$  and  $Q$  be two probability measures on a measurable space  $(\Omega, \mathcal{F})$ , and let  $E \in \mathcal{F}$  be an arbitrary event. Then
$$
P (E) + Q (E ^ {\mathsf {C}}) \geq \frac {1}{2} \exp \left(- D (P \| Q)\right),
$$
where  $E^{\complement} = \Omega \setminus E$  is complement of  $E$ .
Proof with finite  $\Omega$ . In Assumption 1, we assume each node in the causal graph can only take a finite number of values. So the sample space for the causal bandit problem with horizon length  $T$  is finite. We provide proof for the Bretagnolle-Huber inequality assuming  $\Omega$  is finite. First, we show
$$
\begin{array}{l} P (E) + Q \left(E ^ {\complement}\right) = \sum_ {\omega \in E} P (\omega) + \sum_ {\omega \in E ^ {\complement}} Q (\omega) \geq \sum_ {\omega \in E} \min  \left\{P (\omega), Q (\omega) \right\} + \sum_ {\omega \in E ^ {\complement}} \min  \left\{P (\omega), Q (\omega) \right\} \\ = \sum_ {\omega \in \Omega} \min  \left\{P (\omega), Q (\omega) \right\} = \frac {1}{2} \sum_ {\omega \in \Omega} \min  \left\{P (\omega), Q (\omega) \right\} \sum_ {\omega \in \Omega} [ P (\omega) + Q (\omega) ] \\ \geq \frac {1}{2} \sum_ {\omega \in \Omega} \min  \{P (\omega), Q (\omega) \} \sum_ {\omega \in \Omega} \max  \{P (\omega), Q (\omega) \}. \\ \end{array}
$$
Applying Cauchy-Schwarz inequality, we get
$$
\begin{array}{l} P (E) + Q \left(E ^ {\complement}\right) \\ \geq \frac {1}{2} \left(\sum_ {\omega \in \Omega} \sqrt {\min  \{P (\omega) , Q (\omega) \} \max  \{P (\omega) , Q (\omega) \}}\right) ^ {2} = \frac {1}{2} \left(\sum_ {\omega \in \Omega} \sqrt {P (\omega) Q (\omega)}\right) ^ {2} \\ = \frac {1}{2} \exp \left(2 \log \sum_ {\omega \in \Omega} \sqrt {P (\omega) Q (\omega)}\right) = \frac {1}{2} \exp \left(2 \log \sum_ {\omega \in \Omega} P (\omega) \sqrt {\frac {Q (\omega)}{P (\omega)}}\right) \\ \geq \frac {1}{2} \exp \left(2 \sum_ {\omega \in \Omega} P (\omega) \log \sqrt {\frac {Q (\omega)}{P (\omega)}}\right) = \frac {1}{2} \exp \left(- D (P \| Q)\right). \\ \end{array}
$$
We conclude our proof.
To prove the regret lower bound, the remaining work regards applying divergence decomposition and substituting  $D(\mathbb{P} \parallel \mathbb{P}')$  into the Bretagnolle-Huber inequality.
Lemma 5. Given a causal graph  $\mathcal{G} = \langle \mathbf{V},\mathcal{E},\mathcal{B}\rangle$ , let  $\mathcal{P} = \langle P_{\mathbf{s}}\rangle_{\mathbf{s}\in \mathcal{I}}\in \mathcal{M}_{\mathcal{G}}$  be the true interventional distribution tuple with a unique optimal intervention  $\mathbf{s}_*\in \mathcal{I}$  with mean reward  $\mu_{*}$ . The expected regret for any uniformly good causal bandit policy  $\pi$  satisfies
$$
\liminf_{T\to \infty}\frac{R_{T}^{\pi}(\mathcal{P})}{\ln T}\geq C(\mathcal{P},\mathcal{G}),
$$
where  $C(\mathcal{P},\mathcal{M}_{\mathcal{G}})$  is the value of the optimization problem given below
$$
\tilde {\mathbb {O}} \left(\mathcal {P}, \mathcal {M} _ {\mathcal {G}}\right): \underset {\eta_ {\mathrm {s}} \geq 0, \forall \mathbf {s} \in \mathcal {I} _ {- \mathrm {s} _ {*}}} {\text {m i n i m i z e}} \sum_ {\mathbf {s} \in \mathcal {I} _ {- \mathrm {s} _ {*}}} \eta_ {\mathrm {s}} \Delta_ {\mathbf {s}} \tag {12}
$$
$$
s. t. \sum_ {\mathbf {s} \in \mathcal {I}} \eta_ {\mathbf {s}} D \left(P _ {\mathbf {s}} \| \mathcal {P} _ {\mathbf {s}} ^ {\prime}\right) \geq 1, \forall P ^ {\prime} \in \left\{\mathcal {P} ^ {\prime \prime} \in \mathcal {M} _ {\mathcal {G}} \mid \mathbf {s} _ {*} \notin \mathcal {I} _ {*} ^ {\prime \prime} \right\}, \tag {13}
$$
where  $\mathcal{I}_*''$  is the set of optimal interventions according to  $\mathcal{P}''$ .
Remark 3. To interpret Lemma 5, (12) is a minimize of the regret, and its solution indicates an optimal exploration strategy. Each  $\mathbf{s} \in \mathcal{I}_{-\mathbf{s}_*}$  should be selected up to  $\eta_{\mathbf{s}} \ln T$  as  $T \to \infty$ . In (13),  $\sum_{\mathbf{s} \in \mathcal{I}} \eta_{\mathbf{s}} D(P_{\mathbf{s}} \| P_{\mathbf{s}}^{\prime \prime})$  is viewed as the distance generated between true interventional distribution tuple  $\mathcal{P}$  and an alternative  $\mathcal{P}^{\prime \prime} \in \mathcal{M}_{\mathcal{G}}$  by an exploration strategy  $\langle \eta_{\mathbf{s}} \rangle_{\mathbf{s} \in \mathcal{I}_{-\mathbf{s}_*}}$ . So (13) requires  $\langle \eta_{\mathbf{s}} \rangle_{\mathbf{s} \in \mathcal{I}_{-\mathbf{s}_*}}$  must generate distance  $\geq 1$  for any  $\mathcal{P}^{\prime} \in \mathcal{M}_{\mathcal{G}}$  in which intervention  $\mathbf{s}_*$  is not optimal.
Proof. For any  $\mathcal{P}' \in \{\mathcal{P}'' \in \mathcal{M}_{\mathcal{G}} \mid \mathbf{s}_* \notin \mathcal{I}_*''\}$ , the optimal intervention  $\mathbf{s}_*$  according to ground truth  $\mathcal{P}$  is not optimal in  $\mathcal{P}'$ . Recall  $\mathbb{P}'$  is the measures on  $T$ -round plays of causal bandit with interventional distribution tuple  $\mathcal{P}'$ . Also let  $\mu_*'$  be the optimal mean reward and  $\Delta_{\mathbf{s}}' = \mu_*' - \mu_{\mathbf{s}}'$ . For true causal model  $\mathcal{P}$ ,  $\mathbb{P}$  and  $\Delta_{\mathbf{s}}$  are defined similarly. For a fixed policy  $\pi$ , let  $R_T$  and  $R_T'$  be the expected regret from  $T$  round plays in  $\mathcal{P}$  and  $\mathcal{P}'$  respectively. Letting  $\epsilon = \min\{\Delta_{\mathbf{s}_*,'}', \Delta_{\mathbf{s}_*'}'\}$ , we get
$$
R _ {T} \geq \frac {\epsilon T}{2} \mathbb {P} \left(N _ {T} (\mathbf {s} _ {*}) <   \frac {T}{2}\right) \text {a n d} R _ {T} ^ {\prime} \geq \frac {\epsilon T}{2} \mathbb {P} ^ {\prime} \left(N _ {T} (\mathbf {s} _ {*}) \geq \frac {T}{2}\right).
$$
Combining both inequalities, we get
$$
\mathbb {P} \Big (N _ {T} (\mathbf {s} _ {*}) <   \frac {T}{2} \Big) + \mathbb {P} ^ {\prime} \Big (N _ {T} (\mathbf {s} _ {*}) \geq \frac {T}{2} \Big) \leq \frac {2}{\epsilon T} (R _ {T} + R _ {T} ^ {\prime}).
$$
Applying Bretagnolle-Huber inequality in Lemma 4 by substituting event  $\{N_T(\mathbf{s}_*) < \frac{T}{2}\}$  into  $E$ ,
$$
D \left(\mathbb {P} \| \mathbb {P} ^ {\prime}\right) \geq \ln \frac {\epsilon T}{4 \left(R _ {T} + R _ {T} ^ {\prime}\right)}. \tag {14}
$$
With Lemma 3, we substitute  $\sum_{\mathbf{s} \in \mathcal{I}} \mathbb{E}[N_T(\mathbf{s})] D(P_{\mathbf{s}} \| P_{\mathbf{s}}')$  into  $D(\mathbb{P} \| \mathbb{P}')$  and rearrange (14),
$$
\frac {\sum_ {\mathbf {s} \in \mathcal {I}} \mathbb {E} \left[ N _ {T} (\mathbf {s}) \right] D \left(P _ {\mathbf {s}} \| P _ {\mathbf {s}} ^ {\prime}\right)}{\ln T} \geq 1 - \frac {\ln 4 \left(R _ {T} + R _ {T} ^ {\prime}\right) / \epsilon}{\ln T}. \tag {15}
$$
According to Definition 3, that for any  $\alpha >0$ , the regret for a uniformly good policy  $\pi$  satisfies
$$
\lim  _ {T \rightarrow \infty} \frac {R _ {T}}{T ^ {\alpha}} = 0 \text {a n d} \lim  _ {T \rightarrow \infty} \frac {R _ {T} ^ {\prime}}{T ^ {\alpha}} = 0.
$$
It follows from (15) that
$$
\liminf_{T\to \infty}\frac{\sum_{\mathbf{s}\in\mathcal{I}}\mathbb{E}[N_{T}(\mathbf{s})]D\left(P_{\mathbf{s}}\parallel P_{\mathbf{s}}^{\prime}\right)}{\ln T}\geq 1,
$$
which sets the constraints for  $\mathbb{E}[N_T(\mathbf{s})] / \ln T$  as  $T\to \infty$ . Since  $R_{T} = \sum_{\mathbf{s}\in \mathcal{I}_{-s_{*}}}\mathbb{E}[N_{T}(\mathbf{s})]\Delta_{\mathbf{s}}$ , by replacing  $\mathbb{E}[N_T(\mathbf{s})] / \ln T$  with  $\eta_{\mathbf{s}}$ , we finish the proof.
Proof of Theorem 1. We finish the proof by showing the equivalence of  $\tilde{\mathbb{O}} (\mathcal{P},\mathcal{M}_{\mathcal{G}})$  in Lemma 5 and  $\mathbb{O}(\mathcal{P},\mathcal{M}_{\mathcal{G}})$  in Theorem 1. First notice that there is no  $\eta_{\mathbf{s}_*}$  in (12). Accordingly,  $\eta_{\mathbf{s}_*}$  can be assigned to an arbitrarily large value to make  $\sum_{\mathbf{s}\in \mathcal{I}}\eta_{\mathbf{s}}D(P_{\mathbf{s}}\parallel P_{\mathbf{s}}^{\prime})\geq 1$  if  $P_{\mathbf{s}_*}\neq P_{\mathbf{s}_*}^\prime$ . Therefore, (13) is equivalent to
$$
\sum_ {\mathbf {s} \in \mathcal {I} _ {- \mathbf {s} _ {*}}} \eta_ {\mathbf {s}} D (P _ {\mathbf {s}} \| P _ {\mathbf {s}} ^ {\prime}) \geq 1, \forall \mathcal {P} ^ {\prime} \in \left\{\mathcal {P} ^ {\prime \prime} \in \mathcal {M} _ {\mathcal {G}} \mid P _ {\mathbf {s} _ {*}} = P _ {\mathbf {s} _ {*}} ^ {\prime \prime}, \mathbf {s} _ {*} \notin \mathcal {I} _ {*} ^ {\prime \prime} \right\}.
$$
It means if  $\mathbf{s}_*$  is not optimal in  $\mathcal{P}'$ , then  $\sum_{\mathbf{s} \in \mathcal{I}_{-\mathbf{s}_*}} \eta_{\mathbf{s}} D(P_{\mathbf{s}} \| P_{\mathbf{s}}') \geq 1$ . Its contrapositive statement is that if  $\sum_{\mathbf{s} \in \mathcal{I}_{-\mathbf{s}_*}} \eta_{\mathbf{s}} D(P_{\mathbf{s}} \| P_{\mathbf{s}}') < 1$ , then  $\mathbf{s}_*$  is also optimal in  $\mathcal{P}'$ . Putting it into the mathematical expression, we get
$$
\max  _ {\mathbf {s} \in \mathcal {I} _ {- \mathbf {s} _ {*}}} \mu_ {\mathbf {s}} ^ {\prime} \leq \mu_ {*}, \forall \mathcal {P} ^ {\prime} \in \left\{\mathcal {P} ^ {\prime \prime} \in \mathcal {M} _ {\mathcal {G}} \middle | \sum_ {\mathbf {s} \in \mathcal {I} _ {- \mathbf {s} _ {*}}} \eta_ {\mathbf {s}} D (P _ {\mathbf {s}} \| P _ {\mathbf {s}} ^ {\prime \prime}) <   1, P _ {\mathbf {s} _ {*}} ^ {\prime \prime} = P _ {\mathbf {s} _ {*}} \right\},
$$
where we use the fact  $\mu_{*} = \mu_{*}^{\prime}$  since  $P_{\mathbf{s}_{*}} = P_{\mathbf{s}_{*}}^{\prime}$ . The proof is concluded.
B Finite time Regret Analysis of SCM-AAM in Theorem 2
B.1 Guarantee with Forced Exploration
The SCM-AAM algorithm maintains a counter of exploration steps  $N_{t}^{\mathrm{e}}$ . In expiration steps, if  $\min_{\mathbf{s} \in \mathcal{I}} N_{t}(\mathbf{s}) \leq \epsilon N_{t}^{\mathrm{e}}$ , the SCM-AAM algorithm generates forced exploration by selecting one of the least selected intervention. Lemma 6 shows that the forced exploration mechanism ensures every intervention is persistently selected in exploration phases.
Lemma 6. Let  $\mathcal{R} = \{t_i\}_{i\geq 1}$  be the sequence of rounds that SCM-AAM enters the exploration phase. For any time  $t_i\in \mathcal{R}$ , it satisfies that
$$
\min  _ {\mathbf {s} \in \mathcal {I}} N _ {t} (\mathbf {s}) \geq \frac {\epsilon i}{2}.
$$
Proof. We first note that the following two facts are true.
-  $\min_{\mathbf{s} \in \mathcal{I}} N_t(\mathbf{s})$  is non-decreasing over  $t$ .
- If  $\min_{\mathbf{s} \in \mathcal{I}} N_{t_i}(\mathbf{s}) \leq \epsilon N_{t_i}^e$ , then  $\min_{\mathbf{s} \in \mathcal{I}} N_{t_{i+|T|}}(\mathbf{s}) \geq \min N_{t_i}(\mathbf{s}) + 1$ .
Since  $N_{t}(\mathbf{s})$  is non-decreasing over  $t$ , the first statement is true. The second statement is true since otherwise,  $\underline{\mathbf{S}}_t$  is selected by forced exploration at least  $|\mathcal{I}|$  times without increasing  $\min_{\mathbf{s} \in \mathcal{I}} N_t(\mathbf{s})$ . With these two facts, we are ready to provide proof by contradiction. Suppose there exists  $i \in \{1, \dots, |\mathcal{R}|\}$  such that
$$
\min  _ {\mathbf {s} \in \mathcal {I}} N _ {t _ {i}} (\mathbf {s}) <   \frac {\epsilon i}{2}.
$$
According to the first fact, we have  $\forall j\geq i / 2$
$$
\min  _ {\mathbf {s} \in \mathcal {I}} N _ {t _ {j}} (\mathbf {s}) \leq \min  _ {\mathbf {s} \in \mathcal {I}} N _ {t _ {i}} (\mathbf {s}) \leq \epsilon j.
$$
According to SCM-AAM,  $\epsilon \in (0,1 / |\mathcal{I}|)$ . Then we apply the second fact,