SlowGuess/NeurIPS_2023_OCR huggingface dataset:
performs natural Bayesian updates based on both the data available in source environments and its own experimentation as interventional samples  $\pmb{v}_{x^{(1)}},\dots,\pmb{v}_{x^{(t - 1)}}$  become available, matching the intuition of most other Thompson sampling bandits in the literature. The full algorithm, called Thompson sampling with Transportability (tTS), is given in Alg. 1.
3 Regret guarantees conditional on prior data
We define information-theoretic regret bounds that aim to capture the exploration-exploitation trade-off for Alg. 1 when prior information allows it to infer parts of the environment before experimentation takes place.
Performance in MABs is, to a large extent, intimately related with the agent's uncertainty about which action is optimal, represented by a random variable  $\tilde{X}:\Omega_{\pmb{\xi}}\times \Omega_{\pmb{\theta}}\mapsto \Omega_X$  where  $\Omega_{\pmb{\xi}}\times \Omega_{\pmb{\theta}}$  defines the space of all models  $\mathcal{M}(\mathcal{G}^*)$  consistent with our knowledge of the deployment environment  $\pi^{*}$ . For example,  $P_{M\sim \mathcal{M}(\mathcal{G}^{*})}(\tilde{X} = x) = P_{M\sim \mathcal{M}(\mathcal{G}^{*})}\bigl (\mathbb{E}_{P_M}[Y_x] > \mathbb{E}_{P_M}[Y_{x'}],\forall x'\in \Omega_X\backslash \{x\} \bigr)$  where  $P_{M\sim \mathcal{M}(\mathcal{G}^{*})}$  is a probability mass function defined over  $\mathcal{M}(\mathcal{G}^*)$ . It is reasonable to assume that one would only choose actions with large regret when it can reduce the uncertainty in  $\tilde{X}$  substantially. Following [35, Sec. 5], we define a scalar  $\Gamma_t^8$  such that the per-round regret can be bounded by information gain,
$$
\mathbb {E} \left[ Y _ {\tilde {X}} - Y _ {X ^ {(t)}} \mid \bar {\boldsymbol {V}} _ {t}, \bar {\boldsymbol {v}} \right] \leqslant \Gamma_ {t} \sqrt {I _ {P (\cdot | \bar {\boldsymbol {V}} _ {t} , \bar {\boldsymbol {v}})} \left(\tilde {X} ; \boldsymbol {V} _ {X ^ {(t)}}\right)}, \tag {5}
$$
where  $\bar{V}_t\coloneqq \{\pmb{V}_{X^{(1)}},\dots ,\pmb{V}_{X^{(t - 1)}}\}$  denotes the agent's history of interactions with  $\pi^*$  up to round  $t$  and  $I_P(X,Y)\coloneqq \mathcal{D}_{KL}(P(X,Y)\| P(X)P(Y))$  denotes the filtered mutual information defined based on  $P$  (where  $\mathcal{D}_{KL}$  is the Kullback-Leibler divergence). Expectations, unless otherwise stated, are taken with respect to all random quantities. The following proposition, extended from [35, Prop. 1], shows that the Bayesian regret of an agent acting according to Alg. 1 is sub-linear with a dependency on the entropy of the optimal action  $\tilde{X}$ .
Proposition 1. Let  $R_T$  denote the regret incurred by following Thompson sampling (Alg. 1). For any  $T \in \mathbb{N}$  and  $\Gamma \geqslant \Gamma_t$ , then
$$
\mathbb {E} \left[ R _ {T} \mid \bar {\boldsymbol {v}} \right] \leqslant \Gamma \sqrt {\mathcal {H} (\tilde {X} \mid \bar {\boldsymbol {v}}) T}, \tag {6}
$$
where  $\mathcal{H}(\tilde{X}\mid \bar{\upsilon})$  is the conditional entropy of  $\tilde{X}$  given  $\bar{\upsilon}$ .
Proofs are given in Appendix C.
This bound is interesting because it cleanly relates the regret with the uncertainty about the optimal action conditioned on prior data. On one extreme, if data from source environments fully characterizes the optimal action, the entropy equals 0, and no further experimentation is required; on the other extreme, if data from source environments have no relationship with the target query, the entropy equals  $\log (|\Omega_X|)$ , and the bound reverts to conventional worst-case guarantees [35]. The entropy of the optimal action is often not sufficient to capture the information from  $\bar{v}$  as  $\tilde{X}$  may still have a uniform distribution even though posterior distributions over  $(\theta ,\xi)$  have tightened. In other words, there is additional structure among different reward distributions that is not captured by the entropy of the optimal action. Such a setting can be analyzed with a different assumption on the per round regret that explicitly considers model parameters  $(\theta ,\xi)$  to quantify information gain,
$$
\mathbb {E} \left[ Y _ {\bar {X}} - Y _ {X ^ {(t)}} \mid \bar {V} _ {t}, \bar {v} \right] \leqslant \Gamma_ {t} \sqrt {I _ {P (\cdot | \bar {V} _ {t} , \bar {v})} \left(\boldsymbol {\theta} , \boldsymbol {\xi} ; \boldsymbol {V} _ {X ^ {(t)}}\right)} + \epsilon_ {t}. \tag {7}
$$
where  $\epsilon_t > 0$  is an additional slack term. Accordingly, the following proposition provides an alternative bound using a conditional analogue of [25, Prop. 2].
Proposition 2. Let  $R_T$  denote the regret incurred following the policy defined by Alg. 1. For any  $T \in \mathbb{N}$ , if Eq. (7) holds with  $\Gamma \geqslant \Gamma_t$  for all  $t$ ,
$$
\mathbb {E} \left[ R _ {T} \mid \bar {\boldsymbol {v}} \right] \leqslant \Gamma \sqrt {T I _ {P (\cdot | \bar {\boldsymbol {v}})} \left(\boldsymbol {\theta} , \boldsymbol {\xi} ; \boldsymbol {V} _ {X ^ {(1)}}, \dots , \boldsymbol {V} _ {X ^ {(T)}}\right)} + \sum_ {t = 1} ^ {T} \mathbb {E} [ \epsilon_ {t} ]. \tag {8}
$$
$^8\Gamma_t$  is called the information ratio and quantifies the trade-off between incurring low regret and gaining information about the optimal action.  $\Gamma_{t}$  can always be upper-bounded by  $|\Omega_X| / 2$  [35, Sec. 5]. We provide a derivation of a bound for  $\Gamma_{t}$  for a specific set of environments as an example in Appendix C.3.
This proposition shows that if prior data allows the agent to concentrate  $(\theta, \xi)$  around some value, additional experimentation does not provide much more information and the regret should be small. In principle, it is possible to get precise per-round regret values by inferring values for  $\Gamma_t$  and  $\epsilon_t$  through the construction of concentration inequalities for the reward variable, as done by Lu et al. in [25, Lem. 3]. We adapt this result using conditional information-theoretic quantities in the following proposition.
Proposition 3. Fix  $\delta >0$  and choose  $\Gamma_t$  such that  $|Y_{x} - \mathbb{E}[Y_{x}\mid \bar{V}_{t},\bar{v} ]|\leqslant \frac{\Gamma_{t}}{2}\sqrt{I_{P(\cdot|\bar{V}_{t},\bar{v})}(\pmb{\theta},\pmb{\xi};Y_{x})}$  for all  $x\in \Omega_X$  simultaneously with probability greater than  $1 - \delta$ . Then Alg. 1 chooses actions  $X^{(t)}$  that satisfy
$$
\mathbb {E} \left[ Y _ {\tilde {X}} - Y _ {X ^ {(t)}} \mid \bar {\boldsymbol {V}} _ {t}, \bar {\boldsymbol {v}} \right] \leqslant \Gamma_ {t} \sqrt {I _ {P (\cdot | \bar {\boldsymbol {V}} _ {t} , \bar {\boldsymbol {v}})} \left(\boldsymbol {\theta} , \boldsymbol {\xi} ; \boldsymbol {V} _ {X ^ {(t)}}\right)} + \delta B, \tag {9}
$$
where  $B\geqslant 0$  is such that  $\sup_{y,y'\in \Omega_Y}y - y'\leqslant B$
4 Posterior approximations
This section describes a tractable algorithm to evaluate posterior distributions of the form  $P(\pmb{\xi}, \pmb{\theta} \mid \bar{\pmb{v}}, \pmb{v}_{x^{(1)}}, \dots, \pmb{v}_{x^{(t-1)}})$  and its posterior updates when new data is collected. Priors on  $\pmb{\xi}, \pmb{\theta}$  may be defined such as to induce tractable conditional distributions that may be used within a Gibbs sampling framework. The Gibbs sampler starts with some initial value for all latent quantities  $(U, \pmb{\xi}, \pmb{\theta})$  in our target expected reward
$$
\mathbb {E} _ {P ^ {*}} \left[ Y _ {x} \right] = \sum_ {y \in \Omega_ {Y}} y P ^ {*} \left(Y _ {x} = y\right) = \sum_ {\boldsymbol {v} \backslash \{x \}} y \sum_ {\substack {u = 1, \dots , d _ {U}, \\ U \in \boldsymbol {U}}} \prod_ {V \in \boldsymbol {V} \backslash X} \mathbb {1} \left\{\xi_ {V} ^ {\left(\boldsymbol {p a} _ {V}, \boldsymbol {u} _ {V}\right)} = v \right\} \prod_ {U \in \boldsymbol {U}} \theta_ {u}, \tag{10}
$$
and samples each one iteratively using their conditional distributions, each parameter conditioned on the current values of the remaining terms in the parameter vector and the available data [16]. As mentioned, what data point carries information about which parameters depends on the structural differences between environments.
**Prior.** For every  $V \in V, \forall p a_{V}, u_{V}$ , the functional assignment parameters  $\xi_{V}^{(p a_{V}, u_{V})}$  are drawn uniformly in the discrete domain  $\Omega_{V}$ . For every  $U \in U$ , exogenous probabilities  $\theta_{U}$  with dimension  $d_{U} = \prod_{V \in \mathbf{C}_{U}} |\Omega_{P a(V)}|$  are drawn from a prior Dirichlet distribution (here chosen for conjugacy with the categorical distribution of  $U$ ),  $\theta_{U} = (\theta_{1}, \ldots, \theta_{d_{U}}) \sim \text{Dir}(\alpha_{1}, \ldots, \alpha_{d_{U}})$ , with hyperparameters  $\alpha_{1}, \ldots, \alpha_{d_{U}}$ .
Posterior. In a particular round  $t$ , the Gibbs sampler iterates over the following steps.
1. Sample  $\pmb{u}$ . We start by sampling a corresponding exogenous latent variable for each observed sample  $\pmb{v}^{(n)} \in (\bar{\pmb{v}}, \pmb{v}_{x^{(1)}}, \dots, \pmb{v}_{x^{(t-1)}})$ ,  $n = 1, \dots, t-1+\sum_{i} n_{i}$ . Let  $(\bar{\pmb{u}}, \pmb{u}_{x^{(1)}}, \dots, \pmb{u}_{x^{(t-1)}})$  denote the corresponding set of samples of  $\pmb{U}$ . Exogenous variables  $\pmb{U}^{(n)}$  are mutually independent given  $\pmb{V}^{(n)}, \pmb{\xi}, \pmb{\theta}$  and thus we can sample each separately using the conditional
$$
P \left(\boldsymbol {u} ^ {(n)} \mid \boldsymbol {v} ^ {(n)}, \boldsymbol {\xi}, \boldsymbol {\theta}\right) \propto P \left(\boldsymbol {u} ^ {(n)}, \boldsymbol {v} ^ {(n)} \mid \boldsymbol {\xi}, \boldsymbol {\theta}\right) = \prod_ {V \in V} \mathbb {1} \left\{\xi_ {V} ^ {\left(\boldsymbol {p a} _ {V} ^ {(n)}, \boldsymbol {u} _ {V} ^ {(n)}\right)} = v ^ {(n)} \right\} \prod_ {U \in U} \theta_ {u}. \tag {11}
$$
2. Sample  $\xi$ . Similarly, for fixed  $p a_{V}, u_{V}$ , parameters  $\xi_{V}^{(p a_{V}, u_{V})}$  are mutually independent given  $\bar{v}, v_{x^{(1)}}, \ldots, v_{x^{(t-1)}}, \bar{u}, u_{x^{(1)}}, \ldots, u_{x^{(t-1)}}, \theta$ . As mentioned, each parameter is updated with the subset of  $(\bar{v}, v_{x^{(1)}}, \ldots, v_{x^{(t-1)}}, \bar{u}, u_{x^{(1)}}, \ldots, u_{x^{(t-1)}})$  associated with environments in which the functional assignment of  $V$  is invariant across source and deployment environments. As they represent a mapping between variables, its conditional distribution is given by  $P(\xi_{V}^{(p a_{V}, u_{V})} = v \mid \bar{v}, v_{x^{(1)}}, \ldots, v_{x^{(t-1)}, \bar{u}}, u_{x^{(1)}}, \ldots, u_{x^{(t-1)}}) = 1$  if there exists a (relevant) sample  $(v^{(n)}, p a_{V}^{(n)}, u_{V}^{(n)})$  that fixes the mapping  $p a_{V}^{(n)}, u_{V}^{(n)} \mapsto v^{(n)}$ . Otherwise,  $P(\xi_{V}^{(p a_{V}, u_{V})} = v \mid \bar{v}, v_{x^{(1)}}, \ldots, v_{x^{(t-1)}, \bar{u}}, u_{x^{(1)}}, \ldots, u_{x^{(t-1)}})$  is sampled uniformly from a discrete distribution over  $\Omega_{V}$ .
3. Sample  $\pmb{\theta}$ . Fix  $U \in \mathbf{U}$ . By conjugacy of Dirichlet distributions with the categorical distribution, its conditional distribution given all other quantities is given by a Dirichlet distribution  $\pmb{\theta}_U$
(a)  $\mathcal{G}^{a,*}$
(b) Regret without prior data.
(c) Regret with prior data.
Figure 3: Performance figures related to Experiment 1.
$\bar{\pmb{v}}, \pmb{v}_{x^{(1)}}, \dots, \pmb{v}_{x^{(t-1)}}, \bar{\pmb{u}}, \pmb{u}_{x^{(1)}}, \dots, \pmb{u}_{x^{(t-1)}} \sim Dir(\beta_1, \dots, \beta_{d_U})$  where  $\beta_j := \alpha_j + \sum_n \mathbb{1}\{u^{(n)} = u_j\}$ , and, similarly,  $n$  iterates over the samples  $(\bar{\pmb{u}}, \pmb{u}_{x^{(1)}}, \dots, \pmb{u}_{x^{(t-1)}})$  associated with the subset of environments in which exogenous probabilities match the deployment environment.
This procedure eventually forms a Markov chain with the invariant distribution  $P(\pmb{u}, \pmb{\xi}, \pmb{\theta} \mid \bar{\pmb{v}}, \bar{\pmb{v}}_{x^{(1)}}, \dots, \bar{\pmb{v}}_{x^{(t-1)}})$ . We plug in one of these samples  $\pmb{\xi}, \pmb{\theta} \sim P(\pmb{\xi}, \pmb{\theta} \mid \bar{\pmb{v}}, \bar{\pmb{v}}_{x^{(1)}}, \dots, \bar{\pmb{v}}_{x^{(t-1)}})$  into Eq. (10) for different  $x$  to choose the next action  $x^{(t)}$ . This sample initializes the chain for  $P(\pmb{u}, \pmb{\xi}, \pmb{\theta} \mid \bar{\pmb{v}}, \bar{\pmb{v}}_{x^{(1)}}, \dots, \bar{\pmb{v}}_{x^{(t)}})$  in round  $t$  of Alg. 1.
So far, we have described algorithms, approximations, and regret guarantees that pre-suppose the correct specification of causal and selection diagrams across multiple domains. Some degree of misspecification, however, can be tolerated without voiding guarantees on performance improvements. We discuss more details in Appendix B.1.
We evaluate the proposed approach on several synthetic scenarios inspired by the literature on clinical trials and advertising. We compare Thompson sampling with additional data sources (tTS, Alg. 1) with Thompson sampling with uninformative priors (TS) [38], a KL-UCB [9] algorithm with uninformative priors (UCB), and as a baseline also include the algorithm that chooses actions uniformly at random (Uniform)<sup>9</sup>. For all algorithms, we measure their regrets  $R_{T}$ , averaged over 10 repetitions. Details on all data generating mechanisms and a discussion on mis-specification and limitations of the proposed approach can be found in Appendix D and Appendix B, respectively.
Experiment 1. We start by evaluating the usefulness of prior data by comparing learned distributions of expected reward with and without access to prior data. We consider a bandit problem with action, reward and contextual variables  $X, Y, Z \in \{0,1\}$ , respectively, characterized by Fig. 3a in which 1000 prior data samples are given from an environment  $\pi^a$  that differs in the causal assignment of  $Z$  in comparison with the deployment environment  $\pi^*$ . Specifically, with this model,
$$
P ^ {*} \left(y _ {x}\right) = \sum_ {z, u _ {z}, u _ {x y}} \mathbb {1} \left\{\xi_ {Y} ^ {\left(x, z, u _ {x y}\right)} = y \right\} \mathbb {1} \left\{\xi_ {Z} ^ {\left(u _ {z}\right)} = z \right\} \theta_ {u _ {z}} \theta_ {u _ {x y}}, \tag {12}
$$
where  $(\xi_{Y},\theta_{u_{xy}})$  are invariant across environments while  $(\xi_{Z},\theta_{u_{z}})$  are specific to the deployment environment. We start by considering Fig. 3b that gives samples from  $\mathbb{E}_{P*}[Y_x]\mid \pmb{v}_{x^{(1)}},\dots,\pmb{v}_{x^{(t)}}$  as a function of experimentation rounds  $t$ , that is without making use of prior data  $\bar{\pmb{v}}$ . Distributions of expected rewards under action  $x = 0$  and  $x = 1$  overlap substantially until round  $t = 300$  at which point  $x = 1$  is inferred to lead to higher expected rewards. Fig. 3c gives a similar plot with the exception that the left part of the plot gives prior samples  $\mathbb{E}_{P*}[Y_x]\mid \bar{\pmb{v}}$  illustrating the shape of the expected reward distribution learned from prior data only. In particular, we observe  $\mathbb{E}_{P*}[Y_{x = 1}]\mid \bar{\pmb{v}}$  concentrated in the interval [0.3, 0.9] and  $\mathbb{E}_{P*}[Y_{x = 0}]\mid \bar{\pmb{v}}$  concentrated in the interval [0.1, 0.8]. The
In contrast to related work, causal bandit algorithms are designed for single environments with multiple intervention targets, meta-learning bandit algorithms require experimentation in multiple related environments, while in the setting considered in this paper, data from source environments are given as a batch. Both sets of methods thus focus on a different class of problems. See Appendix A for more details.
agent starts experimentation at round  $t = 0$  with this prior and from then onward expected reward samples are drawn from  $\mathbb{E}_{P*}[Y_x] \mid \bar{v}, v_{x^{(1)}}, \ldots, v_{x^{(t)}}$  as a function of experimentation round  $t$ . The bandit algorithm with prior data is remarkably more efficient, being able to determine  $x = 1$  as the superior action after only 80 rounds of experimentation. Overall, prior data leads the bandit algorithm to pull the optimal arm in  $99\%$  of time versus  $93\%$  of the time without prior data.
Experiment 2. We revisit our introductory example to quantify the benefit of leveraging historical patient data from various hospitals. In this example, the objective is to infer the optimal level of hypertension medication  $X$ . We are given a choice among 5 different levels, i.e.  $|\Omega_X| = 5$ , and wish to increase the probability of the presence of a beneficial biomarker  $Y$ , i.e.  $|\Omega_Y| = 2$ , in the clinical trial population  $\pi^*$  given that we have prior observational data in a different hospital  $\pi^a$ . The selection diagram describing this causal protocol is given in Fig. 1. Regret comparisons for all algorithms are given in Fig. 4 (LHS). We observe a significant gain in performance by tTS that chooses the optimal intervention in  $67\%$  of the rounds in contrast with  $35\%$  of the rounds for TS, and  $28\%$  of the rounds for UCB (and  $17\%$  for an algorithm choosing interventions at random).
We use this example also to illustrate empirically the dependence between regret and prior entropy shown in Prop. 1 (RHS). For this, we consider different prior beta distributions for  $P^{*}(W = 1 \mid x)$ , specifically with increasing standard deviations around the true value of  $P^{*}(W = 1 \mid x)$  for each  $x \in \Omega_{X}$ . A larger standard deviations implies a less informative prior and higher entropy of the random variable  $\tilde{X}$  that denotes the optimal action. The
Figure 4: Performance figures related to Experiment 2.
entropy of  $\tilde{X}$  takes values in the interval [0.2, 1.4] for assumed Beta priors for  $P^{*}(W = 1\mid x)$  with standard deviations in the interval [0.001, 0.1]. Fig. 4 (RHS) demonstrates empirically the influence of the entropy of  $\tilde{X}$  on the expected cumulative regret given in Prop. 1. In particular, narrower, more informative priors lead to better regret.
Experiment 3. This example considers an advertiser seeking to optimize which ads to show visitors on a particular website. For each visitor, we choose one out of a collection of 6 ads  $X$ ,  $|\Omega_X| = 6$ , some of which will be more engaging than others, to ultimately optimize whether a user clicks  $Y \in \{0,1\}$ . Each ad has some theoretical but unknown click-through-rate  $P^*(y_x)$ .
In this example, we assume access to 500 data points from an ad recommendation system used on a different website, i.e. a different environment  $\pi^a$ . There, the effect of an add  $X$  on the number of clicks  $Y$  is confounded by the user's age  $A$ , (here categorized into old and young such that  $\Omega_A = \{0,1\}$  and  $|\Omega_A| = 2$ ) and the user's product preferences level  $W$ , which interacts with current ad-recommendation system through a user's
Figure 5: Performance figures related to Experiment 3.
browsing history  $Z$  and location (not observed and therefore represented with a bi-directed edge between  $W$  and  $X$ ). Moreover, in this example, the relationship between  $W$  and  $Y$  is itself confounded by unobserved factors. The population visiting the website of interest  $\pi^{*}$ , where the MAB will be deployed, is known to agree on all causal components with  $\pi^{a}$  except on the distribution of age  $A$ . This causal protocol as well as regret comparisons for this example are shown in Fig. 5. We observe noticeable improvements in regret with prior data and knowledge of structural differences as tTS substantially improves over algorithms agnostic of prior data.
This paper investigated the problem of improving the efficiency of multi-armed bandits using batch data from related environments. As source environments might differ, some knowledge of structure and (potential) discrepancies are necessary to extrapolate consistently. This paper demonstrated that knowledge of selection diagrams that encode causal influence as well as potential discrepancies
across source and target environments, without, however, an explicit specification of functional form and distributions, is sufficient to consistently define an informative prior over reward distributions using data from arbitrary environments. The resulting algorithm guarantees improvements in regret in comparison to algorithms agnostic of prior data. To our knowledge, this serves as one of the first principled approaches to consistently leverage prior data in the context of bandits and we hope it can pave the way for developing more general transfer learning methods in reinforcement learning.
[1] Yasin Abbasi-Yadkori, David Pal, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011.
[2] Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In International Conference on Machine Learning, pages 127-135. PMLR, 2013.
[3] Elias Bareinboim and Judea Pearl. Causal inference and the data-fusion problem. Proceedings of the National Academy of Sciences, 113(27):7345-7352, 2016.
[4] Soumya Basu, Branislav Kveton, Manzil Zaheer, and Csaba Szepesvári. No regrets for learning the prior in bandits. Advances in Neural Information Processing Systems, 34:28029-28041, 2021.
[5] Alexis Bellot and Elias Bareinboim. Partial transportability for domain generalization. Technical Report R-88, Causal AI Lab, Columbia University, 2022.
[6] Alexis Bellot, Junzhe Zhang, and Elias Bareinboim. Scores for learning discrete causal graphs with unobserved confounders. Technical Report R-83, Causal AI Lab, Columbia University, 2022.
[7] Blair Bilodeau, Linbo Wang, and Daniel M Roy. Adaptively exploiting d-separators with causal bandits. arXiv preprint arXiv:2202.05100, 2022.
[8] Léon Bottou, Jonas Peters, Joaquin Quinonero-Candela, Denis X Charles, D Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research, 14(11), 2013.
[9] Olivier Cappé, Aurélien Garivier, Odalric-Embrym Maillard, Rémi Munos, and Gilles Stoltz. Kullback-Leibler upper confidence bounds for optimal sequential allocation. The Annals of Statistics, pages 1516–1541, 2013.
[10] Leonardo Cella and Massimiliano Pontil. Multi-task and meta-learning with sparse linear bandits. In Uncertainty in Artificial Intelligence, pages 1692-1702. PMLR, 2021.
[11] Juan Correa and Elias Baireinboim. General transportability of soft interventions: Completeness results. Advances in Neural Information Processing Systems, 33:10902-10912, 2020.
[12] Arnoud AWM de Kroon, Danielle Belgrave, and Joris M Mooij. Causal discovery for causal bandits utilizing separating sets. arXiv preprint arXiv:2009.07916, 2020.
[13] Aniket Anand Deshmukh, Urun Dogan, and Clay Scott. Multi-task learning for contextual bandits. Advances in neural information processing systems, 30, 2017.
[14] Alun Evans, Veikko Salomaa, Sangita Kulathinal, Kjell Asplund, François Cambien, Marco Ferrario, Markus Perola, Leena Peltonen, Denis Shields, Hugh Tunstall-Pedoe, et al. Morgam (an international pooling of cardiovascular cohorts). International journal of epidemiology, 34(1):21-27, 2005.
[15] Sarah Filippi, Olivier Cappe, Aurélien Garivier, and Csaba Szepesvári. Parametric bandits: The generalized linear case. Advances in Neural Information Processing Systems, 23, 2010.
[16] Alan E Gelfand. Gibbs sampling. Journal of the American statistical Association, 95(452):1300-1304, 2000.